This sounds like a rare map loading issue.
Map loading issues are often easily resolved by reloading the area. Reloading an area can be done by going to a different area, quick-save and reload; or, by staying at an Inn. A 12-hour rest at an Inn resets all game maps.
Talk to prisoners to find out about gate. Interact with loose brick. Open Gate.