The Centurians in the Town Hall Quests are a notable step up in difficulty than the Decurions -- they do more damage and have a lot higher armor. If you are using 1-handed weapons (esp short swords or daggers) you might struggle to do any damage.
Suggestions: lots of potions (might, healing, shield), Grissenda as a Companion, Summon (wolf or elemental) and a mercenary (might be hard to get a mercenary into Blue Rock Castle alive).
Also, when the Centurain appears in Blue Rock Castle, it seems nearly impossible to divide the soldiers apart.Once one gets triggered, they attack in mass.
If they spawn near the dire wolf chamber, you can free the dire wolves to attack the Centurain but the wolves die pretty quickly.