Castles in the GoT World Map

Hi, I'm not able to find any information covering this, but how does one go about defeating castles in the main Game of Thrones map? It's been a while since I played (mid/late 2022), and they seem to be a lot stronger than they used to be. I'm sending stacks of 500 or so random sword/horse/spears at them with almost no one manning them at 170ish defense, and everyone I send is dying.

Send elites