When it was first occuring we thought maybe I somehow got on the whole
fort's hate list (possibly another group's mob wandered into my AOE then
killed them and walked into the fort, tell them about this little punk half
elf outside)

So I zoned (twice actually, as I had to run right after getting back). No
change, as soon as I came up to the group, Giants just starting charging out
of the fort...

I am not going to sue charm and see if it still happenes tonight.