That's because charm works like Lullaby, nix's the aggression list of the
monster, with the only person remaining on the list being you as the caster of
the effect.

Just wait until you are level 40 though, and the fun with giants really begins.
Every other class I know except druids whines they can't solo generally, and I
do so often. 2 giants, a giant + griffin, etc are extremely easy to control.
However, I do not use selo's, because if you get bashed you are at the giant's
mercy. For soloing I use Journeyman Boots. I have also managed to kill a few
giants solo using chains and fear, but it's a much harder battle, and I have to
break contact and heal with lute at least twice to kill it entirely (note what
weapons you use play a large part in what you can realistically solo).