The spectres normally stay on the island. If they were
sweeping the coast, it's because some higher level players
foolishly/inconsiderately tried to attack them, and instead of
gracefully dying when they realized they were outmatched, ran
for the zone with spectres in tow, thus unleashing them on
everyone else.