Little backwater servers can turn into metropolises very fast. Trust me.
Turalyon was a low/medium population three month old server and then two highly progressed Naxx guilds transfered over. Now, I'd say more than half of the level 60s on the server are transfers. I wouldn't count on being the only guild transferring to a new server.