Remember how Ituralde used the tactic of hiding soldiers as farmers in TGS Prologue? Maybe the Seanchan cottoned on. 10,000 is a small army compared to what is there, but all you need is 13 channelers to kidnap Rand.

The thing that cements it for me is that there is no way that farmers would know about the meeting at Merrilor. Most world leaders had to mobilize very quickly to get there, and came in groups via Gateway. However, the "farmers" having been coming in in small groups. Not only does this avoid attracting attention, but it begs the question: If they are all coming in from different directions, in different groups, why not join up with one of the existing bands? For example, Perrin has a bunch of refugees with him, the farmers would fit right in.