.... and scores of thousands able men among the refugees, with blacksmiths able to make weapons, and men able to make bows etc... And they will have to - there's just no way Perrin can feed so many people. They'll have to gather plants and hunt on the way north.

By the time he meets with Galad, there's really no telling how huge Perrin's forces will really be. Perrin intended to send the refugees away, but he was told back the Asha'man can't handle this for a while and many refugees just don't want to return home and wish to join Perrin.

It's also about the right time for Perrin to start getting his army of wolves for TG.... IMO, this is the real reason why he ends up staying in the wilderness so long: the wolves have started gathering to him. He needed to stop in one place for a while, and this place needs to be away from civilization.

In any case, it's all rhetorical. Asunawa is a long way south, the Questionners are no match for the regular WC troops and there's no way the Seanchan would trust Asunawa to hunt Galad, and there's no way Galad would fight Perrin with 100,000 refugees. Valda or Carridin may not have frown on that, but with Galad in charge that's just unthinkable.

Beside, we saw in TGS that he has joined Perrin's camp.