I've coloured the map in question for your convenience. As you can see, there is no way of getting from Green to Blue without going through Red, aside from that tiny border at the far left, which is the Two Rivers. "The road to the east" that you pointed out heads straight into the Red area.

As for that area being desolate, that is currently true, but largely due to the powerlessness of the monarchy in Altara. The Seanchan, on the other hand, are not powerless, and you can expect them not to ignore intrusion within their borders once they have fully secured the province.