Traveling was more likely to kill off those who knew it during the Breaking! Most of them probably ended up in exactly the sorts of circumstances you described, if they did not get lost between the Pattern when they tried to create a relationship between their starting point and a place that no longer existed.

But every few months, someone decides they know better, despite the explicit statements that no sister survived, and everyone goes around telling RJ that the world he created could not possibly work the way he says it does.