The connection I picked up on was in the story of the little crannogman, he asked her to dance on Ned's behalf. There was Barristan Selmy's recollection about "Stark" dishonoring Ashara, but that could have been Brandon or Ned, and dishonor could be anything from a one-night stand to sexual assault. It could be his rather sexually-sheltered perception of a consensual relationship (a celibate knight might have a radically different perception of a quick fling than a woman from Dorne with its much more relaxed sexual mores) or he could be repeating the gossip that Cersei threw in Ned's face about his begetting Jon Snow on her and then causing her despair by taking the baby. The timing of her suicide argues against the rape, IMO, and the fact that Jon was nursed by Wylla, a retainer of House Dayne would seem to suggest that the two families maintained affection and respect up to the end.

On the other hand, if Brandon had raped Ashara, a close companion of Aerys' daughter-in-law, that might have been one of those "hey look, there's another side to the story" things that Martin seems to love doing, regarding the Mad King's treatment of Brandon.

Without the rape, I don't see the attitude Wert attributes to Brandon, and without some other indicators I am missing, I don't see there being enough evidence for a rape or even much arguing for a relationship between Brandon & Ashara. The Stark sibs seem pretty close, and I don't see one of them cock-blocking the others. At the least, Ned seems to have cared for Ashara, what with the hints of their relationship and the courtesy call he made regarding her brother's death and his concern for her reputation, and I can't think he'd really remember Brandon that fondly if he had, in fact, done something like that to her.

What do you think of a rumor I read somewhere that she is actually Lemore, the septa companion of Connington & Aegon VI?