Brandon is a hothead, which we knew from previous books. He's also not a nice person, as we got from Barbry's description of him in ADWD. We know from ASoS that Brandon went to talk to Ashara 'on Ned's behalf' during the Harrenhal tourney during the Year of False Spring. We know he went after what he wanted and he was only marrying Cat because of his father's wishes, not because of his own desire.

Selmy is also vehement in his anger with "Stark" who dishonored Ashara, got a daughter on her (so, definitely not Jon Snow's mother) and she killed herself as a result. He is 100% not talking about Ned. Barristan and Ned got on very well in AGoT and even in ADWD itself he stands up for Ned to Daenerys, telling her he stood vigil for Ned in the Great Sept of Baelor even though he was on the run himself. The chances of him doing that for someone he thought was an honourless person who'd effectively killed an innocent woman are non-existent.