I think I agree with you, on the whole, but the incident with Sheriam raises questions about that. She's panicked, defending herself against allegations of being Black, and answers "Of course not" to Egwene's question whether she has red hair before even realizing what the question is. She doesn't have any intent to deny that particular question, thinks she's denying something else instead, so you would think it should've been possible to say that even with the First Oath. But no, Egwene and the Hall take it as very strong evidence that she's Black, as if a sister bound to the Oaths would've been unable to give that answer under those circumstances.