I don't think it's nonsense to be bothered about the notion of people being accused or even convicted of rape when they were understandably convinced, based on reasonable grounds, that it was consensual. Of course, "reasonable grounds" are impossible to define, but then that's why it's a grey area. Obviously in general it's about consent, not refusal, but in cases where there are "reasonable grounds" to lead one to conclude that it's consensual, it's important for the non-consenting partner to make that refusal clear. And having previously had consensual sex does change one's frame of reference, changes how explicit a consent you'd need to reasonably suppose that sex is welcome.