I'm with you.

I'm fairly averse to using it as a noun except for animals in English too, though I know that's all to do with how you use and don't use femelle (anymore) in French.

It would be extremely derogatory to use the word a noun for a human woman, except in a medical contex or as a short cut for "genre féminin". Another acceptable use would be for humour (non offensive gender humour, for e.g....)

I guess it has a lot to do with culture. Femelle for French speakers reduces the gender to its reproductive function and was mostly used in other contexts by mysogynists or men wishing to dismiss or attack the gender. Since feminism, the word is barely ever used except for animals and a proper scientific term (totally non offensive when used this way). A few decades ago however, a priest could for example (and if you go back far enough, often would) refer to Eve as a "femelle" (while referring to Adam as a Man), but nowadays a priest doing that would really ruffle his flock.

Anyway, culturally I still find it really weird and odd to use female as a noun for women in English when the context isn't scientific, and when others use it sounds derogative and rude (even aggressive), whatever the real intent. Based on the argument, it sounds like with English speakers the word has gained the same derogative stigma for some women, but it's not as widely established culturally/linguistically as it is in French. In French even the adjective could be derogative out of the scientific context, and féminin is much preferred for all the others.

Typically, mâle has none of those connotations. Using it over homme (man) for a human male is often even flattering. French can be incredibly sexist in some of its usages.