Could it be that (s)he represents something that you are jealous of, just in an exaggerated form? Maybe you're just now learning you don't hate that type of person at all, but that you can learn a lot about yourself by learning from this person.
For example, if you're shy, a very outspoken person will likely make your skin itch, but that person may also teach you to be more open. It's just that this person is exaggerating it a little. (And vice versa: a shy person could teach somebody who always talks to be quiet for once.)

Or maybe that person represents a personality (or social class? job? hair color?) that is "not done" within your culture/family/etc, which makes you despise that person more or less "automatically"? I can imagine that certain families automatically dislike "low-brow" culture, and thus despise people who like soap operas, cheap musicals and drag car racing. Or if you're from a family of readers, that a very loud, talkative person who always has music on gives you the creeps (my wife never has music on and immediately turns the radio off when I leave the room).

For this thing, focus on the despise part, I'd say. Then see if you can discard it and, if so: pursue!