Genre is a librarian's tool, in my opinion, made to easily find thematically similar works. Sadly, both authors and critics take genre so seriously that they have the rest of us convinced.

In any case, what you're asking would require a universal definition of fantasy, which is something that hasn't yet been done, and probably won't be. You can't say, for example, that books that contain impossible situations, actions, characters, or creatures is fantasy, because there are so many books containing one or all of these that are considered Literature.

But you'll find plenty of people who disagree