Sometime in my mid 20's I think I started developing allergies, and when I came home on leave I found out I'd become allergic to cats, and just about every local piece flora, and I've had cats my whole life and have a small herd now, but its not too bad, I really only flare up bad when spring thaw hits... so right now I think it comes down to how bad your allergies are.

Anyway, if it is cats, you can take some steps to massively reduce effect, don't let them sleep on the bed, keep them out of the bedroom entirely, wash them more frequently, vacuum more, never touch your eyes after petting them. You also want to consider getting a HEPA filter, steam cleaning carpets with some frequency, and make sure sure to wash bedding with hot water that's at least 140 F or 60 C. Along with allergy medicine for when its bad, this will probably limit cat based allergies, and if you don't get a drop off then its probably not cats.