I am the anti-sayer. I mean, I'll say it light-heartedly and with my friends, etc, but it's not something easily said when I'm being serious.

I think that all you can do at this point is to tell her you do really care for her, and you wouldn't want to lose what you two are building up, but that you're not the type of person who says this without deeper thought and emotion. You have to be careful how you say it, because she could think you're saying that she is not feeling deeper thought and emotion if she said it so soon. But you should definitely stick to your guns on this.