I don't think they want to keep him away from Hogwarts as a way of caring for him. I think it's all about themselves. What if people found out that the boy who has lived with them for 11 years was one of Those Weird People? It's (strangely) better for people to think that he's just a wild, disturbed child than someone who is part of a world so outside their understanding. And I think there's some malice in it as well: trying to prevent Harry from doing something he enjoys.

Petunia's a bit more complex than that. I think there is some love for Harry deep (deep) down. (Of course, we do see a glimpse of that later in the series.) But any love she does have for him is only because he's her sister's child.