It's the latency that's the issue. Bandwidth limits how detailed your graphics can be. I think OnLive requires around 5mbps for their pseudo-720p graphics, and 2ish for their SD stuff. Latency, on the other hand, limits the in-game lag. The time it takes for your character to respond to your input. Under absolutely ideal network conditions it should be hard to notice the lag, but in more common circumstances, it's obviously there. The few times I've tried it, I found certain games to be very playable, if a little less fun than playing it on a local machine. Action games were good, for the most part (Just Cause 2 and Darksiders both worked well) but shooters and other games requiring precision were frustrating.

But yeah, you don't have to have a super high bandwidth connection to enjoy the service, you just need low latency between yourself and an OnLive server location.