What tipped the scales for me is that Ulfric is a huge jerk, as are the people of his inner circle. The Imperial leadership is much more reasonable and even, in some cases, sympathetic with the other side. I don't regret choosing the Imperial side of things at all.

Of course, as jh said, there are essentially zero consequences for choosing a side. I haven't even heard it mentioned again. *shrug*