A restaurant serves the public, it is inherently a public building. While it is privately owned, it depends on the public. Private clubs, however, which often do serve meals and function in much the way restaurants function, are allowed to exclude whomever they want because they are (say it with me now) PRIVATE. Private colleges are allowed to accept whomever they want, based on whatever criteria they want. Again, because they are PRIVATE. There are private schools that only accept women, men, Mormons, Catholics, pentecostals, and anglicans.

The workplace is an interesting exception to this trend, and one I don't know enough about to argue.

Any way you slice it, in the vast majority of examples, private institutions are allowed to do whatever they want so long as it doesn't infringe on the rights of anyone else.