I could easily see this being a cause for racism. After all, if you get used to the idea that everyone around you looks the same, anything that breaks that mental model could more easily be seen as a threat. After all, if they looks so different, who knows what strange habits they might have? Take the Tinkers for example, the lifestyle and clothing style set them apart, and then that takes even villainous undertones (thievery, stealing children).