The problem is that on a large scale, such things can still cause problems. If you take a nation like france, which already has its own cultural and ethnic character and has had it for centuries, and fill it full of large amounts of people who have a different cultural and ethnic character, and these people also have a higher birth rate and are not pressed to assimilate hardly at all because it's considered wrong for the natives to expect such a thing, it will dilute and undermine the original character of the nation. It will either become a rather generic multicultural nation or the newcomers will eventually replace the natives if the trends continue on unstopped.

And there is no going back once this happens. How could you "Re-Frenchify" France if it became predomiantely Muslim? You couldn't do so without ethnic cleansing or other crimes against humanity which one one wants to have happen. So France would either become Muslim or a generic multicultural mini-USA. it would be France in name only, if that. Meanwhile, the immigrants' culture would still exist in its original homeland, as well as in "France," but there would be no more french culture.

This would ultimately result in less cultural diversity, not more.