It's important to note that the largest group of Justices aren't (directly, at least) judging the merits of affirmative action. They're ruling on whether states have the right to take an action, regardless of that action's wisdom. Kennedy outright wrote that the case "is not about the constitutionality, or the merits, of race-conscious admissions policies in higher education."

In any case, I don't think your statement "If society should be color-blind, let it be color-blind" is meaningful. If, for the sake of argument, there still exist wide-spread and significant disparities based on race, ignoring these disparities would not do as much to correct the situation as methods developed to fix them.