Admitting students to colleges (even partly) based on athletic ability or legacy hurts (or at least can’t help) other metrics of quality in the admitted student body. Hiring workers (even partly) based on race hurts other metrics of worker quality. Using any of a number of simple, good models, it’s obvious that introducing an extra criterion with any weight into your selection process will always either do (nearly) nothing, or diminish the performance of the selected candidates on other metrics. The students with the highest (.2(athlete) + .8(SAT)) will almost always have lower SATs than the students with the highest SATs.
Soft-headed types often try to minimize or deny this point in discussions of (e.g.) affirmative action in admissions, as though saying (probably correctly) that diversity can strongly enhance the value of the college experience means that we pay no price for using affirmative action to foster diversity. (The other (and currently disfavored?) argument for affirmative action is that it compensates its beneficiaries for past (or present) disadvantages that stem from their ethnic group membership.)
Apart from all that, though, there are some other (fairly plausible) scenarios in which an affirmative-action-type rule could in fact improve the quality of the admitted pool (probably all of these have to do with measuring problems in the candidate pool). For example, if the SAT is racially biased so that SAT = intelligence – .2*black, then maximizing .2*black + SAT will yield a smarter pool of admitted students than would maximizing SAT.