Not all racial incidents are racist incidents, Lawrence Blum says. We need a more varied and nuanced moral vocabulary for talking about the arena of race. We should not be faced with a choice of 'racism' or nothing. Use of the word racism is pervasive: An article about the NAACP's criticism of television networks for casting too few minority actors in lead roles asks, Is television a racist institution? A white girl in Virginia says it is racist for her African-American teacher to wear African attire.

Blum argues that a growing tendency to castigate as racism everything that goes wrong in the racial domain reduces the term's power to evoke moral outrage. In I'm Not a Racist, But..., Blum develops a historically grounded account of racism as the deeply morally charged notion it has become. He addresses the question whether people of color can be racist, defines types of racism, and identifies debased and inappropriate usages of the term. Though racial insensitivity, racial anxiety, racial ignorance and racial injustice are, in his view, not racism, they are racial ills that should elicit moral concern.

Blum argues that race itself, even when not serving distinct racial malfeasance, is a morally destructive idea, implying moral distance and unequal worth. History and genetic science reveal both the avoidability and the falsity of the idea of race. Blum argues that we can give up the idea of race, but must recognize that racial groups' historical and social experience has been shaped by having been treated as if they were races.