I disagree. Using "labels" is a way of understanding a social phenomenon.

You wouldn't make that argument about sexists, rapists, or pedophiles. You simply don't understand the gravity of racism and thus don't see the purpose of understanding the phenonemon.