I don't know you so I can't say that you are racist or ever behaved racistly (not enough context). But, I will say that working with Black students doesn't automatically mean someone is not racist. Consider a prison warden claiming he's not racist because the jail he works at has 70% + Black inmates. Working in an institution with majority Black people doesn't mean you can't be racist, it just means you don't have a problem being around Black people. However, it's how you treat the Black people you are around that determines if you are racist or not. For instance, I had many White teachers in public school, but I never thought of any of them as racist, or heard my peers refer to them this way. That's because they never treated us as inferior to them in any way. The curriculum was white-centered, but its because they were following orders.

Not sure why some students called you racist and depending on their ages and circumstances, they may have been just name-calling (that's very possible). However, I wouldn't close your mind entirely to what the students are saying. Instead of saying, "I can't be racist because..." try asking, "What did I say or do to make you guys say that about me?"

If they can't answer that question, it's save to ignore.