In this case, if you were to use that as a component, you've retained your peremptory challenges, and then it would be up to the other side to say, “Hey, I think you're using them for a discriminatory reason”, and that would possibly in itself limit the number of times that anybody would ever even contemplate doing that, because they could be called out on it.
Would that be a preferable solution to you than stripping them from your repertoire of things that you can do?