The first and most obvious problem here is that Islam is not a race; its more usually categorized as a religion. Second problem is that there are people of just about every race you can imagine that consider themselves Muslims. That means ... the label of "racist" is simply wrong.
Yep, that's definitely the best way to respond to an accusation of being racist - argue that you're not racist, because your irrational hatred crosses racial boundaries.
Well done, sir.
(That said, he's - technically - right, on this point at least - but even such an inveterate foot-in-mouther nows that arrant pedantry is not usually an effective way to get people on your side).