Do please explain your logic there - that is a typical internet riposte, 'you said brown people so you must be racist', really????
He’s right. Calling someone ‘brown’ is considered a racist term. Don’t ask me why, it just is. I’d stop using it immediately if I were you.