According to Mitchell Green, speech act theory traditionally idealizes away from crucial aspects of conversational contexts, including those in which the speaker’s social position affects the possibility of her performing certain speech acts. In recent times, asymmetries in communicative situations have become a lively object of study for linguists, philosophers of language and moral philosophers: several scholars view hate speech itself in terms of speech acts, namely acts of subordination (acts establishing or reinforcing unfair hierarchies). The aim of this paper is to address one of the main objections to accounts of hate speech in terms of illocutionary speech acts, that is the Authority Problem. While the social role of the speaker is the focus of several approaches (Langton 2018a, 2018b; Maitra 2012; Kukla 2014; Green 2014, 2017a, 2017b), the social role of the audience has too often been neglected. The author will show that not only must the speaker have a certain kind of standing or social position in order to perform speech acts of subordination, but also the audience must typically have a certain kind of standing or social position in order to either license or object to the speaker’s authority, and her acts of subordination.