Attention has been conceptualized as a filter mechanism that suffices the fundamental capacity limitation of sensory processing by selecting the input of most behavioral relevance. Recent research has demonstrated that information can be selected by attention based on the sensory modality it is presented in. However, it is not yet known whether this ‘intermodal’ selection relies on common attentional resources. Alternatively, each sensory modality might draw on an independent pool of resources. The present work investigates the neural mechanisms of sustained intermodal attention by pitting the notions of common vs. modality-specific attentional resources against each other. To this end, concurrently presented, frequency-tagged auditory and visual stimuli elicited continuous electrophysiological brain responses in respective early sensory cortices. Three experiments probed (1) whether attention to a particular sensory modality results in a modality-specific modulation of processing, (2) whether this modulation of processing is a facilitation of the attended, an inhibition of the unattended sensory modality or a combination of both factors, and (3) whether stimuli of different sensory modalities enter a competition for processing, thus necessitating intermodal attention to rely on common attentional resources. Attentional modulation of stimulus processing was found to be modality-specific. This modulation likely involved two separate mechanisms: a facilitation of stimuli presented to attended sensory modalities that led to an inhibition of stimuli presented to unattended modalities. As a complementary result, stimuli were found to enter a competition for processing within but not between modalities. In conclusion, the present findings provide evidence for early sensory processing to rely on modality-specific rather than common attentional resources.