During binocular rivalry, the percept alternates between dissimilar images presented dichoptically. A central question in studies on binocular rivalry is whether its dynamics are affected by attention. Here we approach this question by studying how set size affects the number of alternations reported from a rivalry-inducing display. The results show that the number of alternations reported increased only slightly with increasing set size. In follow-up experiments we show that neither (1) limitations in response speed nor (2) grouping cues are responsible for this result, and that (3) increasing set size decreases the number of alternations reported per rivalry-inducing element. Our results show that the spatial distribution of attention affects the number of alternations reported in a rivalry-inducing display.