The considerable agreement across languages in the way they categorize the color domain, despite independent historical development, demands an explanation. One option is to postulate a universal innate representation of the color categories, 'hardwired' into each observer's brain. An alternative is that observers internalize their color categories through a process of cultural (linguistic) transmission, constrained by some kind of 'optimality hypothesis' to account for the cross-language agreement. A number of optimality hypotheses are reviewed. It is tempting to believe that the vivid experiential quality of the categories can only be explained if they are determined by innate representations rather than by linguistic imprinting. However, linguistic transmission of color categories – perturbed from their optimal boundaries by special circumstances – fits best with the experience of dichromats. Even for the 'primary hue' categories, where the case for innate representations should be strongest, the evidence is far from convincing.