The World Color Survey was a large-scale cross-cultural experiment in which informants used the color lexicons of 110 non-written languages to label a standard set of stimuli. Here those data are explored with a novel analysis which focuses on the averaged location of boundaries within the stimulus set, revealing the system of color categories native to each language. A quantitative index of inter-language similarity was defined, comparing these average boundaries. Analyzing the similarities among color-naming patterns led to a 'language space', in which languages are grouped into clusters according to linguistic families (i.e., descent from common ancestors). This implies that each language's departures from the cross-cultural consensus about color categories are systematic (non-random). Given the non-unanimity about the color lexicon within languages, the persistence of these language families across the course of linguistic evolution is paradoxical.