Phylogenetic Inference from Word Lists Using Weighted Alignment with Empirically Determined Weights

3.1. The Evidence for Relatedness

As spelled out in the previous section, the LDND score aggregates distances between words to distances between languages (i.e.: word lists over a given concept list)

and by

• computing the distances between all word pairs from

  

  

  View Expanded

  
  and
  

  

  View Expanded

  
  ,

• computing the average distance between synonymous and the average distance between non-synonymous words, and

• dividing the former by the latter.

In this section I will propose an alternative method for aggregating a matrix of distances between words from

and to an overall distance measure between and .

To illustrate the underlying intuition, consider again the matrix of LDN scores between English and Swedish words illustrated in Table 1. The distribution of off-diagonal scores is shown in Fig. 2.

The word pair fiS/fisK ‘fish’ has an LDN score of

, marked by the dashed line in Fig. 2. Only 4 off-diagonal entries have a lower score, and 31 entries have the same score. This means that a randomly picked pair of non-synonymous words has only a chance of about to be more similar to each other than fiS/fisk. Intuitively, the fact that this fraction is so small provides evidence that fiS and fisk—and therefore English and Swedish—are related. Likewise, each of the other diagonal-entries provide a certain amount of evidence for the languages to be related, depending on their position within the distribution of off-diagonal entries.

To make this precise, let us assume that

and are the word lists from the languages to be compared. The entry of is denoted by , and likewise for and . It is assumed that and are pairwise synonymous for all . is the LDN distance between and .

The ASJP data contain missing entries at many positions. To deal with this issue, we assume that there are

concepts for which both and contain entries. If does not contain an entry for concept , and are undefined, and likewise for .

The rank of a diagonal entry

—written as —is the position that would assume if it is added to the set of off-diagonal entries and the resulting set is sorted in increasing order. Formally, we have

(It is tacitly assumed that only those pairs

are counted for which is defined.)

For the time being we assume that there are no ties; this issue will be taken up later on.

As the sizes of word lists may differ between languages, we normalize the rank by dividing it by the maximal possible rank, which is the number of off-diagonal entries

. This leads to the definition of the normalized rank , which always assumes a value in the interval :

If the languages in question are unrelated, the entries along the diagonal are drawn from the same distribution as the off-diagonal entries. Therefore we expect each rank (between

and ) to be equally likely. However, if the languages are related, we expect some diagonal entries to be small in comparison to the off-diagonal entries, i.e. we expect their ranks to be small.

This is illustrated in Fig. 3. The left panel shows the distribution of diagonal entries (left boxplot) and off-diagonal entries (right boxplot) for the comparison of English and Swedish.

It is clearly visible that the diagonal scores are on average much lower than the off-diagonal scores.

The right panel shows the same data for the comparison of English with Swahili. The two languages are unrelated, and the diagonal entries are similiarly distributed as the off-diagonal entries.

For a pair of unrelated languages we expect the normalized ranks for diagonal entries to be uniformly distributed between

and . If the languages are related, this distribution should be skewed towards small values. To test this, I drew 100,000 ranks from a random selection of ASJP language pairs that belong to the same genus according to WALS (Haspelmath et al., 2008), and another sample of 100,000 ranks from language pairs that each represent different WALS families. The histograms of the two distributions are shown in Fig. 4.

As expected, the normalized ranks for pairs of related languages are heavily skewed towards small values, while the values for unrelated languages approximately follow a uniform distribution.²

Figure 5 displays the same data as histograms with logarithmic binning in log-log plot.

The values for the related languages lie approximately on a straight line with a negative slope. This indicates that the normalized ranks are distributed according to a power law (see, for instance, Clauset et al., 2009 on power law distributions in empirical data). This means that there are real numbers

and such that

We can thus approximate the empirical distribution by a continuous probability density function

with

The distribution of normalized ranks for unrelated languages can be approximated by a constant density function:

Suppose we have to decide whether or not two languages are related on the basis of normalized ranks of all translation pairs. So we compare two hypotheses:

(languages are unrelated) and (languages are related). According to Bayes’ formula, the posterior odds of the likelihood of the hypotheses are

If we make the simplifying assumption that the normalized ranks for the individual translation pairs are stochastically independent, this amounts to

The posterior log odds thus come out as

While we do not know the prior probabilities

and , we can determine the term empirically. The posterior log odds are a monotonically increasing linear function of this quantity.

However, this only holds for a constant

. Recall that is the number of concepts for which both word lists to be compared contain an entry. Due to missing data in ASJP, may assume different values for different language pairs. The maximal number for is . If , we have to estimate the normalized ranks for the missing entry pairs. The maximum likelihood estimation is that the average value of the missing -values equals the average of the known values. Therefore the quantity provides the maximum likelihood estimator. As is constant, the estimated posterior log odds are a monotonically increasing function of the quantity

Let us call this quantity the Evidence for Relatedness (ER).

To turn this into an operational definition, one further amendment needs to be made.

Recall that there may be ties, i.e. pairs

with . To put it another way, suppose we form the set of distances and sort it in increasing order. The quantity in the numerator of Eq. (1) is the number of items preceding is this sequence; adding 1 yields the rank of . If there are ties, the rank may not be uniquely defined. In this case we compute the evidence for relatedness for the concept for each possible rank and form the average. The possible ranks in the definition below are the set of ranks that can assume in such a sequence. The normalized rank is the geometric mean of all possible ranks, divided by the number of off-diagonal entries . Forming the geometric rather than the arithmetic mean ensures that the logarithm of the normalized rank equals the arithmetic mean of the logarithms of the possible values of .

This leads to the following final definition:

Definition 1 (Evidence for Relatedness)

View Expanded

It is reasonable to assume that the Evidence for Relatedness becomes stronger the closer two languages are related. ER can thus be considered a similarity measure between languages. It can easily be transformed into a distance measure. ER is maximized if we compare a word list to itself: it contains no homonymies and no missing entries. In this case, all

. The number of off-diagonal entries equals , so for all , . Hence the ER score is

The theoretical minimum for the ER score is achieved if all diagonal entries are smaller than all off-diagonal entries. In this scenario all

and hence . The Distance based on Evidence of Relatedness (dER) is then defined as follows:

Definition 2 (Distance based on Evidence of Relatedness)

View Expanded

The dER score always assumes a value between

and . Note that it does not depend on the values of and , so no parameter fitting is necessary.

3.2. Correcting for Missing Entries

If the word lists to be compared contain missing entries, the dER measure relies on a maximum likelihood estimate of the

scores of the missing entries. As a consequence, the absolute value both of positive and of negative evidence is overestimated in this case. While it seems unproblematic to underestimate the similarity between two languages if the available, incomplete word lists do not provide evidence for relatedness, the error in the opposite direction is potentially more serious. In the case of incomplete word lists, chance similarities receive a higher weight than is actually justified. To correct this, high ER scores should be discounted somewhat in proportion to the amount of missing entries.

Suppose the languages

and are completely unrelated. Then the scores are, as a good approximation, drawn from a uniform distribution over the interval . The probability density function for the term then follows a standard exponential distribution with 1 as its mean and standard deviation.³

The ER score is defined as the mean of

(approximately) independent variables that, if the languages are unrelated, are drawn from a distribution with mean and variance . So the ER score is a random variable with mean and variance if the languages are unrelated.⁴

The sum (and thus the average) of

exponentially distributed variables follows an Erlang distribution. However, this distribution can be approximated by a normal distribution (also with mean and variance ) if is sufficiently large. This follows from the Central Limit Theorem. So we can transform the ER score to a variable that is distributed according to a standard normal distribution in the following way:

Definition 3 (Corrected Evidence for Relatedness)

View Expanded

According to this definition, the mean and variance of the ERC scores for unrelated languages do not depend on

, i.e. on the number of missing entries. This enables statistical hypothesis testing for the null hypothesis : “ and are unrelated” vs. the alternative hypothesis : “ and are related.” The -value for a given ERC score is simply the probability that the standard normally distributed variable has a value (technically, this is the converse error function of ), regardless of the number of missing entries.⁵

Just like the ER score, the ERC score is a similarity measure between languages. It can be turned into a distance measure analogously to Definition 2.

Definition 4 (Distance based on Corrected Evidence of Relatedness)

View Expanded

For ASJP data with

, and .

To get an idea for the numerical magnitudes, dERC scores for some language pairs are given in Table 2.

It might seem counter-intuitive that the dERC of English to itself is larger than 0. This reflects the fact that the ASJP-list for English contains one pair of homonyms: both ‘I’ and ‘eye’ are transcribed as Ei. Therefore the probability of a chance identity is assessed as positive, and therefore the probability of the two lists being identical despite the languages being unrelated is assessed as positive, if very small.

5.1. The Method

Levenshtein alignment only distinguishes between identical and non-identical sounds. To achieve a better approximation of the etymologically correct alignment of cognate words, a graded notion of similarity between sounds seems more appropriate. The ALINE system from Kondrak (2002), for instance, uses a sophisticated hand-crafted notion of segment similarity that draws on insights from phonology. ALINE will be discussed in more detail in the next section. The present section reviews an alternative approach that has been used in previous work in bioinformatics (see, for instance, Durbin et al., 1989) and computational dialectometry (cf., among others, Wieling et al., 2009).

The approach emerges from similar considerations that we used in the previous section in the derivation of the ER measure.¹⁰ Suppose we want to compare two strings

and —such as sequences of DNA bases or protein molecules, or words that we suspect to be cognates—and figure out whether or not they developed from a common ancestor. We have two hypotheses: : “ and are unrelated” and : “ and are related.” According to Bayesian logic, the following holds:

The prior odds

are unknown, so we focus on the first term on the right hand side, which expresses the strength of the evidence of the particular data point for .

We start with the (unrealistic) assumption that in case

is true, no insertions or deletions of segments have taken place, so and have the same length. If is true, the segments of and are pairwise historically related. Under the simplifying assumption that point mutations in biological evolution and individual sound shifts in language change are mutually independent, the probability of observing strings and given is the product of the individual probabilities that and are historically related:

Let

and be two segments. The quantity is defined as the probability that a specific segment in some sequence (biomolecule/word) developed into an along one phylogenetic branch and into a along another branch. Under this interpretation, is symmetric, i.e. . If the substitution matrix , i.e. the values of , is known for all segment pairs , we have

If

and are unrelated, the pairings of and are just randomly picked segments. Let be the probability of occurrence of at an arbitrary position within an arbitrary sequence. With the simplifying assumption that the occurrences of segments at different positions within a sequence are independent of each other, i.e. the sequences have no grammar, we have

Of course the assumptions made here—stochastic independence of positions within a sequence and of evolutionary changes at different positions within a sequence—are wildly unrealistic. Nevertheless this null model leads to workable results, as we will see later on.

Putting the pieces together, we have

In the bioinformatics tradition, this quantity is called the log odds score. In computational linguistics, it is also known under the name of Partial Mutual Information (PMI, see Church and Hanks, 1990).¹¹ I will follow the latter terminology here. Some notation:

We now turn to the issue of insertions and deletions. Suppose we evaluate a specific hypothesis about the historical relation between

and , which includes assumptions about segments being inserted or deleted. This leads to an alignment between the sequences including gaps. As an example, consider the German and Swedish words for star, Stern/stjärna, which are StErn/SEnE in the ASJP transcription. The etymologically correct alignment is

The gap symbol “-” represents a position where either a segment has been deleted or a segment has been added in the other language.

and now refer to the positions in the aligned strings. In the example, would be the gap symbol.

Following standard practice in bioinformatics, I assume that there is a uniform PMI score for gaps, regardless of the segment the gap is matched with:

The constant

, which is positive, is referred to as gap penalty.¹²

However, both in biological evolution and in language change, insertions and deletions frequently operate on contiguous chunks of segments. For instance, in language comparison we frequently find partial cognates, i.e. word pairs in which one item is morphologically complex (or is etymologically derived from a morphologically complex word) and the other word is cognate to just one morpheme of the first word. Consider the Latin and Italian words for mountain, mons/montagna, transcribed as mons/monta5a in ASJP. The Italian word is probably derived from the Latin montaneus ‘mountainous,’ a denominal adjectivization of mons. So the correct alignment is

The three gaps at the end of the upper sequence are the reflex of a single historical process, i.e. suffixation plus semantic change.

Since gaps frequently come in chunks, the penalty for a gap in

at position should be lower if is also a gap than if is a regular segment. This is captured by the notion of affine gap penalties. There are two positive constants (penalty for opening a gap) and (penalty for extending a gap) with such that if :

The same applies mutatis mutandis to gaps within

.

With these provisos, the PMI score of an alignment of

and gives an estimate of the strength of evidence that provide for under a specific alignment. The upper bound thereof is the maximal PMI score for any alignment of with . The Needleman-Wunsch algorithm (Needleman and Wunsch, 1970) is a simple generalization of the Levenshtein alignment algorithm that, for a given substition matrix and gap penalties and , efficiently (i.e. in quadratic time) finds this optimal alignment and its PMI score.

5.2. Parameter Estimation

To reliably estimate the PMI scores for all segment pairs, one would ideally need a very large corpus of correctly aligned sequence pairs. In bioinformatics such databases do indeed exist, and several carefully crafted substitution matrices for different domains have been constructed (see Durbin et al., 1989 for details). In dialectometric work (such as Wieling et al., 2012), such data are fairly easy to obtain because dialectometric data are organized in cognate sets, and the linguistically correct alignment between cognate words from different dialects of the same language can be reliably constructed with automatic means.

When dealing with cross-linguistic data from a wide variety of languages such as the ASJP data, the situation is more difficult. Sizeable amounts of expert cognacy judgments only exist for a small number of language families (mainly for Indo-European based on the pioneering work of Dyen et al., 1992, and for Austronesian, see Greenhill et al., 2008). Also, and more importantly, the ultimate goal of this entire enterprise is to do language classification automatically. Therefore, information about language family affiliation should not be utilized for parameter training to avoid circularity.

In the following, a heuristic method is described to extract a large corpus of probable cognate pairs from the ASJP word lists, which can be used for parameter training. The method only relies on the word lists themselves; no additional information about cognacy relations or the genetic affiliation of the languages involved is being used.

To avoid the pitfall of overtraining, I split the ASJP database into two sets of about equal size, the training set and the test set. For training purposes, I use only the former. The resulting model will then be tested against the latter.

To make sure the two sets are really independent—or at least to approximate this ideal as far as possible with cross-linguistic data—the two sets were constructed in such a way that each WALS family either completely belongs to the training set or the test set.¹³ To be more specific, the set of WALS families was placed in a random order and languages and language families were added to the training set in this order, as long as its size did not exceed half the size of the entire database. The remaining families constitute the test set. The training set contains 2,723 and the test set 2,758 word lists. The lists of language families in the two sets are provided in the Online Supporting Material.

Relying on the training set only, I used the following procedure for constructing a sufficiently large corpus of probable cognate pairs, which can be used for parameter training:

• All language pairs that have a dERC distance below a given threshold

  

  

  View Expanded

  
  are considered to be probably related.

• For a pair of probably related languages

  

  

  View Expanded

  
  and
  

  

  View Expanded

  
  and a concept
  

  

  View Expanded

  
  , all entries for
  

  

  View Expanded

  
  in the ASJP list for
  

  

  View Expanded

  
  and in the ASJP list for
  

  

  View Expanded

  
  are considered. The pair of words with the lowest LDN score is considered a potential cognate.

• All pairs of probable cognates are then aligned with the Levenshtein algorithm. If there are multiple optimal alignments, only one of them is considered.¹⁴

This yields a set of aligned sequence pairs. The quantity

is estimated as the relative frequency of alignments of with (in either direction) within this corpus. The quantity is estimated as the relative frequency of occurrence of the segment type within the entire ASJP data base (or the subset thereof that is used for training purposes). This gives an estimate for

for each pair of segment types.

Assuming certain values for the gap penalties (more on this later), in the next step the set of potential cognate pairs is aligned with the Needleman-Wunsch algorithm, using the estimated parameters.

Additionally, I assume a threshold

. All potential cognate pairs with a PMI score are treated as probable cognates. The set of aligned probable cognates is then used to re-estimate the PMI scores in the way described above.

The re-estimation of parameters is repeated 10 times. Experience shows that the estimated parameter values do not change substantially anymore after that.

The appropriate choice for the meta-parameters

and remains to be determined. As for the former, it is instructive to look at the distribution of dERC distances. It is displayed in Fig. 8 as a histogram.

It can be seen that the distribution is dominated by a bell-shaped curve with the maximum at about

. As pointed out in connection with the derivation of dERC (Section 3.2), we do expect that the ERC for unrelated languages approximately follows a standard normal distribution. The transformation leading from ERC to dERC turns this into a normal distribution with mean at , so the histogram shows that the vast majority of language pairs behave under dERC as if they are unrelated. We also see that the distribution is not symmetric—there are more values below than above the predicted mean value. The threshold should be chosen in such a way that the probability that an unrelated language pair has a dERC is very small, while at the same time ensuring that there are still sufficiently many language pairs with a dERC to make parameter training possible. I picked as a somewhat arbitrary choice which fulfills both requirements. It corresponds to an ERC value of , and the probability that a standard normally distributed variable has a value above that point is about , which seems to be sufficiently small. On the other hand, there are 20,505 language pairs with a dERC score in the training set, which gives rise to about pairs of potential cognates. This figure is large enough for parameter estimation.

As pointed out above, there is no straightforward way to estimate gap penalties from a training corpus. Appropriate values for

and have to be found via optimization. The same holds for .

The training procedure supplies a PMI matrix for a given vector

, which in turn, together with and , defines a PMI score for pairs of strings, which is a similarity measure for word pairs. Using the dERC aggregation procedure with LDN scores replaced by negative PMI scores, this gives us a distance measure between languages. Let us call it dERC/PMI.

As a heuristic to assess the quality of a parameter configuration, I sampled 1,000 pairs of probably related languages, i.e. languages with a dERC

. The mean dERC/PMI between these 1,000 language pairs is treated as the target function to be minimized. According to the way dERC/PMI is computed, this amounts to maximizing the ranks of translation pairs, i.e. maximizing the similarity between synonymous word pairs while at the same time minimizing the similarity between non-synonymous word pairs. As we can assume that there are many cognates among translation pairs from probably related languages, minimizing the mean dERC/PMI is tantamount to maximizing similarity between cognates and minimizing similarity between non-cognates.

As even a single evaluation step is computationally quite expensive, advanced methods of optimization such as simulated annealing proved to be impractical. Therefore I performed a simple downhill Nelder-Mead style optimization (cf. Nelder and Mead, 1965), starting from several manually chosen initial positions. The lowest value of the target function was achieved with

, , and .¹⁵

For a selection of sounds, the optimal PMI scores thus derived are shown in Table 7. (The full matrix is provided in the Online Supporting Material.) Not surprisingly, the entries along the diagonal are all positive, i.e. alignment of two identical elements provides the strongest evidence for relatedness. Additionally, we find positive PMI scores for several sound pairs that are known to be frequently historically related via sound shifts, such as p/b, d/t, d/8 (where the ASJP symbol 8 represents voiceless and voiced dental fricatives; cf. Table 15) and s/h. The latter case is especially interesting because the two sounds are articulatorily dissimilar, but the sound shift from s to h is known to be quite common (see, for instance, Ferguson, 1990).

Figure 9 displays a hierarchical clustering of the ASJP sound symbols according to their PMI scores.¹⁶ We find a primary split between vowels and consonants. The consonants are further divided into three large groups, which largely correspond to the dental, labial, and velar/uvular sounds. The only exception to this pattern according to place of articulation is the position of h and x (the voiceless and voiced velar fricatives), which are clustered together with the s-sounds within the larger cluster of dental sounds. This is probably a reflex of the already mentioned diachronic cline from s to h.

Following the example of Wieling et al. (2012)—who obtained PMI scores essentially in the same way but using data from different dialects of the same language—I performed non-metric multidimensional scaling with the PMI scores among the vowels. The result is displayed in Fig. 10. We find that the articulatory vowel triangle is reproduced to a good approximation, with the schwa (ASJP symbol 3) in the center.

Brown et al. (2013) also use the ASJP data to estimate the probability of different sound correspondences across the languages of the world. Their method is quite different from the one developed here, so a comparison of the results provides a certain validity check.

The authors use a highly conservative heuristic to identify regular sound correspondences. According to this method, a pair of languages

exhibits a regular correspondence between the segments and if and only if:

• 

  

  View Expanded

  
  and
  

  

  View Expanded

  
  belong to the same genus, and

• there are at least two concepts

  

  

  View Expanded

  
  such that the ASJP entry for
  

  

  View Expanded

  
  from
  

  

  View Expanded

  
  can be transformed into its translation to
  

  

  View Expanded

  
  by replacing all occurrences of
  

  

  View Expanded

  
  by
  

  

  View Expanded

  
  (and vice versa).

To use the running example of the English/Swedish comparison again, there are only two regular correspondences that can be detected from the 40-item word lists: o-e (bon/ben ‘bone’ and ston/sten ‘stone’); and i-e (liv3r/lev3r ‘liver’ and si/se ‘see’).

A certain genus is available for a correspondence

- if both segments, and , occur in at least one language within this genus. The PG score (“percentage of available genera”) of a correspondence is the relative frequency (expressed in percent) of genera exhibiting the correspondence at least once among all genera that are available for that correspondence.

Figure 11 plots the PG scores of all consonant pairs that have a positive PG score in the supporting online material from Brown et al. (2013) against their corresponding PMI score.

The

-axis is logarithmically transformed while the -axis is linear. We see a strong positive relation (indicated by the regression line) between the two measures. In fact, the correlation between the logarithms of the positive PG scores and the corresponding PMI scores is . This underscores that the empirically induced PMI scores do in fact reflect genuine patterns of regular sound correspondences.

5.3. Aggregation

In Table 1 the LDN scores for several English/Swedish word pairs were given. Table 8 gives the corresponding PMI scores.

It can be seen that the PMI notion of string similarity is more fine-grained than LDN. For instance, while both du/yu and vi/wi receive a positive score (i.e. they are more likely to be related than not), the absolute value for the latter is much higher. This reflects the fact that a correspondence between v and w is more likely than one between d and y. The pair fisk/fiS has an even higher PMI score because (a) the words are longer than vi/wi, i.e. the evidence they provide is stronger, and (b) the correspondence s/S is very likely. This is counterbalanced only by a single gap penalty.

The distribution of PMI scores for the English/Swedish comparison on the diagonal and off the diagonal is shown in the left panel of Fig. 12. The right panel shows the same data for the comparison English/Swahili.

In comparison to the corresponding plots for LDN, the PMI values are much more spread out. Apart from that, we find a similar qualitative pattern (apart from the inessential difference that LDN is a distance and PMI a similarity measure). For a pair of related languages, the diagonal entries are mostly much higher than the off-diagonal entries, while both collections appear to be drawn from the same distribution for a pair of unrelated languages.

The normalized ranks of PMI scores are now computed according to the definition given in Section 3, with LDN scores replaced by PMI scores and

replaced by . As shown in Fig. 13, the PMI-based normalized ranks from related languages follow approximately a power law distribution and those from unrelated languages a uniform distribution, just as the LDN-based normalized ranks.

Therefore the theoretical justification for the dERC-style aggregation of normalized ranks to a distance measure between languages also applies to PMI scores.

Table 9 compares the dERC/LDN scores and dERC/PMI scores for the language pairs from Table 2.

These numbers convey the impression that the dERC/PMI scores for related languages are generally lower than the corresponding dERC/LDN scores, while the scores for unrelated languages are randomly distributed around

for both measures.

5.4. Empirical Evaluation

The methods described in Section 4 to compare different distance measures will now be used to evaluate the quality of the dERC/PMI against LDND and the LDN-based version of dERC (dERC/LDN). Only the word lists from the test set will be used for this comparison.

The triplet distances to the three expert classifications are given in Table 10 and visualized in Fig. 14.

We find a slight improvement from LDND to dERC/LDN and a more substantial improvement to dERC/PMI.¹⁷

From the distance matrices for the test set for LDND, dERC/LDN and dERC/PMI, the corresponding phylogenetic trees were computed with Neighbor Joining. The generalized Robinson-Foulds distances and quartet distances are given in Table 11.

The results are not decisive, with dERC/PMI giving the lowest GRF scores and dERC/LDN the lowest GQD scores. However, as discussed in Section 4, evaluating different Neighbor Joining trees for a single data set can be highly misleading. Therefore the same procedure as above is applied here: 1,000 random samples of word lists from the test set, each comprising 500 doculects, are generated, Neighbor Joining trees for LDND, dERC/LDN and dERC/PMI are computed, and all three trees are compared to the three expert trees regarding both GRF and GQD. The results are depicted in Fig. 15 and the mean values are given in Table 12.

The mean values for the 1,000 samples display a similar pattern as the triplet distances: LDND and dERC/LDN perform about equally well (with a slight advantage for the former), while dERC/PMI leads to lower distance scores. As the aggregation method for dERC/LDN and dERC/PMI is identical, we can conclude that the PMI based method of measuring string similarities leads to better phylogenetic inference than (normalized) Levenshtein distance.

As a further test, I performed a version of cross-validation.¹⁸ In general,

-fold cross-validation means that a data set is split into subsets of equal size. Then one subset is singled out as the test set. The remaining data are used for training, and the model thus obtained is tested against the test set. This is repeated for each subset.

Cross-validation requires the individual subsets to be independent from each other. As discussed above, obtaining mutually independent subsamples of a cross-linguistic data base such as ASJP that are representative for the data set as a whole is a non-trivial issue. As an approximation, I performed 4-fold cross-validation, where the subsets correspond to the four continental areas Africa (including all Afro-Asiatic languages), Eurasia, the Indo-Pacific region (including Australia), and America.

For each continental area

, PMI scores where trained with the languages outside following the method described in the previous section. For the threshold and the gap penalties and , the values obtained from the original training set were used. Using the PMI scores thus induced, the pairwise dERC/PMI-values for the languages in were computed, and the triplet distances to WALS, Ethn and Hstr were determined and compared to the corresponding values for LDND and dERC/LDN. The results are given in Table 13 and displayed in Fig. 16.

In 11 out of 12 cases, dERC/PMI provides the best results (the exception being the Hstr classification for America, where LDND is slightly better). The general pattern for Africa, Eurasia and the Indo-Pacific is similar to the test set above: LDND and dERC/LDN are about equally good, while dERC/PMI is about

– better. For America, all three distance measures perform roughly equally well.

The average correlation of the PMI matrices obtained during cross-validation with the PMI matrix obtained from the training set is

, and the average correlation between the four PMI matrices from cross-validation is . This indicates that the patterns of regular sound correspondences across different samples of language families are highly similar.

5.5. Discussion

A possible objection against the general approach developed here concerns the risk of circularity. As an anonymous reviewer points out, it might be problematic to perform automatic language classification on the basis of parameters that are trained with data from a database “which was […] obtained through some other type of (manual) analysis.” Let us therefore carefully review what kind of information goes into the training procedure and what kind of information we get out of it.

The construction of the training corpus of word pairs relied on guessing a value for

. The guess of is to some degree arbitrary, but it was motivated by a visual inspection of the distribution of dERC/LDN scores.

dERC/LDN scores are determined on the basis of pairwise LDN scores for words from the word lists to be compared. No further information about the genetic affiliation of the languages involved is being used here, and LDN scores are obtained from Levenshtein distances, a general-purpose string comparison method that does not rely on any specifically linguistic information.

Once the training corpus is constructed, initial PMI scores are estimated using Levenshtein alignment. In subsequent steps, Needleman-Wunsch alignment is performed ten times, each time using the PMI score estimates from the previous step. For given values of

and the gap penalties and , the PMI scores thus obtained define a string similarity score, which is in turn fed into the dERC aggregation scheme to yield a distance measure between word lists. , , are estimated via optimization in a way that minimizes the average distance between a sample of language pairs. This sample was collected using only dERC/LDN scores and . So the only information that enters the entire procedure consists of the plain word lists. No knowledge about the languages involved is used anywhere in parameter training. In the terminology of machine learning, PMI scores are obtained via unsupervised learning.

The test procedures in turn use the aggregate distance between word lists thus obtained to do phylogenetic inference and to compare the results to expert classifications. (Triplet distance relies on classifying triplets of languages, so this also involves a kind of phylogenetic inference). So the information that is obtained from the parametrized model—language classification—is of an entirely different nature than the information that went into it, namely word lists.

Another potential objection concerns the fact that the overall gain in accuracy—about

for triplet distances and – for GRF and GQD—may still appear small. However, three considerations should be kept in mind here:

• Both for the triplet distance and for GQD, the baseline of completely randomly distributed distances is not

  

  

  View Expanded

  
  but
  

  

  View Expanded

  
  because there are only three rooted binary trees for a triplet and three butterflies for a quartet of languages.

• Even very crude distance measures achieve a much higher accuracy as suggested by these baselines. To illustrate this point, I defined such a crude measure: for each word list, the vectors of relative frequencies of occurrence of sounds are computed. The cosine similarity between two languages is then defined as the cosine of these vectors, and the cosine distance is the difference between the cosine similarity score and 1. So this distance measure only quantifies how much the frequency patterns of unigrams differ between word lists, without any reference to the meaning of the words. The Neighbor Joining tree derived from these distances for the entire ASJP database already achieves a GQD of

  

  

  View Expanded

  
  to WALS,
  

  

  View Expanded

  
  to Ethn and
  

  

  View Expanded

  
  to Hstr.

• The practically achievable minimum GQD (and likewise for triplet distance and GRF) is arguably somewhat above 0. First, the expert classifications contain controversial units (such as Altaic, Australian, Niger-Congo and Trans-New Guinea in WALS), which may partially be wrong. In this case it would not be a defect of an automatic classification if those units are not detected. Second, the 40-item Swadesh lists arguably do not always contain the information that human experts would need to establish a genetic relationship between a group of languages.

To make a rough guess, the maximum GQD (achievable by a simple-minded distance measure such as the cosine distance) for a given data set may be around 35%–40% and the minimum GQD that can possibly be attained by automatic methods from 40-item Swadesh lists may by around 3%. Each gain in accuracy of a certain percentage thus actually amounts to a much higher proportion (by a factor of about 3) of this range.