Using t-distributed Stochastic Neighbor Embedding (TSNE) to cluster folios

For this attack we’ll use the Takeshi EVA transcription to count the number of times each glyph appears on each folio. This gives us a vector of probabilities for each glyph, for each folio – the vectors are 24 long, as there are 24 EVA glyphs in the alphabet.

For example, here is the probability vector for f1r:

1r 28 lines {‘a’: 0.08917835671342686, ‘c’: 0.08216432865731463, ‘e’: 0.05110220440881764, ‘d’: 0.06212424849699399, ‘f’: 0.00501002004008016, ‘i’: 0.08617234468937876, ‘h’: 0.12324649298597194, ‘k’: 0.045090180360721446, ‘*’: 0.012024048096192385, ‘m’: 0.001002004008016032, ‘l’: 0.03507014028056112, ‘o’: 0.11923847695390781, ‘n’: 0.050100200400801605, ‘p’: 0.012024048096192385, ‘s’: 0.06412825651302605, ‘r’: 0.04408817635270541, ‘t’: 0.03907815631262525, ‘y’: 0.07915831663326653}

(This reads as glyph “a” appears 8.9% of the time on f1r, glyph “c” 8.2% of the time, and so on.)

The question is: how similar are these frequency distributions amongst all the folios? Using tSNE (implemented in Scikit learn here: http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) we can try to find a 3D arrangement of all the folios that minimises the glyph frequency vector difference between nearby folios.

Here’s a typical result: each folio appears as a point in 3D space …

The colour coding is: red dots are folios that Currier identified as “Language A”, blue are “Language B”, and the remaining black dots do not have an assignment.

It’s clear that the red and blue are well separated, reinforcing Currier’s assignments. Thus this is independent support of Currier’s theory.

There are a couple of notable features:

• f57r and f57v are labelled as Language A (red) – but it looks like they should be labelled as Language B (blue)
• The unassigned folios (black dots) look like they are all Language B

Folio Similarities

Something Knox said recently made me wonder how the vocabulary of the VMs folios changes throughout the manuscript.

I made some counts and filled them into an Excel spreadsheet. I defined the Similarity between folio i and j to be computed as follows:

1) List all unique words in Folio i = Ni
2) List all unique words in Folio j = Nj
3) List all unique words appearing in both Folio i and Folio j = Mij

Then compute Similarity = Mij / (Ni + Nj – Mij)

(If Folio i contains exactly the same words as Folio j then S = 1, and if it contains no words in common with Folio j then S = 0)

You can see a visual pattern of of the Similarity distribution here:

(I have a feeling I’ve seen something similar to this for the Voynich before … but can’t find it now – can someone help? – see References below!)

This contour plot is symmetric about a line running diagonally from the left hand bottom corner to the top right hand corner, corresponding to i=j (for which I set the values to 0 for easier viewing).

The rectangular red region around folios 140 to 165 corresponds to strong similarity in the VMs folios f75r to f84v – the Biological Folios. These pages all typically share up to 50% of the same words.

What I found surprising is the generally low level of shared vocabulary between the folios: typically only a few of the words used on one folio are used on the next – but see below.

The spreadsheet answers questions like “Which folio is most similar to folio f1v?” … the answer being f24r by this metric.

Clustering

Using the Similarity number as a connection strength between each pair of folios, we can generate a cluster map that arranges the folios so that similar folios appear together. I used the freely available software called LinLogLayout to do this. Here are the results:

The algorithm has split the folios into two clusters, shown as red and blue circles. Interestingly, the red circles generally match Currier Language A and the blue match Currier Language B. For some folios near the interface, e.g. f68r1, the Currier Language is “unknown” (according to http://voynich.freie-literatur.de/index.php?show=page&id=f68r1) … indicating uncertainty in the attribution, consistent with the folio’s position on the cluster map.

For folio f103v, at the far right edge of the blue cluster, the Currier Language is “BX”.

Comparison with a Latin Text

Here I took the Latin Herb Garden and split it into 20 folios corresponding to each of the herbs described. Then I ran the same code against it to generate the similarities between each folio, and made an Excel spreadsheet.. The corresponding contour plot is shown below, with the same colour scale as the one for the Voynich above.

As you can see, the typical value of “Similarity” between folios is around 0.02 or so … much *lower* than for the Voynich. The conclusion is that the Voynich folios are much more alike than this Latin text, and the Biological Folios in particular are quite unusually similar.

References

This is very similar work to that done by Rene in 1997: http://www.voynich.nu/extra/lang.html although his word counting rules are different (I only count unique words).

Comment by Nick Pelling

Nick sent me the following email and included an annotated version of the LinLogLayout shown above.

Having played with it a bit (as per the attached jpeg), it appears that while some pages’ recto and verso sides are very similar, others are wildly different. For example, just in the recipe section:-
103    good
104    very bad
105    very good
106    bad
107    excellent
108    excellent
109    (missing)
110    (missing)
111    excellent
112    good
113    excellent
114    excellent
115    very bad
116    n/a

Looking at pages within recipe bifolios, however, yields different results again: for example, even though both f104 and f115 are both “bad” above (and are on the same bifolio), f104v is extremely similar to f115r, while f104r is extremely similar to f115v (which is a bit odd). Furthermore, the closeness between f111v and f108r suggests that these originally formed the central bifolio (but reversed), i.e. that the correct page order across the centre was f111r, f111v, f108r, f108v. However, f105 / f114 seem quite unconnected, as do f106 / f113 and f107 / f112.

At this point, however, we may be mining too deeply, and that the presence of so many datapoints in a single overall set may be getting in the way. I suspect that pre-partitioning the dataset (i.e. working on each thematic section in isolation) may yield more informative results.