Language A and B Again

March 13, 2013 12 comments

A tentative conclusion from comparing Language A and Language B  is that the non-gallows glyphs are used in the same way in both Languages.

That is to say, they appear to mean the same thing. So the “o” in A means the same as the “o” in B.
There is some persistent “mixing” between the e/y glyphs, which is illustrated by the example result below:
ABMixing
There is also some doubt about the “8” glyph, which sometimes seems to mix with the gallows glyphs (e.g. in some cases, the “8” appears in A to function in the same way as a gallows glyph in B and vice versa). This may simply be an error in the comparison method, or it may be that the “8” is a null, or it may be due to some other effect.
The gallows glyphs are different – they don’t appear to mean the same in A and B. I’m focussing on those glyphs now.
Advertisements

Language “A” and “B” Conversions

March 5, 2013 12 comments

This is an update to my previous two posts on this topic.

I have been concentrating on searching for the correspondence between glyphs used in Language A, and glyphs used in Language B. As a reminder, the method is to take all words in, say, Language A, and “convert” them to words in Language B by changing the glyphs according to a candidate mapping table. The frequency of the converted Language B words is then compared with the original Language A words: the closer the frequencies, the better the mapping match.

Method Check using only Language A words

As a check of the method, I took the Herbal folios 1-25 (all in Language A) and split them into two groups: 1-12 and 13-25, and I then artificially labelled the latter group as Language B. Then I ran the matching procedure, which produced the following result:

Epoch 62 Best chromosome 0 Value= 5.62272615159e-05
Chromosome ['o', '9', '1', 'i', '8', 'a', 'e', 'c', 'k', 'y', 'h', 'N', '2', '4', 's', 'g', 'p', '?', 'K', 'H']
ngramsA    ['o', '9', '1', 'i', '8', 'a', 'e', 'c', 'h', 'y', 'k', 'N', '2', '4', 's', 'g', 'p', '?', 'K', 'H']

This is good and reassuring, since it shows that the words in folios 13-25 have essentially the same frequency distribution when their glyphs are mapped to the same glyphs in folios 1-12.

Removal of Glyph Variants in Voyn_101

As the tests progressed, it became clear that some of the glyphs GC defined in Voyn_101 were in fact variants of more common glyphs. The most obvious were the “m”, “n”, “N” glyphs mentioned before – with these included, the conversions between Language B and Language A were of much poorer quality than if they were expanded to “iiN”, “iN” and “iiiN” respectively. After some time weeding out these variants, the following table was arrived at:

seek =  ["3", "5", "+", "%", "#", "6", "7", "A", "X", 
         "I", "C", "z", "Z", "j", "u", "d", "U", "P", 
         "Y", "$", "S", "t", "q",
         "m", "M", "n", "Y", "!", ")", "*", "b", "J", "E", "x", "B", "D", "T", "Q", "W", "w", "V", "(", "&"]
repl =  ["2", "2", "2", "2", "2", "8", "8", "a", "y", 
         "ii", "cc", "iy", "iiy", "g", "f", "ccc", "F", "ip",
         "y", "s", "cs", "s", "iip",
         "iiN", "iiiN", "iN", "y", "2", "9", "p", "y", "G", "c", "y", "cccN", "ccN", "s", "p", "h", "h", "K", "9", "8"]

I am very confident that the glyphs remaining after using the above conversion table are the base set.  The base set of glyphs is thus:

Language A frequency order: 'o', 'c', '9', '1', 'a', '8', 'e', 'i', 'h', 'y', 'k', 's', '2', 'N', '4', 'g', 'p', '?', 'K', 'H', 'f', 'G', 'F', 'L', 'l', 'v', 'r', 'R'
Language B frequency order: 'c', 'o', '9', 'a', '8', 'e', '1', 'h', 'i', 'y', 'k', '2', 'N', 's', '4', 'g', 'p', 'f', '?', 'H', 'K', 'G', 'F', 'l', 'L', 'R', 'r', 'v'

where “?” represents all very rare glyphs (such as the “picnic table” glyph). There are thus 27 glyphs (15 gallows and 12 regular) excluding the rare special glyphs like the picnic table.

Glyph Mixing Between A and B

I ran many trials using the base set of glyphs, comparing various sections of the VMs written in the different hands. In particular, the following folio collections were defined:

Special = {'HerbalRecipeAB': range(107,117) + range(1,26),
           'HerbalAB': range(1,57),
           'HerbalBalneoAB': range(1,26) + range(75,85),
           'HerbalAstroAB': range(1,13) + range(67,75),
           'PharmaRecipeAB': [88,89,99,100,101,102] + range(103,117),
           'AllAB': range(1,117)
 }

The collection I used the most was the one called “HerbalBalneoAB”, which contains Herbal folios written in Language A, and Balneo folios written in Language B. The nice feature of this collection is that the number of words is around the same for both Languages, which makes comparing counts very easy:

Total words =  2846  Total Language A =  1581  Total Language B =  1584

As an example, here is a trial result for HerbalBalneoAB:

Language B ['o', '9', '1', 'a', 'i', 'f', 'c', 'y', 'h', 'e', 'K', 'N', '2', 's', '4', 'g', 'p', '8', 'k', 'H']
Language A ['o', '9', '1', 'a', 'i', '8', 'c', 'e', 'h', 'y', 'k', 'N', '2', 's', '4', 'g', 'p', 'K', '?', 'H']

In all the tests I ran, there were some common features in the results:

  • Mixing between “e” and “y” – when writing Language A, the use of “e” appears to be equivalent to the use of  “y” in Language B, and vice versa
  • Mixing between  8,f,F,k,K,g,G,r,R,?  and so on – the Gallows glyphs swap amongst themselves, and “8”

Just about all trials showed the “e”/”y” mixing. Tony Gaffney pointed out that these two glyphs are quite similar in stroke construction. The appearance of “8” amongst the swapping Gallows glyphs is curious.

Single glyphs in Language A and Language B

March 2, 2013 4 comments
As a sanity check, I looked at single glyphs (rather than nGrams > 1), searching for the mapping that takes all the Language B glyphs and maps them to Language A glyphs, so that the Language B words converted with the mapping most closely match the frequency of Language A words. I found the following:
Chromosome  ['o', '9', '1', 'a', 'H', 'c', 'e', 'h', 'y', 'k', '2', 's', 'm', '4', 'i', '(', '8', 'p', 'g', 'n']
ngramsA     ['o', '9', '1', 'a', '8', 'c', 'e', 'h', 'y', 'k', '2', 's', 'm', '4', 'g', 'i', 'K', 'p', '?', 'n']

This shows that most Language B glyphs map to the same glyph in Language A. However, there is some mixing going on here between “H”, “8”, “i”, “g”, “(“, “K” and “?”

It occurred to me that this may be due to GC’s choice of ascribing single glyphs where there should perhaps be several. In particular, he has:
“m” which looks like “iiN”
“n” which looks like “iN”
“M” which looks like “iiiN”
(I think EVA does a better job of recognizing these.) So I adjusted the GC transcription accordingly, replacing n,m,M with the i,N combinations above.
This resulted in a new mapping for B to A:
Chromosome  ['o', '9', '1', 'a', 'i', 'g', 'c', 'y', 'k', 'e', 'h', 'N', '2', 's', '4', '(', '8', 'p', 'f', 'H']
ngramsA     ['o', '9', '1', 'a', 'i', '8', 'c', 'e', 'h', 'y', 'k', 'N', '2', 's', '4', 'g', 'K', 'p', '?', 'H']
(There may be better mappings, but this is the best so far.) This has some interesting features:
  • e and y swap between languages
  • h and k gallows swap between languages
  • some mixing of g,8,(,K,f,? – some of these are relatively rare, so the statistics are poor, which may explain the mixing.
 Note that the simplification table I’m using for Voyn_101 is currently:
    seek = ["3",   "5",    "+",  "%",   "#", "6", "7",    "A", "X",  
            "I",   "C",    "z",  "Z",   "j", "u", "d",    "U", "P", 
            "Y",   "$",    "S",  "t",   "q",
            "m",   "M",    "n",  "Y",   "!"]
    repl = ["2",   "2",    "2",  "2",   "2", "8",  "8",   "a", "y",  
            "ii",  "cc",   "iy", "iiy", "g", "f",  "ccc", "F", "ip",
            "y",   "s",    "cs", "s",   "iip",
            "iiN", "iiiN", "iN",  "y",   "2"]
(Thanks to Tony Gaffney for spotting an error in the conversion for C in a previous version.)
Categories: Algorithms, Languages

The Relationship Between Currier Languages “A” and “B”

March 1, 2013 24 comments

Captain Prescott Currier, a cryptographer, looked at the Voynich many moons ago, and made some very perceptive comments about it, which can be seen here on Rene Zandbergen’s site.

In particular, he noticed that the handwriting was different between some folios and others, and he also noticed (based on glyph/character counts) that there were two “languages” being used.

When I first looked at the manuscript, I was principally considering the initial (roughly) fifty folios, constituting the herbal section. The first twenty-five folios in the herbal section are obviously in one hand and one ‘‘language,’’ which I called ‘‘A.’’ (It could have been called anything at all; it was just the first one I came to.) The second twenty-five or so folios are in two hands, very obviously the work of at least two different men. In addition to this fact, the text of this second portion of the herbal section (that is, the next twenty-five of thirty folios) is in two ‘‘languages,’’ and each ‘‘language’’ is in its own hand. This means that, there being two authors of the second part of the herbal section, each one wrote in his own ‘‘language.’’ Now, I’m stretching a point a bit, I’m aware; my use of the word language is convenient, but it does not have the same connotations as it would have in normal use. Still, it is a convenient word, and I see no reason not to continue using it.

We can look at some statistics to see what he was referring to. Let’s compare the most common words in Folios 1 to 25 (in the Herbal section, Language A, written in Hand 1) and in Folios 107 to 116 (in the Recipes section, Language B, written in a different Hand):

Comparison between word frequencies in Languages A and B

Comparison between word frequencies in Languages A and B

So, for example, in Language A the most common word is “8am” and it occurs 192 times in the folios, whereas in Language B the most common word is “am”, occuring 137 times.

We might expect that these are the same word, enciphered differently. The question then is, how does one convert between words in Language A and words in Language B, and vice versa? In the case of the “8am” to “am” it’s just a question of dropping the “8”, as if “8” is a null character in Language A. In the case of the next most popular words, “1oe”(A) and “1c89″(B) it looks like “oe”(A) converts to “c89″(B). And so on.

If we look at the most popular nGrams (substrings) in both Languages, perhaps there is a mapping that translates between the two. Perhaps the cipher machinery that was used to generate the text had different settings, that produced Language A in one configuration, and Language B in another. Perhaps, if we look at the nGram correspondence that results in the best match between the two Languages, a clue will be revealed as to how that machinery worked.

This involves some software (I’m using Python now, which is fun). The software first calculates the word frequencies for Language A and B in a set of folios (the table above is an output from this stage). It then calculates the nGram frequencies for each Language. Here are the top 10:

nGramFrequencies

The software then runs a Genetic Algorithm to find the best mapping between the two sets of nGrams, so that when the mapping is applied to all words in Language B, it produces a set of words in Language A the frequencies of which most closely match the frequencies of words observed in Language A (i.e.  the frequencies shown in the first table above).

Here is an initial result. With the following mapping, you can take most common words in Language B, and convert them to Language A.

Table for converting between a Language B word and a Language A word

Table for converting between a Language B word and a Language A word

A couple of remarks. This is an early result and probably not the best match. There are some interesting correspondences :

  • “9” and “c” are immutable, and have the same function
  • Another interesting feature is that “4o” in Language B maps to “o” in Language A, and vice versa!
  • in Language B, “ha” maps to “h” in Language A, as if “a” is a null

In the Comments, Dave suggested looking at word pair frequencies between the Languages. Here is a table of the most common pairs in each Language.

Common word pairs in Languages A and B

Common word pairs in Languages A and B

For clarity, I am using what I call the “HerbalRecipesAB” folios for this study i.e.

Using folios for HerbalRecipeAB : [107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]

More results coming …

How was the Voynich Manuscript text written?

August 23, 2012 7 comments

I’ve spent many happy hours poring over the text, and am convinced that it is not as “simple” as it appears (i.e. the “words” are not words at all). Here are some conjectures:

  1. The lines look like they are written left to right i.e. the glyphs were written down from left to right, but were not.
  2. The scribe started with the drawing and started writing glyphs at various positions on the page.
  3. The method used for choosing each glyph and for deciding its position involved a mechanical apparatus, perhaps a set of co-rotating cipher wheels that were used to convert each character in the Latin plaintext into a VMs glyph and page position
  4. The apparatus is set to a new starting position for each folio/page (so e.g. Bettony labels on the three folios the plant appears on are different)
  5.  The density of ink is a clue to the order in which the glyphs were written (nib/quill freshly dipped and full of ink, or almost dry)
  6. At some point the scribe finishes writing the needed glyphs, and then fills out the spaces with pseudo-random words.
  7. There is no punctuation because what is seen are not words. What is seen makes no grammatical sense because the glyphs are not ordered and positioned linearly across the page.
  8. Perhaps the secret to unwinding the cipher is in the labels. The labels on one page are constrained to have been produced by the same initial position of the cipher apparatus, and they must come from the plaintext label.

There are so many clues as to what is going on, yet putting them all together is hugely challenging

For example, Jim Reeds suggested years ago that the order in which the text had been written on the sunflower page, f33v:

f33v

was first the text to the left of the left stalk, second the text in between the stalks, and finally the text to the right of the last stalk. This is compelling, since the ink density looks different, and the lines don’t line up well across the stalks. It becomes clearer if you saturate the image:

f33v Saturated

And in that image, what jumps out are the glyphs that are darker than the others. Those can be seen more clearly in black/white:

f33v monochrome drop

where the “o”, “y”, “8”, “e” stick out like sore thumbs. Most of those are in the left section, some in the middle, and fewer in the right. Why are these glyphs bolder, why are they inked more heavily? Were these the glyphs initially placed on the page, and contain the real information, and the rest, unimportant and pseudo-random, were all added later to make the text look “normal”?

Categories: 8, ay, Characters, e, f33v, Features, Jim Reeds, Latin, o, oy, Theories, Writing, y Tags: , ,

Odd Distributions of “oy” and “ay”

August 23, 2012 6 comments
A few weeks ago I posted some images showing the positions of the 
gallows characters on each of the VMs folios.
(The blog post is here is you missed it: https://voynichattacks.wordpress.com/2012/06/29/page-positional-gallows-mk-ii/ )

With a couple of small changes to the code, I have generated a set of 
images showing the positions of the "oy" and "ay" glyphs on each of the 
folios. (I believe the oy and ay are transcribed in EVA as ol and or, 
not sure.) This was prompted by the observations that
a) these glyph pairs often occur many times on a folio,
b) on some folios they don't appear at all
c) on some other folios only "ay" appears, on others only "oy"
d) often the "oy" glyphs appear to the left of each line, and the "ay" 
to the right, and sometimes vice-versa.

I wanted to link to a few example images from the set. The colour code 
is "oy" yellow and "ay" pink, with the coloured square indicating the 
position of the "o" or "a", a grey square indicating another glyph, and 
a black square a space.

1)  Examples of "oy"s at the left, and "ay"s at the right:
f18v http://imageshack.us/photo/jjbunn/31/gffolio18v.jpg/
f29v http://imageshack.us/photo/jjbunn/571/gffolio29v.jpg/

2) Example of the opposite: "ay"s at the left, "oy"s at the right:
f26v http://imageshack.us/photo/jjbunn/826/gffolio26v.jpg/

3) Example of only "oy" on the folio:
f21r http://imageshack.us/photo/jjbunn/809/gffolio21r.jpg/

4) Example of only "ay" on the folio:
f26r http://imageshack.us/photo/jjbunn/402/gffolio26r.jpg/

5) Example of numerous "oy"s to only one "ay":
f37v http://imageshack.us/photo/jjbunn/51/gffolio37v.jpg/

6) Example of an even mixture of both types, across the lines:
f39v http://imageshack.us/photo/jjbunn/19/gffolio39v.jpg/

What might be going on here? Nick Pelling commented on my blog that GC, 
while working on the Voyn_101 transcription, got the impression that the 
change from dominant "oy" to dominant "ay" was a vocabulary change in 
the text (at least, that's what I understood from Nick's comment).

I'd welcome comments on this. Also, if you would like me to generate 
images for your favourite glyph's distribution, it's a trivial process - 
just let me know
Categories: ay, Glen Caston, Nick Pelling, oy, oy Tags:

Ink density and glyph order

July 31, 2012 4 comments

When using an ink pot and an old fashioned nib, one has to dip the nib in to the pot from time to time to replenish the ink. When the nib has just been dipped, the first few letters or so written often have more ink to them, as there is plenty on the nib.

If one were writing a string of identical letters, the frequency at which one has to re-dip the nib as it runs out of ink should be approximately constant. For example, if I’m writing “ooooooooooooo” then perhaps the nib holds enough ink for 5 “o”s before it needs replenishment. Then the appearance on the paper might be:

ooooooooooooooo

and so on.

On most folios of the VMs it’s apparent that some of the glyphs have been inked more heavily than others. Some possible explanations are:

  1. the nib has been refilled with ink just prior to these glyphs being written
  2. these glyphs have been re-inked for some reason
  3. these glyphs have been written at a different time with a different nib or ink or both

Here is a nice example of this feature

f49v

If the glyphs we see were written from the top down, from left to right, then the heavily inked glyphs are not spaced apart in the way expected by possibility 1 above.

One conclusion is that the glyphs we see were not written top down left to right, but in some other order.

Also, the glyphs that tend to be heavily inked are a subset of all the glyphs. “8” is very commonly heavily inked, but not always, even on the same folio. One of the gallows glyphs is also often heavily inked, but only on one side. Another is GC “1” and the downstroke on GC “y”.

(An interesting feature on this folio is the compound gallows that appears to have been constructed by first writing the standard gallows, then adding a line between each “c” that straddles it. Or perhaps the “c”s were written first, then the gallows was written and the “c”s joined later?)

Categories: f49v, Features, gallows Tags: , ,