New Transliteration Test Files
The Test_*.html files show the transliteration of characters for given
languages. The sample for each language consists of "What Is Unicode"
in Thai, followed by other available text. The text is broken apart into
sentences for ease of viewing (note: we know of some problems with the sentence
rules for Japanese and Chinese). The left column is the original, and the right
is the romanization. The program also converts back to the original script. If
there is a discrepancy between the source and the reverse transformation, that
is indicated by making the background red
from that point on.
  Note: If you have some more text that you would like added to the
  sample, just let me know. I am particularly interested in name lists, since
  they are the typical source.
Standards
The goal is to follow a given standard, such as ISO* or UNGEGN wherever
possible. We also need to round-trip, so in some cases, that means adding some
additional accent marks to disambiguate characters. And often the source
standards are missing some characters, such as characters with combining Hamzas
in Arabic. Remember that the goal for these is transliteration (unambiguously
representing all the letters in the original), not transcription (representing
the best pronunciation).
  - Thai: ISO 11940 < http://homepage.mac.com/sirbinks/pdf/Thai.r2.pdf
    > plus a few items:
    
      - Accents may be added to the Latin for disambiguation.
- In the next release, we'd like to do the UNGEGN version < http://www.eki.ee/wgrs/rom1_th.pdf
        > which is probably more useful (and readable), and follows more
        closely the Thai standard.
- Spaces are provided at word-breaks, using the Thai BreakIterator.
- An inherent vowel (ọ) is added, as in UNGEGN. The dot is for
        disambiguation.
        
          - Note: if the inherent vowel positions cannot be algorithmically
            determined, let me know and I will remove them.
 
 
- Arabic: Generally follows
    UNGEGN < http://www.eki.ee/wgrs/rom1_ar.pdf
    >
    
      - Accents may be added to the Latin for disambiguation.
- Occasionally deviates in the direction of ISO 233 < http://homepage.mac.com/sirbinks/pdf/Arabic.pdf
        >
        
          - with underdot instead of cedilla for letter like SAD, since those
            are explicitly in Unicode for transliteration of Arabic
- adding extra non-Arabic-language letters, like PEH. Note: not all
            extended Arabic characters are handled yet.
 
- Does not do assimilation of "al", nor hyphenation of
        it.
        
          - While it could be done, we need to determine whether a prefix
            "al" could occur other than as the definite article (since
            no space is used).
 
- This is transliteration. For transcription one would want an
        engine that added points appropriately to the Hebrew.
 
- Hebrew: Generally
    follows UNGEGN < http://www.eki.ee/wgrs/rom1_he.pdf
    >, with some exceptions:
    
      - Accents may be added to the Latin for disambiguation.
- Combinations of dagesh, shin/sin dot that would produce different
        letters are not yet called out.
- Note that the final forms are not preserved. Thus, when going from
        Latin to Hebrew, a character is given final form depending on its
        position.
        
          - E.g. מםמם => mmmm =>
            מממם
 
- This is transliteration. For transcription one would want an
        engine that added points appropriately to the Hebrew.
- See also < http://homepage.mac.com/sirbinks/pdf/Hebrew.r1.pdf
        > for the ISO version. The Chicago Manual of Style has a clear table
        of mappings for the vowel marks.
 
- Han: Uses the CEDICT
    data plus Unicode Unihan kMandarin values for pinyin. Doesn't
    roundtrip!
    
      - Note: the Chinese pronunciation of Han characters varies by
        context and grammar, though nowhere near as much a Japanese.
        
          - Ideally we'd have an underlying engine for this. In 2.4 we will
            have a plug-in interface so that people could add one, such as the
            IBM engine.
- The data from CEDICT and Unihan don't list the most frequent
            choice first, so we will be updating that.
 
 
- Greek/UNGEGN: Uses a
    modern Greek transliteration, based on the UNGEGN rules at < http://www.eki.ee/wgrs/rom1_el.pdf
    >. This version will not roundtrip ancient Greek.
- Greek: Uses a classic Greek
    transliteration. This version will not roundtrip modern Greek.
Notes
  - For readability, the files have a few other things besides just the
    transliteration:
    
      - The first word of the sentences are titlecased, as are names (where we
        have a name-list, such as in Thai).
- The Latin in the original is mapped to the private-use zone before
        conversion, and then again after conversion. This does have the downside
        that any rules (such as in Han) that need to know the context (e.g. for
        inserting spaces or capitalization) will gum up a little bit. This is
        just an artifact of the test display.
 
- I don't think that ISO 11940 is a particularly good way to romanize, but
    it is at least complete and a standard. So what I am interested in just for
    now is whether the samples in the file follow it (with the above
    exceptions).
- Some of the files also have a set of characters at the end, one character
    per row, with a following row listing the hex and name.
- The source rules for all of these is in the following URL. So if you want
    to know the details of how the characters are handled, that is the place to
    look.