Last week I attended a training course on Unicode by Jukka K. Korpela. It was interesting, though the subject is… “tough”! 
Following are a few (absolutely non exhaustive) notes I took during the course:
Introduction
Computers just deal with numbers. They store letters and other characters by assigning a number for each one. Before Unicode was invented, there were hundreds of different encoding systems for assigning these numbers. No single encoding could contain enough characters: for example, the European Union alone requires several different encodings to cover all its languages. Even for a single language like English no single encoding was adequate for all the letters, punctuation, and technical symbols in common use. (from What is Unicode)
Unicode is an international standard that, due to its complexity, is still not fully accepted. However, it is the default in several applications (e.g., XML applications).
Unicode is a “unified coding system” that contains more than 100,000 characters. It is dynamic, because new characters are continually added in order include “all” possible human characters. From a theoretical point of view, we could say that it tries to preserve cultural diversity while giving a universal interpretation of all human languages (it is arguable whether it is successful on this: for example, some Chinese characters are still not part of Unicode).
It’s about encoding, not fonts
There is an important difference between a font and the underlying encoding. Unicode is about “encoding glyphs”: given a sign representing a character in a human language, Unicode describes it univocally. Fonts, on the other hand, are a visual representation (a rendering) of those glyphs.
A font usually supports a small subset of Unicode. Western languages fonts, for example, do not support Chinese characters.
In general, only glyphs can be encoded, not abstract ideas. This simple concept has been and still is a matter of discussion whenever a new character needs to be included.
About characters
- Unicode is a 32-bits characters set. Each character has only one encoding, with some exceptions (compatibility reasons with older encoding systems)
- Some characters are obtained as a composition from other characters. The accented letter “è”, for example, is a composition of two characters: è = e + ` (see here for more details)
- The name of a character is its identifier: it contains letters, numbers, spaces, hypens.
A few definitions:
- Code point. A value in the 32-bits space. Each char has a code point, not all code points are assigned to chars. This is the numeric representation (usually hexadecimal) of a character.
- Blocks. Blocks are groups of characters. The assignment to characters to blocks, however, seems a bit confusing: for example, there is a block called “Greek and Coptic”, but it does not include all Greek characters.
- Categories. Each character has a set of properties which can be used for classification. For example, a letter category is anything used to write words in any language. There are properties which distinguish the script to which a character belongs to. There is a math symbols category.
- Normalisation. Technique to translate a complex character in two or more simpler characters.
- In western languages it is usually used to remove diacritics (accents, etc.) by substituting them with apostrophes
- Used for compatibility purposes (eg, to translate to ASCII)
- It might create problems if the process has to be reversed
Unicode in real life and a bit of IDNs
Using Unicode might lead to lots of confusion and extra care should be used when dealing with it:
- ASCII punctuation is different from Unicode punctuation, for example when dealing with quotes: “ ‘ ’ ” ‘ ” but to many the difference is not clear
- Certain characters are repeated in different scripts
- The Latin character A and the Cyrillic character A, for example, look/are the same.
- Different sets of numbers are presents in different scripts
- Sometimes the same punctuation characters can be found in different scripts with different logical meaning (this is the case of math symbols)
- Compatibility characters: they are used to make Unicode compatible with older encodings, it is a very vague concept that may easily induce in confusion. For example, K (Kelvin symbol) is different from K (letter) but identical in their representation.
- To make things worse, characters do not have a property that identify compatibility chars. People “know” which they are only from reading the big books containing the standards.
We discussed a bit about the problem of Internationalised Domain Names (IDNs), which open the doors to typosquatting and phishing. One policy might be to disallow mixing different scripts when registering IDNs. In certain languages it is common practice to use characters or words from the Latin alphabet as part of the sentence and such a solution would constitute a big limitation.
A partial solution, which might work for the most common cases, is to allow mixing any script with the “common” Latin script.
References
- www.unicode.org (hard to read)
- http://www.fileformat.info (easier to read, you can look up for single characters and see what they are)
- www.icu-project.org
