Last week I attended a training course on Unicode by Jukka K. Korpela. It was interesting, though the subject is… “tough”!

Following are a few (absolutely non exhaustive) notes I took during the course:

Introduction

Computers just deal with numbers. They store letters and other characters by assigning a number for each one. Before Unicode was invented, there were hundreds of different encoding systems for assigning these numbers. No single encoding could contain enough characters: for example, the European Union alone requires several different encodings to cover all its languages. Even for a single language like English no single encoding was adequate for all the letters, punctuation, and technical symbols in common use. (from What is Unicode)

Unicode is an international standard that, due to its complexity, is still not fully accepted. However, it is the default in several applications (e.g., XML applications).

Unicode is a “unified coding system” that contains more than 100,000 characters. It is dynamic, because new characters are continually added in order include “all” possible human characters. From a theoretical point of view, we could say that it tries to preserve cultural diversity while giving a universal interpretation of all human languages (it is arguable whether it is successful on this: for example, some Chinese characters are still not part of Unicode).

It’s about encoding, not fonts
There is an important difference between a font and the underlying encoding. Unicode is about “encoding glyphs”: given a sign representing a character in a human language, Unicode describes it univocally. Fonts, on the other hand, are a visual representation (a rendering) of those glyphs.

A font usually supports a small subset of Unicode. Western languages fonts, for example, do not support Chinese characters.

In general, only glyphs can be encoded, not abstract ideas. This simple concept has been and still is a matter of discussion whenever a new character needs to be included.

About characters

• Unicode is a 32-bits characters set. Each character has only one encoding, with some exceptions (compatibility reasons with older encoding systems)
• Some characters are obtained as a composition from other characters. The accented letter “è”, for example, is a composition of two characters: è = e + ` (see here for more details)
• The name of a character is its identifier: it contains letters, numbers, spaces, hypens.

A few definitions:

• Code point. A value in the 32-bits space. Each char has a code point, not all code points are assigned to chars. This is the numeric representation (usually hexadecimal) of a character.
• Blocks. Blocks are groups of characters. The assignment to characters to blocks, however, seems a bit confusing: for example, there is a block called “Greek and Coptic”, but it does not include all Greek characters.
• Categories. Each character has a set of properties which can be used for classification. For example, a letter category is anything used to write words in any language. There are properties which distinguish the script to which a character belongs to. There is a math symbols category.
• Normalisation. Technique to translate a complex character in two or more simpler characters.
• In western languages it is usually used to remove diacritics (accents, etc.) by substituting them with apostrophes
• Used for compatibility purposes (eg, to translate to ASCII)
• It might create problems if the process has to be reversed

Unicode in real life and a bit of IDNs

Using Unicode might lead to lots of confusion and extra care should be used when dealing with it:

• ASCII punctuation is different from Unicode punctuation, for example when dealing with quotes: “ ‘ ’ ” ‘ ” but to many the difference is not clear
• Certain characters are repeated in different scripts
• The Latin character A and the Cyrillic character A, for example, look/are the same.
• Different sets of numbers are presents in different scripts
• Sometimes the same punctuation characters can be found in different scripts with different logical meaning (this is the case of math symbols)
• Compatibility characters: they are used to make Unicode compatible with older encodings, it is a very vague concept that may easily induce in confusion. For example, K (Kelvin symbol) is different from K (letter) but identical in their representation.
• To make things worse, characters do not have a property that identify compatibility chars. People “know” which they are only from reading the big books containing the standards.

We discussed a bit about the problem of Internationalised Domain Names (IDNs), which open the doors to typosquatting and phishing. One policy might be to disallow mixing different scripts when registering IDNs. In certain languages it is common practice to use characters or words from the Latin alphabet as part of the sentence and such a solution would constitute a big limitation.

A partial solution, which might work for the most common cases, is to allow mixing any script with the “common” Latin script.

References

• www.unicode.org (hard to read)
• http://www.fileformat.info (easier to read, you can look up for single characters and see what they are)
• www.icu-project.org

Mapping the Mal Web, Revisited (McAfee, June 4).

A new security report from McAfee has just been released on the spread of malicious web sites among different TLDs. Very informative and detailed, the report integrates last year report. Some of the key findings:

• .ro (Romania) and .ru (Russia) are the most risky European TLDs, i.e., the probability of finding a malicious web site is higher if surfing one of those TLDs.
• Risk related to .biz (business) and .cn (China) is also increasing (if compared to last year)
• .it (Italy) has worsened, but is still “a safe place”
• .hk (Hong Kong) is the riskiest TLDs

The “Hong Kong” case, in particular, is worth a closer attention:

Bonnie Chun, an official [from the .hk] TLD, acknowledged that they had made some decisions that inadvertently encouraged the scammers:
1 . “We enhanced our domain registration online process thus making it more user-friendly. Instances include the capability for registering several domains at one time, auto-copying of administrative contact to technical contact and billing contact, etc. Phishers usually registered eight or more domains at one time.
2 . We offered great domain registration discounts, such as buy-one, get-two domains.
3 . Our overseas service partners promoted .hk domains in overseas markets.”

In a previous post I talked about the recent increased phishing activity in the .uk registry, which, in that particular case, has taken advantage from Nominet’s automatic registration process.

Other country, other problem: the .it registry will implement automatic registration procedures by the end of the year; and I read, a couple of weeks ago on Swartzy’s blog, that the IIT/CNR is also launching an advertisement campaign for .it domains.

I am curious to see if, in analogy to what happened in Hong Kong, we will see an increase of the malicious activity in the .it TLD.