- You SHOULD use Unicode internally, preferably with UTF-8 when exchanging text with others.
- Don't mix encodings: talk with your team to use the same encoding everywhere.
- Explicitly specify encoding: configure your servers, edit your files (HTML/XML/...), etc.
- Write documentation to specify the encoding of your source files and outputs.
- Read documentation/RFCs to know how to treat input texts and which encoding to use for outputs.
- It's easier when using Unicode friendly programming languages or good libraries like Qt or the full featured ICU library.
Text encoding is seen by many developers as something that should work automagically. We just want to manipulate text, that's all!
When it doesn't, we often add a line about
iso-8859-1 and hope it will work. Then we start to read more about encodings when something become... broken.
A character set is a table where numbers called code points are associated with characters: Unicode is a character set where the code point
233 refers to the character
LATIN SMALL LETTER E WITH ACUTE.
A character encoding is a method to store these code points using bits: UTF-32 is a character encoding for Unicode that uses 32 bits per code point.
For example, a 8-bit encoded ASCII character
0b01100101 refers to the code point
101 in the ASCII character set:
LATIN SMALL LETTER E.
For technical reasons, character sets having at most 256 code points are usually encoded using 8 bits (1 byte) per code point.
Thus, the same term is often used to refer both to character set and to character encoding: it's the case for ASCII.
That's why many people don't make the difference and just talk about charsets.
# History and Unicode
In 1963, ASCII appeared with 128 code points to represent English characters. It became the default character encoding on many systems.
Later, other parts of the world created their own character sets to handle custom characters.
In 1985, ISO-8859-1 appeared to extend the 128 ASCII code points with 128 additional code points to represent Western Europe characters.
As it was compatible with ASCII, ISO-8859-1 became the default character encoding on many systems (replacing ASCII).
Regarding to the many character sets in use and incompatibility issues, Unicode appeared in 1991 to unify all these character sets.
Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems. (Wikipedia)
- A character set that aimed to represent most used characters in the world, including mathematical symbols (up to 2 097 152 code points (21 bits)).
- Ways for combining characters together (for example, you can represent the character
LATIN SMALL LETTER E WITH ACUTEor as
LATIN SMALL LETTER Ecombined with the character
- Rules for sorting, comparing, normalizing and transforming texts (like for removing accents, capitalize letters, etc).
- Character encodings to store these code points.
The three common character encodings for Unicode are:
- UTF-8: provides a way to store a Unicode code point using 1 to 4 elements of 8-bit.
- UTF-16: provides a way to store a Unicode code point using 1 to 2 elements of 16-bit.
- UTF-32: provides a way to store a Unicode code point using 1 element of 32-bit.
In 2012, UTF-8 is the default character encoding for many recent Linux distributions.
# Is UTF-8 the way to go?
When mixing character sets or character encodings, it becomes hard for developers and users to avoid issues.
The easiest way to solve most issues is to agree to use the same character encoding.
UTF-8 may already be seen as the de-facto default character encoding to use.
Although it is compatible with ASCII it is not compatible with ISO-8859-1. This slows down the global migration to UTF-8.
However, it's now the default character encoding in many places (Linux, CSS, XML, SIP, ...) and 69% of websites now use UTF-8 on March 2012.
Moreover, many languages and libraries evolve to provide a better support for Unicode (C11, C++11, Python 3...).
# What can you do?
You can use UTF-8 whenever you can and be consistent:
- Configure your systems to use UTF-8 locales.
- Use UTF-8 to encode your files.
- When it's possible, add a header to your files to specify their encoding (HTML, CSS, XML, ...).
- Talk with you team to use the same encoding as you.
- If you provide an API, write which encoding to use for texts.
- Configure your servers to explicit its encoding (for web servers it's a Content-Type header).
However be careful: there is no effective way to know which encoding is used by a text file. If your system was previously using a non-ASCII encoding, switching to UTF-8 can lead to encoding issues: I recommend to start with a fresh Linux install and/or to use
iconv to convert your files.
# What else?
Well, you might still encounter some issues with files and APIs contaning non-ASCII characters and using another encoding.
However, this problem can be avoided with file types allowing to specify their encoding (HTML, CSS, XML, ...).
Otherwise, you will have to convert texts from the target encoding to your encoding (and vice versa) to fix the problem. To know which target encoding to use, you will have to:
- Communicate with the file or API provider to ask its encoding (read the documentation).
- Or heuristically guess the encoding (it's like guessing the language of a sentence - try
Then, you can use
iconv (as command line tool or as C library) or others languages or libraries to convert between encodings during inputs and outputs.
For example, you can know the encoding of Unix files and console by looking at
man 3 nl_langinfo).
You also have to look at how your programming language deals with encoding: some are encoding agnostics and serve what you give them, others convert your texts to a specific encoding.
# That's all?
It depends of your needs.
If you only want to handle all the symbols of the world: yes, that's all.
But if you really want to reach a perfect internationalized software, you'll still have to think about details:
- How to display dates in a familiar way to users?
- How to sort texts depending on contexts (language & place)?
- How to transliterate texts from one script into another?
- How to simplificate searches with non-ASCII characters? (like treating
eas equal to
# Good libraries for C/C++
For basic encoding conversions, you can look at the iconv library (
man 3 iconv).
For easy conversions and manipulations of Unicode texts, the Qt library is really pleasant to use. Note that Qt 4 use Unicode internally, but treats all texts as ISO-8859-1 by default. You can change this behavior and use UTF-8 by default with:
For advanced internationalization, the ICU library seems to be the way to go (also exists for Java).
As you see, mixing encodings leads to issues: work with your team to use the same character encoding and write documentation to explicitly mention which encoding you use.
As everybody migrates to UTF-8, the encoding issues will gradually decrease.
Feel free to post any suggestions/corrections/questions!