Home Original page

Encoding of surrogate code points to UTF-8

Steven D'Aprano steve+comp.lang.python at pearwood.info
Tue Oct 8 18:20:58 EDT 2013 
On Tue, 08 Oct 2013 18:00:58 +0100, MRAB wrote:

> The only time you should get a surrogate pair in a Unicode string is in
> a narrow build, which doesn't exist in Python 3.3 and later.

Incorrect.

py> sys.version
'3.3.0rc3 (default, Sep 27 2012, 18:44:58) \n[GCC 4.1.2 20080704 (Red Hat 
4.1.2-52)]'
py> s = '\ud800\udc01'
py> print(len(s))
2
py> import unicodedata as ud
py> for c in s:
...     print(ud.category(c))
...
Cs
Cs

s is a string containing two code points making up a surrogate pair.


It is very frustrating that the Unicode FAQs don't always clearly 
distinguish between when they are talking about bytes and when they are 
talking about code points. This area about surrogates is one of places 
where they conflate the two.



-- 
Steven
More information about the Python-list mailing list