python-unicode-gotchas
Installation
SKILL.md
Python Unicode Gotchas
str = code points, bytes = encoded output. Visually identical strings can differ in length (precomposed vs combining). str + bytes → TypeError; decode at input, encode at output.
Error handlers
| Handler | dec | enc | Effect |
|---|---|---|---|
strict (default) |
✓ | ✓ | raises UnicodeError |
ignore |
✓ | ✓ | drops offending char |
replace |
✓ | ✓ | U+FFFD / ? |
backslashreplace |
✓ | ✓ | \x80, \uXXXX |
xmlcharrefreplace |
— | ✓ | &#NNN; |
namereplace |
— | ✓ | \N{NAME} |
surrogateescape |
✓ | ✓ | lossless round-trip of unknown bytes |
surrogatepass |
✓ | ✓ | allows lone surrogates in UTF-8/16/32 |
surrogateescape maps bytes 0x80–0xFF ↔ U+DC80–U+DCFF. Use for read-then-write-back of files with unknown encoding.