Menu

CSV vs CSV UTF-8: the same rows in four encodings, byte by byte
📰

CSV vs CSV UTF-8: the same rows in four encodings, byte by byte

#devcommunity#dev#bytes#windows#file
Reading 0:00
15s threshold

"CSV" and "CSV UTF-8" sound like two formats. They are the same format: CSV describes the rows and columns, UTF-8 describes how the characters are stored as bytes. The only other thing that differs is an optional three-byte mark at the start of the file.

To see exactly what changes, I saved the same three rows ( Zoë Müller , José Peña , Chloé Dubois ) in four encodings and read the bytes.

The same rows, four encodings

Encoding Size First bytes ë is stored as

UTF-8 with BOM 88 bytes EF BB BF 69 64 2C C3 AB

UTF-8, no BOM 85 bytes 69 64 2C 6E 61 6D C3 AB

Windows-1252 77 bytes 69 64 2C 6E 61 6D EB

UTF-16 LE with BOM 156 bytes FF FE 69 00 64 00 EB 00

Three things stand out:

For plain English text, UTF-8 and Windows-1252 store identical bytes. Both files start 69 64 2C ("id,"). The difference only shows up with accented or non-Latin letters.

The first two rows differ by exactly three bytes: EF BB BF , the byte order mark (BOM).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.