The tiny encoding problem that can make a perfectly good CSV look broken

A supplier sends a CSV. The columns line up, the row count looks right, and every comma is where it belongs. Then someone opens it and sees José instead of José, a currency symbol becomes gibberish, or a lookup cannot find a header that appears to be spelled correctly.

It is tempting to call this a bad CSV. Often, it is not. The file’s structure may be fine. The problem is that the receiving software read its bytes using the wrong character encoding.

CSV and encoding handle different jobs. CSV tells software where fields and records begin and end: commas, quotation marks, and line breaks. Encoding tells it how a sequence of bytes becomes letters, punctuation, and symbols.

A file can get the first job exactly right and still fail at the second.

Why the error matters beyond appearance

Misread characters are not always cosmetic. A garbled customer name, place name, or product description can fail an exact match. It can create apparent duplicates, weaken a filter, or leave records unmatched during a join.

For example, München and a misdecoded version of the same text are different strings to a computer. A system may treat them as different locations even when the original source was correct.

The same issue can affect column names. If an importer fails to handle an initial UTF-8 byte-order mark, or BOM, the first header may contain an invisible prefix. A lookup for company_id then fails because the imported header is not quite company_id.

That is why a successful preview is not enough. A spreadsheet may display text acceptably while an import process still changes headers or values in ways that matter downstream.

Encoding is part of the delivery contract

UTF-8 is the sensible default for most CSV interchange. It can represent a broad range of characters, and the W3C’s CSV-on-the-Web guidance recommends UTF-8 for CSV files.

But “use UTF-8” is only half a delivery contract. The recipient also has to know, or correctly infer, that the file is UTF-8.

For CSV files delivered over HTTP, the Content-Type header can carry that information: text/csv; charset=utf-8. The charset parameter is part of the registered CSV media type, and it reduces guesswork for software receiving the file through the web.

That metadata may disappear when a file is downloaded, emailed, copied to a shared folder, or attached to a ticket. In those cases, document the encoding with the delivery and make sure the recipient’s import process can select it explicitly.

A double-click is not a reliable test of every receiving workflow. In Excel, for instance, the Text/CSV import path allows the user to select UTF-8 as the file origin when needed. That is more defensible than assuming automatic opening will always make the right choice.

Automated workflows need the same discipline. Make the decoder explicit rather than accepting a server, machine, or locale default. Python’s CSV documentation, for example, notes that opening a file without an encoding argument uses the system default encoding.

Three clues that point to encoding

The symptoms can help narrow the problem, though they do not prove the source encoding on their own.

Readable-looking gibberish such as José usually points to a decoding mismatch. The bytes may be valid, but the reader used the wrong character map.

Replacement symbols, missing characters, or decoding errors can mean the file contains bytes that are invalid under the encoding the receiving tool assumed. The source could use a legacy code page, UTF-16, or contain damaged data. Confirm before converting.

A mismatch only in the first header is a reason to check for an unhandled BOM. A UTF-8 BOM is a three-byte opening signature. UTF-8 does not require it, because UTF-8 has no byte-order ambiguity. Some applications use it as a helpful signal; others do not expect it and may retain it as header text.

That makes a BOM a compatibility decision, not a universal fix.

A short response plan when text looks wrong

Start by preserving the original file byte for byte. Do not manually repair hundreds of garbled values if a reliable source file still exists.

Then ask the sender for the declared encoding, or check the export documentation. Reopen or reimport the original with that encoding selected explicitly. If the confirmed source is not UTF-8, transcode it from the confirmed encoding rather than guessing from what looks plausible on screen.

After import, check more than a few visible rows. Verify:

• exact header names, including the first one;
• expected column and row counts; and
• a small set of known values containing accents, non-Latin characters, curly punctuation, or symbols.

Only after those checks should a team move on to separate CSV concerns such as delimiters, quoting, line endings, and spreadsheet type conversion.

For publishers and data teams, the preventive routine is straightforward: export UTF-8, declare charset=utf-8 when serving CSV over HTTP, document encoding for local-file delivery, and test the file in the exact workflow used by the recipient.

A CSV is not proven sound because it opens cleanly on the sender’s computer. It is proven useful when the receiving system reads the same headers and characters the sender intended.