bcquality/custom/knowledge/architecture/text-artifacts-require-explicit-utf8-handling.md

bc-version	domain	keywords	technologies	countries	application-area
all	architecture	encoding utf-8 powershell mojibake here-string bom raw-bytes	al	w1	all

Text artifacts require explicit UTF-8 handling — no ambient encoding

Description

Two production incidents, same disease, different vectors. (1) smiley.agent.md shipped double-UTF-8-encoded for weeks: fetched as string content over HTTP and re-saved with ambient encoding — every em-dash became â€", and file misclassified the agent as Nim source code. (2) A Mode B batch script wrote the SAME mojibake into three project CLAUDE.md files: Windows PowerShell 5.1 reads a BOM-less .ps1 as cp1252, so UTF-8 em-dashes inside a here-string were mis-decoded before they were ever written. Both incidents corrupted deployed instruction files silently; both were caught by inspection, not by tooling.

The root cause is never the characters — it is trusting an AMBIENT encoding (HTTP string decoding, PS 5.1 script parsing, Out-File defaults) anywhere between a text artifact's source and its destination.

Rule

When creating, fetching or transforming CURABIS text artifacts (agent files, CLAUDE.md, knowledge files, scripts):

1. Transfers are raw bytes. Filesystem Copy-Item or Invoke-WebRequest -OutFile — never via .Content strings.
2. Writes are explicit UTF-8. [System.IO.File]::WriteAllText(path, text, [System.Text.UTF8Encoding]::new($false)) — never Out-File / Set-Content defaults for content with non-ASCII.
3. Text transformation with non-ASCII happens in Python (explicit encoding="utf-8" on read AND write) — or, if it must be PowerShell, the .ps1 file itself carries a UTF-8 BOM so PS 5.1 parses its literals correctly. A BOM-less .ps1 with non-ASCII literals is a latent bug.
4. Verify after generation: grep the output for â€ (the mojibake signature) before committing or deploying. One line, catches the class.

What NOT to do

• Do not put em-dashes, æøå or any non-ASCII inside a here-string in a .ps1 that lacks a BOM — the corruption happens at parse time, before your code runs
• Do not fetch a file via .Content and re-save it — the double-decode is invisible until someone reads the artifact
• Do not assume "it looked fine in my editor" — editors auto-detect; parsers and runtimes do not

Signal to watch for

The literal byte sequence â€ (or æ, ø, Ã¥) in any committed or deployed text file. Also: file/tooling misclassifying a markdown file as source code — that is encoding damage until proven otherwise.

Message to developer

When mojibake is found in an artifact, report which file, repair by cp1252→UTF-8 reversal (line-by-line fallback preserves already-correct lines), and identify WHICH transfer step used ambient encoding — the repair without the root cause just schedules the next incident.