Member Avatar for fnrayosunshine

We have a third party web host who designs our website and hosts it. They use templates and "tokens". What we do is upload our news articles as .txt documents. All of a sudden, our dashes and hyphens show up as question marks. Any ideas?

Thanks,

Heather

  • 6 Contributors
  • 9 Replies
  • 114 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by MidiMagic

Dani AI

Generated

Brief diagnosis: this is almost certainly a character-encoding mismatch (not a font problem). As , and pointed out, “fancy” dash characters (en‑dash / em‑dash / smart quotes) live outside plain ASCII. If a file encoded in Windows‑1252/ISO‑8859‑1 is served or interpreted as UTF‑8, those single bytes become invalid UTF‑8 and show up as replacement characters (often seen as question marks or the black diamond with a question mark).

Why that happens (concise): the ASCII hyphen is U+002D and is safe everywhere. An en‑dash is U+2013; in UTF‑8 it is the three bytes E2 80 93, while in Windows‑1252 it’s the single byte 0x96. If the pipeline (upload → token parser → page output) treats 0x96 as UTF‑8, the browser sees garbage and replaces it. For background on character sets and HTTP charset semantics see Character encoding — MDN.

Quick, practical troubleshooting (do these in order):

  • Confirm what the page is declared/served as:
    curl -I <URL> | grep -i Content-Type
  • Check the raw .txt encoding:
    file -bi news.txt
xxd -g 1 news.txt | head
  • If files are Windows‑1252, convert them to UTF‑8:
    iconv -f WINDOWS-1252 -t UTF-8 old.txt > new.txt

    (see the iconv man page for options.)

Immediate workarounds and final fix: convert all source files to UTF‑8 (no BOM) and ensure the site’s header include or HTTP header declares charset=utf-8 (for example <meta charset="utf-8">). If the host’s token engine is touching files, ask them to verify it preserves encoding or performs a correct conversion (look for PHP functions like utf8_encode()/utf8_decode() being misused). As a safe fallback, replace en/em dashes with &ndash;/&mdash; or plain - until the pipeline is fixed.

Recommended Answers

All 9 Replies

Member Avatar for ¤| battousai |¤

something wrong with the language files or fonts?

Member Avatar for fnrayosunshine

We use simple text and the default font is courier. I do know that there was a "front" folder that was deleted and then we started having the issues. We put the folder back. The host said that the folder is usually empty and that deleting it and putting it back did not cause the issue. However, I am at the mercy of their $100 / hour tech support because I am a net admin - not a developer or designer.

Here is our site: http://www.thelancasternews.com

Member Avatar for sillyboy

Are you using MS Word to create these .txt files? If so, I think you are encoding your text differently to what their web pages are using. If you want, try using notepad (or else, just make sure you use UTF-8).

Member Avatar for jbennet

yes, use notepad

the problem may be that you are making windows text files and the server expects UNIX text files

Member Avatar for sillyboy

Aren't windows & unix text files essentially the same except for their line endings? (Just curious)

Member Avatar for jbennet

I do a lot of work using PHP and MySQL and i notice that if i open the readme files (created on unix type systems) hey often have wierd characters like squares or accents

Member Avatar for MidiMagic

The trouble is that there are several versions of different characters in Word. I would suggest you change some settings:

- Go into the Word spellchecker and turn off the functions whcih make smart quotes and long hyphens.

- Make sure the server, your computer's default, and word's default are all the same character set.

- The only character in this area which is universally defined is the minus sign. Real hyphens and long hypens are in extended character sets.

- If you have trouble after that, choose the US ASCII character set in the editor. Your text won't be as pretty, but it works anywhere.

Member Avatar for Mrtony

If you stil have trouble, try making plain text files with edit under cmd. This will a plain ascii file without extended characters.

Member Avatar for MidiMagic

Notepad also makes plain text files.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.