Member Avatar for shadow95

i have setup my custom content publishing script which is working great except for the part that when i publish content in it by copying it from external sources like from my other websites some special characters like (' " ) becomes this (� ) i have no idea why this is happening,so is there any way that i can fix this.

  • 3 Contributors
  • 3 Replies
  • 76 Views
  • 22 Hours Discussion Span
  • Latest Post Latest Post by Member #120589

Dani AI

Generated

This is most likely an encoding mismatch: the bytes copied from an external site are encoded in one charset but your publishing pipeline or the browser is decoding them as another. described exactly that pattern. Converting characters to entities (as suggested) can hide the symptom, but it does not fix the pipeline. was on the right track: the fix is to make the source, storage, server headers, and HTML agree on one Unicode encoding and to avoid double-encoding.

Practical checklist to follow in order:

  • Confirm the original copy source encoding (browser Save/Encoding, file -i, or use mb_detect_encoding on the raw bytes).
  • Verify the HTTP Content-Type header and the HTML meta charset both declare the same charset. See the meta charset guidance on MDN.
  • Ensure files are saved as UTF-8 without a BOM. A BOM can introduce odd characters.
  • Set the database tables, columns and client connection to a Unicode charset (prefer utf8mb4 for MySQL) and make sure the app opens the connection with that charset. For PHP, set the response header and the connection charset (example below).
  • If data is already stored incorrectly, export a backup and convert bytes with iconv or use a careful ALTER/CONVERT approach in the database.

Example (minimal):

header("Content-Type: text/html; charset=utf-8");
$mysqli->set_charset("utf8mb4");

Back up before converting data. For reference on meta charset and MySQL Unicode handling, see MDN and the MySQL docs: meta charset (MDN) and MySQL utf8mb4 notes.

Recommended Answers

All 3 Replies

Member Avatar for MJ Pieterse

i'm not 100% sure, but my guess is that you used the special characters and not their codes when you created the pages initially, besides that, you can't use special characters (should really not) and rather use the code, for instance:

" should be "
' should be ·

etc. so you should go and change it in any way if it is giving you gibberish symbols

Member Avatar for shadow95

i already did that but same errors

Member Avatar for Member #120589

Change your page encoding and browser encoding to UTF-8. If your data is coming from a DB, ensure that your encoding is right. You mention external sources, from where? Files, DBs, XML, website scrapes?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.