Member Avatar for senzacionale

Hi,

i parse 3 html sites and one is in utf-8 encoding, one in iso-859-1 and one in win-1250. This one in utf-8 wors fine when i parse it, also čćšđž works, but other two doesn't work. čćžđšš is corrupted and ? is shown.

Can you help me how to make that čćšđ will work even if page which i parse it has different encoding then utf-8?

Thx

  • 1 Contributor
  • 2 Replies
  • 117 Views
  • 14 Hours Discussion Span
  • Latest Post Latest Post by senzacionale

Dani AI

Generated

The symptoms you describe (question marks instead of accented letters) mean the page bytes were decoded with the wrong character encoding. The correct approach is to treat the HTTP response as raw bytes, determine the page encoding (HTTP header or the HTML meta tag), then decode the bytes with that encoding. HTTP headers and meta tags are authoritative; if neither is present you can fall back to a detector or heuristic. See general background on encodings for context: Character encoding (MDN).

Concrete checklist and notes for

  • Correct the name: it is ISO-8859-1 (not iso-859-1). Windows Central European is usually windows-1250 (code page 1250).
  • Fetch the response into a byte array (do not use a StreamReader without first knowing the encoding).
  • Check the response Content-Type/charset first (resp.CharacterSet or parse Content-Type).
  • If the header is missing or wrong, examine the first few KB of the raw bytes for a <meta charset=...> or http-equiv declaration (scan as ASCII or ISO-8859-1 since the charset token is ASCII).
  • Use System.Text.Encoding.GetEncoding(foundNameOrCodePage) to get the correct Encoding, then call encoding.GetString(bytes).
  • If detection is still ambiguous, use a charset detector library (for example, Ude) or an HTML parser that can honor declared encodings (Html Agility Pack has options for loading with a specified encoding).

Minimal example pattern (C#): read raw bytes, look at resp.CharacterSet, else scan initial bytes for "charset=", then call Encoding.GetEncoding(...) and decode. If you rely on libraries, Html Agility Pack and Ude can simplify detection and loading for malformed pages (see https://html-agility-pack.net/ and https://github.com/errepi/ude).

Cautions: some sites declare the wrong charset, or put the meta tag after large scripts so your scan must cover enough bytes. Always store/output in a consistent encoding (UTF-8) after correct decoding to avoid later corruption.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.