Member Avatar for dinamit3

I have this string " țara " which contains a character ț which is stored on 2 bytes.

When i do in linux : hexdump -C file that contains the string it shows :

00000000  c8 9b 61 72 61                                    |..ara|
00000005

It is normaly to view c8 and 9b ? This isn't the code point in hex of ț .
When we take the same string and put it here : txt_to_hex
We get :

 021b 61 72 61

Where 021b is the code point of ț showed in hexa.

Why does hexdump shows a wrong result ? c8 9b is different from 021b ,that means is not showing us the same character.

Edited by dinamit3
  • 2 Contributors
  • 1 Reply
  • 321 Views
  • 1 Hour Discussion Span
  • Latest Post Latest Post by L7Sqr

Dani AI

Generated

Short answer: hexdump is working correctly. The difference comes from "code point" versus "encoded bytes". The Romanian small t with comma below is U+021B (the Unicode scalar). In a UTF-8 file that scalar is stored as a two-byte sequence, so a raw-byte dump and a tool that prints the Unicode codepoint will show different hex values for the same visible character. As noted, this is an encoding difference; 's hexdump is showing the file's actual bytes.

Practical checks (all-ASCII examples):

# show declared charset for a file
file -i file.txt

# view raw bytes as hex
xxd -g 1 file.txt

# inspect the Unicode code point and how it's encoded
python3 - <<'PY'
s = '\u021b'               # U+021B (small t with comma below)
print('codepoint =', hex(ord(s)))
print('utf-8 bytes =', s.encode('utf-8').hex())
print('utf-16-be bytes =', s.encode('utf-16-be').hex())
PY

Note about similar characters and normalization: Unicode contains two related code points — U+021B (t with comma below) and U+0163 (t with cedilla) — plus a decomposed form using a base t plus U+0326 (combining comma below). Different tools or legacy encodings might use one or the other. Normalization (NFC/NFD) determines whether the character is a single codepoint or a base + combining mark. Example normalization check:

python3 - <<'PY'
import unicodedata
s = 't\u0326'   # decomposed: t + combining comma below
print('NFC:', unicodedata.normalize('NFC', s).encode('utf-8').hex())
print('NFD:', unicodedata.normalize('NFD', '\u021b').encode('utf-8').hex())
PY

Final notes: treat hexdump/xxd outputs as authoritative for raw bytes, and use a codepoint-aware tool (or Python) to see Unicode scalar values. Converting files consistently to UTF-8 and normalizing (if matching is important) avoids most surprises.

Member Avatar for L7Sqr

It seems to be the difference between UTF-8 hex and UTF-16 hex (See encodings section here)

Edited by L7Sqr
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.