>and I'm sure it's not right.
Nope. fread() returns the number of successfully-read elements. You need to convert the char from buffer into an integer. If you'll take a look at an ASCII table, you'll see that characters are just decimals. You'll also see that numbers are stored as decimals, too.
So... you'll notice that '0' starts at 48. Just subtract 48: