frequency[ grades[ 2 ] / 10 ]++;

here assume that grades[2] includes 87, divided by 10 it equals to 8.7, which equals to

frequency[8.7]++;

so is the decimal removed ?

or is it rounded to 9 ?

Your compiler will throw an error if you try to put a decimal number into an array subscript. Most likely grades[] is defined as an array of integers. The integer 87 divided by the integer 10 would be 8, as ints cannot hold decimal values (in other words, any fractions are chopped after the arithmetic is done).