You said it yourself! So why bother and make your life difficult. When you come back later to your code it will be more easy to understand then if you used pointer arithmetic. If you have a good C++ compiler he would translate both options the same way!
>but (as I've read) pointer arithmetic is faster
Prove it. In your example, the two are quite likely compiled to exactly the same machine code. Array subscripting is literally syntactic sugar for the equivalent pointer offset calculation. A more likely example is this:
int *p = x;
for ( int i = 0; i < N; i++ )
process ( x[i] );
for ( int i = 0; i < N; p += i )
process ( *p );
The latter could very well be faster (though I would question why you're worried about such micro-optimizations) due to the fact that the body of the loop is only performing a dereference rather than an addition and a dereference.
My advice is not to worry about it. Use whichever more clearly displays your intentions.