Every few months (usually shortly after one of the browsers gets a new release) a review comes out with the results of benchmarks on the major browsers showing how each performs in several areas. Usually (surprise surprise) IE comes out as sub-par while browsers like Chrome and Firefox bobble around at the top, trading places occasionally. I've tried both and prefer Firefox because of the user interface. Chrome may outperform it on some benchmarks (at least this month), but just because it may render 3% faster in the benchmark, are you really going to notice the difference?
[personal anecdote follows - feel free to stop reading now]
I worked in the control centre of a major electric utility for 29 years. The centre was manned 24x7 and I was on call frequently. I got many calls from the users of the type "the [name app] or network seems to be running slow, can you please have a look at it". Every test, even background tests which we ended up running automatically because of the complaints, showed no significant difference in the speed of the app or the network. It ended up being a perceptual problem. How slow the app or network seemed depended on how badly the operator wanted the results.
I think, for most users, perceived browser performance is affected by the users' expectations. Except for IE, of course, which just sucks. But I could be biased ;-)