Obviously you can look at their own sites/portfolio to see if they have a good eye for colour, but you could ask:
- how they would improve the site structure
- how that would make it easier for people to find things
and umm I've run out of ideas!

edit: also could ask what software they use - do they use the professional stuff like Photoshop and Dreamweaver, or something else - however that won't tell you much about their abilities. You can only tell that by looking at their past work or samples.

I think what software they use is important and it is very important that they know how to make your site easier to use.

Just and idea:

Show them your site, leave them for five minutes and tell them to look at it and decide five things they would change to make it look better and easier for the user. I think this will give you a good idea of how good they are and what they are capable of doing because they are unlinkely to say somehting they can't do!

But as I said that's just an idea, maybe it is a bit harsh on the Web Designer but they want to work for you!

At the end of the day their ability with that software is far more important than the software itself. The decision has to rely on the visible skills and natural talent they have with the tools. After all, it doesn't matter if your door locks were fitted with a chisel or a router, but the skills of the person using it will affect how well it works for you.