I was just wondering how those of you with large communities handle site searches? With very large forums, even MySQL fulltext isn't good enough, nevermind the search feature built into software like vBulletin or phpBB.

Anyone play with www.sphinxsearch.com or something similar? Or are we all switching to Google's CBE nowadays?

I've been developing a custom search for my site using lucene. I've found lucene fast and very flexible. I have a small site so the flexibility is more important to me than the speed.

I messed around with sphinx a bit. There were a couple of restrictions on how the DB tables were organized that would have made it a pain for me to use (like all primary keys had to be ints). But it was very fast for me. But again, I have a very small site, so YMMV.

When you say "isn't good enough" do you mean with respect to speed or something else?

I mean with respect to speed and also searches tend to be super literal ... I've had the literal problem with Sphinx as well (i.e. it doesn't automatically do searches including/excluding prefixes/suffixes). I've heard really good things about Lucene but never tried it out myself.

Yeah, I agree with you that a lot of search implementations tend to be too literal.

Lucene is very nice in that respect. If you have a domain specific site, you can plug in your own aliases for various terms, as well as different stemming engines.

I'm using Roller for my blog engine, and since they're both written in java, it's a nice fit for me.