Member Avatar for Dani

I was just wondering how those of you with large communities handle site searches? With very large forums, even MySQL fulltext isn't good enough, nevermind the search feature built into software like vBulletin or phpBB.

Anyone play with www.sphinxsearch.com or something similar? Or are we all switching to Google's CBE nowadays?

  • 2 Contributors
  • 3 Replies
  • 92 Views
  • 14 Hours Discussion Span
  • Latest Post Latest Post by brianlevine

Dani AI

Generated

Search for large, active forums needs more than a simple DB fulltext: predictable latency, useful fuzzy/partial matches, tunable relevance, and an index/update model that fits your write volume. flagged the core problems; 's Lucene experiments point toward using a purpose-built search layer rather than relying on the forum software alone.

Two mature self-hosted choices are Solr and Elasticsearch. Both expose HTTP APIs you can call from PHP, give language analyzers, synonym/stemming support, faceting, and relevance controls. If running and managing a cluster is a burden, commercial search-as-a-service providers remove most ops work at the cost of recurring fees and possible privacy tradeoffs.

Practical checklist to move from DB search to a search engine:

  • Define what matters: index title, body, tags, author, timestamps, and any signals (views, replies, upvotes) you can use for boosting.
  • Pick analyzers: language tokenizers, lowercase/normalization, stopwords, synonyms, and edge-ngrams for autocomplete.
  • Indexing strategy: push updates via background jobs/queues (do not block web requests); support delta/near-real-time indexing to keep search fresh.
  • Relevancy tuning: start with BM25, add field boosts and recency/popularity boosts, then tune with a representative query set and small A/B tests.
  • Performance and ops: monitor P95/P99 query latency, indexing lag, memory/IO; plan shards/replicas and warm caches.
  • Privacy and scope: avoid sending private or gated content to third-party services without consent.

Start with a small prototype: index a sample of posts, run your top 50 real queries, measure latency and relevance, and iterate. For vBulletin/phpBB integration the simplest route is a tiny sync worker that posts create/update/delete events into the search API; rely on built-in connectors only if they match your indexing and relevancy needs.

Recommended Answers

All 3 Replies

Member Avatar for brianlevine

I've been developing a custom search for my site using lucene. I've found lucene fast and very flexible. I have a small site so the flexibility is more important to me than the speed.

I messed around with sphinx a bit. There were a couple of restrictions on how the DB tables were organized that would have made it a pain for me to use (like all primary keys had to be ints). But it was very fast for me. But again, I have a very small site, so YMMV.

When you say "isn't good enough" do you mean with respect to speed or something else?

Member Avatar for Dani

I mean with respect to speed and also searches tend to be super literal ... I've had the literal problem with Sphinx as well (i.e. it doesn't automatically do searches including/excluding prefixes/suffixes). I've heard really good things about Lucene but never tried it out myself.

Member Avatar for brianlevine

Yeah, I agree with you that a lot of search implementations tend to be too literal.

Lucene is very nice in that respect. If you have a domain specific site, you can plug in your own aliases for various terms, as well as different stemming engines.

I'm using Roller for my blog engine, and since they're both written in java, it's a nice fit for me.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.