Member Avatar for urtrivedi

I want to integrate local seach engine to my website. Which open source tool is best for it, I want to search pdf files also. Will google custom search do this job.

  • 3 Contributors
  • 2 Replies
  • 224 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by Manish02

Dani AI

Generated

Short, practical update for (and following up on and ):

If you need true “local” (self‑hosted) site search that can index PDFs — including private files behind authentication — use a self‑hosted search engine + a text‑extraction step. Google Programmable Search Engine (Custom Search) can surface PDFs if Google can crawl them, but it’s a hosted service that relies on Google’s index and therefore won’t run on your server or index files behind auth. (developers.google.com)

Recommended, pragmatic choices

  • Apache Solr — mature, scalable, and has built‑in support for extracting text from PDFs via Solr Cell / Apache Tika (good for complex requirements).
  • Elasticsearch — powerful, but check current licensing terms before using it as “open source.”
  • Meilisearch or Typesense — lightweight, developer‑friendly, great for fast site search with minimal ops.
  • Xapian, Sphinx or Whoosh — useful when you want an embeddable library or a small footprint.
    Pick Solr for heavy indexing and tunable relevance; pick Meili/Typesense for quick deployment and low operational cost. (solr.apache.org)

Minimal PDF indexing pipeline (practical)

  1. Extract text/metadata from each PDF (use Apache Tika).
  2. If PDFs are scanned images, run OCR (Tesseract) before/inside the Tika step.
  3. Send the cleaned JSON (text + metadata + path/ID) to your engine’s document API.
  4. Enforce access control in your application (search layer should respect your auth). To sanity‑check extraction, run a Tika server locally and try:
curl -T mydoc.pdf 'http://localhost:9998/tika' -H 'Accept: text/plain'

That returns plain text you can index. Watch for encoding/metadata issues and implement incremental updates (watcher/webhook) rather than full reindexes. For Tika OCR details and Tesseract, see the official docs. (cwiki.apache.org)

If the goal is “public site search” only and you don’t need private indexing, Google’s hosted option can work; otherwise prefer a self‑hosted Solr/Meili/Typesense pipeline so you control extraction, OCR, security, and relevance.

Recommended Answers

All 2 Replies

Member Avatar for pritaeas

Perhaps this answers your question:

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.