Member Avatar for mayank.15

Hey Everyone!
I am a newcomer to Lucene and Nutch usage. I crawled a few webpages to create an Index using Nutch. Now instead of using Tomcat to search through this created index, I wanted to search through this index using a typical search in Lucene. Now, I realized that Nutch doesnt store its index the same way Lucene does when in Indexes documents. Could someone help me out as to how to go about searching this Index created with nutch?

  • 3 Contributors
  • 3 Replies
  • 103 Views
  • 2 Years Discussion Span
  • Latest Post Latest Post by b.s.herbert

Dani AI

Generated

Context from the thread: crawled pages with Nutch and wants to run plain Lucene queries against the result. The missing piece in many setups is whether a filesystem Lucene index actually exists (Nutch can leave data in crawl DB / segments or write a Lucene index depending on the step and version). The practical options are: produce a Lucene index from Nutch, or export/push the data into a search server (Solr/Elasticsearch) and query that. See Nutch indexing docs for the exact indexing step and plugins: Apache Nutch and .

Checklist and troubleshooting (common gaps seen in this thread)

  • Confirm what files are present: a true Lucene index will contain segment files (e.g., segments_*) in an index directory. If only crawldb and segments exist, run Nutch’s indexer/export step first.
  • Version compatibility: use Lucene libraries that match (or are compatible with) the Lucene version used when Nutch wrote the index. Mismatched Lucene versions are a common cause of “cannot open index” errors.
  • Field names and analysis: Nutch’s index uses its own field names (content, title, url, etc.). Query against the correct field and use an Analyzer that matches how the text was indexed.
  • If the index lives in HDFS or was produced by a MapReduce job, copy/export it to local FS before opening with Lucene APIs, or use the appropriate Nutch exporter to Solr.

Minimal Java example to open and query a filesystem Lucene index (API details vary by Lucene version):

Directory dir = FSDirectory.open(Paths.get("/path/to/nutch-index"));
try (IndexReader reader = DirectoryReader.open(dir)) {
  IndexSearcher s = new IndexSearcher(reader);
  Analyzer a = new StandardAnalyzer();
  Query q = new QueryParser("content", a).parse("search terms");
  TopDocs td = s.search(q, 10);
  for (ScoreDoc sd : td.scoreDocs) {
    Document d = s.doc(sd.doc);
    System.out.println(d.get("title") + " - " + d.get("url"));
  }
}

If integrating from .NET (as noted), use the Lucene.NET port but verify index-format compatibility between the .NET port and the original Lucene version. Tools such as Luke are useful to inspect an index before writing code. Additional reading: Apache Lucene and the Luke repository (luke on GitHub).

Member Avatar for InsightsDigital

I have never heard of Lucerne and Nutch. What is it? Is it a city in France?

Member Avatar for mayank.15

If that is the case then, lucene is an apache project for indexing and searching and nutch is a webcrawler.

Member Avatar for b.s.herbert

There is a new .NET port of Lucene called Lucene.NET.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.