Member Avatar for abhinay1111

i want to make my own search engine as google what should i do...

  • 5 Contributors
  • 7 Replies
  • 202 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by happygeek

Dani AI

Generated

A realistic starting note: building “a search engine like Google” is a very different goal from building a useful learning project or a focused site search. ’s point that a quick prototype is possible sits beside and ’s reminder about scale and resources; the standard textbook treatment is a good foundation for understanding why (indexing, retrieval models, evaluation). (nlp.stanford.edu)

Practical, low-risk path: pick a narrow scope (one site vertical or a topical crawl), seed it with an existing corpus rather than trying to crawl the entire web, and rely on mature engines for the hard parts. Apache Lucene / Solr (and the ecosystems built on Lucene) handle indexing and query processing; Common Crawl provides large public crawl dumps to seed data; Apache Nutch can be used when a custom crawler is needed. Start with a BM25 baseline for ranking, measure with held-out queries and nDCG/MRR, then add signals (anchor text, freshness, simple user-feedback features) and iterate. (lucene.apache.org)

Operational cautions and priorities: respect robots.txt and crawler politeness, store raw pages plus parsed text and link graphs so ranking can be re-run, and plan for sharding/replication as index size grows. Expect relevance tuning to be ongoing work — many lessons are about data quality and evaluation, not just clever algorithms. The links above point to practical tools and the canonical IR material for deeper study. (developer.mozilla.org)

Recommended Answers

All 7 Replies

Member Avatar for omol

1) Make a spider to crawl the web that follows every link it can find and scrapes important infomation about the linked page.
2) Log the results into a database. (You will need alot of storage.)
3) Create an algorithym that searches your database based on what a user entered into a search field and return the most logical reply.
4) Optimize 1, 2 and 3.
5) ....
6) Profit. goto 4)

Edited by omol because: n/a
Member Avatar for space2

Huh.. be serious man. This is not easy.

Member Avatar for Lusiphur

Huh.. be serious man. This is not easy.

Nobody said "this will be easy"... All Omol did was outline the base steps involved in response to a single (partial) sentence request for information.

In all actuality, if you're in a position to be asking "i want to make my own search engine as google what should i do" you don't currently have the skillset to do so. At least that's my opinion :twisted:

No faulting a person for being ambitious though.

Member Avatar for happygeek

Indeed. If it were that easy we'd all be doing it and enjoying our billion dollar payday.

Member Avatar for omol

Actually, it is easy to make a search engine. The hard part is making a good one.

I could put one together in an hour or so, a perl spider, a mysql database and a php front end.

It would be rubbish and take years to populate my database with anything useful. Its all but impossible to enter the search engine market now, mainly due to the amount of time the others have had to mine data.

That is, unless you can come up with something so awe inspiringly diffrent that works.

Member Avatar for Lusiphur

That is, unless you can come up with something so awe inspiringly diffrent that works.

My point exactly. Which is the part that isn't "easy" :twisted:

Member Avatar for happygeek

Quite. What isn't easy is making a "search engine as Google" - which implies one with the same depth and the same impact upon the world.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.