Member Avatar for dirtyhiphophead

i'm a computer science undergruaduate in a Nigerian university and somethin like google.com,is the big idea i have in mind.what programming language would i require(learn) to being such a structure(website)?
every bit of details,pls
thnaks in advance

  • 4 Contributors
  • 6 Replies
  • 215 Views
  • 2 Months Discussion Span
  • Latest Post Latest Post by michos

Dani AI

Generated

Good idea, — turning a Google‑scale system into a student project needs a tight scope and a clear MVP. Treat this as an information‑retrieval exercise: limit the crawl footprint (a single country TLD, a city, or a vertical like business listings), collect a few thousand pages, index core fields (URL, title, snippet, timestamp), and iterate on quality before thinking about scale or full monetization.

Core components to build and test: a polite crawler (honor robots.txt and rate limits), a parser/boilerplate stripper, an inverted index (term → postings with docID and positions), a lightweight query layer that supports tokenization and ranking, and a simple UI with query logging. For ranking, combine a lexical score (TF‑IDF/BM25 or similar) with simple signals: domain trust, freshness, and local relevance. Instrument everything (query logs, CTR on results) so tuning is data‑driven.

Operational tips and common traps: deduplicate near‑identical pages, detect and remove boilerplate, handle character encodings and basic language detection, and add simple spam filters to avoid low‑value results. Start on a single VPS and run crawler/indexer as separate worker jobs so they can be paused and resumed. For ads, begin with direct local listings and sponsored placements labelled clearly; measure conversions before automating auctions or complex targeting. Be mindful of legal/privacy considerations when collecting and serving data.

Minimal prototype workflow (pseudo‑code):

seed = load_seed_urls()
queue = seed
while queue:
  url = queue.pop()
  if allowed_by_robots(url):
    resp = fetch(url)
    if resp.status == 200 and is_html(resp):
      doc_id = store_doc(url, resp.title, resp.timestamp)
      text = extract_visible_text(resp)
      for pos, term in enumerate(tokenize(text)):
        add_posting(term, doc_id, pos)
      for link in extract_links(resp):
        if within_scope(link) and not_seen(link):
          queue.push(link)

A realistic short plan: 3–6 months to prototype (scope → crawler+parser → index+query → small UI → simple ads/analytics). The suggestions from and about focusing scope and search quality are especially relevant; ’s encouragement is a good reminder to collaborate — a small team shortens the timeline and covers more skills.

Recommended Answers

All 6 Replies

Member Avatar for nikesh.yadav

i appreciate ur thinking

Member Avatar for dirtyhiphophead

thanx bro,i'd appreciate any material or info u could help me with,i definitely can't do this alone but i goin to see to it that i becomes a reality

Member Avatar for omol

Ok, this is one hell of a task your thinking of undertaking.

Serach engine, you will need to write a spider to scrape sites that are submitted to your serach engine, a spider to find new sites. A mass of servers to hold all the data those spiders will bring back. I would recommend using a programming language that has good support for sockets and ports if you thinking of alowing more than http requests. I have made all my spiders in perl but nothing like a spider. Your spider will also have to work out what the site is about, how relvent the data on that page is and maybe some ranking system like googles Page Rank.

Displaying results, server side code that will search the database the infomation the spiders have baught back. One hell of an algerythum to make sure the search term used brings back infomation in a relavent order and only displays relavent searched items.

As for advertising? How do you mean, sponsoured links or a directory style of adverts?

Good luck!

Member Avatar for dirtyhiphophead

thanks omol for the response.yea someone else also 'prescribe' perl but all i've got now is the tutorials for dummies and still not even sufficient..if u could help with relevant materials,it would be greatly appreciated.

Member Avatar for omol

Ok I can recommend spidering hacks, am o'riely book ISBN 13: 9780596005771. The perl refrance material all available online. The google story.

Just a few to start. Books on server side communicationa and the break down of tcp / ip programming. I have a great book at home, when i get back home i will post up the isbn and name of it. Programming TCP / IP a complete reff off the top of my head.

Member Avatar for michos

Hi
Actually, we are in the BETA stages of something like that. Check out , it actually crawls the internet and does not need user input. Right now, we are crawling only African CCTLD's as when we crawl international TLD's like .com and .net return lots of junk so we fine tuning our scoring and filtering.
I would recommend the Java programing language as it is versatile and i think if you are not ready to sit down and code for a very long time, do it as a group project.

Good Luck

i'm a computer science undergruaduate in a Nigerian university and somethin like google.com,is the big idea i have in mind.what programming language would i require(learn) to being such a structure(website)?
every bit of details,pls
thnaks in advance

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.