0

We want to design a search engine for news websites such as timesofindia.com and indianexpress.com, ie. download various article pages from these websites, index the pages, and answer search queries on the index.

Give a short pseudocode to find an appropriate crawling frequency -- you do not want to crawl too often because the website may not have changed, and you do not want to crawl too infrequently because your index would then be out of date. Assume that your crawling code looks as follows:

while(1) {
     sleep(sleep_interval); // sleep for sleep_interval
     crawl(website);        // crawls the entire website

     // returns a % value of difference between the latest and previous crawls of the website
     diff = diff(currently_crawled_website, previously_crawled_website);

     sleep_interval = infer_sleep_interval(diff, sleep_interval); // this is the method you have to write!
}

Give a pseudocode for the infer_sleep_interval method:

long sleep_interval infer_sleep_interval(int diff_percentage, long previous_sleep_interval) {
    ...
    ...
    ...
}

Design a method which adaptively alters the sleeping interval based on the update frequency of the website.

Edited by pritaeas: Formatted on request

2
Contributors
1
Reply
2
Views
5 Years
Discussion Span
Last Post by Dani
0

Hi,

I'm onfused what your question is? Is this a homework assignment? Please show effort and post your question, along with what you've tried before, in the C++ forum.

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.