We want to design a search engine for news websites such as timesofindia.com and indianexpress.com, ie. download various article pages from these websites, index the pages, and answer search queries on the index.
Give a short pseudocode to find an appropriate crawling frequency -- you do not want to crawl too often because the website may not have changed, and you do not want to crawl too infrequently because your index would then be out of date. Assume that your crawling code looks as follows:
while(1) {
sleep(sleep_interval); // sleep for sleep_interval
crawl(website); // crawls the entire website
// returns a % value of difference between the latest and previous crawls of the website
diff = diff(currently_crawled_website, previously_crawled_website);
sleep_interval = infer_sleep_interval(diff, sleep_interval); // this is the method you have to write!
}
Give a pseudocode for the infer_sleep_interval method:
long sleep_interval infer_sleep_interval(int diff_percentage, long previous_sleep_interval) {
...
...
...
}
Design a method which adaptively alters the sleeping interval based on the update frequency of the website.
