I'm trying to create a web crawler. I've read about web crawler's duty and about how it works and what he does.
But just need more information. Could you please tell me what does a web crawler can do? What kind of duty i can define for my web crawler? What can i ask it to do?
Jump to Post
For example we have two webpages called helloworld.html and hellopython.html
helloworld.html contains the text "Hello World"
if you look at its source it would have some lines like
and for hellopython.html that contains "Hello Python"
Now I can instruct my crawler to filter-out these webpages …
All 4 Replies
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.