Are the crawled URLs stored in the database then they are traversed in BFS/DFS manner .... or is it something else ?

Please Help!

Member Avatar

@apanimesh061

Are the crawled URLs stored in the database then they are traversed in BFS/DFS manner .... or is it something else ?

You are asking about a theory of how it works?

Read this:

http://activities.tjhsst.edu/sct/lectures/1112/bfsdfs100711.pdf

If you want to try how crawl works then read this:

https://developers.google.com/search-appliance/documentation/68/admin_crawl/database_crawl_serve

https://developers.google.com/search-appliance/documentation/68/help_gsa/crawl_database