robot.txt tell the SE'S which page to crawl or which not. there are robot.txt generator sample robot.txt code are this
Sitemap: your site sitemap.xml
hope this helps
Be a part of the DaniWeb community
We're a friendly, industry-focused community of 1.19 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.