Page Crawling Issue

[MasoodDidThat]

My website is an ecommerce website and it has more than 500 pages, the SEMRush is only crawling 4 pages and i checked on google search conscole and all the pages there are indexed. My internal linking is also good. Help me understand why semrush is not crawling all my pages?

[Harry Brook]

It sounds like SEMrush might be facing restrictions while crawling your site. Check if your crawl limit is too low, ensure robots.txt isn't blocking pages, and verify your XML sitemap is updated. Also, if your site relies on JavaScript, SEMrush may struggle to crawl it properly. Try adjusting these settings and re-run the crawl.

[Dani]

As Harry points out, my first guess would be to check robots.txt and ensure you aren't blocking any pages from SEMRush. Also make sure you aren't using a CDN or proxy like Cloudflare that is blocking it from their side.

[naps2]

If you are encountering a page crawling issue, you should check your robots.txt file to ensure that important pages are not blocked. You should also check Google Search Console for crawl errors or indexing issues.

[Olu_610]

I have encountered similar issue. SEMrush appears to focus on crawling only the most prestigious pages because it operates on a 500-page limit. Using the SEMrush API alongside adjusting crawl settings will help raise the crawl depth.

[ashleydent4u]

If Google Search Console shows your pages are indexed but SEMrush is only crawling a few, it could be due to your crawl settings or restrictions. Check your robots.txt file and meta tags to ensure SEMrush isn’t blocked from crawling your site. Also, look at your crawl budget settings in SEMrush—if it’s set too low, it won’t scan all your pages. Another issue could be your website’s server response time; slow-loading pages might not be fully crawled. Try increasing the crawl limit in SEMrush settings and re-run the audit to see if it picks up more pages.

[MasoodDidThat]

Found the solution

I tried a method where I went to the Site Audit settings on Semrush, uploaded a .txt file containing all the URLs I wanted to get indexed, and then re-ran the campaign. It worked.

[jonathannweyer]

If SEMRush is only crawling a few pages, it might be due to the crawl settings or limits in your SEMRush account. Check if there are any crawl restrictions set up and make sure your robots.txt file isn’t blocking any important pages.

[Cliproxy]

SEMrush does not crawl all of your website pages—it only indexes the top 100 ranking pages for specific keywords on Google. If one of your pages ranks within the top 100 for a given keyword, SEMrush will display that corresponding page. If you want to improve your website's SEO performance as reflected in SEMrush, I believe the key lies in enhancing the overall quality of your content, product or service offerings, website design, and even article creation. Continuously improving the quality of your services is essential.

[Dani]

Honestly, I wouldn't worry about it as long as Google is crawling and indexing all your pages.