Question on website's pages

Is there a way to check the total number of pages in a website?
How much crawl limit will you set before running a site audit using Semrush?

Lavender_Strings said:

How much crawl limit will you set before running a site audit using Semrush?

Click to expand...

I set the limit to 10k for my large sites, 5k for medium sites and less than 5k for even small sites. I have to use my quota carefully so I make sure that I am allotting it site wise.

Even if the site has too many pages it will be enough to know the top pages because they are the best performers.

sageshark said:

I set the limit to 10k for my large sites, 5k for medium sites and less than 5k for even small sites. I have to use my quota carefully so I make sure that I am allotting it site wise.

Even if the site has too many pages it will be enough to know the top pages because they are the best performers.

Click to expand...

Which crawl source should I choose for a perfect side audit?
Is it enough if i set the crawl source as 'Sitemaps on site' to only crawl the URLs that are found in the sitemap from the robots.txt file?

Yes, there are several ways to check the total number of pages on a website:

1. Manual Method: You can manually navigate through the website and count the number of pages by checking the sitemap, menu, and any other internal links that lead to unique pages. This method can be time-consuming and may not be accurate for large websites.
2. Google Search Operators: You can use Google search operators to estimate the number of indexed pages for a website. Simply enter "site:example.com" (replace example.com with the domain name of the website you want to check) in Google's search bar, and it will display the number of indexed pages for that website.
3. Web Crawling Tools: Many SEO tools, including Semrush, have web crawling capabilities that can automatically analyze a website and provide information on the total number of pages. These tools can save time and provide more accurate results for larger websites.

As for the crawl limit to set before running a site audit using Semrush or any other web crawling tool, it depends on the size of the website, the complexity of the pages, and the limitations of the tool itself. Semrush allows users to set their own crawl limits based on their subscription plan, ranging from 10,000 pages for the Pro plan to 1 million pages for the Business plan. It's important to carefully consider the size of the website and the limitations of the tool to set an appropriate crawl limit. Setting a crawl limit that is too high may result in longer processing times, increased resource usage, and incomplete or inaccurate audit results. It's recommended to start with a reasonable crawl limit and adjust it based on your specific needs and the capabilities of the tool being used.

To check the total number of pages on a website, you can use a tool like Screaming Frog or Ahrefs Site Explorer. These tools will crawl the site and provide you with a list of all the pages they found.