What\u2019s the secret to Google loving your website and indexing it? The secret is to steer crawlers to the most important and useful pages of your site and ignore those pages with little value to searchers. You may think that you want Google or Bing to index every URL of your website but you could be preventing it from crawling your most important pages by using up your \u2018crawl budget\u2019 on poor or irrelevant pages.<\/p>\n
What is \u2018crawl budget\u2019?<\/strong><\/p>\n When Google\u2019s crawlbot (or Bing\u2019s for that matter) comes to your website, it has a finite amount of resources to spend on accessing your website\u2019s pages; after all, Google has millions of other websites to crawl that day too. Why control access?<\/strong><\/p>\n Give important pages priority<\/strong><\/p>\n By controlling where Googlebot is allowed to crawl within your website, you are increasing the likelihood that important and valuable pages will be crawled every time Google visits your website.<\/p>\n Examples of these could be your product or service pages, blog post pages or even your contact details page. All of these are pages you want to have ranked highly in the search results so users can find this information quicker.<\/p>\n Ignore pages that don\u2019t need to be ranked<\/strong><\/p>\n There will be pages of your website that have no need to be indexed in the search results. These include pages that a user wouldn\u2019t typically look for in the search results but perhaps will browse to whilst on your website. These could be your privacy policy page, terms & conditions page or your blog tag or category pages.<\/p>\n How to help Googlebot access the right pages of your website<\/strong><\/p>\n There are a number of different ways in which you can help Googlebot access your website. The more of the following you can adjust or implement, the more control you should have over Googlebot or Bingbot.<\/p>\n Robots.txt file<\/strong><\/p>\n The first thing to look at is setting disallow rules in your robots file for all pages, folders or files types on your site that do not need to be crawled. Upon visiting a site, the first place a crawler will look at is your robots.txt file (provided it is always located at http:\/\/www.mydomain.com\/robots.txt). This will help indicate to various crawlers which parts of your website it should not attempt to crawl. You can set rules depending on what crawler bot you want to control.<\/p>\n NoIndex tags<\/strong><\/p>\n To help prevent certain pages from being indexed, it is also recommended that you add the NoIndex tag to the header code of those pages. Once added to a page, you should test these tags by doing a \u2018Fetch as Google\u2019 request on the URLs in Google Search Console.<\/p>\n Up-to-date XML sitemaps<\/strong><\/p>\n Although Google won\u2019t take your XML sitemap as a rule of which pages to crawl, it takes it as a hint \u2013 so make sure it\u2019s up to date to help reinforce which pages of your site it should be indexing. Fix internal links<\/strong><\/p>\n Googlebot will follow links it finds in your webpage content so make sure you aren\u2019t going to waste its time by letting it crawl links to missing pages. Use a crawling tool such as Screaming Frog\u2019s SEO Spider tool to find these broken internal links and fix them at the source.<\/p>\n Page load times<\/strong><\/p>\n Googlebot will need to load each of your pages when it visits them so by reducing the load time of each you can allow it to crawl and index more pages within the same overall time. There are a number of free tools available to help you analyse and improve site speed.<\/p>\n Site Structure<\/strong><\/p>\n A good site structure is an underrated method of helping Googlebot crawl your website a lot easier. Clearly categorising page content and not hiding pages away too deep in your site structure increases the likelihood they\u2019ll be found by the crawler.<\/p>\n The SEO Benefits<\/strong><\/p>\n If you\u2019ve managed to implement some or all of the above recommendations and tested them using the tools mentioned, you should begin to see some changes in crawl stats shown within Google Search Console.<\/p>\n Here we\u2019re looking for the number of pages crawled to be similar, or just over, the number of actual pages on your site in the first blue graph. The reduction in kilobytes downloaded (in red) should mimic the reduction in pages crawled if you previously had lots of pages being crawled.<\/p>\n Below is an example of a site with a significant number of URL parameter issues in which Googlebot crawled up to 12,000 URLs when in fact there were just a few hundred actual pages of the site. Through the application of URL parameter rules and the other factors mentioned above, the number of pages crawled became much more consistent and realistic.<\/p>\n Google is crawling your useful pages each time, the rankings of your pages will be more likely to change frequently, and most likely for the better. Fresh content will get indexed and ranked a lot quicker and time won\u2019t be wasted from your \u2018crawl budget\u2019.<\/p>\n
\nYour crawl budget can be determined by the value or quality of your website already, including but not limited to the quality of backlinks to your website. I won\u2019t speculate on more factors here, but there are some good research pieces on the web where people have tried to identify these factors.<\/p>\n
\nRemove any old pages from your site and add any new pages.<\/p>\n