Viralogy

One of the area most commonly overlooked by eCommerce websites is the utilization of their robots.txt file. As eCommerce sites can be one of the most difficult types of websites for search engine crawlers to index, a robots.txt file can help crawlers find content to index without much trouble.

One of the biggest reasons many eCommerce sites fail to have one, is that they don’t understand what a robot.txt file actually does. In simplistic terms, a sites robot.txt file will guide search engines in the direction of content to index. For example, if you didn’t want search engines to index content within your ‘documents’ folder, you should place it within your robots.txt file. This will tell search engines that they shouldn’t index that content. As search engines have a difficult time in deciphering which content should and should not be indexed, giving the web crawlers a guideline as to what is off-limits can positively boost how your site is indexed.

Advantages of Using A Robot.txt File on Your eCommerce Site

1. Outdated content

Occasionally, there might be an item that you just don’t sell anymore, but don’t want to take off your site in case you decide to change your mind later on. As this could be a problem if Google crawlers index it, putting old pages and irrelevant content in your robots.txt file can help keep things organized and up to date.

2. Broken links

Some eCommerce sites have pages that they like to use as ‘test’ pages where they can figure out layouts and designs. These pages are meant to be for private use by the eCommerce team which calls for a placement within the robots.txt file. Every site should go through and look for these pages as you never want your customer to end up on a page they shouldn’t be.

3. Bandwidth

One big area that robots.txt files can be especially helpful is in keeping search engine crawlers out of your shopping cart. The reason for this is that sometimes the crawlers will try to index hundreds of items in the shopping cart. By keeping the search engines out of pages like this, you can save your bandwidth and use it for more important things.

If you aren’t sure if your site has a robots.txt file setup, be sure to check out eCommerce store grader, Grade Your Store, to see if your site gets a ‘passing’ grade!