If you get a chance try to attend the Twitter chat sessions run by SEMrush, the folk behind the well-known marketing tool. It’s a great way to view crowdsourced Q&As from the marketing community.
Last week’s session was run by Ian Laurie, a veteran of the industry. You can see one of the exchanges below – view the whole discussion by viewing the questions (Q1-Q6) on the @semrush feed against the answers tagged with #semrushchat.
— SEMrush (@semrush) May 13, 2015
A6 And finally: Missing a Disallow / in a robots.txt. Headdesk. Fixed by, well, fixing it. #semrushchat
— Ian Lurie (@portentint) May 13, 2015
Disallowing access to Google, via the robots.txt is a very common problem for SEOs. Most developers ensure their working versions are excluded from Google, to prevent duplicate content leaking online. It’s easy to see how their version of the robots.txt can be copied over the live version – quickly resulting in Google deindexing the website in question.
Once this deindexing starts, the clock is ticking on getting the robots.txt file fixed. Dropping out of the search index for a major brand site is an expensive lesson in the importance of this file.
To keep on top of this issue, I recommend signing up to the free service at ChangeDetection.com. Enter the URL of each robots.txt file you wish to monitor and they will email you as soon as changes are detected. In addition, they will store a log of each change so you have a record to check for your forensic SEO investigations (useful when developers have caused and then fixed this issue, but not let anyone know).
Mid-2004 Google Webmaster Tools added their robots.txt testing tool that generated email alerts and logged changes to the file. When this was released it became an additional check for me, but I kept using ChangeDetection on the basis that multiple checks wouldn’t hurt – and in practice GWT access is often setup for agencies with a generic email address, so alerts would not come to my personal inbox.
When IFTTT came on the scene, this warning system got even better. With a few clicks, you can set up ChangeDetection and GWT emails to come to your phone as SMS (the IFTTT recipe for email to SMS here). Well that’s my process, if you have any improvements or tips then please share below.
The post How To Set Up Email & SMS Alerts On Robots.txt Changes appeared first on Nick Wilsdon.