Category: SEO Update

Google Stop Unsupported rules in Robots.txt

In last 25 years REP (Robots Exclusion Protocol) was only a de-facto standard. This had really a frustrating implementation sometimes. Now Google announced that we are open sourcing Google’s production robots.txt parser. So we can say that Google now officially allow that 1st Sep-2019 onwards GoogleBot will no longer obey a robots.txt directive relative to indexing.