SEO6 months ago
Google Confirms Robots.txt Can’t Prevent Unauthorized Access
Google’s Gary Illyes confirmed a common observation that robots.txt has limited control over unauthorized access by crawlers. Gary then offered an overview of access controls that...