Additionally, even reputable organizations ignore some commands that you can put in Robots.txt.
And malicious bots can and will ignore the robots.txt file.
Robots.txt cannot force a bot to follow its directives. That “participating” part is important, though. You can block bots entirely, restrict their access to certain areas of your site, and more. Robots.txt is the practical implementation of that standard – it allows you to control how participating bots interact with your site. The desire to control how web robots interact with websites led to the creation of the robots exclusion standard in the mid-1990s. But that doesn’t necessarily mean that you, or other webmasters, want bots running around unfettered. So, bots are, in general, a good thing for the Internet…or at least a necessary thing. These bots “crawl” around the web to help search engines like Google index and rank the billions of pages on the Internet. The most common example is search engine crawlers. Robots are any type of “bot” that visits websites on the Internet. Before we can talk about the WordPress robots.txt, it’s important to define what a “robot” is in this case.
0 kommentar(er)
