A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.
Other terms for Web crawlers are ants, automatic indexers, bots, or Web spiders, Web robots, or—especially in the FOAF community—Web scutters. This process is called Web crawling or spidering.
A Web crawler is one type of bot, or software agent.
In general, it starts with a list of URLs to visit, called the seeds.
As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
The bot will not pick up the changes in robots.txt immediately, usually it may take up to one week.
If you want the changes to be active immediately - please contact our email@example.com.
How to turn it off?
The quick way to prevent AhrefsBot from visiting your site is to put these two lines into the /robots.txt file on your server:
Alternatively, you can control the frequency of AhrefsBot visiting your site by specifying the minimum acceptable delay between two consecutive requests from our bot as follows (in robots.txt):
where the value of Crawl-Delay parameter is time in seconds (you can use fractional seconds too)
Read about robots.txt at www.robotstxt.org