We’re powered by seriously big data

Ahrefs collects, processes, and stores large amounts of data for search marketing professionals.

Similar to search engines, we crawl the entire web 24/7 and store petabytes (1PB = 1000TB) of information about live websites – like how they link to each other and what keywords they rank for in search results.

We're also really proud of our backend infrastructure. Since no existing solution could keep up with the volume of data we operate, most of our software infrastructure was built in-house.

3400
Servers
692
K
CPU cores
5
PB
RAM
33
PB
HDD
484
PB
SSD

$900
M
The amount we’d have to spend on the cloud in 3 years without our own infrastructure.
Watch reel

Most active web crawler in the SEO industry

AhrefsBot is the most active web crawler in the “Search Engine Optimization” category on Cloudflare Radar. It’s also the 3rd most active web crawler in the world.

Every minute, we crawl
5
M
Pages
There are
170
T
Rows in our key-value database
/

Rank

Bot

Owner

Category

1

GoogleBot

Google

Search Engine Crawler

2

BingBot

Microsoft

Search Engine Crawler

3

DataDog Synthetic Bot

DataDog

Monitoring & Analytics

4

AhrefsBot

Ahrefs

Search Engine Optimization

5

Qualys

Qualys

Search Engine Crawler

6

YandexBot

Yandex

Search Engine Crawler

7

PetalBot

Huawei

Search Engine Crawler

8

Google AdsBot

Google

SAdvertising & Marketing

9

Google Image Proxy

Google

Page Preview

10

Google Images

Google

Search Engine Crawler

Bots ranked by number of requests. Data supplied as of April, 2023


Content index

Our content index powers Content Explorer, which is basically a content search engine.

Every 24 hours, we discover
10
M
New pages
Every 24 hours, we update metrics for
300
M
Pages

Total content index size

15.2B

Pages


Keyword index

This powers Keywords Explorer, our keyword research tool for finding what your prospective customers are searching for online.

216
Locations

Total keyword index size

12.0B

Filtered keywords

74.0B

Keywords ever seen

1.0BUnited States
438MUnited Kingdom
383MIndia
279MRussian Federation
276MCanada
272MGermany
244MBrazil
215MJapan
208MFrance
199MAustralia
177MIndonesia
166MPhilippines
163MSpain
160MMexico
148MItaly
146MTurkey
133MNetherlands
131MVietnam
120MPoland
120MKorea
113MUkraine
110MArgentina
105MMalaysia
94MColombia
93MTaiwan
85MThailand
84MPakistan
79MSouth Africa
78MSwitzerland
77MBelgium
76MEgypt
74MIran
73MSingapore
71MSweden
70MPeru
66MChile
65MAustria
65MUnited Arab Emirates
61MSaudi Arabia
59MBangladesh
58MIreland
55MRomania
52MNew Zealand
51MPortugal
47MHong Kong
46MMorocco
46MNorway
45MCzech Republic

1.0BUnited States
438MUnited Kingdom
383MIndia
279MRussian Federation
276MCanada
272MGermany
244MBrazil
215MJapan
208MFrance
199MAustralia
177MIndonesia
166MPhilippines
163MSpain
160MMexico
148MItaly
146MTurkey
133MNetherlands
131MVietnam
120MPoland
120MKorea
113MUkraine
110MArgentina
105MMalaysia
94MColombia
93MTaiwan
85MThailand
84MPakistan
79MSouth Africa
78MSwitzerland
77MBelgium
76MEgypt
74MIran

1.0BUnited States
438MUnited Kingdom
383MIndia
279MRussian Federation
276MCanada
272MGermany
244MBrazil
215MJapan
208MFrance
199MAustralia
177MIndonesia
166MPhilippines
163MSpain
160MMexico
148MItaly
146MTurkey


Organic search traffic index

Our database of search queries and the amount of search traffic they send to top-ranking pages. It powers search traffic estimations in Site Explorer.

189
Locations
583
M
Keywords
154MUnited States
31MIndia
26MJapan
24MUnited Kingdom
24MFrance
22MGermany
22MBrazil
19MCanada
19MSpain
17MRussian Federation
16MAustralia
15MItaly
13MMexico
13MNetherlands
12MIndonesia
11MTurkey
9.3MPoland
6.8MThailand
6.6MColombia
6.5MPhilippines
6.3MArgentina
5.7MVietnam
4.7MUkraine
4.7MSweden
3.8MTaiwan
3.4MMalaysia
3.2MChile
3.0MSouth Africa
3.0MDenmark
2.9MNorway
2.8MPeru
2.7MSaudi Arabia
2.7MAustria
2.6MPakistan
2.6MEgypt
2.5MCzech Republic
2.4MNew Zealand
2.4MKorea
2.3MBelgium
2.1MUnited Arab Emirates
2.1MPortugal
2.0MRomania
1.9MFinland
1.9MHungary
1.9MSingapore
1.9MGreece
1.9MSwitzerland
1.8MHong Kong

154MUnited States
31MIndia
26MJapan
24MUnited Kingdom
24MFrance
22MGermany
22MBrazil
19MCanada
19MSpain
17MRussian Federation
16MAustralia
15MItaly
13MMexico
13MNetherlands
12MIndonesia
11MTurkey
9.3MPoland
6.8MThailand
6.6MColombia
6.5MPhilippines
6.3MArgentina
5.7MVietnam
4.7MUkraine
4.7MSweden
3.8MTaiwan
3.4MMalaysia
3.2MChile
3.0MSouth Africa
3.0MDenmark
2.9MNorway
2.8MPeru
2.7MSaudi Arabia

154MUnited States
31MIndia
26MJapan
24MUnited Kingdom
24MFrance
22MGermany
22MBrazil
19MCanada
19MSpain
17MRussian Federation
16MAustralia
15MItaly
13MMexico
13MNetherlands
12MIndonesia
11MTurkey


Backlink index

The world's largest index of live backlinks, updated with fresh data every 15–30 minutes. This index powers backlink data in Site Explorer, and is used to calculate our proprietary metrics: Ahrefs Rank, Domain Rating and URL Rating.

External backlinks history

35.0T

Records

Pages in index
334.3
B
Domains (post-vetting)
207.4
M
Internal backlinks
22.7
T

© 2024 Ahrefs Pte. Ltd. (201227417H) 16 Raffles Quay, #33-03 Hong Leong Building, Singapore 048581