We’re powered by seriously big data

Ahrefs collects, processes, and stores large amounts of data for search marketing professionals.

Similar to search engines, we crawl the entire web 24/7 and store petabytes (1PB = 1000TB) of information about live websites – like how they link to each other and what keywords they rank for in search results.

We're also really proud of our backend infrastructure. Since no existing solution could keep up with the volume of data we operate, most of our software infrastructure was built in-house.

3500
Servers
657
K
CPU cores
4
PB
RAM
33
PB
HDD
424
PB
SSD

$900
M
The amount we’d have to spend on the cloud in 3 years without our own infrastructure.
Watch reel

Most active web crawler in the SEO industry

AhrefsBot is the most active web crawler in the “Search Engine Optimization” category on Cloudflare Radar. It’s also the 3rd most active web crawler in the world.

Every minute, we crawl

There are

5
M
170
T

Pages

Rows in our key-value database

/

Rank

Bot

Owner

Category

1

GoogleBot

Google

Search Engine Crawler

2

BingBot

Microsoft

Search Engine Crawler

3

DataDog Synthetic Bot

DataDog

Monitoring & Analytics

4

AhrefsBot

Ahrefs

Search Engine Optimization

5

Qualys

Qualys

Search Engine Crawler

6

YandexBot

Yandex

Search Engine Crawler

7

PetalBot

Huawei

Search Engine Crawler

8

Google AdsBot

Google

SAdvertising & Marketing

9

Google Image Proxy

Google

Page Preview

10

Google Images

Google

Search Engine Crawler

Bots ranked by number of requests. Data supplied as of April, 2023


Content index

Our content index powers Content Explorer, which is basically a content search engine.

Every 24 hours, we discover
10
M
New pages
Every 24 hours, we update metrics for
300
M
Pages

Total content index size

14.7B

Pages


Keyword index

This powers Keywords Explorer, our keyword research tool for finding what your prospective customers are searching for online.

216
Locations

Total keyword index size

16.2B

Keywords

1.4BUnited States
687MUnited Kingdom
610MIndia
532MBrazil
513MRussian Federation
465MGermany
444MCanada
394MFrance
379MIndonesia
344MJapan
342MAustralia
309MSpain
308MPhilippines
304MMexico
303MItaly
300MTurkey
296MVietnam
253MPoland
246MUkraine
243MKorea
241MNetherlands
223MArgentina
207MMalaysia
205MThailand
197MTaiwan
188MColombia
184MPakistan
174MEgypt
166MSingapore
163MBangladesh
162MSouth Africa
151MSweden
150MSwitzerland
150MUnited Arab Emirates
140MBelgium
124MPeru
123MSaudi Arabia
122MRomania
121MIreland
116MChile
115MAustria
113MMorocco
111MIran
110MNew Zealand
109MPortugal
105MCzech Republic
103MAlgeria
99MNorway

1.4BUnited States
687MUnited Kingdom
610MIndia
532MBrazil
513MRussian Federation
465MGermany
444MCanada
394MFrance
379MIndonesia
344MJapan
342MAustralia
309MSpain
308MPhilippines
304MMexico
303MItaly
300MTurkey
296MVietnam
253MPoland
246MUkraine
243MKorea
241MNetherlands
223MArgentina
207MMalaysia
205MThailand
197MTaiwan
188MColombia
184MPakistan
174MEgypt
166MSingapore
163MBangladesh
162MSouth Africa
151MSweden

1.4BUnited States
687MUnited Kingdom
610MIndia
532MBrazil
513MRussian Federation
465MGermany
444MCanada
394MFrance
379MIndonesia
344MJapan
342MAustralia
309MSpain
308MPhilippines
304MMexico
303MItaly
300MTurkey


Organic search traffic index

Our database of search queries and the amount of search traffic they send to top-ranking pages. It powers search traffic estimations in Site Explorer.

189
Locations
621
M
Keywords
165MUnited States
32MIndia
27MJapan
26MGermany
25MFrance
24MUnited Kingdom
21MSpain
21MCanada
21MBrazil
19MAustralia
17MItaly
17MRussian Federation
15MIndonesia
14MNetherlands
13MTurkey
12MMexico
11MPoland
7.1MVietnam
7.0MThailand
6.5MUkraine
6.2MColombia
6.0MArgentina
5.8MPhilippines
5.5MSweden
4.5MTaiwan
3.3MAustria
3.3MNorway
3.2MMalaysia
3.2MDenmark
3.0MChile
2.9MSouth Africa
2.9MSaudi Arabia
2.8MNew Zealand
2.6MCzech Republic
2.5MBelgium
2.5MEgypt
2.5MPeru
2.4MPakistan
2.3MGreece
2.2MHong Kong
2.2MFinland
2.2MRomania
2.1MKorea
2.1MHungary
2.0MPortugal
2.0MSwitzerland
2.0MUnited Arab Emirates
2.0MIsrael

165MUnited States
32MIndia
27MJapan
26MGermany
25MFrance
24MUnited Kingdom
21MSpain
21MCanada
21MBrazil
19MAustralia
17MItaly
17MRussian Federation
15MIndonesia
14MNetherlands
13MTurkey
12MMexico
11MPoland
7.1MVietnam
7.0MThailand
6.5MUkraine
6.2MColombia
6.0MArgentina
5.8MPhilippines
5.5MSweden
4.5MTaiwan
3.3MAustria
3.3MNorway
3.2MMalaysia
3.2MDenmark
3.0MChile
2.9MSouth Africa
2.9MSaudi Arabia

165MUnited States
32MIndia
27MJapan
26MGermany
25MFrance
24MUnited Kingdom
21MSpain
21MCanada
21MBrazil
19MAustralia
17MItaly
17MRussian Federation
15MIndonesia
14MNetherlands
13MTurkey
12MMexico


Backlink index

The world's largest index of live backlinks, updated with fresh data every 15–30 minutes. This index powers backlink data in Site Explorer, and is used to calculate our proprietary metrics: Ahrefs Rank, Domain Rating and URL Rating.

External backlinks history

35.0T

Records

Pages in index
373.2
B
Domains (post-vetting)
213.4
M
Internal backlinks
25.3
T
Language
Extra tools & features
Product
Company
Resources
Social

© 2024 Ahrefs Pte. Ltd. (201227417H) 16 Raffles Quay, #33-03 Hong Leong Building, Singapore 048581