{"id":17473,"date":"2017-10-16T23:45:50","date_gmt":"2017-10-17T07:45:50","guid":{"rendered":"https:\/\/ahrefs.com\/blog\/?p=17473"},"modified":"2023-07-19T03:32:33","modified_gmt":"2023-07-19T08:32:33","slug":"web-scraping-for-marketers","status":"publish","type":"post","link":"https:\/\/ahrefs.com\/blog\/web-scraping-for-marketers\/","title":{"rendered":"6 Actionable Web Scraping Hacks for White Hat Marketers"},"content":{"rendered":"
Have you ever used a program like Screaming Frog<\/a> to extract metadata (e.g. title\/description\/etc.) from a bunch of web pages in bulk? <\/div>\n

If so, you\u2019re already<\/em> familiar with web scraping.<\/p>\n

But, while this can certainly be useful, there\u2019s much more to web scraping than grabbing a few title tags\u2014it can actually be used to extract any<\/em> data from any<\/em> web page in seconds.<\/p>\n

The question is: what<\/em> data would you need to extract and why<\/em>?<\/p>\n

In this post, I\u2019ll aim to answer these questions by showing you 6 web scraping hacks:<\/p>\n

    \n
  1. How to find content \u201cevangelists\u201d in website comments<\/li>\n
  2. How to collect prospects\u2019 data from \u201cexpert roundups\u201d<\/li>\n
  3. How to remove junk \u201cguest post\u201d prospects<\/li>\n
  4. How to analyze performance of your blog categories<\/li>\n
  5. How to choose the right content for Reddit<\/li>\n
  6. How to build relationships with those who love your content<\/li>\n<\/ol>\n

    I\u2019ve also automated as much of the process as possible to make things less daunting for those new to web scraping.<\/p>\n

    But first, let\u2019s talk a bit more about web scraping and how it works.<\/p>\n

    A basic introduction to web scraping<\/h2>\n

    Let\u2019s assume that you want to extract the titles from your competitors\u2019 50 most recent blog posts.<\/p>\n

    You could visit each website individually, check the HTML, locate the title tag, then copy\/paste that data to wherever you needed it (e.g. a spreadsheet).<\/p>\n

    \"view<\/p>\n

    But, this would be very<\/em> time-consuming and boring.<\/p>\n

    That\u2019s why it\u2019s much easier to scrape the data we want using a computer application (i.e. web scraper).<\/p>\n

    In general, there are two ways to \u201cscrape\u201d the data you\u2019re looking for:<\/p>\n

      \n
    1. Using a path-based system (e.g. XPath\/CSS selectors);<\/li>\n
    2. Using a search pattern (e.g. Regex)<\/li>\n<\/ol>\n

      XPath\/CSS (i.e. path-based system) is the best way to scrape most types of data.<\/p>\n

      For example, let\u2019s assume that we wanted to scrape the h1<\/em> tag<\/a> from this document:<\/p>\n

      \"HTML<\/p>\n

      We can see that the h1<\/em> is nested in the body <\/em>tag, which is nested under the html<\/em> tag\u2014here\u2019s how to write this as XPath\/CSS:<\/p>\n