Gone are the days when you have to reach for Python, Ruby, or some other language to accomplish your webpage scraping needs. These days, there is a robust set of JavaScript tools running on Node that help you harvest data from your favorite web pages.

Node and Xray have made web scraping a really simple affair. This video introduces you to the process of scraping all of the "a" tags off of a url and saving them to a .json file.

xray

Intro to Web Scraping with Node and X-ray

Node makes scraping images off the web extremely easy using a couple handy packages: Xray and Download. Simple scrape the img tag, grab all the src attributes, filter out images you don't want, then hand them over to Download to grab them. 

Web Scraping Images with Node, Xray, and Download

When web scraping, you'll often want to get more than just one page of data. Xray supports pagination by finding the "next" or "more" button on each page and cycling through each new page until it can no longer find that link. This lesson demonstrates how to paginate as well as more advanced selectors for when links are difficult to scrape.

Web Scraping with Pagination and Advanced Selectors

Many websites have more than just simple static content. Dynamic content which is rendered by JavaScript requires browser to be able to scrape data. This video demonstrates how to use Nightmare (which is a wrapper around Electron) to launch a url and scrape dynamic data.

nightmare

Scraping Dynamic JavaScript Websites with Nightmare

Many pages you'll want to grab data from will require you to log in first. This video demonstrates how to log in and grab whatever data off the page you will need.

..learn about web scraping in JavaScript

Joel Hooks

Course Content