The ability to reply to discussions is limited to PRO members. Want to join in the discussion? Click here to subscribe now.

Web Scraping with Pagination and Advanced Selectors

Web Scraping with Pagination and Advanced Selectors

3:29
When web scraping, you'll often want to get more than just one page of data. Xray supports pagination by finding the "next" or "more" button on each page and cycling through each new page until it can no longer find that link. This lesson demonstrates how to paginate as well as more advanced selectors for when links are difficult to scrape.
Watch this lesson now
Avatar
egghead.io

When web scraping, you'll often want to get more than just one page of data. Xray supports pagination by finding the "next" or "more" button on each page and cycling through each new page until it can no longer find that link. This lesson demonstrates how to paginate as well as more advanced selectors for when links are difficult to scrape.

Avatar
Thomas

What is the app/development tool you are using in the screen scraping (Xray) examples? It looks like some advanced version of Node REPL.

In reply to egghead.io
Avatar
John

I use WebStorm. It's showing a side-by-side pane of Code on the left and the output.json on the right.

In reply to Thomas
Avatar
Thomas

I have WebStorm. Evidently I have figure out all its features yet...

In reply to John
Avatar
Jim

Could you please give an example of how to use Composition and pagination to another site? I cant get the documented feature to work: https://www.npmjs.com/package/x-ray#paginating-to-another-site

HEY, QUICK QUESTION!
Joel's Head
Why are we asking?