Learn Elasticsearch, Reactfire and Xray

The next web scraper. See through the <html> noise.

xray

When web scraping, you'll often want to get more than just one page of data. Xray supports pagination by finding the "next" or "more" button on each page and cycling through each new page until it can no longer find that link. This lesson demonstrates how to paginate as well as more advanced selectors for when links are difficult to scrape.

Web Scraping with Pagination and Advanced Selectors

Node makes scraping images off the web extremely easy using a couple handy packages: Xray and Download. Simple scrape the img tag, grab all the src attributes, filter out images you don't want, then hand them over to Download to grab them. 

Web Scraping Images with Node, Xray, and Download

Node and Xray have made web scraping a really simple affair. This video introduces you to the process of scraping all of the "a" tags off of a url and saving them to a .json file.

Intro to Web Scraping with Node and X-ray

 We map over the initial set of results then use `Promise.all` to efficiently fetch the details of each pokemon rendered in the table. This allows us to add more columns to our pokemon table.

Fetch and display row details with Promise.all in Tanstack Table

map over the initial set of results then use `Promise.all` to efficiently fetch the details of each pokemon rendered in the tanstack table

Create a custom hook to fetch remote data by passing a promise to the React 19 `use` hook. Then render this data in Tanstack Table by memoizing the results and passing them to `useReactTable`.

Render remote data in a Tanstack Table with React 19's use hook and a promise

Create a custom hook using React 19’s use to fetch data with a promise. Render it in a Tanstack Table by memoizing and using useReactTable.

You know when you're deep in the code trenches and need someone to challenge your assumptions? That's exactly what this is about.

Instead of waiting for someone to poke holes in your master plan, you can paste that strategy straight into AI Studio and let it pressure test your ideas. We're talking about using this tool to validate everything from your feature proposals to your ambitious refactoring strategies.

The real magic happens when you hit that microphone button. Now you're in a real-time verbal sparring match. It's like rubber duck debugging but the duck actually pushes back and makes you defend every architectural decision you've made.

Here's the workflow:
1. Drop your master plan into the system instructions
2. Tell the AI to challenge every assumption
3. Hit that mic and defend your choices like you mean it
4. Watch your plan either emerge stronger or get rightfully dismantled

This approach is perfect when you're:
- Stuck in your own head with a complex problem
- Need to validate if your approach holds water
- Want to catch edge cases before they catch you
- Looking for opportunities you might have missed because you're too close to the code

It's like having a senior dev on call 24/7, ready to challenge your ideas and push you to think deeper about your solutions.

Now go forth and get your ideas challenged. Your projects will thank you for it.


An Uncomfortable and Challenging Conversation with AI Studio

In this lesson, you'll learn how to effectively brainstorm your project ideas before working with AI agents. We'll cover:

1. The importance of having a clear plan before starting work with AI agents
2. Using traditional brainstorming techniques to gather your thoughts
3. Leveraging video recording to capture your brainstorming session
4. How to review and analyze websites for inspiration
5. Documenting likes, dislikes, and potential features for your project
6. Using AI tools (like AI Studio) to organize and process your recorded brainstorming session

The lesson emphasizes a modern approach to brainstorming that combines traditional techniques with video recording, making it easier to later process your ideas through AI tools. This method helps ensure that no ideas are lost and that your thoughts can be effectively organized by AI for future reference.

How to Brainstorm Your Project Features to Prepare for AI Agents

Many people share .cursorrules files, but trying to craft your own from scratch is a huge pain. Instead, this lesson shows you how to grab ALL community-created .cursorrules files to build a custom rule set for your projects. We'll use Gitingest and Google AI Studio to gather these examples and generate an awesome starting point customized to your specific project.

Generate Custom .cursorrules for Your Project Based on Community Examples

The GitHub CLI allows you to search issues and filter on different criteria like amount of comments. This allows you to search for real-world situations and pains/solutions people have using different technology. 

With the GitHub CLI in hand we'll drop 100s of issues as context in Googles AI Studio and have Gemini determine the 10 most frequently asked questions. This will give us a sense of how people use a library and where people get hung up.

Search 100s of Issues with GitHub CLI as Context for AI Studio

I often find myself wondering how to use certain APIs in certain situations, and I have a couple of keywords that I try and search up on Google. 

What you can do in this situation is turn to GitHub and see how other people have actually used this code on public projects. The GitHub CLI enables us to run searches like this, and we can take the results and work on them programmatically.

Inside of Cursor with their Composer AI agent, we can search for the files we want, tell the agent to download them, and inspect them for better understanding of how those things work together.

Search All of GitHub's Code and Pass the Results to AI Agents in Cursor

Implement virtual scrolling to handle large datasets in an HTML table. Start with a basic table that renders 10,000 rows of data. The browser can struggle with this amount of DOM nodes to render at one time, especially on low end devices. 

We implement tanstack react virtual to render only the necessary amount of rows the user is viewing. The implementation includes proper column sizing, accessibility considerations, and convenient scroll controls, resulting in a smooth and performant table experience regardless of the dataset size.

Create a row virtualized HTML table with Tanstack Virtual

Learn how to implement virtual scrolling in React tables using TanStack Table and Tanstack Virtual with the useWindowVirtualizer React hook.

The most difficult tasks will require the strongest AI models. And those models simply aren't available as AI agents inside of your tooling. So the best workflows are to drag and drop the files you need to be changed and analyzed inside of a reasoning model such as OpenAI's O1 Pro, then letting it create a giant plan for your dumber AI agents to work on. The key here is that instead of just copying and pasting the entire plan into Cursors Composer or your AI agent, it's to create a file inside of your project and ask your agent to tackle the plan one step at a time. This allows you ample opportunity between each step to test and verify the micro steps along the way. Otherwise, as I've experienced many times, a session can go way off track and you burn a lot of time and burn a lot of AI requests Cursors Composer and it starts churning through a plan Cursors Composer and making lots of unwanted changes. Cursors Composer and it's to guide an AI agent to hold its hand along the way, especially through the most complex tasks. Cursors Composer and the most complex tasks Cursors Composer and the most complex tasks require a step-by-step plan.

Planning AI Agent Sessions with Reasoning Models

AI agents will inevitably make mistakes—it's just a matter of when. A key tool in your toolbox, once they do make mistakes after working on a problem for a while, is to use **git diff** to capture all of the changes they've made. 

Once you have those changes in a text file, you have a portable document that you can test against other AI models. This allows you to move the problem into smarter reasoning models with smaller context windows, helping you isolate and focus on the issue at hand.

**git diff** has been invaluable when working with AI agents because they'll often make many good changes but also slip in one tiny error. Finding that one small mistake is **git diff**’s superpower.

Search

Search Results

Web Scraping with Pagination and Advanced Selectors

Web Scraping Images with Node, Xray, and Download

Intro to Web Scraping with Node and X-ray