Push Chunks of Large MySQL Data to an Algolia Index with Lodash

Raphael Terrier
InstructorRaphael Terrier
Share this video with your friends

Social Share Links

Send Tweet
Published 6 years ago
Updated 5 years ago

Using the Lodash library, we’ll see how to split large amounts of data from a local MySQL database into smaller chunks and push them to an Algolia index.

Instructor: [00:00] When dealing with a larger number of records, it is recommended to split those into batches of 1,000 to 10,000, depending on the record size. Let's switch tables in our database from actors_sml to actors_big.

[00:14] The records are similar, except this time we don't have 500, but 50,000 rows to push. To push data by batches, we are going to use the Lodash library. Let's install the package and import it.

[00:30] Let's edit the query to target the right table and remove the previous addObjects method. Using Lodash chunks method, we will create an array holding smaller arrays of 1,000 records each.

[00:44] We then iterate over this array of arrays and call the addObject method each time, sending chunks of 1,000 records. We run the script and push the data back to the Algolia dashboard. I can now refresh the index and check that all the 50,000 actors are already there.

egghead
egghead
~ 16 minutes ago

Member comments are a way for members to communicate, interact, and ask questions about a lesson.

The instructor or someone from the community might respond to your question Here are a few basic guidelines to commenting on egghead.io

Be on-Topic

Comments are for discussing a lesson. If you're having a general issue with the website functionality, please contact us at support@egghead.io.

Avoid meta-discussion

  • This was great!
  • This was horrible!
  • I didn't like this because it didn't match my skill level.
  • +1 It will likely be deleted as spam.

Code Problems?

Should be accompanied by code! Codesandbox or Stackblitz provide a way to share code and discuss it in context

Details and Context

Vague question? Vague answer. Any details and context you can provide will lure more interesting answers!

Markdown supported.
Become a member to join the discussionEnroll Today