Transform Stream Data With a Node Transform Stream

Matt Ross
InstructorMatt Ross

Share this video with your friends

Send Tweet
Published 4 years ago
Updated 3 years ago

Node's Stream API provides a stream.Transform class for easily and intuitively transforming incoming data. Piping streams through this class preserves natively provided back-pressure and high watermarking, while also keeping you working in the same paradigm (streams!) from start to finish.

Instructor: [00:01] Here I have some code that sets up a read stream, a read from a data.JSON file that I prepared ahead of time. If that stream encounters an error, we'll log that error.

[00:10] Once that stream becomes readable, we'll read each incoming chunk off the stream and log the chunk value. If you look to the right, you'll see that we're receiving a buffer. That buffer is the JSON data being read from the file.

[00:24] Now, I want to convert that buffer into a JavaScript object, so I'm going to use a transform stream. I'm going to create a class to object that extends stream.transform. Any transform stream must implement transform method.

[00:42] The arguments to that method are an incoming chunk, the encoding of the chunk or stream, and a callback. We can then call the callback with any possible error and apply some transformation to the chunk if we're sending it out.

[00:58] Then, I can just pipe that incoming stream to a new instance of to object. That new instance is going to have object mode set to true. That just means that any outcoming value is expected to be an object, rather than a string or a buffer. As you can see, on the right, we're getting the JSON out from data.JSON, our email and our password.

[01:21] Now, if I want to create additional transform streams, I'll need to implement a new class that extends from transform, or I can pass my transformation function to an option in the constructor. Even better than that, I can use a helper library, like through, provided by Mississippi.

[01:40] Now, I can create a map function that takes some function as an argument and returns a through stream, with object mode set to true. The first argument to this function is going to be a transform function, just like above.

[01:54] The arguments will be chunk, encoding, and callback. Just as above, we will call that callback with a potential error, and the result of applying our function to the chunk. Now, rather than pipe to some new object, I can say map. I take my chunk argument and apply my transformation.

[02:18] I can clean this up a little bit. Since map is easy to implement with any transformation, I can easily extend this. I can take my chunk and, just as an example, I can return just the email field.