Advanced Flattening

[00:02] Welcome back to the End of the Loop. Last time we learned about the absorbable type. This is what we've been building up to thus far. We've been learning to program without loops so that we can use these methods and these ways of building complex programs on not just arrays, which are synchronous collections, but asynchronous collections like Observable.

[00:21] The goal here is to teach you to learn to program without loops so that we can apply these methods to asynchronous programming later on, but first of all we have a few methods to learn and we have a few basic skills we need to really drive home and absorb in order to be successful when we move over to asynchronous programing, which is a good deal more complex than synchronous programing.

[00:39] The first and most important is flattening. Now if you remember a few exercises, we did a flattening exercise in which we went through a array of exchanges and we collected up all of the stocks within each of those exchange where the price was larger than or equal to a hundred. Here's the program down here.

[00:58] In order to flatten without loops what we need to use is the concat all method. The challenge, of course, is that concat all only works on two dimensional arrays, and in this case we have an array of objects which contain arrays.

[01:12] How do we flatten? Well, we can't flatten that with concat all alone because it only works on arrays of arrays. However, by using map to transform each exchange into the stocks inside of that exchange and then filter those stocks for all those stocks where the price was larger than or equal to a hundred, we were able to use map to transform that array of objects that contain arrays into an array of arrays.

[01:39] Then we can apply concat all to flatten it out. Finally we use For Each to consume the collection and do something with it, like log it out to the console. Let's check out to see if this works. We run it, and sure enough it does. We see only those stocks with a price larger than or equal to a hundred and the stock NYN is omitted.

[01:58] This technique of flattening is great. It works. It's something that you're going to need to learn to become very, very familiar with as we go on and move towards async programing, but just to make a hundred percent sure that we really know what we're doing, we're going to try flattening not just two levels of data structure. We're going to try flattening three levels of data structure.

[02:17] Let's imagine that instead of just an array of objects that are exchanges, each of which contain an array of stocks, we had one more level that we had an array of exchanges, each of which contains an array of stocks, but each one of those stocks contains not just one price but an array of close prices, meaning a particular date and a particular price that it closed at on that particular.

[02:42] We have three levels. We have array of exchanges. We have an object that represents an exchange which contains an array of stocks. Within each stock we have array of close dates. Let's say that we want to grab the price of all of the stocks on Christmas Eve, the closing price of all the stocks on Christmas Eve. Let's just start, shall we?

[03:06] I'm going to first start at the very top with exchanges. First I'm going to name the collection I'm creating. I'll call it Christmas Eve Closes. I'm going to first map over all of the exchanges. I'm just going to go ahead and move map to the next line here, because I fully expect there will be operators chained off of it. Then I'm going to map over all of the stocks in each exchange.

[03:48] Now here is where a lot of developers make a mistake, and that's partly because we as developers aren't necessarily comfortable with dealing with multi-nested collections. We don't like going deep, deep down into collections like this, and so we try and get out of them as soon as possible.

[04:03] What a lot of developers will do is, faced with this collection, they will just go ahead and filter the close's collection. They'll take stock closes, and they will filter it. They'll look for the close with the date of December and the day of 24th.

[04:34] They'll assign it to a variable, and then they will want to return their results. In this case I'm going to return the symbol and the price of each stock on Christmas Eve. Where am I going to get the symbol from?

[04:56] Notice that the stock has been introduced by a disclosure. This map function is going to get executed for every single stock, so I have access to the stock symbol right here. As for the price, I'm almost there. Notice that this Christmas Eve Closes is not just one item. It's actually array that contains one item. That's why I've named it pleural here.

[05:22] This array will only ever contain one item because no matter how many closes there are, at least assuming this is only for the year 2014, there will only be one date in this collection that matches this criteria.

[05:33] What a lot of developers will do is just go ahead and pull it out. What's wrong with this? This should work just fine, right? Here's the issue. Now you guys know about Observable. There are some things you just cannot do to an Observable, like synchronously pull a valuable out.

[05:56] Observables are just like events. You can't simply block and wait for an event to give you an item. While it will work just fine for arrays, it will not work for observables and asynchronous data streams, so we're not going to do it.

[06:10] Here's what we're going to do instead. We are going to go ahead and return the results of this filter. Instead of just pulling the first value out like so, how are we going to get a variable that is pointing to this object? It's easy. All we have to do is map over this array that contains only one item -- it will only contain the Christmas Eve Close -- and now we have our Christmas Eve Close.

[06:44] Now inside of here I can return this object because I have access to all of the data that I need. This can be replaced with this. Really the trick to flattening deeply nested structures is to just keep nesting map expressions until you have a variable or function argument, when is basically the same thing here, a variable bound to every single item you need to create your result.

[07:21] That's the first step in flattening any nested data structure. The last step -- and this is the trickiest --you've got to figure out how many levels of collection deep you are. What we've got here is we actually have a nested array. Christmas Eve Closes is not a flat array. It's a nested array.

[07:40] Why is that? Well, it has to do with the way that map works. If you take a collection and you map over it, and for every item in that collection you return another collection, you will end up with a two dimensional collection. In this case we have this. Take a look at this code for a second.

[08:11] If I map over every item in a collection and I just return a regular item, transform that item somehow, we get another flat collection or a collection with the same dimensions as the source collection. As soon as I return an array instead, we end up with a two dimensional collection.

[08:32] If we notice, that's exactly what I'm doing here. I am mapping over all of the exchanges, and within that map expression I'm returning yet another array because the product of mapping over stocks will be yet another array. Then within this map expression I'm returning yet another array.

[08:54] Because there are three nested map expressions, I am going to end up with a three dimensional collection. The first step is figuring out how many levels deep your collection is. The next step is to flatten it n-1 times.

[09:12] If your collection is three dimensional, you need two concat alls. It also matters where you put the concat all. I can't put the concat all here because I want you to notice that this expression by itself is just a one dimensional collection. We're just taking an array of closes, filtering it, and then mapping it not into another collection but just into an object.

[09:33] This expression, this subexpression by itself, is just a one dimensional collection, which means that concat all will err, will throw. You cannot concat all to a one dimensional collection. However, out here what I've done is for every stock I'm returning an array just like in this example up here.

[09:50] For every item in an array I am returning yet another array. I therefore must have a two dimensional collection, so I use concat all. It will work on this because this subexpression is a two dimensional collection.

[10:07] Notice that for every exchange we are returning an array. In this case now it's a flat array because we have applied concat all. So that means we have a two dimensional collection. We've gone from a three dimensional collection to a two dimensional collection. Now we just need one more concat all to flatten it into a flat, one dimensional collection.

[10:26] Once we've built the collection that we want, Christmas Eve Closes, from the collection that we have, Exchanges, there's only one thing left to do, consume that collection and do something with the data. In this case I'm going to use For Each to consume the collection, and I am going to print it out to the console.

[10:48] Now what we should expect to see is only the prices for those stocks, for all these stocks, on Christmas Eve. Let's give this a run. No? What did I miss? Oh, I compared get month to 12 instead of 11, and the month is base 0If I run this now we should see...we see price. We see the symbol for all of the stocks, and we see a price. Let's go and compare that price to see that it worked. $240.10 for XFX, $521.24 for TNZ, yes, that's the Christmas Eve date. $423.22 for JXJ and $16.82 for NYN.

[11:34] As we can see, we've managed to flatten any nested data structure. All we need to do is to keep mapping until we have all of the data that we need bound through the closure arguments and then create our result, figure out how many levels deep we are, and apply concat all n-1 times. That's all there is to flattening. Stay tuned.