In this video we're going to setup so-called "task pipelines" and "task caching" to enhance the DX of your existing monorepo. I'm using an NPM workspaces monorepo specifically, but it'd also apply to Yarn or PNPM workspaces in the very same way.
One of the issues you might encounter in an NPM workspace monorepo is that you have dependencies among projects. Like in this example I'm showing, there's a Vite based React app which depends on a shared-ui package. As such when you serve or build your React app, you need to make sure to have shared-ui built first. A task pipeline can automate that process.
Finally, we're going to also configure caching for our build tasks to speed up our monorepo. Task caching avoids re-running the same task if the underlying input (the src and other files) haven't changed.
Transcript
[00:00] So I have here a very small, super simple npm-workspaces-monorepo and you can see that by this workspaces property here, which is at the root-level package.json. And so I have here a React app generated with the vcli, I have another one with the remix-cli, and both depend on this shared-ui package down here. Now you can also do something like NX graph and I don't have NX installed here but you can run that on any npm, pnpm or Yarn workspace and that allows you to open here this view in the browser and basically visualize how these two packages look like and how our monorepo structure looks like. So this is the situation that we have just to have something shared here. Now, when I mentioned before that speed is not necessarily the first thing you might benefit in terms of adding tools on top of just an npm workspace monorepo, is task pipelines.
[00:49] Because we have seen that there is this dependency between, for instance, let's say React Vite and that shared UI package. And so basically, if I want to build this React Vite package, so I would run npm run build workspace React Vite, It will fail the build because it cannot find that shared ui package because I haven't built it yet. So you can see here it failed to resolve the entry to shared ui package because I haven't built it yet and because the react-vite package here depends on the build output of that package down here in that packages folder. So basically what you would have to do is make sure that you first run the build of shared UI. And so once that is done we can go back and run it for React Vite and now it would succeed properly.
[01:34] So this is called a task pipeline because the tool needs to understand that well, if someone runs the build or even the serving of this React Vite, it should first build the shared UI package down here because otherwise it will fail. And this is one of the first things I feel personally can be enhanced quite a bit by using some of the other tooling on top of an npm workspaces. So let me go ahead and I'm going to use NX here and I go and just install here that latest package. So we need to make sure to install that. And so once we have installed that, we can add a new file here, which is called NxJSON, which allows us to specify some metadata related to Nx.
[02:13] So let me also paste in our schema, so we get some auto-completion. And one of the first things I can do to define such a task pipeline is here tell BasicG whenever you run a build, then you have a property called dependsOn, and you can tell BasicG with that caret symbol, run all the builds of the dependent project. That current symbol specifically tells that NX should go and look at all the dependencies downstream and build them first if they have a build target. And so if we try this again, let me remove actually all these folders that we have created, but now I run it through the NX pipeline, such that NX actually is able to take care of the build. What it will do is just run the scripts, the package.json scripts that we already have.
[02:55] So if I run, for instance, something like NX build react-vite, NX will go here in the react-vite package, look at the package.json, and run that build script. So the build that I mentioned here is specifically mentioning this script or targeting this script dump here. Now, if I run this, you will see now NX is waiting for one dependent project to build first. And that is exactly that shared UI package that I just mentioned below. So in X.Understood, well, there's the dependent package that has a build target.
[03:22] So let me run that first and then invoke the build target of the tool, basically, that I'm running or the package that I'm running, which in this case is React V. And clearly, you can configure that even differently. So you can go ahead and say, well, I also want to do that for the dev script, because clearly once I spin up the dev server, that shared UI package should also be pre-built. And there's other things that you could do, like potentially watch for dependencies and rebuild them automatically. There's much more that you could potentially add.
[03:46] So that's, in my opinion, one of the first immediate benefits that you get. The next one, which I already anticipated before, is clearly speed, right? So if I now have multiple of these packages, I want to make sure that they're built as fast as possible. And many modern tools nowadays have something built in that is called caching. And so in order to enable caching, I can just go again in here and say cache true for this build.
[04:10] And this is an opt-in because I want to make sure that the build is actually cacheable. Because if it has some side effects, it's not. So I want to enable that manually here. And so if I go back and run ReactVBuild, it would rerun it the first time around because there is no cache yet. But you can see if I rerun it again, it is basically immediate, just taking a couple of milliseconds, because there's no change that had happened, so it just went ahead and built it.
[04:36] And so the more packages you have, for instance, if I run multiple targets, if I run build and test and lint, I don't even know what I have here, all of these targets defined, You can see now it runs through, but for instance the ReactV is not being run again because that is already cached and just runs it for the other ones. And if I would rerun clearly all of them would be cached. Now by default the cache here lives just locally, so there's just an NX folder and there's a cache in here. But then if you want to use that on CI, clearly you would go and have to do something like nx connect and connect that with the remote caching. And if I rerun here the build of all the testing and linting, it would run them first time to produce some of the caching.
[05:16] Then you can see it would be pulled down now from the remote cache and synced locally to my local folder here. So clearly on CI you would need some central place where that cash is being hosted.