Automate Package Scaffolding in NPM/Yarn/PNPM Monorepo Workspaces

So I have here already a starting structure where I have a packages folder and a UI library and it's a very simple one it just here has a public export has here one React component in the package.json I have the name I have the main entry points here have the build script I'm using Zup for this so you can see here the Zup config and I'm also having here a tsconfig that is being used here by Zup and that points to some more common base tsconfig which has just some more common settings that I might want to reuse across various packages that I have in here. Now in this specific case I'm actually using pmpm-workspaces here you can see that by that pmpm-workspace yaml file but it's perfectly fine to also use npm or yarn workspaces it wouldn't matter in this case. So to build this project here I could just run pmpm-r build and it would run recursively the builds for all the projects in its workspace. So in this case I just have one and you can see now the dist folder is being produced and the compiled output, basically our TypeScript is being compiled to JavaScript here.

Now if I want to create a second library, which is very often the case, I would have to go and now copy and paste this. So just go here, copy and paste, create a new one, potentially remove the files within it. Now Obviously here I don't have much file so it would be pretty easy but imagine a more real-world project you would have a couple of those and so removing them even maybe tinkering here with specific dependencies for that new app and removing all the old ones could be tedious. So I wanted to show you quickly how we could kind of automate this by leveraging NX generators. Now as a first step let's just add NX here to this monorepo.

So you can just run pmpm dlx nx add latest init And so this now installs NX, can ask a couple of questions like what scripts need to be cached and where the output folders are. And so we end up here with an NX dependency in our package.json and also an NX.json here, which defines things such as the cache operations or such a task pipeline, which defines, for instance, a case where it would run the builds of our dependent projects before running the build of our library itself. So for now we can actually ignore this and now that we have NX installed, we can also just run our commands as follows using the NX pipeline. So we can say run many-t build and this will now run the builds of the projects that are in our workspace. But what we want to actually do is to automate this workspace and NX has the capability of running generators.

Now to better understand this if we go to nx.dev docs there's an entry page there about why NX and further down it is kind of an explanation of what the over architecture of NX looks like. So what we have done now in this PMPM workspace monorep is just install that NX package which gives us the task running as we have seen it also gives things like distribution and caching but there's also section about plugins in particular the dev kit that can be used to develop such plugins. Now plugins are usually very technology specific so there might be a plugin for React, for Angular, for Next.js, for Remix that basically help you set up those projects in a very easy way by generating some of those parts and that's actually a part that we're interested in And then some other things such as like wrapping lower level build toolings by providing dedicated task runners and things like automate migrations. But we are mostly focusing now on the generators part because we want to leverage that for our local workspace. And so there's a whole section about how to develop such plugins and how to can install them in a workspace or use them to automate your workspace so definitely feel free to check out that types of docs but we're going to now do that live here in this PMPM mon repo.

So the first thing that I'm doing is installing the NX plugin package because that provides the facilities to create such automation generators. So let me just run here pmpm add nx-plugin and I'm installing it as a dev dependency and also workspace-wide at the root level package.json because this is something that kind of impacts the entire workspace here. So once this is installed we can now run pmpm nx list add nx plugin and this will give us a list of so-called generators that this plugin already comes with Because we don't want to set up the whole plugin system for ourselves, but we could rather just run a generator again to create, basically generate such a plugin. And so in this case, we are interested in this one specifically and later then in that generator, generator, which is kind of a weird combination, to set up such a code scaffolding mechanism to then create our new package. So as mentioned, first of all, let's run the generation of such a new NX plugin here in this workspace.

And so I'm running nx generate, add nx plugin, plugin. And so it will ask me now, what is the name of the plugin that you want to create and in my case let's just call this automation because this should kind of contain the automation facilities for our workspace here. So you can see here an output of what has been created, but you can also just go in here and you will see now we got a new kind of package inside the packages folder that is empty for now, it has a package.json with just a name but nothing really more in it. So the next step is to generate such a generator. And so again, I'm using nx-generate, nx-plugin, generator.

And let's call this React-lib because we wanna create a generator that scaffolds a new React library based on this model. So it will be fully custom to this workspace. So here it asks me in which project we want to add that generator, and obviously we want to add it in our automation project. And so here we get now such a generator setup. And that includes first of all such a generators.json, which is leveraged by NX at runtime to find potential plugins that have generators installed and shipped as part of them.

And so what the generator's JSON does here is mostly identify the name of the generator, point to the actual implementation function that needs to be run when the generator is executed. It has a description and it has a schema file. So let's have a look at that schema file here. And this is basically simply a description, a JSON schema description of what your generator looks like. So what input it provides, what the type of those inputs are and what are required inputs.

Moreover it also has things like an X prompt so if you don't provide it on your CLI it would actually go and prompt you. Now we will see this and these are actually really useful because it allows to give the user more feedback about what it needs to provide. So without diving too much in the underlying mechanisms, let's actually look at the generator's TS file, which is the actual implementation. Now at the high level, what you get there is just a function that is being exposed and that is exactly the generator's function that is being run once this is being invoked. And it gets a tree object which is, imagine there's a virtual representation of your file system where you can run operations against and so this allows you to perform such things like dry runs and simulations of what would actually be created at runtime.

And then it has the options to add is actually our schema, which is basically the TypeScript definition of that schema.json file. So let's have a look at what we are actually wanting to implement. So we want to implement a generator that creates a new library similar to this one whenever we invoke it. And so we can actually just go ahead and replace the things that are in this files folder here because this is something that gets read by this generate files as you can see here it reads in that files folder and then generates that into the new library wherever we specify that path to be. So first of all let's have a look at what we get here or have in our source folder of the UI library which is our example project basically.

So we have the index.ts which is just the point where we expose things and so we have that here as well. Note we have that .template which is just a way to mark this as actually being a template file. Now here we could just say something like export your public components. Since this is React library we can also say something specific like export your public React components here. So as a next step then we would want to actually have the whole config part.

So I don't think we necessarily need to create a sample component. I usually like it when there's less components around. So things like the package json would be something that we want to generate. So let me just copy that up. And so we have now our source, we have our package json.

Let's suffix this with .template. And here, for instance, we would want to get the name of the library array right. So this would be a template and probably we also want to have a scope of the package. So let's actually add in here some template markers and so this would be the scope and here would be the name of the function, the name of the actual package. And similarly we probably also want to provide here the description here of our package as well.

So we obviously need to provide these, but that's something we can do in a second moment. Let's continue completing here our file setup here. And so what's missing now is the tsconfig which doesn't really have any dynamic replacements here based on the name of our package so it's pretty static and similar to the subconfig here and so let's paste those in as well and we can leave them as is for now. So with that we have all the necessary files to set up our simple package here. So now we need to go and actually provide these.

So first of all the name is something that already comes here from our schema.json so we have that already. Now we could go ahead and actually copy this for our description. So just fill in here the description. We don't have here a default and as the next prompt we could say what is the package about. So the user can provide that.

I wouldn't say this is required so we could by default just have it empty if it's not being provided which I think is fine. Now we need to make sure that these are being passed to our template here and to do that we can use here an option field that is passed to this generate files. Generate files is the one that takes here that finds directory, provides a couple of options such as like where it should be generated and we need to first adjust that as well. And then it will also go and pass in here this options field which is here our type representation of our schema. And so this is the schema DTS file and so we also need to make sure here that we have the description field marked as a string.

Now for things such as the options field here that is being passed in by the user usually want to have it kind of pre-processed and adjusted. For instance we want to have something like resolved options that here passes the options. But then, for instance, for the name, there is a utility function that is exposed directly by the devkit. So that's where most of the utility functions come from in terms of developing NX extensions or NX plugins. And so we have here our name that we kind of process through that names function and that gives a variety of different potential outputs and transformations of that which could be a class name, constant name, which are very handy if you need it in the template or also the file name which is actually pretty handy for something like actual file names, folder names, but also in this case for an npm package name.

And so next we need to make sure that we replace our normal options with these resolved options. Such that they are being passed to our generate files function here. So the description is already being passed in here and we can just leave it as is, but we also need that scope property which we have used in our template here in the name of our package. And so that scope can be something that we either could ask users to give, but it can be kind of annoying. So we could actually infer that from the package.json at the root level of our monorepo.

And so here we have quite a long one, but we could say here something like this is my org, or this could be the name of my monoreap in general and we can leverage that as our scope for all the packages that live within that monoreap workspace. And so for instance our UI package here should rather be named my org UI to make sure that it's kind of following this naming convention. So to read that we can actually use a function again from the NXDevKit which is called readJson and then here have something like const root or scope name And that's exactly what we need. So we kind of need to name here from the package.json at the root of our workspace. And so this would be the scope name.

So with that, we are pretty far already. There are some generated utility functions that we don't really need for this specific setup. Now one thing that we need to adjust though is the project root. Now in this specific case we could actually just go ahead and simplify our lives a bit and hard code this and so say this is always going to be in the packages folder and probably we want to use here the resolved options dot name just to make sure that we also kind of suffix this or make sure that it kind of be resolved based on the file system naming conventions. So with that we should be done.

Let's actually go and try running this. To run such a generator we can just refer it by the actual name that we have here in our package.json and so this is called automation. So we could just run pmpm nx generate or just g for short and then say automation and colon the name of the generator which in this case is custom lib because that's the name that we actually gave our generator here so to do a dry run we can just pass dry run here and this would now invoke directly our generator so it asks properly what name we want to use for our library. So let's say my reactlib. What is the package about?

It is just a simple setup of a react library and now you would see what it is going to generate. So here it would create a new folder here based on the name we provided with a package.json, has the source file, has the configs that we need. And so far it looks actually pretty good. So let's try this without actually a dry run. We can also pass the name directly, so I could say my react lib and give the description, a simple React library.

And now here you can see in this packages folder we got a new myReactLib. It has properly set the name, it has set the description. We have our tsConfig which should still be valid, You have our subconfig which we didn't really change but just generate into this folder here. And we have the index.ts file where we just mentioned here export your public react components. And so we could go ahead here and say create a new hello-world.tsx and just create here a simple React component.

Now notice here it doesn't recognize the TSX yet and the reason is we actually need to run a pmpm install here so it would install all the packages that this specific library here needs which also includes things like React, React DOM and so on. And so now you can see it properly recognizes it. Now since this has already the build script as well, we can just run pmpm nx build and the name of this library is my-org-my-react-lib and we just now build our application here. Now you can see there are some warnings because what happened here when we generated this setup automation is that here in the TS config base, some relative paths have been set and this is necessary for NX automatically recognize this library and pre-build it for us and run it. Because this is written in TypeScript, ByteLinux immediately executes it without us having to do any pre-build step at all.

And so to circumvent this warning, you can just set here the base URL and set it to dot as well as the root there to dot. And so if you rerun the build here of our React lib again, it just produces the proper output. And so you can see here it compiles just as we would expect. And so this generator was actually a pretty straightforward one as we didn't really customize a lot of the output here, but we are just copying pasted files in here, replace a couple of things such as here and here that could be dynamic based on the user input and that's it. But you could do a whole lot of different things here such as using some AST parser, looking at an existing library and for instance augmenting that library with some additional things.

Let's say you have a library or an application in your monorepo and you want to add Prisma support so you could generate that on top of that existing library rather than just always scaffolding new ones.

Automate Package Scaffolding in NPM/Yarn/PNPM Monorepo Workspaces

Relevant Links

Transcript

More Tips