Demystifying MDX

Cole Bemis: [0:00] Hey, everyone. Today, we're going to be demystifying MDX by learning how it works under the hood. Before we do that, let me introduce myself. Hi, I'm Cole. You can find me on Twitter and GitHub @colebemis.

[0:19] I recently graduated Cal Poly San Luis Obispo with a bachelor's degree in computer science. As of this week, I started working at GitHub on the design systems team as a software engineer.

[0:40] Now, before we get into this talk, I have a confession to make. That confession is that when I agreed to do this talk, I didn't know how MDX worked. Let me tell you the story.

[0:59] A few months back, Chris and John came to me and asked me if I wanted to talk at MDX Conf and, of course, said, "Yes." Then I realized that I didn't know what I wanted to talk about. I started brainstorming ideas and I came to them with the idea for explaining how MDX works under the hood.

[1:26] I thought that would be a cool talk and they agreed. I said, "OK, I'll do it." [laughs] Then I thought about it, I was like, "Now, I have to figure out how MDX works." This talk is my journey to figuring out how MDX works and along the way, I'll point out some important takeaways. Let's get started.

[1:57] First, let's talk about why would you even want to bother learning how MDX works? You definitely don't have to know how MDX works in order to use it, but there's a couple of reasons why you might want to know how MDX works.

[2:15] The first is it makes debugging easier. If you run into problems using MDX, if you know how it works internally, some of the error messages might make more sense to you. It also allows you to contribute back to the project.

[2:31] If in your debugging you find out that it's a bug with MDX, you can fix the bug in MDX and contribute back to the project, which is always super appreciated and welcomed by maintainers. It also just makes you more productive when using MDX. You know the full possibilities and you can take full advantage of the tool.

[3:00] Now, that we're a little more motivated to learn how MDX works, now let's get started. I figured it'd be best to start this journey with the fundamentals and put everything in context. The fundamentals of a website rendered into a browser is that every website is backed by HTML. HTML is what is describing the web page and it's what the browser looks to render.

[3:37] HTML is a great markup language and it's very declarative, but it's not the best for writing long documents. If you want to author long documents, it's pretty tedious to write HTML tags when you want to format things, which is why Markdown exist.

[3:58] Markdown is a nicer syntax for authoring documents, the formatting syntax is nicer and more concise, but the browser doesn't understand Markdown directly. You can't just give the browser a Markdown file and expect it to render the formatting correctly.

[4:23] When you're using Markdown, you have to first convert into HTML before it can be rendered in the browser. Markdown, while it's great for authoring long static documents, it has limitation in that it's cumbersome to add interactive elements inside of a Markdown page, which is where MDX comes in.

[4:50] MDX is a syntax that's a superset of Markdown that allows you to embed JSX in your Markdown files, which makes it easy to use your React components maybe from your design component library in your Markdown files.

[5:10] Just like Markdown, the browser doesn't understand its MDX format natively. Before you can render anything in the browser, MDX still needs to be converted into HTML before the browser can render it. This conversion between MDX and HTML is one I wanted to learn about for this talk. This is the starting point. This is where I started.

[5:41] To learn more about how MDX gets converted into HTML so it can be rendered in the browser, I started looking around the MDX docs and I found a quote in the API page that says that "MDX -- the library --, at its core, transformed MDX -- syntax -- into JSX."

[6:03] This adds another element to the picture that we've been constructing in that it's not MDX straight to HTML, it's MDX to JSX, then to HTML, and then rendered in the browser. I'm assuming you know what React is for this talk.

[6:28] React handles JSX rendering in the browser, so I wanted to figure out how MDX gets converted into JSX, because that's the core of the MDX library, as the docs mentioned. What this MDX to JSX looks like in code is there's @mdx-js/mdx package that exports a function, and the function accepts a string of MDX syntax and it returns a string of JSX syntax, which looks like this.

[7:09] Don't worry if this is a little bit overwhelming. The only thing you need to recognize is that this vaguely looks like some React code that you might write. That's our first takeaway is that MDX -- the library -- at its core, is a function that accepts an MDX string and returns a JSX string. This is actually the most important takeaway in the whole talk.

[7:37] If you only get one thing out of this talk, remember that MDX library at its core is a function that accepts an MDX string and returns a JSX string. All of the other stuff that we'll end up looking at at this talk, builds off of this takeaway.

[7:54] We have an MDX function, it takes a string of MDX syntax and returns JSX. To figure out how MDX works, I need to figure out how this function is implemented.

[8:10] I went to the source code for this package, I went to index.js file and I started the reading functions, I started following the import statements, and I came up with this simplified implementation of the MDX function that can fit in a slide.

[8:28] This is obviously oversimplified, but it's representative of what the MDX function is doing and it's helpful to explain how MDX works. Don't worry if you don't understand what all is happening in here, I will go over all of that. When I first looked at this, I definitely didn't know what was going on at all.

[8:54] When I boiled it down to this I, I figured that this unified call right at the top would be a good place to start. This unified function is coming from a package called unified. I went to the documentation for unified.

[9:16] The documentation for unified described it as an interface for manipulating text with syntax trees, which I guess are words, [laughs] but it didn't click with me. Until I started reading more and I came to understand that unified is basically an interface for building a function that takes an input string, manipulates it in some way which you can define, and gives you an output string.

[9:52] This function that you can build with unified is called a processor. In our case for MDX, we're building a processor that takes an MDX string and outputs a JSX string, which is our second takeaway. MDX -- the library -- uses a package called unified to build the function, also known as a processor, that can transform an MDX string into a JSX string.

[10:27] We know what a processor does, but I wanted to figure out, how does it do it? What is inside of a processor? There turns out to be three parts of a processor -- the parser, the transformers, and the compiler. Let's dig into those a little bit.

[10:52] The parser is a function that takes the input string and returns a syntax tree. What's a syntax tree? The syntax tree is basically object representation of some syntax that is easier for programs to understand.

[11:17] For example, on the left here, we have a JavaScript expression, 1 + 2. The abstract syntax representation of this would be something like on the right. You can see that this object describes the syntax on the left.

[11:43] This is an example of Markdown. On the left, we have Markdown syntax for a heading. On the right is the syntax tree for the heading. You can see that it describes that we have type heading, it's a depth of 1, and the children is the text that is the inside of the heading.

[12:07] Now we understand what a parser does. Input string. It gives you a syntax tree that represents the input string. The next part of a processor is a transformer. The transformers take the syntax tree and they transform it. They'll manipulate it, add things to it, remove things from the syntax tree, and they'll return a new syntax tree.

[12:36] The way these work, you can actually chain them together. The output of one transformer becomes the input to the next transformer. You can have as many of these as you want in a processor.

[12:52] Last part of a processor is the compiler. This is the component that is responsible for generating the output. It's a function that takes the syntax tree and converts it into the output string. In our case, our compiler is outputting JSX.

[13:16] This brings me to takeaway number three, which is that processors process strings in three stages. Parsing, which turns the input string into a syntax tree, which you'll often see described as an abstract syntax tree, or AST.

[13:34] The second stage is transformation. This stage takes the syntax tree that was parsed, manipulates it someway, and gives you a new syntax tree. Then the third step is code generation, which takes the result of all the transformations of that syntax tree and turns it into an output string.

[14:02] We understand what a processor is, what it does, how it works, but we don't know how to make one. How do we do this with code? How you do that is, you start with a base processor, unified() gives you that.

[14:20] That's what this unified function call returns a base processor that is essentially a no-op. This is a processor that doesn't do anything. You can extend it with this use method and you give the use method a plugin, which I'll explain what that is, and it'll return you a new processor.

[14:41] A plugin is a function that can change the parser, can add transformers, and it can change the compiler of the processor. An example processor we might build is we start from the unified base processor. That's the first function call, and we chain together use calls and we parse it different plugins that are responsible for different stages of the processing pipeline.

[15:09] In this case, we're parsing in a plugin that handles the parsing, one that handles transformation, and one that handles compilation or generating the output code. This is a trivial example, obviously, but we can substitute that with our actual MDX example. It looks pretty similar. In just a second, I'll go over what all of these plugins are doing.

[15:39] Just recognize that this is how we're building the MDX processor. We're starting with a base processor and extending it with plugins. That is our takeaway number four, that processors are constructed using plugins that define what happen in each stage.

[15:58] Back to the MDX processor that we've built here, let's dive deeper into what each of these plugins are doing. This toMDAST is the plugin that's handling parsing of our MDX string. It's given an MDX string and it returns an MDAST, which stands for Markdown Abstract Syntax Tree.

[16:27] That markdown abstract syntax tree is then transformed into an MDXAST, which is more specific to MDX, and then that syntax tree is then transformed into an MDXHAST. HAST stands for HTML Abstract Syntax Tree.

[16:51] We're converting a syntax tree that represents Markdown syntax into a syntax tree that represents HTML, which will make it easier to turn into JSX. This last step, will return this MDXHAST into a JSX string.

[17:13] That was a lot of information and a lot of acronyms on the screen, and I went pretty fast through it. Let's go through an example of all of the different steps in this pipeline. We're starting with an MDX string. This is an example MDX string that just renders a heading, some text, and a button. This is parsed into an MDAST that describes the syntax that we had initially parsed in. That is then transformed into an MDXAST.

[17:53] Since our example in this case is so simple, that syntax tree doesn't change. Now this MDXAST is then transformed into an MDXHAST, which is the abstract representation of HTML. There's a subtle difference.

[18:14] If you can see in this last step, the heading was marked as type heading with a depth of one, which is more representative of the Markdown syntax, and in this next step, that's been changed into a tagName which is an h1, which directly maps to an h1 tag in HTML.

[18:38] Finally, this MDXHAST is then compiled into a JSX string that the output looks like this. That leads me to takeaway number five, which is that the MDX processor specifically converts an MDX string into a Markdown AST, then into an MDXAST, finally into an MDXHAST, which is converted into a JSX string.

[19:13] I don't expect you to remember this. This is probably the least important takeaway because you can look it up, but there you go. Here is the processor that we've built for MDX to convert an MDX string into a JSX string. To use this, we need to save it into a variable which we can call processor, and then we need to run it on a string.

[19:51] Let's wrap it inside of a function that takes a string and process that string with the processor that we built and returns the result, which is what we have here. This is the first code example that I showed you. This is the implementation of the MDX function. Now, hopefully, all of these different parts make sense.

[20:20] That was a lot of information, let's take a second to recap what we learned. The first thing is that MDX -- the library -- at its core, is a function that accepts an MDX string and returns a JSX string.

[20:38] Second, MDX -- the library -- uses a package called the unified to build the function, which is also called a processor that transforms an MDX string into a JSX string. Processors process strings in three stages -- parsing, transformation, and code generation.

[21:04] Processors are constructed using plugins that define what happens in each stage. The MDX processor specifically converts an MDX string into Markdown AST, which is converted into an MDXAST, then into an MDXHAST, and finally, into a JSX string.

[21:31] Now, I have a little bonus six takeaway before I leave you, and that takeaway is that the tools you use aren't magic. As I've hopefully shown in this talk, the tools you use can often feel like a black box, but you can definitely open that box and learn how they work because I did. I didn't know how MDX works when I started this talk, and now I do. You can definitely do that as well.

[22:09] I'll leave you with a challenge to pick a tool that you use and see if you can figure out how it works. I'm sure that you will. If you don't figure out exactly how it works, you'll learn something new and have fun doing it. Thank you very much.

[22:33] If you want to revisit these slides later, they're on GitHub at this URL. If you have any questions for me or just want to talk about MDX or any other things in this talk, feel free to reach out to me on Twitter. Thank you very much.