All PostsEngineering Blog

Anatomy of JS module systems and building libraries!

PWA
PWA

Lately, we have all been stuck with these giant words “JavaScript Modules”. Each one of us wondering what we can do with them, how do they even play a vital role in our daily life and so on.

 

So what the heck is JS module System? 🤔

As JavaScript development gets more and more common, namespaces and dependencies get much more difficult to handle. Different solutions were developed to deal with this problem in the form of module systems.

Why understanding JS Module System is important? 😕

When we say an application is modular, we generally mean it’s composed of a set of highly decoupled, distinct pieces of functionality stored as modules. As you probably know, loose coupling facilitates easier maintainability of apps by removing dependencies where possible. When this is implemented efficiently, its quite easy to see how changes to one part of a system may affect another.

Story Time!
Telling stories is as basic to human beings as eating. More so, in fact, for while food makes us live, stories are what make our lives worth living – Richard Kearney 😇

Why am I even talking about all this?

So my day job is to design and architect projects at my workplace and soon I realized that there were many common functionalities required across projects and I always end up copy-pasting those functionalities to newer as well as existing projects over and over again.
The problem here was whenever there is a change in one piece of the code I need to manually sync those changes across the projects. Hence, to avoid all the tedious manual tasks I decided to extract the common functionalities which are used across the projects and compose an npm package out of it, so that others in the team can re-use them as a dependency and simply update it whenever a new release is rolled out.
This approach had some advantages:
1.If there is some change in the core library then a change has to be made at one place without refactoring all the applications code for the same thing.
2. All the applications remain in sync. Whenever a change is made all the applications just need to run “npm update” command.

Source Code of Library
Source Code of Library

So, next step was to publish the library. Huh?? 😖
This was the toughest part because there were bunch of things striking my head like:
1. How to make it tree shakeable?
2. What JS module systems shall I target (commonjs, amd, harmony).
3. Shall I transpile the source?
4. Shall I bundle the source?
5. What files to publish?
Everyone of us havethese kind of questions bubbling in our heads while creating a library. Right?
I’ll try to address all the above questions.

Different Types of JS Module Systems 📚
1. CommonJs
  • Implemented by node.
  • Used for server side when you have modules installed.
  • No runtime/async module loading.
  • import via “require”.
  • export via “module.exports”.
  • When you import you get back an object.
  • No tree shaking because when you import you get an object.
  • No static analyzing as you get an object so property lookup is at runtime.
  • Always get a copy of an object so no live changes in the module itself.
  • Poor cyclic dependency management.
  • Simple Syntax.

2. AMD: Async Module Definition
  • Implemented by RequireJs.
  • Used for client side(browser) when you want dynamic loading of modules.
  • import via “require”.
  • Complex Syntax.

3. UMD: Universal Module Definition
  • Combination of CommonJs + AMD i.e Syntax of CommonJs + async loading of AMD.
  • Can be used for both AMD/CommonJs environments.
  • UMD essentially creates a way to use either of the two, while also supporting the global variable definition. As a result, UMD modules are capable of working on both client and server.

4. ECMAScript Harmony (ES6)
  • Used for both server/client side.
  • Runtime/static loading of modules supported
  • When you import you get back bindings value(actual value).
  • import via “import” and export via “export”
  • Static analyzing — You can determine imports and exports at compile time (statically) — you only have to look at the source code, you don’t have to execute it.
  • Tree shakeable because of static analyzing supported by ES6
  • Always get an actual value so live changes in the module itself.
  • Better cyclic dependency management than CommonJs


That was all about different types of JS module systems and how it has evolved. Although ES Harmony module system is supported by all the tools and modern browsers, but we never know while publishing libraries how our consumers will utilize it. Hence, we must always ensure that our libraries should work in all the environments. Let’s dive more deep and design a sample library to answer all the questions related to publishing a library in a proper way.

I’ll build a small UI library, source on GitHub, and then share all my experiences and explorations for transpiling, bundling and publishing it.

Directory Structure
Directory Structure

Here we have a small UI library which has 3 components Button, Card and NavBar. Let’s transpile and publish it step by step.

Best practices before publishing? 📝
  1. Tree Shaking 🌳
  • Tree shaking is a term commonly used in the JavaScript context for dead-code elimination. It relies on the static structure of ES2015 module syntax, i.e. import and export. The name and concept have been popularized by the ES2015 module bundler rollup.
  • Webpack and Rollup both supports Tree Shaking but for that we need to keep certain things in mind, so that our code is tree shakeable.

2. Publish all module variants
  • We should publish all the module variants i.e UMDand ES, because we never know which browser/webpack versions our consumers might use this library/package in.
  • Though all the bundlers like Webpack and Rollup understands ES module but if our consumer is using Webpack 1.x then it cannot understand ES module.
// package.json
{
"name": "js-module-system",
"version": "0.0.1",
...
"main": "dist/index.js",
"module": "dist/index.es.js",
...
}
  • The main fieldof package.json file is usually used to point to the UMDversion of the library/package.

How can I release the ES version of my library/package? 🤔
The module fieldof package.json is used to point to the ES version of the library/package. Previously many fields were used like js:next ,js:mainbut module is now standardized and used by bundlers as a lookup for ESversion of library/package.

Less known fact: Webpack uses resolve.mainfields to determine which fields in package.json are checked.
Performance Tip: Always try to publish ES version of your library/package as well because all the modern browsers now support ES modules. So you can transpile less and ultimately you’ll end up shipping less code to your users, hence your applications performance boosts up.

So now what’s next? Transpilation or Bundling? What tools to use?
Ah, here comes the trickiest part! Let’s dive in. 🏊
Webpack vs Rollup vs Babel? 😕

These are all the modern day tools we use in our day to day life to ship our applications/library/packages. I cannot imagine the modern generation web development without them *BLESSED*. Hence we cannot compare them, therefore it’s a wrong question to ask! ❌

Each tool has it’s own benefits and serves different purpose based on the needs.

It’s a wrong question to ask. Each tool has it’s own benefits and serves different purpose based on the needs.

Webpack: Webpack is a great module bundler 📦 that is widely accepted across the globe and mostly used for building SPAs and gives you all the features out of the box like code splittingasync loading of bundles, tree shakingetc. It uses CommonJS module system.

PS:Webpack-4.0.0 alpha is already out 😎, hopefully with the stable release it will become the universal bundler for all types of module system.

RollupJS: Rollup is also a module bundler similar to Webpack but the main building block of rollup is that it follows new standardized format for code modules included in the ES6 revision, so you can use this to bundle ES module variant of your library/package. It doesn’t supportsasync loading of bundles.

Babel: Babel is a transpiler for JavaScript best known for its ability to turn ES6 version into code that runs in your browser (or on your server) today. Remember it just transpiles and not bundles your code.

Use Rollup for libraries and Webpack for apps.

Transpile(Babelify) the source or Bundle it?

Again there’s a story behind it. 😄

Most of my time was spent to figure out answer to this question when I was building this library. I started digging out my node_modules to lookup all the great libraries and check out their build systems.

Libraries vs Packages build output comparision
Libraries vs Packages build output comparison

After looking at different libraries/packages build output, it gave me a clear picture of what different strategies and purpose the authors of these libraries think before publishing their library/package. Below are the observations.

As you can see the above image, I’ve divided them into 2 groups based on their characteristics:

  1. UI Libraries (styled-componentsmaterial-ui)
  2. Core Packages (reactreact-dom)

If you’re a good observer 😎 you might have figured out the difference between these two groups

1. UI Libraries
  • There is a dist folder that has the bundled and minified version for ESand UMD/CJS module system as a target.
  • There is a lib folder that has the transpiled version of the library.
2. Core Packages
  • There is just one folder which has bundled and minified version for CJS orUMD module system as a target.

But why there is a difference in build output of UI libraries and Core Packages? 🤔

UI Libraries
  • Imagine if we just publish the bundled version of our library and host it on CDN our consumer will use it directly in a<script/> tag. Now if my consumer wants to use just the <Button/> component he has to load the entire library. Also in a browser there is no bundler which will take care of tree shaking and we’ll end up shipping the whole library code to our consumer. ❌
<script type="module">
import {Button} from "https://unpkg.com/uilibrary/index.js";
</script>
  • Now if we simply transpile the src into lib and host the lib on a CDN, our consumers can actually get whatever they want without any overhead.

“Ship less, load faster”. ✅

<script type="module">
import {Button} from "https://unpkg.com/uilibrary/lib/button.js";
</script>
Core Packages
  • Core packages are never utilized via <script/> tag, as they need to be part of main application, hence we can safely release the bundled version (UMD, ES) for these kind of packages and leave up to the consumers build system. Example they can use the UMD variant but no tree shaking or they can use the ES variant if the bundler is capable to identify and get the benefits of tree shaking.
// CJS require
const Button = require("uilibrary/button");
// ES import
import {Button} from "uilibrary";
But…
Transpile(Babelify) the source or Bundle it? 🤔
UI Library
  1. Transpile the source with babel with es module system as a target, place it inlib. We can even host the lib on a CDN.
  2. Bundle and minify the source using rollup, for cjs/umd module system and es module system as a target. Modify the package.json to point to the proper target systems.
// package.json
{
"name": "js-module-system",
"version": "0.0.1",
...
"main": "dist/index.js",      // for umd/cjs builds
"module": "dist/index.es.js", // for es build
...
}
Core Packages
  1. We don’t need the lib version in this case.
  2. Bundle and minify the source using rollup, for cjs/umd module system and es module system as a target. Modify the package.json to point to the proper target systems same as above.

Tip: We can host the dist folder on CDN as well, for the consumers who are willing to download the whole library/package via <script/> tag.

How to Build?

We should have different scripts for each target system in package.json . You can find the rollup config in the in the GitHub repo.

// package.json
{
...
"scripts": {
"clean": "rimraf dist",
"build": "run-s clean && run-p build:es build:cjs build:lib:es",
"build:es": "NODE_ENV=es rollup -c",
"build:cjs": "NODE_ENV=cjs rollup -c",
"build:lib:es": "BABEL_ENV=es babel src -d lib"
}
...
}
What to Publish?
  • License
  • README
  • Changelog
  • Metadata(main , modulebin) — package.json
  • Control through package.jsonfiles property

In package.json"files" field is an array of file patterns that describes the entries to be included when your package is installed as a dependency. If you name a folder in the array, then it will also include the files inside that folder.

We will include lib and dist folder in "files" field in our case.

// package.json
{
...
"files": ["dist", "lib"]
...
}

Finally the library is ready to publish just type npm run build command in the terminal and you can see the following output. Closely look at the dist and lib folder. 🎉

Ready to publish 😎

Wrap up! 🗞

Wow! Where does the time go? That was a wild ride, but I sincerely hope it gave you a better understanding of JavaScript Module system and how you can create your own library and publish it by ensuring following things:

  1. Make it Tree Shakeable. 🌳
  2. Target at least ES Harmony and CJS module systems. 👍
  3. Use Babel and Bundlers for libraries. 💄
  4. Use Bundlers for Core packages. 📦
  5. Set module field of package.json to point to the ES version of your module (PS: It helps in tree shaking). 👻
  6. Publish the folders which has transpiled as well as bundled versions of you module. 🏭
Trending 📈
  1. Webpack-V4 alpha released. 😍
  2. ParcelJs: Blazing fast, zero configuration web application bundler. 📦
  3. Turbo: 5x faster than Yarn & NPM, and runs natively in-browser 🔥

Thanks to Juho Vepsäläinen and Lakshya Ranganath for their reviews & feedbacks, Sean T. Larkin and Tobias Koppers for sharing the insights of webpack at ReactiveConfAddy Osmani for sharing workings of different JS module Systems in “Writing Modular JavaScript With AMD, CommonJS & ES Harmony”.


Comment here