Skip to main content

A gentle intro to npm workspaces, with visuals

Β· 15 min read
Carlos Precioso
Carlos Precioso
Framework Engineer @ Wasp

I am a founding engineer at Wasp - a full-stack TypeScript framework based on React, Node.js, and Prisma. Like Rails, but for JavaScript. A Wasp app is built as a multi-package setup - frontend, backend and shared code all live in a single repo as separate packages managed by npm workspaces.

Getting this right was harder than expected, and this post is what we learned. We'll dig into how npm workspaces actually work: how dependencies get resolved and hoisted, how Node.js finds modules at runtime, and help you build intuition/mental model that will come in useful when debugging this setup.

Your side project started as a single folder with a hundred lines of code. Now it's a full-stack app with a React frontend, an Express backend, and a shared library of types and helpers that both sides need. You might be copy-pasting files between folders like it's 2005, and the cracks are showing: a type definition got updated in one place but not the other, and now your API responses don't match what the frontend expects. Something has to change.

The first step would probably be splitting your code into multiple packages. But without the right tools, you'll quickly find yourself drowning in running npm install in five different folders, or fighting runtime errors because React is installed three times.To fix this, modern package managers introduced workspaces. Intended to allow you to manage multiple packages within a single repository without the dependency headaches, workspaces are widely used in many projects, but their inner workings are often misunderstood.

In this post, we'll dig into the magic that makes them tick: how package managers resolve shared dependencies across packages, and how Node.js finds them at runtime.

Enter workspaces​

Workspaces are a feature in npm, Yarn, pnpm, and others that let you organize multiple JavaScript packages in a single project. And while each package acts as its own encapsulated unit, they can still depend on one another with no extra ceremony.

Diagram comparing dependency management with and without workspaces. Without workspaces two separate projects each install their own copy of their dependencies, and are isolated from one another. With workspaces the same two projects share their common dependencies, can call each other.

Two main behaviors make this work: shared dependencies (if two packages depend on react, they both use the same installed version) and cross-package imports (a pkg-a can just import pkg-b by name).

The building blocks are interesting and require a dive into how package managers and Node.js work. The feature also comes after a long evolution of tooling and JS development practices, so let's start with some history.

A brief history​

It seems like the second half of the 2010s was when the professionalization of JS development started to trickle down to the average developer. The rise of Babel and Webpack gave us a modern development experience, one that more closely resembled what other languages had enjoyed for years. We finally had a proper module system, and the ability to target newer language versions, while still supporting older runtimes.

Our history of workspaces starts in 2015, when the 6to5 project realized they didn't want to work only on ES6 code, but also on everything that came after it. They renamed themselves to Babel and started working on modularizing the codebase, knowing they'd need well-separated concerns to keep up with the ever-evolving language.

Babel maintainers wanted to distribute the new version as a collection of plugins you could pick and choose, but couldn't afford to manage each one independently. So when they split all their code into different packages, they did so in the shape of a monorepo. That decision kickstarted the development of what would eventually be called workspaces.

Their internal tooling was soon released as Lerna. Lerna's main job was to allow packages in a monorepo to seamlessly require() one another as if they were already published, instead of having to edit, publish, and reinstall each one independently.

A few months later in 2017, Yarn came along. Yarn was created to solve some problematic behaviors of npm at the time, and dramatically improved performance and reliability. Alongside features like reproducible lockfiles and heavy caching, Yarn also introduced workspaces as a first-class citizen. This was no surprise, as Yarn was created by some of the same people who had worked on Babel and Lerna.

Soon, Yarn exploded in popularity, and in time, other package managers followed suit. pnpm added workspaces support in 2018, and npm followed in 2020. Nowadays, after a long evolution, workspaces are supported in all the mainstream package managers and widely used in both open and closed source projects.

Problems and solutions​

The exact definition of what a workspace is depends on the package manager you're using. The definition I gave above is more of an observation of what the main package managers have in common than a prescriptive specification. There isn't a common "workspaces spec". Each package manager has its own take, with different configuration formats, commands, and even different ways to declare dependencies.

That said, there's a core of functionality you can expect across package managers: you declare a root folder that contains multiple sub-packages. Each sub-package is called a workspace, and the whole set is a project1. From there, three features usually become available:

  • Each workspace is its own full package, as if it were an independent project. Each workspace has its own package.json, where it declares dependencies, configuration, and scripts without worrying about the insides of other packages.

  • All dependencies are resolved together across the project, as if it were a single package. If two workspaces depend on the same package, it gets installed once and shared (assuming compatible version ranges).

  • Workspaces can depend on each other by name, as if they were regular dependencies. In a project with many workspaces, pkg-a can import pkg-b and everything works.

Workspaces give us the best of both worlds: the encapsulation of small packages with the convenience of shared dependencies.

The first feature is just the natural way of working with multiple packages, so it doesn't require much explanation. Let's look at the other two: why they're useful and how they're implemented. The exact implementation details depend on the package manager, so we'll use npm as a reference here, since it's the most widely used. The general concepts apply across the board.

The dependencies of the whole project are resolved together​

The problem​

Imagine testing a project with many interconnected packages ...say, 157 of them. You'd have to find out which packages depend on which others, run npm install in each folder, and deal with the ongoing headache of keeping dependency versions in sync. Worse: if each package had its own node_modules, you'd end up with tons of copies of the same dependencies installed in different places.

Multiple copies aren't just a waste of disk space, they can cause bugs. In JavaScript, the same class defined in two different files is considered different, even if the code is identical2:

foo/my-class.js
export class MyClass {}
bar/my-class.js
export class MyClass {}
main.js
import * as foo from "./foo/my-class";
import * as bar from "./bar/my-class";

new foo.MyClass() instanceof foo.MyClass; // true
new foo.MyClass() instanceof bar.MyClass; // false!

For shared dependencies to work correctly, they need to be installed once, in a single place. That way, when different pieces of code import the same dependency, they get the same object.

The solution​

Most package managers already deduplicate dependencies for a single project. So to get this behaviour for free with workspaces, they cheat a bit: they treat all workspaces as dependencies of a single, "fake" root package. The existing resolution algorithm gets reused without changes.

You can see this in any workspaces lockfile. Here's a simplified example for a regular package with no workspaces:

The package depends on both left-pad and right-pad, which both depend on core-pad with different version ranges. The package manager found a single version of core-pad that satisfies both and installed it once.

Here's the same scenario in a workspaces project:

By pretending all workspaces are dependencies of a single root package, the package manager reuses its existing algorithm. It finds one version of core-pad that works for both, installs it once, and any workspace that imports it gets the same object.

Workspaces can depend on each other by name​

The problem​

If you have two separate packages and want one to depend on the other, you have two options:

  • Publish the dependency to the registry, then install it in the other package. Straightforward, but if you're iterating quickly, the publish-install cycle kills your flow.

  • Use a local file dependency. Edit the imports to point to the local path. This works, but it's brittle: if you move packages around or share the code with someone else, paths break. If the dependency has its own dependencies, you have to install them manually. And you need to remember to switch paths back before publishing.

Neither is great. What you actually want is to say "my package depends on pkg-b" and have the package manager figure out that pkg-b is just in the folder next to you. That's exactly what workspaces do!

The solution​

This seems straightforward to implement. The package manager knows where all workspaces are, so it could point to the right folder when it finds a matching dependency name. But there's a catch: the dependency resolution of the package manager (npm, Yarn) is separate from the import resolution of the runtime (Node.js).

If you've ever looked into your node_modules and found dependencies nested inside other dependencies instead of a flat structure, this is why. The package manager has no influence on how Node.js finds imports3, all it can do is lay them out on disk in a way that Node.js will understand.

How Node.js finds packages​

The high-level idea is simple:

  • If the import is a relative or absolute path (./, ../, or /), Node.js follows that path directly.
  • Otherwise, it looks for the import name in the nearest node_modules folder, starting from the current file's directory, walking up the folder tree until it finds a match or reaches the filesystem root.

For example, if we want to import e.g. left-pad, Node.js checks these folders in order:

  • From ~/projects/app/src/utils.js
  1. ~/projects/app/src/node_modules/left-pad
  2. ~/projects/app/node_modules/left-pad
  3. ~/projects/node_modules/left-pad
  4. ~/node_modules/left-pad

We know we don't want multiple copies of the same workspace in different node_modules folders, because that would bring back all the problems we discussed. Instead, we can use symlinks. As a refresher: they are special files that just "redirect" to another file or folder when accessed.

So now, this just becomes a problem of how to arrange the symlinks in the node_modules folders for Node.js. And the solution: we'll create a symlink to each workspace in the top-level node_modules folder. Since Node.js walks up the directory tree looking for that folder, every workspace will eventually find the root one:

  • package-root/
    • node_modules/
      • sub-a (symlink to sub-a)
      • sub-b (symlink to sub-b)
      • sub-c (symlink to sub-c)
    • sub-a/
      • package.json
      • ...
    • sub-b/
      • package.json
      • ...
    • sub-c/
      • package.json
      • ...

Here's an example:

package-root/sub-a/index.js
import { foo } from "sub-b";
console.log(foo());
package-root/sub-b/index.js
export function foo() {
  return "Hello from sub-b!";
}

How does Node.js resolve the import in sub-a/index.js?

  • package-root/sub-a/node_modules/sub-b
    β†’ doesn't exist
  • package-root/node_modules/sub-b
    β†’ symlink to package-root/sub-b/, follow it
  • package-root/sub-b
    β†’ found it!

By placing symlinks in the top-level node_modules, any workspace can import any other workspace by name, and Node.js finds it correctly.

When (not) to use workspaces​

While workspaces are pretty useful, they are not a silver bullet. That is, you shouldn't go link every project in your ~/dev folder just yet. Instead, they should be treated as a scoped feature, with a clear sweet spot.

In general, workspaces work best for projects that:

  • Consist of multiple packages
  • Will be developed in tandem
  • Share many dependencies
  • Call each other frequently

That's why workspaces are mostly used in monorepos: almost by definition, they satisfy all of these criteria. Wasp uses them outside the monorepo context, but that's because we generate packages for each part of your app, that are tightly coupled and evolve together.

On the other hand, I'd discourage using workspaces when:

  • You don't have a good reason to split a single codebase or repo into multiple packages. Workspaces solve problems, but it's better if you don't have those problems in the first place.
  • Your packages are mostly independent and don't share many dependencies. I wouldn't use workspaces to link all the random libraries in my ~/dev folder. It could lead to random interference between dependencies, or implicit relationships.
  • You have cyclic dependencies. Without a clear story of which packages depend on which4, workspaces can allow any package depend on any other carelessly, and that can lead to a tangled web of dependencies.
  • Your project is split across multiple repositories. Workspaces would make them no longer self-contained. Unless you're very sure of your use case, either unlink the repos altogether or merge them into a single monorepo.

Why does Wasp care about workspaces?​

When you run the Wasp compiler, it generates app code into a .wasp/ folder inside your project. And part of that generated code is three different packages: one for the SDK we generate from your Wasp spec, and a frontend and backend package that contain the final codebase for your deployed apps.

Those three packages are tightly coupled, with frequent imports across packages, and also to your own code. They also share many dependencies, and we want to avoid any chance of mismatches or duplication between them.

As part of our push for Wasp 1.0, we wanted to make this situation easier to work with, and we realized workspaces were a perfect fit for our problems. We worked on a project internally called Wasp Citizen, so we could capitalize on workspaces to ease our dependency mismatch issues.

Since Wasp v0.19, this is at work in every Wasp project. Your app still works as a single-package project and you don't need to learn anything new; but internally, the generated packages are structured as workspaces. This translates to a better experience for you, fewer weird bugs for us, and quicker installs for everyone.

And with this knowledge, if you want, now you can add your own workspaces to your Wasp project too, and have them play nicely with the generated ones!

Go off!​

Workspaces are a powerful tool that help you organize your code, avoid dependency errors, and speed up your development workflow. I hope this post has given you a good understanding of how they work under the hood and when to use them. If you find yourself in a situation where workspaces could help you, give them a try!

Until next time, happy coding!

Footnotes​

  1. Yarn has a great glossary that you can check out if you want to get specific definitions for a package-manager-related term, and pointers to which feature they're a part of. ↩

  2. This is one of the reasons React requires a single copy in your app. React relies on a single instance to track internal state (hooks, context, and reconciler). If you had two copies, components created with one wouldn't be recognized by the other, leading to cryptic errors. ↩

  3. Some newer runtimes have the package manager integrated directly into execution, so they can control both installation and resolution, and simply point to the right folder. For example, Deno doesn't even install dependencies in your project folder. Instead, it keeps a single global copy in an internal cache that it can programmatically reference. ↩

  4. Yarn has some great features that force you to be exhaustive about how your workspaces are structured. By default, you have to explicitly declare which other workspaces you depend on. They also have a workspace constraints feature that lets you enforce rules on which packages can depend on which others. This one has a steep learning curve, but it's worth it if you're working on a very big monorepo. ↩

Discord

Join our developer community

Wasp is 100% open source. Join our Discord to learn from others and get help whenever you need it!

Join our Discord πŸ‘Ύ
β†’
πŸ“«

Subscribe to our newsletter

Once per month - receive useful blog posts and Wasp news.