Migrating an Ember addon to the next-gen v2 format

Embroider, Ember's next generation build system, introduced a new format v2 for publishing addons. It is specified in detail in the corresponding RFC. Please read that document if you are interested in all the details, as I did, especially again before starting this journey here. But to summarize the key concepts in my own words:

• addons will become less special, less "Ember-ish", more like "just a regular npm package".
• v1 addons were able to hook into Ember CLI's build system using Broccoli hooks (e.g. treeForX), v2 addons aren't. And the vast majority won't need this. They are static, ES modules based vanilla JS.
• the spec defines an Ember Language Standard, that specifies what language features you can use in your published package. Anything beyond that needs to be compiled away, as was the case before with e.g. ember-cli-babel, but this time this does not happen at build-time of the app, but at publish-time of the addon.

But why all the fuzz?

This will yield many benefits when Embroider takes over our legacy build system:

• faster builds and NPM installs.
• existing build tooling, that the Ember community does not have to reinvent, like Webpack and many more, can understand these packages natively. (mostly at least, some remaining Emberisms like Handlebars templates still have to be compiled away)
• this enables faster progress and more experimentation in the domain of build pipelines. Important features that the Ember ecosystem has been lacking for a way too long time like tree-shaking and code splitting are supported easier, without Embroider's current complex compatibility layer.

But I am not on Embroider yet!?

Don't worry! Although v2 addons are only understood natively by Embroider, there is a compatibility layer to make a v2 addon behave like a v1 addon in a classic Ember CLI build. Basically we are shipping a hybrid addon, that is fully backwards-compatible. The only requirement for your app is to use ember-auto-import v2!

Ok, enough theory, let's get our hands dirty!

And we will! In the following I will walk you through the essential steps I needed to do to migrate my ember-stargate addon to the v2 format. I deliberately picked this addon, as it is fairly simple - just two components and a (private) service - and does not need any special build tooling like TypeScript compilation.

⚠️ Disclaimer: if you maintain an Ember addon yourself, you may ask if you should do the same for your addons? The general answer is… no. Probably not. Not yet at least.
This was an experimental journey, to learn und understand what it takes. But it did take me a whole day, to get everything right and working. And I am probably not going to do that again manually for all private and public addons we maintain, at least in the near future. But this experience and that of all other early adopters will hopefully still be very useful, to distill best practices and mold those into better tooling, like eventually migration tools or at least an alternative addon blueprint like ember addon --v2. Having said that, if you identify yourself as enjoying being on the experimental side of things and willing to go the extra mile, then yes, please try it and contribute your experiences!

Choosing your test-app strategy

I promised we will get our hands dirty, but not quite yet unfortunately. We first have to decide how we will structure our packages, especially in regard to the test-app package we need.

We need a test-app package? Why even?

Yes, we need it. We need an app to run our addon within. Ember's test infrastructure expects that. And in fact we always had it, also in v1 addons. In lack of a better name, it was just called "dummy"!

But the way the dummy app was integrated into a classic addon repo is quite awkward for several reasons, to name just a few:

• the devDependencies listed in your addon's package.json are basically dependencies of the dummy app, not your addon.
• the ember-cli-build.js file at the root of the repo is responsible for building the dummy app, and not the addon.

Our v2 addon will not need Ember CLI for its build process, including the test setup we know from v1 addons. So instead we will have a separate regular Ember app inside our addon's repo.

Test-app in a regular repo

One way to do so, is to create a test-app in an arbitrary sub folder of your regular repo, say /test-app. But this requires two things to correctly work:

• first, the app's dependencies need to get installed correctly, i.e. not only the dependencies of your addon in /package.json, but also the devDependencies of /test-app/package.json
• second, there is one exception to that, as the actual addon should not get installed into the test-app via npm, but rather the current addon code of the repo should be sym-linked into the test-app

The @embroider/addon-dev package provides a command to help with these two tasks. But they have to be called manually, or semi-automated in the case of the sym-linking part by creating a prepare npm script.

Monorepo

The other approach is to migrate your addon's repo to a real monorepo. By using yarn workspaces to manage two separate packages in the same repo, the two tasks mentioned above are basically handled for you automatically. While setting up a monorepo can come with its own difficulties, it also helps with other aspects:

• you can make your existing ember-try based test matrix easily work in a monorepo, by only letting it manage the (clearly isolated) dependencies of the test-app.
• other tooling such as dependency managers like Renovate perfectly support monorepos, while they are likely to fail for a regular repo that has separate package.json files as in the first case.
• when you want to create a documentation app for your addon, this can be done easily with a cleanly separated own package.

From my own experience, mixing the concerns of testing and documentation in a single dummy app can lead to painful problems. For example, you may want to use a dependency for your docs app that is not compatibly with your addon's Ember support matrix, thus messing up ember-try test scenarios. Welcome to dependency hell!

While I started with the first approach, as it seemed to be the less invasive change, I later decided to rather go with a monorepo, for the reasons mentioned above. So in the following we will set up the addon to use a monorepo.

Migrating to the new package structure

As we have chosen the monorepo approach, we can now create two separate packages in the same repo:

• /packages/ember-stargate for the addon, and
• /packages/test-app for, well, guess what…

The folder structure here is what I chose, but you are free to organize your packages in any way, as long as the workspaces configuration is correctly configured, as described below.

Setting up the addon package

Tackling the addon package first, we will put all addon code into a src sub folder. So basically we can move all the existing code from /addon to /packages/ember-stargate/src. The classic addon also had an /app subtree which contained only re-exports of addon code to make that available automatically in the app's namespace. A v2 addon does not have this concept anymore, instead references to that "App JavaScript" are defined in the addon meta data of the addon's package.json. We will deal with that later, so for now we can delete the whole old /app tree.

We can also move the package.json and config files for linting into the new package folder.

But now we also need a package.json at the repo's root, to set up the workspaces. In the most simple case that can look like this (but you should add the other usual metadata, like the repository link, license information etc.):

{
  "workspaces": [
    "packages/*"
  ],
  "private": true
}

Cleaning up dependencies

Now let's clean up the addon's dependencies in its packages/ember-stargate/package.json. As said before, most devDependencies were actually related to the test-app (formerly dummy-app). So we can remove these, except for linting tools like eslint or ember-template-lint, which still run on the actual addon.

Also, some dependencies are not needed anymore, which were used to build the addon code within the consuming app. Remember, we are publishing (mostly) static code now. So v1 addons that plug into the classic build pipeline can be removed now from our dependencies. In most cases - also here - these are ember-cli-babel and ember-cli-htmlbars. Other dependencies, that provide runtime code for your addon, need to stay though!

Addon-shim

We mentioned in the beginning that our v2 addon will be backwards-compatible with the classic build pipeline too, by using a shim to make it also behave like a v1 addon. This functionality is provided by the @embroider/addon-shim package, which we will add to the addon's dependencies.

Now we need to create a addon-main.js file (it could have any arbitrary name, but we will stick to this convention). It is basically what our v1 addon's index.js was before, so this is what a classic Ember CLI build will consume as the addon's entrypoint. But now the shim will expose the v1 addon API, by wrapping our v2 addon:

const { addonV1Shim } = require('@embroider/addon-shim');
module.exports = addonV1Shim(__dirname);

Add meta data

Classic addons already used an ember-addon field in their package.json for addon-specific metadata. v2 addons require some additional fields:

{
  "ember-addon": {
    "version": 2,
    "type": "addon",
    "main": "addon-main.js"
  }
}

Creating the test-app

Now it's time to create our second package, our test-app. Unlike the usual dummy app, this is now a plain Ember app with a very conventional setup, so we can use Ember CLI's app blueprint to scaffold the app:

ember new test-app --skip-npm --skip-git --no-welcome --directory packages/test-app

We need to make sure the app has ember-auto-import v2 in its devDependencies, and add the addon itself!

Now when we run yarn in the repo's root folder, it should install all dependencies of both packages, and sym-link the addon.

Finally, we can move the existing tests into the tests folder of the test-app, and remove the root /tests folder.

Build setup

Now we'll set up the v2 addon's build tooling. As we learned before, a v2 addon is publishing mostly static JavaScript that does not need many build steps at built-time of the app. So here we refer to the build steps that we run at publish-time, so the final artefact that gets published into the npm registry adhering to the v2 addon spec.

Note: while it is totally possible to author your addon code in a way that is already v2 compliant, thus eliminating the need for an additional publish-time build, this is not always very convenient. For example, you would need to explicitly import and associate component templates to the corresponding JavaScript module via the setComponentTemplate() API.

Luckily Embroider provides some tooling for this, based on rollup and babel. So we install those in the addon package:

cd packages/ember-stargate
yarn add -D @embroider/addon-dev rollup @babel/core @rollup/plugin-babel @babel/plugin-proposal-decorators @babel/plugin-proposal-class-properties

Then we need to copy the sample rollup config and customize it to our needs. Ours looks like this now:

import babel from '@rollup/plugin-babel';
import { Addon } from '@embroider/addon-dev/rollup';

const addon = new Addon({
  srcDir: 'src',
  destDir: 'dist',
});

export default {
  // This provides defaults that work well alongside `publicEntrypoints` below.
  // You can augment this if you need to.
  output: addon.output(),

  plugins: [
    // These are the modules that users should be able to import from your
    // addon. Anything not listed here may get optimized away.
    addon.publicEntrypoints(['components/**/*.js', 'services/**/*.js']),

    // These are the modules that should get reexported into the traditional
    // "app" tree. Things in here should also be in publicEntrypoints above, but
    // not everything in publicEntrypoints necessarily needs to go here.
    addon.appReexports(['components/**/*.js', 'services/**/*.js']),

    // This babel config should *not* apply presets or compile away ES modules.
    // It exists only to provide development niceties for you, like automatic
    // template colocation.
    // See `babel.config.json` for the actual Babel configuration!
    babel({ babelHelpers: 'bundled' }),

    // Ensure that standalone .hbs files are properly integrated as Javascript.
    addon.hbs(),

    // addons are allowed to contain imports of .css files, which we want rollup
    // to leave alone and keep in the published output.
    addon.keepAssets(['**/*.css']),

    // Remove leftover build artifacts when starting a new build.
    addon.clean(),
  ],
};

And we add a babel.config.json like this:

{
  "plugins": [
    ["@babel/plugin-proposal-decorators", { "legacy": true }],
    "@babel/plugin-proposal-class-properties",
    "@embroider/addon-dev/template-colocation-plugin"
  ]
}

If you need additional transforms for your code, then you can install those plugins and add them here.

The first two plugins are needed for transpiling decorators. Actually this shouldn't be necessary, as the Ember Language Standard explicitly supports decorators. So ideally we would leave code using decorators untransformed. But due to a still unresolved issue this does not work currently, so for now we are using those plugins to compile decorator usage away.

Note that you do not need to transpile your code here to something like ES5 here! The app that consumes your addon will still do this. But you must make sure that the published code adheres to the already mentioned Ember Language Standard!

When running rollup --config this will now do all of this for us:

• process the entrypoints (here our components and the service) through babel.
• associate colocated component templates with their JavaScript class
• add the app-js entrypoints to the addon's package.json. This is the replacement for the classic /app tree mentioned before!
• eventually copy static CSS (not applicable in our case)
• put the final publishable files into a dist folder

Scripts and other metadata

To make building, testing and publishing more convenient, we should add some scripts and additional metadata to our package.json file(s).

What to publish

First we should narrow what files get actually pushed into the npm tarball when publishing the addon. Most often this was done by maintaining an exclude-list in the form of an .npmignore file. But this is easy to get out of sync. And as we have a build step now that puts all the published artefacts into a dist folder, we can now much easier define this with an include-list based approach, by deleting the .npmignore file and adding this to our packages/ember-stargate/package.json:

{
  "files": [
    "addon-main.js",
    "dist"
  ]
}

Where to import from

There are two ways for addon code to be used in the Ember world: implicitly, when the addon exposes the classes (e.g. components or services) in the app namespace, so it can be looked up by Ember's resolver and Dependency Injection system - remember the /app tree in a classic addon! And explicitly, when importing from an addon's module.

The former we have already covered by the build step's app re-exports, which were configured in the rollup config file explained above. But for the latter we have to make sure that whatever bundler is used (most likely webpack when using ember-auto-import or Embroider) it will understand to look for the module files we published into the dist folder when it needs to import them. While keys like main in package.json can be used for this, the newer exports field, which is supported by webpack and node.js itself allows for more granular control:

{
  "exports": {
    "./*": "./dist/*",
    "./addon-main.js": "./addon-main.js"
  }
}

Handy scripts

The scripts added to the part with the same name of your package.json make invoking common tasks easy. We already have those in place for the test-app, given by the Ember CLI app blueprint. But we should add some for our v2 addon, which again is not based on Ember CLI anymore, so no more ember s to easily start your addon.

Well, if you read this blog post carefully, you will have noticed the convoluted interweaving of an addon and its dummy app, so ember s actually did not start your addon, but your addon's dummy app! 🙃

So these are the scripts we defined for our addon (i.e. packages/ember-stargate/package.json):

{
  "scripts": {
    "build": "rollup --config",
    "lint": "npm-run-all --aggregate-output --continue-on-error --parallel \"lint:!(fix)\"",
    "lint:fix": "npm-run-all --aggregate-output --continue-on-error --parallel lint:*:fix",
    "lint:hbs": "ember-template-lint .",
    "lint:hbs:fix": "ember-template-lint . --fix",
    "lint:js": "eslint . --cache",
    "lint:js:fix": "eslint . --fix",
    "start": "rollup --config --watch",
    "test": "echo 'Addon does not have tests, run tests in test-app'",
    "prepare": "yarn build"
  }
}

The linting script is the common stuff you already know from classic addons, but this is new:

• build: will invoke our build step as described in the section above.
• start: will do the same, but watch all affected source files, so whenever you change them a re-build is triggered.
• test: this is just a dummy entry, which is needed for… reasons. As said before, the tests for our addon live in the test app, so this is basically a noop.
• prepare: this one is important, as it runs our previously mentioned build script whenever you do a yarn install or before you publish your addon, so that the compiled dist folder is correctly populated with your current compiled code.

We have now setup scripts for our addon's workspace, but those will only be used when you cd into that folder. But we also should be able to run common scripts right from the repo's root. So we will basically set up some forwarding:

{
  "scripts": {
    "prepare": "yarn workspace ember-stargate run prepare",
    "start": "npm-run-all --parallel start:*",
    "start:addon": "yarn workspace ember-stargate run start",
    "start:test-app": "yarn workspace test-app run start",
    "lint": "npm-run-all --aggregate-output --continue-on-error --parallel lint:*",
    "lint:addon": "yarn workspace ember-stargate run lint",
    "lint:test-app": "yarn workspace test-app run lint",
    "lint:fix": "npm-run-all --aggregate-output --continue-on-error --parallel lint:fix:*",
    "lint:fix:addon": "yarn workspace ember-stargate run lint:fix",
    "lint:fix:test-app": "yarn workspace test-app run lint:fix",
    "test": "npm-run-all --aggregate-output --continue-on-error --parallel test:*",
    "test:addon": "yarn workspace ember-stargate run test",
    "test:test-app": "yarn workspace test-app run test"
  }
}

Here we are

• forwarding the prepare lifecycle script to our addon's workspace, as yarn unfortunately does not support lifecycle scripts for workspaces.
• forwarding *:addon scripts to the addon, and *:test-app scripts to the test-app workspace.
• invoking those in parallel for our common operations like lint, test and start. We have to use npm-run-all here, as the yarn workspaces command does not support parallel execution across workspaces, at least for v1 of yarn which we are (shame on me) still using here.

So with this, when you hit yarn start for example, it will build and watch the v2 addon using rollup, while running ember serve on the test app. Whenever some addon code is changed, it will get re-compiled, and the test app will also correctly pick up those changes, rebuild and reload.

Publishing

When everything is working fine up to this point, i.e. our development setup works and all tests are passing, then it is up for the final step: publishing our addon in the shiny new v2 format!

Before we do so, we should run npm pack in the addon's folder (here packages/ember-stargate). This will emit the same tarball that npm publish would upload to the npm registry. With this we can make sure everything works as expected:

• our prepare script runs and compiles our addon code.
• the tar archive contains those compiled modules in our dist folder, alongside our v1 addon shim addon-main.js.

With this hurdle also mastered, we can think how to actually set up our publishing workflow. While we could do this manually, there are some things that are a bit more involving now with our monorepo setup. For example the version of our addon that is referenced in the test app should also be bumped when cutting a new release of the addon itself.

Others have used tools like lerna for managing their monorepo, but I had good experiences so far with only using the basic workspaces support of yarn for managing dependencies in a monorepo, and using release-it together with some plugins for publishing.

For the latter I recommend Robert Jackson's fantastic setup script, which will autodetect our monorepo and add the appropriate plugins, including one for generating a professional Changelog file. Please refer to its documentation for more details. With this, publishing our new addon will be as simple as running release-it in the root folder of our repo, all the rest will be taken care of!

Don't worry, our test app won't be published, as it has the private flag in its package.json!

Final words

Finally, after this long journey we have come to an end, and have converted our v1 addon into a shiny new modern v2 addon! ✨

As I said in the beginning, this is still experimental stuff. Just as it was not very useful for many React devs to go into panic mode the second the new (back then) hooks concept has been announced and to rewrite their app, Ember addon authors also don't have to rush into this. Likely some things proposed here will not stand the test of time, and will be replaced with better alternatives. This is a moving target, and best practices will evolve. Again, I expect that eventually the experiences of early adopters like me and others will be distilled into a new addon blueprint, that sets the new gold standard for v2 addon development.

But if that warning does not hold you off, then I hope this blog post will be useful for trying this out by yourself!

For the record, besides ember-stargate there are at the time of writing these other v2 addon that I am aware of, some of which follow a different path towards the same goal:

• ember-welcome-page
• glimmer-apollo
• ember-resources

Last but not least, I need to thank Ed Faulkner, not only for an early review of this blog post, but most importantly for the enormous effort that went into Embroider as a project - not only writing the core of it, but also all the tooling, documentation, issue triaging and other communication around it! 🙏