Hacking on Atom Part II: Building a Development Process

Over one year ago, I wrote a blog post: “Hacking on Atom Part I: CoffeeScript”. This post (Part II) is long overdue, but I kept putting it off because I continued to learn and improve the way I developed for Atom such that it was hard to reach a “stopping point” where I felt that the best practices I would write up this week wouldn't end up becoming obsolete by something we discovered next week.

Honestly, I don't feel like we're anywhere near a stopping point, but I still think it's important to share some of the things we have learned thus far. (In fact, I know we still have some exciting improvements in the pipeline for our developer process, but those are gated by the Babel 6.0 release, so I'll save those for another post once we've built them out.)

I should also clarify that when I say “we,” I am referring to the Nuclide team, of which I am the tech lead. Nuclide is a a collection of packages for Atom to provide IDE-like functionality for a variety of programming languages and technologies. The code is available on GitHub (and we accept contributions!), but it is primarily developed by my team at Facebook. At the time of this writing, Nuclide is composed of 40 Atom packages, so we have quite a bit of Atom development experience under our belts.

My Transpiler Quest

When I spun up the Nuclide project, I knew that we were going to produce a lot of JavaScript code. Transpilers such as Traceur were not very mature yet, but I strongly believed that ES6 was the future and I didn't want to start writing Nuclide in ES5 and have to port everything to ES6 later. On a high level, I was primarily concerned about leveraging the following language extensions:

• Standardized class syntax
  
• async/await
  
• JSX (for React)
  
• type annotations
  

Note that of those four things, only the first one is specified by ES6. async/await appears to be on track for ES7, though I don't know if the TC39 will ever be able to agree on a standard for type annotations or let JSX in, but we'll see.

Because of my diverse set of requirements, it was hard to find a transpiler that could provide all of these things:

• Traceur provided class syntax and other ES6 features, but more experimental things like async/await were very buggy.
  
• TypeScript provided class syntax and type annotations, but I knew that Flow was in the works at Facebook, so ultimately, that is what we would use for type annotations.
  
• React came with jstransform, which supported JSX and a number of ES6 features.
  
• recast provided a general JavaScript AST transformation pipeline. Most notably, the regenerator transform to provide support for yield and async/await was built on recast.
  

Given my constraints and the available tools, it seemed like investing in recast by adding more transforms for the other features we wanted seemed like the most promising way to go. In fact, someone had already been working on such an endeavor internally at Facebook, but the performance was so far behind that of jstransform that it was hard to justify the switch.

For awhile, I tried doing crude things with regular expressions to hack up our source so that we could use regenerator and jstransform. The fundamental problem was that transpilers do not compose  because if they both recognize language features that cause parse errors in the other, then you cannot use them together. Once we started adding early versions of Flow into the mix to get type checking (and Flow's parser recognized even less of ES6 than jstransform did), the problem became even worse. For a long time, in an individual file, we could have async/await or type checking, but not both.

To make matters worse, we also had to run a file-watcher service that would write the transpiled version of the code someplace where Atom could load it. We tried using a combination of gulp and other things, but all too often a change to a file would go unnoticed, the version on disk would not get transpiled correctly, and we would then have a subtle bug (this seemed to happen most often when interactive rebasing in Git).

It started to become questionable whether putting up with the JavaScript of the future was worth it. Fortunately, it was around this time that Babel (née 6to5) came on the scene. It did everything that we needed and more. I happily jettisoned the jstransform/regenerator hack I had cobbled together. Our team was so much happier and more productive, but there were still two looming issues:

• Babel accepted a much greater input language than Flow.
  
• It did not eliminate the need for a file-watching service to transpile on the fly for Atom.
  

Rather than continue to hack around the problems we were having, we engaged with the Flow and Atom teams directly. For the Flow team, supporting all of ES6 has been an ongoing issue, but we actively lobbied to prioritize to support features that were most important to us (and caused unrecoverable parse errors if they were not addressed), such as support for async/await and following symlinks through require/import statements.

To eliminate our gulp/file-watching contraption, we upstreamed a pull request to Atom to support Babel natively, just as it already had support for CoffeeScript. Because we didn't want to identify Babel files via a special extension (we wanted the files to be named .js rather than .es6 or something), and because (at least at the time), Babel was too slow to categorically transpile all files with a .js extension, we compromised on using the heuristic “if a file has a .js extension and its contents start with 'use babel', then transpile the file with Babel before evaluating it.” This has worked fairly well for us, but it also meant that everyone had to use the set of Babel options that we hardcoded in Atom. However, once Babel 6.0 comes out, we plan to work with Atom to let packages specify a .babelrc file so that every package can specify its own Babel options.

It has been a long road, but now that we are in a world where we can use Babel and Flow to develop Atom packages, we are very happy.

Node vs. Atom Packages

Atom has its own idea of a package that is very similar to an npm package, but differs in some critical ways:

• Only one version/instance of an Atom package can be loaded globally in the system.
  
• An Atom package cannot declare another Atom package as a dependency. (This somewhat follows from the first bullet point because if two Atom packages declared a dependency on different versions of the same Atom package, it is unclear what the right way to resolve it would be.)
  
• Because Atom packages cannot depend on each other, it is not possible to pull in another one synchronously via require().
  
• Atom packages have special folders such as styles, grammars, snippets, etc. The contents of these folders must adhere to a specific structure, and their corresponding resources are loaded when the Atom package is activated.
  

This architecture is particularly problematic when trying to build reusable UI components. Organizationally, it makes sense to put something like a combobox widget in its own package that can be reused by other packages. However, because it likely comes with its own stylesheet, it must be bundled as an Atom package rather than a Node package.

To work around these issues, we introduced the concept of three types of packages in Nuclide: Node/npm, Node/apm, and Atom/apm. We have a README that explains this in detail, but the key insight is that a Node/apm package is a package that is available via npm (and can be loaded via require()), but is structured like an Atom package. We achieved this by introducing a utility, nuclide-atom-npm, which loads the code from a Node package, but also installs the resources from the styles/ and grammars/ directories in the package, if present. It also adds a hidden property on global to ensure that the package is loaded only once globally.

Nuclide has many packages that correspond to UI widgets that we use in Atom: nuclide-ui-checkbox, nuclide-ui-dropdown, nuclide-ui-panel, etc. If you look at the implementation of any of these packages, you will see that they load their code using nuclide-atom-npm. Using this technique, we can reliably require the building blocks of our UI synchronously, which would not be the case if our UI components were published as Atom packages. This is especially important to us because we build our UI in Atom using React.

React

In July 2014, the Atom team had a big blog post that celebrated their move to React for the editor. Months later, they had a very quiet pull request that removed the use of React in Atom. Basically, the Atom team felt that to achieve the best possible performance for their editor, they needed to hand-tune that code rather than risk the overhead of an abstraction, such as React.

This was unfortunate for Nuclide because one of the design limitations of React is that it expects there to be only one instance of the library installed in the environment. When you own all of the code in a web page, it is not a big deal to commit to using only one version (if anything, it's desirable!), but when you are trying to create an extensible platform like Atom, it presents a problem. Either Atom has to choose the version of React that every third-party package must use, or every third-party package must include its own version of React.

The downside of Atom mandating the version of React is that, at some point, some package authors will want them to update it when a newer version of React comes out whereas other package authors will want them to hold back so their packages don't break. The downside of every third-party package (that wants to use React) including its own version is that multiple instances of React are not guaranteed to work together when used in the same environment. (It also increases the amount of code loaded globally at runtime.)

Further, React and Atom also conflict because they both want to control how events propagate through the system. To that end, when Atom was using React for its editor, it created a fork of React that did not interfere with Atom's event dispatch. This was based on React v0.11, which quickly became an outdated version of React.

Because we knew that Atom planned to remove their fork of React from their codebase before doing their 1.0 release, we needed to create our own fork that we could use. To that end, Jonas Gebhardt created the react-for-atom package, which is an Atom-compatible fork of React based off of React v0.13. As we did for the nuclide-atom-npm package, we added special logic that ensured that require('react-for-atom') could be called by various packages loaded by Atom, but it cached the result on the global environment so that subsequent calls to require() would return the cached result rather than load it again, making it act as a singleton.

Atom does not provide any sort of built-in UI library by default. The Atom UI itself does not use a standard library: most of the work is done via raw DOM operations. Although this gives package authors a lot of freedom, it also arrests some via a paradox of choice. On Nuclide, we have been using React very happily and successfully inside of Atom. The combination of Babel/JSX/React has facilitated producing performant and maintainable UI code.

Monorepo

Atom is developed as a collection of over 100 packages, each contained in its own repository. From the outside, this seems like an exhausting way to develop a large software project. Many, many commits to Atom are just minor version bumps to dependencies in package.json files across the various repos. To me, this is a lot of unnecessary noise.

Both Facebook and Google have extolled the benefits of a single, monolithic codebase. One of the key advantages over the multi-repo approach is that it makes it easier to develop and land atomic changes that span multiple parts of a project. For this and other reasons, we decided to develop the 100+ packages for Nuclide in a single repository.

Unfortunately, the Node/npm ecosystem makes developing multiple packages locally out of one repository a challenge. It has a heavy focus on semantic versioning and encourages dependencies to be fetched from https://www.npmjs.org/. Yes, it is true that you can specify local dependencies in a package.json file, but then you cannot publish your package.json file as-is.

Rather than embed local, relative paths in package.json files that get rewritten upon publishing to npm (which I think would be a reasonable approach), we created a script that walks our tree, symlinking local dependendencies under node_modules while using npm install to fetch the rest. By design, we specify the semantic version of a local dependency as 0.0.0 in a package.json file. These version numbers get rewritten when we publish packages to npm/apm.

We also reject the idea of semantic versioning. (Jeremy Ashkenas has a great note about the false promises of semantic versioning, calling it “romantic versioning.”) Most Node packages are published independently, leaving the package author to do the work of determining what the new version number should be based on the changes in the new release. Practically speaking, it is not possible to automate the decision of whether the new release merits a minor or major version bump, which basically means the process is imperfect. Even when the author gets the new version number right, it is likely that he/she sunk a lot of time into doing so.

By comparison, we never publish a new version of an individual Nuclide package: we publish new versions of all of our packages at once, and it is always a minor-minor version bump. (More accurately, we disallow cycles in our own packages' dependencies, so we publish them in topological order, all with the same version number where the code is derived from the same commit hash in our GitHub repo.) It is indeed the case that for many of our packages, there is nothing new from version N to N+1. That is, the only reason N+1 was published is because some other package in our repo needed a new version to be published. We prefer to have some superfluous package publications than to sink time into debating new version numbers and updating scores of package.json files.

Having all of the code in one place makes it easier to add processes that are applied across all packages, such as pre-transpiling Babel code before publishing it to npm or apm or running ESLint. Also, because our packages can be topologically sorted, many of the processes that we run over all of our packages can be parallelized, such as building or publishing. Although this seems to fly in the face of traditional Node development, it makes our workflow dramatically simpler.

async/await

I can't say enough good things about using async/await, which is something that we can do because we use Babel. Truth be told, as a side-project, I have been trying to put all of my thoughts around it into writing. I'm at 35 pages and counting, so we'll see where that goes.

An oversimplified explanation of the benefit is that it makes asynchronous code no harder to write than synchronous code. Many times, JavaScript developers know that designing code to be asynchronous will provide a better user experience, but give in to writing synchronous code because it's easier. With async/await, you no longer have to choose, and everyone wins as a result.

Flow

Similar to async/await, I can't say enough good things about static typing, particularly optional static typing. I gave a talk at the first mloc.js, “Keeping Your Code in Check,” demonstrating different approaches to static typing in JavaScript, and argued why it is particularly important for large codebases. Flow is a fantastic implementation of static typing in JavaScript.

Closing Thoughts

In creating Nuclide, we have written over 50,000 lines of JavaScript[1] and have created over 100 packages, 40 of which are Atom packages[2]. Our development process and modularized codebase has empowered us to move quickly. As you can tell from this essay (or from our scripts directory), building out this process has been a substantial, deliberate investment, but I think it has been well worth it. We get to use the best JavaScript technologies available to us today with minimal setup and an edit/refresh cycle of a few seconds.

Honestly, the only downside is that we seem to be ahead of Atom in terms of the number of packages it can support. We have an open issue against Atom about how to install a large number of packages more efficiently (note this is a problem when you install Nuclide via nuclide-installer, but not when you build from source). Fortunately, this should [mostly] be a simple matter of programming™: there is no fundamental design flaw in Atom that is getting in the way of a solution. We have had an outstanding working relationship with Atom thus far, finding solutions that improve things for not just Nuclide, but the entire Atom community. It has been a lot of fun to build on top of their platform. For me, working on Atom/Nuclide every day has been extremely satisfying due to the rapid speed of development and the tools we are able to provide to our fellow developers as a result of our work.

[1] To exclude tests and third-party code, I got this number by running:

git clone https://github.com/facebook/nuclide.git
cd nuclide
git ls-files pkg | grep -v VendorLib | grep -v hh_ide | grep -v spec | grep -e '\.js$' | xargs wc -l

[2] There is a script in Nuclide that we use to list our packages in topologically sorted order, so I got these numbers by running:

./scripts/dev/packages | wc -l
./scripts/dev/packages --package-type Atom | wc -l