Recently Next.js 8 was introduced. The release included a massive build-time memory usage reduction. This blog post will explore how we have helped optimize webpack for the community.

Next.js is zero-configuration and is built on top of tools like webpack and Babel. Its goal is to help you focus on what’s important: your application code.

Modern web applications consist of one or more pages. For example, a homepage, blog, dashboard, or product listing.

With Next.js, these pages become files in a special pages directory in the root of your project.

For example: the file pages/about.js maps to the URL /about.

One of the key design constraints of the framework is that it has to work well for both a single page and thousands of pages.

While implementing Serverless Next.js it quickly became apparent that running next build on a project with hundreds of pages caused high memory usage. Sometimes exceeding the approximately 1.4 GB memory heap limit that Node.js has.

We started profiling memory usage of the build process using the Chrome developer tools.

In the resulting profiles we discovered a point at which webpack would allocate a chunk of 548 MB memory all at once.

The amount of memory allocated directly correlated to the amount of pages, meaning more pages resulted in more memory usage.

By going through the memory profile's stacktrace we were able to track down the function that caused the memory allocation spike.

The allocation itself came from source.source() method being called which generates the resulting file and stores it into memory.

However by looking further up the function that calls the source() method you can see that compilation.assets was being iterated over using asyncLib.forEach. Meaning that the provided function would be called for every file in the compilation.assets array at the same time.

So this meant that if there are for example 100 pages, and each page has to be written to disk, above code would try to write all 100 at the same time, including generating all 100 files at the same time.

The solution for this issue is using a semaphore to limit the amount of concurrent writes. Generally we use async-sema for this, but in this case webpack already had a suitable method available on neo-async:

asyncLib.forEach(compilation.assets, (source, file, callback) => {
  // etc
})

Previous code that ran the function concurrently for all assets

asyncLib.forEachLimit(compilation.assets, 15, (source, file, callback) => {
  // etc
})

New code that runs the function concurrently for a maximum of 15 at a time

After implementing this concurrency limit and profiling the build memory usage again. We could see the memory allocation being split into smaller pieces of 34 MB.

This change showed very promising results, however in practice the build still ran out of memory, so we kept profiling and investigating the issue.

By further inspecting the memory profile we noticed how after the source.source() method was called the memory did not get cleaned up afterwards (garbage collected).

In webpack assets are generally instances of Source classes. These classes all implement a source() method that will generate the file source.

The profile showed that many assets were instances of CachedSource. The way CachedSource works is that when source() is called the result is cached in-memory until the asset is disposed.

Inspecting the webpack plugins Next.js uses showed that we had no plugins that called source() after webpack had written the file, meaning that caching the written value had no benefit.

After collaborating with Tobias Koppers he has implemented a new option called output.futureEmitAssets which allows opting-in to the new asset writing behavior.

With this new behavior the chunks being allocated were reduced to 182 KB over time.

Next.js 8 already has all these optimizations built-in. There is no need to change anything when using Next.js.

This optimization was introduced on webpack, meaning not just Next.js users, but all webpack users will benefit from these optimizations.

We will actively continue to improve Next.js and webpack memory usage and performance.