Node.js Design Patterns BlogFrom the authors of Node.js Design Patterns, useful bits to enrich your Node.js knowledge2022-03-11T12:30:00Zhttps://www.nodejsdesignpatterns.com/blogLuciano Mammino and Mario Casciaroauthors@nodejsdesignpatterns.com5 Ways to install Node.js2020-12-24T18:30:00Zhttps://www.nodejsdesignpatterns.com/blog/5-ways-to-install-node-js/<p>In this article, we will explore some of the most common ways to install Node.js in your development system. We will see how to install Node.js using the official installer for various platforms, how to use a Node.js version manager such as <code>n</code> or <code>nvm</code> and, finally, we will also see how to compile and install Node.js from source. Along the way, we will try to disclose one or two tips to get you even more productive with Node.js!</p>
<p>Let's get started!</p>
<h2 id="which-option-should-i-pick%3F" tabindex="-1" class="title is-2">Which option should I pick?</h2>
<p>There are many different ways to install Node.js and every one of them comes with its own perks and drawbacks. In this article, we will try to explore the most common ones and by the end of it, you should have a good understanding of which ones should be more suitable for you.</p>
<h3 id="tldr%3B" tabindex="-1" class="title is-3">TLDR;</h3>
<ul>
<li>Use <code>nvm</code> or <code>n</code> if you develop with Node.js frequently and you expect to be needing to switch Node.js version while moving from one project to another or to debug potential compatibility issues in your project or library.</li>
<li>Use the system package manager like <code>apt</code>, <code>brew</code> or <code>winget</code> if you tend to install all your software this way and if you don't expect to be needing to switch or upgrade Node.js version too often.</li>
<li>Install Node.js from source if you are an advanced user and if you want to contribute back to Node.js itself.</li>
<li>Use the official Node.js installer if you don't fall in any of the previous options...</li>
</ul>
<h3 id="what-other-people-seem-to-like" tabindex="-1" class="title is-3">What other people seem to like</h3>
<p>Before writing this article, I was actually curious to find out what are the options that most folks in my network prefer. For this reason, I run a <a href="https://twitter.com/loige/status/1340999569807712257">poll on Twitter</a>. In this poll I asked how you prefer to install Node.js and provided 4 options:</p>
<ul>
<li>Official Installer</li>
<li>Version manager (<code>nvm</code> or <code>n</code>)</li>
<li>Package Manager (<code>apt</code>, <code>brew</code>, etc.)</li>
<li>From source</li>
</ul>
<p>The results are quite interesting:</p>
<a href="https://twitter.com/loige/status/1340999569807712257" rel="nofollow noreferrer">
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 593px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-64.png 64w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-128.png 128w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-256.png 256w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-512.png 512w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-633.png 633w" sizes="(max-width: 593px) 100vw, 593px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-64.webp 64w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-128.webp 128w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-256.webp 256w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-512.webp 512w, https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-633.webp 633w" sizes="(max-width: 593px) 100vw, 593px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="Install Node.js Twitter poll results" src="https://www.nodejsdesignpatterns.com/img/poll-results-M4yStsOXME-64.png" width="64" height="45" />
</picture>
</span>
</a>
<p>It seems quite obvious that people in my network, mostly fellow software engineers, prefer to use version managers such as <code>nvm</code> or <code>n</code>.</p>
<p>The second place (actually very tight with the third one) is the official installer, followed by a system package manager and, last one, installing Node.js from source.</p>
<h3 id="lts-and-stable-releases" tabindex="-1" class="title is-3">LTS and stable releases</h3>
<p>Before moving on and exploring all the different installation options, it is definitely worth spending few words to learn about the types of release the Node.js project maintains.</p>
<p>Node.js offers 2 main release lines:</p>
<ul>
<li><strong>Stable</strong> (or <em>Current</em>): every new major Node.js release is considered "Current" for the first 6 months after the publish date. The idea is to give library authors the time to test their compatibility with the new release and do any necessary change. After the 6 months period, all the odd release numbers (9, 11, 13, 15, etc.) move to the state of <em>Unsupported</em>, while even releases (10, 12, 14, etc.) are promoted to <em>Long Term Support</em> (or "LTS").</li>
<li><strong>LTS</strong>: releases marked as "Long Term Support" get critical bug fixes for a total of 30 months since the initial publish date. This makes LTS releases particularly suitable for production deployments. The most recent LTS is also called <em>Active LTS</em>, while previous LTS versions (still under the 30 months support timeframe) are called <em>Maintenance LTS</em>.</li>
</ul>
<p>Finally, the release coming from the current <em>master</em> branch is considered <strong>Unstable</strong>. This is generally a release dedicated to people maintaining Node.js or developers who want to explore new experimental features that haven't been yet included in any of the major releases.</p>
<p>Node.js publishes an <a href="https://nodejs.org/en/about/releases/">official timeline of current and future releases</a>. At the time of writing (December 2020), this how the timeline looks like:</p>
<a href="https://nodejs.org/en/about/releases/" target="_blank" rel="noreferrer noopener">
<p style="text-align: center"><img loading="lazy" decoding="async" style="max-width: 100%; margin: 0px; vertical-align: middle;" alt="Node.js release timeline" src="https://www.nodejsdesignpatterns.com/img/nodejs-release-schedule_9b4bf060.svg" width="760" height="396" /></p>
</a>
<p>If you are still wondering which release should you use, going with the <em>Active LTS</em> is almost always the best choice, especially if you are building production applications.</p>
<h2 id="install-node.js-using-n" tabindex="-1" class="title is-2">Install Node.js using n</h2>
<p>Since installing Node.js using a version manager seems to be the favourite option (and it's also my personal favourite!) let's start with it.</p>
<p>My favourite Node.js version manager is <a href="https://github.com/tj/n"><code>n</code> by TJ Holowaychuk</a>. The reason why I like it is because it is quite simple to install and use and it is generally up to date with the latest releases of Node.js.
The main issue with it is that it does not support Windows, so if Windows is your operative system, this is not an option for you!</p>
<p>Let's see how to install <code>n</code>:</p>
<p>If you are on macOS and you have <code>brew</code> (Homebrew) installed, the simplest way to install <code>n</code> is to just do it with <code>brew</code>:</p>
<pre class="language-bash"><code class="language-bash">brew <span class="token function">install</span> n</code></pre>
<p>Alternatively, you can use the custom install script:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">curl</span> <span class="token parameter variable">-L</span> https://git.io/n-install <span class="token operator">|</span> <span class="token function">bash</span></code></pre>
<p><strong>Note:</strong> if you are concerned about running a script downloaded from the web (as you should because <a href="https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/"><code>curl | bash</code> might be dangerous</a>), you can always download the script first, READ IT, and then run it locally...</p>
<p>If all goes well, you should now be able to use the <code>n</code> executable from your shell.</p>
<p>These are some of the commands you can run:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># shows the version of `n` installed in your system</span>
n <span class="token parameter variable">--version</span>
<span class="token comment"># installs the latest LTS release of Node.js</span>
n lts
<span class="token comment"># lists all the versions of Node.js currently available</span>
n list
<span class="token comment"># install the given version of Node.js and switch to it</span>
n <span class="token operator"><</span>some_version<span class="token operator">></span></code></pre>
<p>Or you can simply run:</p>
<pre class="language-bash"><code class="language-bash">n</code></pre>
<p>For an interactive prompt that will show you all the available versions, highlight the ones you have already installed and let you pick the version you want to switch to.</p>
<p style="text-align: center"><img loading="lazy" decoding="async" style="max-width: 100%; margin: 0px; vertical-align: middle;" alt="n Node.js version manager in action" src="https://www.nodejsdesignpatterns.com/img/n_ac172e26.gif" width="640" height="428" /></p>
<p>In summary, this is where <code>n</code> shines or falls short:</p>
<ul>
<li>👎 No official support for Windows</li>
<li>👍 Very easy to install on macOS and unix systems</li>
<li>👍 Very easy to keep your Node.js install up to date and switch version on demand</li>
<li>👍 It keeps all the installed versions cached, so you can switch quickly between versions (no full re-install)</li>
<li>👍 Allows to keep the setup local to the user so you don't have to use admin permission to install global packages</li>
</ul>
<h2 id="install-node.js-using-nvm" tabindex="-1" class="title is-2">Install Node.js using nvm</h2>
<p>With more than 45 thousand stars on GitHub, <a href="https://github.com/nvm-sh/nvm"><code>nvm</code></a>, which stands for "Node.js Version Manager" (no surprises!), is probably the most famous Node.js version manager currently available.</p>
<p><code>nvm</code> works on any POSIX-compliant shell (<code>sh</code>, <code>dash</code>, <code>ksh</code>, <code>zsh</code>, <code>bash</code>, etc.) and it has been strongly tested against the following systems: unix, macOS, and windows WSL (if you are on Windows, you can also check out <a href="https://github.com/coreybutler/nvm-windows"><code>nvm-windows</code></a>).</p>
<p>The easiest way to install <code>nvm</code> on your system is to use the official installer script:</p>
<pre class="language-bash"><code class="language-bash"><span class="token assign-left variable">VERSION</span><span class="token operator">=</span>v0.37.2
<span class="token function">curl</span> -o- <span class="token string">"https://raw.githubusercontent.com/nvm-sh/nvm/<span class="token variable">${VERSION}</span>/install.sh"</span> <span class="token operator">|</span> <span class="token function">bash</span></code></pre>
<p><strong>Note</strong>: At the time of writing, version <code>v0.37.2</code> is the latest version available. Make sure to check out if there is any new version available if you are installing <code>nvm</code> following this tutorial.</p>
<p>Once <code>nvm</code> is installed in your system, here are some examples showing what you can do with it:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># installs the latest version of Node.js</span>
nvm <span class="token function">install</span> <span class="token function">node</span>
<span class="token comment"># installs the latest LTS version of Node.js</span>
nvm <span class="token function">install</span> <span class="token parameter variable">--lts</span>
<span class="token comment"># installs a specific version of Node.js</span>
nvm <span class="token function">install</span> <span class="token string">"10.10.0"</span>
<span class="token comment"># switch to a specific version of Node.js</span>
nvm use <span class="token string">"8.9.1"</span>
<span class="token comment"># runs a specific script with a given version of Node.js (no switch)</span>
nvm <span class="token builtin class-name">exec</span> <span class="token string">"4.2"</span> <span class="token function">node</span> somescript.js
<span class="token comment"># shows the full path where a given version of Node.js was installed</span>
nvm <span class="token function">which</span> <span class="token string">"4.2"</span>
<span class="token comment"># lists all the versions of Node.js available</span>
nvm <span class="token function">ls</span></code></pre>
<p>One great thing about <code>nvm</code> is that it allows to specify the Node.js version you want to use for a given project.</p>
<p>For instance, if you are working on a project that requires you to use Node.js <code>10.10</code> you can do the following (in the root folder of the project):</p>
<pre class="language-bash"><code class="language-bash"><span class="token builtin class-name">echo</span> <span class="token string">"10.10"</span> <span class="token operator">></span> .nvmrc</code></pre>
<p>Then every time you work on that project, you only need to run:</p>
<pre class="language-bash"><code class="language-bash">nvm use</code></pre>
<p>Which should print something like this:</p>
<pre><code>Found '/path/to/project/.nvmrc' with version <10.10>
Now using node v10.10.1 (npm v6.7.3)
</code></pre>
<p>At this point, you can be sure that you working using the correct Node.js version for your project.</p>
<p>If you don't want to do manually, you can enable <a href="https://github.com/nvm-sh/nvm#deeper-shell-integration">deeper shell integration</a> to make this happen automatically when you <code>cd</code> into a folder that has a <code>.nvmrc</code> file.</p>
<p><strong>PRO tip</strong>: You can also do that by using <a href="https://asdf-vm.com/"><code>asdf</code></a>, a <em>meta</em> version manager that offers a unified interface for various programming languages and version managers (including Node.js, of course).</p>
<p>Finally, here are some pros and cons of <code>nvm</code>:</p>
<ul>
<li>👍 Most popular version manager for Node.js with a large community of users.</li>
<li>👍 Very easy to install on POSIX systems.</li>
<li>👍 It allows for easy (and even automated) switch of Node.js version based on the project you are working on.</li>
<li>👍 It keeps all the installed versions cached, so you can switch quicly between versions (no full re-install)</li>
<li>👍 You can run once off commands on a given version of Node.js without having to switch the entire system to that version.</li>
<li>👎 You might have to take a bit of time to go through the documentation and make sure you install it and use it correctly.</li>
</ul>
<p><strong>Note</strong>: if you like to use version managers like <code>n</code> or <code>nvm</code>, you can also check out <a href="https://volta.sh/"><code>volta.sh</code></a>, another interesting alternative in this space, which defines itself as <em>"The Hassle-Free JavaScript Tool Manager"</em>.</p>
<h2 id="install-node.js-using-the-official-installer" tabindex="-1" class="title is-2">Install Node.js using the official installer</h2>
<p>The second most common way to install Node.js is through one of the official installers or the pre-compiled binaries.</p>
<p><a href="https://nodejs.org/en/download/">Official installers</a> are available on the official Node.js website for Windows and macOS and they cover the latest <em>Active LTS</em> release and the latest <em>Current</em> release.</p>
<p>The installer for Windows is an executable <em>.msi</em> installer, while the one for macOS is a <em>.pkg</em> one.</p>
<p>These installers behave and look like most of the installers you see while installing software on Windows or macOS. You will be presented with clickable UI which will allow you to customise and install Node.js into your system.</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 732px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-64.png 64w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-128.png 128w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-256.png 256w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-512.png 512w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-732.png 732w" sizes="(max-width: 732px) 100vw, 732px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-64.webp 64w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-128.webp 128w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-256.webp 256w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-512.webp 512w, https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-732.webp 732w" sizes="(max-width: 732px) 100vw, 732px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="Install Node.js using the official macOS installer" src="https://www.nodejsdesignpatterns.com/img/node-js-macos-installer-screenshot-ceJZLISO1x-64.png" width="64" height="48" />
</picture>
</span>
<p>This is probably the easiest way to install Node.js as you don't need to be a POSIX expert or do any kind of manual configuration. The installer will suggest sensible defaults to you and allow you to customise the main parameters (e.g. installation path).</p>
<p>If you are running a unix system, there is no official graphical installer available, but the <a href="https://nodejs.org/dist/">official Node.js download page</a> offers a set of pre-compiled binaries for most architectures (32-bit, 64-bit, ARMv7 and ARMv8) for Linux, Windows and macOS.</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 700px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-64.png 64w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-128.png 128w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-256.png 256w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-512.png 512w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-1077.png 1077w" sizes="(max-width: 700px) 100vw, 700px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-64.webp 64w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-128.webp 128w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-256.webp 256w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-512.webp 512w, https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-1077.webp 1077w" sizes="(max-width: 700px) 100vw, 700px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="Install Node.js using the official macOS installer" src="https://www.nodejsdesignpatterns.com/img/node-js-macos-precompiled-binary-XVV50zv_zZ-64.png" width="64" height="26" />
</picture>
</span>
<p>With the binary distribution, it is up to you to copy the necessary files in the right place. A version manager tool such as <code>nvm</code> and <code>n</code> makes things simple, because it takes care of downloading the correct binary release for the desired version (and for your system), then it places the files in the correct folder as expected by your operative system. If you choose to download the binaries manually, all the wiring is up to you.</p>
<p>While installing Node.js using the official installers is probably the simplest option, doing it using the binaries is a lot more complicated and definitely more complicated than using a version manager.</p>
<p>If you still want to go down this path, make sure to check out the <a href="https://github.com/nodejs/help/wiki/Installation">official tutorial for installing from Node.js pre-compiled binaries</a>.</p>
<p>It is definitely worth mentioning that the official installer is not the only option. <a href="https://nodesource.com/">NodeSource</a> maintains alternative installers for Debian, Red Hat, macOS and Windows. If you are interested in this approach checkout <a href="https://node.dev/node-binary">NodeSource Node.js Binary distributions page</a>.</p>
<p>To summarise, these are the main pros and cons of Node.js installers and binary distributions:</p>
<ul>
<li>👍 Installers are quite easy to use and they don't require specific POSIX experience.</li>
<li>👎 Hard to switch between version or upgrade. If you want to do that, you basically have to download the specific installer for the desired version and run through the full process again.</li>
<li>👎 Installer often will install Node.js as admin, which means that you can't install global packages unless you do that as admin.</li>
<li>👎 Binary packages require you to manually manage all the files and configuration.</li>
</ul>
<h2 id="install-node.js-using-a-package-manager" tabindex="-1" class="title is-2">Install Node.js using a package manager</h2>
<p>If you are the kind of person that loves to install and manage all the software in your device using system package managers such as <code>apt</code> (Debian / Ubuntu), <code>brew</code> (macOS), or <code>winget</code> (Windows), installing Node.js through a package manager is definitely an option.</p>
<p>A word of warning though, the various Node.js packages in every package repository are not officially maintained by the Node.js core team, so your mileage might vary quite a lot. This also means that you might not have fresh releases available straight away in your package manager of choice.</p>
<p>The Node.js core team has compiled an official documentation page on <a href="https://nodejs.org/en/download/package-manager/">how to install Node.js using the most common system package managers</a>.</p>
<p>Let's see here a summary for the most common options:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Homebrew (macOS)</span>
brew <span class="token function">install</span> <span class="token function">node</span>
<span class="token comment"># Arch Linux</span>
pacman <span class="token parameter variable">-S</span> nodejs <span class="token function">npm</span>
<span class="token comment"># CentOS, Fedora and Red Hat Enterprise Linux</span>
dnf module list nodejs
<span class="token comment"># Debian and Ubuntu based Linux distributions</span>
<span class="token function">apt-get</span> <span class="token function">install</span> <span class="token parameter variable">-y</span> nodejs
<span class="token comment"># FreeBSD</span>
pkg <span class="token function">install</span> <span class="token function">node</span>
<span class="token comment"># Gentoo</span>
emerge nodejs
<span class="token comment"># Winget (Windows)</span>
winget <span class="token function">install</span> <span class="token parameter variable">-e</span> <span class="token parameter variable">--id</span> OpenJS.Nodejs
<span class="token comment"># Chocolatey (Windows)</span>
cinst nodejs.install
<span class="token comment"># Scoop (Windows)</span>
scoop <span class="token function">install</span> nodejs</code></pre>
<p>In short, this is "the good" and "the bad" of following this approach:</p>
<ul>
<li>👍 Familiar approach if you install software often using your system package manager.</li>
<li>👎 Latest Node.js versions might not be immediately available in your package manager of choice. Some versions might not be available at all.</li>
<li>👎 In most cases, Node.js is installed as super user, which makes it harder to install global packages with <code>npm</code>.</li>
</ul>
<h2 id="install-node.js-from-source" tabindex="-1" class="title is-2">Install Node.js from source</h2>
<p>If you are brave enough to be willing to build and install Node.js from source, your first stop should be the <a href="https://github.com/nodejs/node/blob/main/BUILDING.md">official documentation on how to build Node.js from source</a>.</p>
<p>Here is a brief summary of all the steps involved:</p>
<ol>
<li>Install the necessary build dependencies (C++ compiler and build toolchains) for your target system.</li>
<li>Install Python (used by the build process).</li>
<li>Download the source code from the <a href="https://github.com/nodejs/node">official repository</a>.</li>
<li>Launch <code>./configure</code> and then <code>make</code>.</li>
<li>Test your compiled version with <code>make test</code>.</li>
<li>Install it with <code>make install</code>.</li>
</ol>
<p>If all went well, you should have the <code>node</code> binary available on your system and be able to run:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">node</span> <span class="token parameter variable">--version</span></code></pre>
<p>Finally, here is the usual summary of pros and cons:</p>
<ul>
<li>👍 You can install any version of Node.js, including master or even work in progress from a dev branch or a PR. You can even play around with custom changes and get to the point where you might decide to contribute back to Node.js.</li>
<li>👍 You have full control on how to compile and install Node.js and don't have to follow pre-defined structures.</li>
<li>👎 You might need to install a bunch of additional build requirements (compilers, build tools, etc.) before you can even start with the process.</li>
<li>👎 Definitely the most complicated and the slowest way to get Node.js in your machine.</li>
</ul>
<h2 id="node.js-with-docker" tabindex="-1" class="title is-2">Node.js with Docker</h2>
<p>If you just want to "play" a bit with a Node.js REPL, you don't need to install Node.js in your system. If you have <code>docker</code> installed in your system, running a Node.js REPL in a container is as easy as running:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> run <span class="token parameter variable">-it</span> <span class="token function">node</span></code></pre>
<p>Here's a super quick demo:</p>
<p style="text-align: center"><img loading="lazy" decoding="async" style="max-width: 100%; margin: 0px; vertical-align: middle;" alt="Running a Node.js REPL using Docker" src="https://www.nodejsdesignpatterns.com/img/node-repl-with-docker_edc66675.gif" width="750" height="482" /></p>
<p>If you want to run a shell in a container with Node.js and <code>npm</code> installed, then you can do the following:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> run <span class="token parameter variable">-it</span> <span class="token function">node</span> <span class="token function">bash</span></code></pre>
<p>This way you can install third-party modules using <code>npm</code>, create your own scripts and run them with <code>node</code>. When you close the session the container and all the generated files will be destroyed.</p>
<p>This is the perfect environment for quick and dirty experiments.</p>
<p>Note, that you can also use Docker as a complete environment for development and not just for quick tests. Docker is actually great for keeping different Node.js version and other dependencies isolated on a per-project basis. Exploring this setup goes beyond the scope of this article, but there is ton of reference on the web about how you might use Docker for Node.js development.</p>
<h2 id="node.js-online" tabindex="-1" class="title is-2">Node.js online</h2>
<p>But what if you don't have docker installed and still want to have an environment where you can write and run some Node.js code?</p>
<p>Well, there is no shortage of platforms online that will give you a Node.js environment and an IDE that you can use to write and run JavaScript online.</p>
<p>These environments often offer delightful additional features like collaborative edit and the possibility to host and share your applications.</p>
<p>Here's a non-exhaustive list of services that you might want to try if you just need a quick way to write and share some Node.js examples:</p>
<ul>
<li><a href="https://codesandbox.io/">CodeSandbox</a></li>
<li><a href="https://repl.it/">Repl.it</a></li>
<li><a href="https://glitch.com/">Glitch</a></li>
<li><a href="https://stackblitz.com/">Stackblitz</a></li>
</ul>
<p>Most of these services offer a quite generous free plan, so you only need to sign up to start coding!</p>
<h2 id="conclusion" tabindex="-1" class="title is-2">Conclusion</h2>
<p>This concludes our list of ways to install Node.js. At this point, I hope you feel comfortable enough picking one of the options suggested here and that along the way you learned a trick or two.</p>
<p>If you enjoyed this article please consider sharing it and don't hesitate to reach out to me <a href="https://twitter.com/loige">on Twitter</a>. I am quite curious to find out what is your favourite way to install Node.js and why!</p>
<p>Until next time!</p>
<h3 id="credits" tabindex="-1" class="title is-3">Credits</h3>
<p>This article was possible only thanks to the great support and feedback of some amazing engineers. Here are some of the names that helped me (and sorry if I am forgetting someone): <a href="https://twitter.com/_Don_Quijote_">@_Don_Quijote_</a>, <a href="https://twitter.com/giuseppemorelli">@GiuseppeMorelli</a>, <a href="https://twitter.com/oliverturner">@oliverturner</a>, <a href="https://twitter.com/aetheon">@aetheon</a>, <a href="https://twitter.com/dottorblaster">@dottorblaster</a>, <a href="https://twitter.com/bcomnes">@bcomnes</a> & <a href="https://twitter.com/wa7son">@wa7son</a>.</p>
Node.js race conditions2021-01-24T18:35:00Zhttps://www.nodejsdesignpatterns.com/blog/node-js-race-conditions/<p>A single-threaded event loop like the one used by JavaScript and Node.js, makes it somewhat harder to have race conditions, but, SPOILER ALERT: race conditions are still possible!</p>
<p>In this article, we will explore the topic of race conditions in Node.js. We will discuss some examples and present a few different solutions that can help us to make our code <em>race condition free</em>.</p>
<h2 id="what-is-a-race-condition%3F" tabindex="-1" class="title is-2">What is a race condition?</h2>
<p>First of all, let's try to clarify what a <em>race condition</em> actually is.</p>
<p>A race condition is a type of <em>programming error</em> that can occur when multiple processes or threads are accessing the same shared resource, for instance, a file on a file system or a record in a database, and at least one of them is trying to modify the resource.</p>
<p>Let's try to present an example. Imagine that while a thread is trying to rename a file, another thread is trying to delete the same file. In this case, the second thread will receive an error because, when it's trying to delete the file, the file has already been renamed. Or, the other way around, while one thread is trying to rename the file, the file was already deleted by the other thread and it's not available on the filesystem anymore.</p>
<p>In other cases, race conditions can be more subtle, because they wouldn't result in the program crashing, but they might just be the source of an incorrect or inconsistent behaviour. In these cases, since there is no explicit error and no stack trace, the issue is generally much harder to troubleshoot and fix.</p>
<p>A classic example is when 2 threads are trying to update the same data source and the new information is the result of a function applied to the current value.</p>
<p>Let's pretend we are building a Roman Empire simulation game in which we can manage some cash flow and we have a global balance in <a href="https://en.wiktionary.org/wiki/aureus"><em>aureus</em></a> (a currency used in the Roman Empire around 100 B.C.E.). Now, let's say that our initial balance is <code>0</code> <em>aurei</em> and that there are two independent game components (possibly running on separate threads) that are trying to increase the balance by <code>50</code> <em>aurei</em> each, we should expect that in the end, the balance is <code>100</code> <em>aurei</em>, right?</p>
<pre class="language-text"><code class="language-text">0 + 50 + 50 = 100 🤑</code></pre>
<p>If we implement this in a naive way, we might have the two components performing three distinct operations each:</p>
<ol>
<li>Read the current value for <code>balance</code></li>
<li>Add <code>50</code> <em>aurei</em> to it</li>
<li>Save the resulting value into <code>balance</code></li>
</ol>
<p>Since the two components are running in parallel, without any synchronisation mechanism, the following case could happen:</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 848px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-64.png 64w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-128.png 128w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-256.png 256w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-512.png 512w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-2002.png 2002w" sizes="(max-width: 848px) 100vw, 848px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-64.webp 64w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-128.webp 128w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-256.webp 256w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-512.webp 512w, https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-2002.webp 2002w" sizes="(max-width: 848px) 100vw, 848px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="A race condition example showing 2 processes trying to update a balance" src="https://www.nodejsdesignpatterns.com/img/aurei-race-condition-node-js-Fngk3b5IPO-64.png" width="64" height="30" />
</picture>
</span>
<p>In the picture above you can see that <strong>Component 2</strong> ends up having a <em>stale</em> view of the balance: the balance gets changed by <strong>Component 1</strong> after <strong>Component 2</strong> has read the balance. For this reason, when <strong>Component 2</strong> performs its own update, it is effectively overriding any change previously made by <strong>Component 1</strong>. This is why we have a race condition: the two components are effectively racing to complete their own tasks and they might end up stepping onto each other's toes! This doesn't make <em>Julius</em> happy I am afraid...</p>
<p>One way to solve this problem is to isolate the 2 concurrent operations into <em>transactions</em> and make sure that there is only one transaction running at a given time. This idea might look like this:</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 848px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-64.png 64w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-128.png 128w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-256.png 256w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-512.png 512w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-2022.png 2022w" sizes="(max-width: 848px) 100vw, 848px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-64.webp 64w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-128.webp 128w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-256.webp 256w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-512.webp 512w, https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-2022.webp 2022w" sizes="(max-width: 848px) 100vw, 848px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="Fixing a race condition using a transaction" src="https://www.nodejsdesignpatterns.com/img/aurei-fixed-race-condition-node-js-JsApfpFGe9-64.png" width="64" height="31" />
</picture>
</span>
<p>In the last picture, we are using transactions to make sure that all the steps of <strong>Component 1</strong> happen in order before all the steps of <strong>Component 2</strong>. This prevents any <em>stale read</em> and makes sure that every component always has an up to date view of the world before doing any change. You can stop holding your breath now, Julius!</p>
<p>In the rest of this article, we will zoom in more on race conditions in the context of Node.js and we will see some other approaches to deal with them.</p>
<h2 id="can-we-have-race-conditions-in-node.js%3F" tabindex="-1" class="title is-2">Can we have race conditions in Node.js?</h2>
<p>It is a common misconception to think that Node.js does not have race conditions because of its single-threaded nature. While it is true that in Node.js you would not have multiple threads competing for resources, you might still end up with tasks belonging to different logical transactions being executed in an order that might result in <em>stale reads</em> and generate a race condition.</p>
<p>In the example that we illustrated above, we intentionally represented the various tasks (<em>read</em>, <em>increase</em> and <em>save</em>) as discrete units. Note how the system is never executing more than one task at the same time. This is a simple but accurate representation of how the Node.js event loop processes tasks on a single thread. Nonetheless, you can see that there might be situations where multiple logical transactions (e.g. multiple deposits) are scheduled concurrently on the event loop and the discrete tasks might end up being intermingled, which results in a race condition.</p>
<p>So... <strong>Yes</strong>, we can have race conditions in Node.js!</p>
<h2 id="a-node.js-example-with-a-race-condition" tabindex="-1" class="title is-2">A Node.js example with a race condition</h2>
<p>Now, let's talk some code! Let's try to re-create the Roman Empire simulation game example that we discussed above.</p>
<p>In ancient Rome, Romans used to export olives and grapes. No wonder Italy is still famous worldwide for olive oil and wine! In our game, we want to be able to harvest olives and grapes and then sell them as a means to acquire more <em>aurei</em>.</p>
<p>We are going to have two functions that can increase the balance by <code>50</code> <em>aurei</em> which we are going to call <code>sellOlives()</code> and <code>sellGrapes()</code>. We will also assume that every time the balance is changed, it is persisted to a data storage of sort (e.g. a database). For the sake of this example, we won't be using a real data storage, but we will just simulate some random asynchronous delay before reading or modifying a global value. This will be enough to illustrate how we can end up with a race condition.</p>
<p>For starts, let's see what a buggy implementation might look like:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token comment">// Utility function to simulate some delay (e.g. reading from or writing to a database).</span>
<span class="token comment">// It will take from 0 to 50ms in a random fashion.</span>
<span class="token keyword">const</span> <span class="token function-variable function">randomDelay</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token keyword">new</span> <span class="token class-name">Promise</span><span class="token punctuation">(</span><span class="token parameter">resolve</span> <span class="token operator">=></span>
<span class="token function">setTimeout</span><span class="token punctuation">(</span>resolve<span class="token punctuation">,</span> Math<span class="token punctuation">.</span><span class="token function">random</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">100</span><span class="token punctuation">)</span>
<span class="token punctuation">)</span>
<span class="token comment">// Our global balance.</span>
<span class="token comment">// In a more complete implementation, this will live in the persistent data storage.</span>
<span class="token keyword">let</span> balance <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">loadBalance</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// simulates random delay to retrieve data from data storage</span>
<span class="token keyword">await</span> <span class="token function">randomDelay</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> balance
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">saveBalance</span> <span class="token punctuation">(</span><span class="token parameter">value</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// simulates random delay to write the data to the data storage</span>
<span class="token keyword">await</span> <span class="token function">randomDelay</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
balance <span class="token operator">=</span> value
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellGrapes</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellOlives</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">main</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> transaction1 <span class="token operator">=</span> <span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// NOTE: no `await`</span>
<span class="token keyword">const</span> transaction2 <span class="token operator">=</span> <span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// NOTE: no `await`</span>
<span class="token keyword">await</span> transaction1 <span class="token comment">// NOTE: awaiting here does not stop `transaction2` </span>
<span class="token comment">// from being scheduled before transaction 1 is completed</span>
<span class="token keyword">await</span> transaction2
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Final balance: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>If we execute this code we might end up with different results. In one case we might get the correct outcome:</p>
<pre class="language-text"><code class="language-text">sellOlives - balance loaded: 0
sellOlives - balance updated: 50
sellGrapes - balance loaded: 50
sellGrapes - balance updated: 100
Final balance: 100</code></pre>
<p>But in other cases we might end up in a bad state:</p>
<pre class="language-text"><code class="language-text">sellGrapes - balance loaded: 0
sellOlives - balance loaded: 0
sellGrapes - balance updated: 50
sellOlives - balance updated: 50
Final balance: 50</code></pre>
<p>Note how in this last case, <code>sellOlives</code> is essentially a stale read and therefore it will end up overriding the balance disregarding any work already done by <code>sellGrapes</code>. Yes, we do have a race condition, unfortunately!</p>
<p>Now, this example is simple ad it is not too hard to pinpoint exactly where the race condition has originated by just looking at the code.</p>
<p>Take a minute or two to read the code again. Check out the output from the 2 cases as well. Pay attention to the notes and the log messages and try to imagine how the Node.js runtime might execute this code in the 2 different scenarios.</p>
<p>Ok, now that you have done that, let's discuss together what happens.</p>
<p>In our <code>main</code> function, when we execute <code>sellGrapes()</code> and <code>sellOlives()</code>, since we are not awaiting the two operations independently, we are essentially scheduling both operations onto the event loop.</p>
<p>We only await the two transactions after they have been already scheduled, which means that they will work concurrently. After the two transactions have been scheduled, we wait for <code>transaction1</code> to complete and only then we wait for <code>transaction2</code> to complete. Note that <code>transaction2</code> might complete even before <code>transaction1</code>. In other words, awaiting for <code>transaction1</code> doesn't block <code>transaction2</code> in any way.</p>
<p>This approach is similar to writing the following code:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">await</span> Promise<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<p>Using <code>Promise.all()</code> is a more commonly used way to schedule different tasks to run concurrently.</p>
<p>Note that with <code>Promise.all()</code>, the resulting promise will reject as soon as any of the promises rejects. In our previous example, since we await the two promises independently, we will always catch errors in <code>transaction1</code> before <code>transaction2</code>.</p>
<p>But let's not digress too much into this. Now that we understand the problem, how do we fix the race condition?</p>
<p>Well, it turns out that in this simple case, we might make things right quite easily:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">main</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> <span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// <- schedule the first transaction and wait for completion</span>
<span class="token keyword">await</span> <span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// <- when it's completed, we start the second transaction </span>
<span class="token comment">// and wait for completion</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Final balance: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>This implementation will consistently produce the following output:</p>
<pre class="language-text"><code class="language-text">sellGrapes - balance loaded: 0
sellGrapes - balance updated: 50
sellOlives - balance loaded: 50
sellOlives - balance updated: 100
Final balance: 100</code></pre>
<p>As we can observe, <code>sellGrapes</code> is always started and completed <em>before</em> we start <code>sellOlives</code>. This makes the two logical transactions isolated and makes sure their tasks won't end up being mixed together in random order.</p>
<p>Problem solved... <em>vade in pacem</em> dear race condition!</p>
<h2 id="using-a-mutex-in-node.js" tabindex="-1" class="title is-2">Using a mutex in Node.js</h2>
<p>OK, the previous example was illustrative, but if we are building a real game, chances are things will end up being a lot more complicated. We will probably end up having many different actions that might cause a change of balance. Those actions might be the result of a particular sequence of events and it might become hard to track down the discrete logical transactions that we have to <em>serialize</em> in order to avoid race conditions.</p>
<p>Ideally, we don't want to think in terms of transactions, we just need to make sure that we never read the balance if there is another concurrent operation that is ready to change its value.</p>
<p>To be able to do this we need two things:</p>
<ol>
<li>Have a way to identify when <em>we are about to change</em> the balance</li>
<li>Let other events wait in line until the change is completed before reading the balance</li>
</ol>
<p>We could say that when <em>we are about to change</em> the balance we enter a <em>critical path</em> and that we don't want to intermingle events from different logical transactions in a critical path.</p>
<p>One way to achieve this is by using a <em>Mutex</em> (which stands for <a href="https://en.wikipedia.org/wiki/Mutual_exclusion"><strong>mut</strong>ual <strong>ex</strong>clusion</a>).</p>
<p>A mutex is a mechanism that allows synchronising access to a shared resource.</p>
<p>We can see a mutex as a shared object that allows us to mark when the code execution is entering and exiting from a critical path. In addition to that, a mutex can help us to queue other logical transactions that want to access the same critical path while one transaction is being processed.</p>
<p>Before talking code, be aware that using a mutex might have a performance impact in your application and that this solution won't work if you use a distributed or a multi-process setup. More details on this later.</p>
<h2 id="using-async-mutex" tabindex="-1" class="title is-2">Using <code>async-mutex</code></h2>
<p>A very useful library that we can useg is <a href="https://npm.im/async-mutex"><code>async-mutex</code></a>. This library provides a promise-based implementation of the mutex pattern.</p>
<p>You can install this library from <code>npm</code>:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">--save</span> async-mutex</code></pre>
<p>Now, here's an example of how we could use this library to mark the beginning and the end of a critical path:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> Mutex <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'async-mutex'</span>
<span class="token keyword">const</span> mutex <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Mutex</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// creates a shared mutex instance</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">doingSomethingCritical</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> release <span class="token operator">=</span> <span class="token keyword">await</span> mutex<span class="token punctuation">.</span><span class="token function">acquire</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// acquires access to the critical path</span>
<span class="token keyword">try</span> <span class="token punctuation">{</span>
<span class="token comment">// ... do stuff on the critical path</span>
<span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span>
<span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// completes the work on the critical path</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>In this example, we are using a global mutex instance to mark the beginning and the end of a critical path which happens inside our <code>doingSomethingCritical()</code> function.</p>
<p>When we call <code>mutex.acquire()</code>, this method will return a promise. If no other concurrent operation is currently on the same critical path, the promise resolves to a function that we call <code>release</code>. In this situation, we are essentially granted exclusive access to the critical path. If some concurrent operation is on the critical path already, the promise won't resolve until the concurrent operation already on the critical path has completed. This is how concurrent operations <em>wait in line</em> for our exclusive access to the critical path.</p>
<p>The <code>release</code> function must be invoked to mark the completion of the work on the critical path. It effectively <em>releases</em> the exclusive access to the critical path and makes it available to the next task in line. Note that we are using a <code>try</code>/<code>finally</code> block here to make sure that <code>release</code> is called even in case of an exception. It is very important to do so. In fact, failing to call <code>release</code>, will leave all the other events waiting in line forever!</p>
<p>Now let's try to use <code>async-mutex</code> to avoid race conditions in our game:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> Mutex <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'async-mutex'</span>
<span class="token keyword">const</span> <span class="token function-variable function">randomDelay</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">let</span> balance <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">const</span> mutex <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Mutex</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// global mutex instance</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">loadBalance</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">saveBalance</span> <span class="token punctuation">(</span><span class="token parameter">value</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellGrapes</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// this code will need exclusive access to the balance</span>
<span class="token comment">// so we consider this to be a critical path</span>
<span class="token keyword">const</span> release <span class="token operator">=</span> <span class="token keyword">await</span> mutex<span class="token punctuation">.</span><span class="token function">acquire</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// get access to the critical path (or wait in line)</span>
<span class="token keyword">try</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span>
<span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// completes work on the critical path</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellOlives</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// similar to `sellGrapes` this is a critical path because</span>
<span class="token comment">// it needs exclusive access to balance</span>
<span class="token keyword">const</span> release <span class="token operator">=</span> <span class="token keyword">await</span> mutex<span class="token punctuation">.</span><span class="token function">acquire</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">try</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span> <span class="token keyword">finally</span> <span class="token punctuation">{</span>
<span class="token function">release</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">main</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// Here we can call many events safely, the mutex will guarantee that the</span>
<span class="token comment">// competing events are executed in the right order!</span>
<span class="token keyword">await</span> Promise<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Final balance: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>The code above will consistently produce the following output:</p>
<pre class="language-text"><code class="language-text">sellGrapes - balance loaded: 0
sellGrapes - balance updated: 50
sellOlives - balance loaded: 50
sellOlives - balance updated: 100
sellGrapes - balance loaded: 100
sellGrapes - balance updated: 150
sellOlives - balance loaded: 150
sellOlives - balance updated: 200
sellGrapes - balance loaded: 200
sellGrapes - balance updated: 250
sellOlives - balance loaded: 250
sellOlives - balance updated: 300
Final balance: 300</code></pre>
<p>Some of the code has been truncated for simplicity. You can find all the examples in <a href="https://github.com/lmammino/node-js-race-conditions">this repository</a>.</p>
<p>From the example above, you can see how mutexes can provide a convenient way of thinking about exclusive access and how they can help to avoid race conditions. We are intentionally triggering multiple calls to <code>sellGrapes()</code> and <code>sellOlives()</code> concurrently, to make obvious that we don't have to think about potential race conditions at the <em>calling point</em>. This means that, as our game grows more complicated, we can keep invoking these functions without having to worry about generating new race conditions.</p>
<h2 id="let's-implement-a-mutex" tabindex="-1" class="title is-2">Let's implement a mutex</h2>
<p>But what if we are dealing with a race condition only in one place in our entire application? Is it worth to include and manage an external dependecy just because of that? Can we come up with a simpler alternative that does not require us to install a new dependency?</p>
<p>It turns out that we can easily do that! Let's see how we can implement our own mutex.</p>
<p>Note that the solution we are going to present here is effectively a variation of the <strong>sequential execution pattern</strong> using promises that is presented in <em>Chapter 5</em> of <a href="https://www.nodejsdesignpatterns.com/">Node.js Design Patterns</a>.</p>
<p>The idea is to inizialize our global mutex as an instance of a resolved promise:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">let</span> mutex <span class="token operator">=</span> Promise<span class="token punctuation">.</span><span class="token function">resolve</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>Then in our critical path we can do something like this:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">doingSomethingCritical</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
mutex <span class="token operator">=</span> mutex<span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token comment">// ... do stuff on the critical path</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token punctuation">.</span><span class="token function">catch</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token comment">// ... manage errors on the critical path</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> mutex
<span class="token punctuation">}</span></code></pre>
<p>The idea is that every time we are invoking the function <code>doingSomethingCritical()</code> we are effectively "queueing" the execution of the code on the critical path using <code>mutex.then()</code>. If this is the first call, our initial instance of the <code>mutex</code> promise is a resolved promise, so the code on the critical path will be executed straight away on the next cycle of the event loop.</p>
<p>Calling <code>.then()</code> on a promise returns a new promise instance that is used to replace the original <code>mutex</code> instance and it's also returned by the <code>doingSomethingCritical()</code> function.</p>
<p>This allows us to have concurrent calls to <code>doingSomethingCritical()</code> being queued to be executed sequentially.</p>
<p>Note that we also specify a <code>mutex.catch()</code>. This allows us to catch and react to specific errors, but it also allows us not to break the chain of sequential execution in case an operation fails.</p>
<p>Ok, now that we have explored this idea, let's apply it to our example.</p>
<p>This is how our code is going to look like:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> <span class="token function-variable function">randomDelay</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">let</span> balance <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">let</span> mutex <span class="token operator">=</span> Promise<span class="token punctuation">.</span><span class="token function">resolve</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment">// global mutex instance</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">loadBalance</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">saveBalance</span> <span class="token punctuation">(</span><span class="token parameter">value</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><span class="token comment">/* ... */</span><span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellGrapes</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
mutex <span class="token operator">=</span> mutex<span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellGrapes - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">catch</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> mutex
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">sellOlives</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
mutex <span class="token operator">=</span> mutex<span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance loaded: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token keyword">const</span> newBalance <span class="token operator">=</span> balance <span class="token operator">+</span> <span class="token number">50</span>
<span class="token keyword">await</span> <span class="token function">saveBalance</span><span class="token punctuation">(</span>newBalance<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">sellOlives - balance updated: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>newBalance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">catch</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> mutex
<span class="token punctuation">}</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token function">main</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> Promise<span class="token punctuation">.</span><span class="token function">all</span><span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellGrapes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
<span class="token function">sellOlives</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> balance <span class="token operator">=</span> <span class="token keyword">await</span> <span class="token function">loadBalance</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Final balance: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>balance<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<p>If you try to run this code, you will see that it consistently prints the same output as per our previous implementation using <code>async-mutex</code>!</p>
<p>So, here we have it, a simple mutex implementation in just few lines of code leveraging promise chainability!</p>
<h2 id="mutex-with-multiple-processes" tabindex="-1" class="title is-2">Mutex with multiple processes</h2>
<p>It is important to mention that the solutions presented in this article only work in a Node.js application running on a single process.</p>
<p>If you are running your application on multiple processes (for instance, by using the <a href="https://nodejs.org/api/cluster.html"><code>cluster</code> module</a>, <a href="https://nodejs.org/api/worker_threads.html">worker threads</a> or a multi-process runner like <a href="https://pm2.keymetrics.io/"><code>pm2</code></a>) using a mutex within our code is not going to solve race conditions across processes. This is also the case if you are running your application on multiple servers.</p>
<p>In these cases you have to rely on more complicated solutions like <a href="https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html">distributed locks</a> or, if you are using a central database, you can rely on solutions provided by your own database systems. We will discuss a simple example in the next section.</p>
<h2 id="mutex-performance" tabindex="-1" class="title is-2">Mutex performance</h2>
<p>We already mentioned that using a mutex might have a relevant performance impact in your application.</p>
<p>To try to visualize why a mutex has a performance impact in your application let's try to think about the case when an operation is trying to acquire a lock on a mutex but the mutex is already locked. In this case, our operation is simply waiting without doing nothing, while for instance it could be doing some IO operation like connecting to the database or sending a query. It will probably take the event loop several spins before the lock is released and the operation that is waiting in line can acquire the lock. This get worse with a high number of operations waiting in line.</p>
<p>With a mutex we are effectively serializing tasks, making sure that executed in sequence and non-concurrently. If you abuse this pattern, you might end up in a situation where you could effectively eliminate all concurrency from your application.</p>
<p>Measuring how a mutex might impact your specific application is not something that can be done holistically and we recommend you to run your own benchmarks to find out what is the effect of introducing one or more mutex instances in your application.</p>
<p>Our general recommendation is to use a mutex only when you are sure you have to protect your code from a race condition and to try to make the critical path as short as possible.</p>
<p>Be aware that a mutex is not the only solution to race conditions. For instance, in our example, if we were to use a real relational database as a data storage, we could have avoided any race condition (at the application level) by letting the database itself do the increment using a SQL query:</p>
<pre class="language-sql"><code class="language-sql"><span class="token keyword">UPDATE</span> game <span class="token keyword">SET</span> aurei <span class="token operator">=</span> aurei <span class="token operator">+</span> <span class="token number">50</span><span class="token punctuation">;</span></code></pre>
<p>With this approach, we are trusting the database to do the right thing and we are not slowing down our application.</p>
<p>And there are other alternative approches. Just to name one, <a href="https://en.wikipedia.org/wiki/Optimistic_concurrency_control">optimistic locks</a> might provide a great alternative if race conditions are possible but they actually happen only in rare occasions.</p>
<h2 id="conclusion" tabindex="-1" class="title is-2">Conclusion</h2>
<p>In this article, we have explored race conditions and learned why they can be harmful. We showed how race conditions can happen in Node.js and several techniques to address them including the adoptopm of a mutex.</p>
<p>This is an interesting topic which often gets explored in the context of multi-threaded languages. The theory isn't much different but there are some important differences when dealing with concurrent, single-threaded languages like Node.js.</p>
<p>If you are curious to understand better the difference between <strong>Parallelism</strong> and <strong>Concurrency</strong> I strongly recommend you to read this great essay titled <a href="http://yosefk.com/blog/parallelism-and-concurrency-need-different-tools.html">parallelism and concurrency need different tools</a>. You can also watch this wonderful talk by <a href="https://twitter.com/steveklabnik">Steve Klabnik</a> called <a href="https://www.youtube.com/watch?v=lJ3NC-R3gSI">Rust's Journey to Async/Await</a> (yes, it's not only about Rust, trust me).</p>
<p>I really hope you enjoyed this article. Make sure to <a href="https://twitter.com/loige">reach out to me on Twitter</a> and let me know what you think!</p>
<p>Bye 😋</p>
<h2 id="credits" tabindex="-1" class="title is-2">Credits</h2>
<p>Thanks to <a href="https://github.com/Jack-Barry">Jack Barry</a> for the inspiration for this post on the <a href="https://github.com/PacktPublishing/Node.js-Design-Patterns-Third-Edition/discussions/25">Node.js Design Patterns discussion board</a>. Thanks to <a href="https://twitter.com/quasi_modal">Peter Caulfield</a>, <a href="https://twitter.com/StefanoAbalsamo">Stefano Abalsamo</a>, <a href="https://twitter.com/gbinside">Roberto Gambuzzi</a> and <a href="https://twitter.com/mariocasciaro">Mario Casciaro</a> for kindly reviewing this post.</p>
Node.js development with Docker and Docker Compose2021-04-07T18:00:00Zhttps://www.nodejsdesignpatterns.com/blog/node-js-development-with-docker-and-docker-compose/<p>In this article, we are going to learn how to use Docker and Docker Compose for Node.js development. We will discuss the main benefits of this approach and explore some interesting examples. While doing that we will also learn what Docker is and why you should consider it, especially if you are developing multiple projects or if you are working in a team.</p>
<h2 id="what-is-a-docker-container-in-simple-terms%3F" tabindex="-1" class="title is-2">What is a Docker container in simple terms?</h2>
<p>To explain what a Docker container is, let's just start by imagining a virtual machine or a virtual server provisioned to run a specific program.</p>
<p>Virtual machines are great because they allow you to run some useful software in an isolated way and they are easy to distribute. You can create an image and run it in different environments. You can even run it in your own desktop machine or share it with your own colleagues. It's a consistent medium to develop and deploy software.</p>
<p>In many ways, Docker containers, are very similar to virtual machines. Docker containers are another way to package (or "containerise") software and run it across different environments.</p>
<p>But there's a fundamental difference with virtual machines. With Docker containers, you don't have the overhead of having to include an entire operating system as part of your image, but only the basic dependencies needed to run some programs.</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 800px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-64.png 64w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-128.png 128w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-256.png 256w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-512.png 512w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-2900.png 2900w" sizes="(max-width: 800px) 100vw, 800px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-64.webp 64w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-128.webp 128w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-256.webp 256w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-512.webp 512w, https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-2900.webp 2900w" sizes="(max-width: 800px) 100vw, 800px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="Docker VS virtual machines" src="https://www.nodejsdesignpatterns.com/img/docker-vs-virtual-machines-m-F-bSuk1W-64.png" width="64" height="23" />
</picture>
</span>
<p>From the image above you can see that the <strong>Container Engine</strong> runs on top of the operative system and that you don't have to include a guest operative system into your container images. For this reason, containers are much more lightweight than traditional virtual machines, while still providing the benefits of isolation and portability.</p>
<p>But how can we create a new Docker image?</p>
<p>This is generally done by using a special configuration file called <code>Dockerfile</code>. Let's not indulge with more theory and let's see an example instead!</p>
<p>Before getting started let's make sure that you have Docker installed in your machine. You can get docker for most platform using the installers available on the <a href="https://docs.docker.com/get-docker/">official Docker website</a>.</p>
<p>Ok, if you are ready, let's create our first <code>Dockerfile</code>:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">touch</span> Dockerfile</code></pre>
<p>What do we want to put into our Dockerfile? Let's keep things simple for now and let's say that we want to create a container that just run <code>npm</code>. To achieve this, this is what we can write into our <code>Dockerfile</code>:</p>
<pre class="language-dockerfile"><code class="language-dockerfile"><span class="token instruction"><span class="token keyword">FROM</span> debian:buster</span>
<span class="token instruction"><span class="token keyword">RUN</span> apt-get update && apt-get install nodejs npm -y</span>
<span class="token instruction"><span class="token keyword">CMD</span> [<span class="token string">'npm'</span>]</span></code></pre>
<p>Ok, let's stop for a second... there's a lot to unpack here!</p>
<p>The first thing that we can see is that a <code>Dockerfile</code> is a file with a specific syntax. It is somewhat similar to a Bash script, but there's also something different about it...</p>
<p>The order of lines is important and every line starts with a keyword that defines the type of instruction: in our example, <code>FROM</code>, <code>RUN</code> and <code>CMD</code>.</p>
<ul>
<li><code>FROM</code> is generally the first line of any <code>Dockerfile</code> and it is used to define which <em>base image</em> do we want to extend. A base image provides a sort of a starting point that can make our life easier. For instance, here by specifying <code>debian:buster</code>, we are inheriting all the binaries and libraries that can be found in the <em>Buster</em> version of <em>Debian</em>. This includes the package manager <code>apt</code> that we can use to install additional software.</li>
<li><code>RUN</code> is used to run some scripting. This is generally done to install additional software, create configuration files, generate keys, download remote resources, etc. You can see <code>RUN</code> instructions as provisioning steps, in other words, instructions you need to run to configure your image. It is quite common to have multiple <code>RUN</code> instructions in a <code>Dockerfile</code>.</li>
<li><code>CMD</code> generally appears only once per <code>Dockerfile</code> and it indicates which command should be executed when the container runs. In our case, we are using it to specify that we want <code>npm</code> to be executed.</li>
</ul>
<p>There are other commands like <code>ENV</code> (define environment variables) and <code>COPY</code> (copy a file from your system into the image), but providing a comprehensive guide on how to write a <code>Dockerfile</code> goes beyond the scope of this article. If you want to find out more, check out the <a href="https://docs.docker.com/engine/reference/builder/">official <code>Dockerfile</code> reference</a>.</p>
<p>Ok, now that we have defined our <code>Dockerfile</code>, how do we build a container image?</p>
<p>That's easy, we just need to execute <code>docker build .</code> (where <code>.</code> means "look for a <code>Dockerfile</code> in the current folder). We can also give a name to our image by using tags, so the final command we want to execute is actually going to look like this:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> build <span class="token parameter variable">-t</span> demo-docker/npmdemo <span class="token builtin class-name">.</span></code></pre>
<p>With this command we are "tagging" our image with the name <code>demo-docker/npmdemo</code>. We can use this name later to run our container.</p>
<p>So ok, let's run the container then!</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> run demo-docker/npmdemo</code></pre>
<p>If everything went as expected we should see the following output:</p>
<pre class="language-text"><code class="language-text">Usage: npm <command />
where <command /> is one of:
access, adduser, audit, bin, bugs, c, cache, ci, cit,
clean-install, clean-install-test, completion, config,
create, ddp, dedupe, deprecate, dist-tag, docs, doctor,
edit, explore, fund, get, help, help-search, hook, i, init,
install, install-ci-test, install-test, it, link, list, ln,
login, logout, ls, org, outdated, owner, pack, ping, prefix,
profile, prune, publish, rb, rebuild, repo, restart, root,
run, run-script, s, se, search, set, shrinkwrap, star,
stars, start, stop, t, team, test, token, tst, un,
uninstall, unpublish, unstar, up, update, v, version, view,
whoami
npm <command /> -h quick help on <command />
npm -l display full usage info
npm help <term> search for help on <term>
npm help npm involved overview
...</term></term></code></pre>
<p>After showing this output, the container is stopped (simply because <code>npm</code> exits after showing the help message) and we are back to our prompt.</p>
<p>Remember that we specified in our <code>CMD</code> instruction that we just wanted to run <code>npm</code> at container start? This is why we are seeing the <code>npm</code> help message when we start the container. What if we want to run the container again but this time we want to run a different command? Can we override the default command? We can easily do that! For instance, we can run the following:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> run demo-docker/npmdemo <span class="token function">npm</span> <span class="token parameter variable">--version</span></code></pre>
<p>As you might expect, this is going to print something like this:</p>
<pre class="language-text"><code class="language-text">6.14.11</code></pre>
<p>I know this is not super useful just yet! But hold on for few other minutes so we can discuss why you might want to use Docker in your team. After that, we will actually go building a Docker image for a Node.js HTTP server! I promise that will be interesting :)</p>
<h2 id="why-docker-and-docker-compose-can-help-you-and-your-team" tabindex="-1" class="title is-2">Why Docker and Docker Compose can help you and your team</h2>
<p>Any active project will be constantly updated with new features, bug fixes or security patches. This is true for our project as well for the dependencies of our project. Of course, it is desirable to keep the dependencies of our project as much up to date as possible. But, inevitably, different projects might end up using different versions of the same dependency.</p>
<p>Keep in mind that, when we talk about dependencies, we don't mean only libraries but also other systems the project relies on, like a database. Let's take for example MySQL. In 2021 it is common to see project using <strong>MySQL 5.6, MySQL 5.7</strong> or <strong>MySQL 8</strong>.</p>
<p>Now, if we are working on multiple projects that require different versions of MySQL, how can we manage different versions of MySQL in our development machines? Is it always possible to switch from one version to another without messing up our configuration or corrupting the data?</p>
<p>With Docker you can avoid to spend hours to find a solution to this problem and run your entire stack on containers. Different projects can run locally by spinning up a different MySQL container and since every container is isolated, you won't have any issue when switching from one project to another: you won't accidentally corrupt MySQL configuration or its data.</p>
<p>So, the idea is to use Docker to manage every single "process" that is needed to run your application. Every process will run in its own container. For instance, in a regular web application, you might have a container for the web server process and a container for the database.</p>
<p>To manage and integrate many containers together, Docker offers a utility called Docker Compose. Docker Compose looks for a file called <code>docker-compose.yml</code> where you can define all the necessary configuration and containers you want to run for your project.</p>
<p>If you work in a team, you don't have to agree in advance on every single configuration detail with your teammates or make sure that everyone has exactly the same setup. In fact, with Docker compose, you can just share the <code>docker-compose.yml</code> configuration as part of your project and Docker will take care to spin up the same containers in every machine.</p>
<p>At any point in time, if someone on the team decides to update a dependency or introduce a new dependency, they will do so by updating the <code>docker-compose.yml</code> and everyone else will easily be able to get the latest changes.</p>
<p>Even if you have a new joiner in the team, they don't have to understand and install all the necessary dependencies one by one, they can simply install Docker rely on the latest version of <code>docker-compose.yml</code> to run the project on their development machine.</p>
<p>This is one of the main advantages that you can get by using Docker and Docker Compose for development.</p>
<p>Let's now see an example of a <code>docker-compose.yml</code>:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">'3'</span>
<span class="token comment">################################################################################</span>
<span class="token comment"># SERVICES</span>
<span class="token comment">################################################################################</span>
<span class="token key atrule">services</span><span class="token punctuation">:</span>
<span class="token comment"># ------------------------------------------------------------</span>
<span class="token comment"># MySQL Database</span>
<span class="token comment"># ------------------------------------------------------------</span>
<span class="token key atrule">mysqlserver</span><span class="token punctuation">:</span>
<span class="token key atrule">container_name</span><span class="token punctuation">:</span> myproject_mysql
<span class="token key atrule">image</span><span class="token punctuation">:</span> mysql<span class="token punctuation">:</span><span class="token number">5.7</span>
<span class="token key atrule">environment</span><span class="token punctuation">:</span>
<span class="token key atrule">MYSQL_ROOT_PASSWORD</span><span class="token punctuation">:</span> <span class="token string">"docker"</span>
<span class="token key atrule">MYSQL_USER</span><span class="token punctuation">:</span> <span class="token string">"local"</span>
<span class="token key atrule">MYSQL_PASSWORD</span><span class="token punctuation">:</span> <span class="token string">"local"</span>
<span class="token key atrule">volumes</span><span class="token punctuation">:</span>
<span class="token comment"># ---- Format: ----</span>
<span class="token comment"># HOST-DIRECTORY : DOCKER-DIRECTORY</span>
<span class="token punctuation">-</span> $<span class="token punctuation">{</span>MYSQL_BACKUP_FOLDER<span class="token punctuation">}</span><span class="token punctuation">:</span>/backup/
<span class="token key atrule">networks</span><span class="token punctuation">:</span>
<span class="token key atrule">app_net</span><span class="token punctuation">:</span>
<span class="token key atrule">ipv4_address</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>IP_MYSQL_SERVER<span class="token punctuation">}</span>
<span class="token comment">################################################################################</span>
<span class="token comment"># NETWORK</span>
<span class="token comment">################################################################################</span>
<span class="token key atrule">networks</span><span class="token punctuation">:</span>
<span class="token key atrule">app_net</span><span class="token punctuation">:</span>
<span class="token key atrule">driver</span><span class="token punctuation">:</span> bridge
<span class="token key atrule">ipam</span><span class="token punctuation">:</span>
<span class="token key atrule">driver</span><span class="token punctuation">:</span> default
<span class="token key atrule">config</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">subnet</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>IP_SUBNET<span class="token punctuation">}</span></code></pre>
<p>What is inside this this file?</p>
<ul>
<li><code>services</code> : this is the list of the containers (and their settings) that we want to run in our project</li>
<li><code>mysqlserver</code> : the name of the service. This name is used when you use <code>docker-compose</code> command and you want to refer to this specific service (container).</li>
<li><code>container_name</code> : overrides the default alias of the container. It's a good practice to use something like <code><name of the project>_<name of the service></code> to avoid conflict with other projects.</li>
<li><code>image</code> : the name of the Docker image that we want to use (as defined in the Docker registry).</li>
<li><code>environment</code> : environment variables used inside the container.</li>
<li><code>volumes</code> : shared folder between your computer and the container. You can crate a mapping of your destination folder even if doesn't exist. In our example we map our backup folder with <code>/backup</code> folder inside the container. Very useful if we want to crate a standard script for restoring the data into the database (e.g. standard seed data shared across devs).</li>
<li><code>networks</code> : add specific ipv4 IP to our container. Setting up a specific IP for your container (service) is very useful when you have to work with different projects or you want to create local DNS aliases for your project.</li>
<li><code>networks → app_net → driver: bridge</code> this is a default configuration to allow the container to connect to the internet through the host system. The name <code>app_net</code> is a the name we want to give to the Docker network, so you can name it wathever makes sense to you. Refer to the <a href="https://docs.docker.com/compose/networking/#specify-custom-networks">official documentation on custom networks</a> for more information.</li>
</ul>
<p>Now let's see our <code>.env</code> file that we are going to use to store the enviornment variables needed for this project:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment">################################################################################</span>
<span class="token comment">### MySQL Settings</span>
<span class="token comment">################################################################################</span>
<span class="token assign-left variable">MYSQL_BACKUP_FOLDER</span><span class="token operator">=</span>/home/user/backup_db/project_x/
<span class="token comment">################################################################################</span>
<span class="token comment">### IP Settings</span>
<span class="token comment">################################################################################</span>
<span class="token assign-left variable">IP_SUBNET</span><span class="token operator">=</span><span class="token number">172.16</span>.250.0/24
<span class="token assign-left variable">IP_LOCAL_COMPUTER</span><span class="token operator">=</span><span class="token number">172.16</span>.250.1
<span class="token assign-left variable">IP_MYSQL_SERVER</span><span class="token operator">=</span><span class="token number">172.16</span>.250.11</code></pre>
<p>It is not always necessary to use an <code>.env</code> file, especially in simple examples like this one. It is a good practice to use one, though. The main advantage is when you have to share the <code>docker-compose.yml</code> and the <code>.env</code> with the team.</p>
<p>Generally the <code>.env</code> file is not committed to the repository. The idea is to use it to provide "user specific" configuration, so every member of the team can choose your own range of IPs or reference their own work folders.</p>
<p>Let's see an example. If we add in <code>docker-compose</code> a service for Node.js we can add some "link" to other services.
(official docker documentation: <a href="https://docs.docker.com/compose/compose-file/compose-file-v3/#extra_hosts">https://docs.docker.com/compose/compose-file/compose-file-v3/#extra_hosts</a>)</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token comment"># ...</span>
<span class="token comment"># ------------------------------------------------------------</span>
<span class="token comment"># Node.js Server</span>
<span class="token comment"># ------------------------------------------------------------</span>
<span class="token key atrule">nodejs</span><span class="token punctuation">:</span>
<span class="token comment"># ...</span>
<span class="token key atrule">extra_hosts</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> mysql<span class="token punctuation">-</span>server<span class="token punctuation">-</span>service<span class="token punctuation">:</span>$<span class="token punctuation">{</span>IP_MYSQL_SERVER<span class="token punctuation">}</span>
<span class="token comment"># ...</span></code></pre>
<p>In <code>extra_hosts</code> we can add all entries that are stored in our <code>/etc/hosts</code> file so we can create "custom domain" via local configuration without have a real domain.</p>
<p>In this specific case in our Node.js app we can refer to mysql server not via IP but via "mysql-server-service" name.</p>
<p>Let's go back to our docker-compose file. At this point we can run <code>docker-compose</code> to start all the docker services specified in our project:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker-compose</span> up <span class="token parameter variable">-d</span></code></pre>
<p>Note that the <code>-d</code> option will start all the services in background.</p>
<p>After a few seconds, all our services should be running. So let's see, for example, how we can access the MySQL shell and run a query:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker-compose</span> <span class="token builtin class-name">exec</span> mysqlserver /bin/bash
root@2f88599b3b1f:/<span class="token comment"># mysql -u root -pdocker (...enter root password)</span>
mysql<span class="token operator">></span> SHOW DATABASES<span class="token punctuation">;</span></code></pre>
<p>First of all: we need to "open" a bash shell inside the container so we can run any command. We can do that with: <code>docker-compose exec mysqlserver /bin/bash</code>.</p>
<p>These are the arguments of this <code>docker-compose</code> command:</p>
<ul>
<li><code>exec</code> : Docker searches for a running container (with the name <code>mysqlserver</code>) to run a given command. If you don't run <code>docker-compose up -d</code> before, this command will fail.</li>
<li><code>/bin/bash</code> : is the name of the command we want to run inside the container.</li>
</ul>
<p>Once we are "inside" the container we can do anything we want. In this case we want open a mysql terminal, so we run <code>mysql -u root -pdocker</code>.</p>
<p>Note that the root password is defined by environment variable <code>MYSQL_ROOT_PASSWORD</code>.</p>
<h2 id="create-and-configure-a-node.js-project-with-docker" tabindex="-1" class="title is-2">Create and configure a Node.js project with Docker</h2>
<p>Based on the complexity of the project we might decide to use Docker directly one or more containers or to rely on Docker Compose to orchestrate multiple containers.</p>
<p>Let's take a look at this simple "Hello World" web server (file <code>app.js</code>).</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> http <span class="token operator">=</span> <span class="token function">require</span><span class="token punctuation">(</span><span class="token string">'http'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> hostname <span class="token operator">=</span> <span class="token string">'0.0.0.0'</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> port <span class="token operator">=</span> <span class="token number">4040</span><span class="token punctuation">;</span>
<span class="token keyword">const</span> server <span class="token operator">=</span> http<span class="token punctuation">.</span><span class="token function">createServer</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">req<span class="token punctuation">,</span> res</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
res<span class="token punctuation">.</span>statusCode <span class="token operator">=</span> <span class="token number">200</span><span class="token punctuation">;</span>
res<span class="token punctuation">.</span><span class="token function">setHeader</span><span class="token punctuation">(</span><span class="token string">'Content-Type'</span><span class="token punctuation">,</span> <span class="token string">'text/plain'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
res<span class="token punctuation">.</span><span class="token function">end</span><span class="token punctuation">(</span><span class="token string">'Hello World'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
server<span class="token punctuation">.</span><span class="token function">listen</span><span class="token punctuation">(</span>port<span class="token punctuation">,</span> hostname<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">Server running at http://</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>hostname<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">:</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>port<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">/</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>Nothing particularly complicated here, we are simply creating a web server that responds to every request with an "Hello World!".</p>
<p>But pay special attention to <code>hostname</code>! With docker you can have connectivity problems if you use <code>"localhost"</code> or <code>"127.0.0.1"</code>, because it refers to the docker container and not to your local machine, therefore the container will only listen for connections from inside the container itself. It's better to use use <code>"0.0.0.0"</code> or directly the ip associated to the container so that we can issue requests from our local machine or from other containers.</p>
<h3 id="work-with-docker-directly-(no-configuration-file)" tabindex="-1" class="title is-3">Work with Docker directly (no configuration file)</h3>
<p>To start this application with Docker, we don't necessarily need to create a <code>Dockerfile</code> or a <code>docker-compose.yml</code>. We can simply run (in the same folder of <code>app.js</code>) the following command:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker</span> run <span class="token punctuation">\</span>
<span class="token parameter variable">--rm</span> <span class="token punctuation">\</span>
<span class="token parameter variable">-p</span> <span class="token string">"4040:4040"</span> <span class="token punctuation">\</span>
<span class="token parameter variable">-v</span> <span class="token variable">${<span class="token environment constant">PWD</span>}</span>:/home/node/project <span class="token punctuation">\</span>
node:14 <span class="token punctuation">\</span>
<span class="token function">node</span> /home/node/project/app.js</code></pre>
<p>At this point we should be able to connect to our web server using a browser by simply visiting <code>http://127.0.01:4040</code>.</p>
<span style="position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 600px; ">
<picture>
<source type="image/png" srcset="https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-64.png 64w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-128.png 128w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-256.png 256w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-512.png 512w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-564.png 564w" sizes="(max-width: 600px) 100vw, 600px" />
<source type="image/webp" srcset="https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-64.webp 64w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-128.webp 128w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-256.webp 256w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-512.webp 512w, https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-564.webp 564w" sizes="(max-width: 600px) 100vw, 600px" />
<img loading="lazy" decoding="async" style="max-width: 100%; width: 100%; margin: 0px; vertical-align: middle;" alt="A simple Node.js web server seen from the browser" src="https://www.nodejsdesignpatterns.com/img/node-js-hello-world-http-server-from-browser-yCAvls1S5S-64.png" width="64" height="18" />
</picture>
</span>
<p>Let's discuss in more detail what's happening when we run the command above:</p>
<ul>
<li><code>--rm</code> means "destroy the container once it is stopped".</li>
<li><code>-p</code> maps a port of your pc to a port in the container (in this case we are saying map the local port <code>4040</code> to the port <code>4040</code> inside the container).</li>
<li><code>-v</code> stands for "volume" and it is used to map the folder (<code>$PWD</code>) in the development machine to the <code>/home/node/project</code> folder inside the container.</li>
<li><code>node:14</code> is the name and version of the docker image we want to use (this comes from the <a href="https://hub.docker.com/">official Docker Registry</a>)</li>
<li><code>node /home/node/project/app.js</code> is the command we want to run once the container starts. We are simply executing <code>node</code> and running our app file.</li>
</ul>
<p>Note that <code>/home/node/project/app.js</code> refers to the mapping of the local folder to volume in the container. Remember that Docker runs isolated processes, so by default they don't have access to our filesystem and we can only share files by explicitly defining and attaching volumes to running containers.</p>
<h3 id="work-with-docker-compose.yml-and-the-docker-compose-command" tabindex="-1" class="title is-3">Work with docker-compose.yml and the docker-compose command</h3>
<p>Let's now look at doing the same thing but with a different approach. By using an <code>.env</code> file, a <code>docker-compose.yml</code> file and <code>docker-compose</code>, we can define all the necessary settings there so we can keep our command line as lean as possible.</p>
<p>This is how we can define our <code>docker-compose.yml</code>:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">'3'</span>
<span class="token key atrule">services</span><span class="token punctuation">:</span>
<span class="token key atrule">nodejs</span><span class="token punctuation">:</span>
<span class="token key atrule">container_name</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>COMPOSE_PROJECT_NAME<span class="token punctuation">}</span>_$<span class="token punctuation">{</span>NODEJS_SERVER_NAME<span class="token punctuation">}</span>
<span class="token key atrule">image</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>NODEJS_CONTAINER_IMAGE<span class="token punctuation">}</span>
<span class="token key atrule">user</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>NODEJS_USER<span class="token punctuation">}</span>
<span class="token key atrule">command</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>NODEJS_COMMAND<span class="token punctuation">}</span>
<span class="token key atrule">ports</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token string">"4040:4040"</span>
<span class="token key atrule">environment</span><span class="token punctuation">:</span>
<span class="token key atrule">NODE_ENV</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>NODEJS_ENV<span class="token punctuation">}</span>
<span class="token key atrule">volumes</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> $<span class="token punctuation">{</span>PWD<span class="token punctuation">}</span><span class="token punctuation">:</span>/home/node/project
<span class="token key atrule">working_dir</span><span class="token punctuation">:</span> /home/node/project
<span class="token key atrule">networks</span><span class="token punctuation">:</span>
<span class="token key atrule">app_net</span><span class="token punctuation">:</span>
<span class="token key atrule">ipv4_address</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>IP_NODE_SERVER<span class="token punctuation">}</span>
<span class="token key atrule">networks</span><span class="token punctuation">:</span>
<span class="token key atrule">app_net</span><span class="token punctuation">:</span>
<span class="token key atrule">driver</span><span class="token punctuation">:</span> bridge
<span class="token key atrule">ipam</span><span class="token punctuation">:</span>
<span class="token key atrule">driver</span><span class="token punctuation">:</span> default
<span class="token key atrule">config</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> <span class="token key atrule">subnet</span><span class="token punctuation">:</span> $<span class="token punctuation">{</span>IP_SUBNET<span class="token punctuation">}</span></code></pre>
<p>And this is how we can define our <code>.env</code> file:</p>
<pre class="language-bash"><code class="language-bash"><span class="token assign-left variable">COMPOSE_PROJECT_NAME</span><span class="token operator">=</span>nodetest
<span class="token assign-left variable">NODEJS_CONTAINER_IMAGE</span><span class="token operator">=</span>node:14
<span class="token assign-left variable">NODEJS_SERVER_NAME</span><span class="token operator">=</span>nodejs
<span class="token assign-left variable">NODEJS_USER</span><span class="token operator">=</span>node
<span class="token assign-left variable">NODEJS_ENV</span><span class="token operator">=</span>development
<span class="token assign-left variable">NODEJS_COMMAND</span><span class="token operator">=</span>node app.js
<span class="token assign-left variable">IP_SUBNET</span><span class="token operator">=</span><span class="token number">172.16</span>.250.0/24
<span class="token assign-left variable">IP_LOCAL_COMPUTER</span><span class="token operator">=</span><span class="token number">172.16</span>.250.1
<span class="token assign-left variable">IP_NODE_SERVER</span><span class="token operator">=</span><span class="token number">172.16</span>.250.10</code></pre>
<p>At this point we just need to run <code>docker-compose up</code> and we should have the same result as the command described before.</p>
<p>If you want to run in background mode (so you can close the terminal) just add <code>-d</code> to the command like <code>docker-compose up -d</code>.</p>
<p>If you want to change something like node version, just edit the <code>.env</code> file in the same folder and done!</p>
<p>Once you are finished working on the project and you want to stop the containers, you can simply run:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">docker-compose</span> down</code></pre>
<h2 id="pros-and-cons-of-using-docker-with-node.js" tabindex="-1" class="title is-2">Pros and cons of using Docker with Node.js</h2>
<p>It might seem that Docker is like magic for your project but, as with many things, "all that glitters is not gold". Of course, Docker is great, but it can have its fair share of unexpected "surprises" that can be tedious to resolve.</p>
<p>Let's conclude this article, by discussing some of the main pros and cons of using Docker for Node.js development.</p>
<h3 id="pros" tabindex="-1" class="title is-3">PROS</h3>
<ul>
<li>You can manage multiple version of Node.js without having to install all of them or needing complicated setup to be able to switch version on demand.</li>
<li>You can use different version of Node.js at the same time. Imagine you have a microservice-oriented architecture and 2 different services need to use 2 different versions of Node.js!</li>
<li>It is easier to share a consistent setup with the members of your team. Docker becomes the only shared dependency that needs to be pre-installed.</li>
</ul>
<h3 id="cons" tabindex="-1" class="title is-3">CONS</h3>
<ul>
<li>For small projects running only one monolithic service setting up Docker and Docker Compose can be a bit of over-engineering.</li>
<li>With more advanced projects, you might need to setup some bash scripts to run some Docker command (because they might be long and difficult to remember).</li>
<li>For Mac OS and Windows you can have some setup problems and degraded performances because Docker is not running natively in these platforms.</li>
</ul>
<p>That's all! Hopefully you found this article interesting and you will decide to give Docker a shot for your Node.js development.</p>
<p>You can check out the code examples in the following repository: <a href="https://github.com/giuseppemorelli/docker-node-example">https://github.com/giuseppemorelli/docker-node-example</a>. If you liked this article, please consider giving it a star, everyone needs some vanity metrics, after all! :)</p>
<h2 id="about-the-author" tabindex="-1" class="title is-2">About the author</h2>
<p>Giuseppe Morelli is the guest author of this post. This is Giuseppe's bio.</p>
<p>Since I was a baby I came to love technology and programming. Today I am a remote worker for Italian and European companies. I like to work with Agile and Time-Material practices.
I have been a PHP Developer by trade since 2006, and I am particularly experienced with e-commerce development. Every day I try to learn something new by being a part of the open-source community. In 2020 I've started to study Node.js by reading Mario and Luciano's book :)</p>
<p>If you want to connect with me, check out <a href="https://giuseppemorelli.net/">my personal website</a> or my <a href="https://twitter.com/giuseppemorelli">Twitter account</a>.</p>
JavaScript async iterators2021-05-04T13:10:00Zhttps://www.nodejsdesignpatterns.com/blog/javascript-async-iterators/<p>Did you know that JavaScript offers a few protocols to allow iteration over certain objects? Of course, we know we can easily iterate over arrays, but with these protocols, you can make your own custom objects iterable as well.</p>
<p>When you have an iterable object representing a collection, you can use the <code>for...of</code> syntax to iterate over every single item of the collection.</p>
<p>But what if an object abstracts data generated asynchronously? For instance, think of an abstraction that allows us to fetch data from a paginated API, or think about some records consumed in batches from a database, or something as simple as a countdown timer. Well in these cases you can use the <code>for await...of</code> syntax!</p>
<p>In this article, we will learn more about the <em>iterator</em> and the <em>iterable</em> protocol (and their async counterparts) and we will see how to create custom objects that can expose their internal data in an ergonomic and idiomatic way.</p>
<h2 id="javascript-iteration-with-for...of" tabindex="-1" class="title is-2">JavaScript iteration with <code>for...of</code></h2>
<p>With ECMAScript 2015, JavaScript got the <code>for...of</code> syntax. This syntax provides a very easy way to iterate over collections, such as arrays, string, sets, and maps.</p>
<p>If you have never seen this syntax in action here are some examples:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> judokas <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">'Driulis Gonzalez Morales'</span><span class="token punctuation">,</span>
<span class="token string">'Ilias Iliadis'</span><span class="token punctuation">,</span>
<span class="token string">'Tadahiro Nomura'</span><span class="token punctuation">,</span>
<span class="token string">'Anton Geesink'</span><span class="token punctuation">,</span>
<span class="token string">'Teddy Riner'</span><span class="token punctuation">,</span>
<span class="token string">'Ryoko Tani'</span>
<span class="token punctuation">]</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> judoka <span class="token keyword">of</span> judokas<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>judoka<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>In the example above, we are iterating over an array using the <code>for...of</code> syntax. If we run this code, this is what we will get as output:</p>
<pre class="language-text"><code class="language-text">Driulis Gonzalez Morales
Ilias Iliadis
Tadahiro Nomura
Anton Geesink
Teddy Riner
Ryoko Tani</code></pre>
<p>The same syntax works also for iterating over the characters of a string:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> judoka <span class="token operator">=</span> <span class="token string">'Ryoko Tani'</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> char <span class="token keyword">of</span> judoka<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>char<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>The above will print:</p>
<pre class="language-text"><code class="language-text">R
y
o
k
o
T
a
n
i</code></pre>
<p>And we can even use this for <code>Set</code> and <code>Map</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> medals <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'gold'</span><span class="token punctuation">,</span> <span class="token string">'silver'</span><span class="token punctuation">,</span> <span class="token string">'bronze'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> medal <span class="token keyword">of</span> medals<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>medal<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>Which is going to output:</p>
<pre class="language-text"><code class="language-text">gold
silver
bronze</code></pre>
<p><code>Map</code> is especially interesting because we can use <em>destructuring</em> to iterate over key-value pairs:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> medallists <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Map</span><span class="token punctuation">(</span><span class="token punctuation">[</span>
<span class="token punctuation">[</span><span class="token string">'Teddy Riner'</span><span class="token punctuation">,</span> <span class="token number">33</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token string">'Driulis Gonzalez Morales'</span><span class="token punctuation">,</span> <span class="token number">16</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token string">'Ryoko Tani'</span><span class="token punctuation">,</span> <span class="token number">16</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span><span class="token string">'Ilias Iliadis'</span><span class="token punctuation">,</span> <span class="token number">15</span><span class="token punctuation">]</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token punctuation">[</span>judoka<span class="token punctuation">,</span> medals<span class="token punctuation">]</span> <span class="token keyword">of</span> medallists<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>judoka<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> has won </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>medals<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> medals</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>The above example will output:</p>
<pre class="language-text"><code class="language-text">Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals</code></pre>
<p>Finally, if you want to iterate over the key-value pairs of an object literal using the <code>for...of</code> syntax, we can do that by using the helper <code>Object.entries</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> medallists <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token string-property property">'Teddy Riner'</span><span class="token operator">:</span> <span class="token number">33</span><span class="token punctuation">,</span>
<span class="token string-property property">'Driulis Gonzalez Morales'</span><span class="token operator">:</span> <span class="token number">16</span><span class="token punctuation">,</span>
<span class="token string-property property">'Ryoko Tani'</span><span class="token operator">:</span> <span class="token number">16</span><span class="token punctuation">,</span>
<span class="token string-property property">'Ilias Iliadis'</span><span class="token operator">:</span> <span class="token number">15</span>
<span class="token punctuation">}</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token punctuation">[</span>judoka<span class="token punctuation">,</span> medals<span class="token punctuation">]</span> <span class="token keyword">of</span> Object<span class="token punctuation">.</span><span class="token function">entries</span><span class="token punctuation">(</span>medallists<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>judoka<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> has won </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>medals<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> medals</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>The code snippet above will produce the same output as the previous example.</p>
<p>What's interesting here is that, if we try to use the <code>for...of</code> syntax directly on the object <code>medallists</code> (without <code>Object.entries</code>), we get the following error:</p>
<pre class="language-text"><code class="language-text">for (const [judoka, medals] of medallists) {
^
TypeError: medallists is not iterable
at Object.<anonymous> (.../05-for-of-object.js:8:32)
at Module._compile (node:internal/modules/cjs/loader:1108:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10)
at Module.load (node:internal/modules/cjs/loader:988:32)
at Function.Module._load (node:internal/modules/cjs/loader:828:14)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
at node:internal/main/run_main_module:17:47</anonymous></code></pre>
<p>Let's read this error once again: <code>medallists is not iterable</code>!</p>
<p>Yeah, the error is clear: a regular JavaScript object is not <em>iterable</em>, while arrays, strings, maps, and sets are!</p>
<p>But what does it mean for an object to be <em>iterable</em>?</p>
<p>During the rest of this article, we will learn how JavaScript knows if a given object is <strong>iterable</strong> and how we can make our own custom <em>iterable</em> objects.</p>
<p>But first let's quickly take a look at how we can use async iterators.</p>
<h2 id="javascript-iteration-with-for-await...of" tabindex="-1" class="title is-2">JavaScript iteration with <code>for await...of</code></h2>
<p>ECMAScript 2018 introduced a new syntax called <code>for await...of</code>. This syntax is somewhat similar to <code>for...of</code> but it allows us to iterate over <em>asynchronous collections</em> where data becomes available over time in an asynchronous fashion.</p>
<p>A good use case for this syntax is reading data from a remote source like a database.</p>
<p>Here's an example that uses AWS DynamoDB and the <code>for await...of</code> syntax to list all the tables available in our account:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> DynamoDBClient<span class="token punctuation">,</span> paginateListTables <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'@aws-sdk/client-dynamodb'</span>
<span class="token keyword">const</span> client <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">DynamoDBClient</span><span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> page <span class="token keyword">of</span> <span class="token function">paginateListTables</span><span class="token punctuation">(</span><span class="token punctuation">{</span> client <span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// page.TableNames is an array of table names</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> tableName <span class="token keyword">of</span> page<span class="token punctuation">.</span>TableNames<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>tableName<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>In the example above, <code>paginateListTables</code> will <em>produce</em> pages over time, and every page will contain a portion of the data (information about all the available tables).</p>
<p>This approach allows us to list hundreds or even thousands of table names efficiently. In fact, the data can be printed as soon as it is available and we don't have to wait for the entire data set to be received.</p>
<p>Note how we are combining here <code>for await...of</code> and <code>for...of</code>. Pages become available over time asynchronously, so we need to use <code>for await...of</code> to iterate over this data. Every page contains an array of table names, so in this case, to iterate over every single table name we can simply use <code>for...of</code>.</p>
<p>In general, we can use the <code>for await...of</code> syntax with objects that are <strong>async iterable</strong>.</p>
<p>In the next few sections, we will see how JavaScript classifies a given object as <em>async iterable</em> and how we can build our custom <em>async iterable</em> objects.</p>
<h2 id="the-javascript-iterator-protocol" tabindex="-1" class="title is-2">The JavaScript iterator protocol</h2>
<p>JavaScript defines a number of protocols that are used to make objects iterable (or async iterable).</p>
<p>The first one we are going to start with is the <strong>iterator protocol</strong>.</p>
<blockquote>
<p>In JavaScript, an object is <strong>an iterator</strong> if it has a <code>next()</code> method. Every time you call it, it returns an object with the keys <code>done</code> (boolean) and <code>value</code>.</p>
</blockquote>
<p>Let's see an example. Let's say we want to build a countdown. This countdown is initialized with a positive integer and it will produce all the numbers from that value down to <code>0</code>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">createCountdown</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">from</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">let</span> nextVal <span class="token operator">=</span> from
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token function">next</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>nextVal <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
<span class="token literal-property property">value</span><span class="token operator">:</span> nextVal<span class="token operator">--</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>In this example, <code>createCountdown</code> is a simple factory function. From this function, we return an <em>iterator</em> object. In fact, the object implements the <em>iterator protocol</em> defined above. Note that the returned object implements a <code>next()</code> method and that this method returns either <code>{done: true}</code> or <code>{done: false, value: someNumber}</code>.</p>
<p>Let's see now how can we use this object to extract all the values:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 3 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 2 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 1 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 0 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: true }</span></code></pre>
<p>Or if we want to use this object with a more generic loop:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
<span class="token keyword">let</span> result <span class="token operator">=</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">while</span> <span class="token punctuation">(</span><span class="token operator">!</span>result<span class="token punctuation">.</span>done<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>result<span class="token punctuation">.</span>value<span class="token punctuation">)</span>
result <span class="token operator">=</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>The code above will produce the following output:</p>
<pre class="language-text"><code class="language-text">3
2
1
0</code></pre>
<p>This is not the most intuitive or ergonomic approach, but the iterator protocol is the basic building block for the <em>iterable protocol</em> which enables the <code>for...of</code> syntax.</p>
<h2 id="the-javascript-iterable-protocol" tabindex="-1" class="title is-2">The JavaScript iterable protocol</h2>
<p>As we said, the <em>iterable protocol</em> builds on top of the <em>iterator protocol</em> that we just explored. Let's see how:</p>
<blockquote>
<p>An object is <strong>iterable</strong> if it implements the <code>@@iterator</code> method, a zero-argument function that <strong>returns an iterator</strong>.</p>
</blockquote>
<p>Note that with <code>@@iterator</code> we indicate a symbol that is accessed with the global value <code>Symbol.iterator</code>.</p>
<p>Can we make our countdown example <em>iterable</em>? We certainly can!</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">createCountdown</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">from</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">let</span> nextVal <span class="token operator">=</span> from
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token punctuation">[</span>Symbol<span class="token punctuation">.</span>iterator<span class="token punctuation">]</span><span class="token operator">:</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">(</span><span class="token punctuation">{</span>
<span class="token function">next</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>nextVal <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> nextVal<span class="token operator">--</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>In this new example, our factory function returns an <em>iterable</em> object. This object in fact has a method referenced with <code>Symbol.iterator</code> that returns an <em>iterator</em> object.</p>
<p>At this point, once we have an instance of a countdown, we can use the <code>for..of</code> syntax to iterate over all the numbers from the countdown:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> value <span class="token keyword">of</span> countdown<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>The example above will output:</p>
<pre class="language-text"><code class="language-text">3
2
1
0</code></pre>
<p>Hooray! Now we know how to make iterators and iterable objects. If you find the two terms confusing, don't worry, that's quite common!</p>
<p>One good way to try to remember and distinguish these 2 concepts is the following:</p>
<ul>
<li>An <em>iterator</em> is a lower-level object that allows us to iterate over some data set using <code>next()</code></li>
<li>An <em>iterable</em> is an object on which we can iterate over using the <code>for...of</code> syntax.</li>
</ul>
<h3 id="using-javascript-generator-functions" tabindex="-1" class="title is-3">Using JavaScript generator functions</h3>
<p>An interesting detail is that JavaScript generator functions <em>produce</em> iterators.</p>
<p>This allows us to simplify the way we can implement both the <em>iterator</em> and the <em>iterable</em> protocols.</p>
<p>Let's see how can we rewrite our countdown logic using a generator function:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token operator">*</span> <span class="token function">createCountdown</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">from</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>If we call <code>createCountdown(3)</code> we get an <em>iterator</em>. So this is perfectly compatible with our previous <em>iterator</em> implementation:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { value: 3, done: false }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { value: 2, done: false }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { value: 1, done: false }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { value: 0, done: false }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { value: undefined, done: true }</span></code></pre>
<p>Similarly, we can use generators to implement the <em>iterable protocol</em>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token function">createCountdown</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">from</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token punctuation">[</span>Symbol<span class="token punctuation">.</span>iterator<span class="token punctuation">]</span><span class="token operator">:</span> <span class="token keyword">function</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>And this factory will produce iterable objects, exactly as before:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> value <span class="token keyword">of</span> countdown<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>In general, generators can be considered great syntactic sugars to write iterators.</p>
<h3 id="the-spread-syntax-for-iterable-objects" tabindex="-1" class="title is-3">The spread syntax for iterable objects</h3>
<p>Another interesting detail is that all iterable objects can be used with the <em>spread syntax</em>.</p>
<p>The spread syntax looks like <code>...someIterable</code> and it basically allows us to apply every single element from the iterable to a given context.</p>
<p>The most common use cases are found with array literals and function calls.</p>
<p>Let's see a couple of examples:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> from5to0 <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token operator">...</span>countdown<span class="token punctuation">]</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>from5to0<span class="token punctuation">)</span> <span class="token comment">// [ 5, 4, 3, 2, 1, 0 ]</span>
<span class="token keyword">const</span> countdown2 <span class="token operator">=</span> <span class="token function">createCountdown</span><span class="token punctuation">(</span><span class="token number">6</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'countdown2 data:'</span><span class="token punctuation">,</span> <span class="token operator">...</span>countdown2<span class="token punctuation">)</span>
<span class="token comment">// countdown2 data: 6 5 4 3 2 1 0</span></code></pre>
<p>This is something we see most often with arrays, but it's important to note that any iterable object can be used with the spread syntax.</p>
<h2 id="the-javascript-async-iterator-protocol" tabindex="-1" class="title is-2">The JavaScript async iterator protocol</h2>
<p>Ok, so far we have explored only synchronous iteration protocols. What about async?</p>
<p>Unsurprisingly, both the iterator protocol and the iterable protocol have their async counterparts!</p>
<p>Let's start with the <strong>async iterator protocol</strong>:</p>
<blockquote>
<p>An object is an <strong>async iterator</strong> if it has a <code>next()</code> method. Every time you call it, it returns <strong>a promise that resolves</strong> to an object with the keys <code>done</code> (boolean) and <code>value</code>.</p>
</blockquote>
<p>Note how this is quite similar to the synchronous version of the iterator protocol. The main difference here is that the <code>next()</code> function won't return an object straight away. Instead, it will return a promise that will eventually resolve to an object.</p>
<p>Let's now revisit our countdown example and let's say we want some time to pass before numbers are <em>produced</em>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> setTimeout <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'timers/promises'</span>
<span class="token keyword">function</span> <span class="token function">createAsyncCountdown</span> <span class="token punctuation">(</span><span class="token parameter">from<span class="token punctuation">,</span> delay <span class="token operator">=</span> <span class="token number">1000</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">let</span> nextVal <span class="token operator">=</span> from
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token keyword">async</span> <span class="token function">next</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> <span class="token function">setTimeout</span><span class="token punctuation">(</span>delay<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span>nextVal <span class="token operator"><</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> nextVal<span class="token operator">--</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>Note that this time we are using an <em>async</em> function to implement <code>next()</code>. This will make this method immediately return a promise, that will later resolve when we run one of the <code>return</code> statements from within the <em>async</em> function.</p>
<p>Also, note that here we are using <code>setTimeout</code> from <code>timers/promises</code>, a new core module available from Node.js 16.</p>
<p>Ok, now we are ready to use this iterator:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createAsyncCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token keyword">await</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 3 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token keyword">await</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 2 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token keyword">await</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 1 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token keyword">await</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: false, value: 0 }</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token keyword">await</span> countdown<span class="token punctuation">.</span><span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// { done: true }</span></code></pre>
<p>This works very similarly to its synchronous counterpart with two notable exceptions:</p>
<ul>
<li>We need to use <code>await</code> to wait for the next element to be produced.</li>
<li>Between one element and another about 1 second will pass, so this iteration is much slower.</li>
</ul>
<p style="text-align: center"><img loading="lazy" decoding="async" style="max-width: 100%; margin: 0px; vertical-align: middle;" alt="An example of JavaScript async iterator" src="https://www.nodejsdesignpatterns.com/img/javascript-async-iterator-countdown_fe19a712.gif" /></p>
<p>Of course, here we can use generators as well as a nice syntactic sugar:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> setTimeout <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'timers/promises'</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token operator">*</span> <span class="token function">createAsyncCountdown</span> <span class="token punctuation">(</span><span class="token parameter">from<span class="token punctuation">,</span> delay <span class="token operator">=</span> <span class="token number">1000</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> <span class="token function">setTimeout</span><span class="token punctuation">(</span>delay<span class="token punctuation">)</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>This code is more concise and probably more readable, at least to those accustomed to async functions and generator functions.</p>
<h2 id="the-javascript-async-iterable-protocol" tabindex="-1" class="title is-2">The JavaScript async iterable protocol</h2>
<p>Let's now discuss the last iteration protocol: the <em>async iterable protocol</em>!</p>
<blockquote>
<p>An object is an <strong>async iterable</strong> if it implements the <code>@@asyncIterator</code> method, a zero-argument function that returns an <strong>async iterator</strong>.</p>
</blockquote>
<p>Note that with <code>@@asyncIterator</code> we indicate a symbol that can be accessed with the global value <code>Symbol.asyncIterator</code>.</p>
<p>Once again, this definition is quite similar to its synchronous counterpart. The main difference is that this type we have to use <code>Symbol.asyncIterator</code> and that it must return an <em>async</em> iterator.</p>
<p>Let's revisit our async countdown example:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> setTimeout <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'timers/promises'</span>
<span class="token keyword">function</span> <span class="token function">createAsyncCountdown</span> <span class="token punctuation">(</span><span class="token parameter">from<span class="token punctuation">,</span> delay <span class="token operator">=</span> <span class="token number">1000</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span>
<span class="token punctuation">[</span>Symbol<span class="token punctuation">.</span>asyncIterator<span class="token punctuation">]</span><span class="token operator">:</span> <span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token operator">*</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> <span class="token function">setTimeout</span><span class="token punctuation">(</span>delay<span class="token punctuation">)</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>At this point, our <code>createAsyncCountdown</code> returns a valid async iterator, so we can finally use the <code>for await...of</code> syntax:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> countdown <span class="token operator">=</span> <span class="token function">createAsyncCountdown</span><span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> value <span class="token keyword">of</span> countdown<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>As you might expect, this code will produce <code>3</code>, <code>2</code>, <code>1</code> and <code>0</code> with a delay:</p>
<p style="text-align: center"><img loading="lazy" decoding="async" style="max-width: 100%; margin: 0px; vertical-align: middle;" alt="An example of JavaScript async iterator" src="https://www.nodejsdesignpatterns.com/img/javascript-async-iterable-countdown_6150da48.gif" /></p>
<p>Great!</p>
<p>At this point, we know how the JavaScript iteration protocol work and how to create iterator and iterable objects in a synchronous and asynchronous fashion!</p>
<h2 id="combining-iterator-and-iterable" tabindex="-1" class="title is-2">Combining iterator and iterable</h2>
<p>Can an object be both an iterator and an iterable at the same time?</p>
<p>Yes! Nothing is stopping us from implementing both protocols for a given object. This basically means that <code>@@iterator</code> or <code>@@asyncIterator</code> will have to return the same object as in the following example:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> iterableIterator <span class="token operator">=</span> <span class="token punctuation">{</span>
<span class="token function">next</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token punctuation">{</span> <span class="token literal-property property">done</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span> <span class="token literal-property property">value</span><span class="token operator">:</span> <span class="token string">"hello"</span> <span class="token punctuation">}</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">[</span>Symbol<span class="token punctuation">.</span>iterator<span class="token punctuation">]</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">return</span> <span class="token keyword">this</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> value <span class="token keyword">of</span> iterableIterator<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>value<span class="token punctuation">)</span> <span class="token comment">// "hello"</span>
<span class="token punctuation">}</span></code></pre>
<p>The example above will print "hello" endlessly.</p>
<p>What's even cooler is that generator functions are also iterable. This means that we can greatly simplify our countdown examples.</p>
<p>Let's see how the syncrhonous countdown would look like:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">function</span> <span class="token operator">*</span> <span class="token function">createCountdown</span> <span class="token punctuation">(</span><span class="token parameter"><span class="token keyword">from</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>We don't even need to bother with <code>Symbol.iterator</code>!</p>
<p>The same goes for the asynchronous version of our countdown:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> setTimeout <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'timers/promises'</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token operator">*</span> <span class="token function">createAsyncCountdown</span> <span class="token punctuation">(</span><span class="token parameter">from<span class="token punctuation">,</span> delay <span class="token operator">=</span> <span class="token number">1000</span></span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">let</span> i <span class="token operator">=</span> from<span class="token punctuation">;</span> i <span class="token operator">>=</span> <span class="token number">0</span><span class="token punctuation">;</span> i<span class="token operator">--</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">await</span> <span class="token function">setTimeout</span><span class="token punctuation">(</span>delay<span class="token punctuation">)</span>
<span class="token keyword">yield</span> i
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>And here we don't have to explicitly use <code>Symbol.asyncIterator</code>, in fact, an async generator function is already an async iterable!</p>
<p>If we decide to use generators, this will help us to write even more concise iterator and iterable objects.</p>
<h2 id="using-javascript-iteration-protocols-with-node.js" tabindex="-1" class="title is-2">Using JavaScript iteration protocols with Node.js</h2>
<p>Everything we have been discussing so far is part of the JavaScript specification, but what about Node.js?</p>
<p>Actually, support for these features looks quite good in Node.js!</p>
<p>Synchronous iteration protocols have been supported in Node.js for a long time (since Node.js 0.12).</p>
<p>Recent versions of Node.js (Node.js 10.3) introduced support for async iterators and the <code>for await...of</code> syntax.</p>
<p>Synchronous iterable objects and the <code>for...of</code> syntax are quite widespread, so in the next sections, we will focus on providing some examples of how you can take advantage of its asynchronous counterpart and the <code>for await...of</code> syntax.</p>
<h3 id="node.js-readable-streams-and-async-iterators" tabindex="-1" class="title is-3">Node.js readable streams and async iterators</h3>
<p>One interesting detail that needs a bit more visibility is that Node.js <em>Readable</em> streams are async iterable objects since Node.js 11.14.</p>
<p>This basically means that we can consume data from a Readable stream using <code>for await...of</code>.</p>
<p>Let's see a simple example of a CLI utility that allows us to read the content of a given file and count the number of bytes:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> createReadStream <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'fs'</span>
<span class="token keyword">const</span> sourcePath <span class="token operator">=</span> process<span class="token punctuation">.</span>argv<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span>
<span class="token keyword">const</span> sourceStream <span class="token operator">=</span> <span class="token function">createReadStream</span><span class="token punctuation">(</span>sourcePath<span class="token punctuation">)</span>
<span class="token keyword">let</span> bytes <span class="token operator">=</span> <span class="token number">0</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> chunk <span class="token keyword">of</span> sourceStream<span class="token punctuation">)</span> <span class="token punctuation">{</span>
bytes <span class="token operator">+=</span> chunk<span class="token punctuation">.</span>length
<span class="token punctuation">}</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>process<span class="token punctuation">.</span>argv<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>bytes<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"> bytes</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span></code></pre>
<p>The interesting thing is that when we are using this approach the stream is consumed in <em>non-flowing</em> (or <em>paused</em>) mode which can help us to handle backpressure in a very simple way.</p>
<p>Let's say that we want to write every chunk to a very slow transform stream (that we are going to identify with <code>SlowTransform</code>), this is how we can handle backpressure:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> createReadStream <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'fs'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> once <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'events'</span>
<span class="token keyword">const</span> sourcePath <span class="token operator">=</span> process<span class="token punctuation">.</span>argv<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span>
<span class="token keyword">const</span> sourceStream <span class="token operator">=</span> <span class="token function">createReadStream</span><span class="token punctuation">(</span>sourcePath<span class="token punctuation">)</span>
<span class="token keyword">const</span> destStream <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">SlowTransform</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> chunk <span class="token keyword">of</span> sourceStream<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> canContinue <span class="token operator">=</span> destStream<span class="token punctuation">.</span><span class="token function">write</span><span class="token punctuation">(</span>chunk<span class="token punctuation">)</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>canContinue<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token comment">// backpressure, now we stop and we need to wait for drain</span>
<span class="token keyword">await</span> <span class="token function">once</span><span class="token punctuation">(</span>destStream<span class="token punctuation">,</span> <span class="token string">'drain'</span><span class="token punctuation">)</span>
<span class="token comment">// ok now it's safe to resume writing</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>Note that having an <code>await</code> inside the <code>for await...of</code> block will effectively pause the iteration. This will stop consuming data from the source stream until the destination stream is drained.</p>
<h3 id="converting-a-node.js-event-emitter-to-an-async-iterable" tabindex="-1" class="title is-3">Converting a Node.js event emitter to an async iterable</h3>
<p>Another interesting use case for async iteration in Node.js is when dealing with repeated events happening over time.</p>
<p>Events are generally fired by an <em>event emitter</em> and, since version 12.16, Node.js offers an interesting utility to convert a sequence of events into an async iterable.</p>
<p>We can see a simple example by using the third party module <a href="https://npm.im/glob"><code>glob</code></a> which allows us to find files matching a specific glob expression.</p>
<p>In this example we will find and print all the JavaScript files (<code>.js</code> extension) in the current folder (and subfolders):</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> on <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'events'</span>
<span class="token keyword">import</span> glob <span class="token keyword">from</span> <span class="token string">'glob'</span>
<span class="token keyword">const</span> matcher <span class="token operator">=</span> <span class="token function">glob</span><span class="token punctuation">(</span><span class="token string">'**/*.js'</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token punctuation">[</span>filePath<span class="token punctuation">]</span> <span class="token keyword">of</span> <span class="token function">on</span><span class="token punctuation">(</span>matcher<span class="token punctuation">,</span> <span class="token string">'match'</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>filePath<span class="token punctuation">)</span>
<span class="token punctuation">}</span></code></pre>
<p>As you can see, we are using <code>on(matcher, 'match')</code> to create an async iterable that will <em>produce</em> a new value every time the <code>matcher</code> instance fires a <code>match</code> event.</p>
<p>Note that the value produced by this async iterable at every iteration is an array containing all the values contained in the original <code>match</code> event. This is the reason why we need to use destructuring to extract the <code>filePath</code>.</p>
<p>At this point you might ask: "wait a second, but how do we know, with this approach, when there are no more events to process?"</p>
<p>And that's a great question... we don't!</p>
<p>In fact, we are only listening for <code>match</code> events and we don't really have a way to stop the loop.</p>
<p>If we put any code just after the <code>for await...of</code> loop, that code will never be executed.</p>
<p>One solution to this problem is the <code>AbortController</code>, which allows us to create an Async Iterable that can be aborted.</p>
<p>With that, we could listen for the <code>end</code> event on our <code>matcher</code> instance and, once that happens, we can use the <code>AbortController</code> to stop the iteration.</p>
<p>Let's see some code:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> on <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'events'</span>
<span class="token keyword">import</span> glob <span class="token keyword">from</span> <span class="token string">'glob'</span>
<span class="token keyword">const</span> matcher <span class="token operator">=</span> <span class="token function">glob</span><span class="token punctuation">(</span><span class="token string">'**/*.js'</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> ac <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">global<span class="token punctuation">.</span>AbortController</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
matcher<span class="token punctuation">.</span><span class="token function">once</span><span class="token punctuation">(</span><span class="token string">'end'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> ac<span class="token punctuation">.</span><span class="token function">abort</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">try</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> <span class="token punctuation">[</span>filePath<span class="token punctuation">]</span> <span class="token keyword">of</span> <span class="token function">on</span><span class="token punctuation">(</span>matcher<span class="token punctuation">,</span> <span class="token string">'match'</span><span class="token punctuation">,</span> <span class="token punctuation">{</span> <span class="token literal-property property">signal</span><span class="token operator">:</span> ac<span class="token punctuation">.</span>signal <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">./</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>filePath<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span>err<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span>ac<span class="token punctuation">.</span>signal<span class="token punctuation">.</span>aborted<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span>
process<span class="token punctuation">.</span><span class="token function">exit</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'NOW WE GETTING HERE! :)'</span><span class="token punctuation">)</span></code></pre>
<p>In the code example above, you can see that we are creating a new instance of <code>AbortController</code> by using <code>new global.AbortController()</code>.</p>
<p>Then, we listen for the <code>end</code> event on our <code>matcher</code> and when that happens we invoke <code>abort()</code> on our <code>AbortController</code> instance.</p>
<p>The last step is to pass the <code>AbortController</code> instance to the <code>on()</code> function. We do that by passing an options object and using the <code>signal</code> option.</p>
<p>You might have noticed that we also added a <code>try/catch</code> block. This is actually very important. When we stop the iteration using an <code>AbortController</code> this will not simply stop the iteration, but it will raise an exception.</p>
<p>In this case the exception is expected, so we handle it gracefully. We also want to distinguish the abort exception from other unintended exceptions, so we make sure to check wheter our abort signal was raised, otherwise we exit the program with an error.</p>
<p>Note that this is a lot of work, so this pattern, while it's cute, might not always give you great benefits compared to simply handling events using regular listeners.</p>
<h2 id="consuming-paginated-data-with-async-iterators" tabindex="-1" class="title is-2">Consuming paginated data with async iterators</h2>
<p>As we mentioned before with the DynamoDB examples, another great use case for async iteration is when we need to fetch data from a remote paginated dataset. Even more so when we cannot determine how to access the next page until we have fetched the previous one. This is a typical example of asynchronous sequential iteration and it's probably the most adequate use case for async iterators.</p>
<p>Just to present a very simple example, let's use <a href="https://swapi.dev/">a free and open-source Star Wars API</a> (happy May 4th everyone!) which allows us to access all the Star Wars characters in a paginated fashion.</p>
<p>To get data from this API, we can make a GET request to the following endpoint:</p>
<pre class="language-text"><code class="language-text">https://swapi.dev/api/people</code></pre>
<p>This request will respond with a JSON message that looks like this:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
<span class="token property">"count"</span><span class="token operator">:</span> <span class="token number">82</span><span class="token punctuation">,</span>
<span class="token property">"next"</span><span class="token operator">:</span> <span class="token string">"http://swapi.dev/api/people/?page=2"</span><span class="token punctuation">,</span>
<span class="token property">"results"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
<span class="token punctuation">{</span>
<span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Sly Moore"</span><span class="token punctuation">,</span>
<span class="token property">"height"</span><span class="token operator">:</span> <span class="token string">"178"</span><span class="token punctuation">,</span>
<span class="token property">"..."</span><span class="token operator">:</span> <span class="token string">"more fields..."</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Another character"</span><span class="token punctuation">,</span>
<span class="token property">"height"</span><span class="token operator">:</span> <span class="token string">"whatever"</span><span class="token punctuation">,</span>
<span class="token property">"..."</span><span class="token operator">:</span> <span class="token string">"more fields..."</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
<span class="token punctuation">{</span>
<span class="token property">"..."</span><span class="token operator">:</span> <span class="token string">"more characters"</span>
<span class="token punctuation">}</span>
<span class="token punctuation">]</span>
<span class="token punctuation">}</span></code></pre>
<p>Note that the <code>next</code> field contains the URL that we can use to fetch the data from the following page. All the records for the current page are presented in the <code>results</code> field.</p>
<p>With these details in mind, this is how we can create a custom client that allows us to fetch all the characters using the <code>for await...of</code> syntax:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> axios <span class="token keyword">from</span> <span class="token string">'axios'</span>
<span class="token keyword">async</span> <span class="token keyword">function</span> <span class="token operator">*</span> <span class="token function">starWarsCharacters</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">let</span> nextUrl <span class="token operator">=</span> <span class="token string">'https://swapi.dev/api/people'</span>
<span class="token keyword">while</span> <span class="token punctuation">(</span>nextUrl<span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> response <span class="token operator">=</span> <span class="token keyword">await</span> axios<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span>nextUrl<span class="token punctuation">)</span>
nextUrl <span class="token operator">=</span> response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>next
<span class="token keyword">yield</span> response<span class="token punctuation">.</span>data<span class="token punctuation">.</span>results
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>Now we can use this function as follows:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> page <span class="token keyword">of</span> <span class="token function">starWarsCharacters</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
<span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token keyword">const</span> char <span class="token keyword">of</span> page<span class="token punctuation">)</span> <span class="token punctuation">{</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>char<span class="token punctuation">.</span>name<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token punctuation">}</span></code></pre>
<p>If we run this code we should see the following output:</p>
<pre class="language-text"><code class="language-text">Luke Skywalker
C-3PO
R2-D2
Darth Vader
Leia Organa
[... other 77 names]</code></pre>
<h2 id="wrapping-up" tabindex="-1" class="title is-2">Wrapping up</h2>
<p>This concludes our exploration of JavaScript iteration protocols. At this point, you should feel comfortable understanding what the various protocols are and how to use <code>for...of</code> and <code>for await...of</code> effectively in both JavaScript and Node.js.</p>
<p>These techniques are often ideal to implement synchronous and asynchronous sequential iteration patterns, which makes them very effective tools in our toolbelt.</p>
<p>If you are interested in learning more patterns and interesting Node.js techniques, consider checking out <a href="https://www.nodejsdesignpatterns.com/">Node.js Design Patterns</a>. You can grab a free chapter for free by filling the form at the end of this page. Among other things, this free chapter contains some other examples of iteration and async iterators!</p>
<p>See you at the next post!</p>
<p>CIAO 👋</p>
<p>P.S. All the examples presented in this article are available on GitHub at <a href="https://github.com/lmammino/javascript-iteration-protocols">lmammino/javascript-iteration-protocols</a>.</p>
<p><small>Thanks to <a href="https://twitter.com/mariocasciaro">Mario Casciaro</a> for the kind review of this article and to <a href="https://twitter.com/Dominus_Kelvin">Kelvin Omereshone</a> for finding and fixing a few typos.</small></p>
Node.js stream consumer utilities2022-03-11T12:30:00Zhttps://www.nodejsdesignpatterns.com/blog/node-js-stream-consumer/<p>How many times did you need to read the entire content of a <code>Readable</code> stream into memory and ended up writing something like this?</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> chunks <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
someReadableStream<span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">'data'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">chunk</span><span class="token punctuation">)</span> <span class="token operator">=></span> chunks<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>chunk<span class="token punctuation">)</span><span class="token punctuation">)</span>
someReadableStream<span class="token punctuation">.</span><span class="token function">on</span><span class="token punctuation">(</span><span class="token string">'end'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> data <span class="token operator">=</span> Buffer<span class="token punctuation">.</span><span class="token function">concat</span><span class="token punctuation">(</span>chunks<span class="token punctuation">)</span>
<span class="token comment">// do something with `data`</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Or using <a href="https://www.nodejsdesignpatterns.com/blog/javascript-async-iterators/">async iterators</a>:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">const</span> chunks <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> <span class="token keyword">await</span> <span class="token punctuation">(</span><span class="token keyword">const</span> chunk <span class="token keyword">of</span> someReadableStream<span class="token punctuation">)</span> <span class="token punctuation">{</span>
chunks<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>chunk<span class="token punctuation">)</span>
<span class="token punctuation">}</span>
<span class="token keyword">const</span> data <span class="token operator">=</span> Buffer<span class="token punctuation">.</span><span class="token function">concat</span><span class="token punctuation">(</span>chunks<span class="token punctuation">)</span>
<span class="token comment">// do something with `data`</span></code></pre>
<p>This is a bit of a boilerplate-heavy solution for just consuming an entire readable stream. Consider that here we are not even handling errors, trying to do that (as we should!) will add even more boilerplate!</p>
<p>If you wish there was an easier way, well keep reading, this article is for you!</p>
<h2 id="the-stream%2Fconsumers-module" tabindex="-1" class="title is-2">The <code>stream/consumers</code> module</h2>
<p>Since Node.js version 16, there is a new built in stream utility library called <code>stream/consumers</code> which offers a bunch of useful utilities to consume the entire content of a <code>ReadableStream</code>.</p>
<p><s>At the time of writing this article, <code>stream/consumers</code> does not even appear in the official Node.js documentation, so it's still of a hidden gem. Hopefully this article will help to spread the word a little bit.</s></p>
<p><strong>UPDATE</strong>: It turns out that this module is documented under the <a href="https://nodejs.org/api/webstreams.html#streamconsumersjsonstream">Web Streams API section</a> and in fact these utilities are both compatible with Node.js streams and web streams.</p>
<p>Without further ado, let's see what's inside this module:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> consumers <span class="token keyword">from</span> <span class="token string">'stream/consumers'</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>consumers<span class="token punctuation">)</span></code></pre>
<p>If we run this code, we will see the following output:</p>
<pre class="language-plain"><code class="language-plain">{
arrayBuffer: [AsyncFunction: arrayBuffer],
blob: [AsyncFunction: blob],
buffer: [AsyncFunction: buffer],
text: [AsyncFunction: text],
json: [AsyncFunction: json]
}</code></pre>
<p>So, what we can tell is that the <code>stream/consumers</code> module exposes some async function that seem to be helpful to consume <code>Readable</code> streams in different ways:</p>
<ul>
<li>As binary data (<code>ArrayBuffer</code>, <code>Blob</code>, <code>Buffer</code>)</li>
<li>As text</li>
<li>As JSON</li>
</ul>
<p>In the next sections we will see some examples on how to use these functions.</p>
<h2 id="reading-a-binary-file-from-a-readable-stream" tabindex="-1" class="title is-2">Reading a binary file from a Readable stream</h2>
<p>Ok, let's say that we have to do some processing on a picture and, in order to do that, we need to load the entire binary content representing the picture from a file to memory.</p>
<p>We could easily use the <code>buffer</code> function from the <code>stream/consumers</code> library to do that:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> path <span class="token keyword">from</span> <span class="token string">'path'</span>
<span class="token keyword">import</span> <span class="token punctuation">{</span> createReadStream <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'fs'</span>
<span class="token keyword">import</span> consumers <span class="token keyword">from</span> <span class="token string">'stream/consumers'</span>
<span class="token keyword">const</span> __dirname <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">URL</span><span class="token punctuation">(</span><span class="token string">'.'</span><span class="token punctuation">,</span> <span class="token keyword">import</span><span class="token punctuation">.</span>meta<span class="token punctuation">.</span>url<span class="token punctuation">)</span><span class="token punctuation">.</span>pathname<span class="token punctuation">;</span>
<span class="token keyword">const</span> readable <span class="token operator">=</span> <span class="token function">createReadStream</span><span class="token punctuation">(</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>__dirname<span class="token punctuation">,</span> <span class="token string">'picture.png'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">const</span> data <span class="token operator">=</span> <span class="token keyword">await</span> consumers<span class="token punctuation">.</span><span class="token function">buffer</span><span class="token punctuation">(</span>readable<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span></code></pre>
<p>If we execute this code, we will see the following output:</p>
<pre class="language-plain"><code class="language-plain">(node:7685) ExperimentalWarning: buffer.Blob is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
<Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 02 80 00 00 01 c1 08 02 00 00 00 76 43 9d 20 00 00 00 01 73 52 47 42 00 ae ce 1c e9 00 00 00 04 ... 347824 more bytes></code></pre>
<p>You can see that all the binary data (~300Kb) was loaded in the buffer, but also that this feature (as of v17.7.1) is still experimental and therefore we get a warning. You will get a similar warning also when trying to use <code>consumers.arrayBuffer</code> and <code>consumers.blob</code>. This will be the case until <code>buffer.Blob</code> is stabilised.</p>
<h2 id="reading-a-json-object-from-a-readable-stream" tabindex="-1" class="title is-2">Reading a JSON object from a Readable stream</h2>
<p>Similarly to what we just saw in the previous section, we can use the <code>stream/consumers</code> library to consume the entire content of a <code>ReadableStream</code> as a JSON encoded string. For instance, we could use this to process the response body from an HTTP request:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> <span class="token punctuation">{</span> get <span class="token punctuation">}</span> <span class="token keyword">from</span> <span class="token string">'https'</span>
<span class="token keyword">import</span> consumers <span class="token keyword">from</span> <span class="token string">'stream/consumers'</span>
<span class="token keyword">const</span> url <span class="token operator">=</span> <span class="token string">'https://rickandmortyapi.com/api/character/639'</span>
<span class="token function">get</span><span class="token punctuation">(</span>url<span class="token punctuation">,</span> <span class="token keyword">async</span> <span class="token punctuation">(</span><span class="token parameter">response</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span>
<span class="token keyword">const</span> data <span class="token operator">=</span> <span class="token keyword">await</span> consumers<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span>response<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span>
<span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Here we are using the awesome (and free) <a href="https://rickandmortyapi.com/">The Rick and Morty API</a>. If we run this code we should see the following output:</p>
<pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
id<span class="token operator">:</span> <span class="token number">639</span><span class="token punctuation">,</span>
name<span class="token operator">:</span> 'Uncle Nibbles'<span class="token punctuation">,</span>
status<span class="token operator">:</span> 'Dead'<span class="token punctuation">,</span>
species<span class="token operator">:</span> 'Alien'<span class="token punctuation">,</span>
type<span class="token operator">:</span> 'Soulless Puppet'<span class="token punctuation">,</span>
gender<span class="token operator">:</span> 'Male'<span class="token punctuation">,</span>
origin<span class="token operator">:</span> <span class="token punctuation">{</span>
name<span class="token operator">:</span> 'Tickets Please Guy Nightmare'<span class="token punctuation">,</span>
url<span class="token operator">:</span> 'https<span class="token operator">:</span><span class="token comment">//rickandmortyapi.com/api/location/98'</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
location<span class="token operator">:</span> <span class="token punctuation">{</span>
name<span class="token operator">:</span> 'Tickets Please Guy Nightmare'<span class="token punctuation">,</span>
url<span class="token operator">:</span> 'https<span class="token operator">:</span><span class="token comment">//rickandmortyapi.com/api/location/98'</span>
<span class="token punctuation">}</span><span class="token punctuation">,</span>
image<span class="token operator">:</span> 'https<span class="token operator">:</span><span class="token comment">//rickandmortyapi.com/api/character/avatar/639.jpeg',</span>
episode<span class="token operator">:</span> <span class="token punctuation">[</span> 'https<span class="token operator">:</span><span class="token comment">//rickandmortyapi.com/api/episode/37' ],</span>
url<span class="token operator">:</span> 'https<span class="token operator">:</span><span class="token comment">//rickandmortyapi.com/api/character/639',</span>
created<span class="token operator">:</span> '<span class="token number">2020</span><span class="token number">-08</span>-06T16<span class="token operator">:</span><span class="token number">51</span><span class="token operator">:</span><span class="token number">23</span>.084Z'
<span class="token punctuation">}</span></code></pre>
<p>It's also worth mentioning that <code>consumers.json</code> does not produce any warning, so this feature can be considered stable in Node.js 16.</p>
<h2 id="reading-a-text-from-a-readable-stream" tabindex="-1" class="title is-2">Reading a text from a Readable stream</h2>
<p>Let's discuss one more example. Let's try to consume an entire readable stream as text, which means that we will be consuming the stream assuming it's a valid UTF-8 encoded string and save the result into a string variable.</p>
<p>One simple example could be to try to read a string from the standard input in a CLI application:</p>
<pre class="language-javascript"><code class="language-javascript"><span class="token keyword">import</span> consumers <span class="token keyword">from</span> <span class="token string">'stream/consumers'</span>
<span class="token keyword">const</span> input <span class="token operator">=</span> <span class="token keyword">await</span> consumers<span class="token punctuation">.</span><span class="token function">text</span><span class="token punctuation">(</span>process<span class="token punctuation">.</span>stdin<span class="token punctuation">)</span>
console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>input<span class="token punctuation">)</span></code></pre>
<p>If we try to run the script as follows:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">cat</span> mobydick.txt <span class="token operator">|</span> <span class="token function">node</span> stdin.js</code></pre>
<p>We should see something like this in the output:</p>
<pre class="language-plain"><code class="language-plain">CHAPTER 1
Loomings.
Call me Ishmael. Some years ago--never mind how long
precisely--having little or no money in my purse, and nothing
particular to interest me on shore, I thought I would sail about a
little and see the watery part of the world. It is a way I have of
driving off the spleen and regulating the circulation. Whenever I
find myself growing grim about the mouth; whenever it is a damp,
drizzly November in my soul; whenever I find myself involuntarily
pausing before coffin warehouses, and bringing up the rear of every
funeral I meet; and especially whenever my hypos get such an upper
hand of me, that it requires a strong moral principle to prevent me
from deliberately stepping into the street, and methodically knocking
people's hats off--then, I account it high time to get to sea as soon
as I can. This is my substitute for pistol and ball. With a
philosophical flourish Cato throws himself upon his sword; I quietly
take to the ship. There is nothing surprising in this. If they but
knew it, almost all men in their degree, some time or other, cherish
very nearly the same feelings towards the ocean with me.
[...]</code></pre>
<p>Again, no warnings, so this feature is stable in Node.js v16.</p>
<h2 id="is-this-even-a-good-idea%3F" tabindex="-1" class="title is-2">Is this even a good idea?</h2>
<p>Now that you know how to use this utility, it's worth mentioning that as with all good things it should be used with moderation.</p>
<p>In fact, accumulating all the content of a stream in memory is something that should not be done lightly.</p>
<p>Streams are an abstraction that has been built into Node.js to allow developers to handle arbitrary amounts of data (even infinite streams) and process such data as soon as possible, while still keeping the memory footprint low.</p>
<p>By processing the data in chunks, we can keep the amount of memory being allocated at any given time low and have our processing logic run efficiently.</p>
<p>When we accumulate an entire stream we are effectively defeating all the advantages of Node.js streams, so this is something that is recommended only when you are absolutely certain you are dealing with small amounts of data.</p>
<h2 id="wapping-up" tabindex="-1" class="title is-2">Wapping up</h2>
<p>This is all for this article, feel free to <a href="https://twitter.com/loige">reach out to me on Twitter</a> if you found this article interesting and if you think you learned something useful.</p>
<p>If you are curious, you can also <a href="https://github.com/nodejs/node/blob/main/lib/stream/consumers.js">read the code of the <code>stream/consumers</code> module</a>, it's actually a really thin layer (less than 100 lines) and you can learn a trick or two by doing that.</p>
<p>See you in the next article!</p>