There are lots of engineering blogs that showcase product features like a tourist destination described in a travel agency brochure, but not enough stories about how the feature came to be, from starting point to the destination, like a travelogue.
Today I’m recounting the story of how I added support for V8, the Google Chrome JS/Wasm runtime, to WasmX, Kong's upcoming Wasm framework for Nginx. This will give some insights into WasmX and its tech stack, being a new contributor to a project, and the steps (and detours!) that go into adding a feature like this. Follow along as I break down the process chronologically (for reference, I've been putting an average of 20 hours/week on this).
Weeks 1 and 2: Getting started
To give a bit of context, this was my first project since rejoining Kong in late April after a year-long sabbatical. I used to work on the Kong Gateway, so WasmX (and Wasm) was uncharted territory for me, in spite of my familiarity with Nginx and my background in compilers and languages, I had a lot to read. So the first two weeks were about onboarding: getting acquainted with new projects. WasmX already supported two Wasm virtual machines: WasmTime and Wasmer, and this project consisted of adding support for a third one, so I started by getting the other two running, and I made a small PR updating WasmTime.
WasmX general architecture and the plan to support V8
I started to play with V8 on its own to learn about it, and I quickly realized the project is very much focused on being a JavaScript engine, and Wasm, while well supported, is an "extra": it does provide a C API for embedding into applications, but it doesn't match the latest spec. Also, only the bare virtual machine is provided: the Web Assembly System Interfaces (WASI), which provide the typical functions you'd expect an OS to give you, are nowhere to be found (V8 relies on the browser to provide host functionality, so it doesn't ship with any).
Weeks 3 and 4: Getting V8 into Nginx
One thing V8 does provide is a build mode that produces a stand-alone Wasm library, called libwee8 (those jokesters!), with an almost-compliant WebAssembly C API. I quickly got the simple hello-world examples running, producing a stand-alone C program that bundled it and ran some Wasm. Now it was time to do it from within Nginx.
WasmX is the framework for getting WebAssembly running within Nginx. It lives on the ngx_wasm_module repository, the module's "formal" name in Nginx parlance. My onboarding experience with the project itself had already been pretty smooth: the DEVELOPER.md document I had followed in the very first week explained how to get things up and running, with instructions for both WasmTime and Wasmer. From a user point of view, they boiled down to: download WasmTime and Wasmer, point a couple of environment variables to where the library and header files are, and run make. You end up with a nice build of nginx bundling the Wasm module, which in turn bundles the Wasm VM.
So it was my goal to add a third mode for V8. And I did it the simplest way possible: I just grepped the entire sources of ngx_wasm_module looking for “wasmtime” and/or “wasmer” and added an equivalent V8 codepath in every location. WasmX includes a src/wasm/wrt/ folder with a file for each Wasm runtime implementation, so I added a new file ngx_wrt_v8.c there. What’s curious is that all three runtimes supposedly follow the standard WebAssembly C API, however, none of them implement them exactly the same way — one of the reasons being that the standard itself is rather vague (it's a young standard, after all) — and that's why we need an implementation file for each VM. Turns out the one for V8 has some parts looking more like Wasmer's and others looking like WasmTime's, though Wasmer was the closest reference.
One thing I noticed soon enough was that a key function was missing in V8: the one for converting WebAssembly inputs in text format (known as .wat files — there's the Wasm team's sense of humor again) into binary Wasm (.wasm files). Both Wasmer and WasmTime provided this non-standard function (with different names, of course: wat2wasm and wasmtime_wat2wasm), and the WasmX test suite relies on this function in order to load its test cases.
Weeks 3 and 4: Getting V8 into Nginx (Continued)
So off I went to get an implementation of this function I could use with V8, and I found the reference one from the WebAssembly Binary Toolkit, WABT. This is, however, a C++ set of CLI programs, and it does not provide an embeddable C library… except for the Emscripten build target that uses a semi-private C API in order to transpile it into JS to use WABT from the browser(!). Amazingly, this is exactly what the Rust bindings of WABT use to build wabt-rs: they bundle and patch WABT to hijack the Emscripten build process and obtain C symbols usable from Rust.
So guess what I did: I wrote a small Rust module that takes those Rust symbols and re-exposes them as a static C library! I called it "cwabt", and made it an internal part of our V8 integration for WasmX. This was the lowest-overhead solution from a maintenance perspective, because it relies on wabt-rs’s clean and embeddable interface, and leaves all the Emscripten shenanigans for them to maintain upstream. Now I have my own wat2wasm function that I can use with V8, and it didn't take long until the first tests from the WasmX test suite were running!
I soon hit the next snag, which was foreshadowed above: the lack of WASI. But to understand where that comes into play, we need to understand how WasmX runs in practice. WasmX runs by launching a WebAssembly VM inside the Nginx process (we got that part running!) and then loading into it Wasm modules that implement the proxy-wasm API: proxy-wasm is an intended standard API with functions for proxy operations (think Kong's Plugin Development Kit: get header", "set header", and so on). In proxy-wasm terms, those modules loaded by WasmX are "filters", modules which implement a series of callbacks that are issued at various stages, not unlike Lua plugins in the Kong Gateway. You write a filter using any Wasm-supported language, such as Rust, and you compile it by linking to a proxy-wasm SDK, which contains internally a series of calls to standard proxy-wasm functions that are meant to be loaded by the host (the proxy) into the Wasm VM running inside it.
For example, you write a Rust filter that calls ctx.get_property in Rust, and then Rust's proxy-wasm-rust-sdk will turn that internally into a proxy_get_property call, which the host must implement (in our case, ngx_wasm_module) and inserted into the Wasm VM (in our case, V8). WasmX implements those proxy-wasm functions, but any Rust code compiled into a filter ends up making a few system interface calls that the Rust compiler for Wasm translates into WASI calls. WasmTime and Wasmer provide implementations for those internally, but in V8 those are nowhere to be found.
The solution, this time, was to actually implement those functions… which was easier said than done, because first of all I had to figure out which missing functions were causing the Wasm modules to fail to load, and then figure out what were the correct interfaces that the compiled filters expected. Remember when I talked about how young and vague the Wasm C API standard looked? Well, the situation for WASI is not that different — being new pieces of tech, all these standards are in flux, and their written specs often don't match their current implementations.
To be able to get to the source of truth for the latest version of the needed interfaces, I had to resort to a WebAssembly disassembler, converting the .wasm binaries from our test case filters back into .wat files and reading through the disassembly. Once I got the interfaces right — and fortunately, it was just a handful of functions that were missing to get things running — the next step was the implementations: writing C implementations of those functions inside WasmX that behave the way the Rust filters expected.
WasmX general architecture — now with V8, cwabt, and WASI
By the end of those two weeks, having implemented the V8 interfaces in ngx_wrt_v8.c, the cwabt wrapper for wat2wasm, and a minimal compliant WASI implementation, I got all tests in WasmX’s make test to pass, and opened my Draft PR! At this point, V8 is effectively running in WasmX. The journey was far from over though…
Weeks 5 and 6: Going beyond "runs on my machine"
By the end of Week 4, I had V8 running on WasmX on a "runs on my machine" basis. As we all know, that's far from being done. I opened the Draft PR anyway because I wanted to get review feedback from Thibault, WasmX’s mastermind at Kong, as soon as possible. As I received review comments, I shifted my focus away from the C and Rust code to the build scripts and Makefiles.
As I mentioned in the beginning, WasmTime and Wasmer can be used in WasmX by setting a couple of environment variables pointing to their library and header files… which are provided by the upstream projects as ready-to-use binary packages. That's not the case with V8; we need to build libwee8 ourselves. We still needed to decide how we would ship that, but I wanted to encapsulate the logic for building all needed dependencies (libwee8 and cwabt) in an automatable manner, so I wrote a shell script that did the entire job. And then, to test it beyond "runs on my machine" I tested it on "another machine": a brand-new container — and, of course, that exposed all the assumptions the script was making about my environment. Once I had it running cleanly on that container, I sent the script Thibault's way, and happily enough, it ran fine on his machine on the first try.
As we went on with the code review, I started addressing one issue I had spotted while developing, which I had sidestepped in order to continue getting the tests to pass: V8 would crash whenever I tried to run the validation function — the one that checks whether the input Wasm bytecode is valid. While developing, I just commented that out and assumed all Wasm was valid, but we needed a working validator for the final version. WABT provided one, so I considered exposing that in cwabt as I did for wat2wasm, but V8's error message gave me a hunch that it was some bad state in their C API implementation, so I looked at their C++ implementation and, sure enough, the validate function looked suspiciously different from the others.
I opened an issue and sent (what started as) a one-line patch to the V8 developers with a fix, which then extended into a pleasant back-and-forth interaction with a Chromium developer on how to implement a proper test for it in their C++ test suite, and after a week, my fix and test were merged. The most unusual part of the interaction was learning the ropes of sending patches via Gerrit, after being used to GitHub for so long. In the meantime, I also sent a PR that got merged to the Wasm C API spec repo, which extended their example code to exercise the validate function (which clearly was the only test code used by V8 to test their C API implementation).
Weeks 7 and 8: CI, the weeks of pain
At this point we had code that passes the tests, runs locally on my machine and Thibault's machine, builds the dependencies in an automated manner, and we didn't need to apply patches anymore because the V8 fix was merged upstream. It was time to get the code to run on CI.
The WasmX CI test matrix does a great job at testing the builds thoroughly: it runs the tests with all supported Wasm runtimes, both with and without Valgrind, and also tests build modes given different combinations of compilation flags, as well as a few linters for C and Rust code.
Before even attempting to run on CI, I decided to exercise all those modes locally, and sure enough, Valgrind did catch a bunch of fixes to be made. Most of them were simple — Valgrind is a lifesaver! — but as I started to run experiments both inside and outside of the test suite, I started noticing weird behaviors: depending on how I changed my C code one way or another, I'd get Valgrind happy about it in the test suite and complain about it when running manually, or vice-versa. Turns out that the Nginx directives for managing master/worker processes change the order that termination functions run, so it was pretty tricky to get Valgrind happy on every possible combination, especially when some are triggered with certain nginx.conf flags that never happen when using the Test::Nginx suite, but we got there.
We’ve all been there, haven’t we?
It was time to run it all on CI. To my surprise, the code that ran so happily on our machines plain and simple failed to compile at all on CI, even though those were the same Ubuntu versions I had tested previously! I'll spare you the gory details, but it took me a week to get a full green test run, and for half of that time I couldn't even get Nginx to build with V8 on CI. Turns out the main source of incompatibility was that V8's build system downloads its own compiler and uses by default a customized C++ standard library implementation which ends up being ABI-incompatible with the host system's; disabling their optional "custom libcxx" solved the issue.
Weeks 9 and 10: The final stretch
It was halfway through Week 9 that I finally got a reliable green build. Did I mention that an uncached build of V8 takes a whole hour to run on GitHub Actions? Yeah, until I got Nginx to link correctly that was my turnaround time. I got it somewhat sped up by using Act locally, which mimics the GitHub Actions environment on your own machine using containers (and Thibault had helpfully set it up already so it was a matter of installing it and running make act), but annoyingly I still needed to tweak the CI YAML every time I wanted to run it locally due to a small bug on Act I managed to eventually fix.
Around that time, once I got V8 to build reliably on CI, I cherry-picked a few minor fixes I had made in WasmX that were not directly related to V8 and moved them into separate easy-to-review, easy-to-merge PRs, which got merged quickly. I rebased my branch and ended Week 9 by getting the PR out of draft.
It’s a wrap!
During Week 10 I was already mainly focusing on my follow-up project, while we went through our final round of reviews, mostly concerning the CI changes. Once we finished the third and final review cycle, and a whopping 95 review comments later, the PR got merged. Keep an eye on our upcoming technical preview of WasmX, including V8 support!
Special thanks to Thibault Charbonnier for the thoughtful and attentive code reviews, and to you for reading (or skimming) all the way to the end! I hope this write-up turns out to be useful in some way, be it to spread some background info about the workings of WasmX, or just to give you encouragement in case you're toiling on a long project yourself — they do end!
Authored by: Hisham Muhammad