Benchmarks: DotNet vs PHP vs Go vs NodeJS

I know people love benchmarks. I know I do, I always check out some before I choose a tool for the next project. It comes with the job as I work as an architect on a great variety of projects under a lot of constraints with respect to resources / deployments / performance targets / etc. Knowing some tools always helps.

I also know that performance isn’t always a top criteria (I would always choose something that helps developers get their hands dirty fast and just develop reliably rather than aim at performance with a tool they can’t handle). But I digress.

Here’s what I wanted to do: checkout the performance for a basic REST-like API for a close to real-use setup with a couple of languages and/or their top frameworks for the purpose. That’s to say, I aim at using their HTTP server with routing, JSON encoding and some basic computational capability.

For the purpose the test, I created a set of APIs that exposed 2 endpoints: one that generates a random set of data (structure: numeric id, full name, email, ip and street name), 6000 of them, encode it as JSON. The second one should provide some static data (the version of the deployment, the configurable element count read from environment and the hostname).

What I used:

• DotNet Core with Bogus for generating data
• PHP “Micro”: this is my own framework made of ThePHPLeague Router, PHPDI container, Faker for generating data.
• PHP Symfony 5: the latest and the greatest
• NodeJS Fastify: I was told this is the top performer in the ecosystem, with Faker for generating data
• Go Echo: with its own Faker library

How I tested (aside from the API structure)

• 3 tests: one with the APIs running on my local machine with full access to resources (i7, 32 Gb RAM), one in my local machine but in a Docker environment (4 cores allocated, 4 Gb of RAM), one in a Kubernetes cluster where each API has 2 running pods behind Nginx Ingress running in Azure in compute-optimized instances.
• where applicable, logging was disabled
• minimal setup for PHP (I used the official Apache Docker container for one set of tests, as well as another run using the RoadRunner platform which is written in Go).
• otherwise, as out of the box as possible. Endpoints served by closures where possible.

Evaluation:

• using Autocannon (autocannon -c 100 -d 40 -p 10 -t 120) from my local machine (1Gbps available broadband)
• Basically, it’s a 40 seconds test with concurrency of 100 and 40 requests in pipeline.

Results:

1. Directly on local machine: https://imgur.com/Yzhv6E2
2. Local machine in Docker: https://imgur.com/72N9y72
3. In K8s cluster: https://imgur.com/zRFGRO5

Notes:

• I know the test isn’t really fair for interpreted languages (like PHP). I will add a more “real” scenario, perhaps fetching something from a DB
• In my local environment I wanted to also use Python using Stilette framework with GUnicorn. I was shocked to see how poorly it did on the dynamic endpoint with data generation, perhaps I did something wrong as I’m not much of a Python expert.
• For the first test, I didn’t setup Apache with PHP locally, I only performed tests with RoadRunner for both PHP apps (Symfony and ‘frameworkless’).
• The results aren’t any average, I took an actual result set that was slightly above-average. In each case the tests were ran at least 10 times each for each of the endpoints.
• The Docker on my local machine test used the exact same containers as those that ran in K8s. For each build the code/binary was copied to the container.

Observations:

• I’m puzzled by Fastify. I haven no explanation why when using Docker with my local machine it only managed 104 (always ~100) requests on the dynamic endpoint. In all other tests it did very much close to my expectations
• On the PHP side, the ‘regular’ setup (aka, with Apache) saw the ‘frameworkless’ API usually under Symfony, though a few months ago I did a similar test with Symfony 4 and I have to say that Symfony 5 is way better
• Roadrunner is a MAJOR boost for PHP. It’s the power of Go and the popularity of PHP all-in-one. It feels like PHP can compete in some cases!
• Roadrunner also levelled the playing field between the ‘frameworkless’ setup and Symfony 5. Honestly, in this case, I can’t see why I’d choose Symfony and all the yml nonsense over defining my dependencies in plain PHP with PHPDI!
• Go’s performance just explodes when it has more system threads to abuse!
• I don’t know what happened to DotNet when used in Docker container (ran under the official AspNetCore image) but it was insane on the static endpoint! I ran that test maybe 20 times and I can say that 700k requests in container over 40s wasn’t the top result (it also achieve ~850k one time). It felt strange that it outperformed Go.
• Fastify also outperformed Go in the same scenario. I didn’t expect that, since in the Docker environment it still had 2 vCPUs available and the GOMAXPROCS was set to 50 for good measure (I tried with settings between 2 and 50, such as 5, 10, 20, 25 and 35 after landing on 50, with little practical difference)
• Node and DotNet seem to fare better with a single but better container environment, whereas Go made better use of the limited resources in K8s with some horizontal scaling in the mix (each pod ran a single container with 250m vCPU and up to 512Mb RAM)
• With RoadRunner, the PHP ‘frameworkless’ (micro) also resulted in some errors on the RoadRunner side — so I made a bug report for RoadRunner.

The bottom line

• I do find the test relevant, despite it being not fair to interpreted languages (life is not fair, the tool needs to do the job). Generating random data tests the computational capabilities of the platform. It’s a heavy task.
• I looked into platforms in a way that should be similar to how developers would approach a task. Sure, you may want to build things from scratch but generally you’d pick a library / router or a framework depending on how much manual work you think it’s worth doing.
• In Go world: I looked into lightweight things as well. The wonder library fasthttp + phi router is a great performer (in local test it did constant > 4 million requests for static endpoint with average of 1ms latency, but no difference for dynamic endpoint). But you need json encoder and logging to match (please, no logrus or stdlib json marshalling!)
• Today I wouldn’t use PHP without Roadrunner. IMHO, PHP has too many moving parts and tools to configure in order to a) be production and b) provide a high performance production environment. Apache/fpm, phpunit, xdebug, etc — a wide array of third party tools *required* to deploy/develop something. A platform should provide them!

I may continue to add some more frameworks / languages there. I know I’d like to give Stilette another short, definitely Java (though I don’t like it at all), Rust for sure and time-permitting PHP with Ubiquity framework.

Cheers!