How fast is your programming language? New contest and benchmarks spark debate • DEVCLASS

Java programmers are being pressed to post the fastest possible solution to a “one billion row challenge,” prompting both Java and non-Java solutions along with debate about programming language performance. At the same time a developer has posted new benchmarks comparing the performance of popular languages.

Software engineer Gunnar Morling (formerly at Red Hat, now working at Decodable) has posted the new year contest to Java developers, the goal being to “explore the benefits of modern Java and find out how far you can push this platform.” The deadline for submissions is 31 January.

The problem is not complicated. Developers are asked to “write a Java program for retrieving temperature measurement values from a text file and calculating the min, mean, and max temperature per weather station. There’s just one caveat: the file has 1,000,000,000 rows.”

At the time of writing the leading solution runs in 14.848 seconds using the GraalVM for JDK Community Edition 21, a JDK implementation that uses ahead of time (AOT) compilation for fast start-up. Solutions are evaluated on a cloud VM with 8 dedicated vCPUs and 32GB RAM, running on an AMD processor, and with Fedora 39 Linux.

How fast though is Java versus other languages? A show and tell page has submissions in Rust, C#, Go, Python, PostgreSQL, Python, C, C++, and more. These are hard to compare with one another since they have been run on different hardware, but there are some impressive results, including one under 5 seconds done with C on an AMD laptop, and a C# solution that runs in 5.3 seconds on a Core i5-12500 with 6 cores.

The matter of evaluating performance across different programming languages and runtimes is vexed though, as it is rarely like with like. Many of the solutions in this case have comments from other developers about how the algorithm might be further optimized, so it is code quality as well as raw performance that is being compared.

Another developer who goes by the name attractivechaos has posted benchmarks for 25 languages and runtimes with the aim of focusing “on the performance of the language itself without library functions.” The developer is “mostly a C programmer” and confesses that some implementations may be sub-optimal. Nevertheless it is an illuminating benchmark, including heavy use of nested loops, integer operations, matrix multiplication and array access, to solve problems including a 15-queens problem: a classic computational exercise to calculate how to place n queens on an n x n chessboard such that no queen threatens another.

Programming language performance compared in new benchmark. Matmul is short for matrix multiplication, and bedcov finds overlaps between large arrays

Results for the 15-queen problem show clear groupings, with system languages like C and Rust more than fifty times faster than interpreted languages Python and Perl. JIT (just-in-time) languages including PHP and Ruby come in between.

It is notable though that well optimized languages with AOT compilation can perform well. The 15-queen problem languages, including C#, Go, Swift and Zig have performance close to that of C. A Sudoku-solving test opens up bigger gaps, with C# and Java taking twice as long as C and Rust, but still many times faster than Python or Ruby.

We also noticed that although the four JavaScript runtimes tested (Bun, Deno, Node.js and the developer’s own k8) perform closely, Bun is consistently the fastest, and Deno the slowest. Another key point: on this kind of test, PyPy is many times quicker than CPython.

“Python remains one of the two most used scripting languages despite its poor performance,” the developer reflects. “Will there be a language to displace Python in the next decade? I am not optimistic.”

Source: devclass.com