Google boosts Python by turning it into Go

The experimental Grumpy project compiles Python apps into Go, for faster runtimes and closer interoperability

Google boosts Python by turning it into Go
Credit: Mike

Python may be getting a performance boost from a competitor: the Google Go language.

Grumpy, an experimental project from Google, transpiles Python code into Go, allowing Python programs to be compiled and run as static binaries using the Go toolchain. It's as creative an application for Go as it is a transformation of Python's use cases.

In a blog post announcing the open source release, Google stated the project stemmed from its efforts to speed up the Python-powered front end for YouTube. But Google hit an obstacle that's familiar to folks who've deployed Python in production: It's hard to get CPython -- the default Python interpreter written in C -- to scale efficiently.

"We think Grumpy has the potential to scale more gracefully than CPython for many real world workloads," writes Google.

Lose a little, gain a lot

While Grumpy compiles Python code to a static binary via the Go toolchain, that's not the chief reason for its touted speed. After all, other projects to compile Python to binaries exist, but they don't necessarily provide a speed boost because of Python's dynamic type system. Cython, for instance, provides major speedups only when the developer supplies static C-style type information.

According to Google, Grumpy speeds Python code by doing away with some CPython legacy baggage. The largest is a deliberate lack of support for C extension modules, which allows C/C++ code to be made available to Python. A huge number of libraries use C extension modules, including many supplied with Python's standard library (such as ctypes, for interfacing with C code from Python).

Because it doesn't support C extensions, Grumpy doesn't have CPython's Global Interpreter Lock, which is commonly cited as a roadblock to running Python concurrent workloads smoothly. Grumpy also uses Go's garbage collection mechanisms to manage memory under the hood, instead of CPython's.

Grumpy creates close interoperability between Python scripts and Go by allowing Go packages to be imported and used with the same syntax as Go modules. Other Python projects allow interoperation with different language environments -- pythonnet, for instance, allows to make calls between CPython and .NET/Mono runtimes. Grumpy, though, translates Python's AST to Go first, so it's better thought of as a transpiler from Python to Go, rather than an actual bridge between both language runtimes.

Missing pieces

Go and its toolchain have already been used by some third parties to develop new languages. The Have project employed Go as a transpilation target, and the Oden project experimented with Go's toolchain to create a functional language in the vein of Haskell or Lisp. Work on Oden ceased at the end of last year, although Have remains under active development.

With Grumpy, Go's toolchain helps re-implement an existing language with a broad, existing user base. It's possible Google will see good internal results with it since it's part of a custom-built use case. But success with the broader Python community depends on lifting or working around many existing roadblocks.

For one, Grumpy only supports Python's older Python 2.x syntax, though Python 2 itself is scheduled for decommission in 2020. Much of the standard library hasn't been ported, and functions like exec or eval, which execute Python code on the fly, are not yet implemented.

The biggest missing piece isn't support for C extensions themselves, but the libraries that depend on them. Without C extensions, only the most vanilla Python scripts can be compiled and run under the current interation of Grumpy. In time, it might be possible to devise replacements for them written in Go rather than C, but that's not a trivial project. Many Python libraries with big followings, like NumPy, would be completely unusable -- and such libraries are a big part of Python's draw.

[An earlier version of this article had a different and less accurate comparison between Grumpy and the "pythonnet" project. This version provides more details about the distinctions between the two projects.]