Python 3.6 is packed with goodness

More async support, more type-hinting power, more speed, and more efficient memory usage top a long list of useful additions to Python

Debuting a little more than a year ago, Python 3.5 hinted at how the language could become faster and more powerful without sacrificing the convenience and ease of use that characterize Python -- without forcing everyone to toss out existing Python code and start over.

Python 3.6 picks up where many of those improvements left off and nudges them into new realms. Python 3.5 added syntax used by static type checking tools to ensure software quality; Python 3.6 expands on that idea, which could eventually lead to high-speed statically compiled Python programs. Python 3.5 gave us options to write asynchronous functions; Python 3.6 bolsters them. But the biggest changes in Python 3.6 lie under the hood, and they open up possibilities that didn't exist before.

Below I’ve outlined the most important stuff that's new in Python 3.6: what's been added, why it's useful, and what you can do to take advantage of it in your own applications.

Async in more places

What: With Python 3.6, the async and await keywords in Python can now be used inside generators and comprehensions, instead of simply to decorate functions.

Why: Processing that normally involves long delays, such as reading from a network socket, can now be done without holding up the rest of the app -- and via syntax built into the language.

The nitty-gritty: Earlier versions of Python allowed developers to build functions that could run asynchronously, without blocking each other, but not in a particularly elegant manner. Python 3.5 introduced asynchronous behavior to Python's syntax through the async and await keywords. Now Python 3.6 has extended async and await to work in two other major parts of Python: generators and comprehensions.

Generators are functions that yield values iteratively, like a list that's computed on demand. When paired with async syntax, generators can be used to concisely create functions that yield values when external conditions are met. Python's documentation provides this as an example:

   """Yield numbers from 0 to *to* every *delay* seconds."""
   for i in range(to):
       await asyncio.sleep(delay)

Comprehensions let you assemble data structures like lists or dictionaries using a concise syntax. The async/await syntax can now be used in the context of a comprehension:

result = [i async for i in aiter()]

This would append the output of the function aiter() to the list result as values are yielded from the asynchronous function. With this it's possible to use asynchronous syntax within functions like Python lambdas or simply as a convenient option to condense the syntax of a list declaration.

Other major changes in this vein elevate the status of asyncio, a Python library used to perform asynchronous I/O such as reading from network sockets. Bearing new and refined methods, asyncio becomes a full-blown member of Python's standard library; before it was included provisionally while its API was being nailed down.

Note that the async/await keywords aren’t really a replacement for asyncio. As core Python developer Brett Cannon has pointed outasync/await can be used to create all kinds of asynchronous behaviors; asyncio is only one of the many ways async/await can be put to use.

Speed and memory usage improvements

What: Python 3.6 introduces a number of improvements that render CPython, the default implementation of the Python runtime, both faster and more economical with memory -- without breaking backward compatibility.

Why: Python has traditionally been slower than C by an order of magnitude or more, but attempts to dramatically speed up Python tend to come at the expense of compatibility. The increased quickness in Python 3.6 aren't warp-speed improvements, but they help. Any changes that help Python use a little less memory are welcome as well, especially when dealing with applications that create large data structures.

The best part is that developers can make use of these improvements in speed and memory efficiency with little or no reworking of existing Python code. They come for free.

The nitty-gritty: The performance optimizations in Python 3.6 come through bytecode parsing improvements, faster function calling, and more efficient dictionaries.

Bytecode speedups. Python scripts are compiled to cached bytecode to speed execution. As a result, many optimizations of Python have focused on making the bytecode faster to parse. A slew of improvements has landed in Python 3.6 around bytecode, such as using a 16-bit format for bytecode, providing more opcodes for common use cases (essentially, condensing several instructions into one), simplifying some opcodes to remove less-used ones, and simplifying the interpreter loop that handles bytecodes.

The improvements aren't dramatic -- a fraction of a percent here and there across various benchmarks. But those slivers of improvement add up over time, and they come for free. That is, the only effort the developer has to make is to migrate to the newest version of the language runtime.

The one place where these changes break compatibility is with anything that works directly with Python bytecode. That said, Python has never guaranteed bytecode compatibility across versions, so this is well within the scope of what seasoned Python devs can expect from any major revision of the language.

Faster function calling. Core Python developer Victor Stinner has made speeding up CPython a big part of his work with the language. One of his projects, which landed in Python 3.6, amounts to a faster option to perform function calls in the language.

Tests show the improvement isn't enormous, but they're measurable and demonstrably faster than Python 2.x, so it works to defray a common criticism that the "new" Python is less performant than the "old" one. Such criticisms have lent legitimacy to the idea that Python 2.x is worth keeping alive. (At least one project is working to keep Python 2.x current in the face of eventually losing official support.)

If you use Cython, the Python library that compiles Python code to C, note that it has been updated to take advantage of the this new calling convention.

More efficient dictionaries. Python dictionaries, or the dict type, are an efficient method to store information by way of key/value pairs. The folks at the PyPy project, the just-in-time compiled version of Python that many look to as a quick speed-up for the language, created their own implementation of dict that was faster and used up to 25 percent less memory. It was an impressive achievement -- so impressive that it got ported back into the mainline version of the language.

Pluggable support for JITs, tracers, and debuggers

What: The official term for this addition is a “frame evaluation API,” which lets third-party applications hook into CPython and intercept code evaluation.

Why: With this API, one could create JIT (just-in-time) compilers, tracers, debuggers, or other low-level utilities that could plug straight into CPython and accelerate its performance in a more modular manner.

The nitty-gritty: The frame evaluation API lets you hand off the execution of code to an external method when code is being evaluated. The code could be recompiled by a JIT to machine code or inspected in ways that the current CPython interpreter doesn't allow.

Microsoft co-proposed the idea, and in fact an existing Microsoft project, Pyjion, allows Python code to be JIT compiled via the CoreCLR execution engine.

In theory all of the existing JITs that exist for Python could be enhanced by this. Aside from PyPy (the best-known JIT for Python) and the Dropbox project Pyston, one project that could take advantage of this is Numba, which is used for JIT compiling individual math-heavy functions via the LLVM compiler.

Wait, there’s more

Of course that’s not all. Python 3.6 also provides support for DTrace and SystemTap, brings a secrets module to the standard library, introduces new string and number formats, and adds type annotations for variables. It also gives us easier methods to customize the creation of subclasses.

CPython can now use the DTrace and SystemTap monitoring tools (used in MacOS and Linux, respectively) to provide debugging hooks for certain operations. If you're trying to debug a really low-level problem with Python -- for instance, one involving a C extension module -- these provide highly advanced methods for such work.

Developers are often left to their own devices to generate cryptographically strong values for logins, authentication tokens, or other such functions. Reinventing this wheel is a bad idea, as many people don't know how to write robust code for such items. The new secrets module saves you the heavy lifting and provides utilities for generating tokens, including tokens that are safe to use in URLs.

For those developing APIs or web apps with logins, it's a must-have and a must-use. Best of all, it's not "locked" to Python 3.6; you can copy it out and use it in an earlier version of Python if need be.

A new kind of string format, the "f-string," allows strings to include expressions evaluated at runtime. A string like f'Four={2+2}' would yield Four=4. Think of this as a more concise, less boilerplate method to use the format() protocol for strings.

You can also use the underscore character as a separator for numeric literals, to make them easier to read (for example, 1_000_000 for 1 million decimal). If you hard-code a lot of number literals, this can help keep your eyes from crossing, although like f-strings it isn't backward compatible.

Python officially added type hinting, as used by linting tools like Mypy to perform pre-emptive type checking on applications. Python 3.6 expands on the idea by allowing annotations for class and instance variables, so the values assigned to items later on can be type-checked as well.

A new method, __init_subclass__, is now called on a given class whenever subclasses are created from it. This makes it possible to automatically change the default behavior of subclasses without having to tediously do so by hand for each subclass.

A minor but useful internal improvement: Python 3.6 preserves the order of elements for more structures. Keyword arguments passed to a function, attribute definitions in a class, and dictionaries all preserve the order of elements as they were defined. Serialization APIs or test suites can both benefit from this, as it makes the order of the output from serialized objects more predictable.

Where from here?

The improvements in Python 3.6 fall into roughly three categories. First are the "free" improvements -- the minor performance upticks and other goodies that you get by switching to the new version of the interpreter. It's least effort, but also the least payoff.

Beyond that are the changes that require reworking existing Python applications, but come with a reward -- using async/await, for instance. There is a fair amount of labor involved, but it’s strictly optional. There's no penalty for not using those features, except you won't be able to take advantage of what they provide. Existing apps will continue to run as-is.

Finally, there's the unexplored territory, as seen most prominently in the new frame evaluation API. The possibilities for this haven't even been touched yet, so it will be exciting to see what people do with it once Python 3.6 finds its way into more hands.

Copyright © 2016 IDG Communications, Inc.