Python Hands-on

Virtualenv and venv: Python virtual environments explained

Take advantage of virtual environments in Python 2 and Python 3 to manage conflicts between Python projects

Virtualenv and venv: Python virtual environments explained
Thinkstock

Python Hands-on

Show More

Of all the reasons Python is a hit with developers, one of the biggest is its broad and ever-expanding selection of third-party packages. Convenient toolkits for everything from ingesting and formatting data to high-speed math and machine learning are just an import or pip install away.

But what happens when those packages don’t play nice with each other? What do you do when different Python projects need competing or incompatible versions of the same add-ons? That’s where Python virtual environments come into play.

You can create and work with virtual environments in both Python 2 and Python 3, though the tools are different. Virtualenv is the tool of choice for Python 2, while venv handles the task in Python 3. 

What are Python virtual environments?

A virtual environment is a way to have multiple, parallel instances of the Python interpreter, each with different package sets and different configurations. Each virtual environment contains a discrete copy of the Python interpreter, including copies of its support utilities.

The packages installed in each virtual environment are seen only in that virtual environment and no other. Even large, complex packages with platform-dependent binaries can be corralled off from each other in virtual environments.

There are a few common use cases for a virtual environment:

  1. You’re developing multiple projects that depend on different versions of the same packages, or you have a project that must be isolated from certain packages because of a namespace collision. This is the most standard use case.
  2. You’re working in a Python environment where you can’t modify the site-packages directory. This may be because you’re working in a highly controlled environment, such as managed hosting, or on a server where the choice of interpreter (or packages used in it) can’t be changed because of production requirements.
  3. You want to experiment with a specific combination of packages under highly controlled circumstances, for instance to test cross-compatibility or backward compatibility.
  4. You want to run a “baseline” version of the Python interpreter on a system with no third-party packages, and only install third-party packages for each individual project as needed.

Nothing says you can’t simply unpack a Python library into a subfolder of a project and use it that way. Likewise, you could download a standalone copy of the Python interpreter, unpack it into a folder, and use it to run scripts and packages devoted to it.

But managing such cobbled-together projects soon becomes difficult. It only seems easier to do that at first. Working with packages that have binary components, or that rely on elaborate third-party dependencies, can be a nightmare. The best long-term solution is to use Python’s native mechanisms for creating and working with virtual environments.

Virtual environments in Python 3

Virtualenv has proven indispensible to countless Python developers, but it is not part of Python’s standard library. Python 3 has native tooling for virtual environments that makes the whole process quite simple.

Create the virtual environment

To create a virtual environment in a given directory, type:

python3 -m venv /path/to/directory

(Note that you can just use python instead of python3 if your system recognizes python as the default Python 3 interpreter.)

The whole process of setting up the virtual environment may take a minute or two. When it’s finished, you should have a directory with a few subdirectories in it. The most important subdirectory is bin on Unix or Scripts on Windows, which is where you’ll find the copy of the Python interpreter for the virtual environment along with its utilities.

Note that because each virtual environment contains its own copy of the Python interpreter, it can be fairly large. On Windows and Linux alike, a Python 3.6 virtual environment will consume about 23 MB of disk space.

Activate the virtual environment

Before you can use this virtual environment, you need to explicitly activate it. Activation makes the virtual environment the default Python interpreter for the duration of the session.

You’ll need to use different syntax for activating the virtual environment depending on which operating system and command shell you’re using.

  • On Unix or MacOS, using the bash shell: source /path/to/venv/bin/activate
  • On Unix or MacOS, using the csh shell: source /path/to/venv/bin/activate.csh
  • On Unix or MacOS, using the fish shell: source /path/to/venv/bin/activate.fish
  • On Windows using the Command Prompt: path\to\venv\Scripts\activate.bat
  • On Windows using PowerShell: path\to\venv\Scripts\Activate.ps1

Note that the activated environment only works for the context it was activated in. For instance, if you launch two instances of PowerShell, A and B, and you only activate the virtual environment in instance A, that environment will only apply to A. It wouldn’t apply anywhere else.

Configure and use the virtual environment

Once you’ve activated the new virtual environment, you can use the pip package manager to add and change packages for it. You’ll find pip in the Scripts subdirectory of the virtual environment on Windows, and in the bin subdirectory on Unix OSes.

If you’re already familiar with the way pip works, you’re set. It should be just the same in a virtual environment. Just make sure you’re using the instance of pip that manages packages for the virtual environment in the context where it was activated—e.g., the bash session or Windows CLI/PowerShell session. If you want to verify that you’re using the right pip and the right virtual environment, type pip -V and check that the path it displays points to a subdirectory of your virtual environment.

To use the virtual environment you created to run Python scripts, simply invoke Python from the command line in the context where you activated it.

Deactivating the virtual environment

When you’re done using the virtual environment, you can just terminate the session where you were using it. If you want to continue to work in the environment but with the default Python interpreter instead, type deactivate at the prompt. Windows users on the Command Prompt need to run deactivate.bat from the Scripts subdirectory, but Unix users and Windows users running PowerShell can simply type deactivate in any directory.

Removing the virtual environment

Virtual environments are self-contained. When you no longer need the virtual environment, you can just delete its directory.

Virtual environments in Python 2

With Python 2, virtual environments aren’t a native feature of the language. Instead, you need to install third-party libraries to create and manage virtual environments.

The most popular and widely used of these projects is virtualenv, which handles creating the directory structure and copying the needed files into a virtual environment. To install virtualenv, just use pip install virtualenv. To create a virtual environment directory with it, type virtualenv /path/to/directory. Activating and deactivating the virtual environment works the same way as it does for virtual environments in Python 3 (see above).

Using virtual environments with Jupyter notebooks

If you’re using Jupyter notebooks (aka IPython notebooks), and you already have Jupyter installed systemwide, create your virtual environment and activate it. Then, from your virtual environment directory, run pip install ipykernel to add the needed components for IPython. Finally, run ipython kernel install —user —name=<project_name>, where project_name is a name you want to associate with that particular project. From there you should be able to launch Jupyter and switch to the IPython kernel you installed inside the virtual environment.

Copyright © 2017 IDG Communications, Inc.