Manage Python dependencies with virtualenv¶

virtualenv lets each project run on its own Python version and its own set of installed libraries, so changes to one project’s dependencies (i.e., packages it depends on) don’t affect other projects.

To explain the basic mechanism, a virtualenv is really just a directory containing a mini-installation of Python. Once you ‘activate’ the virtualenv, running ‘python’ (or running most command line python scripts) doesn’t run the system’s python, but the virtualenv’s python, which also uses its own set of libraries. Once you ‘deactivate’ the virtualenv, you are back to normal Python.

You can get technical details and instructions for use in virtualenv’s docs. These should suffice for the experienced or well-caffeinated. But many beginners have a tougher time with them and don’t fully understand the uses of virtualenv and why we bother with it. So this document tries to explain the reasoning and introduce basic usage in simple language.

If you aren’t interested in detailed reasons, you can go directly to the section titled Basic Usage or directly to virtualenv’s docs.

Examples of Dependency Problems¶

A lot of operating systems come with Python, because they need it to run some system scripts. Since it’s already installed, lots of people just use it to work on their projects. And often they install all their libraries where the system Python will get them (e.g.: with sudo pip install), which means a ton of libraries accumulate in the same place, alongside and overlapping with whatever the OS scripts need. Alternatively, one may use an OS which doesn’t depend on Python at all, but still do everything from one Python installation.

Maybe nothing has gone wrong yet. But this is the foundation of many real-world confusions and disasters.

Upgrades¶

If an OS upgrade or downgrade changes the Python version to one which your program doesn’t support, the program will break. If you do a crude job of installing a different Python version to use, or install and then partly uninstall one, and it overrides the system Python, then OS scripts will break. Same for libraries.

Using virtualenv to separate OS dependencies from project dependencies lets you stop worrying about this, because you can easily change versions for one thing without affecting the other.

Coexisting Projects¶

You have two projects, A and B. A uses version 1 of a library. B uses version 2 of the same library. The versions aren’t compatible, so they won’t both run on the same system. It isn’t very predictable when this will happen, so things may seem to be fine for a while, until cropping up at some inconvenient time in an unpredictable form.

Instead of heavyweight solutions like waiting to see these problems and then fixing code or running each on a different machine, you can just give each project its own virtualenv as a matter of course.

Implicit Dependencies¶

You’ve installed hundreds of libraries for other projects, all sharing the same environment. You write some new code which imports some of these libraries. Everything seems to be working great for a while. But when you go to share or deploy your code on a new machine, it fails because of some libraries which are missing or have the wrong versions. These errors can be painful to track down, and this might be the worst possible time to have to do that.

If you start from a clean virtualenv when you start the project, you would install each thing needed for that project far ahead of time. Then you would use pip freeze > requirements.txt to dump a list of everything you installed, along with its version. Then any time you needed to install in a new place, you would create a virtualenv, change to the directory containing requirements.txt (using cd), and use pip install -r requirements.txt to install all the dependencies automatically.

Even if you didn’t normally work inside virtualenvs, you might still want to test in them just to check that you have the right list of Python dependencies. This is something that tox can do for you without requiring you to directly use virtualenvs.

Playing Around¶

Make a virtualenv when you just want to play around with installing Python packages, or try to reproduce a problem with them. When you are done, you don’t have to individually uninstall things; just remove the virtualenv and none of the changes should be left behind.

A huge bonus of virtualenv is that all the files to manage can be installed using normal user privileges. For example, while you should never install a package you don’t trust, it only makes matters worse to install a possibly broken package with root privileges (as with sudo easy_install example, assuming the name of the package is example). Using virtualenv, that is not necessary to try the package, because the virtualenv is usually in some directory owned by a user and sudo is therefore not necessary. virtualenv makes it simpler to work effectively in Python without root, including if you want to deploy something as a limited user on a shared host.

People used to do all kinds of awful things before virtualenv came along. If you are curious about these or want more detail on why people use virtualenv to have a better life with Python, check out Virtualenv Workarounds.

Basic Usage¶

Assuming you use pip, you can install virtualenv with:

pip install virtualenv

(It is also possible to use virtualenv as a standalone file, virtualenv.py, but I’ll leave you to look into that yourself if you need it).

If you see old instructions that say to use easy_install, please just skip them. Normally it will be easy to just use pip instead. If you are using virtualenvwrapper (not necessary but OK), the rest of this is all a little different, but basically a different spelling of the same thing.

The next step would be to create a virtualenv to use for some purpose. A virtualenv is really just a disposable directory containing some Python files. Here one is created with the arbitrary name ‘example’ - but choose your own descriptive names.

virtualenv example

(where example is the name you want the directory to have). Normally you won’t mess with the contents of this directory, and you definitely shouldn’t put your own code there. It should always be OK to destroy and recreate the directories that virtualenv creates.

Now, after you’ve created a virtualenv (directory), you need to ‘activate’ it each time you want to use it. The exact method depends on your OS.

In most cases, Unix-like OSes using the bash shell (or compatible) will let you use:

source example/bin/activate

On Windows using cmd.exe, instead run:

example\Scripts\activate

The same should work in Powershell, but there is an additional issue with execution policies; see virtualenv docs on activate, in the section about using Powershell.

When you are done using a virtualenv for a bit, but don’t want to actually remove it or anything, you can type:

deactivate

When you are done using a virtualenv for good, you can just remove the directory that virtualenv created, e.g. example. Of course, be careful not to remove your own valuable code, which should never be inside virtualenv directories.

There are more advanced uses of virtualenv. For one example, if you want to create a virtualenv using a non-default python executable (e.g. an older or newer one you have somewhere, for example at example/python), you could use this command:

virtualenv -p example/python

here’s an example in Windows:

virtualenv -p C:\Python27\python.exe

Another fancy use is to make a virtualenv to be able to use globally installed packages. This isn’t the best habit, but if you need it, it’s possible to create a virtualenv like this using:

virtualenv --system-site-packages example

But the default for a virtualenv is NOT to use system site packages, and that is typically a better choice.

Many people like using virtualenvwrapper to keep their virtualenvs in one centralized directory. Consider it optional and try it if you want to.

If anything in this document is unclear or seems wrong, please refer directly to the actual documentation for virtualenv.

Tips on Dependency Management¶

Some additional tips on reducing pain with Python dependencies and virtualenv.

Try not to install libraries for your projects into a global python installation (i.e.: try to avoid sudo pip install or making projects depend much on system libraries) but rather in virtualenvs specific to each project, or related group of projects.
Treat virtualenv directories as disposable. For example, don’t put them into version control. Instead, put a requirements.txt and/or a script for recreating the virtualenv into version control with the project that uses it, and document it briefly in a README file.
It’s really useful to write at least a basic set of automated tests, so you can quickly run one command to check if anything you did has broken the project (including dependencies changes). If you don’t know what to use, I might suggest pytest. But the point is to get started early. You can’t very easily or reliably ensure everything works without automated testing, so you can’t safely mess with dependencies either.
When you have things working, dump the installed dependencies to requirements.txt using pip freeze > requirements.txt so you and others will know which libraries and versions are needed to run the project. Review that file make sure it only includes things that are requirements of the project, and update and retest it as needed. Of course, the point of this is when you make a new virtualenv, you can install dependencies using pip install -r requirements.txt.
If you need to upgrade one or more libraries for an important project that’s currently working with older versions, at least create a separate virtualenv to install the new versions in and test thoroughly before the upgrade, leaving the existing stuff in place until you can really verify the new dependencies will work.
If you need to make the same code work with multiple Python versions, use tox. This makes sure that your package can install and pass its tests in all the Python versions you want to support.
If you need to develop against many versions of Python including the very old, consider developing on Ubuntu and installing old pythons from the Deadsnakes PPA (or something comparable if you know it on another distribution). Compiling Python is OK as compiling things goes, but if your interest is development then it is nicer to have someone doing this for you, or at worst supplying a script to do it.