Manage Python dependencies with virtualenv¶
virtualenv lets each project run on its own Python version and its own set of installed libraries, so changes to one project’s dependencies (i.e., packages it depends on) don’t affect other projects.
To explain the basic mechanism, a virtualenv is really just a directory containing a mini-installation of Python. Once you ‘activate’ the virtualenv, running ‘python’ (or running most command line python scripts) doesn’t run the system’s python, but the virtualenv’s python, which also uses its own set of libraries. Once you ‘deactivate’ the virtualenv, you are back to normal Python.
You can get technical details and instructions for use in virtualenv’s docs. These should suffice for the experienced or well-caffeinated. But many beginners have a tougher time with them and don’t fully understand the uses of virtualenv and why we bother with it. So this document tries to explain the reasoning and introduce basic usage in simple language.
Examples of Dependency Problems¶
A lot of operating systems come with Python, because they need it to run some
system scripts. Since it’s already installed, lots of people just use it to
work on their projects. And often they install all their libraries where the
system Python will get them (e.g.: with
sudo pip install), which means a ton of
libraries accumulate in the same place, alongside and overlapping with whatever
the OS scripts need. Alternatively, one may use an OS which doesn’t depend on
Python at all, but still do everything from one Python installation.
Maybe nothing has gone wrong yet. But this is the foundation of many real-world confusions and disasters.
If an OS upgrade or downgrade changes the Python version to one which your program doesn’t support, the program will break. If you do a crude job of installing a different Python version to use, or install and then partly uninstall one, and it overrides the system Python, then OS scripts will break. Same for libraries.
Using virtualenv to separate OS dependencies from project dependencies lets you stop worrying about this, because you can easily change versions for one thing without affecting the other.
You have two projects, A and B. A uses version 1 of a library. B uses version 2 of the same library. The versions aren’t compatible, so they won’t both run on the same system. It isn’t very predictable when this will happen, so things may seem to be fine for a while, until cropping up at some inconvenient time in an unpredictable form.
Instead of heavyweight solutions like waiting to see these problems and then fixing code or running each on a different machine, you can just give each project its own virtualenv as a matter of course.
You’ve installed hundreds of libraries for other projects, all sharing the same environment. You write some new code which imports some of these libraries. Everything seems to be working great for a while. But when you go to share or deploy your code on a new machine, it fails because of some libraries which are missing or have the wrong versions. These errors can be painful to track down, and this might be the worst possible time to have to do that.
If you start from a clean virtualenv when you start the project, you would
install each thing needed for that project far ahead of time. Then you would
pip freeze > requirements.txt to dump a list of everything you
installed, along with its version. Then any time you needed to install in a new
place, you would create a virtualenv, change to the directory containing
cd), and use
pip install -r requirements.txt to install all
the dependencies automatically.
Even if you didn’t normally work inside virtualenvs, you might still want to test in them just to check that you have the right list of Python dependencies. This is something that tox can do for you without requiring you to directly use virtualenvs.
Make a virtualenv when you just want to play around with installing Python packages, or try to reproduce a problem with them. When you are done, you don’t have to individually uninstall things; just remove the virtualenv and none of the changes should be left behind.
A huge bonus of virtualenv is that all the files to manage can be installed
using normal user privileges.
For example, while you should never install a package you don’t trust, it only
makes matters worse to install a possibly broken package with root privileges
sudo easy_install example, assuming the name of the package is
Using virtualenv, that is not necessary to try the
package, because the virtualenv is usually in some directory owned by a user
and sudo is therefore not necessary.
virtualenv makes it simpler to work effectively in Python without root,
including if you want to deploy something as a limited user on a shared host.
People used to do all kinds of awful things before virtualenv came along. If you are curious about these or want more detail on why people use virtualenv to have a better life with Python, check out Virtualenv Workarounds.
Assuming you use pip, you can install virtualenv with:
pip install virtualenv
(It is also possible to use virtualenv as a standalone file, virtualenv.py, but I’ll leave you to look into that yourself if you need it).
If you see old instructions that say to use easy_install, please just skip them. Normally it will be easy to just use pip instead. If you are using virtualenvwrapper (not necessary but OK), the rest of this is all a little different, but basically a different spelling of the same thing.
The next step would be to create a virtualenv to use for some purpose. A virtualenv is really just a disposable directory containing some Python files. Here one is created with the arbitrary name ‘example’ - but choose your own descriptive names.
example is the name you want the directory to have). Normally you
won’t mess with the contents of this directory, and you definitely shouldn’t
put your own code there. It should always be OK to destroy and recreate the
directories that virtualenv creates.
Now, after you’ve created a virtualenv (directory), you need to ‘activate’ it each time you want to use it. The exact method depends on your OS.
In most cases, Unix-like OSes using the bash shell (or compatible) will let you use:
On Windows using cmd.exe, instead run:
The same should work in Powershell, but there is an additional issue with execution policies; see virtualenv docs on activate, in the section about using Powershell.
When you are done using a virtualenv for a bit, but don’t want to actually remove it or anything, you can type:
When you are done using a virtualenv for good, you can just remove the
directory that virtualenv created, e.g.
example. Of course, be careful
not to remove your own valuable code, which should never be inside virtualenv
There are more advanced uses of virtualenv. For one example, if you want to
create a virtualenv using a non-default python executable (e.g. an older or
newer one you have somewhere, for example at
example/python), you could
use this command:
virtualenv -p example/python
here’s an example in Windows:
virtualenv -p C:\Python27\python.exe
Another fancy use is to make a virtualenv to be able to use globally installed packages. This isn’t the best habit, but if you need it, it’s possible to create a virtualenv like this using:
virtualenv --system-site-packages example
But the default for a virtualenv is NOT to use system site packages, and that is typically a better choice.
Many people like using virtualenvwrapper to keep their virtualenvs in one centralized directory. Consider it optional and try it if you want to.
If anything in this document is unclear or seems wrong, please refer directly to the actual documentation for virtualenv.
Tips on Dependency Management¶
Some additional tips on reducing pain with Python dependencies and virtualenv.
- Try not to install libraries for your projects into a global python
installation (i.e.: try to avoid
sudo pip installor making projects depend much on system libraries) but rather in virtualenvs specific to each project, or related group of projects.
- Treat virtualenv directories as disposable. For example, don’t put them into version control. Instead, put a requirements.txt and/or a script for recreating the virtualenv into version control with the project that uses it, and document it briefly in a README file.
- It’s really useful to write at least a basic set of automated tests, so you can quickly run one command to check if anything you did has broken the project (including dependencies changes). If you don’t know what to use, I might suggest pytest. But the point is to get started early. You can’t very easily or reliably ensure everything works without automated testing, so you can’t safely mess with dependencies either.
- When you have things working, dump the installed dependencies to
pip freeze > requirements.txtso you and others will know which libraries and versions are needed to run the project. Review that file make sure it only includes things that are requirements of the project, and update and retest it as needed. Of course, the point of this is when you make a new virtualenv, you can install dependencies using
pip install -r requirements.txt.
- If you need to upgrade one or more libraries for an important project that’s currently working with older versions, at least create a separate virtualenv to install the new versions in and test thoroughly before the upgrade, leaving the existing stuff in place until you can really verify the new dependencies will work.
- If you need to make the same code work with multiple Python versions, use tox. This makes sure that your package can install and pass its tests in all the Python versions you want to support.
- If you need to develop against many versions of Python including the very old, consider developing on Ubuntu and installing old pythons from the Deadsnakes PPA (or something comparable if you know it on another distribution). Compiling Python is OK as compiling things goes, but if your interest is development then it is nicer to have someone doing this for you, or at worst supplying a script to do it.