How python virtual environments actually work

In this post I talk about how virtual environemts work in a more in depth view, some basic stuff that a python developer needs to know and remember. Let’s start with a couple of questions.

Where are python packages installed?

Open up a python shell and run:

import site site.getsitepackages()

This will print a list of strings which represent absolute system paths, these locations store system level python packages that are available to all projects, and this is without using a virtual environment.
If you install a python package with pip install package-name, the default behaviour for python is to save this new package's files under lib/site-packages relative to the location of the python executable's parent directory, and again this is for system level packages.
Of course different projects need a differnet set of packages (with different versions). this is where virtual environmetns play a role by simply changing the site-packages folder path when a new virtual environment is created, which will be different for each project.

What does creating a virtual environment do?

Creation

Creating a virtual environment simply adds a new directory named by the virtual environment’s name (let’s call it myenv) to a specified place that you choose, that directory will contain a python executable under myenv/bin/ (which is actually a symbolic link to the system's python executable).
Note that python saves all installed packages to ../lib/site-packages (going up a directory -> /lib/site-packages/), and that's relative to the path of the python's executable.

Activation

Activating the virtual environment will prepend the path to the virtual environment’s python executable directory (/var/www/projectX/myenv/bin) to the $PATH environment variable. Therefore, when you run python at the console, the one that's on myenv/bin/python will get executed.
Now, when installing new packages, these will get stored under myenv/lib/site-packages/. and will only be used by the target project, making the project isolated with it's own packages (with the required versions).

Creating Virtual environments for different versions of python

Python 2.x

You need first to install the virtualenv package using pip, pip comes by deafult with python 2:

pip install virtualenv

then navigate to where you need to create the virtual environment and:

virtualenv env

env is the virtual environemt's name, remember it's just a folder.

python 3.x

Python 3 comes with a preinstalled tool called venv, which is used to create the virtual environment:

python3 -m venv env

If you create the virtual environmet inside the project’s repository, make sure to ignore the entire folder by adding /env to the .gitignore file if you're using git for example. you can just track the names and versions of the used packages if you want to share or deploy the project.

Inspecting the virtual environment folder

Creating a virtual environmet folder will generate the following structure:

├── bin │ ├── activate │ ├── python -> python3.5 ├── include ├── lib │ └── python3.5 │ └── site-packages └── ..

What you actually need to know about are these:

• bin/activate: A script where you will often run to activate the virtual environment, this prepends the absolute path to the virtual environment’s python executable to the $PATH system variable.
• lib/site-packages: Where all python project dependencies get stored, meaning if you activate this environment and install any package with pip, it will get stored here.

Standard usage of virtualenv

Summing up what we mensioned above, you can use virtualenv in a simple and standard way like so (I do it like this in most cases):

1. Make sure python and pip are installed (see above).
2. Navigate to where you need to install the virtual environment (this could be the project’s repository or some other common place like: ~/.venvs).
3. Create a virtual environment for the spicific python version.
4. Activate it, then install your packages and include them in your project.

There are more tools that are built on this virtualenv package, these tools make it a bit easier to interact with the process of managing, creating and activating virtual environments. some of them are:

1. virtualenvwrapper: Provides commands to interact with virtual environments like mkvirtualenv to create a new environment, cdvirtualenv, activate and deactivate. It needs to be setup first.
2. pipenv: It’s more like an extention of both pip and virtualenv, this automatically creates a virtual environment with a hashed name and set’s a refrence to it’s name in a file that you include in your project. I will make a separate post for this tool.