Home Explore Blog CI



nixpkgs

10th chunk of `doc/languages-frameworks/python.section.md`
a82b91d78af28cd1fdeb70d5e361b8d9b6faf8c815a53d520000000100000faa
$ nix-shell -p 'python312.withPackages(ps: with ps; [ numpy toolz ])'
```

By default `nix-shell` will start a `bash` session with this interpreter in our
`PATH`, so if we then run:

```Python console
[nix-shell:~/src/nixpkgs]$ python3
Python 3.12.4 (main, Jun  6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy; import toolz
```

Note that no other modules are in scope, even if they were imperatively
installed into our user environment as a dependency of a Python application:

```Python console
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
```

We can add as many additional modules onto the `nix-shell` as we need, and we
will still get 1 wrapped Python interpreter. We can start the interpreter
directly like so:

```sh
$ nix-shell -p "python312.withPackages (ps: with ps; [ numpy toolz requests ])" --run python3
Python 3.12.4 (main, Jun  6 2024, 18:26:44) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>
```

Notice that this time it built a new Python environment, which now includes
`requests`. Building an environment just creates wrapper scripts that expose the
selected dependencies to the interpreter while re-using the actual modules. This
means if any other env has installed `requests` or `numpy` in a different
context, we don't need to recompile them -- we just recompile the wrapper script
that sets up an interpreter pointing to them. This matters much more for "big"
modules like `pytorch` or `tensorflow`.

Module names usually match their names on [pypi.org](https://pypi.org/), but
normalized according to PEP 503/508. (e.g. Foo__Bar.baz -> foo-bar-baz)
You can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html)
to find them as well (along with non-python packages).

At this point we can create throwaway experimental Python environments with
arbitrary dependencies. This is a good way to get a feel for how the Python
interpreter and dependencies work in Nix and NixOS, but to do some actual
development, we'll want to make it a bit more persistent.

##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang}

Sometimes, we have a script whose header looks like this:

```python
#!/usr/bin/env python3
import numpy as np
a = np.array([1,2])
b = np.array([3,4])
print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
```

Executing this script requires a `python3` that has `numpy`. Using what we learned
in the previous section, we could startup a shell and just run it like so:

```ShellSession
$ nix-shell -p 'python312.withPackages (ps: with ps; [ numpy ])' --run 'python3 foo.py'
The dot product of [1 2] and [3 4] is: 11
```

But if we maintain the script ourselves, and if there are more dependencies, it
may be nice to encode those dependencies in source to make the script re-usable
without that bit of knowledge. That can be done by using `nix-shell` as a
[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so:

```python
#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])"
import numpy as np
a = np.array([1,2])
b = np.array([3,4])
print(f"The dot product of {a} and {b} is: {np.dot(a, b)}")
```

Then we execute it, without requiring any environment setup at all!

```sh
$ ./foo.py
The dot product of [1 2] and [3 4] is: 11
```

If the dependencies are not available on the host where `foo.py` is executed, it
will build or download them from a Nix binary cache prior to starting up, prior
that it is executed on a machine with a multi-user nix installation.

This provides a way to ship a self bootstrapping Python script, akin to a
statically linked binary, where it can be run on any machine (provided nix is
installed) without having to assume that `numpy` is installed globally on the

Title: Creating and Running Python Environments with Nix-Shell
Summary
This section explains how Nix-Shell is used to create isolated Python environments with specific packages. It highlights that each environment only includes the specified packages, and demonstrates adding 'requests' to the environment. It also explains the module naming convention that mirrors pypi.org. The section then shows how to use Nix-Shell as a shebang to encode dependencies directly in a Python script, allowing the script to be self-bootstrapping and executable on any machine with Nix installed, without relying on globally installed dependencies.