Nix 2.4+ introduces replacements for commonly known nix commands. The `nix-shell` replacements cause some confusion which this post aims to solve.

A new Nix command

This post follows the release of Nix 2.4, which among many other things introduced an all new (experimental) nix command along with the previously discussed flake feature.

The nix command aims to collect most common commands such as nix-build, nix-copy-closure, nix-env, … as subcommands of one common program. Unsurprisingly, this does not spare nix-shell.

Yet, unlike some of the other commands which received a more or less one-to-one replacement it is not so easy with nix-shell. This command was actually broken up into multiple commands with different semantics: nix shell, nix develop and nix run. Yet, depending on what you used nix-shell for in the past the new commands may not exactly do what you would expect. Indeed, they happen to cause quite some confusion already. Most notably, while nix-shell invoked without any other flags previously set up the build environment of a derivation (which could be somewhat abused to define general development environments) that is not what nix shell will do…

What the shell…¹

Yes, but from the start…

If you are here just for the commands you can read the tl;dr for nix develop, nix shell, or nix run directly or jump to the notes section.

Development shells (`nix-shell [DERIVATION]`)

Development shells using the new nix command are now created with nix develop.

That command is a more focused version with additional tools for development. It is meant to be used with flakes but supports standalone derivations using the -f flag. How to use it then and what will it do?

A development aid

The intended use of nix develop is to recreate build environments for single packages. That is, you call nix develop nixpkgs#hello and are dropped into a shell that is as close as possible to the nix builder environment.

“yes,.. and?”

This allows you to build a package step by step, or better phase by phase, meaning that in particular the build instructions used by nix can be reused for development purposes. Thus, we can use nix without needing to repeatedly go through full build processes, including wasteful copying, configuring, etc.

To make this process even easier, nix develop now comes with special arguments to run those phases directly.

nix develop creates a shell with all buildInputs and environment variables of a derivation loaded and shellHooks executed.

This allows to run phases individually in the shell using $ unpackPhase, $ configurePhase $ buildPhase, etc.

…or directly using nix develop --<PHASE> or nix develop --phase PHASE (for non-standard phases).

…or run an arbitrary command in the shell using nix develop --command COMMAND [ARGS…]

Why should you use this?

It is most useful for locally developed packages.

Use it to set up the environment using e.g. the configurePhase and perform the subsequent development using build and check phases. If your workflow includes things that are not part of a phase use nix develop --command

In essence, this is exactly what nix-shell was intended for!

A slight annoyance with phases arises when the targeted derivation overrides standard phases, i.e. {unpack,configure,build,install}Phases. As the default implementation in nix’s stdenv is done as functions, an internal use of runHook will give precedence to those functions over the overridden phases stored as environment variables.

Solution

Enter a shell using nix develop and run the overridden phases using eval $buildPhase or --command eval ‘$buildPhase’.

Practically, nix-shell was also used for another purpose; reproducible development environments.

Development environments

Particularly useful combined with tools like direnv, one can leverage the fact that the resulting shell of ~~nix-shell~~ nix develop includes all declared buildInputs and environment variables to put together an environment with all sorts of dependencies and development tools available. Importantly, nix will also ensure all setupHooks are run when the shell is opened allowing for some impure setup to happen.

A helpful tool to achieve this is mkShell. This function provides an easy interface to collect packages for an environment.

pkgs.mkShell = {
  # a list of packages to add to the shell environment
  packages ? [ ]
, # propagate all the inputs from the given derivations
  inputsFrom ? [ ]
, buildInputs ? [ ]
, nativeBuildInputs ? [ ]
, propagatedBuildInputs ? [ ]
, propagatedNativeBuildInputs ? [ ]
, ...
}: ...

All extra attributes unknown to mkDerivation are applied as env variables.

You can provide a shellHook to run commands whenever you enter the shell

Being closely connected to flakes, nix develop supports loading a flake’s development shell directly if a devShell output is defined.

{
  description = "Flake utils demo";

  inputs.nixpkgs.url = "github:nixos/nixpkgs";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    let
    in
      flake-utils.lib.eachDefaultSystem (
        system:
          let
            pkgs' = import nixpkgs { inherit system; };
          in
            rec {
              packages = { myPackage = ... }; # packages defined here
              devShell  = pkgs'.mkShell = {
                # a list of packages to add to the shell environment
                packages ? [ jq ]
                , # propagate all the inputs from the given derivations
                  # this adds all the tools that can build myPackage to the environment
                inputsFrom ? [ pacakges.myPackage ]
              };
            }
      );
}

(Adapted from my previous post).

Temporary Programs

The fact that nix is built on the idea of the nix store from which user environments are created by cherry-picking the desired packages may raise the question, whether we may be able to amend our current environment imperatively². And in fact, we can. It is possible to add software to a user’s profile by imperatively by the means of ~~nix-env -iA~~ or nowadays nix profile install.

Beware that nix profile is incompatible with nix-env and therefore (today) also with home-manager.

Yet, we also know that installing software this way is not the Nix way of doing things.

For the times when we do want to have some program at our disposal, either to try it out, use a different version or just needing it only temporarily, there should be a way to get this without going through the effort of adding it to your configuration.nix, home.nix, project default.nix, etc. and rebuilding your environment. In these cases, traditional distributions reach back to installing software or relying on containerization (i.e. docker). In Nix, while you could install the piece of software, the aforementioned usage of the nix store allows making software available temporarily without installing it.

This is what nix-shell -p <package+> is used for. nix-shell will retrieve the desired packages and open a shell with these packages mixed in.

Given you need to quickly convert some asciidoc to HTML usually the response of your terminal will be

$ asciidoc -b html5 manual.adoc
zsh: command not found: asciidoc

While you could go and add asciidoc to your configuration you might need it just once. In that case we can make use of nix-shell:

$ nix-shell -p asciidoc
[nix-shell] $ asciidoc -b html5 manual.adoc

This should now just work. Likewise, this works with almost anything available to install through nixpkgs.

Internally, what happens when nix-shell -p asciidoc is called is that nix constructs a derivation with the programs as buildInputs and popularizes them through the same mechanism described above.

In this case the derivation shell’ed into is:

with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (asciidoc) ]; } ""

Note that instead of derivations you can also add expressions as the -p argument as these are just plugged in, i.e.:

$ nix-shell -p "import ./some.nix {}"

But this is not about nix-shell…

Nix 2.4 allows the same thing using nix shell now focussing on flakes.

With nix shell any output of a flake can be added to the environment by running

$ nix shell nixpkgs#asciidoc

Here, nix strictly expects a flake URL.

Like nix-shell this command supports multiple arguments.

nix shell creates a shell from the specified inputs.

This is useful to install temporary software

…from a flake specifier.

…from a *.nix file/arbitrary expression using --impure --expr EXPR flags

Why should you use this?

The strong point about Nix is its declarative way to manage installations. Software that is used constantly can and should be packaged by the respective tool, be it a system configuration, home configuration.

For project development tools one can use development shells as discussed above.

Yet, sometimes a program or library is needed temporarily only, or once in a different version etc. In these cases programs can be loaded into the shell using nix shell. Derivations from this kind command are eventually garbage collected and removed from the nix store, so they do not use up dist space unnecessarily.

Notably, not mentioned here is the use of nix shell to load FANCYLANGUAGE with FANCYLANGUAGEPACKAGES. Sadly, this hits the limits of the new command. See the section about shellHooks below.

Run scripts

Apart from dropping into development shells, nix-shell can also be used to run commands and programs from derivation not currently installed to the user’s profile. This is it can build a shell as before and run a command inside transparently.

We discussed the use of nix-shell --command COMMAND ARGS above, where we would run a command from within the build environment of a derivation. Similarly, we may want to just run a program provided by a derivation. For this nix-shell provided the --run argument

As a development aid, --command is interactive, meaning among other things, that if a command fails or is interrupted by the user, the user is dropped into the shell with the build environment loaded.

This behavior translates into an invocation using the -p PROGRAM argument as well as seen in the following box.

$ asciidoc 
zsh: command not found: asciidoc

$ nix-shell -p asciidoc --command asciidoc
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

$ asciidoc # still available as were in the build shell with asciidoc present
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

--run runs non-interactive and closes the shell after the command returns

$ asciidoc 
zsh: command not found: asciidoc

$ nix-shell -p asciidoc --run asciidoc
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

$ asciidoc # not available anymore
zsh: command not found: asciidoc

`nix shell -c`

As the functions of nix-shell DERIVATION and nix-shell -p DERIVATION were separated, the new tools come with new clearer semantics.

The generic nix-shell --run function is now nix shell -c. Given an installable, nix allows to run any command in an environment where the installable is present. Note that this command is run in a non-interactive shell. The shell is dropped as the command ends.

The above example using the new command would look like this:

$ nix shell nixpkgs#asciidoc -c asciidoc

`nix run`

Yet, nix shell -c will still require to type the name of the executed program. As for most programs this command is the same as the derivation name e.g. nix shell nixpkgs#asciidoc -c asciidoc another command was introduced named nix run. With nix run the previous command can be run as nix run nixpkgs#asciidoc.

Naturally, the functionality of nix run goes further and as is the case for many other new commands mainly concerns flakes. Next to packages, nixosModules and others, flakes can now also define apps. Written as records with two fields -type (currently, necessarily app) and program (an executable path) - these apps can be run directly using nix run.

This app definition

...
outputs = {self}: {
  ...
  apps.x86_64-linux.watch = {
    type = "app";
    program = "${generator-watch}";
  };
}

… could be used to watch and build this blogs source directly using nix run .#watch, given generator-watch is a script in the nix store. Note that program only accepts paths in the store and no default arguments.

If an attribute to nix run is not found as an app, nix will look up a program using this key instead and execute programs.${<program>}/bin/<program> instead.

The new nix command comes with a new way to run programs not installed in your system for an even greater “run and forget” experience.

With nix shell DERIVATION+ -c COMMAND

… run any command in an environment with all specified DERIVATIONs present (again consider the section about shellHooks)

With nix run INSTALLABLE you can

… run scripts defined in a flake under the apps output

… run the executables of any derivation as long as it is located in bin/INSTALLABLE of the derivation with the attribute name INSTALLABLE

Shell interpreter `#!/usr/bin/env nix-shell`

Lastly, a useful feature of nix-shell is its usage as a shell interpreter. What that means is that nix-shell can be used to dynamically fetch dependencies for a script file and execute the file in that context. Shell interpreters are defined using a special syntax at the start of script files.

Bash, Zsh and other shells interpret a first line starting with #! as an interpreter instruction. This way, instead of running a file using e.g. bash or python explicitly, we can write #!/usr/bin/env python to instruct the shell to use python to execute the file.

#!/usr/bin/env python

# file.py

print("hello")

Here $ python file.py would be equivalent to ./file.py, given the user has permission to execute the file.

The nix-shell command can be used to use nix to provide the actual interpreter using

#! /usr/bin/env nix-shell
#! nix-shell -i real-interpreter -p packages

Therefore, the above example would therefore continue to work even without python installed, if defined as

#! /usr/bin/env nix-shell
#! nix-shell -i python -p python3
print("hello")

As it stands this function does not yet have a replacement.

Notes and Resources

To flake or not to flake

Most of the new nix commands are designed in a flake first way. Most notably nix {shell,develop,run} expect flake URLs as argument. Traditional *.nix files can be used with the --expr argument all commands support. As flake mode imposes greater purity strictness, imports have to happen with the --impure flag given:

$ nix shell --impure --expr "import my.nix {}"

A word about `shellHook`s

While shell hooks were previously introduced as a means to run certain commands to set up a development shell with nix develop and nix-shell, they find use in other places. Concretely, neither nix run nor nix shell are running these hooks. This is even though their nix-shell equivalent does so.

What that means is that nix shell cannot be used to load e.g. a Python environment with packages directly as python modules use the shellHook to set up the PYTHONPATH. The same goes for Perl and other ecosystems with similar principles. This means

nix shell nixpkgs#{python,python3Packages.numpy}

and

nix-shell -p python python3Packages.numpy

Do not evaluate to the same thing.

Instead, one needs to work around with custom expressions or mkShells passed to nix develop.

Workarounds

For Python in particular we can use python3.withPackages to build an ad-hoc derivation with the paths set up correctly:

nix shell --impure --expr "with import <nixpkgs> {}; python.withPackages (pkgs: with pkgs; [ prettytable ])"

This might work as well with other language ecosystems

Alternatively, we can (ab)use nix develop by passing it a devShell:

nix develop --impure --expr "with import <nixpkgs> {}; pkgs.mkShell { packages = [python3 python3Packages.numpy ];}"

Both approaches work to some degree but are clunky (i.e. not improving UX as promised) and rely on the supposed-to-be-superseded channels.

The second-hardest problem

Picking up the confusion mentioned in the beginning, there is another problem with nix shell… naming. Being so closely named to its semantically different predecessor, it is impossible to query google for meaningful, targeted results. This is a pain for newcomers and more experienced nix’ers alike. And indeed, there is a heated discussion on renaming the shell command. Yet, until that is resolved, I hope this guide helps to understand the differences a bit better.

newnixshellguide

Appendix

Summary table

/	`nix-shell`	`nix develop`	`nix shell`	`nix run`
runs `shellHook`s	yes	yes	no	no
use as interpreter	yes	no	no	no
supports flakes	no	yes	yes	only
evaluate nix file	yes	with `--impure`, `-f` or `--expr`	with `--impure`, `-f` or `--expr`	with `--impure`, `-f` or `--expr`
modifies environment	`PATH`, attributes mk`mkDerivation` and changes by `shellHooks`	`PATH`, attributes mk`mkDerivation` and changes by `shellHooks`	`PATH`	nothing

Seemingly, no-one has given any thought about google’ability of the new commands… google “newnixshellguide” to find this post in the meantime↩︎
Functional purists, please bear with me here↩︎

One too many shell Clearing up with nix' shells nix shell and nix-shell

Credits

A new Nix command

What the shell…¹

Development shells (`nix-shell [DERIVATION]`)

A development aid

Development environments

Temporary Programs

Run scripts

`nix shell -c`

`nix run`

Shell interpreter `#!/usr/bin/env nix-shell`

Notes and Resources

To flake or not to flake

A word about `shellHook`s

Workarounds

The second-hardest problem

Appendix

Summary table

Credits

A new Nix command

What the shell…1

Development shells (nix-shell [DERIVATION])

A development aid

Development environments

Temporary Programs

Run scripts

nix shell -c

nix run

Shell interpreter #!/usr/bin/env nix-shell

Notes and Resources

To flake or not to flake

A word about shellHooks

Workarounds

The second-hardest problem

Appendix

Summary table

What the shell…¹

Development shells (`nix-shell [DERIVATION]`)

`nix shell -c`

`nix run`

Shell interpreter `#!/usr/bin/env nix-shell`

A word about `shellHook`s