Setting up a development environment

Nix is used heavily in the Batsim ecosystem. Not only to distribute the software to end users, but also to manage the build and test environments of the various tools. In the end, we recommend Nix to any Batsim user or developer:

• For end users, it provides controlled environments to execute your simulations, which helps a lot in making your experiments repeatable (yes, you/your advisors/your reviewers/etc. want them to be repeatable :p). It also avoids the hassle of installing dependencies. Please give a look at Using Batsim from a well-defined Nix environment for a simple Nix usage in this case, or follow our Doing a reproducible experiment tutorial to see more advanced usage (tuning/pinning the versions of the various tools).

• For developers getting started with Batsim, we provide environments to build or test Batsim, which means no hassle to install dependencies regardless of your Linux distribution or operating system. It also means that you can trivially share your development environment, which will enable us to help you more easily if you Contact us.

• For advanced Batsim developers, Nix’s full power can be unleashed to tune various packages at the same time while remaining clean/reproducible/shareable with other developers. This is especially useful when adding a new Batsim feature that involves modifying Batsim and schedulers at the same time — or when you also need to modify SimGrid for your work.

Nix architecture

Most of our packages are defined in the NUR-Kapack repository. The default versions of packages defined in NUR-Kapack are meant to be used for end users: Optimized binaries without debug information.

The daily environment used to build and test Batsim (by developers or Continuous Integration machines) is different, as in this case we want debug information on Batsim but also on its dependencies, notably SimGrid. All of this environment is defined in default.nix, located at the root of Batsim’s git repository. This file reuses the package definitions from NUR-Kapack and overrides them to have the desired behavior. This file defines a Nix set, that is to say an associative structure that here associates names (attributes) to packages or environments (Nix derivations). Here is an overview of the various attributes of this file:

• batsim defines a package that builds the Batsim executable (and coverage information generated by gcov at compile time). This package may also run unit tests and generate coverage information about them.

• integration_tests defines a package that runs Batsim integration tests and generates test report and coverage information.

• coverage-report combines gcov information to generate readable coverage reports.

• sphinx_doc generates Batsim’s sphinx documentation (you are currently reading it).

• doxydoc generates Batsim’s code doxygen documentation.

Basic usage with nix-build

Most of our attributes are meant to be used easily with nix-build. As the name says, nix-build builds a Nix expression and puts the result files in the Nix store — a ./result symlink is also produced so you can easily give a look at the files generated by the command.

You can for example build a debug version of Batsim from local sources with nix-build -A batsim, then run it with ./result/bin/batsim (do not worry if you see many warning lines about profiling/gcda files, this is a normal artifact of our gcov usage).

Note

The first time you call this command, Nix may compile many dependencies if they are not present in your machine’s cache. Most dependencies are likely to be present in Batsim’s binary cache, so you can simply download them from there if you want to save some time — refer to Using Nix to enable fetching data from Batsim’s binary cache with cachix.

You can then run integration tests with nix-build -A integration_tests. Please note that Nix will compile/download any needed dependency when this command is called. Here, this means that Batsim (from the batsim attribute) will be compiled if needed, or schedulers (batsched and pybatsim attributes), or tools to run the tests (pytest and batexpe).

Finally, you can see which Batsim code lines are covered by our tests by calling nix-build -A coverage-report. By default, this should generate textual reports (plain text, CSV and JSON) — cat ./result/file-summary.txt prints one of the generated files. You can enable many other output formats by setting Nix variables when calling commands. For example, nix-build --arg coverageHtml true -A coverage-report will generate human readable html files where you can browse lines of code and see whether they are covered or not with firefox ./result/html/index.html. For a list of available arguments, please read the first lines of default.nix.

Warning

By default, the integration_tests attribute will always be rebuilt (when you call nix-build -A integration_tests directly or when you use any attribute that requires it such as nix-build -A coverage-report). This is done voluntarily as some tests may have non-deterministic outcome and we want to easily run them many times in a row.

If for some reason you want to avoid rebuilds (typically if you want to generate coverage reports from an existing test run), you can define the testVersion Nix variable to a fixed value when running your commands: nix-build --argstr testVersion fixed -A integration_tests then nix-build --argstr testVersion fixed -A coverage-report. You can put any value instead of fixed as long as you use the same value in all your commands.

Iterative builds with nix-shell

nix-build -A batsim makes sure that Batsim is built correctly by building Batsim sources from scratch every time. This is an interesting property, but sometimes you just want to compile many times in rapid succession, and keeping/updating a build cache to compile Batsim can be very useful. The simplest way to achieve this is to get into an environment that can build Batsim, and to compile it manually from there.

• Enter shell that can build Batsim: nix-shell -A batsim

• From inside the shell, generate a Meson build directory if it does not already exist: meson build

• From inside the shell, compile Batsim: ninja -C build

You should now be able to edit your code however you want then call ninja -C build to trigger a Batsim build with cache.

Warning

Please note that you may need to recreate the build directory depending on what you do. Typically, whenever you add new files or change the include graph of the project, creating a new build directory is advised.

Using IDEs such as qtcreator

Depending on what development you are doing, using an IDE such as qtcreator can be useful to edit source code. This is completely doable, but you have to make sure to run your IDE from the right environment. Assuming that qtcreator is already installed in your local environment, you can follow these steps to make it able to build Batsim.

• Enter a shell that can build Batsim with CMake: nix-shell -A batsim_cmake

• From inside the shell, generate a CMake build directory if it does not already exist: (mkdir build-cmake && cd build-cmake && cmake .. && make -j $(nproc))

• From inside the shell, run qtcreator to import an existing CMake project: qtcreator ./CMakeLists.txt &. When qtcreator prompts you about how to import the project, tell it to only use the existing build environment/directory and not to create a new one.

• You should now be able to compile Batsim from qtcreator.

Run tests from iteratively-built Batsim

Sometimes, you want to run some Batsim integration tests from an iteratively-built Batsim. Typically, this happens when you are making a single test pass and you want to do fast build/test iterations. This can be done by using several environments at the same time:

• Keep a first environment open, in which you build Batsim. These environments can be Iterative builds with nix-shell or Using IDEs such as qtcreator for example.

• Keep a second hacky environment open, in which you run the tests.

You can create the second environment by calling nix-shell -A integration_tests — the following commands of this section are to be executed in the shell created by this nix-shell command. The default batsim executable in this environment is the one built cleanly by Nix. You can call which batsim to see where the batsim executable is located (spoiler: in the Nix store). As Batsim integration tests simply run Batsim by calling a batsim command, you can hack the test environment to run your iteratively built Batsim by hacking your PATH environment variable. For example, if your Batsim build cache directory is in ${HOME}/proj/batsim/build, you can hack your call path via this command: export PATH="${HOME}/proj/batsim/build:${PATH}". To make sure the hack worked, call which batsim to make sure that batsim now references your iteratively built file.

Now, in the second shell, you can go into the test directory (from Batsim’s root directory) and run tests how you want (pytest will run all tests, pytest test_energy.py will run energy tests…).

Running a simulation instance with gdb

Sometimes, you want to run Batsim/the scheduler/both with a debugger to see what’s going on — typically when a test fails and you modify Batsim’s source code in a trial and error method. This section shows how this can be done (we will only run Batsim with a debugger here).

First, please read Run tests from iteratively-built Batsim, as this section is a direct continuation to it. Here, we will need three environments:

• A first environment to build Batsim iteratively. Make sure to build Batsim without optimization and with debug information (DWARF symbols). This should be done by default by Meson but you can force it if needed, refer to Running Meson and Meson’s built-in options in this case. Calling file ./build/batsim on your iteratively built batsim should show whether debug information is present or not (with debug_info, not stripped).

• A second environment to execute Batsim. A starting point of this environment is the one defined in Run tests from iteratively-built Batsim. Unless stated explicitly, all commands listed in the following of this section are to be run in this environment.

• A third environment to execute the scheduler. As here we won’t modify the scheduler, just use nix-shell -A integration_tests.

The main entry point of Batsim’s integration tests is to call pytest, that will generate many simulation instances and run them. Most simulation instances consist in calling batsim with some command-line arguments, and a scheduler (typically batsched, sometimes pybatsim) with some command-line arguments. A wrapper process called robin is in charge of executing both commands and of handling errors (typically, kill batsim when scheduler crashes and vice versa, or kill everyone when a timeout is reached to avoid infinite loops). When robin runs a simulation instance, it generates separate scripts to run Batsim and the scheduler. Here, we will see how these scripts can be hacked to run Batsim from gdb.

First, clean the output directory of Batsim’s tests, which is located in test/test-out from Batsim’s source directory. Then, run integration tests with pytest from the test directory — here we will assume that one of the tests defined in test_walltime.py fails so we will run pytest test_walltime.py. Running this command will populate the test-out directory with many tests: A directory is created per simulation instance, and an instance.yaml file describes how to run each instance. You can manually run a simulation instance by calling robin instance.yaml (assuming your have moved your current working directory to the test you want to work on). You can run several tests until you find the one that fails. Now, assuming you found the failing test you want to work on, you can give a look at the test directory structure — tree should output something like this:

.
├── batres_jobs.csv
├── batres_machine_states.csv
├── batres_schedule.csv
├── batres_schedule.trace
├── cmd
│   ├── batsim.bash
│   └── sched.bash
├── instance.yaml
└── log
    ├── batsim.log
    ├── sched.err.log
    └── sched.out.log

• Files that start with batres_ are simulation results generated by Batsim.

• The log directory contains the logs of the batsim/scheduler commands.

• The cmd directory contains the commands that were run by robin.

You can now run the simulation instance without robin, by calling batsim and the scheduler in their own shells.

• In the shell configured to run your iteratively built Batsim, you can run ./cmd/batsim.bash.

• In the shell for the scheduler, you can run ./cmd/sched.bash.

Now you just have to hack ./cmd/batsim.bash so that it runs batsim with your favorite debugger, and execute both scripts in their own shells again. As I personally use gdb with the cgdb terminal interface, so I just prefix the Batsim command with cgdb --args to run it inside my debugger.

Note

Your debugger may not display some source files. This should not happen for Batsim source code, but it may happen for dependencies such as SimGrid, in which case you can observe something like this:

(gdb)
#8  0x00007ffff7a1270c in simgrid::kernel::context::Context::operator() (this=0x7f9d00) at ../src/kernel/context/Context.hpp:65
65      ../src/kernel/context/Context.hpp: No such file or directory.

If you want to display SimGrid source code, clone SimGrid’s source code somewhere in your filesystem (make sure to checkout the exact same version as the one defined in default.nix/NUR-Kapack) and tell your debugger where to find source files. If you cloned SimGrid in /home/user/proj/simgrid-release and you use gdb, this is done by typing dir /home/user/proj/simgrid-release/src in gdb’s interactive interface.

You can do the exact same trick for any source code that your debugger cannot find. You can also create initialization files for your debugger to load these directories automatically.

Changing Batsim dependencies

Sometimes, you not only need to hack Batsim but also one of its dependencies, which happens quite often with SimGrid. In this section we will see how default.nix can be hacked to use a custom SimGrid version, that can either come from your own git repository (fork) or just from your local filesystem.

As simgrid is an input of default.nix, its value can be overridden when default.nix is evaluated by giving command-line arguments to nix-build or nix-shell without modifying the file at all. However, we will not take this approach here as modifying default.nix is easier to do. We will instead create a new SimGrid package called my-custom-simgrid then tell the batsim package to use our new custom SimGrid. As I write these lines, default.nix defines and returns a Nix set named jobs. It’s inside jobs that we will add our new package. Here is a first version of my-custom-simgrid

my-custom-simgrid = (kapack.simgrid-light.override { inherit debug; }).overrideAttrs (attr: rec {
  src = pkgs.fetchgit {
    rev = "44bca631e482bd6e48a926393437d0959b661218";
    url = "https://framagit.org/mpoquet/simgrid.git";
    sha256 = "sha256:1fs0sdwx77yrakbffh00g7hxqladbjpqcs15lcx37viahjlk7fp0";
  };
});

Here is an explanation of the various subparts of this Nix expression:

• my-custom-simgrid = ...; defines a new attribute named my-custom-simgrid in the embracing set (named jobs).

• (kapack.simgrid-light.override { inherit debug; }) overrides the inputs of the simgrid-light SimGrid version defined in NUR-Kapack, by customizing its debug input. inherit debug; is strictly equivalent to debug=debug;. Most packages in NUR-Kapack have a debug input that make packages generate and keep debug information (DWARF symbols).

• <package>.overrideAttrs (attr: rec {...}) overrides the definition of <package> (here package is a lightweight SimGrid with debug information). This will enable us in next lines to override the package source code.

• src = pkgs.fetchgit {rev="..."; url="..."; sha256="...";}; defines a new source for the package we are overriding. Here, the source code will be fetched from a Git repository on url https://framagit.org/mpoquet/simgrid.git and commit 44bca631e482bd6e48a926393437d0959b661218. The sha256 attribute is a checksum used by Nix to make sure the fetched data is the expected one.

Note

Filling the sha256 field can seem tricky for Nix newcomers. A fast way to do it is to:

1. Fill the field with a random SHA-256 string. For example, calling echo simgrid | sha256sum will generate a valid SHA-256 string.

2. Try to build your package (here, nix-build -A my-custom-simgrid). Nix will cry about hash mismatch.

   hash mismatch in fixed-output derivation '/nix/storesk57v94s4y55b6r0xfzzy6g3sfsg20mi-simgrid-44bca63':
     wanted: sha256:1g3jwfnij8016b866frc8jl46fp39ivlznm1ib726xjz700lr3kr
     got:    sha256:1fs0sdwx77yrakbffh00g7hxqladbjpqcs15lcx37viahjlk7fp0
   

3. Copy the got value computed by Nix into your Nix expression.

You can now modify the batsim package to use my-custom-simgrid instead of simgrid. This is done by changing the simgrid input when we override the Batsim package defined in NUR-Kapack, as shown in this diff:

-    batsim = (kapack.batsim.override { inherit debug simgrid; }).overrideAttrs (attr: rec {
+    batsim = (kapack.batsim.override { inherit debug; simgrid=my-custom-simgrid; }).overrideAttrs (attr: rec {

And that’s it, the batsim package now uses the SimGrid version we just defined :).

If you want to use a local SimGrid version rather than one from a Git repository, the steps to follow are the same, you just need to change the source of your my-custom-simgrid package:

my-custom-simgrid = (kapack.simgrid-light.override { inherit debug; }).overrideAttrs (attr: rec {
  src = /path/to/your/local/simgrid/source/repository;
});

Hacking Batsim and a scheduler at the same time

Similarly to Changing Batsim dependencies, it is quite common to work on both Batsim and a scheduler at the same time. This is notably the case when modifying the Protocol. This section shows how to hack batsched/pybatsim and Batsim at the same time.

A first simple way to do this is to bypass our Nix environments and put your own version of batsched/pybatsim in the PATH of the shell that launches the integration tests. The procedure to achieve this is very similar to what we have done in Run tests from iteratively-built Batsim so you can refer to it for details on how to setup your PATH environment variable. Here are short instructions on how to build your own local version of batsched and pybatsim:

• Batsched can be obtained from batsched’s git repository and is built in the same way as Batsim. From the root of batsched’s git repository, you can enter a shell able to build the batsched executable by calling nix-shell -A batsched. meson build should then generate a build directory. And finally, ninja -C build should compile a batsched executable in the build directory.

• Pybatsim can be obtained from pybatsim’s git repository. As I write these lines, pybatsim does not have a clean Nix environment support for now, but you can enter a virtualenv to work on pybatsim with the following commands (from the root of pybatsim’s git repository). First, either install python/virtualenv in your local machine or enter a Nix shell that has python and virtualenv: nix-shell -p python3Packages.virtualenv. Then, create a new virtualenv by calling virtualenv venv and set your environment variables by calling source ./venv/bin/activate. You can then build/install a local pybatsim by calling pip install ..

From there, hacking the environment of a shell that runs Batsim integration tests should be very easy for batsched, but it can be a bit tricky for pybatsim as it uses Python. Instead, you can decide to go for a Nix setup to change the versions of batsched or pybatsim, in a very similar fashion to what we did in Changing Batsim dependencies. Here is an example on what to add in the jobs Nix set in Batsim’s default.nix to create your custom versions of batsched and pybatsim:

my-custom-batsched = (kapack.batsched.override { inherit debug; }).overrideAttrs (attr: rec {
  src = /path/to/your/local/batsched/source/repository;
  preConfigure = "rm -rf build";
});

my-custom-pybatsim = kapack.pybatsim.overridePythonAttrs(attr: rec {
  src = /path/to/your/local/pybatsim/source/repository;
});

This Nix expression is very similar to what we have done and explained in Changing Batsim dependencies, so please refer to it for a detailed explanation of this snippet. A new Nix trick here is overridePythonAttrs, which does exactly the same as overrideAttrs but with the additional dark magic to make it work with Python. We also added preConfigure = "rm -rf build"; in the my-custom-batsched package definition, this line makes sure the Nix build starts from a clean build directory (this does not removes the build directory in your local repository that contain batsched’s sources, the only directory deleted by this command is in the temporary copy done by Nix when it builds the package). You can also of course use a git repository as the package source instead of a directory in your local filesystem (once again, see Changing Batsim dependencies for details).

Then, you can change the integration_tests attribute in default.nix to use your versions of batsched and pybatsim, as shown in the following diff:

    buildInputs = with pkgs.python37Packages; [
-     batsim batsched batexpe pkgs.redis
-     pybatsim pytest pytest_html pandas] ++
+     batsim my-custom-batsched batexpe pkgs.redis
+     my-custom-pybatsim pytest pytest_html pandas] ++
    pkgs.lib.optional doValgrindAnalysis [ pkgs.valgrind ];

Now, whenever you run the integration_tests (nix-build -A integration_tests) or enter a shell to do so (nix-shell -A integration_tests), the environment should use your custom versions of batsched and pybatsim.