.. _setting_up_dev_env: Setting up a development environment ==================================== Nix_ is used heavily in the Batsim ecosystem. Not only to distribute the software to end users, but also to manage the build and test environments of the various tools. In the end, we recommend Nix to any Batsim user or developer: - For end users, it provides controlled environments to execute your simulations, which helps a lot in making your experiments repeatable (yes, you/your advisors/your reviewers/etc. want them to be repeatable :p). It also avoids the hassle of installing dependencies. Please give a look at :ref:`using_batsim_from_well_defined_nix_env` for a simple Nix usage in this case, or follow our :ref:`tuto_reproducible_experiment` tutorial to see more advanced usage (tuning/pinning the versions of the various tools). - For developers getting started with Batsim, we provide environments to build or test Batsim, which means no hassle to install dependencies regardless of your Linux distribution or operating system. It also means that you can trivially share your development environment, which will enable us to help you more easily if you :ref:`contact_us`. - For advanced Batsim developers, Nix's full power can be unleashed to tune various packages at the same time while remaining clean/reproducible/shareable with other developers. This is especially useful when adding a new Batsim feature that involves modifying Batsim and schedulers at the same time — or when you also need to modify SimGrid for your work. Nix architecture ---------------- Most of our packages are defined in the `NUR-Kapack`_ repository. The default versions of packages defined in NUR-Kapack are meant to be used for end users: Optimized binaries without debug information. The daily environment used to build and test Batsim (by developers or :ref:`ci` machines) is different, as in this case we want debug information on Batsim but also on its dependencies, notably SimGrid. All of this environment is defined in :download:`default.nix <../default.nix>`, located at the root of Batsim's git repository. This file reuses the package definitions from NUR-Kapack and *overrides* them to have the desired behavior. This file defines a `Nix set`_, that is to say an associative structure that here associates names (*attributes*) to packages or environments (`Nix derivations`_). Here is an overview of the various attributes of this file: - ``batsim`` defines a package that builds the Batsim executable (and coverage information generated by gcov at compile time). This package may also run unit tests and generate coverage information about them. - ``integration_tests`` defines a package that runs Batsim integration tests and generates test report and coverage information. - ``coverage-report`` combines gcov information to generate readable coverage reports. - ``sphinx_doc`` generates Batsim's sphinx documentation (you are currently reading it). - ``doxydoc`` generates Batsim's code doxygen documentation. Basic usage with ``nix-build`` ------------------------------ Most of our *attributes* are meant to be used easily with ``nix-build``. As the name says, ``nix-build`` builds a Nix expression and puts the result files in the Nix store — a ``./result`` symlink is also produced so you can easily give a look at the files generated by the command. You can for example build a debug version of Batsim from local sources with ``nix-build -A batsim``, then run it with ``./result/bin/batsim`` (do not worry if you see many warning lines about profiling/gcda files, this is a normal artifact of our gcov usage). .. note:: The first time you call this command, Nix may compile many dependencies if they are not present in your machine's cache. Most dependencies are likely to be present in Batsim's binary cache, so you can simply download them from there if you want to save some time — refer to :ref:`using_nix_introduction` to enable fetching data from Batsim's binary cache with ``cachix``. You can then run integration tests with ``nix-build -A integration_tests``. Please note that Nix will compile/download any needed dependency when this command is called. Here, this means that Batsim (from the ``batsim`` attribute) will be compiled if needed, or schedulers (``batsched`` and ``pybatsim`` attributes), or tools to run the tests (``pytest`` and ``batexpe``). Finally, you can see which Batsim code lines are covered by our tests by calling ``nix-build -A coverage-report``. By default, this should generate textual reports (plain text, CSV and JSON) — ``cat ./result/file-summary.txt`` prints one of the generated files. You can enable many other output formats by setting Nix variables when calling commands. For example, ``nix-build --arg coverageHtml true -A coverage-report`` will generate human readable html files where you can browse lines of code and see whether they are covered or not with ``firefox ./result/html/index.html``. For a list of available arguments, please read the first lines of :download:`default.nix <../default.nix>`. .. warning:: By default, the ``integration_tests`` attribute will always be rebuilt (when you call ``nix-build -A integration_tests`` directly or when you use any attribute that requires it such as ``nix-build -A coverage-report``). This is done voluntarily as some tests may have non-deterministic outcome and we want to easily run them many times in a row. If for some reason you want to avoid rebuilds (typically if you want to generate coverage reports from an existing test run), you can define the ``testVersion`` Nix variable to a fixed value when running your commands: ``nix-build --argstr testVersion fixed -A integration_tests`` then ``nix-build --argstr testVersion fixed -A coverage-report``. You can put any value instead of ``fixed`` as long as you use the same value in all your commands. .. _iterative_builds_with_nix_shell: Iterative builds with ``nix-shell`` ----------------------------------- ``nix-build -A batsim`` makes sure that Batsim is built correctly by building Batsim sources from scratch every time. This is an interesting property, but sometimes you just want to compile many times in rapid succession, and keeping/updating a build cache to compile Batsim can be very useful. The simplest way to achieve this is to get into an environment that can build Batsim, and to compile it manually from there. - Enter shell that can build Batsim: ``nix-shell -A batsim`` - From inside the shell, generate a Meson build directory if it does not already exist: ``meson build`` - From inside the shell, compile Batsim: ``ninja -C build`` You should now be able to edit your code however you want then call ``ninja -C build`` to trigger a Batsim build with cache. .. warning:: Please note that you may need to recreate the build directory depending on what you do. Typically, whenever you add new files or change the include graph of the project, creating a new build directory is advised. .. _using_ides_such_as_qtcreator: Using IDEs such as ``qtcreator`` -------------------------------- Depending on what development you are doing, using an IDE such as ``qtcreator`` can be useful to edit source code. This is completely doable, but you have to make sure to run your IDE from the right environment. Assuming that ``qtcreator`` is already installed in your local environment, you can follow these steps to make it able to build Batsim. - Enter a shell that can build Batsim with CMake: ``nix-shell -A batsim_cmake`` - From inside the shell, generate a CMake build directory if it does not already exist: ``(mkdir build-cmake && cd build-cmake && cmake .. && make -j $(nproc))`` - From inside the shell, run qtcreator to import an existing CMake project: ``qtcreator ./CMakeLists.txt &``. When qtcreator prompts you about how to import the project, tell it to only use the existing build environment/directory and **not** to create a new one. - You should now be able to compile Batsim from qtcreator. .. _run_tests_from_iteratively_built_batsim: Run tests from iteratively-built Batsim --------------------------------------- Sometimes, you want to run some Batsim integration tests from an iteratively-built Batsim. Typically, this happens when you are making a single test pass and you want to do fast build/test iterations. This can be done by using several environments at the same time: - Keep a first environment open, in which you build Batsim. These environments can be :ref:`iterative_builds_with_nix_shell` or :ref:`using_ides_such_as_qtcreator` for example. - Keep a second *hacky* environment open, in which you run the tests. You can create the second environment by calling ``nix-shell -A integration_tests`` — **the following commands of this section are to be executed in the shell created by this nix-shell command**. The default ``batsim`` executable in this environment is the one built cleanly by Nix. You can call ``which batsim`` to see where the ``batsim`` executable is located (spoiler: in the Nix store). As Batsim integration tests simply run Batsim by calling a ``batsim`` command, **you can hack the test environment to run your iteratively built Batsim by hacking your PATH environment variable**. For example, if your Batsim build cache directory is in ``${HOME}/proj/batsim/build``, you can hack your call path via this command: ``export PATH="${HOME}/proj/batsim/build:${PATH}"``. To make sure the hack worked, call ``which batsim`` to make sure that ``batsim`` now references your iteratively built file. Now, in the second shell, you can go into the ``test`` directory (from Batsim's root directory) and run tests how you want (``pytest`` will run all tests, ``pytest test_energy.py`` will run energy tests...). Running a simulation instance with gdb -------------------------------------- Sometimes, you want to run Batsim/the scheduler/both with a debugger to see what's going on — typically when a test fails and you modify Batsim's source code in a trial and error method. This section shows how this can be done (we will only run Batsim with a debugger here). First, please read :ref:`run_tests_from_iteratively_built_batsim`, as this section is a direct continuation to it. Here, we will need three environments: - A first environment to build Batsim iteratively. Make sure to build Batsim without optimization and with debug information (DWARF symbols). This should be done by default by Meson but you can force it if needed, refer to `Running Meson`_ and `Meson's built-in options`_ in this case. Calling ``file ./build/batsim`` on your iteratively built batsim should show whether debug information is present or not (``with debug_info, not stripped``). - A second environment to execute Batsim. A starting point of this environment is the one defined in :ref:`run_tests_from_iteratively_built_batsim`. **Unless stated explicitly, all commands listed in the following of this section are to be run in this environment**. - A third environment to execute the scheduler. As here we won't modify the scheduler, just use ``nix-shell -A integration_tests``. The main entry point of Batsim's integration tests is to call pytest, that will generate many simulation instances and run them. Most simulation instances consist in calling ``batsim`` with some command-line arguments, and a scheduler (typically ``batsched``, sometimes ``pybatsim``) with some command-line arguments. A wrapper process called ``robin`` is in charge of executing both commands and of handling errors (typically, kill batsim when scheduler crashes and vice versa, or kill everyone when a timeout is reached to avoid infinite loops). When ``robin`` runs a simulation instance, it generates separate scripts to run Batsim and the scheduler. Here, we will see how these scripts can be hacked to run Batsim from gdb. First, clean the output directory of Batsim's tests, which is located in ``test/test-out`` from Batsim's source directory. Then, run integration tests with ``pytest`` from the ``test`` directory — here we will assume that one of the tests defined in ``test_walltime.py`` fails so we will run ``pytest test_walltime.py``. Running this command will populate the ``test-out`` directory with many tests: A directory is created per simulation instance, and an ``instance.yaml`` file describes how to run each instance. You can manually run a simulation instance by calling ``robin instance.yaml`` (assuming your have moved your current working directory to the test you want to work on). You can run several tests until you find the one that fails. Now, assuming you found the failing test you want to work on, you can give a look at the test directory structure — ``tree`` should output something like this: .. code-block:: text . ├── batres_jobs.csv ├── batres_machine_states.csv ├── batres_schedule.csv ├── batres_schedule.trace ├── cmd │   ├── batsim.bash │   └── sched.bash ├── instance.yaml └── log ├── batsim.log ├── sched.err.log └── sched.out.log - Files that start with ``batres_`` are simulation results generated by Batsim. - The ``log`` directory contains the logs of the batsim/scheduler commands. - The ``cmd`` directory contains the commands that were run by robin. You can now run the simulation instance without robin, by calling ``batsim`` and the scheduler in their own shells. - In the shell configured to run your iteratively built Batsim, you can run ``./cmd/batsim.bash``. - In the shell for the scheduler, you can run ``./cmd/sched.bash``. Now you just have to hack ``./cmd/batsim.bash`` so that it runs ``batsim`` with your favorite debugger, and execute both scripts in their own shells again. As I personally use ``gdb`` with the cgdb_ terminal interface, so I just prefix the Batsim command with ``cgdb --args`` to run it inside my debugger. .. note:: Your debugger may not display some source files. This should not happen for Batsim source code, but it may happen for dependencies such as SimGrid, in which case you can observe something like this: .. code-block:: text (gdb) #8 0x00007ffff7a1270c in simgrid::kernel::context::Context::operator() (this=0x7f9d00) at ../src/kernel/context/Context.hpp:65 65 ../src/kernel/context/Context.hpp: No such file or directory. If you want to display SimGrid source code, clone SimGrid's source code somewhere in your filesystem (make sure to checkout the exact same version as the one defined in :download:`default.nix <../default.nix>`/`NUR-Kapack`_) and tell your debugger where to find source files. If you cloned SimGrid in ``/home/user/proj/simgrid-release`` and you use ``gdb``, this is done by typing ``dir /home/user/proj/simgrid-release/src`` in gdb's interactive interface. You can do the exact same trick for any source code that your debugger cannot find. You can also create initialization files for your debugger to load these directories automatically. .. _changing_batsim_dependencies: Changing Batsim dependencies ---------------------------- Sometimes, you not only need to hack Batsim but also one of its dependencies, which happens quite often with SimGrid. In this section we will see how :download:`default.nix <../default.nix>` can be hacked to use a custom SimGrid version, that can either come from your own git repository (fork) or just from your local filesystem. As ``simgrid`` is an *input* of :download:`default.nix <../default.nix>`, its value can be overridden when :download:`default.nix <../default.nix>` is evaluated by giving command-line arguments to ``nix-build`` or ``nix-shell`` without modifying the file at all. However, we will **not** take this approach here as modifying :download:`default.nix <../default.nix>` is easier to do. We will instead create a new SimGrid package called ``my-custom-simgrid`` then tell the ``batsim`` package to use our new custom SimGrid. As I write these lines, :download:`default.nix <../default.nix>` defines and returns a `Nix set`_ named ``jobs``. It's **inside** ``jobs`` that we will add our new package. Here is a first version of ``my-custom-simgrid`` .. code-block:: nix my-custom-simgrid = (kapack.simgrid-light.override { inherit debug; }).overrideAttrs (attr: rec { src = pkgs.fetchgit { rev = "44bca631e482bd6e48a926393437d0959b661218"; url = "https://framagit.org/mpoquet/simgrid.git"; sha256 = "sha256:1fs0sdwx77yrakbffh00g7hxqladbjpqcs15lcx37viahjlk7fp0"; }; }); Here is an explanation of the various subparts of this Nix expression: - ``my-custom-simgrid = ...;`` defines a new *attribute* named ``my-custom-simgrid`` in the embracing set (named ``jobs``). - ``(kapack.simgrid-light.override { inherit debug; })`` *overrides* the **inputs** of the ``simgrid-light`` SimGrid version defined in `NUR-Kapack`_, by customizing its ``debug`` input. ``inherit debug;`` is strictly equivalent to ``debug=debug;``. Most packages in `NUR-Kapack`_ have a ``debug`` input that make packages generate and keep debug information (DWARF symbols). - ``.overrideAttrs (attr: rec {...})`` *overrides* the **definition** of ```` (here ``package`` is a lightweight SimGrid with debug information). This will enable us in next lines to *override* the package source code. - ``src = pkgs.fetchgit {rev="..."; url="..."; sha256="...";};`` defines a new source for the package we are overriding. Here, the source code will be fetched from a Git repository on url ``https://framagit.org/mpoquet/simgrid.git`` and commit ``44bca631e482bd6e48a926393437d0959b661218``. The ``sha256`` attribute is a checksum used by Nix to make sure the fetched data is the expected one. .. note:: Filling the ``sha256`` field can seem tricky for Nix newcomers. A fast way to do it is to: 1. Fill the field with a random `SHA-256`_ string. For example, calling ``echo simgrid | sha256sum`` will generate a valid SHA-256 string. 2. Try to build your package (here, ``nix-build -A my-custom-simgrid``). Nix will cry about hash mismatch. .. code-block:: text hash mismatch in fixed-output derivation '/nix/storesk57v94s4y55b6r0xfzzy6g3sfsg20mi-simgrid-44bca63': wanted: sha256:1g3jwfnij8016b866frc8jl46fp39ivlznm1ib726xjz700lr3kr got: sha256:1fs0sdwx77yrakbffh00g7hxqladbjpqcs15lcx37viahjlk7fp0 3. Copy the ``got`` value computed by Nix into your Nix expression. You can now modify the ``batsim`` package to use ``my-custom-simgrid`` instead of ``simgrid``. This is done by changing the ``simgrid`` input when we *override* the Batsim package defined in `NUR-Kapack`_, as shown in this diff: .. code-block:: diff - batsim = (kapack.batsim.override { inherit debug simgrid; }).overrideAttrs (attr: rec { + batsim = (kapack.batsim.override { inherit debug; simgrid=my-custom-simgrid; }).overrideAttrs (attr: rec { And that's it, the ``batsim`` package now uses the SimGrid version we just defined :). If you want to use a local SimGrid version rather than one from a Git repository, the steps to follow are the same, you just need to change the source of your ``my-custom-simgrid`` package: .. code-block:: nix my-custom-simgrid = (kapack.simgrid-light.override { inherit debug; }).overrideAttrs (attr: rec { src = /path/to/your/local/simgrid/source/repository; }); .. _hacking_batsim_and_scheduler_at_same_time: Hacking Batsim and a scheduler at the same time ----------------------------------------------- Similarly to :ref:`changing_batsim_dependencies`, it is quite common to work on both Batsim and a scheduler at the same time. This is notably the case when modifying the :ref:`protocol`. This section shows how to hack batsched/pybatsim and Batsim at the same time. A first simple way to do this is to bypass our Nix environments and put your own version of batsched/pybatsim in the ``PATH`` of the shell that launches the integration tests. The procedure to achieve this is very similar to what we have done in :ref:`run_tests_from_iteratively_built_batsim` so you can refer to it for details on how to setup your ``PATH`` environment variable. Here are short instructions on how to build your own local version of batsched and pybatsim: - **Batsched** can be obtained from `batsched's git repository`_ and is built in the same way as Batsim. From the root of batsched's git repository, you can enter a shell able to build the batsched executable by calling ``nix-shell -A batsched``. ``meson build`` should then generate a build directory. And finally, ``ninja -C build`` should compile a ``batsched`` executable in the build directory. - **Pybatsim** can be obtained from `pybatsim's git repository`_. As I write these lines, pybatsim does not have a clean Nix environment support for now, but you can enter a virtualenv_ to work on pybatsim with the following commands (from the root of pybatsim's git repository). First, either install python/virtualenv in your local machine or enter a Nix shell that has python and virtualenv: ``nix-shell -p python3Packages.virtualenv``. Then, create a new virtualenv by calling ``virtualenv venv`` and set your environment variables by calling ``source ./venv/bin/activate``. You can then build/install a local pybatsim by calling ``pip install .``. From there, hacking the environment of a shell that runs Batsim integration tests should be very easy for batsched, but it can be a bit tricky for pybatsim as it uses Python. Instead, you can decide to go for a Nix setup to change the versions of batsched or pybatsim, in a very similar fashion to what we did in :ref:`changing_batsim_dependencies`. Here is an example on what to add in the ``jobs`` `Nix set`_ in Batsim's :download:`default.nix <../default.nix>` to create your custom versions of batsched and pybatsim: .. code-block:: nix my-custom-batsched = (kapack.batsched.override { inherit debug; }).overrideAttrs (attr: rec { src = /path/to/your/local/batsched/source/repository; preConfigure = "rm -rf build"; }); my-custom-pybatsim = kapack.pybatsim.overridePythonAttrs(attr: rec { src = /path/to/your/local/pybatsim/source/repository; }); This Nix expression is very similar to what we have done and explained in :ref:`changing_batsim_dependencies`, so please refer to it for a detailed explanation of this snippet. A new Nix trick here is ``overridePythonAttrs``, which does exactly the same as ``overrideAttrs`` but with the additional dark magic to make it work with Python. We also added ``preConfigure = "rm -rf build";`` in the ``my-custom-batsched`` package definition, this line makes sure the Nix build starts from a clean ``build`` directory (this does **not** removes the ``build`` directory in your local repository that contain batsched's sources, the only directory deleted by this command is in the temporary copy done by Nix when it builds the package). You can also of course use a git repository as the package source instead of a directory in your local filesystem (once again, see :ref:`changing_batsim_dependencies` for details). Then, you can change the ``integration_tests`` attribute in :download:`default.nix <../default.nix>` to use your versions of batsched and pybatsim, as shown in the following diff: .. code-block:: diff buildInputs = with pkgs.python37Packages; [ - batsim batsched batexpe pkgs.redis - pybatsim pytest pytest_html pandas] ++ + batsim my-custom-batsched batexpe pkgs.redis + my-custom-pybatsim pytest pytest_html pandas] ++ pkgs.lib.optional doValgrindAnalysis [ pkgs.valgrind ]; Now, whenever you run the ``integration_tests`` (``nix-build -A integration_tests``) or enter a shell to do so (``nix-shell -A integration_tests``), the environment should use your custom versions of batsched and pybatsim. .. _Nix: https://nixos.org/nix/ .. _NUR-Kapack: https://github.com/oar-team/nur-kapack .. _Running Meson: https://mesonbuild.com/Running-Meson.html .. _Meson's built-in options: https://mesonbuild.com/Builtin-options.html .. _cgdb: https://cgdb.github.io/ .. _Nix set: https://nixos.wiki/wiki/Nix_Expression_Language#Types .. _Nix derivations: https://nixos.org/manual/nix/stable/#ssec-derivation .. _SHA-256: https://en.wikipedia.org/wiki/SHA-2 .. _batsched's git repository: https://framagit.org/batsim/batsched/ .. _pybatsim's git repository: https://gitlab.inria.fr/batsim/pybatsim .. _virtualenv: https://virtualenv.pypa.io/en/latest/