File System model ----------------- Storage ~~~~~~~ The model chosen for the storage is to use a simple SimGrid host, linked to a compute node for local disk, or anywhere in the topology for centralized storage. The storage access latency and read/write bandwidth is modeled by two SimGrid links that attached the storage node to the rest of the platform. This model allows to simulate any kind of file system topology, from classical DFS, by adding a storage to every node of the platform, to centralized storage with a single large storage node, and including any mix of the two. Here is an example of storage node added to the Taurus platform that represents an HDD: .. code:: xml I/O transfer ~~~~~~~~~~~~ Now that the platform contains storage node, we need to model the I/O transfer between them. Batsim implementation of I/O transfer are based on the parallel task model described above. Every storage get a resource ID that identify them in the platform with a unique number, just like the computation resources. Using the storage ID and the normal resource ID, data movement is simply modeled by a matrix of communication involving computation node and storage nodes. The genericity of the model is ensured by keeping it outside of Batsim. Thus, the burden of maintaining the file system model is delegated to the user’s defined process that is connected to Batsim called the decision process, also called the scheduler. The scheduler uses dynamic job to model file system internal moves like DFS load balance, or data staging from PFS to Burst Buffers, but most of I/O transfer are triggered by applications. But the application model also need to be independent from the file system. So the application model that is defined by the Batsim job profiles, provided to Batsim with the workload file, needs to contain enough information for the scheduler to generate actual I/O transfer at execution time. Thus, the application model can be statically defined in the Batsim workload file, but the I/O that is triggered by this application is dynamically generated depending on the context by the scheduler. Concretely, when a job is submitted in the workload, Batsim informs the scheduler of this event with a job profile attached. This job profile contains information about the quantity of I/O that need to be done by the job, or any arbitrary information required by the file system model. Then, the scheduler can choose where the application will be allocated, and also, what are the I/O transfers that will occur during the execution of this application. Once the decision is taken, the scheduler send the job allocation to Batsim inside the job execution order as usual, but includes also an additional I/O job. This I/O job has its own allocation and the associated communication matrix that represent the transfer between the compute nodes of the jobs and the storages. Finally, Batsim will merge this I/O job profile with the normal job profile before to delegate the actual simulation to SimGrid. Full example ~~~~~~~~~~~~ .. todo:: Add more detailed to the example: - the platform file - the full workload file - the launch command Here is an example that illustrate this mechanism. Let’s assume that the ``job1`` was submitted to the scheduler with a request of 2 computing resources. The profile of the job called is defined by a Json object, ``job1_profile``. #. The scheduler receive the submission job from Batsim with this forwarded profile, where the quantity of I/O is just an information given to the scheduler (not read by Batsim): .. code:: json "job1_profile": { "type": "parallel_homogeneous_total", "cpu": 1e10, "com": 1e7, "io_reads": 8e8, "io_writes": 2e8 } #. The scheduler takes the decision to allocate this job on the resources 0 and 1. Then it decides, regarding its file system model, that each node will read half of the data from only one centralized storage (located at the resource id 10), but the nodes of the application will write on local their local disks (id 2 and 3). A new profile of type ``parallel`` is generated an submitted to Batsim with the name ``io_for_this_alloc_on_job1``. The kind job profile is composed of a computation matrix that may reflect I/O related computation (here set to 0), and a communication matrix that represents the aforementioned assumptions. Note that the direction of the transfer is from row to column, and that the indices of the matrix are mapped to the IO allocation, e.g. for the allocation of :math:`(0,1,2,3,10)` a value of :math:`4e8` on the 1st column and at the 5th row means that 400MB will be transferred from the resource 10 to the resource 0. .. code:: json "io_for_this_alloc_on_job1": { "type": "parallel", "cpu": [0 ,0 ,0 ,0 ,0] "com": [0 ,0 ,1e8,0 ,0 0 ,0 ,0 ,1e8,0 0 ,0 ,0 ,0 ,0 0 ,0 ,0 ,0 ,0 4e8,4e8,0 ,0 ,0] } #. The scheduler ask Batsim to execute the job with the given job allocation, and the additional I/O job. .. code:: json { "job_id":"08a582!1", "alloc":"0-1", "additional_io_job": { "alloc":"0-3,10", "profile": "io_for_this_alloc_on_job1" } } #. Batsim merges the 2 profiles, and generates a new job with IO matrix that is then sent to SimGrid in order to be simulated: .. code:: json { "alloc": "0-3,10", "cpu": [5e9,5e9,0 ,0 ,0], "com": [0 ,5e6,1e8,0 ,0 5e6,0 ,0 ,1e8,0 0 ,0 ,0 ,0 ,0 0 ,0 ,0 ,0 ,0 4e8,4e8,0 ,0 ,0] } Note the difference of allocation between the job itself and the IO that it generates. Batsim is capable to merge any interval set of resource allocation, even if only part of the job’s nodes are taking part in the IO transfer. This simple file system model is generic enough to simulate any centralized and decentralized file system, because it not assume any kind of I/O behavior. For example, it is possible to simulated hierarchical file system like a PFS with I/O nodes, or a multi-tiers storage setup with two different centralized file system, e.g. NFS and Lustre, or even a mix of DFS and PFS. It is fully dynamic: the I/O transfer inside the application are generated online by the decision process, which allows to take the I/O into account for any decision, from job allocation, to I/O gateway selection. Also, dynamic job can be created to model internal file system I/O transfer at any time during the simulation. Limitations and Evolutions ~~~~~~~~~~~~~~~~~~~~~~~~~~ The file system model described above has some limitations that are discuss here. First, the storage model is very simple and do not reflect the fine grain behavior of different kind of storage like HDD, SSD, of NVM. Also, The storage model not include disk size limitation enforcement, and even if this can be done the decision process, the contention behavior that it implies is not modeled. To overcome the storage model limitation, it would be possible to add a more realistic model into SimGrid, but it may induce large changes in the underneath contention model. The fact that the file system model is hold by the decision process increase flexibility and permits to the Batsim users to model any kind of behavior, but it has the drawback to lead to multiple implementation of the same model (one for each scheduler). This can be overcome by putting the file system model into an external process, which even more realistic, but it increase the maintenance cost and hinder the model genericity. Another limitation is the absence of cache effects in the model, either from the storage itself or from a second level like burst buffers. Also, any cluster file system have metadata servers are also not taken into account in our model. To allows to model fine grain behavior like the metadata server, or the cache effects, it requires to add new features inside Batsim, built on top of SimGrid directly, and thus putting the file system model inside Batsim.