Multivac Maintenance

This section provides detailed instructions on how to maintain the cluster. This manual was prepared based on an existing guide, Installation of slurm, together with the official Slurm documentation, slurm workload manager.

How to add a new node

The following steps must be executed:

# ### MUNGE ###
# Install munge
sudo apt install munge
# Copy the munge key from iocex to the node
sudo scp root@iocex:/etc/munge/munge.key /etc/munge
# Configure permissions
sudo chown -R munge: /etc/munge/ /var/log/munge/ /var/lib/munge/ /run/munge/
sudo chmod 0700 /etc/munge/ /var/log/munge/ /var/lib/munge/ /run/munge/
# Copy /etc/systemd/system/multi-user.target.wants/munge.service to our node,
# ensuring munge waits for nslcd to be activated
sudo scp root@iocex:/etc/systemd/system/multi-user.target.wants/munge.service /etc/systemd/system/multi-user.target.wants/munge.service
# Reload systemd
sudo systemctl daemon-reload
# Enable munge and verify it works correctly
sudo systemctl enable munge
sudo systemctl start munge
# Check from iocex that it can access the node
sudo munge -n | unmunge
sudo munge -n | ssh nom_del_node_nou unmunge

# ### slurm ###
sudo apt install slurmd
# Create directories on the node
sudo mkdir /var/spool/slurmd
sudo chown slurm: /var/spool/slurmd
sudo chmod 755 /var/spool/slurmd
sudo mkdir /var/log/slurm/
sudo touch /var/log/slurm/slurmd.log
sudo chown -R slurm:slurm /var/log/slurm/slurmd.log

# Edit /etc/slurm/slurm.conf on iocex with the new node and copy it to all nodes, including the new one
nano /etc/slurm/slurm.conf
# Copy slurm.conf to nodeX
sudo scp root@i-iocex:/etc/slurm/slurm.conf /etc/slurm

# Copy cgroup.conf and slurmdbd.conf
sudo scp root@psi:/etc/slurm/cgroup.conf /etc/slurm
sudo scp root@i-iocex:/etc/slurm/slurmdbd.conf /etc/slurm

# Copy the service file from an existing node to ensure slurmd waits for munge
sudo scp root@psi:/etc/systemd/system/multi-user.target.wants/slurmd.service /etc/systemd/system/multi-user.target.wants/slurmd.service
# Reload systemd
sudo systemctl daemon-reload
# Enable slurmd.service
systemctl enable slurmd.service
systemctl start slurmd.service
# Check that the node is idle on iocex, otherwise restart slurmd.service
systemctl status slurmd.service
# If it is not healthy, restart munge and slurmd
systemctl status munge.service
systemctl restart munge.service
systemctl restart slurmd.service
# Change node state from down to idle
scontrol update NodeName=nodeX State=RESUME
# Reconfigure all nodes after changing slurm.conf
sudo scontrol reconfigure

How to check node status

From iocex, we can check node status with the sinfo command. If any node appears as down, we can reactivate it. We can also deactivate nodes when needed.

It is important to know that when a node fails, the reason can be found in /var/log/slurm/slurmd.log. There we can fix the issue (usually munge has been deactivated or slurmd has crashed if users attempted an invalid operation), and then reactivate the node once fixed. The order is: first reactivate munge, then slurmd.

# Activate node
sudo scontrol update NodeName=nodeX State=RESUME
# Deactivate node (for maintenance, for example)
sudo scontrol update NodeName=nodeX State=Down Reason='Manteniment'
# Command to reconfigure all nodes
sudo scontrol reconfigure

Recommendations

Users share a social queue, meaning everyone is aware of what is being executed and must use the cluster responsibly. If one person launches very long jobs, other users may not be able to launch theirs.

Custom scripts

There is a set of custom Bash scripts that simplify execution workflows. These scripts are available at the following paths:

multivac

Description: This is the main script. It reads a file with a .slurm extension, builds a temporary script that runs an sbatch command, and then deletes the temporary script.

  • Usage: multivac conf.slurm

  • Ruta: /usr/local/bin/multivac

mvac_crear_venv

Description: This script creates a venv in the folder specified when running the script.

  • Usage: mvac_crear_venv

  • Ruta: /usr/local/bin/mvac_crear_venv

mvac_deploy_jupyter

Description: This script creates a venv with Jupyter Notebook in the user’s home directory. It is required to run Jupyter remotely.

  • Usage: mvac_deploy_jupyter

  • Ruta: /usr/local/bin/mvac_deploy_jupyter

mvacinteract

Description: This script opens an SSH session to a specific node, reserving 2 cores and 512MB of RAM.

  • Usage: mvacinteract nodeX

  • Ruta: /usr/local/bin/mvacinteract

mvachtop

Description: This script opens an SSH session to a specific node and runs htop.

  • Usage: mvachtop nodeX

  • Ruta: /usr/local/bin/mvachtop

mvac_jupyter

Description: This script opens an SSH session and a JupyterHub connection on one of the available nodes. It reserves 4 cores and 3968M.

  • Usage: mvac_jupyter

  • Ruta: /usr/local/bin/mvacinteract

mvac_jupyter_remote

Description: This script opens an SSH session and a JupyterHub connection on one of the available nodes. It reserves 4 cores and 3968M. It allows running Jupyter in the background and exiting the terminal. Maximum duration is 24h. When finished, the job must be terminated manually; otherwise resources remain reserved until the 24h limit is reached.

  • Usage: mvac_jupyter_remote

  • Ruta: /usr/local/bin/mvac_jupyter_remote

sjob

Description: This script retrieves information about a finished job.

  • Usage: sjob

  • Ruta: /usr/local/bin/sjob

Execution history

In /var/log/slurm/job_completions there is a plain-text record of all executions run on Multivac.

Multivac documentation

This documentation is stored in a gitioc repository: Multivac manual.

It is written in rst format, and to deploy the documentation we need a venv with the following requirements.

alabaster==0.7.13
 Babel==2.13.1
 certifi==2023.11.17
 charset-normalizer==3.3.2
 docutils==0.20.1
 idna==3.6
 imagesize==1.4.1
 Jinja2==3.1.2
 MarkupSafe==2.1.3
 packaging==23.2
 Pygments==2.17.2
 requests==2.31.0
 snowballstemmer==2.2.0
 Sphinx==7.2.6
 sphinx-rtd-theme==2.0.0
 sphinxcontrib-applehelp==1.0.7
 sphinxcontrib-devhelp==1.0.5
 sphinxcontrib-htmlhelp==2.0.4
 sphinxcontrib-jquery==4.1
 sphinxcontrib-jsmath==1.0.1
 sphinxcontrib-qthelp==1.0.6
 sphinxcontrib-serializinghtml==1.1.9
 urllib3==2.1.0

To add new pages, create a .rst file and add its name to index.rst.

To deploy the documentation automatically, run the deploy_doc.sh script from the documentation folder.

Recomanacions

  • Users share a social queue, meaning everyone is aware of what is being executed and must use the cluster responsibly. If one person launches very long jobs, other users may not be able to launch theirs.

  • There are two important folders:

    • /etc/slurm/ This folder contains Slurm configuration files.

    • /var/log/slurm This folder contains Slurm logs.

To inspect why a node failed, we can run:

  • sudo journalctl -u slurmctld --no-pager -n 100 if checking from iocex.

  • sudo journalctl -u slurmd --no-pager -n 100 if checking from a node.

System improvements and upgrades

One proposed improvement is to enable the MariaDB database (already installed), change the JobCompType=jobcomp/filetxt field in slurm.conf, and store all execution data in the database. This would allow using sacct to see real script resource usage and better tune future executions.