Parallel Python

Introduction

Sometimes it is necessary to run the same model with different parameters. This set of scripts explains how to parallelize processes using Bash and Python. A future version of the manual will include an explanation using specialized Python and C++ libraries.

Example Code

Suppose we want to run our script called parallel_python.py, which receives 3 input parameters. In this case, it prints them on screen. An example command would be:

python3 parallel_python.py 5 2 3

import sys
import socket

def main():
   # We check that there are 3 input parameters. Add 1 because arg[0] is the script name itself.
   if len(sys.argv) != 4:
      print("Usage: provide the correct number of input parameters")
      sys.exit()

   arg1 = sys.argv[1]
   arg2 = sys.argv[2]
   arg3 = sys.argv[3]
   print('Hostname: ', socket.gethostname(),' ', 'Arg1: ', arg1, ' ', 'Arg2: ', arg2, ' ', 'Arg3: ', arg3, ' ')

if __name__ == '__main__':
   main()

We will create a file called parallel.sh where we include the values passed as parameters. We can see that we create three arrays with (in this case) 2 values each, and we want to iterate over all of them. We also activate a virtual environment to ensure we have an isolated Python environment.

#!/bin/bash

ROUTE=~/tests/parallel_example
params1=(1 2)
params2=(a b)
params3=(A B)

for param1 in "${params1[@]}"
   do
   for param2 in "${params2[@]}"
      do
      for param3 in "${params3[@]}"
         do
         srun --mem=512M --ntasks=1 --cpus-per-task=20 --output="$ROUTE/%j_${param1}_${param2}_${param3}.out" bash -c "source $ROUTE/venv/bin/activate && python3 $ROUTE/parallel_python.py $param1 $param2 $param3 && deactivate" &
         done
      done
   done

Execution

Since this is a Bash script, it may not be executable at first. To fix this, we need to change the file execution permissions:

chmod +x parallel.sh

Then we only need to run it:

./parallel.sh

Once execution finishes, the output files will be visible in the same directory where the script was executed.