Simple Linux Utility for Resource Management

# IBM AIX User and Administrator Guide

## Overview

This document describes the unique features of SLURM on the IBM AIX computers with a Federation switch. SLURM support for AIX has been thoroughly tested, but we know of no AIX installations using SLURM after 2008. You should be familiar with the SLURM's mode of operation on Linux clusters before studying the relatively few differences in IBM system operation described in this document.

## User Tools

The normal set of SLURM user tools: srun, scancel, sinfo, smap, squeue and scontrol provide all of the expected services except support for job steps. While the srun command will launch the tasks of a job step on an IBM AIX system, it does not support use of the Federation switch or IBM's MPI. Job steps should be launched using IBM's poe command. This architecture insures proper operation of all IBM tools.

You will use srun to submit a batch script to SLURM. This script should contain one or more invocations of poe to launch the tasks. If you want to run a job interactively, just execute poe directly. Poe will recognize that it lacks a SLURM job allocation (the SLURM_JOB_ID environment variable will be missing) and create the SLURM allocation prior to launching tasks.

Each poe invocation (or SLURM job step) can have it's own network specification. For example one poe may use IP mode communications and the next use User Space (US) mode communications. This enhancement to normal poe functionality may be accomplished by setting the SLURM_NETWORK environment variable. The format of SLURM_NETWORK is "network.[protocol],[type],[usage],[mode]". For example "network.mpi,en0,shared,ip". See LoadLeveler documentation for more details.

## Checkpoint

SLURM supports checkpoint via poe. In order to enable checkpoint, the shell executing the poe command must itself be initiated with the environment variable CHECKPOINT=yes. One file is written for each node on which the job is executing, plus another for the script executing poe.a By default, the checkpoint files will be written to the current working directory of the job. Names and locations of these files can be controlled using the environment variables MP_CKPTFILE and MP_CKPTDIR. Use the squeue command to identify the job and job step of interest. To initiate a checkpoint in which the job step will continue execution, use the command:
scontrol check create job_id.step_id
To initiate a checkpoint in which the job step will terminate afterwards, use the command:
scontrol check vacate job_id.step_id

## System Administration

Three unique components are required to use SLURM on an IBM system.

1. The Federation switch plugin is required. This component is packaged with the SLURM distribution.
2. There is a process tracking kernel extension required. This is used to insure that all processes associated with a job are tracked. SLURM normally uses session ID and process group ID on Linux systems, but these mechanisms can not prevent user processes from establishing their own session or process group and thus "escape" from SLURM tracking. This kernel extension is not packaged with SLURM, but is available upon request.
3. The final component is a library that accepts poe library calls and performs actions in SLURM to satisfy these requests, such as launching tasks. This library is based upon IBM Confidential information and is not at this time available for distribution. Interested parties are welcome to pursue the possible distribution of this library with IBM and SLURM developers.

Until this last issue is resolved, use of SLURM on an IBM AIX system should not be viewed as a supported configuration (at least outside of LLNL, which established a contract with IBM for this purpose).

Last modified 5 October 2012