Simple Linux Utility for Resource Management

# Maui Scheduler Integration Guide

## Overview

Maui configuration is quite complicated and is really beyond the scope of any documents we could supply with SLURM. The best resource for Maui configuration information is the online documents at Cluster Resources Inc.: http://www.clusterresources.com/products/maui/docs/mauiadmin.shtml.

Maui uses SLURM commands and a wiki interface to communicate. See the Wiki Interface Specification and Wiki Socket Protocol Description for more information.

## Configuration

First, download the Maui scheduler kit from their web site http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php. Note: maui-3.2.6p9 has been validated with SLURM, other versions should also work properly. We anticipate the Maui Scheduler to be upgraded to utilize a more extensive interface to Slurm in early 2007. The newer Maui Scheduler will be able to utilize a more ful featured interface to Slurm as descripted in the Moab Cluster Suite Integration Guide. This guide will be upgrade at that time.

### Maui configuration

Make sure that SLURM is installed and running before building Maui. Then build Maui from its source distribution. This is a two step process:

1. ./configure --with-key=42 --with-wiki
2. gmake

The key of 42 is arbitrary. You can use any value, but it will need to be a number no larger than 4,294,967,295 (2^32) and specify the same value as a SLURM configuration parameter described below. Maui developers have assured us the authentication key will eventually be set in a configuration file rather than at build time.

Update the Maui configuration file maui.conf (Copy the file maui-3.2.6p9/maui.cfg.dist to maui.conf). Add the following configuration parameters to maui.conf:

    RMCFG[host]       TYPE=WIKI
    RMPORT            7321            # selected port
    RMHOST            host
    RMAUTHTYPE[host]  CHECKSUM

host is the hostname where the SLURM controller is running. This must match the value of ControlMachine configured in slurm.conf. Note that localhost doesn't work. If you run Maui and SLURM on the same machine, you must specify the actual host name. The above example uses a TCP port number of 7321 for communications between SLURM and Maui, but you can pick any port that is available and accessible. You can also set a polling interval with

    RMPOLLINTERVAL  00:00:20

It may be desired to have Maui poll SLURM quite often -- in this case every 20 seconds. Note that a job submitted to an idle cluster will not be initiated until the Maui daemon polls SLURM and decides to make it run, so the value of RMPOLLINTERVAL should be set to a value appropriate for your site considering both the desired system responsiveness and the overhead of executing Maui daemons too frequently.

In order for Maui to be able to access your SLURM partition, you will need to define in maui.conf a partition with the same name as the SLURM partition(s). For example if nodes "linux[0-3]" are in SLURM partition "PartA", slurm.conf includes a line of this sort:

    PartitionName=PartA Default=yes Nodes=linux[0-3]

The add the corresponding lines to maui.cfg:

    PARTITIONMODE ON
    NODECFG[linux0]   PARTITION=PartA
    NODECFG[linux1]   PARTITION=PartA
    NODECFG[linux2]   PARTITION=PartA
    NODECFG[linux3]   PARTITION=PartA

Set the following environment variables and path:

    set path=(/root/MAUI/maui-3.2.6p9/bin $path)
    setenv MAUIHOMEDIR /root/MAUI/maui-3.2.6p9

### SLURM configuration

Set the slurm.conf scheduler parameters as follows:

    SchedulerType=sched/wiki
    SchedulerPort=7321
    SchedulerAuth=42 (for Slurm version 1.1 and earlier only)

In this case, "SchedulerAuth" has been set to 42, which was the authentication key specified when Maui was configured above. Just make sure the numbers match.

For SLURM version 1.2 or higher, the authentication key is stored in a file specific to the wiki-plugin named wiki.conf. This file should be protected from reading by users. It only needs to be readable by SlurmUser (as configured in slurm.conf) and only needs to exist on computers where the slurmctld daemon executes. More information about wiki.conf is available in a man page distributed with SLURM, although that includes a description of keywords presently only supported by the sched/wiki2 plugin for use with the Moab Scheduler.

SLURM version 2.0 and higher have internal scheduling capabilities that are not compatible with Maui.

1. Do not configure SLURM to use the "priority/multifactor" plugin as it would set job priorities which conflict with those set by Maui.
2. Do not use SLURM's reservation mechanism, but use that offered by Maui.
3. Do not use SLURM's resource limits as those may conflict with those managed by Maui.

The wiki.conf keywords currently supported by Maui include:

AuthKey is a DES based encryption key used to sign communications between SLURM and Maui or Moab. This use of this key is essential to insure that a user not build his own program to cancel other user's jobs in SLURM. This should be no more than 32-bit unsigned integer and match the the encryption key in Maui (--with-key on the configure line) or Moab (KEY parameter in the moab-private.cfg file). Note that SLURM's wiki plugin does not include a mechanism to submit new jobs, so even without this key nobody could run jobs as another user.

ExcludePartitions is used to identify partitions whose jobs are to be scheduled directly by SLURM rather than Maui. These jobs will be scheduled on a First-Come-First-Served basis. This may provide faster response times than Maui scheduling. Maui will account for and report the jobs, but their initiation will be outside of Maui's control. Note that Maui controls for resource reservation, fair share scheduling, etc. will not apply to the initiation of these jobs. If more than one partition is to be scheduled directly by Slurm, use a comma separator between their names.

HidePartitionJobs identifies partitions whose jobs are not to be reported to Maui. These jobs will not be accounted for or otherwise visible to Maui. Any partitions listed here must also be listed in ExcludePartitions. If more than one partition is to have its jobs hidden, use a comma separator between their names.

Here is a sample wiki.conf file

    # wiki.conf
    # SLURM's wiki plugin configuration file
    #
    # Matches Maui's --with-key configuration parameter
    AuthKey=42
    #
    # SLURM to directly schedule "debug" partition
    # and hide the jobs from Maui
    ExcludePartitions=debug
    HidePartitionJobs=debug

Last modified 8 May 2009