DQS Ref. Manual

COMPUTATIONAL SCIENCE AND INFORMATION TECHNOLOGY

DISTRIBUTED QUEUEING SYSTEM - 3.3.2

##
REFERENCE MANUAL

#
December 12, 2000

Attention
Note that all default DQS-3.3.2 ancillary commands referenced have the digits "332" appended to the command. That is, if this document cites "qsub", use the default "qsub332" instead.

Table of Contents

Introduction

qalter - Alter a Batch Job

qconf - Queue configuration tasks

qdel - Delete on or more jobs from the system

qhold - Place a hold on one or more jobs to defer scheduling

qidle - X Windows console monitor

qmod - Modify the state of a queue

qmove - Move jobs from one cell to another

qrls - Release hold(s) on one or more jobs

qstat - Display status of selected queues and jobs

qsub - Submit a job to the queuing system

the daemons

Glossary

## Introduction

This manual is based on the draft POSIX Specification
"Part 2: Shell and Utilities -- Amendment: Batch Environment".
DQS 3.3.2 possesses many extensions and refinements to that basic specification. Refer to the DQS 3.3.2 Appendix and the POSIX Specification for a comparison of the differences.

##
the utilities

## qalter- Alter a Batch Job

qalter [-a date_time] [-A account_string] [-cell cell_name] [-clean] [-cwd] [-C directive_prefix] [-e path_name] [-h hold_list] [-hard] [-help] [-j join_list] [-jid job_identifier] [-l resource_list] [-m mail_options] [-master parallel_destination_identifier...] [-M mail_list] [-N name] [-notify] [-o path_name] [-p priority] [-par paradigm] [-passwd] [-passwd_file filename] [-q destination_identifier_list] [-r y|n] [-reauth seconds] [-soft] [-S path_name_list] [-v variable_list] [-verify] [-V] [script file]

Description

The attributes of a job are altered by a request to the qmaster. The "qalter" ancillary utility is a user-accessible batch client that requests the alteration of the attributes of one or more jobs.

The "qalter" ancillary utility alters the attributes of those jobs, and only those jobs, for which a job_identifier is presented to the utility.

The "qalter" utility alters the attributes of jobs in the order in which the job_identifiers are presented to the utility.

If the "qalter" utility fails to process a job_identifier successfully, the utility will proceed to process the remaining job_identifiers, if any.

For each job_identifier for which the "qalter" utility succeeds, each attribute of the identified job will be altered as indicated by all of the options presented to the utility.

For each identified job for which the "qalter" utility fails the utility will not alter any attribute of the job.

For each job that the "qalter" utility processes, the utility will not modify any attribute other than those required by the options and option arguments presented to the utility.

The "qalter" utility alters jobs by sending a Modify Job Request to the qmaster. At the time the "qalter" utility exits, it will have modified the job corresponding to each successfully processed job_identifier.

Jobs in the RUNNING state cannot be modified by "qalter".

Options

-a date time
Redefines the time at which the job becomes eligible for execution. The "date_time" option argument is described in the Glossary.

The "qalter" utility sets the Execution_Time attribute of the job to the number of seconds since Epoch that is equivalent to the local times expressed by the value of the "date_time" option argument. Specifying a "date_time" option argument that represents a time (number of seconds since Epoch) earlier than the time at which the utility exists will have the same effect on job execution as if the "-a" option had not been presented to the utility.

-A account string
Redefines the account to which the resource consumption of the job should be charged. The "account_string" option argument is described in the Glossary.

-cell cell_name
The cell to which the "qalter" request is to be sent.

-clean
Causes all elements of the job to be reset to the initial default status prior to applying any modifications (if any) appearing in this specific "qalter" command.

-cwd
Redefines the current working directory to be used when the job begins execution

-C directive_prefix
The directive prefix consists of two ASCII characters which when appearing in the first two bytes of a script line indicate that what follows is a DQS command.

-e path_name
Redefines the path to be used for the standard error stream of the job.

If the "path_name" option argument constitutes an absolute path name, the "qalter" utility will set the Error_Path attribute of the job to the value of the path_name option argument including the hostname, if present.

If the "path_name" option argument constitutes a relative path name and no hostname is specified, the "qalter" utility will set the Error_Path attribute of the job to the value of the absolute path name. The absolute path name is derived from the relative path name by expanding the "path_name" option argument relative to the current working directory of the process that executed the "qalter" utility.

If the "path_name" option argument constitutes a relative path name and a hostname is specified, the "qalter" utility will set the Error_Path attribute of the job to the value of the option argument without expansion.

If the "path_name" option argument does not include a hostname, the "qalter" utility will prefix the path name in the Error_Path attribute with "hostname:", where hostname is the name of the host upon which the "qalter" utility is being executed.

The default file name for standard error output has the following form:

"job_name.e.sequence_number.pid"

If the "-e" option possess no option argument, the utility will clear the Error_Path attribute of the job so that the default standard error output will be used when the job is executed.

-h hold list
Redefines the types of holds, if any, on the job. The "qalter -h" will accept a value for the "hold_list" option argument that is a string of alphanumeric characters in the portable character set. The "hold_list" option argument is described in the Glossary.

-hard
Signifies that all resource requirements following in the "qalter" command will be hard requirements and must be satisfied in full before a job can be scheduled.

-help
Prints a listing of all options for qalter.

-j join list
Redefines which streams of the job are to be merged. The "qalter -j" option will accept a value for the "join_list" option argument that is a string of alphanumeric characters in the portable character set. The "join_list" option argument is described in the Glossary.

The only characters permitted in this implementation are "y" or "n". Specifies whether the standard error stream of the job is merged into the standard output stream. The meaning of the "-j" option is different from the standard.

-jid job_identifier
Defines the job identifier on which the "qalter" command will act.

-l resource list
Redefines the resources that are allowed or required by the job. The "resource_list" option argument is described in the Glossary.

-m mail options
Redefines the points in the execution of the job at which the server is to send mail about a change in the state of the job.

The "mail_options" option argument is described in the Glossary.

-master parallel destination identifier
Redefines the queue to serve as a master queue for a parallel job. The "parallel_destination_identifier" option argument is described in the Glossary.

-M mail_list (Default - the login name of the user submitting the job)
Redefines the list of users to which the server that executes the job is to send mail, if the server sends mail about the job.

mail_address[,mail_address,....]

The interpretation of mail_address is implementation defined.

-notify
Redefines the Notify attribute for a job. This attribute, when TRUE causes the batch server to send "warning" signals to a running job prior to sending the signals themselves. If a SIGSTOP is pending the job will receive a SIGUSR1 several seconds before the SIGSTOP. If a SIGKILL is pending the job will receive a SIGUSR2 several seconds before the SIGKILL. The amount of time delay is controlled by the "notify" parameter in each queue configuration.

-N name
Redefines the name of a job.

The "qalter" utility will set the Job_Name attribute of the job to the value of the name option, truncated to eight characters, if necessary.

If the "-N" option is not presented to the "qalter" utility, the utility will set the Job_Name attribute of the job to the name of the script -name argument after any directory "path-name" has been removed from the script-name.

If the "-N" option is not presented to the "qalter" utility, and the script is read from standard input, the utility will set the job-name attribute of the job to the value STDIN.

-o path_name
Redefines the path for standard output of the job.

If the "path_name" option argument constitutes an absolute path name, the "qalter" utility will set the Output_Path attribute of the job to the value of the "path_name" option argument including the hostname, if present.

If the "path_name" option argument constitutes a relative path name and no hostname is specified, the "qalter" utility will set the Output_Path attribute of the job to the value of the absolute path name. The absolute path name is derived from the relative path name by expanding the "path_name" option argument relative to the current working directory of the process that executed the "qalter" utility.

If the "path_name" option argument constitutes a relative path name and a hostname is specified, the "qalter" utility will set the Output_Path attribute of the job to the value of the option argument without expansion.

If the "path_name" option argument does not include a hostname, the "qalter" utility will prefix the path name in the Output_Path attribute with "hostname:", where hostname is the name of the host upon which the "qalter" utility is being executed.

The default file name for standard output has the following form:

job_name.o.sequence_number.pid

If the "-o" option possess no option argument, the utility will clear the Output_Path attribute of the job so that the default standard error output will be used when the job is executed.

-p priority
Redefines the priority of the job.

The "qalter" utility will accept a value for the priority option argument that conforms to the syntax for signed decimal integers, and which is not less than -1024 and not greater than 1023.

-par parallel paradigm
DQS can process, monitor and initiate parallel jobs using one of several message passing system paradigms. This option redefines the paradigm to use for a job. The "parallel_paradigm" option argument is described in the Glossary.

-passwd
Specifies that a password list will be requested for AFS and/or DFS re-authentication. If given, DQS will utilize the password list to authenticate the process prior to executing the job, so that re-authentication will be unnecessary.

-passwd_file filename
Redefines the name of the file containing the AFS passwords. (See AFS support in the User Guide.)

-q destination identifier
Resets the destination identifier list, i.e. queue name(s), for the job. The "destination_identifier" option argument is described in the Glossary.

-r y | n
Redefines whether the job is Rerunable.

If the value of the option argument is 'y' the "qalter" utility will set the Rerunable attribute of the job to TRUE.

If the value of the option argument is 'n' the "qalter" utility will set the Rerunable attribute of the job to FALSE.

The "qalter" utility will consider it an error if any character other than 'y' or 'n' is specified in the option argument.

-reauth seconds
Specifies the time interval for AFS/DFS re-authentication.

-soft
Signifies that all resource requirements following in the "qalter" command will be soft requirements and will be filled on a "as available" basis before a job can be scheduled.

-S path_name_list
Redefines the shell that interprets the script at the destination system.

The "qalter" utility will accept a "path_name_list" option argument that conforms to the following syntax:

path-name[@host][,path-name[@host],...]

The "qalter" utility will accept only one path name that is missing a corresponding host name. The "qalter" utility will allow only one path name per named host.

The "qalter" utility will add a value to the Shell_Path_List attribute of the job for each entry in the "path_name_list" option argument.

If the "-S" option has no arguments the utility will set the path-list attribute of the job to the null string.

-v variable list
Redefines the environment variables to be exported to the job.

The "variable_list" option argument is described in the Glossary.

If the -v option is presented to the "qalter" utility, the utility will also add to the "environment_variable_list" attribute of the job every variable named in the "variable_list" option argument and, optionally, values of specified variables.

-verify
This does not affect any job attributes. It causes the qalter utility to send the contents of the "qalter" command to standard out for verification by the user.

-V
Specifies that all environment active within the dqs_execd at the time it initiates the job be exported to the job.

Operands

Script File

The path to the script of the job. The "Script_File" operand argument is described in the Glossary.

If the "qalter" utility is presented with a script file it will replace the job script file originally submitted with the "qsub" command. The "qalter" utility scans the script file interpreting lines flagged with the DQS_DELIMETER pair of characters. All command options in the "qalter" command line and the script file are processed and replace or modify job attributes for the job.

A script can contain directives to the "qalter" utility.

The "qalter" utility will scan the lines of the script for directives until the first line that begins with a string other than the directive string; if directives occur on subsequent lines, the utility will ignore those directives.
go to toc

## qconf- Queue configuration tasks

qconf [-ac complex_name] [-acons consumable_name] [-ah server_list] [-am user_list] [-ao user_list ] [-aq [queue_template]] [-au user_list | name_list] [-Aq filename] [-cell cell_name] [-cq destination_id_list] [-dc complex_name] [-dcons consumable_name] [-dh server_list] [-dm user_list] [-do user_list] [-dq destination_id_list] [-du user_list | name_list] [-help] [-mc complex_name] [-mcons consumable_name] [-Mq file_name] [-sc complex_list] [-scl] [-sconf [cell]] [-sconsl] [-scons consumable_name] [-sh] [-sm] [-so] [-sq [destination_id_list]] [-sql] [-su name] [-sul] [-srl]

Description

The "qconf" utility provides the user interface for managing the queues which make up a DQS environment.

The "qconf" may be utilized by managers, operators and users with increasing levels of restrictions for each class of user. The DQS manager has unrestricted control over queue management.

Options

-ac complex_name
Adds a complex to the DQS environment. Complex entries contain one or more resources which may be requested by jobs submitted to the system. The entries will conform to the following three forms:

string
string =string
string =integer quantity

There are no reserved words in complex lists. Thus the "string" may contain any combinations of alphanumeric characters and be up to 1023 bytes in length.

This option requires root/manager privileges.

-acons consumable_name
Add a consumable to the DQS environment Consumable list. Each consumable is described by a four-line entry in an internal file with the consumable_name. An entry:

Consumable consumable_name
Available = <the amount of resources available>
Consume_by <quantum by which resource is reduced by a request>
Current = <currently available resources>

The Current value starts out with the available value and is reduced or increased as resources are used and returned by executing jobs.

This option requires root/manager privileges.

-ah server_list
Adds one or more hosts to the DQS host list. A host must reside in the DQS list to process any other DQS commands. The host on which the qmaster is running is automatically added to the host list. In addition host names are added when they appear in a newly created queue configuration. The host-name_list will conform to the following:

hostname[,hostname,..]

This option requires root/manager privileges.

-am user list
Adds one or more managers to the manager list.

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-ao user list
Adds one or more operators to the operator list

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-aq [queue_template]
Add A NEW QUEUE
Retrieves a default queue configuration, or the optional "queue_template" and executes the editor specified by the EDITOR environment variable to permit the customization of the queue configuration. As a minimum, the new queue must be assigned a unique queue name and a host.

Upon exit from the editor the queue is registered with the qmaster.

This option requires root/manager privileges.

-au user list| name list
Adds one or more users to the DQS access list (ACL). The user is used for "qusage" authentication.

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-Aq filename
Add New Queue From File
Adds a new queue description from an ASCII file. The default queue template, "generic_queue" may be found in the "common_dir" and used as a basis for preparation of the queue configuration file.

This option requires root/manager privileges.

-cell cell_name
Specifies the cell to which the "qconf" request will be Sent.

-cq destination identifier
Clean Queue(s)
Cleans out residual information from a queue where a job has been completed but for some reason the system has retained the job in the queue. This step is still necessary in DQS 3.3.2 due to some cases which the qmaster cannot verify job completion. It most often occurs when the DQS manager is starting and terminating the qmaster and dqs_execd processes with a kill -9 while doing system testing. The "destination_identifier" option argument is described in the Glossary.

This option requires root/manager privileges.

-dc complex_name
Deletes the named complex.

This option requires root/manager privileges.

-dcons consumable_name
Deletes the named Consumable.

This option requires root/manager privileges.

-dh server_list
Deletes one or more hosts from the DQS host list. The host-name_list will conform to the following:

hostname[,hostname,..]

This option requires root/manager privileges.

-dm user list
Deletes one or more managers from the manager list.

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-do user list
Deletes one or more operators from the operator list.

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-dq destination identifier
Removes the specified queue(s).

The "destination_identifier" option argument is described in the Glossary.

This option requires root/manager privileges.

-du user list
Deletes one or more users from the DQS access list (ACL).

The "user_list" option argument is described in the Glossary.

This option requires root/manager privileges.

-help
Sends a listing of the qconf options to standard output

-mc complex_name
Modify Complex
Invokes the DQS editor and displays the named complex. When the editor is closed the complex is updated by the qmaster.

This option requires root/manager privileges.

-mcons consumable_name
Modify Consumable
Invokes the editor which displays the entry for the named consumable. When the editor is closed the consumable values are updated. This action will be used in subsequent passes of the job scheduler however no RUNNING jobs will be affected.

This option requires root/manager privileges.

-mq queue_name
Modify Queue Configuration
Invokes the editor which displays the entry for the named queue. When the editor is closed the queue configuration is updated. This action will be used in subsequent passes of the job scheduler however no RUNNING jobs will be affected.

Note: the "qname" and "qhostname" fields CANNOT be changed with this option. Modification of these fields requires deletion of the queue definition with the "-dq queue_name" option.

This option requires root/manager privileges.

-Mq filename
Modify Queue Configuration using named file
The editor is invoked to apply the named file to the queue configuration. These modifiactions are then used in subsequent passes of the job scheduler. No running jobs will be affected.

To create the new file for use with this option, copy generic_queue from the DQS directory tree under conf/qmaster/qmaster_hostname/common_dir to a new file and make the appropriate changes. Then execute qconf -Mq new_file to make the changes take affect.

Note: the "qname" and "qhostname" fields CANNOT be changed with this option. Modification of these fields requires deletion of the queue definition with the "-dq queue_name" option.

This option requires root/manager privileges.

-scl
Show Complex List Names

-sc complex_list
Show complexes

-sconf [cell]
Show master configuration for cell

-sconsl
Shown the names of the consumables

-scons consumable_name
Shown the named consumable entry

-sh
Show Hosts

-sm
Show Managers

-so
Show Operators

-sq [destination identifier]
Show queue configurations

The "destination_identifier" option argument is described in the Glossary.

-sql
Show list of "destination identifiers" (queue names)

-su name list
Show users

The "name_list" option argument is described in the Glossary.

-sul
Show User Lists

-srl
Show Resolve List
go to toc

## qdel- Delete on or more jobs from the system

qdel [-cell cell_name] [-f] [-help ] [-verify] job_identifier_list

Description

A job is deleted by sending a request to the batch server that manages the job. A job that has been deleted is no longer subject to management by batch services. The "qdel" utility is user-accessible client of batch services that requests the deletion of one or more jobs.

A "qdel" utility requests a batch server to delete those jobs for which a job_identifier appears in the "job_identifier_list".

The "qdel" utility deletes the specified jobs in the order in which their job_identifiers appear in the "job_identifier_list".

If the "qdel" utility fails to process any job_identifier successfully, the utility will proceed to process the remaining job_identifiers, if any.

The "qdel" utility deletes each job by sending a Delete Job Request to the qmaster.

The "qdel" utility will not exit until the job corresponding to each successfully processed job_identifier has been deleted.

Options

-cell cell_name
The cell to which the "qdel" request is sent.

-f
Attempt to force a deletion even if the job has been handed to a host that cannot be reached. This does not guarantee that running processes are destroyed.

-help
Prints a listing of all options

-verify
Causes DQS to print the request to standard out. no action is taken.

Operands

job identifier list
Specifies the job number(s) of the jobs which are to be deleted. The job numbers were assigned by the batch server when the job was submitted.

The "job_identifier_list" operand argument is described in Glossary.
go to toc

## qhold- Place a hold on one or more jobs to defer scheduling

qhold [-h hold_list] [-help] [-verify] [job_identifier_list]

Description

The "qhold" utility provides a means for a user to place one or more types of holds on identified job(s). A job that has one or more holds is not eligible for execution.

"qhold" places holds on jobs in the order in which their identifiers are presented to the utility, by sending a Hold Job Request to the batch server that manages the job.

If "qhold" fails to process any identified job successfully, the utility will proceed to process the remaining job identifiers, if any.

"qhold" will not exit until holds have been placed on the job corresponding to each successfully processed job identifier.

Options

-h hold list
Defines the types of holds, if any, on the job. The "qhold -h" will accept a value for the "hold_list" option argument that is a string of alphanumeric characters in the portable character set. The "hold_list" option argument is described in the Glossary.

-help
Prints a listing of all options

-verify
Causes DQS to print the request to standard out. No action is taken.

Operands

job identifier list
Specifies a list of one or more job sequence numbers that are affected by the "qhold" utility.

If the application presents the server portion of a job identifier to the "qhold" utility, the utility will send the Hold Job Request to the specified server.

The "job_identifier_list" operand argument is described in Glossary.
go to toc

## qidle- X Windows console monitor

qidle [-d display] [-s seconds] [-g geometry]

Description

"qidle" monitors the local X server for activity, if a user is interactively using that host's console, the local DQS queue(s) is (are) automatically suspended. If the server is idle for a certain period of time, "qidle" re-enables the queue causing the suspended job to resume. The sleep button in the X-windows "qidle" icon allows the console user to modify the length of time for which the queue is suspended.

Options

-d display
Name of the local X display. "qidle" will use "unix:0" as a default display unless the environment variable "DISPLAY" is present. The "-d" option will override these choices for the display name.

-s seconds
Length of time that the queue is suspended before retrying an enable of the queue

-g geometry
X windows geometry for the "qidle" window, in the form used to describe location length and width of a window for X-windows applications.
go to toc

## qmod- Modify the state of a queue

qmod [-cell cell_name] [-s or -us or -soc] [-d] [-e] [-f] [-help] [-verify] [-q destination_identifier] or [-l resource_list]

Description

The "qmod" command enables users classified as owners of a workstation to modify the state of a DQS queue for their machine. A manager, operator or the super-user can execute "qmod" for any queue in their system.

Options

-cell cell_name
The cell to which the request is sent.

-d
Disables the queue

-e
Enables the queue

-f
Force a modification action for the queue despite the apparent current state of the queue. For example if a queue appears to be suspended but the job execution seems to be continuing the manager can force a suspend operation which will send a SIGSTOP to the job.

-help
Send a list of all options to standard output.

-l resource list
Redefines the resources that are allowed or required by the job.

The "resource_list" option argument is described in the Glossary.

-q destination identifier
Name of the queue(s) to be modified.

The "destination_identifier" option argument is described in the Glossary.

-s
Suspends the queue and any jobs in execution.

-soc
Suspends the queue on completion of currently executing job.

-us
Removes the suspend status from the queue and restarts any jobs which were running when the queue was suspended.

-verify
Send the contents of the "qmod" request to standard output. No actions are taken.

-xsoc
Remove the suspend on completion mandate from the queue.
go to toc

## qmove- Move jobs from one cell to another

qmove [-cell cell_name] job_identifier_list

Description

Move a job to <cell_name>. This function is a small subset of the POSIX "qmove" command. If a job must be moved from one queue to another within a cell, the "qalter" utility should be used.

Options

-cell cell_name
The destination cell for the moved jobs. If this option is absent, the cell within which this utility is executing will become the destination.

Operands

job identifier list
Specifies a list of one or more job sequence numbers that are affected by the qmove command.

To use this utility, a user must be authenticated in both cells as well as being the owner of the jobs to be moved.

The "job_identifier_list" operand argument is described in Glossary.
go to toc

## qrls- Release hold(s) on one or more jobs

qrls [-h hold_list] job_identifier

Description

A job might have one or more holds which prevent the job from executing. A job from which all the holds have been removed becomes eligible for execution, and is said to have been released.

A job hold is removed by sending a Release Job Request to the qmaster. The "qrls" utility is a user-accessible client of a batch server that requests holds to be removed from one or more jobs.

The "qrls" utility will remove one or more holds from those jobs for which a job_identifier is presented to the utility, in the order in which the job_identifiers are presented to the utility.

If the "qrls" utility fails to process a job_identifier successfully, the utility will proceed to process the remaining job_identifiers, if any.

The "qrls" utility will not exit until the holds have been removed from the job corresponding to each successfully processed job_identifier.

Options

-h hold list
Specifies the types of holds to be removed from the job.

The "hold_list" option argument is described in the Glossary.

Operands

job identifier list
Specifies a list of one or more job sequence numbers that are affected by the "qrls" command.

If the application presents the server portion of a job identifier to the "qrls" utility, the utility will send the Hold Job Request to the specified server.

The "job_identifier_list" operand argument is described in Glossary.
go to toc

## qstat- Display status of selected queues and jobs

qstat [-cell cell_name] [-ext] [-f] [-help] [-l resource_list] [-q queue-name] [-u user_list] [-verify]

Description

The status of a job, queue, or batch server is obtained by a request to the qmaster. The "qstat" utility is a user accessible batch client that requests the status of one or more jobs, queues, or servers, and writes the status information to standard output.

For each successfully processed destination, the "qstat" utility will display information about the corresponding queue.

For each successfully processed server name, the "qstat" utility will display information about the corresponding server.

For each successfully processed resource request, the "qstat" utility will display information about the queues that possess resources that match those specified.

For each successfully processed user id, the "qstat" utility will display information about the jobs owned by that user.

The "qstat" utility acquires all job and queue status information in a single request to the batch server/qmaster. Server information is displayed for each queue with one of the following status symbols

RUNNING - The queue is in the running state or a job is RUNNING.

ENABLED - The queue is enabled for scheduling jobs.

DISABLED - The queue is disabled and will not be included in scheduling.

SUBORDINATED - The queue has been subordinated by a "superior" queue.

SUSPENDED - The queue any running jobs in that queue are suspended.

UNDEFINED - The queue's dqs_execd has not reported in to the qmaster.

ALARM - The load average threshold for the queue has been exceeded.

Options

-cell cell_name
The cell to which the "qstat" request is to be forwarded.

-ext
Specifies that an extended description of queues and jobs are to be sent to standard output

-f
Specifies that a "full format" output format is to be used. This format shows the full queue status and list of jobs in the queues, but not the queue and job descriptions provided with the "-ext" option.

-help
Sends a list of all "qstat" options to standard output

-l resource list
This option allows users to define which resources queues must satisfy in order to show up under the "qstat" listing.

The "resource_list" option argument is described in the Glossary.

Examples:

qstat -l ibms
displays all the queues of group "ibms"

qstat -l mem.gt.256
displays all the queues having mem > 256

-q destination identifier
Specify the destination identifier i.e. queue name(s), that "qstat" should pertain to.

The "destination_identifier" option argument is described in the Glossary.

-u user list
Specify the user(s) that "qstat" should pertain to.

The "user_list" option argument is described in the Glossary.

-verify
Causes the contents of the "qstat" command to be sent to standard output, no status is requested or displayed.

Standard Output

If the -f option is not specified, the utility will display the following items on a single line in the stated order, with white space separators, for each successfully processed operand:

-- queue name

-- job identifier

-- job name

-- job owner

-- job state

-- job location

If the -f option is specified, the utility will display the following items for each successfully processed operand:

-- queue name

-- type of queue (execution or routing )

-- maximum number of concurrent jobs permitted

-- total number of jobs in queue

-- enabled or disabled status of the queue

-- started or stopped status of the queue

-- job identifier

-- job name

-- job owner

-- job state

-- job location

-- why a pending job has not been scheduled
go to toc

## qsub- Submit a job to the queuing system

qsub [-a date_time] [-A account_string] [-cell cell_name] [-cwd] [-C directive_prefix] [-e path_name] [-F] [-h] [-hard] [-help ] [-hold_jid jid[,jid,...]] [-j y | n] [-l resource_list] [-m mail_options] [-master parallel_destination_identifier...] [-M mail_list] [-N name] [-notify ] [-o path_name] [-p priority] [-par paradigm] [-passwd] [-passwd_file filename] [-q destination_identifier_list] [-r y|n] [-reauth seconds] [-soft] [-S path_name_list] [-v variable_list] [-verify] [-V] [script file]

Description

To submit a script is to create a job that executes the script. A script is submitted by a request to a batch server. The "qsub" utility is a user accessible batch client that submits a script.

Upon successful completion, the "qsub" utility shall have created a job that will execute the submitted script.

The "qsub" utility shall submit a script by sending a Queue Job Request to a batch server.

The "qsub" utility shall place the value of the following environment variables in the Variable-List attributes of the job:
HOME LOGNAME PATH MAIL SHELL TZ
The name of the environment variable shall be the current name prefixed with the string "DQS_O_".

In addition to he variables described above, the "qsub" utility shall add the following variables with the indicated values to the variable list:

-- DQS_O_WORKDIR The absolute path of the current working directory of the "qsub" utility process.

-- DQS_O_HOST The name of the host on which the "qsub" utility is running.

Summary of DQS environment variables

Variable Name | Value at qsub Time
DQS_O_HOME | $HOME
DQS_O_HOST | client host name
DQS_O_LOGNAME | $LOGNAME
DQS_O_PATH | $PATH
DQS_O_MAIL | $MAIL
DQS_O_SHELL | $SHELL
DQS_O_TZ | $TZ
DQS_O_WORKDIR | current working directory

Options

-a date time
Defines the time at which the job becomes eligible for execution.

The "date_time" option argument is described in the Glossary..

The "qsub" utility sets the Execution_Time attribute of the job to the number of seconds since Epoch that is equivalent to the local times expressed by the value of the "date_time" option argument. Specifying a date_time option argument that represents a time (number of seconds since Epoch) earlier than the time at which the utility exists will have the same effect on job execution as if the "-a" option had not been presented to the utility.

-A account string
Defines the account to which the resource consumption of the job should be charged.

The "account_string" option argument is described in the Glossary.

-cell cell_name
The cell to which the "qsub" request is sent.

-cwd
Declares that the job should begin execution in the current working directory, rather than in the user's home directory. Obviously this directory must exist on all potential hosts. Inside your job script, you can, of course, set the current directory using the "cd" command, again with the caveat that the named directory must exist on any potential execution host.
Obeying "-cwd" takes place as follows:

1. If execution takes place on the same machine that submitted the job, just "cd" to the submitting directory.
2. If executing on a different machine, then
   1. attempt to automount the submitting directory
   2. use a local directory with the exact same absolute path as the submitting directory
3. If neither option of #2 works, the job silently fails.

-C directive_prefix
The directive prefix consists of two ASCII characters which when appearing in the first two bytes of a script line indicate that what follows is a DQS command. Default is "#?"

-e path_name
Defines the path to be used for the standard error stream of the job.

If the "path_name" option argument constitutes an absolute path name, the "qsub" utility will set the Error_Path attribute of the job to the value of the "path_name" option argument including the hostname, if present.

If the "path_name" option argument constitutes a relative path name and no hostname is specified, the "qsub" utility will set the Error_Path attribute of the job to the value of the absolute path name. The absolute path name is derived from the relative path name by expanding the "path_name" option argument relative to the current working directory of the process that executed the "qsub" utility.

If the "path_name" option argument constitutes a relative path name and a hostname is specified, the "qsub" utility will set the Error_Path attribute of the job to the value of the option argument without expansion.

If the "path_name" option argument does not include a hostname, the "qsub" utility will prefix the path name in the Error_Path attribute with "hostname:", where hostname is the name of the host upon which the "qsub" utility is being executed.

The default file name for standard error output has the following form:

"job_name.e.sequence_number.pid"

If the "-e" option possess no option argument, the utility will clear the Error_Path attribute of the job so that the default standard error output will be used when the job is executed.

-F
Forces the submission of the job to the queuing system despite the fact that a requested resource (-l option) does not exist in any queue at the time of job submission. This is provided where the user anticipates a resource to be introduced into the system at a later time, and wishes to queue their job to wait for that resource.

-h
Place a "USER" hold on the job. A job that has one or more holds is not eligible for execution. See the "qhold" command.

-hard
Signifies that all resource requirements following in the "qsub" command will be hard requirements and must be satisfied in full before a job can be scheduled.

-help
Prints a listing of all options for "qsub".

-hold_jid jid[,jid,...]
Do not schedule this job until all the cited "jid's" finish.

-j join list

Redefines which streams of the job are to be merged. The "qsub -j" option will accept a value for the "join_list" option argument that is a string of alphanumeric characters in the portable character set. The "join_list" option argument is described in the Glossary.

The only characters permitted in this implementation are "y" or "n". Specifies whether the standard error stream of the job is merged into the standard output stream. The meaning of the "-j" option is different from the standard.

If the value of the option argument is 'y' the Join_Path attribute of the job is set to TRUE,
if the value is 'n' the Join_Path attribute of the job is set to FALSE.

If both the "-j y" and the "-e" options are specified, the batch server will set but ignore the Error_Path attribute.

If the "-j" option is not presented to the -"qsub" utility, the utility will set the value of the Join_Path attribute to FALSE.

-l resource list
Defines the resources that are allowed or required by the job.

The "resource_list" option argument is described in the Glossary.

-m mail options
Defines the points in the execution of the job at which the server is to send mail about a change in the state of the job.

The "mail_options" option argument is described in the Glossary.

-master parallel destination identifier
Defines the queue to serve as a master queue for a parallel job.

The "parallel_destination_identifier" option argument is described in the Glossary.

-M mail_list
Defines the list of users to which the server that executes the job is to send mail, if the server sends mail about the job.

mail_address[,mail_address,....]
(Default- the login name of the user submitting the job)

NOTE: The interpretation of mail_address is implementation defined.

-N name
Defines the name of the job. The name can be any printable set of characters up to 8 chars long. Names in excess of 8 characters will be truncated.

The "qsub" utility will set the Job_Name attribute of the job to the value of the name option , truncated to eight characters, if necessary.

If the "-N" option is not presented to the "qsub" utility, the utility will set the Job_Name attribute of the job to the name of the Script_File argument after any directory "path-name" has been removed. If the script is read from standard input, "qsub" will set the job-name attribute of the job to the value STDIN.

-notify
Defines the Notify attribute for a job. This attribute, when TRUE, causes the batch server to send "warning" signals to a running job prior to sending the signals themselves. If a SIGSTOP is pending the job will receive a SIGUSR1 several seconds before the SIGSTOP. If a SIGKILL is pending the job will receive a SIGUSR2 several seconds before the SIGKILL. The amount of time delay is controlled by the "notify" parameter in each queue configuration.

-o path_name
Redefines the path for standard output of the job.

If the "path_name" option argument constitutes an absolute path name, the "qsub" utility will set the Output_Path attribute of the job to the value of the "path_name" option argument including the hostname, if present.

If the "path_name" option argument constitutes a relative path name and no hostname is specified, the "qsub" utility will set the Output_Path attribute of the job to the value of the absolute path name. The absolute path name is derived from the relative path name by expanding the "path_name" option argument relative to the current working directory of the process that executed the "qsub" utility.

If the "path_name" option argument constitutes a relative path name and a hostname is specified, the "qsub" utility will set the Output_Path attribute of the job to the value of the option argument without expansion.

If the "path_name" option argument does not include a hostname, the "qsub" utility will prefix the path name in the Output_Path attribute with "hostname:", where hostname is the name of the host upon which the "qsub" utility is being executed.

The default file name for standard output has the following form:

job_name.o.sequence_number.pid

If the -o option possess no option argument, the utility will clear the Output_Path attribute of the job so that the default standard error output will be used when the job is executed.

-p priority
Defines the priority of the job relative to other jobs.

"qsub" will accept a value for the priority option argument that conforms to the syntax for signed decimal integers, and which is not less than -1024 and not greater than 1023.

-par parallel paradigm
DQS can process, monitor and initiate parallel jobs using one of several message passing system paradigms. This option defines the paradigm to use for a job. The "parallel_paradigm" option argument is described in the Glossary.

-passwd
Specifies that a password list will be requested for AFS and/or DFS re-authentication. If given, DQS will utilize the password list to authenticate the process prior to executing the job, so that re-authentication will be unnecessary.

-passwd_file filename
Defines the name of the file containing the AFS passwords. (See AFS support in the User Guide.)

-q destination identifier
Assigns the destination identifier, i.e. queue name(s), for the job.
DQS attempts to assign the host based on a first to last traversal of the "destination_identifier" list. If the "destination_identifier" list is specified as a "soft" request and DQS is unable to fill the request based on the respective "destination_identifier" list, DQS will assign a host as available.

The "destination_identifier" option argument is described in the Glossary.

-r y | n
Defines whether the job is Rerunable.

If the value of the option argument is 'y', "qsub" will set the Rerunable attribute of the job to TRUE, if the option argument is 'n', the Rerunable attribute will be set to FALSE.

The "qsub" utility will consider it an error if any character other than 'y' or 'n' is specified in the option argument.

-reauth seconds
Specifies the time interval for AFS/DFS re-authentication.

-soft
Signifies that all resource requirements following in the "qsub" command will be soft requirements and will be filled on a "as available" basis before a job can be scheduled.

-S path_name_list
Defines the shell that interprets the script at the destination system.

"qsub" accepts a "path_name_list" option argument that conforms to the following syntax:

path-name[@host][,path-name[@host],...]

The "qsub" utility will accept only one path name that is missing a corresponding host name. The "qsub" utility will allow only one path name per named host.

"qsub" adds a value to the Shell_Path_List attribute of the job for each entry in the "path_name_list" option argument.

If the -S option has no arguments the utility will set the path-list attribute of the job to the null string.

-v variable list
Defines the environment variables to be exported to the job.

The "variable_list" option argument is described in the Glossary.

If the -v option is presented to the "qsub" utility, the utility will also add to the environment_variable_list attribute of the job every variable named in the "variable_list" option argument and, optionally, values of specified variables.

-verify
This does not affect any job attributes. It causes the "qsub" utility to send the contents of the "qsub" command to standard out so it may be verified by the user.

-V
Specifies that all environment variables be exported to the context of the job.

Script File
The path to the script of the job. The "Script_File" operand argument is described in the Glossary.

Defining the Directive Prefix
The "qsub" utility will select the directive prefix from one of the following sources:

-- if the -C option is presented to the utility, the value of the directive_prefix option argument.

-- if the environment variable POSIX2_DPREFIX is defined, the value of that variable.

If the -C option is present in the script file, it will be ignored.
go to toc

##
The daemons

The backbone of DQS is the operation of the qmaster and the dqs_execd daemons. Each host which will execute batch jobs will have a dqs_execd running as a background daemon. The qmaster runs as a background daemon on each host functioning as a DQS cell manager. The dqs_execd daemons communicate only with the qmaster except the DSH utility is invoked by a master process during parallel job startup. The qmaster handles all requests from DQS utilities and communicates with the dqs_execd daemons in its own cell and with other qmaster daemons.

# qmaster

# dqs_execd

go to toc

##
Glossary

account string

The account string is a list containing account_names as follows:

Account_Name[@cell][,Account_Name[@cell],... ]

The Account_Name may be any arbitrary ASCII alphanumeric string of bytes but may contain no blanks or separator characters. The under-bar "_" is considered a non-separator.

date time

The date_time option argument is of the form:
[[CC]YY]MMDDhhmm[.ss]
where :

CC Specifies the first two digits of the year

YY Specifies the last two digits of the year

MM Specifies the month of the year (01 to 12)

DD Specifies the day of the month (01 to 31)

hh Specifies the hour of the day (00 to 23 )

mm Specifies the minute of the hour (00 to 59 )

ss Specifies the number of seconds in the hour (00 to 59 )
(note the decimal point must be present to flag the ss )

destination identifier

The destination_identifier(_list) option argument conforms to the following:
destination_id "queue" "queue@cell"
destination_id_list destin_id[,destin_id,...]
parallel destination identifier

For parallel jobs, the destination_identifier can have an additional component:

destination_id_list[-exec .eq. exec_str]

The string "-exec.eq." is reserved for "generic" parallel job execution. At the moment of job initiation the optional execution string will be used to startup a user supplied program.

For example:
qsub ... -master myqueue -exec.eq. "parallel N2 params"

This entry requests "myqueue" as the queue for the master process. DQS will execute the program "parallel" and pass to it the parameters "N2" and "params".

DQS attempts to assign the master host based on a first to last traversal of the destination_identifier list. If the destination_identifier list is specified as a "soft" request and DQS is unable to fill the master request based on the respective destination_identifier list, DQS will assign a master as available.

The exec_str may contain DQS reserved "wildcards" which will be expanded to provide dynamic information to the startup program. The reserved "wildcards" are:

%execer_id Set to DQS

%master_host Set to the master's hostname at random

%my_hostname Set to my hostname at random

%my_nodenum Set to my node number at random

%total_numnodes Set to the total number of nodes at random

%remote_info Set to remote exec_str at random

Using the previous example with a wildcard:
qsub ... -master myqueue -exec.eq. "parallel %total_numnodes params"

hold list

Utilities accept a value for the "hold_list" option argument that is a string one or more characters 'u' 's' 'o' or the single character 'n'.

For each unique character in the "hold_list", the utility will add a value to the Hold_Type attribute of the job as follows, each representing a different hold type:

u USER

s SYSTEM

o OPERATOR

n NO_HOLD

It is considered an error if any hold type other than 'n' is combined with hold type 'n'.

job identifier list

sequence_number[@cell][,sequence_number[@cell],...]

If the application omits the cell portion of a job identifier, the utility will use either:

-- the name of the cell specified on the command line, if any

-- the name of the cell specified in the environment variable DQS_CELL, if it is set

-- the name of the default cell

join list

The utility will accept a "join_list" option argument that consists of one or more characters 'e' and 'o' or the single character 'n'.

All other job output streams specified will be merged into the output stream represented by the character listed first in the "join_list" option argument.

For each unique character in the "join_list" option argument, the utility will add a value to the Join_Path attribute of the job as follows, each representing a different of stream to join:

e The standard error of the batch job JOIN_STD_ERROR

o The standard output of the batch job JOIN_STD_OUTPUT

An existing Join_Path attribute can be cleared by the job type:

n NO_JOIN

If 'n' is specified, then no files are joined. The utility will consider it an error if any type other than 'n' is combined with the join type 'n'.

If the value of the option argument is 'y' the Join_Path attribute of the job is set to TRUE,
if the value is 'n' the Join_Path attribute of the job is set to FALSE.

If both the "-j y" and the "-e" options are specified, the batch server will set but ignore the Error_Path attribute.

If the "-j" option is not presented to the utility, the utility will set the value of the Join_Path attribute to FALSE.

mail options

The utility will accept a value for the "mail_options" option argument that is a string one or more of the characters:'

e MAIL_AT_EXIT

b MAIL_AT_BEGINNING

a MAIL_AT_ABORT

s MAIL_AT_SUSPENSION

If any of these characters are duplicated in the "mail_options" option argument, the duplicates will be ignored.

n NO_MAIL

If 'n' is specified then mail is not sent. The utility will consider it an error if any mail type other than 'n' is combined with mail type 'n'.

parallel paradigm

The "paradigm" can be one of the following:

p4 - support for ANL p4 language interface

mpi - support for the standard MPI interface

generic_all - a simple parallel interface

pvm - The ORNL PVM interface

tcgmsg - The Argonne / PNL interface

generic_sla - ???

(See parallel job execution in the User Manual.)

resource list

The "resource_list" option argument will conform to the following:

resource[relational-op resource-field[logical-op resource[relational-op….

The "resource_list" should not contain ANY embedded blanks, as the space character is used to delimit each of the option fields in commands.

There are two character strings which act as "reserved words" in the "resource_list":

qty The quantity of a requested resource, default=1.

-exec.eq. Identifies special parallel resource definitions. See Parallel Job Submission in the User Guide.

Each relational-op may be selected from the FORTRAN or "C" list of operators, which consists of
.eq. .ne. .gt. .lt. .ge. .le. or '==' '!=' '>' '<' '>=' '<='

A special relationship is flagged when a single equal-sign is used '='.
This indicates that a comparison of character strings will be used in the relational test rather than a numeric comparison.

Each logical-op may be selected from the FORTRAN or "C" list of operators, which consists of
.and. .or. .not. or '&&' '||' '!'

The "logical-op" may be preceded or followed by an opening or closing parenthesis to ease readability. For example "(mem.gt.32).and.(disk.gt.100)"

Script File

If the script is not presented to the utility, or if the specified operand is the single character '-' (minus), the utility will read the script from standard input.

If the script argument represents a partial path, the utility will expand the path relative to the current directory of the process executing the utility.

If the utility is presented with a script file it will replace the job script file originally submitted with the "qsub" command. The utility scans the script file interpreting lines flagged with the DQS_DELIMETER pair of characters. All command options in the command line and the script file are processed and replace or modify job attributes for the job.

A script can contain directives to the utility.

The utility will scan the lines of the script for directives until the first line that begins with a string other than the directive string; if directives occur on subsequent lines, the utility will ignore those directives.

The utility will process a line in the script as a directive if, and only, if, the string of characters from the first non-white space on the line until the first space or tab character on the line match the directive prefix.

The utility will ignore any blank lines and shell comment lines prior to the first executable line.

The utility will process the options and option arguments contained on the directive prefix line using the same syntax as if the options were input on the command line.

The utility will continue to process a directive prefix line until after a <newline> or an non-quoted comment character of the shell that will interpret the script is encountered.

If an option is present in both a directive and arguments to the utility, it will ignore the option and the corresponding option argument, if any, in the directive.

If an option that is present in the directive is not present in the arguments to the utility, the utility will process the option and the option argument, if any.

In order of preference, the utility will select the directive prefix from one of the following sources:

-- if the -C option is presented to the utility, the value of the directive_prefix option argument.

-- if the environment variable POSIX2_DPREFIX is defined, the value of that variable.

If the -C option is present in the script file, it will be ignored.

user list

The "user_list" option argument will conform to the following:

user[@host][,user[@host],...]
or
name[,name,...]

variable list

A variable_list is a set of strings delimited by commas. A string will be of either of these forms:

variable
or
variable = value (with no intervening spaces)

A variable cannot be repeated in the environment variable_list option argument.

go to toc