slurm-slurm-17-11-2-1/doc/html/heterogeneous_jobs.shtml000066400000000000000000000710051322351151600227760ustar00rootroot00000000000000

Heterogeneous Job Support

Overview

Slurm version 17.11 and later supports the ability to submit and manage heterogeneous jobs, in which each component has virtually all job options available including partition, account and QOS (Quality Of Service). For example, part of a job might require four cores and 4 GB for each of 128 tasks while another part of the job would require 16 GB of memory and one CPU.

Submitting Jobs

The salloc, sbatch and srun commands can all be used to submit heterogeneous jobs. Resource specifications for each component of the heterogeneous job should be separated with ":" character. For example:

$ sbatch --cpus-per-task=4 --mem-per-cpu=1  --ntasks=128 : \
         --cpus-per-task=1 --mem-per-cpu=16 --ntasks=1 my.bash

Options specified for one component of a heterogeneous job (or job step) will be used for subsequent components to the extent which is expected to be helpful. Propagated options can be reset as desired for each component (e.g. a different account name could be specified for each job component. For example, --immediate and --job-name are propagated, while --ntasks and --mem-per-cpu are reset to default values for each component. A list of propagated options follows.

  • --account
  • --acctg-freq
  • --begin
  • --checkpoint
  • --checkpoint-dir
  • --cluster-constraint
  • --clusters
  • --comment
  • --deadline
  • --delay-boot
  • --dependency
  • --distribution
  • --epilog (option available only in srun)
  • --error
  • --export
  • --export-file
  • --exclude
  • --get-user-env
  • --gid
  • --hold
  • --ignore-pbs
  • --immediate
  • --input
  • --job-name
  • --kill-on-bad-exit (option available only in srun)
  • --label (option available only in srun)
  • --max-exit-timeout (option available only in srun)
  • --mcs-label
  • --msg-timeout (option available only in srun)
  • --no-allocate (option available only in srun)
  • --no-requeue
  • --nice
  • --no-kill
  • --open-mode (option available only in srun)
  • --output
  • --parsable
  • --priority
  • --profile
  • --propagate
  • --prolog (option available only in srun)
  • --pty (option available only in srun)
  • --qos
  • --quiet
  • --quit-on-interrupt (option available only in srun)
  • --reboot
  • --reservation
  • --requeue
  • --signal
  • --slurmd-debug (option available only in srun)
  • --task-epilog (option available only in srun)
  • --task-prolog (option available only in srun)
  • --time
  • --test-only
  • --time-min
  • --uid
  • --unbuffered (option available only in srun)
  • --verbose
  • --wait
  • --wait-all-nodes
  • --wckey
  • --workdir

The task distribution specification applies separately within each job component. Consider for example a heterogeneous job with each component being allocated 4 CPUs on 2 nodes. In our example, job component zero is allocated 2 CPUs on node "nid00001" and 2 CPUs on node "nid00002". Job component zero is allocated 2 CPUs on node "nid00003" and 2 CPUs on node "nid00004". A task distribution of "cyclic" will distribute the first 4 tasks in a cyclic fashion on nodes "nid00001" and "nid00002", then distribute the next 4 tasks in a cyclic fashion on nodes "nid00003" and "nid00004" as shown below.

Node nid00001Node nid00002Node nid00003Node nid00004
Rank 0Rank 1Rank 4Rank 5
Rank 2Rank 3Rank 6Rank 7

Some options should be specified only in the first job component. For example, specifyihg a batch job output file in the second job component's options will result in the first job component (where the batch script executes) using the default output file name.

Environment variables used to specify default options for the job submit command will be applied to every component of the heterogeneous job (e.g. SBATCH_ACCOUNT).

Batch job options can be included in the submitted script for multiple heterogeneous job components. Each component should be separated by a line containing the line "#SBATCH packjob" as shown below.

$ cat new.bash
#!/bin/bash
#SBATCH --cpus-per-task=4 --mem-per-cpu=16g --ntasks=1
#SBATCH packjob
#SBATCH --cpus-per-task=2 --mem-per-cpu=1g  --ntasks=8

srun run.app

$ sbatch new.bash

Is equivalent to the following:

$ cat my.bash
#!/bin/bash
srun run.app

$ sbatch --cpus-per-task=4 --mem-per-cpu=16g --ntasks=1 : \
         --cpus-per-task=2 --mem-per-cpu=1g  --ntasks=8 my.bash

The batch script will be executed in the first node in the first component of the heterogeneous job. For the above example, that will be the job component with 1 task, 4 CPUs and 64 GB of memory (16 GB for each of the 4 CPUs).

If a heterogeneous job is submitted to run in multiple clusters not part of a federation (e.g. "sbatch --cluster=alpha,beta ...") then the entire job will be sent to the cluster expected to be able to start all components at the earliest time.

A resource limit test is performed when a heterogeneous job is submitted in order to immediately reject jobs that will not be able to start with current limits. The individual components of the heterogeneous job are validated, like all regular jobs. The heterogeneous job as a whole is also tested, but in a more limited fashion with respect to quality of service (QOS) limits. Each component of a heterogeneous job counts as a "job" with respect to resource limits.

Burst Buffers

A burst buffer can either be persistent or linked to a specific job ID (at least the Cray implementation). Since a heterogeneous job consists of multiple job IDs, a job-specific burst buffer will be associated with only one heterogeneous job component. Only a persistent burst buffer can be accessed by all components of a heterogeneous job. A sample batch script demonstrating this for a Cray system is appended.

#!/bin/bash
#SBATCH --nodes=1 --constraint=haswell
#BB create_persistent name=alpha capacity=10 access=striped type=scratch
#DW persistentdw name=alpha
#SBATCH packjob
#SBATCH --nodes=16 --constraint=knl
#DW persistentdw name=alpha
...

NOTE: Cray's DataWarp interface directly reads the job script, but has no knowledge of "Slurm's "packjob" directive, so Slurm internally rebuilds the script for each job component so that only that job components burst buffer directives are included in that script. The batch script first component of the job will be modified in order to replace the "#DW" directives of other job components with "#EXCLUDED DW" to prevent their interpretation by Cray infrastructure. Since the batch script will only be executed by the first job component, the subsequent job components will not include commands from the original script. These scripts are build and managed by Slurm for internal purposes (and visiable from various Slurm commands) from a user script as shown above. An example is shown below:

Rebuilt script for first job component

#!/bin/bash
#SBATCH --nodes=1 --constraint=haswell
#BB create_persistent name=alpha capacity=10 access=striped type=scratch
#DW persistentdw name=alpha
#SBATCH packjob
#SBATCH --nodes=16 --constraint=knl
#EXCLUDED DW persistentdw name=alpha
...


Rebuilt script for second job component

#!/bin/bash
#SBATCH --nodes=16 --constraint=knl
#DW persistentdw name=alpha
exit 0

Managing Jobs

Information maintained in Slurm for a heterogeneous job includes:

  • job_id: Each component of a heterogeneous job will have its own unique job_id.
  • pack_job_id: This identification number applies to all components of the heterogeneous job. All components of the same job will have the same pack_job_id value and it will be equal to the job_id of the first component. We refer to this as the "heterogeneous job leader".
  • pack_job_id_set: Regular expression identifying all job_id values associated with the job.
  • pack_job_offset: A unique sequence number applied to each component of the heterogeneous job. The first component will have a >pack_job_offset value of 0, the next a value of 1, etc.
job_id pack_job_id pack_job_offset pack_job_id_set
1231230123-127
1241231123-127
1251232123-127
1261233123-127
1271234123-127

Table 1: Example job IDs

The smap, smap, squeue and sview report the components of a heterogeneous job using the format "<pack_job_id>+<pack_job_offset>". For example "123+4" would represent heterogeneous job id 123 and it's fifth component (note: the first component has a pack_job_offset value of 0).

A request for a specific job ID that identifies a ID of the first component of a heterogeneous job (i.e. the "heterogeneous job leader") will return information about all components of that job. For example:

$ squeue --job=93
JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
 93+0     debug  bash  adam  R 18:18      1 nid00001
 93+1     debug  bash  adam  R 18:18      1 nid00011
 93+2     debug  bash  adam  R 18:18      1 nid00021

A request to cancel or otherwise signal a heterogeneous job leader will be applied to all components of that heterogeneous job. A request to cancel a specific component of the heterogeneous job using the "#+#" notation will apply only to that specific component. For example:

$ squeue --job=93
JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
 93+0     debug  bash  adam  R 19:18      1 nid00001
 93+1     debug  bash  adam  R 19:18      1 nid00011
 93+2     debug  bash  adam  R 19:18      1 nid00021
$ scancel 93+1
$ squeue --job=93
JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
 93+0     debug  bash  adam  R 19:38      1 nid00001
 93+2     debug  bash  adam  R 19:38      1 nid00021
$ scancel 93
$ squeue --job=93
JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST

While a heterogeneous job is in pending state, only the entire job can be cancelled rather than it's individual components. A request to cancel an individual component of a heterogeneous job not in pending state will return an error. After the job has begun execution, the individual component can be cancelled.

Email notification for job state changes (the --mail-type option) is only supported for a heterogeneous job leader. Requests for email notifications for other components of a heterogeneous job will be silently ignored.

Requests to modify an individual component of a job using the scontrol command must specify the job ID with the "#+#" notation. A request to modify a job by specifying the pack_job_id will modify all components of a heterogeneous job. For example:

# Change the account of compnent 2 of heterogeneous job 123:
$ scontrol update jobid=123+2 account=abc

# Change the time limit of all components of heterogeneous job 123:
$ scontrol update jobid=123 timelimit=60

Requests to perform the following operations a job can only be requested for a heterogeneous job leader and will be applied to all components of that heterogeneous job. Requests to operate on individual components of the heterogeneous will return an error.

  • requeue
  • resume
  • suspend

The sbcast command supports heterogeneous job allocations. By default, sbcast will will copy files to all nodes in the job allocation. The -j/--jobid option can be used to copy files to individual components as shown below.

$ sbcast --jobid=123   data /tmp/data
$ sbcast --jobid=123.0 app0 /tmp/app0
$ sbcast --jobid=123.1 app1 /tmp/app1

The srun commands --bcast option will transfer files to the nodes associated with the application to be launched as specified by the --pack-group option.

Slurm has a configuration option to control behavior of some commands with respect to heterogeneous jobs. By default a request to cancel, hold or release a job ID that is not the pack_job_id, but that of a job component will only operate that one component of the heterogeneous job. If SchedulerParameters configuration parameter includes the option "whole_pack" then the operation would apply to all components of the job if any job component is specified to be operated upon. In the below example, the scancel command will either cancel all components of job 93 if SchedulingParameters=whole_pack is configured, otherwise only job 93+1 will be cancelled. If a specific heterogeneous job component is specified (e.g. "scancel 93+1"), then only that one component will be effected.

$ squeue --job=93
JOBID PARTITION  NAME  USER ST  TIME  NODES NODELIST
 93+0     debug  bash  adam  R 19:18      1 nid00001
 93+1     debug  bash  adam  R 19:18      1 nid00011
 93+2     debug  bash  adam  R 19:18      1 nid00021
$ scancel 94 (where job ID 94 is equivalent to 93+1)
# Cancel 93+0, 93+1 and 93+2 if SchedulerParameters includes "whole_pack"
# Cancel only 93+1 if SchedulerParameters does not include "whole_pack"

Accounting

Slurm's accounting database records the pack_job_id and pack_job_offset fields. The sacct command reports job's using the format "<pack_job_id>+<pack_job_offset>" and can accept a job ID specification for filtering using the same format. If a pack_job_id value is specified as a job filter, then information about all components of that job will be reported as shown below.

$ sacct -j 67767
  JobID JobName Partition Account AllocCPUS     State ExitCode 
------- ------- --------- ------- --------- --------- -------- 
67767+0     foo     debug    test         2 COMPLETED      0:0 
67767+1     foo     debug    test         4 COMPLETED      0:0 

$  sacct -j 67767+1
  JobID JobName Partition Account AllocCPUS     State ExitCode 
------- ------- --------- ------- --------- --------- -------- 
67767+1     foo     debug    test         4 COMPLETED      0:0

Launching Applications (Job Steps)

The srun command is used to launch applications. By default, the application is launched only on the first component of a heterogeneous job, but options are available to support different behaviors.

srun's "--pack-group" option defines which pack job component(s) are to have applications launched for them. The --pack-group option takes an expression defining which component(s) are to launch an application for an individual execution of the srun command. The expression can contain one or more component index values in a comma separated list. Ranges of index values can be specified in a hyphen separated list. By default, an application is launched only on component number zero. Some examples follow:

  • --pack-group=2
  • --pack-group=0,4
  • --pack-group=1,3-5

IMPORTANT: The ability to execute a single application across more than one job allocation does not work in Slurm version 17.11 except in the case of OpenMPI with Slurm's mpi/pmi2 plugin or a non-MPI job with Slurm's mpi/none. We expect support for other MPI implementations in a future release of Slurm. Until that time, an appropriate error message will be reported unless Slurm's SchedulerParameters configuration parameter includes the option "enable_hetero_steps".

By default, the applications launched by a single execution of the srun command (even for different components of the heterogeneous job) are combined into one MPI_COMM_WORLD with non-overlapping task IDs.

As with the salloc and sbatch commands, the ":" character is used to separate multiple components of a heterogeneous job. This convention means that the stand-alone ":" character can not be used as an argument to an application launched by srun. This includes the ability to execute different applications and arguments for each job component. If some heterogeneous job component lacks an application specification, the next application specification provided will be used for earlier components lacking one as shown below.

$ srun --label -n2 : -n1 hostname
0: nid00012
1: nid00012
2: nid00013

If multiple srun commands are executed concurrently, this may result in resource contention (e.g. memory limits preventing some job steps components from being allocated resources because of two srun commands executing at the same time). If the srun --pack-group option is used to create multiple job steps (for the different components of a heterogeneous job), those job steps will be created sequentially. When multiple srun commmands execute at the same time, this may result in some step allocations taking place, while others are delayed. Only after all job step allocations have been granted will the application being launched.

All components of a job step will have the same step ID value. If job steps are launched on subsets of the job components there may be gaps in the step ID values for individual job components.

$ salloc -n1 : -n2 beta bash
salloc: Pending job allocation 1721
salloc: Granted job allocation 1721
$ srun --pack-group=0,1 true   # Launches steps 1721.0 and 1722.0
$ srun --pack-group=0 t  rue   # Launches step  1721.1, no 1722.1
$ srun --pack-group=0,1 true   # Launches steps 1721.2 and 1722.2

The maximum pack-group specified in a job step allocation (either explicitly specified or implied by the ":" separator) must not exceed the number of components in the heterogeneous job allocation. For example

$ salloc -n1 -C alpha : -n2 -C beta bash
salloc: Pending job allocation 1728
salloc: Granted job allocation 1728
$ srun --pack-group=0,1 hostname
nid00001
nid00008
nid00008
$ srun hostname : date : id
error: Attempt to run a job step with pack-group value of 2,
       but the job allocation has maximum value of 1

Environment Variables

Slurm environment variables will be set independently for each component of the job by appending "_PACK_GROUP_" and a sequence number the the usual name. In addition, the "SLURM_JOB_ID" environment variable will contain the job ID of the heterogeneous job leader and "SLURM_PACK_SIZE" will contain the number of components in the job. For example:

$ salloc -N1 : -N2 bash
salloc: Pending job allocation 11741
salloc: job 11741 queued and waiting for resources
salloc: job 11741 has been allocated resources
$ env | grep SLURM
SLURM_JOB_ID=11741
SLURM_PACK_SIZE=2
SLURM_JOB_ID_PACK_GROUP_0=11741
SLURM_JOB_ID_PACK_GROUP_1=11742
SLURM_JOB_NODES_PACK_GROUP_0=1
SLURM_JOB_NODES_PACK_GROUP_1=2
SLURM_JOB_NODELIST_PACK_GROUP_0=nid00001
SLURM_JOB_NODELIST_PACK_GROUP_1=nid[00011-00012]
...

The various MPI implementations rely heavily upon Slurm environment variables for proper operation. A single MPI applications executing in a single MPI_COMM_WORLD require a uniform set of environment variables that reflect a single job allocation. The example below shows how Slurm sets environment variables for MPI.

$ salloc -N1 : -N2 bash
salloc: Pending job allocation 11741
salloc: job 11751 queued and waiting for resources
salloc: job 11751 has been allocated resources
$ env | grep SLURM
SLURM_JOB_ID=11751
SLURM_PACK_SIZE=2
SLURM_JOB_ID_PACK_GROUP_0=11751
SLURM_JOB_ID_PACK_GROUP_1=11752
SLURM_JOB_NODELIST_PACK_GROUP_0=nid00001
SLURM_JOB_NODELIST_PACK_GROUP_1=nid[00011-00012]
...
$ srun --pack-group=0,1 env | grep SLURM
SLURM_JOB_ID=11751
SLURM_JOB_NODELIST=nid[00001,00011-00012]
...

Examples

Create a heterogeneous resource allocation containing one node with 256GB of memory and a feature of "haswell" plus 2176 cores on 32 nodes with a feature of "knl". Then launch a program called "master" on the "haswell" node and "slave" on the "knl" nodes. Each application will be in its own MPI_COMM_WORLD.

salloc -N1 --mem=256GB -C haswell : \
       -n2176 -N32 --ntasks-per-core=1 -C knl bash
srun master &
srun --pack-group=1 slave &
wait

This variation of the above example launches programs "master" and "slave" in a single MPI_COMM_WORLD.

salloc -N1 --mem=256GB -C haswell : \
       -n2176 -N32 --ntasks-per-core=1 -C knl bash
srun master : slave &

The SLURM_PROCID and SLURM_TASKID environment variables will be set to reflect a global task rank (both environment variables will have the same value). Each spawned process will have a unique SLURM_PROCID.

Similarly, the SLURM_NPROCS and SLURM_NTASKS environment variables will be set to reflect a global task count (both environment variables will have the same value). SLURM_NTASKS will be set to the total count of tasks in all components. Note that the task rank and count values are needed by MPI and typically determined by examining Slurm environment variables.

Limitations

The backfill scheduler has limiations in how it tracks usage of CPUs and memory in the future. This typically requires the backfill scheduler be able to allocate each component of a heterogeneous job on a different node in order to begin its resource allocation, even if multiple compoents of the job do actually get allocated resources on the same node.

In a federation of clusters, a heterogeneous job will execute entirely on the cluster from which the job is submitted. The heterogeneous job will not be eligible to migrate between clusters or to have different components of the job execute on different clusters in the federation.

Job arrays of heterogeneous jobs are not supported.

The srun command's --no-allocate option is not supported for heterogeneous jobs.

Only one job step per heterogeneous job component can be launched by a single srun command (e.g. "srun --pack-group=0 alpha : --pack-group=0 beta" is not supported).

The sattach command can only be used to attach to a single component of a heterogeneous job at a time.

Heterogeneous jobs are only scheduled by the backfill scheduler plugin. The more frequently executed scheduling logic only starts jobs on a first-in first-out (FIFO) basis and lacks logic for concurrently scheduling all components of a heterogeneous job.

Heterogeneous jobs are not supported with Slurm's select/serial plugin.

Heterogeneous jobs are not supported on Cray ALPS systems.

Heterogeneous jobs are not supported on IBM PE systems.

Slurm's PERL APIs currently do not support heterogeneous jobs.

The srun --multi-prog option can not be used to span more than one heterogeneous job component.

The srun --open-mode option is by default set to "append".

Ancient versions of OpenMPI and their derivatives (i.e. Cray MPI) are dependent upon communication ports being assigned to them by Slurm. Such MPI jobs will experience step launch failure if any component of a heterogeneous job step is unable to acquire the allocated ports. Non-heterogeneous job steps will retry step launch using a new set of communication ports (no change in Slurm behavior).

System Administrator Information

The job submit plugin is invoked independently for each component of a heterogeneous job.

The spank_init_post_opt() function is invoked once for each component of a heterogeneous job. This permits site defined options on a per job component basis.

Scheduling of heterogeneous jobs is performed only by the sched/backfill plugin and all heterogeneous job components are either all scheduled at the same time or deferred. In order to ensure the timely initiation of both heterogeneous and non-heterogeneous jobs, the backfill scheduler alternates between two different modes on each iteration. In the first mode, if a heterogeneous job component can not be initiated immediately, its expected start time is recorded and all subsequent components of that job will be considered for starting no earlier than the latest component's expected start time. In the second mode, all heterogeneous job components will be considered for starting no earlier than the latest component's expected start time. After completion of the second mode, all heterogeneous job expected start time data is cleared and the first mode will be used in the next backfill scheduler iteration. Regular (non-heterogeneous jobs) are scheduled independently on each iteration of the backfill scheduler.

For example, consider a heterogeneous job with three components. When considered as independent jobs, the components could be initiated at times now (component 0), now plus 2 hour (component 1), and now plus 1 hours (component 2). When the backfill scheduler runs in the first mode:

  1. Component 0 will be noted to possible to start now, but not initiated due to the additional components to be initiated
  2. Component 1 will be noted to be possible to start in 2 hours
  3. Component 2 will not be considered for scheduling until 2 hours in the future, which leave some additional resources available for scheduling to other jobs

When the backfill scheduler executes next, it will use the second mode and (assuming no other state changes) all three job components will be considered available for scheduling no earlier than 2 hours in the future, which may allow other jobs to be allocated resources before heterogeneous job component 0 could be initiated.

The heterogeneous job start time data will be cleared before the first mode is used in the next iteration in order to consider system status changes which might permit the heterogeneous to be initiated at an earlier time than previously determined.

A resource limit test is performed when a heterogeneous job is submitted in order to immediately reject jobs that will not be able to start with current limits. The individual components of the heterogeneous job are validated, like all regular jobs. The heterogeneous job as a whole is also tested, but in a more limited fashion with respect to quality of service (QOS) limits. This is due to the complexity of each job component having up to three sets of limits (association, job QOS and partition QOS). Note that successful submission of any job (heterogeneous or otherwise) does not ensure the job will be able to start without exceeding some limit. For example a job's CPU limit test does not consider that CPUs might not be allocated individually, but resource allocations might be performed by whole core, socket or node. Each component of a heterogeneous job counts as a "job" with respect to resource limits.

For example, a user might have a limit of 2 concurrent running jobs and submit a heterogeneous job with 3 components. Such a situation will have an adverse effect upon scheduling other jobs, especially other heterogeneous jobs.

top

Last modified 20 December 2017

slurm-slurm-17-11-2-1/doc/html/high_throughput.shtml000066400000000000000000000264601322351151600223220ustar00rootroot00000000000000

High Throughput Computing Administration Guide

This document contains Slurm administrator information specifically for high throughput computing, namely the execution of many short jobs. Getting optimal performance for high throughput computing does require some tuning and this document should help you off to a good start. A working knowledge of Slurm should be considered a prerequisite for this material.

Performance Results

Slurm has also been validated to execute 500 simple batch jobs per second on a sustained basis with short bursts of activity at a much higher level. Actual performance depends upon the jobs to be executed plus the hardware and configuration used.

System configuration

Several system configuration parameters may require modification to support a large number of open files and TCP connections with large bursts of messages. Changes can be made using the /etc/rc.d/rc.local or /etc/sysctl.conf script to preserve changes after reboot. In either case, you can write values directly into these files (e.g. "echo 32832 > /proc/sys/fs/file-max").

  • /proc/sys/fs/file-max: The maximum number of concurrently open files. We recommend a limit of at least 32,832.
  • /proc/sys/net/ipv4/tcp_max_syn_backlog: The maximum number of SYN requests to keep in memory that we have yet to get the third packet in a 3-way handshake from. The tcp_max_syn_backlog variable is overridden by the tcp_syncookies variable, which needs to be turned on for this variable to have any effect. The default value is 1024 for systems with more than 128Mb of memory, and 128 for low memory machines. If server suffers of overload, try to increase this number.
  • /proc/sys/net/ipv4/tcp_syncookies: Used to send out syncookies to hosts when the kernels syn backlog queue for a specific socket is overflowed. The default value is 0, which disables this functionality. Set the value to 1.
  • /proc/sys/net/ipv4/tcp_synack_retries: How many times to retransmit the SYN,ACK reply to an SYN request. In other words, this tells the system how many times to try to establish a passive TCP connection that was started by another host. This variable takes an integer value, but should under no circumstances be larger than 255. Each retransmission will take aproximately 30 to 40 seconds. The default value of 5, which results in a timeout of passive TCP connections of aproximately 180 seconds and is generally satisfactory.
  • /proc/sys/net/core/somaxconn: Limit of socket listen() backlog, known in userspace as SOMAXCONN. Defaults to 128. The value should be raised substantially to support bursts of request. For example, to support a burst of 1024 requests, set somaxconn to 1024.
  • /proc/sys/net/ipv4/ip_local_port_range: Identify the ephermeral ports available, which are used for many Slurm communcations. The value may be raised to support a high volume of communcations. For example, write the value "32768 65535" into the ip_local_port_range file in order to make that range of ports available.

The transmit queue length (txqueuelen) may also need to be modified using the ifconfig command. A value of 4096 has been found to work well for one site with a very large cluster (e.g. "ifconfig txqueuelen 4096").

Munge configuration

By default the Munge daemon runs with two threads, but a higher thread count can improve its throughput. We suggest starting the Munge daemon with ten threads for high throughput support (e.g. "munged --num-threads 10").

User limits

The ulimit values in effect for the slurmctld daemon should be set quite high for memory size, open file count and stack size.

Slurm Configuration

Several Slurm configuration parameters should be adjusted to reflect the needs of high throughput computing. The changes described below will not be possible in all environments, but these are the configuration options that you may want to consider for higher throughput.

  • AccountingStorageType: Disabling storing of accounting by using the accounting_storage/none plugin. Turning accounting off provides minimal improvement in performance. If using the SlurmDBD increased speedup can be achieved by setting the CommitDelay option in the slurmdbd.conf
  • JobAcctGatherType: Disabling the collection of job accounting information will improve job throughput. Disable collection of accounting by using the jobacct_gather/none plugin.
  • JobCompType: Disabling recording of job completion information will improve job throughput. Disable recording of job completion information by using the jobcomp/none plugin.
  • MaxJobCount: Controls how many jobs may be in the slurmctld daemon records at any point in time (pending, running, suspended or completed[temporarily]). The default value is 10,000.
  • MessageTimeout: Controls how long to wait for a response to messages. The default value is 10 seconds. While the slurmctld daemon is highly threaded, its responsiveness is load dependent. This value might need to be increased somewhat.
  • MinJobAge: Controls how soon the record of a completed job can be purged from the slurmctld memory and thus not visible using the squeue command. The record of jobs run will be preserved in accounting records and logs. The default value is 300 seconds. The value should be reduced to a few seconds if possible. Use of accounting records for older jobs can increase the job throughput rate compared with retaining old jobs in the memory of the slurmctld daemon.
  • PriorityType: The priority/builtin is considerably faster than other options, but schedules jobs only on a First In First Out (FIFO) basis.
  • SchedulerParameters: Several scheduling parameters are available.
    • Setting option batch_sched_delay will control how long the scheduling of batch jobs can be delayed. This effects only batch jobs. For example, if many jobs are submitted each second, the overhead of trying to schedule each one will adversely impact the rate at which jobs can be submitted. The default value is 3 seconds.
    • Setting option defer will avoid attempting to schedule each job individually at job submit time, but defer it until a later time when scheduling multiple jobs simultaneously may be possible. This option may improve system responsiveness when large numbers of jobs (many hundreds) are submitted at the same time, but it will delay the initiation time of individual jobs.
    • sched_min_interval is yet another configuration parameter to control how frequently the scheduling logic runs. It can still be triggered on each job submit, job terminiation, or other state change which could permit a new job to be started. However that triggering does not cause the scheduling logic to be started immediately, but only within the configured sched_interval. For example, if sched_min_interval=2000000 (microseconds) and 100 jobs are submitted within a 2 second time window, then the scheduling logic will be executed one time rather than 100 times if sched_min_interval was set to 0 (no delay).
    • Besides controlling how frequently the scheduling logic is executed, the default_queue_depth configuration parameter controls how many jobs are considered to be started in each scheduler iteration. The default value of default_queue_depth is 100 (jobs), which should be fine in most cases.
    • The sched/backfill plugin has relatively high overhead if used with large numbers of job. Configuring bf_max_job_test to a modest size (say 100 jobs or less) and bf_interval to 30 seconds or more will limit the overhead of backfill scheduling (NOTE: the default values are fine for both of these parameters). Other backfill options available for tuning backfill scheduling include bf_max_job_user, bf_resolution and bf_window. See the slurm.conf man page for details.
  • SchedulerType: If most jobs are short lived then use of the sched/builtin plugin is recommended. This manages a queue of jobs on a First-In-First-Out (FIFO) basis and eliminates logic used to sort the queue by priority.
  • SelectType: If only serial jobs (single CPU jobs) are to be executed, use of the select/serial plugin is recommended. This plugin eliminates much of the logic found in other select plugins to optimize job allocations with respect to network topology. It also reduces communications required to executed batch jobs by 75 percent through the use of a "pull" model, where the completeion of each job by the slurmd daemon initiates a single message to the slurmctld daemon upon completion of the job's epilog script. The response to this message can be information required to initiate another job utilizing the resources just released by the previous job. NOTE: The use of select/serial prevents the job's Epilog program from being initiated with any SPANK environment variables.
  • SlurmctldPort: It is desirable to configure the slurmctld daemon to accept incoming messages on more than one port in order to avoid having incoming messages discarded by the operating system due to exceeding the SOMAXCONN limit described above. Using between two and ten ports is suggested when large numbers of simultaneous requests are to be supported.
  • SlurmctldDebug: More detailed logging will decrease system throughput. Set to 2 (log errors only) or 3 (general information logging). Each increment in the logging level will increase the number of message by a factor of about 3.
  • SlurmdDebug: More detailed logging will decrease system throughput. Set to 2 (log errors only) or 3 (general information logging). Each increment in the logging level will increase the number of message by a factor of about 3.
  • SlurmdLogFile: Writing to local storage is recommended.
  • TaskPlugin: Avoid using task/cgroup with the combination of ConstrainRAMSpace it is slower than other alternatives. On the same note task/affinity does not appear to add any measurable overhead. Using task/affinity for affinity is advised in any case.
  • Other: Configure logging, accounting and other overhead to a minimum appropriate for your environment.

SlurmDBD Configuration

Turning accounting off provides a minimal improvement in performance. If using SlurmDBD increased speedup can be achieved by setting the CommitDelay option in the slurmdbd.conf

You might also consider setting the 'Purge*' options in your slurmdbd.conf to clear out old Data. A Typical configuration would look like this...

  • PurgeEventAfter=12months
  • PurgeJobAfter=12months
  • PurgeResvAfter=2months
  • PurgeStepAfter=2months
  • PurgeSuspendAfter=1month
  • PurgeTXNAfter=12months
  • PurgeUsageAfter=12months

Last modified 15 November 2017

slurm-slurm-17-11-2-1/doc/html/ibm-pe.shtml000066400000000000000000000603061322351151600202600ustar00rootroot00000000000000

IBM Parallel Environment
User and Administrator Guide

Overview
User Tools
System Administration

Overview

This document describes the unique features of Slurm on the IBM computers with the Parallel Environment (PE) software. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in operation on systems with PE, which are described in this document.

Note that Slurm is designed to be a replacement for IBM's LoadLeveler. They are not designed to concurrently schedule resources. Slurm provides manages network resources and provides the POE command with a library that emulates LoadLeveler functionality.

User Tools

The normal set of Slurm user tools: srun, scancel, sinfo, squeue, scontrol, etc. provide all of the expected services. The onl. Slurm command not supported is sattach. Job steps are launched using the srun command, which translates its options and invokes IBM's poe command. The poe command actually launches the tasks. The poe command may also be invoked directly if desired. The actual task launch process is as follows:

  1. Invoke srun command with desired options.
  2. The srun command creates a job allocation (if necessary).
  3. The srun command translates its options and invokes the poe command.
  4. The poe command loads a Slurm library that provides various resource management functionality.
  5. The poe command, through the Slurm library, creates a Slurm step allocation and launches a process named "pmdv12" on the appropriate compute nodes. Note that the "v12" on the end of the process name represents the version number of the "pmd" process and is subject to change.
  6. The poe command interacts with the pmdv12 process to launch the application tasks, handle their I/O, etc. Since the task launch procedure occurs outside of Slurm's control, none of the normal task-level Slurm support is available.
  7. The poe command, through the Slurm library, reports the completion of the job step.

Network Options

Each job step can specify it's desired network options. For example, one job step may use IP mode communications and the next use User Space (US) mode communications. Network specifications may be specified using srun's --network option or the SLURM_NETWORK environment variable. Supported network options include:

  • Network protocol
    • ip Internet protocol, version 4
    • ipv4 Internet protocol, version 4 (default)
    • ipv6 Internet protocol, version 6
    • us User Space protocol, may be combined with ibv4 or ipv6
  • Programming interface
    • lapi Low-level Application Programming Interface
    • mpi Message Passing Interface (default)
    • pami Parallel Active Message Interface
    • pgas Partitioned Global Address Programming
    • shmem OpenSHMEM interface
    • test Test Protocol
    • upc Unified Parallel C Interface

      • NOTE: Slurm supports the values listed above, and does not support poe user_defined_parallelAPI values.
  • Other options
    • bulk_xfer [=resources] Enable bulk transfer of data using Remote Direct-Memory Access (RDMA). The optional resources specification is a numeric value which can have a suffix of "k", "K", "m", "M", "g" or "G" for kilobytes, megabytes or gigabytes. NOTE: The resources specification is not supported by the underlying IBM infrastructure as of Parallel Environment version 2.2 and no value should be specified at this time.
    • cau=count Specify the count of Collective Acceleration Units (CAU) required per programming interface. Default value is zero. Applies only to IBM Power7-IH processors. POE requires that if cau has a non-zero value then us, devtype=IB or devtype=HFI must be explicitly specified otherwise the request may attempt to allocate CAU with IP communications and fail.
    • devname=name Specify the name of an individual network adapter to use. For example: "eth0" or "mlx4_0".
    • devtype=type Specify the device type to use for communications. The supported values of type are: "IB" (InfiniBand), "HFI" (P7 Host Fabric Interface), "IPONLY" (IP-Only interfaces), "HPCE" (HPC Ethernet), and "KMUX" (Kernel Emulation of HPCE). The devices allocated to a job must all be of the same type. The default value depends upon depends upon what hardware is available and in order of preferences is IPONLY (which is not considered in User Space mode), HFI, IB, HPCE, and KMUX.
    • immed=count Specify the count of immediate send slots per adapter window. Default value is zero. Applies only to IBM Power7-IH processors.
    • instances=count Specify number of network connections for each task on each network connection. The default instance count is 1.
    • sn_all Use all available switch adapters (default). This option can not be combined with sn_single.
    • sn_single Use only one switch adapters. This option can not be combined with sn_all. If multiple adapters of different types exist, the devname and/or devtype option can also be used to select one of them.

Examples of network option use:

--network=sn_all,mpi
Allocate one switch window per task on each node and every network supporting MPI.

--network=sn_all,mpi,bulk_xfer,us
Allocate one switch window per task on each node and every network supporting MPI and user space communications. Reserve resources for RDMA.

--network=sn_all,instances=3,mpi
Allocate three switch window per task on each node and every network supporting MPI.

--network=sn_all,mpi,pami
Allocate one switch window per task on each node and every network supporting MPI and a second window supporting PAMI.

--network=devtype=ib,instances=2,lapi,mpi
On every InfiniBand network connection, allocate two switch windows each for both lapi and mpi interfaces. If each node has one InfiniBand network connection, this would result in four switch windows per task.

NOTE: Switch resources on a node are shared between all job steps on that node. If a job step can not be initiated due to insufficient switch resources being available, that job step will periodically retry allocating resources for the lifetime of the job unless srun's --immediate option is used.

Debugging

Most debuggers require detailed information about launched tasks such as host name, process ID, etc. Since that information is only available from poe (which launches those tasks), the srun command wrapper can not be used for most debugging purposes. You or the debugging tool must invoke the poe command directly. In order to facilitate the direct use of poe, srun's
--launch-cmd option may be used with the options normally used. srun will then print the equivalent poe command line, which can subsequently be used with the debugger. The poe options must be explicitly set even if the command is executed from within an existing Slurm allocation (i.e. from within an allocation created by the salloc or sbatch command).

Checkpoint

Checkpoint/restart is only supported with LoadLeveler.

Unsupported Options

Some Slurm options can not be supported by PE and the following srun options are silently ignored:

  • -D, --chdir (set working directory)
  • -K, --kill-on-bad-exit (terminate step if any task has a non-zero exit code)
  • -k, --no-kill (set to not kill job upon node failure)
  • --ntasks-per-core (number of tasks to invoke per code)
  • --ntasks-per-socket (number of tasks to invoke per socket)
  • -O, --overcommit (over subscribe resources)
  • --resv-ports (communication ports reserved for OpenMPI)
  • --runjob-opts (used only on IBM BlueGene/Q systems)
  • --signal (signal to send when near time limit and the remaining time required)
  • --sockets-per-node (number of sockets per node required)
  • --task-epilog (per-task epilog program)
  • --task-prolog (per-task prolog program)
  • -u, --unbuffered (avoid line buffering)
  • -W, --wait (specify job swait time after first task exit)
  • -Z, --no-allocate (launch tasks without creating job allocation>

A limited subset of srun's --cpu-bind options are supported as shown below. If the --cpus-per-task option is not specified, a value of one is used by default. Note tha. Slurm's mask_cpu and map_cpu options are not supported, nor are options to bind to sockets or boards.

Slurm option POE equivalent
--cpu-bind=threads --cpus-per-task=# -task_affinity=cpu:#
--cpu-bind=cores --cpus-per-task=# -task_affinity=core:#
--cpu-bind=rank --cpus-per-task=# -task_affinity=cpu:#

In addition, file name specifications with expression substitution (e.g. file names including "%j" for job_ID, "%J" for job_ID.step_ID, "%s" for step_ID, "%t" for task_ID, or "%n" for node_ID) are not supported. This effects the following options:

  • -e, --error
  • -i, --input
  • -o, --output

For the srun command's --multi-prog option (Multiple Program, Multiple Data configurations), the command file will be translated from Slurm's format to a POE format. POE does not suppor. Slurm expressions in the MPMD configuration file (e.g. "%t" will not be replaced with the task's number and "%o" will not be replaced with the task's offset within this range). The command file will be stored in a temporary file in the ".slurm" subdirectory of your home directory. The file name will have a prefix of "slurm_cmdfile." followed by srun's process id. If the srun command does not terminate gracefully, this file may persist after the job step's termination and not be purged. You can find and purge these files using commands of the form shown below. You should only purge older files, after their job steps are completed.

$ ls -l ~/.slurm/slurm_cmdfile.*
$ rm    ~/.slurm/slurm_cmdfile.*

The -L/--label option differs slightly in that when the output from multiple tasks are identical, they are combined on a single line with the prefix identifying which task(s) generated the output. In addition, there is a colon but no space between the task IDs and output. For example:

# Slurm OUTPUT
0: foo
1: foo
2: foo
0: bar
1: barr
2: bar

# POE OUTPUT
0-2:foo
0 2:bar
1:barr

In addition, when srun's --multi-prog option (for Multiple Program, Multiple Data configurations) is used with the -L/--label option then a job step ID, colon and space will precede the task ID and colon. For example:

# Slurm OUTPUT
0: zero
1: one
2: two

# POE OUTPUT (FOR STEP ID 1)
1: 0: zero
1: 1: one
1: 2: two

The srun command is not able to report task status upon receipt of a SIGINT signal (ctrl-c interrupt from keyboard), however two SIGINT signals within a one second interval will terminate the job as on other Slurm configurations.

Environment Variables

Since Slurm is not directly launching user tasks, the following environment variables are NOT available with POE:

  • SLURM_CPU_BIND_LIST
  • SLURM_CPU_BIND_TYPE
  • SLURM_CPU_BIND_VERBOSE
  • SLURM_CPUS_ON_NODE
  • SLURM_GTIDS
  • SLURM_LAUNCH_NODE_IPADDR
  • SLURM_LOCALID
  • SLURM_MEM_BIND_LIST
  • SLURM_MEM_BIND_TYPE
  • SLURM_MEM_BIND_VERBOSE
  • SLURM_NODEID
  • SLURM_PROCID
  • SLURM_SRUN_COMM_HOST
  • SLURM_SRUN_COMM_PORT
  • SLURM_TASK_PID
  • SLURM_TASKS_PER_NODE
  • SLURM_TOPOLOGY_ADDR
  • SLURM_TOPOLOGY_ADDR_PATTERN

Note that POE sets a variety of environment variables that provide similar information to some the missing Slurm environment variables. Particularly note the following environment variables:

  • MP_I_UPMD_HOSTNAME (local hostname)
  • MP_CHILD (global task ID)

Gang Scheduling

Slurm can be configured to gang schedule (time slice) parallel jobs by alternately suspending and resuming them. Depending upon the number of jobs configured to time slice and the time slice interval (as specified in the slurm.conf file using the OverSubscribe and SchedulerTimeSlice options), the job may experience communication timeouts. Set the environment variable MP_TIMEOUT to specify an appropriate communication timeout value. Note that the default timeout is 150 seconds. See Gang Scheduling for more information.

export MP_TIMEOUT=600

Other User Notes

POE can not support a step ID of zero. In POE installations, a job's first step ID will be 1 rather than 0.

Since the Slurm step launches the PE PMD process instead of the users tasks the exit code stored in accounting will be that of the PMD instead of the users tasks. The exit code of the job allocation if started with srun will be correct as we will grab the exit code from the wrapped poe.

Use of the srun command rather than the poe command is recommended to launch tasks. If the poe command is used with a hostlist file (e.g. "-hfile" option or "MP_HOSTFILE" environment variable or a "host.list" file in the current working directory), the resd option must be set to yes (e.g. "-resd yes" option or "MP_RESD=yes" environment variable).

System Administration

There are several critical Slurm configuration parameters for use with PE. These configuration parameters should be set in your slurm.conf file. LaunchType defines the task launch mechanism to be used and must be set to launch/poe. This configuration means that poe will be used to launch all applications. SwitchType defines the mechanism used to manage the network switch and it must be set to switch/nrt and use IBM's Network Resource Table (NRT) interface on ALL nodes in the cluster (only a few of them will actually interact with IBM's NRT library, but all need to work with the NRT data structures). Task launch is slower in this environment than with a typical Linux cluster and the MessageTimeout must be configured to a sufficiently large value so that large parallel jobs can be launched withou. Slurm's job step credentials expiring. When switch resources are allocated to a job, all processes spawned by that job must be terminated before the switch resources can be released for use by another program. This means that reliable tracking of all spawned processes is critical for switch use. Use of ProctrackType=proctrack/cgroup is strongly recommended. Use of any other process tracking plugin significantly increases the likelihood of orphan processes that must be manually identified and killed in order to release switch resources. While it is possible to to configure distinct NodeName and NodeHostName parameters for the compute nodes, this is discouraged for performance reasons (the switch/nrt plugin is not optimized for such a configuration).

# Excerpt of slurm.conf
LaunchType=launch/poe
SwitchType=switch/nrt
MessageTimeout=30
ProctrackType=proctrack/cgroup

In order for these plugins to be built, the locations of the POE Resource Manager header file (permapi.h) the NRT header file (nrt.h) and NRT library (libnrt.so) must be identified at the time the Slurm is built. The header files are needed at build time to get NRT data structures, function return codes, etc. The NRT library location is needed to identify where the library is to be loaded from by Slurm's switch/nrt plugin, but the library is only actually when needed and only by the slurmd daemon.

Slurm searches for the header files in the /usr/include directory by default. If the files are not installed there, you can specify a different location using the --with-nrth=PATH option to the configure program, where "PATH" is the fully qualified pathname of the parent directory(ies) of the nrt.h and permapi.h files. Slurm searches for the libnrt.so file in the /usr/lib and /usr/lib64 directories by default. If the file is not installed there, you can specify a different location using the --with-nrtlib=PATH option to the configure program, where "PATH" is the fully qualified pathname of the parent directory of the libnrt.so file. Alternately these values may be specified in your ~/.rpmmacros file. For example:

%_with_nrth      "/opt/ibmhpc/pecurrent/base/include"
%_with_libnrt    "/opt/ibmhpc/pecurrent/base/intel/lib64"

IMPORTANT:The poe command interacts with Slurm by loading a Slurm library providing a variety of functions for its use. The library name is "libpermapi.so" and it is in installed with the other Slurm libraries in the subdirectory "lib/slurm". You must modify the link of /usr/lib64/libpermapi.so to point to the location of the slurm version of this library.

Modifying the "/etc/poe.limits" file is not enough. The poe command is loading and using the libpermapi.so library initially from the /usr/lib64 directory. It later reads the /etc/poe.limits file and loads the library listed there. In order for poe to work with Slurm, it needs to use the "libpermapi.so" generated by Slurm for all of its functions. Until poe is modified to only load the correct library, it is necessary for /usr/lib64/libpermapi.so to contain Slurm's library or a link to it.

If you are having problems running on more than 32 nodes this is most likely your issue.

Changes to the count of dynmamic switch windows necessitate cold-starting Slurm (without jobs). The procedure is as follows:

  1. Prevent new jobs from being started (e.g. Drain the compute nodes).
  2. Cancel all jobs.
  3. Change the dynamic window count on the compute nodes.
  4. Restart Slurm daemons without preserving state (e.g. "/etc/init.d/slurm startclean" or initiate the daemons using the "-c" option).

Job Scheduling

Slurm can be configured to gang schedule (time slice) parallel jobs by alternately suspending and resuming them. Depending upon the number of jobs configured to time slice and the time slice interval (as specified in the slurm.conf file using the OverSubscribe and SchedulerTimeSlice options), the job may experience communication timeouts. Set the environment variable MP_TIMEOUT to specify an appropriate communication timeout value. Note that the default timeout is 150 seconds. See Gang Scheduling for more information.

export MP_TIMEOUT=600

Slurm also can support long term job preemption with IBM's Parallel Environment. Job's can be explicitly preempted and later resumed using the scontrol suspend <jobid> and scontrol resume <jobid> commands. This functionality relies upon NRT functions to suspend/resume programs and reset MPI timeouts. Note tha. Slurm supports the preemption only of whole jobs rather than individual job steps. A suspended job will relinquish CPU resources, but retain memory and switch window resources. Note that the long term suspension of jobs with any allocated Collective Acceleration Units (CAU) is disabled and an error message to that effect will be generated in response to such a request. In addition, version 1200 or higher of IBM's NRT API is required to support this functionality.

Cold Starting

If the slurmctld daemon is cold started (without saved state), then information about previously allocated network resources is lost. Slurm will release those resources to the best of its ability based upon information available from the Network Resource Table (NRT) library functions. These function provide sufficient information to release all resources except for CAU on a Torrent network (e.g. a PERCS system). In order to release CAU, it is necessary to cold start the Protocol Network Services Daemon (PNSD) on compute nodes following the sequence shown below.

Stop  Slurm daemons:       /etc/init.d/slurm stop
Stop  PNSD:                stopsrc -s pnsd
Start PNSD clean:          startsrc -s pnsd -a -c
Start Slurm daemons clean: /etc/init.d/slurm startclean

Design Notes

It is necessary for all nodes that can be used for scheduling a single job have the same network adapter types and count. For example, if node "tux1" has two ethernet adapters then the node "tux2" in the same cluster must also have two ethernet adapters on the same networks or be in a different Slurm partition so that one job can not be allocated resources on both nodes. Without this restriction, a job may allocated adapter resources on one node and be unable to allocate the corresponding adapter resources on another node.

It is possible to configure Slurm and LoadLeveler to simultaneously exist on a cluster, however each scheduler must be configured to manage different compute nodes (e.g. LoadLeveler can manage compute nodes "tux[1-8]" and Slurm can manage compute nodes "tux[9-16]" on the same cluster). In addition, the /etc/poe.limits file on each node must identify the MP_PE_RMLIB appropriate for that node (e.g. IBM's or Slurm's libpermapi.so). If Slurm and LoadLeveler are configured to simultaneously manage the same nodes, you should expect both resource managers to try assigning the same resources. This will result in job failures.

The srun command uses the launch/poe plugin to launch the poe program. Then poe uses the launch/slurm plugin to launch the "pmd" process on the compute nodes, so two launch plugins are actually used.

Depending upon job size and network options, allocating and deallocating switch resources can take multiple seconds per node and the process of launching applications on multiple nodes is not well parallelized. This is outside of Slurm's control.

The two figures below show a high-level overview of the switch/nrt and launch/poe plugins. A typical Slurm installation with IBM PE would make use of both plugins, but the operation of each is shown independently for improved clarity. Note the the switch/nrt plugin is needed by not only the slurmd daemon, but also the slurmctld daemon (for managing switch allocation data structures) and the srun command (for packing and unpacking switch allocation information used at task launch time). In figure 2, note that the libpermapi library issues the job and job step creation requests. The srun command is an optional front-end for the poe command and the poe command can be invoked directly by the user if desired.

Figure 1: Use of the switch/nrt plugin

Figure 2: Use of the launch/poe plugin

Debugging Notes

It is possible to generate detailed logging of all switch/nrt actions and data by configuring DebugFlags=switch.

The environment variable MP_INFOLEVEL can be used to enable the logging of POE debug messages. To enable fairly detailed logging, set MP_INFOLEVEL=6.

The Protocol Network Services Daemon (PNSD) manages the Network Resource Table (NRT) information on each node. It's logs are written to the file /tmp/serverlog, which may be useful to diagnose problems. In order to execute PNSD in debug node (for extra debugging information), run the following commands as user root:

stopsrc -s pnsd
startsrc -s pnsd -a -D

top

Last modified 31 March 2016

slurm-slurm-17-11-2-1/doc/html/ibm_pe_fig1.png000066400000000000000000001123061322351151600207030ustar00rootroot00000000000000PNG  IHDR_wIDATx?hQ?z"E)RH"E \~),,REX,,,BXXB ٵpzl /XEps|9;'OL4̼^pЙg<PNS))>gS(ʔm_}>;>&-䗏y vS`_XF``@A``@`Nmmq 0 O ;Zu{/e}Ng@Xxe`'.0 ז#}[NgZkmvVv.Stj<޵X<SwՖeknwʫN)_kyW9):{zx|Y| gD-WH%~˖"㲵u# 7| J~U*P . eS~~?î00p|k{900p,tʷ!IrSr_7y`reWT?*8000GOneq³ tڛ+?{T`BQ)kZ000RwWVŊr2''yf~t[EQ˷`N󗨸ϛgo2k 0rEgNq|/##3zSf)` H,Gι.##Sts^ ԝu, a Ke^8r;#3l7]J)|CZnZ 7) K|ꔇYvNq1G,3}Oݓ?SnwʊN|JZS+i ;ʗ^K(!_w>*y[CZnĿ{;7 4wO.%OEvߢR<.byoٍtt8)v gf 0G^ JSSIÙ( JǯoAb9Sr>enp8o],_T| ZnWUjQ7[Qs8'g-o8P_[w?c:lNpX) i9`?m^5Ofnopv%uORnʍԝjy׃Kݩ]n &9D,-밟 l3ֱr+ܱ/ϣdKr9ݹA_\KS=.vJmZxKڧz].oqm4!=Ftta a&RH?KÛFe-U X\8 I2# K75o=5#|Y7^g׍3a7[m#}.YYޏ?>#ཉu=2mx&N\"> onkJS C> >24^hw=bi!XMݮo[p?׫]O+Uxb_5%^5WN\korJ4w.b5]M)lj::ѕ!.NMs(z^~Xyj_(qܞ=En7K~F8_^n vZ~۴?Z#7h706rX8ZrsU8kLCީޟ_rz*t(h_6I^@]|o _-` [0QSqٷR; NzĖܦo% \܋}e[z2ssr3>E.pN|j;Zwl%3J4ApV>9Is4s{|=jjˠS/UBy㹳O\^#t_Oe!u* /Q-dkcy`7ez icN{$BVÏtt^x>nYNm#>:|HVA8qx#zJK>>2k9ߋA| 8-\ATr u-c^ay.O5%GUٟlI/}[}WZ^r,9(eX۶R9-w,e |'pyy 2F=w4{;u}7Q]rߟӲ)s1\\mcxl;pr˖sb,09٫ϵ{ױ.-?rd}VR cn1]N |YZYgGoK]8GK`5MTaBS<ʃ]sŴ-aD ñS8 Užp~`,eژQ[W듯U4&9(~g-uG ϣE<׬#y~8w^z]Oq4Q[on!oxIl{Fwzۻ0jJTy۩9/;_cW;Qι"># ɽrW;^k |GZ^߅u[Ta7;&_\>sK]2 = .O2Bpn ::|J7Q2-7OI6f.Ai_8s/O}׼mjX> R[N}ѿ*+Zn=y]{#QunOYz~x9oexIt/-+U ƻkp`Syc[,?qgS>9o[iu] n5rDnܿ._"O2Q4߬*z~?mc΁{2{2o0ROS4n3a)_n Kon߷\'ўW-X`HCTy3 h+-71õ ?{ey~6"HJڐT(Ro x8n_lXn/'~haKLszٺZݟ[4hN˴[ hl%^}=  r-̲AURFK4^}(?+^4F3/Wa}=?I:y_f r;ږCYK.9h {~ݹ&t? '|bzIr-wϏϤ-e>kq<8[g94~Zw9vE?_1u$ټR>9ae 2F{}- ppq5ʕaw9~Kݮp2A].Ab[_|Jy@r.NpV;4 3Y* 3VFF'7;b9c.Vݭ/UeQ)Gb cJFؾ=aE>;2[ xˡnu]ޤnwI8 m1u{OVj[Ɔꚯ-#M+`G/d}J.U3k9Q5Hӓ%UFk~X >n]<!>T]?K=<]Y=~z`rzϹziWFNy@2@Vc.{ܲK.Hnw59֮a'6Wy*:dܵLÑ[`r+oW``nKʥ)v mG9bK7|)O``,>HH4=Lnu[Z::vg۽$W#V#ZzC3ǽ-#sS u/]ٶK+Mezl:lq.ܺ"wENky4}[zKkn^:e~HVkCZzl eoRd`ukn )>A. iE$BMTykv 0nr{>Os1ltتűO%r$4y!G}7o.|M9֣+nob]k)TxG(Ϭoo ʁ^T)+v 0*pĨ8ʕ{>Q\3"k1r*uKwWDZitVc,~ Cnusv^p ҙ;nU(6RwQ0ץ)pnIg4p Lz<(Vur}P[%'V`*[_c7 ~^ ù1&2a]TK}ڹ/Sdk>Lz`i].i0nU/q,&Lf8==;x&k[Ó2Y01Gq戃ttaFq>Y'O-`Jlű\g|-~J([Oݑ1?@+ .| }X\Nqkǯ`}`{qCrRJmkA~r<A^t.H|yc}[G4[۩ەmGk{V;")$.vUg|層׃o0r7eL.J=W}n8||E0v=u[YB#^2[pyv (/ZNݖ|6?8MkC)oRz|O V::@%x`WOq>&r4|cf cԣ*Ar}XKv "Wc4‛W:tq.[NݮN?$h^JnخA'_I3˥H\H8}9,eXkKa)N=`kv E{{67aͽ:.ٳ*9!v .NGGw:5ui1<)`,kikk7FOg]r(6ػ)=nظ|Hr_;m #叒˗| [\荣}\ /{Bɗ.cyY](0!u8~KɃk^.<@8g)_K[F?]|>J鹵z\:mC8sy:UQ{#duCpml_`[OqtoLR˺A3rlݝ`6Vic0ar2Qj_~)2 p72e ?LB)# 0,#00 0 0 # 0 00 0  00al 0 sQ] Ox0y{AYye0fB@`IS^ 0, Ny)CKיw#|;S8h4EYwwZgn[{)˸뙗2w]T{Sw~w0xX>3}ݸoZkU=}9_#=U!{-{#\k梮Q2O1 ?֟",)_;{[zp'xUк;^ ~|SAwuV1⠲?q{2!X'|_G܋'k8ިo^puRyȣ{<,70V8(}M3l~q=n_o9PIIp#ǵo#LΜQu*u{-Ι* 0ꔕJA.gMOpu^qh 7i2IU>5+=)!zӆ,ޤYC}?^l?g~1'[{o]/886˷ 0< f#LzǨqxZ 0.^^m^$ަ@X=BY8xo r;uD _U&:`@O9u>m;uc4_WA>0/qޏuo7o;ԕԻx&堕ϻ Oj'ˠOsO-$ Cae4tI).7ܕy.?z  00L<3`@FIVtM:;<_ 00):fA`J]mܿGzo! 00 g>ݥ*H7"`A&Ew|H] 0 $|-^:x.A`A``A`A@`A```bX Zvʜ \Ӕ 8WH0{OHg#?,plA<2\4=~D{ݿ/T~å>)4||ΗFm4=s'?xjGqL㹗b[^+QNXF}yi>+?:\:  [:[ܟݏ.`7lϧA`gGs‹{5O{oon:(=FVΚެ%3k\0'=/ıyLޏ@4W1?:\:Cm'g5NL4_o5۫u[i,{gi {Ө}mY 3Zp?sՏy{kqߵ_u7W:3[~~l5֡Lخ=gRoƁ%܍[ g+o+nRoTu|ܟ:ר#ܨL~R}7uufxߨ_-W;n[֪m)`@\m f3?wGkǧ 4sN;ZvQ*.}-n{ixTGSvq#57{Oh`YuUg:> XڻXfkA^:϶-] =my3Qκ p( c6И{]ZmFUS"x\O_P!"9u[y[6oFh[Z:ۧ`,7-ǁW^*E _k{JViq%ػFH$pԽr' . !?n9zԥkРqj;x$:5 9|l2CW#4pCzjjQ$)&2_|(##QhJQݿtB}Njy1e['{˼9o CUNm'Kz u_>3`s]k`/nx.q wʧNy)K'r5b}/fNy)?;[#7Զ?lNy)-ȷs߾`@'N4 KdpOQ_y^ۼ,}3/ W?֋^ޮ3vVu\,,SN8uvp xݖ?|s E y0p[}b%4opy|zIxǺ]z >ARx, 'g0/ 07[V/ ӟ=, "hj:͸YG#_nGkXxt8b?cj[Ale' ?ڇuRiNwqAq/Xxp7#`'q_&`! Qy޽ X, De),7>O~ݎ0 Rz\_J` /'SH{?U% m9\Iݖ2'^k, ?=^W43uGpu\+ \` |6u;m# Z>߈ۋٖXLNTl2Y X'3c{ɽzFr>/}WQ۶eY,<|^rt|2)L K|I!8Ϥ  yJg3) Wέ|v^!"- fbLG1~]`@G|ytz|~&`'.L|s}! _S```````A@FAF``XF``@tt8E9ykd@na4eWƙ øA@``@@A``A`A@`A```;+$0?˔)(`8? (T|.%.e #_}8{sZ(4 _{ i1LiS/mQO"nw;mlugY`2㸛{0^G /6޺]Vђ%Fa:>enD-X7hxl\~wʂpwծHz\n=A)K鰥W~ O}¯nK]k'k~iouNylS'oGS~T9?픯q<8D3p/؝꾹N)?gG,kniމGInmA3\Vԝ9``Z\J[ϓ<):v:Vō_zOqs$]}j˚RIǝ p<`Q"VkEtHَ, # _OZ=e-}`ako}7iw`)X9J}^`Iމ@&Bٽ3\n/mWsKؗNyYsE&,?t3u>j+9p/ # mUn+/"`R7`RSf`R-zc׮ںA/-a?ǎ]pm7kDDs 7`RSf`RnЏ `nЗ#E޶+&] 1cL{*J.#?m\鳔·(P |Sh9t8Lry}_}|`W0v```@A``@A`X[)Lܞ۳c 7:e)?;߲Mr{ oyN,xُm)r#i}.uʭS*lS[)-[忬[\.خA+X4@ޏS\:%g&(0 Od~և_OSIm#h _G=S>܈`@g%ZG7;A픕rV]?X/KMݠbn 1Z` Z׍*4>6Bc.zP?,s}#uGR4h NCt3o.x)wdO_畖ӃFh go(_ Z?#&*iJ.e^{b0],vXil Ru#LkO-7nйU8jog&$8 OU I_ 'G=p޷}7B I4y_喠ڼ]w#C?+ϥnk/HmytaWjoYٲf.ݠz{<T=nj_pw;t);:̀ typ ;0 _^:s8>-soYﲞJnMxkw˼OOā #r|^nn/:y:y%^`۵ORw^AwZKG[s65˽t*#`7[nZby_OgDx"E,B8_w,0-4d`1u+^C@SS0  0 0 ixeK`EWͤePZ^SNY+`p{r|̶+ϻ-r>؍/KNayW=[x}SD362W;eႮ{<dS~AΕcmo`ƵاcYe9io?[>_vS!|ӟLNy-H9{5;c9̯.1./`-D:W~֤aeF܅prx\.x)ruZxLZXs_iYZVzZQfq g'_N]=lc-X+}Hbm\-gٶo=m9>G;in:ut1m]Ve]XHGJku^sվ٬1ίlՖfپKr[Ӳ~͵peuوu;Is7<-]t|gBYp~xV8Wz7v=(=y)خrEg5[*[gt7?~GoU|߲g61{= fe\oBc|t ='-7Uh٫^_U~=xj5^>W=E4Uvc~F{YXo|nvb7߇ן}A[|XzPνmZ~ON[Z헖кղZ~Z~onf~- QA.kUڌǽJ|ͪ!W#H ҵ{ߊ@V΍y+\MwRUh)ßYnVoT˝`\j {R^W\LqJK^rWf!Qi߫BNZ|&+~lƙN-7E2߿ضm߇Cu[^s b_܎缮\[UX߫VmW}~=vu'gwlOhrZz^_4NW:7h^ǫzuaڞϱw)'wb\嗓pתr|VfwUElQmVi#oIQ!u|qzoy] mY-Hݷ2uYT^K jUK{6pw-ۍ mYXsҨ5>ulTcx7{_tmqVtڶ.yӲNiQ߷O?X>[ǽq2 >yX~{6T'kj8y<g};]N= Iǻ2TZwx[6'&Kx\ZFv[6]ڔkK Ft/5͐V_׻(A-?l*G9Z*}F} %n_np?PS5uռwGܥqA$3,$Ez}5.z̞GwBH׺&^o~UwOn֕S%/T߳+_?GW+''>V;8؟%?lY~ qOnk׮4*m(gۮk>lu;UY'u}z|~FQaN7"tZ_T߬Bj{{ V#`#/wyb]?6u:Ww,42oc\si}uR6{|=g_g/ ?G'}?~ӏ_yp *m?(-Wb+K-G} 6_*2'.SR~b[U0~>;{G:⾘-C_|ގ'.79+}>s1< |8 tx=wϳ/'flipJz\>zo oT6_S7>7\oiUԮn{[- nz'g/cr7ύ A=,ig=E~۸gO|JՇ=}RY-|?crτ/} xܼzi.gwf#T]j0#g8('oɋt|Te]OƉ0I9'9*FeG[Fm!^cW*gsw|R\Fmu%m;"F,#5*_#͟2zRW5yNK^W۵qBX+(xZ F V\>7_? X흋}x(7,SN;Bjw (n(8 'J|x {,+'A.O/fƲfSw| Lt^0DܯJ娐J ~l=*wq?@v"%^c?ʗ3 %0J([i @ =@ @ D@ @  @ Q@3Nd7-˽;gPv$Γd>\SX.b.}5<\l)te:C:W;{ԃ%2kul}UmQ!rcʎ+сj먛>yp`O]/D;no~aaE~r@jxK /T|;c窟˳&[A?x ezE}=,%@yЗl^j0kߴ]~UB!_a3>GǼ L>@|uu4v<\JBձ|N>s%ʾFn#j P.݌1yLc :AC#==c6|[]m׻{- WKw6.۰K^dg >6!jfgT_I쳻w"{Q}92ֶlDSqU[p Ӟb6邶pqо#wܧvP ӞО]mgƕtɽ܅/_~Ԉ5U\$!B!=O!B!PB!B!B!B?l !B!B!B!B!B!B!d{!;`)sqj%/#pi&PTWUɾK<@Zo3Y򔿹u/Gl77?B!?Bq~x'ԟcد#"Q?I4P#o`?v͠}_7[ne_'i7B!pxu,Ϧ~/GN{ԟ l pIAO +(D,oPж"Q݃Y0!B7RGPt7z7ne=pյFVnM>B6w |łN(A0ϞQ*񞱂ۍy+$p}G{Nq\ߩxm 6nç6j7Ϫ݄~ ut/GT_zT-ϰm3>(cNцZ'BȠ#`bF@iK]7;H8o\ *_ vY4yu!6!ߒ)-mKxǙ( z,+;/o0p+yy.0QusFOYʷ:yp!}`L U+ g`i}|X%h_(i72:cc 6v6^yz=‡Ү[bl}"d۳MvNe@@ 8RBE|qwot}FG##`Oo';l&Rc[3p/~I澹DΧnI>3f0C۶hw`62ў_`˶ׯ{6] `^#Ww&S>T<+,З͵M%&zFl6HB}~Q5}9su~CW X|Q`B!"&jlDYDJ{8fxʻ 1g H&[vF oޚz3.\V&"-!~=JPMz jDep#)˖E4eFsKYb-*a46"8f߃fYb?V q.DLu%k_,c6)CjL.ޞ6z}-m)]|,>t3"D߬ɧ̺˖`g/rp-gmB!J.A13j*KfbE<o/Af"MϐRnis }S=8rfځ >l_~u8p 1`4 lm ҬRvPc]{ 򵳳rWy9 G?lj7jP#t1)w)QǞMHytᝐ/L>ʫX?G|Ue gB+R( !7""|*8>5 w@j̼gS;}C*#~e 0_em&O!Rva!}z-ln7cM 50a">#m4YK͊˖ K!|Y 8{e}* @( Z$:Yp#%5t٫Z#곖PgE{*e Qw ;ӬSʸ ?Hvho.{G*i `B!Ӽ>k,ŔEvp~U+U;\d|3[<\|\q&6E^!Li#]׻sz7G@WҖo|Rb< ̯Rm﨧D_js=B!Qdv&﨨 z/BוsX>Y #*GLBw.!X^D0XM,oͅqg9WE웅؉-+{'LbO) fo=MWISp ?+9UF{4_Ʃ}z87Sǵm|.i(>}<(k9ꚚNFX?ݷsKI̻?N6҇k(,?}/*h7 9}'cac*j'B!B!B!B!B!B!B!B!B!B!B!BGB!B!*wW{!#Ogl пӳ&M~օ?y+#K!BrXFtW!~?:5b^ Co֯B!C<:_|(7(([;0!BQugh"7 `_{Lc9^T_c^V|XB~&u2Ok"|U93m:6:g _ةP]˾5Pee/֯c֜32Jt[T8\Xue5yïgz]fpU[7vH&k/|; ,GӰa uY._7cǶ:j+Ƿ#5&K{ ) !B ?|sQ. ~p˳9JSq[ Nl4%m-evSxt`>ms;_/3~$C︶s|||_ ԕ%#fGOGFfwKǹ)P;f0"HCb˩]QQ=k; ]+|H{92@9}^[ Sզt}L' ̴U_J.{&tE Ay?!_P92>f}67ަ:TImr`B}^fLt e30\pY Yq>D[UPBjP}seewe#~ 57*!BIMQ>0~pgGJ9 vJH鿸edd XfK5FTKm#v>B#ߗ&-OE>0BRDEL\ =m9o_ J  `vD>UNJ9S.B2h/ l f_cjQD|zc>_/85m4QOreB!$y];3s-^| q "k*=2~eq- V4T "/v鯷0"uju}}|uz+s2Yɱ;žlۉ^`=PsѯPyoAK^&s x>ǾeNzsjpR>?+W">= 5VgO( !Bg?W$ ou~9G~%{Pqby}yҳJ "2>G95$>#kƯ;66U?3į9%VL+0l'~+Ld%XNB-^^4슟+Hs aR>Y=9D<bH*y;PB!3;~rkckp3?JrEXfC+9) !Bqt 6@3*f]y-q.[,Mim ~s@wÜ7`}㳙m^, sij_.fb@X= Vrē<³xxGP[&YRSob,3 ku6򪣘Xڏ`vgM-FڮKm/ `/SRdPn=$s~z܂6;Kȯ( !B)#0{q]cm]ˑEGq~UlS f@ Н}\nI Sz-S=KUXdá㯆oT;69nyo ~lh1*F*-e f\y[{0m*w_'F׻be%?漮s.+JT?[ Zf ߝ;S[5ȵ\xQziʠf ɵR=Wځ铗46k_>w !\ PB!$}m95m};t/%R; rY\eFAسv{R|Ẉ+k kS_?gluC=~_&ґ~C8U#睨< nůc9>*5T,yYǃT!zlK*)}A]QԽ}ϒQqFʲvMu_:p; מv}cRu3X#}{?t^|:9sٻΏ!B!mrF!B!PB!B!B!B??7[B!B!B!B!B!B!Byg? }=%Kt[iwCmU層) p;Ww?N=K!6f?#:7+ǭf>_A?MqhUek~ޔ_i+yNS-#Ӵ}_U{q=d#{K!3:  }y<nU_g|8ƺ/D'ǯzuBH?CwW `Oy~a鋿?j]ľ|B,H&85_/D R (R* @6 eG;!y=DkU݇]۸ ?kǕ@hrLdO[&Ծ\SJ.Dr/'u͡4lUDK W]|SҖ^Q(s~\W "I?@ޛh_OFOq^ ~&2r`>uPbCĠn-#~8oeDpgܥA7W?l?[JOD~[&FOFqxs+yCic- {b]&-ٚ:Ousoyqsu7' NuTo?/0}w+꾿a?H.{d !Ͼs#~c-G7B!;A:G3n0Q \ zY ~fؼJ8,p > ]k"|"yHR5#9;Kql@HZ5DE~_@o >J/F.BL ~ m;KWv͈-fTH_8w <l}ZE]¨ \FLwZ2׊M߸ և0x2}vV"mOVz_Rut,A@6xd-B@zwǻ8;z<~5}*A콣~qJaۧ)'B3!YG@GIuلQ9zg2 p"Ak(0B %'cԑ\+-U?1AЬ\]Rܳ>pş!B9o6ADV9n!o]v{ˤF cw x-bSLFQ-r^OD(kO|2i'"W#2紽cܠߛ1+zȽs*i̥>iPp,#siöX;>'s? dMO.{NYf|9Q9)yG)̆gwG; WʇE96J iy+R>kNNRE;Bo\Vt`ƞDP5tZȡ26сaB?`<@>*T${Ї gmfѮJYK~XK~׾Z}ބBI·!e}@G0?Kdl#W6g2A9<1$fr-ٴ~iDdl#z\MT@ 2k X|qkR_%eÚ@ja ^&x'"+7O{"Ҿd"{qP,})0hrڹWVDm0p}}XچfOȔ4"->ιw{en{ƣK[ )OO=|?4Խ-d~8//Fs\CľDbڻ.8!B syXrY 0_VpA>E6QϹ G$ݞ6_8kC>*@ZȱoEٗl*[F]6C 15S^1xwpݴ^2v\*!gߐh太}cEEoW#M`{{JT>C5^_\+pMugƅ_"S6-o W}N|mD\-TZ҆.]LgS@y쫖C>v ;&G; .],iW,ϧ#B@;eKN '|w|3D Yvs>k߫O6BCK Ipގ tzwwï qۮ9| %i t^/2BTTP*җD߭I} x{3U׻$]uZD|AW,/˙:U dXDx۔}|w+Au؟\2 `6d͡ǡMk! 2eK!DoGeC +7$!=^V~T_ .ri@\6<;>O*B!@P)"&\xs%>TLrc_\7>$?7^ѲDd 8vW Ҫ%_~%˟ͫWvl]ZC9' |UuL}?5  LՈ;_S~d ˟ !B?|@7ynJWw 3B!kɛ_2뗗6Z}Ϳ3tohߩy}Xt{n3fI%?& !C(_4 .c~mHygL@y~'_$/v˻FMeGR !0!B>$׾1 AWtF7pP{4}y"k yGO~&ND>U/|z/7ԠQ|j=sCy;rv4GM:!tÞHo2ެRU|~yȭ!MnF#<+?iA }G;Z}T=oF(jqx߻|׺յ[Źw/>֑:wkeKFEG\=n'7._TGG)kh7H[mGP3/-2A;ߧUݩzZԯ`}TGɽV7PB>nU=_>Ң+MHWSBq?だX$3oZǕ U%/<_zm&yxIUH:)^ -#.P dֱfϣ`vc7xV] A x꭮꼞c8}zgNM5PE/fEHba/sAa1P]3إ]R`T b⼶jK3Yߧ.U~vxf-hK Խk_b3埒L8Vjpd"Adm30lQw\ L=K\w BTг^`;p75X&L@}kg hI0>3IHX!fߤ/8thic@ `nv척- yo u#Pm3P"WkKۅ6ANYeO2[v'(o?:9i3Ƚ{K(*}_K 8Ȍ{ŜS1]9oޅWU/LЃMg C $nUVԀcHڡ+G+f~D!/b4@3 ` tK`Lkqe͢9u%|lG搠W"1IDATk&cG5US'팱Ff{j*H,W.&@NLڣF"R|Rm38qUː}ޚҥ=5xt~PI~̹}87U dʮ\oHPGȣ3܅Bf (\c .P߱ RdN wXmFYJ?Cu=:nDܺ>ڷ׳=̻H?*V]3KBFXƞW"u"7+6W/GɄy,ޙu!G TEVHDeK^CH4?U [5K'Z,;nCf#loJX>1\># u2V];*VBvB!_ugP> Xp]R~]25 `~d岡\%m9jՈI?<\4#t|C#D\)8#}n AV]w[ uOz6OlZW}h@?K;yP$ O\)o.uly/ca`J[K.{iA/-N~M T,e=ۮ4}W%e:B!Qfr Y2D ecq%^]x:vْm~V0h+9К b;lNt 7žDhZ6*'9PN׳9d,uZ,|nx݅g~J(lYP1>n쪆<\ ŭ˖n~V Md,2u݈Sur?%o-R.w5vӥ@׻aKOں~@+~Xv.q+߇d`mWB=>@43w]vi)z <_|ԻJ&WVHXq zRf p+8˞yA<&վy\+Kk*+γe-:g9nͅ^:W=(A\G>Fp߆ioAd; ; `+.F\*<B,w׻HqyzWVRxX-+c%QM!2$0έ2+bDf#1K%=˸ BPN@`B ܅{j=&w.gץxM_\J֧}!&\S y識F(.{-)M5v]l |wWgsec,yN3t([%OyDNڏ i'Axj/Vw݋})X*_v]Ѧ*EA=V|]lbcэd~Xx}|.tP5LOϽj.{\N3NS&guS u 0PVd@O7̸gŞ uؽ%>#@|Gt?~]e(B!|/u)VeyUzy+tL(˾ n5wc݅'{Z;\Nת /3N/]N\ = ?ϙq [6V>m@ފ Փ3#.ۙ5Tof"TT͠H=QIAX-!o[;.Tz^Xp^@;j"jV=ƶ5׻ܶu3Ө7)rDLI9Fv{OyNjh7#fMywD{~|ߕ#}O͎9՞uhE СGGҖWI| kM8B!B! y;B!B!B!B0!B!<5]MB!B!B!B!B!B!BGޫx~&w._ǽ!?s7^Y -yf<!9>{tH_6cGN!B;qw"xXJq3/K,sig?[ʶ>0L[G&gFL6}+h~"6%B!qVlUBx]=/!>Oa0?!y!B!/8#qt4_|-E~-<[Tw/Ymv=B $3̥G1lbW>)8/e{-ApUqN}@<(jy#+a:'mi\iGB!f}̍_`Npw"z" sa?V<-,rM$~Q&M]#KEn;(q{^ʹA~>KUm}ô/&O7sO\y*@ymFwα1ߐm+12bm]f}b7'g,oCs>˾yM7i3Et*C_-N1@\37 !Be^>Ⱦ@y>._]з-;>[Tl8٘<23~K׻ HćyxBZwi"yZg8g_F8g'H[ Uf(\V ?CWl<$u*(/ʡm^r 6%ve|vosH40 5MY|J?U~>I|sGr=RiAU!̀NT{>ʷDy im!;7YhB!YVlƀʌigSc32;gr.]U(¯|2cP#l$7FEUU%65_!ڀ.ϝ5D8}5c_Y.T,'QU 6uh˯'Z%u\ 2+y6\7eq-!@Ǧ{h GHz%ΫEj@kv]B!G6z3AoeeO_C'w}}˃QO.<;}fQqsΌv0n'3~s*~"oa@l嵡baQ ֋>%Q9Ƿ( Pʪ.ra [BYB̩*_P#6*wrکxU-PewƯB! Or!7Y&/zejiEɲ,}`o\]Cy^;N0gA/Tέ|yFw҇}1Jkv=b/ATHOTD4\)UYWоKe_ rtMnhw7N<|61ˮ3Mw7\ڊ +SS}ܮ.A>p&Xzo!B?0 $$7׺@ڂHFD~t*w`3Uy#u36K_(M/.{]]Ϲ A,\ F_2#,Yr<&}C2׌̏& (5H6#vE 71 Μr6sz_\6ˮ_$WhOy]R7Qdωke4 <>Ic`0?Wm^QuBmcB!%@JuP(3WiNlw%. {=z6]s#]&Lz .I#Diw$A\]%jvw>j3( 0,]A^si_U׻#<>m]%J5ˁ6_4_v86>z5rLwOnPXkg!rSB$pϘ`}\&"KW6qwrsPO~B!7{ w64? ДrIVXJi .yտѾXFTj¹#mF.<#> :?'j}O?!B!st!B!?S뗎RqB!Bȿt?ؤ;!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!R,KwK&IENDB`slurm-slurm-17-11-2-1/doc/html/ibm_pe_fig2.png000066400000000000000000001341631322351151600207110ustar00rootroot00000000000000PNG  IHDR_wIDATx?ha7H"E )g"E"E8,,RXXE rA8,,r'E 9{g9R>o~qf16t'3wgRĬEDeW|KX|U%%,#0 0  0 0 0 eaWf._m^7Co!>m/ ncf,^m@XGX|S rWm*֜tnM7ތݪMw_Y^O\9: l^1aWɅL}C"^v|4dW޷-Yx䠷pmGcL}K[ 0 |67E~tVuƈ^x|LGmz?>C(-W s~l& xmF}, X@``8sU!|kBh#E,e^.qp(l 0G@IQƹ,\<*`X```@XAF`9[c?V08ۈu*`XAF````@XAF```:=U!'g&W{n6>Wt.xL if>_'zVM@?|EKFAڿJ|pWŰwBFSiZH+tx1Qnw1!# &cYUbW~F}L͞g[[8?j2ކ&#Aȹ ^+>Mp,ۧ A>Mp0i0EO,} ``` c&8X6cO @F:``#c@XA8:``#c@XA8|c @F\-U!,Xw@(G(0Y:YS7{k(ϧ^ o{wW& +|+ƿ" l|872ۣ,ssʇ!%X>oRH+7u~Kn-[\hy^y+~3<\=_I戏oy>ozZG|z2^lyY``@!}N ߨsyP6߽olLJyBp yZc>ǰύe7'N#/ϘǍcX..J'Dy?^vF!XF#rUB/00||Tfhn|jϱ^ͽ`^|myߨ{vG] 0<J"L{=jn>j#8Mݏ e|6m Z^^ijoە``@\|}? ->.[n Y|E~< x]E>1WLgZF **ԇ" Q7-ӛl[Ա u]$# p9W~W-o?V4{o1c[_3MS:9o0 1Lז{b˘CDcooN&"`~?b;<~n<{$^7MF}}q#7X|EGK~Ex5?TA\Q~p<1})Xė#$Lf1,7.WJԫ0ž:G/+ no~ո_pTz1ͽ:B=1<_y  q 5-݉. 00|閿}KMoxOZ[2?X[`meU!G{>1}jq& ~I J K_We,g5Ϊ~ʇp:5s xiue,gߧnG8M" cyԏU``@>u5^[.ٌ|n+iC?o{*I`{cӪf5,_1~[?w?sd +/A뭎y\K-<-  )c%``@*ol5,w+7``@%?")7ދP|n0Q*`hnqO_Qfg0Ʊ ؟ ```<DW00gl1dSU؟ ```G ~,rXy+# T 8(]Q , k*``  8 qP*@S/wVY%p"nwꃪ;&YԆ.k00cgW(00cI}LNr``R?jhnn,ˤU iB``@c?zN5,[ @AT.=kYU _ƀ4(TE;02n7=άTUl- Q9 ?U=X&gJ.+'ro>&J3Q%?P-c\[m[w|ʞsMt3In27VP'}c~# @cm1~q*9$kվF׌$Eh܄nJ.|'7cLTq(ZtR}q*NndLOlK-ewWVc??Uq+cX('niֲ`ӱ!0J3Rz??)AaKq>Θ8i8@ΏVۋ}ʸ@i*.*%t1GHVi3%Ø.?ltF$#_<|`n {^}u 0ȚU*ONF@UFhºNA{6&#4a] ʋt<2~R^gsh `T"~eAhºDl8?d%.A>B% 0|&a5s.u*AO}d]Zo߸@WAO}d]Z  𩏃Ǿl6^/F^ʇG R`AS#j[iD,|z.G^Y?$[-Ӻ]KzWʇ^߈8cRYbW&?``|$W@h^+۱!uWz}gËjn:|޷Qq>`)|UF jl_ OZ|o,y^1L˼oï1zڵe;'s:wjQS{*O*}d[;j+>[,g=n#ߌa 0`ipIx&Ϫ}ޟ8zҲO|OU6[e?hl+s1רfk}8?n9uO+OF'OA8ED>~e Y^q ļGa1Կ=_m߱-̧y};i r47,WXιS Xec~w_\.r+˹N~F &Z 7;𲯽5`[_At)ApxHA>E{$Y!w%jiix>CT~uGw<^ c~^|\< ,ݬ;9 v^ub&jG~2c/R;bv'rBqa'JYXJ|KWcn7~T;bSD  z?_}/=me/;\qm{{2cV_h%B^AD,1:YO+]!aV:ZRVѯ5)܋志6l ^1]Wc{Xձ :o-/,i~wwn:Z3tfaONI'fձͯ49`/j_˻[sONi8%oG{u+>t(RF'K^<׫^=\9 ʽXNOEètv Ӷ4rᠸy+T@peU-mҕ^ XA|z7rGNfet/q&G!UzPamVoFp܉@~PZZٗ}tlN4g鵝Xo +㯶lWy"kzވi :Qƺ v[߿7^: ֽ,7;$JX޿tf6[7lf3r w[Lxͫ'nvq֡d] \l1߮6uwN4D^>tf3蟩zP^JM`! B% pΟ CcG$&йp22^nx5 i9[.^iWZtoC2, B% _1~caTo;D|_['XWM٣v&'X`m@~Z Kq|>(_ZMX &R%kY_WʤiqJ8ml떰J+,^n"7㋏ {h{ZW6~v~=0wR{ YUYg5/2^qz+Z}ǽrUutUUYvo?_ϻmO{eAa嘡o􃶜)n0,'T\ZP8qCn~}9gң4s<$[E\tz~( )߲;,M/!7cd>(Kz~7Gn?sUy`RQcjQ:Giwq6: ޕlQ:֣4]K|:߂4#Ms;0;g&R0tv>3ymDOңݣtzno7W6#{͡(=?]-=+ͻ1l-pv'`8ңޣGiݖ9}.0u(mo`rҹ)t[~ 17}oZ6`|{JҿR{ҹ-=J0~]wf>`<_7/wn3 `LCt(OQ.}>ʡs+qMs;ԣt`K\ToʹՍsX|JZb@?H=JNz^7;lo On]ΚnHӹ CysH^UE0JsZznw n7Tz^O=JNz_zocu*=J_QMN`ATz( M[o>a 9K?S:wx3W&ze9V=wyO{c߾h)XkL[o+k1ɖ:Ye^, \(ĥG髪 (`( x?"MPދ ۊMϪ4?cW1ռVn2[>*N6>j>y71| QUM@,'n*z^Ozr|7o'̻\)]v ?F@iwx:Yy4>ߚ131/8uVܣGir?aaUA}M7"Ou:.U_io]龏p$8onZ֣tnΦGi2z6o_<7;!r֝"kf^IupNL~6`{z=3 $'6[ uǀ|/ƹu|ryg\j;CTzgz=Q\?m /~G.{V -g~QF,*lۍ~jj?ղT{'ƫ?Iz.ң%|Jݽ@zW<|=!u?VAX4V~<|K~|` &=Jupw+"輍9xϚ[<{*(u܄yn͉Z3c,wn}|2IFy3Bu|'1rG=.2l||G0pQNh_B |?_ù)\QzV5,(1(Q``+޸|W[ߒAf-~s`CN# NB ңݣtz``ssH( 0)=JN=JrQ`ݣtz``sS&Uӱ7q.`s;*=JJ=JN(=ժr~oT 0h(=JH=Jo$=J&75M̿T4A8mU 00znKitÅ`;ң^^4ߋU01GIo^E w;w0905=Jnt:7qd:+eWl. (;jQ:7 D'J|xf 00^J멽Gi<{^O=N|ipHO")8mg~s0=J_5ڼ1Ryn(e(7|qWz{=%_+m眲 "+gW^ߨ3/w|f/} Rcxi*UYWFsSTG#r3X^{`A3Vz~{~FG|U;׼^~a|L!w1(FpN`y#m!7/{M&@`WSw2 '9O=Jޗ/Z+&o*=Jg(, Hi+ sq;{GGi@Cn^( p,G鯩G|O{q7)Op~ 'TrҏS{ҹX=J[l| ίL.Qzq&83%Fg?|Wk`Q_yIxCQvjQz;{TUpbw Ow:pGGI:pdv>4- G{^K=JOz(GNa0q֣ۜjCvӸ@櫿Rw r3'/NibT5ңt?GIPw㴾1MU Qz;(};Q+_=`8'*WգtrGi. 0=zf]m)v0=J5qQjb p2|XцSznQ:?X XHM=Jt( 0SޣGihs?#T\\ޣGi{*`<\K=JLWfT͡z~.wk(WϦ/6>ζlhePz{~.W%4Ln)pY0u((-DN5p^rҹ)t[~s00NrXG_G܌uz("(.v00FQ:7ͨ``.z00p$xzFGig tQw# ӹ `'GYV[ҹsQZFN\Qz=(:}ϔ;y'EfcX\T5!mL+we`Ȯax7{>"aQujl#uOV?=${7VNCQSjQa:Z7pͪ {e;JE~i>|ZG1`6ciGC9aY>;Kۉ[Y_B{?Ul42FX2B]=Jl ŷ/Y^:' -DP,!ers">-D݈y! + 1$u7.|=/ Y+=JJ=J/ͨRq 9(+q}ڿ;l[>˨I>p#ۊן #c;1jK^ekk(>|GN]4m=J7M^$g"ӍϞbո 1}UezתaU}R>;BމeF`u~ڛAI= 4` por%95s7>g_i>_on {+yrb,#uZOufWc؝Cӎsw\ӣ1mvK{tj@~2 "?ĺt9&oR=%7A}P.ӹ%1 manZf@Ӏxր镫7:p ܥD0xEFIoZ;n] F['͹5&!pP. `V,ǁ~. tv%8[.{k;j>D^NWV%h+謟A 8 s&O f;pXK\?Bn~>o[.'vˣ&`@#^S\-_p6M?Gx 5e˓ t{PI #sMŸF˽^O7[fjo6O3'>Owi5l^i)R{Kqo6/ƶ$hQ}~ӷ1`s;ͨGۙl I(i=ߏ.\N71+z8"`\MOTNߋUu0ާ2i ؙ핽F ~$fkWz[앹vUh +g#^_?Apf&{>+c*,\ ooǘ~[W2k5pF]ib1vHi!tu@8^;g~gZWw1潒wfUީnʕ1U#s LiC>m8 I˭~~=wZ.~nln`]n}8< 妛S?g<} EW+00} lw8|GS0WZ2ϭMgvnDz,ǼnVLt矟zFzߪ|VcyG|9Uz;buQWci1X:i&SU=jg15QyNLyj}^K+?^/]o |;D>olk}^g>y']oO;>vzw zl|z/˳Se7e'p (B} x^y~˩/bO{;+[n,ï8}dj{,v,Ã(_cnu]9Q`;QLI ۮ0f"DnV_eAKQ:3`\=_̳>˗Zjg-@z?+U{unQvLo6j{ܫB佪NV&"E}ߊme;f{1|569s >?Wg9!r{-'"g@^χ1ϻIֲ}>NnnĴļRmG->tS_Wv<{C,E#~1\K#F*SЍ`4WBQF}dX4|P>i|:`p.&s]y^o{ˏΕctp8 S>?n{*,~mDWؼz\BLǶr4x"98 2ؘjW(X;mv:xw"m%_;ܶbwTa<5Nlgb9YIū'8{\ŸMGk8~e_n5*2樂}SNиXwA#rGHv=Ϊk ;5i5>Qm;;׍ApW_϶{qګ^4BjKg}ݹX`_ه軎i/;|n_k,.2n7߶| P^A :{~64#EY_-uPnzըӗ:,.c'Qjq5U0[z}XVW'e\b,Q$F}u=CY1k>Mسtxo6VRvh< KAf4-#_-+ axưpx+j_1vr=۲RKw y)*oW3[(wl['K9g߫s="|Xر}ވWYm{'@8mcmm@Kط~Pൖ V@8]Qjqp{Vq7ĴKs::eyfA͋'DB}ozHx{oQ.pKQ_S'=m=-F V'~6 Y/]ۖ'-Ӛj> mMt~~Sƾ_m `pO[jF*!- |owHsP^y=뎀QTt;tG3|:$mNG}rHx+k4Q.tl?xĿtqQtl?va`}mLt rRܧAx%~h>xY99;-^x``LBI_w;2t5K(~~tJ}P1ޝoPgI~j쀀MW:,)z߫NN_jcqrïf#0`wx?B^]>T')@}abBO^gl/7c?cSc2OT-5? 0 2}}3˩[C߳t)Dxk:/,Wt,Qs-u3ަ'u^F]Q(%_9dxXJܵ3 :a 3'Khl`p秖iv,6;> }-\m;/,K=o|_~uLÕTpLܐ;1xɕX^rA_εu/Yc8o'O' .`ԔZ?|yIU#J皋:8f&;tQ5q\ܻƎ1f|t>lJ$VU7g>7f:brVj-Ǖ/T 0*ƎCڿ"6wlbY<7k^m9x8y[\r)uy;zAN*F͇ߏc./1m$6U0t5}^9_W?(7}a,e!vZ/"tE[>6"s_IW˵jҼ,b^y\Mo5 w/i3/r5pq߈Әtu2}گ&MwcZy1vy-TIueZ1fTjYs1eYl#PMoU5rg!V~m{nvu2~ ?]_{4/7u:uB%/ߥ^|"<i1_1^o^#~O??5p96{i{?İyƲv b}x%迫vyV?Q m4G5f,c@Z|cxR9}uRFUlt^ݙm9ۨG|Hɖ+o˟t~ Bo ǩAD~u@+\58wRj%Va"wޜrcE ?V&mdmW料v[⯘N|5nU*DN7>c*`2m u kO2弄W勵[9lUvZ-;W'7ƵoQ$֢]@籊}鹕t~5*``*Z;f`3LM^Wz?MU'bо>zR,b#4;x>7}g%oatq@@YlӨ ́ \x[Y_MG@La|:|:pMIͧ~ĹlUFx$ @Z~Q4 \fPJWQХtc t9iNF}˔～:~>Ph@X\J+\M<=Q:x|#]-WIB^0+wa:$~ e00\: qN!GmFN?LWXsܼz<J0;;" V}e^'Z,w'"O>W^:9|uIx}@8~Ɇ]O,7;;,|/y61 0W[J5`Q|W#.FHkP[*Ė6ۖʳ{[|m`-5trWs}VY͎Jn kUOt8IrfMƟH56>!Wƶ KgeZSvo0p0ccl×\!݉N͉o_SӮ|+m.+;vR9;*g` _0YΥ3Kaˇ|apys=j K}~J0U˳=hړ u-_߳F Tm#?V۪M 1va::d @ӃJY4>⠿pf"4`t?xZsr-ryU1γu5_ˍ+sbxhkz}74tJsǴ'4y6|YWZeW1Tuǽ[H˶vG=r|zZg-WA=쮺y/ 0@::_F~6m`x\o6Ц n.͜r̞<`X>\}ix,'l%vtz@Ο9s~GG ]?y~O;ޜ<`@ϼ5VKF.QOfOOOP'\'o4ǫ(QT, ܎iy*Z>-S&&Ǵ&~Dۓr;11,OX-Ǯ'L +2."8E$w]c[^r_"O7'ASکv0!AQ@od0a1R]*wogGNTBk%ܫtO%+v倈{20td%Spn .~tmJmeĎ=a?o>ЏڱYW5eZL[5q{GJk;`#.X^b:|XY:oBɚz _;& \Gh J~w@W31tYV>]?X7y[G޳ֆi'Ĕm߯v4ŋԩ4ڼ,7].TQ tdE>4ߡPg޸FKCN껟m !W]|nl(2#Bo7 `iM)ͱjvFؤK6,3wEu4ՋWop ?UXMh>*p_UbY5e)[>|8`_[gN5f"n`-qGS;@pqWuF>jͦK֤ͫF|+m%cyQBI*: Jj^( e))*{܀Kp5Q #Rw8z0OF2.D;U֖ :H\:}+ꝴDD]>*N5EDi>Ӆ*>$͕vy(BގCp X2/7ۗvӈ}㮀x{QR_DAoZ]41]NL {3MUn8iPMX^F:j3+.{m%cw[Òr c癈Һ)dĚPsi{RDhdu]wG^)﮿z ۛ _3OhR;\ڃ+^7Υ&?"^;xi< *nz)s%^t܌O# mFTJ.AS$-zTLESV`yA uZDyCCb$lݣ'+C V }9𮹿]FCoT/9ْ +'Mdm((i=&R݈MmtL2:+n-f5G8.9 CgOhS=s=yp%Kn]X^"MvءtGw SglQB k)nˑmTC8hx^Ћt/hΡFjsd㭧\0CT>$FOB,贝_˕\}PG9Let2[*""mCAҗgj?Af,/ ` q<|7m?<5yCj}i,"B) pn.uw̞窩^};w(PZ{]I? tmf>Fݴjwl q:PlyY -t_ҵ `BrP0I,#o ,k;6T[o2B4lⷅ!XB#N7)/ՋeALϠ{/r!p# Aҗ+Wv5 _Rž4rbeT,Y_H9w'*n,WIǎ O `B<>#0!ߊoXݻDzpc׊DsȐoLURD|n&QڶQm"4&s `9Ǯ) ߔ Yp_z3fŔD):@Y7ٺ7_)sY=%ϳvJoPlbm2r_NyѿHX.!Ug+͡yGOHQHI3y;HKהk9J-@?9::Ou^B[Ʈ݀ hkw.qKsƎ1JX:N"@lgq=Vֵ˰S5k&gWʿrȻ׿nJ>y< ,![D~<ßRwΛ}-7]+އ&/T&w*Sevc.dFVA0!_GǦ&oMdt,yM0!Äw `BD( !PB( & `B( !dziiZbs(4! `B!B0!B! `B( !B!P_7;Į2Mym,p7.jqμ+ aTY#7( &$#yuQIq1+ o{ݻvMYS(n%?R@<#`~^OPٹF^q_GWUG4JwQ6l|ݼ>^]hwO`y_)A[GRBLȷrF_$c k?yd:%o c_~a `bYF>LPfz!TSJ5blӺZ+;|pwtZ;27`h!y+u?@/oypn;ϖ †.~=`+v/;6mձv_>[JPf%\5EslcWMNZm,~].6qӰؽqWmPnp7Irz)4O yQHY[nPJ@K~M0s*n'.|WYs[qo*H]ȗN* ;\"_hyGN` oG#2fԙ {]Pq:E*e|奠ʈx_2oJ8yu#N'<؊+i2WqmD|SCfSu&nټ.Iۛy=Nw<~6Dw ”wfRtN:.9ůʿb>]p76SLk/s7_^Y bD|.Џ9Ҡpe RiܕHm%!qW 4Zϼ Q_iG/KsV 5Ȥ͔r@K9tG[u7k| Z,ejNז_Qcm Y)_Ѧ%Z({ R@)>)RO8UNҼ`De}kتO UL t6#k+o혌| OCIuľ&oPMas 6u]xkW՞_͗;Fs_ן9".cg%`N7)-r'^V#o(F "ИGח`GLzz XF+#iD5#bi>F>3~JH9!*S-S('Cϧhˮ 7CVu,!iLϴy|fw{Íʷfc)ToP77Q[K߇fʻzCm ߇X^LfqC2B!R.YvÝ[<5@JF?bSЗFV4p" Am/Sn/ĺK֋z#+:79~pcL nkcX8GCඩciiA|J~8Tm^FY!V ǭL]Vn^ {a^o?B$#`[>>+}xr#,B!46Ү2tpi1biʬjtze$6ps `e 5#3l6QHP1G8Ķ:iTM#λ4^OݕX;[F_,e*{f_K)gNEB<~P41 ॔p{,2B:eDl%kOCr%\xFLK=: 1 #BH߹ w4DArcϝvM5fF/Vn2/K˚NpnSaK[>[g `iߺdR m~$BMhn~YXkF}Z) ac"B:#y縲4A9PQ:X_]pbQpZsJ^,P8q!\TSvnPyYqu&M=o<Ϭ*|s~s ಲEf`UHNr"!BR ڰD%ZkFƃpMՠ+*wrH4.92Z)=7^%=YعdoKc/< *}@Z)͑KTx;u_Gq"Lm%k`ildm*M^5xcV%G#ƾLEDzd]r48pY?rpdʘ1ghr9FDVY)GDE6nӶ;b%G^*EHul:îJK]BޕZ]&T LޝX NoE?'RޟO>d16g45Bp!gdh+U?V jȏؕ\KFJv׿]54`}zoH_Q/.9}69.YO,(Ic^,U^g:-#[w_wZ=Pvޏش|U ͮ]o\ ezXRUrJ 0%::P$RһlR'%e}ݡy\0YꞎYwKO.WqY t*mwp6>D벽1}޾P>S9-}YB!f]ٌ0 w QwBDH3 pƼ*W`ˉ+?9gwgڻ%>?þ?0}(> }%}2~MB!BO?/vn]xMǗh&beoG!B1,d3kGjר#Ag;4~:7G !B_˚L9!g B!d8_ɜgYO B!B!B!B!B!B!L#0~^yu-ֲk!E4 m\.q(smM2S{o`.9woB! Lˏ¿=3b̄Dӏcض#㸹D| "Q64_Gr1"ԗ)ho*3XYןcK}@!oq:hZ]}S8 A xo(|_*~P6徆{Ki=: ،B!ce  ~So?sb:mv )_=򻏧=eӀ.mI_ZC߇y~]v)WOie~ͩg*3F^P=霊&TC孤_PY5nqRp3[G-3>-3n>o.6)PUBL+*PuѦ1iV竪wmEҳ}4N)C=Cj ĈB!#FƼW#F7|Oěo]1p1@-陰.L0obBNQv Iuڻjhlxl#Ɩj+ܔI es3'Q=-}H=q57OeNIx]J~C9L޸yC8BS 'n bՔYή{5%:(3g9ˌKK&(*[.w>o Hm<4 Wb#evfZރ9%i oy Pݺv<BQT# ap %Fu#a3x,F u@54zh>.~oB#M=H F[ ᾣ1WWШKӍI $o^uL {"uo y,u_GH_>Ωp6:U+64$z k黃;r=.INJ:+.(w=뒑`q?t}x[zrHo2<~FYa!ܛT~SVVB!针eƉ*32c)+^dDyG=ρpR}n50KHkuzÔ+]J"(#Y,@7i"nvETR&]2YԳ+!. `kvJ{%C"+8Nβ3,@[_H!ob PAL%ж=qP \J,,Yۅ׼Il3[%#HC'kK7\2.)-H*!O+1"XK#6-?&4eNgA͙o5pvmʻ-K5u$g˷tЄ:`JuxO%B! EV׌.dc5!m-|;R+;[ 5huCmcQtMȷ IMjVeO6Uy5ypBK}pE4Z^2v ԭ^3G)rKL=ӬJUk:iBA8pbұV~ˑ_ {t2H-öJ}GoM!Gַn\֯yjYo 1h<BjX-ȿie! -U7 [д-Uf+cx7*C{Qˌ*(Ŷe)27+G)fB~h~)c?=fI5Ҧ#BbяPjZaS -!ԳBֵJB5)QCNy6EHn#]5lB5^LIg XYva/.~^T*p Ң˷ 뾮s pUuh\fMK MamEUf6·)QΥ3t.SD/\#_hzC,3)z=CxKF˳ +{ZF=}B!9]##k# `璑FqTۅ7Fu;r]~U| .i"E%bl?OGXR-ꥡ8hdSZN܌v$wO.Uʸ۠,)6uv|2,d1` )eOa?b@N78.ǀXX% ]Д~s_7<*(7MF٨@ IUߴ26> @Ǥԃ̣HݙY3Pĸ);'cPhp+Rg]#8tNw&=87nލc>B!|(Bm~񐧡%T譀h:E<|=ĭro}=&g^MchF\?~%T.zF:|j X5)3j:Bn -t,%HK; +ը6]bM!`|cNZ5A8.IDATP6_+i|;p҂<5_9?X7 vM={!`m"7*Oc[tlNr^%/UY?Ube닁)efG4(1-硪Vfp:m}? S?Ӧ^?2}Ȼʔ{|Q^u)bi J<б\vܾE^ª}tOB!B! 5ׄB!B0!B!BL!B!5~]o B!B!B!B!B!B! g0Y1md3'ɀ;~o797u<_eqˍ?T<BbPi B!Nj'k73cPO:4n۸_ lymR\R00!Bo` 1;oׇ-~$94M{1P `B!xqq=f?oSv+f>?-\~tBd͛~{Dw{y.دᗈOYD$O5n|~yl4?F Yey׏" Ap6)N͊)wU&U<euylbÐ|1q.gLžAz@\ *}'Gu$}^)w8:)L]23+*N*b ƙ>xJ m")TJ7pt%sU-PAYx@X@\6}ʍϣ\ad^x}*nP $B!;)PnYDFbs_SHiHJ8hH!]"\{vBŻy7 S{sN<=ltѰ}Ao0Gpdȳ"v?3rҳj;~3/Ӣ ~9ʾ{gk\@Y%3`YMͼrsw:]Yx]Uybԅ6}GlPi{C<SP ,QhC#/ʮg~wUSCulF׷+#bA9L=8ޚI#~U*Cwt\Ror `Msbrx> .maﵫ@M+.#BDMWLϦUٺckB&0zDDDڻ 桀j4FQ+"GJM[70g@㮅QDe5.)cW.vw?^4L(w%4x_UU#i/L"[EslBS@omתÒKֺe].a*EMܘ0d}@K6!UY(zMɻGSq<5nzt y7V**?#XC6Ml{y\Ҷm{W1mCfޟS3fSulӷK6H+^yO-6\xyt!B7PbL56uaql(+FB_Oip }3GC(Z(DS8TnW BAΰiYXv')ƭԅ]|iΕ@̮ :>oR 'yC `+#uu )C"g^w,0~!&Jw9UBlxa)71HRN.;[J<[){2PsWODo&\tB! x6Fd! lyu6Nȧq,KYM"26lpwhl,8D7ݔmYMMDFJ/e–xk+YT\Rwbe_&gءcOuHL@<D+dd/K6j(7&!O"s-^ȜLv}tݸ2<#}y*;:Ux<%B4n ELx/~HJi,dwN(FFz\ A9pלj$>YAQDn_2bmt\I5{p_.dKvb&Qi)(*.9fZ姌>Qk]C%%7`]Ҧ~(UZ&?.iQBXo i+ C򢄕I߬8t]r\l6钣rjGLK2^#)IU_rzvJ#^BN|E8¹pS,G[X3bi Uo]2m:>h!l@$9!V8םms:7r!P\ܙ3R 2 gѤ*z\UbUud$<7z%wa̞w/bZd'].-|p)(B!?6 5cG(&r+.O ƩDŽäm]xƵd%#.}#U\lV6v_BH:ܖ6Yζ[NG.dQ-w2_MĶcEo> 3T}6F,B!?Gexı˷;ܒcy@Td8N,B!/ߖ7,B!䟣~MBwf2_4̧2?y-~^ܓ)uvY=;ˈ?WV"g}ZM/iVKN-]YeĦ ;塦jϮ'#6 &%?fUY/(w bZ} qon?Lyx2b7RC66TJF~A1iB!c3r_SDcX̀~뒩r*W@8DpH]Huȣ@ٹ 8eʶL;Ui㫗B!!Bq+qjL bn HFG]S"Bێ>/I x7*pGqZe481Bk]E5py7g %>M]R'9 `E~Z}qG%;Qeh *,ݍϟS~*.< r~ީrZC &`oj:M p1R6U ;3^ %#Z %e݌pLs)e2㲊K훴=uZa2_B!'IF;Zp_ĽsΈl{"&tC;"ēMA6["&s@?E2<#7TǃtN\-]2h[Fl[85!>E:D 6.EiygZ;t TBl.df/gS:c6 &B!?WhU? xw%X!(㞽^NcHT^FBBgHTb3/nTDW]ɻAx21)C:^;-Il1R5Sl}Xu=9MKgJlN""#ә FJeSz2u|I-TC낯_B!QK׿QMNbAk:3 n^]ƥeC֐*bv\is+azKf*zծ״]&M9ap7GP(ϷUL??ɑ:錼8Ķzl!l{Sŷ 97B!B٥\52ZN,LA|)!\1tM4Fv7ibc{H,ӜC2gx\sɆ_HDQgH.:j| 2ryrS;vMFg %3)v/ݤ+v !BO2:5 v๟.>eDAhDQ6Ich3ا!`7 mo0]'2ʋ<[_$ke 8ϋs2%x..)$.G~MEh=Zt|rˌ/+Ӂ ]ze [1m:.4_B!䧑_>@`L\NB DF[֢ r3/Pz![66'#.gpZ#oUW;(x v|A-ɞgm@3qݪpF?[.d#k%2MdHF2\2>VaVPR Vq7P>s9{ܱeSX˦_.9wͤMK]sQ~B!BwhcvАJhA*EK3 kqr5Ɵ¼S"ۏu *N]4ۮh$;׾ O-D!ܼMpo׿)y 7f=c9t(H+`6?Ŝbؿ(Q_@G.{9-e`6?({=reeIKlPub"g_Sv{esa>E6ſ|B!g!MS^m2_uf!B!"׏M^ڹ^ `B!BY %kP݉(6# `B!BȏPk?_37i:P/4nd]{&I!B!B!B!B!B!B!id\#3BOH9 sfq%B!7quG΋yMv#B!D#r]A\-&"W?sxUfJLL!Be!\Ϗ?QSSB!B~'~?{zwV`7SD]\M57t û}^Yu^ uO!e͇zo? 㿤qnqVW =5p;a"}:OT߬ynGYsŮ-¾%ls^cM<0na'M !/ǯe/|Jy4R@~I 9p|*CͻEtxwW.YO?QnC~y/uqo޼">ZA~="o}B!C)Џ V7wø;J4{h">Hw%=DḊ{ʿc%~C L>2BB~1B\]hx noĿC%Vu- u!.<@ցkXBZwxm.Y;nmr>k^rw noϟ!.뎔]A҂o𳛳N<5#Ȼ7^u|#,6~@<:t{t\TDJ,;W~I=kB!54@;#SHӛiݣA<ﾎ**[4u^4-a&e#hkd%mxjG6TZҐkDY}bGc#0{d:\DH.$b\f%Hǁ3H y4*w>W_S'L"g&5YG_|{mnR쒑Igq^/M"V`KVq3 slB!#GLq}ÓcwNЈVrthҸp{5Mߌ>YVo[F,]xeLlXQ1S݅2"SgWSҕ&W[QN)[cP]z@(;[V\ͼ߈!P'Eky7 [>YFW?B!BF㢌ƮL}RbT|݈xd[C>Tc~; nu>V"(=6<}@{qi;E,&yQ[gs{EǹfJ MRF!e q)0GPk=^!k)ni:&B!MݟO]]27xW!6\24"nQ:}E@HTvYo%#"n r\L`N֏(!90hݴK6b>VBfDJǤ_SЈ+-d4;V}VL}y4[B;LB;/< :F7Up6Boe`|QW׻Ϻ").9;মʈL"~]set\ouL!BZ ~.aX1E84sww=-cmDgzpS}Č\o tI }]d=fMyaX0ivr4g;yEC e*,rna8dKe^gh:Gp%gtq+4:]oUu`<כd/+4!BV .ZƄ oƔFo ЮekV UxCĪ p_w-DQ%LHDΜU}GL mzKv 5q!W9 wWSm1y'L" BwvىH:k#-oS߬ݪ@]_YpwmT1LB!B2b;]j. !YB!BLLL!B!*~_:eqoH&9B!B!B!B!B!B!B!GThBȱXSqD??h2p\;L!BHju_/K_CB[A_,f(RngEڽԷ[|]B!B#Eܯ݉Ѐ>Qx;v8kFzaUD+ILO7={^ n!o-yB!B~FF0#]iӋ^m^-|Q7߿,knqO "\XkQ7K,B!@cYS <AHy(.5-߯,Bķ!Ɗ 2~3`pRqLq ֎sa!T )bw )pQY ;Q!޿]fMC†EC4JFJ8m|L"2a/4NFk0UZ?7WF#B!ppkB h0)m-5푹C^'QuYﶋpߍM&L7WFSuMf[kң]2jEVMZv7eWt 4qS>S6}Cvsw|g=w,_pIrtqI~@|07.!]˪;>NuG!BQs4!!<3M[6B%)BPAȚƝ*~.(xFSH=a;]P݀&~`*()7:p_.uoϞ)Q>>dt ŀ@H|Ȱ_}=lZI\p!ޢrWUyw+O)#]}dS7Cuݭt^YCx~Lf(V#B!uDȽ%La}3`m㦤Ғɀik9Θ7Jv"d1&l攽Bt-[4- 3z Wko(Jn;t8'.*7)Y%c<(Ȱ 16-#cl=-%B%t#B!e*)"!0,SC+Z6+:d7ano )?mG,W"k ͳ沧ن ;b~ gO/.|#9֪6tU_n"R{!B!IcaPzeoO `^-+FSu'\2ڦDVW q `Qn/.owƎZ!HfQOm;"vS;qS.ہT:dmADKweX:^NM. Kv 8? LBISDB!򟣎oD_@xGDa7I R{9.96!g<$;=#ldl{&GڇYM6:qəFWp bqee¾qɑJiT\nb{_r벧c2lvuo}e y3v.ZEWU$e;&`']r%g߸6#B! ?t s\MZ9 Kpok8!Ah %<ׁ %\d?8_rMڟ`IB9`=7;B!2ϫy}|^O~;9_zn$dp%wf:q۹F٭܄B!=4?M@ܫ{߅"x8`7 CL!B_E K <70000!B!?ِ?QgMu钩WnqO>r_(o_SWo | {:Vyw7(٧KFW߼yg3㧨Q{N QO*]>NFgsTsGS^TrkŲ'Rwo([} qsk}uΟefqoU ,t܃}x鸄Ѝ B!DaIv [ѻp+#ܿX ^McnNxNMT*gwo@Jxvw!9*vT8 7% HUb_sla,}&}{3e Sf>_>@<8!Cܚț{RZpUǎspl8tܪv([B!%.<#yp\sh1_5#oՃL@\g\2*'s12 p^៵;<+̺dZ|ň:^0_~I6hRf(O툷)7Ɲ1 ρx1t 'Sk%f\x -8'}bomZ;+V&TP5U<;w]W-DY!B ##Pem t\HcCYy6y͔_G?x~}9S_>} [/w/FhuבOmC.YN )<җء<'}'.Zo"Ϯdo\`G^W L_3v7Ȟ \rȿL? t$B!B ue4K6o]-8gĽ{;\1yZuziM.vKF!BH?7,v5Ruȋ^F< .\Yk,?/S. #8%^H?BĻp_Dx/"(OOa?.%\2J~I⻜wDX+E:díHǒH 鰹BZmJ"B!?M%+_*k'78HI9Pďͻ(x+1 l3#Nd9ID)oPsx- |Jk1wn!K(E\]2]#?.q^2}3 #]¹4i~Dye / 4s 12et5B!B҅l#Geby;Tʺ阝g& =J+{څEl%Gfs7c̓J@N3܎*V ˷ΚB!BRJ`B!B!B!BJMA!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!B!?wS+Z':IENDB`slurm-slurm-17-11-2-1/doc/html/intel_knl.shtml000066400000000000000000000366351322351151600210760ustar00rootroot00000000000000

Intel Knights Landing (KNL) User and Administrator Guide

Overview

This document describes the unique features of Slurm on the computers with the Intel Knights Landing processor. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in Intel KNL system operation described in this document.

User Tools

The desired NUMA and MCDRAM modes for a KNL processor should be specified using the -C or --constraints option of Slurm's job submission commands: salloc, sbatch, and srun. Currently available NUMA and MCDRAM modes are shown in the table below. Each node's available and current NUMA and MCDRAM modes are visible in the "available features" and "active features" fields respectively, which may be seen using the scontrol, sinfo, or sview commands. Note that a node may need to be rebooted to get the desired NUMA and MCDRAM modes and nodes may only be rebooted when they contain no running jobs (i.e. sufficient resources may be available to run a pending job, but until the node is idle and can be rebooted, the pending job may not be allocated resources). Also note that the job will be charged for resources from the time of resource allocation, which may include time to reboot a node into the desired NUMA and MCDRAM configuration.

While Slurm generally supports a very rich set of options for the node constraint options (exclusive OR, node counts for each constraint, etc.), only a simple AND operation is supported for KNL systems (specified using a comma or ampersand separator between specified constraints). Jobs may specify their desired NUMA and/or MCDRAM configuration. If no NUMA and/or MCDRAM configuration is specified, then a node with any possible value for that configuration will be used.

Type Name Description
MCDRAMcacheAll of MCDRAM to be used as cache
MCDRAMequalMCDRAM to be used partly as cache and partly combined with primary memory
MCDRAMflatMCDRAM to be combined with primary memory into a "flat" memory space
NUMAa2aAll to all
NUMAhemiHemisphere
NUMAsnc2Sub-NUMA cluster 2
NUMAsnc4Sub-NUMA cluster 4 (NOTE)
NUMAquadQuadrant (NOTE)

Jobs requiring some or all of the KNL high bandwidth memory (HBM) should explicitly request that memory using Slurm's Generic RESource (GRES) options. The HBM will always be known by Slurm GRES name of "hbm". Examples below demonstrate use of HBM.

Sorting of the free cache pages at step startup using Intel's zonesort module can be configred as the default for all steps using the "LaunchParameters=mem_sort" option in the slurm.conf file. Individual job steps can enable or disable sorting using the "--mem-bind=sort" or "--mem-bind=nosort" command line options for srun. Sorting will be performed only for the NUMA nodes allocated to the job step.

NOTE: Slurm provides limited support for restricting use of HBM. At some point in the future, the amount of HBM requested by the job will be compared with the total amount of HBM and number of memory-containing NUMA nodes available on the KNL processor. The job will then be bound to specific NUMA nodes in order to limit the total amount of HBM available to the job, and thus reserve the remaining HBM for other jobs running on that KNL processor.

NOTE: Slurm can only support homogeneous nodes (e.g. the same number of cores per NUMA node). KNL processors with 68 cores (a subset of KNL models) will not have homogeneous NUMA nodes in snc4 mode, but each each NUMA node will have either 16 or 18 cores. This will result in Slurm using the lower core count, finding a total of 256 threads rather than 272 threads and setting the node to a DOWN state.

Accounting

If a node requires rebooting for a job's required configuration, the job will be charged for the resource allocation from the time of allocation through the lifetime of the job, including the time consumed for booting the nodes. The job's time limit will be calculated from the time that all nodes are ready for use. For example, a job with a 10 minute time limit may be allocated resources at 10:00:00. If the nodes require rebooting, they might not be available for use until 10:20:00, 20 minutes after allocation, and the job will begin execution at that time. The job must complete no later than 10:30:00 in order to satisfy it's time limit (10 minutes after execution actually begins). However, the job will be charged for 30 minutes of resource use, which includes the boot time.

Sample Use Cases

$ sbatch -C flat,a2a -N2 --gres=hbm:8g --exclusive my.script
$ srun --constraint=hemi,cache -n36 a.out
$ srun --constraint=flat --gres=hbm:2g -n36 a.out

$ sinfo -o "%30N %20b %f"
NODELIST       ACTIVE_FEATURES  AVAIL_FEATURES
nid000[10-11]
nid000[12-35]  flat,a2a         flat,a2a,snc2,hemi
nid000[36-43]  cache,a2a        flat,equal,cache,a2a,hemi

Network Topology

Slurm will optimize performance using those resources available without rebooting. If node rebooting is required, then it will optimize layout with respect to network bandwidth using both nodes currently in the desired configuration and those which can be made available after rebooting. This can result in more nodes being rebooted than strictly needed, but will improve application performance.

Users can specify they want all resources allocated on a specific count of leaf switches (Dragonfly group) using Slurm's --switches option. They can also specify how much additional time they are willing to wait for such a configuration. If the desired configuration can not be made available within the specified time interval, the job will be allocated nodes optimized with respect to network bandwidth to the extent possible. On a Dragonfly network, this means allocating resources over either single group or distributed evenly over as many groups as possible. For example:

srun --switches=1@10:00 N16 a.out

Note that system administrators can disable use of the --switches option or limit the amount of time the job can be deferred using the SchedulerParameters max-switch-wait option.

Booting Problems

If node boots fail, those nodes are drained and the job is requeued so that it can be allocated a different set of nodes. The nodes originally allocated to the job will remain available to the job, so likely a small number of additional nodes will be required.

System Administration

Four important components are required to use Slurm on an Intel KNL system.

  1. Slurm needs a mechanism to determine the node's current topology (e.g. how many NUMA exist and which cores are associated with each NUMA). Slurm relies upon Portable Hardware Locality (HWLOC) for this functionality. Please install HWLOC before building Slurm.
  2. The node features plugin manages the available and active features information available for each KNL node.
  3. A configuration file is used to define various timeouts, default configuration, etc. The configuration file name and contents will depend upon the node features plugins used. See the knl.conf man page for more information.
  4. A mechanism is required to boot nodes in the desired configuration. This mechanism must be integrated with existing Slurm infrastructure for rebooting nodes on user request (--reboot) plus (for Cray systems only) power saving (powering down idle nodes and restarting them on demand).

In addition, there is a DebugFlags option of "NodeFeatures" which will generate detailed information about KNL operations.

The KNL-specific available and active features are configured differently based upon the plugin configured.
For the knl_cray plugin, KNL-specific available and active features are not included in the "slurm.conf" configuration file, but are set and the managed by the NodeFeatures plugin when the slurmctld daemon starts.
For the knl_generic plugin, KNL-specific features should be defined in the "slurm.conf" configuration file. When the slurmd daemon starts on each compute node, it will update the available and active features as needed.
Features which are not KNL-specific (e.g. rack number, "knl", etc.) will be copied from the node's "Features" configuration in "slurm.conf" to both the available and active feature fields and not modified by the NodeFeatures plugin.

NOTE: For Dell KNL systems you must also include theSystemType=Dell option for successful operation.

Slurm does not support the concept of multiple NUMA nodes within a single socket. If a KNL node is booted with multiple NUMA, then each NUMA node will appear in Slurm as a separate socket. In the slurm.conf configuration file, set "FastSchedule=1" and node socket and core counts to values which are appropriate for some NUMA mode to be used on the node. When the node boots and the slurmd daemon on the node starts, it will report to the slurmctld daemon the node's actual socket (NUMA) and core counts, which will update Slurm data structures for the node to the values which are currently configured. Note that Slurm currently does not support the concept of differing numbers of cores in each socket (or NUMA node). We are currently working to address these issues.

Mode of Operation

  1. The node's configured "Features" are copied to the available and active feature fields.
  2. The node features plugin determines the node's current MCDRAM and NUMA values as well as those which are available and adds those values to the node's active and available feature fields respectively. Note that these values may not be available until the node has booted and the slurmd daemon on the compute node sends that information to the slurmctld daemon.
  3. Jobs will be allocated nodes already in the requested MCDRAM and NUMA mode if possible. If insufficient resources are available with the requested configuration then other nodes will be selected and booted into the desired configuration once no other jobs are active on the node. Until a node is idle, its configuration can not be changed. Note that node reboot time is roughly on the order of 20 minutes.

Cray Configuration

On Cray systems, NodeFeaturesPlugins should be configured as "knl_cray".

The configuration file will be named "knl_cray.conf". The file will include the path to the capmc program (CapmcPath), which is used to get a node's available MCDRAM and NUMA modes, change the modes, power the node down, reboot it, etc. Note the "CapmcTimeout" parameter is the time required for the capmc program to respond to a request and NOT the time for a boot operation to complete.

Power saving mode is integrated with rebooting nodes in the desired mode. Programs named "capmc_resume" and "capmc_suspend" are provided to boot nodes in the desired mode. The programs are included in the main Slurm RPM and installed in the "sbin" directory and must be installed on the Cray "sdb" node. If powering down of idle nodes is not desired, then configure "ResumeProgram" in "slurm.conf" to the path of the "capmc_resume" file and configure "SuspendTime" to a huge value (e.g. "SuspendTime=30000000" will only power down a node which has been idle for about one year).

Note that getting a compute node's current MCDRAM and NUMA mode, modifying its MCDRAM and NUMA mode, and rebooting it are operations performed by the slurmctld daemon on the head node.

The GresTypes configuration parameter should include "hbm" to identify High Bandwidth Memory (HBM) as being a consumable resources on compute nodes. Additional GresTypes can be specified as needed in a comma separated list. The amount of HBM on each node should not be configured in a Slurm configuration file, but that information will be loaded by the knl_cray plugin using information provided by the capmc program.

Sample slurm.conf file for knl_cray plugin

# Sample slurm.conf
NodeFeaturesPlugins=knl_cray
DebugFlags=NodeFeatures
GresTypes=hbm
#
ResumeProgram=/opt/slurm/default/sbin/capmc_resume
SuspendTime=30000000
ResumeTimeout=1800
...
Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl
NodeName=nid[00000-00127] State=UNKNOWN

Sample knl_cray.conf file

# Sample knl_cray.conf
CapmcPath=/opt/cray/capmc/default/bin/capmc
CapmcTimeout=2000	# msec
DefaultNUMA=a2a         # NUMA=all2all
AllowNUMA=a2a,snc2,hemi
DefaultMCDRAM=cache     # MCDRAM=cache

Sample slurm.conf File

# Sample slurm.conf
NodeFeaturesPlugins=knl_cray
DebugFlags=NodeFeatures
GresTypes=hbm
#
ResumeProgram=/opt/slurm/default/sbin/capmc_resume
SuspendProgram=/opt/slurm/default/sbin/capmc_suspend
SuspendTime=30000000
ResumeTimeout=1800
...
Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl
NodeName=nid[00000-00127] State=UNKNOWN

Generic Cluster Configuration

All other clusters should have NodeFeaturesPlugins configured to "knl_generic". This plugin performs all operations directly on the compute nodes using Intel's syscfg program to get and modify the node's MCDRAM and NUMA mode and uses the Linux reboot program to reboot the compute node in order for modifications in MCDRAM and/or NUMA mode to take effect. Make sure that RebootProgram is defined in the slurm.conf file. This plugin currently does not permit the specification of ResumeProgram, SuspendProgram, SuspendTime, etc. in slurm.conf, however that limitation may be removed in the future (the ResumeProgram currently has no means of changing the node's MCDRAM and/or NUMA mode prior to reboot).

NOTE:The syscfg program reports the MCDRAM and NUMA mode to be used when the node is next booted. If the syscfg program is used to modify the MCDRAM or NUMA mode of a node, but it is not rebooted, then Slurm will be making scheduling decisions based upon incorrect state information. If you want to change node state information outside of Slurm then use the following proceedure:

  1. Drain the nodes of interest
  2. Change their MCDRAM and/or NUMA mode
  3. Reboot the nodes, then
  4. Restore them to service in Slurm

Sample knl_generic.conf File

# Sample knl_generic.conf
SyscfgPath=/usr/bin/syscfg
DefaultNUMA=a2a         # NUMA=all2all
AllowNUMA=a2a,snc2,hemi
DefaultMCDRAM=cache     # MCDRAM=cache

Sample slurm.conf File

# Sample slurm.conf
NodeFeaturesPlugins=knl_generic
DebugFlags=NodeFeatures
GresTypes=hbm
RebootProgram=/sbin/reboot
...
Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl
NodeName=nid[00000-00127] State=UNKNOWN

top

Last modified 27 October 2017

slurm-slurm-17-11-2-1/doc/html/job_array.shtml000066400000000000000000000322561322351151600210620ustar00rootroot00000000000000

Job Array Support

Overview

Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily; job arrays with millions of tasks can be submitted in milliseconds (subject to configured size limits). All jobs must have the same initial options (e.g. size, time limit, etc.), however it is possible to change some of these options after the job has begun execution using the scontrol command specifying the JobID of the array or individual ArrayJobID.

$ scontrol update job=101 ...
$ scontrol update job=101_1 ...

Job arrays are only supported for batch jobs and the array index values are specified using the --array or -a option of the sbatch command. The option argument can be specific array index values, a range of index values, and an optional step size as shown in the examples below. Note that the minimum index value is zero and the maximum value is a Slurm configuration parameter (MaxArraySize minus one). Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value.

# Submit a job array with index values between 0 and 31
$ sbatch --array=0-31    -N1 tmp

# Submit a job array with index values of 1, 3, 5 and 7
$ sbatch --array=1,3,5,7 -N1 tmp

# Submit a job array with index values between 1 and 7
# with a step size of 2 (i.e. 1, 3, 5 and 7)
$ sbatch --array=1-7:2   -N1 tmp

A maximum number of simultaneously running tasks from the job array may be specified using a "%" separator. For example "--array=0-15%4" will limit the number of simultaneously running tasks from this job array to 4.

Job ID and Environment Variables

Job arrays will have two additional environment variable set. SLURM_ARRAY_JOB_ID will be set to the first job ID of the array. SLURM_ARRAY_TASK_ID will be set to the job array index value. SLURM_ARRAY_TASK_COUNT will be set to the number of tasks in the job array. SLURM_ARRAY_TASK_MAX will be set to the highest job array index value. SLURM_ARRAY_TASK_MIN will be set to the lowest job array index value. For example a job submission of this sort
sbatch --array=1-3 -N1 tmp
will generate a job array containing three jobs. If the sbatch command responds
Submitted batch job 36
then the environment variables will be set as follows:

SLURM_JOB_ID=36
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=37
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=38
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

All Slurm commands and APIs recognize the SLURM_JOB_ID value. Most commands also recognize the SLURM_ARRAY_JOB_ID plus SLURM_ARRAY_TASK_ID values separated by an underscore as identifying an element of a job array. Using the example above, "37" or "36_2" would be equivalent ways to identify the second array element of job 36. A set of APIs has been developed to operate on an entire job array or select tasks of a job array in a single function call. The function response consists of an array identifying the various error codes for various tasks of a job ID. For example the job_resume2() function might return an array of error codes indicating that tasks 1 and 2 have already completed; tasks 3 through 5 are resumed successfully, and tasks 6 through 99 have not yet started.

File Names

Two additional options are available to specify a job's stdin, stdout, and stderr file names: %A will be replaced by the value of SLURM_ARRAY_JOB_ID (as defined above) and %a will be replaced by the value of SLURM_ARRAY_TASK_ID (as defined above). The default output file format for a job array is "slurm-%A_%a.out". An example of explicit use of the formatting is:
sbatch -o slurm-%A_%a.out --array=1-3 -N1 tmp
which would generate output files names of this sort "slurm-36_1.out", "slurm-36_2.out" and "slurm-36_3.out". If these file name options are used without being part of a job array then "%A" will be replaced by the current job ID and "%a" will be replaced by 4,294,967,294 (equivalent to 0xfffffffe or NO_VAL).

Scancel Command Use

If the job ID of a job array is specified as input to the scancel command then all elements of that job array will be cancelled. Alternately an array ID, optionally using regular expressions, may be specified for job cancellation.

# Cancel array ID 1 to 3 from job array 20
$ scancel 20_[1-3]

# Cancel array ID 4 and 5 from job array 20
$ scancel 20_4 20_5

# Cancel all elements from job array 20
$ scancel 20

# Cancel the current job or job array element (if job array)
if [[-z $SLURM_ARRAY_JOB_ID]]; then
  scancel $SLURM_JOB_ID
else
  scancel ${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}
fi

Squeue Command Use

When a job array is submitted to Slurm, only one job record is created. Additional job records will only be created when the state of a task in the job array changes, typically when a task is allocated resources or its state is modified using the scontrol command. By default, the squeue command will report all of the tasks associated with a single job record on one line and use a regular expression to indicate the "array_task_id" values as shown below.

$ squeue
 JOBID     PARTITION  NAME  USER  ST  TIME  NODES NODELIST(REASON)
1080_[5-1024]  debug   tmp   mac  PD  0:00      1 (Resources)
1080_1         debug   tmp   mac   R  0:17      1 tux0
1080_2         debug   tmp   mac   R  0:16      1 tux1
1080_3         debug   tmp   mac   R  0:03      1 tux2
1080_4         debug   tmp   mac   R  0:03      1 tux3

An option of "--array" or "-r" has also been added to the squeue command to print one job array element per line as shown below. The environment variable "SQUEUE_ARRAY" is equivalent to including the "--array" option on the squeue command line.

$ squeue -r
 JOBID PARTITION  NAME  USER  ST  TIME  NODES NODELIST(REASON)
1082_3     debug   tmp   mac  PD  0:00      1 (Resources)
1082_4     debug   tmp   mac  PD  0:00      1 (Priority)
  1080     debug   tmp   mac   R  0:17      1 tux0
  1081     debug   tmp   mac   R  0:16      1 tux1
1082_1     debug   tmp   mac   R  0:03      1 tux2
1082_2     debug   tmp   mac   R  0:03      1 tux3

The squeue --step/-s and --job/-j options can accept job or step specifications of the same format.

$ squeue -j 1234_2,1234_3
...
$ squeue -s 1234_2.0,1234_3.0
...

Two additional job output format field options have been added to squeue:
%F prints the array_job_id value
%K prints the array_task_id value
(all of the obvious letters to use were already assigned to other job fields).

Scontrol Command Use

Use of the scontrol show job option shows two new fields related to job array support. The JobID is a unique identifier for the job. The ArrayJobID is the JobID of the first element of the job array. The ArrayTaskID is the array index of this particular entry, either a single number of an expression identifying the entries represented by this job record (e.g. "5-1024"). Neither field is displayed if the job is not part of a job array. The optional job ID specified with the scontrol show job or scontrol show step commands can identify job array elements by specifying ArrayJobId and ArrayTaskId with an underscore between them (eg. <ArrayJobID>_<ArrayTaskId>).

The scontrol command will operate on all elements of a job array if the job ID specified is ArrayJobID. Individual job array tasks can be modifed using the ArrayJobID_ArrayTaskID as shown below.

$ sbatch --array=1-4 -J array ./sleepme 86400
Submitted batch job 21845

$ squeue
 JOBID   PARTITION     NAME     USER  ST  TIME NODES NODELIST
 21845_1    canopo    array    david  R  0:13  1     dario
 21845_2    canopo    array    david  R  0:13  1     dario
 21845_3    canopo    array    david  R  0:13  1     dario
 21845_4    canopo    array    david  R  0:13  1     dario

$ scontrol update JobID=21845_2 name=arturo
$ squeue
 JOBID   PARTITION     NAME     USER  ST   TIME  NODES NODELIST
 21845_1    canopo    array    david  R   17:03   1    dario
 21845_2    canopo   arturo    david  R   17:03   1    dario
 21845_3    canopo    array    david  R   17:03   1    dario
 21845_4    canopo    array    david  R   17:03   1    dario

The scontrol hold, holdu, release, requeue, requeuehold, suspend and resume commands can also either operate on all elements of a job array or individual elements as shown below. 

$ scontrol suspend 21845
$ squeue
 JOBID PARTITION      NAME     USER  ST TIME  NODES NODELIST
21845_1    canopo    array    david  S 25:12  1     dario
21845_2    canopo   arturo    david  S 25:12  1     dario
21845_3    canopo    array    david  S 25:12  1     dario
21845_4    canopo    array    david  S 25:12  1     dario
$ scontrol resume 21845
$ squeue
 JOBID PARTITION      NAME     USER  ST TIME  NODES NODELIST
21845_1    canopo    array    david  R 25:14  1     dario
21845_2    canopo   arturo    david  R 25:14  1     dario
21845_3    canopo    array    david  R 25:14  1     dario
21845_4    canopo    array    david  R 25:14  1     dario

scontrol suspend 21845_3
$ squeue
 JOBID PARTITION      NAME     USER  ST TIME  NODES NODELIST
21845_1    canopo    array    david  R 25:14  1     dario
21845_2    canopo   arturo    david  R 25:14  1     dario
21845_3    canopo    array    david  S 25:14  1     dario
21845_4    canopo    array    david  R 25:14  1     dario
scontrol resume 21845_3
$ squeue
 JOBID PARTITION      NAME     USER  ST TIME  NODES NODELIST
21845_1    canopo    array    david  R 25:14  1     dario
21845_2    canopo   arturo    david  R 25:14  1     dario
21845_3    canopo    array    david  R 25:14  1     dario
21845_4    canopo    array    david  R 25:14  1     dario

Job Dependencies

A job which is to be dependent upon an entire job array should specify itself dependent upon the ArrayJobID. Since each array element can have a different exit code, the interpretation of the afterok and afternotok clauses will be based upon the highest exit code from any task in the job array.

When a job dependency specifies the job ID of a job array:
The after clause is satisfied after all tasks in the job array start.
The afterany clause is satisfied after all tasks in the job array complete.
The afterok clause is satisfied after all tasks in the job array complete successfully.
The afternotok clause is satisfied after all tasks in the job array complete with at least one tasks not completing successfully.

Examples of use are shown below:

# Wait for specific job array elements
sbatch --depend=after:123_4 my.job
sbatch --depend=afterok:123_4:123_8 my.job2

# Wait for entire job array to complete
sbatch --depend=afterany:123 my.job

# Wait for entire job array to complete successfully
sbatch --depend=afterok:123 my.job

# Wait for entire job array to complete and at least one task fails
sbatch --depend=afternotok:123 my.job

Other Command Use

The following Slurm commands do not currently recognize job arrays and their use requires the use of Slurm job IDs, which are unique for each array element: sacct, sbcast, smap, sreport, sshare, sstat, and strigger. The sattach, sprio and sstat commands have been modified to permit specification of either job IDs or job array elements. The sview command has been modified to permit display of a job's ArrayJobId and ArrayTaskId fields. Both fields are displayed with a value of "N/A" if the job is not part of a job array.

System Administration

A new configuration parameter has been added to control the maximum job array size: MaxArraySize. The smallest index that can be specified by a user is zero and the maximum index is MaxArraySize minus one. The default value of MaxArraySize is 1001. The maximum MaxArraySize supported in Slurm is 4000001. Be mindful about the value of MaxArraySize as job arrays offer an easy way for users to submit large numbers of jobs very quickly.

The sched/backfill plugin has been modified to improve performance with job arrays. Once one element of a job array is discovered to not be runable or impact the scheduling of pending jobs, the remaining elements of that job array will be quickly skipped.

Slurm creates a single job record when a job array is submitted. Additional job records are only created as needed, typically when a task of a job array is started, which provides a very scalable mechanism to manage large job counts. Each task of the job array will have a unique Slurm "job_id", but all will have the same "array_job_id" value.

Last modified 7 November 2017

slurm-slurm-17-11-2-1/doc/html/job_container_plugins.shtml000066400000000000000000000155321322351151600234650ustar00rootroot00000000000000

Job Container Plugin API

Overview

This document describes Slurm job container plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own Slurm job container plugins. Note that job container plugin is designed for use with Slurm jobs. It also applies to the sbcast server process on compute nodes. There is a proctrack plugin designed for use with Slurm job steps.

Slurm job container plugins are Slurm plugins that implement the Slurm job container API described herein. They must conform to the Slurm Plugin API with the following specifications:

const char plugin_type[]
The major type must be "job_container." The minor type can be any recognizable abbreviation for the type of proctrack. We recommend, for example:

  • cncu — Designed for use on Cray systems only and interface with Compute Node Clean Up (CNCU) the Cray infrastructure.
  • none — Designed for all other systems.

const char plugin_name[]
Some descriptive name for the plugin. There is no requirement with respect to its format.

const uint32_t plugin_version
If specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loadeed by Slurm commands and daemons from any version, however this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.

The programmer is urged to study src/plugins/proctrack/job_container/job_container_cncu.c for an example implementation of a Slurm proctrack plugin.

top

Data Objects

The implementation must support a container ID of type uint64_t. This container ID is generated by the proctrack plugin.

The implementation must maintain (though not necessarily directly export) an enumerated errno to allow Slurm to discover as practically as possible the reason for any failed API call. These values must not be used as return values in integer-valued functions in the API. The proper error return value from integer-valued functions is Slurm_ERROR. The implementation should endeavor to provide useful and pertinent information by whatever means is practical. Successful API calls are not required to reset errno to a known value.

top

API Functions

The following functions must appear. Functions which are not implemented should be stubbed.

int init (void)

Description:
Called when the plugin is loaded, before any other functions are called. Put global initialization here.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void fini (void)

Description:
Called when the plugin is removed. Clear any allocated storage here.

Returns: None.

Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the SLURM init(), and the SLURM fini() is called before the system's _fini().

int container_p_create (uint32_t job_id);

Description: Create a container. The caller should ensure that the valid container_p_delete() is called.

Argument: job_id    (input) Job ID.

Returns: Slurm_SUCCESS if successful. On failure, the plugin should return Slurm_ERROR and set the errno to an appropriate value to indicate the reason for failure.

int container_p_add_cont (uint32_t job_id, uint64_t cont_id);

Description: Add a specific process tracking container (PAGG) to a given job's container.

Arguments:
job_id    (input) Job ID.
cont_id    (input) Process tracking container value as set by the proctrack plugin.

Returns: Slurm_SUCCESS if successful. On failure, the plugin should return Slurm_ERROR and set the errno to an appropriate value to indicate the reason for failure.

int container_p_add_pid (uint32_t job_id, pid_t pid, uid_t uid);

Description: Add a specific process ID to a given job's container. The process is first placed into a process tracking container (PAGG).

Arguments:
job_id    (input) Job ID.
pid    (input) Process ID.
uid    (input) Owning user ID.

Returns: Slurm_SUCCESS if successful. On failure, the plugin should return Slurm_ERROR and set the errno to an appropriate value to indicate the reason for failure.

int container_p_delete (uint32_t job_id);

Description: Destroy or otherwise invalidate a job container. This does not imply the container is empty, just that it is no longer needed.

Arguments: job_id    (input) Job ID.

Returns: Slurm_SUCCESS if successful. On failure, the plugin should return Slurm_ERROR and set the errno to an appropriate value to indicate the reason for failure.

void container_p_reconfig (void);

Description: Note change in configuration, especially the value of the DebugFlags with respect to JobContainer.

top

Last modified 27 March 2015

slurm-slurm-17-11-2-1/doc/html/job_exit_code.shtml000066400000000000000000000115711322351151600217040ustar00rootroot00000000000000

Job Exit Codes

A job's exit code (aka exit status, return code and completion code) is captured by Slurm and saved as part of the job record. For sbatch jobs, the exit code that is captured is the output of the batch script. For salloc jobs, the exit code will be the return value of the exit call that terminates the salloc session. For srun, the exit code will be the return value of the command that srun executes.

Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a Reason of "NonZeroExitCode".

The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, Slurm will display it as an unsigned value in the 0 - 255 range.

Job Step Exit Codes

When a job contains multiple job steps, the exit code of each executable invoked by srun is saved individually to the job step record.

Signaled Jobs

When a job or step is sent a signal that causes its termination, Slurm also captures the signal number and saves it to the job or step record.

Displaying Exit Codes and Signals

Slurm displays a job's exit code in the output of the scontrol show job and the sview utility. Slurm displays job step exit codes in the output of the scontrol show step and the sview utility.

When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).

Database Job/Step Records

The Slurm control daemon sends job and step records to the Slurm database when the Slurm accounting_storage plugin is installed. Job and step records sent to the Slurm db can be viewed using the sacct command. The default sacct output contains an ExitCode field whose format mirrors the output of scontrol and sview described above.

Derived Exit Code and Comment String

After reading the above description of a job's exit code, one can imagine a scenario where a central task of a batch job fails but the script returns an exit code of zero, indicating success. In many cases, a user may not be able to ascertain the success or failure of a job until after they have examined the job's output files.

The job includes a "derived exit code" field. It is initially set to the value of the highest exit code returned by all of the job's steps (srun invocations). The job's derived exit code is determined by the Slurm control daemon and sent to the database when the accounting_storage plugin is enabled.

In addition to the derived exit code, the job record in the Slurm database contains a comment string. This is initialized to the job's comment string (when AccountingStoreJobComment parameter in the slurm.conf is set) and can only be changed by the user.

A new option has been added to the sacctmgr command to provide the user the means to modify these two fields of the job record. No other modification to the job record is allowed. For those who prefer a simpler command specifically designed to view and modify the derived exit code and comment string, the sjobexitmod wrapper has been created (see below).

The user now has the means to annotate a job's exit code after it completes and provide a description of what failed. This includes the ability to annotate a successful completion to jobs that appear to have failed but actually succeeded.

The sjobexitmod command

The sjobexitmod command is available to display and update the two derived exit fields of the Slurm db's job record. sjobexitmod can first be used to display the existing exit code / string for a job:

> sjobexitmod -l 123
JobID Account NNodes NodeList     State ExitCode DerivedExitCode Comment
----- ------- ------ -------- --------- -------- --------------- -------
123        lc      1     tux0 COMPLETED      0:0             0:0
If a change is desired, sjobexitmod can modify the derived fields: 
> sjobexitmod -e 49 -r "out of memory" 123

 Modification of job 123 was successful.

> sjobexitmod -l 123
JobID Account NNodes NodeList     State ExitCode DerivedExitCode Comment
----- ------- ------ -------- --------- -------- --------------- -------
123        lc      1     tux0 COMPLETED      0:0            49:0 out of memory

The existing sacct command also supports the two new derived exit fields:

> sacct -X -j 123 -o JobID,NNodes,State,ExitCode,DerivedExitcode,Comment
JobID   NNodes      State ExitCode DerivedExitCode        Comment
------ ------- ---------- -------- --------------- --------------
123          1  COMPLETED      0:0            49:0  out of memory

Last modified 15 April 2015

slurm-slurm-17-11-2-1/doc/html/job_launch.shtml000066400000000000000000000164511322351151600212150ustar00rootroot00000000000000

Job Launch Design Guide

Overview

This guide describes at a high level the processes which occur in order to initiate a job including the daemons and plugins involved in the process. It describes the process of job allocation, step allocation, task launch and job termination. The functionality of tens of thousands of lines of code has been distilled here to a couple of pages of text, so much detail is missing.

Job Allocation

The first step of the process is to create a job allocation, which is a claim on compute resources. A job allocation can be created using the salloc, sbatch or srun command. The salloc and sbatch commands create resource allocations while the srun command will create a resource allocation (if not already running within one) plus launch tasks. Each of these commands will fill in a data structure identifying the specifications of the job allocation requirement (e.g. node count, task count, etc.) based upon command line options and environment variables and send the RPC to the slurmctld daemon. If the new job request is the highest priority, the slurmctld daemon will attempt to select resources for it immediately, otherwise it will validate that the job request can be satisfied at some time and queue the request. In either case the request will receive a response almost immediately containing one of the following:

  • A job ID and the resource allocation specification (nodes, cpus, etc.)
  • A job ID and notification of the job being in a queued state OR
  • An error code

The process of selecting resources for a job request involves multiple steps, some of which involve plugins. The process is as follows:

  1. Call job_submit plugins to modify the request as appropriate
  2. Validate that the options are valid for this user (e.g. valid partition name, valid limits, etc.)
  3. Determine if this job is the highest priority runnable job, if so then really try to allocate resources for it now, otherwise only validate that it could run if no other jobs existed
  4. Determine which nodes could be used for the job. If the feature specification uses an exclusive OR option, then multiple iterations of the selection process below will be required with disjoint sets of nodes
  5. Call the select plugin to select the best resources for the request
  6. The select plugin will consider network topology and the topology within a node (e.g. sockets, cores, and threads) to select the best resources for the job
  7. If the job can not be initiated using available resources and preemption support is configured, the select plugin will also determine if the job can be initiated after preempting lower priority jobs. If so then initiate preemption as needed to start the job

Step Allocation

The srun command is always used for job step creation. It fills in a job step request RPC using information from the command line and environment variables then sends that request to the slurmctld daemon. It is important to note that many of the srun options are intended for job allocation and are not supported by the job step request RPC (for example the socket, core and thread information is not supported). If a job step uses all of the resources allocated to the job then the lack of support for some options is not important. If one wants to execute multiple job steps using various subsets of resources allocated to the job, this shortcoming could prove problematic. It is also worth noting that the logic used to select resources for a job step is relatively simple and entirely contained within the slurmctld daemon code (the select plugin is not used for job steps). If the request can not be immediately satisfied due to a request for exclusive access to resources, the appropriate error message will be sent and the srun command will retry the request on a periodic basis. (NOTE: It would be desirable to queue the job step requests to support job step dependencies and better performance in the initiation of job steps, but that is not currently supported.) If the request can be satisfied, the response contains a digitally signed credential (by the crypto plugin) identifying the resources to be used.

Task Launch

The srun command builds a task launch request data structure including the credential, executable name, file names, etc. and sends it to the slurmd daemon on node zero of the job step allocation. The slurmd daemon validates the signature and forwards the request to the slurmd daemons on other nodes to launch tasks for that job step. The degree of fanout in this message forwarding is configurable using the TreeWidth parameter. Each slurmd daemon tests that the job has not been cancelled since the credential was issued (due to a possible race condition) and spawns a slurmstepd program to manage the job step. Note that the slurmctld daemon is not directly involved in task launch in order to minimize the overhead on this critical resource.

Each slurmstepd program executes a single job step. Besides the functions listed below, the slurmstepd program also executes several SPANK plugin functions at various times.

  1. Performs MPI setup (using the appropriate plugin)
  2. Calls the switch plugin to perform any needed network configuration
  3. Creates a container for the job step using a proctrack plugin
  4. Change user ID to that of the user
  5. Configures I/O for the tasks (either using files or a socket connection back to the srun command
  6. Sets up environment variables for the tasks including many task-specific environment variables
  7. Fork/exec the tasks

Job Step Termination

There are several ways in which a job step or job can terminate, each with slight variation in the logic executed. The simplest case is if the tasks run to completion. The srun will note the termination of output from the tasks and notify the slurmctld daemon that the job step has completed. slurmctld will simply log the job step termination. The job step can also be explicitly cancelled by a user, reach the end of its time limit, etc. and those follow a sequence of steps very similar to that for job termination, which is described below.

Job Termination

Job termination can either be user initiated (e.g. scancel command) or system initiated (e.g. time limit reached). The termination ultimately requires the slurmctld daemon to notify the slurmd daemons on allocated nodes that the job is to be ended. The slurmd daemon does the following:

  1. Send a SIGCONT and SIGTERM signal to any user tasks
  2. Wait KilLWait seconds if there are any user tasks
  3. Send a SIGKILL signal to any user tasks
  4. Wait for all tasks to complete
  5. Execute any Epilog program
  6. Send an epilog_complete RPC to the slurmctld daemon

top

Last modified 18 May 2011

slurm-slurm-17-11-2-1/doc/html/job_submit_plugins.shtml000066400000000000000000000277541322351151600230170ustar00rootroot00000000000000

Job Submit Plugin API

Overview

This document describes Slurm job submit plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own Slurm job submit plugins. This is version 100 of the API.

Slurm job submit plugins must conform to the Slurm Plugin API with the following specifications:

const char plugin_name[]="full text name"

A free-formatted ASCII text string that identifies the plugin.

const char plugin_type[]="major/minor"

The major type must be "job_submit." The minor type can be any suitable name for the type of job submission package. We include samples in the Slurm distribution for

  • all_partitions — Set default partition to all partitions on the cluster.
  • defaults — Set default values for job submission or modify requests.
  • logging — Log select job submission and modification parameters.
  • lua — Interface to Lua scripts implementing these functions (actually a slight variation of them). Sample Lua scripts can be found with the Slurm distribution in the directory contribs/lua. The default installation location of the Lua scripts is the same location as the Slurm configuration file, slurm.conf.
  • partition — Sets a job's default partition based upon job submission parameters and available partitions.

const uint32_t plugin_version
If specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loadeed by Slurm commands and daemons from any version, however this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.

Slurm can be configured to use multiple job_submit plugins if desired, however the lua plugin will only execute one lua script named "job_submit.lua" located in the default script directory (typically the subdirectory "etc" of the installation directory).

top

API Functions

All of the following functions are required. Functions which are not implemented must be stubbed.

int init (void)

Description:
Called when the plugin is loaded, before any other functions are called. Put global initialization here.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void fini (void)

Description:
Called when the plugin is removed. Clear any allocated storage here.

Returns: None.

Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the Slurm init(), and the Slurm fini() is called before the system's _fini().

int job_submit(struct job_descriptor *job_desc, uint32_t submit_uid, char **error_msg)

Description:
This function is called by the slurmctld daemon with job submission parameters supplied by the salloc, sbatch or srun command. It can be used to log and/or modify the job parameters supplied by the user as desired. Note that this function has access to the slurmctld's global data structures, for example to examine the available partitions, reservations, etc.

Arguments:
job_desc (input/output) the job allocation request specifications.
submit_uid (input) user ID initiating the request.
error_msg (output) If the argument is not null, then a plugin generated error message can be stored here. The error message is expected to have allocated memory which Slurm will release using the xfree function. The error message is always propagated to the caller, no matter the return code.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int job_modify(struct job_descriptor *job_desc, struct job_record *job_ptr, uint32_t modify_uid)

Description:
This function is called by the slurmctld daemon with job modification parameters supplied by the scontrol or sview command. It can be used to log and/or modify the job parameters supplied by the user as desired. Note that this function has access to the slurmctld's global data structures, for example to examine the available partitions, reservations, etc.

Arguments:
job_desc (input/output) the job allocation request specifications.
job_ptr (input/output) slurmctld daemon's current data structure for the job to be modified.
modify_uid (input) user ID initiating the request.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

Lua Functions

The Lua functions differ slightly from those implemented in C for better ease of use. Sample Lua scripts can be found with the Slurm distribution in the directory contribs/lua. The default installation location of the Lua scripts is the same location as the Slurm configuration file, slurm.conf. Reading and writing of job environment variables using Lua is possible by referencing the environment variables as a data structure containing named elements. For example:
if (job_desc.environment.LANGUAGE == "en_US") then
....

int job_submit(struct job_descriptor *job_desc, List part_list, uint32_t submit_uid)

Description:
This function is called by the slurmctld daemon with job submission parameters supplied by the salloc, sbatch or srun command. It can be used to log and/or modify the job parameters supplied by the user as desired. Note that this function has access to the slurmctld's global data structures, for example to examine the available partitions, reservations, etc.

Arguments:
job_desc (input/output) the job allocation request specifications.
part_list (input) List of pointer to partitions which this user is authorized to use.
submit_uid (input) user ID initiating the request.

Returns:
0 on success, or an errno on failure. Slurm specific error numbers from slurm/slurm_errno.h may be used. On failure, the request will be rejected and the user will have an appropriate error message printed for that errno.

int job_modify(struct job_descriptor *job_desc, struct job_record *job_ptr, List part_list, int modify_uid)

Description:
This function is called by the slurmctld daemon with job modification parameters supplied by the scontrol or sview command. It can be used to log and/or modify the job parameters supplied by the user as desired. Note that this function has access to the slurmctld's global data structures, for example to examine the available partitions, reservations, etc.

Arguments:
job_desc (input/output) the job allocation request specifications.
job_ptr (input/output) slurmctld daemon's current data structure for the job to be modified.
part_list (input) List of pointer to partitions which this user is authorized to use.
modify_uid (input) user ID initiating the request.
0 on success, or an errno on failure. Slurm specific error numbers from slurm/slurm_errno.h may be used. On failure, the request will be rejected and the user will have an appropriate error message printed for that errno.

top

Building

Generally using a LUA interface for a job submit plugin is best: It is simple to write and maintain with minimal dependencies upon the Slurm data structures. However using C does provide a mechanism to get more information than available using LUA including full access to all of the data structures and functions in the slurmctld daemon. The simplest way to build a C program would be to just replace one of the job submit plugins included in the Slurm distribution with your own code (i.e. use a patch with your own code). Then just build and install Slurm with that new code. Building a new plugin outside of the Slurm distribution is possible, but far more complex. It also requires access to a multitude of Slurm header files as shown in the proceedure below.

  1. You will need to at least partly build Slurm first. The "configure" command must be executed in order to build the "config.h" file in the build directory.
  2. Create a local directory somewhere for your files to build with. Also create subdirectories named ".libs" and ".deps".
  3. Copy a ".deps/job_submit_*Plo" file from another job_submit plugin's ".deps" directory (made as part of the build process) into your local ".deps" subdirectory. Rename the file as appropriate to reflect your plugins name (e.g. rename "job_submit_partition.Plo" to be something like "job_submit_mine.Plo").
  4. Compile and link your plugin. Those options might differ depending upon your build environment. Check the options used for building the other job_submit plugins and modify the example below as required.
  5. Install the plugin.
# Example:
# The Slurm source is in ~/SLURM/slurm.git
# The Slurm build directory is ~/SLURM/slurm.build
# The plugin build is to take place in the directory
#   "~/SLURM/my_submit"
# The installation locaiton is "/usr/local"

# Build Slurm from ~/SLURM/slurm.build
# (or at least run "~/SLURM/slurm.git/configure")

# Set up your plugin files
cd ~/SLURM
mkdir my_submit
cd my_submit
mkdir .libs
mkdir .deps
# Create your plugin code
vi job_submit_mine.c

# Copy up a dependency file
cp ~/SLURM/slurm.build/src/plugins/job_submit/partition/.deps/job_submit_partition.Plo \
   .deps/job_submit_mine.Plo

# Compile
gcc -DHAVE_CONFIG_H -I~/SLURM/slurm.build -I~/slurm.git \
   -g -O2 -pthread -fno-gcse -Werror -Wall -g -O0       \
   -fno-strict-aliasing -MT job_submit_mine.lo          \
   -MD -MP -MF .deps/job_submit_mine.Tpo                \
   -c job_submit_mine.c -o .libs/job_submit_mine.o

# Some clean up
mv -f .deps/job_submit_mine.Tpo .deps/job_submit_mine.Plo
rm -fr .libs/job_submit_mine.a .libs/job_submit_mine.la \
   .libs/job_submit_mine.lai job_submit_mine.so

# Link
gcc -shared -fPIC -DPIC .libs/job_submit_mine.o -O2         \
   -pthread -O0 -pthread -Wl,-soname -Wl,job_submit_mine.so \
   -o job_submit_mine.so

# Install
cp job_submit_mine.so file \
   /usr/local/lib/slurm/job_submit_mine.so

Last modified 29 September 2017

slurm-slurm-17-11-2-1/doc/html/jobacct_gatherplugins.shtml000066400000000000000000000353741322351151600234570ustar00rootroot00000000000000

Slurm Job Accounting Gather Plugin API

Overview

This document describes Slurm job accounting gather plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own Slurm job accounting gather plugins.

Slurm job accounting gather plugins must conform to the Slurm Plugin API with the following specifications:

const char plugin_name[]="full text name"

A free-formatted ASCII text string that identifies the plugin.

const char plugin_type[]="major/minor"

The major type must be "jobacct_gather." The minor type can be any suitable name for the type of accounting package. We currently use

  • cgroup — Gathers information from Linux cgroup infrastructure and adds this information to the standard rusage information also gathered for each job. (Experimental, not to be used in production.)
  • linux — Gathers information from Linux /proc table and adds this information to the standard rusage information also gathered for each job.
  • none — No information gathered.

const uint32_t plugin_version
If specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loadeed by Slurm commands and daemons from any version, however this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.

The sacct program can be used to display gathered data from regular accounting and from these plugins.

The programmer is urged to study src/plugins/jobacct_gather/linux and src/common/slurm_jobacct_gather.[c|h] for a sample implementation of a Slurm job accounting gather plugin.

top

API Functions

All of the following functions are required. Functions which are not implemented must be stubbed.

int init (void)

Description:
Called when the plugin is loaded, before any other functions are called. Put global initialization here.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void fini (void)

Description:
Called when the plugin is removed. Clear any allocated storage here.

Returns: None.

Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the Slurm init(), and the Slurm fini() is called before the system's _fini().

int jobacct_gather_p_poll_data(List task_list, bool pgid_plugin, uint64_t cont_id)

Description:
Build a table of all current processes.

Arguments:
task_list (in/out) List containing current processes
pgid_plugin (input) if we are running with the pgid plugin
cont_id (input) container id of processes if not running with pgid

int jobacct_gather_p_endpoll(void)

Description:
Called when the process is finished to stop the polling thread.

Arguments:
none

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacct_gather_p_add_task(pid_t pid, uint16_t tid)

Description:
Used to add a task to the poller.

Arguments:
pid (input) Process id
tid (input) slurm global task id

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

Job Account Gathering

All of the following functions are not required but may be used.

int jobacct_gather_init(void)

Description:
Loads the job account gather plugin.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacct_gather_fini(void)

Description:
Unloads the job account gathering plugin.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacct_gather_startpoll(uin16_t frequency)

Description:
Creates and starts the polling thread.

Arguments:
frequency (input) frequency of the polling.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacct_gather_change_poll(uint16_t frequency)

Description:
Changes the polling thread to a new frequency.

Arguments:
frequency (input) frequency of the polling

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacct_gather_suspend_poll(void)

Description:
Temporarily stops the polling thread.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacct_gather_resume_poll(void)

Description:
Resumes the polling thread that was stopped.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

jobacctinfo_t *jobacct_gather_stat_task(pid_t pid)

Description:
Gets the basis of the information of the task.

Arguments:
pid (input) process id.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

jobacctinfo_t *jobacct_gather_remove_task(pid_t pid)

Description:
Removes the task.

Arguments:
pid (input) process id.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacct_gather_set_proctrack_container_id(uint64_t id)

Description:
Sets the proctrack container to a given id.

Arguments:
id (input) id to set.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacct_gather_set_mem_limit(uint32_t job_id, uint32_t step_id, uint64_t mem_limit)

Description:
Sets the memory limit of the job account.

Arguments:
job_id (input) id of the job.
sted_id (input) id of the step.
mem_limit (input) memory limit in megabytes.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacct_gather_handle_mem_limit(uint64_t total_job_mem, uint64_t total_job_vsize)

Description:
Called to find out how much memory is used.

Arguments:
total_job_mem (input) total amount of memory for jobs.
total_job_vsize (input) the total job size.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

Job Account Info

All of the following functions are not required but may be used.

jobacctinfo_t *jobacctinfo_create(jobacct_id_t *jobacct_id)

Description:
Creates the job account info.

Arguments:
jobacct_id (input) the job account id.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacctinfo_destroy(void *object)

Description:
Destroys the job account info.

Arguments:
object (input) the job that needs to be destroyed

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacctinfo_setinfo(jobacctinfo_t *jobacct, enum jobacct_data_type type, void *data)

Description:
Set the information for the job.

Arguments:
jobacct (input) job account
type(input) enum telling the plugin how to transform the data.
data (input/output) Is a void * and the actual data type depends upon the first argument to this function (type).

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacctinfo_getinfo(jobacctinfo_t *jobacct, enum jobacct_data_type type, void *data)

Description:
Gets the information about the job.

Arguments:
jobacct (input) job account.
type (input) the data type of the job account. data

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacctinfo_pack(jobacctinfo_t *jobacct, uint16_t rpc_version, Buf buffer)

Description:
Packs the job account information.

Arguments:
jobacct (input) the job account.
rpc_version (input) the rpc version.
buffer (input) the buffer.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

int jobacctinfo_unpack(jobacctinfo_t **jobacct, uint16_t rpc_version, Buf buffer)

Description:
Unpacks the job account information.

Arguments:
jobacct (input) the job account.
rpc_version (input) the rpc version.
buffer (input) the buffer.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacctinfo_aggregate(jobacctinfo_t *dest, jobacctinfo_t *from)

Description:
Aggregates the jobs.

Arguments:
dest (input) New destination of the job.
from (input) Original location of job.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void jobacctinfo_2_stats(slurmdb_stats_t *stats, jobacctinfo_t *jobacct)

Description:
Gets the stats of the job in accounting.

Arguments:
stats (input) slurm data base stat.
jobacct (input) the job account.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

top

Parameters

These parameters can be used in the slurm.conf to configure the plugin and the frequency at which to gather information about running jobs.

JobAcctGatherType
Specifies which plugin should be used.
JobAcctGatherFrequency
Time interval between pollings in seconds.

top

Last modified 27 March 2015

slurm-slurm-17-11-2-1/doc/html/jobcompplugins.shtml000066400000000000000000000210371322351151600221400ustar00rootroot00000000000000

Slurm Job Completion Logging Plugin API

Overview

This document describes Slurm job completion logging plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own SLURM job completion logging plugins.

Slurm job completion logging plugins are Slurm plugins that implement the SLURM API for logging job information upon their completion. This may be used to log job information to a text file, database, etc. The plugins must conform to the Slurm Plugin API with the following specifications:

const char plugin_type[]
The major type must be "jobcomp." The minor type can be any recognizable abbreviation for the type of scheduler. We recommend, for example:

  • none — No job logging.
  • filetxt — Log job information to a text file.
  • mysql — Job completion is written to a mysql database.
  • script — Execute a script passing in job information in environment variables.

const char plugin_name[]
Some descriptive name for the plugin. There is no requirement with respect to its format.

const uint32_t plugin_version
If specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loadeed by Slurm commands and daemons from any version, however this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.

The programmer is urged to study src/plugins/jobcomp/filetxt/jobcomp_filetxt.c and src/plugins/jobcomp/none/jobcomp_none.c for sample implementations of a Slurm job completion logging plugin.

top

Data Objects

The implementation must maintain (though not necessarily directly export) an enumerated errno to allow Slurm to discover as practically as possible the reason for any failed API call. Plugin-specific enumerated integer values should be used when appropriate. It is desirable that these values be mapped into the range ESLURM_JOBCOMP_MIN and ESLURM_JOBCOMP_MAX as defined in slurm/slurm_errno.h. The error number should be returned by the function slurm_jobcomp_get_errno() and this error number can be converted to an appropriate string description using the slurm_jobcomp_strerror() function described below.

These values must not be used as return values in integer-valued functions in the API. The proper error return value from integer-valued functions is SLURM_ERROR. The implementation should endeavor to provide useful and pertinent information by whatever means is practical. Successful API calls are not required to reset any errno to a known value. However, the initial value of any errno, prior to any error condition arising, should be SLURM_SUCCESS.

top

API Functions

The following functions must appear. Functions which are not implemented should be stubbed.

int init (void)

Description:
Called when the plugin is loaded, before any other functions are called. Put global initialization here.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void fini (void)

Description:
Called when the plugin is removed. Clear any allocated storage here.

Returns: None.

Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the SLURM init(), and the SLURM fini() is called before the system's _fini().

int slurm_jobcomp_set_location (char * location);

Description: Specify the location to be used for job logging.

Argument: location    (input) specification of where logging should be done. The interpretation of this string is at the discretion of the plugin implementation.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR and set the errno to an appropriate value to indicate the reason for failure.

top

int slurm_jobcomp_log_record (struct job_record *job_ptr);

Description: Note that a job is about to terminate or change size. The job's state will include the JOB_RESIZING flag if and only if it is about to change size. Otherwise the job is terminating. Note the existence of resize_time in the job record if one wishes to record information about a job at each size (i.e. a history of the job as its size changes through time).

Argument:
job_ptr   (input) Pointer to job record as defined in src/slurmctld/slurmctld.h

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR and set the errno to an appropriate value to indicate the reason for failure.

top

int slurm_jobcomp_get_errno (void);

Description: Return the number of a job completion logger specific error.

Arguments: None

Returns: Error number for the last failure encountered by the job completion logging plugin.

top

const char *slurm_jobcomp_strerror(int errnum);

Description: Return a string description of a job completion logger specific error code.

Arguments: errnum    (input) a job completion logger specific error code.

Returns: Pointer to string describing the error or NULL if no description found in this plugin.

top

List slurm_jobcomp_get_jobs(acct_job_cond_t *job_cond);

Description: Get completed job info from storage.

Arguments:
job_cond     (input) specification of filters to identify the jobs we wish information about (start time, end time, cluster name, user id, etc). acct_job_cond_t is defined in common/slurm_accounting_storage.h.

Returns: A list of job records or NULL on error. Elements on the list are of type jobcomp_job_rec_t, which is defined in common/slurm_jobcomp.h. Any returned list must be destroyed to avoid memory leaks.

top

void slurm_jobcomp_archive(List selected_parts, void *params)

Description: used to archive old data.

Arguments:
List selected_parts (input) list containing char *'s of names of partitions to query against.
void *params (input) to be cast as sacct_parameters_t in the plugin.

Returns: None

top

top

Last modified 27 March 2015

slurm-slurm-17-11-2-1/doc/html/jquery.min.js000066400000000000000000002505171322351151600205020ustar00rootroot00000000000000/*! jQuery v3.1.0 | (c) jQuery Foundation | jquery.org/license */ !function(a,b){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){"use strict";var c=[],d=a.document,e=Object.getPrototypeOf,f=c.slice,g=c.concat,h=c.push,i=c.indexOf,j={},k=j.toString,l=j.hasOwnProperty,m=l.toString,n=m.call(Object),o={};function p(a,b){b=b||d;var c=b.createElement("script");c.text=a,b.head.appendChild(c).parentNode.removeChild(c)}var q="3.1.0",r=function(a,b){return new r.fn.init(a,b)},s=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,t=/^-ms-/,u=/-([a-z])/g,v=function(a,b){return b.toUpperCase()};r.fn=r.prototype={jquery:q,constructor:r,length:0,toArray:function(){return f.call(this)},get:function(a){return null!=a?a<0?this[a+this.length]:this[a]:f.call(this)},pushStack:function(a){var b=r.merge(this.constructor(),a);return b.prevObject=this,b},each:function(a){return r.each(this,a)},map:function(a){return this.pushStack(r.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(f.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(a<0?b:0);return this.pushStack(c>=0&&c0&&b-1 in a)}var x=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+1*new Date,v=a.document,w=0,x=0,y=ha(),z=ha(),A=ha(),B=function(a,b){return a===b&&(l=!0),0},C={}.hasOwnProperty,D=[],E=D.pop,F=D.push,G=D.push,H=D.slice,I=function(a,b){for(var c=0,d=a.length;c+~]|"+K+")"+K+"*"),S=new RegExp("="+K+"*([^\\]'\"]*?)"+K+"*\\]","g"),T=new RegExp(N),U=new RegExp("^"+L+"$"),V={ID:new RegExp("^#("+L+")"),CLASS:new RegExp("^\\.("+L+")"),TAG:new RegExp("^("+L+"|[*])"),ATTR:new RegExp("^"+M),PSEUDO:new RegExp("^"+N),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+K+"*(even|odd|(([+-]|)(\\d*)n|)"+K+"*(?:([+-]|)"+K+"*(\\d+)|))"+K+"*\\)|)","i"),bool:new RegExp("^(?:"+J+")$","i"),needsContext:new RegExp("^"+K+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+K+"*((?:-\\d)?\\d*)"+K+"*\\)|)(?=[^-]|$)","i")},W=/^(?:input|select|textarea|button)$/i,X=/^h\d$/i,Y=/^[^{]+\{\s*\[native \w/,Z=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,$=/[+~]/,_=new RegExp("\\\\([\\da-f]{1,6}"+K+"?|("+K+")|.)","ig"),aa=function(a,b,c){var d="0x"+b-65536;return d!==d||c?b:d<0?String.fromCharCode(d+65536):String.fromCharCode(d>>10|55296,1023&d|56320)},ba=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\x80-\uFFFF\w-]/g,ca=function(a,b){return b?"\0"===a?"\ufffd":a.slice(0,-1)+"\\"+a.charCodeAt(a.length-1).toString(16)+" ":"\\"+a},da=function(){m()},ea=ta(function(a){return a.disabled===!0},{dir:"parentNode",next:"legend"});try{G.apply(D=H.call(v.childNodes),v.childNodes),D[v.childNodes.length].nodeType}catch(fa){G={apply:D.length?function(a,b){F.apply(a,H.call(b))}:function(a,b){var c=a.length,d=0;while(a[c++]=b[d++]);a.length=c-1}}}function ga(a,b,d,e){var f,h,j,k,l,o,r,s=b&&b.ownerDocument,w=b?b.nodeType:9;if(d=d||[],"string"!=typeof a||!a||1!==w&&9!==w&&11!==w)return d;if(!e&&((b?b.ownerDocument||b:v)!==n&&m(b),b=b||n,p)){if(11!==w&&(l=Z.exec(a)))if(f=l[1]){if(9===w){if(!(j=b.getElementById(f)))return d;if(j.id===f)return d.push(j),d}else if(s&&(j=s.getElementById(f))&&t(b,j)&&j.id===f)return d.push(j),d}else{if(l[2])return G.apply(d,b.getElementsByTagName(a)),d;if((f=l[3])&&c.getElementsByClassName&&b.getElementsByClassName)return G.apply(d,b.getElementsByClassName(f)),d}if(c.qsa&&!A[a+" "]&&(!q||!q.test(a))){if(1!==w)s=b,r=a;else if("object"!==b.nodeName.toLowerCase()){(k=b.getAttribute("id"))?k=k.replace(ba,ca):b.setAttribute("id",k=u),o=g(a),h=o.length;while(h--)o[h]="#"+k+" "+sa(o[h]);r=o.join(","),s=$.test(a)&&qa(b.parentNode)||b}if(r)try{return G.apply(d,s.querySelectorAll(r)),d}catch(x){}finally{k===u&&b.removeAttribute("id")}}}return i(a.replace(P,"$1"),b,d,e)}function ha(){var a=[];function b(c,e){return a.push(c+" ")>d.cacheLength&&delete b[a.shift()],b[c+" "]=e}return b}function ia(a){return a[u]=!0,a}function ja(a){var b=n.createElement("fieldset");try{return!!a(b)}catch(c){return!1}finally{b.parentNode&&b.parentNode.removeChild(b),b=null}}function ka(a,b){var c=a.split("|"),e=c.length;while(e--)d.attrHandle[c[e]]=b}function la(a,b){var c=b&&a,d=c&&1===a.nodeType&&1===b.nodeType&&a.sourceIndex-b.sourceIndex;if(d)return d;if(c)while(c=c.nextSibling)if(c===b)return-1;return a?1:-1}function ma(a){return function(b){var c=b.nodeName.toLowerCase();return"input"===c&&b.type===a}}function na(a){return function(b){var c=b.nodeName.toLowerCase();return("input"===c||"button"===c)&&b.type===a}}function oa(a){return function(b){return"label"in b&&b.disabled===a||"form"in b&&b.disabled===a||"form"in b&&b.disabled===!1&&(b.isDisabled===a||b.isDisabled!==!a&&("label"in b||!ea(b))!==a)}}function pa(a){return ia(function(b){return b=+b,ia(function(c,d){var e,f=a([],c.length,b),g=f.length;while(g--)c[e=f[g]]&&(c[e]=!(d[e]=c[e]))})})}function qa(a){return a&&"undefined"!=typeof a.getElementsByTagName&&a}c=ga.support={},f=ga.isXML=function(a){var b=a&&(a.ownerDocument||a).documentElement;return!!b&&"HTML"!==b.nodeName},m=ga.setDocument=function(a){var b,e,g=a?a.ownerDocument||a:v;return g!==n&&9===g.nodeType&&g.documentElement?(n=g,o=n.documentElement,p=!f(n),v!==n&&(e=n.defaultView)&&e.top!==e&&(e.addEventListener?e.addEventListener("unload",da,!1):e.attachEvent&&e.attachEvent("onunload",da)),c.attributes=ja(function(a){return a.className="i",!a.getAttribute("className")}),c.getElementsByTagName=ja(function(a){return a.appendChild(n.createComment("")),!a.getElementsByTagName("*").length}),c.getElementsByClassName=Y.test(n.getElementsByClassName),c.getById=ja(function(a){return o.appendChild(a).id=u,!n.getElementsByName||!n.getElementsByName(u).length}),c.getById?(d.find.ID=function(a,b){if("undefined"!=typeof b.getElementById&&p){var c=b.getElementById(a);return c?[c]:[]}},d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){return a.getAttribute("id")===b}}):(delete d.find.ID,d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){var c="undefined"!=typeof a.getAttributeNode&&a.getAttributeNode("id");return c&&c.value===b}}),d.find.TAG=c.getElementsByTagName?function(a,b){return"undefined"!=typeof b.getElementsByTagName?b.getElementsByTagName(a):c.qsa?b.querySelectorAll(a):void 0}:function(a,b){var c,d=[],e=0,f=b.getElementsByTagName(a);if("*"===a){while(c=f[e++])1===c.nodeType&&d.push(c);return d}return f},d.find.CLASS=c.getElementsByClassName&&function(a,b){if("undefined"!=typeof b.getElementsByClassName&&p)return b.getElementsByClassName(a)},r=[],q=[],(c.qsa=Y.test(n.querySelectorAll))&&(ja(function(a){o.appendChild(a).innerHTML="",a.querySelectorAll("[msallowcapture^='']").length&&q.push("[*^$]="+K+"*(?:''|\"\")"),a.querySelectorAll("[selected]").length||q.push("\\["+K+"*(?:value|"+J+")"),a.querySelectorAll("[id~="+u+"-]").length||q.push("~="),a.querySelectorAll(":checked").length||q.push(":checked"),a.querySelectorAll("a#"+u+"+*").length||q.push(".#.+[+~]")}),ja(function(a){a.innerHTML="";var b=n.createElement("input");b.setAttribute("type","hidden"),a.appendChild(b).setAttribute("name","D"),a.querySelectorAll("[name=d]").length&&q.push("name"+K+"*[*^$|!~]?="),2!==a.querySelectorAll(":enabled").length&&q.push(":enabled",":disabled"),o.appendChild(a).disabled=!0,2!==a.querySelectorAll(":disabled").length&&q.push(":enabled",":disabled"),a.querySelectorAll("*,:x"),q.push(",.*:")})),(c.matchesSelector=Y.test(s=o.matches||o.webkitMatchesSelector||o.mozMatchesSelector||o.oMatchesSelector||o.msMatchesSelector))&&ja(function(a){c.disconnectedMatch=s.call(a,"*"),s.call(a,"[s!='']:x"),r.push("!=",N)}),q=q.length&&new RegExp(q.join("|")),r=r.length&&new RegExp(r.join("|")),b=Y.test(o.compareDocumentPosition),t=b||Y.test(o.contains)?function(a,b){var c=9===a.nodeType?a.documentElement:a,d=b&&b.parentNode;return a===d||!(!d||1!==d.nodeType||!(c.contains?c.contains(d):a.compareDocumentPosition&&16&a.compareDocumentPosition(d)))}:function(a,b){if(b)while(b=b.parentNode)if(b===a)return!0;return!1},B=b?function(a,b){if(a===b)return l=!0,0;var d=!a.compareDocumentPosition-!b.compareDocumentPosition;return d?d:(d=(a.ownerDocument||a)===(b.ownerDocument||b)?a.compareDocumentPosition(b):1,1&d||!c.sortDetached&&b.compareDocumentPosition(a)===d?a===n||a.ownerDocument===v&&t(v,a)?-1:b===n||b.ownerDocument===v&&t(v,b)?1:k?I(k,a)-I(k,b):0:4&d?-1:1)}:function(a,b){if(a===b)return l=!0,0;var c,d=0,e=a.parentNode,f=b.parentNode,g=[a],h=[b];if(!e||!f)return a===n?-1:b===n?1:e?-1:f?1:k?I(k,a)-I(k,b):0;if(e===f)return la(a,b);c=a;while(c=c.parentNode)g.unshift(c);c=b;while(c=c.parentNode)h.unshift(c);while(g[d]===h[d])d++;return d?la(g[d],h[d]):g[d]===v?-1:h[d]===v?1:0},n):n},ga.matches=function(a,b){return ga(a,null,null,b)},ga.matchesSelector=function(a,b){if((a.ownerDocument||a)!==n&&m(a),b=b.replace(S,"='$1']"),c.matchesSelector&&p&&!A[b+" "]&&(!r||!r.test(b))&&(!q||!q.test(b)))try{var d=s.call(a,b);if(d||c.disconnectedMatch||a.document&&11!==a.document.nodeType)return d}catch(e){}return ga(b,n,null,[a]).length>0},ga.contains=function(a,b){return(a.ownerDocument||a)!==n&&m(a),t(a,b)},ga.attr=function(a,b){(a.ownerDocument||a)!==n&&m(a);var e=d.attrHandle[b.toLowerCase()],f=e&&C.call(d.attrHandle,b.toLowerCase())?e(a,b,!p):void 0;return void 0!==f?f:c.attributes||!p?a.getAttribute(b):(f=a.getAttributeNode(b))&&f.specified?f.value:null},ga.escape=function(a){return(a+"").replace(ba,ca)},ga.error=function(a){throw new Error("Syntax error, unrecognized expression: "+a)},ga.uniqueSort=function(a){var b,d=[],e=0,f=0;if(l=!c.detectDuplicates,k=!c.sortStable&&a.slice(0),a.sort(B),l){while(b=a[f++])b===a[f]&&(e=d.push(f));while(e--)a.splice(d[e],1)}return k=null,a},e=ga.getText=function(a){var b,c="",d=0,f=a.nodeType;if(f){if(1===f||9===f||11===f){if("string"==typeof a.textContent)return a.textContent;for(a=a.firstChild;a;a=a.nextSibling)c+=e(a)}else if(3===f||4===f)return a.nodeValue}else while(b=a[d++])c+=e(b);return c},d=ga.selectors={cacheLength:50,createPseudo:ia,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(a){return a[1]=a[1].replace(_,aa),a[3]=(a[3]||a[4]||a[5]||"").replace(_,aa),"~="===a[2]&&(a[3]=" "+a[3]+" "),a.slice(0,4)},CHILD:function(a){return a[1]=a[1].toLowerCase(),"nth"===a[1].slice(0,3)?(a[3]||ga.error(a[0]),a[4]=+(a[4]?a[5]+(a[6]||1):2*("even"===a[3]||"odd"===a[3])),a[5]=+(a[7]+a[8]||"odd"===a[3])):a[3]&&ga.error(a[0]),a},PSEUDO:function(a){var b,c=!a[6]&&a[2];return V.CHILD.test(a[0])?null:(a[3]?a[2]=a[4]||a[5]||"":c&&T.test(c)&&(b=g(c,!0))&&(b=c.indexOf(")",c.length-b)-c.length)&&(a[0]=a[0].slice(0,b),a[2]=c.slice(0,b)),a.slice(0,3))}},filter:{TAG:function(a){var b=a.replace(_,aa).toLowerCase();return"*"===a?function(){return!0}:function(a){return a.nodeName&&a.nodeName.toLowerCase()===b}},CLASS:function(a){var b=y[a+" "];return b||(b=new RegExp("(^|"+K+")"+a+"("+K+"|$)"))&&y(a,function(a){return b.test("string"==typeof a.className&&a.className||"undefined"!=typeof a.getAttribute&&a.getAttribute("class")||"")})},ATTR:function(a,b,c){return function(d){var e=ga.attr(d,a);return null==e?"!="===b:!b||(e+="","="===b?e===c:"!="===b?e!==c:"^="===b?c&&0===e.indexOf(c):"*="===b?c&&e.indexOf(c)>-1:"$="===b?c&&e.slice(-c.length)===c:"~="===b?(" "+e.replace(O," ")+" ").indexOf(c)>-1:"|="===b&&(e===c||e.slice(0,c.length+1)===c+"-"))}},CHILD:function(a,b,c,d,e){var f="nth"!==a.slice(0,3),g="last"!==a.slice(-4),h="of-type"===b;return 1===d&&0===e?function(a){return!!a.parentNode}:function(b,c,i){var j,k,l,m,n,o,p=f!==g?"nextSibling":"previousSibling",q=b.parentNode,r=h&&b.nodeName.toLowerCase(),s=!i&&!h,t=!1;if(q){if(f){while(p){m=b;while(m=m[p])if(h?m.nodeName.toLowerCase()===r:1===m.nodeType)return!1;o=p="only"===a&&!o&&"nextSibling"}return!0}if(o=[g?q.firstChild:q.lastChild],g&&s){m=q,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n&&j[2],m=n&&q.childNodes[n];while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if(1===m.nodeType&&++t&&m===b){k[a]=[w,n,t];break}}else if(s&&(m=b,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n),t===!1)while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if((h?m.nodeName.toLowerCase()===r:1===m.nodeType)&&++t&&(s&&(l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),k[a]=[w,t]),m===b))break;return t-=e,t===d||t%d===0&&t/d>=0}}},PSEUDO:function(a,b){var c,e=d.pseudos[a]||d.setFilters[a.toLowerCase()]||ga.error("unsupported pseudo: "+a);return e[u]?e(b):e.length>1?(c=[a,a,"",b],d.setFilters.hasOwnProperty(a.toLowerCase())?ia(function(a,c){var d,f=e(a,b),g=f.length;while(g--)d=I(a,f[g]),a[d]=!(c[d]=f[g])}):function(a){return e(a,0,c)}):e}},pseudos:{not:ia(function(a){var b=[],c=[],d=h(a.replace(P,"$1"));return d[u]?ia(function(a,b,c,e){var f,g=d(a,null,e,[]),h=a.length;while(h--)(f=g[h])&&(a[h]=!(b[h]=f))}):function(a,e,f){return b[0]=a,d(b,null,f,c),b[0]=null,!c.pop()}}),has:ia(function(a){return function(b){return ga(a,b).length>0}}),contains:ia(function(a){return a=a.replace(_,aa),function(b){return(b.textContent||b.innerText||e(b)).indexOf(a)>-1}}),lang:ia(function(a){return U.test(a||"")||ga.error("unsupported lang: "+a),a=a.replace(_,aa).toLowerCase(),function(b){var c;do if(c=p?b.lang:b.getAttribute("xml:lang")||b.getAttribute("lang"))return c=c.toLowerCase(),c===a||0===c.indexOf(a+"-");while((b=b.parentNode)&&1===b.nodeType);return!1}}),target:function(b){var c=a.location&&a.location.hash;return c&&c.slice(1)===b.id},root:function(a){return a===o},focus:function(a){return a===n.activeElement&&(!n.hasFocus||n.hasFocus())&&!!(a.type||a.href||~a.tabIndex)},enabled:oa(!1),disabled:oa(!0),checked:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&!!a.checked||"option"===b&&!!a.selected},selected:function(a){return a.parentNode&&a.parentNode.selectedIndex,a.selected===!0},empty:function(a){for(a=a.firstChild;a;a=a.nextSibling)if(a.nodeType<6)return!1;return!0},parent:function(a){return!d.pseudos.empty(a)},header:function(a){return X.test(a.nodeName)},input:function(a){return W.test(a.nodeName)},button:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&"button"===a.type||"button"===b},text:function(a){var b;return"input"===a.nodeName.toLowerCase()&&"text"===a.type&&(null==(b=a.getAttribute("type"))||"text"===b.toLowerCase())},first:pa(function(){return[0]}),last:pa(function(a,b){return[b-1]}),eq:pa(function(a,b,c){return[c<0?c+b:c]}),even:pa(function(a,b){for(var c=0;c=0;)a.push(d);return a}),gt:pa(function(a,b,c){for(var d=c<0?c+b:c;++d1?function(b,c,d){var e=a.length;while(e--)if(!a[e](b,c,d))return!1;return!0}:a[0]}function va(a,b,c){for(var d=0,e=b.length;d-1&&(f[j]=!(g[j]=l))}}else r=wa(r===g?r.splice(o,r.length):r),e?e(null,g,r,i):G.apply(g,r)})}function ya(a){for(var b,c,e,f=a.length,g=d.relative[a[0].type],h=g||d.relative[" "],i=g?1:0,k=ta(function(a){return a===b},h,!0),l=ta(function(a){return I(b,a)>-1},h,!0),m=[function(a,c,d){var e=!g&&(d||c!==j)||((b=c).nodeType?k(a,c,d):l(a,c,d));return b=null,e}];i1&&ua(m),i>1&&sa(a.slice(0,i-1).concat({value:" "===a[i-2].type?"*":""})).replace(P,"$1"),c,i0,e=a.length>0,f=function(f,g,h,i,k){var l,o,q,r=0,s="0",t=f&&[],u=[],v=j,x=f||e&&d.find.TAG("*",k),y=w+=null==v?1:Math.random()||.1,z=x.length;for(k&&(j=g===n||g||k);s!==z&&null!=(l=x[s]);s++){if(e&&l){o=0,g||l.ownerDocument===n||(m(l),h=!p);while(q=a[o++])if(q(l,g||n,h)){i.push(l);break}k&&(w=y)}c&&((l=!q&&l)&&r--,f&&t.push(l))}if(r+=s,c&&s!==r){o=0;while(q=b[o++])q(t,u,g,h);if(f){if(r>0)while(s--)t[s]||u[s]||(u[s]=E.call(i));u=wa(u)}G.apply(i,u),k&&!f&&u.length>0&&r+b.length>1&&ga.uniqueSort(i)}return k&&(w=y,j=v),t};return c?ia(f):f}return h=ga.compile=function(a,b){var c,d=[],e=[],f=A[a+" "];if(!f){b||(b=g(a)),c=b.length;while(c--)f=ya(b[c]),f[u]?d.push(f):e.push(f);f=A(a,za(e,d)),f.selector=a}return f},i=ga.select=function(a,b,e,f){var i,j,k,l,m,n="function"==typeof a&&a,o=!f&&g(a=n.selector||a);if(e=e||[],1===o.length){if(j=o[0]=o[0].slice(0),j.length>2&&"ID"===(k=j[0]).type&&c.getById&&9===b.nodeType&&p&&d.relative[j[1].type]){if(b=(d.find.ID(k.matches[0].replace(_,aa),b)||[])[0],!b)return e;n&&(b=b.parentNode),a=a.slice(j.shift().value.length)}i=V.needsContext.test(a)?0:j.length;while(i--){if(k=j[i],d.relative[l=k.type])break;if((m=d.find[l])&&(f=m(k.matches[0].replace(_,aa),$.test(j[0].type)&&qa(b.parentNode)||b))){if(j.splice(i,1),a=f.length&&sa(j),!a)return G.apply(e,f),e;break}}}return(n||h(a,o))(f,b,!p,e,!b||$.test(a)&&qa(b.parentNode)||b),e},c.sortStable=u.split("").sort(B).join("")===u,c.detectDuplicates=!!l,m(),c.sortDetached=ja(function(a){return 1&a.compareDocumentPosition(n.createElement("fieldset"))}),ja(function(a){return a.innerHTML="","#"===a.firstChild.getAttribute("href")})||ka("type|href|height|width",function(a,b,c){if(!c)return a.getAttribute(b,"type"===b.toLowerCase()?1:2)}),c.attributes&&ja(function(a){return a.innerHTML="",a.firstChild.setAttribute("value",""),""===a.firstChild.getAttribute("value")})||ka("value",function(a,b,c){if(!c&&"input"===a.nodeName.toLowerCase())return a.defaultValue}),ja(function(a){return null==a.getAttribute("disabled")})||ka(J,function(a,b,c){var d;if(!c)return a[b]===!0?b.toLowerCase():(d=a.getAttributeNode(b))&&d.specified?d.value:null}),ga}(a);r.find=x,r.expr=x.selectors,r.expr[":"]=r.expr.pseudos,r.uniqueSort=r.unique=x.uniqueSort,r.text=x.getText,r.isXMLDoc=x.isXML,r.contains=x.contains,r.escapeSelector=x.escape;var y=function(a,b,c){var d=[],e=void 0!==c;while((a=a[b])&&9!==a.nodeType)if(1===a.nodeType){if(e&&r(a).is(c))break;d.push(a)}return d},z=function(a,b){for(var c=[];a;a=a.nextSibling)1===a.nodeType&&a!==b&&c.push(a);return c},A=r.expr.match.needsContext,B=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i,C=/^.[^:#\[\.,]*$/;function D(a,b,c){if(r.isFunction(b))return r.grep(a,function(a,d){return!!b.call(a,d,a)!==c});if(b.nodeType)return r.grep(a,function(a){return a===b!==c});if("string"==typeof b){if(C.test(b))return r.filter(b,a,c);b=r.filter(b,a)}return r.grep(a,function(a){return i.call(b,a)>-1!==c&&1===a.nodeType})}r.filter=function(a,b,c){var d=b[0];return c&&(a=":not("+a+")"),1===b.length&&1===d.nodeType?r.find.matchesSelector(d,a)?[d]:[]:r.find.matches(a,r.grep(b,function(a){return 1===a.nodeType}))},r.fn.extend({find:function(a){var b,c,d=this.length,e=this;if("string"!=typeof a)return this.pushStack(r(a).filter(function(){for(b=0;b1?r.uniqueSort(c):c},filter:function(a){return this.pushStack(D(this,a||[],!1))},not:function(a){return this.pushStack(D(this,a||[],!0))},is:function(a){return!!D(this,"string"==typeof a&&A.test(a)?r(a):a||[],!1).length}});var E,F=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/,G=r.fn.init=function(a,b,c){var e,f;if(!a)return this;if(c=c||E,"string"==typeof a){if(e="<"===a[0]&&">"===a[a.length-1]&&a.length>=3?[null,a,null]:F.exec(a),!e||!e[1]&&b)return!b||b.jquery?(b||c).find(a):this.constructor(b).find(a);if(e[1]){if(b=b instanceof r?b[0]:b,r.merge(this,r.parseHTML(e[1],b&&b.nodeType?b.ownerDocument||b:d,!0)),B.test(e[1])&&r.isPlainObject(b))for(e in b)r.isFunction(this[e])?this[e](b[e]):this.attr(e,b[e]);return this}return f=d.getElementById(e[2]),f&&(this[0]=f,this.length=1),this}return a.nodeType?(this[0]=a,this.length=1,this):r.isFunction(a)?void 0!==c.ready?c.ready(a):a(r):r.makeArray(a,this)};G.prototype=r.fn,E=r(d);var H=/^(?:parents|prev(?:Until|All))/,I={children:!0,contents:!0,next:!0,prev:!0};r.fn.extend({has:function(a){var b=r(a,this),c=b.length;return this.filter(function(){for(var a=0;a-1:1===c.nodeType&&r.find.matchesSelector(c,a))){f.push(c);break}return this.pushStack(f.length>1?r.uniqueSort(f):f)},index:function(a){return a?"string"==typeof a?i.call(r(a),this[0]):i.call(this,a.jquery?a[0]:a):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(a,b){return this.pushStack(r.uniqueSort(r.merge(this.get(),r(a,b))))},addBack:function(a){return this.add(null==a?this.prevObject:this.prevObject.filter(a))}});function J(a,b){while((a=a[b])&&1!==a.nodeType);return a}r.each({parent:function(a){var b=a.parentNode;return b&&11!==b.nodeType?b:null},parents:function(a){return y(a,"parentNode")},parentsUntil:function(a,b,c){return y(a,"parentNode",c)},next:function(a){return J(a,"nextSibling")},prev:function(a){return J(a,"previousSibling")},nextAll:function(a){return y(a,"nextSibling")},prevAll:function(a){return y(a,"previousSibling")},nextUntil:function(a,b,c){return y(a,"nextSibling",c)},prevUntil:function(a,b,c){return y(a,"previousSibling",c)},siblings:function(a){return z((a.parentNode||{}).firstChild,a)},children:function(a){return z(a.firstChild)},contents:function(a){return a.contentDocument||r.merge([],a.childNodes)}},function(a,b){r.fn[a]=function(c,d){var e=r.map(this,b,c);return"Until"!==a.slice(-5)&&(d=c),d&&"string"==typeof d&&(e=r.filter(d,e)),this.length>1&&(I[a]||r.uniqueSort(e),H.test(a)&&e.reverse()),this.pushStack(e)}});var K=/\S+/g;function L(a){var b={};return r.each(a.match(K)||[],function(a,c){b[c]=!0}),b}r.Callbacks=function(a){a="string"==typeof a?L(a):r.extend({},a);var b,c,d,e,f=[],g=[],h=-1,i=function(){for(e=a.once,d=b=!0;g.length;h=-1){c=g.shift();while(++h-1)f.splice(c,1),c<=h&&h--}),this},has:function(a){return a?r.inArray(a,f)>-1:f.length>0},empty:function(){return f&&(f=[]),this},disable:function(){return e=g=[],f=c="",this},disabled:function(){return!f},lock:function(){return e=g=[],c||b||(f=c=""),this},locked:function(){return!!e},fireWith:function(a,c){return e||(c=c||[],c=[a,c.slice?c.slice():c],g.push(c),b||i()),this},fire:function(){return j.fireWith(this,arguments),this},fired:function(){return!!d}};return j};function M(a){return a}function N(a){throw a}function O(a,b,c){var d;try{a&&r.isFunction(d=a.promise)?d.call(a).done(b).fail(c):a&&r.isFunction(d=a.then)?d.call(a,b,c):b.call(void 0,a)}catch(a){c.call(void 0,a)}}r.extend({Deferred:function(b){var c=[["notify","progress",r.Callbacks("memory"),r.Callbacks("memory"),2],["resolve","done",r.Callbacks("once memory"),r.Callbacks("once memory"),0,"resolved"],["reject","fail",r.Callbacks("once memory"),r.Callbacks("once memory"),1,"rejected"]],d="pending",e={state:function(){return d},always:function(){return f.done(arguments).fail(arguments),this},"catch":function(a){return e.then(null,a)},pipe:function(){var a=arguments;return r.Deferred(function(b){r.each(c,function(c,d){var e=r.isFunction(a[d[4]])&&a[d[4]];f[d[1]](function(){var a=e&&e.apply(this,arguments);a&&r.isFunction(a.promise)?a.promise().progress(b.notify).done(b.resolve).fail(b.reject):b[d[0]+"With"](this,e?[a]:arguments)})}),a=null}).promise()},then:function(b,d,e){var f=0;function g(b,c,d,e){return function(){var h=this,i=arguments,j=function(){var a,j;if(!(b=f&&(d!==N&&(h=void 0,i=[a]),c.rejectWith(h,i))}};b?k():(r.Deferred.getStackHook&&(k.stackTrace=r.Deferred.getStackHook()),a.setTimeout(k))}}return r.Deferred(function(a){c[0][3].add(g(0,a,r.isFunction(e)?e:M,a.notifyWith)),c[1][3].add(g(0,a,r.isFunction(b)?b:M)),c[2][3].add(g(0,a,r.isFunction(d)?d:N))}).promise()},promise:function(a){return null!=a?r.extend(a,e):e}},f={};return r.each(c,function(a,b){var g=b[2],h=b[5];e[b[1]]=g.add,h&&g.add(function(){d=h},c[3-a][2].disable,c[0][2].lock),g.add(b[3].fire),f[b[0]]=function(){return f[b[0]+"With"](this===f?void 0:this,arguments),this},f[b[0]+"With"]=g.fireWith}),e.promise(f),b&&b.call(f,f),f},when:function(a){var b=arguments.length,c=b,d=Array(c),e=f.call(arguments),g=r.Deferred(),h=function(a){return function(c){d[a]=this,e[a]=arguments.length>1?f.call(arguments):c,--b||g.resolveWith(d,e)}};if(b<=1&&(O(a,g.done(h(c)).resolve,g.reject),"pending"===g.state()||r.isFunction(e[c]&&e[c].then)))return g.then();while(c--)O(e[c],h(c),g.reject);return g.promise()}});var P=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;r.Deferred.exceptionHook=function(b,c){a.console&&a.console.warn&&b&&P.test(b.name)&&a.console.warn("jQuery.Deferred exception: "+b.message,b.stack,c)},r.readyException=function(b){a.setTimeout(function(){throw b})};var Q=r.Deferred();r.fn.ready=function(a){return Q.then(a)["catch"](function(a){r.readyException(a)}),this},r.extend({isReady:!1,readyWait:1,holdReady:function(a){a?r.readyWait++:r.ready(!0)},ready:function(a){(a===!0?--r.readyWait:r.isReady)||(r.isReady=!0,a!==!0&&--r.readyWait>0||Q.resolveWith(d,[r]))}}),r.ready.then=Q.then;function R(){d.removeEventListener("DOMContentLoaded",R),a.removeEventListener("load",R),r.ready()}"complete"===d.readyState||"loading"!==d.readyState&&!d.documentElement.doScroll?a.setTimeout(r.ready):(d.addEventListener("DOMContentLoaded",R),a.addEventListener("load",R));var S=function(a,b,c,d,e,f,g){var h=0,i=a.length,j=null==c;if("object"===r.type(c)){e=!0;for(h in c)S(a,b,h,c[h],!0,f,g)}else if(void 0!==d&&(e=!0, r.isFunction(d)||(g=!0),j&&(g?(b.call(a,d),b=null):(j=b,b=function(a,b,c){return j.call(r(a),c)})),b))for(;h1,null,!0)},removeData:function(a){return this.each(function(){W.remove(this,a)})}}),r.extend({queue:function(a,b,c){var d;if(a)return b=(b||"fx")+"queue",d=V.get(a,b),c&&(!d||r.isArray(c)?d=V.access(a,b,r.makeArray(c)):d.push(c)),d||[]},dequeue:function(a,b){b=b||"fx";var c=r.queue(a,b),d=c.length,e=c.shift(),f=r._queueHooks(a,b),g=function(){r.dequeue(a,b)};"inprogress"===e&&(e=c.shift(),d--),e&&("fx"===b&&c.unshift("inprogress"),delete f.stop,e.call(a,g,f)),!d&&f&&f.empty.fire()},_queueHooks:function(a,b){var c=b+"queueHooks";return V.get(a,c)||V.access(a,c,{empty:r.Callbacks("once memory").add(function(){V.remove(a,[b+"queue",c])})})}}),r.fn.extend({queue:function(a,b){var c=2;return"string"!=typeof a&&(b=a,a="fx",c--),arguments.length\x20\t\r\n\f]+)/i,ja=/^$|\/(?:java|ecma)script/i,ka={option:[1,""],thead:[1,"","
"],col:[2,"","
"],tr:[2,"","
"],td:[3,"","
"],_default:[0,"",""]};ka.optgroup=ka.option,ka.tbody=ka.tfoot=ka.colgroup=ka.caption=ka.thead,ka.th=ka.td;function la(a,b){var c="undefined"!=typeof a.getElementsByTagName?a.getElementsByTagName(b||"*"):"undefined"!=typeof a.querySelectorAll?a.querySelectorAll(b||"*"):[];return void 0===b||b&&r.nodeName(a,b)?r.merge([a],c):c}function ma(a,b){for(var c=0,d=a.length;c-1)e&&e.push(f);else if(j=r.contains(f.ownerDocument,f),g=la(l.appendChild(f),"script"),j&&ma(g),c){k=0;while(f=g[k++])ja.test(f.type||"")&&c.push(f)}return l}!function(){var a=d.createDocumentFragment(),b=a.appendChild(d.createElement("div")),c=d.createElement("input");c.setAttribute("type","radio"),c.setAttribute("checked","checked"),c.setAttribute("name","t"),b.appendChild(c),o.checkClone=b.cloneNode(!0).cloneNode(!0).lastChild.checked,b.innerHTML="",o.noCloneChecked=!!b.cloneNode(!0).lastChild.defaultValue}();var pa=d.documentElement,qa=/^key/,ra=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,sa=/^([^.]*)(?:\.(.+)|)/;function ta(){return!0}function ua(){return!1}function va(){try{return d.activeElement}catch(a){}}function wa(a,b,c,d,e,f){var g,h;if("object"==typeof b){"string"!=typeof c&&(d=d||c,c=void 0);for(h in b)wa(a,h,c,d,b[h],f);return a}if(null==d&&null==e?(e=c,d=c=void 0):null==e&&("string"==typeof c?(e=d,d=void 0):(e=d,d=c,c=void 0)),e===!1)e=ua;else if(!e)return a;return 1===f&&(g=e,e=function(a){return r().off(a),g.apply(this,arguments)},e.guid=g.guid||(g.guid=r.guid++)),a.each(function(){r.event.add(this,b,e,d,c)})}r.event={global:{},add:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=V.get(a);if(q){c.handler&&(f=c,c=f.handler,e=f.selector),e&&r.find.matchesSelector(pa,e),c.guid||(c.guid=r.guid++),(i=q.events)||(i=q.events={}),(g=q.handle)||(g=q.handle=function(b){return"undefined"!=typeof r&&r.event.triggered!==b.type?r.event.dispatch.apply(a,arguments):void 0}),b=(b||"").match(K)||[""],j=b.length;while(j--)h=sa.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n&&(l=r.event.special[n]||{},n=(e?l.delegateType:l.bindType)||n,l=r.event.special[n]||{},k=r.extend({type:n,origType:p,data:d,handler:c,guid:c.guid,selector:e,needsContext:e&&r.expr.match.needsContext.test(e),namespace:o.join(".")},f),(m=i[n])||(m=i[n]=[],m.delegateCount=0,l.setup&&l.setup.call(a,d,o,g)!==!1||a.addEventListener&&a.addEventListener(n,g)),l.add&&(l.add.call(a,k),k.handler.guid||(k.handler.guid=c.guid)),e?m.splice(m.delegateCount++,0,k):m.push(k),r.event.global[n]=!0)}},remove:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=V.hasData(a)&&V.get(a);if(q&&(i=q.events)){b=(b||"").match(K)||[""],j=b.length;while(j--)if(h=sa.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n){l=r.event.special[n]||{},n=(d?l.delegateType:l.bindType)||n,m=i[n]||[],h=h[2]&&new RegExp("(^|\\.)"+o.join("\\.(?:.*\\.|)")+"(\\.|$)"),g=f=m.length;while(f--)k=m[f],!e&&p!==k.origType||c&&c.guid!==k.guid||h&&!h.test(k.namespace)||d&&d!==k.selector&&("**"!==d||!k.selector)||(m.splice(f,1),k.selector&&m.delegateCount--,l.remove&&l.remove.call(a,k));g&&!m.length&&(l.teardown&&l.teardown.call(a,o,q.handle)!==!1||r.removeEvent(a,n,q.handle),delete i[n])}else for(n in i)r.event.remove(a,n+b[j],c,d,!0);r.isEmptyObject(i)&&V.remove(a,"handle events")}},dispatch:function(a){var b=r.event.fix(a),c,d,e,f,g,h,i=new Array(arguments.length),j=(V.get(this,"events")||{})[b.type]||[],k=r.event.special[b.type]||{};for(i[0]=b,c=1;c-1:r.find(e,this,null,[i]).length),d[e]&&d.push(f);d.length&&g.push({elem:i,handlers:d})}return h\x20\t\r\n\f]*)[^>]*)\/>/gi,ya=/\s*$/g;function Ca(a,b){return r.nodeName(a,"table")&&r.nodeName(11!==b.nodeType?b:b.firstChild,"tr")?a.getElementsByTagName("tbody")[0]||a:a}function Da(a){return a.type=(null!==a.getAttribute("type"))+"/"+a.type,a}function Ea(a){var b=Aa.exec(a.type);return b?a.type=b[1]:a.removeAttribute("type"),a}function Fa(a,b){var c,d,e,f,g,h,i,j;if(1===b.nodeType){if(V.hasData(a)&&(f=V.access(a),g=V.set(b,f),j=f.events)){delete g.handle,g.events={};for(e in j)for(c=0,d=j[e].length;c1&&"string"==typeof q&&!o.checkClone&&za.test(q))return a.each(function(e){var f=a.eq(e);s&&(b[0]=q.call(this,e,f.html())),Ha(f,b,c,d)});if(m&&(e=oa(b,a[0].ownerDocument,!1,a,d),f=e.firstChild,1===e.childNodes.length&&(e=f),f||d)){for(h=r.map(la(e,"script"),Da),i=h.length;l")},clone:function(a,b,c){var d,e,f,g,h=a.cloneNode(!0),i=r.contains(a.ownerDocument,a);if(!(o.noCloneChecked||1!==a.nodeType&&11!==a.nodeType||r.isXMLDoc(a)))for(g=la(h),f=la(a),d=0,e=f.length;d0&&ma(g,!i&&la(a,"script")),h},cleanData:function(a){for(var b,c,d,e=r.event.special,f=0;void 0!==(c=a[f]);f++)if(T(c)){if(b=c[V.expando]){if(b.events)for(d in b.events)e[d]?r.event.remove(c,d):r.removeEvent(c,d,b.handle);c[V.expando]=void 0}c[W.expando]&&(c[W.expando]=void 0)}}}),r.fn.extend({detach:function(a){return Ia(this,a,!0)},remove:function(a){return Ia(this,a)},text:function(a){return S(this,function(a){return void 0===a?r.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=a)})},null,a,arguments.length)},append:function(){return Ha(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Ca(this,a);b.appendChild(a)}})},prepend:function(){return Ha(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Ca(this,a);b.insertBefore(a,b.firstChild)}})},before:function(){return Ha(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this)})},after:function(){return Ha(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this.nextSibling)})},empty:function(){for(var a,b=0;null!=(a=this[b]);b++)1===a.nodeType&&(r.cleanData(la(a,!1)),a.textContent="");return this},clone:function(a,b){return a=null!=a&&a,b=null==b?a:b,this.map(function(){return r.clone(this,a,b)})},html:function(a){return S(this,function(a){var b=this[0]||{},c=0,d=this.length;if(void 0===a&&1===b.nodeType)return b.innerHTML;if("string"==typeof a&&!ya.test(a)&&!ka[(ia.exec(a)||["",""])[1].toLowerCase()]){a=r.htmlPrefilter(a);try{for(;c1)}});function Xa(a,b,c,d,e){return new Xa.prototype.init(a,b,c,d,e)}r.Tween=Xa,Xa.prototype={constructor:Xa,init:function(a,b,c,d,e,f){this.elem=a,this.prop=c,this.easing=e||r.easing._default,this.options=b,this.start=this.now=this.cur(),this.end=d,this.unit=f||(r.cssNumber[c]?"":"px")},cur:function(){var a=Xa.propHooks[this.prop];return a&&a.get?a.get(this):Xa.propHooks._default.get(this)},run:function(a){var b,c=Xa.propHooks[this.prop];return this.options.duration?this.pos=b=r.easing[this.easing](a,this.options.duration*a,0,1,this.options.duration):this.pos=b=a,this.now=(this.end-this.start)*b+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),c&&c.set?c.set(this):Xa.propHooks._default.set(this),this}},Xa.prototype.init.prototype=Xa.prototype,Xa.propHooks={_default:{get:function(a){var b;return 1!==a.elem.nodeType||null!=a.elem[a.prop]&&null==a.elem.style[a.prop]?a.elem[a.prop]:(b=r.css(a.elem,a.prop,""),b&&"auto"!==b?b:0)},set:function(a){r.fx.step[a.prop]?r.fx.step[a.prop](a):1!==a.elem.nodeType||null==a.elem.style[r.cssProps[a.prop]]&&!r.cssHooks[a.prop]?a.elem[a.prop]=a.now:r.style(a.elem,a.prop,a.now+a.unit)}}},Xa.propHooks.scrollTop=Xa.propHooks.scrollLeft={set:function(a){a.elem.nodeType&&a.elem.parentNode&&(a.elem[a.prop]=a.now)}},r.easing={linear:function(a){return a},swing:function(a){return.5-Math.cos(a*Math.PI)/2},_default:"swing"},r.fx=Xa.prototype.init,r.fx.step={};var Ya,Za,$a=/^(?:toggle|show|hide)$/,_a=/queueHooks$/;function ab(){Za&&(a.requestAnimationFrame(ab),r.fx.tick())}function bb(){return a.setTimeout(function(){Ya=void 0}),Ya=r.now()}function cb(a,b){var c,d=0,e={height:a};for(b=b?1:0;d<4;d+=2-b)c=aa[d],e["margin"+c]=e["padding"+c]=a;return b&&(e.opacity=e.width=a),e}function db(a,b,c){for(var d,e=(gb.tweeners[b]||[]).concat(gb.tweeners["*"]),f=0,g=e.length;f1)},removeAttr:function(a){return this.each(function(){r.removeAttr(this,a)})}}),r.extend({attr:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return"undefined"==typeof a.getAttribute?r.prop(a,b,c):(1===f&&r.isXMLDoc(a)||(e=r.attrHooks[b.toLowerCase()]||(r.expr.match.bool.test(b)?hb:void 0)),void 0!==c?null===c?void r.removeAttr(a,b):e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:(a.setAttribute(b,c+""),c):e&&"get"in e&&null!==(d=e.get(a,b))?d:(d=r.find.attr(a,b),null==d?void 0:d))},attrHooks:{type:{set:function(a,b){if(!o.radioValue&&"radio"===b&&r.nodeName(a,"input")){var c=a.value;return a.setAttribute("type",b),c&&(a.value=c),b}}}},removeAttr:function(a,b){var c,d=0,e=b&&b.match(K); if(e&&1===a.nodeType)while(c=e[d++])a.removeAttribute(c)}}),hb={set:function(a,b,c){return b===!1?r.removeAttr(a,c):a.setAttribute(c,c),c}},r.each(r.expr.match.bool.source.match(/\w+/g),function(a,b){var c=ib[b]||r.find.attr;ib[b]=function(a,b,d){var e,f,g=b.toLowerCase();return d||(f=ib[g],ib[g]=e,e=null!=c(a,b,d)?g:null,ib[g]=f),e}});var jb=/^(?:input|select|textarea|button)$/i,kb=/^(?:a|area)$/i;r.fn.extend({prop:function(a,b){return S(this,r.prop,a,b,arguments.length>1)},removeProp:function(a){return this.each(function(){delete this[r.propFix[a]||a]})}}),r.extend({prop:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return 1===f&&r.isXMLDoc(a)||(b=r.propFix[b]||b,e=r.propHooks[b]),void 0!==c?e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:a[b]=c:e&&"get"in e&&null!==(d=e.get(a,b))?d:a[b]},propHooks:{tabIndex:{get:function(a){var b=r.find.attr(a,"tabindex");return b?parseInt(b,10):jb.test(a.nodeName)||kb.test(a.nodeName)&&a.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),o.optSelected||(r.propHooks.selected={get:function(a){var b=a.parentNode;return b&&b.parentNode&&b.parentNode.selectedIndex,null},set:function(a){var b=a.parentNode;b&&(b.selectedIndex,b.parentNode&&b.parentNode.selectedIndex)}}),r.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){r.propFix[this.toLowerCase()]=this});var lb=/[\t\r\n\f]/g;function mb(a){return a.getAttribute&&a.getAttribute("class")||""}r.fn.extend({addClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).addClass(a.call(this,b,mb(this)))});if("string"==typeof a&&a){b=a.match(K)||[];while(c=this[i++])if(e=mb(c),d=1===c.nodeType&&(" "+e+" ").replace(lb," ")){g=0;while(f=b[g++])d.indexOf(" "+f+" ")<0&&(d+=f+" ");h=r.trim(d),e!==h&&c.setAttribute("class",h)}}return this},removeClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).removeClass(a.call(this,b,mb(this)))});if(!arguments.length)return this.attr("class","");if("string"==typeof a&&a){b=a.match(K)||[];while(c=this[i++])if(e=mb(c),d=1===c.nodeType&&(" "+e+" ").replace(lb," ")){g=0;while(f=b[g++])while(d.indexOf(" "+f+" ")>-1)d=d.replace(" "+f+" "," ");h=r.trim(d),e!==h&&c.setAttribute("class",h)}}return this},toggleClass:function(a,b){var c=typeof a;return"boolean"==typeof b&&"string"===c?b?this.addClass(a):this.removeClass(a):r.isFunction(a)?this.each(function(c){r(this).toggleClass(a.call(this,c,mb(this),b),b)}):this.each(function(){var b,d,e,f;if("string"===c){d=0,e=r(this),f=a.match(K)||[];while(b=f[d++])e.hasClass(b)?e.removeClass(b):e.addClass(b)}else void 0!==a&&"boolean"!==c||(b=mb(this),b&&V.set(this,"__className__",b),this.setAttribute&&this.setAttribute("class",b||a===!1?"":V.get(this,"__className__")||""))})},hasClass:function(a){var b,c,d=0;b=" "+a+" ";while(c=this[d++])if(1===c.nodeType&&(" "+mb(c)+" ").replace(lb," ").indexOf(b)>-1)return!0;return!1}});var nb=/\r/g,ob=/[\x20\t\r\n\f]+/g;r.fn.extend({val:function(a){var b,c,d,e=this[0];{if(arguments.length)return d=r.isFunction(a),this.each(function(c){var e;1===this.nodeType&&(e=d?a.call(this,c,r(this).val()):a,null==e?e="":"number"==typeof e?e+="":r.isArray(e)&&(e=r.map(e,function(a){return null==a?"":a+""})),b=r.valHooks[this.type]||r.valHooks[this.nodeName.toLowerCase()],b&&"set"in b&&void 0!==b.set(this,e,"value")||(this.value=e))});if(e)return b=r.valHooks[e.type]||r.valHooks[e.nodeName.toLowerCase()],b&&"get"in b&&void 0!==(c=b.get(e,"value"))?c:(c=e.value,"string"==typeof c?c.replace(nb,""):null==c?"":c)}}}),r.extend({valHooks:{option:{get:function(a){var b=r.find.attr(a,"value");return null!=b?b:r.trim(r.text(a)).replace(ob," ")}},select:{get:function(a){for(var b,c,d=a.options,e=a.selectedIndex,f="select-one"===a.type,g=f?null:[],h=f?e+1:d.length,i=e<0?h:f?e:0;i-1)&&(c=!0);return c||(a.selectedIndex=-1),f}}}}),r.each(["radio","checkbox"],function(){r.valHooks[this]={set:function(a,b){if(r.isArray(b))return a.checked=r.inArray(r(a).val(),b)>-1}},o.checkOn||(r.valHooks[this].get=function(a){return null===a.getAttribute("value")?"on":a.value})});var pb=/^(?:focusinfocus|focusoutblur)$/;r.extend(r.event,{trigger:function(b,c,e,f){var g,h,i,j,k,m,n,o=[e||d],p=l.call(b,"type")?b.type:b,q=l.call(b,"namespace")?b.namespace.split("."):[];if(h=i=e=e||d,3!==e.nodeType&&8!==e.nodeType&&!pb.test(p+r.event.triggered)&&(p.indexOf(".")>-1&&(q=p.split("."),p=q.shift(),q.sort()),k=p.indexOf(":")<0&&"on"+p,b=b[r.expando]?b:new r.Event(p,"object"==typeof b&&b),b.isTrigger=f?2:3,b.namespace=q.join("."),b.rnamespace=b.namespace?new RegExp("(^|\\.)"+q.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,b.result=void 0,b.target||(b.target=e),c=null==c?[b]:r.makeArray(c,[b]),n=r.event.special[p]||{},f||!n.trigger||n.trigger.apply(e,c)!==!1)){if(!f&&!n.noBubble&&!r.isWindow(e)){for(j=n.delegateType||p,pb.test(j+p)||(h=h.parentNode);h;h=h.parentNode)o.push(h),i=h;i===(e.ownerDocument||d)&&o.push(i.defaultView||i.parentWindow||a)}g=0;while((h=o[g++])&&!b.isPropagationStopped())b.type=g>1?j:n.bindType||p,m=(V.get(h,"events")||{})[b.type]&&V.get(h,"handle"),m&&m.apply(h,c),m=k&&h[k],m&&m.apply&&T(h)&&(b.result=m.apply(h,c),b.result===!1&&b.preventDefault());return b.type=p,f||b.isDefaultPrevented()||n._default&&n._default.apply(o.pop(),c)!==!1||!T(e)||k&&r.isFunction(e[p])&&!r.isWindow(e)&&(i=e[k],i&&(e[k]=null),r.event.triggered=p,e[p](),r.event.triggered=void 0,i&&(e[k]=i)),b.result}},simulate:function(a,b,c){var d=r.extend(new r.Event,c,{type:a,isSimulated:!0});r.event.trigger(d,null,b)}}),r.fn.extend({trigger:function(a,b){return this.each(function(){r.event.trigger(a,b,this)})},triggerHandler:function(a,b){var c=this[0];if(c)return r.event.trigger(a,b,c,!0)}}),r.each("blur focus focusin focusout resize scroll click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup contextmenu".split(" "),function(a,b){r.fn[b]=function(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),r.fn.extend({hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)}}),o.focusin="onfocusin"in a,o.focusin||r.each({focus:"focusin",blur:"focusout"},function(a,b){var c=function(a){r.event.simulate(b,a.target,r.event.fix(a))};r.event.special[b]={setup:function(){var d=this.ownerDocument||this,e=V.access(d,b);e||d.addEventListener(a,c,!0),V.access(d,b,(e||0)+1)},teardown:function(){var d=this.ownerDocument||this,e=V.access(d,b)-1;e?V.access(d,b,e):(d.removeEventListener(a,c,!0),V.remove(d,b))}}});var qb=a.location,rb=r.now(),sb=/\?/;r.parseXML=function(b){var c;if(!b||"string"!=typeof b)return null;try{c=(new a.DOMParser).parseFromString(b,"text/xml")}catch(d){c=void 0}return c&&!c.getElementsByTagName("parsererror").length||r.error("Invalid XML: "+b),c};var tb=/\[\]$/,ub=/\r?\n/g,vb=/^(?:submit|button|image|reset|file)$/i,wb=/^(?:input|select|textarea|keygen)/i;function xb(a,b,c,d){var e;if(r.isArray(b))r.each(b,function(b,e){c||tb.test(a)?d(a,e):xb(a+"["+("object"==typeof e&&null!=e?b:"")+"]",e,c,d)});else if(c||"object"!==r.type(b))d(a,b);else for(e in b)xb(a+"["+e+"]",b[e],c,d)}r.param=function(a,b){var c,d=[],e=function(a,b){var c=r.isFunction(b)?b():b;d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(null==c?"":c)};if(r.isArray(a)||a.jquery&&!r.isPlainObject(a))r.each(a,function(){e(this.name,this.value)});else for(c in a)xb(c,a[c],b,e);return d.join("&")},r.fn.extend({serialize:function(){return r.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var a=r.prop(this,"elements");return a?r.makeArray(a):this}).filter(function(){var a=this.type;return this.name&&!r(this).is(":disabled")&&wb.test(this.nodeName)&&!vb.test(a)&&(this.checked||!ha.test(a))}).map(function(a,b){var c=r(this).val();return null==c?null:r.isArray(c)?r.map(c,function(a){return{name:b.name,value:a.replace(ub,"\r\n")}}):{name:b.name,value:c.replace(ub,"\r\n")}}).get()}});var yb=/%20/g,zb=/#.*$/,Ab=/([?&])_=[^&]*/,Bb=/^(.*?):[ \t]*([^\r\n]*)$/gm,Cb=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Db=/^(?:GET|HEAD)$/,Eb=/^\/\//,Fb={},Gb={},Hb="*/".concat("*"),Ib=d.createElement("a");Ib.href=qb.href;function Jb(a){return function(b,c){"string"!=typeof b&&(c=b,b="*");var d,e=0,f=b.toLowerCase().match(K)||[];if(r.isFunction(c))while(d=f[e++])"+"===d[0]?(d=d.slice(1)||"*",(a[d]=a[d]||[]).unshift(c)):(a[d]=a[d]||[]).push(c)}}function Kb(a,b,c,d){var e={},f=a===Gb;function g(h){var i;return e[h]=!0,r.each(a[h]||[],function(a,h){var j=h(b,c,d);return"string"!=typeof j||f||e[j]?f?!(i=j):void 0:(b.dataTypes.unshift(j),g(j),!1)}),i}return g(b.dataTypes[0])||!e["*"]&&g("*")}function Lb(a,b){var c,d,e=r.ajaxSettings.flatOptions||{};for(c in b)void 0!==b[c]&&((e[c]?a:d||(d={}))[c]=b[c]);return d&&r.extend(!0,a,d),a}function Mb(a,b,c){var d,e,f,g,h=a.contents,i=a.dataTypes;while("*"===i[0])i.shift(),void 0===d&&(d=a.mimeType||b.getResponseHeader("Content-Type"));if(d)for(e in h)if(h[e]&&h[e].test(d)){i.unshift(e);break}if(i[0]in c)f=i[0];else{for(e in c){if(!i[0]||a.converters[e+" "+i[0]]){f=e;break}g||(g=e)}f=f||g}if(f)return f!==i[0]&&i.unshift(f),c[f]}function Nb(a,b,c,d){var e,f,g,h,i,j={},k=a.dataTypes.slice();if(k[1])for(g in a.converters)j[g.toLowerCase()]=a.converters[g];f=k.shift();while(f)if(a.responseFields[f]&&(c[a.responseFields[f]]=b),!i&&d&&a.dataFilter&&(b=a.dataFilter(b,a.dataType)),i=f,f=k.shift())if("*"===f)f=i;else if("*"!==i&&i!==f){if(g=j[i+" "+f]||j["* "+f],!g)for(e in j)if(h=e.split(" "),h[1]===f&&(g=j[i+" "+h[0]]||j["* "+h[0]])){g===!0?g=j[e]:j[e]!==!0&&(f=h[0],k.unshift(h[1]));break}if(g!==!0)if(g&&a["throws"])b=g(b);else try{b=g(b)}catch(l){return{state:"parsererror",error:g?l:"No conversion from "+i+" to "+f}}}return{state:"success",data:b}}r.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:qb.href,type:"GET",isLocal:Cb.test(qb.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":Hb,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":r.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(a,b){return b?Lb(Lb(a,r.ajaxSettings),b):Lb(r.ajaxSettings,a)},ajaxPrefilter:Jb(Fb),ajaxTransport:Jb(Gb),ajax:function(b,c){"object"==typeof b&&(c=b,b=void 0),c=c||{};var e,f,g,h,i,j,k,l,m,n,o=r.ajaxSetup({},c),p=o.context||o,q=o.context&&(p.nodeType||p.jquery)?r(p):r.event,s=r.Deferred(),t=r.Callbacks("once memory"),u=o.statusCode||{},v={},w={},x="canceled",y={readyState:0,getResponseHeader:function(a){var b;if(k){if(!h){h={};while(b=Bb.exec(g))h[b[1].toLowerCase()]=b[2]}b=h[a.toLowerCase()]}return null==b?null:b},getAllResponseHeaders:function(){return k?g:null},setRequestHeader:function(a,b){return null==k&&(a=w[a.toLowerCase()]=w[a.toLowerCase()]||a,v[a]=b),this},overrideMimeType:function(a){return null==k&&(o.mimeType=a),this},statusCode:function(a){var b;if(a)if(k)y.always(a[y.status]);else for(b in a)u[b]=[u[b],a[b]];return this},abort:function(a){var b=a||x;return e&&e.abort(b),A(0,b),this}};if(s.promise(y),o.url=((b||o.url||qb.href)+"").replace(Eb,qb.protocol+"//"),o.type=c.method||c.type||o.method||o.type,o.dataTypes=(o.dataType||"*").toLowerCase().match(K)||[""],null==o.crossDomain){j=d.createElement("a");try{j.href=o.url,j.href=j.href,o.crossDomain=Ib.protocol+"//"+Ib.host!=j.protocol+"//"+j.host}catch(z){o.crossDomain=!0}}if(o.data&&o.processData&&"string"!=typeof o.data&&(o.data=r.param(o.data,o.traditional)),Kb(Fb,o,c,y),k)return y;l=r.event&&o.global,l&&0===r.active++&&r.event.trigger("ajaxStart"),o.type=o.type.toUpperCase(),o.hasContent=!Db.test(o.type),f=o.url.replace(zb,""),o.hasContent?o.data&&o.processData&&0===(o.contentType||"").indexOf("application/x-www-form-urlencoded")&&(o.data=o.data.replace(yb,"+")):(n=o.url.slice(f.length),o.data&&(f+=(sb.test(f)?"&":"?")+o.data,delete o.data),o.cache===!1&&(f=f.replace(Ab,""),n=(sb.test(f)?"&":"?")+"_="+rb++ +n),o.url=f+n),o.ifModified&&(r.lastModified[f]&&y.setRequestHeader("If-Modified-Since",r.lastModified[f]),r.etag[f]&&y.setRequestHeader("If-None-Match",r.etag[f])),(o.data&&o.hasContent&&o.contentType!==!1||c.contentType)&&y.setRequestHeader("Content-Type",o.contentType),y.setRequestHeader("Accept",o.dataTypes[0]&&o.accepts[o.dataTypes[0]]?o.accepts[o.dataTypes[0]]+("*"!==o.dataTypes[0]?", "+Hb+"; q=0.01":""):o.accepts["*"]);for(m in o.headers)y.setRequestHeader(m,o.headers[m]);if(o.beforeSend&&(o.beforeSend.call(p,y,o)===!1||k))return y.abort();if(x="abort",t.add(o.complete),y.done(o.success),y.fail(o.error),e=Kb(Gb,o,c,y)){if(y.readyState=1,l&&q.trigger("ajaxSend",[y,o]),k)return y;o.async&&o.timeout>0&&(i=a.setTimeout(function(){y.abort("timeout")},o.timeout));try{k=!1,e.send(v,A)}catch(z){if(k)throw z;A(-1,z)}}else A(-1,"No Transport");function A(b,c,d,h){var j,m,n,v,w,x=c;k||(k=!0,i&&a.clearTimeout(i),e=void 0,g=h||"",y.readyState=b>0?4:0,j=b>=200&&b<300||304===b,d&&(v=Mb(o,y,d)),v=Nb(o,v,y,j),j?(o.ifModified&&(w=y.getResponseHeader("Last-Modified"),w&&(r.lastModified[f]=w),w=y.getResponseHeader("etag"),w&&(r.etag[f]=w)),204===b||"HEAD"===o.type?x="nocontent":304===b?x="notmodified":(x=v.state,m=v.data,n=v.error,j=!n)):(n=x,!b&&x||(x="error",b<0&&(b=0))),y.status=b,y.statusText=(c||x)+"",j?s.resolveWith(p,[m,x,y]):s.rejectWith(p,[y,x,n]),y.statusCode(u),u=void 0,l&&q.trigger(j?"ajaxSuccess":"ajaxError",[y,o,j?m:n]),t.fireWith(p,[y,x]),l&&(q.trigger("ajaxComplete",[y,o]),--r.active||r.event.trigger("ajaxStop")))}return y},getJSON:function(a,b,c){return r.get(a,b,c,"json")},getScript:function(a,b){return r.get(a,void 0,b,"script")}}),r.each(["get","post"],function(a,b){r[b]=function(a,c,d,e){return r.isFunction(c)&&(e=e||d,d=c,c=void 0),r.ajax(r.extend({url:a,type:b,dataType:e,data:c,success:d},r.isPlainObject(a)&&a))}}),r._evalUrl=function(a){return r.ajax({url:a,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},r.fn.extend({wrapAll:function(a){var b;return this[0]&&(r.isFunction(a)&&(a=a.call(this[0])),b=r(a,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstElementChild)a=a.firstElementChild;return a}).append(this)),this},wrapInner:function(a){return r.isFunction(a)?this.each(function(b){r(this).wrapInner(a.call(this,b))}):this.each(function(){var b=r(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=r.isFunction(a);return this.each(function(c){r(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(a){return this.parent(a).not("body").each(function(){r(this).replaceWith(this.childNodes)}),this}}),r.expr.pseudos.hidden=function(a){return!r.expr.pseudos.visible(a)},r.expr.pseudos.visible=function(a){return!!(a.offsetWidth||a.offsetHeight||a.getClientRects().length)},r.ajaxSettings.xhr=function(){try{return new a.XMLHttpRequest}catch(b){}};var Ob={0:200,1223:204},Pb=r.ajaxSettings.xhr();o.cors=!!Pb&&"withCredentials"in Pb,o.ajax=Pb=!!Pb,r.ajaxTransport(function(b){var c,d;if(o.cors||Pb&&!b.crossDomain)return{send:function(e,f){var g,h=b.xhr();if(h.open(b.type,b.url,b.async,b.username,b.password),b.xhrFields)for(g in b.xhrFields)h[g]=b.xhrFields[g];b.mimeType&&h.overrideMimeType&&h.overrideMimeType(b.mimeType),b.crossDomain||e["X-Requested-With"]||(e["X-Requested-With"]="XMLHttpRequest");for(g in e)h.setRequestHeader(g,e[g]);c=function(a){return function(){c&&(c=d=h.onload=h.onerror=h.onabort=h.onreadystatechange=null,"abort"===a?h.abort():"error"===a?"number"!=typeof h.status?f(0,"error"):f(h.status,h.statusText):f(Ob[h.status]||h.status,h.statusText,"text"!==(h.responseType||"text")||"string"!=typeof h.responseText?{binary:h.response}:{text:h.responseText},h.getAllResponseHeaders()))}},h.onload=c(),d=h.onerror=c("error"),void 0!==h.onabort?h.onabort=d:h.onreadystatechange=function(){4===h.readyState&&a.setTimeout(function(){c&&d()})},c=c("abort");try{h.send(b.hasContent&&b.data||null)}catch(i){if(c)throw i}},abort:function(){c&&c()}}}),r.ajaxPrefilter(function(a){a.crossDomain&&(a.contents.script=!1)}),r.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(a){return r.globalEval(a),a}}}),r.ajaxPrefilter("script",function(a){void 0===a.cache&&(a.cache=!1),a.crossDomain&&(a.type="GET")}),r.ajaxTransport("script",function(a){if(a.crossDomain){var b,c;return{send:function(e,f){b=r("