I_MPI_ADJUST Family Environment Variables

I_MPI_ADJUST_<opname>

Control collective operation algorithm selection.

Syntax

I_MPI_ADJUST_<opname>="<algid>[:<conditions>][;<algid>:<conditions>[...]]"

Arguments

<algid>

Algorithm identifier

>= 0

The default value of zero selects the reasonable settings

 

<conditions>

A comma separated list of conditions. An empty list selects all message sizes and process combinations

<l>

Messages of size <l>

<l>-<m>

Messages of size from <l> to <m>, inclusive

<l>@<p>

Messages of size <l> and number of processes <p>

<l>-<m>@<p>-<q>

Messages of size from <l> to <m> and number of processes from <p> to <q>, inclusive

Description

Set this environment variable to select the desired algorithm(s) for the collective operation <opname> under particular conditions. Each collective operation has its own environment variable and algorithms.

Environment Variables, Collective Operations, and Algorithms

Collective Operation

Environment Variable

Algorithms

MPI_Allgather

I_MPI_ADJUST_ALLGATHER
  1. Recursive doubling
  2. Bruck's
  3. Ring
  4. Topology aware Gatherv + Bcast
  5. Knomial

MPI_Allgatherv

I_MPI_ADJUST_ALLGATHERV
  1. Recursive doubling
  2. Bruck's
  3. Ring
  4. Topology aware Gatherv + Bcast

MPI_Allreduce

I_MPI_ADJUST_ALLREDUCE
  1. Recursive doubling
  2. Rabenseifner's
  3. Reduce + Bcast
  4. Topology aware Reduce + Bcast
  5. Binomial gather + scatter
  6. Topology aware binominal gather + scatter
  7. Shumilin's ring
  8. Ring
  9. Knomial
  10. Topology aware SHM-based flat
  11. Topology aware SHM-based Knomial
  12. Topology aware SHM-based Knary

MPI_Alltoall

I_MPI_ADJUST_ALLTOALL
  1. Bruck's
  2. Isend/Irecv + waitall
  3. Pair wise exchange
  4. Plum's

MPI_Alltoallv

I_MPI_ADJUST_ALLTOALLV
  1. Isend/Irecv + waitall
  2. Plum's

MPI_Alltoallw

I_MPI_ADJUST_ALLTOALLW

Isend/Irecv + waitall

MPI_Barrier

I_MPI_ADJUST_BARRIER
  1. Dissemination
  2. Recursive doubling
  3. Topology aware dissemination
  4. Topology aware recursive doubling
  5. Binominal gather + scatter
  6. Topology aware binominal gather + scatter
  7. Topology aware SHM-based flat
  8. Topology aware SHM-based Knomial
  9. Topology aware SHM-based Knary

MPI_Bcast

I_MPI_ADJUST_BCAST
  1. Binomial
  2. Recursive doubling
  3. Ring
  4. Topology aware binomial
  5. Topology aware recursive doubling
  6. Topology aware ring
  7. Shumilin's
  8. Knomial
  9. Topology aware SHM-based flat
  10. Topology aware SHM-based Knomial
  11. Topology aware SHM-based Knary

MPI_Exscan

I_MPI_ADJUST_EXSCAN
  1. Partial results gathering
  2. Partial results gathering regarding layout of processes

MPI_Gather

I_MPI_ADJUST_GATHER
  1. Binomial
  2. Topology aware binomial
  3. Shumilin's
  4. Binomial with segmentation

MPI_Gatherv

I_MPI_ADJUST_GATHERV
  1. Linear
  2. Topology aware linear
  3. Knomial

MPI_Reduce_scatter

I_MPI_ADJUST_REDUCE_SCATTER
  1. Recursive halving
  2. Pair wise exchange
  3. Recursive doubling
  4. Reduce + Scatterv
  5. Topology aware Reduce + Scatterv

MPI_Reduce

I_MPI_ADJUST_REDUCE
  1. Shumilin's
  2. Binomial
  3. Topology aware Shumilin's
  4. Topology aware binomial
  5. Rabenseifner's
  6. Topology aware Rabenseifner's
  7. Knomial
  8. Topology aware SHM-based flat
  9. Topology aware SHM-based Knomial
  10. Topology aware SHM-based Knary
  11. Topology aware SHM-based binomial

MPI_Scan

I_MPI_ADJUST_SCAN
  1. Partial results gathering
  2. Topology aware partial results gathering

MPI_Scatter

I_MPI_ADJUST_SCATTER
  1. Binomial
  2. Topology aware binomial
  3. Shumilin's

MPI_Scatterv

I_MPI_ADJUST_SCATTERV
  1. Linear
  2. Topology aware linear

MPI_Iallgather

I_MPI_ADJUST_IALLGATHER
  1. Recursive doubling
  2. Bruck’s
  3. Ring

I_MPI_ADJUST_IALLGATHER_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLGATHER_NETWORK
  1. Recursive doubling
  2. Bruck’s
  3. Ring

I_MPI_ADJUST_IALLGATHER_NODE
  1. Recursive doubling
  2. Bruck’s
  3. Ring

MPI_Iallgatherv

I_MPI_ADJUST_IALLGATHERV
  1. Recursive doubling
  2. Bruck’s
  3. Ring

I_MPI_ADJUST_IALLGATHERV_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLGATHERV_NETWORK
  1. Recursive doubling
  2. Bruck’s
  3. Ring

I_MPI_ADJUST_IALLGATHERV_NODE
  1. Recursive doubling
  2. Bruck’s
  3. Ring

MPI_Iallreduce

I_MPI_ADJUST_IALLREDUCE

 
  1. Recursive doubling
  2. Rabenseifner’s
  3. Reduce + Bcast
  4. Ring
  5. Knomial
  6. Binomial
  7. Reduce scatter allgather
  8. SMP
  9. Nreduce

I_MPI_ADJUST_IALLREDUCE_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLREDUCE_NETWORK
  1. Naïve (Reduce + Bcast)
  2. Recursive doubling
  3. Reduce scatter allgather
  4. Nreduce
  5. Rabenseifner's
  6. Ring
  7. SMP
  8. Knomial

I_MPI_ADJUST_IALLREDUCE_NODE
  1. Naïve
  2. Recursive doubling
  3. Reduce scatter allgather
  4. Nreduce
  5. Rabenseifner's
  6. Ring
  7. SMP
  8. Knomial

MPI_Ialltoall

I_MPI_ADJUST_IALLTOALL
  1. Bruck’s
  2. Isend/Irecv + Waitall
  3. Pairwise exchange

I_MPI_ADJUST_IALLTOALL_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLTOALL_NETWORK
  1. Pairwise exchange
  2. Bruck’s
  3. Inplace
  4. Isend/Irecv + Waitall

I_MPI_ADJUST_IALLTOALL_NODE
  1. Pairwise exchange
  2. Bruck’s
  3. Inplace
  4. Isend/Irecv + Waitall

MPI_Ialltoallv

I_MPI_ADJUST_IALLTOALLV

Isend/Irecv + Waitall

I_MPI_ADJUST_IALLTOALLV_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLTOALLV_NETWORK
  1. Isend/Irecv + Waitall
  2. Inplace

I_MPI_ADJUST_IALLTOALLV_NODE
  1. Isend/Irecv + Waitall
  2. Inplace

MPI_Ialltoallw

I_MPI_ADJUST_IALLTOALLW

Isend/Irecv + Waitall

I_MPI_ADJUST_IALLTOALLW_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IALLTOALLW_NETWORK
  1. Isend/Irecv + Waitall
  2. Inplace

I_MPI_ADJUST_IALLTOALLW_NODE
  1. Isend/Irecv + Waitall
  2. Inplace

MPI_Ibarrier

I_MPI_ADJUST_IBARRIER

Dissemination

I_MPI_ADJUST_IBARRIER_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IBARRIER_NETWORK

Dissemination

I_MPI_ADJUST_IBARRIER_NODE

Dissemination

MPI_Ibcast

I_MPI_ADJUST_IBCAST
  1. Binomial
  2. Recursive doubling
  3. Ring
  4. Knomial
  5. SMP
  6. Tree knominal
  7. Tree kary

I_MPI_ADJUST_IBCAST_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IBCAST_NETWORK
  1. Binomial
  2. Scatter recursive doubling allgather
  3. Ring allgather
  4. SMP
  5. Tree knomial
  6. Tree kary
  7. Knomial

I_MPI_ADJUST_IBCAST_NODE
  1. Binomial
  2. Scatter recursive doubling allgather
  3. Ring allgather
  4. SMP
  5. Tree knomial
  6. Tree kary
  7. Knomial

MPI_Iexscan

I_MPI_ADJUST_IEXSCAN

Recursive doubling

I_MPI_ADJUST_IEXSCAN_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IEXSCAN_NETWORK

Recursive doubling

I_MPI_ADJUST_IEXSCAN_NODE

Recursive doubling

MPI_Igather

I_MPI_ADJUST_IGATHER
  1. Binomial
  2. Knomial

I_MPI_ADJUST_IGATHER_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IGATHER_NETWORK
  1. Binomial
  2. Knomial

I_MPI_ADJUST_IGATHER_NODE
  1. Binomial
  2. Knomial

MPI_Igatherv

I_MPI_ADJUST_IGATHERV
  1. Linear
  2. Linear ssend

I_MPI_ADJUST_IGATHERV_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IGATHERV_NETWORK
  1. Linear
  2. Linear ssend

I_MPI_ADJUST_IGATHERV_NODE
  1. Linear
  2. Linear ssend

MPI_Ireduce_scatter

I_MPI_ADJUST_IREDUCE_SCATTER
  1. Recursive halving
  2. Pairwise
  3. Recursive doubling

I_MPI_ADJUST_IREDUCE_SCATTER_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IREDUCE_SCATTER_NETWORK
  1. Noncommutative
  2. Pairwise
  3. Recursive doubling
  4. Recursive halving

I_MPI_ADJUST_IREDUCE_SCATTER_NODE
  1. Noncommutative
  2. Pairwise
  3. Recursive doubling
  4. Recursive halving

MPI_Ireduce_scatter_block

I_MPI_ADJUST_IREDUCE_SCATTER_BLOCK
  1. Recursive halving
  2. Pairwise
  3. Recursive doubling

I_MPI_ADJUST_IREDUCE_SCATTER_BLOCK_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IREDUCE_SCATTER_BLOCK_NETWORK
  1. Noncommutative
  2. Pairwise
  3. Recursive doubling
  4. Recursive halving

I_MPI_ADJUST_IREDUCE_SCATTER_BLOCK_NODE
  1. Noncommutative
  2. Pairwise
  3. Recursive doubling
  4. Recursive halving

MPI_Ireduce

I_MPI_ADJUST_IREDUCE
  1. Rabenseifner’s
  2. Binomial
  3. Knomial
  4. SMP

I_MPI_ADJUST_IREDUCE_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_IREDUCE_NETWORK
  1. Binomial
  2. Rabenseifner’s
  3. SMP
  4. Knomial

I_MPI_ADJUST_IREDUCE_NODE
  1. Binomial
  2. Rabenseifner’s
  3. SMP
  4. Knomial

MPI_Iscan

I_MPI_ADJUST_ISCAN
  1. Recursive Doubling
  2. SMP

I_MPI_ADJUST_ISCAN_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_ISCAN_NETWORK
  1. Recursive Doubling
  2. SMP

I_MPI_ADJUST_ISCAN_NODE
  1. Recursive Doubling
  2. SMP

MPI_Iscatter

I_MPI_ADJUST_ISCATTER
  1. Binomial
  2. Knomial

I_MPI_ADJUST_ISCATTER_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_ISCATTER_NETWORK
  1. Binomial
  2. Knomial

I_MPI_ADJUST_ISCATTER_NODE
  1. Binomial
  2. Knomial

MPI_Iscatterv

I_MPI_ADJUST_ISCATTERV

Linear

I_MPI_ADJUST_ISCATTERV_COMPOSITION
  1. Composition alpha
  2. Composition beta

I_MPI_ADJUST_ISCATTERV_NETWORK

Linear

I_MPI_ADJUST_ISCATTERV_NODE

Linear

The message size calculation rules for the collective operations are described in the table. In the following table, "n/a" means that the corresponding interval <l>-<m> should be omitted.

Message Collective Functions

Collective Function

Message Size Formula

MPI_Allgather

recv_count*recv_type_size

MPI_Allgatherv

total_recv_count*recv_type_size

MPI_Allreduce

count*type_size

MPI_Alltoall

send_count*send_type_size

MPI_Alltoallv

n/a

MPI_Alltoallw

n/a

MPI_Barrier

n/a

MPI_Bcast

count*type_size

MPI_Exscan

count*type_size

MPI_Gather

recv_count*recv_type_size if MPI_IN_PLACE is used, otherwise send_count*send_type_size

MPI_Gatherv

n/a

MPI_Reduce_scatter

total_recv_count*type_size

MPI_Reduce

count*type_size

MPI_Scan

count*type_size

MPI_Scatter

send_count*send_type_size if MPI_IN_PLACE is used, otherwise recv_count*recv_type_size

MPI_Scatterv

n/a

Examples

Use the following settings to select the second algorithm for MPI_Reduce operation:
I_MPI_ADJUST_REDUCE=2

Use the following settings to define the algorithms for MPI_Reduce_scatter operation:
I_MPI_ADJUST_REDUCE_SCATTER="4:0-100,5001-10000;1:101-3200,2:3201-5000;3"

In this case. algorithm 4 is used for the message sizes between 0 and 100 bytes and from 5001 and 10000 bytes, algorithm 1 is used for the message sizes between 101 and 3200 bytes, algorithm 2 is used for the message sizes between 3201 and 5000 bytes, and algorithm 3 is used for all other messages.

I_MPI_ADJUST_REDUCE_SEGMENT

Syntax

I_MPI_ADJUST_REDUCE_SEGMENT=<block_size>|<algid>:<block_size>[,<algid>:<block_size>[...]]

Arguments

<algid>

Algorithm identifier

1

Shumilin’s algorithm

3

Topology aware Shumilin’s algorithm

<block_size>

Size of a message segment in bytes

> 0

The default value is 14000

Description

Set an internal block size to control MPI_Reduce message segmentation for the specified algorithm. If the <algid> value is not set, the <block_size> value is applied for all the algorithms, where it is relevant.

Note

This environment variable is relevant for Shumilin’s and topology aware Shumilin’s algorithms only (algorithm N1 and algorithm N3 correspondingly).

I_MPI_ADJUST_BCAST_SEGMENT

Syntax

I_MPI_ADJUST_BCAST_SEGMENT=<block_size>|<algid>:<block_size>[,<algid>:<block_size>[...]]

Arguments

<algid>

Algorithm identifier

1

Binomial

4

Topology aware binomial

7

Shumilin's

8

Knomial

<block_size>

Size of a message segment in bytes

> 0

The default value is 12288

Description

Set an internal block size to control MPI_Bcast message segmentation for the specified algorithm. If the <algid> value is not set, the <block_size> value is applied for all the algorithms, where it is relevant.

Note

This environment variable is relevant only for Binomial, Topology-aware binomial, Shumilin’s and Knomial algorithms.

I_MPI_ADJUST_ALLGATHER_KN_RADIX

Syntax

I_MPI_ADJUST_ALLGATHER_KN_RADIX=<radix>

Arguments

<radix>

An integer that specifies a radix used by the Knomial MPI_Allgather algorithm to build a knomial communication tree

> 1

The default value is 2

Description

Set this environment variable together with I_MPI_ADJUST_ALLGATHER=5 to select the knomial tree radix for the corresponding MPI_Allgather algorithm.

I_MPI_ADJUST_BCAST_KN_RADIX

Syntax

I_MPI_ADJUST_BCAST_KN_RADIX=<radix>

Arguments

<radix>

An integer that specifies a radix used by the Knomial MPI_Bcast algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_BCAST=8 to select the knomial tree radix for the corresponding MPI_Bcast algorithm.

I_MPI_ADJUST_ALLREDUCE_KN_RADIX

Syntax

I_MPI_ADJUST_ALLREDUCE_KN_RADIX=<radix>

Arguments

<radix>

An integer that specifies a radix used by the Knomial MPI_Allreduce algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_ALLREDUCE=9 to select the knomial tree radix for the corresponding MPI_Allreduce algorithm.

I_MPI_ADJUST_REDUCE_KN_RADIX

Syntax

I_MPI_ADJUST_REDUCE_KN_RADIX=<radix>

Arguments

<radix>

An integer that specifies a radix used by the Knomial MPI_Reduce algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_REDUCE=7 to select the knomial tree radix for the corresponding MPI_Reduce algorithm.

I_MPI_ADJUST_GATHERV_KN_RADIX

Syntax

I_MPI_ADJUST_GATHERV_KN_RADIX=<radix>

Arguments

<radix>

An integer that specifies a radix used by the Knomial MPI_Gatherv algorithm to build a knomial communication tree

> 1

The default value is 2

Description

Set this environment variable together with I_MPI_ADJUST_GATHERV=3 to select the knomial tree radix for the corresponding MPI_Gatherv algorithm.

I_MPI_ADJUST_IALLREDUCE_KN_RADIX

Syntax

I_MPI_ADJUST_IALLREDUCE_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial MPI_Iallreduce algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_IALLREDUCE=5 to select the knomial tree radix for the corresponding MPI_Iallreduce algorithm.

I_MPI_ADJUST_IBCAST_KN_RADIX

Syntax

I_MPI_ADJUST_IBCAST_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial MPI_Ibcast algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_IBCAST=4 to select the knomial tree radix for the corresponding MPI_Ibcast algorithm.

I_MPI_ADJUST_IREDUCE_KN_RADIX

Syntax

I_MPI_ADJUST_IREDUCE_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial MPI_Ireduce algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_IREDUCE=3 to select the knomial tree radix for the corresponding MPI_Ireduce algorithm.

I_MPI_ADJUST_IGATHER_KN_RADIX

Syntax

I_MPI_ADJUST_IGATHER_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial MPI_Igather algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_IGATHER=2 to select the knomial tree radix for the corresponding MPI_Igather algorithm.

I_MPI_ADJUST_ISCATTER_KN_RADIX

Syntax

I_MPI_ADJUST_ISCATTER_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial MPI_Iscatter algorithm to build a knomial communication tree

> 1

The default value is 4

Description

Set this environment variable together with I_MPI_ADJUST_ISCATTER=2 to select the knomial tree radix for the corresponding MPI_Iscatter algorithm.

I_MPI_ADJUST_<COLLECTIVE>_SHM_KN_RADIX

Syntax

I_MPI_ADJUST_<COLLECTIVE>_SHM_KN_RADIX=<radix>

Arguments

<radix> 

An integer that specifies a radix used by the Knomial or Knary SHM-based algorithm to build a knomial or knary communication tree

> 0
  • If you specify the environment variables I_MPI_ADJUST_BCAST_SHM_KN_RADIX and I_MPI_ADJUST_BARRIER_SHM_KN_RADIX, the default value is 3
  • If you specify the environment variables I_MPI_ADJUST_REDUCE_SHM_KN_RADIX and I_MPI_ADJUST_ALLREDUCE_SHM_KN_RADIX, the default value is 4

Description

This environment variable includes the following variables:

Set this environment variable to select the knomial or knary tree radix for the corresponding tree SHM-based algorithms. When you build a knomial communication tree, the specified value is used as the power for 2 to generate resulting radix (2^<radix>). When you build a knary communication tree, the specified value is used for the radix.

I_MPI_COLL_INTRANODE

Syntax

I_MPI_COLL_INTRANODE=<mode>

Arguments

<mode> 

Intranode collectives type

pt2pt

Use only point-to-point communication-based collectives

shm

Enables shared memory collectives. This is the default value

Description

Set this environment variable to switch intranode communication type for collective operations. If there is large set of communicators, you can switch off the SHM-collectives to avoid memory overconsumption.

I_MPI_COLL_INTRANODE_SHM_THRESHOLD

Syntax

I_MPI_COLL_INTRANODE_SHM_THRESHOLD=<nbytes>

Arguments

<nbytes> 

Define the maximal data block size processed by shared memory collectives.

> 0

Use the specified size. The default value is 16384 bytes.

Description

Set this environment variable to define the size of shared memory area available for each rank for data placement. Messages greater than this value will not be processed by SHM-based collective operation, but will be processed by point-to-point based collective operation. The value must be a multiple of 4096.

I_MPI_ADJUST_GATHER_SEGMENT

Syntax

I_MPI_ADJUST_GATHER_SEGMENT=<block_size>

Arguments

<block_size>

Size of a message segment in bytes.

> 0

Use the specified size. The default value is 16384 bytes.

Description

Set an internal block size to control the MPI_Gather message segmentation for the binomial algorithm with segmentation.