SGI machines are often configured such that a distributed memory system appears as a shared memory system. Such SGI machines are often equipped with a MPI library that can take advantage of this specific memory architecture. STAR-CCM+ can take advantage of this on single-host, multiple-CPU machines where the SGI MPI library has been installed. STAR-CCM+ probes for SGI MPI and uses it automatically without you having to enter a separate command, provided you have not used the -machinefile or -on options.
SGI MPI must be installed separately on your machine; it is not distributed with STAR-CCM+.
If SGI MPI is not installed under the default directory (/usr), the environment variable SGI_MPI_HOME must be set such that libmpi.so and libmpi++.so exist in $SGI_MPI_HOME/lib/. Also, the executable mpirun must exist in $SGI_MPI_HOME/bin/. The starccm+ script can show how this works. As only single-host machines are supported, the number of worker processes should be prescribed using the -np rather than the -on or -machinefile options.
SGI MPI uses a concept of cpusets to control which processes are run on which CPUs. To use SGI MPI either you must specify a particular cpuset or STAR-CCM+ will run on the default cpuset. Additional information about cpusets can be found in the SGI Technical Documentation.
Processes will be placed on the CPUs within the cpuset in a round-robin fashion unless the -oncpu option is invoked to specify which CPUs relative to the cpuset are used. The SGI command dplace must be available to use -oncpu. The starccm+ script checks for this.
On some SGI systems you can prescribe the CPUs where the STAR-CCM+ processes will be run using the command line option:
-oncpu <CPU1,CPU2,...,CPUN>
The series shown in brackets may either be a comma-separated list or a range given by a hyphen. The exact format is exactly the same as the SGI dplace -c command. This might be helpful when you wish to run on one shared-memory machine while restricting which CPUs of that machine STAR-CCM+ uses. For example:
starccm+ -np 10 -oncpu 0-12
To achieve good performance you should ensure that the cpuset has enough CPUs allocated to it to permit each worker process and the controller process to be placed on their own CPUs. Running the controller process on the same CPU as a worker process will degrade performance.
Polling is off by default. This driver is able to support polling to enhance performance by adding the option:
-mpidriver sgi:-polling
To deactivate it again, use the following line command:
-mpidriver sgi
You should be careful not to over-subscribe a CPU with processes which will result in a performance degradation.
When starting SGI MPI, you may encounter a warning like this:
"MPI: This executable was not linked with MPT version 1.6.1 or later. Memory mapping features such as single copy transfers, MPI one-sided communication and shmem functions are not available unless you relink your executable with the current MPT library."
This is a known problem with SGI and can be ignored.