Ошибка intelmpi 2021.1 на RHEL 8 с ошибкой целочисленного деления на ноль ⇐ Linux

[Anonymous]

После обновления вычислительных узлов кластера HPC с CentOS 7 до дистрибутива Linux RHEL 8 (KeyarchOS 5.8) я получаю сообщение о том, что некоторое программное обеспечение, скомпилированное с помощью Intel OneAPI 2021.1, не работает с mpirun. Типичная ошибка следующая:
[cu345:1485183:0:1485183] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485184:0:1485184] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485185:0:1485185] Caught signal 8 (Floating point exception: integer divide by zero)
[cu345:1485186:0:1485186] Caught signal 8 (Floating point exception: integer divide by zero)
==== backtrace (tid:1485126) ====
0 0x0000000000012ce0 __funlockfile() :0
1 0x0000000000b696ed next_random() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_types.h:1809
2 0x0000000000b696ed impi_bcast_intra_huge() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_bcast.h:667
3 0x0000000000b6630d impi_bcast_intra_heap() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/posix/eager/include/intel_transport_bcast.h:798
4 0x000000000018ef6d MPIDI_POSIX_mpi_bcast() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/src/../src/../posix/intel/posix_coll.h:124
5 0x000000000017335e MPIDI_SHM_mpi_bcast() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/shm/src/../src/shm_coll.h:39
6 0x000000000017335e MPIDI_Bcast_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:303
7 0x000000000017335e MPID_Bcast_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:1726
8 0x000000000017335e MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3356
9 0x0000000000153bee MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129
10 0x000000000021c02d MPID_Bcast() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:51
11 0x00000000001386e9 PMPI_Bcast() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/bcast/bcast.c:416

Когда я пытаюсь воспроизвести проблему, я обнаруживаю, что вероятность прохождения теста составляет около 20%. При работе на CentOS 7 все проходит на 100 %, и я понятия не имею, что произойдет.
Чтобы воспроизвести эту проблему, можно скомпилировать собственный LAMMPS
module load intel/2021.1
module load dev/cmake/3.26.3
module load gcc/9.3

# find the ver in https://download.lammps.org/tars/index.html
# find the ver in https://download.lammps.org/tars/index.html
wget -c https://download.lammps.org/tars/lammps ... 022.tar.gz
tar -zxvf lammps-23Jun2022.tar.gz
cd lammps-23Jun2022
mkdir -p build
cd build
cmake ../cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-D BUILD_MPI=yes -D BUILD_OMP=yes -D LAMMPS_MACHINE=intel_cpu_intelmpi \
-D CMAKE_INSTALL_PREFIX=/public/home/jxzhu/apps/lammps/install/lammps-23Jun2022 \
-D CMAKE_INSTALL_LIBDIR=lib \
-D CMAKE_INSTALL_FULL_LIBDIR=/public/home/jxzhu/apps/lammps/install/lammps-23Jun2022/lib \
-C ../cmake/presets/most.cmake -C ../cmake/presets/nolib.cmake \
-D BUILD_SHARED_LIBS=yes
make -j 32
make install

А затем используйте следующий пакетный скрипт для запуска
#!/bin/bash -l
#SBATCH -p cpu
#SBATCH --qos normal

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=64
#SBATCH -J std

module load lammps/2024.8.29
module list

export I_MPI_DEBUG=6

mpirun lmp_mpi -i input.lammps && echo "mpi_success $SLURM_NODELIST" || echo "mpi_fail $SLURM_NODELIST"


Подробнее здесь: https://stackoverflow.com/questions/791 ... de-by-zero