Testing decomp: ./ne30_F_case_48602x72_512p.dat pio_readdof start pio_readdof end, read time = 0.50187202600000003 [chr-0500:685345:0:685345] cma_ep.c:97 process_vm_readv(pid=685344 length=524288) returned -1: No such process [chr-0499:744006:0:744006] cma_ep.c:97 process_vm_readv(pid=744005 length=524288) returned -1: No such process [chr-0501:718190:0:718190] cma_ep.c:97 process_vm_readv(pid=718189 length=524288) returned -1: No such process [chr-0496:956720:0:956720] cma_ep.c:97 process_vm_readv(pid=956719 length=524288) returned -1: No such process srun: error: chr-0499: task 368: Killed srun: error: chr-0501: task 501: Killed [chr-0497:831816:0:831816] cma_ep.c:97 process_vm_readv(pid=831815 length=524288) returned -1: No such process srun: error: chr-0497: task 216: Killed srun: error: chr-0498: task 299: Killed [chr-0494:1632087:0:1632087] cma_ep.c:97 process_vm_readv(pid=1632086 length=524288) returned -1: No such process ==== backtrace (tid: 685345) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 718190) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 831816) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? srun: error: chr-0497: task 214: Killed srun: error: chr-0494: tasks 0,12: Killed [chr-0500:685350:0:685350] cma_ep.c:97 process_vm_readv(pid=685349 length=524288) returned -1: Bad address #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? [chr-0499:743995:0:743995] cma_ep.c:97 process_vm_readv(pid=743994 length=524288) returned -1: No such process [chr-0495:1436499:0:1436499] cma_ep.c:97 process_vm_readv(pid=1436498 length=524288) returned -1: No such process #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? ==== backtrace (tid: 744006) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 ================================= #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 Program received signal SIGABRT: Process abort signal. Backtrace for this error: #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? srun: error: chr-0499: tasks 357,365: Killed srun: error: chr-0498: task 296: Killed ==== backtrace (tid:1632087) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x000000000003e6fc opal_progress() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/opal/runtime/opal_progress.c:231 6 0x000000000008bacd ompi_request_wait_completion() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/../ompi/request/request.h:440 7 0x000000000008bacd ompi_request_default_wait() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/request/req_wait.c:42 8 0x00000000000d7c4a ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:62 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: srun: error: chr-0495: tasks 91,105,107: Killed srun: error: chr-0494: task 9: Killed srun: error: chr-0501: tasks 496,504: Killed [chr-0500:685353:0:685353] cma_ep.c:97 process_vm_readv(pid=685352 length=524288) returned -1: No such process #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? ==== backtrace (tid: 743995) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 ==== backtrace (tid: 685350) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #28 0x155554c3a279 in ??? #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #28 0x155554c3a279 in ??? #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #27 0x155554c3a279 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #28 0x46b732 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? ==== backtrace (tid: 685353) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? srun: error: chr-0499: task 353: Killed srun: error: chr-0498: task 291: Killed srun: error: chr-0495: task 87: Killed srun: error: chr-0496: tasks 128,132,136: Killed ==== backtrace (tid:1436499) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 ================================= #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 Program received signal SIGABRT: Process abort signal. Backtrace for this error: #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? [chr-0494:1632100:0:1632100] cma_ep.c:97 process_vm_readv(pid=1632099 length=524288) returned -1: Bad address #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x1555515826fb in opal_progress at runtime/opal_progress.c:231 #12 0x155553a38acc in ompi_request_wait_completion at ../ompi/request/request.h:440 #13 0x155553a38acc in ompi_request_default_wait at request/req_wait.c:42 #14 0x155553a84c49 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:62 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? [chr-0496:956726:0:956726] cma_ep.c:97 process_vm_readv(pid=956725 length=524288) returned -1: Bad address ==== backtrace (tid: 956720) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? srun: error: chr-0501: task 491: Killed srun: error: chr-0501: task 502: Aborted (core dumped) srun: error: chr-0499: task 383: Killed srun: error: chr-0497: task 212: Killed srun: error: chr-0497: task 217: Aborted (core dumped) [chr-0496:956735:0:956735] cma_ep.c:97 process_vm_readv(pid=956734 length=524288) returned -1: No such process ==== backtrace (tid:1632100) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 956726) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x000000000003e6fc opal_progress() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/opal/runtime/opal_progress.c:231 6 0x000000000008bacd ompi_request_wait_completion() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/../ompi/request/request.h:440 7 0x000000000008bacd ompi_request_default_wait() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/request/req_wait.c:42 8 0x00000000000d7c4a ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:62 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x1555515826fb in opal_progress at runtime/opal_progress.c:231 #12 0x155553a38acc in ompi_request_wait_completion at ../ompi/request/request.h:440 #13 0x155553a38acc in ompi_request_default_wait at request/req_wait.c:42 #14 0x155553a84c49 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:62 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? srun: error: chr-0499: tasks 358,369: Aborted (core dumped) srun: error: chr-0500: tasks 390,401,404,409,412: Killed srun: error: chr-0500: tasks 405,410,413: Aborted (core dumped) srun: error: chr-0494: tasks 11,25: Killed srun: error: chr-0494: task 13: Aborted (core dumped) srun: error: chr-0494: task 26: Aborted (core dumped) [chr-0496:956742:0:956742] cma_ep.c:97 process_vm_readv(pid=956741 length=524288) returned -1: No such process ==== backtrace (tid: 956735) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: srun: error: chr-0495: task 80: Killed srun: error: chr-0495: task 92: Aborted (core dumped) #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 [chr-0498:787616:0:787616] cma_ep.c:97 process_vm_readv(pid=787615 length=524288) returned -1: No such process #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 ==== backtrace (tid: 787616) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 ================================= #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 Program received signal SIGABRT: Process abort signal. Backtrace for this error: #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #28 0x155554c3a279 in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? [chr-0494:1632120:0:1632120] cma_ep.c:97 process_vm_readv(pid=1632119 length=524288) returned -1: No such process srun: error: chr-0498: tasks 285,288,312: Killed ==== backtrace (tid: 956742) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: [chr-0496:956748:0:956748] cma_ep.c:97 process_vm_readv(pid=956747 length=524288) returned -1: No such process [chr-0497:831832:0:831832] cma_ep.c:97 process_vm_readv(pid=831831 length=524288) returned -1: No such process [chr-0500:685369:0:685369] cma_ep.c:97 process_vm_readv(pid=685368 length=524288) returned -1: No such process #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? srun: error: chr-0499: task 342: Killed ==== backtrace (tid:1632120) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? [chr-0501:718145:0:718145] cma_ep.c:97 process_vm_readv(pid=718144 length=524288) returned -1: No such process ==== backtrace (tid: 956748) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? srun: error: chr-0494: task 45: Killed ==== backtrace (tid: 685369) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 ================================= #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 Program received signal SIGABRT: Process abort signal. Backtrace for this error: #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #11 0x1555512dc1d9 in ??? #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? srun: error: chr-0500: task 428: Killed srun: error: chr-0496: tasks 133,139,148: Aborted (core dumped) srun: error: chr-0496: tasks 138,147,154,160: Killed [chr-0494:1632126:0:1632126] cma_ep.c:97 process_vm_readv(pid=1632125 length=524288) returned -1: No such process srun: error: chr-0498: task 313: Killed ==== backtrace (tid:1632126) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 831832) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? [chr-0497:831841:0:831841] cma_ep.c:97 process_vm_readv(pid=831840 length=524288) returned -1: No such process #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? [chr-0501:718158:0:718158] cma_ep.c:97 process_vm_readv(pid=718157 length=524288) returned -1: Bad address ==== backtrace (tid: 831841) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? ==== backtrace (tid: 718145) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x0000000000053fab ucs_callbackq_get_id() ???:0 5 0x00000000000351da ucp_worker_progress() ???:0 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 8 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 9 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 10 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 11 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 12 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 13 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 14 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 15 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 16 0x000000000016fbc2 ncmpio_read_write() ???:0 17 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 18 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 19 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 20 0x00000000001661a4 req_commit() ncmpio_wait.c:0 21 0x0000000000166a0c ncmpio_wait() ???:0 22 0x00000000000b727a ncmpi_wait_all() ???:0 23 0x000000000046b733 flush_output_buffer() ???:0 24 0x000000000042d108 sync_file() pio_file.c:0 25 0x000000000042d432 PIOc_closefile() ???:0 26 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 27 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 28 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 29 0x0000000000410ff3 main() ???:0 30 0x0000000000023493 __libc_start_main() ???:0 31 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 718158) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 6 0x0000000000232b77 mca_pml_ucx_send_nbr() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 7 0x0000000000232b77 mca_pml_ucx_send() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 8 0x00000000000d7c32 ompi_coll_base_sendrecv_actual() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.c:58 9 0x00000000000d707b ompi_coll_base_sendrecv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/base/coll_base_util.h:133 10 0x000000000010ced0 ompi_coll_tuned_allgatherv_intra_dec_fixed() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 11 0x000000000016697a mca_fcoll_vulcan_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 12 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 13 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 14 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 15 0x000000000016fbc2 ncmpio_read_write() ???:0 16 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 17 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 18 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 19 0x00000000001661a4 req_commit() ncmpio_wait.c:0 20 0x0000000000166a0c ncmpio_wait() ???:0 21 0x00000000000b727a ncmpi_wait_all() ???:0 22 0x000000000046b733 flush_output_buffer() ???:0 23 0x000000000042d108 sync_file() pio_file.c:0 24 0x000000000042d432 PIOc_closefile() ???:0 25 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 26 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 27 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 28 0x0000000000410ff3 main() ???:0 29 0x0000000000023493 __libc_start_main() ???:0 30 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x155550b15faa in ??? #11 0x1555512dc1d9 in ??? #12 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #13 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #14 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #15 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #16 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #17 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #18 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #19 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #20 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #21 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #22 0x155554cf2bc1 in ??? #23 0x155554ced7f5 in ??? #24 0x155554ceb2bb in ??? #25 0x155554cece3f in ??? #26 0x155554ce91a3 in ??? #27 0x155554ce9a0b in ??? #28 0x155554c3a279 in ??? #29 0x46b732 in ??? #30 0x42d107 in ??? #31 0x42d431 in ??? #32 0x4137fe in ??? #33 0x40dd3e in ??? #34 0x40ad14 in ??? #35 0x410ff2 in ??? #36 0x155552779492 in ??? #37 0x40a48d in ??? #38 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bdfb76 in mca_pml_ucx_send_nbr at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:923 #12 0x155553bdfb76 in mca_pml_ucx_send at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:944 #13 0x155553a84c31 in ompi_coll_base_sendrecv_actual at base/coll_base_util.c:58 #14 0x155553a8407a in ompi_coll_base_sendrecv at base/coll_base_util.h:133 #15 0x155553a8407a in ompi_coll_base_allgatherv_intra_ring at base/coll_base_allgatherv.c:272 #16 0x155553ab9ecf in ompi_coll_tuned_allgatherv_intra_dec_fixed at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/coll/tuned/coll_tuned_decision_fixed.c:1363 #17 0x155553b13979 in mca_fcoll_vulcan_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/vulcan/fcoll_vulcan_file_write_all.c:418 #18 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #19 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #20 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #21 0x155554cf2bc1 in ??? #22 0x155554ced7f5 in ??? #23 0x155554ceb2bb in ??? #24 0x155554cece3f in ??? #25 0x155554ce91a3 in ??? #26 0x155554ce9a0b in ??? #27 0x155554c3a279 in ??? #28 0x46b732 in ??? #29 0x42d107 in ??? #30 0x42d431 in ??? #31 0x4137fe in ??? #32 0x40dd3e in ??? #33 0x40ad14 in ??? #34 0x410ff2 in ??? #35 0x155552779492 in ??? #36 0x40a48d in ??? #37 0xffffffffffffffff in ??? srun: error: chr-0500: task 429: Aborted (core dumped) srun: error: chr-0500: task 433: Killed srun: error: chr-0494: task 46: Aborted (core dumped) srun: error: chr-0494: task 51: Killed srun: error: chr-0496: tasks 155,161: Aborted (core dumped) srun: error: chr-0501: tasks 456,469,487-488: Killed srun: error: chr-0497: tasks 201,229,232,241: Killed srun: error: chr-0498: task 283: Killed srun: error: chr-0499: task 334: Killed srun: error: chr-0494: task 52: Aborted (core dumped) srun: error: chr-0498: task 273: Killed srun: error: chr-0501: tasks 457,470: Aborted (core dumped) srun: error: chr-0497: task 192: Killed srun: error: chr-0497: tasks 233,242: Aborted (core dumped) srun: error: chr-0498: task 257: Killed srun: Job step aborted: Waiting up to 92 seconds for job step to finish. slurmstepd: error: *** STEP 195622.0 ON chr-0494 CANCELLED AT 2022-06-29T10:13:42 DUE TO TIME LIMIT *** slurmstepd: error: *** JOB 195622 ON chr-0494 CANCELLED AT 2022-06-29T10:13:42 DUE TO TIME LIMIT *** srun: got SIGCONT srun: forcing job termination