Testing decomp: ./ne30_F_case_48602x72_512p.dat pio_readdof start pio_readdof end, read time = 0.46044078199999999 srun: error: chr-0509: task 487: Killed [chr-0502:667331:0:667331] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667359:0:667359] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667301:0:667301] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667303:0:667303] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667329:0:667329] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667316:0:667316] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667298:0:667298] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667341:0:667341] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667343:0:667343] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667355:0:667355] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667351:0:667351] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667333:0:667333] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667335:0:667335] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667347:0:667347] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667337:0:667337] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667339:0:667339] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667338:0:667338] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667346:0:667346] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667345:0:667345] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667349:0:667349] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667310:0:667310] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667312:0:667312] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667311:0:667311] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667357:0:667357] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667330:0:667330] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667332:0:667332] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667354:0:667354] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667356:0:667356] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667342:0:667342] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667344:0:667344] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667334:0:667334] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667336:0:667336] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667302:0:667302] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667304:0:667304] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667360:0:667360] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667315:0:667315] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667358:0:667358] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667300:0:667300] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667340:0:667340] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667299:0:667299] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667306:0:667306] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667308:0:667308] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667324:0:667324] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667322:0:667322] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667350:0:667350] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667352:0:667352] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667318:0:667318] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667320:0:667320] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667348:0:667348] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667326:0:667326] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667328:0:667328] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667353:0:667353] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667313:0:667313] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667321:0:667321] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667327:0:667327] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667309:0:667309] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667317:0:667317] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667319:0:667319] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667323:0:667323] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667325:0:667325] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667305:0:667305] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667307:0:667307] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address [chr-0502:667314:0:667314] cma_ep.c:97 process_vm_readv(pid=667297 length=524288) returned -1: Bad address ==== backtrace (tid: 667303) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667301) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667299) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667305) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667311) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667307) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667309) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667331) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667347) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667329) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667316) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667341) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667343) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? ==== backtrace (tid: 667298) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667314) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667359) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667337) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667339) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667315) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667306) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667317) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667319) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667325) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667327) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667313) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667323) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667321) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667355) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667351) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? ==== backtrace (tid: 667338) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667346) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667333) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667335) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667349) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667332) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667330) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667345) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667357) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667340) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667342) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667344) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667334) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667336) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667310) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667312) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667348) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667352) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667350) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667356) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667354) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667358) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667360) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667324) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667322) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667318) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667326) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667320) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667328) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667353) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667308) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667302) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= ==== backtrace (tid: 667304) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: ==== backtrace (tid: 667300) ==== 0 0x00000000000026a2 uct_cma_ep_tx() ???:0 1 0x000000000001aff9 uct_scopy_ep_progress_tx() ???:0 2 0x0000000000053414 ucs_arbiter_dispatch_nonempty() ???:0 3 0x000000000001aaf1 uct_scopy_iface_progress() ???:0 4 0x00000000000351da ucp_worker_progress() ???:0 5 0x0000000000230c20 mca_pml_ucx_recv() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 6 0x0000000000161d5b ompi_fcoll_base_coll_allgatherv_array() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/base/fcoll_base_coll_array.c:108 7 0x000000000017534b mca_fcoll_dynamic_file_write_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 8 0x00000000000c2b39 mca_common_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 9 0x00000000001aff57 mca_io_ompio_file_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 10 0x00000000000aaaae PMPI_File_write_at_all() /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 11 0x000000000016fbc2 ncmpio_read_write() ???:0 12 0x000000000016a7f6 mgetput() ncmpio_wait.c:0 13 0x00000000001682bc req_aggregation() ncmpio_wait.c:0 14 0x0000000000169e40 wait_getput() ncmpio_wait.c:0 15 0x00000000001661a4 req_commit() ncmpio_wait.c:0 16 0x0000000000166a0c ncmpio_wait() ???:0 17 0x00000000000b727a ncmpi_wait_all() ???:0 18 0x000000000046b733 flush_output_buffer() ???:0 19 0x000000000042d108 sync_file() pio_file.c:0 20 0x000000000042d432 PIOc_closefile() ???:0 21 0x00000000004137ff __piolib_mod_MOD_closefile() ???:0 22 0x000000000040dd3f pioperformance_rearrtest.4019() pioperformance_rearr.F90:0 23 0x000000000040ad15 MAIN__() pioperformance_rearr.F90:0 24 0x0000000000410ff3 main() ???:0 25 0x0000000000023493 __libc_start_main() ???:0 26 0x000000000040a48e _start() ???:0 ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? #0 0x15555278d3ff in ??? #1 0x15555278d37f in ??? #2 0x155552777db4 in ??? #3 0x155550b1afb5 in ??? #4 0x155550b203c4 in ??? #5 0x155550b20563 in ??? #6 0x15554d8746a1 in ??? #7 0x15555108aff8 in ??? #8 0x155550b15413 in ??? #9 0x15555108aaf0 in ??? #10 0x1555512dc1d9 in ??? #11 0x155553bddc1f in mca_pml_ucx_recv at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/pml/ucx/pml_ucx.c:646 #12 0x155553b0ed5a in ompi_fcoll_base_coll_allgatherv_array at base/fcoll_base_coll_array.c:108 #13 0x155553b2234a in mca_fcoll_dynamic_file_write_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c:302 #14 0x155553a6fb38 in mca_common_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/common/ompio/common_ompio_file_write.c:452 #15 0x155553b5cf56 in mca_io_ompio_file_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mca/io/ompio/io_ompio_file_write.c:174 #16 0x155553a57aad in PMPI_File_write_at_all at /tmp/svcbuilder/spack-stage-openmpi-4.1.3-sxfyy4knvddpewshfcc45heice7tzs7f/spack-src/ompi/mpi/c/profile/pfile_write_at_all.c:75 #17 0x155554cf2bc1 in ??? #18 0x155554ced7f5 in ??? #19 0x155554ceb2bb in ??? #20 0x155554cece3f in ??? #21 0x155554ce91a3 in ??? #22 0x155554ce9a0b in ??? #23 0x155554c3a279 in ??? #24 0x46b732 in ??? #25 0x42d107 in ??? #26 0x42d431 in ??? #27 0x4137fe in ??? #28 0x40dd3e in ??? #29 0x40ad14 in ??? #30 0x410ff2 in ??? #31 0x155552779492 in ??? #32 0x40a48d in ??? #33 0xffffffffffffffff in ??? srun: error: chr-0502: task 0: Killed srun: error: chr-0502: tasks 2,4,6,8: Aborted (core dumped) srun: error: chr-0502: tasks 1,3,5,7,9-63: Aborted (core dumped) srun: Job step aborted: Waiting up to 92 seconds for job step to finish. slurmstepd: error: *** JOB 195680 ON chr-0502 CANCELLED AT 2022-06-29T13:41:42 DUE TO TIME LIMIT *** srun: got SIGCONT srun: forcing job termination