+ [22-12-29 13:02:22] '[' multi == multi ']' + [22-12-29 13:02:22] echo 'Clients: rdma-cki-01.rdma.lab.eng.rdu2.redhat.com' Clients: rdma-cki-01.rdma.lab.eng.rdu2.redhat.com + [22-12-29 13:02:22] echo 'Servers: rdma-dev-21.rdma.lab.eng.rdu2.redhat.com' Servers: rdma-dev-21.rdma.lab.eng.rdu2.redhat.com + [22-12-29 13:02:22] RQA_system_info_for_debug + [22-12-29 13:02:22] grep -i distro /etc/motd + [22-12-29 13:02:22] tr -d ' ' DISTRO=CentOS-Stream-9-20221219.0 + [22-12-29 13:02:22] cat /etc/redhat-release CentOS Stream release 9 + [22-12-29 13:02:22] uname -a Linux rdma-cki-01.rdma.lab.eng.rdu2.redhat.com 5.14.0-226.1773_734386600.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Dec 29 16:22:43 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux + [22-12-29 13:02:22] cat /proc/cmdline BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-226.1773_734386600.el9.x86_64 root=UUID=11404fb6-7936-492a-b769-723f1beb2070 ro resume=UUID=cd04ddec-f114-4d12-a049-904ba18ffce4 console=ttyS0,115200n81 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G + [22-12-29 13:02:22] rpm -q rdma-core linux-firmware rdma-core-41.0-3.el9.x86_64 linux-firmware-20221214-129.el9.noarch + [22-12-29 13:02:22] tail /sys/class/infiniband/mlx5_bond_0/fw_ver 22.99.7864 + [22-12-29 13:02:22] lspci + [22-12-29 13:02:22] grep -i -e ethernet -e infiniband -e omni -e ConnectX 84:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 84:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] c5:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) + [22-12-29 13:02:22] lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. Model name: AMD EPYC 7402P 24-Core Processor BIOS Model name: AMD EPYC 7402P 24-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU max MHz: 2800.0000 CPU min MHz: 1500.0000 BogoMIPS: 5589.79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca Virtualization: AMD-V L1d cache: 768 KiB (24 instances) L1i cache: 768 KiB (24 instances) L2 cache: 12 MiB (24 instances) L3 cache: 128 MiB (8 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-47 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected + [22-12-29 13:02:22] ibstat CA 'mlx5_bond_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.99.7864 Hardware version: 0 Node GUID: 0xb8599f0300f07fd0 System image GUID: 0xb8599f0300f07fd0 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba599ffffef07fd0 Link layer: Ethernet + [22-12-29 13:02:22] ibstatus Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 100 Gb/sec (4X EDR) link_layer: Ethernet + [22-12-29 13:02:22] /usr/bin/ibv_devinfo -v hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 22.99.7864 node_guid: b859:9f03:00f0:7fd0 sys_image_guid: b859:9f03:00f0:7fd0 vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000359 phys_port_cnt: 1 max_mr_size: 0xffffffffffffffff page_size_cap: 0xfffffffffffff000 max_qp: 262144 max_qp_wr: 32768 device_cap_flags: 0x25321c36 BAD_PKEY_CNTR BAD_QKEY_CNTR AUTO_PATH_MIG CHANGE_PHY_PORT PORT_ACTIVE_EVENT SYS_IMAGE_GUID RC_RNR_NAK_GEN MEM_WINDOW XRC MEM_MGT_EXTENSIONS MEM_WINDOW_TYPE_2B RAW_IP_CSUM MANAGED_FLOW_STEERING max_sge: 30 max_sge_rd: 30 max_cq: 16777216 max_cqe: 4194303 max_mr: 16777216 max_pd: 16777216 max_qp_rd_atom: 16 max_ee_rd_atom: 0 max_res_rd_atom: 4194304 max_qp_init_rd_atom: 16 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_HCA (1) max_ee: 0 max_rdd: 0 max_mw: 16777216 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 0 max_mcast_grp: 2097152 max_mcast_qp_attach: 240 max_total_mcast_qp_attach: 503316480 max_ah: 2147483647 max_fmr: 0 max_srq: 8388608 max_srq_wr: 32767 max_srq_sge: 31 max_pkeys: 128 local_ca_ack_delay: 16 general_odp_caps: ODP_SUPPORT ODP_SUPPORT_IMPLICIT rc_odp_caps: SUPPORT_SEND SUPPORT_RECV SUPPORT_WRITE SUPPORT_READ SUPPORT_SRQ uc_odp_caps: NO SUPPORT ud_odp_caps: SUPPORT_SEND xrc_odp_caps: SUPPORT_SEND SUPPORT_WRITE SUPPORT_READ SUPPORT_SRQ completion timestamp_mask: 0x7fffffffffffffff hca_core_clock: 156250kHZ raw packet caps: C-VLAN stripping offload Scatter FCS offload IP csum offload Delay drop device_cap_flags_ex: 0x1425321C36 RAW_SCATTER_FCS PCI_WRITE_END_PADDING tso_caps: max_tso: 262144 supported_qp: SUPPORT_RAW_PACKET rss_caps: max_rwq_indirection_tables: 65536 max_rwq_indirection_table_size: 2048 rx_hash_function: 0x1 rx_hash_fields_mask: 0x800000FF supported_qp: SUPPORT_RAW_PACKET max_wq_type_rq: 8388608 packet_pacing_caps: qp_rate_limit_min: 1kbps qp_rate_limit_max: 100000000kbps supported_qp: SUPPORT_RAW_PACKET tag matching not supported cq moderation caps: max_cq_count: 65535 max_cq_period: 4095 us maximum available device memory: 131072Bytes num_comp_vectors: 48 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet max_msg_sz: 0x40000000 port_cap_flags: 0x04010000 port_cap_flags2: 0x0000 max_vl_num: invalid value (0) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 1 gid_tbl_len: 256 subnet_timeout: 0 init_type_reply: 0 active_width: 4X (2) active_speed: 25.0 Gbps (32) phys_state: LINK_UP (5) GID[ 0]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 1]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 2]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 3]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 4]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 5]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 6]: 0000:0000:0000:0000:0000:ffff:ac1f:2b51, RoCE v1 GID[ 7]: ::ffff:172.31.43.81, RoCE v2 GID[ 8]: 0000:0000:0000:0000:0000:ffff:ac1f:2851, RoCE v1 GID[ 9]: ::ffff:172.31.40.81, RoCE v2 GID[ 10]: 0000:0000:0000:0000:0000:ffff:ac1f:2d51, RoCE v1 GID[ 11]: ::ffff:172.31.45.81, RoCE v2 + [22-12-29 13:02:23] iblinkinfo ibwarn: [11130] mad_rpc_open_port: client_register for mgmt 1 failed Failed to open (null) port 0 + [22-12-29 13:02:23] [[ '' == *\o\p\a* ]] + [22-12-29 13:02:23] ip addr show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: lom_1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:ad:f7:84 brd ff:ff:ff:ff:ff:ff altname enp197s0f0 altname eno5 inet 10.1.217.16/24 brd 10.1.217.255 scope global dynamic noprefixroute lom_1 valid_lft 86176sec preferred_lft 86176sec inet6 2620:52:0:1d9:4adf:37ff:fead:f784/64 scope global dynamic noprefixroute valid_lft 2591935sec preferred_lft 604735sec inet6 fe80::4adf:37ff:fead:f784/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: lom_2: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:85 brd ff:ff:ff:ff:ff:ff altname enp197s0f1 altname eno6 4: lom_3: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:86 brd ff:ff:ff:ff:ff:ff altname enp197s0f2 altname eno7 5: lom_4: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:87 brd ff:ff:ff:ff:ff:ff altname enp197s0f3 altname eno8 6: mlx5_slave1: mtu 9000 qdisc mq master mlx5_bond_roce state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff altname enp132s0f0np0 altname ens1f0np0 7: mlx5_slave2: mtu 9000 qdisc mq master mlx5_bond_roce state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff permaddr b8:59:9f:f0:7f:d1 altname enp132s0f1np1 altname ens1f1np1 8: mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.40.81/24 brd 172.31.40.255 scope global dynamic noprefixroute mlx5_bond_roce valid_lft 3392sec preferred_lft 3392sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever 9: mlx5_bond_ro.43@mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.43.81/24 brd 172.31.43.255 scope global dynamic noprefixroute mlx5_bond_ro.43 valid_lft 3392sec preferred_lft 3392sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever 10: mlx5_bond_ro.45@mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.45.81/24 brd 172.31.45.255 scope global dynamic noprefixroute mlx5_bond_ro.45 valid_lft 3392sec preferred_lft 3392sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever + [22-12-29 13:02:23] grep -i fedora /etc/redhat-release + [22-12-29 13:02:23] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:02:23] hostname -A + [22-12-29 13:02:23] grep rdma-cki-01 + [22-12-29 13:02:23] echo '------- start test -------' ------- start test ------- + [22-12-29 13:02:23] client + [22-12-29 13:02:23] source ./setup.sh + [22-12-29 13:02:23] client_setup + [22-12-29 13:02:23] is_cki + [22-12-29 13:02:23] typeset -l flag=true + [22-12-29 13:02:23] [[ true == \t\r\u\e ]] + [22-12-29 13:02:23] return 0 + [22-12-29 13:02:23] run_task_env_setup + [22-12-29 13:02:23] echo 'Now start to run task env_setup, please be patient ...' Now start to run task env_setup, please be patient ... + [22-12-29 13:02:23] cd ../../env_setup + [22-12-29 13:02:23] make run make[1]: Entering directory '/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.zip/infiniband/env_setup' test -x runtest.sh || chmod a+x runtest.sh ./runtest.sh + trap 'exit 1' SIGHUP SIGINT SIGQUIT SIGTERM + which wget + alias + eval declare -f ++ declare -f + /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot wget /usr/bin/wget + source /usr/bin/rhts_environment.sh ++ '[' -z /mnt/testarea/tmp.BcDPZz ']' ++ '[' -z '' ']' +++ uname -i ++ ARCH=x86_64 ++ '[' -z '' ']' +++ cat /etc/redhat-release +++ sed -e 's/\(.*\)release\s\([0-9]*\).*/\1\2/; s/\s//g' ++ FAMILY=CentOSStream9 ++ '[' -h /mnt/testarea/current.log ']' ++ ln -sf /mnt/testarea/tmp.BcDPZz /mnt/testarea/current.log + '[' -f /root/rdma-qa.sh ']' + source /root/rdma-qa.sh ++ PS4='$(date +"+ [%y-%m-%d %H:%M:%S]") ' ++ [22-12-29 13:02:23] export NIC_PREF= ++ [22-12-29 13:02:23] NIC_PREF= ++ [22-12-29 13:02:23] export PERFTEST_FLAGS= ++ [22-12-29 13:02:23] PERFTEST_FLAGS= +++ [22-12-29 13:02:23] hostname -s ++ [22-12-29 13:02:23] export s_hostname=rdma-cki-01 ++ [22-12-29 13:02:23] s_hostname=rdma-cki-01 ++ [22-12-29 13:02:23] RQA_set_pyexec ++ [22-12-29 13:02:23] PYEXEC= ++ [22-12-29 13:02:23] which python ++ [22-12-29 13:02:23] export PYEXEC=python ++ [22-12-29 13:02:23] PYEXEC=python ++ [22-12-29 13:02:23] return 0 +++ [22-12-29 13:02:23] grep -i fedora /etc/redhat-release ++ [22-12-29 13:02:23] [[ -n '' ]] +++ [22-12-29 13:02:23] RQA_get_rhel_major +++ [22-12-29 13:02:23] awk -F . '{print $1}' ++++ [22-12-29 13:02:23] RQA_get_rhel_release ++++ [22-12-29 13:02:23] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:23] echo 9 ++ [22-12-29 13:02:23] [[ 9 -ge 8 ]] ++ [22-12-29 13:02:23] export 'PKGINSTALL=dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:02:23] PKGINSTALL='dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:02:23] export 'PKGREMOVE=dnf remove --noautoremove -y' ++ [22-12-29 13:02:23] PKGREMOVE='dnf remove --noautoremove -y' ++ [22-12-29 13:02:23] [[ rdma-cki-01 == \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:02:23] export LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:02:23] LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:02:23] '[' -f /root/known-issues.sh ']' ++ [22-12-29 13:02:23] source /root/known-issues.sh ++ [22-12-29 13:02:26] RQA_load_tech_preview_modules ++ [22-12-29 13:02:26] [[ ! -e /etc/rdma/rdma.conf ]] ++ [22-12-29 13:02:26] return ++ [22-12-29 13:02:26] RQA_fix_uefi_reboot ++ [22-12-29 13:02:26] ls -l /usr/sbin/reboot ++ [22-12-29 13:02:26] grep rhts-reboot ++ [22-12-29 13:02:26] which rhts-reboot ++ [22-12-29 13:02:26] rpm -q rhts-test-env ++ [22-12-29 13:02:26] RQA_fix_ssh_ipv6_auth +++ [22-12-29 13:02:26] RQA_get_rhel_major +++ [22-12-29 13:02:26] awk -F . '{print $1}' ++++ [22-12-29 13:02:26] RQA_get_rhel_release ++++ [22-12-29 13:02:26] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:26] echo 9 ++ [22-12-29 13:02:26] '[' 9 -ge 7 ']' ++ [22-12-29 13:02:26] local need_restart=0 ++ [22-12-29 13:02:26] grep -E '^UseDNS yes' /etc/ssh/sshd_config ++ [22-12-29 13:02:26] grep -E '^GSSAPIStrictAcceptorCheck yes' /etc/ssh/sshd_config ++ [22-12-29 13:02:26] grep 'AddressFamily inet' /root/.ssh/config ++ [22-12-29 13:02:26] '[' 0 -eq 1 ']' ++ [22-12-29 13:02:26] RQA_fix_mlx5_ib_connected_mode +++ [22-12-29 13:02:26] RQA_get_rhel_major +++ [22-12-29 13:02:26] awk -F . '{print $1}' ++++ [22-12-29 13:02:26] RQA_get_rhel_release ++++ [22-12-29 13:02:26] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:26] echo 9 ++ [22-12-29 13:02:26] [[ 9 -ge 7 ]] +++ [22-12-29 13:02:26] RQA_get_rhel_minor +++ [22-12-29 13:02:26] awk -F . '{print $2}' ++++ [22-12-29 13:02:26] RQA_get_rhel_release ++++ [22-12-29 13:02:26] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:26] echo 9 ++ [22-12-29 13:02:26] [[ '' -ge 5 ]] +++ [22-12-29 13:02:26] RQA_get_rhel_major +++ [22-12-29 13:02:26] awk -F . '{print $1}' ++++ [22-12-29 13:02:26] RQA_get_rhel_release ++++ [22-12-29 13:02:27] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:27] echo 9 ++ [22-12-29 13:02:27] [[ 9 -gt 7 ]] +++ [22-12-29 13:02:27] ls /etc/sysconfig/network-scripts/ +++ [22-12-29 13:02:27] grep mlx5 +++ [22-12-29 13:02:27] grep ib +++ [22-12-29 13:02:27] grep -v '~' ++ [22-12-29 13:02:27] RQA_nfs_rhel8_temp_fix +++ [22-12-29 13:02:27] RQA_get_rhel_major +++ [22-12-29 13:02:27] awk -F . '{print $1}' ++++ [22-12-29 13:02:27] RQA_get_rhel_release ++++ [22-12-29 13:02:27] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:02:27] echo 9 ++ [22-12-29 13:02:27] (( 9 == 8 )) ++ [22-12-29 13:02:27] source /root/.bashrc ++ [22-12-29 13:02:27] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:02:27] export BASHRC_FILE=/root/.bashrc + [22-12-29 13:02:27] BASHRC_FILE=/root/.bashrc + [22-12-29 13:02:27] [[ rdma-cki-01 == \r\d\m\a\-\p\e\r\f\-\0\2 ]] + [22-12-29 13:02:27] [[ rdma-cki-01 == \r\d\m\a\-\p\e\r\f\-\0\3 ]] + [22-12-29 13:02:27] export ENV_DRIVER= + [22-12-29 13:02:27] ENV_DRIVER= + [22-12-29 13:02:27] export ENV_NETWORK=roce + [22-12-29 13:02:27] ENV_NETWORK=roce ++ [22-12-29 13:02:27] hostname -s + [22-12-29 13:02:27] export host_json=./json/rdma-cki-01.json + [22-12-29 13:02:27] host_json=./json/rdma-cki-01.json + [22-12-29 13:02:27] [[ rdma-dev-21.rdma.lab.eng.rdu2.redhat.com == *\r\d\m\a\-\d\e\v\-\3\0* ]] + [22-12-29 13:02:27] [[ rdma-dev-21.rdma.lab.eng.rdu2.redhat.com == *\r\d\m\a\-\d\e\v\-\3\1* ]] + [22-12-29 13:02:27] [[ rdma-cki-01.rdma.lab.eng.rdu2.redhat.com == *\r\d\m\a\-\d\e\v\-\3\0* ]] + [22-12-29 13:02:27] [[ rdma-cki-01.rdma.lab.eng.rdu2.redhat.com == *\r\d\m\a\-\d\e\v\-\3\1* ]] + [22-12-29 13:02:27] '[' -z rdma-cki-01.rdma.lab.eng.rdu2.redhat.com -a -z rdma-dev-21.rdma.lab.eng.rdu2.redhat.com ']' + [22-12-29 13:02:27] RQA_set_servers_clients ++ [22-12-29 13:02:27] echo rdma-dev-21.rdma.lab.eng.rdu2.redhat.com ++ [22-12-29 13:02:27] awk -F . '{ print $1}' + [22-12-29 13:02:27] s_server_name=rdma-dev-21 ++ [22-12-29 13:02:27] echo rdma-cki-01.rdma.lab.eng.rdu2.redhat.com ++ [22-12-29 13:02:27] awk -F . '{ print $1}' + [22-12-29 13:02:27] s_client_name=rdma-cki-01 + [22-12-29 13:02:27] grep 'export SERVERS=' /root/.bashrc + [22-12-29 13:02:27] echo 'export SERVERS="rdma-dev-21"' + [22-12-29 13:02:27] grep 'export CLIENTS=' /root/.bashrc + [22-12-29 13:02:27] echo 'export CLIENTS="rdma-cki-01"' + [22-12-29 13:02:27] source /root/.bashrc ++ [22-12-29 13:02:27] '[' -f /etc/bashrc ']' ++ [22-12-29 13:02:27] . /etc/bashrc +++ [22-12-29 13:02:27] '[' -z Y ']' ++ [22-12-29 13:02:27] '[' vt100 = vt102 ']' ++ [22-12-29 13:02:27] alias 'grep=/bin/grep --color=tty ' ++ [22-12-29 13:02:27] alias vi=vim ++ [22-12-29 13:02:27] export EDITOR=vim ++ [22-12-29 13:02:27] EDITOR=vim ++ [22-12-29 13:02:27] RDMA_LOOKASIDE=http://download-node-02.eng.bos.redhat.com/qa/rhts/lookaside/rdma-testing +++ [22-12-29 13:02:27] which rhts-reboot +++ [22-12-29 13:02:27] alias +++ [22-12-29 13:02:27] /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot rhts-reboot +++ [22-12-29 13:02:27] eval declare -f ++++ [22-12-29 13:02:27] declare -f ++ [22-12-29 13:02:27] '[' '!' -x /usr/bin/rhts-reboot ']' ++ [22-12-29 13:02:27] alias shutdown=rhts-reboot ++ [22-12-29 13:02:27] export SERVERS=rdma-dev-21 ++ [22-12-29 13:02:27] SERVERS=rdma-dev-21 ++ [22-12-29 13:02:27] export CLIENTS=rdma-cki-01 ++ [22-12-29 13:02:27] CLIENTS=rdma-cki-01 + [22-12-29 13:02:27] RQA_rhts_or_dev_mode multi env + [22-12-29 13:02:27] export RUN_NUMBER=0 + [22-12-29 13:02:27] RUN_NUMBER=0 ++ [22-12-29 13:02:27] grep -i 'Job whiteboard:' /etc/motd ++ [22-12-29 13:02:27] awk -F '[' '{print $2}' ++ [22-12-29 13:02:27] sed 's/\]//g' + [22-12-29 13:02:27] merge_request= + [22-12-29 13:02:27] [[ ! -z '' ]] + [22-12-29 13:02:27] test -z 7381428 + [22-12-29 13:02:27] echo 'Variable JOBID set, we'\''re running on RHTS' Variable JOBID set, we're running on RHTS +++ [22-12-29 13:02:27] pwd ++ [22-12-29 13:02:27] basename /mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.zip/infiniband/env_setup + [22-12-29 13:02:27] '[' env_setup == nfsordma_sh ']' + [22-12-29 13:02:27] exec + [22-12-29 13:02:27] exec + [22-12-29 13:02:27] '[' multi == multi ']' + [22-12-29 13:02:27] echo 'Clients: rdma-cki-01' Clients: rdma-cki-01 + [22-12-29 13:02:27] echo 'Servers: rdma-dev-21' Servers: rdma-dev-21 + [22-12-29 13:02:27] RQA_system_info_for_debug + [22-12-29 13:02:27] tr -d ' ' + [22-12-29 13:02:27] grep -i distro /etc/motd DISTRO=CentOS-Stream-9-20221219.0 + [22-12-29 13:02:27] cat /etc/redhat-release CentOS Stream release 9 + [22-12-29 13:02:27] uname -a Linux rdma-cki-01.rdma.lab.eng.rdu2.redhat.com 5.14.0-226.1773_734386600.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Dec 29 16:22:43 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux + [22-12-29 13:02:27] cat /proc/cmdline BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-226.1773_734386600.el9.x86_64 root=UUID=11404fb6-7936-492a-b769-723f1beb2070 ro resume=UUID=cd04ddec-f114-4d12-a049-904ba18ffce4 console=ttyS0,115200n81 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G + [22-12-29 13:02:27] rpm -q rdma-core linux-firmware rdma-core-41.0-3.el9.x86_64 linux-firmware-20221214-129.el9.noarch + [22-12-29 13:02:27] tail /sys/class/infiniband/mlx5_bond_0/fw_ver 22.99.7864 + [22-12-29 13:02:27] lspci + [22-12-29 13:02:27] grep -i -e ethernet -e infiniband -e omni -e ConnectX 84:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] 84:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] c5:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) c5:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) + [22-12-29 13:02:27] lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. Model name: AMD EPYC 7402P 24-Core Processor BIOS Model name: AMD EPYC 7402P 24-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU max MHz: 2800.0000 CPU min MHz: 1500.0000 BogoMIPS: 5589.79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca Virtualization: AMD-V L1d cache: 768 KiB (24 instances) L1i cache: 768 KiB (24 instances) L2 cache: 12 MiB (24 instances) L3 cache: 128 MiB (8 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-47 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected + [22-12-29 13:02:27] ibstat CA 'mlx5_bond_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.99.7864 Hardware version: 0 Node GUID: 0xb8599f0300f07fd0 System image GUID: 0xb8599f0300f07fd0 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba599ffffef07fd0 Link layer: Ethernet + [22-12-29 13:02:27] ibstatus Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 100 Gb/sec (4X EDR) link_layer: Ethernet + [22-12-29 13:02:27] /usr/bin/ibv_devinfo -v hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 22.99.7864 node_guid: b859:9f03:00f0:7fd0 sys_image_guid: b859:9f03:00f0:7fd0 vendor_id: 0x02c9 vendor_part_id: 4125 hw_ver: 0x0 board_id: MT_0000000359 phys_port_cnt: 1 max_mr_size: 0xffffffffffffffff page_size_cap: 0xfffffffffffff000 max_qp: 262144 max_qp_wr: 32768 device_cap_flags: 0x25321c36 BAD_PKEY_CNTR BAD_QKEY_CNTR AUTO_PATH_MIG CHANGE_PHY_PORT PORT_ACTIVE_EVENT SYS_IMAGE_GUID RC_RNR_NAK_GEN MEM_WINDOW XRC MEM_MGT_EXTENSIONS MEM_WINDOW_TYPE_2B RAW_IP_CSUM MANAGED_FLOW_STEERING max_sge: 30 max_sge_rd: 30 max_cq: 16777216 max_cqe: 4194303 max_mr: 16777216 max_pd: 16777216 max_qp_rd_atom: 16 max_ee_rd_atom: 0 max_res_rd_atom: 4194304 max_qp_init_rd_atom: 16 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_HCA (1) max_ee: 0 max_rdd: 0 max_mw: 16777216 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 0 max_mcast_grp: 2097152 max_mcast_qp_attach: 240 max_total_mcast_qp_attach: 503316480 max_ah: 2147483647 max_fmr: 0 max_srq: 8388608 max_srq_wr: 32767 max_srq_sge: 31 max_pkeys: 128 local_ca_ack_delay: 16 general_odp_caps: ODP_SUPPORT ODP_SUPPORT_IMPLICIT rc_odp_caps: SUPPORT_SEND SUPPORT_RECV SUPPORT_WRITE SUPPORT_READ SUPPORT_SRQ uc_odp_caps: NO SUPPORT ud_odp_caps: SUPPORT_SEND xrc_odp_caps: SUPPORT_SEND SUPPORT_WRITE SUPPORT_READ SUPPORT_SRQ completion timestamp_mask: 0x7fffffffffffffff hca_core_clock: 156250kHZ raw packet caps: C-VLAN stripping offload Scatter FCS offload IP csum offload Delay drop device_cap_flags_ex: 0x1425321C36 RAW_SCATTER_FCS PCI_WRITE_END_PADDING tso_caps: max_tso: 262144 supported_qp: SUPPORT_RAW_PACKET rss_caps: max_rwq_indirection_tables: 65536 max_rwq_indirection_table_size: 2048 rx_hash_function: 0x1 rx_hash_fields_mask: 0x800000FF supported_qp: SUPPORT_RAW_PACKET max_wq_type_rq: 8388608 packet_pacing_caps: qp_rate_limit_min: 1kbps qp_rate_limit_max: 100000000kbps supported_qp: SUPPORT_RAW_PACKET tag matching not supported cq moderation caps: max_cq_count: 65535 max_cq_period: 4095 us maximum available device memory: 131072Bytes num_comp_vectors: 48 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet max_msg_sz: 0x40000000 port_cap_flags: 0x04010000 port_cap_flags2: 0x0000 max_vl_num: invalid value (0) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 1 gid_tbl_len: 256 subnet_timeout: 0 init_type_reply: 0 active_width: 4X (2) active_speed: 25.0 Gbps (32) phys_state: LINK_UP (5) GID[ 0]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 1]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 2]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 3]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 4]: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0, RoCE v1 GID[ 5]: fe80::ba59:9fff:fef0:7fd0, RoCE v2 GID[ 6]: 0000:0000:0000:0000:0000:ffff:ac1f:2b51, RoCE v1 GID[ 7]: ::ffff:172.31.43.81, RoCE v2 GID[ 8]: 0000:0000:0000:0000:0000:ffff:ac1f:2851, RoCE v1 GID[ 9]: ::ffff:172.31.40.81, RoCE v2 GID[ 10]: 0000:0000:0000:0000:0000:ffff:ac1f:2d51, RoCE v1 GID[ 11]: ::ffff:172.31.45.81, RoCE v2 + [22-12-29 13:02:27] iblinkinfo ibwarn: [14657] mad_rpc_open_port: client_register for mgmt 1 failed Failed to open (null) port 0 + [22-12-29 13:02:27] [[ '' == *\o\p\a* ]] + [22-12-29 13:02:27] ip addr show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: lom_1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 48:df:37:ad:f7:84 brd ff:ff:ff:ff:ff:ff altname enp197s0f0 altname eno5 inet 10.1.217.16/24 brd 10.1.217.255 scope global dynamic noprefixroute lom_1 valid_lft 86171sec preferred_lft 86171sec inet6 2620:52:0:1d9:4adf:37ff:fead:f784/64 scope global dynamic noprefixroute valid_lft 2591930sec preferred_lft 604730sec inet6 fe80::4adf:37ff:fead:f784/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: lom_2: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:85 brd ff:ff:ff:ff:ff:ff altname enp197s0f1 altname eno6 4: lom_3: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:86 brd ff:ff:ff:ff:ff:ff altname enp197s0f2 altname eno7 5: lom_4: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 48:df:37:ad:f7:87 brd ff:ff:ff:ff:ff:ff altname enp197s0f3 altname eno8 6: mlx5_slave1: mtu 9000 qdisc mq master mlx5_bond_roce state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff altname enp132s0f0np0 altname ens1f0np0 7: mlx5_slave2: mtu 9000 qdisc mq master mlx5_bond_roce state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff permaddr b8:59:9f:f0:7f:d1 altname enp132s0f1np1 altname ens1f1np1 8: mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.40.81/24 brd 172.31.40.255 scope global dynamic noprefixroute mlx5_bond_roce valid_lft 3387sec preferred_lft 3387sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever 9: mlx5_bond_ro.43@mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.43.81/24 brd 172.31.43.255 scope global dynamic noprefixroute mlx5_bond_ro.43 valid_lft 3387sec preferred_lft 3387sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever 10: mlx5_bond_ro.45@mlx5_bond_roce: mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether b8:59:9f:f0:7f:d0 brd ff:ff:ff:ff:ff:ff inet 172.31.45.81/24 brd 172.31.45.255 scope global dynamic noprefixroute mlx5_bond_ro.45 valid_lft 3387sec preferred_lft 3387sec inet6 fe80::ba59:9fff:fef0:7fd0/64 scope link noprefixroute valid_lft forever preferred_lft forever + [22-12-29 13:02:27] PY_FILE=/root/py_parse_out.txt + [22-12-29 13:02:27] rm -f /root/py_parse_out.txt + [22-12-29 13:02:27] export result=PASS + [22-12-29 13:02:27] result=PASS + [22-12-29 13:02:27] export TEST=/kernel/infiniband/env_setup/standalone + [22-12-29 13:02:27] TEST=/kernel/infiniband/env_setup/standalone + [22-12-29 13:02:27] ip address show + [22-12-29 13:02:27] grep 'inet 172.31.' + [22-12-29 13:02:27] [[ 0 -ne 0 ]] + [22-12-29 13:02:27] install_ca_trust + [22-12-29 13:02:27] local cert_path=/etc/pki/ca-trust/source/anchors + [22-12-29 13:02:27] '[' -f /etc/pki/ca-trust/source/anchors/legacy.crt -a -f /etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt ']' + [22-12-29 13:02:27] wget --no-check-certificate -P /etc/pki/ca-trust/source/anchors https://password.corp.redhat.com/legacy.crt --2022-12-29 13:02:27-- https://password.corp.redhat.com/legacy.crt Resolving password.corp.redhat.com (password.corp.redhat.com)... 10.2.111.154 Connecting to password.corp.redhat.com (password.corp.redhat.com)|10.2.111.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1338 (1.3K) Saving to: ‘/etc/pki/ca-trust/source/anchors/legacy.crt’ 0K . 100% 65.7M=0s 2022-12-29 13:02:28 (65.7 MB/s) - ‘/etc/pki/ca-trust/source/anchors/legacy.crt’ saved [1338/1338] + [22-12-29 13:02:28] wget --no-check-certificate -P /etc/pki/ca-trust/source/anchors https://password.corp.redhat.com/RH-IT-Root-CA.crt --2022-12-29 13:02:28-- https://password.corp.redhat.com/RH-IT-Root-CA.crt Resolving password.corp.redhat.com (password.corp.redhat.com)... 10.2.111.154 Connecting to password.corp.redhat.com (password.corp.redhat.com)|10.2.111.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1517 (1.5K) Saving to: ‘/etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt.1’ 0K . 100% 75.0M=0s 2022-12-29 13:02:28 (75.0 MB/s) - ‘/etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt.1’ saved [1517/1517] + [22-12-29 13:02:28] update-ca-trust enable + [22-12-29 13:02:29] update-ca-trust extract + [22-12-29 13:02:30] RQA_rhel8_brew_repo ++ [22-12-29 13:02:30] RQA_get_rhel_major ++ [22-12-29 13:02:30] awk -F . '{print $1}' +++ [22-12-29 13:02:30] RQA_get_rhel_release +++ [22-12-29 13:02:30] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:02:30] echo 9 + [22-12-29 13:02:30] local major=9 + [22-12-29 13:02:30] (( major == 8 )) + [22-12-29 13:02:30] pek2_lab_setup ++ [22-12-29 13:02:30] RQA_get_rdma_hostname ++ [22-12-29 13:02:30] hostname -A ++ [22-12-29 13:02:30] grep -o 'rdma.*' ++ [22-12-29 13:02:30] awk '{print $1}' ++ [22-12-29 13:02:30] awk -F . '{print $1}' + [22-12-29 13:02:30] [[ rdma-cki-01 == *\r\d\m\a\0* ]] + [22-12-29 13:02:30] common_setup ++ [22-12-29 13:02:30] RQA_get_rdma_hostname ++ [22-12-29 13:02:30] hostname -A ++ [22-12-29 13:02:30] grep -o 'rdma.*' ++ [22-12-29 13:02:30] awk '{print $1}' ++ [22-12-29 13:02:30] awk -F . '{print $1}' + [22-12-29 13:02:30] short_hostname=rdma-cki-01 + [22-12-29 13:02:30] RDMA_DRIVER= + [22-12-29 13:02:30] RDMA_NETWORK= + [22-12-29 13:02:30] '[' '!' -z ']' + [22-12-29 13:02:30] '[' '!' -z roce ']' + [22-12-29 13:02:30] RDMA_NETWORK=roce + [22-12-29 13:02:30] inv_file=json/rdma-cki-01.json + [22-12-29 13:02:30] '[' '!' -f json/rdma-cki-01.json ']' + [22-12-29 13:02:30] RQA_check_result -r 1 -t 'No inventory file for rdma-cki-01 exists' + [22-12-29 13:02:30] local test_pass=0 + [22-12-29 13:02:30] local test_skip=777 + [22-12-29 13:02:30] test 4 -gt 0 + [22-12-29 13:02:30] case $1 in + [22-12-29 13:02:30] local rc=1 + [22-12-29 13:02:30] shift + [22-12-29 13:02:30] shift + [22-12-29 13:02:30] test 2 -gt 0 + [22-12-29 13:02:30] case $1 in + [22-12-29 13:02:30] local 'msg=No inventory file for rdma-cki-01 exists' + [22-12-29 13:02:30] shift + [22-12-29 13:02:30] shift + [22-12-29 13:02:30] test 0 -gt 0 + [22-12-29 13:02:30] '[' -z 1 -o -z 'No inventory file for rdma-cki-01 exists' ']' + [22-12-29 13:02:30] '[' -z '' ']' + [22-12-29 13:02:30] source /etc/environment + [22-12-29 13:02:30] '[' -z '' ']' + [22-12-29 13:02:30] __RQA_create_test_result_file ++ [22-12-29 13:02:30] rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}' rdma-core + [22-12-29 13:02:30] local rdma_version=rdma-core-41.0-3.el9 ++ [22-12-29 13:02:30] echo /kernel/infiniband/env_setup/standalone ++ [22-12-29 13:02:30] sed -r 's/\/kernel\/infiniband\///;s/\/server//;s/\/client//;s/\/standalone//' + [22-12-29 13:02:31] TEST_NAME=env_setup + [22-12-29 13:02:31] [[ ! -z rdma-dev-21 ]] + [22-12-29 13:02:31] [[ ! -z rdma-cki-01 ]] + [22-12-29 13:02:31] hosts_list=rdma-dev-21/rdma-cki-01 + [22-12-29 13:02:31] local _bkrjb= ++ [22-12-29 13:02:31] grep JOBID= /etc/motd ++ [22-12-29 13:02:31] tr -d ' ' ++ [22-12-29 13:02:31] sed s/JOBID=/J:/g ++ [22-12-29 13:02:31] head -1 + [22-12-29 13:02:31] _bkrjb=J:7381428 + [22-12-29 13:02:31] [[ ! -z J:7381428 ]] + [22-12-29 13:02:31] _bkrjb=' & Beaker job J:7381428' ++ [22-12-29 13:02:31] echo results_env_setup.txt ++ [22-12-29 13:02:31] sed 's/\//-/g' + [22-12-29 13:02:31] local result_filename=results_env_setup.txt ++ [22-12-29 13:02:31] mktemp -d + [22-12-29 13:02:31] TEST_RESULT_FILE=/tmp/tmp.HlJeyFDMkE/results_env_setup.txt + [22-12-29 13:02:31] echo 'env_setup test results on rdma-dev-21/rdma-cki-01 & Beaker job J:7381428:' ++ [22-12-29 13:02:31] uname -r + [22-12-29 13:02:31] echo '5.14.0-226.1773_734386600.el9.x86_64, rdma-core-41.0-3.el9, , roce, & ' + [22-12-29 13:02:31] echo ' Result | Status | Test' + [22-12-29 13:02:31] echo ' ---------+--------+------------------------------------' + [22-12-29 13:02:31] sed -i /TEST_RESULT_FILE/d /etc/environment + [22-12-29 13:02:31] echo TEST_RESULT_FILE=/tmp/tmp.HlJeyFDMkE/results_env_setup.txt + [22-12-29 13:02:31] source /etc/environment ++ [22-12-29 13:02:31] TEST_RESULT_FILE=/tmp/tmp.HlJeyFDMkE/results_env_setup.txt + [22-12-29 13:02:31] '[' 1 -eq 0 ']' + [22-12-29 13:02:31] '[' 1 -eq 777 ']' + [22-12-29 13:02:31] local test_result=FAIL + [22-12-29 13:02:31] export result=FAIL + [22-12-29 13:02:31] result=FAIL + [22-12-29 13:02:31] [[ ! -z '' ]] + [22-12-29 13:02:31] printf '%10s | %6s | %s\n' FAIL 1 'No inventory file for rdma-cki-01 exists' + [22-12-29 13:02:31] set +x --- - TEST RESULT FOR env_setup - Test: No inventory file for rdma-cki-01 exists - Result: FAIL - Return: 1 --- + [22-12-29 13:02:31] return 1 + [22-12-29 13:02:31] standalone + [22-12-29 13:02:31] echo -en '\n# /kernel/infiniband/env_setup variables\n' + [22-12-29 13:02:31] [[ -s /root/py_parse_out.txt ]] + [22-12-29 13:02:31] echo 'export RDMA_IPV4=""' + [22-12-29 13:02:31] echo 'export RDMA_IPV6=""' + [22-12-29 13:02:31] source /root/.bashrc + [22-12-29 13:02:31] RQA_check_result -r 0 -t 'Source .bashrc after updates' + [22-12-29 13:02:31] local test_pass=0 + [22-12-29 13:02:31] local test_skip=777 + [22-12-29 13:02:31] test 4 -gt 0 + [22-12-29 13:02:31] case $1 in + [22-12-29 13:02:31] local rc=0 + [22-12-29 13:02:31] shift + [22-12-29 13:02:31] shift + [22-12-29 13:02:31] test 2 -gt 0 + [22-12-29 13:02:31] case $1 in + [22-12-29 13:02:31] local 'msg=Source .bashrc after updates' + [22-12-29 13:02:31] shift + [22-12-29 13:02:31] shift + [22-12-29 13:02:31] test 0 -gt 0 + [22-12-29 13:02:31] '[' -z 0 -o -z 'Source .bashrc after updates' ']' + [22-12-29 13:02:31] '[' -z /tmp/tmp.HlJeyFDMkE/results_env_setup.txt ']' + [22-12-29 13:02:31] '[' -z /tmp/tmp.HlJeyFDMkE/results_env_setup.txt ']' + [22-12-29 13:02:31] '[' 0 -eq 0 ']' + [22-12-29 13:02:31] local test_result=PASS + [22-12-29 13:02:31] printf '%10s | %6s | %s\n' PASS 0 'Source .bashrc after updates' + [22-12-29 13:02:31] set +x --- - TEST RESULT FOR env_setup - Test: Source .bashrc after updates - Result: PASS - Return: 0 --- + [22-12-29 13:02:31] RQA_overall_result + [22-12-29 13:02:31] source /root/known-issues.sh + [22-12-29 13:02:34] '[' -z /tmp/tmp.HlJeyFDMkE/results_env_setup.txt ']' + [22-12-29 13:02:34] '[' -z /tmp/tmp.HlJeyFDMkE/results_env_setup.txt ']' + [22-12-29 13:02:34] local overall_result=PASS + [22-12-29 13:02:34] echo 'Checking for failures and known issues:' + [22-12-29 13:02:34] grep FAIL /tmp/tmp.HlJeyFDMkE/results_env_setup.txt FAIL | 1 | No inventory file for rdma-cki-01 exists + [22-12-29 13:02:35] [[ ! -z '' ]] + [22-12-29 13:02:35] local driver= + [22-12-29 13:02:35] local hca= + [22-12-29 13:02:35] local network=roce ++ [22-12-29 13:02:35] RQA_get_rhel_major ++ [22-12-29 13:02:35] awk -F . '{print $1}' +++ [22-12-29 13:02:35] RQA_get_rhel_release +++ [22-12-29 13:02:35] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:02:35] echo 9 + [22-12-29 13:02:35] local major=9 ++ [22-12-29 13:02:35] RQA_get_rhel_minor ++ [22-12-29 13:02:35] awk -F . '{print $2}' +++ [22-12-29 13:02:35] RQA_get_rhel_release +++ [22-12-29 13:02:35] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:02:35] echo 9 + [22-12-29 13:02:35] local minor= ++ [22-12-29 13:02:35] mktemp + [22-12-29 13:02:35] local test_fail_file=/tmp/tmp.ztxacQoBbI + [22-12-29 13:02:35] grep FAIL /tmp/tmp.HlJeyFDMkE/results_env_setup.txt + [22-12-29 13:02:35] read line ++ [22-12-29 13:02:35] echo FAIL '|' 1 '|' No inventory file for rdma-cki-01 exists ++ [22-12-29 13:02:35] awk -F '|' '{print $3}' ++ [22-12-29 13:02:35] xargs + [22-12-29 13:02:35] local 'test_key=No inventory file for rdma-cki-01 exists' + [22-12-29 13:02:35] local value= + [22-12-29 13:02:35] '[' -z '' ']' + [22-12-29 13:02:35] overall_result=FAIL + [22-12-29 13:02:35] echo ' No inventory file for rdma-cki-01 exists is NOT a known issue on any environment - consider filing a BZ' + [22-12-29 13:02:35] continue + [22-12-29 13:02:35] read line + [22-12-29 13:02:35] [[ FAIL == \P\A\S\S ]] + [22-12-29 13:02:35] echo '' + [22-12-29 13:02:35] report_result /kernel/infiniband/env_setup/standalone FAIL 0 + [22-12-29 13:02:35] OUTPUTFILE=/mnt/testarea/tmp.BcDPZz + [22-12-29 13:02:35] rstrnt-report-result /kernel/infiniband/env_setup/standalone FAIL 0 ** /kernel/infiniband/env_setup/standalone FAIL Score:0 Uploading resultoutputfile.log .done + [22-12-29 13:02:35] rhts-submit-log -l /tmp/tmp.HlJeyFDMkE/results_env_setup.txt Uploading results_env_setup.txt .done + [22-12-29 13:02:35] rm /tmp/tmp.ztxacQoBbI + [22-12-29 13:02:35] __RQA_unset_test_result_file + [22-12-29 13:02:35] cat /tmp/tmp.HlJeyFDMkE/results_env_setup.txt env_setup test results on rdma-dev-21/rdma-cki-01 & Beaker job J:7381428: 5.14.0-226.1773_734386600.el9.x86_64, rdma-core-41.0-3.el9, , roce, & Result | Status | Test ---------+--------+------------------------------------ FAIL | 1 | No inventory file for rdma-cki-01 exists PASS | 0 | Source .bashrc after updates Checking for failures and known issues: No inventory file for rdma-cki-01 exists is NOT a known issue on any environment - consider filing a BZ + [22-12-29 13:02:35] export TEST_RESULT_FILE= + [22-12-29 13:02:35] TEST_RESULT_FILE= + [22-12-29 13:02:35] unset TEST_RESULT_FILE + [22-12-29 13:02:35] sed -i /TEST_RESULT_FILE/d /etc/environment + [22-12-29 13:02:35] echo 'TEST_RESULT_FILE=""' + [22-12-29 13:02:35] source /etc/environment ++ [22-12-29 13:02:35] TEST_RESULT_FILE= + [22-12-29 13:02:35] return 0 + [22-12-29 13:02:35] hostname -A + [22-12-29 13:02:35] grep rdma-cki-01 + [22-12-29 13:02:35] echo '------- client start test -------' ------- client start test ------- + [22-12-29 13:02:35] export result=PASS + [22-12-29 13:02:35] result=PASS + [22-12-29 13:02:35] export TEST=/kernel/infiniband/env_setup/standalone/client + [22-12-29 13:02:35] TEST=/kernel/infiniband/env_setup/standalone/client + [22-12-29 13:02:35] client + [22-12-29 13:02:35] [[ '' == \s\i\w ]] + [22-12-29 13:02:35] rhts_sync_set -s ES_CLIENT_READY_-roce-0 /usr/bin/rhts_sync_set -s ES_CLIENT_READY_-roce-0 -- State set = 12_ES_CLIENT_READY_-roce-0 + [22-12-29 13:02:35] rhts_sync_block -s ES_SERVER_READY_-roce-0 rdma-dev-21 /usr/bin/rhts_sync_block -s ES_SERVER_READY_-roce-0 rdma-dev-21 -- Blocking state(s) = 12_ES_SERVER_READY_-roce-0 Failed to connect to remote server rdma-dev-21: Connection refused Retrying rstrnt-sync-block for state 12_ES_SERVER_READY_-roce-0 + [22-12-29 13:03:35] rhts_sync_block -s ES_SERVER_DONE_-roce-0 rdma-dev-21 /usr/bin/rhts_sync_block -s ES_SERVER_DONE_-roce-0 rdma-dev-21 -- Blocking state(s) = 12_ES_SERVER_DONE_-roce-0 ++ [22-12-29 13:03:35] echo rdma-dev-21 ++ [22-12-29 13:03:35] awk -F . '{print $1}' + [22-12-29 13:03:35] srv_ssh=rdma-dev-21 ++ [22-12-29 13:03:35] ssh rdma-dev-21 echo '${RDMA_DRIVER}' + [22-12-29 13:03:36] SERVER_DRIVER=mlx5 ++ [22-12-29 13:03:36] ssh rdma-dev-21 echo '${RDMA_NETWORK}' + [22-12-29 13:03:36] SERVER_NETWORK=roce ++ [22-12-29 13:03:36] ssh rdma-dev-21 echo '${HCA_ID}' + [22-12-29 13:03:36] SERVER_HCA_ID=mlx5_0 ++ [22-12-29 13:03:36] ssh rdma-dev-21 echo '${HCA_ABRV}' + [22-12-29 13:03:37] SERVER_HCA_ABRV=connectx4 ++ [22-12-29 13:03:37] ssh rdma-dev-21 echo '${DEVICE_ID}' + [22-12-29 13:03:37] SERVER_DEVICE_ID=mlx5_roce ++ [22-12-29 13:03:37] ssh rdma-dev-21 echo '${DEVICE_MAC}' + [22-12-29 13:03:37] SERVER_DEVICE_MAC=24:8a:07:4b:f0:94 ++ [22-12-29 13:03:37] ssh rdma-dev-21 echo '${DEVICE_PORT}' + [22-12-29 13:03:37] SERVER_DEVICE_PORT=1 ++ [22-12-29 13:03:37] ssh rdma-dev-21 echo '${RDMA_IPV4}' + [22-12-29 13:03:38] SERVER_IPV4=172.31.40.121 ++ [22-12-29 13:03:38] ssh rdma-dev-21 echo '${RDMA_IPV6}' + [22-12-29 13:03:38] SERVER_IPV6=fe80::268a:7ff:fe4b:f094 + [22-12-29 13:03:38] '[' -z mlx5 ']' + [22-12-29 13:03:38] '[' -z roce ']' + [22-12-29 13:03:38] '[' -z mlx5_0 ']' + [22-12-29 13:03:38] '[' -z connectx4 ']' + [22-12-29 13:03:38] '[' -z mlx5_roce ']' + [22-12-29 13:03:38] '[' -z 24:8a:07:4b:f0:94 ']' + [22-12-29 13:03:38] '[' -z 1 ']' + [22-12-29 13:03:38] '[' -z 172.31.40.121 ']' + [22-12-29 13:03:38] '[' -z fe80::268a:7ff:fe4b:f094 ']' + [22-12-29 13:03:38] echo 'export CLIENT_DRIVER=""' + [22-12-29 13:03:38] echo 'export CLIENT_NETWORK="roce"' + [22-12-29 13:03:38] echo 'export CLIENT_HCA_ID=""' + [22-12-29 13:03:38] echo 'export CLIENT_HCA_ABRV=""' + [22-12-29 13:03:38] echo 'export CLIENT_DEVICE_ID=""' + [22-12-29 13:03:38] echo 'export CLIENT_DEVICE_MAC=""' + [22-12-29 13:03:38] echo 'export CLIENT_DEVICE_PORT=""' + [22-12-29 13:03:38] echo 'export CLIENT_IPV4=""' + [22-12-29 13:03:38] echo 'export CLIENT_IPV6=""' + [22-12-29 13:03:38] echo 'export SERVER_DRIVER="mlx5"' + [22-12-29 13:03:38] echo 'export SERVER_NETWORK="roce"' + [22-12-29 13:03:38] echo 'export SERVER_HCA_ID="mlx5_0"' + [22-12-29 13:03:38] echo 'export SERVER_HCA_ABRV="connectx4"' + [22-12-29 13:03:38] echo 'export SERVER_DEVICE_ID="mlx5_roce"' + [22-12-29 13:03:38] echo 'export SERVER_DEVICE_MAC="24:8a:07:4b:f0:94"' + [22-12-29 13:03:38] echo 'export SERVER_DEVICE_PORT="1"' + [22-12-29 13:03:38] echo 'export SERVER_IPV4="172.31.40.121"' + [22-12-29 13:03:38] echo 'export SERVER_IPV6="fe80::268a:7ff:fe4b:f094"' + [22-12-29 13:03:38] echo 'export REMOTE_HOST="rdma-dev-21"' + [22-12-29 13:03:38] source /root/.bashrc + [22-12-29 13:03:38] RQA_check_result -r 0 -t 'source .bashrc in client function' + [22-12-29 13:03:38] local test_pass=0 + [22-12-29 13:03:38] local test_skip=777 + [22-12-29 13:03:38] test 4 -gt 0 + [22-12-29 13:03:38] case $1 in + [22-12-29 13:03:38] local rc=0 + [22-12-29 13:03:38] shift + [22-12-29 13:03:38] shift + [22-12-29 13:03:38] test 2 -gt 0 + [22-12-29 13:03:38] case $1 in + [22-12-29 13:03:38] local 'msg=source .bashrc in client function' + [22-12-29 13:03:38] shift + [22-12-29 13:03:38] shift + [22-12-29 13:03:38] test 0 -gt 0 + [22-12-29 13:03:38] '[' -z 0 -o -z 'source .bashrc in client function' ']' + [22-12-29 13:03:38] '[' -z '' ']' + [22-12-29 13:03:38] source /etc/environment ++ [22-12-29 13:03:38] TEST_RESULT_FILE= + [22-12-29 13:03:38] '[' -z '' ']' + [22-12-29 13:03:38] __RQA_create_test_result_file ++ [22-12-29 13:03:38] rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}' rdma-core + [22-12-29 13:03:38] local rdma_version=rdma-core-41.0-3.el9 ++ [22-12-29 13:03:38] echo /kernel/infiniband/env_setup/standalone/client ++ [22-12-29 13:03:38] sed -r 's/\/kernel\/infiniband\///;s/\/server//;s/\/client//;s/\/standalone//' + [22-12-29 13:03:38] TEST_NAME=env_setup + [22-12-29 13:03:38] [[ ! -z rdma-dev-21 ]] + [22-12-29 13:03:38] [[ ! -z rdma-cki-01 ]] + [22-12-29 13:03:38] hosts_list=rdma-dev-21/rdma-cki-01 + [22-12-29 13:03:38] local _bkrjb= ++ [22-12-29 13:03:38] grep JOBID= /etc/motd ++ [22-12-29 13:03:38] tr -d ' ' ++ [22-12-29 13:03:38] sed s/JOBID=/J:/g ++ [22-12-29 13:03:38] head -1 + [22-12-29 13:03:38] _bkrjb=J:7381428 + [22-12-29 13:03:38] [[ ! -z J:7381428 ]] + [22-12-29 13:03:38] _bkrjb=' & Beaker job J:7381428' ++ [22-12-29 13:03:38] echo results_env_setup.txt ++ [22-12-29 13:03:38] sed 's/\//-/g' + [22-12-29 13:03:38] local result_filename=results_env_setup.txt ++ [22-12-29 13:03:38] mktemp -d + [22-12-29 13:03:38] TEST_RESULT_FILE=/tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt + [22-12-29 13:03:38] echo 'env_setup test results on rdma-dev-21/rdma-cki-01 & Beaker job J:7381428:' ++ [22-12-29 13:03:38] uname -r + [22-12-29 13:03:38] echo '5.14.0-226.1773_734386600.el9.x86_64, rdma-core-41.0-3.el9, , roce, & ' + [22-12-29 13:03:38] echo ' Result | Status | Test' + [22-12-29 13:03:38] echo ' ---------+--------+------------------------------------' + [22-12-29 13:03:38] sed -i /TEST_RESULT_FILE/d /etc/environment + [22-12-29 13:03:38] echo TEST_RESULT_FILE=/tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt + [22-12-29 13:03:38] source /etc/environment ++ [22-12-29 13:03:38] TEST_RESULT_FILE=/tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt + [22-12-29 13:03:38] '[' 0 -eq 0 ']' + [22-12-29 13:03:38] local test_result=PASS + [22-12-29 13:03:38] printf '%10s | %6s | %s\n' PASS 0 'source .bashrc in client function' + [22-12-29 13:03:38] set +x --- - TEST RESULT FOR env_setup - Test: source .bashrc in client function - Result: PASS - Return: 0 --- + [22-12-29 13:03:38] echo '--- client finishes.' --- client finishes. + [22-12-29 13:03:38] rhts_sync_set -s ES_CLIENT_DONE_-roce-0 /usr/bin/rhts_sync_set -s ES_CLIENT_DONE_-roce-0 -- State set = 12_ES_CLIENT_DONE_-roce-0 + [22-12-29 13:03:38] RQA_overall_result + [22-12-29 13:03:38] source /root/known-issues.sh + [22-12-29 13:03:42] '[' -z /tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt ']' + [22-12-29 13:03:42] '[' -z /tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt ']' + [22-12-29 13:03:42] local overall_result=PASS + [22-12-29 13:03:42] echo 'Checking for failures and known issues:' + [22-12-29 13:03:42] grep FAIL /tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt + [22-12-29 13:03:42] echo ' no test failures' + [22-12-29 13:03:42] echo '' + [22-12-29 13:03:42] report_result /kernel/infiniband/env_setup/standalone/client PASS 0 + [22-12-29 13:03:42] OUTPUTFILE=/mnt/testarea/tmp.BcDPZz + [22-12-29 13:03:42] rstrnt-report-result /kernel/infiniband/env_setup/standalone/client PASS 0 ** /kernel/infiniband/env_setup/standalone/client PASS Score:0 Uploading resultoutputfile.log .done + [22-12-29 13:03:43] rhts-submit-log -l /tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt Uploading results_env_setup.txt .done + [22-12-29 13:03:43] __RQA_unset_test_result_file + [22-12-29 13:03:43] cat /tmp/tmp.yZ4Rt2BE0x/results_env_setup.txt env_setup test results on rdma-dev-21/rdma-cki-01 & Beaker job J:7381428: 5.14.0-226.1773_734386600.el9.x86_64, rdma-core-41.0-3.el9, , roce, & Result | Status | Test ---------+--------+------------------------------------ PASS | 0 | source .bashrc in client function Checking for failures and known issues: no test failures + [22-12-29 13:03:43] export TEST_RESULT_FILE= + [22-12-29 13:03:43] TEST_RESULT_FILE= + [22-12-29 13:03:43] unset TEST_RESULT_FILE + [22-12-29 13:03:43] sed -i /TEST_RESULT_FILE/d /etc/environment + [22-12-29 13:03:43] echo 'TEST_RESULT_FILE=""' + [22-12-29 13:03:43] source /etc/environment ++ [22-12-29 13:03:43] TEST_RESULT_FILE= + [22-12-29 13:03:43] return 0 + [22-12-29 13:03:43] hostname -A + [22-12-29 13:03:43] grep rdma-dev-21 + [22-12-29 13:03:43] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:03:43] rm -f + [22-12-29 13:03:43] unset RUN_NUMBER TNAME + [22-12-29 13:03:43] echo ' ------ end of runtest.sh.' ------ end of runtest.sh. + [22-12-29 13:03:43] exit 0 make[1]: Leaving directory '/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.zip/infiniband/env_setup' + [22-12-29 13:03:43] echo 'OKAY - task env_setup is done!' OKAY - task env_setup is done! + [22-12-29 13:03:43] bash -x ./tier1.sh common + trap 'exit 1' SIGHUP SIGINT SIGQUIT SIGTERM + source /root/rdma-qa.sh ++ PS4='$(date +"+ [%y-%m-%d %H:%M:%S]") ' ++ [22-12-29 13:03:43] export NIC_PREF= ++ [22-12-29 13:03:43] NIC_PREF= ++ [22-12-29 13:03:43] export PERFTEST_FLAGS= ++ [22-12-29 13:03:43] PERFTEST_FLAGS= +++ [22-12-29 13:03:43] hostname -s ++ [22-12-29 13:03:43] export s_hostname=rdma-cki-01 ++ [22-12-29 13:03:43] s_hostname=rdma-cki-01 ++ [22-12-29 13:03:43] RQA_set_pyexec ++ [22-12-29 13:03:43] PYEXEC= ++ [22-12-29 13:03:43] which python ++ [22-12-29 13:03:43] export PYEXEC=python ++ [22-12-29 13:03:43] PYEXEC=python ++ [22-12-29 13:03:43] return 0 +++ [22-12-29 13:03:43] grep -i fedora /etc/redhat-release ++ [22-12-29 13:03:43] [[ -n '' ]] +++ [22-12-29 13:03:43] RQA_get_rhel_major +++ [22-12-29 13:03:43] awk -F . '{print $1}' ++++ [22-12-29 13:03:43] RQA_get_rhel_release ++++ [22-12-29 13:03:43] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:43] echo 9 ++ [22-12-29 13:03:43] [[ 9 -ge 8 ]] ++ [22-12-29 13:03:43] export 'PKGINSTALL=dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:03:43] PKGINSTALL='dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:03:43] export 'PKGREMOVE=dnf remove --noautoremove -y' ++ [22-12-29 13:03:43] PKGREMOVE='dnf remove --noautoremove -y' ++ [22-12-29 13:03:43] [[ rdma-cki-01 == \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:03:43] export LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:03:43] LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:03:43] '[' -f /root/known-issues.sh ']' ++ [22-12-29 13:03:43] source /root/known-issues.sh ++ [22-12-29 13:03:47] RQA_load_tech_preview_modules ++ [22-12-29 13:03:47] [[ ! -e /etc/rdma/rdma.conf ]] ++ [22-12-29 13:03:47] return ++ [22-12-29 13:03:47] RQA_fix_uefi_reboot ++ [22-12-29 13:03:47] ls -l /usr/sbin/reboot ++ [22-12-29 13:03:47] grep rhts-reboot ++ [22-12-29 13:03:47] which rhts-reboot ++ [22-12-29 13:03:47] rpm -q rhts-test-env ++ [22-12-29 13:03:47] RQA_fix_ssh_ipv6_auth +++ [22-12-29 13:03:47] RQA_get_rhel_major +++ [22-12-29 13:03:47] awk -F . '{print $1}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] '[' 9 -ge 7 ']' ++ [22-12-29 13:03:47] local need_restart=0 ++ [22-12-29 13:03:47] grep -E '^UseDNS yes' /etc/ssh/sshd_config ++ [22-12-29 13:03:47] grep -E '^GSSAPIStrictAcceptorCheck yes' /etc/ssh/sshd_config ++ [22-12-29 13:03:47] grep 'AddressFamily inet' /root/.ssh/config ++ [22-12-29 13:03:47] '[' 0 -eq 1 ']' ++ [22-12-29 13:03:47] RQA_fix_mlx5_ib_connected_mode +++ [22-12-29 13:03:47] RQA_get_rhel_major +++ [22-12-29 13:03:47] awk -F . '{print $1}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] [[ 9 -ge 7 ]] +++ [22-12-29 13:03:47] RQA_get_rhel_minor +++ [22-12-29 13:03:47] awk -F . '{print $2}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] [[ '' -ge 5 ]] +++ [22-12-29 13:03:47] RQA_get_rhel_major +++ [22-12-29 13:03:47] awk -F . '{print $1}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] [[ 9 -gt 7 ]] +++ [22-12-29 13:03:47] ls /etc/sysconfig/network-scripts/ +++ [22-12-29 13:03:47] grep mlx5 +++ [22-12-29 13:03:47] grep ib +++ [22-12-29 13:03:47] grep -v '~' ++ [22-12-29 13:03:47] RQA_nfs_rhel8_temp_fix +++ [22-12-29 13:03:47] RQA_get_rhel_major +++ [22-12-29 13:03:47] awk -F . '{print $1}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] (( 9 == 8 )) ++ [22-12-29 13:03:47] source /root/.bashrc ++ [22-12-29 13:03:47] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:03:47] source /root/mpi-functions.sh ++ [22-12-29 13:03:47] export 'MPI_TIMEOUT=timeout --preserve-status --kill-after=5m 3m' ++ [22-12-29 13:03:47] MPI_TIMEOUT='timeout --preserve-status --kill-after=5m 3m' + [22-12-29 13:03:47] source /root/rdma-qa.sh ++ [22-12-29 13:03:47] PS4='$(date +"+ [%y-%m-%d %H:%M:%S]") ' ++ [22-12-29 13:03:47] export NIC_PREF= ++ [22-12-29 13:03:47] NIC_PREF= ++ [22-12-29 13:03:47] export PERFTEST_FLAGS= ++ [22-12-29 13:03:47] PERFTEST_FLAGS= +++ [22-12-29 13:03:47] hostname -s ++ [22-12-29 13:03:47] export s_hostname=rdma-cki-01 ++ [22-12-29 13:03:47] s_hostname=rdma-cki-01 ++ [22-12-29 13:03:47] RQA_set_pyexec ++ [22-12-29 13:03:47] PYEXEC= ++ [22-12-29 13:03:47] which python ++ [22-12-29 13:03:47] export PYEXEC=python ++ [22-12-29 13:03:47] PYEXEC=python ++ [22-12-29 13:03:47] return 0 +++ [22-12-29 13:03:47] grep -i fedora /etc/redhat-release ++ [22-12-29 13:03:47] [[ -n '' ]] +++ [22-12-29 13:03:47] RQA_get_rhel_major +++ [22-12-29 13:03:47] awk -F . '{print $1}' ++++ [22-12-29 13:03:47] RQA_get_rhel_release ++++ [22-12-29 13:03:47] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:47] echo 9 ++ [22-12-29 13:03:47] [[ 9 -ge 8 ]] ++ [22-12-29 13:03:47] export 'PKGINSTALL=dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:03:47] PKGINSTALL='dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:03:47] export 'PKGREMOVE=dnf remove --noautoremove -y' ++ [22-12-29 13:03:47] PKGREMOVE='dnf remove --noautoremove -y' ++ [22-12-29 13:03:47] [[ rdma-cki-01 == \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:03:47] export LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:03:47] LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:03:47] '[' -f /root/known-issues.sh ']' ++ [22-12-29 13:03:47] source /root/known-issues.sh ++ [22-12-29 13:03:52] RQA_load_tech_preview_modules ++ [22-12-29 13:03:52] [[ ! -e /etc/rdma/rdma.conf ]] ++ [22-12-29 13:03:52] return ++ [22-12-29 13:03:52] RQA_fix_uefi_reboot ++ [22-12-29 13:03:52] ls -l /usr/sbin/reboot ++ [22-12-29 13:03:52] grep rhts-reboot ++ [22-12-29 13:03:52] which rhts-reboot ++ [22-12-29 13:03:52] rpm -q rhts-test-env ++ [22-12-29 13:03:52] RQA_fix_ssh_ipv6_auth +++ [22-12-29 13:03:52] RQA_get_rhel_major +++ [22-12-29 13:03:52] awk -F . '{print $1}' ++++ [22-12-29 13:03:52] RQA_get_rhel_release ++++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:52] echo 9 ++ [22-12-29 13:03:52] '[' 9 -ge 7 ']' ++ [22-12-29 13:03:52] local need_restart=0 ++ [22-12-29 13:03:52] grep -E '^UseDNS yes' /etc/ssh/sshd_config ++ [22-12-29 13:03:52] grep -E '^GSSAPIStrictAcceptorCheck yes' /etc/ssh/sshd_config ++ [22-12-29 13:03:52] grep 'AddressFamily inet' /root/.ssh/config ++ [22-12-29 13:03:52] '[' 0 -eq 1 ']' ++ [22-12-29 13:03:52] RQA_fix_mlx5_ib_connected_mode +++ [22-12-29 13:03:52] RQA_get_rhel_major +++ [22-12-29 13:03:52] awk -F . '{print $1}' ++++ [22-12-29 13:03:52] RQA_get_rhel_release ++++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:52] echo 9 ++ [22-12-29 13:03:52] [[ 9 -ge 7 ]] +++ [22-12-29 13:03:52] RQA_get_rhel_minor +++ [22-12-29 13:03:52] awk -F . '{print $2}' ++++ [22-12-29 13:03:52] RQA_get_rhel_release ++++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:52] echo 9 ++ [22-12-29 13:03:52] [[ '' -ge 5 ]] +++ [22-12-29 13:03:52] RQA_get_rhel_major +++ [22-12-29 13:03:52] awk -F . '{print $1}' ++++ [22-12-29 13:03:52] RQA_get_rhel_release ++++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:52] echo 9 ++ [22-12-29 13:03:52] [[ 9 -gt 7 ]] +++ [22-12-29 13:03:52] ls /etc/sysconfig/network-scripts/ +++ [22-12-29 13:03:52] grep mlx5 +++ [22-12-29 13:03:52] grep ib +++ [22-12-29 13:03:52] grep -v '~' ++ [22-12-29 13:03:52] RQA_nfs_rhel8_temp_fix +++ [22-12-29 13:03:52] RQA_get_rhel_major +++ [22-12-29 13:03:52] awk -F . '{print $1}' ++++ [22-12-29 13:03:52] RQA_get_rhel_release ++++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:03:52] echo 9 ++ [22-12-29 13:03:52] (( 9 == 8 )) ++ [22-12-29 13:03:52] source /root/.bashrc ++ [22-12-29 13:03:52] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:03:52] source /root/mpi-functions.sh ++ [22-12-29 13:03:52] export 'MPI_TIMEOUT=timeout --preserve-status --kill-after=5m 3m' ++ [22-12-29 13:03:52] MPI_TIMEOUT='timeout --preserve-status --kill-after=5m 3m' + [22-12-29 13:03:52] RQA_set_inline_default + [22-12-29 13:03:52] [[ '' != *\m\l\x\4* ]] + [22-12-29 13:03:52] mkdir -p /etc/rdma/rsocket + [22-12-29 13:03:52] echo 0 + [22-12-29 13:03:52] RQA_set_perftest_flags + [22-12-29 13:03:52] PERFTEST_FLAGS='-d -i -F' + [22-12-29 13:03:52] [[ '' == \m\l\x\4 ]] + [22-12-29 13:03:52] PERFTEST_FLAGS='-d -i -F -R' + [22-12-29 13:03:52] export PERFTEST_FLAGS + [22-12-29 13:03:52] MPI_PKG=openmpi + [22-12-29 13:03:52] TEST_PKG=mpitests-openmpi ++ [22-12-29 13:03:52] RQA_get_rhel_major ++ [22-12-29 13:03:52] awk -F . '{print $1}' +++ [22-12-29 13:03:52] RQA_get_rhel_release +++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:03:52] echo 9 + [22-12-29 13:03:52] [[ 9 -eq 6 ]] ++ [22-12-29 13:03:52] RQA_get_rhel_major ++ [22-12-29 13:03:52] awk -F . '{print $1}' +++ [22-12-29 13:03:52] RQA_get_rhel_release +++ [22-12-29 13:03:52] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:03:52] echo 9 + [22-12-29 13:03:52] [[ 9 -eq 7 ]] + [22-12-29 13:03:52] MPI_setup -m openmpi -t mpitests-openmpi + [22-12-29 13:03:52] test 4 -gt 0 + [22-12-29 13:03:52] case $1 in + [22-12-29 13:03:52] local mpi_pkg=openmpi + [22-12-29 13:03:52] shift + [22-12-29 13:03:52] shift + [22-12-29 13:03:52] test 2 -gt 0 + [22-12-29 13:03:52] case $1 in + [22-12-29 13:03:52] local test_pkg=mpitests-openmpi + [22-12-29 13:03:52] shift + [22-12-29 13:03:52] shift + [22-12-29 13:03:52] test 0 -gt 0 + [22-12-29 13:03:52] '[' -z openmpi -o -z mpitests-openmpi ']' + [22-12-29 13:03:52] rpm -q environment-modules environment-modules-5.0.1-2.el9.x86_64 + [22-12-29 13:03:52] '[' -f /etc/profile.d/modules.sh ']' + [22-12-29 13:03:52] source /etc/profile.d/modules.sh ++ [22-12-29 13:03:52] '[' -n /usr/bin/bash ']' ++ [22-12-29 13:03:52] '[' bash = sh ']' ++ [22-12-29 13:03:52] shell=bash ++ [22-12-29 13:03:52] '[' -f /usr/share/Modules/init/bash ']' ++ [22-12-29 13:03:52] . /usr/share/Modules/init/bash +++ [22-12-29 13:03:52] unset _mlshdbg +++ [22-12-29 13:03:52] '[' 0 = 1 ']' +++ [22-12-29 13:03:52] export 'MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:03:52] MODULES_RUN_QUARANTINE='LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:03:52] unset _mlre _mlIFS +++ [22-12-29 13:03:52] '[' -n x ']' +++ [22-12-29 13:03:52] _mlIFS=' ' +++ [22-12-29 13:03:52] IFS=' ' +++ [22-12-29 13:03:52] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:03:52] '[' LD_LIBRARY_PATH = LD_LIBRARY_PATH -a LD_LIBRARY_PATH = LD_LIBRARY_PATH ']' ++++ [22-12-29 13:03:52] eval 'echo ${LD_LIBRARY_PATH+x}' +++++ [22-12-29 13:03:52] echo +++ [22-12-29 13:03:52] '[' -n '' ']' +++ [22-12-29 13:03:52] _mlrv=MODULES_RUNENV_LD_LIBRARY_PATH ++++ [22-12-29 13:03:52] eval 'echo ${MODULES_RUNENV_LD_LIBRARY_PATH:-}' +++++ [22-12-29 13:03:52] echo +++ [22-12-29 13:03:52] _mlre='LD_LIBRARY_PATH='\'''\'' ' +++ [22-12-29 13:03:52] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:03:52] '[' LD_PRELOAD = LD_PRELOAD -a LD_PRELOAD = LD_PRELOAD ']' ++++ [22-12-29 13:03:52] eval 'echo ${LD_PRELOAD+x}' +++++ [22-12-29 13:03:52] echo +++ [22-12-29 13:03:52] '[' -n '' ']' +++ [22-12-29 13:03:52] _mlrv=MODULES_RUNENV_LD_PRELOAD ++++ [22-12-29 13:03:52] eval 'echo ${MODULES_RUNENV_LD_PRELOAD:-}' +++++ [22-12-29 13:03:52] echo +++ [22-12-29 13:03:52] _mlre='LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' +++ [22-12-29 13:03:52] '[' -n 'LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' ']' +++ [22-12-29 13:03:52] _mlre='eval LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' __MODULES_QUARANTINE_SET=1 ' ++++ [22-12-29 13:03:52] eval 'LD_LIBRARY_PATH='\'''\''' 'LD_PRELOAD='\'''\''' __MODULES_QUARANTINE_SET=1 /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++++ [22-12-29 13:03:52] LD_LIBRARY_PATH= +++++ [22-12-29 13:03:52] LD_PRELOAD= +++++ [22-12-29 13:03:52] __MODULES_QUARANTINE_SET=1 +++++ [22-12-29 13:03:52] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++ [22-12-29 13:03:52] _mlcode='module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' +++ [22-12-29 13:03:52] _mlret=0 +++ [22-12-29 13:03:52] '[' -n x ']' +++ [22-12-29 13:03:52] IFS=' ' +++ [22-12-29 13:03:52] unset _mlIFS +++ [22-12-29 13:03:52] unset _mlre _mlv _mlrv +++ [22-12-29 13:03:52] '[' 0 -eq 0 ']' +++ [22-12-29 13:03:52] eval 'module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' ++++ [22-12-29 13:03:52] MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl ++++ [22-12-29 13:03:52] export MODULES_CMD ++++ [22-12-29 13:03:52] unset __MODULES_AUTOINIT_INPROGRESS ++++ [22-12-29 13:03:52] MODULESHOME=/usr/share/Modules ++++ [22-12-29 13:03:52] export MODULESHOME ++++ [22-12-29 13:03:52] test 0 +++ [22-12-29 13:03:52] '[' -t 2 ']' +++ [22-12-29 13:03:52] export -f module ++++ [22-12-29 13:03:52] type -t ml +++ [22-12-29 13:03:52] '[' function = function ']' +++ [22-12-29 13:03:52] export -f ml +++ [22-12-29 13:03:52] '[' 5 -ge 3 ']' +++ [22-12-29 13:03:52] [[ hxB =~ i ]] +++ [22-12-29 13:03:52] [[ ! :/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin: =~ :/usr/share/Modules/bin: ]] +++ [22-12-29 13:03:52] '[' '!' -n x ']' +++ [22-12-29 13:03:52] [[ ! :/usr/share/man:: =~ :/usr/share/man: ]] +++ [22-12-29 13:03:52] unset _mlcode _mlret +++ [22-12-29 13:03:52] '[' -n '' ']' + [22-12-29 13:03:52] type module module is a function module () { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus } + [22-12-29 13:03:52] module purge ++ [22-12-29 13:03:52] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash purge + [22-12-29 13:03:52] eval + [22-12-29 13:03:52] _mlstatus=0 + [22-12-29 13:03:52] return 0 + [22-12-29 13:03:52] sed -e '/^module/d' -i /root/.bashrc + [22-12-29 13:03:52] dnf remove --noautoremove -y 'mpitests*' 'mpich*' 'mvapich*' 'openmpi*' No match for argument: mpitests* No match for argument: mpich* Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Removing: mvapich2 x86_64 2.3.6-3.el9 @beaker-AppStream 15 M mvapich2-devel x86_64 2.3.6-3.el9 @beaker-AppStream 28 M openmpi x86_64 1:4.1.1-5.el9 @beaker-AppStream 10 M openmpi-devel x86_64 1:4.1.1-5.el9 @beaker-AppStream 1.9 M Transaction Summary ================================================================================ Remove 4 Packages Freed space: 55 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : mvapich2-devel-2.3.6-3.el9.x86_64 1/4 Erasing : openmpi-1:4.1.1-5.el9.x86_64 2/4 Erasing : openmpi-devel-1:4.1.1-5.el9.x86_64 3/4 Erasing : mvapich2-2.3.6-3.el9.x86_64 4/4 Running scriptlet: mvapich2-2.3.6-3.el9.x86_64 4/4 Verifying : mvapich2-2.3.6-3.el9.x86_64 1/4 Verifying : mvapich2-devel-2.3.6-3.el9.x86_64 2/4 Verifying : openmpi-1:4.1.1-5.el9.x86_64 3/4 Verifying : openmpi-devel-1:4.1.1-5.el9.x86_64 4/4 Removed: mvapich2-2.3.6-3.el9.x86_64 mvapich2-devel-2.3.6-3.el9.x86_64 openmpi-1:4.1.1-5.el9.x86_64 openmpi-devel-1:4.1.1-5.el9.x86_64 Complete! + [22-12-29 13:03:56] dnf install -y --setopt=strict=0 --nogpgcheck openmpi mpitests-openmpi environment-modules Last metadata expiration check: 0:11:58 ago on Thu 29 Dec 2022 12:51:59 PM EST. Package environment-modules-5.0.1-2.el9.x86_64 is already installed. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: mpitests-openmpi x86_64 5.8-1.el9 BUILDROOT-C9S 897 k openmpi x86_64 1:4.1.1-5.el9 BUILDROOT-C9S 2.7 M Installing dependencies: openmpi-devel x86_64 1:4.1.1-5.el9 BUILDROOT-C9S 1.2 M Transaction Summary ================================================================================ Install 3 Packages Total download size: 4.7 M Installed size: 17 M Downloading Packages: (1/3): openmpi-devel-4.1.1-5.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 (2/3): mpitests-openmpi-5.8-1.el9.x86_64.rpm 5.7 MB/s | 897 kB 00:00 (3/3): openmpi-4.1.1-5.el9.x86_64.rpm 8.5 MB/s | 2.7 MB 00:00 -------------------------------------------------------------------------------- Total 15 MB/s | 4.7 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : openmpi-devel-1:4.1.1-5.el9.x86_64 1/3 Installing : openmpi-1:4.1.1-5.el9.x86_64 2/3 Installing : mpitests-openmpi-5.8-1.el9.x86_64 3/3 Running scriptlet: mpitests-openmpi-5.8-1.el9.x86_64 3/3 Verifying : mpitests-openmpi-5.8-1.el9.x86_64 1/3 Verifying : openmpi-1:4.1.1-5.el9.x86_64 2/3 Verifying : openmpi-devel-1:4.1.1-5.el9.x86_64 3/3 Installed: mpitests-openmpi-5.8-1.el9.x86_64 openmpi-1:4.1.1-5.el9.x86_64 openmpi-devel-1:4.1.1-5.el9.x86_64 Complete! + [22-12-29 13:04:03] '[' -f /etc/profile.d/modules.sh ']' + [22-12-29 13:04:03] source /etc/profile.d/modules.sh ++ [22-12-29 13:04:03] '[' -n /usr/bin/bash ']' ++ [22-12-29 13:04:03] '[' bash = sh ']' ++ [22-12-29 13:04:03] shell=bash ++ [22-12-29 13:04:03] '[' -f /usr/share/Modules/init/bash ']' ++ [22-12-29 13:04:03] . /usr/share/Modules/init/bash +++ [22-12-29 13:04:03] unset _mlshdbg +++ [22-12-29 13:04:03] '[' 0 = 1 ']' +++ [22-12-29 13:04:03] export 'MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:04:03] MODULES_RUN_QUARANTINE='LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:04:03] unset _mlre _mlIFS +++ [22-12-29 13:04:03] '[' -n x ']' +++ [22-12-29 13:04:03] _mlIFS=' ' +++ [22-12-29 13:04:03] IFS=' ' +++ [22-12-29 13:04:03] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:04:03] '[' LD_LIBRARY_PATH = LD_LIBRARY_PATH -a LD_LIBRARY_PATH = LD_LIBRARY_PATH ']' ++++ [22-12-29 13:04:03] eval 'echo ${LD_LIBRARY_PATH+x}' +++++ [22-12-29 13:04:03] echo +++ [22-12-29 13:04:03] '[' -n '' ']' +++ [22-12-29 13:04:03] _mlrv=MODULES_RUNENV_LD_LIBRARY_PATH ++++ [22-12-29 13:04:03] eval 'echo ${MODULES_RUNENV_LD_LIBRARY_PATH:-}' +++++ [22-12-29 13:04:03] echo +++ [22-12-29 13:04:03] _mlre='LD_LIBRARY_PATH='\'''\'' ' +++ [22-12-29 13:04:03] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:04:03] '[' LD_PRELOAD = LD_PRELOAD -a LD_PRELOAD = LD_PRELOAD ']' ++++ [22-12-29 13:04:03] eval 'echo ${LD_PRELOAD+x}' +++++ [22-12-29 13:04:03] echo +++ [22-12-29 13:04:03] '[' -n '' ']' +++ [22-12-29 13:04:03] _mlrv=MODULES_RUNENV_LD_PRELOAD ++++ [22-12-29 13:04:03] eval 'echo ${MODULES_RUNENV_LD_PRELOAD:-}' +++++ [22-12-29 13:04:03] echo +++ [22-12-29 13:04:03] _mlre='LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' +++ [22-12-29 13:04:03] '[' -n 'LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' ']' +++ [22-12-29 13:04:03] _mlre='eval LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' __MODULES_QUARANTINE_SET=1 ' ++++ [22-12-29 13:04:03] eval 'LD_LIBRARY_PATH='\'''\''' 'LD_PRELOAD='\'''\''' __MODULES_QUARANTINE_SET=1 /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++++ [22-12-29 13:04:03] LD_LIBRARY_PATH= +++++ [22-12-29 13:04:03] LD_PRELOAD= +++++ [22-12-29 13:04:03] __MODULES_QUARANTINE_SET=1 +++++ [22-12-29 13:04:03] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++ [22-12-29 13:04:03] _mlcode='module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' +++ [22-12-29 13:04:03] _mlret=0 +++ [22-12-29 13:04:03] '[' -n x ']' +++ [22-12-29 13:04:03] IFS=' ' +++ [22-12-29 13:04:03] unset _mlIFS +++ [22-12-29 13:04:03] unset _mlre _mlv _mlrv +++ [22-12-29 13:04:03] '[' 0 -eq 0 ']' +++ [22-12-29 13:04:03] eval 'module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' ++++ [22-12-29 13:04:03] MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl ++++ [22-12-29 13:04:03] export MODULES_CMD ++++ [22-12-29 13:04:03] unset __MODULES_AUTOINIT_INPROGRESS ++++ [22-12-29 13:04:03] MODULESHOME=/usr/share/Modules ++++ [22-12-29 13:04:03] export MODULESHOME ++++ [22-12-29 13:04:03] test 0 +++ [22-12-29 13:04:03] '[' -t 2 ']' +++ [22-12-29 13:04:03] export -f module ++++ [22-12-29 13:04:03] type -t ml +++ [22-12-29 13:04:03] '[' function = function ']' +++ [22-12-29 13:04:03] export -f ml +++ [22-12-29 13:04:03] '[' 5 -ge 3 ']' +++ [22-12-29 13:04:03] [[ hxB =~ i ]] +++ [22-12-29 13:04:03] [[ ! :/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin: =~ :/usr/share/Modules/bin: ]] +++ [22-12-29 13:04:03] '[' '!' -n x ']' +++ [22-12-29 13:04:03] [[ ! :/usr/share/man:: =~ :/usr/share/man: ]] +++ [22-12-29 13:04:03] unset _mlcode _mlret +++ [22-12-29 13:04:03] '[' -n '' ']' + [22-12-29 13:04:03] [[ openmpi == *\p\s\m\2* ]] ++ [22-12-29 13:04:03] rpm -ql openmpi ++ [22-12-29 13:04:03] grep modulefiles ++ [22-12-29 13:04:03] tail -n 1 + [22-12-29 13:04:03] local mod_file=/usr/share/modulefiles/mpi/openmpi-x86_64 + [22-12-29 13:04:03] echo 'module purge' + [22-12-29 13:04:03] echo 'module load /usr/share/modulefiles/mpi/openmpi-x86_64' + [22-12-29 13:04:03] source /root/.bashrc + [22-12-29 13:04:04] module list ++ [22-12-29 13:04:04] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash list Currently Loaded Modulefiles: 1) /usr/share/modulefiles/mpi/openmpi-x86_64 + [22-12-29 13:04:04] eval + [22-12-29 13:04:04] _mlstatus=0 + [22-12-29 13:04:04] return 0 ++ [22-12-29 13:04:04] sed -e 's|/usr/lib64/openmpi[^/]*/bin:||g' ++ [22-12-29 13:04:04] echo /usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:04:04] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:04:04] echo /usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:04:04] sed -e 's|/usr/lib64/mvapich[^/]*/bin:||g' + [22-12-29 13:04:04] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:04:04] echo /usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:04:04] sed -e 's|/usr/lib64/mpich[^/]*/bin:||g' + [22-12-29 13:04:04] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:04:04] export PATH=/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:04:04] PATH=/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:04:04] MPI_create_hostfiles + [22-12-29 13:04:04] '[' -z 172.31.40.121 -o -z '' ']' + [22-12-29 13:04:04] echo 'Please run /kernel/infiniband/env_setup before MPI_create_hostfiles' Please run /kernel/infiniband/env_setup before MPI_create_hostfiles + [22-12-29 13:04:04] return 1 + [22-12-29 13:04:04] echo 'MPI_setup: Failed to generate MPI hostfiles' MPI_setup: Failed to generate MPI hostfiles + [22-12-29 13:04:04] echo 'MPI_setup: (This is expected on singlehost environment setup)' MPI_setup: (This is expected on singlehost environment setup) + [22-12-29 13:04:04] '[' -x /bin/systemctl ']' + [22-12-29 13:04:04] systemctl stop firewalld.service + [22-12-29 13:04:04] systemctl disable firewalld.service + [22-12-29 13:04:04] systemctl status firewalld.service ○ firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) + [22-12-29 13:04:04] MPI_dual_port_fix + [22-12-29 13:04:04] [[ '' == *\q\i\b* ]] ++ [22-12-29 13:04:04] RQA_get_rhel_major ++ [22-12-29 13:04:04] awk -F . '{print $1}' +++ [22-12-29 13:04:04] RQA_get_rhel_release +++ [22-12-29 13:04:04] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:04] echo 9 + [22-12-29 13:04:04] '[' 9 -ge 7 ']' + [22-12-29 13:04:04] RQA_pkg_install targetcli iscsi-initiator-utils lsscsi + [22-12-29 13:04:04] PKG_LIST= + [22-12-29 13:04:04] for p in "$@" + [22-12-29 13:04:04] rpm -q targetcli package targetcli is not installed + [22-12-29 13:04:04] PKG_LIST=' targetcli' + [22-12-29 13:04:04] for p in "$@" + [22-12-29 13:04:04] rpm -q iscsi-initiator-utils iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 + [22-12-29 13:04:04] for p in "$@" + [22-12-29 13:04:04] rpm -q lsscsi lsscsi-0.32-6.el9.x86_64 + [22-12-29 13:04:04] '[' '!' -z ' targetcli' ']' + [22-12-29 13:04:04] dnf install -y --setopt=strict=0 --nogpgcheck targetcli Last metadata expiration check: 0:12:06 ago on Thu 29 Dec 2022 12:51:59 PM EST. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: targetcli noarch 2.1.53-7.el9 BUILDROOT-C9S 71 k Installing dependencies: python3-configshell noarch 1:1.1.28-7.el9 BUILDROOT-C9S 65 k python3-kmod x86_64 0.9-32.el9 BUILDROOT-C9S 81 k python3-pyparsing noarch 2.4.7-9.el9 BUILDROOT-C9S 149 k python3-pyudev noarch 0.22.0-6.el9 BUILDROOT-C9S 76 k python3-rtslib noarch 2.1.75-1.el9 BUILDROOT-C9S 90 k python3-urwid x86_64 2.1.2-4.el9 BUILDROOT-C9S 768 k target-restore noarch 2.1.75-1.el9 BUILDROOT-C9S 14 k Transaction Summary ================================================================================ Install 8 Packages Total download size: 1.3 M Installed size: 5.3 M Downloading Packages: (1/8): python3-configshell-1.1.28-7.el9.noarch. 3.9 MB/s | 65 kB 00:00 (2/8): python3-kmod-0.9-32.el9.x86_64.rpm 2.9 MB/s | 81 kB 00:00 (3/8): python3-pyudev-0.22.0-6.el9.noarch.rpm 3.5 MB/s | 76 kB 00:00 (4/8): python3-pyparsing-2.4.7-9.el9.noarch.rpm 2.4 MB/s | 149 kB 00:00 (5/8): python3-rtslib-2.1.75-1.el9.noarch.rpm 2.0 MB/s | 90 kB 00:00 (6/8): target-restore-2.1.75-1.el9.noarch.rpm 949 kB/s | 14 kB 00:00 (7/8): targetcli-2.1.53-7.el9.noarch.rpm 5.1 MB/s | 71 kB 00:00 (8/8): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.6 MB/s | 768 kB 00:00 -------------------------------------------------------------------------------- Total 9.2 MB/s | 1.3 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : python3-urwid-2.1.2-4.el9.x86_64 1/8 Installing : python3-pyudev-0.22.0-6.el9.noarch 2/8 Installing : python3-pyparsing-2.4.7-9.el9.noarch 3/8 Installing : python3-configshell-1:1.1.28-7.el9.noarch 4/8 Installing : python3-kmod-0.9-32.el9.x86_64 5/8 Installing : python3-rtslib-2.1.75-1.el9.noarch 6/8 Installing : target-restore-2.1.75-1.el9.noarch 7/8 Running scriptlet: target-restore-2.1.75-1.el9.noarch 7/8 Installing : targetcli-2.1.53-7.el9.noarch 8/8 Running scriptlet: targetcli-2.1.53-7.el9.noarch 8/8 Verifying : python3-configshell-1:1.1.28-7.el9.noarch 1/8 Verifying : python3-kmod-0.9-32.el9.x86_64 2/8 Verifying : python3-pyparsing-2.4.7-9.el9.noarch 3/8 Verifying : python3-pyudev-0.22.0-6.el9.noarch 4/8 Verifying : python3-rtslib-2.1.75-1.el9.noarch 5/8 Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/8 Verifying : target-restore-2.1.75-1.el9.noarch 7/8 Verifying : targetcli-2.1.53-7.el9.noarch 8/8 Installed: python3-configshell-1:1.1.28-7.el9.noarch python3-kmod-0.9-32.el9.x86_64 python3-pyparsing-2.4.7-9.el9.noarch python3-pyudev-0.22.0-6.el9.noarch python3-rtslib-2.1.75-1.el9.noarch python3-urwid-2.1.2-4.el9.x86_64 target-restore-2.1.75-1.el9.noarch targetcli-2.1.53-7.el9.noarch Complete! + [22-12-29 13:04:10] target_name=iqn.2003-01.org.linux-iscsi.rdma.x8664:test-target ++ [22-12-29 13:04:10] RQA_get_rhel_major ++ [22-12-29 13:04:10] awk -F . '{print $1}' +++ [22-12-29 13:04:10] RQA_get_rhel_release +++ [22-12-29 13:04:10] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:10] echo 9 + [22-12-29 13:04:10] [[ 9 -ge 8 ]] + [22-12-29 13:04:10] RQA_pkg_install iperf3 + [22-12-29 13:04:10] PKG_LIST= + [22-12-29 13:04:10] for p in "$@" + [22-12-29 13:04:10] rpm -q iperf3 iperf3-3.9-9.el9.x86_64 + [22-12-29 13:04:10] '[' '!' -z '' ']' + [22-12-29 13:04:10] RQA_pkg_install net-tools + [22-12-29 13:04:10] PKG_LIST= + [22-12-29 13:04:10] for p in "$@" + [22-12-29 13:04:10] rpm -q net-tools package net-tools is not installed + [22-12-29 13:04:10] PKG_LIST=' net-tools' + [22-12-29 13:04:10] '[' '!' -z ' net-tools' ']' + [22-12-29 13:04:10] dnf install -y --setopt=strict=0 --nogpgcheck net-tools Last metadata expiration check: 0:12:12 ago on Thu 29 Dec 2022 12:51:59 PM EST. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: net-tools x86_64 2.0-0.62.20160912git.el9 BUILDROOT-C9S 292 k Transaction Summary ================================================================================ Install 1 Package Total download size: 292 k Installed size: 912 k Downloading Packages: net-tools-2.0-0.62.20160912git.el9.x86_64.rpm 10 MB/s | 292 kB 00:00 -------------------------------------------------------------------------------- Total 9.5 MB/s | 292 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : net-tools-2.0-0.62.20160912git.el9.x86_64 1/1 Running scriptlet: net-tools-2.0-0.62.20160912git.el9.x86_64 1/1 Verifying : net-tools-2.0-0.62.20160912git.el9.x86_64 1/1 Installed: net-tools-2.0-0.62.20160912git.el9.x86_64 Complete! + [22-12-29 13:04:14] [[ common == \c\o\m\m\o\n ]] + [22-12-29 13:04:14] common_tests ++ [22-12-29 13:04:14] RQA_get_driver_modules ++ [22-12-29 13:04:14] case ${RDMA_DRIVER} in ++ [22-12-29 13:04:14] echo '' + [22-12-29 13:04:14] driver_modules= + [22-12-29 13:04:14] [[ '' == *\o\p\a* ]] + [22-12-29 13:04:14] [[ '' == *\i\b* ]] ++ [22-12-29 13:04:14] grep -A 7 'port ' ++ [22-12-29 13:04:14] awk '/rate:/ {print $2 " " $3}' ++ [22-12-29 13:04:14] ibstatus + [22-12-29 13:04:14] ibstatus_rate='100 Gb/sec' + [22-12-29 13:04:14] [[ -z '' ]] + [22-12-29 13:04:14] RQA_check_result -r -1 -t 'HCA_RATE not set & can'\''t compare HCA rate' + [22-12-29 13:04:14] local test_pass=0 + [22-12-29 13:04:14] local test_skip=777 + [22-12-29 13:04:14] test 4 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local rc=-1 + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 2 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local 'msg=HCA_RATE not set & can'\''t compare HCA rate' + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 0 -gt 0 + [22-12-29 13:04:14] '[' -z -1 -o -z 'HCA_RATE not set & can'\''t compare HCA rate' ']' + [22-12-29 13:04:14] '[' -z '' ']' + [22-12-29 13:04:14] source /etc/environment ++ [22-12-29 13:04:14] TEST_RESULT_FILE= + [22-12-29 13:04:14] '[' -z '' ']' + [22-12-29 13:04:14] __RQA_create_test_result_file ++ [22-12-29 13:04:14] rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}' rdma-core + [22-12-29 13:04:14] local rdma_version=rdma-core-41.0-3.el9 ++ [22-12-29 13:04:14] echo /kernel/infiniband/cki/sanity ++ [22-12-29 13:04:14] sed -r 's/\/kernel\/infiniband\///;s/\/server//;s/\/client//;s/\/standalone//' + [22-12-29 13:04:14] TEST_NAME=cki/sanity + [22-12-29 13:04:14] [[ ! -z rdma-dev-21 ]] + [22-12-29 13:04:14] [[ ! -z rdma-cki-01 ]] + [22-12-29 13:04:14] hosts_list=rdma-dev-21/rdma-cki-01 + [22-12-29 13:04:14] local _bkrjb= ++ [22-12-29 13:04:14] grep JOBID= /etc/motd ++ [22-12-29 13:04:14] tr -d ' ' ++ [22-12-29 13:04:14] sed s/JOBID=/J:/g ++ [22-12-29 13:04:14] head -1 + [22-12-29 13:04:14] _bkrjb=J:7381428 + [22-12-29 13:04:14] [[ ! -z J:7381428 ]] + [22-12-29 13:04:14] _bkrjb=' & Beaker job J:7381428' ++ [22-12-29 13:04:14] echo results_cki/sanity.txt ++ [22-12-29 13:04:14] sed 's/\//-/g' + [22-12-29 13:04:14] local result_filename=results_cki-sanity.txt ++ [22-12-29 13:04:14] mktemp -d + [22-12-29 13:04:14] TEST_RESULT_FILE=/tmp/tmp.zXxktCJnMx/results_cki-sanity.txt + [22-12-29 13:04:14] echo 'cki/sanity test results on rdma-dev-21/rdma-cki-01 & Beaker job J:7381428:' ++ [22-12-29 13:04:14] uname -r + [22-12-29 13:04:14] echo '5.14.0-226.1773_734386600.el9.x86_64, rdma-core-41.0-3.el9, , , & ' + [22-12-29 13:04:14] echo ' Result | Status | Test' + [22-12-29 13:04:14] echo ' ---------+--------+------------------------------------' + [22-12-29 13:04:14] sed -i /TEST_RESULT_FILE/d /etc/environment + [22-12-29 13:04:14] echo TEST_RESULT_FILE=/tmp/tmp.zXxktCJnMx/results_cki-sanity.txt + [22-12-29 13:04:14] source /etc/environment ++ [22-12-29 13:04:14] TEST_RESULT_FILE=/tmp/tmp.zXxktCJnMx/results_cki-sanity.txt + [22-12-29 13:04:14] '[' -1 -eq 0 ']' + [22-12-29 13:04:14] '[' -1 -eq 777 ']' + [22-12-29 13:04:14] local test_result=FAIL + [22-12-29 13:04:14] export result=FAIL + [22-12-29 13:04:14] result=FAIL + [22-12-29 13:04:14] [[ ! -z '' ]] + [22-12-29 13:04:14] printf '%10s | %6s | %s\n' FAIL -1 'HCA_RATE not set & can'\''t compare HCA rate' + [22-12-29 13:04:14] set +x --- - TEST RESULT FOR sanity - Test: HCA_RATE not set & can't compare HCA rate - Result: FAIL - Return: -1 --- ++ [22-12-29 13:04:14] RQA_get_rhel_major ++ [22-12-29 13:04:14] awk -F . '{print $1}' +++ [22-12-29 13:04:14] RQA_get_rhel_release +++ [22-12-29 13:04:14] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:14] echo 9 + [22-12-29 13:04:14] [[ 9 -ge 8 ]] + [22-12-29 13:04:14] [[ '' == \i\b\0 ]] + [22-12-29 13:04:14] [[ '' == \i\b\1 ]] + [22-12-29 13:04:14] [[ '' == \o\p\a\0 ]] + [22-12-29 13:04:14] [[ '' == \r\o\c\e ]] + [22-12-29 13:04:14] [[ '' == \i\w ]] + [22-12-29 13:04:14] [[ '' != *\i\w* ]] + [22-12-29 13:04:14] /usr/sbin/ibstat CA 'mlx5_bond_0' CA type: MT4125 Number of ports: 1 Firmware version: 22.99.7864 Hardware version: 0 Node GUID: 0xb8599f0300f07fd0 System image GUID: 0xb8599f0300f07fd0 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xba599ffffef07fd0 Link layer: Ethernet + [22-12-29 13:04:14] RQA_check_result -r 0 -t /usr/sbin/ibstat + [22-12-29 13:04:14] local test_pass=0 + [22-12-29 13:04:14] local test_skip=777 + [22-12-29 13:04:14] test 4 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local rc=0 + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 2 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local msg=/usr/sbin/ibstat + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 0 -gt 0 + [22-12-29 13:04:14] '[' -z 0 -o -z /usr/sbin/ibstat ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' 0 -eq 0 ']' + [22-12-29 13:04:14] local test_result=PASS + [22-12-29 13:04:14] printf '%10s | %6s | %s\n' PASS 0 /usr/sbin/ibstat + [22-12-29 13:04:14] set +x --- - TEST RESULT FOR sanity - Test: /usr/sbin/ibstat - Result: PASS - Return: 0 --- + [22-12-29 13:04:14] /usr/sbin/ibstatus Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:ba59:9fff:fef0:7fd0 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 100 Gb/sec (4X EDR) link_layer: Ethernet + [22-12-29 13:04:14] RQA_check_result -r 0 -t /usr/sbin/ibstatus + [22-12-29 13:04:14] local test_pass=0 + [22-12-29 13:04:14] local test_skip=777 + [22-12-29 13:04:14] test 4 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local rc=0 + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 2 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local msg=/usr/sbin/ibstatus + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 0 -gt 0 + [22-12-29 13:04:14] '[' -z 0 -o -z /usr/sbin/ibstatus ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' 0 -eq 0 ']' + [22-12-29 13:04:14] local test_result=PASS + [22-12-29 13:04:14] printf '%10s | %6s | %s\n' PASS 0 /usr/sbin/ibstatus + [22-12-29 13:04:14] set +x --- - TEST RESULT FOR sanity - Test: /usr/sbin/ibstatus - Result: PASS - Return: 0 --- + [22-12-29 13:04:14] rpm -q srptools srp_daemon package srptools is not installed srp_daemon-41.0-3.el9.x86_64 + [22-12-29 13:04:14] guid_st_01=0x001175000078b784 + [22-12-29 13:04:14] guid_st_03=0xe41d2d03001d6470 + [22-12-29 13:04:14] echo 'a 0x001175000078b784 pkey=ffff' + [22-12-29 13:04:14] echo 'a 0xe41d2d03001d6470 pkey=ffff' + [22-12-29 13:04:14] echo d + [22-12-29 13:04:14] local srpd_serv=srpd.service + [22-12-29 13:04:14] '[' -f /usr/lib/systemd/system/srp_daemon.service ']' + [22-12-29 13:04:14] srpd_serv=srp_daemon.service + [22-12-29 13:04:14] systemctl is-active srp_daemon.service + [22-12-29 13:04:14] systemctl start srp_daemon.service + [22-12-29 13:04:14] RQA_check_result -r 0 -t 'systemctl start srp_daemon.service' + [22-12-29 13:04:14] local test_pass=0 + [22-12-29 13:04:14] local test_skip=777 + [22-12-29 13:04:14] test 4 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local rc=0 + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 2 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local 'msg=systemctl start srp_daemon.service' + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 0 -gt 0 + [22-12-29 13:04:14] '[' -z 0 -o -z 'systemctl start srp_daemon.service' ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' 0 -eq 0 ']' + [22-12-29 13:04:14] local test_result=PASS + [22-12-29 13:04:14] printf '%10s | %6s | %s\n' PASS 0 'systemctl start srp_daemon.service' + [22-12-29 13:04:14] set +x --- - TEST RESULT FOR sanity - Test: systemctl start srp_daemon.service - Result: PASS - Return: 0 --- + [22-12-29 13:04:14] RQA_skip_test ibsrpdm + [22-12-29 13:04:14] local skp=ibsrpdm + [22-12-29 13:04:14] local skp_rtn=0 + [22-12-29 13:04:14] '[' '!' -z ibsrpdm ']' + [22-12-29 13:04:14] [[ '' == *\i\b\s\r\p\d\m* ]] + [22-12-29 13:04:14] return 0 + [22-12-29 13:04:14] skip_ibsrpdm=0 + [22-12-29 13:04:14] '[' 0 -eq 0 ']' + [22-12-29 13:04:14] /usr/sbin/ibsrpdm -vc Unsupported link layer Ethernet Resource allocation failed Using device mlx5_bond_0 port 1 + [22-12-29 13:04:14] RQA_check_result -r 1 -t '/usr/sbin/ibsrpdm -vc' + [22-12-29 13:04:14] local test_pass=0 + [22-12-29 13:04:14] local test_skip=777 + [22-12-29 13:04:14] test 4 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local rc=1 + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 2 -gt 0 + [22-12-29 13:04:14] case $1 in + [22-12-29 13:04:14] local 'msg=/usr/sbin/ibsrpdm -vc' + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] shift + [22-12-29 13:04:14] test 0 -gt 0 + [22-12-29 13:04:14] '[' -z 1 -o -z '/usr/sbin/ibsrpdm -vc' ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:14] '[' 1 -eq 0 ']' + [22-12-29 13:04:14] '[' 1 -eq 777 ']' + [22-12-29 13:04:14] local test_result=FAIL + [22-12-29 13:04:14] export result=FAIL + [22-12-29 13:04:14] result=FAIL + [22-12-29 13:04:14] [[ ! -z '' ]] + [22-12-29 13:04:14] printf '%10s | %6s | %s\n' FAIL 1 '/usr/sbin/ibsrpdm -vc' + [22-12-29 13:04:14] set +x --- - TEST RESULT FOR sanity - Test: /usr/sbin/ibsrpdm -vc - Result: FAIL - Return: 1 --- + [22-12-29 13:04:14] systemctl stop srp_daemon + [22-12-29 13:04:15] RQA_check_result -r 0 -t 'systemctl stop srp_daemon' + [22-12-29 13:04:15] local test_pass=0 + [22-12-29 13:04:15] local test_skip=777 + [22-12-29 13:04:15] test 4 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local rc=0 + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 2 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local 'msg=systemctl stop srp_daemon' + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 0 -gt 0 + [22-12-29 13:04:15] '[' -z 0 -o -z 'systemctl stop srp_daemon' ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' 0 -eq 0 ']' + [22-12-29 13:04:15] local test_result=PASS + [22-12-29 13:04:15] printf '%10s | %6s | %s\n' PASS 0 'systemctl stop srp_daemon' + [22-12-29 13:04:15] set +x --- - TEST RESULT FOR sanity - Test: systemctl stop srp_daemon - Result: PASS - Return: 0 --- ++ [22-12-29 13:04:15] RQA_get_rhel_major ++ [22-12-29 13:04:15] awk -F . '{print $1}' +++ [22-12-29 13:04:15] RQA_get_rhel_release +++ [22-12-29 13:04:15] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:15] echo 9 + [22-12-29 13:04:15] '[' 9 -lt 8 ']' + [22-12-29 13:04:15] timeout 3m ping -i 0.2 -c 100 ping: usage error: Destination address required + [22-12-29 13:04:15] RQA_check_result -r 1 -t 'ping self - ' + [22-12-29 13:04:15] local test_pass=0 + [22-12-29 13:04:15] local test_skip=777 + [22-12-29 13:04:15] test 4 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local rc=1 + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 2 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local 'msg=ping self - ' + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 0 -gt 0 + [22-12-29 13:04:15] '[' -z 1 -o -z 'ping self - ' ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' 1 -eq 0 ']' + [22-12-29 13:04:15] '[' 1 -eq 777 ']' + [22-12-29 13:04:15] local test_result=FAIL + [22-12-29 13:04:15] export result=FAIL + [22-12-29 13:04:15] result=FAIL + [22-12-29 13:04:15] [[ ! -z '' ]] + [22-12-29 13:04:15] printf '%10s | %6s | %s\n' FAIL 1 'ping self - ' + [22-12-29 13:04:15] set +x --- - TEST RESULT FOR sanity - Test: ping self - - Result: FAIL - Return: 1 --- + [22-12-29 13:04:15] timeout 3m ping6 -i 0.2 -c 100 % ping6: %: Name or service not known + [22-12-29 13:04:15] RQA_check_result -r 2 -t 'ping6 self - %' + [22-12-29 13:04:15] local test_pass=0 + [22-12-29 13:04:15] local test_skip=777 + [22-12-29 13:04:15] test 4 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local rc=2 + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 2 -gt 0 + [22-12-29 13:04:15] case $1 in + [22-12-29 13:04:15] local 'msg=ping6 self - %' + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] shift + [22-12-29 13:04:15] test 0 -gt 0 + [22-12-29 13:04:15] '[' -z 2 -o -z 'ping6 self - %' ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:04:15] '[' 2 -eq 0 ']' + [22-12-29 13:04:15] '[' 2 -eq 777 ']' + [22-12-29 13:04:15] local test_result=FAIL + [22-12-29 13:04:15] export result=FAIL + [22-12-29 13:04:15] result=FAIL + [22-12-29 13:04:15] [[ ! -z '' ]] + [22-12-29 13:04:15] printf '%10s | %6s | %s\n' FAIL 2 'ping6 self - %' + [22-12-29 13:04:15] set +x --- - TEST RESULT FOR sanity - Test: ping6 self - % - Result: FAIL - Return: 2 --- ++ [22-12-29 13:04:15] RQA_get_rhel_major ++ [22-12-29 13:04:15] awk -F . '{print $1}' +++ [22-12-29 13:04:15] RQA_get_rhel_release +++ [22-12-29 13:04:15] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:15] echo 9 + [22-12-29 13:04:15] [[ 9 -gt 7 ]] ++ [22-12-29 13:04:15] RQA_get_rhel_minor ++ [22-12-29 13:04:15] awk -F . '{print $2}' +++ [22-12-29 13:04:15] RQA_get_rhel_release +++ [22-12-29 13:04:15] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:04:15] echo 9 + [22-12-29 13:04:15] [[ '' -gt 2 ]] + [22-12-29 13:04:15] echo '--- wait the server to get ready.' --- wait the server to get ready. + [22-12-29 13:04:15] rhts_sync_block -s server-ready rdma-dev-21.rdma.lab.eng.rdu2.redhat.com /usr/bin/rhts_sync_block -s server-ready rdma-dev-21.rdma.lab.eng.rdu2.redhat.com -- Blocking state(s) = 12_server-ready Server rdma-dev-21.rdma.lab.eng.rdu2.redhat.com not reported state 12_server-ready for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_server-ready + [22-12-29 13:05:25] rhts_sync_set -s client-ready /usr/bin/rhts_sync_set -s client-ready -- State set = 12_client-ready + [22-12-29 13:05:25] bash -x ./tier1.sh client + trap 'exit 1' SIGHUP SIGINT SIGQUIT SIGTERM + source /root/rdma-qa.sh ++ PS4='$(date +"+ [%y-%m-%d %H:%M:%S]") ' ++ [22-12-29 13:05:25] export NIC_PREF= ++ [22-12-29 13:05:25] NIC_PREF= ++ [22-12-29 13:05:25] export PERFTEST_FLAGS= ++ [22-12-29 13:05:25] PERFTEST_FLAGS= +++ [22-12-29 13:05:25] hostname -s ++ [22-12-29 13:05:25] export s_hostname=rdma-cki-01 ++ [22-12-29 13:05:25] s_hostname=rdma-cki-01 ++ [22-12-29 13:05:25] RQA_set_pyexec ++ [22-12-29 13:05:25] PYEXEC= ++ [22-12-29 13:05:25] which python ++ [22-12-29 13:05:25] export PYEXEC=python ++ [22-12-29 13:05:25] PYEXEC=python ++ [22-12-29 13:05:25] return 0 +++ [22-12-29 13:05:25] grep -i fedora /etc/redhat-release ++ [22-12-29 13:05:25] [[ -n '' ]] +++ [22-12-29 13:05:25] RQA_get_rhel_major +++ [22-12-29 13:05:25] awk -F . '{print $1}' ++++ [22-12-29 13:05:25] RQA_get_rhel_release ++++ [22-12-29 13:05:25] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:25] echo 9 ++ [22-12-29 13:05:25] [[ 9 -ge 8 ]] ++ [22-12-29 13:05:25] export 'PKGINSTALL=dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:05:25] PKGINSTALL='dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:05:25] export 'PKGREMOVE=dnf remove --noautoremove -y' ++ [22-12-29 13:05:25] PKGREMOVE='dnf remove --noautoremove -y' ++ [22-12-29 13:05:25] [[ rdma-cki-01 == \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:05:25] export LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:05:25] LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:05:25] '[' -f /root/known-issues.sh ']' ++ [22-12-29 13:05:25] source /root/known-issues.sh ++ [22-12-29 13:05:28] RQA_load_tech_preview_modules ++ [22-12-29 13:05:28] [[ ! -e /etc/rdma/rdma.conf ]] ++ [22-12-29 13:05:28] return ++ [22-12-29 13:05:28] RQA_fix_uefi_reboot ++ [22-12-29 13:05:28] ls -l /usr/sbin/reboot ++ [22-12-29 13:05:28] grep rhts-reboot ++ [22-12-29 13:05:28] which rhts-reboot ++ [22-12-29 13:05:28] rpm -q rhts-test-env ++ [22-12-29 13:05:28] RQA_fix_ssh_ipv6_auth +++ [22-12-29 13:05:28] RQA_get_rhel_major +++ [22-12-29 13:05:28] awk -F . '{print $1}' ++++ [22-12-29 13:05:28] RQA_get_rhel_release ++++ [22-12-29 13:05:28] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:28] echo 9 ++ [22-12-29 13:05:28] '[' 9 -ge 7 ']' ++ [22-12-29 13:05:28] local need_restart=0 ++ [22-12-29 13:05:28] grep -E '^UseDNS yes' /etc/ssh/sshd_config ++ [22-12-29 13:05:28] grep -E '^GSSAPIStrictAcceptorCheck yes' /etc/ssh/sshd_config ++ [22-12-29 13:05:28] grep 'AddressFamily inet' /root/.ssh/config ++ [22-12-29 13:05:28] '[' 0 -eq 1 ']' ++ [22-12-29 13:05:28] RQA_fix_mlx5_ib_connected_mode +++ [22-12-29 13:05:28] RQA_get_rhel_major +++ [22-12-29 13:05:28] awk -F . '{print $1}' ++++ [22-12-29 13:05:28] RQA_get_rhel_release ++++ [22-12-29 13:05:28] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:28] echo 9 ++ [22-12-29 13:05:28] [[ 9 -ge 7 ]] +++ [22-12-29 13:05:28] RQA_get_rhel_minor +++ [22-12-29 13:05:28] awk -F . '{print $2}' ++++ [22-12-29 13:05:28] RQA_get_rhel_release ++++ [22-12-29 13:05:28] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:28] echo 9 ++ [22-12-29 13:05:28] [[ '' -ge 5 ]] +++ [22-12-29 13:05:29] RQA_get_rhel_major +++ [22-12-29 13:05:29] awk -F . '{print $1}' ++++ [22-12-29 13:05:29] RQA_get_rhel_release ++++ [22-12-29 13:05:29] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:29] echo 9 ++ [22-12-29 13:05:29] [[ 9 -gt 7 ]] +++ [22-12-29 13:05:29] ls /etc/sysconfig/network-scripts/ +++ [22-12-29 13:05:29] grep mlx5 +++ [22-12-29 13:05:29] grep ib +++ [22-12-29 13:05:29] grep -v '~' ++ [22-12-29 13:05:29] RQA_nfs_rhel8_temp_fix +++ [22-12-29 13:05:29] RQA_get_rhel_major +++ [22-12-29 13:05:29] awk -F . '{print $1}' ++++ [22-12-29 13:05:29] RQA_get_rhel_release ++++ [22-12-29 13:05:29] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:29] echo 9 ++ [22-12-29 13:05:29] (( 9 == 8 )) ++ [22-12-29 13:05:29] source /root/.bashrc ++ [22-12-29 13:05:29] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:05:29] source /root/mpi-functions.sh ++ [22-12-29 13:05:29] export 'MPI_TIMEOUT=timeout --preserve-status --kill-after=5m 3m' ++ [22-12-29 13:05:29] MPI_TIMEOUT='timeout --preserve-status --kill-after=5m 3m' + [22-12-29 13:05:29] source /root/rdma-qa.sh ++ [22-12-29 13:05:29] PS4='$(date +"+ [%y-%m-%d %H:%M:%S]") ' ++ [22-12-29 13:05:29] export NIC_PREF= ++ [22-12-29 13:05:29] NIC_PREF= ++ [22-12-29 13:05:29] export PERFTEST_FLAGS= ++ [22-12-29 13:05:29] PERFTEST_FLAGS= +++ [22-12-29 13:05:29] hostname -s ++ [22-12-29 13:05:29] export s_hostname=rdma-cki-01 ++ [22-12-29 13:05:29] s_hostname=rdma-cki-01 ++ [22-12-29 13:05:29] RQA_set_pyexec ++ [22-12-29 13:05:29] PYEXEC= ++ [22-12-29 13:05:29] which python ++ [22-12-29 13:05:29] export PYEXEC=python ++ [22-12-29 13:05:29] PYEXEC=python ++ [22-12-29 13:05:29] return 0 +++ [22-12-29 13:05:29] grep -i fedora /etc/redhat-release ++ [22-12-29 13:05:29] [[ -n '' ]] +++ [22-12-29 13:05:29] RQA_get_rhel_major +++ [22-12-29 13:05:29] awk -F . '{print $1}' ++++ [22-12-29 13:05:29] RQA_get_rhel_release ++++ [22-12-29 13:05:29] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:29] echo 9 ++ [22-12-29 13:05:29] [[ 9 -ge 8 ]] ++ [22-12-29 13:05:29] export 'PKGINSTALL=dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:05:29] PKGINSTALL='dnf install -y --setopt=strict=0 --nogpgcheck' ++ [22-12-29 13:05:29] export 'PKGREMOVE=dnf remove --noautoremove -y' ++ [22-12-29 13:05:29] PKGREMOVE='dnf remove --noautoremove -y' ++ [22-12-29 13:05:29] [[ rdma-cki-01 == \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:05:29] export LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:05:29] LATEST_MFT=http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz ++ [22-12-29 13:05:29] '[' -f /root/known-issues.sh ']' ++ [22-12-29 13:05:29] source /root/known-issues.sh ++ [22-12-29 13:05:34] RQA_load_tech_preview_modules ++ [22-12-29 13:05:34] [[ ! -e /etc/rdma/rdma.conf ]] ++ [22-12-29 13:05:34] return ++ [22-12-29 13:05:34] RQA_fix_uefi_reboot ++ [22-12-29 13:05:34] ls -l /usr/sbin/reboot ++ [22-12-29 13:05:34] grep rhts-reboot ++ [22-12-29 13:05:34] which rhts-reboot ++ [22-12-29 13:05:34] rpm -q rhts-test-env ++ [22-12-29 13:05:34] RQA_fix_ssh_ipv6_auth +++ [22-12-29 13:05:34] RQA_get_rhel_major +++ [22-12-29 13:05:34] awk -F . '{print $1}' ++++ [22-12-29 13:05:34] RQA_get_rhel_release ++++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:34] echo 9 ++ [22-12-29 13:05:34] '[' 9 -ge 7 ']' ++ [22-12-29 13:05:34] local need_restart=0 ++ [22-12-29 13:05:34] grep -E '^UseDNS yes' /etc/ssh/sshd_config ++ [22-12-29 13:05:34] grep -E '^GSSAPIStrictAcceptorCheck yes' /etc/ssh/sshd_config ++ [22-12-29 13:05:34] grep 'AddressFamily inet' /root/.ssh/config ++ [22-12-29 13:05:34] '[' 0 -eq 1 ']' ++ [22-12-29 13:05:34] RQA_fix_mlx5_ib_connected_mode +++ [22-12-29 13:05:34] RQA_get_rhel_major +++ [22-12-29 13:05:34] awk -F . '{print $1}' ++++ [22-12-29 13:05:34] RQA_get_rhel_release ++++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:34] echo 9 ++ [22-12-29 13:05:34] [[ 9 -ge 7 ]] +++ [22-12-29 13:05:34] RQA_get_rhel_minor +++ [22-12-29 13:05:34] awk -F . '{print $2}' ++++ [22-12-29 13:05:34] RQA_get_rhel_release ++++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:34] echo 9 ++ [22-12-29 13:05:34] [[ '' -ge 5 ]] +++ [22-12-29 13:05:34] RQA_get_rhel_major +++ [22-12-29 13:05:34] awk -F . '{print $1}' ++++ [22-12-29 13:05:34] RQA_get_rhel_release ++++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:34] echo 9 ++ [22-12-29 13:05:34] [[ 9 -gt 7 ]] +++ [22-12-29 13:05:34] ls /etc/sysconfig/network-scripts/ +++ [22-12-29 13:05:34] grep mlx5 +++ [22-12-29 13:05:34] grep ib +++ [22-12-29 13:05:34] grep -v '~' ++ [22-12-29 13:05:34] RQA_nfs_rhel8_temp_fix +++ [22-12-29 13:05:34] RQA_get_rhel_major +++ [22-12-29 13:05:34] awk -F . '{print $1}' ++++ [22-12-29 13:05:34] RQA_get_rhel_release ++++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release +++ [22-12-29 13:05:34] echo 9 ++ [22-12-29 13:05:34] (( 9 == 8 )) ++ [22-12-29 13:05:34] source /root/.bashrc ++ [22-12-29 13:05:34] [[ '' == \i\4\0\i\w ]] + [22-12-29 13:05:34] source /root/mpi-functions.sh ++ [22-12-29 13:05:34] export 'MPI_TIMEOUT=timeout --preserve-status --kill-after=5m 3m' ++ [22-12-29 13:05:34] MPI_TIMEOUT='timeout --preserve-status --kill-after=5m 3m' + [22-12-29 13:05:34] RQA_set_inline_default + [22-12-29 13:05:34] [[ '' != *\m\l\x\4* ]] + [22-12-29 13:05:34] mkdir -p /etc/rdma/rsocket + [22-12-29 13:05:34] echo 0 + [22-12-29 13:05:34] RQA_set_perftest_flags + [22-12-29 13:05:34] PERFTEST_FLAGS='-d -i -F' + [22-12-29 13:05:34] [[ '' == \m\l\x\4 ]] + [22-12-29 13:05:34] PERFTEST_FLAGS='-d -i -F -R' + [22-12-29 13:05:34] export PERFTEST_FLAGS + [22-12-29 13:05:34] MPI_PKG=openmpi + [22-12-29 13:05:34] TEST_PKG=mpitests-openmpi ++ [22-12-29 13:05:34] RQA_get_rhel_major ++ [22-12-29 13:05:34] awk -F . '{print $1}' +++ [22-12-29 13:05:34] RQA_get_rhel_release +++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:05:34] echo 9 + [22-12-29 13:05:34] [[ 9 -eq 6 ]] ++ [22-12-29 13:05:34] RQA_get_rhel_major ++ [22-12-29 13:05:34] awk -F . '{print $1}' +++ [22-12-29 13:05:34] RQA_get_rhel_release +++ [22-12-29 13:05:34] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:05:34] echo 9 + [22-12-29 13:05:34] [[ 9 -eq 7 ]] + [22-12-29 13:05:34] MPI_setup -m openmpi -t mpitests-openmpi + [22-12-29 13:05:34] test 4 -gt 0 + [22-12-29 13:05:34] case $1 in + [22-12-29 13:05:34] local mpi_pkg=openmpi + [22-12-29 13:05:34] shift + [22-12-29 13:05:34] shift + [22-12-29 13:05:34] test 2 -gt 0 + [22-12-29 13:05:34] case $1 in + [22-12-29 13:05:34] local test_pkg=mpitests-openmpi + [22-12-29 13:05:34] shift + [22-12-29 13:05:34] shift + [22-12-29 13:05:34] test 0 -gt 0 + [22-12-29 13:05:34] '[' -z openmpi -o -z mpitests-openmpi ']' + [22-12-29 13:05:34] rpm -q environment-modules environment-modules-5.0.1-2.el9.x86_64 + [22-12-29 13:05:34] '[' -f /etc/profile.d/modules.sh ']' + [22-12-29 13:05:34] source /etc/profile.d/modules.sh ++ [22-12-29 13:05:34] '[' -n /usr/bin/bash ']' ++ [22-12-29 13:05:34] '[' bash = sh ']' ++ [22-12-29 13:05:34] shell=bash ++ [22-12-29 13:05:34] '[' -f /usr/share/Modules/init/bash ']' ++ [22-12-29 13:05:34] . /usr/share/Modules/init/bash +++ [22-12-29 13:05:34] unset _mlshdbg +++ [22-12-29 13:05:34] '[' 0 = 1 ']' +++ [22-12-29 13:05:34] export 'MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:05:34] MODULES_RUN_QUARANTINE='LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:05:34] unset _mlre _mlIFS +++ [22-12-29 13:05:34] '[' -n x ']' +++ [22-12-29 13:05:34] _mlIFS=' ' +++ [22-12-29 13:05:34] IFS=' ' +++ [22-12-29 13:05:34] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:05:34] '[' LD_LIBRARY_PATH = LD_LIBRARY_PATH -a LD_LIBRARY_PATH = LD_LIBRARY_PATH ']' ++++ [22-12-29 13:05:34] eval 'echo ${LD_LIBRARY_PATH+x}' +++++ [22-12-29 13:05:34] echo x +++ [22-12-29 13:05:34] '[' -n x ']' ++++ [22-12-29 13:05:34] eval 'echo ${LD_LIBRARY_PATH}' +++++ [22-12-29 13:05:34] echo /usr/lib64/openmpi/lib +++ [22-12-29 13:05:34] _mlre='__MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\'' ' +++ [22-12-29 13:05:34] _mlrv=MODULES_RUNENV_LD_LIBRARY_PATH ++++ [22-12-29 13:05:34] eval 'echo ${MODULES_RUNENV_LD_LIBRARY_PATH:-}' +++++ [22-12-29 13:05:34] echo +++ [22-12-29 13:05:34] _mlre='__MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\'' LD_LIBRARY_PATH='\'''\'' ' +++ [22-12-29 13:05:34] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:05:34] '[' LD_PRELOAD = LD_PRELOAD -a LD_PRELOAD = LD_PRELOAD ']' ++++ [22-12-29 13:05:34] eval 'echo ${LD_PRELOAD+x}' +++++ [22-12-29 13:05:34] echo +++ [22-12-29 13:05:34] '[' -n '' ']' +++ [22-12-29 13:05:34] _mlrv=MODULES_RUNENV_LD_PRELOAD ++++ [22-12-29 13:05:34] eval 'echo ${MODULES_RUNENV_LD_PRELOAD:-}' +++++ [22-12-29 13:05:34] echo +++ [22-12-29 13:05:34] _mlre='__MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\'' LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' +++ [22-12-29 13:05:34] '[' -n '__MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\'' LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' ']' +++ [22-12-29 13:05:34] _mlre='eval __MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\'' LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' __MODULES_QUARANTINE_SET=1 ' ++++ [22-12-29 13:05:34] eval '__MODULES_QUAR_LD_LIBRARY_PATH='\''/usr/lib64/openmpi/lib'\''' 'LD_LIBRARY_PATH='\'''\''' 'LD_PRELOAD='\'''\''' __MODULES_QUARANTINE_SET=1 /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++++ [22-12-29 13:05:34] __MODULES_QUAR_LD_LIBRARY_PATH=/usr/lib64/openmpi/lib +++++ [22-12-29 13:05:34] LD_LIBRARY_PATH= +++++ [22-12-29 13:05:34] LD_PRELOAD= +++++ [22-12-29 13:05:34] __MODULES_QUARANTINE_SET=1 +++++ [22-12-29 13:05:34] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++ [22-12-29 13:05:34] _mlcode='module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' +++ [22-12-29 13:05:34] _mlret=0 +++ [22-12-29 13:05:35] '[' -n x ']' +++ [22-12-29 13:05:35] IFS=' ' +++ [22-12-29 13:05:35] unset _mlIFS +++ [22-12-29 13:05:35] unset _mlre _mlv _mlrv +++ [22-12-29 13:05:35] '[' 0 -eq 0 ']' +++ [22-12-29 13:05:35] eval 'module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; test 0;' ++++ [22-12-29 13:05:35] MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl ++++ [22-12-29 13:05:35] export MODULES_CMD ++++ [22-12-29 13:05:35] unset __MODULES_AUTOINIT_INPROGRESS ++++ [22-12-29 13:05:35] MODULESHOME=/usr/share/Modules ++++ [22-12-29 13:05:35] export MODULESHOME ++++ [22-12-29 13:05:35] test 0 +++ [22-12-29 13:05:35] '[' -t 2 ']' +++ [22-12-29 13:05:35] export -f module ++++ [22-12-29 13:05:35] type -t ml +++ [22-12-29 13:05:35] '[' function = function ']' +++ [22-12-29 13:05:35] export -f ml +++ [22-12-29 13:05:35] '[' 5 -ge 3 ']' +++ [22-12-29 13:05:35] [[ hxB =~ i ]] +++ [22-12-29 13:05:35] [[ ! :/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin: =~ :/usr/share/Modules/bin: ]] +++ [22-12-29 13:05:35] '[' '!' -n x ']' +++ [22-12-29 13:05:35] [[ ! :/usr/share/man/openmpi-x86_64:/usr/share/man:: =~ :/usr/share/man: ]] +++ [22-12-29 13:05:35] unset _mlcode _mlret +++ [22-12-29 13:05:35] '[' -n '' ']' + [22-12-29 13:05:35] type module module is a function module () { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus } + [22-12-29 13:05:35] module purge ++ [22-12-29 13:05:35] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash purge + [22-12-29 13:05:35] eval unset 'MPI_COMPILER;' unset 'LD_LIBRARY_PATH;' 'MANPATH=/usr/share/man:;' export 'MANPATH;' unset 'MPI_INCLUDE;' unset 'MPI_MAN;' unset '_LMFILES_;' unset 'LOADEDMODULES;' unset 'PKG_CONFIG_PATH;' unset 'MPI_PYTHON3_SITEARCH;' unset 'MPI_HOME;' unset 'MPI_FORTRAN_MOD_DIR;' unset '__MODULES_LMCONFLICT;' '__MODULES_SHARE_MANPATH=:1;' export '__MODULES_SHARE_MANPATH;' unset 'MPI_SUFFIX;' unset 'MPI_SYSCONFIG;' unset 'MPI_LIB;' 'PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin;' export 'PATH;' unset 'MPI_BIN;' test '0;' ++ [22-12-29 13:05:35] unset MPI_COMPILER ++ [22-12-29 13:05:35] unset LD_LIBRARY_PATH ++ [22-12-29 13:05:35] MANPATH=/usr/share/man: ++ [22-12-29 13:05:35] export MANPATH ++ [22-12-29 13:05:35] unset MPI_INCLUDE ++ [22-12-29 13:05:35] unset MPI_MAN ++ [22-12-29 13:05:35] unset _LMFILES_ ++ [22-12-29 13:05:35] unset LOADEDMODULES ++ [22-12-29 13:05:35] unset PKG_CONFIG_PATH ++ [22-12-29 13:05:35] unset MPI_PYTHON3_SITEARCH ++ [22-12-29 13:05:35] unset MPI_HOME ++ [22-12-29 13:05:35] unset MPI_FORTRAN_MOD_DIR ++ [22-12-29 13:05:35] unset __MODULES_LMCONFLICT ++ [22-12-29 13:05:35] __MODULES_SHARE_MANPATH=:1 ++ [22-12-29 13:05:35] export __MODULES_SHARE_MANPATH ++ [22-12-29 13:05:35] unset MPI_SUFFIX ++ [22-12-29 13:05:35] unset MPI_SYSCONFIG ++ [22-12-29 13:05:35] unset MPI_LIB ++ [22-12-29 13:05:35] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:05:35] export PATH ++ [22-12-29 13:05:35] unset MPI_BIN ++ [22-12-29 13:05:35] test 0 + [22-12-29 13:05:35] _mlstatus=0 + [22-12-29 13:05:35] return 0 + [22-12-29 13:05:35] sed -e '/^module/d' -i /root/.bashrc + [22-12-29 13:05:35] dnf remove --noautoremove -y 'mpitests*' 'mpich*' 'mvapich*' 'openmpi*' No match for argument: mpich* No match for argument: mvapich* Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Removing: mpitests-openmpi x86_64 5.8-1.el9 @BUILDROOT-C9S 5.3 M openmpi x86_64 1:4.1.1-5.el9 @BUILDROOT-C9S 10 M openmpi-devel x86_64 1:4.1.1-5.el9 @BUILDROOT-C9S 1.9 M Transaction Summary ================================================================================ Remove 3 Packages Freed space: 17 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : mpitests-openmpi-5.8-1.el9.x86_64 1/3 Erasing : openmpi-1:4.1.1-5.el9.x86_64 2/3 Erasing : openmpi-devel-1:4.1.1-5.el9.x86_64 3/3 Running scriptlet: openmpi-devel-1:4.1.1-5.el9.x86_64 3/3 Verifying : mpitests-openmpi-5.8-1.el9.x86_64 1/3 Verifying : openmpi-1:4.1.1-5.el9.x86_64 2/3 Verifying : openmpi-devel-1:4.1.1-5.el9.x86_64 3/3 Removed: mpitests-openmpi-5.8-1.el9.x86_64 openmpi-1:4.1.1-5.el9.x86_64 openmpi-devel-1:4.1.1-5.el9.x86_64 Complete! + [22-12-29 13:05:38] dnf install -y --setopt=strict=0 --nogpgcheck openmpi mpitests-openmpi environment-modules Last metadata expiration check: 0:13:40 ago on Thu 29 Dec 2022 12:51:59 PM EST. Package environment-modules-5.0.1-2.el9.x86_64 is already installed. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: mpitests-openmpi x86_64 5.8-1.el9 BUILDROOT-C9S 897 k openmpi x86_64 1:4.1.1-5.el9 BUILDROOT-C9S 2.7 M Installing dependencies: openmpi-devel x86_64 1:4.1.1-5.el9 BUILDROOT-C9S 1.2 M Transaction Summary ================================================================================ Install 3 Packages Total download size: 4.7 M Installed size: 17 M Downloading Packages: (1/3): mpitests-openmpi-5.8-1.el9.x86_64.rpm 10 MB/s | 897 kB 00:00 (2/3): openmpi-devel-4.1.1-5.el9.x86_64.rpm 7.3 MB/s | 1.2 MB 00:00 (3/3): openmpi-4.1.1-5.el9.x86_64.rpm 8.2 MB/s | 2.7 MB 00:00 -------------------------------------------------------------------------------- Total 14 MB/s | 4.7 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : openmpi-devel-1:4.1.1-5.el9.x86_64 1/3 Installing : openmpi-1:4.1.1-5.el9.x86_64 2/3 Installing : mpitests-openmpi-5.8-1.el9.x86_64 3/3 Running scriptlet: mpitests-openmpi-5.8-1.el9.x86_64 3/3 Verifying : mpitests-openmpi-5.8-1.el9.x86_64 1/3 Verifying : openmpi-1:4.1.1-5.el9.x86_64 2/3 Verifying : openmpi-devel-1:4.1.1-5.el9.x86_64 3/3 Installed: mpitests-openmpi-5.8-1.el9.x86_64 openmpi-1:4.1.1-5.el9.x86_64 openmpi-devel-1:4.1.1-5.el9.x86_64 Complete! + [22-12-29 13:05:44] '[' -f /etc/profile.d/modules.sh ']' + [22-12-29 13:05:44] source /etc/profile.d/modules.sh ++ [22-12-29 13:05:44] '[' -n /usr/bin/bash ']' ++ [22-12-29 13:05:44] '[' bash = sh ']' ++ [22-12-29 13:05:44] shell=bash ++ [22-12-29 13:05:44] '[' -f /usr/share/Modules/init/bash ']' ++ [22-12-29 13:05:44] . /usr/share/Modules/init/bash +++ [22-12-29 13:05:44] unset _mlshdbg +++ [22-12-29 13:05:44] '[' 0 = 1 ']' +++ [22-12-29 13:05:44] export 'MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:05:44] MODULES_RUN_QUARANTINE='LD_LIBRARY_PATH LD_PRELOAD' +++ [22-12-29 13:05:44] unset _mlre _mlIFS +++ [22-12-29 13:05:44] '[' -n x ']' +++ [22-12-29 13:05:44] _mlIFS=' ' +++ [22-12-29 13:05:44] IFS=' ' +++ [22-12-29 13:05:44] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:05:44] '[' LD_LIBRARY_PATH = LD_LIBRARY_PATH -a LD_LIBRARY_PATH = LD_LIBRARY_PATH ']' ++++ [22-12-29 13:05:44] eval 'echo ${LD_LIBRARY_PATH+x}' +++++ [22-12-29 13:05:44] echo +++ [22-12-29 13:05:44] '[' -n '' ']' +++ [22-12-29 13:05:44] _mlrv=MODULES_RUNENV_LD_LIBRARY_PATH ++++ [22-12-29 13:05:44] eval 'echo ${MODULES_RUNENV_LD_LIBRARY_PATH:-}' +++++ [22-12-29 13:05:44] echo +++ [22-12-29 13:05:44] _mlre='LD_LIBRARY_PATH='\'''\'' ' +++ [22-12-29 13:05:44] for _mlv in ${MODULES_RUN_QUARANTINE:-} +++ [22-12-29 13:05:44] '[' LD_PRELOAD = LD_PRELOAD -a LD_PRELOAD = LD_PRELOAD ']' ++++ [22-12-29 13:05:44] eval 'echo ${LD_PRELOAD+x}' +++++ [22-12-29 13:05:44] echo +++ [22-12-29 13:05:44] '[' -n '' ']' +++ [22-12-29 13:05:44] _mlrv=MODULES_RUNENV_LD_PRELOAD ++++ [22-12-29 13:05:44] eval 'echo ${MODULES_RUNENV_LD_PRELOAD:-}' +++++ [22-12-29 13:05:44] echo +++ [22-12-29 13:05:44] _mlre='LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' +++ [22-12-29 13:05:44] '[' -n 'LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' ' ']' +++ [22-12-29 13:05:44] _mlre='eval LD_LIBRARY_PATH='\'''\'' LD_PRELOAD='\'''\'' __MODULES_QUARANTINE_SET=1 ' ++++ [22-12-29 13:05:44] eval 'LD_LIBRARY_PATH='\'''\''' 'LD_PRELOAD='\'''\''' __MODULES_QUARANTINE_SET=1 /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++++ [22-12-29 13:05:44] LD_LIBRARY_PATH= +++++ [22-12-29 13:05:44] LD_PRELOAD= +++++ [22-12-29 13:05:44] __MODULES_QUARANTINE_SET=1 +++++ [22-12-29 13:05:44] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash autoinit +++ [22-12-29 13:05:44] _mlcode='module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; LOADEDMODULES=; export LOADEDMODULES; test 0;' +++ [22-12-29 13:05:44] _mlret=0 +++ [22-12-29 13:05:44] '[' -n x ']' +++ [22-12-29 13:05:44] IFS=' ' +++ [22-12-29 13:05:44] unset _mlIFS +++ [22-12-29 13:05:44] unset _mlre _mlv _mlrv +++ [22-12-29 13:05:44] '[' 0 -eq 0 ']' +++ [22-12-29 13:05:44] eval 'module() { eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`; _mlstatus=$?; return $_mlstatus; }; ml() { module ml "$@"; }; MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl; export MODULES_CMD; unset __MODULES_AUTOINIT_INPROGRESS; MODULESHOME=/usr/share/Modules; export MODULESHOME; LOADEDMODULES=; export LOADEDMODULES; test 0;' ++++ [22-12-29 13:05:44] MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl ++++ [22-12-29 13:05:44] export MODULES_CMD ++++ [22-12-29 13:05:44] unset __MODULES_AUTOINIT_INPROGRESS ++++ [22-12-29 13:05:44] MODULESHOME=/usr/share/Modules ++++ [22-12-29 13:05:44] export MODULESHOME ++++ [22-12-29 13:05:44] LOADEDMODULES= ++++ [22-12-29 13:05:44] export LOADEDMODULES ++++ [22-12-29 13:05:44] test 0 +++ [22-12-29 13:05:44] '[' -t 2 ']' +++ [22-12-29 13:05:44] export -f module ++++ [22-12-29 13:05:44] type -t ml +++ [22-12-29 13:05:44] '[' function = function ']' +++ [22-12-29 13:05:44] export -f ml +++ [22-12-29 13:05:44] '[' 5 -ge 3 ']' +++ [22-12-29 13:05:44] [[ hxB =~ i ]] +++ [22-12-29 13:05:44] [[ ! :/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin: =~ :/usr/share/Modules/bin: ]] +++ [22-12-29 13:05:44] '[' '!' -n x ']' +++ [22-12-29 13:05:44] [[ ! :/usr/share/man:: =~ :/usr/share/man: ]] +++ [22-12-29 13:05:44] unset _mlcode _mlret +++ [22-12-29 13:05:44] '[' -n '' ']' + [22-12-29 13:05:44] [[ openmpi == *\p\s\m\2* ]] ++ [22-12-29 13:05:44] rpm -ql openmpi ++ [22-12-29 13:05:44] grep modulefiles ++ [22-12-29 13:05:44] tail -n 1 + [22-12-29 13:05:44] local mod_file=/usr/share/modulefiles/mpi/openmpi-x86_64 + [22-12-29 13:05:44] echo 'module purge' + [22-12-29 13:05:44] echo 'module load /usr/share/modulefiles/mpi/openmpi-x86_64' + [22-12-29 13:05:44] source /root/.bashrc + [22-12-29 13:05:45] module list ++ [22-12-29 13:05:45] /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash list Currently Loaded Modulefiles: 1) /usr/share/modulefiles/mpi/openmpi-x86_64 + [22-12-29 13:05:45] eval + [22-12-29 13:05:45] _mlstatus=0 + [22-12-29 13:05:45] return 0 ++ [22-12-29 13:05:45] echo /usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:05:45] sed -e 's|/usr/lib64/openmpi[^/]*/bin:||g' + [22-12-29 13:05:45] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:05:45] echo /usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:05:45] sed -e 's|/usr/lib64/mvapich[^/]*/bin:||g' + [22-12-29 13:05:45] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin ++ [22-12-29 13:05:45] sed -e 's|/usr/lib64/mpich[^/]*/bin:||g' ++ [22-12-29 13:05:45] echo /usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:05:45] PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:05:45] export PATH=/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:05:45] PATH=/usr/lib64/openmpi/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin + [22-12-29 13:05:45] MPI_create_hostfiles + [22-12-29 13:05:45] '[' -z 172.31.40.121 -o -z '' ']' + [22-12-29 13:05:45] echo 'Please run /kernel/infiniband/env_setup before MPI_create_hostfiles' Please run /kernel/infiniband/env_setup before MPI_create_hostfiles + [22-12-29 13:05:45] return 1 + [22-12-29 13:05:45] echo 'MPI_setup: Failed to generate MPI hostfiles' MPI_setup: Failed to generate MPI hostfiles + [22-12-29 13:05:45] echo 'MPI_setup: (This is expected on singlehost environment setup)' MPI_setup: (This is expected on singlehost environment setup) + [22-12-29 13:05:45] '[' -x /bin/systemctl ']' + [22-12-29 13:05:45] systemctl stop firewalld.service + [22-12-29 13:05:45] systemctl disable firewalld.service + [22-12-29 13:05:45] systemctl status firewalld.service ○ firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) + [22-12-29 13:05:45] MPI_dual_port_fix + [22-12-29 13:05:45] [[ '' == *\q\i\b* ]] ++ [22-12-29 13:05:45] RQA_get_rhel_major ++ [22-12-29 13:05:45] awk -F . '{print $1}' +++ [22-12-29 13:05:45] RQA_get_rhel_release +++ [22-12-29 13:05:45] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:05:45] echo 9 + [22-12-29 13:05:45] '[' 9 -ge 7 ']' + [22-12-29 13:05:45] RQA_pkg_install targetcli iscsi-initiator-utils lsscsi + [22-12-29 13:05:45] PKG_LIST= + [22-12-29 13:05:45] for p in "$@" + [22-12-29 13:05:45] rpm -q targetcli targetcli-2.1.53-7.el9.noarch + [22-12-29 13:05:45] for p in "$@" + [22-12-29 13:05:45] rpm -q iscsi-initiator-utils iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 + [22-12-29 13:05:45] for p in "$@" + [22-12-29 13:05:45] rpm -q lsscsi lsscsi-0.32-6.el9.x86_64 + [22-12-29 13:05:45] '[' '!' -z '' ']' + [22-12-29 13:05:45] target_name=iqn.2003-01.org.linux-iscsi.rdma.x8664:test-target ++ [22-12-29 13:05:45] RQA_get_rhel_major ++ [22-12-29 13:05:45] awk -F . '{print $1}' +++ [22-12-29 13:05:45] RQA_get_rhel_release +++ [22-12-29 13:05:45] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:05:45] echo 9 + [22-12-29 13:05:45] [[ 9 -ge 8 ]] + [22-12-29 13:05:45] RQA_pkg_install iperf3 + [22-12-29 13:05:45] PKG_LIST= + [22-12-29 13:05:45] for p in "$@" + [22-12-29 13:05:45] rpm -q iperf3 iperf3-3.9-9.el9.x86_64 + [22-12-29 13:05:45] '[' '!' -z '' ']' + [22-12-29 13:05:45] RQA_pkg_install net-tools + [22-12-29 13:05:45] PKG_LIST= + [22-12-29 13:05:45] for p in "$@" + [22-12-29 13:05:45] rpm -q net-tools net-tools-2.0-0.62.20160912git.el9.x86_64 + [22-12-29 13:05:45] '[' '!' -z '' ']' + [22-12-29 13:05:45] [[ client == \c\o\m\m\o\n ]] + [22-12-29 13:05:45] [[ client == \s\e\r\v\e\r ]] + [22-12-29 13:05:45] [[ client == \c\l\i\e\n\t ]] + [22-12-29 13:05:45] client_tests + [22-12-29 13:05:45] [[ -z 172.31.40.121 ]] + [22-12-29 13:05:45] timeout 3m ping -i 0.2 -c 100 172.31.40.121 PING 172.31.40.121 (172.31.40.121) 56(84) bytes of data. 64 bytes from 172.31.40.121: icmp_seq=1 ttl=64 time=0.247 ms 64 bytes from 172.31.40.121: icmp_seq=2 ttl=64 time=0.107 ms 64 bytes from 172.31.40.121: icmp_seq=3 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=4 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=5 ttl=64 time=0.073 ms 64 bytes from 172.31.40.121: icmp_seq=6 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=7 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=8 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=9 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=10 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=11 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=12 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=13 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=14 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=15 ttl=64 time=0.062 ms 64 bytes from 172.31.40.121: icmp_seq=16 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=17 ttl=64 time=0.076 ms 64 bytes from 172.31.40.121: icmp_seq=18 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=19 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=20 ttl=64 time=0.058 ms 64 bytes from 172.31.40.121: icmp_seq=21 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=22 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=23 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=24 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=25 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=26 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=27 ttl=64 time=0.059 ms 64 bytes from 172.31.40.121: icmp_seq=28 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=29 ttl=64 time=0.072 ms 64 bytes from 172.31.40.121: icmp_seq=30 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=31 ttl=64 time=0.076 ms 64 bytes from 172.31.40.121: icmp_seq=32 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=33 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=34 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=35 ttl=64 time=0.068 ms 64 bytes from 172.31.40.121: icmp_seq=36 ttl=64 time=0.068 ms 64 bytes from 172.31.40.121: icmp_seq=37 ttl=64 time=0.068 ms 64 bytes from 172.31.40.121: icmp_seq=38 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=39 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=40 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=41 ttl=64 time=0.073 ms 64 bytes from 172.31.40.121: icmp_seq=42 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=43 ttl=64 time=0.062 ms 64 bytes from 172.31.40.121: icmp_seq=44 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=45 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=46 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=47 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=48 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=49 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=50 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=51 ttl=64 time=0.069 ms 64 bytes from 172.31.40.121: icmp_seq=52 ttl=64 time=0.067 ms 64 bytes from 172.31.40.121: icmp_seq=53 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=54 ttl=64 time=0.069 ms 64 bytes from 172.31.40.121: icmp_seq=55 ttl=64 time=0.068 ms 64 bytes from 172.31.40.121: icmp_seq=56 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=57 ttl=64 time=0.069 ms 64 bytes from 172.31.40.121: icmp_seq=58 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=59 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=60 ttl=64 time=0.062 ms 64 bytes from 172.31.40.121: icmp_seq=61 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=62 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=63 ttl=64 time=0.062 ms 64 bytes from 172.31.40.121: icmp_seq=64 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=65 ttl=64 time=0.068 ms 64 bytes from 172.31.40.121: icmp_seq=66 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=67 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=68 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=69 ttl=64 time=0.066 ms 64 bytes from 172.31.40.121: icmp_seq=70 ttl=64 time=0.065 ms 64 bytes from 172.31.40.121: icmp_seq=71 ttl=64 time=0.069 ms 64 bytes from 172.31.40.121: icmp_seq=72 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=73 ttl=64 time=0.063 ms 64 bytes from 172.31.40.121: icmp_seq=74 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=75 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=76 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=77 ttl=64 time=0.064 ms 64 bytes from 172.31.40.121: icmp_seq=78 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=79 ttl=64 time=0.094 ms 64 bytes from 172.31.40.121: icmp_seq=80 ttl=64 time=0.082 ms 64 bytes from 172.31.40.121: icmp_seq=81 ttl=64 time=0.084 ms 64 bytes from 172.31.40.121: icmp_seq=82 ttl=64 time=0.073 ms 64 bytes from 172.31.40.121: icmp_seq=83 ttl=64 time=0.076 ms 64 bytes from 172.31.40.121: icmp_seq=84 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=85 ttl=64 time=0.074 ms 64 bytes from 172.31.40.121: icmp_seq=86 ttl=64 time=0.083 ms 64 bytes from 172.31.40.121: icmp_seq=87 ttl=64 time=0.092 ms 64 bytes from 172.31.40.121: icmp_seq=88 ttl=64 time=0.086 ms 64 bytes from 172.31.40.121: icmp_seq=89 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=90 ttl=64 time=0.072 ms 64 bytes from 172.31.40.121: icmp_seq=91 ttl=64 time=0.096 ms 64 bytes from 172.31.40.121: icmp_seq=92 ttl=64 time=0.095 ms 64 bytes from 172.31.40.121: icmp_seq=93 ttl=64 time=0.092 ms 64 bytes from 172.31.40.121: icmp_seq=94 ttl=64 time=0.084 ms 64 bytes from 172.31.40.121: icmp_seq=95 ttl=64 time=0.070 ms 64 bytes from 172.31.40.121: icmp_seq=96 ttl=64 time=0.081 ms 64 bytes from 172.31.40.121: icmp_seq=97 ttl=64 time=0.085 ms 64 bytes from 172.31.40.121: icmp_seq=98 ttl=64 time=0.087 ms 64 bytes from 172.31.40.121: icmp_seq=99 ttl=64 time=0.071 ms 64 bytes from 172.31.40.121: icmp_seq=100 ttl=64 time=0.088 ms --- 172.31.40.121 ping statistics --- 100 packets transmitted, 100 received, 0% packet loss, time 20586ms rtt min/avg/max/mdev = 0.058/0.071/0.247/0.019 ms + [22-12-29 13:06:06] RQA_check_result -r 0 -t 'ping server - 172.31.40.121' + [22-12-29 13:06:06] local test_pass=0 + [22-12-29 13:06:06] local test_skip=777 + [22-12-29 13:06:06] test 4 -gt 0 + [22-12-29 13:06:06] case $1 in + [22-12-29 13:06:06] local rc=0 + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] test 2 -gt 0 + [22-12-29 13:06:06] case $1 in + [22-12-29 13:06:06] local 'msg=ping server - 172.31.40.121' + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] test 0 -gt 0 + [22-12-29 13:06:06] '[' -z 0 -o -z 'ping server - 172.31.40.121' ']' + [22-12-29 13:06:06] '[' -z '' ']' + [22-12-29 13:06:06] source /etc/environment ++ [22-12-29 13:06:06] TEST_RESULT_FILE=/tmp/tmp.zXxktCJnMx/results_cki-sanity.txt + [22-12-29 13:06:06] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:06:06] '[' 0 -eq 0 ']' + [22-12-29 13:06:06] local test_result=PASS + [22-12-29 13:06:06] printf '%10s | %6s | %s\n' PASS 0 'ping server - 172.31.40.121' + [22-12-29 13:06:06] set +x --- - TEST RESULT FOR sanity - Test: ping server - 172.31.40.121 - Result: PASS - Return: 0 --- + [22-12-29 13:06:06] timeout 3m ping6 -i 0.2 -c 100 fe80::268a:7ff:fe4b:f094% ping6: fe80::268a:7ff:fe4b:f094%: Name or service not known + [22-12-29 13:06:06] RQA_check_result -r 2 -t 'ping6 server - fe80::268a:7ff:fe4b:f094%' + [22-12-29 13:06:06] local test_pass=0 + [22-12-29 13:06:06] local test_skip=777 + [22-12-29 13:06:06] test 4 -gt 0 + [22-12-29 13:06:06] case $1 in + [22-12-29 13:06:06] local rc=2 + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] test 2 -gt 0 + [22-12-29 13:06:06] case $1 in + [22-12-29 13:06:06] local 'msg=ping6 server - fe80::268a:7ff:fe4b:f094%' + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] shift + [22-12-29 13:06:06] test 0 -gt 0 + [22-12-29 13:06:06] '[' -z 2 -o -z 'ping6 server - fe80::268a:7ff:fe4b:f094%' ']' + [22-12-29 13:06:06] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:06:06] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:06:06] '[' 2 -eq 0 ']' + [22-12-29 13:06:06] '[' 2 -eq 777 ']' + [22-12-29 13:06:06] local test_result=FAIL + [22-12-29 13:06:06] export result=FAIL + [22-12-29 13:06:06] result=FAIL + [22-12-29 13:06:06] [[ ! -z '' ]] + [22-12-29 13:06:06] printf '%10s | %6s | %s\n' FAIL 2 'ping6 server - fe80::268a:7ff:fe4b:f094%' + [22-12-29 13:06:06] set +x --- - TEST RESULT FOR sanity - Test: ping6 server - fe80::268a:7ff:fe4b:f094% - Result: FAIL - Return: 2 --- + [22-12-29 13:06:06] RQA_skip_test mpi + [22-12-29 13:06:06] local skp=mpi + [22-12-29 13:06:06] local skp_rtn=0 + [22-12-29 13:06:06] '[' '!' -z mpi ']' + [22-12-29 13:06:06] [[ '' == *\m\p\i* ]] + [22-12-29 13:06:06] return 0 + [22-12-29 13:06:06] skip_mpi=0 + [22-12-29 13:06:06] '[' 0 -eq 0 ']' + [22-12-29 13:06:06] source /root/.bashrc + [22-12-29 13:06:07] MPIPATH=/usr/lib64/openmpi/bin + [22-12-29 13:06:07] [[ openmpi == \o\p\e\n\m\p\i\3 ]] ++ [22-12-29 13:06:07] MPI_get_openmpi_mca_param ++ [22-12-29 13:06:07] local server_dev=mlx5_0:1 ++ [22-12-29 13:06:07] local client_dev=: ++ [22-12-29 13:06:07] local 'ompi_flags=-mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include' ++ [22-12-29 13:06:07] local __mca_verbose= ++ [22-12-29 13:06:07] [[ mlx5_0:1 == \: ]] ++ [22-12-29 13:06:07] ompi_flags='-mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1' ++ [22-12-29 13:06:07] case ${RDMA_DRIVER} in ++ [22-12-29 13:06:07] ompi_flags='-mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl openib,self' ++ [22-12-29 13:06:07] __mca_verbose='--mca mtl_base_verbose 100 --mca btl_openib_verbose 100' ++ [22-12-29 13:06:07] echo '-mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100' + [22-12-29 13:06:07] mpirun_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100' ++ [22-12-29 13:06:07] ucx_mpi_flags '--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100' ++ [22-12-29 13:06:07] local '_local_mpi_flags=--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100' +++ [22-12-29 13:06:07] echo rdma-cki-01 ++ [22-12-29 13:06:07] local _client_nm=rdma-cki-01 +++ [22-12-29 13:06:07] echo rdma-dev-21 ++ [22-12-29 13:06:07] local _server_nm=rdma-dev-21 ++ [22-12-29 13:06:07] local '__mca_verbose=--mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100' ++ [22-12-29 13:06:07] rpm -q ucx ++ [22-12-29 13:06:07] [[ 0 -eq 0 ]] +++ [22-12-29 13:06:07] echo --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl '^psm2,psm,ofi' -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 +++ [22-12-29 13:06:07] awk -F - '{for (i=1; i<=NF; i++) if ($i ~ /mca btl /) print $i}' ++ [22-12-29 13:06:07] mca_btl='mca btl openib,self ' ++ [22-12-29 13:06:07] [[ ! -z mca btl openib,self ]] +++ [22-12-29 13:06:07] echo mca btl openib,self +++ [22-12-29 13:06:07] sed 's/mca btl .*/mca btl ^openib /g' ++ [22-12-29 13:06:07] x_openib='mca btl ^openib ' +++ [22-12-29 13:06:07] echo --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl '^psm2,psm,ofi' -mca btl openib,self --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 +++ [22-12-29 13:06:07] sed 's|mca btl openib,self |mca btl ^openib |g' ++ [22-12-29 13:06:07] _local_mpi_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100' +++ [22-12-29 13:06:07] echo --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl '^psm2,psm,ofi' -mca btl '^openib' --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 +++ [22-12-29 13:06:07] awk -F - '{for (i=1; i<=NF; i++) if ($i ~ /mca pml /) print $i}' ++ [22-12-29 13:06:07] mca_pml= ++ [22-12-29 13:06:07] [[ ! -z '' ]] ++ [22-12-29 13:06:07] _local_mpi_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx ' ++ [22-12-29 13:06:07] [[ --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx != *\m\c\a\ \o\s\c\ \u\c\x* ]] ++ [22-12-29 13:06:07] [[ --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx == *\m\c\a\ \p\m\l\ \u\c\x* ]] ++ [22-12-29 13:06:07] _local_mpi_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx ' ++ [22-12-29 13:06:07] [[ --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx != *\-\x\ \U\C\X\_\N\E\T\_\D\E\V\I\C\E\S\=* ]] ++ [22-12-29 13:06:07] [[ rdma-cki-01 != \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:06:07] [[ rdma-dev-21 != \r\d\m\a\-\d\e\v\-\2\0 ]] ++ [22-12-29 13:06:07] _local_mpi_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES=' ++ [22-12-29 13:06:07] echo '--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES= --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100' + [22-12-29 13:06:07] ucx_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES= --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100' + [22-12-29 13:06:07] mpirun_flags='--allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include :,mlx5_0:1 -mca mtl ^psm2,psm,ofi -mca btl ^openib --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES= --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100' + [22-12-29 13:06:07] hfiles=($(echo $HFILES)) ++ [22-12-29 13:06:07] echo + [22-12-29 13:06:07] hfile= + [22-12-29 13:06:07] '[' '!' -z ']' + [22-12-29 13:06:07] rhts_sync_set -s mpi_done_--0 /usr/bin/rhts_sync_set -s mpi_done_--0 -- State set = 12_mpi_done_--0 + [22-12-29 13:06:07] route add -net 224.0.0.0 netmask 240.0.0.0 dev Usage: inet_route [-vF] del {-host|-net} Target[/prefix] [gw Gw] [metric M] [[dev] If] inet_route [-vF] add {-host|-net} Target[/prefix] [gw Gw] [metric M] [netmask N] [mss Mss] [window W] [irtt I] [mod] [dyn] [reinstate] [[dev] If] inet_route [-vF] add {-host|-net} Target[/prefix] [metric M] reject inet_route [-FC] flush NOT supported ++ [22-12-29 13:06:07] RQA_get_rhel_major ++ [22-12-29 13:06:07] awk -F . '{print $1}' +++ [22-12-29 13:06:07] RQA_get_rhel_release +++ [22-12-29 13:06:07] grep -o '[0-9]*\.*[0-9]*' /etc/redhat-release ++ [22-12-29 13:06:07] echo 9 + [22-12-29 13:06:07] '[' 9 -ge 8 ']' + [22-12-29 13:06:07] ip maddr show dev Command line is not complete. Try option "help" + [22-12-29 13:06:07] RQA_check_result -r 255 -t 'ip multicast addr' + [22-12-29 13:06:07] local test_pass=0 + [22-12-29 13:06:07] local test_skip=777 + [22-12-29 13:06:07] test 4 -gt 0 + [22-12-29 13:06:07] case $1 in + [22-12-29 13:06:07] local rc=255 + [22-12-29 13:06:07] shift + [22-12-29 13:06:07] shift + [22-12-29 13:06:07] test 2 -gt 0 + [22-12-29 13:06:07] case $1 in + [22-12-29 13:06:07] local 'msg=ip multicast addr' + [22-12-29 13:06:07] shift + [22-12-29 13:06:07] shift + [22-12-29 13:06:07] test 0 -gt 0 + [22-12-29 13:06:07] '[' -z 255 -o -z 'ip multicast addr' ']' + [22-12-29 13:06:07] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:06:07] '[' -z /tmp/tmp.zXxktCJnMx/results_cki-sanity.txt ']' + [22-12-29 13:06:07] '[' 255 -eq 0 ']' + [22-12-29 13:06:07] '[' 255 -eq 777 ']' + [22-12-29 13:06:07] local test_result=FAIL + [22-12-29 13:06:07] export result=FAIL + [22-12-29 13:06:07] result=FAIL + [22-12-29 13:06:07] [[ ! -z '' ]] + [22-12-29 13:06:07] printf '%10s | %6s | %s\n' FAIL 255 'ip multicast addr' + [22-12-29 13:06:07] set +x --- - TEST RESULT FOR sanity - Test: ip multicast addr - Result: FAIL - Return: 255 --- + [22-12-29 13:06:07] [[ ! '' == *\c\x\g\b* ]] + [22-12-29 13:06:07] RQA_skip_test rping + [22-12-29 13:06:07] local skp=rping + [22-12-29 13:06:07] local skp_rtn=0 + [22-12-29 13:06:07] '[' '!' -z rping ']' + [22-12-29 13:06:07] [[ '' == *\r\p\i\n\g* ]] + [22-12-29 13:06:07] return 0 + [22-12-29 13:06:07] skip_rping=0 + [22-12-29 13:06:07] '[' 0 -eq 0 ']' + [22-12-29 13:06:07] rhts_sync_block -s rping-ready_--0 rdma-dev-21 /usr/bin/rhts_sync_block -s rping-ready_--0 rdma-dev-21 -- Blocking state(s) = 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 5 bytes, ret 0 [1, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 5 bytes, ret 0 [1, 0] Retrying rstrnt-sync-block for state 12_rping-ready_--0 Server rdma-dev-21 not reported state 12_rping-ready_--0 for Multihost Sync rcvd 0 bytes, ret 0 [0, 0]