[ 115.877742] Running test [R:12688642 T:5 - Storage - blktests - nvmeof-mp - Kernel: 6.0.0-0.rc7.987a926c1d8a.51.test.fc38.aarch64] [ 118.344369] run blktests nvmeof-mp/001 at 2022-09-30 18:54:18 [ 118.495681] alua: device handler registered [ 118.505174] emc: device handler registered [ 118.514529] rdac: device handler registered [ 118.529367] null_blk: module loaded [ 118.571450] null_blk: disk nullb0 created [ 118.587098] null_blk: disk nullb1 created [ 118.639013] rdma_rxe: loaded [ 118.662335] infiniband eno1_rxe: set active [ 118.666511] infiniband eno1_rxe: added eno1 [ 118.844054] nvmet: adding nsid 1 to subsystem nvme-test [ 118.865116] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 118.992835] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 119.007033] nvme nvme0: creating 128 I/O queues. [ 119.111515] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 119.147091] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 119.193108] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 120.345883] rdma_rxe: unloaded [ 121.244309] run blktests nvmeof-mp/001 at 2022-09-30 18:54:21 [ 121.375038] null_blk: module loaded [ 121.406731] null_blk: disk nullb0 created [ 121.420710] null_blk: disk nullb1 created [ 121.469953] SoftiWARP attached [ 121.612623] nvmet: adding nsid 1 to subsystem nvme-test [ 121.632618] iwpm_register_pid: Unable to send a nlmsg (client = 2) [ 121.638843] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 121.729146] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 121.743343] nvme nvme0: creating 128 I/O queues. [ 121.918675] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 121.959934] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 122.007614] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 123.056564] SoftiWARP detached [ 123.675518] run blktests nvmeof-mp/002 at 2022-09-30 18:54:24 [ 123.794889] null_blk: module loaded [ 123.826393] null_blk: disk nullb0 created [ 123.842365] null_blk: disk nullb1 created [ 123.892768] rdma_rxe: loaded [ 123.906859] infiniband eno1_rxe: set active [ 123.911045] infiniband eno1_rxe: added eno1 [ 124.052340] nvmet: adding nsid 1 to subsystem nvme-test [ 124.073202] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 124.192353] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 124.206585] nvme nvme0: creating 128 I/O queues. [ 124.345144] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 124.373647] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 124.466211] device-mapper: multipath service-time: version 0.3.0 loaded [ 125.228244] EXT4-fs (dm-1): mounted filesystem without journal. Quota mode: none. [-- MARK -- Fri Sep 30 22:55:00 2022] [ 160.486771] EXT4-fs (dm-1): unmounting filesystem. [ 160.755246] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 161.814262] rdma_rxe: unloaded [ 162.875411] run blktests nvmeof-mp/002 at 2022-09-30 18:55:03 [ 163.015312] null_blk: module loaded [ 163.046241] null_blk: disk nullb0 created [ 163.059966] null_blk: disk nullb1 created [ 163.108458] SoftiWARP attached [ 163.249633] nvmet: adding nsid 1 to subsystem nvme-test [ 163.270991] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 163.349840] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 163.364062] nvme nvme0: creating 128 I/O queues. [ 163.550710] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 163.590890] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 164.370183] EXT4-fs (dm-1): mounted filesystem without journal. Quota mode: none. [ 196.188305] nvme nvme0: I/O 5 QID 80 timeout [ 196.192585] nvme nvme0: starting error recovery [ 196.258041] nvme nvme0: I/O 19 QID 98 timeout [ 196.262545] nvme0n1: I/O Cmd(0x1) @ LBA 51460, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.270557] recoverable transport error, dev nvme0n1, sector 51460 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.273751] nvme nvme0: nvme_rdma_post_recv failed with error code -107 [ 196.280427] nvme0n1: I/O Cmd(0x1) @ LBA 44130, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280432] nvme0n1: I/O Cmd(0x1) @ LBA 20684, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280438] nvme0n1: I/O Cmd(0x0) @ LBA 18446744073709551615, 0 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280447] recoverable transport error, dev nvme0n1, sector 44130 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280448] recoverable transport error, dev nvme0n1, sector 20684 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280449] nvme0n1: I/O Cmd(0x1) @ LBA 58962, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280453] nvme0n1: I/O Cmd(0x1) @ LBA 8880, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280458] device-mapper: multipath: 253:1: Failing path 259:0. [ 196.280462] nvme0n1: I/O Cmd(0x1) @ LBA 63546, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280467] recoverable transport error, dev nvme0n1, sector 58962 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280465] nvme0n1: I/O Cmd(0x1) @ LBA 6352, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280470] recoverable transport error, dev nvme0n1, sector 8880 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280468] nvme0n1: I/O Cmd(0x1) @ LBA 29074, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280474] recoverable transport error, dev nvme0n1, sector 63546 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280476] nvme0n1: I/O Cmd(0x1) @ LBA 36234, 8 blocks, I/O Error (sct 0x3 / sc 0x70) [ 196.280481] recoverable transport error, dev nvme0n1, sector 6352 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280486] recoverable transport error, dev nvme0n1, sector 29074 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.280493] recoverable transport error, dev nvme0n1, sector 36234 op 0x1:(WRITE) flags 0xca00 phys_seg 1 prio class 2 [ 196.533181] nvme nvme0: Reconnecting in 10 seconds... [ 207.069502] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 207.083645] nvme nvme0: creating 128 I/O queues. [ 207.308988] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 207.325916] nvme nvme0: Successfully reconnected (1 attempts) [ 212.105944] device-mapper: multipath: 253:1: Reinstating path 259:0. [ 212.382260] EXT4-fs (dm-1): unmounting filesystem. [ 212.743978] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 213.916038] SoftiWARP detached [ 214.794383] run blktests nvmeof-mp/005 at 2022-09-30 18:55:55 [ 214.924903] null_blk: module loaded [ 214.957418] null_blk: disk nullb0 created [ 214.968635] null_blk: disk nullb1 created [ 215.012800] rdma_rxe: loaded [ 215.026894] infiniband eno1_rxe: set active [ 215.031080] infiniband eno1_rxe: added eno1 [ 215.174801] nvmet: adding nsid 1 to subsystem nvme-test [ 215.195640] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 215.323060] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 215.337141] nvme nvme0: creating 128 I/O queues. [ 215.488546] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 215.521622] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 227.883769] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 228.988528] rdma_rxe: unloaded [ 229.822112] run blktests nvmeof-mp/005 at 2022-09-30 18:56:10 [ 229.944903] null_blk: module loaded [ 229.975717] null_blk: disk nullb0 created [ 229.986567] null_blk: disk nullb1 created [ 230.028941] SoftiWARP attached [ 230.170221] nvmet: adding nsid 1 to subsystem nvme-test [ 230.190596] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 230.279682] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 230.293800] nvme nvme0: creating 128 I/O queues. [ 230.488918] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 230.530454] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 243.073868] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 244.197472] SoftiWARP detached [ 244.862719] run blktests nvmeof-mp/006 at 2022-09-30 18:56:25 [ 244.984703] null_blk: module loaded [ 245.024250] null_blk: disk nullb0 created [ 245.035388] null_blk: disk nullb1 created [ 245.084302] rdma_rxe: loaded [ 245.098442] infiniband eno1_rxe: set active [ 245.102617] infiniband eno1_rxe: added eno1 [ 245.244988] nvmet: adding nsid 1 to subsystem nvme-test [ 245.265078] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 245.386503] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 245.400507] nvme nvme0: creating 128 I/O queues. [ 245.558385] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 245.591740] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 258.174244] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 259.368301] rdma_rxe: unloaded [ 260.284003] run blktests nvmeof-mp/006 at 2022-09-30 18:56:40 [ 260.414389] null_blk: module loaded [ 260.459756] null_blk: disk nullb0 created [ 260.475230] null_blk: disk nullb1 created [ 260.525199] SoftiWARP attached [ 260.668307] nvmet: adding nsid 1 to subsystem nvme-test [ 260.688579] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 260.769355] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 260.783483] nvme nvme0: creating 128 I/O queues. [ 260.992190] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 261.037404] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 273.764896] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 274.896926] SoftiWARP detached [ 275.568368] run blktests nvmeof-mp/009 at 2022-09-30 18:56:56 [ 275.694161] null_blk: module loaded [ 275.724914] null_blk: disk nullb0 created [ 275.734779] null_blk: disk nullb1 created [ 275.782865] rdma_rxe: loaded [ 275.796271] infiniband eno1_rxe: set active [ 275.800457] infiniband eno1_rxe: added eno1 [ 275.939536] nvmet: adding nsid 1 to subsystem nvme-test [ 275.960153] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 276.075082] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 276.088865] nvme nvme0: creating 128 I/O queues. [ 276.246722] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 276.277264] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 288.574984] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 289.666819] rdma_rxe: unloaded [ 290.560008] run blktests nvmeof-mp/009 at 2022-09-30 18:57:11 [ 290.704093] null_blk: module loaded [ 290.746863] null_blk: disk nullb0 created [ 290.757735] null_blk: disk nullb1 created [ 290.806389] SoftiWARP attached [ 290.947267] nvmet: adding nsid 1 to subsystem nvme-test [ 290.966967] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 291.037232] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 291.051264] nvme nvme0: creating 128 I/O queues. [ 291.219204] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 291.259339] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 303.824897] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 304.915540] SoftiWARP detached [ 305.555767] run blktests nvmeof-mp/010 at 2022-09-30 18:57:26 [ 305.693865] null_blk: module loaded [ 305.726677] null_blk: disk nullb0 created [ 305.735245] null_blk: disk nullb1 created [ 305.782878] rdma_rxe: loaded [ 305.795548] infiniband eno1_rxe: set active [ 305.799741] infiniband eno1_rxe: added eno1 [ 305.936245] nvmet: adding nsid 1 to subsystem nvme-test [ 305.956224] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 306.072235] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 306.086106] nvme nvme0: creating 128 I/O queues. [ 306.272427] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 306.301611] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 318.944232] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 320.177598] rdma_rxe: unloaded [ 321.013410] run blktests nvmeof-mp/010 at 2022-09-30 18:57:41 [ 321.153537] null_blk: module loaded [ 321.183465] null_blk: disk nullb0 created [ 321.192675] null_blk: disk nullb1 created [ 321.239317] SoftiWARP attached [ 321.379435] nvmet: adding nsid 1 to subsystem nvme-test [ 321.399258] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 321.475774] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 321.489805] nvme nvme0: creating 128 I/O queues. [ 321.672138] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 321.711726] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 334.143154] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 335.404316] SoftiWARP detached [ 336.065024] run blktests nvmeof-mp/011 at 2022-09-30 18:57:56 [ 336.201302] null_blk: module loaded [ 336.243205] null_blk: disk nullb0 created [ 336.252280] null_blk: disk nullb1 created [ 336.300291] rdma_rxe: loaded [ 336.314162] infiniband eno1_rxe: set active [ 336.318354] infiniband eno1_rxe: added eno1 [ 336.457409] nvmet: adding nsid 1 to subsystem nvme-test [ 336.477404] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 336.600110] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 336.614149] nvme nvme0: creating 128 I/O queues. [ 336.761946] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 336.795476] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 372.533037] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 373.753704] rdma_rxe: unloaded [ 374.767752] run blktests nvmeof-mp/011 at 2022-09-30 18:58:35 [ 374.902402] null_blk: module loaded [ 374.936935] null_blk: disk nullb0 created [ 374.947445] null_blk: disk nullb1 created [ 374.989099] SoftiWARP attached [ 375.130943] nvmet: adding nsid 1 to subsystem nvme-test [ 375.151104] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 375.260220] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 375.274419] nvme nvme0: creating 128 I/O queues. [ 375.466107] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 375.512580] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 411.823482] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 412.934627] SoftiWARP detached [ 413.818692] run blktests nvmeof-mp/012 at 2022-09-30 18:59:14 [ 413.963546] null_blk: module loaded [ 414.002868] null_blk: disk nullb0 created [ 414.011825] null_blk: disk nullb1 created [ 414.054491] rdma_rxe: loaded [ 414.069055] infiniband eno1_rxe: set active [ 414.073230] infiniband eno1_rxe: added eno1 [ 414.216578] nvmet: adding nsid 1 to subsystem nvme-test [ 414.238148] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 414.360852] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 414.374936] nvme nvme0: creating 128 I/O queues. [ 414.557579] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 414.592669] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 423.366724] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 423.871278] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 423.885396] nvme nvme0: creating 128 I/O queues. [ 424.039481] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 424.068573] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 433.052804] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 434.135928] rdma_rxe: unloaded [ 435.068660] run blktests nvmeof-mp/012 at 2022-09-30 18:59:35 [ 435.203355] null_blk: module loaded [ 435.236354] null_blk: disk nullb0 created [ 435.247043] null_blk: disk nullb1 created [ 435.294487] SoftiWARP attached [ 435.438044] nvmet: adding nsid 1 to subsystem nvme-test [ 435.458638] nvmet_rdma: enabling port 1 (10.19.242.43:7777) [ 435.548352] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 435.562320] nvme nvme0: creating 128 I/O queues. [ 435.752753] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 435.792535] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 444.254652] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 444.697696] nvmet: creating nvm controller 1 for subsystem nvme-test for NQN nqn.2014-08.org.nvmexpress:uuid:f8643fdb-e1a7-481c-af6a-dad13393d89a. [ 444.712157] nvme nvme0: creating 128 I/O queues. [ 444.871557] nvme nvme0: mapped 128/0/0 default/read/poll queues. [ 444.911730] nvme nvme0: new ctrl: NQN "nvme-test", addr 10.19.242.43:7777 [ 453.193832] nvme nvme0: Removing ctrl: NQN "nvme-test" [ 454.264413] SoftiWARP detached