[ 222.959695] Running test [R:13258406 T:7 - RDMA Sanity Test - RoCE - Server - Kernel: 5.14.0-239.rt14.239.1773_752168396.el9.x86_64] [ 253.180594] systemd-rc-local-generator[15465]: /etc/rc.d/rc.local is not marked executable, skipping. [ 255.029277] systemd-rc-local-generator[15645]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Thu Jan 19 17:45:00 2023] [ 641.815404] systemd-rc-local-generator[31421]: /etc/rc.d/rc.local is not marked executable, skipping. [ 646.286681] systemd-rc-local-generator[31489]: /etc/rc.d/rc.local is not marked executable, skipping. [ 655.633196] systemd-rc-local-generator[31663]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Thu Jan 19 17:50:00 2023] [ 805.495761] systemd-rc-local-generator[40398]: /etc/rc.d/rc.local is not marked executable, skipping. [ 1051.761800] XFS (sda4): Unmounting Filesystem [-- MARK -- Thu Jan 19 17:55:00 2023] [ 1124.601610] XFS (sda4): Mounting V5 Filesystem [ 1124.743907] XFS (sda4): Ending clean mount [ 1125.304430] NFSD: Using nfsdcld client tracking operations. [ 1125.304434] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000) [ 1355.494603] INFO: task nfsd:42162 blocked for more than 122 seconds. [ 1355.501008] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1355.508273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.516150] task:nfsd state:D stack: 0 pid:42162 ppid: 2 flags:0x00004000 [ 1355.524554] Call Trace: [ 1355.527018] [ 1355.529135] __schedule+0x25b/0x660 [ 1355.532654] schedule+0x5d/0xd0 [ 1355.535817] schedule_timeout+0x151/0x190 [ 1355.539855] ? preempt_count_add+0x70/0xa0 [ 1355.543980] __wait_for_common+0x8e/0x1c0 [ 1355.548016] ? usleep_range_state+0x90/0x90 [ 1355.552233] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.558033] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.563042] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.567819] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.571888] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.576999] nfsd+0xc3/0x190 [nfsd] [ 1355.580535] kthread+0x106/0x130 [ 1355.583783] ? kthread_complete_and_exit+0x20/0x20 [ 1355.588606] ret_from_fork+0x22/0x30 [ 1355.592210] [ 1355.594411] INFO: task nfsd:42163 blocked for more than 122 seconds. [ 1355.600804] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1355.608071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.615947] task:nfsd state:D stack: 0 pid:42163 ppid: 2 flags:0x00004000 [ 1355.624357] Call Trace: [ 1355.626820] [ 1355.628933] __schedule+0x25b/0x660 [ 1355.632444] schedule+0x5d/0xd0 [ 1355.635605] schedule_timeout+0x151/0x190 [ 1355.639640] ? preempt_count_add+0x70/0xa0 [ 1355.643765] __wait_for_common+0x8e/0x1c0 [ 1355.647800] ? usleep_range_state+0x90/0x90 [ 1355.652011] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.657808] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.662819] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.667586] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.671651] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.676757] nfsd+0xc3/0x190 [nfsd] [ 1355.680292] kthread+0x106/0x130 [ 1355.683541] ? kthread_complete_and_exit+0x20/0x20 [ 1355.688369] ret_from_fork+0x22/0x30 [ 1355.691969] [ 1355.694172] INFO: task nfsd:42164 blocked for more than 123 seconds. [ 1355.700563] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1355.707828] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.715706] task:nfsd state:D stack: 0 pid:42164 ppid: 2 flags:0x00004000 [ 1355.724109] Call Trace: [ 1355.726572] [ 1355.728685] __schedule+0x25b/0x660 [ 1355.732198] schedule+0x5d/0xd0 [ 1355.735358] schedule_timeout+0x151/0x190 [ 1355.739399] ? preempt_count_add+0x70/0xa0 [ 1355.743523] __wait_for_common+0x8e/0x1c0 [ 1355.747556] ? usleep_range_state+0x90/0x90 [ 1355.751772] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.757567] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.762577] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.767343] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.771408] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.776515] nfsd+0xc3/0x190 [nfsd] [ 1355.780050] kthread+0x106/0x130 [ 1355.783298] ? kthread_complete_and_exit+0x20/0x20 [ 1355.788118] ret_from_fork+0x22/0x30 [ 1355.791719] [ 1355.793919] INFO: task nfsd:42165 blocked for more than 123 seconds. [ 1355.800312] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1355.807579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.815462] task:nfsd state:D stack: 0 pid:42165 ppid: 2 flags:0x00004000 [ 1355.823867] Call Trace: [ 1355.826327] [ 1355.828439] __schedule+0x25b/0x660 [ 1355.831950] schedule+0x5d/0xd0 [ 1355.835113] schedule_timeout+0x151/0x190 [ 1355.839152] ? preempt_count_add+0x70/0xa0 [ 1355.843275] __wait_for_common+0x8e/0x1c0 [ 1355.847310] ? usleep_range_state+0x90/0x90 [ 1355.851521] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.857315] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.862327] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.867091] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.871156] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.876264] nfsd+0xc3/0x190 [nfsd] [ 1355.879803] kthread+0x106/0x130 [ 1355.883050] ? kthread_complete_and_exit+0x20/0x20 [ 1355.887870] ret_from_fork+0x22/0x30 [ 1355.891470] [ 1355.893672] INFO: task nfsd:42166 blocked for more than 123 seconds. [ 1355.900065] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1355.907331] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.915208] task:nfsd state:D stack: 0 pid:42166 ppid: 2 flags:0x00004000 [ 1355.923611] Call Trace: [ 1355.926071] [ 1355.928184] __schedule+0x25b/0x660 [ 1355.931698] schedule+0x5d/0xd0 [ 1355.934859] schedule_timeout+0x151/0x190 [ 1355.938894] ? preempt_count_add+0x70/0xa0 [ 1355.943024] __wait_for_common+0x8e/0x1c0 [ 1355.947057] ? usleep_range_state+0x90/0x90 [ 1355.951269] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.957066] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.962074] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.966837] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.970900] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.976006] nfsd+0xc3/0x190 [nfsd] [ 1355.979538] kthread+0x106/0x130 [ 1355.982786] ? kthread_complete_and_exit+0x20/0x20 [ 1355.987607] ret_from_fork+0x22/0x30 [ 1355.991208] [ 1355.993408] INFO: task nfsd:42167 blocked for more than 123 seconds. [ 1355.999799] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1356.007070] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.014948] task:nfsd state:D stack: 0 pid:42167 ppid: 2 flags:0x00004000 [ 1356.023352] Call Trace: [ 1356.025814] [ 1356.027928] __schedule+0x25b/0x660 [ 1356.031439] schedule+0x5d/0xd0 [ 1356.034599] schedule_timeout+0x151/0x190 [ 1356.038638] ? preempt_count_add+0x70/0xa0 [ 1356.042761] __wait_for_common+0x8e/0x1c0 [ 1356.046796] ? usleep_range_state+0x90/0x90 [ 1356.051008] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.056806] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.061815] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.066580] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.070650] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.075758] nfsd+0xc3/0x190 [nfsd] [ 1356.079292] kthread+0x106/0x130 [ 1356.082539] ? kthread_complete_and_exit+0x20/0x20 [ 1356.087362] ret_from_fork+0x22/0x30 [ 1356.090964] [ 1356.093165] INFO: task nfsd:42168 blocked for more than 123 seconds. [ 1356.099558] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1356.106822] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.114699] task:nfsd state:D stack: 0 pid:42168 ppid: 2 flags:0x00004000 [ 1356.123100] Call Trace: [ 1356.125560] [ 1356.127672] __schedule+0x25b/0x660 [ 1356.131186] schedule+0x5d/0xd0 [ 1356.134355] schedule_timeout+0x151/0x190 [ 1356.138391] ? preempt_count_add+0x70/0xa0 [ 1356.142527] __wait_for_common+0x8e/0x1c0 [ 1356.146563] ? usleep_range_state+0x90/0x90 [ 1356.150775] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.156570] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.161578] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.166343] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.170410] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.175516] nfsd+0xc3/0x190 [nfsd] [ 1356.179048] kthread+0x106/0x130 [ 1356.182296] ? kthread_complete_and_exit+0x20/0x20 [ 1356.187119] ret_from_fork+0x22/0x30 [ 1356.190722] [ 1356.192922] INFO: task nfsd:42169 blocked for more than 123 seconds. [ 1356.199321] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1356.206586] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.214465] task:nfsd state:D stack: 0 pid:42169 ppid: 2 flags:0x00004000 [ 1356.222866] Call Trace: [ 1356.225325] [ 1356.227437] __schedule+0x25b/0x660 [ 1356.230949] schedule+0x5d/0xd0 [ 1356.234110] schedule_timeout+0x151/0x190 [ 1356.238145] ? preempt_count_add+0x70/0xa0 [ 1356.242269] __wait_for_common+0x8e/0x1c0 [ 1356.246305] ? usleep_range_state+0x90/0x90 [ 1356.250516] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.256312] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.261320] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.266085] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.270149] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.275254] nfsd+0xc3/0x190 [nfsd] [ 1356.278790] kthread+0x106/0x130 [ 1356.282037] ? kthread_complete_and_exit+0x20/0x20 [ 1356.286858] ret_from_fork+0x22/0x30 [ 1356.290460] [-- MARK -- Thu Jan 19 18:00:00 2023] [ 1478.376034] INFO: task nfsd:42162 blocked for more than 245 seconds. [ 1478.382435] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1478.389703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1478.397581] task:nfsd state:D stack: 0 pid:42162 ppid: 2 flags:0x00004000 [ 1478.405988] Call Trace: [ 1478.408449] [ 1478.410567] __schedule+0x25b/0x660 [ 1478.414083] schedule+0x5d/0xd0 [ 1478.417244] schedule_timeout+0x151/0x190 [ 1478.421282] ? preempt_count_add+0x70/0xa0 [ 1478.425409] __wait_for_common+0x8e/0x1c0 [ 1478.429445] ? usleep_range_state+0x90/0x90 [ 1478.433660] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1478.439461] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1478.444471] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1478.449248] svc_recv+0xa1/0x1d0 [sunrpc] [ 1478.453315] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1478.458427] nfsd+0xc3/0x190 [nfsd] [ 1478.461963] kthread+0x106/0x130 [ 1478.465212] ? kthread_complete_and_exit+0x20/0x20 [ 1478.470033] ret_from_fork+0x22/0x30 [ 1478.473637] [ 1478.475834] INFO: task nfsd:42163 blocked for more than 245 seconds. [ 1478.482235] Not tainted 5.14.0-239.rt14.239.1773_752168396.el9.x86_64 #1 [ 1478.489500] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1478.497377] task:nfsd state:D stack: 0 pid:42163 ppid: 2 flags:0x00004000 [ 1478.505781] Call Trace: [ 1478.508242] [ 1478.510355] __schedule+0x25b/0x660 [ 1478.513869] schedule+0x5d/0xd0 [ 1478.517030] schedule_timeout+0x151/0x190 [ 1478.521068] ? preempt_count_add+0x70/0xa0 [ 1478.525191] __wait_for_common+0x8e/0x1c0 [ 1478.529227] ? usleep_range_state+0x90/0x90 [ 1478.533440] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1478.539236] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1478.544248] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1478.549017] svc_recv+0xa1/0x1d0 [sunrpc] [ 1478.553083] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1478.558188] nfsd+0xc3/0x190 [nfsd] [ 1478.561722] kthread+0x106/0x130 [ 1478.564970] ? kthread_complete_and_exit+0x20/0x20 [ 1478.569792] ret_from_fork+0x22/0x30 [ 1478.573391] [-- MARK -- Thu Jan 19 18:05:00 2023] [-- MARK -- Thu Jan 19 18:10:00 2023] [-- MARK -- Thu Jan 19 18:15:00 2023] [-- MARK -- Thu Jan 19 18:20:00 2023] [-- MARK -- Thu Jan 19 18:25:00 2023] [-- MARK -- Thu Jan 19 18:30:00 2023] [-- MARK -- Thu Jan 19 18:35:00 2023] [-- MARK -- Thu Jan 19 18:40:00 2023]