[ 229.092874] Running test [R:13230884 T:7 - RDMA Sanity Test - RoCE - Server - Kernel: 5.14.0-235.rt14.236.1639_747671795.el9.x86_64] [-- MARK -- Sun Jan 15 14:20:00 2023] [ 259.063490] systemd-rc-local-generator[15464]: /etc/rc.d/rc.local is not marked executable, skipping. [ 261.070232] systemd-rc-local-generator[15638]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Sun Jan 15 14:25:00 2023] [ 713.805619] systemd-rc-local-generator[31559]: /etc/rc.d/rc.local is not marked executable, skipping. [ 718.215344] systemd-rc-local-generator[31626]: /etc/rc.d/rc.local is not marked executable, skipping. [ 724.541250] systemd-rc-local-generator[31801]: /etc/rc.d/rc.local is not marked executable, skipping. [ 802.899080] systemd-rc-local-generator[40362]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Sun Jan 15 14:30:00 2023] [ 944.452128] XFS (sda4): Unmounting Filesystem [ 1017.309258] XFS (sda4): Mounting V5 Filesystem [ 1017.447480] XFS (sda4): Ending clean mount [ 1017.995776] NFSD: Using nfsdcld client tracking operations. [ 1017.995780] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000) [-- MARK -- Sun Jan 15 14:35:00 2023] [ 1232.304926] INFO: task nfsd:41911 blocked for more than 122 seconds. [ 1232.311331] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1232.318600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1232.326477] task:nfsd state:D stack: 0 pid:41911 ppid: 2 flags:0x00004000 [ 1232.334884] Call Trace: [ 1232.337348] [ 1232.339461] __schedule+0x25b/0x660 [ 1232.342980] schedule+0x5d/0xd0 [ 1232.346142] schedule_timeout+0x151/0x190 [ 1232.350178] ? preempt_count_add+0x70/0xa0 [ 1232.354302] __wait_for_common+0x8e/0x1c0 [ 1232.358337] ? usleep_range_state+0x90/0x90 [ 1232.362552] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1232.368354] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1232.373368] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1232.378145] svc_recv+0xa1/0x1d0 [sunrpc] [ 1232.382210] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1232.387320] nfsd+0xc3/0x190 [nfsd] [ 1232.390858] kthread+0x106/0x130 [ 1232.394110] ? kthread_complete_and_exit+0x20/0x20 [ 1232.398931] ret_from_fork+0x22/0x30 [ 1232.402533] [ 1232.404733] INFO: task nfsd:41913 blocked for more than 122 seconds. [ 1232.411132] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1232.418398] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1232.426274] task:nfsd state:D stack: 0 pid:41913 ppid: 2 flags:0x00004000 [ 1232.434680] Call Trace: [ 1232.437143] [ 1232.439257] __schedule+0x25b/0x660 [ 1232.442770] schedule+0x5d/0xd0 [ 1232.445930] schedule_timeout+0x151/0x190 [ 1232.449967] ? preempt_count_add+0x70/0xa0 [ 1232.454091] __wait_for_common+0x8e/0x1c0 [ 1232.458127] ? usleep_range_state+0x90/0x90 [ 1232.462337] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1232.468135] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1232.473151] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1232.477919] svc_recv+0xa1/0x1d0 [sunrpc] [ 1232.481982] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1232.487088] nfsd+0xc3/0x190 [nfsd] [ 1232.490620] kthread+0x106/0x130 [ 1232.493870] ? kthread_complete_and_exit+0x20/0x20 [ 1232.498693] ret_from_fork+0x22/0x30 [ 1232.502291] [ 1232.504494] INFO: task nfsd:41915 blocked for more than 123 seconds. [ 1232.510886] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1232.518153] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1232.526030] task:nfsd state:D stack: 0 pid:41915 ppid: 2 flags:0x00004000 [ 1232.534432] Call Trace: [ 1232.536901] [ 1232.539015] __schedule+0x25b/0x660 [ 1232.542527] schedule+0x5d/0xd0 [ 1232.545691] schedule_timeout+0x151/0x190 [ 1232.549730] ? preempt_count_add+0x70/0xa0 [ 1232.553856] __wait_for_common+0x8e/0x1c0 [ 1232.557890] ? usleep_range_state+0x90/0x90 [ 1232.562103] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1232.567900] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1232.572910] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1232.577675] svc_recv+0xa1/0x1d0 [sunrpc] [ 1232.581739] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1232.586846] nfsd+0xc3/0x190 [nfsd] [ 1232.590378] kthread+0x106/0x130 [ 1232.593629] ? kthread_complete_and_exit+0x20/0x20 [ 1232.598451] ret_from_fork+0x22/0x30 [ 1232.602056] [ 1232.604256] INFO: task nfsd:41916 blocked for more than 123 seconds. [ 1232.610648] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1232.617915] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1232.625791] task:nfsd state:D stack: 0 pid:41916 ppid: 2 flags:0x00004000 [ 1232.634195] Call Trace: [ 1232.636658] [ 1232.638772] __schedule+0x25b/0x660 [ 1232.642285] schedule+0x5d/0xd0 [ 1232.645448] schedule_timeout+0x151/0x190 [ 1232.649484] ? preempt_count_add+0x70/0xa0 [ 1232.653606] __wait_for_common+0x8e/0x1c0 [ 1232.657640] ? usleep_range_state+0x90/0x90 [ 1232.661850] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1232.667652] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1232.672662] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1232.677426] svc_recv+0xa1/0x1d0 [sunrpc] [ 1232.681489] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1232.686597] nfsd+0xc3/0x190 [nfsd] [ 1232.690132] kthread+0x106/0x130 [ 1232.693382] ? kthread_complete_and_exit+0x20/0x20 [ 1232.698203] ret_from_fork+0x22/0x30 [ 1232.701803] [ 1355.186606] INFO: task nfsd:41911 blocked for more than 245 seconds. [ 1355.193009] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1355.200277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.208154] task:nfsd state:D stack: 0 pid:41911 ppid: 2 flags:0x00004000 [ 1355.216560] Call Trace: [ 1355.219021] [ 1355.221135] __schedule+0x25b/0x660 [ 1355.224651] schedule+0x5d/0xd0 [ 1355.227815] schedule_timeout+0x151/0x190 [ 1355.231851] ? preempt_count_add+0x70/0xa0 [ 1355.235978] __wait_for_common+0x8e/0x1c0 [ 1355.240015] ? usleep_range_state+0x90/0x90 [ 1355.244230] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.250033] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.255047] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.259825] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.263892] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.269001] nfsd+0xc3/0x190 [nfsd] [ 1355.272537] kthread+0x106/0x130 [ 1355.275788] ? kthread_complete_and_exit+0x20/0x20 [ 1355.280608] ret_from_fork+0x22/0x30 [ 1355.284213] [ 1355.286414] INFO: task nfsd:41913 blocked for more than 245 seconds. [ 1355.292815] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1355.300081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.307956] task:nfsd state:D stack: 0 pid:41913 ppid: 2 flags:0x00004000 [ 1355.316361] Call Trace: [ 1355.318822] [ 1355.320935] __schedule+0x25b/0x660 [ 1355.324447] schedule+0x5d/0xd0 [ 1355.327607] schedule_timeout+0x151/0x190 [ 1355.331644] ? preempt_count_add+0x70/0xa0 [ 1355.335768] __wait_for_common+0x8e/0x1c0 [ 1355.339802] ? usleep_range_state+0x90/0x90 [ 1355.344013] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.349815] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.354826] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.359590] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.363655] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.368761] nfsd+0xc3/0x190 [nfsd] [ 1355.372297] kthread+0x106/0x130 [ 1355.375547] ? kthread_complete_and_exit+0x20/0x20 [ 1355.380369] ret_from_fork+0x22/0x30 [ 1355.383971] [ 1355.386176] INFO: task nfsd:41915 blocked for more than 245 seconds. [ 1355.392569] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1355.399835] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.407710] task:nfsd state:D stack: 0 pid:41915 ppid: 2 flags:0x00004000 [ 1355.416112] Call Trace: [ 1355.418581] [ 1355.420694] __schedule+0x25b/0x660 [ 1355.424207] schedule+0x5d/0xd0 [ 1355.427370] schedule_timeout+0x151/0x190 [ 1355.431405] ? preempt_count_add+0x70/0xa0 [ 1355.435528] __wait_for_common+0x8e/0x1c0 [ 1355.439563] ? usleep_range_state+0x90/0x90 [ 1355.443775] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.449572] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.454581] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.459344] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.463409] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.468516] nfsd+0xc3/0x190 [nfsd] [ 1355.472051] kthread+0x106/0x130 [ 1355.475302] ? kthread_complete_and_exit+0x20/0x20 [ 1355.480123] ret_from_fork+0x22/0x30 [ 1355.483728] [ 1355.485927] INFO: task nfsd:41916 blocked for more than 246 seconds. [ 1355.492318] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1355.499585] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1355.507463] task:nfsd state:D stack: 0 pid:41916 ppid: 2 flags:0x00004000 [ 1355.515868] Call Trace: [ 1355.518329] [ 1355.520439] __schedule+0x25b/0x660 [ 1355.523950] schedule+0x5d/0xd0 [ 1355.527112] schedule_timeout+0x151/0x190 [ 1355.531149] ? preempt_count_add+0x70/0xa0 [ 1355.535271] __wait_for_common+0x8e/0x1c0 [ 1355.539305] ? usleep_range_state+0x90/0x90 [ 1355.543522] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1355.549322] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1355.554331] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1355.559095] svc_recv+0xa1/0x1d0 [sunrpc] [ 1355.563158] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1355.568265] nfsd+0xc3/0x190 [nfsd] [ 1355.571800] kthread+0x106/0x130 [ 1355.575048] ? kthread_complete_and_exit+0x20/0x20 [ 1355.579869] ret_from_fork+0x22/0x30 [ 1355.583470] [-- MARK -- Sun Jan 15 14:40:00 2023] [ 1478.068267] INFO: task nfsd:41911 blocked for more than 368 seconds. [ 1478.074671] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1478.081936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1478.089813] task:nfsd state:D stack: 0 pid:41911 ppid: 2 flags:0x00004000 [ 1478.098222] Call Trace: [ 1478.100683] [ 1478.102795] __schedule+0x25b/0x660 [ 1478.106313] schedule+0x5d/0xd0 [ 1478.109473] schedule_timeout+0x151/0x190 [ 1478.113509] ? preempt_count_add+0x70/0xa0 [ 1478.117634] __wait_for_common+0x8e/0x1c0 [ 1478.121669] ? usleep_range_state+0x90/0x90 [ 1478.125885] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1478.131686] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1478.136698] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1478.141476] svc_recv+0xa1/0x1d0 [sunrpc] [ 1478.145541] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1478.150650] nfsd+0xc3/0x190 [nfsd] [ 1478.154187] kthread+0x106/0x130 [ 1478.157437] ? kthread_complete_and_exit+0x20/0x20 [ 1478.162265] ret_from_fork+0x22/0x30 [ 1478.165867] [ 1478.168070] INFO: task nfsd:41913 blocked for more than 368 seconds. [ 1478.174462] Not tainted 5.14.0-235.rt14.236.1639_747671795.el9.x86_64 #1 [ 1478.181729] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1478.189605] task:nfsd state:D stack: 0 pid:41913 ppid: 2 flags:0x00004000 [ 1478.198012] Call Trace: [ 1478.200473] [ 1478.202586] __schedule+0x25b/0x660 [ 1478.206097] schedule+0x5d/0xd0 [ 1478.209257] schedule_timeout+0x151/0x190 [ 1478.213296] ? preempt_count_add+0x70/0xa0 [ 1478.217421] __wait_for_common+0x8e/0x1c0 [ 1478.221455] ? usleep_range_state+0x90/0x90 [ 1478.225672] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1478.231469] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1478.236478] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1478.241242] svc_recv+0xa1/0x1d0 [sunrpc] [ 1478.245305] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1478.250411] nfsd+0xc3/0x190 [nfsd] [ 1478.253946] kthread+0x106/0x130 [ 1478.257195] ? kthread_complete_and_exit+0x20/0x20 [ 1478.262017] ret_from_fork+0x22/0x30 [ 1478.265618] [-- MARK -- Sun Jan 15 14:45:00 2023] [-- MARK -- Sun Jan 15 14:50:00 2023] [-- MARK -- Sun Jan 15 14:55:00 2023] [-- MARK -- Sun Jan 15 15:00:00 2023] [-- MARK -- Sun Jan 15 15:05:00 2023] [-- MARK -- Sun Jan 15 15:10:00 2023] [-- MARK -- Sun Jan 15 15:15:00 2023]