[ 228.031034] Running test [R:13122417 T:7 - RDMA Sanity Test - RoCE - Server - Kernel: 5.14.0-217.rt14.218.1639_725911687.el9.x86_64] [ 258.209901] systemd-rc-local-generator[15419]: /etc/rc.d/rc.local is not marked executable, skipping. [ 260.238684] systemd-rc-local-generator[15595]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Sun Dec 18 12:25:00 2022] [-- MARK -- Sun Dec 18 12:30:00 2022] [ 648.398058] systemd-rc-local-generator[31363]: /etc/rc.d/rc.local is not marked executable, skipping. [ 652.794102] systemd-rc-local-generator[31434]: /etc/rc.d/rc.local is not marked executable, skipping. [ 657.516955] systemd-rc-local-generator[31601]: /etc/rc.d/rc.local is not marked executable, skipping. [ 806.185696] systemd-rc-local-generator[40336]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Sun Dec 18 12:35:00 2022] [ 1053.718969] XFS (sda4): Unmounting Filesystem [ 1126.514450] XFS (sda4): Mounting V5 Filesystem [ 1126.650914] XFS (sda4): Ending clean mount [ 1127.185395] NFSD: Using nfsdcld client tracking operations. [ 1127.185399] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000) [-- MARK -- Sun Dec 18 12:40:00 2022] [ 1356.214550] INFO: task nfsd:42089 blocked for more than 122 seconds. [ 1356.220954] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1356.228222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.236102] task:nfsd state:D stack: 0 pid:42089 ppid: 2 flags:0x00004000 [ 1356.244508] Call Trace: [ 1356.246970] [ 1356.249088] __schedule+0x25b/0x660 [ 1356.252606] schedule+0x5d/0xd0 [ 1356.255771] schedule_timeout+0x151/0x190 [ 1356.259810] ? preempt_count_add+0x70/0xa0 [ 1356.263935] __wait_for_common+0x8e/0x1c0 [ 1356.267971] ? usleep_range_state+0x90/0x90 [ 1356.272187] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.277995] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.283007] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.287787] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.291854] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.296964] nfsd+0xc3/0x190 [nfsd] [ 1356.300502] kthread+0x106/0x130 [ 1356.303753] ? kthread_complete_and_exit+0x20/0x20 [ 1356.308576] ret_from_fork+0x22/0x30 [ 1356.312179] [ 1356.314378] INFO: task nfsd:42091 blocked for more than 122 seconds. [ 1356.320769] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1356.328036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.335913] task:nfsd state:D stack: 0 pid:42091 ppid: 2 flags:0x00004000 [ 1356.344323] Call Trace: [ 1356.346784] [ 1356.348898] __schedule+0x25b/0x660 [ 1356.352409] schedule+0x5d/0xd0 [ 1356.355571] schedule_timeout+0x151/0x190 [ 1356.359607] ? preempt_count_add+0x70/0xa0 [ 1356.363731] __wait_for_common+0x8e/0x1c0 [ 1356.367766] ? usleep_range_state+0x90/0x90 [ 1356.371976] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.377773] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.382782] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.387550] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.391614] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.396721] nfsd+0xc3/0x190 [nfsd] [ 1356.400254] kthread+0x106/0x130 [ 1356.403509] ? kthread_complete_and_exit+0x20/0x20 [ 1356.408342] ret_from_fork+0x22/0x30 [ 1356.411942] [ 1356.414146] INFO: task nfsd:42094 blocked for more than 123 seconds. [ 1356.420539] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1356.427807] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.435686] task:nfsd state:D stack: 0 pid:42094 ppid: 2 flags:0x00004000 [ 1356.444089] Call Trace: [ 1356.446553] [ 1356.448667] __schedule+0x25b/0x660 [ 1356.452180] schedule+0x5d/0xd0 [ 1356.455344] schedule_timeout+0x151/0x190 [ 1356.459381] ? preempt_count_add+0x70/0xa0 [ 1356.463505] __wait_for_common+0x8e/0x1c0 [ 1356.467538] ? usleep_range_state+0x90/0x90 [ 1356.471749] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1356.477545] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.482553] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1356.487321] svc_recv+0xa1/0x1d0 [sunrpc] [ 1356.491398] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.496506] nfsd+0xc3/0x190 [nfsd] [ 1356.500040] kthread+0x106/0x130 [ 1356.503288] ? kthread_complete_and_exit+0x20/0x20 [ 1356.508111] ret_from_fork+0x22/0x30 [ 1356.511713] [ 1356.513912] INFO: task nfsd:42095 blocked for more than 123 seconds. [ 1356.520304] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1356.527569] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.535452] task:nfsd state:D stack: 0 pid:42095 ppid: 2 flags:0x00004000 [ 1356.543855] Call Trace: [ 1356.546315] [ 1356.548429] __schedule+0x25b/0x660 [ 1356.551939] schedule+0x5d/0xd0 [ 1356.555102] schedule_timeout+0x151/0x190 [ 1356.559140] ? preempt_count_add+0x70/0xa0 [ 1356.563264] __wait_for_common+0x8e/0x1c0 [ 1356.567299] ? usleep_range_s[ 1479.096134] INFO: task nfsd:42089 blocked for more than 245 seconds. [ 1479.102536] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1479.109803] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.117681] task:nfsd state:D stack: 0 pid:42089 ppid: 2 flags:0x00004000 [ 1479.126095] Call Trace: [ 1479.128559] [ 1479.130675] __schedule+0x25b/0x660 [ 1479.134192] schedule+0x5d/0xd0 [ 1479.137356] schedule_timeout+0x151/0x190 [ 1479.141394] ? preempt_count_add+0x70/0xa0 [ 1479.145518] __wait_for_common+0x8e/0x1c0 [ 1479.149555] ? usleep_range_state+0x90/0x90 [ 1479.153769] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1479.159574] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.164586] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1479.169363] svc_recv+0xa1/0x1d0 [sunrpc] [ 1479.173432] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.178545] nfsd+0xc3/0x190 [nfsd] [ 1479.182082] kthread+0x106/0x130 [ 1479.185332] ? kthread_complete_and_exit+0x20/0x20 [ 1479.190160] ret_from_fork+0x22/0x30 [ 1479.193764] [ 1479.195964] INFO: task nfsd:42091 blocked for more than 245 seconds. [ 1479.202359] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1479.209624] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.217501] task:nfsd state:D stack: 0 pid:42091 ppid: 2 flags:0x00004000 [ 1479.225906] Call Trace: [ 1479.228381] [ 1479.230496] __schedule+0x25b/0x660 [ 1479.234009] schedule+0x5d/0xd0 [ 1479.237170] schedule_timeout+0x151/0x190 [ 1479.241206] ? preempt_count_add+0x70/0xa0 [ 1479.245329] __wait_for_common+0x8e/0x1c0 [ 1479.249370] ? usleep_range_state+0x90/0x90 [ 1479.253587] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1479.259384] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.264393] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1479.269160] svc_recv+0xa1/0x1d0 [sunrpc] [ 1479.273226] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.278332] nfsd+0xc3/0x190 [nfsd] [ 1479.281864] kthread+0x106/0x130 [ 1479.285115] ? kthread_complete_and_exit+0x20/0x20 [ 1479.289937] ret_from_fork+0x22/0x30 [ 1479.293539] [ 1479.295742] INFO: task nfsd:42094 blocked for more than 245 seconds. [ 1479.302134] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1479.309399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.317280] task:nfsd state:D stack: 0 pid:42094 ppid: 2 flags:0x00004000 [ 1479.325683] Call Trace: [ 1479.328146] [ 1479.330257] __schedule+0x25b/0x660 [ 1479.333770] schedule+0x5d/0xd0 [ 1479.336933] schedule_timeout+0x151/0x190 [ 1479.340972] ? preempt_count_add+0x70/0xa0 [ 1479.345098] __wait_for_common+0x8e/0x1c0 [ 1479.349135] ? usleep_range_state+0x90/0x90 [ 1479.353345] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1479.359143] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.364153] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1479.368927] svc_recv+0xa1/0x1d0 [sunrpc] [ 1479.372990] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.378095] nfsd+0xc3/0x190 [nfsd] [ 1479.381634] kthread+0x106/0x130 [ 1479.384883] ? kthread_complete_and_exit+0x20/0x20 [ 1479.389704] ret_from_fork+0x22/0x30 [ 1479.393306] [ 1479.395503] INFO: task nfsd:42095 blocked for more than 246 seconds. [ 1479.401896] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1479.409163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.417040] task:nfsd state:D stack: 0 pid:42095 ppid: 2 flags:0x00004000 [ 1479.425445] Call Trace: [ 1479.427907] [ 1479.430033] __schedule+0x25b/0x660 [ 1479.433545] schedule+0x5d/0xd0 [ 1479.436707] schedule_timeout+0x151/0x190 [ 1479.440750] ? preempt_count_add+0x70/0xa0 [ 1479.444878] __wait_for_common+0x8e/0x1c0 [ 1479.448928] ? usleep_range_state+0x90/0x90 [ 1479.453138] svc_rdma_process_read_list+0x103/0x300 [rpcrdma] [ 1479.458935] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.463943] svc_handle_xprt+0x350/0x400 [sunrpc] [ 1479.468707] svc_recv+0xa1/0x1d0 [sunrpc] [ 1479.472769] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.477877] nfsd+0xc3/0x190 [nfsd] [ 1479.481411] kthread+0x106/0x130 [ 1479.484658] ? kthread_complete_and_exit+0x20/0x20 [ 1479.489481] ret_from_fork+0x22/0x30 [ 1479.493082] [ 1479.495286] INFO: task rpc.nfsd:42347 blocked for more than 246 seconds. [ 1479.502026] Not tainted 5.14.0-217.rt14.218.1639_725911687.el9.x86_64 #1 [ 1479.509298] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.517176] task:rpc.nfsd state:D stack: 0 pid:42347 ppid: 1 flags:0x00000006 [ 1479.525579] Call Trace: [ 1479.528039] [ 1479.530152] __schedule+0x25b/0x660 [ 1479.533665] schedule+0x5d/0xd0 [ 1479.536826] schedule_timeout+0x151/0x190 [ 1479.540862] ? default_send_IPI_single_phys+0x26/0x40 [ 1479.545946] ? preempt_count_add+0x70/0xa0 [ 1479.550070] __wait_for_common+0x8e/0x1c0 [ 1479.554106] ? usleep_range_state+0x90/0x90 [ 1479.558321] kthread_stop+0x62/0x1d0 [ 1479.561925] svc_set_num_threads+0x10d/0x200 [sunrpc] [ 1479.567044] nfsd_svc+0x123/0x190 [nfsd] [ 1479.571016] write_threads+0x95/0x100 [nfsd] [ 1479.575336] ? _copy_from_user+0x3a/0x60 [ 1479.579288] ? simple_transaction_get+0xc4/0xf0 [ 1479.583850] ? write_pool_threads+0x230/0x230 [nfsd] [ 1479.588867] nfsctl_transaction_write+0x46/0x80 [nfsd] [ 1479.594059] vfs_write+0xb5/0x2e0 [ 1479.597398] ksys_write+0x62/0xe0 [ 1479.600736] do_syscall_64+0x5c/0x90 [ 1479.604337] ? exc_page_fault+0x65/0x180 [ 1479.608287] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1479.613373] RIP: 0033:0x7fa6a3d73be7 [ 1479.616972] RSP: 002b:00007ffd8337e928 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1479.624590] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fa6a3d73be7 [ 1479.631774] RDX: 0000000000000002 RSI: 0000563961759c20 RDI: 0000000000000003 [ 1479.638957] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffd8337e7c0 [ 1479.646136] R10: 0000000000000000 R11: 0000000000000246 R12: 0000563961759c20 [ 1479.653316] R13: 00007fa6a3c326c0 R14: 00007ffd8337e9f0 R15: 00000000ffffffff [ 1479.660498] [-- MARK -- Sun Dec 18 12:45:00 2022] [-- MARK -- Sun Dec 18 12:50:00 2022] [-- MARK -- Sun Dec 18 12:55:00 2022] [-- MARK -- Sun Dec 18 13:00:00 2022] [-- MARK -- Sun Dec 18 13:05:00 2022] [-- MARK -- Sun Dec 18 13:10:00 2022] [-- MARK -- Sun Dec 18 13:15:00 2022] [-- MARK -- Sun Dec 18 13:20:00 2022]