[ 225.373494] Running test [R:13084803 T:7 - RDMA Sanity Test - RoCE - Server - Kernel: 5.14.0-210.mr1773_719190109.el9.x86_64] [ 251.010025] systemd-rc-local-generator[14912]: /etc/rc.d/rc.local is not marked executable, skipping. [ 252.579628] systemd-rc-local-generator[15084]: /etc/rc.d/rc.local is not marked executable, skipping. [-- MARK -- Sun Dec 11 20:20:00 2022] [ 620.555420] systemd-rc-local-generator[30372]: /etc/rc.d/rc.local is not marked executable, skipping. [ 625.372740] systemd-rc-local-generator[30494]: /etc/rc.d/rc.local is not mark[-- MARK -- Sun Dec 11 20:25:00 2022] ed executable, skipping. [ 630.754375] systemd-rc-local-generator[30671]: /etc/rc.d/rc.local is not marked executable, skipping. [ 770.929274] systemd-rc-local-generator[39362]: /etc/rc.d/rc.local is not marked executable, skipping. [ 919.156877] XFS (sda4): Unmounting Filesystem [-- MARK -- Sun Dec 11 20:30:00 2022] [ 991.929436] XFS (sda4): Mounting V5 Filesystem [ 992.064682] XFS (sda4): Ending clean mount [ 992.541116] NFSD: Using nfsdcld client tracking operations. [ 992.546741] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000) [-- MARK -- Sun Dec 11 20:35:00 2022] [ 1233.588074] INFO: task nfsd:40913 blocked for more than 122 seconds. [ 1233.594477] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1233.601139] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1233.609045] task:nfsd state:D stack: 0 pid:40913 ppid: 2 flags:0x00004000 [ 1233.617455] Call Trace: [ 1233.619919] [ 1233.622039] __schedule+0x248/0x620 [ 1233.625562] schedule+0x5a/0xc0 [ 1233.628730] schedule_timeout+0x11d/0x160 [ 1233.632774] __wait_for_common+0x93/0x1d0 [ 1233.636815] ? usleep_range_state+0x90/0x90 [ 1233.641041] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1233.646851] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1233.651871] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1233.656677] svc_recv+0xab/0x1b0 [sunrpc] [ 1233.660756] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1233.665878] nfsd+0xc3/0x190 [nfsd] [ 1233.669436] kthread+0xd9/0x100 [ 1233.672604] ? kthread_complete_and_exit+0x20/0x20 [ 1233.677429] ret_from_fork+0x22/0x30 [ 1233.681044] [ 1233.683247] INFO: task nfsd:40914 blocked for more than 122 seconds. [ 1233.689645] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1233.696303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1233.704194] task:nfsd state:D stack: 0 pid:40914 ppid: 2 flags:0x00004000 [ 1233.712604] Call Trace: [ 1233.715072] [ 1233.717191] __schedule+0x248/0x620 [ 1233.720717] schedule+0x5a/0xc0 [ 1233.723884] schedule_timeout+0x11d/0x160 [ 1233.727926] __wait_for_common+0x93/0x1d0 [ 1233.731967] ? usleep_range_state+0x90/0x90 [ 1233.736194] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1233.741999] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1233.747020] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1233.751810] svc_recv+0xab/0x1b0 [sunrpc] [ 1233.755888] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1233.761009] nfsd+0xc3/0x190 [nfsd] [ 1233.764557] kthread+0xd9/0x100 [ 1233.767735] ? kthread_complete_and_exit+0x20/0x20 [ 1233.772561] ret_from_fork+0x22/0x30 [ 1233.776165] [ 1233.778369] INFO: task nfsd:40915 blocked for more than 123 seconds. [ 1233.784777] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1233.791435] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1233.799318] task:nfsd state:D stack: 0 pid:40915 ppid: 2 flags:0x00004000 [ 1233.807735] Call Trace: [ 1233.810203] [ 1233.812320] __schedule+0x248/0x620 [ 1233.815836] schedule+0x5a/0xc0 [ 1233.819003] schedule_timeout+0x11d/0x160 [ 1233.823055] __wait_for_common+0x93/0x1d0 [ 1233.827095] ? usleep_range_state+0x90/0x90 [ 1233.831311] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1233.837129] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1233.842148] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1233.846928] svc_recv+0xab/0x1b0 [sunrpc] [ 1233.851009] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1233.856130] nfsd+0xc3/0x190 [nfsd] [ 1233.859676] kthread+0xd9/0x100 [ 1233.862844] ? kthread_complete_and_exit+0x20/0x20 [ 1233.867688] ret_from_fork+0x22/0x30 [ 1233.871294] [ 1233.873501] INFO: task rpc.nfsd:41220 blocked for more than 123 seconds. [ 1233.880255] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1233.886915] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1233.894797] task:rpc.nfsd state:D stack: 0 pid:41220 ppid: 1 flags:0x00004006 [ 1233.903215] Call Trace: [ 1233.905682] [ 1233.907800] __schedule+0x248/0x620 [ 1233.911319] schedule+0x5a/0xc0 [ 1233.914488] schedule_timeout+0x11d/0x160 [ 1233.918541] ? default_send_IPI_single_phys+0x26/0x40 [ 1233.923631] ? ttwu_queue_wakelist+0xf4/0x110 [ 1233.928022] __wait_for_common+0x93/0x1d0 [ 1233.932068] ? usleep_range_state+0x90/0x90 [ 1233.936285] kthread_stop+0x62/0x170 [ 1233.939888] svc_set_num_threads+0x10d/0x1a0 [sunrpc] [ 1233.945032] nfsd_svc+0x123/0x190 [nfsd] [ 1233.949017] write_threads+0x95/0x100 [nfsd] [ 1233.953348] ? _copy_from_user+0x3a/0x60 [ 1233.957310] ? simple_transaction_get+0xc4/0xf0 [ 1233.961875] ? write_pool_threads+0x230/0x230 [nfsd] [ 1233.966903] nfsctl_transaction_write+0x46/0x80 [nfsd] [ 1233.972121] vfs_write+0xb5/0x280 [ 1233.975466] ksys_write+0x5f/0xe0 [ 1233.978808] do_syscall_64+0x5c/0x90 [ 1233.982415] ? exc_page_fault+0x62/0x150 [ 1233.986369] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1233.991470] RIP: 0033:0x7fe81473ebe7 [ 1233.995073] RSP: 002b:00007ffdeae81c88 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 1234.002703] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fe81473ebe7 [ 1234.009889] RDX: 0000000000000002 RSI: 0000563abbbc1c20 RDI: 0000000000000003 [ 1234.017083] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffdeae81b20 [ 1234.024265] R10: 0000000000000000 R11: 0000000000000246 R12: 0000563abbbc1c20 [ 1234.031450] R13: 00007fe81480f6c0 R14: 00007ffdeae81d50 R15: 00000000ffffffff [ 1234.038647] [ 1356.469811] INFO: task nfsd:40913 blocked for more than 245 seconds. [ 1356.476218] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1356.482879] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.490762] task:nfsd state:D stack: 0 pid:40913 ppid: 2 flags:0x00004000 [ 1356.499173] Call Trace: [ 1356.501638] [ 1356.503753] __schedule+0x248/0x620 [ 1356.507276] schedule+0x5a/0xc0 [ 1356.510444] schedule_timeout+0x11d/0x160 [ 1356.514486] __wait_for_common+0x93/0x1d0 [ 1356.518528] ? usleep_range_state+0x90/0x90 [ 1356.522748] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1356.528559] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.533579] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1356.538374] svc_recv+0xab/0x1b0 [sunrpc] [ 1356.542455] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.547578] nfsd+0xc3/0x190 [nfsd] [ 1356.551128] kthread+0xd9/0x100 [ 1356.554294] ? kthread_complete_and_exit+0x20/0x20 [ 1356.559121] ret_from_fork+0x22/0x30 [ 1356.562729] [ 1356.564931] INFO: task nfsd:40914 blocked for more than 245 seconds. [ 1356.571330] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1356.577989] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.585871] task:nfsd state:D stack: 0 pid:40914 ppid: 2 flags:0x00004000 [ 1356.594278] Call Trace: [ 1356.596747] [ 1356.598863] __schedule+0x248/0x620 [ 1356.602380] schedule+0x5a/0xc0 [ 1356.605547] schedule_timeout+0x11d/0x160 [ 1356.609589] __wait_for_common+0x93/0x1d0 [ 1356.613629] ? usleep_range_state+0x90/0x90 [ 1356.617845] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1356.623650] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.628667] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1356.633452] svc_recv+0xab/0x1b0 [sunrpc] [ 1356.637531] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.642652] nfsd+0xc3/0x190 [nfsd] [ 1356.646200] kthread+0xd9/0x100 [ 1356.649365] ? kthread_complete_and_exit+0x20/0x20 [ 1356.654192] ret_from_fork+0x22/0x30 [ 1356.657796] [ 1356.660000] INFO: task nfsd:40915 blocked for more than 245 seconds. [ 1356.666402] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1356.673062] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.680945] task:nfsd state:D stack: 0 pid:40915 ppid: 2 flags:0x00004000 [ 1356.689355] Call Trace: [ 1356.691822] [ 1356.693938] __schedule+0x248/0x620 [ 1356.697455] schedule+0x5a/0xc0 [ 1356.700621] schedule_timeout+0x11d/0x160 [ 1356.704664] __wait_for_common+0x93/0x1d0 [ 1356.708704] ? usleep_range_state+0x90/0x90 [ 1356.712920] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1356.718726] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1356.723747] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1356.728528] svc_recv+0xab/0x1b0 [sunrpc] [ 1356.732607] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1356.737726] nfsd+0xc3/0x190 [nfsd] [ 1356.741272] kthread+0xd9/0x100 [ 1356.744438] ? kthread_complete_and_exit+0x20/0x20 [ 1356.749265] ret_from_fork+0x22/0x30 [ 1356.752872] [ 1356.755076] INFO: task rpc.nfsd:41220 blocked for more than 246 seconds. [ 1356.761821] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1356.768480] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1356.776361] task:rpc.nfsd state:D stack[ 1479.351537] INFO: task nfsd:40913 blocked for more than 368 seconds. [ 1479.357937] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1479.364598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.372480] task:nfsd state:D stack: 0 pid:40913 ppid: 2 flags:0x00004000 [ 1479.380890] Call Trace: [ 1479.383357] [ 1479.385472] __schedule+0x248/0x620 [ 1479.388991] schedule+0x5a/0xc0 [ 1479.392156] schedule_timeout+0x11d/0x160 [ 1479.396199] __wait_for_common+0x93/0x1d0 [ 1479.400238] ? usleep_range_state+0x90/0x90 [ 1479.404458] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1479.410266] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.415286] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1479.420079] svc_recv+0xab/0x1b0 [sunrpc] [ 1479.424160] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.429284] nfsd+0xc3/0x190 [nfsd] [ 1479.432832] kthread+0xd9/0x100 [ 1479.435998] ? kthread_complete_and_exit+0x20/0x20 [ 1479.440824] ret_from_fork+0x22/0x30 [ 1479.444430] [ 1479.446635] INFO: task nfsd:40914 blocked for more than 368 seconds. [ 1479.453034] Not tainted 5.14.0-210.mr1773_719190109.el9.x86_64 #1 [ 1479.459691] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1479.467572] task:nfsd state:D stack: 0 pid:40914 ppid: 2 flags:0x00004000 [ 1479.475980] Call Trace: [ 1479.478447] [ 1479.480571] __schedule+0x248/0x620 [ 1479.484093] schedule+0x5a/0xc0 [ 1479.487264] schedule_timeout+0x11d/0x160 [ 1479.491306] __wait_for_common+0x93/0x1d0 [ 1479.495348] ? usleep_range_state+0x90/0x90 [ 1479.499566] svc_rdma_process_read_list+0x103/0x2d0 [rpcrdma] [ 1479.505374] svc_rdma_recvfrom+0x217/0x470 [rpcrdma] [ 1479.510391] svc_handle_xprt+0x2ea/0x370 [sunrpc] [ 1479.515178] svc_recv+0xab/0x1b0 [sunrpc] [ 1479.519259] ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd] [ 1479.524382] nfsd+0xc3/0x190 [nfsd] [ 1479.527936] kthread+0xd9/0x100 [ 1479.531102] ? kthread_complete_and_exit+0x20/0x20 [ 1479.535931] ret_from_fork+0x22/0x30 [ 1479.539540] [-- MARK -- Sun Dec 11 20:40:00 2022] [-- MARK -- Sun Dec 11 20:45:00 2022] [-- MARK -- Sun Dec 11 20:50:00 2022] [-- MARK -- Sun Dec 11 20:55:00 2022] [-- MARK -- Sun Dec 11 21:00:00 2022] [-- MARK -- Sun Dec 11 21:05:00 2022] [-- MARK -- Sun Dec 11 21:10:00 2022] [-- MARK -- Sun Dec 11 21:15:00 2022]