Linux version 6.0.0-rc7 [ 0.000000] efi: EFI v2.70 by American Megatrends [ 0.000000] efi: ACPI 2.0=0xec1d0000 SMBIOS 3.0=0xf0a1ff98 ESRT=0xe585df98 MOKvar=0xee650000 RNG=0xf765e398 MEMRESERVE=0xe5873018 [ 0.000000] efi: seeding entropy pool [ 0.000000] random: crng init done [ 0.000000] esrt: Reserving ESRT space from 0x00000000e585df98 to 0x00000000e585dfd0. [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000EC1D0000 000024 (v02 Ampere) [ 0.000000] ACPI: XSDT 0x00000000EC1C0000 0000A4 (v01 Ampere Altra 00000000 AMI 01000013) [ 0.000000] ACPI: FACP 0x00000000EC1A0000 000114 (v06 Ampere Altra 00000000 INTL 20190509) [ 0.000000] ACPI: DSDT 0x00000000EC140000 0174C2 (v02 Ampere Jade 00000001 INTL 20200717) [ 0.000000] ACPI: DBG2 0x00000000EC1B0000 00005C (v00 Ampere Altra 00000000 INTL 20190509) [ 0.000000] ACPI: GTDT 0x00000000EC190000 000110 (v03 Ampere Altra 00000000 INTL 20190509) [ 0.000000] ACPI: SSDT 0x00000000EC180000 00002D (v02 Ampere Altra 00000001 INTL 20190509) [ 0.000000] ACPI: FIDT 0x00000000EC130000 00009C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: SPCR 0x00000000EC120000 000050 (v02 ALASKA A M I 01072009 AMI 0005000F) [ 0.000000] ACPI: BGRT 0x00000000EC110000 000038 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: MCFG 0x00000000EC100000 0000AC (v01 Ampere Altra 00000001 AMP. 01000013) [ 0.000000] ACPI: IORT 0x00000000EC0F0000 000610 (v00 Ampere Altra 00000000 AMP. 01000013) [ 0.000000] ACPI: PPTT 0x00000000EC0D0000 006E60 (v02 Ampere Altra 00000000 AMP. 01000013) [ 0.000000] ACPI: SLIT 0x00000000EC0C0000 00002D (v01 Ampere Altra 00000000 AMP. 01000013) [ 0.000000] ACPI: SRAT 0x00000000EC0B0000 000A30 (v03 Ampere Altra 00000000 AMP. 01000013) [ 0.000000] ACPI: APIC 0x00000000EC0E0000 0028F4 (v05 Ampere Altra 00000003 AMI 01000013) [ 0.000000] ACPI: PCCT 0x00000000EC090000 000576 (v02 Ampere Altra 00000003 AMP. 01000013) [ 0.000000] ACPI: WSMT 0x00000000EC080000 000028 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 0x00000000EC070000 000044 (v01 ALASKA A M I 01072009 AMI 01000013) [ 0.000000] ACPI: SPCR: console: pl011,mmio32,0x100002600000,115200 [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x88300000-0x883fffff] [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x90000000-0xffffffff] [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000000-0x8007fffffff] [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80100000000-0x80fffffffff] [ 0.000000] NUMA: NODE_DATA [mem 0x80ff8042a80-0x80ff8058fff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000088300000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000100000000-0x0000080fffffffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000088300000-0x00000000883fffff] [ 0.000000] node 0: [mem 0x0000000090000000-0x0000000091ffffff] [ 0.000000] node 0: [mem 0x0000000092000000-0x0000000093ffffff] [ 0.000000] node 0: [mem 0x0000000094000000-0x00000000ebc90fff] [ 0.000000] node 0: [mem 0x00000000ebc91000-0x00000000ec014fff] [ 0.000000] node 0: [mem 0x00000000ec015000-0x00000000ec017fff] [ 0.000000] node 0: [mem 0x00000000ec018000-0x00000000ec01cfff] [ 0.000000] node 0: [mem 0x00000000ec01d000-0x00000000ec01dfff] [ 0.000000] node 0: [mem 0x00000000ec01e000-0x00000000ec01ffff] [ 0.000000] node 0: [mem 0x00000000ec020000-0x00000000ec23ffff] [ 0.000000] node 0: [mem 0x00000000ec240000-0x00000000ec24ffff] [ 0.000000] node 0: [mem 0x00000000ec250000-0x00000000ee64ffff] [ 0.000000] node 0: [mem 0x00000000ee650000-0x00000000f765ffff] [ 0.000000] node 0: [mem 0x00000000f7660000-0x00000000f784ffff] [ 0.000000] node 0: [mem 0x00000000f7850000-0x00000000f7fdffff] [ 0.000000] node 0: [mem 0x00000000f7fe0000-0x00000000ffc8efff] [ 0.000000] node 0: [mem 0x00000000ffc8f000-0x00000000ffc8ffff] [ 0.000000] node 0: [mem 0x00000000ffc90000-0x00000000ffffffff] [ 0.000000] node 0: [mem 0x0000080000000000-0x000008007fffffff] [ 0.000000] node 0: [mem 0x0000080100000000-0x0000080fffffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000088300000-0x0000080fffffffff] [ 0.000000] On node 0, zone DMA: 768 pages in unavailable ranges [ 0.000000] On node 0, zone DMA: 31744 pages in unavailable ranges [ 0.000000] cma: Reserved 64 MiB at 0x00000000fbc00000 [ 0.000000] psci: probing for conduit method from ACPI. [ 0.000000] psci: PSCIv1.1 detected in firmware. [ 0.000000] psci: Using standard PSCI v0.2 function IDs [ 0.000000] psci: MIGRATE_INFO_TYPE not supported. [ 0.000000] psci: SMC Calling Convention v1.2 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x0 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x10000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x10100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x20000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x20100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x30000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x30100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x40000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x40100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x50000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x50100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x60000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x60100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x70000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x70100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x80000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x80100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x90000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x90100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xa0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xa0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xb0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xb0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xc0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xc0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xd0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xd0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xe0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xe0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xf0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xf0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x100000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x100100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x110000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x110100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x120000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x120100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x130000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x130100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x140000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x140100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x150000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x150100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x160000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x160100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x170000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x170100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x180000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x180100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x190000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x190100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1a0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1a0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1b0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1b0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1c0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1c0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1d0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1d0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1e0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1e0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1f0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1f0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x200000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x200100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x210000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x210100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x220000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x220100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x230000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x230100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x240000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x240100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x250000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x250100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x260000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x260100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x270000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x270100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x280000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x280100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x290000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x290100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2a0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2a0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2b0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2b0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2c0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2c0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2d0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2d0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2e0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2e0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2f0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2f0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x300000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x300100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x310000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x310100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x320000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x320100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x330000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x330100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x340000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x340100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x350000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x350100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x360000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x360100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x370000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x370100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x380000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x380100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x390000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x390100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3a0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3a0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3b0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3b0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3c0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3c0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3d0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3d0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3e0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3e0100 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3f0000 -> Node 0 [ 0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3f0100 -> Node 0 [ 0.000000] percpu: Embedded 31 pages/cpu s89000 r8192 d29784 u126976 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] CPU features: detected: GIC system register CPU interface [ 0.000000] CPU features: detected: Virtualization Host Extensions [ 0.000000] CPU features: detected: Hardware dirty bit management [ 0.000000] CPU features: detected: Spectre-v4 [ 0.000000] CPU features: detected: Spectre-BHB [ 0.000000] CPU features: kernel page table isolation forced ON by KASLR [ 0.000000] CPU features: detected: Kernel page table isolation (KPTI) [ 0.000000] CPU features: detected: ARM erratum 1418040 [ 0.000000] alternatives: patching kernel code [ 0.000000] Fallback order for Node 0: 0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 16450812 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.0.0-rc7 root=/dev/mapper/fedora_ampere--mtsnow--altramax--01-root ro rd.lvm.lv=fedora_ampere-mtsnow-altramax-01/root acpi=force [ 0.000000] Unknown kernel command line parameters "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.0.0-rc7", will be passed to user space. [ 0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] printk: log_buf_len total cpu_extra contributions: 520192 bytes [ 0.000000] printk: log_buf_len min size: 262144 bytes [ 0.000000] printk: log_buf_len: 1048576 bytes [ 0.000000] printk: early log buf free: 245848(93%) [ 0.000000] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear) [ 0.000000] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear) [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] software IO TLB: area num 128. [ 0.000000] software IO TLB: mapped [mem 0x00000000e7c91000-0x00000000ebc91000] (64MB) [ 0.000000] Memory: 65144248K/66847744K available (16704K kernel code, 4178K rwdata, 15188K rodata, 9152K init, 10736K bss, 1637960K reserved, 65536K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=128, Nodes=1 [ 0.000000] ftrace: allocating 57546 entries in 225 pages [ 0.000000] ftrace: allocated 225 pages with 4 groups [ 0.000000] trace event string verifier disabled [ 0.000000] Dynamic Preempt: voluntary [ 0.000000] rcu: Preemptible hierarchical RCU implementation. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=4096 to nr_cpu_ids=128. [ 0.000000] Trampoline variant of Tasks RCU enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=128 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode [ 0.000000] GICv3: 672 SPIs implemented [ 0.000000] GICv3: 0 Extended SPIs implemented [ 0.000000] Root IRQ handler: gic_handle_irq [ 0.000000] GICv3: GICv3 features: 16 PPIs [ 0.000000] GICv3: CPU0: found redistributor 120000 region 0:0x00001001005c0000 [ 0.000000] SRAT: PXM 0 -> ITS 0 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 1 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 2 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 3 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 4 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 5 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 6 -> Node 0 [ 0.000000] SRAT: PXM 0 -> ITS 7 -> Node 0 [ 0.000000] ITS [mem 0x100100040000-0x10010005ffff] [ 0.000000] ITS@0x0000100100040000: allocated 8192 Devices @80000830000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x0000100100040000: allocated 32768 Interrupt Collections @80000840000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x100100060000-0x10010007ffff] [ 0.000000] ITS@0x0000100100060000: allocated 8192 Devices @80000860000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x0000100100060000: allocated 32768 Interrupt Collections @80000870000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x100100080000-0x10010009ffff] [ 0.000000] ITS@0x0000100100080000: allocated 8192 Devices @80000890000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x0000100100080000: allocated 32768 Interrupt Collections @800008a0000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x1001000a0000-0x1001000bffff] [ 0.000000] ITS@0x00001001000a0000: allocated 8192 Devices @800008c0000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x00001001000a0000: allocated 32768 Interrupt Collections @800008d0000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x1001000c0000-0x1001000dffff] [ 0.000000] ITS@0x00001001000c0000: allocated 8192 Devices @800008f0000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x00001001000c0000: allocated 32768 Interrupt Collections @80000900000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x1001000e0000-0x1001000fffff] [ 0.000000] ITS@0x00001001000e0000: allocated 8192 Devices @80000920000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x00001001000e0000: allocated 32768 Interrupt Collections @80000930000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x100100100000-0x10010011ffff] [ 0.000000] ITS@0x0000100100100000: allocated 8192 Devices @80000960000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x0000100100100000: allocated 32768 Interrupt Collections @80000970000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] ITS [mem 0x100100120000-0x10010013ffff] [ 0.000000] ITS@0x0000100100120000: allocated 8192 Devices @80000990000 (indirect, esz 8, psz 64K, shr 1) [ 0.000000] ITS@0x0000100100120000: allocated 32768 Interrupt Collections @800009a0000 (flat, esz 2, psz 64K, shr 1) [ 0.000000] GICv3: using LPI property table @0x00000800009b0000 [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000800009c0000 [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes to big. [ 0.000000] kfence: initialized - using 2097152 bytes for 255 objects at 0x000000000d15f764-0x0000000060d831da [ 0.000000] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.000000] ACPI GTDT: found 1 memory-mapped timer block(s). [ 0.000000] arch_timer: cp15 and mmio timer(s) running at 25.00MHz (phys/phys). [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns [ 0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns [ 0.000116] Console: colour dummy device 80x25 [ 0.000166] ACPI: Core revision 20220331 [ 0.000320] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=250000) [ 0.000323] pid_max: default: 131072 minimum: 1024 [ 0.000374] LSM: Security Framework initializing [ 0.000388] Yama: becoming mindful. [ 0.000395] SELinux: Initializing. [ 0.000439] LSM support for eBPF active [ 0.000440] landlock: Up and running. [ 0.000492] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) [ 0.000515] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) [ 0.001483] cblist_init_generic: Setting adjustable number of callback queues. [ 0.001503] cblist_init_generic: Setting shift to 7 and lim to 1. [ 0.001535] cblist_init_generic: Setting shift to 7 and lim to 1. [ 0.001570] cblist_init_generic: Setting shift to 7 and lim to 1. [ 0.001614] rcu: Hierarchical SRCU implementation. [ 0.001615] rcu: Max phase no-delay instances is 1000. [ 0.001866] Platform MSI: ITS@0x100100040000 domain created [ 0.001868] Platform MSI: ITS@0x100100060000 domain created [ 0.001869] Platform MSI: ITS@0x100100080000 domain created [ 0.001871] Platform MSI: ITS@0x1001000a0000 domain created [ 0.001872] Platform MSI: ITS@0x1001000c0000 domain created [ 0.001873] Platform MSI: ITS@0x1001000e0000 domain created [ 0.001874] Platform MSI: ITS@0x100100100000 domain created [ 0.001876] Platform MSI: ITS@0x100100120000 domain created [ 0.001881] PCI/MSI: ITS@0x100100040000 domain created [ 0.001882] PCI/MSI: ITS@0x100100060000 domain created [ 0.001883] PCI/MSI: ITS@0x100100080000 domain created [ 0.001884] PCI/MSI: ITS@0x1001000a0000 domain created [ 0.001886] PCI/MSI: ITS@0x1001000c0000 domain created [ 0.001887] PCI/MSI: ITS@0x1001000e0000 domain created [ 0.001888] PCI/MSI: ITS@0x100100100000 domain created [ 0.001890] PCI/MSI: ITS@0x100100120000 domain created [ 0.001893] fsl-mc MSI: ITS@0x100100040000 domain created [ 0.001894] fsl-mc MSI: ITS@0x100100060000 domain created [ 0.001896] fsl-mc MSI: ITS@0x100100080000 domain created [ 0.001897] fsl-mc MSI: ITS@0x1001000a0000 domain created [ 0.001900] fsl-mc MSI: ITS@0x1001000c0000 domain created [ 0.001901] fsl-mc MSI: ITS@0x1001000e0000 domain created [ 0.001902] fsl-mc MSI: ITS@0x100100100000 domain created [ 0.001903] fsl-mc MSI: ITS@0x100100120000 domain created [ 0.001911] Remapping and enabling EFI services. [ 0.005632] smp: Bringing up secondary CPUs ... [ 0.005884] Detected PIPT I-cache on CPU1 [ 0.005940] GICv3: CPU1: found redistributor 120100 region 0:0x00001001005e0000 [ 0.005948] GICv3: CPU1: using allocated LPI pending table @0x00000800009d0000 [ 0.005990] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.006003] CPU1: Booted secondary processor 0x0000120100 [0x413fd0c1] [ 0.006676] Detected PIPT I-cache on CPU2 [ 0.006734] GICv3: CPU2: found redistributor 140000 region 0:0x0000100100640000 [ 0.006743] GICv3: CPU2: using allocated LPI pending table @0x00000800009e0000 [ 0.006787] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.006800] CPU2: Booted secondary processor 0x0000140000 [0x413fd0c1] [ 0.007656] Detected PIPT I-cache on CPU3 [ 0.007708] GICv3: CPU3: found redistributor 140100 region 0:0x0000100100660000 [ 0.007717] GICv3: CPU3: using allocated LPI pending table @0x00000800009f0000 [ 0.007760] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.007771] CPU3: Booted secondary processor 0x0000140100 [0x413fd0c1] [ 0.009551] Detected PIPT I-cache on CPU4 [ 0.009608] GICv3: CPU4: found redistributor 1a0000 region 0:0x00001001007c0000 [ 0.009617] GICv3: CPU4: using allocated LPI pending table @0x0000080000a00000 [ 0.009660] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.009673] CPU4: Booted secondary processor 0x00001a0000 [0x413fd0c1] [ 0.011048] Detected PIPT I-cache on CPU5 [ 0.011104] GICv3: CPU5: found redistributor 1a0100 region 0:0x00001001007e0000 [ 0.011112] GICv3: CPU5: using allocated LPI pending table @0x0000080000a10000 [ 0.011154] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.011166] CPU5: Booted secondary processor 0x00001a0100 [0x413fd0c1] [ 0.012469] Detected PIPT I-cache on CPU6 [ 0.012529] GICv3: CPU6: found redistributor 1c0000 region 0:0x0000100100840000 [ 0.012538] GICv3: CPU6: using allocated LPI pending table @0x0000080000a20000 [ 0.012582] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.012595] CPU6: Booted secondary processor 0x00001c0000 [0x413fd0c1] [ 0.013381] Detected PIPT I-cache on CPU7 [ 0.013439] GICv3: CPU7: found redistributor 1c0100 region 0:0x0000100100860000 [ 0.013447] GICv3: CPU7: using allocated LPI pending table @0x0000080000a30000 [ 0.013490] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.013502] CPU7: Booted secondary processor 0x00001c0100 [0x413fd0c1] [ 0.014679] Detected PIPT I-cache on CPU8 [ 0.014733] GICv3: CPU8: found redistributor 100000 region 0:0x0000100100540000 [ 0.014742] GICv3: CPU8: using allocated LPI pending table @0x0000080000a40000 [ 0.014786] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.014798] CPU8: Booted secondary processor 0x0000100000 [0x413fd0c1] [ 0.016064] Detected PIPT I-cache on CPU9 [ 0.016121] GICv3: CPU9: found redistributor 100100 region 0:0x0000100100560000 [ 0.016130] GICv3: CPU9: using allocated LPI pending table @0x0000080000a50000 [ 0.016173] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.016186] CPU9: Booted secondary processor 0x0000100100 [0x413fd0c1] [ 0.016981] Detected PIPT I-cache on CPU10 [ 0.017040] GICv3: CPU10: found redistributor 160000 region 0:0x00001001006c0000 [ 0.017049] GICv3: CPU10: using allocated LPI pending table @0x0000080000a60000 [ 0.017094] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.017106] CPU10: Booted secondary processor 0x0000160000 [0x413fd0c1] [ 0.018760] Detected PIPT I-cache on CPU11 [ 0.018821] GICv3: CPU11: found redistributor 160100 region 0:0x00001001006e0000 [ 0.018831] GICv3: CPU11: using allocated LPI pending table @0x0000080000a70000 [ 0.018874] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.018887] CPU11: Booted secondary processor 0x0000160100 [0x413fd0c1] [ 0.019937] Detected PIPT I-cache on CPU12 [ 0.019997] GICv3: CPU12: found redistributor 180000 region 0:0x0000100100740000 [ 0.020006] GICv3: CPU12: using allocated LPI pending table @0x0000080000a80000 [ 0.020049] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.020062] CPU12: Booted secondary processor 0x0000180000 [0x413fd0c1] [ 0.021249] Detected PIPT I-cache on CPU13 [ 0.021306] GICv3: CPU13: found redistributor 180100 region 0:0x0000100100760000 [ 0.021315] GICv3: CPU13: using allocated LPI pending table @0x0000080000a90000 [ 0.021358] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.021369] CPU13: Booted secondary processor 0x0000180100 [0x413fd0c1] [ 0.022671] Detected PIPT I-cache on CPU14 [ 0.022736] GICv3: CPU14: found redistributor 1e0000 region 0:0x00001001008c0000 [ 0.022745] GICv3: CPU14: using allocated LPI pending table @0x0000080000aa0000 [ 0.022790] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.022803] CPU14: Booted secondary processor 0x00001e0000 [0x413fd0c1] [ 0.023762] Detected PIPT I-cache on CPU15 [ 0.023824] GICv3: CPU15: found redistributor 1e0100 region 0:0x00001001008e0000 [ 0.023833] GICv3: CPU15: using allocated LPI pending table @0x0000080000ab0000 [ 0.023876] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.023889] CPU15: Booted secondary processor 0x00001e0100 [0x413fd0c1] [ 0.025132] Detected PIPT I-cache on CPU16 [ 0.025187] GICv3: CPU16: found redistributor a0000 region 0:0x00001001003c0000 [ 0.025196] GICv3: CPU16: using allocated LPI pending table @0x0000080000ac0000 [ 0.025240] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.025253] CPU16: Booted secondary processor 0x00000a0000 [0x413fd0c1] [ 0.026442] Detected PIPT I-cache on CPU17 [ 0.026493] GICv3: CPU17: found redistributor a0100 region 0:0x00001001003e0000 [ 0.026502] GICv3: CPU17: using allocated LPI pending table @0x0000080000ad0000 [ 0.026545] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.026558] CPU17: Booted secondary processor 0x00000a0100 [0x413fd0c1] [ 0.028657] Detected PIPT I-cache on CPU18 [ 0.028713] GICv3: CPU18: found redistributor c0000 region 0:0x0000100100440000 [ 0.028723] GICv3: CPU18: using allocated LPI pending table @0x0000080000ae0000 [ 0.028768] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.028781] CPU18: Booted secondary processor 0x00000c0000 [0x413fd0c1] [ 0.029845] Detected PIPT I-cache on CPU19 [ 0.029899] GICv3: CPU19: found redistributor c0100 region 0:0x0000100100460000 [ 0.029908] GICv3: CPU19: using allocated LPI pending table @0x0000080000af0000 [ 0.029951] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.029964] CPU19: Booted secondary processor 0x00000c0100 [0x413fd0c1] [ 0.031152] Detected PIPT I-cache on CPU20 [ 0.031220] GICv3: CPU20: found redistributor 220000 region 0:0x00001001009c0000 [ 0.031229] GICv3: CPU20: using allocated LPI pending table @0x0000080000b00000 [ 0.031273] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.031286] CPU20: Booted secondary processor 0x0000220000 [0x413fd0c1] [ 0.032580] Detected PIPT I-cache on CPU21 [ 0.032645] GICv3: CPU21: found redistributor 220100 region 0:0x00001001009e0000 [ 0.032654] GICv3: CPU21: using allocated LPI pending table @0x0000080000b10000 [ 0.032696] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.032709] CPU21: Booted secondary processor 0x0000220100 [0x413fd0c1] [ 0.034975] Detected PIPT I-cache on CPU22 [ 0.035046] GICv3: CPU22: found redistributor 240000 region 0:0x0000100100a40000 [ 0.035056] GICv3: CPU22: using allocated LPI pending table @0x0000080000b20000 [ 0.035100] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.035113] CPU22: Booted secondary processor 0x0000240000 [0x413fd0c1] [ 0.036244] Detected PIPT I-cache on CPU23 [ 0.036313] GICv3: CPU23: found redistributor 240100 region 0:0x0000100100a60000 [ 0.036322] GICv3: CPU23: using allocated LPI pending table @0x0000080000b30000 [ 0.036365] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.036376] CPU23: Booted secondary processor 0x0000240100 [0x413fd0c1] [ 0.038782] Detected PIPT I-cache on CPU24 [ 0.038838] GICv3: CPU24: found redistributor 80000 region 0:0x0000100100340000 [ 0.038848] GICv3: CPU24: using allocated LPI pending table @0x0000080000b40000 [ 0.038892] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.038905] CPU24: Booted secondary processor 0x0000080000 [0x413fd0c1] [ 0.040955] Detected PIPT I-cache on CPU25 [ 0.041013] GICv3: CPU25: found redistributor 80100 region 0:0x0000100100360000 [ 0.041023] GICv3: CPU25: using allocated LPI pending table @0x0000080000b50000 [ 0.041067] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.041080] CPU25: Booted secondary processor 0x0000080100 [0x413fd0c1] [ 0.042469] Detected PIPT I-cache on CPU26 [ 0.042530] GICv3: CPU26: found redistributor e0000 region 0:0x00001001004c0000 [ 0.042540] GICv3: CPU26: using allocated LPI pending table @0x0000080000b60000 [ 0.042585] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.042598] CPU26: Booted secondary processor 0x00000e0000 [0x413fd0c1] [ 0.043868] Detected PIPT I-cache on CPU27 [ 0.043931] GICv3: CPU27: found redistributor e0100 region 0:0x00001001004e0000 [ 0.043941] GICv3: CPU27: using allocated LPI pending table @0x0000080000b70000 [ 0.043985] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.043998] CPU27: Booted secondary processor 0x00000e0100 [0x413fd0c1] [ 0.045060] Detected PIPT I-cache on CPU28 [ 0.045130] GICv3: CPU28: found redistributor 200000 region 0:0x0000100100940000 [ 0.045140] GICv3: CPU28: using allocated LPI pending table @0x0000080000b80000 [ 0.045183] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.045197] CPU28: Booted secondary processor 0x0000200000 [0x413fd0c1] [ 0.048081] Detected PIPT I-cache on CPU29 [ 0.048155] GICv3: CPU29: found redistributor 200100 region 0:0x0000100100960000 [ 0.048165] GICv3: CPU29: using allocated LPI pending table @0x0000080000b90000 [ 0.048207] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.048220] CPU29: Booted secondary processor 0x0000200100 [0x413fd0c1] [ 0.049662] Detected PIPT I-cache on CPU30 [ 0.049738] GICv3: CPU30: found redistributor 260000 region 0:0x0000100100ac0000 [ 0.049748] GICv3: CPU30: using allocated LPI pending table @0x0000080000ba0000 [ 0.049793] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.049806] CPU30: Booted secondary processor 0x0000260000 [0x413fd0c1] [ 0.051153] Detected PIPT I-cache on CPU31 [ 0.051230] GICv3: CPU31: found redistributor 260100 region 0:0x0000100100ae0000 [ 0.051240] GICv3: CPU31: using allocated LPI pending table @0x0000080000bb0000 [ 0.051284] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.051296] CPU31: Booted secondary processor 0x0000260100 [0x413fd0c1] [ 0.052666] Detected PIPT I-cache on CPU32 [ 0.052723] GICv3: CPU32: found redistributor 20000 region 0:0x00001001001c0000 [ 0.052734] GICv3: CPU32: using allocated LPI pending table @0x0000080000bc0000 [ 0.052779] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.052792] CPU32: Booted secondary processor 0x0000020000 [0x413fd0c1] [ 0.053842] Detected PIPT I-cache on CPU33 [ 0.053894] GICv3: CPU33: found redistributor 20100 region 0:0x00001001001e0000 [ 0.053904] GICv3: CPU33: using allocated LPI pending table @0x0000080000bd0000 [ 0.053948] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.053960] CPU33: Booted secondary processor 0x0000020100 [0x413fd0c1] [ 0.055066] Detected PIPT I-cache on CPU34 [ 0.055124] GICv3: CPU34: found redistributor 40000 region 0:0x0000100100240000 [ 0.055134] GICv3: CPU34: using allocated LPI pending table @0x0000080000be0000 [ 0.055179] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.055192] CPU34: Booted secondary processor 0x0000040000 [0x413fd0c1] [ 0.056354] Detected PIPT I-cache on CPU35 [ 0.056408] GICv3: CPU35: found redistributor 40100 region 0:0x0000100100260000 [ 0.056418] GICv3: CPU35: using allocated LPI pending table @0x0000080000bf0000 [ 0.056461] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.056474] CPU35: Booted secondary processor 0x0000040100 [0x413fd0c1] [ 0.057981] Detected PIPT I-cache on CPU36 [ 0.058060] GICv3: CPU36: found redistributor 2a0000 region 0:0x0000100100bc0000 [ 0.058070] GICv3: CPU36: using allocated LPI pending table @0x0000080000c00000 [ 0.058113] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.058127] CPU36: Booted secondary processor 0x00002a0000 [0x413fd0c1] [ 0.059987] Detected PIPT I-cache on CPU37 [ 0.060068] GICv3: CPU37: found redistributor 2a0100 region 0:0x0000100100be0000 [ 0.060078] GICv3: CPU37: using allocated LPI pending table @0x0000080000c10000 [ 0.060120] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.060133] CPU37: Booted secondary processor 0x00002a0100 [0x413fd0c1] [ 0.061179] Detected PIPT I-cache on CPU38 [ 0.061259] GICv3: CPU38: found redistributor 2c0000 region 0:0x0000100100c40000 [ 0.061269] GICv3: CPU38: using allocated LPI pending table @0x0000080000c20000 [ 0.061313] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.061326] CPU38: Booted secondary processor 0x00002c0000 [0x413fd0c1] [ 0.062675] Detected PIPT I-cache on CPU39 [ 0.062759] GICv3: CPU39: found redistributor 2c0100 region 0:0x0000100100c60000 [ 0.062769] GICv3: CPU39: using allocated LPI pending table @0x0000080000c30000 [ 0.062811] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.062824] CPU39: Booted secondary processor 0x00002c0100 [0x413fd0c1] [ 0.065061] Detected PIPT I-cache on CPU40 [ 0.065120] GICv3: CPU40: found redistributor 0 region 0:0x0000100100140000 [ 0.065130] GICv3: CPU40: using allocated LPI pending table @0x0000080000c40000 [ 0.065175] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.065188] CPU40: Booted secondary processor 0x0000000000 [0x413fd0c1] [ 0.066350] Detected PIPT I-cache on CPU41 [ 0.066402] GICv3: CPU41: found redistributor 100 region 0:0x0000100100160000 [ 0.066413] GICv3: CPU41: using allocated LPI pending table @0x0000080000c50000 [ 0.066456] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.066469] CPU41: Booted secondary processor 0x0000000100 [0x413fd0c1] [ 0.067982] Detected PIPT I-cache on CPU42 [ 0.068045] GICv3: CPU42: found redistributor 60000 region 0:0x00001001002c0000 [ 0.068056] GICv3: CPU42: using allocated LPI pending table @0x0000080000c60000 [ 0.068102] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.068115] CPU42: Booted secondary processor 0x0000060000 [0x413fd0c1] [ 0.071246] Detected PIPT I-cache on CPU43 [ 0.071309] GICv3: CPU43: found redistributor 60100 region 0:0x00001001002e0000 [ 0.071320] GICv3: CPU43: using allocated LPI pending table @0x0000080000c70000 [ 0.071363] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.071376] CPU43: Booted secondary processor 0x0000060100 [0x413fd0c1] [ 0.072470] Detected PIPT I-cache on CPU44 [ 0.072551] GICv3: CPU44: found redistributor 280000 region 0:0x0000100100b40000 [ 0.072562] GICv3: CPU44: using allocated LPI pending table @0x0000080000c80000 [ 0.072605] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.072619] CPU44: Booted secondary processor 0x0000280000 [0x413fd0c1] [ 0.075138] Detected PIPT I-cache on CPU45 [ 0.075221] GICv3: CPU45: found redistributor 280100 region 0:0x0000100100b60000 [ 0.075232] GICv3: CPU45: using allocated LPI pending table @0x0000080000c90000 [ 0.075274] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.075286] CPU45: Booted secondary processor 0x0000280100 [0x413fd0c1] [ 0.076368] Detected PIPT I-cache on CPU46 [ 0.076453] GICv3: CPU46: found redistributor 2e0000 region 0:0x0000100100cc0000 [ 0.076464] GICv3: CPU46: using allocated LPI pending table @0x0000080000ca0000 [ 0.076508] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.076521] CPU46: Booted secondary processor 0x00002e0000 [0x413fd0c1] [ 0.079962] Detected PIPT I-cache on CPU47 [ 0.080049] GICv3: CPU47: found redistributor 2e0100 region 0:0x0000100100ce0000 [ 0.080060] GICv3: CPU47: using allocated LPI pending table @0x0000080000cb0000 [ 0.080102] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.080115] CPU47: Booted secondary processor 0x00002e0100 [0x413fd0c1] [ 0.081254] Detected PIPT I-cache on CPU48 [ 0.081342] GICv3: CPU48: found redistributor 320000 region 0:0x0000100100dc0000 [ 0.081353] GICv3: CPU48: using allocated LPI pending table @0x0000080000cc0000 [ 0.081395] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.081409] CPU48: Booted secondary processor 0x0000320000 [0x413fd0c1] [ 0.082473] Detected PIPT I-cache on CPU49 [ 0.082554] GICv3: CPU49: found redistributor 320100 region 0:0x0000100100de0000 [ 0.082564] GICv3: CPU49: using allocated LPI pending table @0x0000080000cd0000 [ 0.082605] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.082618] CPU49: Booted secondary processor 0x0000320100 [0x413fd0c1] [ 0.085161] Detected PIPT I-cache on CPU50 [ 0.085253] GICv3: CPU50: found redistributor 340000 region 0:0x0000100100e40000 [ 0.085264] GICv3: CPU50: using allocated LPI pending table @0x0000080000ce0000 [ 0.085307] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.085321] CPU50: Booted secondary processor 0x0000340000 [0x413fd0c1] [ 0.086383] Detected PIPT I-cache on CPU51 [ 0.086465] GICv3: CPU51: found redistributor 340100 region 0:0x0000100100e60000 [ 0.086475] GICv3: CPU51: using allocated LPI pending table @0x0000080000cf0000 [ 0.086517] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.086530] CPU51: Booted secondary processor 0x0000340100 [0x413fd0c1] [ 0.089986] Detected PIPT I-cache on CPU52 [ 0.090075] GICv3: CPU52: found redistributor 300000 region 0:0x0000100100d40000 [ 0.090087] GICv3: CPU52: using allocated LPI pending table @0x0000080000d00000 [ 0.090130] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.090144] CPU52: Booted secondary processor 0x0000300000 [0x413fd0c1] [ 0.091260] Detected PIPT I-cache on CPU53 [ 0.091340] GICv3: CPU53: found redistributor 300100 region 0:0x0000100100d60000 [ 0.091351] GICv3: CPU53: using allocated LPI pending table @0x0000080000d10000 [ 0.091392] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.091405] CPU53: Booted secondary processor 0x0000300100 [0x413fd0c1] [ 0.092473] Detected PIPT I-cache on CPU54 [ 0.092566] GICv3: CPU54: found redistributor 360000 region 0:0x0000100100ec0000 [ 0.092577] GICv3: CPU54: using allocated LPI pending table @0x0000080000d20000 [ 0.092621] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.092634] CPU54: Booted secondary processor 0x0000360000 [0x413fd0c1] [ 0.095138] Detected PIPT I-cache on CPU55 [ 0.095233] GICv3: CPU55: found redistributor 360100 region 0:0x0000100100ee0000 [ 0.095245] GICv3: CPU55: using allocated LPI pending table @0x0000080000d30000 [ 0.095287] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.095300] CPU55: Booted secondary processor 0x0000360100 [0x413fd0c1] [ 0.096374] Detected PIPT I-cache on CPU56 [ 0.096469] GICv3: CPU56: found redistributor 3a0000 region 0:0x0000100100fc0000 [ 0.096480] GICv3: CPU56: using allocated LPI pending table @0x0000080000d40000 [ 0.096522] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.096536] CPU56: Booted secondary processor 0x00003a0000 [0x413fd0c1] [ 0.099955] Detected PIPT I-cache on CPU57 [ 0.100051] GICv3: CPU57: found redistributor 3a0100 region 0:0x0000100100fe0000 [ 0.100063] GICv3: CPU57: using allocated LPI pending table @0x0000080000d50000 [ 0.100104] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.100117] CPU57: Booted secondary processor 0x00003a0100 [0x413fd0c1] [ 0.101257] Detected PIPT I-cache on CPU58 [ 0.101354] GICv3: CPU58: found redistributor 3c0000 region 0:0x0000100101040000 [ 0.101366] GICv3: CPU58: using allocated LPI pending table @0x0000080000d60000 [ 0.101409] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.101422] CPU58: Booted secondary processor 0x00003c0000 [0x413fd0c1] [ 0.102562] Detected PIPT I-cache on CPU59 [ 0.102651] GICv3: CPU59: found redistributor 3c0100 region 0:0x0000100101060000 [ 0.102662] GICv3: CPU59: using allocated LPI pending table @0x0000080000d70000 [ 0.102703] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.102715] CPU59: Booted secondary processor 0x00003c0100 [0x413fd0c1] [ 0.105075] Detected PIPT I-cache on CPU60 [ 0.105173] GICv3: CPU60: found redistributor 380000 region 0:0x0000100100f40000 [ 0.105184] GICv3: CPU60: using allocated LPI pending table @0x0000080000d80000 [ 0.105227] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.105241] CPU60: Booted secondary processor 0x0000380000 [0x413fd0c1] [ 0.106353] Detected PIPT I-cache on CPU61 [ 0.106441] GICv3: CPU61: found redistributor 380100 region 0:0x0000100100f60000 [ 0.106452] GICv3: CPU61: using allocated LPI pending table @0x0000080000d90000 [ 0.106494] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.106505] CPU61: Booted secondary processor 0x0000380100 [0x413fd0c1] [ 0.108577] Detected PIPT I-cache on CPU62 [ 0.108679] GICv3: CPU62: found redistributor 3e0000 region 0:0x00001001010c0000 [ 0.108691] GICv3: CPU62: using allocated LPI pending table @0x0000080000da0000 [ 0.108735] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.108748] CPU62: Booted secondary processor 0x00003e0000 [0x413fd0c1] [ 0.111179] Detected PIPT I-cache on CPU63 [ 0.111281] GICv3: CPU63: found redistributor 3e0100 region 0:0x00001001010e0000 [ 0.111292] GICv3: CPU63: using allocated LPI pending table @0x0000080000db0000 [ 0.111334] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.111347] CPU63: Booted secondary processor 0x00003e0100 [0x413fd0c1] [ 0.112671] Detected PIPT I-cache on CPU64 [ 0.112747] GICv3: CPU64: found redistributor 130000 region 0:0x0000100100600000 [ 0.112759] GICv3: CPU64: using allocated LPI pending table @0x0000080000dc0000 [ 0.112803] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.112816] CPU64: Booted secondary processor 0x0000130000 [0x413fd0c1] [ 0.115068] Detected PIPT I-cache on CPU65 [ 0.115147] GICv3: CPU65: found redistributor 130100 region 0:0x0000100100620000 [ 0.115158] GICv3: CPU65: using allocated LPI pending table @0x0000080000dd0000 [ 0.115201] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.115214] CPU65: Booted secondary processor 0x0000130100 [0x413fd0c1] [ 0.116344] Detected PIPT I-cache on CPU66 [ 0.116419] GICv3: CPU66: found redistributor 150000 region 0:0x0000100100680000 [ 0.116430] GICv3: CPU66: using allocated LPI pending table @0x0000080000de0000 [ 0.116474] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.116487] CPU66: Booted secondary processor 0x0000150000 [0x413fd0c1] [ 0.118565] Detected PIPT I-cache on CPU67 [ 0.118645] GICv3: CPU67: found redistributor 150100 region 0:0x00001001006a0000 [ 0.118657] GICv3: CPU67: using allocated LPI pending table @0x0000080000df0000 [ 0.118700] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.118714] CPU67: Booted secondary processor 0x0000150100 [0x413fd0c1] [ 0.121154] Detected PIPT I-cache on CPU68 [ 0.121234] GICv3: CPU68: found redistributor 1b0000 region 0:0x0000100100800000 [ 0.121246] GICv3: CPU68: using allocated LPI pending table @0x0000080000e00000 [ 0.121290] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.121302] CPU68: Booted secondary processor 0x00001b0000 [0x413fd0c1] [ 0.122674] Detected PIPT I-cache on CPU69 [ 0.122758] GICv3: CPU69: found redistributor 1b0100 region 0:0x0000100100820000 [ 0.122771] GICv3: CPU69: using allocated LPI pending table @0x0000080000e10000 [ 0.122813] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.122826] CPU69: Booted secondary processor 0x00001b0100 [0x413fd0c1] [ 0.125065] Detected PIPT I-cache on CPU70 [ 0.125147] GICv3: CPU70: found redistributor 1d0000 region 0:0x0000100100880000 [ 0.125158] GICv3: CPU70: using allocated LPI pending table @0x0000080000e20000 [ 0.125202] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.125215] CPU70: Booted secondary processor 0x00001d0000 [0x413fd0c1] [ 0.126351] Detected PIPT I-cache on CPU71 [ 0.126429] GICv3: CPU71: found redistributor 1d0100 region 0:0x00001001008a0000 [ 0.126441] GICv3: CPU71: using allocated LPI pending table @0x0000080000e30000 [ 0.126483] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.126496] CPU71: Booted secondary processor 0x00001d0100 [0x413fd0c1] [ 0.128563] Detected PIPT I-cache on CPU72 [ 0.128641] GICv3: CPU72: found redistributor 110000 region 0:0x0000100100580000 [ 0.128653] GICv3: CPU72: using allocated LPI pending table @0x0000080000e40000 [ 0.128697] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.128710] CPU72: Booted secondary processor 0x0000110000 [0x413fd0c1] [ 0.131165] Detected PIPT I-cache on CPU73 [ 0.131245] GICv3: CPU73: found redistributor 110100 region 0:0x00001001005a0000 [ 0.131257] GICv3: CPU73: using allocated LPI pending table @0x0000080000e50000 [ 0.131299] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.131312] CPU73: Booted secondary processor 0x0000110100 [0x413fd0c1] [ 0.132672] Detected PIPT I-cache on CPU74 [ 0.132753] GICv3: CPU74: found redistributor 170000 region 0:0x0000100100700000 [ 0.132765] GICv3: CPU74: using allocated LPI pending table @0x0000080000e60000 [ 0.132810] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.132823] CPU74: Booted secondary processor 0x0000170000 [0x413fd0c1] [ 0.135056] Detected PIPT I-cache on CPU75 [ 0.135140] GICv3: CPU75: found redistributor 170100 region 0:0x0000100100720000 [ 0.135153] GICv3: CPU75: using allocated LPI pending table @0x0000080000e70000 [ 0.135196] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.135209] CPU75: Booted secondary processor 0x0000170100 [0x413fd0c1] [ 0.136351] Detected PIPT I-cache on CPU76 [ 0.136432] GICv3: CPU76: found redistributor 190000 region 0:0x0000100100780000 [ 0.136444] GICv3: CPU76: using allocated LPI pending table @0x0000080000e80000 [ 0.136488] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.136501] CPU76: Booted secondary processor 0x0000190000 [0x413fd0c1] [ 0.138572] Detected PIPT I-cache on CPU77 [ 0.138657] GICv3: CPU77: found redistributor 190100 region 0:0x00001001007a0000 [ 0.138670] GICv3: CPU77: using allocated LPI pending table @0x0000080000e90000 [ 0.138712] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.138725] CPU77: Booted secondary processor 0x0000190100 [0x413fd0c1] [ 0.141161] Detected PIPT I-cache on CPU78 [ 0.141248] GICv3: CPU78: found redistributor 1f0000 region 0:0x0000100100900000 [ 0.141260] GICv3: CPU78: using allocated LPI pending table @0x0000080000ea0000 [ 0.141304] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.141317] CPU78: Booted secondary processor 0x00001f0000 [0x413fd0c1] [ 0.142672] Detected PIPT I-cache on CPU79 [ 0.142762] GICv3: CPU79: found redistributor 1f0100 region 0:0x0000100100920000 [ 0.142774] GICv3: CPU79: using allocated LPI pending table @0x0000080000eb0000 [ 0.142817] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.142830] CPU79: Booted secondary processor 0x00001f0100 [0x413fd0c1] [ 0.145138] Detected PIPT I-cache on CPU80 [ 0.145215] GICv3: CPU80: found redistributor b0000 region 0:0x0000100100400000 [ 0.145227] GICv3: CPU80: using allocated LPI pending table @0x0000080000ec0000 [ 0.145271] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.145284] CPU80: Booted secondary processor 0x00000b0000 [0x413fd0c1] [ 0.146469] Detected PIPT I-cache on CPU81 [ 0.146539] GICv3: CPU81: found redistributor b0100 region 0:0x0000100100420000 [ 0.146551] GICv3: CPU81: using allocated LPI pending table @0x0000080000ed0000 [ 0.146594] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.146607] CPU81: Booted secondary processor 0x00000b0100 [0x413fd0c1] [ 0.148082] Detected PIPT I-cache on CPU82 [ 0.148159] GICv3: CPU82: found redistributor d0000 region 0:0x0000100100480000 [ 0.148171] GICv3: CPU82: using allocated LPI pending table @0x0000080000ee0000 [ 0.148216] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.148229] CPU82: Booted secondary processor 0x00000d0000 [0x413fd0c1] [ 0.149755] Detected PIPT I-cache on CPU83 [ 0.149835] GICv3: CPU83: found redistributor d0100 region 0:0x00001001004a0000 [ 0.149848] GICv3: CPU83: using allocated LPI pending table @0x0000080000ef0000 [ 0.149891] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.149904] CPU83: Booted secondary processor 0x00000d0100 [0x413fd0c1] [ 0.151280] Detected PIPT I-cache on CPU84 [ 0.151370] GICv3: CPU84: found redistributor 230000 region 0:0x0000100100a00000 [ 0.151382] GICv3: CPU84: using allocated LPI pending table @0x0000080000f00000 [ 0.151426] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.151439] CPU84: Booted secondary processor 0x0000230000 [0x413fd0c1] [ 0.152571] Detected PIPT I-cache on CPU85 [ 0.152655] GICv3: CPU85: found redistributor 230100 region 0:0x0000100100a20000 [ 0.152667] GICv3: CPU85: using allocated LPI pending table @0x0000080000f10000 [ 0.152708] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.152721] CPU85: Booted secondary processor 0x0000230100 [0x413fd0c1] [ 0.155067] Detected PIPT I-cache on CPU86 [ 0.155158] GICv3: CPU86: found redistributor 250000 region 0:0x0000100100a80000 [ 0.155171] GICv3: CPU86: using allocated LPI pending table @0x0000080000f20000 [ 0.155214] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.155227] CPU86: Booted secondary processor 0x0000250000 [0x413fd0c1] [ 0.156359] Detected PIPT I-cache on CPU87 [ 0.156444] GICv3: CPU87: found redistributor 250100 region 0:0x0000100100aa0000 [ 0.156456] GICv3: CPU87: using allocated LPI pending table @0x0000080000f30000 [ 0.156498] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.156511] CPU87: Booted secondary processor 0x0000250100 [0x413fd0c1] [ 0.158558] Detected PIPT I-cache on CPU88 [ 0.158637] GICv3: CPU88: found redistributor 90000 region 0:0x0000100100380000 [ 0.158650] GICv3: CPU88: using allocated LPI pending table @0x0000080000f40000 [ 0.158694] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.158707] CPU88: Booted secondary processor 0x0000090000 [0x413fd0c1] [ 0.161164] Detected PIPT I-cache on CPU89 [ 0.161245] GICv3: CPU89: found redistributor 90100 region 0:0x00001001003a0000 [ 0.161258] GICv3: CPU89: using allocated LPI pending table @0x0000080000f50000 [ 0.161301] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.161314] CPU89: Booted secondary processor 0x0000090100 [0x413fd0c1] [ 0.162672] Detected PIPT I-cache on CPU90 [ 0.162755] GICv3: CPU90: found redistributor f0000 region 0:0x0000100100500000 [ 0.162768] GICv3: CPU90: using allocated LPI pending table @0x0000080000f60000 [ 0.162813] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.162826] CPU90: Booted secondary processor 0x00000f0000 [0x413fd0c1] [ 0.165072] Detected PIPT I-cache on CPU91 [ 0.165156] GICv3: CPU91: found redistributor f0100 region 0:0x0000100100520000 [ 0.165169] GICv3: CPU91: using allocated LPI pending table @0x0000080000f70000 [ 0.165213] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.165226] CPU91: Booted secondary processor 0x00000f0100 [0x413fd0c1] [ 0.166356] Detected PIPT I-cache on CPU92 [ 0.166447] GICv3: CPU92: found redistributor 210000 region 0:0x0000100100980000 [ 0.166459] GICv3: CPU92: using allocated LPI pending table @0x0000080000f80000 [ 0.166502] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.166516] CPU92: Booted secondary processor 0x0000210000 [0x413fd0c1] [ 0.168570] Detected PIPT I-cache on CPU93 [ 0.168664] GICv3: CPU93: found redistributor 210100 region 0:0x00001001009a0000 [ 0.168677] GICv3: CPU93: using allocated LPI pending table @0x0000080000f90000 [ 0.168719] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.168732] CPU93: Booted secondary processor 0x0000210100 [0x413fd0c1] [ 0.171183] Detected PIPT I-cache on CPU94 [ 0.171279] GICv3: CPU94: found redistributor 270000 region 0:0x0000100100b00000 [ 0.171292] GICv3: CPU94: using allocated LPI pending table @0x0000080000fa0000 [ 0.171336] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.171349] CPU94: Booted secondary processor 0x0000270000 [0x413fd0c1] [ 0.172684] Detected PIPT I-cache on CPU95 [ 0.172784] GICv3: CPU95: found redistributor 270100 region 0:0x0000100100b20000 [ 0.172798] GICv3: CPU95: using allocated LPI pending table @0x0000080000fb0000 [ 0.172840] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.172853] CPU95: Booted secondary processor 0x0000270100 [0x413fd0c1] [ 0.175072] Detected PIPT I-cache on CPU96 [ 0.175153] GICv3: CPU96: found redistributor 30000 region 0:0x0000100100200000 [ 0.175166] GICv3: CPU96: using allocated LPI pending table @0x0000080000fc0000 [ 0.175211] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.175224] CPU96: Booted secondary processor 0x0000030000 [0x413fd0c1] [ 0.176353] Detected PIPT I-cache on CPU97 [ 0.176425] GICv3: CPU97: found redistributor 30100 region 0:0x0000100100220000 [ 0.176438] GICv3: CPU97: using allocated LPI pending table @0x0000080000fd0000 [ 0.176481] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.176494] CPU97: Booted secondary processor 0x0000030100 [0x413fd0c1] [ 0.177985] Detected PIPT I-cache on CPU98 [ 0.178065] GICv3: CPU98: found redistributor 50000 region 0:0x0000100100280000 [ 0.178078] GICv3: CPU98: using allocated LPI pending table @0x0000080000fe0000 [ 0.178123] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.178136] CPU98: Booted secondary processor 0x0000050000 [0x413fd0c1] [ 0.181241] Detected PIPT I-cache on CPU99 [ 0.181322] GICv3: CPU99: found redistributor 50100 region 0:0x00001001002a0000 [ 0.181336] GICv3: CPU99: using allocated LPI pending table @0x0000080000ff0000 [ 0.181380] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.181393] CPU99: Booted secondary processor 0x0000050100 [0x413fd0c1] [ 0.182554] Detected PIPT I-cache on CPU100 [ 0.182653] GICv3: CPU100: found redistributor 2b0000 region 0:0x0000100100c00000 [ 0.182667] GICv3: CPU100: using allocated LPI pending table @0x0000080001000000 [ 0.182710] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.182722] CPU100: Booted secondary processor 0x00002b0000 [0x413fd0c1] [ 0.186336] Detected PIPT I-cache on CPU101 [ 0.186439] GICv3: CPU101: found redistributor 2b0100 region 0:0x0000100100c20000 [ 0.186453] GICv3: CPU101: using allocated LPI pending table @0x0000080001010000 [ 0.186494] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.186507] CPU101: Booted secondary processor 0x00002b0100 [0x413fd0c1] [ 0.191158] Detected PIPT I-cache on CPU102 [ 0.191261] GICv3: CPU102: found redistributor 2d0000 region 0:0x0000100100c80000 [ 0.191274] GICv3: CPU102: using allocated LPI pending table @0x0000080001020000 [ 0.191317] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.191330] CPU102: Booted secondary processor 0x00002d0000 [0x413fd0c1] [ 0.192688] Detected PIPT I-cache on CPU103 [ 0.192794] GICv3: CPU103: found redistributor 2d0100 region 0:0x0000100100ca0000 [ 0.192808] GICv3: CPU103: using allocated LPI pending table @0x0000080001030000 [ 0.192850] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.192863] CPU103: Booted secondary processor 0x00002d0100 [0x413fd0c1] [ 0.196338] Detected PIPT I-cache on CPU104 [ 0.196421] GICv3: CPU104: found redistributor 10000 region 0:0x0000100100180000 [ 0.196434] GICv3: CPU104: using allocated LPI pending table @0x0000080001040000 [ 0.196479] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.196492] CPU104: Booted secondary processor 0x0000010000 [0x413fd0c1] [ 0.198579] Detected PIPT I-cache on CPU105 [ 0.198660] GICv3: CPU105: found redistributor 10100 region 0:0x00001001001a0000 [ 0.198674] GICv3: CPU105: using allocated LPI pending table @0x0000080001050000 [ 0.198718] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.198731] CPU105: Booted secondary processor 0x0000010100 [0x413fd0c1] [ 0.201170] Detected PIPT I-cache on CPU106 [ 0.201256] GICv3: CPU106: found redistributor 70000 region 0:0x0000100100300000 [ 0.201269] GICv3: CPU106: using allocated LPI pending table @0x0000080001060000 [ 0.201315] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.201328] CPU106: Booted secondary processor 0x0000070000 [0x413fd0c1] [ 0.202684] Detected PIPT I-cache on CPU107 [ 0.202770] GICv3: CPU107: found redistributor 70100 region 0:0x0000100100320000 [ 0.202785] GICv3: CPU107: using allocated LPI pending table @0x0000080001070000 [ 0.202829] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.202842] CPU107: Booted secondary processor 0x0000070100 [0x413fd0c1] [ 0.205082] Detected PIPT I-cache on CPU108 [ 0.205187] GICv3: CPU108: found redistributor 290000 region 0:0x0000100100b80000 [ 0.205201] GICv3: CPU108: using allocated LPI pending table @0x0000080001080000 [ 0.205244] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.205257] CPU108: Booted secondary processor 0x0000290000 [0x413fd0c1] [ 0.206358] Detected PIPT I-cache on CPU109 [ 0.206453] GICv3: CPU109: found redistributor 290100 region 0:0x0000100100ba0000 [ 0.206466] GICv3: CPU109: using allocated LPI pending table @0x0000080001090000 [ 0.206507] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.206520] CPU109: Booted secondary processor 0x0000290100 [0x413fd0c1] [ 0.208578] Detected PIPT I-cache on CPU110 [ 0.208687] GICv3: CPU110: found redistributor 2f0000 region 0:0x0000100100d00000 [ 0.208701] GICv3: CPU110: using allocated LPI pending table @0x00000800010a0000 [ 0.208745] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.208758] CPU110: Booted secondary processor 0x00002f0000 [0x413fd0c1] [ 0.211174] Detected PIPT I-cache on CPU111 [ 0.211283] GICv3: CPU111: found redistributor 2f0100 region 0:0x0000100100d20000 [ 0.211297] GICv3: CPU111: using allocated LPI pending table @0x00000800010b0000 [ 0.211340] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.211353] CPU111: Booted secondary processor 0x00002f0100 [0x413fd0c1] [ 0.212865] Detected PIPT I-cache on CPU112 [ 0.212977] GICv3: CPU112: found redistributor 330000 region 0:0x0000100100e00000 [ 0.212991] GICv3: CPU112: using allocated LPI pending table @0x00000800010c0000 [ 0.213034] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.213047] CPU112: Booted secondary processor 0x0000330000 [0x413fd0c1] [ 0.215070] Detected PIPT I-cache on CPU113 [ 0.215182] GICv3: CPU113: found redistributor 330100 region 0:0x0000100100e20000 [ 0.215195] GICv3: CPU113: using allocated LPI pending table @0x00000800010d0000 [ 0.215237] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.215250] CPU113: Booted secondary processor 0x0000330100 [0x413fd0c1] [ 0.216363] Detected PIPT I-cache on CPU114 [ 0.216474] GICv3: CPU114: found redistributor 350000 region 0:0x0000100100e80000 [ 0.216488] GICv3: CPU114: using allocated LPI pending table @0x00000800010e0000 [ 0.216531] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.216544] CPU114: Booted secondary processor 0x0000350000 [0x413fd0c1] [ 0.218584] Detected PIPT I-cache on CPU115 [ 0.218697] GICv3: CPU115: found redistributor 350100 region 0:0x0000100100ea0000 [ 0.218711] GICv3: CPU115: using allocated LPI pending table @0x00000800010f0000 [ 0.218752] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.218766] CPU115: Booted secondary processor 0x0000350100 [0x413fd0c1] [ 0.221189] Detected PIPT I-cache on CPU116 [ 0.221302] GICv3: CPU116: found redistributor 310000 region 0:0x0000100100d80000 [ 0.221315] GICv3: CPU116: using allocated LPI pending table @0x0000080001100000 [ 0.221358] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.221372] CPU116: Booted secondary processor 0x0000310000 [0x413fd0c1] [ 0.222859] Detected PIPT I-cache on CPU117 [ 0.222971] GICv3: CPU117: found redistributor 310100 region 0:0x0000100100da0000 [ 0.222986] GICv3: CPU117: using allocated LPI pending table @0x0000080001110000 [ 0.223027] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.223040] CPU117: Booted secondary processor 0x0000310100 [0x413fd0c1] [ 0.225169] Detected PIPT I-cache on CPU118 [ 0.225286] GICv3: CPU118: found redistributor 370000 region 0:0x0000100100f00000 [ 0.225300] GICv3: CPU118: using allocated LPI pending table @0x0000080001120000 [ 0.225344] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.225357] CPU118: Booted secondary processor 0x0000370000 [0x413fd0c1] [ 0.226468] Detected PIPT I-cache on CPU119 [ 0.226572] GICv3: CPU119: found redistributor 370100 region 0:0x0000100100f20000 [ 0.226586] GICv3: CPU119: using allocated LPI pending table @0x0000080001130000 [ 0.226628] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.226641] CPU119: Booted secondary processor 0x0000370100 [0x413fd0c1] [ 0.230085] Detected PIPT I-cache on CPU120 [ 0.230206] GICv3: CPU120: found redistributor 3b0000 region 0:0x0000100101000000 [ 0.230220] GICv3: CPU120: using allocated LPI pending table @0x0000080001140000 [ 0.230263] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.230277] CPU120: Booted secondary processor 0x00003b0000 [0x413fd0c1] [ 0.231283] Detected PIPT I-cache on CPU121 [ 0.231390] GICv3: CPU121: found redistributor 3b0100 region 0:0x0000100101020000 [ 0.231404] GICv3: CPU121: using allocated LPI pending table @0x0000080001150000 [ 0.231444] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.231457] CPU121: Booted secondary processor 0x00003b0100 [0x413fd0c1] [ 0.232573] Detected PIPT I-cache on CPU122 [ 0.232694] GICv3: CPU122: found redistributor 3d0000 region 0:0x0000100101080000 [ 0.232708] GICv3: CPU122: using allocated LPI pending table @0x0000080001160000 [ 0.232751] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.232765] CPU122: Booted secondary processor 0x00003d0000 [0x413fd0c1] [ 0.235169] Detected PIPT I-cache on CPU123 [ 0.235289] GICv3: CPU123: found redistributor 3d0100 region 0:0x00001001010a0000 [ 0.235304] GICv3: CPU123: using allocated LPI pending table @0x0000080001170000 [ 0.235345] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.235358] CPU123: Booted secondary processor 0x00003d0100 [0x413fd0c1] [ 0.236478] Detected PIPT I-cache on CPU124 [ 0.236596] GICv3: CPU124: found redistributor 390000 region 0:0x0000100100f80000 [ 0.236611] GICv3: CPU124: using allocated LPI pending table @0x0000080001180000 [ 0.236654] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.236668] CPU124: Booted secondary processor 0x0000390000 [0x413fd0c1] [ 0.240068] Detected PIPT I-cache on CPU125 [ 0.240187] GICv3: CPU125: found redistributor 390100 region 0:0x0000100100fa0000 [ 0.240202] GICv3: CPU125: using allocated LPI pending table @0x0000080001190000 [ 0.240243] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.240256] CPU125: Booted secondary processor 0x0000390100 [0x413fd0c1] [ 0.241361] Detected PIPT I-cache on CPU126 [ 0.241485] GICv3: CPU126: found redistributor 3f0000 region 0:0x0000100101100000 [ 0.241499] GICv3: CPU126: using allocated LPI pending table @0x00000800011a0000 [ 0.241543] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.241558] CPU126: Booted secondary processor 0x00003f0000 [0x413fd0c1] [ 0.242770] Detected PIPT I-cache on CPU127 [ 0.242882] GICv3: CPU127: found redistributor 3f0100 region 0:0x0000100101120000 [ 0.242896] GICv3: CPU127: using allocated LPI pending table @0x00000800011b0000 [ 0.242938] arch_timer: Enabling local workaround for ARM erratum 1418040 [ 0.242951] CPU127: Booted secondary processor 0x00003f0100 [0x413fd0c1] [ 0.243114] smp: Brought up 1 node, 128 CPUs [ 0.243120] SMP: Total of 128 processors activated. [ 0.243124] CPU features: detected: 32-bit EL0 Support [ 0.243126] CPU features: detected: Data cache clean to the PoU not required for I/D coherence [ 0.243129] CPU features: detected: Common not Private translations [ 0.243131] CPU features: detected: CRC32 instructions [ 0.243134] CPU features: detected: Data cache clean to Point of Persistence [ 0.243137] CPU features: detected: RCpc load-acquire (LDAPR) [ 0.243139] CPU features: detected: LSE atomic instructions [ 0.243141] CPU features: detected: Privileged Access Never [ 0.243143] CPU features: detected: RAS Extension Support [ 0.243147] CPU features: detected: Speculative Store Bypassing Safe (SSBS) [ 0.262561] CPU: All CPU(s) started at EL2 [ 0.277302] devtmpfs: initialized [ 0.282499] Registered cp15_barrier emulation handler [ 0.282534] Registered setend emulation handler [ 0.282618] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.282625] futex hash table entries: 32768 (order: 9, 2097152 bytes, linear) [ 0.283126] pinctrl core: initialized pinctrl subsystem [ 0.283292] SMBIOS 3.4.0 present. [ 0.283296] DMI: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 0.283498] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.284772] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations [ 0.284906] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285034] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.285050] audit: initializing netlink subsys (disabled) [ 0.285169] audit: type=2000 audit(0.280:1): state=initialized audit_enabled=0 res=1 [ 0.285222] thermal_sys: Registered thermal governor 'fair_share' [ 0.285223] thermal_sys: Registered thermal governor 'step_wise' [ 0.285224] thermal_sys: Registered thermal governor 'user_space' [ 0.285251] cpuidle: using governor menu [ 0.285402] Detected 15 PCC Subspaces [ 0.285403] Registering PCC driver as Mailbox controller [ 0.285473] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. [ 0.286806] ASID allocator initialised with 32768 entries [ 0.286842] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286899] Serial: AMBA PL011 UART driver [ 0.288145] KASLR enabled [ 0.299446] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299448] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page [ 0.299449] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages [ 0.299450] HugeTLB: 508 KiB vmemmap can be freed for a 32.0 MiB page [ 0.299451] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 0.299451] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page [ 0.299453] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages [ 0.299453] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page [ 0.299759] cryptd: max_cpu_qlen set to 1000 [ 0.300252] raid6: skipped pq benchmark and selected neonx8 [ 0.300254] raid6: using neon recovery algorithm [ 0.300362] ACPI: Added _OSI(Module Device) [ 0.300363] ACPI: Added _OSI(Processor Device) [ 0.300364] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.300365] ACPI: Added _OSI(Processor Aggregator Device) [ 0.300366] ACPI: Added _OSI(Linux-Dell-Video) [ 0.300367] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) [ 0.300368] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics) [ 0.306980] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.309282] ACPI: Interpreter enabled [ 0.309283] ACPI: Using GIC for interrupt routing [ 0.309292] ACPI: MCFG table detected, 8 entries [ 0.309296] ACPI: IORT: SMMU-v3[33ffe0000000] Mapped to Proximity domain 0 [ 0.309320] ACPI: IORT: SMMU-v3[37ffe0000000] Mapped to Proximity domain 0 [ 0.309333] ACPI: IORT: SMMU-v3[3bffe0000000] Mapped to Proximity domain 0 [ 0.309345] ACPI: IORT: SMMU-v3[3fffe0000000] Mapped to Proximity domain 0 [ 0.309358] ACPI: IORT: SMMU-v3[23ffe0000000] Mapped to Proximity domain 0 [ 0.309370] ACPI: IORT: SMMU-v3[27ffe0000000] Mapped to Proximity domain 0 [ 0.309383] ACPI: IORT: SMMU-v3[2bffe0000000] Mapped to Proximity domain 0 [ 0.309396] ACPI: IORT: SMMU-v3[2fffe0000000] Mapped to Proximity domain 0 [ 0.319963] ARMH0011:00: ttyAMA0 at MMIO 0x100002600000 (irq = 34, base_baud = 0) is a SBSA [ 6.834577] printk: console [ttyAMA0] enabled [ 6.839935] ARMH0011:01: ttyAMA1 at MMIO 0x100002620000 (irq = 35, base_baud = 0) is a SBSA [ 6.849230] ACPI: PCI Root Bridge [PCI1] (domain 000d [bus 00-ff]) [ 6.855405] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 6.864832] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 6.872807] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability] [ 6.879412] acpi PNP0A08:00: MCFG quirk: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 6.890721] acpi PNP0A08:00: ECAM area [mem 0x37fff0000000-0x37ffffffffff] reserved by PNP0C02:00 [ 6.899586] acpi PNP0A08:00: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] [ 6.907762] PCI host bridge to bus 000d:00 [ 6.911845] pci_bus 000d:00: root bus resource [mem 0x50000000-0x5fffffff window] [ 6.919316] pci_bus 000d:00: root bus resource [mem 0x340000000000-0x37ffdfffffff window] [ 6.927479] pci_bus 000d:00: root bus resource [bus 00-ff] [ 6.932976] pci 000d:00:00.0: [1def:e100] type 00 class 0x060000 [ 6.939033] pci 000d:00:01.0: [1def:e101] type 01 class 0x060400 [ 6.945042] pci 000d:00:01.0: enabling Extended Tags [ 6.950022] pci 000d:00:01.0: supports D1 D2 [ 6.954278] pci 000d:00:01.0: PME# supported from D0 D1 D3hot [ 6.960094] pci 000d:00:02.0: [1def:e102] type 01 class 0x060400 [ 6.966124] pci 000d:00:02.0: supports D1 D2 [ 6.970382] pci 000d:00:02.0: PME# supported from D0 D1 D3hot [ 6.976188] pci 000d:00:03.0: [1def:e103] type 01 class 0x060400 [ 6.982238] pci 000d:00:03.0: supports D1 D2 [ 6.986494] pci 000d:00:03.0: PME# supported from D0 D1 D3hot [ 6.992302] pci 000d:00:04.0: [1def:e104] type 01 class 0x060400 [ 6.998332] pci 000d:00:04.0: supports D1 D2 [ 7.002590] pci 000d:00:04.0: PME# supported from D0 D1 D3hot [ 7.008436] acpiphp: Slot [1] registered [ 7.012376] acpiphp: Slot [2] registered [ 7.016319] acpiphp: Slot [3] registered [ 7.020257] acpiphp: Slot [4] registered [ 7.024179] pci 000d:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 7.032345] pci 000d:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 7.043809] pci 000d:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 [ 7.054317] pci 000d:00:02.0: bridge window [io 0x1000-0x0fff] to [bus 02] add_size 1000 [ 7.062481] pci 000d:00:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 02] add_size 200000 add_align 100000 [ 7.073943] pci 000d:00:02.0: bridge window [mem 0x00100000-0x000fffff] to [bus 02] add_size 200000 add_align 100000 [ 7.084452] pci 000d:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 03] add_size 1000 [ 7.092614] pci 000d:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000 [ 7.104077] pci 000d:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 03] add_size 200000 add_align 100000 [ 7.114585] pci 000d:00:04.0: bridge window [io 0x1000-0x0fff] to [bus 04] add_size 1000 [ 7.122748] pci 000d:00:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000 [ 7.134219] pci 000d:00:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000 [ 7.144733] pci 000d:00:01.0: BAR 14: assigned [mem 0x50000000-0x501fffff] [ 7.151595] pci 000d:00:01.0: BAR 15: assigned [mem 0x340000000000-0x3400001fffff 64bit pref] [ 7.160107] pci 000d:00:02.0: BAR 14: assigned [mem 0x50200000-0x503fffff] [ 7.166968] pci 000d:00:02.0: BAR 15: assigned [mem 0x340000200000-0x3400003fffff 64bit pref] [ 7.175480] pci 000d:00:03.0: BAR 14: assigned [mem 0x50400000-0x505fffff] [ 7.182341] pci 000d:00:03.0: BAR 15: assigned [mem 0x340000400000-0x3400005fffff 64bit pref] [ 7.190852] pci 000d:00:04.0: BAR 14: assigned [mem 0x50600000-0x507fffff] [ 7.197714] pci 000d:00:04.0: BAR 15: assigned [mem 0x340000600000-0x3400007fffff 64bit pref] [ 7.206225] pci 000d:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.212564] pci 000d:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.219253] pci 000d:00:02.0: BAR 13: no space for [io size 0x1000] [ 7.225593] pci 000d:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 7.232282] pci 000d:00:03.0: BAR 13: no space for [io size 0x1000] [ 7.238622] pci 000d:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 7.245310] pci 000d:00:04.0: BAR 13: no space for [io size 0x1000] [ 7.251650] pci 000d:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 7.258346] pci 000d:00:04.0: BAR 13: no space for [io size 0x1000] [ 7.264686] pci 000d:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 7.271374] pci 000d:00:03.0: BAR 13: no space for [io size 0x1000] [ 7.277714] pci 000d:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 7.284404] pci 000d:00:02.0: BAR 13: no space for [io size 0x1000] [ 7.290743] pci 000d:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 7.297432] pci 000d:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.303772] pci 000d:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.310461] pci 000d:00:01.0: PCI bridge to [bus 01] [ 7.315413] pci 000d:00:01.0: bridge window [mem 0x50000000-0x501fffff] [ 7.322189] pci 000d:00:01.0: bridge window [mem 0x340000000000-0x3400001fffff 64bit pref] [ 7.330614] pci 000d:00:02.0: PCI bridge to [bus 02] [ 7.335566] pci 000d:00:02.0: bridge window [mem 0x50200000-0x503fffff] [ 7.342340] pci 000d:00:02.0: bridge window [mem 0x340000200000-0x3400003fffff 64bit pref] [ 7.350766] pci 000d:00:03.0: PCI bridge to [bus 03] [ 7.355718] pci 000d:00:03.0: bridge window [mem 0x50400000-0x505fffff] [ 7.362493] pci 000d:00:03.0: bridge window [mem 0x340000400000-0x3400005fffff 64bit pref] [ 7.370918] pci 000d:00:04.0: PCI bridge to [bus 04] [ 7.375871] pci 000d:00:04.0: bridge window [mem 0x50600000-0x507fffff] [ 7.382645] pci 000d:00:04.0: bridge window [mem 0x340000600000-0x3400007fffff 64bit pref] [ 7.391071] pci_bus 000d:00: resource 4 [mem 0x50000000-0x5fffffff window] [ 7.397932] pci_bus 000d:00: resource 5 [mem 0x340000000000-0x37ffdfffffff window] [ 7.405489] pci_bus 000d:01: resource 1 [mem 0x50000000-0x501fffff] [ 7.411742] pci_bus 000d:01: resource 2 [mem 0x340000000000-0x3400001fffff 64bit pref] [ 7.419646] pci_bus 000d:02: resource 1 [mem 0x50200000-0x503fffff] [ 7.425899] pci_bus 000d:02: resource 2 [mem 0x340000200000-0x3400003fffff 64bit pref] [ 7.433803] pci_bus 000d:03: resource 1 [mem 0x50400000-0x505fffff] [ 7.440056] pci_bus 000d:03: resource 2 [mem 0x340000400000-0x3400005fffff 64bit pref] [ 7.447960] pci_bus 000d:04: resource 1 [mem 0x50600000-0x507fffff] [ 7.454213] pci_bus 000d:04: resource 2 [mem 0x340000600000-0x3400007fffff 64bit pref] [ 7.462245] ACPI: PCI Root Bridge [PCI3] (domain 0000 [bus 00-ff]) [ 7.468416] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 7.477838] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 7.485812] acpi PNP0A08:01: _OSC: OS now controls [AER PCIeCapability] [ 7.492414] acpi PNP0A08:01: MCFG quirk: ECAM at [mem 0x3ffff0000000-0x3fffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 7.503722] acpi PNP0A08:01: ECAM area [mem 0x3ffff0000000-0x3fffffffffff] reserved by PNP0C02:00 [ 7.512588] acpi PNP0A08:01: ECAM at [mem 0x3ffff0000000-0x3fffffffffff] for [bus 00-ff] [ 7.520724] PCI host bridge to bus 0000:00 [ 7.524807] pci_bus 0000:00: root bus resource [mem 0x70000000-0x7fffffff window] [ 7.532277] pci_bus 0000:00: root bus resource [mem 0x3c0000000000-0x3fffdfffffff window] [ 7.540440] pci_bus 0000:00: root bus resource [bus 00-ff] [ 7.545934] pci 0000:00:00.0: [1def:e100] type 00 class 0x060000 [ 7.551979] pci 0000:00:01.0: [1def:e101] type 01 class 0x060400 [ 7.557987] pci 0000:00:01.0: enabling Extended Tags [ 7.562967] pci 0000:00:01.0: supports D1 D2 [ 7.567223] pci 0000:00:01.0: PME# supported from D0 D1 D3hot [ 7.573039] pci 0000:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 7.581202] pci 0000:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 7.592667] pci 0000:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 [ 7.603175] pci 0000:00:01.0: BAR 14: assigned [mem 0x70000000-0x701fffff] [ 7.610037] pci 0000:00:01.0: BAR 15: assigned [mem 0x3c0000000000-0x3c00001fffff 64bit pref] [ 7.618551] pci 0000:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.624891] pci 0000:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.631592] pci 0000:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.637932] pci 0000:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.644621] pci 0000:00:01.0: PCI bridge to [bus 01] [ 7.649573] pci 0000:00:01.0: bridge window [mem 0x70000000-0x701fffff] [ 7.656348] pci 0000:00:01.0: bridge window [mem 0x3c0000000000-0x3c00001fffff 64bit pref] [ 7.664772] pci_bus 0000:00: resource 4 [mem 0x70000000-0x7fffffff window] [ 7.671635] pci_bus 0000:00: resource 5 [mem 0x3c0000000000-0x3fffdfffffff window] [ 7.679191] pci_bus 0000:01: resource 1 [mem 0x70000000-0x701fffff] [ 7.685446] pci_bus 0000:01: resource 2 [mem 0x3c0000000000-0x3c00001fffff 64bit pref] [ 7.693378] ACPI: PCI Root Bridge [PCI7] (domain 0005 [bus 00-ff]) [ 7.699552] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 7.708970] acpi PNP0A08:02: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 7.716946] acpi PNP0A08:02: _OSC: OS now controls [AER PCIeCapability] [ 7.723547] acpi PNP0A08:02: MCFG quirk: ECAM at [mem 0x2ffff0000000-0x2fffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 7.734854] acpi PNP0A08:02: ECAM area [mem 0x2ffff0000000-0x2fffffffffff] reserved by PNP0C02:00 [ 7.743718] acpi PNP0A08:02: ECAM at [mem 0x2ffff0000000-0x2fffffffffff] for [bus 00-ff] [ 7.751854] PCI host bridge to bus 0005:00 [ 7.755937] pci_bus 0005:00: root bus resource [mem 0x30000000-0x3fffffff window] [ 7.763407] pci_bus 0005:00: root bus resource [mem 0x2c0000000000-0x2fffdfffffff window] [ 7.771570] pci_bus 0005:00: root bus resource [bus 00-ff] [ 7.777063] pci 0005:00:00.0: [1def:e100] type 00 class 0x060000 [ 7.783107] pci 0005:00:01.0: [1def:e101] type 01 class 0x060400 [ 7.789119] pci 0005:00:01.0: enabling Extended Tags [ 7.794137] pci 0005:00:01.0: supports D1 D2 [ 7.798393] pci 0005:00:01.0: PME# supported from D0 D1 D3hot [ 7.804192] pci 0005:00:03.0: [1def:e103] type 01 class 0x060400 [ 7.810225] pci 0005:00:03.0: supports D1 D2 [ 7.814483] pci 0005:00:03.0: PME# supported from D0 D1 D3hot [ 7.820309] pci 0005:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 7.828474] pci 0005:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 7.839937] pci 0005:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 [ 7.850445] pci 0005:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 02] add_size 1000 [ 7.858609] pci 0005:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 02] add_size 200000 add_align 100000 [ 7.870071] pci 0005:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 02] add_size 200000 add_align 100000 [ 7.880581] pci 0005:00:01.0: BAR 14: assigned [mem 0x30000000-0x301fffff] [ 7.887442] pci 0005:00:01.0: BAR 15: assigned [mem 0x2c0000000000-0x2c00001fffff 64bit pref] [ 7.895954] pci 0005:00:03.0: BAR 14: assigned [mem 0x30200000-0x303fffff] [ 7.902816] pci 0005:00:03.0: BAR 15: assigned [mem 0x2c0000200000-0x2c00003fffff 64bit pref] [ 7.911329] pci 0005:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.917669] pci 0005:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.924358] pci 0005:00:03.0: BAR 13: no space for [io size 0x1000] [ 7.930698] pci 0005:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 7.937400] pci 0005:00:03.0: BAR 13: no space for [io size 0x1000] [ 7.943740] pci 0005:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 7.950429] pci 0005:00:01.0: BAR 13: no space for [io size 0x1000] [ 7.956769] pci 0005:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 7.963458] pci 0005:00:01.0: PCI bridge to [bus 01] [ 7.968410] pci 0005:00:01.0: bridge window [mem 0x30000000-0x301fffff] [ 7.975186] pci 0005:00:01.0: bridge window [mem 0x2c0000000000-0x2c00001fffff 64bit pref] [ 7.983610] pci 0005:00:03.0: PCI bridge to [bus 02] [ 7.988563] pci 0005:00:03.0: bridge window [mem 0x30200000-0x303fffff] [ 7.995337] pci 0005:00:03.0: bridge window [mem 0x2c0000200000-0x2c00003fffff 64bit pref] [ 8.003764] pci_bus 0005:00: resource 4 [mem 0x30000000-0x3fffffff window] [ 8.010625] pci_bus 0005:00: resource 5 [mem 0x2c0000000000-0x2fffdfffffff window] [ 8.018182] pci_bus 0005:01: resource 1 [mem 0x30000000-0x301fffff] [ 8.024435] pci_bus 0005:01: resource 2 [mem 0x2c0000000000-0x2c00001fffff 64bit pref] [ 8.032339] pci_bus 0005:02: resource 1 [mem 0x30200000-0x303fffff] [ 8.038591] pci_bus 0005:02: resource 2 [mem 0x2c0000200000-0x2c00003fffff 64bit pref] [ 8.046544] ACPI: PCI Root Bridge [PCI5] (domain 0003 [bus 00-ff]) [ 8.052715] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 8.062137] acpi PNP0A08:03: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 8.070111] acpi PNP0A08:03: _OSC: OS now controls [AER PCIeCapability] [ 8.076713] acpi PNP0A08:03: MCFG quirk: ECAM at [mem 0x27fff0000000-0x27ffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 8.088034] acpi PNP0A08:03: ECAM area [mem 0x27fff0000000-0x27ffffffffff] reserved by PNP0C02:00 [ 8.096896] acpi PNP0A08:03: ECAM at [mem 0x27fff0000000-0x27ffffffffff] for [bus 00-ff] [ 8.105030] PCI host bridge to bus 0003:00 [ 8.109113] pci_bus 0003:00: root bus resource [mem 0x10000000-0x1fffffff window] [ 8.116583] pci_bus 0003:00: root bus resource [mem 0x240000000000-0x27ffdfffffff window] [ 8.124746] pci_bus 0003:00: root bus resource [bus 00-ff] [ 8.130242] pci 0003:00:00.0: [1def:e100] type 00 class 0x060000 [ 8.136288] pci 0003:00:01.0: [1def:e101] type 01 class 0x060400 [ 8.142297] pci 0003:00:01.0: enabling Extended Tags [ 8.147277] pci 0003:00:01.0: supports D1 D2 [ 8.151534] pci 0003:00:01.0: PME# supported from D0 D1 D3hot [ 8.157328] pci 0003:00:03.0: [1def:e103] type 01 class 0x060400 [ 8.163361] pci 0003:00:03.0: supports D1 D2 [ 8.167619] pci 0003:00:03.0: PME# supported from D0 D1 D3hot [ 8.173442] pci 0003:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 8.181607] pci 0003:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 8.193070] pci 0003:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 [ 8.203615] pci 0003:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 02] add_size 1000 [ 8.211778] pci 0003:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 02] add_size 200000 add_align 100000 [ 8.223242] pci 0003:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 02] add_size 200000 add_align 100000 [ 8.233761] pci 0003:00:01.0: BAR 14: assigned [mem 0x10000000-0x101fffff] [ 8.240623] pci 0003:00:01.0: BAR 15: assigned [mem 0x240000000000-0x2400001fffff 64bit pref] [ 8.249135] pci 0003:00:03.0: BAR 14: assigned [mem 0x10200000-0x103fffff] [ 8.255996] pci 0003:00:03.0: BAR 15: assigned [mem 0x240000200000-0x2400003fffff 64bit pref] [ 8.264507] pci 0003:00:01.0: BAR 13: no space for [io size 0x1000] [ 8.270847] pci 0003:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 8.277535] pci 0003:00:03.0: BAR 13: no space for [io size 0x1000] [ 8.283875] pci 0003:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 8.290565] pci 0003:00:03.0: BAR 13: no space for [io size 0x1000] [ 8.296905] pci 0003:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 8.303594] pci 0003:00:01.0: BAR 13: no space for [io size 0x1000] [ 8.309934] pci 0003:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 8.316627] pci 0003:00:01.0: PCI bridge to [bus 01] [ 8.321579] pci 0003:00:01.0: bridge window [mem 0x10000000-0x101fffff] [ 8.328355] pci 0003:00:01.0: bridge window [mem 0x240000000000-0x2400001fffff 64bit pref] [ 8.336780] pci 0003:00:03.0: PCI bridge to [bus 02] [ 8.341733] pci 0003:00:03.0: bridge window [mem 0x10200000-0x103fffff] [ 8.348507] pci 0003:00:03.0: bridge window [mem 0x240000200000-0x2400003fffff 64bit pref] [ 8.356933] pci_bus 0003:00: resource 4 [mem 0x10000000-0x1fffffff window] [ 8.363794] pci_bus 0003:00: resource 5 [mem 0x240000000000-0x27ffdfffffff window] [ 8.371351] pci_bus 0003:01: resource 1 [mem 0x10000000-0x101fffff] [ 8.377604] pci_bus 0003:01: resource 2 [mem 0x240000000000-0x2400001fffff 64bit pref] [ 8.385509] pci_bus 0003:02: resource 1 [mem 0x10200000-0x103fffff] [ 8.391762] pci_bus 0003:02: resource 2 [mem 0x240000200000-0x2400003fffff 64bit pref] [ 8.399705] ACPI: PCI Root Bridge [PCI0] (domain 000c [bus 00-ff]) [ 8.405877] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 8.415302] acpi PNP0A08:04: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 8.423275] acpi PNP0A08:04: _OSC: OS now controls [AER PCIeCapability] [ 8.429878] acpi PNP0A08:04: MCFG quirk: ECAM at [mem 0x33fff0000000-0x33ffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 8.441187] acpi PNP0A08:04: ECAM area [mem 0x33fff0000000-0x33ffffffffff] reserved by PNP0C02:00 [ 8.450051] acpi PNP0A08:04: ECAM at [mem 0x33fff0000000-0x33ffffffffff] for [bus 00-ff] [ 8.458188] PCI host bridge to bus 000c:00 [ 8.462271] pci_bus 000c:00: root bus resource [mem 0x40000000-0x4fffffff window] [ 8.469741] pci_bus 000c:00: root bus resource [mem 0x300000000000-0x33ffdfffffff window] [ 8.477904] pci_bus 000c:00: root bus resource [bus 00-ff] [ 8.483398] pci 000c:00:00.0: [1def:e100] type 00 class 0x060000 [ 8.489442] pci 000c:00:01.0: [1def:e101] type 01 class 0x060400 [ 8.495445] pci 000c:00:01.0: enabling Extended Tags [ 8.500414] pci 000c:00:01.0: supports D1 D2 [ 8.504670] pci 000c:00:01.0: PME# supported from D0 D1 D3hot [ 8.510448] pci 000c:00:02.0: [1def:e102] type 01 class 0x060400 [ 8.516464] pci 000c:00:02.0: supports D1 D2 [ 8.520722] pci 000c:00:02.0: PME# supported from D0 D1 D3hot [ 8.526495] pci 000c:00:03.0: [1def:e103] type 01 class 0x060400 [ 8.532526] pci 000c:00:03.0: supports D1 D2 [ 8.536782] pci 000c:00:03.0: PME# supported from D0 D1 D3hot [ 8.542574] pci 000c:00:04.0: [1def:e104] type 01 class 0x060400 [ 8.548604] pci 000c:00:04.0: supports D1 D2 [ 8.552862] pci 000c:00:04.0: PME# supported from D0 D1 D3hot [ 8.558673] pci 000c:01:00.0: [1b21:1164] type 00 class 0x010601 [ 8.564679] pci 000c:01:00.0: reg 0x10: [mem 0x40182000-0x40183fff] [ 8.570964] pci 000c:01:00.0: reg 0x24: [mem 0x40180000-0x40181fff] [ 8.577224] pci 000c:01:00.0: reg 0x30: [mem 0x40100000-0x4017ffff pref] [ 8.583916] pci 000c:01:00.0: enabling Extended Tags [ 8.588911] pci 000c:01:00.0: PME# supported from D0 D3hot D3cold [ 8.595079] pci 000c:02:00.0: [1b21:1164] type 00 class 0x010601 [ 8.601085] pci 000c:02:00.0: reg 0x10: [mem 0x40082000-0x40083fff] [ 8.607367] pci 000c:02:00.0: reg 0x24: [mem 0x40080000-0x40081fff] [ 8.613627] pci 000c:02:00.0: reg 0x30: [mem 0x40000000-0x4007ffff pref] [ 8.620319] pci 000c:02:00.0: enabling Extended Tags [ 8.625311] pci 000c:02:00.0: PME# supported from D0 D3hot D3cold [ 8.631488] pci 000c:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 8.639675] pci 000c:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 8.651139] pci 000c:00:01.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01] add_size 100000 add_align 100000 [ 8.661648] pci 000c:00:02.0: bridge window [io 0x1000-0x0fff] to [bus 02] add_size 1000 [ 8.669811] pci 000c:00:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 02] add_size 200000 add_align 100000 [ 8.681282] pci 000c:00:02.0: bridge window [mem 0x00100000-0x001fffff] to [bus 02] add_size 100000 add_align 100000 [ 8.691790] pci 000c:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 03] add_size 1000 [ 8.699953] pci 000c:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000 [ 8.711416] pci 000c:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 03] add_size 200000 add_align 100000 [ 8.721924] pci 000c:00:04.0: bridge window [io 0x1000-0x0fff] to [bus 04] add_size 1000 [ 8.730087] pci 000c:00:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000 [ 8.741550] pci 000c:00:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000 [ 8.752063] pci 000c:00:01.0: BAR 14: assigned [mem 0x40000000-0x401fffff] [ 8.758924] pci 000c:00:01.0: BAR 15: assigned [mem 0x300000000000-0x3000001fffff 64bit pref] [ 8.767436] pci 000c:00:02.0: BAR 14: assigned [mem 0x40200000-0x403fffff] [ 8.774297] pci 000c:00:02.0: BAR 15: assigned [mem 0x300000200000-0x3000003fffff 64bit pref] [ 8.782808] pci 000c:00:03.0: BAR 14: assigned [mem 0x40400000-0x405fffff] [ 8.789669] pci 000c:00:03.0: BAR 15: assigned [mem 0x300000400000-0x3000005fffff 64bit pref] [ 8.798180] pci 000c:00:04.0: BAR 14: assigned [mem 0x40600000-0x407fffff] [ 8.805041] pci 000c:00:04.0: BAR 15: assigned [mem 0x300000600000-0x3000007fffff 64bit pref] [ 8.813552] pci 000c:00:01.0: BAR 13: no space for [io size 0x1000] [ 8.819892] pci 000c:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 8.826581] pci 000c:00:02.0: BAR 13: no space for [io size 0x1000] [ 8.832920] pci 000c:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 8.839615] pci 000c:00:03.0: BAR 13: no space for [io size 0x1000] [ 8.845955] pci 000c:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 8.852643] pci 000c:00:04.0: BAR 13: no space for [io size 0x1000] [ 8.858983] pci 000c:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 8.865683] pci 000c:00:04.0: BAR 13: no space for [io size 0x1000] [ 8.872023] pci 000c:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 8.878712] pci 000c:00:03.0: BAR 13: no space for [io size 0x1000] [ 8.885051] pci 000c:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 8.891739] pci 000c:00:02.0: BAR 13: no space for [io size 0x1000] [ 8.898079] pci 000c:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 8.904767] pci 000c:00:01.0: BAR 13: no space for [io size 0x1000] [ 8.911107] pci 000c:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 8.917796] pci 000c:01:00.0: BAR 6: assigned [mem 0x40000000-0x4007ffff pref] [ 8.925004] pci 000c:01:00.0: BAR 0: assigned [mem 0x40080000-0x40081fff] [ 8.931781] pci 000c:01:00.0: BAR 5: assigned [mem 0x40082000-0x40083fff] [ 8.938557] pci 000c:00:01.0: PCI bridge to [bus 01] [ 8.943510] pci 000c:00:01.0: bridge window [mem 0x40000000-0x401fffff] [ 8.950284] pci 000c:00:01.0: bridge window [mem 0x300000000000-0x3000001fffff 64bit pref] [ 8.958709] pci 000c:02:00.0: BAR 6: assigned [mem 0x40200000-0x4027ffff pref] [ 8.965918] pci 000c:02:00.0: BAR 0: assigned [mem 0x40280000-0x40281fff] [ 8.972695] pci 000c:02:00.0: BAR 5: assigned [mem 0x40282000-0x40283fff] [ 8.979470] pci 000c:00:02.0: PCI bridge to [bus 02] [ 8.984424] pci 000c:00:02.0: bridge window [mem 0x40200000-0x403fffff] [ 8.991198] pci 000c:00:02.0: bridge window [mem 0x300000200000-0x3000003fffff 64bit pref] [ 8.999624] pci 000c:00:03.0: PCI bridge to [bus 03] [ 9.004575] pci 000c:00:03.0: bridge window [mem 0x40400000-0x405fffff] [ 9.011351] pci 000c:00:03.0: bridge window [mem 0x300000400000-0x3000005fffff 64bit pref] [ 9.019775] pci 000c:00:04.0: PCI bridge to [bus 04] [ 9.024728] pci 000c:00:04.0: bridge window [mem 0x40600000-0x407fffff] [ 9.031502] pci 000c:00:04.0: bridge window [mem 0x300000600000-0x3000007fffff 64bit pref] [ 9.039928] pci_bus 000c:00: resource 4 [mem 0x40000000-0x4fffffff window] [ 9.046789] pci_bus 000c:00: resource 5 [mem 0x300000000000-0x33ffdfffffff window] [ 9.054347] pci_bus 000c:01: resource 1 [mem 0x40000000-0x401fffff] [ 9.060600] pci_bus 000c:01: resource 2 [mem 0x300000000000-0x3000001fffff 64bit pref] [ 9.068504] pci_bus 000c:02: resource 1 [mem 0x40200000-0x403fffff] [ 9.074757] pci_bus 000c:02: resource 2 [mem 0x300000200000-0x3000003fffff 64bit pref] [ 9.082661] pci_bus 000c:03: resource 1 [mem 0x40400000-0x405fffff] [ 9.088914] pci_bus 000c:03: resource 2 [mem 0x300000400000-0x3000005fffff 64bit pref] [ 9.096818] pci_bus 000c:04: resource 1 [mem 0x40600000-0x407fffff] [ 9.103071] pci_bus 000c:04: resource 2 [mem 0x300000600000-0x3000007fffff 64bit pref] [ 9.111014] ACPI: PCI Root Bridge [PCI4] (domain 0002 [bus 00-ff]) [ 9.117186] acpi PNP0A08:05: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 9.126608] acpi PNP0A08:05: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 9.134583] acpi PNP0A08:05: _OSC: OS now controls [AER PCIeCapability] [ 9.141185] acpi PNP0A08:05: MCFG quirk: ECAM at [mem 0x23fff0000000-0x23ffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 9.152493] acpi PNP0A08:05: ECAM area [mem 0x23fff0000000-0x23ffffffffff] reserved by PNP0C02:00 [ 9.161360] acpi PNP0A08:05: ECAM at [mem 0x23fff0000000-0x23ffffffffff] for [bus 00-ff] [ 9.169497] PCI host bridge to bus 0002:00 [ 9.173580] pci_bus 0002:00: root bus resource [mem 0x08000000-0x0fffffff window] [ 9.181051] pci_bus 0002:00: root bus resource [mem 0x200000000000-0x23ffdfffffff window] [ 9.189214] pci_bus 0002:00: root bus resource [bus 00-ff] [ 9.194734] pci 0002:00:00.0: [1def:e100] type 00 class 0x060000 [ 9.200779] pci 0002:00:01.0: [1def:e101] type 01 class 0x060400 [ 9.206785] pci 0002:00:01.0: enabling Extended Tags [ 9.211759] pci 0002:00:01.0: supports D1 D2 [ 9.216015] pci 0002:00:01.0: PME# supported from D0 D1 D3hot [ 9.221800] pci 0002:00:03.0: [1def:e103] type 01 class 0x060400 [ 9.227832] pci 0002:00:03.0: supports D1 D2 [ 9.232090] pci 0002:00:03.0: PME# supported from D0 D1 D3hot [ 9.237918] pci 0002:01:00.0: [8086:1521] type 00 class 0x020000 [ 9.243928] pci 0002:01:00.0: reg 0x10: [mem 0x08020000-0x0803ffff] [ 9.250198] pci 0002:01:00.0: reg 0x18: [io 0x0020-0x003f] [ 9.255767] pci 0002:01:00.0: reg 0x1c: [mem 0x08044000-0x08047fff] [ 9.262141] pci 0002:01:00.0: PME# supported from D0 D3hot D3cold [ 9.268262] pci 0002:01:00.0: reg 0x184: [mem 0x200000060000-0x200000063fff 64bit pref] [ 9.276251] pci 0002:01:00.0: VF(n) BAR0 space: [mem 0x200000060000-0x20000007ffff 64bit pref] (contains BAR0 for 8 VFs) [ 9.287129] pci 0002:01:00.0: reg 0x190: [mem 0x200000040000-0x200000043fff 64bit pref] [ 9.295118] pci 0002:01:00.0: VF(n) BAR3 space: [mem 0x200000040000-0x20000005ffff 64bit pref] (contains BAR3 for 8 VFs) [ 9.306083] pci 0002:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0002:00:01.0 (capable of 16.000 Gb/s with 5.0 GT/s PCIe x4 link) [ 9.321091] pci 0002:01:00.1: [8086:1521] type 00 class 0x020000 [ 9.327100] pci 0002:01:00.1: reg 0x10: [mem 0x08000000-0x0801ffff] [ 9.333372] pci 0002:01:00.1: reg 0x18: [io 0x0000-0x001f] [ 9.338939] pci 0002:01:00.1: reg 0x1c: [mem 0x08040000-0x08043fff] [ 9.345311] pci 0002:01:00.1: PME# supported from D0 D3hot D3cold [ 9.351425] pci 0002:01:00.1: reg 0x184: [mem 0x200000020000-0x200000023fff 64bit pref] [ 9.359418] pci 0002:01:00.1: VF(n) BAR0 space: [mem 0x200000020000-0x20000003ffff 64bit pref] (contains BAR0 for 8 VFs) [ 9.370295] pci 0002:01:00.1: reg 0x190: [mem 0x200000000000-0x200000003fff 64bit pref] [ 9.378285] pci 0002:01:00.1: VF(n) BAR3 space: [mem 0x200000000000-0x20000001ffff 64bit pref] (contains BAR3 for 8 VFs) [ 9.389305] pci 0002:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01-02] add_size 300000 add_align 100000 [ 9.401029] pci 0002:00:01.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-02] add_size 100000 add_align 100000 [ 9.411798] pci 0002:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 03] add_size 1000 [ 9.419963] pci 0002:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000 [ 9.431427] pci 0002:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 03] add_size 200000 add_align 100000 [ 9.441936] pci 0002:00:01.0: BAR 14: assigned [mem 0x08000000-0x081fffff] [ 9.448798] pci 0002:00:01.0: BAR 15: assigned [mem 0x200000000000-0x2000002fffff 64bit pref] [ 9.457310] pci 0002:00:03.0: BAR 14: assigned [mem 0x08200000-0x083fffff] [ 9.464171] pci 0002:00:03.0: BAR 15: assigned [mem 0x200000300000-0x2000004fffff 64bit pref] [ 9.472682] pci 0002:00:01.0: BAR 13: no space for [io size 0x1000] [ 9.479022] pci 0002:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 9.485715] pci 0002:00:03.0: BAR 13: no space for [io size 0x1000] [ 9.492055] pci 0002:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 9.498750] pci 0002:00:01.0: BAR 13: no space for [io size 0x1000] [ 9.505089] pci 0002:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 9.511778] pci 0002:00:03.0: BAR 13: no space for [io size 0x1000] [ 9.518118] pci 0002:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 9.524809] pci 0002:01:00.0: BAR 0: assigned [mem 0x08000000-0x0801ffff] [ 9.531585] pci 0002:01:00.1: BAR 0: assigned [mem 0x08020000-0x0803ffff] [ 9.538364] pci 0002:01:00.0: BAR 3: assigned [mem 0x08040000-0x08043fff] [ 9.545141] pci 0002:01:00.0: BAR 7: assigned [mem 0x200000000000-0x20000001ffff 64bit pref] [ 9.553569] pci 0002:01:00.0: BAR 10: assigned [mem 0x200000020000-0x20000003ffff 64bit pref] [ 9.562083] pci 0002:01:00.1: BAR 3: assigned [mem 0x08044000-0x08047fff] [ 9.568862] pci 0002:01:00.1: BAR 7: assigned [mem 0x200000040000-0x20000005ffff 64bit pref] [ 9.577288] pci 0002:01:00.1: BAR 10: assigned [mem 0x200000060000-0x20000007ffff 64bit pref] [ 9.585803] pci 0002:01:00.0: BAR 2: no space for [io size 0x0020] [ 9.592056] pci 0002:01:00.0: BAR 2: failed to assign [io size 0x0020] [ 9.598658] pci 0002:01:00.1: BAR 2: no space for [io size 0x0020] [ 9.604910] pci 0002:01:00.1: BAR 2: failed to assign [io size 0x0020] [ 9.611513] pci 0002:01:00.0: BAR 2: no space for [io size 0x0020] [ 9.617766] pci 0002:01:00.0: BAR 2: failed to assign [io size 0x0020] [ 9.624368] pci 0002:01:00.1: BAR 2: no space for [io size 0x0020] [ 9.630621] pci 0002:01:00.1: BAR 2: failed to assign [io size 0x0020] [ 9.637227] pci 0002:00:01.0: PCI bridge to [bus 01-02] [ 9.642439] pci 0002:00:01.0: bridge window [mem 0x08000000-0x081fffff] [ 9.649214] pci 0002:00:01.0: bridge window [mem 0x200000000000-0x2000002fffff 64bit pref] [ 9.657639] pci 0002:00:03.0: PCI bridge to [bus 03] [ 9.662591] pci 0002:00:03.0: bridge window [mem 0x08200000-0x083fffff] [ 9.669365] pci 0002:00:03.0: bridge window [mem 0x200000300000-0x2000004fffff 64bit pref] [ 9.677791] pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc [ 9.686826] pci_bus 0002:00: resource 4 [mem 0x08000000-0x0fffffff window] [ 9.693689] pci_bus 0002:00: resource 5 [mem 0x200000000000-0x23ffdfffffff window] [ 9.701244] pci_bus 0002:01: resource 1 [mem 0x08000000-0x081fffff] [ 9.707499] pci_bus 0002:01: resource 2 [mem 0x200000000000-0x2000002fffff 64bit pref] [ 9.715401] pci_bus 0002:03: resource 1 [mem 0x08200000-0x083fffff] [ 9.721656] pci_bus 0002:03: resource 2 [mem 0x200000300000-0x2000004fffff 64bit pref] [ 9.729591] ACPI: PCI Root Bridge [PCI2] (domain 0001 [bus 00-ff]) [ 9.735764] acpi PNP0A08:06: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 9.745184] acpi PNP0A08:06: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 9.753160] acpi PNP0A08:06: _OSC: OS now controls [AER PCIeCapability] [ 9.759760] acpi PNP0A08:06: MCFG quirk: ECAM at [mem 0x3bfff0000000-0x3bffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 9.771067] acpi PNP0A08:06: ECAM area [mem 0x3bfff0000000-0x3bffffffffff] reserved by PNP0C02:00 [ 9.779932] acpi PNP0A08:06: ECAM at [mem 0x3bfff0000000-0x3bffffffffff] for [bus 00-ff] [ 9.788067] PCI host bridge to bus 0001:00 [ 9.792154] pci_bus 0001:00: root bus resource [mem 0x60000000-0x6fffffff window] [ 9.799624] pci_bus 0001:00: root bus resource [mem 0x380000000000-0x3bffdfffffff window] [ 9.807787] pci_bus 0001:00: root bus resource [bus 00-ff] [ 9.813280] pci 0001:00:00.0: [1def:e100] type 00 class 0x060000 [ 9.819326] pci 0001:00:01.0: [1def:e101] type 01 class 0x060400 [ 9.825334] pci 0001:00:01.0: enabling Extended Tags [ 9.830314] pci 0001:00:01.0: supports D1 D2 [ 9.834570] pci 0001:00:01.0: PME# supported from D0 D1 D3hot [ 9.840384] pci 0001:00:01.0: bridge window [io 0x1000-0x0fff] to [bus 01] add_size 1000 [ 9.848547] pci 0001:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000 [ 9.860011] pci 0001:00:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000 [ 9.870520] pci 0001:00:01.0: BAR 14: assigned [mem 0x60000000-0x601fffff] [ 9.877381] pci 0001:00:01.0: BAR 15: assigned [mem 0x380000000000-0x3800001fffff 64bit pref] [ 9.885893] pci 0001:00:01.0: BAR 13: no space for [io size 0x1000] [ 9.892233] pci 0001:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 9.898922] pci 0001:00:01.0: BAR 13: no space for [io size 0x1000] [ 9.905263] pci 0001:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 9.911952] pci 0001:00:01.0: PCI bridge to [bus 01] [ 9.916904] pci 0001:00:01.0: bridge window [mem 0x60000000-0x601fffff] [ 9.923679] pci 0001:00:01.0: bridge window [mem 0x380000000000-0x3800001fffff 64bit pref] [ 9.932104] pci_bus 0001:00: resource 4 [mem 0x60000000-0x6fffffff window] [ 9.938967] pci_bus 0001:00: resource 5 [mem 0x380000000000-0x3bffdfffffff window] [ 9.946522] pci_bus 0001:01: resource 1 [mem 0x60000000-0x601fffff] [ 9.952776] pci_bus 0001:01: resource 2 [mem 0x380000000000-0x3800001fffff 64bit pref] [ 9.960713] ACPI: PCI Root Bridge [PCI6] (domain 0004 [bus 00-ff]) [ 9.966887] acpi PNP0A08:07: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] [ 9.976305] acpi PNP0A08:07: _OSC: platform does not support [PCIeHotplug PME LTR DPC] [ 9.984280] acpi PNP0A08:07: _OSC: OS now controls [AER PCIeCapability] [ 9.990881] acpi PNP0A08:07: MCFG quirk: ECAM at [mem 0x2bfff0000000-0x2bffffffffff] for [bus 00-ff] with pci_32b_read_ops [ 10.002187] acpi PNP0A08:07: ECAM area [mem 0x2bfff0000000-0x2bffffffffff] reserved by PNP0C02:00 [ 10.011050] acpi PNP0A08:07: ECAM at [mem 0x2bfff0000000-0x2bffffffffff] for [bus 00-ff] [ 10.019186] PCI host bridge to bus 0004:00 [ 10.023268] pci_bus 0004:00: root bus resource [mem 0x20000000-0x2fffffff window] [ 10.030739] pci_bus 0004:00: root bus resource [mem 0x280000000000-0x2bffdfffffff window] [ 10.038901] pci_bus 0004:00: root bus resource [bus 00-ff] [ 10.044394] pci 0004:00:00.0: [1def:e100] type 00 class 0x060000 [ 10.050439] pci 0004:00:01.0: [1def:e101] type 01 class 0x060400 [ 10.056445] pci 0004:00:01.0: enabling Extended Tags [ 10.061419] pci 0004:00:01.0: supports D1 D2 [ 10.065676] pci 0004:00:01.0: PME# supported from D0 D1 D3hot [ 10.071460] pci 0004:00:02.0: [1def:e102] type 01 class 0x060400 [ 10.077483] pci 0004:00:02.0: supports D1 D2 [ 10.081742] pci 0004:00:02.0: PME# supported from D0 D1 D3hot [ 10.087522] pci 0004:00:03.0: [1def:e103] type 01 class 0x060400 [ 10.093556] pci 0004:00:03.0: supports D1 D2 [ 10.097813] pci 0004:00:03.0: PME# supported from D0 D1 D3hot [ 10.103604] pci 0004:00:04.0: [1def:e104] type 01 class 0x060400 [ 10.109636] pci 0004:00:04.0: supports D1 D2 [ 10.113894] pci 0004:00:04.0: PME# supported from D0 D1 D3hot [ 10.119710] pci 0004:01:00.0: [1a03:1150] type 01 class 0x060400 [ 10.125759] pci 0004:01:00.0: enabling Extended Tags [ 10.130771] pci 0004:01:00.0: supports D1 D2 [ 10.135027] pci 0004:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 10.141740] pci_bus 0004:02: extended config space not accessible [ 10.147838] pci 0004:02:00.0: [1a03:2000] type 00 class 0x030000 [ 10.153851] pci 0004:02:00.0: reg 0x10: [mem 0x20000000-0x21ffffff] [ 10.160115] pci 0004:02:00.0: reg 0x14: [mem 0x22000000-0x2201ffff] [ 10.166379] pci 0004:02:00.0: reg 0x18: [io 0x0000-0x007f] [ 10.172023] pci 0004:02:00.0: supports D1 D2 [ 10.176279] pci 0004:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 10.183005] pci 0004:03:00.0: [1912:0014] type 00 class 0x0c0330 [ 10.189019] pci 0004:03:00.0: reg 0x10: [mem 0x22200000-0x22201fff 64bit] [ 10.195903] pci 0004:03:00.0: PME# supported from D0 D3hot D3cold [ 10.202105] pci 0004:00:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01-02] add_size 200000 add_align 100000 [ 10.213829] pci 0004:00:02.0: bridge window [io 0x1000-0x0fff] to [bus 03] add_size 1000 [ 10.221992] pci 0004:00:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000 [ 10.233455] pci 0004:00:02.0: bridge window [mem 0x00100000-0x001fffff] to [bus 03] add_size 100000 add_align 100000 [ 10.243963] pci 0004:00:03.0: bridge window [io 0x1000-0x0fff] to [bus 04] add_size 1000 [ 10.252130] pci 0004:00:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000 [ 10.263593] pci 0004:00:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000 [ 10.274101] pci 0004:00:04.0: bridge window [io 0x1000-0x0fff] to [bus 05] add_size 1000 [ 10.282264] pci 0004:00:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000 [ 10.293727] pci 0004:00:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 05] add_size 200000 add_align 100000 [ 10.304238] pci 0004:00:01.0: BAR 14: assigned [mem 0x20000000-0x22ffffff] [ 10.311099] pci 0004:00:01.0: BAR 15: assigned [mem 0x280000000000-0x2800001fffff 64bit pref] [ 10.319611] pci 0004:00:02.0: BAR 14: assigned [mem 0x23000000-0x231fffff] [ 10.326472] pci 0004:00:02.0: BAR 15: assigned [mem 0x280000200000-0x2800003fffff 64bit pref] [ 10.334984] pci 0004:00:03.0: BAR 14: assigned [mem 0x23200000-0x233fffff] [ 10.341845] pci 0004:00:03.0: BAR 15: assigned [mem 0x280000400000-0x2800005fffff 64bit pref] [ 10.350356] pci 0004:00:04.0: BAR 14: assigned [mem 0x23400000-0x235fffff] [ 10.357217] pci 0004:00:04.0: BAR 15: assigned [mem 0x280000600000-0x2800007fffff 64bit pref] [ 10.365728] pci 0004:00:01.0: BAR 13: no space for [io size 0x1000] [ 10.372068] pci 0004:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 10.378756] pci 0004:00:02.0: BAR 13: no space for [io size 0x1000] [ 10.385096] pci 0004:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 10.391784] pci 0004:00:03.0: BAR 13: no space for [io size 0x1000] [ 10.398124] pci 0004:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 10.404812] pci 0004:00:04.0: BAR 13: no space for [io size 0x1000] [ 10.411152] pci 0004:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 10.417855] pci 0004:00:01.0: BAR 13: no space for [io size 0x1000] [ 10.424195] pci 0004:00:01.0: BAR 13: failed to assign [io size 0x1000] [ 10.430890] pci 0004:00:04.0: BAR 13: no space for [io size 0x1000] [ 10.437230] pci 0004:00:04.0: BAR 13: failed to assign [io size 0x1000] [ 10.443919] pci 0004:00:03.0: BAR 13: no space for [io size 0x1000] [ 10.450259] pci 0004:00:03.0: BAR 13: failed to assign [io size 0x1000] [ 10.456948] pci 0004:00:02.0: BAR 13: no space for [io size 0x1000] [ 10.463287] pci 0004:00:02.0: BAR 13: failed to assign [io size 0x1000] [ 10.469977] pci 0004:01:00.0: BAR 14: assigned [mem 0x20000000-0x22ffffff] [ 10.476838] pci 0004:01:00.0: BAR 13: no space for [io size 0x1000] [ 10.483180] pci 0004:01:00.0: BAR 13: failed to assign [io size 0x1000] [ 10.489868] pci 0004:02:00.0: BAR 0: assigned [mem 0x20000000-0x21ffffff] [ 10.496647] pci 0004:02:00.0: BAR 1: assigned [mem 0x22000000-0x2201ffff] [ 10.503424] pci 0004:02:00.0: BAR 2: no space for [io size 0x0080] [ 10.509681] pci 0004:02:00.0: BAR 2: failed to assign [io size 0x0080] [ 10.516281] pci 0004:01:00.0: PCI bridge to [bus 02] [ 10.521237] pci 0004:01:00.0: bridge window [mem 0x20000000-0x22ffffff] [ 10.528017] pci 0004:00:01.0: PCI bridge to [bus 01-02] [ 10.533234] pci 0004:00:01.0: bridge window [mem 0x20000000-0x22ffffff] [ 10.540008] pci 0004:00:01.0: bridge window [mem 0x280000000000-0x2800001fffff 64bit pref] [ 10.548456] pci 0004:03:00.0: BAR 0: assigned [mem 0x23000000-0x23001fff 64bit] [ 10.555759] pci 0004:00:02.0: PCI bridge to [bus 03] [ 10.560713] pci 0004:00:02.0: bridge window [mem 0x23000000-0x231fffff] [ 10.567487] pci 0004:00:02.0: bridge window [mem 0x280000200000-0x2800003fffff 64bit pref] [ 10.575912] pci 0004:00:03.0: PCI bridge to [bus 04] [ 10.580864] pci 0004:00:03.0: bridge window [mem 0x23200000-0x233fffff] [ 10.587639] pci 0004:00:03.0: bridge window [mem 0x280000400000-0x2800005fffff 64bit pref] [ 10.596064] pci 0004:00:04.0: PCI bridge to [bus 05] [ 10.601016] pci 0004:00:04.0: bridge window [mem 0x23400000-0x235fffff] [ 10.607790] pci 0004:00:04.0: bridge window [mem 0x280000600000-0x2800007fffff 64bit pref] [ 10.616216] pci_bus 0004:00: Some PCI device resources are unassigned, try booting with pci=realloc [ 10.625247] pci_bus 0004:00: resource 4 [mem 0x20000000-0x2fffffff window] [ 10.632109] pci_bus 0004:00: resource 5 [mem 0x280000000000-0x2bffdfffffff window] [ 10.639664] pci_bus 0004:01: resource 1 [mem 0x20000000-0x22ffffff] [ 10.645920] pci_bus 0004:01: resource 2 [mem 0x280000000000-0x2800001fffff 64bit pref] [ 10.653823] pci_bus 0004:02: resource 1 [mem 0x20000000-0x22ffffff] [ 10.660078] pci_bus 0004:03: resource 1 [mem 0x23000000-0x231fffff] [ 10.666331] pci_bus 0004:03: resource 2 [mem 0x280000200000-0x2800003fffff 64bit pref] [ 10.674235] pci_bus 0004:04: resource 1 [mem 0x23200000-0x233fffff] [ 10.680488] pci_bus 0004:04: resource 2 [mem 0x280000400000-0x2800005fffff 64bit pref] [ 10.688393] pci_bus 0004:05: resource 1 [mem 0x23400000-0x235fffff] [ 10.694646] pci_bus 0004:05: resource 2 [mem 0x280000600000-0x2800007fffff 64bit pref] [ 10.705650] iommu: Default domain type: Translated [ 10.710515] iommu: DMA domain TLB invalidation policy: lazy mode [ 10.716767] SCSI subsystem initialized [ 10.720668] ACPI: bus type USB registered [ 10.724677] usbcore: registered new interface driver usbfs [ 10.730158] usbcore: registered new interface driver hub [ 10.735620] usbcore: registered new device driver usb [ 10.742906] pps_core: LinuxPPS API ver. 1 registered [ 10.747858] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 10.756990] PTP clock support registered [ 10.761093] EDAC MC: Ver: 3.0.0 [ 10.764463] Registered efivars operations [ 10.770376] NetLabel: Initializing [ 10.773765] NetLabel: domain hash size = 128 [ 10.778108] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 10.783791] NetLabel: unlabeled traffic allowed by default [ 10.789352] mctp: management component transport protocol core [ 10.795174] NET: Registered PF_MCTP protocol family [ 10.800089] pci 0004:02:00.0: vgaarb: setting as boot VGA device [ 10.806083] pci 0004:02:00.0: vgaarb: bridge control possible [ 10.811815] pci 0004:02:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none [ 10.820154] vgaarb: loaded [ 10.823134] clocksource: Switched to clocksource arch_sys_counter [ 10.829404] VFS: Disk quotas dquot_6.6.0 [ 10.833349] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 10.840311] pnp: PnP ACPI init [ 10.843630] system 00:00: [mem 0x3bfff0000000-0x3bffffffffff window] could not be reserved [ 10.851882] system 00:00: [mem 0x3ffff0000000-0x3fffffffffff window] could not be reserved [ 10.860137] system 00:00: [mem 0x23fff0000000-0x23ffffffffff window] could not be reserved [ 10.868415] system 00:00: [mem 0x27fff0000000-0x27ffffffffff window] could not be reserved [ 10.876679] system 00:00: [mem 0x2bfff0000000-0x2bffffffffff window] could not be reserved [ 10.884931] system 00:00: [mem 0x2ffff0000000-0x2fffffffffff window] could not be reserved [ 10.893183] system 00:00: [mem 0x33fff0000000-0x33ffffffffff window] could not be reserved [ 10.901433] system 00:00: [mem 0x37fff0000000-0x37ffffffffff window] could not be reserved [ 10.910121] pnp: PnP ACPI: found 1 devices [ 10.915615] NET: Registered PF_INET protocol family [ 10.920548] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 10.930599] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, linear) [ 10.939431] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 10.947180] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 10.956409] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear) [ 10.964075] TCP: Hash tables configured (established 524288 bind 65536) [ 10.970957] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, linear) [ 10.978746] UDP hash table entries: 32768 (order: 8, 1048576 bytes, linear) [ 10.986096] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes, linear) [ 10.994032] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 10.999684] NET: Registered PF_XDP protocol family [ 11.004535] PCI: CLS 64 bytes, default 64 [ 11.008539] ACPI: bus type thunderbolt registered [ 11.008620] Trying to unpack rootfs image as initramfs... [ 11.025762] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available [ 11.033755] kvm [1]: IPA Size Limit: 48 bits [ 11.038051] kvm [1]: GICv3: no GICV resource entry [ 11.042829] kvm [1]: disabling GICv2 emulation [ 11.047272] kvm [1]: GIC system register CPU interface enabled [ 11.054539] kvm [1]: vgic interrupt IRQ9 [ 11.060048] kvm [1]: VHE mode initialized successfully [ 11.066746] Initialise system trusted keyrings [ 11.071186] Key type blacklist registered [ 11.075260] workingset: timestamp_bits=37 max_order=24 bucket_order=0 [ 11.083086] zbud: loaded [ 11.086434] integrity: Platform Keyring initialized [ 11.091303] integrity: Machine keyring initialized [ 11.104351] NET: Registered PF_ALG protocol family [ 11.109130] xor: measuring software checksum speed [ 11.114411] 8regs : 19873 MB/sec [ 11.119259] 32regs : 19678 MB/sec [ 11.124037] arm64_neon : 22959 MB/sec [ 11.128380] xor: using function: arm64_neon (22959 MB/sec) [ 11.133855] Key type asymmetric registered [ 11.137938] Asymmetric key parser 'x509' registered [ 11.262344] Freeing initrd memory: 20556K [ 11.271498] alg: self-tests for CTR-KDF (hmac(sha256)) passed [ 11.277260] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244) [ 11.284745] io scheduler mq-deadline registered [ 11.289263] io scheduler kyber registered [ 11.293299] io scheduler bfq registered [ 11.300674] atomic64_test: passed [ 11.305874] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0 [ 11.314234] ACPI: button: Power Button [PWRB] [ 11.381024] ACPI GTDT: found 1 SBSA generic Watchdog(s). [ 11.387225] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 11.394713] msm_serial: driver initialized [ 11.399008] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0 [ 11.404582] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.412797] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 65536 entries for cmdq [ 11.419888] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 32768 entries for evtq [ 11.427020] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 65536 entries for priq [ 11.434636] arm-smmu-v3 arm-smmu-v3.1.auto: option mask 0x0 [ 11.440204] arm-smmu-v3 arm-smmu-v3.1.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.448376] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 65536 entries for cmdq [ 11.455466] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 32768 entries for evtq [ 11.462586] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 65536 entries for priq [ 11.470110] arm-smmu-v3 arm-smmu-v3.2.auto: option mask 0x0 [ 11.475964] arm-smmu-v3 arm-smmu-v3.2.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.484104] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 65536 entries for cmdq [ 11.491222] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 32768 entries for evtq [ 11.498320] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 65536 entries for priq [ 11.505887] arm-smmu-v3 arm-smmu-v3.3.auto: option mask 0x0 [ 11.511455] arm-smmu-v3 arm-smmu-v3.3.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.519602] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 65536 entries for cmdq [ 11.526722] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 32768 entries for evtq [ 11.533814] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 65536 entries for priq [ 11.541374] arm-smmu-v3 arm-smmu-v3.4.auto: option mask 0x0 [ 11.546947] arm-smmu-v3 arm-smmu-v3.4.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.555120] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 65536 entries for cmdq [ 11.562210] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 32768 entries for evtq [ 11.569330] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 65536 entries for priq [ 11.576906] arm-smmu-v3 arm-smmu-v3.5.auto: option mask 0x0 [ 11.582473] arm-smmu-v3 arm-smmu-v3.5.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.590649] arm-smmu-v3 arm-smmu-v3.5.auto: allocated 65536 entries for cmdq [ 11.597743] arm-smmu-v3 arm-smmu-v3.5.auto: allocated 32768 entries for evtq [ 11.604867] arm-smmu-v3 arm-smmu-v3.5.auto: allocated 65536 entries for priq [ 11.612394] arm-smmu-v3 arm-smmu-v3.6.auto: option mask 0x0 [ 11.617967] arm-smmu-v3 arm-smmu-v3.6.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.626103] arm-smmu-v3 arm-smmu-v3.6.auto: allocated 65536 entries for cmdq [ 11.633229] arm-smmu-v3 arm-smmu-v3.6.auto: allocated 32768 entries for evtq [ 11.640322] arm-smmu-v3 arm-smmu-v3.6.auto: allocated 65536 entries for priq [ 11.647903] arm-smmu-v3 arm-smmu-v3.7.auto: option mask 0x0 [ 11.653674] arm-smmu-v3 arm-smmu-v3.7.auto: ias 48-bit, oas 48-bit (features 0x00061fff) [ 11.661812] arm-smmu-v3 arm-smmu-v3.7.auto: allocated 65536 entries for cmdq [ 11.668942] arm-smmu-v3 arm-smmu-v3.7.auto: allocated 32768 entries for evtq [ 11.676038] arm-smmu-v3 arm-smmu-v3.7.auto: allocated 65536 entries for priq [ 11.683696] ACPI: bus type drm_connector registered [ 11.701748] ahci 000c:01:00.0: Adding to iommu group 0 [ 11.707590] ahci 000c:01:00.0: SSS flag set, parallel bus scan disabled [ 11.728491] ahci 000c:01:00.0: AHCI 0001.0301 32 slots 24 ports 6 Gbps 0xffff0f impl SATA mode [ 11.737099] ahci 000c:01:00.0: flags: 64bit ncq sntf stag pm led only pio sxs deso sadm sds apst [ 11.747455] scsi host0: ahci [ 11.750625] scsi host1: ahci [ 11.753718] scsi host2: ahci [ 11.756844] scsi host3: ahci [ 11.759971] scsi host4: ahci [ 11.763100] scsi host5: ahci [ 11.766223] scsi host6: ahci [ 11.769365] scsi host7: ahci [ 11.772493] scsi host8: ahci [ 11.775611] scsi host9: ahci [ 11.778713] scsi host10: ahci [ 11.781915] scsi host11: ahci [ 11.785125] scsi host12: ahci [ 11.788341] scsi host13: ahci [ 11.791553] scsi host14: ahci [ 11.794764] scsi host15: ahci [ 11.797987] scsi host16: ahci [ 11.801195] scsi host17: ahci [ 11.804295] scsi host18: ahci [ 11.807508] scsi host19: ahci [ 11.810717] scsi host20: ahci [ 11.813813] scsi host21: ahci [ 11.817038] scsi host22: ahci [ 11.820252] scsi host23: ahci [ 11.823241] ata1: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082100 irq 108 [ 11.830711] ata2: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082180 irq 108 [ 11.838182] ata3: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082200 irq 108 [ 11.845654] ata4: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082280 irq 108 [ 11.853123] ata5: DUMMY [ 11.855556] ata6: DUMMY [ 11.857989] ata7: DUMMY [ 11.860423] ata8: DUMMY [ 11.862857] ata9: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082500 irq 108 [ 11.870327] ata10: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082580 irq 108 [ 11.877885] ata11: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082600 irq 108 [ 11.885441] ata12: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082680 irq 108 [ 11.892997] ata13: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082700 irq 108 [ 11.900555] ata14: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082780 irq 108 [ 11.908112] ata15: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082800 irq 108 [ 11.915669] ata16: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082880 irq 108 [ 11.923226] ata17: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082900 irq 108 [ 11.930782] ata18: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082980 irq 108 [ 11.938341] ata19: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082a00 irq 108 [ 11.945898] ata20: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082a80 irq 108 [ 11.953457] ata21: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082b00 irq 108 [ 11.961013] ata22: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082b80 irq 108 [ 11.968570] ata23: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082c00 irq 108 [ 11.976127] ata24: SATA max UDMA/133 abar m8192@0x40082000 port 0x40082c80 irq 108 [ 11.983823] ahci 000c:02:00.0: Adding to iommu group 1 [ 11.989618] ahci 000c:02:00.0: SSS flag set, parallel bus scan disabled [ 12.006421] ahci 000c:02:00.0: AHCI 0001.0301 32 slots 24 ports 6 Gbps 0xffff0f impl SATA mode [ 12.015028] ahci 000c:02:00.0: flags: 64bit ncq sntf stag pm led only pio sxs deso sadm sds apst [ 12.025583] scsi host24: ahci [ 12.028821] scsi host25: ahci [ 12.032035] scsi host26: ahci [ 12.035268] scsi host27: ahci [ 12.038485] scsi host28: ahci [ 12.041698] scsi host29: ahci [ 12.044917] scsi host30: ahci [ 12.048132] scsi host31: ahci [ 12.051355] scsi host32: ahci [ 12.054506] scsi host33: ahci [ 12.057718] scsi host34: ahci [ 12.060937] scsi host35: ahci [ 12.064037] scsi host36: ahci [ 12.067250] scsi host37: ahci [ 12.070466] scsi host38: ahci [ 12.073618] scsi host39: ahci [ 12.076836] scsi host40: ahci [ 12.080047] scsi host41: ahci [ 12.083337] scsi host42: ahci [ 12.086554] scsi host43: ahci [ 12.089766] scsi host44: ahci [ 12.092958] scsi host45: ahci [ 12.096172] scsi host46: ahci [ 12.099391] scsi host47: ahci [ 12.102375] ata25: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282100 irq 109 [ 12.109933] ata26: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282180 irq 109 [ 12.117491] ata27: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282200 irq 109 [ 12.125049] ata28: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282280 irq 109 [ 12.132604] ata29: DUMMY [ 12.135127] ata30: DUMMY [ 12.137646] ata31: DUMMY [ 12.140166] ata32: DUMMY [ 12.142687] ata33: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282500 irq 109 [ 12.150244] ata34: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282580 irq 109 [ 12.157804] ata35: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282600 irq 109 [ 12.165363] ata36: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282680 irq 109 [ 12.172918] ata37: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282700 irq 109 [ 12.180475] ata38: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282780 irq 109 [ 12.188032] ata39: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282800 irq 109 [ 12.195589] ata40: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282880 irq 109 [ 12.203146] ata41: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282900 irq 109 [ 12.210703] ata42: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282980 irq 109 [ 12.218260] ata43: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282a00 irq 109 [ 12.225818] ata44: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282a80 irq 109 [ 12.233375] ata45: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282b00 irq 109 [ 12.240931] ata46: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282b80 irq 109 [ 12.248488] ata47: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282c00 irq 109 [ 12.256045] ata48: SATA max UDMA/133 abar m8192@0x40282000 port 0x40282c80 irq 109 [ 12.264010] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 12.270527] ehci-pci: EHCI PCI platform driver [ 12.275066] xhci_hcd 0004:03:00.0: Adding to iommu group 2 [ 12.281080] xhci_hcd 0004:03:00.0: failed to load firmware renesas_usb_fw.mem, fallback to ROM [ 12.289713] xhci_hcd 0004:03:00.0: xHCI Host Controller [ 12.294995] xhci_hcd 0004:03:00.0: new USB bus registered, assigned bus number 1 [ 12.302398] xhci_hcd 0004:03:00.0: Zeroing 64bit base registers, expecting fault [ 12.314471] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 12.320891] ata1.00: supports DRM functions and may not be fully accessible [ 12.327841] ata1.00: ATA-11: SAMSUNG MZ7L3960HCJR-00A07, JXTC104Q, max UDMA/133 [ 12.335787] ata1.00: 1875385008 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 12.343987] ata1.00: Features: Trust NCQ-sndrcv [ 12.348789] ata1.00: supports DRM functions and may not be fully accessible [ 12.357854] ata1.00: configured for UDMA/133 [ 12.362383] scsi 0:0:0:0: Direct-Access ATA SAMSUNG MZ7L3960 104Q PQ: 0 ANSI: 5 [ 12.370645] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 12.371193] sd 0:0:0:0: [sda] 1875385008 512-byte logical blocks: (960 GB/894 GiB) [ 12.383507] sd 0:0:0:0: [sda] 4096-byte physical blocks [ 12.388725] sd 0:0:0:0: [sda] Write Protect is off [ 12.393516] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 12.402566] sd 0:0:0:0: [sda] Preferred minimum I/O size 4096 bytes [ 12.455876] xhci_hcd 0004:03:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x0000001100000410 [ 12.464284] sda: sda1 sda2 sda3 [ 12.465425] xhci_hcd 0004:03:00.0: xHCI Host Controller [ 12.470608] sd 0:0:0:0: [sda] supports TCG Opal [ 12.473465] xhci_hcd 0004:03:00.0: new USB bus registered, assigned bus number 2 [ 12.477944] sd 0:0:0:0: [sda] Attached SCSI disk [ 12.485339] xhci_hcd 0004:03:00.0: Host supports USB 3.0 SuperSpeed [ 12.496261] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.00 [ 12.504514] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 12.511722] usb usb1: Product: xHCI Host Controller [ 12.516588] usb usb1: Manufacturer: Linux 6.0.0-rc7 xhci-hcd [ 12.522233] usb usb1: SerialNumber: 0004:03:00.0 [ 12.526944] hub 1-0:1.0: USB hub found [ 12.530689] hub 1-0:1.0: 4 ports detected [ 12.534797] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 12.542888] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.00 [ 12.551141] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 12.558351] usb usb2: Product: xHCI Host Controller [ 12.563217] usb usb2: Manufacturer: Linux 6.0.0-rc7 xhci-hcd [ 12.568862] usb usb2: SerialNumber: 0004:03:00.0 [ 12.573577] hub 2-0:1.0: USB hub found [ 12.577321] hub 2-0:1.0: 4 ports detected [ 12.581516] usbcore: registered new interface driver usbserial_generic [ 12.588047] usbserial: USB Serial support registered for generic [ 12.594075] mousedev: PS/2 mouse device common for all mice [ 12.594473] ata25: SATA link down (SStatus 0 SControl 330) [ 12.600564] rtc-efi rtc-efi.0: registered as rtc0 [ 12.610280] rtc-efi rtc-efi.0: setting system clock to 2022-09-27T23:57:10 UTC (1664323030) [ 12.618828] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. [ 12.631012] device-mapper: uevent: version 1.0.3 [ 12.635708] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com [ 12.646964] ledtrig-cpu: registered to indicate activity on CPUs [ 12.653522] [drm] Initialized simpledrm 1.0.0 20200625 for simple-framebuffer.0 on minor 0 [ 12.662090] fbcon: Deferring console take-over [ 12.666527] simple-framebuffer simple-framebuffer.0: [drm] fb0: simpledrmdrmfb frame buffer device [ 12.675780] SMCCC: SOC_ID: ID = jep106:0a16:0002 Revision = 0x000000a1 [ 12.682358] hid: raw HID events driver (C) Jiri Kosina [ 12.687517] usbcore: registered new interface driver usbhid [ 12.693075] usbhid: USB HID core driver [ 12.697350] drop_monitor: Initializing network drop monitor service [ 12.704482] ata2: SATA link down (SStatus 0 SControl 330) [ 12.713597] Initializing XFRM netlink socket [ 12.717977] NET: Registered PF_INET6 protocol family [ 12.726914] Segment Routing with IPv6 [ 12.730565] RPL Segment Routing with IPv6 [ 12.734575] In-situ OAM (IOAM) with IPv6 [ 12.738503] mip6: Mobile IPv6 [ 12.741457] NET: Registered PF_PACKET protocol family [ 12.748375] registered taskstats version 1 [ 12.756806] Loading compiled-in X.509 certificates [ 12.771737] Loaded X.509 cert 'Build time autogenerated kernel key: 6bea37a56da2f694262360cc00512050644a6cfa' [ 12.785388] zswap: loaded using pool lzo/zbud [ 12.789920] page_owner is disabled [ 12.793487] Key type ._fscrypt registered [ 12.797484] Key type .fscrypt registered [ 12.801393] Key type fscrypt-provisioning registered [ 12.806699] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity=yes [ 12.813410] Key type big_key registered [ 12.817613] Key type encrypted registered [ 12.830299] integrity: Loading X.509 certificate: UEFI:db [ 12.844713] integrity: Loaded X.509 cert 'SUSE Linux Enterprise Secure Boot CA: 3d4d40cf938539024b1cfc5a12dedfe8b17e755f' [ 12.855659] integrity: Loading X.509 certificate: UEFI:db [ 12.861046] usb 1-3: new high-speed USB device number 2 using xhci_hcd [ 12.863621] integrity: Loaded X.509 cert 'Canonical Ltd. Master Certificate Authority: ad91990bc22ab1f517048c23b6655a268e345a63' [ 12.879113] integrity: Loading X.509 certificate: UEFI:db [ 12.884690] integrity: Loaded X.509 cert 'Fedora Secure Boot CA: fde32599c2d61db1bf5807335d7b20e4cd963b42' [ 12.894332] integrity: Loading X.509 certificate: UEFI:db [ 12.899726] integrity: Loaded X.509 cert 'Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53' [ 12.910756] integrity: Loading X.509 certificate: UEFI:db [ 12.916164] integrity: Loaded X.509 cert 'Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4' [ 12.926933] integrity: Loading X.509 certificate: UEFI:db [ 12.932327] integrity: Loaded X.509 cert 'Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e4f9ae17c55af53' [ 12.947813] ima: secureboot mode disabled [ 12.951814] ima: No TPM chip found, activating TPM-bypass! [ 12.957290] Loading compiled-in module X.509 certificates [ 12.963199] Loaded X.509 cert 'Build time autogenerated kernel key: 6bea37a56da2f694262360cc00512050644a6cfa' [ 12.973115] ima: Allocated hash algorithm: sha256 [ 12.977817] ima: No architecture policies found [ 12.982356] evm: Initialising EVM extended attributes: [ 12.987487] evm: security.selinux [ 12.990788] evm: security.SMACK64 (disabled) [ 12.995049] evm: security.SMACK64EXEC (disabled) [ 12.999652] evm: security.SMACK64TRANSMUTE (disabled) [ 13.004691] evm: security.SMACK64MMAP (disabled) [ 13.009294] evm: security.apparmor (disabled) [ 13.013639] evm: security.ima [ 13.016593] evm: security.capability [ 13.020156] evm: HMAC attrs: 0x1 [ 13.034432] ata3: SATA link down (SStatus 0 SControl 330) [ 13.053525] alg: No test for 842 (842-scomp) [ 13.057966] alg: No test for 842 (842-generic) [ 13.058600] usb 1-3: New USB device found, idVendor=05e3, idProduct=0610, bcdDevice=93.10 [ 13.070570] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 13.077694] usb 1-3: Product: USB2.0 Hub [ 13.081603] usb 1-3: Manufacturer: GenesysLogic [ 13.089497] hub 1-3:1.0: USB hub found [ 13.093851] hub 1-3:1.0: 4 ports detected [ 13.178998] pcieport 000d:00:01.0: Adding to iommu group 3 [ 13.185180] pcieport 000d:00:01.0: AER: enabled with IRQ 119 [ 13.190981] pcieport 000d:00:02.0: Adding to iommu group 4 [ 13.197106] pcieport 000d:00:02.0: AER: enabled with IRQ 119 [ 13.202894] pcieport 000d:00:03.0: Adding to iommu group 5 [ 13.209004] pcieport 000d:00:03.0: AER: enabled with IRQ 119 [ 13.214740] pcieport 000d:00:04.0: Adding to iommu group 6 [ 13.215385] usb 2-3: new SuperSpeed USB device number 2 using xhci_hcd [ 13.226887] pcieport 000d:00:04.0: AER: enabled with IRQ 119 [ 13.232783] pcieport 0000:00:01.0: Adding to iommu group 7 [ 13.239037] pcieport 0000:00:01.0: AER: enabled with IRQ 120 [ 13.244815] pcieport 0005:00:01.0: Adding to iommu group 8 [ 13.251018] pcieport 0005:00:01.0: AER: enabled with IRQ 121 [ 13.256805] pcieport 0005:00:03.0: Adding to iommu group 9 [ 13.261844] usb 2-3: New USB device found, idVendor=05e3, idProduct=0620, bcdDevice=93.10 [ 13.270447] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 13.277573] usb 2-3: Product: USB3.1 Hub [ 13.281483] usb 2-3: Manufacturer: GenesysLogic [ 13.286208] pcieport 0005:00:03.0: AER: enabled with IRQ 121 [ 13.292092] pcieport 0003:00:01.0: Adding to iommu group 10 [ 13.294292] hub 2-3:1.0: USB hub found [ 13.301561] pcieport 0003:00:01.0: AER: enabled with IRQ 122 [ 13.307269] hub 2-3:1.0: 4 ports detected [ 13.307355] pcieport 0003:00:03.0: Adding to iommu group 11 [ 13.317602] pcieport 0003:00:03.0: AER: enabled with IRQ 122 [ 13.323626] pcieport 000c:00:01.0: Adding to iommu group 12 [ 13.329935] pcieport 000c:00:01.0: AER: enabled with IRQ 107 [ 13.335893] pcieport 000c:00:02.0: Adding to iommu group 13 [ 13.342139] pcieport 000c:00:02.0: AER: enabled with IRQ 107 [ 13.349017] pcieport 000c:00:03.0: Adding to iommu group 14 [ 13.361802] pcieport 000c:00:03.0: AER: enabled with IRQ 107 [ 13.366824] ata4: SATA link down (SStatus 0 SControl 330) [ 13.367578] pcieport 000c:00:04.0: Adding to iommu group 15 [ 13.379086] pcieport 000c:00:04.0: AER: enabled with IRQ 107 [ 13.386017] pcieport 0002:00:01.0: Adding to iommu group 16 [ 13.392432] pcieport 0002:00:01.0: AER: enabled with IRQ 123 [ 13.398208] pcieport 0002:00:03.0: Adding to iommu group 17 [ 13.404433] pcieport 0002:00:03.0: AER: enabled with IRQ 123 [ 13.410300] pcieport 0001:00:01.0: Adding to iommu group 18 [ 13.416543] pcieport 0001:00:01.0: AER: enabled with IRQ 124 [ 13.422430] pcieport 0004:00:01.0: Adding to iommu group 19 [ 13.428674] pcieport 0004:00:01.0: AER: enabled with IRQ 110 [ 13.434490] pcieport 0004:00:02.0: Adding to iommu group 20 [ 13.440705] pcieport 0004:00:02.0: AER: enabled with IRQ 110 [ 13.446471] pcieport 0004:00:03.0: Adding to iommu group 21 [ 13.452903] pcieport 0004:00:03.0: AER: enabled with IRQ 110 [ 13.458685] pcieport 0004:00:04.0: Adding to iommu group 22 [ 13.464846] pcieport 0004:00:04.0: AER: enabled with IRQ 110 [ 13.470720] pcieport 0004:01:00.0: Adding to iommu group 23 [ 13.712534] ata9: SATA link down (SStatus 0 SControl 300) [ 14.043579] ata10: SATA link down (SStatus 0 SControl 300) [ 14.374512] ata11: SATA link down (SStatus 0 SControl 300) [ 14.707611] ata12: SATA link down (SStatus 0 SControl 300) [ 15.044948] ata13: SATA link down (SStatus 0 SControl 300) [ 15.377957] ata14: SATA link down (SStatus 0 SControl 300) [ 15.717576] ata15: SATA link down (SStatus 0 SControl 300) [ 16.055481] ata16: SATA link down (SStatus 0 SControl 300) [ 16.385221] ata17: SATA link down (SStatus 0 SControl 300) [ 16.713240] ata18: SATA link down (SStatus 0 SControl 300) [ 17.045813] ata19: SATA link down (SStatus 0 SControl 300) [ 17.374151] ata20: SATA link down (SStatus 0 SControl 300) [ 17.704450] ata21: SATA link down (SStatus 0 SControl 300) [ 18.043649] ata22: SATA link down (SStatus 0 SControl 300) [ 18.378917] ata23: SATA link down (SStatus 0 SControl 300) [ 18.719550] ata24: SATA link down (SStatus 0 SControl 300) [ 19.055292] ata26: SATA link down (SStatus 0 SControl 330) [ 19.385131] ata27: SATA link down (SStatus 0 SControl 330) [ 19.714530] ata28: SATA link down (SStatus 0 SControl 330) [ 20.048311] ata33: SATA link down (SStatus 0 SControl 300) [ 20.383694] ata34: SATA link down (SStatus 0 SControl 300) [ 20.713119] ata35: SATA link down (SStatus 0 SControl 300) [ 21.047959] ata36: SATA link down (SStatus 0 SControl 300) [ 21.386373] ata37: SATA link down (SStatus 0 SControl 300) [ 21.725702] ata38: SATA link down (SStatus 0 SControl 300) [ 22.057471] ata39: SATA link down (SStatus 0 SControl 300) [ 22.395947] ata40: SATA link down (SStatus 0 SControl 300) [ 22.734135] ata41: SATA link down (SStatus 0 SControl 300) [ 23.063139] ata42: SATA link down (SStatus 0 SControl 300) [ 23.394415] ata43: SATA link down (SStatus 0 SControl 300) [ 23.733402] ata44: SATA link down (SStatus 0 SControl 300) [ 24.069715] ata45: SATA link down (SStatus 0 SControl 300) [ 24.407178] ata46: SATA link down (SStatus 0 SControl 300) [ 24.743904] ata47: SATA link down (SStatus 0 SControl 300) [ 25.077860] ata48: SATA link down (SStatus 0 SControl 300) [ 25.086257] Freeing unused kernel memory: 9152K [ 25.206551] Checked W+X mappings: passed, no W+X pages found [ 25.212201] rodata_test: all tests were successful [ 25.216998] Run /init as init process [ 25.246222] systemd[1]: systemd 251.4-53.fc38 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) [ 25.278634] systemd[1]: Detected architecture arm64. [ 25.283609] systemd[1]: Running in initial RAM disk. Welcome to Fedora Linux 38 (Rawhide Prerelease) dracut-057-3.fc38 (Initramfs) ! [ 25.323556] systemd[1]: Hostname set to . [ 25.420077] systemd[1]: Failed to open libbpf, cgroup BPF features disabled: Operation not supported [ 25.477633] systemd[1]: Queued start job for default target initrd.target. [ 25.484794] systemd[1]: Reached target initrd-usr-fs.target - Initrd /usr File System. [ OK ] Reached target initrd-usr-…get - Initrd /usr File System. [ 25.536292] systemd[1]: Reached target local-fs.target - Local File Systems. [ OK ] Reached target local-fs.target - Local File Systems. [ 25.575104] systemd[1]: Reached target slices.target - Slice Units. [ OK ] Reached target slices.target - Slice Units. [ 25.614642] systemd[1]: Reached target swap.target - Swaps. [ OK ] Reached target swap.target - Swaps. [ 25.653290] systemd[1]: Reached target timers.target - Timer Units. [ OK ] Reached target timers.target - Timer Units. [ 25.680299] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket. [ OK ] Listening on systemd-journ…socket - Journal Audit Socket. [ 25.733598] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log). [ OK ] Listening on systemd-journ…t - Journal Socket (/dev/log). [ 25.773645] systemd[1]: Listening on systemd-journald.socket - Journal Socket. [ OK ] Listening on systemd-journald.socket - Journal Socket. [ 25.813555] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket. [ OK ] Listening on systemd-udevd….socket - udev Control Socket. [ 25.853680] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket. [ OK ] Listening on systemd-udevd…l.socket - udev Kernel Socket. [ 25.894203] systemd[1]: Reached target sockets.target - Socket Units. [ OK ] Reached target sockets.target - Socket Units. [ 25.994994] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes... Starting kmod-static-nodes…ate List of Static Device Nodes... [ 26.034303] systemd[1]: memstrack.service - Memstrack Anylazing Service was skipped because all trigger condition checks failed. [ 26.047951] systemd[1]: Starting systemd-journald.service - Journal Service... Starting systemd-journald.service - Journal Service... [ 26.083535] systemd[1]: systemd-modules-load.service - Load Kernel Modules was skipped because all trigger condition checks failed. [ 26.096278] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables... Starting systemd-sysctl.se…ce - Apply Kernel Variables... [ 26.134173] systemd[1]: Starting systemd-vconsole-setup.service - Setup Virtual Console... Starting systemd-vconsole-…ice - Setup Virtual Console... [ 26.173725] systemd[1]: Started systemd-journald.service - Journal Service. [ OK ] Started systemd-journald.service - Journal Service. [ 26.214759] audit: type=1130 audit(1664323044.100:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Finished kmod-static-nodes…reate List of Static Device Nodes. [ 26.263441] audit: type=1130 audit(1664323044.150:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Finished systemd-sysctl.service - Apply Kernel Variables. [ 26.314596] audit: type=1130 audit(1664323044.200:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Finished systemd-vconsole-…rvice - Setup Virtual Console. [ 26.363426] audit: type=1130 audit(1664323044.250:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Starting dracut-cmdline.service - dracut cmdline hook... Starting systemd-tmpfiles-…ate Static Device Nodes in /dev... Starting systemd-tmpfiles-… Volatile Files and Directories... [ OK ] Finished dracut-cmdline.service - dracut cmdline hook. [ 26.544273] audit: type=1130 audit(1664323044.430:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Finished systemd-tmpfiles-…reate Static Device Nodes in /dev. [ 26.582635] audit: type=1130 audit(1664323044.460:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Finished systemd-tmpfiles-…te Volatile Files and Directories. [ 26.634164] audit: type=1130 audit(1664323044.520:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Starting dracut-pre-udev.s…vice - dracut pre-udev hook... [ OK ] Finished dracut-pre-udev.service - dracut pre-udev hook. [ 26.793344] audit: type=1130 audit(1664323044.680:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 26.813190] audit: type=1334 audit(1664323044.700:10): prog-id=6 op=LOAD [ 26.819921] audit: type=1334 audit(1664323044.700:11): prog-id=7 op=LOAD Starting systemd-udevd.ser…ger for Device Events and Files... [ OK ] Started systemd-udevd.serv…nager for Device Events and Files. Starting systemd-udev-trig…[0m - Coldplug All udev Devices... [ OK ] Finished systemd-udev-trig…e -[ 27.155005] xgene-slimpro-hwmon APMC0D29:00: APM X-Gene SoC HW monitor driver registered Coldplug All udev Devices. [ 27.166154] sbsa-gwdt sbsa-gwdt.0: Initialized with 10s timeout @ 25000000 Hz, action=0. [ 27.179990] ast 0004:02:00.0: stream 512 already in tree [ OK ] Reached target sysinit.target - System Initialization. [ 27.306438] ast 0004:02:00.0: [drm] platform has no IO space, trying MMIO [ 27.313235] ast 0004:02:00.0: [drm] P2A bridge disabled, using default configuration [ 27.320964] ast 0004:02:00.0: [drm] AST 2500 detected [ 27.326013] ast 0004:02:00.0: [drm] Using analog VGA [ 27.330964] ast 0004:02:00.0: [drm] dram MCLK=800 Mhz type=1 bus_width=16 Starting dracut[ 27.338052] [drm] Initialized ast 0.1.0 20120228 for 0004:02:00.0 on minor 0 -initqueue.…ice - dracut initqueue hook... [ 27.351112] fbcon: Deferring console take-over [ 27.356542] ast 0004:02:00.0: [drm] fb0: astdrmfb frame buffer device Starting plymouth-start.se…[0m - Show Plymouth Boot Screen... [-1;-1f[ OK ] Started plymouth-start.ser…e - Show Plymouth Boot Screen. [ OK ] Started systemd-ask-passwo…uests to Plymouth Directory Watch. [ OK ] Reached target paths.target - Path Units. [ OK ] Reached target basic.target - Basic System. [ OK ] Found device dev-mapper-fe…ampere--mtsnow--altramax--01-root. [ OK ] Reached target initrd-root…e.target - Initrd Root Device. [ OK ] Finished dracut-initqueue.…rvice - dracut initqueue hook. [ OK ] Reached target remote-fs-p…eparation for Remote File Systems. [ OK ] Reached target remote-fs.target - Remote File Systems. Starting systemd-fsck-root…pere--mtsnow--altramax--01-root... [ OK ] Finished systemd-fsck-root…ampere--mtsnow--altramax--01-root. Mounting sysroot.mount - /sysroot... [ 28.076836] SGI XFS with ACLs, security attributes, scrub, quota, no debug enabled [ 28.094563] XFS (dm-0): Mounting V5 Filesystem [ 28.162971] XFS (dm-0): Ending clean mount [ OK ] Mounted sysroot.mount - /sysroot. [ OK ] Reached target initrd-root…get - Initrd Root File System. Starting initrd-parse-etc.…onfiguration from the Real Root... [ OK ] Finished initrd-parse-etc.… Configuration from the Real Root. [ OK ] Reached target initrd-fs.target - Initrd File Systems. [ OK ] Reached target initrd.target - Initrd Default Target. Starting dracut-pre-pivot.…acut pre-pivot and cleanup hook... [ OK ] Finished dracut-pre-pivot.…dracut pre-pivot and cleanup hook. Starting initrd-cleanup.se…ng Up and Shutting Down Daemons... [ OK ] Stopped target timers.target - Timer Units. [ OK ] Stopped dracut-pre-pivot.s…dracut pre-pivot and cleanup hook. [ OK ] Stopped target initrd.target - Initrd Default Target. [ OK ] Stopped target basic.target - Basic System. [ OK ] Stopped target initrd-root…e.target - Initrd Root Device. [ OK ] Stopped target initrd-usr-…get - Initrd /usr File System. [ OK ] Stopped target paths.target - Path Units. [ OK ] Stopped target remote-fs.target - Remote File Systems. [ OK ] Stopped target remote-fs-p…eparation for Remote File Systems. [ OK ] Stopped target slices.target - Slice Units. [ OK ] Stopped target sockets.target - Socket Units. [ OK ] Stopped target sysinit.target - System Initialization. [ OK ] Stopped target swap.target - Swaps. [ OK ] Stopped dracut-initqueue.service - dracut initqueue hook. Starting plymouth-switch-r… - Plymouth switch root service... [ OK ] Stopped systemd-sysctl.service - Apply Kernel Variables. [ OK ] Stop[ 28.705773] systemd-journald[1152]: Received SIGTERM from PID 1 (systemd). ped systemd-tmpfiles-s…te Volatile Files and Directories. [ OK ] Stopped target local-fs.target - Local File Systems. [ OK ] Stopped systemd-udev-trigg…e - Coldplug All udev Devices. Stopping systemd-udevd.ser…ger for Device Events and Files... [ OK ] Finished initrd-cleanup.se…ning Up and Shutting Down Daemons. [ OK ] Finished plymouth-switch-r…0m - Plymouth switch root service. [ OK ] Stopped systemd-udevd.serv…nager for Device Events and Files. [ OK ] Closed systemd-udevd-contr….socket - udev Control Socket. [ OK ] Closed systemd-udevd-kernel.socket - udev Kernel Socket. [ OK ] Stopped dracut-pre-udev.service - dracut pre-udev hook. [ OK ] Stopped dracut-cmdline.service - dracut cm[ 28.797648] SELinux: policy capability network_peer_controls=1 dline hook. Starting [ 28.805120] SELinux: policy capability open_perms=1 initrd-udevadm-cl…ice[ 28.812838] SELinux: policy capability extended_socket_class=1 - Cleanup udev Database... [ 28.821522] SELinux: policy capability always_check_network=0 [ OK ] Stopped [[ 28.830116] SELinux: policy capability cgroup_seclabel=1 0;1;39msystemd-tmpfiles-s…reat[ 28.838278] SELinux: policy capability nnp_nosuid_transition=1 e Static Device Nodes in /dev. [ 28.846960] SELinux: policy capability genfs_seclabel_symlinks=1 [ OK ] Stopped [0[ 28.855816] SELinux: policy capability ioctl_skip_cloexec=0 ;1;39mkmod-static-nodes.…reate List of Static Device Nodes. [ OK ] Finished initrd-udevadm-cl…rvice - Cleanup udev Database. [ OK ] Reached target initrd-switch-root.target - Switch Root. Starting initrd-switch-root.service - Switch Root... [ 29.046240] systemd[1]: Successfully loaded SELinux policy in 279.635ms. [ 29.061895] systemd[1]: RTC configured in localtime, applying delta of -240 minutes to system time. [ 29.102430] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 27.080ms. [ 29.115350] systemd[1]: systemd 251.4-53.fc38 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) [ 29.147817] systemd[1]: Detected architecture arm64. Welcome to Fedora Linux 38 (Rawhide Prerelease) ! [ 29.276627] systemd[1]: bpf-lsm: Failed to load BPF object: No such process [ 29.402442] zram: Added device: zram0 [ 29.459706] systemd[1]: /usr/lib/systemd/system/restraintd.service:8: Standard output type syslog+console is obsolete, automatically updating to journal+console. Please update your unit file, and consider removing the setting altogether. [ 29.544989] systemd[1]: initrd-switch-root.service: Deactivated successfully. [ 29.552266] systemd[1]: Stopped initrd-switch-root.service - Switch Root. [ OK ] Stopped [0;1[ 29.559490] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. ;39minitrd-switch-root.service [[ 29.571464] systemd[1]: Created slice system-getty.slice - Slice /system/getty. 0m - Switch Root. [ OK [ 29.581540] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe. ] Created slice syst[ 29.592276] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty. em-getty.slice - Slice /system[ 29.603993] systemd[1]: Created slice system-sshd\x2dkeygen.slice - Slice /system/sshd-keygen. /getty. [ OK ] Create[ 29.615726] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck. d slice system-modpr…l[ 29.627338] systemd[1]: Created slice system-systemd\x2dzram\x2dsetup.slice - Slice /system/systemd-zram-setup. ice - Slice /system/modprobe. [ 29.640330] systemd[1]: Created slice user.slice - User and Session Slice. [ OK ] Created sl[ 29.649711] systemd[1]: systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch was skipped because of a failed condition check (ConditionPathExists=!/run/plymouth/pid). ice system-seria… [ 29.670639] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch. - Slice /system/serial-getty. [[ 29.684223] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point. OK ] Created slice[ 29.699215] systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes. system-sshd\…e - [ 29.709606] systemd[1]: Stopped target initrd-switch-root.target - Switch Root. Slice /system/sshd-keygen. [ [0[ 29.719755] systemd[1]: Stopped target initrd-fs.target - Initrd File Systems. ;32m OK ] Created slice [0[ 29.729822] systemd[1]: Stopped target initrd-root-fs.target - Initrd Root File System. ;1;39msystem-syste… - Slice[ 29.740679] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes. /system/systemd-fsck. [ [ 29.752417] systemd[1]: Reached target paths.target - Path Units. OK ] Created slice [0;1;3[ 29.761351] systemd[1]: Reached target slices.target - Slice Units. 9msystem-syste… Slice /system/s[ 29.770471] systemd[1]: Reached target time-set.target - System Time Set. ystemd-zram-setup. [ OK[ 29.780104] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes. ] Created slice us[ 29.791666] systemd[1]: Listening on dm-event.socket - Device-mapper event daemon FIFOs. er.slice - User and Session S[ 29.803323] systemd[1]: Listening on lvm2-lvmpolld.socket - LVM2 poll daemon socket. lice. [ OK ] Start[ 29.814118] systemd[1]: Listening on systemd-coredump.socket - Process Core Dump Socket. ed systemd-ask-passwo…[ 29.823907] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe. Requests to Wall Directory Watch[ 29.835792] systemd[1]: Listening on systemd-oomd.socket - Userspace Out-Of-Memory (OOM) Killer Socket. . [ OK ] Set up auto[ 29.848557] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket. mount proc-sys-…rmats [ 29.859003] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket. File System Automount Point. [ [ 29.869942] systemd[1]: Listening on systemd-userdbd.socket - User Database Manager Socket. [0;32m OK ] Reached target cryptsetup.…get - Local Encrypted Volumes. [ OK ] Stopped target initrd-switch-root.target - Switch Root. [ OK ] Stopped target initrd-fs.target - Initrd File Systems. [ OK ] Stopped target initrd-root…get - Initrd Root File System. [ OK ] Reached target integrityse…Local Integrity Protected Volumes. [ OK [ 29.947385] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System... ] Reached target paths.ta[ 29.958367] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System... rget - Path Units. [ [ 29.969307] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System... OK ] Reached target [0;1;39[ 29.979931] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System... mslices.target - Slice Units. [ 29.989802] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of a failed condition check (ConditionPathExists=/etc/krb5.keytab). [ OK ] Reached ta[ 30.009576] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes... rget time-set.target [ 30.021194] systemd[1]: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... - System Time Set. [ OK[ 30.036513] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs... ] Reached target v[ 30.047700] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm... eritysetup… - Local Verity Prote[ 30.058038] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse... cted Volumes. [ OK [ 30.067509] systemd[1]: plymouth-switch-root.service: Deactivated successfully. ] Listening on dm-event[ 30.077735] systemd[1]: Stopped plymouth-switch-root.service - Plymouth switch root service. .sock… Device-mapper event dae[ 30.078869] fuse: init (API version 7.36) mon FIFOs. [ OK ] L[ 30.096047] systemd[1]: systemd-fsck-root.service: Deactivated successfully. istening on lvm2-lvmpol[ 30.105652] systemd[1]: Stopped systemd-fsck-root.service - File System Check on Root Device. ld…ket - LVM2 poll daemon [ 30.116969] systemd[1]: Stopped systemd-journald.service - Journal Service. socket. [ OK ] Listening on [0;1;[ 30.129105] systemd[1]: Starting systemd-journald.service - Journal Service... 39msystemd-cored…et - Proce[ 30.138466] systemd[1]: systemd-modules-load.service - Load Kernel Modules was skipped because all trigger condition checks failed. ss Core Dump Socket. [ [ 30.154073] systemd[1]: Starting systemd-network-generator.service - Generate network units from Kernel command line... OK ] Listening on [ 30.167566] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems... systemd-initc… initctl Compatibi[ 30.178484] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because all trigger condition checks failed. lity Named Pipe. [ OK [ 30.194008] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables... ] Listening on syste[ 30.204461] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices... md-oomd.…Out-Of-Memory (OOM) Kil[ 30.215964] systemd[1]: Started systemd-journald.service - Journal Service. ler Socket. [ OK ] Listening on systemd-udevd….socket - udev Control Socket. [ OK ] Listening on systemd-udevd…l.socket - udev Kernel Socket. [ [0;32[ 30.241145] systemd-journald[1625]: Received client request to flush runtime journal. m OK ] Listening on systemd-userd…0m - User Database Manager Socket. Mounting dev-hugepages.mount - Huge Pages File System... Mounting dev-mqueue.mount …POSIX Message Queue File System... Mounting sys-kernel-debug.… - Kernel Debug File System... Mounting sys-kernel-tracin… - Kernel Trace File System... Starting kmod-static-nodes…ate List of Static Device Nodes... Starting lvm2-monitor.serv…ng dmeventd or progress polling... Starting modprobe@configfs…m - Load Kernel Module configfs... Starting modprobe@drm.service - Load Kernel Module drm... Starting modprobe@fuse.ser…e - Load Kernel Module fuse... [ OK ] Stopped plymouth-switch-ro…0m - Plymouth switch root service. [ OK ] Stopped systemd-fsck-root.… File System Check on Root Device. [ OK ] Stopped systemd-journald.service - Journal Service. Starting systemd-journald.service - Journal Service... Starting systemd-network-g… units from Kernel command line... Starting systemd-remount-f…nt Root and Kernel File Systems... Starting systemd-sysctl.se…ce - Apply Kernel Variables... Starting systemd-udev-trig…[0m - Coldplug All udev Devices... [ OK ] Started systemd-journald.service - Journal Service. [ OK ] Mounted dev-hugepages.mount - Huge Pages File System. [ OK ] Mounted dev-mqueue.mount […- POSIX Message Queue File System. [ OK ] Mounted sys-kernel-debug.m…nt - Kernel Debug File System. [ OK ] Mounted sys-kernel-tracing…nt - Kernel Trace File System. [ OK ] Finished kmod-static-nodes…reate List of Static Device Nodes. [ OK ] Finished lvm2-monitor.serv…sing dmeventd or progress polling. [ OK ] Finished modprobe@configfs…[0m - Load Kernel Module configfs. [ OK ] Finished modprobe@drm.service - Load Kernel Module drm. [ OK ] Finished modprobe@fuse.service - Load Kernel Module fuse. [ OK ] Finished systemd-network-g…rk units from Kernel command line. [ OK ] Finished systemd-remount-f…ount Root and Kernel File Systems. [ OK ] Finished systemd-sysctl.service - Apply Kernel Variables. [ OK ] Reached target network-pre…get - Preparation for Network. Mounting sys-fs-fuse-conne… - FUSE Control File System... Mounting sys-kernel-config…ernel Configuration File System... Starting systemd-journal-f…h Journal to Persistent Storage... Starting systemd-random-se…ice - Load/Save Random Seed... Starting systemd-tmpfiles-…ate Static Device Nodes in /dev... [ OK ] Mounted sys-fs-fuse-connec…nt - FUSE Control File System. [ OK ] Mounted sys-kernel-config.… Kernel Configuration File System. [ OK ] Finished systemd-random-se…rvice[ 30.553766] zram0: detected capacity change from 0 to 16777216 - Load/Save Random Seed. [ OK ] Finished systemd-tmpfiles-…reate Static Device Nodes in /dev. [ OK ] Reached target local-fs-pr…reparation for Local File Systems. Starting systemd-udevd.ser…ger for Device Events and Files... [ OK ] Started systemd-udevd.serv…nager for Device Events and Files. [ OK ] Finished systemd-journal-f…ush Journal to Persistent Storage. Starting modprobe@configfs…m - Load Kernel Module configfs... [ OK ] Finished systemd-udev-trig…e - Coldplug All udev Devices. [ OK ] Found device dev-zram0.device - /dev/zram0. Starting systemd-zram-setu…[0m - Create swap on /dev/zram0... [ OK ] Finished modprobe@configfs…[0m - Load Kernel Module configfs. [ 30.710407] acpi-tad ACPI000E:00: Unsupported capabilities [ 30.710897] IPMI message handler: version 39.2 [ OK ] Finished systemd-zra[ 30.724766] arm_spe_pmu arm,spe-v1: profiling buffer inaccessible. Try passing "kpti=off" on the kernel command line m-setu…e - Create swap on [ 30.737021] arm_spe_pmu: probe of arm,spe-v1 failed with error -1 /dev/zram0. [ 30.739332] igb: Intel(R) Gigabit Ethernet Network Driver [ 30.752469] igb: Copyright (c) 2007-2014 Intel Corporation. Activating swap dev-zram0.…- Compressed Swap on /dev/zram0...[ 30.763562] ipmi device interface [ 30.777509] ipmi_ssif: IPMI SSIF Interface driver [ 30.779719] ipmi_si: IPMI System Interface driver [ 30.787179] ipmi_si: Unable to find any System Interface(s) [ 30.797144] ipmi_ssif i2c-AMPC0004:00: ipmi_ssif: Trying ACPI-specified SSIF interface at i2c address 0x10, adapter Synopsys DesignWare I2C adapter, slave address 0x20 [ 30.838881] igb 0002:01:00.0: Adding to iommu group 24 [ 30.838914] Adding 8388604k swap on /dev/zram0. Priority:100 extents:1 across:8388604k SSDscFS [ 30.852093] XFS (sda2): Mounting V5 Filesystem Mounting boot.mount - /boot... [ OK ] Activated swap dev-zram0.s…m - Compressed Swap on /dev/zram0. [ OK ] Reached targ[ 30.871160] XFS (sda2): Ending clean mount et swap.target - Swaps. Mounting tmp.mount - Temporary Directory /tmp... [ OK ] Mounted tmp.mount - Temporary Directory /tmp. Starting systemd-fsck@dev-… on /dev/disk/by-uuid/213E-8B82... [ OK ] Mounted boot.mount - /boot. [ OK ] Finished systemd-fsck@dev-…ck on /dev/disk/by-uuid/213E-8B82. Mounting boot-efi.mount - /boot/efi... [ 30.935628] igb 0002:01:00.0: added PHC on eth0 [ 30.940154] igb 0002:01:00.0: Intel(R) Gigabit Ethernet Network Connection [ 30.947027] igb 0002:01:00.0: eth0: (PCIe:5.0Gb/s:Width x2) d8:5e:d3:8f:a4:a4 [ 30.954233] igb 0002:01:00.0: eth0: PBA No: 106300-000 [ 30.959358] igb 0002:01:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 30.967042] igb 0002:01:00.1: Adding to iommu group 25 [ 30.995697] ipmi_si: IPMI System Interface driver [ 31.000650] ipmi_si: Unable to find any System Interface(s) [ OK ] Mounted boot-efi.mount - /boot/efi. [ OK ] Reached target local-fs.target - Local File Systems. Starting plymouth-read-wri…mouth To Write Out Runtime Data... Starting systemd-boot-upda… - Automatic Boot Loader Update... Starting systemd-tmpfiles-… Volatile Files and Directories... [ OK ] Fi[ 31.042155] igb 0002:01:00.1: added PHC on eth1 nished plymouth-read-wr[ 31.048725] igb 0002:01:00.1: Intel(R) Gigabit Ethernet Network Connection i…lymouth To Write Out Runtime[ 31.058367] igb 0002:01:00.1: eth1: (PCIe:5.0Gb/s:Width x2) d8:5e:d3:8f:a4:a5 Data. [ OK ] Fin[ 31.068347] igb 0002:01:00.1: eth1: PBA No: 106300-000 ished systemd-boot-upda…0m - Automatic Boot Loader Update. [ 31.081017] igb 0002:01:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 31.086626] igb 0002:01:00.0 eno1: renamed from eth0 [ OK ] Finished systemd-tmpfiles-…te Volatile Files and Directories. Starting auditd.service - Security Auditing Service... Starting systemd-oomd.serv…pace Out-Of-[ 31.112818] ipmi_ssif i2c-AMPC0004:00: IPMI message handler: Found new BMC (man_id: 0x003c0a, prod_id: 0x0212, dev_id: 0x20) Memory (OOM) Killer... Starting systemd-resolved.…e - Network Name Resolution... Starting systemd-userdbd.s…ice - User Database Manager... [ OK ] Started lvm-activate-fedor… fedora_ampere-mtsnow-altramax-01. [ OK ] Started systemd-userdbd.service - User Database Manager. [ OK ] Started auditd.service - Security Auditing Service. Starting systemd-update-ut…rd System Boot/Shutdown in UTMP... [ OK ] Finished systemd-update-ut…cord System Boot/Shutdown in UTMP. Mounting var-lib-nfs-rpc_p…ount - RPC Pipe File System... [ OK ] Started systemd-oomd.servi…rspace Out-Of-Memory (OOM) Killer. [ 31.233808] igb 0002:01:00.1 eno2: renamed from eth1 [ 31.300548] RPC: Registered named UNIX socket transport module. [ 31.306464] RPC: Registered udp transport module. [ 31.311155] RPC: Registered tcp transport module. [ 31.315850] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started systemd-resolved.s…ice - Network Name Resolution. [ OK ] Reached target nss-lookup.…m - Host and Network Name Lookups. [ OK ] Reached target sysinit.target - System Initialization. [ OK ] Started dnf-makecache.timer - dnf makecache --timer. [ OK ] Started systemd-tmpfiles-c… Cleanup of Temporary Directories. [ OK ] Listening on dbus.socket […- D-Bus System Message Bus Socket. [ OK ] Listening on pcscd.socket …art Card Daemon Activation Socket. [ OK ] Listening on sssd-kcm.sock…os Cache Manager responder socket. [ OK ] Reached target sockets.target - Socket Units. [ OK ] Reached target basic.target - Basic System. Starting NetworkManager.service - Network Manager... Starting chronyd.service - NTP client/server... Starting dracut-shutdown.s…tore /run/initramfs on shutdown... Starting flatpak-add-fedor…Add Fedora flatpak repositories... [ OK ] Reached target sshd-keygen.target . [ OK ] Reached target nss-user-lo…[0m - User and Group Name Lookups. Starting systemd-logind.se…ice - User Login Management... [ OK ] Mounted var-lib-nfs-rpc_pi….mount - RPC Pipe File System. [ OK ] Finished dracut-shutdown.s…estore /run/initramfs on shutdown. [ OK ] Reached target rpc_pipefs.target . Starting dbus-broker.servi… - D-Bus System Message Bus... [ OK ] Started chronyd.service - NTP client/server. Starting chrony-wait.servi…ony to synchronize system clock... [ OK ] Started dbus-broker.service - D-Bus System Message Bus. [ OK ] Started NetworkManager.service - Network Manager. [ OK ] Reached target network.target - Network. Starting NetworkManager-wa…m - Network Manager Wait Online... Starting gssproxy.service - GSSAPI Proxy Daemon... Starting sshd.service - OpenSSH server daemon... Starting systemd-hostnamed.service - Hostname Service... [ OK ] Started gssproxy.service - GSSAPI Proxy Daemon. [ OK ] Reached target nfs-client.target - NFS client services. [ OK ] Reached target remote-fs-p…eparation for Remote File Systems. [ OK ] Reached target remote-fs.target - Remote File Systems. Starting systemd-user-sess…vice - Permit User Sessions... [ OK ] Started sshd.service - OpenSSH server daemon. [ OK ] Started systemd-logind.service - User Login Management. [ OK ] Finished systemd-user-sess…ervice - Permit User Sessions. [ OK ] Created slice user-0.slice - User Slice of UID 0. Starting plymouth-quit-wai… until boot process finishes up... Starting plymouth-quit.ser… Terminate Plymouth Boot Screen... Starting user-runtime-dir@…r Runtime Directory /run/user/0... [ OK ] Finished flatpak-add-fedor…- Add Fedora flatpak repositories. [ 34.983508] igb 0002:01:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 34.998021] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 36.827548] fbcon: Taking over console [ 36.846805] Console: switching to colour frame buffer device 128x48 Fedora Linux 38 (Rawhide Prerelease) Kernel 6.0.0-rc7 on an aarch64 (ttyAMA0) ampere-mtsnow-altramax-01 login: [ 45.089728] restraintd[2274]: * Fetching recipe: http://lab-02.rhts.eng.bos.redhat.com:8000//recipes/12671085/ [ 45.182494] restraintd[2274]: * Parsing recipe [ 45.182826] restraintd[2274]: * Running recipe [ 45.182929] restraintd[2274]: ** Continuing task: 150741231 [/mnt/tests/github.com/beaker-project/beaker-core-tasks/archive/master.tar.gz/reservesys] [ 45.185391] restraintd[2274]: ** Preparing metadata [ 45.283379] restraintd[2274]: ** Refreshing peer role hostnames: Retries 0 [ 45.387852] restraintd[2274]: ** Updating env vars [ 45.387966] restraintd[2274]: *** Current Time: Tue Sep 27 23:57:45 2022 Localwatchdog at: * Disabled! * [ 45.390516] restraintd[2274]: ** Running task: 150741231 [/distribution/reservesys] [ 48.701901] Running test [R:12671085 T:150741231 - /distribution/reservesys - Kernel: 6.0.0-rc7] [ 89.462280] Running test [R:12671085 T:4 - Boot test - Kernel: 6.0.0-rc7] [ 175.669816] Running test [R:12671085 T:5 - Storage - blktests - blk - Kernel: 6.0.0-rc7] [ 177.904455] run blktests block/001 at 2022-09-27 23:59:55 [ 177.928421] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 177.937509] scsi host48: scsi_debug: version 0191 [20210520] [ 177.937509] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 177.950028] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 177.950470] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 177.958813] sd 48:0:0:0: Power-on or device reset occurred [ 177.958991] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 177.967240] scsi host49: scsi_debug: version 0191 [20210520] [ 177.967240] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 177.967588] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 177.967760] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 177.967763] scsi host50: scsi_debug: version 0191 [20210520] [ 177.967763] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 177.968071] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 177.968200] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 177.968202] scsi host51: scsi_debug: version 0191 [20210520] [ 177.968202] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 177.968476] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 177.992810] sd 48:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 177.998653] sd 49:0:0:0: Attached scsi generic sg2 type 0 [ 177.998819] sd 49:0:0:0: Power-on or device reset occurred [ 178.017843] sd 48:0:0:0: [sdb] Write Protect is off [ 178.018938] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 178.019690] sd 50:0:0:0: Attached scsi generic sg3 type 0 [ 178.019763] sd 50:0:0:0: Power-on or device reset occurred [ 178.028995] sd 49:0:0:0: [sdc] Write Protect is off [ 178.036817] sd 51:0:0:0: Power-on or device reset occurred [ 178.036848] sd 51:0:0:0: Attached scsi generic sg4 type 0 [ 178.039838] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 178.049103] sd 49:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 178.049858] sd 50:0:0:0: [sdd] Write Protect is off [ 178.068612] sd 48:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 178.069897] sd 50:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 178.076733] sd 51:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 178.099568] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 178.104961] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 178.108364] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 178.109830] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 178.109834] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 178.112907] sd 51:0:0:0: [sde] Write Protect is off [ 178.113751] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 178.132950] sd 51:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 178.239826] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 178.246086] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes [ 178.283810] sd 50:0:0:0: [sdd] Attached SCSI disk [ 178.343777] sd 48:0:0:0: [sdb] Attached SCSI disk [ 178.343788] sd 49:0:0:0: [sdc] Attached SCSI disk [ 178.463774] sd 51:0:0:0: [sde] Attached SCSI disk [ 179.303542] sd 51:0:0:0: [sde] Synchronizing SCSI cache [ 179.309137] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.317791] sd 51:0:0:0: Power-on or device reset occurred [ 179.317823] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 179.333332] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.342806] sd 51:0:0:0: [sde] Sense not available. [ 179.347682] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.353625] sd 50:0:0:0: [sdd] Synchronizing SCSI cache [ 179.357154] sd 51:0:0:0: [sde] Sense not available. [ 179.362670] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.363471] sd 48:0:0:0: [sdb] Synchronizing SCSI cache [ 179.363550] sd 49:0:0:0: [sdc] Synchronizing SCSI cache [ 179.363922] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.364164] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.364274] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 179.364402] sd 49:0:0:0: Attached scsi generic sg2 type 0 [ 179.364770] sd 49:0:0:0: Power-on or device reset occurred [ 179.364945] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.364951] sd 48:0:0:0: [sdb] Sense not available. [ 179.364964] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.364967] sd 48:0:0:0: [sdb] Sense not available. [ 179.364976] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 179.364979] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 179.364986] sd 48:0:0:0: [sdb] Write Protect is off [ 179.364996] sd 48:0:0:0: [sdb] Asking for cache data failed [ 179.364998] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 179.365013] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 179.365017] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 179.367229] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 179.367231] sd 51:0:0:0: [sde] 0-byte physical blocks [ 179.367909] sd 48:0:0:0: [sdb] Attached SCSI disk [ 179.374832] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.374836] sd 49:0:0:0: [sdc] Sense not available. [ 179.374842] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.374844] sd 49:0:0:0: [sdc] Sense not available. [ 179.374849] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 179.374851] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 179.374855] sd 49:0:0:0: [sdc] Write Protect is off [ 179.374860] sd 49:0:0:0: [sdc] Asking for cache data failed [ 179.374861] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 179.374869] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 179.374871] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 179.375584] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 179.375812] sd 50:0:0:0: Power-on or device reset occurred [ 179.377694] sd 49:0:0:0: [sdc] Attached SCSI disk [ 179.380615] sd 51:0:0:0: [sde] Write Protect is off [ 179.395908] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 179.407572] sd 50:0:0:0: [sdd] Write Protect is off [ 179.412960] sd 51:0:0:0: [sde] Asking for cache data failed [ 179.427882] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 179.428009] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 179.432755] sd 50:0:0:0: [sdd] Asking for cache data failed [ 179.442221] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 179.444510] sd 51:0:0:0: [sde] Attached SCSI disk [ 179.447096] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 179.670482] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 179.676744] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 179.684367] sd 50:0:0:0: [sdd] Attached SCSI disk [ 179.854370] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.862712] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 179.862819] sd 48:0:0:0: Power-on or device reset occurred [ 179.873631] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.883106] sd 48:0:0:0: [sdb] Sense not available. [ 179.884582] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.887984] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 179.896576] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 179.896812] sd 49:0:0:0: Power-on or device reset occurred [ 179.905606] sd 48:0:0:0: [sdb] Sense not available. [ 179.905612] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 179.905614] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 179.905618] sd 48:0:0:0: [sdb] Write Protect is off [ 179.916897] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 179.927599] sd 49:0:0:0: [sdc] Write Protect is off [ 179.932636] sd 48:0:0:0: [sdb] Asking for cache data failed [ 179.937506] sd 49:0:0:0: [sdc] Asking for cache data failed [ 179.944887] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 179.944911] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 179.949753] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 179.955318] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 179.960878] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 179.968712] sd 48:0:0:0: [sdb] Attached SCSI disk [ 179.977491] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 179.978238] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 179.982390] sd 49:0:0:0: [sdc] Attached SCSI disk [ 179.994409] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.000424] scsi 51:0:0:0: Attached scsi generic sg1 type 0 [ 180.000555] sd 51:0:0:0: Power-on or device reset occurred [ 180.005456] sd 50:0:0:0: Power-on or device reset occurred [ 180.011070] sd 50:0:0:0: Attached scsi generic sg2 type 0 [ 180.020660] sd 51:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 180.029198] sd 50:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.032002] sd 51:0:0:0: [sdd] Write Protect is off [ 180.037566] sd 50:0:0:0: [sde] Sense not available. [ 180.048565] sd 50:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.053900] sd 51:0:0:0: [sdd] Asking for cache data failed [ 180.061277] sd 50:0:0:0: [sde] Sense not available. [ 180.070751] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 180.075636] sd 50:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 180.080590] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 180.089948] sd 50:0:0:0: [sde] 0-byte physical blocks [ 180.095508] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 180.100381] sd 50:0:0:0: [sde] Write Protect is off [ 180.108649] sd 51:0:0:0: [sdd] Attached SCSI disk [ 180.139891] sd 50:0:0:0: [sde] Asking for cache data failed [ 180.145453] sd 50:0:0:0: [sde] Assuming drive cache: write through [ 180.151628] sd 50:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.162051] sd 50:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.176209] sd 50:0:0:0: [sde] Attached SCSI disk [ 180.224127] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.232482] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 180.232707] sd 49:0:0:0: Power-on or device reset occurred [ 180.243456] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.252932] sd 49:0:0:0: [sdc] Sense not available. [ 180.257849] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.267328] sd 49:0:0:0: [sdc] Sense not available. [ 180.272212] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 180.278485] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 180.283615] sd 49:0:0:0: [sdc] Write Protect is off [ 180.288499] sd 49:0:0:0: [sdc] Asking for cache data failed [ 180.294077] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 180.300361] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.310795] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.327248] sd 49:0:0:0: [sdc] Attached SCSI disk [ 180.344201] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.352555] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 180.352775] sd 48:0:0:0: Power-on or device reset occurred [ 180.363561] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.373036] sd 48:0:0:0: [sdb] Sense not available. [ 180.377936] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.387406] sd 48:0:0:0: [sdb] Sense not available. [ 180.392273] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 180.398529] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 180.403574] sd 48:0:0:0: [sdb] Write Protect is off [ 180.408441] sd 48:0:0:0: [sdb] Asking for cache data failed [ 180.414004] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 180.414381] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.420175] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.428674] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 180.428796] sd 51:0:0:0: Power-on or device reset occurred [ 180.438764] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.448873] sd 51:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 180.451076] sd 48:0:0:0: [sdb] Attached SCSI disk [ 180.459877] sd 51:0:0:0: [sdd] Write Protect is off [ 180.476835] sd 51:0:0:0: [sdd] Asking for cache data failed [ 180.482394] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 180.488568] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 180.494824] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 180.503842] sd 51:0:0:0: [sdd] Attached SCSI disk [ 180.564357] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.572954] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 180.573088] sd 50:0:0:0: Power-on or device reset occurred [ 180.583854] sd 50:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.593321] sd 50:0:0:0: [sde] Sense not available. [ 180.598242] sd 50:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.607713] sd 50:0:0:0: [sde] Sense not available. [ 180.612581] sd 50:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 180.618839] sd 50:0:0:0: [sde] 0-byte physical blocks [ 180.623883] sd 50:0:0:0: [sde] Write Protect is off [ 180.628752] sd 50:0:0:0: [sde] Asking for cache data failed [ 180.634318] sd 50:0:0:0: [sde] Assuming drive cache: write through [ 180.640508] sd 50:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.650936] sd 50:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.664955] sd 50:0:0:0: [sde] Attached SCSI disk [ 180.674439] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.683481] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 180.683772] sd 49:0:0:0: Power-on or device reset occurred [ 180.694388] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.703870] sd 49:0:0:0: [sdc] Sense not available. [ 180.708746] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.718218] sd 49:0:0:0: [sdc] Sense not available. [ 180.723086] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 180.729344] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 180.734389] sd 49:0:0:0: [sdc] Write Protect is off [ 180.739258] sd 49:0:0:0: [sdc] Asking for cache data failed [ 180.744820] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 180.750992] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.761428] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.773443] sd 49:0:0:0: [sdc] Attached SCSI disk [ 180.804349] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.812938] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 180.813201] sd 48:0:0:0: Power-on or device reset occurred [ 180.823873] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.833349] sd 48:0:0:0: [sdb] Sense not available. [ 180.838232] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.847709] sd 48:0:0:0: [sdb] Sense not available. [ 180.852587] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 180.858890] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 180.863961] sd 48:0:0:0: [sdb] Write Protect is off [ 180.864251] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.877036] sd 48:0:0:0: [sdb] Asking for cache data failed [ 180.877209] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 180.878505] sd 51:0:0:0: Power-on or device reset occurred [ 180.882597] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 180.882613] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.888550] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.893467] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.899632] sd 51:0:0:0: [sdd] Sense not available. [ 180.911936] sd 48:0:0:0: [sdb] Attached SCSI disk [ 180.919530] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 180.919532] sd 51:0:0:0: [sdd] Sense not available. [ 180.919535] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 180.959938] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 180.965001] sd 51:0:0:0: [sdd] Write Protect is off [ 180.969873] sd 51:0:0:0: [sdd] Asking for cache data failed [ 180.975437] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 180.981607] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 180.984574] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 180.992033] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 180.993803] sd 51:0:0:0: [sdd] Attached SCSI disk [ 181.000649] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 181.000785] sd 50:0:0:0: Power-on or device reset occurred [ 181.020869] sd 50:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 181.033448] sd 50:0:0:0: [sde] Write Protect is off [ 181.038327] sd 50:0:0:0: [sde] Asking for cache data failed [ 181.043894] sd 50:0:0:0: [sde] Assuming drive cache: write through [ 181.050065] sd 50:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 181.056324] sd 50:0:0:0: [sde] Optimal transfer size 524288 bytes [ 181.063612] sd 50:0:0:0: [sde] Attached SCSI disk [ 181.114169] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.122522] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 181.122709] sd 49:0:0:0: Power-on or device reset occurred [ 181.133438] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.142912] sd 49:0:0:0: [sdc] Sense not available. [ 181.147788] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.157257] sd 49:0:0:0: [sdc] Sense not available. [ 181.162126] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 181.168394] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 181.173447] sd 49:0:0:0: [sdc] Write Protect is off [ 181.178316] sd 49:0:0:0: [sdc] Asking for cache data failed [ 181.183880] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 181.190053] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.200480] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.214211] sd 49:0:0:0: [sdc] Attached SCSI disk [ 181.274476] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.283043] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 181.283060] sd 48:0:0:0: Power-on or device reset occurred [ 181.295947] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.305436] sd 48:0:0:0: [sdb] Sense not available. [ 181.310316] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.319791] sd 48:0:0:0: [sdb] Sense not available. [ 181.324667] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 181.330922] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 181.334322] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.334327] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.334800] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 181.335180] sd 50:0:0:0: Power-on or device reset occurred [ 181.336085] sd 48:0:0:0: [sdb] Write Protect is off [ 181.344556] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 181.344572] sd 51:0:0:0: Power-on or device reset occurred [ 181.345285] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.345289] sd 50:0:0:0: [sdd] Sense not available. [ 181.345320] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.345322] sd 50:0:0:0: [sdd] Sense not available. [ 181.345333] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 181.345335] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 181.345351] sd 50:0:0:0: [sdd] Write Protect is off [ 181.345362] sd 50:0:0:0: [sdd] Asking for cache data failed [ 181.345364] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 181.345374] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.345377] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.347552] sd 50:0:0:0: [sdd] Attached SCSI disk [ 181.364609] sd 51:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 181.368059] sd 48:0:0:0: [sdb] Asking for cache data failed [ 181.373417] sd 51:0:0:0: [sde] Write Protect is off [ 181.378888] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 181.393511] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.402693] sd 51:0:0:0: [sde] Asking for cache data failed [ 181.402694] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 181.402699] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 181.407573] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.413819] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes [ 181.415048] sd 51:0:0:0: [sde] Attached SCSI disk [ 181.425812] sd 48:0:0:0: [sdb] Attached SCSI disk [ 181.584369] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.593085] sd 49:0:0:0: Power-on or device reset occurred [ 181.593704] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 181.608740] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.618219] sd 49:0:0:0: [sdc] Sense not available. [ 181.623094] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.632565] sd 49:0:0:0: [sdc] Sense not available. [ 181.637437] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 181.643694] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 181.648735] sd 49:0:0:0: [sdc] Write Protect is off [ 181.653608] sd 49:0:0:0: [sdc] Asking for cache data failed [ 181.659167] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 181.665356] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.675786] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.687318] sd 49:0:0:0: [sdc] Attached SCSI disk [ 181.744648] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.753254] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 181.753352] sd 50:0:0:0: Power-on or device reset occurred [ 181.764158] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.773634] sd 50:0:0:0: [sdd] Sense not available. [ 181.778514] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.784288] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.787987] sd 50:0:0:0: [sdd] Sense not available. [ 181.796951] sd 48:0:0:0: Power-on or device reset occurred [ 181.801025] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 181.801270] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 181.807213] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.812741] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 181.818138] sd 48:0:0:0: [sdb] Sense not available. [ 181.824365] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 181.824972] scsi 51:0:0:0: Attached scsi generic sg1 type 0 [ 181.825237] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.825240] sd 51:0:0:0: [sde] Sense not available. [ 181.825249] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.825252] sd 51:0:0:0: [sde] Sense not available. [ 181.825260] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 181.825263] sd 51:0:0:0: [sde] 0-byte physical blocks [ 181.825271] sd 51:0:0:0: [sde] Write Protect is off [ 181.825281] sd 51:0:0:0: [sde] Asking for cache data failed [ 181.825284] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 181.825299] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.825304] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.826727] sd 51:0:0:0: [sde] Attached SCSI disk [ 181.827612] sd 50:0:0:0: [sdd] Write Protect is off [ 181.832659] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 181.845672] sd 48:0:0:0: [sdb] Sense not available. [ 181.851241] sd 50:0:0:0: [sdd] Asking for cache data failed [ 181.860723] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 181.865568] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 181.865578] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 181.875041] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 181.875096] sd 48:0:0:0: [sdb] Write Protect is off [ 181.879908] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 181.893156] sd 50:0:0:0: [sdd] Attached SCSI disk [ 181.896077] sd 48:0:0:0: [sdb] Asking for cache data failed [ 182.011176] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 182.017392] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.024283] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.027821] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.036422] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 182.036549] sd 49:0:0:0: Power-on or device reset occurred [ 182.047427] sd 48:0:0:0: [sdb] Attached SCSI disk [ 182.056633] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 182.069190] sd 49:0:0:0: [sdc] Write Protect is off [ 182.074074] sd 49:0:0:0: [sdc] Asking for cache data failed [ 182.079635] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 182.085821] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 182.092077] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 182.101440] sd 49:0:0:0: [sdc] Attached SCSI disk [ 182.134406] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.142919] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 182.143080] sd 51:0:0:0: Power-on or device reset occurred [ 182.153843] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.163319] sd 51:0:0:0: [sde] Sense not available. [ 182.168201] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.177669] sd 51:0:0:0: [sde] Sense not available. [ 182.182536] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 182.188815] sd 51:0:0:0: [sde] 0-byte physical blocks [ 182.193879] sd 51:0:0:0: [sde] Write Protect is off [ 182.198747] sd 51:0:0:0: [sde] Asking for cache data failed [ 182.204315] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 182.210491] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0[-- MARK -- Wed Sep 28 04:00:00 2022] bytes) [ 182.220915] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.232427] sd 51:0:0:0: [sde] Attached SCSI disk [ 182.254476] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.263210] sd 50:0:0:0: Power-on or device reset occurred [ 182.264133] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 182.278759] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.288262] sd 50:0:0:0: [sdd] Sense not available. [ 182.293164] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.302639] sd 50:0:0:0: [sdd] Sense not available. [ 182.307968] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 182.314240] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 182.319290] sd 50:0:0:0: [sdd] Write Protect is off [ 182.324201] sd 50:0:0:0: [sdd] Asking for cache data failed [ 182.329761] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 182.336050] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.346492] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.361019] sd 50:0:0:0: [sdd] Attached SCSI disk [ 182.384385] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.392987] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 182.393127] sd 48:0:0:0: Power-on or device reset occurred [ 182.403891] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.413367] sd 48:0:0:0: [sdb] Sense not available. [ 182.418284] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.424306] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.427760] sd 48:0:0:0: [sdb] Sense not available. [ 182.436342] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 182.436634] sd 49:0:0:0: Power-on or device reset occurred [ 182.440801] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 182.446771] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.451652] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 182.457909] sd 49:0:0:0: [sdc] Sense not available. [ 182.457917] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.467655] sd 48:0:0:0: [sdb] Write Protect is off [ 182.472422] sd 49:0:0:0: [sdc] Sense not available. [ 182.486761] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 182.491635] sd 48:0:0:0: [sdb] Asking for cache data failed [ 182.496493] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 182.502744] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 182.508316] sd 49:0:0:0: [sdc] Write Protect is off [ 182.513357] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.524382] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.526999] sd 48:0:0:0: [sdb] Attached SCSI disk [ 182.534822] sd 49:0:0:0: [sdc] Asking for cache data failed [ 182.555314] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 182.561497] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.564207] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.571924] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.580526] sd 51:0:0:0: Power-on or device reset occurred [ 182.580904] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 182.591626] sd 49:0:0:0: [sdc] Attached SCSI disk [ 182.595859] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.615378] sd 51:0:0:0: [sde] Sense not available. [ 182.620272] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.629748] sd 51:0:0:0: [sde] Sense not available. [ 182.634652] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 182.640907] sd 51:0:0:0: [sde] 0-byte physical blocks [ 182.644297] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.646000] sd 51:0:0:0: [sde] Write Protect is off [ 182.654693] sd 50:0:0:0: Power-on or device reset occurred [ 182.663558] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 182.666679] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.669973] sd 51:0:0:0: [sde] Asking for cache data failed [ 182.679323] sd 50:0:0:0: [sdd] Sense not available. [ 182.679332] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.679334] sd 50:0:0:0: [sdd] Sense not available. [ 182.684897] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 182.685045] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.689767] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 182.699234] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.703555] sd 51:0:0:0: [sde] Attached SCSI disk [ 182.704112] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 182.747010] sd 50:0:0:0: [sdd] Write Protect is off [ 182.751885] sd 50:0:0:0: [sdd] Asking for cache data failed [ 182.757452] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 182.763839] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.774266] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.785802] sd 50:0:0:0: [sdd] Attached SCSI disk [ 182.814377] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.822976] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 182.823110] sd 48:0:0:0: Power-on or device reset occurred [ 182.833867] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.843336] sd 48:0:0:0: [sdb] Sense not available. [ 182.848211] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.854337] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 182.857681] sd 48:0:0:0: [sdb] Sense not available. [ 182.866288] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 182.866407] sd 49:0:0:0: Power-on or device reset occurred [ 182.870712] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 182.876478] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.881598] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 182.881602] sd 48:0:0:0: [sdb] Write Protect is off [ 182.887863] sd 49:0:0:0: [sdc] Sense not available. [ 182.902374] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 182.907232] sd 48:0:0:0: [sdb] Asking for cache data failed [ 182.907233] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 182.907239] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 182.907241] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 182.908477] sd 48:0:0:0: [sdb] Attached SCSI disk [ 182.912134] sd 49:0:0:0: [sdc] Sense not available. [ 182.963606] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 182.969862] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 182.975030] sd 49:0:0:0: [sdc] Write Protect is off [ 182.979905] sd 49:0:0:0: [sdc] Asking for cache data failed [ 182.985471] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 182.991661] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.002090] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.014058] sd 49:0:0:0: [sdc] Attached SCSI disk [ 183.024450] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.033043] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 183.033179] sd 51:0:0:0: Power-on or device reset occurred [ 183.043956] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.053432] sd 51:0:0:0: [sde] Sense not available. [ 183.058308] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.064323] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.067788] sd 51:0:0:0: [sde] Sense not available. [ 183.076376] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 183.076510] sd 50:0:0:0: Power-on or device reset occurred [ 183.080841] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 183.086600] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.091677] sd 51:0:0:0: [sde] 0-byte physical blocks [ 183.097947] sd 50:0:0:0: [sdd] Sense not available. [ 183.107541] sd 51:0:0:0: [sde] Write Protect is off [ 183.112494] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.122175] sd 50:0:0:0: [sdd] Sense not available. [ 183.131668] sd 51:0:0:0: [sde] Asking for cache data failed [ 183.136645] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 183.142075] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 183.148337] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 183.154526] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.159556] sd 50:0:0:0: [sdd] Write Protect is off [ 183.169967] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.185265] sd 50:0:0:0: [sdd] Asking for cache data failed [ 183.186406] sd 51:0:0:0: [sde] Attached SCSI disk [ 183.190827] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 183.201704] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.212130] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.214219] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.223736] sd 50:0:0:0: [sdd] Attached SCSI disk [ 183.230730] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 183.231156] sd 48:0:0:0: Power-on or device reset occurred [ 183.251241] sd 48:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 183.258647] sd 48:0:0:0: [sdb] Write Protect is off [ 183.263524] sd 48:0:0:0: [sdb] Asking for cache data failed [ 183.269083] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 183.275262] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 183.281517] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 183.289607] sd 48:0:0:0: [sdb] Attached SCSI disk [ 183.294495] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.303210] sd 49:0:0:0: Power-on or device reset occurred [ 183.303661] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 183.318758] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.328240] sd 49:0:0:0: [sdc] Sense not available. [ 183.333117] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.342590] sd 49:0:0:0: [sdc] Sense not available. [ 183.347481] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 183.353748] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 183.358791] sd 49:0:0:0: [sdc] Write Protect is off [ 183.363688] sd 49:0:0:0: [sdc] Asking for cache data failed [ 183.369250] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 183.375552] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.385984] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.398011] sd 49:0:0:0: [sdc] Attached SCSI disk [ 183.484266] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.492845] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 183.493057] sd 51:0:0:0: Power-on or device reset occurred [ 183.503760] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.513236] sd 51:0:0:0: [sde] Sense not available. [ 183.518116] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.527585] sd 51:0:0:0: [sde] Sense not available. [ 183.532454] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 183.534420] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.538717] sd 51:0:0:0: [sde] 0-byte physical blocks [ 183.547323] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 183.547435] sd 50:0:0:0: Power-on or device reset occurred [ 183.551944] sd 51:0:0:0: [sde] Write Protect is off [ 183.567513] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 183.575032] sd 51:0:0:0: [sde] Asking for cache data failed [ 183.575160] sd 50:0:0:0: [sdd] Write Protect is off [ 183.580591] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 183.580596] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.591626] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.594068] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.594790] sd 48:0:0:0: Power-on or device reset occurred [ 183.594878] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 183.602058] sd 50:0:0:0: [sdd] Asking for cache data failed [ 183.602061] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 183.602073] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 183.604892] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.604900] sd 48:0:0:0: [sdb] Sense not available. [ 183.604949] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.604952] sd 48:0:0:0: [sdb] Sense not available. [ 183.604962] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 183.604966] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 183.604985] sd 48:0:0:0: [sdb] Write Protect is off [ 183.605007] sd 48:0:0:0: [sdb] Asking for cache data failed [ 183.605010] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 183.605060] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.605064] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.608572] sd 48:0:0:0: [sdb] Attached SCSI disk [ 183.613625] sd 51:0:0:0: [sde] Attached SCSI disk [ 183.620502] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 183.743633] sd 50:0:0:0: [sdd] Attached SCSI disk [ 183.804303] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 183.812893] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 183.813060] sd 49:0:0:0: Power-on or device reset occurred [ 183.823818] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.833288] sd 49:0:0:0: [sdc] Sense not available. [ 183.838185] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 183.847662] sd 49:0:0:0: [sdc] Sense not available. [ 183.852533] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 183.858802] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 183.863885] sd 49:0:0:0: [sdc] Write Protect is off [ 183.868766] sd 49:0:0:0: [sdc] Asking for cache data failed [ 183.874341] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 183.880565] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 183.890997] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 183.903304] sd 49:0:0:0: [sdc] Attached SCSI disk [ 184.054171] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.054359] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.062521] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 184.062715] sd 48:0:0:0: Power-on or device reset occurred [ 184.071209] sd 51:0:0:0: Power-on or device reset occurred [ 184.074750] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.075227] sd 50:0:0:0: Power-on or device reset occurred [ 184.076018] sd 51:0:0:0: Attached scsi generic sg2 type 0 [ 184.082795] sd 48:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 184.087000] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 184.091456] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.091463] sd 51:0:0:0: [sde] Sense not available. [ 184.091478] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.091481] sd 51:0:0:0: [sde] Sense not available. [ 184.091492] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 184.091495] sd 51:0:0:0: [sde] 0-byte physical blocks [ 184.091504] sd 51:0:0:0: [sde] Write Protect is off [ 184.091514] sd 51:0:0:0: [sde] Asking for cache data failed [ 184.091517] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 184.091533] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.091537] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.094392] sd 51:0:0:0: [sde] Attached SCSI disk [ 184.095029] sd 48:0:0:0: [sdb] Write Protect is off [ 184.095313] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 184.105340] sd 50:0:0:0: [sdd] Write Protect is off [ 184.118666] sd 48:0:0:0: [sdb] Asking for cache data failed [ 184.128124] sd 50:0:0:0: [sdd] Asking for cache data failed [ 184.128125] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 184.128130] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 184.128132] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 184.129367] sd 50:0:0:0: [sdd] Attached SCSI disk [ 184.133000] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 184.134289] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.134867] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 184.134981] sd 49:0:0:0: Power-on or device reset occurred [ 184.145051] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.147458] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 184.153602] sd 49:0:0:0: [sdc] Sense not available. [ 184.153629] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.158644] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 184.163540] sd 49:0:0:0: [sdc] Sense not available. [ 184.171063] sd 48:0:0:0: [sdb] Attached SCSI disk [ 184.175326] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 184.329220] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 184.334325] sd 49:0:0:0: [sdc] Write Protect is off [ 184.339198] sd 49:0:0:0: [sdc] Asking for cache data failed [ 184.344761] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 184.350933] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.361359] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.372851] sd 49:0:0:0: [sdc] Attached SCSI disk [ 184.474302] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.482886] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 184.482983] sd 51:0:0:0: Power-on or device reset occurred [ 184.493813] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.503283] sd 51:0:0:0: [sde] Sense not available. [ 184.504240] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.508271] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.516705] sd 50:0:0:0: Power-on or device reset occurred [ 184.523515] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 184.525798] sd 51:0:0:0: [sde] Sense not available. [ 184.531317] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.534575] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.534942] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 184.535215] sd 48:0:0:0: Power-on or device reset occurred [ 184.536835] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 184.541543] sd 50:0:0:0: [sdd] Sense not available. [ 184.545285] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.545289] sd 48:0:0:0: [sdb] Sense not available. [ 184.545303] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.545307] sd 48:0:0:0: [sdb] Sense not available. [ 184.545316] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 184.545319] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 184.545327] sd 48:0:0:0: [sdb] Write Protect is off [ 184.545338] sd 48:0:0:0: [sdb] Asking for cache data failed [ 184.545341] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 184.545356] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.545360] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.547647] sd 48:0:0:0: [sdb] Attached SCSI disk [ 184.551017] sd 51:0:0:0: [sde] 0-byte physical blocks [ 184.559186] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.564588] sd 51:0:0:0: [sde] Write Protect is off [ 184.570043] sd 50:0:0:0: [sdd] Sense not available. [ 184.570049] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 184.581170] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 184.590653] sd 51:0:0:0: [sde] Asking for cache data failed [ 184.595519] sd 50:0:0:0: [sdd] Write Protect is off [ 184.604978] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 184.604991] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.614346] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 184.614754] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 184.614788] sd 49:0:0:0: Power-on or device reset occurred [ 184.616110] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.621232] sd 50:0:0:0: [sdd] Asking for cache data failed [ 184.624837] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.624840] sd 49:0:0:0: [sdc] Sense not available. [ 184.624847] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 184.624848] sd 49:0:0:0: [sdc] Sense not available. [ 184.624853] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 184.624855] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 184.624858] sd 49:0:0:0: [sdc] Write Protect is off [ 184.624863] sd 49:0:0:0: [sdc] Asking for cache data failed [ 184.624864] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 184.624871] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.624873] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.626515] sd 49:0:0:0: [sdc] Attached SCSI disk [ 184.629757] sd 51:0:0:0: [sde] Attached SCSI disk [ 184.631580] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 184.853279] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 184.863707] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 184.875205] sd 50:0:0:0: [sdd] Attached SCSI disk [ 184.994463] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.003137] sd 48:0:0:0: Power-on or device reset occurred [ 185.003524] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 185.018698] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.028201] sd 48:0:0:0: [sdb] Sense not available. [ 185.033166] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.042643] sd 48:0:0:0: [sdb] Sense not available. [ 185.047525] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 185.053787] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 185.058833] sd 48:0:0:0: [sdb] Write Protect is off [ 185.064444] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.064466] sd 48:0:0:0: [sdb] Asking for cache data failed [ 185.074006] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 185.074115] sd 51:0:0:0: Power-on or device reset occurred [ 185.078187] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 185.084269] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.089057] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 185.095215] sd 51:0:0:0: [sde] Sense not available. [ 185.095231] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.104311] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.104715] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 185.104936] sd 49:0:0:0: Power-on or device reset occurred [ 185.104982] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 185.115006] sd 49:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.115013] sd 49:0:0:0: [sdc] Sense not available. [ 185.115027] sd 49:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.115030] sd 49:0:0:0: [sdc] Sense not available. [ 185.115040] sd 49:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 185.115043] sd 49:0:0:0: [sdc] 0-byte physical blocks [ 185.115051] sd 49:0:0:0: [sdc] Write Protect is off [ 185.115062] sd 49:0:0:0: [sdc] Asking for cache data failed [ 185.115065] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 185.115080] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 185.115085] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 185.115113] sd 51:0:0:0: [sde] Sense not available. [ 185.115120] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 185.117969] sd 49:0:0:0: [sdc] Attached SCSI disk [ 185.122312] sd 48:0:0:0: [sdb] Attached SCSI disk [ 185.129451] sd 51:0:0:0: [sde] 0-byte physical blocks [ 185.129460] sd 51:0:0:0: [sde] Write Protect is off [ 185.266431] sd 51:0:0:0: [sde] Asking for cache data failed [ 185.271991] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 185.278284] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 185.284383] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.288710] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 185.297284] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 185.297409] sd 50:0:0:0: Power-on or device reset occurred [ 185.308585] sd 51:0:0:0: [sde] Attached SCSI disk [ 185.317489] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.330079] sd 50:0:0:0: [sdd] Write Protect is off [ 185.334952] sd 50:0:0:0: [sdd] Asking for cache data failed [ 185.340510] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 185.346685] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 185.352938] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 185.360377] sd 50:0:0:0: [sdd] Attached SCSI disk [ 185.454438] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.463039] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 185.463178] sd 48:0:0:0: Power-on or device reset occurred [ 185.463892] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.466136] sd 49:0:0:0: Power-on or device reset occurred [ 185.468591] sd 49:0:0:0: Attached scsi generic sg2 type 0 [ 185.473939] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.486223] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.487555] sd 48:0:0:0: [sdb] Sense not available. [ 185.493400] sd 49:0:0:0: [sdc] Write Protect is off [ 185.502485] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.514653] sd 48:0:0:0: [sdb] Sense not available. [ 185.514658] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 185.519537] sd 49:0:0:0: [sdc] Asking for cache data failed [ 185.528993] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 185.533886] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 185.540212] sd 48:0:0:0: [sdb] Write Protect is off [ 185.545706] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 185.556889] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 185.561760] sd 48:0:0:0: [sdb] Asking for cache data failed [ 185.569759] sd 49:0:0:0: [sdc] Attached SCSI disk [ 185.574092] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 185.574109] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 185.574112] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 185.584179] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.586267] sd 48:0:0:0: [sdb] Attached SCSI disk [ 185.590786] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 185.591019] sd 51:0:0:0: Power-on or device reset occurred [ 185.604495] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.611095] sd 51:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.621127] sd 51:0:0:0: [sde] Write Protect is off [ 185.624477] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 185.624502] sd 50:0:0:0: Power-on or device reset occurred [ 185.644535] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.650491] sd 51:0:0:0: [sde] Asking for cache data failed [ 185.655665] sd 50:0:0:0: [sdd] Write Protect is off [ 185.660740] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 185.673662] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 185.679163] sd 50:0:0:0: [sdd] Asking for cache data failed [ 185.684028] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes [ 185.690190] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 185.698116] sd 51:0:0:0: [sde] Attached SCSI disk [ 185.702025] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 185.725218] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 185.732567] sd 50:0:0:0: [sdd] Attached SCSI disk [ 185.794338] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.794430] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.803193] sd 48:0:0:0: Power-on or device reset occurred [ 185.804102] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 185.811120] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 185.811427] sd 49:0:0:0: Power-on or device reset occurred [ 185.816211] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.831647] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.832442] sd 48:0:0:0: [sdb] Sense not available. [ 185.841924] sd 49:0:0:0: [sdc] Write Protect is off [ 185.849464] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 185.854173] sd 49:0:0:0: [sdc] Asking for cache data failed [ 185.859038] sd 48:0:0:0: [sdb] Sense not available. [ 185.868508] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 185.868563] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 185.874103] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 185.878940] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 185.885113] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 185.892976] sd 49:0:0:0: [sdc] Attached SCSI disk [ 185.897788] sd 48:0:0:0: [sdb] Write Protect is off [ 185.918331] sd 48:0:0:0: [sdb] Asking for cache data failed [ 185.923896] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 185.930071] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 185.940499] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 185.944260] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.952190] sd 48:0:0:0: [sdb] Attached SCSI disk [ 185.959321] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 185.959691] sd 50:0:0:0: Power-on or device reset occurred [ 185.979772] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 185.984457] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 185.987174] sd 50:0:0:0: [sdd] Write Protect is off [ 185.995816] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 185.995935] sd 51:0:0:0: Power-on or device reset occurred [ 186.006005] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.011062] sd 50:0:0:0: [sdd] Asking for cache data failed [ 186.020532] sd 51:0:0:0: [sde] Sense not available. [ 186.026096] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 186.026144] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 186.030979] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.037141] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 186.043396] sd 51:0:0:0: [sde] Sense not available. [ 186.043403] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 186.054629] sd 50:0:0:0: [sdd] Attached SCSI disk [ 186.058956] sd 51:0:0:0: [sde] 0-byte physical blocks [ 186.079839] sd 51:0:0:0: [sde] Write Protect is off [ 186.084731] sd 51:0:0:0: [sde] Asking for cache data failed [ 186.090293] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 186.096546] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.106976] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.119320] sd 51:0:0:0: [sde] Attached SCSI disk [ 186.164254] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.172841] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 186.173265] sd 48:0:0:0: Power-on or device reset occurred [ 186.183767] sd 48:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.193248] sd 48:0:0:0: [sdb] Sense not available. [ 186.198248] sd 48:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.207725] sd 48:0:0:0: [sdb] Sense not available. [ 186.212598] sd 48:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 186.214316] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.218860] sd 48:0:0:0: [sdb] 0-byte physical blocks [ 186.227458] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 186.227574] sd 49:0:0:0: Power-on or device reset occurred [ 186.232127] sd 48:0:0:0: [sdb] Write Protect is off [ 186.247668] sd 49:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 186.255194] sd 49:0:0:0: [sdc] Write Protect is off [ 186.255236] sd 48:0:0:0: [sdb] Asking for cache data failed [ 186.265632] sd 48:0:0:0: [sdb] Assuming drive cache: write through [ 186.265660] sd 49:0:0:0: [sdc] Asking for cache data failed [ 186.271809] sd 48:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.277368] sd 49:0:0:0: [sdc] Assuming drive cache: write through [ 186.287787] sd 48:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.289878] sd 48:0:0:0: [sdb] Attached SCSI disk [ 186.293993] sd 49:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 186.315194] sd 49:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 186.323507] sd 49:0:0:0: [sdc] Attached SCSI disk [ 186.324258] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.336786] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 186.336887] sd 50:0:0:0: Power-on or device reset occurred [ 186.347708] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.357191] sd 50:0:0:0: [sdd] Sense not available. [ 186.362066] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.371559] sd 50:0:0:0: [sdd] Sense not available. [ 186.376437] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 186.382691] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 186.387737] sd 50:0:0:0: [sdd] Write Protect is off [ 186.392606] sd 50:0:0:0: [sdd] Asking for cache data failed [ 186.398182] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 186.404385] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.414811] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.426579] sd 50:0:0:0: [sdd] Attached SCSI disk [ 186.434223] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.442808] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 186.442824] sd 51:0:0:0: Power-on or device reset occurred [ 186.453719] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.463191] sd 51:0:0:0: [sde] Sense not available. [ 186.468075] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.477578] sd 51:0:0:0: [sde] Sense not available. [ 186.482480] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 186.488738] sd 51:0:0:0: [sde] 0-byte physical blocks [ 186.493803] sd 51:0:0:0: [sde] Write Protect is off [ 186.498676] sd 51:0:0:0: [sde] Asking for cache data failed [ 186.504250] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 186.510481] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.520906] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.532433] sd 51:0:0:0: [sde] Attached SCSI disk [ 186.584432] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.584449] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.593134] sd 49:0:0:0: Power-on or device reset occurred [ 186.593564] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 186.601193] scsi 48:0:0:0: Attached scsi generic sg1 type 0 [ 186.606344] sd 48:0:0:0: Power-on or device reset occurred [ 186.606450] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.606458] sd 49:0:0:0: [sdb] Sense not available. [ 186.606473] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.606477] sd 49:0:0:0: [sdb] Sense not available. [ 186.606488] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 186.606491] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 186.606500] sd 49:0:0:0: [sdb] Write Protect is off [ 186.606510] sd 49:0:0:0: [sdb] Asking for cache data failed [ 186.606514] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 186.606530] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.606534] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.609610] sd 49:0:0:0: [sdb] Attached SCSI disk [ 186.704625] sd 48:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.714095] sd 48:0:0:0: [sdc] Sense not available. [ 186.718969] sd 48:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.728439] sd 48:0:0:0: [sdc] Sense not available. [ 186.733305] sd 48:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 186.739569] sd 48:0:0:0: [sdc] 0-byte physical blocks [ 186.744622] sd 48:0:0:0: [sdc] Write Protect is off [ 186.749489] sd 48:0:0:0: [sdc] Asking for cache data failed [ 186.755050] sd 48:0:0:0: [sdc] Assuming drive cache: write through [ 186.761221] sd 48:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 186.771644] sd 48:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 186.783132] sd 48:0:0:0: [sdc] Attached SCSI disk [ 186.854334] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.862928] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 186.863070] sd 50:0:0:0: Power-on or device reset occurred [ 186.873882] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.883362] sd 50:0:0:0: [sdd] Sense not available. [ 186.888268] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 186.897743] sd 50:0:0:0: [sdd] Sense not available. [ 186.902624] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 186.904233] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 186.908897] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 186.917467] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 186.917645] sd 51:0:0:0: Power-on or device reset occurred [ 186.922147] sd 50:0:0:0: [sdd] Write Protect is off [ 186.937724] sd 51:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 186.945207] sd 51:0:0:0: [sde] Write Protect is off [ 186.945211] sd 51:0:0:0: [sde] Asking for cache data failed [ 186.950284] sd 50:0:0:0: [sdd] Asking for cache data failed [ 186.955635] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 186.955641] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 186.955642] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes [ 186.961200] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 186.968575] sd 51:0:0:0: [sde] Attached SCSI disk [ 186.973661] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.000998] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.015089] sd 50:0:0:0: [sdd] Attached SCSI disk [ 187.034658] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.043264] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 187.043446] sd 49:0:0:0: Power-on or device reset occurred [ 187.054210] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.063692] sd 49:0:0:0: [sdb] Sense not available. [ 187.068763] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.078251] sd 49:0:0:0: [sdb] Sense not available. [ 187.083145] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 187.089407] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 187.094462] sd 49:0:0:0: [sdb] Write Protect is off [ 187.099333] sd 49:0:0:0: [sdb] Asking for cache data failed [ 187.104901] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 187.111102] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.121533] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.133499] sd 49:0:0:0: [sdb] Attached SCSI disk [ 187.144488] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.153196] sd 48:0:0:0: Power-on or device reset occurred [ 187.153562] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 187.168757] sd 48:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.178240] sd 48:0:0:0: [sdc] Sense not available. [ 187.183118] sd 48:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.192610] sd 48:0:0:0: [sdc] Sense not available. [ 187.197505] sd 48:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 187.203768] sd 48:0:0:0: [sdc] 0-byte physical blocks [ 187.208815] sd 48:0:0:0: [sdc] Write Protect is off [ 187.213730] sd 48:0:0:0: [sdc] Asking for cache data failed [ 187.219292] sd 48:0:0:0: [sdc] Assuming drive cache: write through [ 187.225645] sd 48:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.236076] sd 48:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.249134] sd 48:0:0:0: [sdc] Attached SCSI disk [ 187.274734] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.284060] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 187.284198] sd 51:0:0:0: Power-on or device reset occurred [ 187.294980] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.304471] sd 51:0:0:0: [sde] Sense not available. [ 187.309359] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.318836] sd 51:0:0:0: [sde] Sense not available. [ 187.323728] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 187.329985] sd 51:0:0:0: [sde] 0-byte physical blocks [ 187.335066] sd 51:0:0:0: [sde] Write Protect is off [ 187.339940] sd 51:0:0:0: [sde] Asking for cache data failed [ 187.345515] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 187.351754] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.362189] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.374575] sd 51:0:0:0: [sde] Attached SCSI disk [ 187.384289] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.392874] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 187.393023] sd 50:0:0:0: Power-on or device reset occurred [ 187.403797] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.413277] sd 50:0:0:0: [sdd] Sense not available. [ 187.418177] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.427701] sd 50:0:0:0: [sdd] Sense not available. [ 187.432590] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 187.434331] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.438855] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 187.447446] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 187.447598] sd 49:0:0:0: Power-on or device reset occurred [ 187.452061] sd 50:0:0:0: [sdd] Write Protect is off [ 187.467683] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 187.475184] sd 49:0:0:0: [sdb] Write Protect is off [ 187.475190] sd 49:0:0:0: [sdb] Asking for cache data failed [ 187.480160] sd 50:0:0:0: [sdd] Asking for cache data failed [ 187.485619] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 187.491178] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 187.497356] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 187.503537] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.509773] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 187.511826] sd 49:0:0:0: [sdb] Attached SCSI disk [ 187.520201] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.521883] sd 50:0:0:0: [sdd] Attached SCSI disk [ 187.564383] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.572979] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 187.573118] sd 48:0:0:0: Power-on or device reset occurred [ 187.583876] sd 48:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.593359] sd 48:0:0:0: [sdc] Sense not available. [ 187.598259] sd 48:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.607742] sd 48:0:0:0: [sdc] Sense not available. [ 187.612666] sd 48:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 187.618926] sd 48:0:0:0: [sdc] 0-byte physical blocks [ 187.623990] sd 48:0:0:0: [sdc] Write Protect is off [ 187.628860] sd 48:0:0:0: [sdc] Asking for cache data failed [ 187.634432] sd 48:0:0:0: [sdc] Assuming drive cache: write through [ 187.640606] sd 48:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.644019] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.651035] sd 48:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.659681] sd 51:0:0:0: Power-on or device reset occurred [ 187.663359] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 187.671299] sd 48:0:0:0: [sdc] Attached SCSI disk [ 187.674973] sd 51:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.694524] sd 51:0:0:0: [sde] Sense not available. [ 187.699414] sd 51:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.708896] sd 51:0:0:0: [sde] Sense not available. [ 187.713807] sd 51:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 187.720063] sd 51:0:0:0: [sde] 0-byte physical blocks [ 187.725249] sd 51:0:0:0: [sde] Write Protect is off [ 187.730129] sd 51:0:0:0: [sde] Asking for cache data failed [ 187.735700] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 187.741900] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.752330] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.765456] sd 51:0:0:0: [sde] Attached SCSI disk [ 187.774046] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.782561] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 187.782727] sd 50:0:0:0: Power-on or device reset occurred [ 187.793482] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.802957] sd 50:0:0:0: [sdd] Sense not available. [ 187.807834] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 187.817335] sd 50:0:0:0: [sdd] Sense not available. [ 187.822213] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 187.824184] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.828472] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 187.836884] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 187.837070] sd 49:0:0:0: Power-on or device reset occurred [ 187.841675] sd 50:0:0:0: [sdd] Write Protect is off [ 187.857157] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 187.864796] sd 49:0:0:0: [sdb] Write Protect is off [ 187.864802] sd 50:0:0:0: [sdd] Asking for cache data failed [ 187.875226] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 187.875236] sd 49:0:0:0: [sdb] Asking for cache data failed [ 187.881520] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 187.886956] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 187.897378] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 187.898749] sd 50:0:0:0: [sdd] Attached SCSI disk [ 187.903617] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 187.924759] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 187.935349] sd 49:0:0:0: [sdb] Attached SCSI disk [ 187.974327] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 187.982917] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 187.983057] sd 48:0:0:0: Power-on or device reset occurred [ 187.993914] sd 48:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.003392] sd 48:0:0:0: [sdc] Sense not available. [ 188.008276] sd 48:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.017753] sd 48:0:0:0: [sdc] Sense not available. [ 188.022624] sd 48:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 188.028888] sd 48:0:0:0: [sdc] 0-byte physical blocks [ 188.033965] sd 48:0:0:0: [sdc] Write Protect is off [ 188.038845] sd 48:0:0:0: [sdc] Asking for cache data failed [ 188.044418] sd 48:0:0:0: [sdc] Assuming drive cache: write through [ 188.050641] sd 48:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 188.054151] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.061073] sd 48:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 188.069444] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 188.069645] sd 51:0:0:0: Power-on or device reset occurred [ 188.081376] sd 48:0:0:0: [sdc] Attached SCSI disk [ 188.089737] sd 51:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.102445] sd 51:0:0:0: [sde] Write Protect is off [ 188.107330] sd 51:0:0:0: [sde] Asking for cache data failed [ 188.112891] sd 51:0:0:0: [sde] Assuming drive cache: write through [ 188.119077] sd 51:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 188.124100] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.125340] sd 51:0:0:0: [sde] Optimal transfer size 524288 bytes [ 188.133696] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 188.133890] sd 50:0:0:0: Power-on or device reset occurred [ 188.142386] sd 51:0:0:0: [sde] Attached SCSI disk [ 188.154011] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.162544] sd 50:0:0:0: [sdd] Write Protect is off [ 188.167427] sd 50:0:0:0: [sdd] Asking for cache data failed [ 188.172988] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 188.179176] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 188.185441] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 188.193925] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.194382] sd 50:0:0:0: [sdd] Attached SCSI disk [ 188.202325] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 188.202485] sd 49:0:0:0: Power-on or device reset occurred [ 188.222564] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.229963] sd 49:0:0:0: [sdb] Write Protect is off [ 188.234846] sd 49:0:0:0: [sdb] Asking for cache data failed [ 188.240406] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 188.246592] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 188.252848] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 188.261808] sd 49:0:0:0: [sdb] Attached SCSI disk [ 188.464138] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.472500] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 188.472686] sd 50:0:0:0: Power-on or device reset occurred [ 188.474118] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.474372] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.474499] sd 51:0:0:0: Power-on or device reset occurred [ 188.475155] sd 48:0:0:0: Power-on or device reset occurred [ 188.477967] sd 51:0:0:0: Attached scsi generic sg2 type 0 [ 188.483427] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.492122] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 188.494573] sd 51:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.494584] sd 51:0:0:0: [sdc] Write Protect is off [ 188.494594] sd 51:0:0:0: [sdc] Asking for cache data failed [ 188.494597] sd 51:0:0:0: [sdc] Assuming drive cache: write through [ 188.494612] sd 51:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 188.494616] sd 51:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 188.495480] sd 48:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.497444] sd 51:0:0:0: [sdc] Attached SCSI disk [ 188.499718] sd 50:0:0:0: [sdd] Sense not available. [ 188.505523] sd 48:0:0:0: [sde] Write Protect is off [ 188.510693] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.516401] sd 48:0:0:0: [sde] Asking for cache data failed [ 188.525573] sd 50:0:0:0: [sdd] Sense not available. [ 188.525582] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 188.525585] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 188.530974] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 188.538406] sd 50:0:0:0: [sdd] Write Protect is off [ 188.543241] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 188.554984] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes [ 188.561361] sd 50:0:0:0: [sdd] Asking for cache data failed [ 188.570418] sd 48:0:0:0: [sde] Attached SCSI disk [ 188.574705] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 188.574812] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 188.584336] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.589155] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 188.598837] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 188.599359] sd 49:0:0:0: Power-on or device reset occurred [ 188.611213] sd 50:0:0:0: [sdd] Attached SCSI disk [ 188.619462] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.711986] sd 49:0:0:0: [sdb] Write Protect is off [ 188.716870] sd 49:0:0:0: [sdb] Asking for cache data failed [ 188.722430] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 188.728643] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 188.734904] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 188.745380] sd 49:0:0:0: [sdb] Attached SCSI disk [ 188.874268] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.882718] scsi 51:0:0:0: Attached scsi generic sg1 type 0 [ 188.883010] sd 51:0:0:0: Power-on or device reset occurred [ 188.903915] sd 51:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.913389] sd 51:0:0:0: [sdc] Sense not available. [ 188.918285] sd 51:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.927759] sd 51:0:0:0: [sdc] Sense not available. [ 188.932629] sd 51:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 188.934071] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.938891] sd 51:0:0:0: [sdc] 0-byte physical blocks [ 188.947270] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 188.947470] sd 48:0:0:0: Power-on or device reset occurred [ 188.952135] sd 51:0:0:0: [sdc] Write Protect is off [ 188.967576] sd 48:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 188.974182] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 188.974570] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 188.974740] sd 50:0:0:0: Power-on or device reset occurred [ 188.975240] sd 48:0:0:0: [sde] Write Protect is off [ 188.983402] sd 51:0:0:0: [sdc] Asking for cache data failed [ 188.984800] sd 50:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.984802] sd 50:0:0:0: [sdd] Sense not available. [ 188.984807] sd 50:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 188.984808] sd 50:0:0:0: [sdd] Sense not available. [ 188.984812] sd 50:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 188.984813] sd 50:0:0:0: [sdd] 0-byte physical blocks [ 188.984815] sd 50:0:0:0: [sdd] Write Protect is off [ 188.984819] sd 50:0:0:0: [sdd] Asking for cache data failed [ 188.984820] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 188.984825] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 188.984827] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 188.986017] sd 50:0:0:0: [sdd] Attached SCSI disk [ 188.994248] sd 51:0:0:0: [sdc] Assuming drive cache: write through [ 188.999129] sd 48:0:0:0: [sde] Asking for cache data failed [ 189.004716] sd 51:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 189.014148] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 189.019007] sd 51:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 189.028527] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 189.036985] sd 51:0:0:0: [sdc] Attached SCSI disk [ 189.039608] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes [ 189.143185] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.144494] sd 48:0:0:0: [sde] Attached SCSI disk [ 189.151564] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 189.151778] sd 49:0:0:0: Power-on or device reset occurred [ 189.171897] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.179305] sd 49:0:0:0: [sdb] Write Protect is off [ 189.184202] sd 49:0:0:0: [sdb] Asking for cache data failed [ 189.189761] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 189.195958] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 189.202213] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 189.210849] sd 49:0:0:0: [sdb] Attached SCSI disk [ 189.394173] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.402691] sd 51:0:0:0: Power-on or device reset occurred [ 189.403483] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 189.404296] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.404981] sd 50:0:0:0: Power-on or device reset occurred [ 189.425051] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.427290] sd 50:0:0:0: Attached scsi generic sg2 type 0 [ 189.428262] sd 51:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.428275] sd 51:0:0:0: [sdc] Write Protect is off [ 189.428286] sd 51:0:0:0: [sdc] Asking for cache data failed [ 189.428288] sd 51:0:0:0: [sdc] Assuming drive cache: write through [ 189.428301] sd 51:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 189.428304] sd 51:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 189.431199] sd 51:0:0:0: [sdc] Attached SCSI disk [ 189.434053] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.434425] sd 48:0:0:0: Power-on or device reset occurred [ 189.444620] sd 50:0:0:0: [sdd] Write Protect is off [ 189.447530] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 189.454501] sd 48:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.464535] sd 48:0:0:0: [sde] Write Protect is off [ 189.470217] sd 50:0:0:0: [sdd] Asking for cache data failed [ 189.480987] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 189.489161] sd 48:0:0:0: [sde] Asking for cache data failed [ 189.494698] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 189.499506] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 189.504054] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.504270] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 189.504432] sd 49:0:0:0: Power-on or device reset occurred [ 189.504897] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 189.512319] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 189.514499] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.514503] sd 49:0:0:0: [sdb] Sense not available. [ 189.514517] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.514520] sd 49:0:0:0: [sdb] Sense not available. [ 189.514529] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 189.514532] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 189.514540] sd 49:0:0:0: [sdb] Write Protect is off [ 189.514550] sd 49:0:0:0: [sdb] Asking for cache data failed [ 189.514553] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 189.514568] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 189.514572] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 189.517482] sd 49:0:0:0: [sdb] Attached SCSI disk [ 189.519809] sd 50:0:0:0: [sdd] Attached SCSI disk [ 189.522715] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes [ 189.528315] sd 48:0:0:0: [sde] Attached SCSI disk [ 189.784748] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.785405] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.793138] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 189.793448] sd 49:0:0:0: Power-on or device reset occurred [ 189.801682] sd 51:0:0:0: Power-on or device reset occurred [ 189.806563] sd 51:0:0:0: Attached scsi generic sg2 type 0 [ 189.813515] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.822033] sd 51:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.822819] sd 49:0:0:0: [sdb] Write Protect is off [ 189.830218] sd 51:0:0:0: [sdc] Sense not available. [ 189.839676] sd 49:0:0:0: [sdb] Asking for cache data failed [ 189.844696] sd 51:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.849408] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 189.854972] sd 51:0:0:0: [sdc] Sense not available. [ 189.864445] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 189.870642] sd 51:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 189.875482] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 189.881733] sd 51:0:0:0: [sdc] 0-byte physical blocks [ 189.889778] sd 49:0:0:0: [sdb] Attached SCSI disk [ 189.895119] sd 51:0:0:0: [sdc] Write Protect is off [ 189.904337] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.917199] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 189.917367] sd 51:0:0:0: [sdc] Asking for cache data failed [ 189.917373] sd 48:0:0:0: Power-on or device reset occurred [ 189.924251] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 189.927439] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.927444] sd 48:0:0:0: [sde] Sense not available. [ 189.927458] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 189.927461] sd 48:0:0:0: [sde] Sense not available. [ 189.927469] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 189.927472] sd 48:0:0:0: [sde] 0-byte physical blocks [ 189.927480] sd 48:0:0:0: [sde] Write Protect is off [ 189.927490] sd 48:0:0:0: [sde] Asking for cache data failed [ 189.927493] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 189.927508] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 189.927512] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 189.928161] sd 51:0:0:0: [sdc] Assuming drive cache: write through [ 189.929106] sd 48:0:0:0: [sde] Attached SCSI disk [ 189.933983] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 189.934139] sd 50:0:0:0: Power-on or device reset occurred [ 189.941830] sd 51:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 189.954212] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 189.956140] sd 51:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 189.965620] sd 50:0:0:0: [sdd] Write Protect is off [ 189.972500] sd 51:0:0:0: [sdc] Attached SCSI disk [ 190.078570] sd 50:0:0:0: [sdd] Asking for cache data failed [ 190.084149] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 190.090358] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 190.096620] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 190.104001] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.104537] sd 50:0:0:0: [sdd] Attached SCSI disk [ 190.112578] sd 49:0:0:0: Power-on or device reset occurred [ 190.113184] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 190.127751] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.137227] sd 49:0:0:0: [sdb] Sense not available. [ 190.142514] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.151996] sd 49:0:0:0: [sdb] Sense not available. [ 190.156898] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 190.163151] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 190.168603] sd 49:0:0:0: [sdb] Write Protect is off [ 190.173834] sd 49:0:0:0: [sdb] Asking for cache data failed [ 190.179394] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 190.185571] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 190.195994] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 190.207573] sd 49:0:0:0: [sdb] Attached SCSI disk [ 190.214447] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.223054] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 190.223186] sd 48:0:0:0: Power-on or device reset occurred [ 190.233998] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.243478] sd 48:0:0:0: [sde] Sense not available. [ 190.248364] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.257835] sd 48:0:0:0: [sde] Sense not available. [ 190.262706] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 190.268963] sd 48:0:0:0: [sde] 0-byte physical blocks [ 190.274006] sd 48:0:0:0: [sde] Write Protect is off [ 190.274263] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.278875] sd 48:0:0:0: [sde] Asking for cache data failed [ 190.287518] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 190.287794] sd 51:0:0:0: Power-on or device reset occurred [ 190.292598] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 190.292604] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 190.307875] sd 51:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 190.309629] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 190.320061] sd 51:0:0:0: [sdc] Write Protect is off [ 190.328762] sd 48:0:0:0: [sde] Attached SCSI disk [ 190.337697] sd 51:0:0:0: [sdc] Asking for cache data failed [ 190.337700] sd 51:0:0:0: [sdc] Assuming drive cache: write through [ 190.344434] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.347277] sd 51:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 190.353423] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 190.353553] sd 50:0:0:0: Power-on or device reset occurred [ 190.358999] sd 51:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 190.373637] sd 50:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 190.381053] sd 51:0:0:0: [sdc] Attached SCSI disk [ 190.384280] sd 50:0:0:0: [sdd] Write Protect is off [ 190.407314] sd 50:0:0:0: [sdd] Asking for cache data failed [ 190.412875] sd 50:0:0:0: [sdd] Assuming drive cache: write through [ 190.419055] sd 50:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 190.425319] sd 50:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 190.433546] sd 50:0:0:0: [sdd] Attached SCSI disk [ 190.564250] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.572635] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 190.572839] sd 49:0:0:0: Power-on or device reset occurred [ 190.583592] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.593068] sd 49:0:0:0: [sdb] Sense not available. [ 190.594059] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.597982] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.606283] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 190.606473] sd 48:0:0:0: Power-on or device reset occurred [ 190.615570] sd 49:0:0:0: [sdb] Sense not available. [ 190.615576] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 190.615578] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 190.626554] sd 48:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 190.631308] sd 49:0:0:0: [sdb] Write Protect is off [ 190.637568] sd 48:0:0:0: [sde] Write Protect is off [ 190.649993] sd 48:0:0:0: [sde] Asking for cache data failed [ 190.654961] sd 49:0:0:0: [sdb] Asking for cache data failed [ 190.659725] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 190.665287] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 190.670857] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 190.677025] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 190.683188] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes [ 190.689444] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 190.701986] sd 48:0:0:0: [sde] Attached SCSI disk [ 190.708732] sd 49:0:0:0: [sdb] Attached SCSI disk [ 190.754098] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.754234] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 190.762502] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 190.762689] sd 50:0:0:0: Power-on or device reset occurred [ 190.771308] sd 51:0:0:0: Power-on or device reset occurred [ 190.775934] sd 51:0:0:0: Attached scsi generic sg2 type 0 [ 190.781321] sd 50:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.796876] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.801622] sd 50:0:0:0: [sdc] Sense not available. [ 190.811091] sd 51:0:0:0: [sdd] Sense not available. [ 190.815956] sd 50:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.820829] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 190.830288] sd 50:0:0:0: [sdc] Sense not available. [ 190.839753] sd 51:0:0:0: [sdd] Sense not available. [ 190.839758] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 190.844624] sd 50:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 190.849486] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 190.855741] sd 50:0:0:0: [sdc] 0-byte physical blocks [ 190.861996] sd 51:0:0:0: [sdd] Write Protect is off [ 190.867036] sd 50:0:0:0: [sdc] Write Protect is off [ 190.881804] sd 51:0:0:0: [sdd] Asking for cache data failed [ 190.881804] sd 50:0:0:0: [sdc] Asking for cache data failed [ 190.887366] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 190.892924] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 190.899097] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 190.905265] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 190.915689] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 190.926109] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 190.927272] sd 50:0:0:0: [sdc] Attached SCSI disk [ 190.939281] sd 51:0:0:0: [sdd] Attached SCSI disk [ 191.014146] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.022549] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 191.022737] sd 48:0:0:0: Power-on or device reset occurred [ 191.033562] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.043033] sd 48:0:0:0: [sde] Sense not available. [ 191.044065] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.048130] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.056295] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 191.056472] sd 49:0:0:0: Power-on or device reset occurred [ 191.065536] sd 48:0:0:0: [sde] Sense not available. [ 191.065660] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 191.076553] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 191.081263] sd 48:0:0:0: [sde] 0-byte physical blocks [ 191.081269] sd 48:0:0:0: [sde] Write Protect is off [ 191.087531] sd 49:0:0:0: [sdb] Write Protect is off [ 191.094907] sd 48:0:0:0: [sde] Asking for cache data failed [ 191.094908] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 191.094913] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.094915] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.096183] sd 48:0:0:0: [sde] Attached SCSI disk [ 191.146876] sd 49:0:0:0: [sdb] Asking for cache data failed [ 191.152438] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 191.158633] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 191.164896] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 191.173183] sd 49:0:0:0: [sdb] Attached SCSI disk [ 191.214081] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.222455] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 191.222715] sd 51:0:0:0: Power-on or device reset occurred [ 191.233380] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.242856] sd 51:0:0:0: [sdd] Sense not available. [ 191.247804] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.254261] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.257280] sd 51:0:0:0: [sdd] Sense not available. [ 191.265865] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 191.266159] sd 50:0:0:0: Power-on or device reset occurred [ 191.270315] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 191.276226] sd 50:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.281172] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 191.287432] sd 50:0:0:0: [sdc] Sense not available. [ 191.287446] sd 50:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.297030] sd 51:0:0:0: [sdd] Write Protect is off [ 191.301955] sd 50:0:0:0: [sdc] Sense not available. [ 191.316299] sd 50:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 191.321287] sd 51:0:0:0: [sdd] Asking for cache data failed [ 191.326034] sd 50:0:0:0: [sdc] 0-byte physical blocks [ 191.332283] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 191.337859] sd 50:0:0:0: [sdc] Write Protect is off [ 191.342897] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.353964] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.364375] sd 50:0:0:0: [sdc] Asking for cache data failed [ 191.376293] sd 51:0:0:0: [sdd] Attached SCSI disk [ 191.380161] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 191.391046] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.401475] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.413271] sd 50:0:0:0: [sdc] Attached SCSI disk [ 191.454153] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.462544] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 191.462804] sd 48:0:0:0: Power-on or device reset occurred [ 191.473530] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.483001] sd 48:0:0:0: [sde] Sense not available. [ 191.487902] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.494360] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.497379] sd 48:0:0:0: [sde] Sense not available. [ 191.505863] scsi 49:0:0:0: Attached scsi generic sg1 type 0 [ 191.510636] sd 49:0:0:0: Power-on or device reset occurred [ 191.512100] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 191.512106] sd 48:0:0:0: [sde] 0-byte physical blocks [ 191.512144] sd 48:0:0:0: [sde] Write Protect is off [ 191.512168] sd 48:0:0:0: [sde] Asking for cache data failed [ 191.512171] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 191.512223] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.512227] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.514404] sd 48:0:0:0: [sde] Attached SCSI disk [ 191.574739] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.584210] sd 49:0:0:0: [sdb] Sense not available. [ 191.589201] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.598670] sd 49:0:0:0: [sdb] Sense not available. [ 191.603660] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 191.609921] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 191.615086] sd 49:0:0:0: [sdb] Write Protect is off [ 191.619957] sd 49:0:0:0: [sdb] Asking for cache data failed [ 191.625519] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 191.631690] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.642113] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.653675] sd 49:0:0:0: [sdb] Attached SCSI disk [ 191.734165] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.742564] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 191.742752] sd 51:0:0:0: Power-on or device reset occurred [ 191.753487] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.762963] sd 51:0:0:0: [sdd] Sense not available. [ 191.767853] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.777333] sd 51:0:0:0: [sdd] Sense not available. [ 191.782215] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 191.788528] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 191.793628] sd 51:0:0:0: [sdd] Write Protect is off [ 191.794340] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.806711] sd 51:0:0:0: [sdd] Asking for cache data failed [ 191.806993] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 191.807151] sd 50:0:0:0: Power-on or device reset occurred [ 191.812280] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 191.827248] sd 50:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 191.829322] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 191.836762] sd 50:0:0:0: [sdc] Write Protect is off [ 191.847143] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 191.862237] sd 50:0:0:0: [sdc] Asking for cache data failed [ 191.863998] sd 51:0:0:0: [sdd] Attached SCSI disk [ 191.867801] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 191.878666] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 191.884925] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 191.892835] sd 50:0:0:0: [sdc] Attached SCSI disk [ 191.914261] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 191.922869] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 191.923028] sd 48:0:0:0: Power-on or device reset occurred [ 191.933796] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.943263] sd 48:0:0:0: [sde] Sense not available. [ 191.948176] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 191.957650] sd 48:0:0:0: [sde] Sense not available. [ 191.962519] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 191.968814] sd 48:0:0:0: [sde] 0-byte physical blocks [ 191.974171] sd 48:0:0:0: [sde] Write Protect is off [ 191.979053] sd 48:0:0:0: [sde] Asking for cache data failed [ 191.984622] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 191.990837] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 192.001266] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 192.013310] sd 48:0:0:0: [sde] Attached SCSI disk [ 192.054323] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.062823] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 192.062997] sd 49:0:0:0: Power-on or device reset occurred [ 192.073771] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.083244] sd 49:0:0:0: [sdb] Sense not available. [ 192.088123] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.097593] sd 49:0:0:0: [sdb] Sense not available. [ 192.102463] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 192.108720] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 192.113764] sd 49:0:0:0: [sdb] Write Protect is off [ 192.118633] sd 49:0:0:0: [sdb] Asking for cache data failed [ 192.124195] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 192.130367] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 192.140794] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 192.153650] sd 49:0:0:0: [sdb] Attached SCSI disk [ 192.214260] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.222761] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 192.222931] sd 50:0:0:0: Power-on or device reset occurred [ 192.224285] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.224838] sd 51:0:0:0: Power-on or device reset occurred [ 192.228280] sd 51:0:0:0: Attached scsi generic sg2 type 0 [ 192.233676] sd 50:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.244916] sd 51:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 192.247294] sd 50:0:0:0: [sdc] Sense not available. [ 192.252674] sd 51:0:0:0: [sdd] Write Protect is off [ 192.262149] sd 50:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.269527] sd 51:0:0:0: [sdd] Asking for cache data failed [ 192.274395] sd 50:0:0:0: [sdc] Sense not available. [ 192.279257] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 192.288774] sd 50:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 192.294303] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 192.299159] sd 50:0:0:0: [sdc] 0-byte physical blocks [ 192.305331] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 192.311597] sd 50:0:0:0: [sdc] Write Protect is off [ 192.321182] sd 51:0:0:0: [sdd] Attached SCSI disk [ 192.338542] sd 50:0:0:0: [sdc] Asking for cache data failed [ 192.344109] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 192.350411] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 192.360839] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 192.372969] sd 50:0:0:0: [sdc] Attached SCSI disk [ 192.374917] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.386301] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 192.386394] sd 48:0:0:0: Power-on or device reset occurred [ 192.397200] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.406674] sd 48:0:0:0: [sde] Sense not available. [ 192.411545] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.421017] sd 48:0:0:0: [sde] Sense not available. [ 192.425901] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 192.432165] sd 48:0:0:0: [sde] 0-byte physical blocks [ 192.437242] sd 48:0:0:0: [sde] Write Protect is off [ 192.442119] sd 48:0:0:0: [sde] Asking for cache data failed [ 192.447689] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 192.453888] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 192.454337] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.464315] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 192.466053] sd 48:0:0:0: [sde] Attached SCSI disk [ 192.472813] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 192.473033] sd 49:0:0:0: Power-on or device reset occurred [ 192.493107] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 192.505689] sd 49:0:0:0: [sdb] Write Protect is off [ 192.510561] sd 49:0:0:0: [sdb] Asking for cache data failed [ 192.516128] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 192.522305] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 192.528566] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 192.536786] sd 49:0:0:0: [sdb] Attached SCSI disk [ 192.684467] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.693064] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 192.693197] sd 51:0:0:0: Power-on or device reset occurred [ 192.703996] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.713480] sd 51:0:0:0: [sdd] Sense not available. [ 192.718358] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 192.727834] sd 51:0:0:0: [sdd] Sense not available. [ 192.732707] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 192.738970] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 192.744023] sd 51:0:0:0: [sdd] Write Protect is off [ 192.744290] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.757105] sd 51:0:0:0: [sdd] Asking for cache data failed [ 192.757497] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 192.757648] sd 50:0:0:0: Power-on or device reset occurred [ 192.762667] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 192.774399] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.777733] sd 50:0:0:0: [sdc] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 192.777744] sd 50:0:0:0: [sdc] Write Protect is off [ 192.777755] sd 50:0:0:0: [sdc] Asking for cache data failed [ 192.777758] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 192.777773] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes [ 192.777777] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes [ 192.779732] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 192.780587] sd 50:0:0:0: [sdc] Attached SCSI disk [ 192.788344] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 192.788479] sd 48:0:0:0: Power-on or device reset occurred [ 192.795268] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 192.797496] sd 51:0:0:0: [sdd] Attached SCSI disk [ 192.808550] sd 48:0:0:0: [sde] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 192.854233] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 192.860622] sd 48:0:0:0: [sde] Write Protect is off [ 192.865469] scsi 49:0:0:0: Attached scsi generic sg1 type 0 [ 192.865794] sd 49:0:0:0: Power-on or device reset occurred [ 192.885881] sd 49:0:0:0: [sdb] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 192.891213] sd 48:0:0:0: [sde] Asking for cache data failed [ 192.896671] sd 49:0:0:0: [sdb] Write Protect is off [ 192.904006] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 192.904015] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes [ 192.914442] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes [ 192.925193] sd 48:0:0:0: [sde] Attached SCSI disk [ 192.927012] sd 49:0:0:0: [sdb] Asking for cache data failed [ 192.943213] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 192.949415] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes [ 192.955680] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes [ 192.963945] sd 49:0:0:0: [sdb] Attached SCSI disk [ 193.044224] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.052632] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 193.052832] sd 50:0:0:0: Power-on or device reset occurred [ 193.063553] sd 50:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.073028] sd 50:0:0:0: [sdc] Sense not available. [ 193.074674] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.077910] sd 50:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.086527] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 193.086833] sd 51:0:0:0: Power-on or device reset occurred [ 193.095537] sd 50:0:0:0: [sdc] Sense not available. [ 193.095553] sd 50:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 193.106919] sd 51:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 193.111269] sd 50:0:0:0: [sdc] 0-byte physical blocks [ 193.117533] sd 51:0:0:0: [sdd] Write Protect is off [ 193.124949] sd 50:0:0:0: [sdc] Write Protect is off [ 193.139698] sd 51:0:0:0: [sdd] Asking for cache data failed [ 193.139818] sd 50:0:0:0: [sdc] Asking for cache data failed [ 193.145265] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 193.150824] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 193.157016] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 193.163175] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.169427] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 193.174437] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.174901] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 193.174938] sd 48:0:0:0: Power-on or device reset occurred [ 193.179847] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.184984] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.184987] sd 48:0:0:0: [sde] Sense not available. [ 193.184994] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.184995] sd 48:0:0:0: [sde] Sense not available. [ 193.185001] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 193.185003] sd 48:0:0:0: [sde] 0-byte physical blocks [ 193.185007] sd 48:0:0:0: [sde] Write Protect is off [ 193.185012] sd 48:0:0:0: [sde] Asking for cache data failed [ 193.185014] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 193.185028] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.185033] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.187987] sd 48:0:0:0: [sde] Attached SCSI disk [ 193.188436] sd 51:0:0:0: [sdd] Attached SCSI disk [ 193.195847] sd 50:0:0:0: [sdc] Attached SCSI disk [ 193.345005] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.353763] scsi 49:0:0:0: Attached scsi generic sg1 type 0 [ 193.354768] sd 49:0:0:0: Power-on or device reset occurred [ 193.364853] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.374335] sd 49:0:0:0: [sdb] Sense not available. [ 193.379225] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.388696] sd 49:0:0:0: [sdb] Sense not available. [ 193.393567] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 193.399821] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 193.404865] sd 49:0:0:0: [sdb] Write Protect is off [ 193.409733] sd 49:0:0:0: [sdb] Asking for cache data failed [ 193.415295] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 193.421467] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.431909] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.443566] sd 49:0:0:0: [sdb] Attached SCSI disk [ 193.464239] scsi 50:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.464342] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.473004] sd 50:0:0:0: Power-on or device reset occurred [ 193.473476] sd 50:0:0:0: Attached scsi generic sg1 type 0 [ 193.481052] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 193.481211] sd 51:0:0:0: Power-on or device reset occurred [ 193.486089] sd 50:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.501300] sd 51:0:0:0: [sdd] 16384 512-byte logical blocks: (8.39 MB/8.00 MiB) [ 193.502320] sd 50:0:0:0: [sdc] Sense not available. [ 193.511798] sd 51:0:0:0: [sdd] Write Protect is off [ 193.519226] sd 50:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.524250] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.524815] sd 48:0:0:0: Power-on or device reset occurred [ 193.524823] sd 48:0:0:0: Attached scsi generic sg1 type 0 [ 193.528911] sd 50:0:0:0: [sdc] Sense not available. [ 193.534864] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.534869] sd 48:0:0:0: [sde] Sense not available. [ 193.534882] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.534885] sd 48:0:0:0: [sde] Sense not available. [ 193.534894] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 193.534897] sd 48:0:0:0: [sde] 0-byte physical blocks [ 193.534905] sd 48:0:0:0: [sde] Write Protect is off [ 193.534915] sd 48:0:0:0: [sde] Asking for cache data failed [ 193.534918] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 193.534932] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.534937] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.536181] sd 48:0:0:0: [sde] Attached SCSI disk [ 193.538388] sd 51:0:0:0: [sdd] Asking for cache data failed [ 193.546570] sd 50:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B) [ 193.552027] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 193.557425] sd 50:0:0:0: [sdc] 0-byte physical blocks [ 193.562301] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes [ 193.571764] sd 50:0:0:0: [sdc] Write Protect is off [ 193.576627] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes [ 193.579353] sd 51:0:0:0: [sdd] Attached SCSI disk [ 193.586121] sd 50:0:0:0: [sdc] Asking for cache data failed [ 193.694694] sd 50:0:0:0: [sdc] Assuming drive cache: write through [ 193.700880] sd 50:0:0:0: [sdc] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.711308] sd 50:0:0:0: [sdc] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.722785] sd 50:0:0:0: [sdc] Attached SCSI disk [ 193.814040] scsi 49:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.822602] sd 49:0:0:0: Power-on or device reset occurred [ 193.822878] sd 49:0:0:0: Attached scsi generic sg1 type 0 [ 193.838160] sd 49:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.847647] sd 49:0:0:0: [sdb] Sense not available. [ 193.852524] sd 49:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.854422] scsi 51:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.862007] sd 49:0:0:0: [sdb] Sense not available. [ 193.871863] sd 51:0:0:0: Attached scsi generic sg1 type 0 [ 193.872037] sd 51:0:0:0: Power-on or device reset occurred [ 193.875044] sd 49:0:0:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 193.882094] sd 51:0:0:0: [sdd] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.885915] sd 49:0:0:0: [sdb] 0-byte physical blocks [ 193.892153] sd 51:0:0:0: [sdd] Sense not available. [ 193.894450] scsi 48:0:0:0: Direct-Access Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 193.895203] scsi 48:0:0:0: Attached scsi generic sg1 type 0 [ 193.895705] sd 48:0:0:0: [sde] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.895711] sd 48:0:0:0: [sde] Sense not available. [ 193.895725] sd 48:0:0:0: [sde] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.895728] sd 48:0:0:0: [sde] Sense not available. [ 193.895737] sd 48:0:0:0: [sde] 0 512-byte logical blocks: (0 B/0 B) [ 193.895740] sd 48:0:0:0: [sde] 0-byte physical blocks [ 193.895748] sd 48:0:0:0: [sde] Write Protect is off [ 193.895758] sd 48:0:0:0: [sde] Asking for cache data failed [ 193.895761] sd 48:0:0:0: [sde] Assuming drive cache: write through [ 193.895776] sd 48:0:0:0: [sde] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.895780] sd 48:0:0:0: [sde] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.898757] sd 48:0:0:0: [sde] Attached SCSI disk [ 193.901890] sd 49:0:0:0: [sdb] Write Protect is off [ 193.906668] sd 51:0:0:0: [sdd] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK [ 193.906669] sd 51:0:0:0: [sdd] Sense not available. [ 193.919702] sd 51:0:0:0: [sdd] 0 512-byte logical blocks: (0 B/0 B) [ 193.925380] sd 49:0:0:0: [sdb] Asking for cache data failed [ 193.934727] sd 51:0:0:0: [sdd] 0-byte physical blocks [ 193.934730] sd 51:0:0:0: [sdd] Write Protect is off [ 193.939595] sd 49:0:0:0: [sdb] Assuming drive cache: write through [ 193.953942] sd 49:0:0:0: [sdb] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 193.960182] sd 51:0:0:0: [sdd] Asking for cache data failed [ 193.965225] sd 49:0:0:0: [sdb] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 193.970086] sd 51:0:0:0: [sdd] Assuming drive cache: write through [ 193.979243] sd 49:0:0:0: [sdb] Attached SCSI disk [ 193.981817] sd 51:0:0:0: [sdd] Preferred minimum I/O size 512 bytes not a multiple of physical block size (0 bytes) [ 194.101747] sd 51:0:0:0: [sdd] Optimal transfer size 524288 bytes not a multiple of physical block size (0 bytes) [ 194.113245] sd 51:0:0:0: [sdd] Attached SCSI disk [ 194.856462] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 194.865523] scsi host48: scsi_debug: version 0191 [20210520] [ 194.865523] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 194.877983] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 194.878055] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 194.886539] sr 48:0:0:0: Power-on or device reset occurred [ 194.895201] scsi host49: scsi_debug: version 0191 [20210520] [ 194.895201] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 194.895928] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 194.896003] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 194.920754] sr 48:0:0:0: [sr0] scsi-1 drive [ 194.921723] scsi host50: scsi_debug: version 0191 [20210520] [ 194.921723] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 194.929894] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 194.934643] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw = (0/1) [ 194.934723] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 194.968469] scsi host51: scsi_debug: version 0191 [20210520] [ 194.968469] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0 [ 194.980599] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 194.980835] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 194.986362] sr 49:0:0:0: Power-on or device reset occurred [ 195.019709] sr 49:0:0:0: [sr1] scsi-1 drive [ 195.048363] sr 49:0:0:0: Attached scsi generic sg2 type 5 [ 195.054576] sr 50:0:0:0: Power-on or device reset occurred [ 195.080132] sr 50:0:0:0: [sr2] scsi-1 drive [ 195.109051] sr 50:0:0:0: Attached scsi generic sg3 type 5 [ 195.115682] sr 51:0:0:0: Power-on or device reset occurred [ 195.141236] sr 51:0:0:0: [sr3] scsi-1 drive [ 195.170320] sr 51:0:0:0: Attached scsi generic sg4 type 5 [ 196.123936] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.123958] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.132283] sr 50:0:0:0: Power-on or device reset occurred [ 196.140502] sr 49:0:0:0: Power-on or device reset occurred [ 196.153929] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.162312] sr 48:0:0:0: Power-on or device reset occurred [ 196.163951] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.165934] sr 50:0:0:0: [sr1] scsi-1 drive [ 196.171321] sr 49:0:0:0: [sr2] scsi-1 drive [ 196.176167] sr 51:0:0:0: Power-on or device reset occurred [ 196.187904] sr 48:0:0:0: [sr0] scsi-1 drive [ 196.194628] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 196.203531] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 196.209894] sr 51:0:0:0: [sr3] scsi-1 drive [ 196.217649] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 196.239834] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 196.394058] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.402449] sr 50:0:0:0: Power-on or device reset occurred [ 196.428014] sr 50:0:0:0: [sr1] scsi-1 drive [ 196.443903] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.452257] sr 51:0:0:0: Power-on or device reset occurred [ 196.457868] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 196.474527] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.477804] sr 51:0:0:0: [sr3] scsi-1 drive [ 196.483057] sr 49:0:0:0: Power-on or device reset occurred [ 196.512475] sr 49:0:0:0: [sr2] scsi-1 drive [ 196.516882] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 196.524367] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.532774] sr 48:0:0:0: Power-on or device reset occurred [ 196.540631] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 196.558359] sr 48:0:0:0: [sr0] scsi-1 drive [ 196.586571] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 196.744026] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.744536] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.752423] sr 49:0:0:0: Power-on or device reset occurred [ 196.760746] sr 50:0:0:0: Power-on or device reset occurred [ 196.784546] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.785933] sr 49:0:0:0: [sr1] scsi-1 drive [ 196.786413] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 196.787379] sr 48:0:0:0: Power-on or device reset occurred [ 196.793045] sr 51:0:0:0: Power-on or device reset occurred [ 196.793409] sr 50:0:0:0: [sr2] scsi-1 drive [ 196.807460] sr 48:0:0:0: [sr0] scsi-1 drive [ 196.821345] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 196.829910] sr 50:0:0:0: Attached scsi generic sg2 type 5 [ 196.830618] sr 51:0:0:0: [sr3] scsi-1 drive [ 196.837394] sr 50:0:0:0: [sr2] Hmm, seems the drive doesn't support multisession CD's [ 196.847401] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 196.863209] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 197.024028] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.027146] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.032406] sr 48:0:0:0: Power-on or device reset occurred [ 197.040691] sr 51:0:0:0: Power-on or device reset occurred [ 197.064631] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.065905] sr 48:0:0:0: [sr0] scsi-1 drive [ 197.071408] sr 51:0:0:0: [sr3] scsi-1 drive [ 197.073098] sr 50:0:0:0: Power-on or device reset occurred [ 197.095393] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 197.100868] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 197.106716] sr 50:0:0:0: [sr2] scsi-1 drive [ 197.106803] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.119280] sr 49:0:0:0: Power-on or device reset occurred [ 197.134671] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 197.144828] sr 49:0:0:0: [sr1] scsi-1 drive [ 197.172676] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 197.353946] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.354479] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.362455] sr 49:0:0:0: Power-on or device reset occurred [ 197.370647] sr 48:0:0:0: Power-on or device reset occurred [ 197.384565] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.384569] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.393064] sr 50:0:0:0: Power-on or device reset occurred [ 197.396596] sr 49:0:0:0: [sr0] scsi-1 drive [ 197.401199] sr 51:0:0:0: Power-on or device reset occurred [ 197.401331] sr 48:0:0:0: [sr1] scsi-1 drive [ 197.420303] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 197.425864] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 197.426460] sr 50:0:0:0: [sr2] scsi-1 drive [ 197.440301] sr 51:0:0:0: [sr3] scsi-1 drive [ 197.459210] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 197.468435] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 197.643970] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.652666] sr 51:0:0:0: Power-on or device reset occurred [ 197.664455] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.664675] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.664770] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.665444] sr 50:0:0:0: Power-on or device reset occurred [ 197.672947] sr 49:0:0:0: Power-on or device reset occurred [ 197.678252] sr 51:0:0:0: [sr3] scsi-1 drive [ 197.681139] sr 48:0:0:0: Power-on or device reset occurred [ 197.685521] sr 50:0:0:0: [sr0] scsi-1 drive [ 197.709044] sr 49:0:0:0: [sr1] scsi-1 drive [ 197.714643] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 197.723429] sr 50:0:0:0: Attached scsi generic sg2 type 5 [ 197.724187] sr 48:0:0:0: [sr2] scsi-1 drive [ 197.735147] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 197.757467] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 197.873882] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.882210] sr 49:0:0:0: Power-on or device reset occurred [ 197.907749] sr 49:0:0:0: [sr1] scsi-1 drive [ 197.913932] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.922533] sr 48:0:0:0: Power-on or device reset occurred [ 197.934090] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 197.944261] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.948104] sr 48:0:0:0: [sr2] scsi-1 drive [ 197.952632] sr 51:0:0:0: Power-on or device reset occurred [ 197.981444] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 197.982180] sr 51:0:0:0: [sr3] scsi-1 drive [ 197.983983] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 197.984148] sr 50:0:0:0: Power-on or device reset occurred [ 198.004176] sr 50:0:0:0: [sr0] scsi-1 drive [ 198.028607] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 198.034735] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 198.093951] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.102458] sr 49:0:0:0: Power-on or device reset occurred [ 198.128021] sr 49:0:0:0: [sr1] scsi-1 drive [ 198.156402] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 198.224067] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.224595] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.232585] sr 50:0:0:0: Power-on or device reset occurred [ 198.240745] sr 48:0:0:0: Power-on or device reset occurred [ 198.265959] sr 50:0:0:0: [sr0] scsi-1 drive [ 198.271437] sr 48:0:0:0: [sr2] scsi-1 drive [ 198.292275] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 198.301391] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 198.304443] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.315321] sr 51:0:0:0: Power-on or device reset occurred [ 198.340894] sr 51:0:0:0: [sr3] scsi-1 drive [ 198.367013] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 198.444175] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.452527] sr 49:0:0:0: Power-on or device reset occurred [ 198.478072] sr 49:0:0:0: [sr1] scsi-1 drive [ 198.506261] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 198.514425] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.522996] sr 48:0:0:0: Power-on or device reset occurred [ 198.524641] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.537408] sr 50:0:0:0: Power-on or device reset occurred [ 198.548568] sr 48:0:0:0: [sr2] scsi-1 drive [ 198.562941] sr 50:0:0:0: [sr0] scsi-1 drive [ 198.563912] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.575460] sr 51:0:0:0: Power-on or device reset occurred [ 198.581079] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 198.589446] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 198.604892] sr 51:0:0:0: [sr3] scsi-1 drive [ 198.631534] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 198.754049] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.762554] sr 50:0:0:0: Power-on or device reset occurred [ 198.764534] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.776584] sr 49:0:0:0: Power-on or device reset occurred [ 198.783928] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.788191] sr 50:0:0:0: [sr0] scsi-1 drive [ 198.792304] sr 51:0:0:0: Power-on or device reset occurred [ 198.802155] sr 49:0:0:0: [sr1] scsi-1 drive [ 198.820055] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 198.821986] sr 51:0:0:0: [sr3] scsi-1 drive [ 198.829748] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 198.853406] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 198.864636] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 198.873146] sr 48:0:0:0: Power-on or device reset occurred [ 198.898734] sr 48:0:0:0: [sr2] scsi-1 drive [ 198.926791] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 199.043970] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.044668] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.052503] sr 49:0:0:0: Power-on or device reset occurred [ 199.060694] sr 50:0:0:0: Power-on or device reset occurred [ 199.085890] sr 49:0:0:0: [sr0] scsi-1 drive [ 199.091365] sr 50:0:0:0: [sr1] scsi-1 drive [ 199.104234] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.112596] sr 51:0:0:0: Power-on or device reset occurred [ 199.118722] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 199.124276] sr 50:0:0:0: Attached scsi generic sg2 type 5 [ 199.133380] sr 49:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 199.138192] sr 51:0:0:0: [sr3] scsi-1 drive [ 199.139548] sr 50:0:0:0: [sr1] Hmm, seems the drive doesn't support multisession CD's [ 199.177752] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 199.244371] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.252759] sr 48:0:0:0: Power-on or device reset occurred [ 199.278313] sr 48:0:0:0: [sr2] scsi-1 drive [ 199.304870] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 199.324186] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.332711] sr 51:0:0:0: Power-on or device reset occurred [ 199.358279] sr 51:0:0:0: [sr3] scsi-1 drive [ 199.364427] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.372943] sr 50:0:0:0: Power-on or device reset occurred [ 199.386276] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 199.394495] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.398529] sr 50:0:0:0: [sr1] scsi-1 drive [ 199.403001] sr 49:0:0:0: Power-on or device reset occurred [ 199.430693] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 199.432400] sr 49:0:0:0: [sr0] scsi-1 drive [ 199.462653] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 199.468481] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.476881] sr 48:0:0:0: Power-on or device reset occurred [ 199.502426] sr 48:0:0:0: [sr2] scsi-1 drive [ 199.528915] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 199.614486] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.623026] sr 51:0:0:0: Power-on or device reset occurred [ 199.643972] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.648617] sr 51:0:0:0: [sr3] scsi-1 drive [ 199.652459] sr 50:0:0:0: Power-on or device reset occurred [ 199.680233] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 199.681867] sr 50:0:0:0: [sr1] scsi-1 drive [ 199.712167] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 199.714438] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.726104] sr 49:0:0:0: Power-on or device reset occurred [ 199.751689] sr 49:0:0:0: [sr0] scsi-1 drive [ 199.753981] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.764245] sr 48:0:0:0: Power-on or device reset occurred [ 199.778541] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 199.793972] sr 48:0:0:0: [sr2] scsi-1 drive [ 199.820661] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 199.853952] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.862327] sr 51:0:0:0: Power-on or device reset occurred [ 199.887917] sr 51:0:0:0: [sr3] scsi-1 drive [ 199.914664] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 199.993897] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 199.994165] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.002280] sr 48:0:0:0: Power-on or device reset occurred [ 200.003705] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.003908] sr 49:0:0:0: Power-on or device reset occurred [ 200.010440] sr 50:0:0:0: Power-on or device reset occurred [ 200.023946] sr 49:0:0:0: [sr0] scsi-1 drive [ 200.035777] sr 48:0:0:0: [sr1] scsi-1 drive [ 200.049431] sr 50:0:0:0: [sr2] scsi-1 drive [ 200.061341] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 200.066842] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 200.075653] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 200.174180] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.182508] sr 51:0:0:0: Power-on or device reset occurred [ 200.208057] sr 51:0:0:0: [sr3] scsi-1 drive [ 200.234305] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 200.253945] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.262385] sr 50:0:0:0: Power-on or device reset occurred [ 200.287966] sr 50:0:0:0: [sr2] scsi-1 drive [ 200.316053] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 200.334430] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.342939] sr 48:0:0:0: Power-on or device reset occurred [ 200.344120] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.356797] sr 49:0:0:0: Power-on or device reset occurred [ 200.368502] sr 48:0:0:0: [sr1] scsi-1 drive [ 200.382358] sr 49:0:0:0: [sr0] scsi-1 drive [ 200.395152] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 200.409539] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 200.504071] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.512449] sr 51:0:0:0: Power-on or device reset occurred [ 200.537999] sr 51:0:0:0: [sr3] scsi-1 drive [ 200.563899] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.572159] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 200.572251] sr 50:0:0:0: Power-on or device reset occurred [ 200.584571] sr 51:0:0:0: [sr3] Hmm, seems the drive doesn't support multisession CD's [ 200.603097] sr 50:0:0:0: [sr2] scsi-1 drive [ 200.613890] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.622265] sr 48:0:0:0: Power-on or device reset occurred [ 200.629846] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 200.647846] sr 48:0:0:0: [sr1] scsi-1 drive [ 200.674287] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.682619] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 200.682645] sr 49:0:0:0: Power-on or device reset occurred [ 200.713570] sr 49:0:0:0: [sr0] scsi-1 drive [ 200.741660] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 200.813819] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.822172] sr 51:0:0:0: Power-on or device reset occurred [ 200.847726] sr 51:0:0:0: [sr3] scsi-1 drive [ 200.875836] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 200.924301] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 200.932643] sr 50:0:0:0: Power-on or device reset occurred [ 200.958199] sr 50:0:0:0: [sr2] scsi-1 drive [ 200.986188] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 201.004131] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.012487] sr 48:0:0:0: Power-on or device reset occurred [ 201.034425] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.038029] sr 48:0:0:0: [sr1] scsi-1 drive [ 201.042938] sr 49:0:0:0: Power-on or device reset occurred [ 201.047186] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.060643] sr 51:0:0:0: Power-on or device reset occurred [ 201.071961] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 201.072341] sr 49:0:0:0: [sr0] scsi-1 drive [ 201.086215] sr 51:0:0:0: [sr3] scsi-1 drive [ 201.104925] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 201.114613] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 201.203950] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.212465] sr 50:0:0:0: Power-on or device reset occurred [ 201.238027] sr 50:0:0:0: [sr2] scsi-1 drive [ 201.254055] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.262561] sr 48:0:0:0: Power-on or device reset occurred [ 201.268174] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 201.288113] sr 48:0:0:0: [sr1] scsi-1 drive [ 201.316092] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 201.373879] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.374358] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.382280] sr 51:0:0:0: Power-on or device reset occurred [ 201.390450] sr 49:0:0:0: Power-on or device reset occurred [ 201.415788] sr 51:0:0:0: [sr0] scsi-1 drive [ 201.421255] sr 49:0:0:0: [sr3] scsi-1 drive [ 201.443903] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.444478] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 201.452296] sr 50:0:0:0: Power-on or device reset occurred [ 201.457538] sr 49:0:0:0: Attached scsi generic sg2 type 5 [ 201.463994] sr 51:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 201.483013] sr 50:0:0:0: [sr2] scsi-1 drive [ 201.513052] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 201.564427] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.573027] sr 48:0:0:0: Power-on or device reset occurred [ 201.598596] sr 48:0:0:0: [sr1] scsi-1 drive [ 201.628287] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 201.653859] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.654023] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.662270] sr 49:0:0:0: Power-on or device reset occurred [ 201.670713] sr 51:0:0:0: Power-on or device reset occurred [ 201.695742] sr 49:0:0:0: [sr0] scsi-1 drive [ 201.701267] sr 51:0:0:0: [sr3] scsi-1 drive [ 201.714056] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.722368] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 201.722578] sr 50:0:0:0: Power-on or device reset occurred [ 201.733886] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 201.749487] sr 51:0:0:0: [sr3] Hmm, seems the drive doesn't support multisession CD's [ 201.753404] sr 50:0:0:0: [sr2] scsi-1 drive [ 201.785367] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 201.813974] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.822348] sr 48:0:0:0: Power-on or device reset occurred [ 201.848033] sr 48:0:0:0: [sr1] scsi-1 drive [ 201.875138] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 201.934476] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.943889] sr 51:0:0:0: Power-on or device reset occurred [ 201.969418] sr 51:0:0:0: [sr3] scsi-1 drive [ 201.984044] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.984748] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 201.992545] sr 50:0:0:0: Power-on or device reset occurred [ 201.996388] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 202.000614] sr 49:0:0:0: Power-on or device reset occurred [ 202.025950] sr 50:0:0:0: [sr0] scsi-1 drive [ 202.036816] sr 49:0:0:0: [sr2] scsi-1 drive [ 202.053739] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.062157] sr 48:0:0:0: Power-on or device reset occurred [ 202.063334] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 202.073119] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 202.083287] sr 49:0:0:0: [sr2] Hmm, seems the drive doesn't support multisession CD's [ 202.087739] sr 48:0:0:0: [sr1] scsi-1 drive [ 202.120543] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 202.264253] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.272629] sr 51:0:0:0: Power-on or device reset occurred [ 202.283995] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.284590] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.292497] sr 48:0:0:0: Power-on or device reset occurred [ 202.298192] sr 51:0:0:0: [sr3] scsi-1 drive [ 202.300684] sr 50:0:0:0: Power-on or device reset occurred [ 202.324513] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.325886] sr 48:0:0:0: [sr0] scsi-1 drive [ 202.328981] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 202.333021] sr 49:0:0:0: Power-on or device reset occurred [ 202.335546] sr 50:0:0:0: [sr1] scsi-1 drive [ 202.359503] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 202.365072] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 202.371976] sr 49:0:0:0: [sr2] scsi-1 drive [ 202.379944] sr 48:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 202.398761] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 202.576899] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.585296] sr 48:0:0:0: Power-on or device reset occurred [ 202.610882] sr 48:0:0:0: [sr0] scsi-1 drive [ 202.634603] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.642902] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 202.643101] sr 51:0:0:0: Power-on or device reset occurred [ 202.664598] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.673106] sr 50:0:0:0: Power-on or device reset occurred [ 202.673869] sr 51:0:0:0: [sr3] scsi-1 drive [ 202.698696] sr 50:0:0:0: [sr1] scsi-1 drive [ 202.706573] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 202.730511] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 202.774537] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.783056] sr 49:0:0:0: Power-on or device reset occurred [ 202.808644] sr 49:0:0:0: [sr2] scsi-1 drive [ 202.836229] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 202.903803] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.912194] sr 50:0:0:0: Power-on or device reset occurred [ 202.937756] sr 50:0:0:0: [sr1] scsi-1 drive [ 202.954343] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.962725] sr 48:0:0:0: Power-on or device reset occurred [ 202.964502] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 202.968345] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 202.976728] sr 51:0:0:0: Power-on or device reset occurred [ 202.985896] sr 50:0:0:0: [sr1] Hmm, seems the drive doesn't support multisession CD's [ 202.988278] sr 48:0:0:0: [sr0] scsi-1 drive [ 203.007321] sr 51:0:0:0: [sr3] scsi-1 drive [ 203.021731] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 203.034162] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 203.043718] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.052102] sr 49:0:0:0: Power-on or device reset occurred [ 203.077695] sr 49:0:0:0: [sr2] scsi-1 drive [ 203.104276] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 203.183917] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.192430] sr 48:0:0:0: Power-on or device reset occurred [ 203.214505] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.217998] sr 48:0:0:0: [sr0] scsi-1 drive [ 203.223006] sr 50:0:0:0: Power-on or device reset occurred [ 203.249185] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 203.252404] sr 50:0:0:0: [sr1] scsi-1 drive [ 203.280973] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 203.303714] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.312103] sr 49:0:0:0: Power-on or device reset occurred [ 203.334437] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.337675] sr 49:0:0:0: [sr2] scsi-1 drive [ 203.342815] sr 51:0:0:0: Power-on or device reset occurred [ 203.369116] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 203.372336] sr 51:0:0:0: [sr3] scsi-1 drive [ 203.383939] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.392427] sr 48:0:0:0: Power-on or device reset occurred [ 203.402442] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 203.418011] sr 48:0:0:0: [sr0] scsi-1 drive [ 203.433835] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.442179] sr 50:0:0:0: Power-on or device reset occurred [ 203.451485] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 203.467758] sr 50:0:0:0: [sr1] scsi-1 drive [ 203.496624] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 203.583744] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.592139] sr 51:0:0:0: Power-on or device reset occurred [ 203.617718] sr 51:0:0:0: [sr3] scsi-1 drive [ 203.624033] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.632547] sr 48:0:0:0: Power-on or device reset occurred [ 203.634604] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.646486] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 203.648297] sr 49:0:0:0: Power-on or device reset occurred [ 203.658119] sr 48:0:0:0: [sr0] scsi-1 drive [ 203.677456] sr 49:0:0:0: [sr2] scsi-1 drive [ 203.684538] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 203.703702] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 203.713928] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.722450] sr 50:0:0:0: Power-on or device reset occurred [ 203.748014] sr 50:0:0:0: [sr1] scsi-1 drive [ 203.776091] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 203.883842] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.884188] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.892214] sr 48:0:0:0: Power-on or device reset occurred [ 203.900429] sr 51:0:0:0: Power-on or device reset occurred [ 203.925753] sr 48:0:0:0: [sr0] scsi-1 drive [ 203.931214] sr 51:0:0:0: [sr3] scsi-1 drive [ 203.952558] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 203.953799] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.954529] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 203.954928] sr 49:0:0:0: Power-on or device reset occurred [ 203.966451] sr 50:0:0:0: Power-on or device reset occurred [ 203.974471] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 203.975015] sr 49:0:0:0: [sr1] scsi-1 drive [ 203.999622] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 204.005054] sr 50:0:0:0: [sr2] scsi-1 drive [ 204.019816] sr 49:0:0:0: [sr1] Hmm, seems the drive doesn't support multisession CD's [ 204.033387] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 204.223702] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.224029] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.232069] sr 49:0:0:0: Power-on or device reset occurred [ 204.240276] sr 48:0:0:0: Power-on or device reset occurred [ 204.265594] sr 49:0:0:0: [sr0] scsi-1 drive [ 204.271086] sr 48:0:0:0: [sr1] scsi-1 drive [ 204.274606] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.283784] sr 51:0:0:0: Power-on or device reset occurred [ 204.284283] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.297547] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 204.297646] sr 50:0:0:0: Power-on or device reset occurred [ 204.303014] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 204.309950] sr 51:0:0:0: [sr3] scsi-1 drive [ 204.312550] sr 49:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 204.319275] sr 48:0:0:0: [sr1] Hmm, seems the drive doesn't support multisession CD's [ 204.336001] sr 50:0:0:0: [sr2] scsi-1 drive [ 204.350935] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 204.363987] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 204.563874] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.563898] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.564242] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.564452] sr 49:0:0:0: Power-on or device reset occurred [ 204.572225] sr 48:0:0:0: Power-on or device reset occurred [ 204.580536] sr 50:0:0:0: Power-on or device reset occurred [ 204.584519] sr 49:0:0:0: [sr0] scsi-1 drive [ 204.609231] sr 48:0:0:0: [sr1] scsi-1 drive [ 204.609352] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 204.629225] sr 50:0:0:0: [sr2] scsi-1 drive [ 204.636485] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 204.657199] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 204.674424] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.684336] sr 51:0:0:0: Power-on or device reset occurred [ 204.709915] sr 51:0:0:0: [sr3] scsi-1 drive [ 204.737987] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 204.813853] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.813858] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.822217] sr 49:0:0:0: Power-on or device reset occurred [ 204.830408] sr 50:0:0:0: Power-on or device reset occurred [ 204.855743] sr 49:0:0:0: [sr0] scsi-1 drive [ 204.861215] sr 50:0:0:0: [sr2] scsi-1 drive [ 204.882206] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 204.889951] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 204.964037] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.964622] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 204.972544] sr 51:0:0:0: Power-on or device reset occurred [ 204.980731] sr 48:0:0:0: Power-on or device reset occurred [ 205.005941] sr 51:0:0:0: [sr1] scsi-1 drive [ 205.011443] sr 48:0:0:0: [sr3] scsi-1 drive [ 205.033883] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 205.039462] sr 48:0:0:0: Attached scsi generic sg2 type 5 [ 205.073915] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.082298] sr 50:0:0:0: Power-on or device reset occurred [ 205.107873] sr 50:0:0:0: [sr2] scsi-1 drive [ 205.136110] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 205.154486] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.162995] sr 49:0:0:0: Power-on or device reset occurred [ 205.188598] sr 49:0:0:0: [sr0] scsi-1 drive [ 205.218414] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 205.263969] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.272487] sr 48:0:0:0: Power-on or device reset occurred [ 205.298053] sr 48:0:0:0: [sr3] scsi-1 drive [ 205.324913] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 205.353995] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.354582] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.362536] sr 50:0:0:0: Power-on or device reset occurred [ 205.370816] sr 51:0:0:0: Power-on or device reset occurred [ 205.393979] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.395899] sr 50:0:0:0: [sr1] scsi-1 drive [ 205.401418] sr 51:0:0:0: [sr2] scsi-1 drive [ 205.402338] sr 49:0:0:0: Power-on or device reset occurred [ 205.423969] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 205.429926] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 205.439404] sr 49:0:0:0: [sr0] scsi-1 drive [ 205.466080] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 205.513897] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.522275] sr 48:0:0:0: Power-on or device reset occurred [ 205.547859] sr 48:0:0:0: [sr3] scsi-1 drive [ 205.576512] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 205.624445] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.632861] sr 49:0:0:0: Power-on or device reset occurred [ 205.634058] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.634172] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.634418] sr 51:0:0:0: Power-on or device reset occurred [ 205.654459] sr 51:0:0:0: [sr1] scsi-1 drive [ 205.654872] sr 50:0:0:0: Power-on or device reset occurred [ 205.658423] sr 49:0:0:0: [sr0] scsi-1 drive [ 205.682586] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 205.688193] sr 49:0:0:0: Attached scsi generic sg2 type 5 [ 205.694043] sr 50:0:0:0: [sr2] scsi-1 drive [ 205.720784] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 205.813831] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.822207] sr 48:0:0:0: Power-on or device reset occurred [ 205.847778] sr 48:0:0:0: [sr3] scsi-1 drive [ 205.874111] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 205.893891] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 205.902256] sr 49:0:0:0: Power-on or device reset occurred [ 205.927817] sr 49:0:0:0: [sr0] scsi-1 drive [ 205.955065] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 206.014078] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.022594] sr 50:0:0:0: Power-on or device reset occurred [ 206.044227] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.048161] sr 50:0:0:0: [sr2] scsi-1 drive [ 206.052602] sr 51:0:0:0: Power-on or device reset occurred [ 206.078891] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 206.082162] sr 51:0:0:0: [sr1] scsi-1 drive [ 206.112154] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 206.153629] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.161977] sr 48:0:0:0: Power-on or device reset occurred [ 206.187533] sr 48:0:0:0: [sr3] scsi-1 drive [ 206.203862] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.212294] sr 49:0:0:0: Power-on or device reset occurred [ 206.217935] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 206.237879] sr 49:0:0:0: [sr0] scsi-1 drive [ 206.264895] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 206.333943] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.342451] sr 51:0:0:0: Power-on or device reset occurred [ 206.368046] sr 51:0:0:0: [sr1] scsi-1 drive [ 206.394704] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 206.404038] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.404536] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.412543] sr 48:0:0:0: Power-on or device reset occurred [ 206.420583] sr 50:0:0:0: Power-on or device reset occurred [ 206.445967] sr 48:0:0:0: [sr2] scsi-1 drive [ 206.451408] sr 50:0:0:0: [sr3] scsi-1 drive [ 206.472766] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 206.482111] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 206.523970] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.532327] sr 49:0:0:0: Power-on or device reset occurred [ 206.557906] sr 49:0:0:0: [sr0] scsi-1 drive [ 206.584579] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 206.674493] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.682868] sr 51:0:0:0: Power-on or device reset occurred [ 206.708439] sr 51:0:0:0: [sr1] scsi-1 drive [ 206.733968] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.742414] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 206.742467] sr 50:0:0:0: Power-on or device reset occurred [ 206.773354] sr 50:0:0:0: [sr3] scsi-1 drive [ 206.802659] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 206.853905] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.854226] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.862267] sr 49:0:0:0: Power-on or device reset occurred [ 206.870463] sr 48:0:0:0: Power-on or device reset occurred [ 206.895802] sr 49:0:0:0: [sr0] scsi-1 drive [ 206.901278] sr 48:0:0:0: [sr2] scsi-1 drive [ 206.923663] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 206.929871] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 206.983873] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 206.992229] sr 50:0:0:0: Power-on or device reset occurred [ 207.017788] sr 50:0:0:0: [sr3] scsi-1 drive [ 207.044196] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.052548] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 207.053455] sr 51:0:0:0: Power-on or device reset occurred [ 207.065845] sr 50:0:0:0: [sr3] Hmm, seems the drive doesn't support multisession CD's [ 207.083495] sr 51:0:0:0: [sr1] scsi-1 drive [ 207.111965] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 207.123769] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.132161] sr 48:0:0:0: Power-on or device reset occurred [ 207.157695] sr 48:0:0:0: [sr2] scsi-1 drive [ 207.184010] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 207.184566] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.197944] sr 49:0:0:0: Power-on or device reset occurred [ 207.223553] sr 49:0:0:0: [sr0] scsi-1 drive [ 207.250194] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 207.284314] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.292824] sr 50:0:0:0: Power-on or device reset occurred [ 207.318380] sr 50:0:0:0: [sr3] scsi-1 drive [ 207.323907] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.332416] sr 51:0:0:0: Power-on or device reset occurred [ 207.344785] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 207.360207] sr 51:0:0:0: [sr1] scsi-1 drive [ 207.386689] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 207.454020] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.462684] sr 49:0:0:0: Power-on or device reset occurred [ 207.488236] sr 49:0:0:0: [sr0] scsi-1 drive [ 207.504160] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.512505] sr 48:0:0:0: Power-on or device reset occurred [ 207.518093] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 207.538048] sr 48:0:0:0: [sr2] scsi-1 drive [ 207.543909] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.543981] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.552421] sr 51:0:0:0: Power-on or device reset occurred [ 207.560652] sr 50:0:0:0: Power-on or device reset occurred [ 207.574459] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 207.585795] sr 51:0:0:0: [sr1] scsi-1 drive [ 207.591304] sr 50:0:0:0: [sr3] scsi-1 drive [ 207.615067] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 207.620604] sr 50:0:0:0: Attached scsi generic sg2 type 5 [ 207.743811] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.752159] sr 49:0:0:0: Power-on or device reset occurred [ 207.777729] sr 49:0:0:0: [sr0] scsi-1 drive [ 207.803826] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.812165] sr 50:0:0:0: Power-on or device reset occurred [ 207.812916] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 207.824598] sr 49:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 207.837714] sr 50:0:0:0: [sr3] scsi-1 drive [ 207.864370] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.864437] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 207.865827] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 207.872870] sr 51:0:0:0: Power-on or device reset occurred [ 207.881048] sr 48:0:0:0: Power-on or device reset occurred [ 207.906176] sr 51:0:0:0: [sr1] scsi-1 drive [ 207.917155] sr 48:0:0:0: [sr2] scsi-1 drive [ 207.934131] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 207.945230] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 207.994032] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.002407] sr 49:0:0:0: Power-on or device reset occurred [ 208.027955] sr 49:0:0:0: [sr0] scsi-1 drive [ 208.053749] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 208.114487] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.123078] sr 50:0:0:0: Power-on or device reset occurred [ 208.148635] sr 50:0:0:0: [sr3] scsi-1 drive [ 208.153886] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.162392] sr 48:0:0:0: Power-on or device reset occurred [ 208.174334] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.182682] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 208.182844] sr 51:0:0:0: Power-on or device reset occurred [ 208.192570] sr 48:0:0:0: [sr2] scsi-1 drive [ 208.196711] sr 50:0:0:0: [sr3] Hmm, seems the drive doesn't support multisession CD's [ 208.213633] sr 51:0:0:0: [sr1] scsi-1 drive [ 208.221531] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 208.223868] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.235515] sr 49:0:0:0: Power-on or device reset occurred [ 208.241089] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 208.261091] sr 49:0:0:0: [sr0] scsi-1 drive [ 208.287499] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 208.443872] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.452379] sr 51:0:0:0: Power-on or device reset occurred [ 208.463849] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.472215] sr 49:0:0:0: Power-on or device reset occurred [ 208.474322] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.477959] sr 51:0:0:0: [sr1] scsi-1 drive [ 208.486205] sr 50:0:0:0: Power-on or device reset occurred [ 208.496867] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.500095] sr 49:0:0:0: [sr0] scsi-1 drive [ 208.505388] sr 48:0:0:0: Power-on or device reset occurred [ 208.519516] sr 50:0:0:0: [sr3] scsi-1 drive [ 208.526282] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 208.534788] sr 48:0:0:0: [sr2] scsi-1 drive [ 208.539095] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 208.556120] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 208.563573] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 208.773907] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.782416] sr 49:0:0:0: Power-on or device reset occurred [ 208.783607] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.796260] sr 48:0:0:0: Power-on or device reset occurred [ 208.804440] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.807987] sr 49:0:0:0: [sr0] scsi-1 drive [ 208.812950] sr 51:0:0:0: Power-on or device reset occurred [ 208.822437] sr 48:0:0:0: [sr2] scsi-1 drive [ 208.840514] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 208.842344] sr 51:0:0:0: [sr1] scsi-1 drive [ 208.851199] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 208.858368] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 208.867241] sr 50:0:0:0: Power-on or device reset occurred [ 208.872830] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 208.892817] sr 50:0:0:0: [sr3] scsi-1 drive [ 208.919001] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 209.075086] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.083638] sr 51:0:0:0: Power-on or device reset occurred [ 209.109182] sr 51:0:0:0: [sr1] scsi-1 drive [ 209.123894] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.132248] sr 50:0:0:0: Power-on or device reset occurred [ 209.137841] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 209.157797] sr 50:0:0:0: [sr3] scsi-1 drive [ 209.164440] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.172943] sr 49:0:0:0: Power-on or device reset occurred [ 209.185810] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 209.198505] sr 49:0:0:0: [sr0] scsi-1 drive [ 209.224783] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.233300] sr 48:0:0:0: Power-on or device reset occurred [ 209.233458] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 209.254230] sr 49:0:0:0: [sr0] Hmm, seems the drive doesn't support multisession CD's [ 209.258860] sr 48:0:0:0: [sr2] scsi-1 drive [ 209.289959] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 209.343871] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.343884] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.352283] sr 51:0:0:0: Power-on or device reset occurred [ 209.360570] sr 50:0:0:0: Power-on or device reset occurred [ 209.385758] sr 51:0:0:0: [sr1] scsi-1 drive [ 209.391259] sr 50:0:0:0: [sr3] scsi-1 drive [ 209.413839] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 209.420318] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 209.469013] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.478282] sr 48:0:0:0: Power-on or device reset occurred [ 209.503809] sr 48:0:0:0: [sr2] scsi-1 drive [ 209.533369] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 209.543934] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.552320] sr 49:0:0:0: Power-on or device reset occurred [ 209.577872] sr 49:0:0:0: [sr0] scsi-1 drive [ 209.607020] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 209.713962] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.722469] sr 48:0:0:0: Power-on or device reset occurred [ 209.748036] sr 48:0:0:0: [sr2] scsi-1 drive [ 209.754358] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.754696] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.762744] sr 51:0:0:0: Power-on or device reset occurred [ 209.771057] sr 50:0:0:0: Power-on or device reset occurred [ 209.774390] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 209.796246] sr 51:0:0:0: [sr1] scsi-1 drive [ 209.807137] sr 50:0:0:0: [sr3] scsi-1 drive [ 209.825233] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 209.836258] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 209.853749] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 209.862130] sr 49:0:0:0: Power-on or device reset occurred [ 209.887722] sr 49:0:0:0: [sr0] scsi-1 drive [ 209.915732] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 210.023997] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.032509] sr 50:0:0:0: Power-on or device reset occurred [ 210.054568] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.058094] sr 50:0:0:0: [sr3] scsi-1 drive [ 210.063263] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.064232] sr 48:0:0:0: Power-on or device reset occurred [ 210.080860] sr 51:0:0:0: Power-on or device reset occurred [ 210.084299] sr 48:0:0:0: [sr1] scsi-1 drive [ 210.092100] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 210.106429] sr 51:0:0:0: [sr2] scsi-1 drive [ 210.112750] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 210.133732] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.142123] sr 49:0:0:0: Power-on or device reset occurred [ 210.142163] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 210.154615] sr 51:0:0:0: [sr2] Hmm, seems the drive doesn't support multisession CD's [ 210.167718] sr 49:0:0:0: [sr0] scsi-1 drive [ 210.196385] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 210.343930] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.344160] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.352303] sr 50:0:0:0: Power-on or device reset occurred [ 210.360754] sr 48:0:0:0: Power-on or device reset occurred [ 210.373909] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.382436] sr 51:0:0:0: Power-on or device reset occurred [ 210.385820] sr 50:0:0:0: [sr1] scsi-1 drive [ 210.391342] sr 48:0:0:0: [sr3] scsi-1 drive [ 210.408008] sr 51:0:0:0: [sr2] scsi-1 drive [ 210.414204] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.420129] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 210.423676] sr 49:0:0:0: Power-on or device reset occurred [ 210.427952] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 210.438752] sr 51:0:0:0: Attached scsi generic sg2 type 5 [ 210.442866] sr 50:0:0:0: [sr1] Hmm, seems the drive doesn't support multisession CD's [ 210.453310] sr 49:0:0:0: [sr0] scsi-1 drive [ 210.479266] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 210.584101] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.592502] sr 48:0:0:0: Power-on or device reset occurred [ 210.618056] sr 48:0:0:0: [sr3] scsi-1 drive [ 210.633811] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.642181] sr 50:0:0:0: Power-on or device reset occurred [ 210.647809] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 210.667737] sr 50:0:0:0: [sr1] scsi-1 drive [ 210.673903] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.682333] sr 49:0:0:0: Power-on or device reset occurred [ 210.695827] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 210.707900] sr 49:0:0:0: [sr0] scsi-1 drive [ 210.737626] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 210.754414] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.762921] sr 51:0:0:0: Power-on or device reset occurred [ 210.788546] sr 51:0:0:0: [sr2] scsi-1 drive [ 210.816515] sr 51:0:0:0: Attached scsi generic sg1 type 5 [ 210.854086] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.862666] sr 48:0:0:0: Power-on or device reset occurred [ 210.888259] sr 48:0:0:0: [sr3] scsi-1 drive [ 210.918058] sr 48:0:0:0: Attached scsi generic sg1 type 5 [ 210.933641] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.933823] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 210.942045] sr 49:0:0:0: Power-on or device reset occurred [ 210.950336] sr 50:0:0:0: Power-on or device reset occurred [ 210.975560] sr 49:0:0:0: [sr0] scsi-1 drive [ 210.981018] sr 50:0:0:0: [sr1] scsi-1 drive [ 211.003603] sr 49:0:0:0: Attached scsi generic sg1 type 5 [ 211.009644] sr 50:0:0:0: Attached scsi generic sg2 type 5 [ 213.302333] run blktests block/006 at 2022-09-28 00:00:30 [ 213.321355] null_blk: disk nullb0 created [ 213.325372] null_blk: module loaded [ 228.560546] run blktests block/016 at 2022-09-28 00:00:45 [ 230.583696] null_blk: disk nullb0 created [ 230.587706] null_blk: module loaded [ 234.210118] run blktests block/017 at 2022-09-28 00:00:51 [ 234.734620] null_blk: disk nullb0 created [ 234.738624] null_blk: module loaded [ 235.944685] run blktests block/018 at 2022-09-28 00:00:53 [ 236.972991] null_blk: disk nullb0 created [ 236.977003] null_blk: module loaded [ 240.589378] run blktests block/021 at 2022-09-28 00:00:57 [ 240.610594] null_blk: disk nullb0 created [ 240.614622] null_blk: module loaded [ 295.156400] run blktests block/023 at 2022-09-28 00:01:52 [ 295.175582] null_blk: disk nullb0 created [ 295.179582] null_blk: module loaded [ 295.544518] null_blk: disk nullb0 created [ 295.548530] null_blk: module loaded [ 296.230365] run blktests loop/001 at 2022-09-28 00:01:53 [ 296.260900] loop: module loaded [ 296.266585] loop0: detected capacity change from 0 to 2097152 [ 296.272880] loop0: p1 p2 [ 296.830195] run blktests loop/003 at 2022-09-28 00:01:54 [ 297.208410] run blktests loop/005 at 2022-09-28 00:01:54 [ 297.222356] loop0: detected capacity change from 0 to 2048 [ 297.303421] I/O error, dev loop0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 [ 297.652681] run blktests nvme/004 at 2022-09-28 00:01:54 [ 297.695371] loop0: detected capacity change from 0 to 2097152 [ 297.704965] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 297.718035] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 297.732511] nvme nvme0: creating 128 I/O queues. [ 297.746419] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 298.791893] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 306.568065] run blktests nvme/006 at 2022-09-28 00:02:03 [ 306.600175] loop0: detected capacity change from 0 to 2097152 [ 306.610560] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 307.378385] run blktests nvme/007 at 2022-09-28 00:02:04 [ 307.409721] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 308.200946] run blktests nvme/008 at 2022-09-28 00:02:05 [ 308.229852] loop0: detected capacity change from 0 to 2097152 [ 308.239627] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 308.256306] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 308.270812] nvme nvme0: creating 128 I/O queues. [ 308.286219] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 309.367954] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 317.018368] run blktests nvme/009 at 2022-09-28 00:02:14 [ 317.049441] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 317.065503] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 317.079991] nvme nvme0: creating 128 I/O queues. [ 317.093180] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 318.157961] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 326.461126] run blktests nvme/010 at 2022-09-28 00:02:23 [ 326.488362] loop0: detected capacity change from 0 to 2097152 [ 326.497919] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 326.515349] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 326.529803] nvme nvme0: creating 128 I/O queues. [ 326.543311] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 334.511466] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 344.023688] run blktests nvme/011 at 2022-09-28 00:02:41 [ 344.055550] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 344.072197] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 344.086621] nvme nvme0: creating 128 I/O queues. [ 344.100623] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 360.644639] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 369.919208] run blktests nvme/012 at 2022-09-28 00:03:07 [ 369.949617] loop0: detected capacity change from 0 to 2097152 [ 369.959722] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 369.977607] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 369.992205] nvme nvme0: creating 128 I/O queues. [ 370.009073] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 380.505244] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 389.961724] run blktests nvme/013 at 2022-09-28 00:03:27 [ 389.988804] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 390.006116] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 390.020626] nvme nvme0: creating 128 I/O queues. [ 390.036119] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 391.089148] ISOFS: unsupported/invalid hardware sector size 4096 [ 400.225033] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 408.793397] run blktests nvme/014 at 2022-09-28 00:03:46 [ 408.823431] loop0: detected capacity change from 0 to 2097152 [ 408.832950] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 408.848842] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 408.863333] nvme nvme0: creating 128 I/O queues. [ 408.876582] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 411.926906] nvme nvme0: using deprecated NVME_IOCTL_IO_CMD ioctl on the char device! [ 412.917951] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1" [ 421.236297] run blktests nvme/015 at 2022-09-28 00:03:58 [ 421.267476] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 421.283974] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-d85ed38fa4a4. [ 421.298402] nvme nvme0: creating 128 I/O queues. [ 421.312469] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 451.111759] watchdog: BUG: soft lockup - CPU#8 stuck for 26s! [kworker/8:12:19399] [ 451.119320] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 451.119345] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 451.207140] CPU: 8 PID: 19399 Comm: kworker/8:12 Not tainted 6.0.0-rc7 #1 [ 451.213915] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 451.221758] watchdog: BUG: soft lockup - CPU#68 stuck for 26s! [swapper/68:0] [ 451.223206] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 451.230328] Modules linked in: nvme_loop nvme_fabrics nvmet [ 451.236495] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 451.242054] nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls [ 451.249001] pc : __sbq_wake_up+0x84/0x1f0 [ 451.255518] rfkill sunrpc [ 451.259514] lr : sbitmap_queue_clear+0x74/0xc0 [ 451.259515] vfat fat [ 451.266638] sp : ffff80000d79b280 [ 451.266638] acpi_ipmi [ 451.268898] x29: ffff80000d79b280 [ 451.272199] ipmi_ssif [ 451.274545] x28: ffffd5becb6aec50 [ 451.277847] cppc_cpufreq [ 451.280192] x27: ffffd5beca4f7000 [ 451.283580] ipmi_devintf [ 451.286186] [ 451.289574] igb [ 451.292180] x26: ffffd5becaf68a98 [ 451.293658] ipmi_msghandler [ 451.295483] x25: ffffd5becb6a60c0 [ 451.298784] arm_cmn [ 451.301651] x24: fffffbffef180dc0 [ 451.305038] arm_dmc620_pmu [ 451.307211] [ 451.310598] arm_spe_pmu [ 451.313379] x23: ffff07ff81d381b0 [ 451.314857] arm_dsu_pmu [ 451.317377] x22: ffff80000d79b30c [ 451.320679] acpi_tad [ 451.323199] x21: ffff07ff84e93c00 [ 451.326586] fuse [ 451.328845] [ 451.332233] zram [ 451.334145] x20: 00000000000001c0 [ 451.335622] xfs [ 451.337534] x19: ffff07ff91ccdb10 [ 451.340835] crct10dif_ce [ 451.342660] x18: ffffffffffffffff [ 451.346048] polyval_ce [ 451.348654] [ 451.352042] ast [ 451.354474] x17: ffff32506c412000 [ 451.355952] polyval_generic [ 451.357778] x16: ffff800008044000 [ 451.361080] drm_vram_helper [ 451.363947] x15: 0000000000004000 [ 451.367334] ghash_ce [ 451.370201] [ 451.373588] drm_ttm_helper [ 451.375847] x14: 010320bb28ec0000 [ 451.377325] sbsa_gwdt [ 451.380105] x13: 0000000000000087 [ 451.383406] ttm [ 451.385752] x12: 071c71c71c71c71c [ 451.389140] xgene_hwmon [ 451.390965] [ 451.394352] Unloaded tainted modules: [ 451.396872] x11: 0000000000000040 [ 451.398350] nvmet():1 [ 451.401997] x10: ffff07ff801a9c70 [ 451.405298] nvme_fabrics():1 [ 451.407644] x9 : ffffd5bec9545fd8 [ 451.411032] nvme_loop():1 [ 451.413985] [ 451.417373] nvmet():1 [ 451.420066] x8 : ffff07ff80404b70 [ 451.421544] nvme_fabrics():1 [ 451.423890] x7 : ffff07ff84e93dc8 [ 451.427191] nvme_loop():1 [ 451.430144] x6 : 0000000000000008 [ 451.433532] nvmet():1 [ 451.436225] [ 451.439612] nvme_fabrics():1 [ 451.441958] x5 : ffff80000d043790 [ 451.443436] nvme_loop():1 [ 451.446390] x4 : ffff07ff84e93dc0 [ 451.449691] nvmet():1 [ 451.452384] x3 : 0000000000000008 [ 451.455771] nvme_fabrics():1 [ 451.458117] [ 451.461505] nvme_loop():1 [ 451.464458] x2 : 0000000000000007 [ 451.465936] nvmet():1 [ 451.468629] x1 : ffff80000d79b30c [ 451.471930] nvme_fabrics():1 [ 451.474276] x0 : ffff07ff84e93dd0 [ 451.477665] nvme_loop():1 [ 451.480619] [ 451.484006] nvmet():1 [ 451.486699] Call trace: [ 451.488178] nvme_fabrics():1 [ 451.490525] __sbq_wake_up+0x84/0x1f0 [ 451.492958] nvme_loop():1 [ 451.495912] sbitmap_queue_clear+0x74/0xc0 [ 451.499560] nvmet():1 [ 451.502253] blk_mq_put_tag+0x44/0x70 [ 451.506335] nvme_fabrics():1 [ 451.508681] __blk_mq_free_request+0x88/0xf0 [ 451.512330] nvme_loop():1 [ 451.515283] blk_mq_free_request+0xc0/0x160 [ 451.519539] nvmet():1 [ 451.522232] __blk_mq_end_request+0x78/0x150 [ 451.526401] nvme_fabrics():1 [ 451.528747] scsi_end_request+0x114/0x260 [ 451.533002] nvme_loop():1 [ 451.535956] scsi_io_completion+0x64/0x174 [ 451.539951] nvmet():1 [ 451.542644] scsi_finish_command+0xe0/0x120 [ 451.546726] nvme_fabrics():1 [ 451.549072] scsi_complete+0x8c/0x1c4 [ 451.553241] nvme_loop():1 [ 451.556194] blk_complete_reqs+0x5c/0x74 [ 451.559843] nvmet():1 [ 451.562536] blk_done_softirq+0x38/0x4c [ 451.566444] nvme_fabrics():1 [ 451.568790] __do_softirq+0x168/0x418 [ 451.572612] nvme_loop():1 [ 451.575565] __irq_exit_rcu+0x120/0x170 [ 451.579213] null_blk():1 [ 451.581906] irq_exit_rcu+0x18/0x24 [ 451.585728] null_blk():1 [ 451.588334] el1_interrupt+0x38/0x70 [ 451.591809] null_blk():1 [ 451.594415] el1h_64_irq_handler+0x18/0x24 [ 451.597977] null_blk():1 [ 451.600585] el1h_64_irq+0x68/0x6c [ 451.604667] null_blk():1 [ 451.607273] _raw_spin_unlock_irqrestore+0x20/0x50 [ 451.610661] null_blk():1 [ 451.613267] __wake_up_common_lock+0x84/0xac [ 451.618043] null_blk():1 [ 451.620650] __wake_up+0x20/0x30 [ 451.624905] scsi_debug():1 [ 451.627511] __sbq_wake_up+0x13c/0x1f0 [ 451.630726] scsi_debug():1 [ 451.633505] sbitmap_queue_clear+0x74/0xc0 [ 451.637240] ipmi_si():1 [ 451.640020] blk_mq_put_tag+0x44/0x70 [ 451.644102] ipmi_si():1 [ 451.646622] __blk_mq_free_request+0x88/0xf0 [ 451.650270] [last unloaded: nvmet] [ 451.652789] blk_mq_free_request+0xc0/0x160 [ 451.657045] [ 451.660519] bfq_insert_request.isra.0+0x204/0x270 [ 451.664688] CPU: 68 PID: 0 Comm: swapper/68 Not tainted 6.0.0-rc7 #1 [ 451.666166] bfq_insert_requests+0x70/0x90 [ 451.670942] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 451.677281] blk_mq_sched_insert_requests+0xac/0x250 [ 451.681364] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 451.690654] blk_mq_flush_plug_list+0x110/0x2cc [ 451.695604] pc : __sbq_wake_up+0x58/0x1f0 [ 451.702550] __blk_flush_plug+0x100/0x140 [ 451.707066] lr : sbitmap_queue_clear+0x74/0xc0 [ 451.711061] blk_finish_plug+0x40/0x25c [ 451.715057] sp : ffff8000088438b0 [ 451.719486] __iomap_dio_rw+0x1b4/0x500 [ 451.723307] x29: ffff8000088438b0 [ 451.726608] iomap_dio_rw+0x18/0x50 [ 451.730430] x28: ffffd5becb6aec50 [ 451.733731] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 451.737205] x27: ffffd5beca4f7000 [ 451.740593] xfs_file_write_iter+0xf4/0x15c [xfs] [ 451.745890] [ 451.749278] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 451.753969] x26: ffffd5becaf68a98 [ 451.755447] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 451.760484] x25: ffffd5becb6a60c0 [ 451.763785] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 451.768909] x24: fffffbffef8c4dc0 [ 451.772297] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 451.777420] [ 451.780808] process_one_work+0x1ec/0x470 [ 451.786192] x23: ffff07ff81d381b0 [ 451.787670] worker_thread+0x74/0x410 [ 451.791665] x22: ffff80000884393c [ 451.794966] kthread+0xec/0x100 [ 451.798614] x21: ffff07ff84e93c00 [ 451.802001] ret_from_fork+0x10/0x20 [ 451.805128] [ 451.813556] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 451.820678] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 451.827800] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 451.834922] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 451.842044] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 451.849166] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 451.856288] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 451.863410] Call trace: [ 451.865843] __sbq_wake_up+0x58/0x1f0 [ 451.869492] sbitmap_queue_clear+0x74/0xc0 [ 451.873576] blk_mq_put_tag+0x44/0x70 [ 451.877226] __blk_mq_free_request+0x88/0xf0 [ 451.881483] blk_mq_free_request+0xc0/0x160 [ 451.885653] __blk_mq_end_request+0x78/0x150 [ 451.889910] scsi_end_request+0x114/0x260 [ 451.893907] scsi_io_completion+0x64/0x174 [ 451.897990] scsi_finish_command+0xe0/0x120 [ 451.902161] scsi_complete+0x8c/0x1c4 [ 451.905811] blk_complete_reqs+0x5c/0x74 [ 451.909721] blk_done_softirq+0x38/0x4c [ 451.913543] __do_softirq+0x168/0x418 [ 451.917193] __irq_exit_rcu+0x120/0x170 [ 451.921016] irq_exit_rcu+0x18/0x24 [ 451.924492] el1_interrupt+0x38/0x70 [ 451.928054] el1h_64_irq_handler+0x18/0x24 [ 451.932137] el1h_64_irq+0x68/0x6c [ 451.935526] cpuidle_enter_state+0xc4/0x4e0 [ 451.939698] cpuidle_enter+0x40/0x60 [ 451.943260] cpuidle_idle_call+0x118/0x1b0 [ 451.947345] do_idle+0xac/0x100 [ 451.950474] cpu_startup_entry+0x30/0x34 [ 451.954385] secondary_start_kernel+0xd8/0xf0 [ 451.958729] __secondary_switched+0xb0/0xb4 [ 472.051686] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 2109 jiffies s: 2093 root: 0x11/. [ 472.063263] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 472.072482] Task dump for CPU 8: [ 472.075700] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 472.085611] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 472.091792] Call trace: [ 472.094228] __switch_to+0xd4/0x100 [ 472.097711] 0xffff6c0200000080 [ 472.100844] Task dump for CPU 68: [ 472.104159] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 472.114069] Call trace: [ 472.116503] __switch_to+0xd4/0x100 [ 472.119984] 0x0 [ 475.111612] watchdog: BUG: soft lockup - CPU#8 stuck for 48s! [kworker/8:12:19399] [ 475.119168] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 475.119185] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 475.206973] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 475.215223] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 475.221611] watchdog: BUG: soft lockup - CPU#68 stuck for 48s! [swapper/68:0] [ 475.224514] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 475.231634] Modules linked in: [ 475.231634] [ 475.231635] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 475.237800] nvme_loop [ 475.240841] pc : __sbq_wake_up+0x84/0x1f0 [ 475.242319] nvme_fabrics [ 475.249265] lr : sbitmap_queue_clear+0x74/0xc0 [ 475.251611] nvmet [ 475.255606] sp : ffff80000d79b280 [ 475.258213] nls_utf8 [ 475.262643] x29: ffff80000d79b280 [ 475.264641] hfsplus [ 475.267942] x28: ffffd5becb6aec50 [ 475.270202] hfs [ 475.273502] x27: ffffd5beca4f7000 [ 475.275675] isofs [ 475.279062] [ 475.280887] nvme_core [ 475.284275] x26: ffffd5becaf68a98 [ 475.286274] nvme_common [ 475.287751] x25: ffffd5becb6a60c0 [ 475.290097] loop [ 475.293398] x24: fffffbffef180dc0 [ 475.295918] tls [ 475.299305] [ 475.301217] rfkill [ 475.304605] x23: ffff07ff81d381b0 [ 475.306430] sunrpc [ 475.307908] x22: ffff80000d79b30c [ 475.309993] vfat [ 475.313294] x21: ffff07ff84e93c00 [ 475.315380] fat [ 475.318769] [ 475.320681] acpi_ipmi [ 475.324070] x20: 00000000000001c0 [ 475.325895] ipmi_ssif [ 475.327373] x19: ffff07ff91ccdb10 [ 475.329719] cppc_cpufreq [ 475.333020] x18: ffffffffffffffff [ 475.335366] ipmi_devintf [ 475.338753] [ 475.341360] igb [ 475.344748] x17: ffff32506c412000 [ 475.347354] ipmi_msghandler [ 475.348832] x16: ffff800008044000 [ 475.350657] arm_cmn [ 475.353958] x15: 0000000000004000 [ 475.356825] arm_dmc620_pmu [ 475.360213] [ 475.362385] arm_spe_pmu [ 475.365772] x14: 010320bb28ec0000 [ 475.368552] arm_dsu_pmu [ 475.370030] x13: 0000000000000087 [ 475.372550] acpi_tad [ 475.375851] x12: 071c71c71c71c71c [ 475.378370] fuse [ 475.381758] [ 475.384017] zram [ 475.387405] x11: 0000000000000040 [ 475.389317] xfs [ 475.390795] x10: ffff07ff801a9c70 [ 475.392707] crct10dif_ce [ 475.396007] x9 : ffffd5bec9545fd8 [ 475.397832] polyval_ce [ 475.401220] [ 475.403827] ast [ 475.407214] x8 : ffff07ff80404b70 [ 475.409647] polyval_generic [ 475.411125] x7 : ffff07ff84e93dc8 [ 475.412950] drm_vram_helper [ 475.416251] x6 : 0000000000000008 [ 475.419118] ghash_ce [ 475.422506] [ 475.425372] drm_ttm_helper [ 475.428760] x5 : ffff80000d043790 [ 475.431019] sbsa_gwdt [ 475.432497] x4 : ffff07ff84e93dc0 [ 475.435278] ttm [ 475.438580] x3 : 0000000000000008 [ 475.440928] xgene_hwmon [ 475.444315] [ 475.446141] Unloaded tainted modules: [ 475.449528] x2 : 0000000000000007 [ 475.452048] nvmet():1 [ 475.453525] x1 : ffff80000d79b30c [ 475.457173] nvme_fabrics():1 [ 475.460474] x0 : ffff07ff84e93dd0 [ 475.462820] nvme_loop():1 [ 475.466208] [ 475.469161] nvmet():1 [ 475.472549] Call trace: [ 475.475242] nvme_fabrics():1 [ 475.476720] __sbq_wake_up+0x84/0x1f0 [ 475.479066] nvme_loop():1 [ 475.481499] sbitmap_queue_clear+0x74/0xc0 [ 475.484453] nvmet():1 [ 475.488101] blk_mq_put_tag+0x44/0x70 [ 475.490794] nvme_fabrics():1 [ 475.494876] __blk_mq_free_request+0x88/0xf0 [ 475.497222] nvme_loop():1 [ 475.500870] blk_mq_free_request+0xc0/0x160 [ 475.503823] nvmet():1 [ 475.508079] __blk_mq_end_request+0x78/0x150 [ 475.510772] nvme_fabrics():1 [ 475.514941] scsi_end_request+0x114/0x260 [ 475.517287] nvme_loop():1 [ 475.521543] scsi_io_completion+0x64/0x174 [ 475.524496] nvmet():1 [ 475.528492] scsi_finish_command+0xe0/0x120 [ 475.531185] nvme_fabrics():1 [ 475.535267] scsi_complete+0x8c/0x1c4 [ 475.537613] nvme_loop():1 [ 475.541781] blk_complete_reqs+0x5c/0x74 [ 475.544736] nvmet():1 [ 475.548384] blk_done_softirq+0x38/0x4c [ 475.551077] nvme_fabrics():1 [ 475.554985] __do_softirq+0x168/0x418 [ 475.557331] nvme_loop():1 [ 475.561153] __irq_exit_rcu+0x120/0x170 [ 475.564106] nvmet():1 [ 475.567754] irq_exit_rcu+0x18/0x24 [ 475.570447] nvme_fabrics():1 [ 475.574269] el1_interrupt+0x38/0x70 [ 475.576615] nvme_loop():1 [ 475.580089] el1h_64_irq_handler+0x18/0x24 [ 475.583043] nvmet():1 [ 475.586604] el1h_64_irq+0x68/0x6c [ 475.589297] nvme_fabrics():1 [ 475.593379] _raw_spin_unlock_irqrestore+0x20/0x50 [ 475.595725] nvme_loop():1 [ 475.599113] __wake_up_common_lock+0x84/0xac [ 475.602066] nvmet():1 [ 475.606843] __wake_up+0x20/0x30 [ 475.609537] nvme_fabrics():1 [ 475.613793] __sbq_wake_up+0x13c/0x1f0 [ 475.616139] nvme_loop():1 [ 475.619353] sbitmap_queue_clear+0x74/0xc0 [ 475.622306] nvmet():1 [ 475.626041] blk_mq_put_tag+0x44/0x70 [ 475.628734] nvme_fabrics():1 [ 475.632816] __blk_mq_free_request+0x88/0xf0 [ 475.635162] nvme_loop():1 [ 475.638810] blk_mq_free_request+0xc0/0x160 [ 475.641764] null_blk():1 [ 475.646019] bfq_insert_request.isra.0+0x204/0x270 [ 475.648712] null_blk():1 [ 475.652881] bfq_insert_requests+0x70/0x90 [ 475.655488] null_blk():1 [ 475.660264] blk_mq_sched_insert_requests+0xac/0x250 [ 475.662870] null_blk():1 [ 475.666952] blk_mq_flush_plug_list+0x110/0x2cc [ 475.669560] null_blk():1 [ 475.674510] __blk_flush_plug+0x100/0x140 [ 475.677117] null_blk():1 [ 475.681633] blk_finish_plug+0x40/0x25c [ 475.684239] null_blk():1 [ 475.688234] __iomap_dio_rw+0x1b4/0x500 [ 475.690841] scsi_debug():1 [ 475.694662] iomap_dio_rw+0x18/0x50 [ 475.697269] scsi_debug():1 [ 475.701090] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 475.703870] ipmi_si():1 [ 475.707344] xfs_file_write_iter+0xf4/0x15c [xfs] [ 475.710124] ipmi_si():1 [ 475.715422] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 475.717941] [last unloaded: nvmet] [ 475.722631] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 475.725150] [ 475.730187] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 475.733662] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 475.738785] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 475.740263] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 475.745387] process_one_work+0x1ec/0x470 [ 475.753201] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 475.758586] worker_thread+0x74/0x410 [ 475.767876] pc : __sbq_wake_up+0x3c/0x1f0 [ 475.771871] kthread+0xec/0x100 [ 475.778818] lr : sbitmap_queue_clear+0x74/0xc0 [ 475.782466] ret_from_fork+0x10/0x20 [ 475.786461] sp : ffff8000088438b0 [ 475.800880] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 475.808003] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 475.815125] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 475.822247] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 475.829369] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 475.836491] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 475.843612] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 475.850734] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 475.857856] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 475.864978] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000036 [ 475.872100] Call trace: [ 475.874533] __sbq_wake_up+0x3c/0x1f0 [ 475.878183] sbitmap_queue_clear+0x74/0xc0 [ 475.882266] blk_mq_put_tag+0x44/0x70 [ 475.885916] __blk_mq_free_request+0x88/0xf0 [ 475.890172] blk_mq_free_request+0xc0/0x160 [ 475.894342] __blk_mq_end_request+0x78/0x150 [ 475.898599] scsi_end_request+0x114/0x260 [ 475.902595] scsi_io_completion+0x64/0x174 [ 475.906679] scsi_finish_command+0xe0/0x120 [ 475.910849] scsi_complete+0x8c/0x1c4 [ 475.914498] blk_complete_reqs+0x5c/0x74 [ 475.918407] blk_done_softirq+0x38/0x4c [ 475.922230] __do_softirq+0x168/0x418 [ 475.925879] __irq_exit_rcu+0x120/0x170 [ 475.929703] irq_exit_rcu+0x18/0x24 [ 475.933178] el1_interrupt+0x38/0x70 [ 475.936742] el1h_64_irq_handler+0x18/0x24 [ 475.940825] el1h_64_irq+0x68/0x6c [ 475.944214] cpuidle_enter_state+0xc4/0x4e0 [ 475.948383] cpuidle_enter+0x40/0x60 [ 475.951946] cpuidle_idle_call+0x118/0x1b0 [ 475.956029] do_idle+0xac/0x100 [ 475.959158] cpu_startup_entry+0x30/0x34 [ 475.963067] secondary_start_kernel+0xd8/0xf0 [ 475.967411] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:05:00 2022] [ 484.901552] rcu: INFO: rcu_preempt self-detected stall on CPU [ 484.907285] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=3000 [ 484.916234] (t=6001 jiffies g=36445 q=1808 ncpus=128) [ 484.921359] Task dump for CPU 8: [ 484.924574] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 484.934474] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 484.940643] Call trace: [ 484.943076] dump_backtrace+0xe8/0x130 [ 484.946813] show_stack+0x20/0x60 [ 484.950115] sched_show_task+0x160/0x190 [ 484.954025] dump_cpu_task+0x4c/0x78 [ 484.957588] rcu_dump_cpu_stacks+0xc4/0x150 [ 484.961760] print_cpu_stall+0x180/0x210 [ 484.965670] check_cpu_stall+0x20c/0x2b4 [ 484.969579] rcu_pending+0x4c/0x190 [ 484.973054] rcu_sched_clock_irq+0x80/0x204 [ 484.977224] update_process_times+0x7c/0xc0 [ 484.981395] tick_sched_handle+0x38/0x74 [ 484.985306] tick_sched_timer+0x60/0xc0 [ 484.989129] __hrtimer_run_queues+0x170/0x300 [ 484.993473] hrtimer_interrupt+0xfc/0x260 [ 484.997469] arch_timer_handler_phys+0x34/0x50 [ 485.001901] handle_percpu_devid_irq+0xac/0x240 [ 485.006419] generic_handle_domain_irq+0x34/0x50 [ 485.011024] gic_handle_irq+0x5c/0xe0 [ 485.014673] call_on_irq_stack+0x2c/0x38 [ 485.018583] do_interrupt_handler+0xdc/0xe0 [ 485.022754] el1_interrupt+0x34/0x70 [ 485.026316] el1h_64_irq_handler+0x18/0x24 [ 485.030399] el1h_64_irq+0x68/0x6c [ 485.033788] __sbq_wake_up+0x4/0x1f0 [ 485.037351] blk_mq_put_tag+0x44/0x70 [ 485.041000] __blk_mq_free_request+0x88/0xf0 [ 485.045257] blk_mq_free_request+0xc0/0x160 [ 485.049427] __blk_mq_end_request+0x78/0x150 [ 485.053684] scsi_end_request+0x114/0x260 [ 485.057680] scsi_io_completion+0x64/0x174 [ 485.061763] scsi_finish_command+0xe0/0x120 [ 485.065934] scsi_complete+0x8c/0x1c4 [ 485.069583] blk_complete_reqs+0x5c/0x74 [ 485.073492] blk_done_softirq+0x38/0x4c [ 485.077315] __do_softirq+0x168/0x418 [ 485.080964] __irq_exit_rcu+0x120/0x170 [ 485.084787] irq_exit_rcu+0x18/0x24 [ 485.088262] el1_interrupt+0x38/0x70 [ 485.091825] el1h_64_irq_handler+0x18/0x24 [ 485.095908] el1h_64_irq+0x68/0x6c [ 485.099296] _raw_spin_unlock_irqrestore+0x20/0x50 [ 485.104074] __wake_up_common_lock+0x84/0xac [ 485.108332] __wake_up+0x20/0x30 [ 485.111548] __sbq_wake_up+0x13c/0x1f0 [ 485.115284] sbitmap_queue_clear+0x74/0xc0 [ 485.119367] blk_mq_put_tag+0x44/0x70 [ 485.123017] __blk_mq_free_request+0x88/0xf0 [ 485.127274] blk_mq_free_request+0xc0/0x160 [ 485.131444] bfq_insert_request.isra.0+0x204/0x270 [ 485.136222] bfq_insert_requests+0x70/0x90 [ 485.140305] blk_mq_sched_insert_requests+0xac/0x250 [ 485.145258] blk_mq_flush_plug_list+0x110/0x2cc [ 485.149775] __blk_flush_plug+0x100/0x140 [ 485.153772] blk_finish_plug+0x40/0x25c [ 485.157595] __iomap_dio_rw+0x1b4/0x500 [ 485.161417] iomap_dio_rw+0x18/0x50 [ 485.164893] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 485.170244] xfs_file_write_iter+0xf4/0x15c [xfs] [ 485.174987] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 485.180031] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 485.185161] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 485.190291] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 485.195677] process_one_work+0x1ec/0x470 [ 485.199674] worker_thread+0x74/0x410 [ 485.203323] kthread+0xec/0x100 [ 485.206451] ret_from_fork+0x10/0x20 [ 485.210015] Task dump for CPU 68: [ 485.213316] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 485.223216] Call trace: [ 485.225650] __switch_to+0xd4/0x100 [ 485.229126] 0x0 [ 499.221464] watchdog: BUG: soft lockup - CPU#68 stuck for 70s! [swapper/68:0] [ 499.228586] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 499.228602] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 499.316389] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 499.324205] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 499.333496] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 499.340444] pc : __sbq_wake_up+0x7c/0x1f0 [ 499.344441] lr : sbitmap_queue_clear+0x74/0xc0 [ 499.348871] sp : ffff800008843900 [ 499.352172] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 499.359294] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 499.366416] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 499.373538] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 499.380660] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 499.387781] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 499.394903] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 499.402025] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 499.409147] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 499.416269] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 499.423391] Call trace: [ 499.425824] __sbq_wake_up+0x7c/0x1f0 [ 499.429473] blk_mq_put_tag+0x44/0x70 [ 499.433122] __blk_mq_free_request+0x88/0xf0 [ 499.437379] blk_mq_free_request+0xc0/0x160 [ 499.441551] __blk_mq_end_request+0x78/0x150 [ 499.445807] scsi_end_request+0x114/0x260 [ 499.449804] scsi_io_completion+0x64/0x174 [ 499.453887] scsi_finish_command+0xe0/0x120 [ 499.458057] scsi_complete+0x8c/0x1c4 [ 499.461706] blk_complete_reqs+0x5c/0x74 [ 499.465616] blk_done_softirq+0x38/0x4c [ 499.469438] __do_softirq+0x168/0x418 [ 499.473088] __irq_exit_rcu+0x120/0x170 [ 499.476911] irq_exit_rcu+0x18/0x24 [ 499.480386] el1_interrupt+0x38/0x70 [ 499.483949] el1h_64_irq_handler+0x18/0x24 [ 499.488032] el1h_64_irq+0x68/0x6c [ 499.491420] cpuidle_enter_state+0xc4/0x4e0 [ 499.495591] cpuidle_enter+0x40/0x60 [ 499.499153] cpuidle_idle_call+0x118/0x1b0 [ 499.503237] do_idle+0xac/0x100 [ 499.506365] cpu_startup_entry+0x30/0x34 [ 499.510275] secondary_start_kernel+0xd8/0xf0 [ 499.514619] __secondary_switched+0xb0/0xb4 [ 511.111390] watchdog: BUG: soft lockup - CPU#8 stuck for 82s! [kworker/8:12:19399] [ 511.118946] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 511.118963] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 511.206752] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 511.215002] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 511.224293] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 511.230460] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 511.237409] pc : __sbq_wake_up+0x58/0x1f0 [ 511.241405] lr : sbitmap_queue_clear+0x74/0xc0 [ 511.245836] sp : ffff80000d79b280 [ 511.249137] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 511.256260] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 511.263382] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 511.270504] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 511.277625] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 511.284748] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 511.291870] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 511.298992] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 511.306114] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 511.313236] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 511.320357] Call trace: [ 511.322791] __sbq_wake_up+0x58/0x1f0 [ 511.326440] sbitmap_queue_clear+0x74/0xc0 [ 511.330523] blk_mq_put_tag+0x44/0x70 [ 511.334172] __blk_mq_free_request+0x88/0xf0 [ 511.338429] blk_mq_free_request+0xc0/0x160 [ 511.342599] __blk_mq_end_request+0x78/0x150 [ 511.346856] scsi_end_request+0x114/0x260 [ 511.350852] scsi_io_completion+0x64/0x174 [ 511.354935] scsi_finish_command+0xe0/0x120 [ 511.359106] scsi_complete+0x8c/0x1c4 [ 511.362755] blk_complete_reqs+0x5c/0x74 [ 511.366664] blk_done_softirq+0x38/0x4c [ 511.370487] __do_softirq+0x168/0x418 [ 511.374136] __irq_exit_rcu+0x120/0x170 [ 511.377959] irq_exit_rcu+0x18/0x24 [ 511.381434] el1_interrupt+0x38/0x70 [ 511.384996] el1h_64_irq_handler+0x18/0x24 [ 511.389079] el1h_64_irq+0x68/0x6c [ 511.392469] _raw_spin_unlock_irqrestore+0x20/0x50 [ 511.397247] __wake_up_common_lock+0x84/0xac [ 511.401504] __wake_up+0x20/0x30 [ 511.404720] __sbq_wake_up+0x13c/0x1f0 [ 511.408456] sbitmap_queue_clear+0x74/0xc0 [ 511.412539] blk_mq_put_tag+0x44/0x70 [ 511.416189] __blk_mq_free_request+0x88/0xf0 [ 511.420445] blk_mq_free_request+0xc0/0x160 [ 511.424615] bfq_insert_request.isra.0+0x204/0x270 [ 511.429393] bfq_insert_requests+0x70/0x90 [ 511.433477] blk_mq_sched_insert_requests+0xac/0x250 [ 511.438429] blk_mq_flush_plug_list+0x110/0x2cc [ 511.442948] __blk_flush_plug+0x100/0x140 [ 511.446944] blk_finish_plug+0x40/0x25c [ 511.450767] __iomap_dio_rw+0x1b4/0x500 [ 511.454590] iomap_dio_rw+0x18/0x50 [ 511.458065] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 511.463416] xfs_file_write_iter+0xf4/0x15c [xfs] [ 511.468159] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 511.473203] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 511.478333] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 511.483463] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 511.488849] process_one_work+0x1ec/0x470 [ 511.492845] worker_thread+0x74/0x410 [ 511.496494] kthread+0xec/0x100 [ 511.499623] ret_from_fork+0x10/0x20 [ 523.221316] watchdog: BUG: soft lockup - CPU#68 stuck for 93s! [swapper/68:0] [ 523.228438] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 523.228454] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 523.316241] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 523.324056] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 523.333347] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 523.340295] pc : __sbq_wake_up+0x20/0x1f0 [ 523.344292] lr : sbitmap_queue_clear+0x74/0xc0 [ 523.348722] sp : ffff8000088438b0 [ 523.352023] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 523.359145] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 523.366267] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 523.373389] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 523.380510] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 523.387632] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 523.394754] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 523.401876] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 523.408998] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 523.416120] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 523.423241] Call trace: [ 523.425674] __sbq_wake_up+0x20/0x1f0 [ 523.429323] sbitmap_queue_clear+0x74/0xc0 [ 523.433406] blk_mq_put_tag+0x44/0x70 [ 523.437056] __blk_mq_free_request+0x88/0xf0 [ 523.441313] blk_mq_free_request+0xc0/0x160 [ 523.445483] __blk_mq_end_request+0x78/0x150 [ 523.449740] scsi_end_request+0x114/0x260 [ 523.453737] scsi_io_completion+0x64/0x174 [ 523.457820] scsi_finish_command+0xe0/0x120 [ 523.461990] scsi_complete+0x8c/0x1c4 [ 523.465639] blk_complete_reqs+0x5c/0x74 [ 523.469549] blk_done_softirq+0x38/0x4c [ 523.473371] __do_softirq+0x168/0x418 [ 523.477020] __irq_exit_rcu+0x120/0x170 [ 523.480843] irq_exit_rcu+0x18/0x24 [ 523.484319] el1_interrupt+0x38/0x70 [ 523.487881] el1h_64_irq_handler+0x18/0x24 [ 523.491966] el1h_64_irq+0x68/0x6c [ 523.495354] cpuidle_enter_state+0xc4/0x4e0 [ 523.499525] cpuidle_enter+0x40/0x60 [ 523.503087] cpuidle_idle_call+0x118/0x1b0 [ 523.507171] do_idle+0xac/0x100 [ 523.510299] cpu_startup_entry+0x30/0x34 [ 523.514208] secondary_start_kernel+0xd8/0xf0 [ 523.518553] __secondary_switched+0xb0/0xb4 [ 535.111242] watchdog: BUG: soft lockup - CPU#8 stuck for 104s! [kworker/8:12:19399] [ 535.118885] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 535.118902] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 535.206690] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 535.214939] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 535.224230] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 535.230398] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 535.237346] pc : __sbq_wake_up+0x4/0x1f0 [ 535.241256] lr : sbitmap_queue_clear+0x74/0xc0 [ 535.245687] sp : ffff80000d79b2d0 [ 535.248988] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 535.256110] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 535.263232] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 535.270354] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 535.277476] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 535.284598] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 535.291719] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 535.298841] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 535.305963] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 535.313085] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 535.320207] Call trace: [ 535.322640] __sbq_wake_up+0x4/0x1f0 [ 535.326204] blk_mq_put_tag+0x44/0x70 [ 535.329854] __blk_mq_free_request+0x88/0xf0 [ 535.334111] blk_mq_free_request+0xc0/0x160 [ 535.338281] __blk_mq_end_request+0x78/0x150 [ 535.342539] scsi_end_request+0x114/0x260 [ 535.346536] scsi_io_completion+0x64/0x174 [ 535.350619] scsi_finish_command+0xe0/0x120 [ 535.354789] scsi_complete+0x8c/0x1c4 [ 535.358438] blk_complete_reqs+0x5c/0x74 [ 535.362348] blk_done_softirq+0x38/0x4c [ 535.366171] __do_softirq+0x168/0x418 [ 535.369820] __irq_exit_rcu+0x120/0x170 [ 535.373643] irq_exit_rcu+0x18/0x24 [ 535.377118] el1_interrupt+0x38/0x70 [ 535.380681] el1h_64_irq_handler+0x18/0x24 [ 535.384764] el1h_64_irq+0x68/0x6c [ 535.388152] _raw_spin_unlock_irqrestore+0x20/0x50 [ 535.392930] __wake_up_common_lock+0x84/0xac [ 535.397187] __wake_up+0x20/0x30 [ 535.400403] __sbq_wake_up+0x13c/0x1f0 [ 535.404139] sbitmap_queue_clear+0x74/0xc0 [ 535.408222] blk_mq_put_tag+0x44/0x70 [ 535.411872] __blk_mq_free_request+0x88/0xf0 [ 535.416129] blk_mq_free_request+0xc0/0x160 [ 535.420300] bfq_insert_request.isra.0+0x204/0x270 [ 535.425078] bfq_insert_requests+0x70/0x90 [ 535.429161] blk_mq_sched_insert_requests+0xac/0x250 [ 535.434114] blk_mq_flush_plug_list+0x110/0x2cc [ 535.438631] __blk_flush_plug+0x100/0x140 [ 535.442628] blk_finish_plug+0x40/0x25c [ 535.446450] __iomap_dio_rw+0x1b4/0x500 [ 535.450273] iomap_dio_rw+0x18/0x50 [ 535.453748] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 535.459100] xfs_file_write_iter+0xf4/0x15c [xfs] [ 535.463843] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 535.468887] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 535.474017] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 535.479147] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 535.484533] process_one_work+0x1ec/0x470 [ 535.488530] worker_thread+0x74/0x410 [ 535.492179] kthread+0xec/0x100 [ 535.495308] ret_from_fork+0x10/0x20 [ 538.611300] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 8765 jiffies s: 2093 root: 0x11/. [ 538.622872] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 538.632086] Task dump for CPU 8: [ 538.635301] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 538.645208] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 538.651383] Call trace: [ 538.653817] __switch_to+0xd4/0x100 [ 538.657297] 0xffff6c0200000080 [ 538.660427] Task dump for CPU 68: [ 538.663735] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 538.673640] Call trace: [ 538.676074] __switch_to+0xd4/0x100 [ 538.679552] 0x0 [ 547.221168] watchdog: BUG: soft lockup - CPU#68 stuck for 115s! [swapper/68:0] [ 547.228376] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 547.228392] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 547.316179] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 547.323996] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 547.333287] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 547.340235] pc : __sbq_wake_up+0x58/0x1f0 [ 547.344232] lr : sbitmap_queue_clear+0x74/0xc0 [ 547.348662] sp : ffff8000088438b0 [ 547.351963] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 547.359085] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 547.366207] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 547.373329] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 547.380452] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 547.387573] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 547.394695] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 547.401817] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 547.408939] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 547.416061] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 547.423183] Call trace: [ 547.425616] __sbq_wake_up+0x58/0x1f0 [ 547.429265] sbitmap_queue_clear+0x74/0xc0 [ 547.433348] blk_mq_put_tag+0x44/0x70 [ 547.436998] __blk_mq_free_request+0x88/0xf0 [ 547.441254] blk_mq_free_request+0xc0/0x160 [ 547.445425] __blk_mq_end_request+0x78/0x150 [ 547.449681] scsi_end_request+0x114/0x260 [ 547.453678] scsi_io_completion+0x64/0x174 [ 547.457761] scsi_finish_command+0xe0/0x120 [ 547.461931] scsi_complete+0x8c/0x1c4 [ 547.465580] blk_complete_reqs+0x5c/0x74 [ 547.469491] blk_done_softirq+0x38/0x4c [ 547.473314] __do_softirq+0x168/0x418 [ 547.476962] __irq_exit_rcu+0x120/0x170 [ 547.480785] irq_exit_rcu+0x18/0x24 [ 547.484260] el1_interrupt+0x38/0x70 [ 547.487823] el1h_64_irq_handler+0x18/0x24 [ 547.491906] el1h_64_irq+0x68/0x6c [ 547.495294] cpuidle_enter_state+0xc4/0x4e0 [ 547.499464] cpuidle_enter+0x40/0x60 [ 547.503027] cpuidle_idle_call+0x118/0x1b0 [ 547.507110] do_idle+0xac/0x100 [ 547.510239] cpu_startup_entry+0x30/0x34 [ 547.514148] secondary_start_kernel+0xd8/0xf0 [ 547.518492] __secondary_switched+0xb0/0xb4 [ 559.111095] watchdog: BUG: soft lockup - CPU#8 stuck for 126s! [kworker/8:12:19399] [ 559.118737] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 559.118754] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 559.206541] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 559.214791] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 559.224082] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 559.230249] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 559.237197] pc : __sbq_wake_up+0x4/0x1f0 [ 559.241107] lr : sbitmap_queue_clear+0x74/0xc0 [ 559.245537] sp : ffff80000d79b2d0 [ 559.248839] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 559.255961] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 559.263083] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 559.270205] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 559.277327] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 559.284449] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 559.291571] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 559.298693] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 559.305815] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 559.312937] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 559.320059] Call trace: [ 559.322492] __sbq_wake_up+0x4/0x1f0 [ 559.326054] blk_mq_put_tag+0x44/0x70 [ 559.329704] __blk_mq_free_request+0x88/0xf0 [ 559.333961] blk_mq_free_request+0xc0/0x160 [ 559.338131] __blk_mq_end_request+0x78/0x150 [ 559.342388] scsi_end_request+0x114/0x260 [ 559.346384] scsi_io_completion+0x64/0x174 [ 559.350467] scsi_finish_command+0xe0/0x120 [ 559.354638] scsi_complete+0x8c/0x1c4 [ 559.358287] blk_complete_reqs+0x5c/0x74 [ 559.362197] blk_done_softirq+0x38/0x4c [ 559.366019] __do_softirq+0x168/0x418 [ 559.369668] __irq_exit_rcu+0x120/0x170 [ 559.373491] irq_exit_rcu+0x18/0x24 [ 559.376966] el1_interrupt+0x38/0x70 [ 559.380529] el1h_64_irq_handler+0x18/0x24 [ 559.384612] el1h_64_irq+0x68/0x6c [ 559.388000] _raw_spin_unlock_irqrestore+0x20/0x50 [ 559.392778] __wake_up_common_lock+0x84/0xac [ 559.397036] __wake_up+0x20/0x30 [ 559.400251] __sbq_wake_up+0x13c/0x1f0 [ 559.403987] sbitmap_queue_clear+0x74/0xc0 [ 559.408070] blk_mq_put_tag+0x44/0x70 [ 559.411720] __blk_mq_free_request+0x88/0xf0 [ 559.415977] blk_mq_free_request+0xc0/0x160 [ 559.420146] bfq_insert_request.isra.0+0x204/0x270 [ 559.424925] bfq_insert_requests+0x70/0x90 [ 559.429008] blk_mq_sched_insert_requests+0xac/0x250 [ 559.433960] blk_mq_flush_plug_list+0x110/0x2cc [ 559.438478] __blk_flush_plug+0x100/0x140 [ 559.442474] blk_finish_plug+0x40/0x25c [ 559.446297] __iomap_dio_rw+0x1b4/0x500 [ 559.450120] iomap_dio_rw+0x18/0x50 [ 559.453595] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 559.458946] xfs_file_write_iter+0xf4/0x15c [xfs] [ 559.463689] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 559.468732] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 559.473863] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 559.478992] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 559.484379] process_one_work+0x1ec/0x470 [ 559.488375] worker_thread+0x74/0x410 [ 559.492023] kthread+0xec/0x100 [ 559.495152] ret_from_fork+0x10/0x20 [ 571.221020] watchdog: BUG: soft lockup - CPU#68 stuck for 137s! [swapper/68:0] [ 571.228228] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 571.228245] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 571.316032] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 571.323848] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 571.333139] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 571.340086] pc : __sbq_wake_up+0x4/0x1f0 [ 571.343997] lr : sbitmap_queue_clear+0x74/0xc0 [ 571.348427] sp : ffff800008843900 [ 571.351729] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 571.358850] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 571.365972] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 571.373095] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 571.380216] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 571.387339] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 571.394461] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 571.401583] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 571.408704] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 571.415826] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 571.422948] Call trace: [ 571.425381] __sbq_wake_up+0x4/0x1f0 [ 571.428943] blk_mq_put_tag+0x44/0x70 [ 571.432593] __blk_mq_free_request+0x88/0xf0 [ 571.436849] blk_mq_free_request+0xc0/0x160 [ 571.441019] __blk_mq_end_request+0x78/0x150 [ 571.445276] scsi_end_request+0x114/0x260 [ 571.449272] scsi_io_completion+0x64/0x174 [ 571.453355] scsi_finish_command+0xe0/0x120 [ 571.457526] scsi_complete+0x8c/0x1c4 [ 571.461175] blk_complete_reqs+0x5c/0x74 [ 571.465085] blk_done_softirq+0x38/0x4c [ 571.468909] __do_softirq+0x168/0x418 [ 571.472558] __irq_exit_rcu+0x120/0x170 [ 571.476380] irq_exit_rcu+0x18/0x24 [ 571.479856] el1_interrupt+0x38/0x70 [ 571.483418] el1h_64_irq_handler+0x18/0x24 [ 571.487501] el1h_64_irq+0x68/0x6c [ 571.490889] cpuidle_enter_state+0xc4/0x4e0 [ 571.495059] cpuidle_enter+0x40/0x60 [ 571.498622] cpuidle_idle_call+0x118/0x1b0 [ 571.502706] do_idle+0xac/0x100 [ 571.505835] cpu_startup_entry+0x30/0x34 [ 571.509744] secondary_start_kernel+0xd8/0xf0 [ 571.514088] __secondary_switched+0xb0/0xb4 [ 583.110946] watchdog: BUG: soft lockup - CPU#8 stuck for 149s! [kworker/8:12:19399] [ 583.118588] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 583.118605] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 583.206393] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 583.214642] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 583.223933] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 583.230102] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 583.237049] pc : __sbq_wake_up+0x58/0x1f0 [ 583.241046] lr : sbitmap_queue_clear+0x74/0xc0 [ 583.245477] sp : ffff80000d79b280 [ 583.248778] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 583.255900] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 583.263022] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 583.270144] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 583.277266] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 583.284388] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 583.291510] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 583.298632] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 583.305754] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 583.312876] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 583.319998] Call trace: [ 583.322431] __sbq_wake_up+0x58/0x1f0 [ 583.326080] sbitmap_queue_clear+0x74/0xc0 [ 583.330164] blk_mq_put_tag+0x44/0x70 [ 583.333814] __blk_mq_free_request+0x88/0xf0 [ 583.338070] blk_mq_free_request+0xc0/0x160 [ 583.342241] __blk_mq_end_request+0x78/0x150 [ 583.346499] scsi_end_request+0x114/0x260 [ 583.350495] scsi_io_completion+0x64/0x174 [ 583.354578] scsi_finish_command+0xe0/0x120 [ 583.358748] scsi_complete+0x8c/0x1c4 [ 583.362398] blk_complete_reqs+0x5c/0x74 [ 583.366307] blk_done_softirq+0x38/0x4c [ 583.370130] __do_softirq+0x168/0x418 [ 583.373779] __irq_exit_rcu+0x120/0x170 [ 583.377602] irq_exit_rcu+0x18/0x24 [ 583.381077] el1_interrupt+0x38/0x70 [ 583.384639] el1h_64_irq_handler+0x18/0x24 [ 583.388723] el1h_64_irq+0x68/0x6c [ 583.392112] _raw_spin_unlock_irqrestore+0x20/0x50 [ 583.396889] __wake_up_common_lock+0x84/0xac [ 583.401147] __wake_up+0x20/0x30 [ 583.404362] __sbq_wake_up+0x13c/0x1f0 [ 583.408098] sbitmap_queue_clear+0x74/0xc0 [ 583.412181] blk_mq_put_tag+0x44/0x70 [ 583.415831] __blk_mq_free_request+0x88/0xf0 [ 583.420089] blk_mq_free_request+0xc0/0x160 [ 583.424259] bfq_insert_request.isra.0+0x204/0x270 [ 583.429037] bfq_insert_requests+0x70/0x90 [ 583.433121] blk_mq_sched_insert_requests+0xac/0x250 [ 583.438073] blk_mq_flush_plug_list+0x110/0x2cc [ 583.442590] __blk_flush_plug+0x100/0x140 [ 583.446587] blk_finish_plug+0x40/0x25c [ 583.450409] __iomap_dio_rw+0x1b4/0x500 [ 583.454232] iomap_dio_rw+0x18/0x50 [ 583.457708] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 583.463059] xfs_file_write_iter+0xf4/0x15c [xfs] [ 583.467801] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 583.472845] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 583.477974] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 583.483104] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 583.488490] process_one_work+0x1ec/0x470 [ 583.492487] worker_thread+0x74/0x410 [ 583.496135] kthread+0xec/0x100 [ 583.499264] ret_from_fork+0x10/0x20 [ 595.220871] watchdog: BUG: soft lockup - CPU#68 stuck for 160s! [swapper/68:0] [ 595.228080] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 595.228096] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 595.315883] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 595.323700] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 595.332991] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 595.339938] pc : __sbq_wake_up+0x58/0x1f0 [ 595.343935] lr : sbitmap_queue_clear+0x74/0xc0 [ 595.348365] sp : ffff8000088438b0 [ 595.351667] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 595.358789] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 595.365910] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 595.373032] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 595.380154] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 595.387276] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 595.394398] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 595.401520] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 595.408641] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 595.415763] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 595.422885] Call trace: [ 595.425318] __sbq_wake_up+0x58/0x1f0 [ 595.428967] sbitmap_queue_clear+0x74/0xc0 [ 595.433051] blk_mq_put_tag+0x44/0x70 [ 595.436701] __blk_mq_free_request+0x88/0xf0 [ 595.440957] blk_mq_free_request+0xc0/0x160 [ 595.445127] __blk_mq_end_request+0x78/0x150 [ 595.449384] scsi_end_request+0x114/0x260 [ 595.453380] scsi_io_completion+0x64/0x174 [ 595.457463] scsi_finish_command+0xe0/0x120 [ 595.461634] scsi_complete+0x8c/0x1c4 [ 595.465283] blk_complete_reqs+0x5c/0x74 [ 595.469192] blk_done_softirq+0x38/0x4c [ 595.473015] __do_softirq+0x168/0x418 [ 595.476664] __irq_exit_rcu+0x120/0x170 [ 595.480487] irq_exit_rcu+0x18/0x24 [ 595.483962] el1_interrupt+0x38/0x70 [ 595.487525] el1h_64_irq_handler+0x18/0x24 [ 595.491608] el1h_64_irq+0x68/0x6c [ 595.494996] cpuidle_enter_state+0xc4/0x4e0 [ 595.499166] cpuidle_enter+0x40/0x60 [ 595.502729] cpuidle_idle_call+0x118/0x1b0 [ 595.506812] do_idle+0xac/0x100 [ 595.509941] cpu_startup_entry+0x30/0x34 [ 595.513850] secondary_start_kernel+0xd8/0xf0 [ 595.518195] __secondary_switched+0xb0/0xb4 [ 605.170864] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 15421 jiffies s: 2093 root: 0x11/. [ 605.182519] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 605.191730] Task dump for CPU 8: [ 605.194946] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 605.204850] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 605.211023] Call trace: [ 605.213456] __switch_to+0xd4/0x100 [ 605.216934] 0xffff6c0200000080 [ 605.220064] Task dump for CPU 68: [ 605.223368] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 605.233274] Call trace: [ 605.235708] __switch_to+0xd4/0x100 [ 605.239185] 0x0 [ 607.110798] watchdog: BUG: soft lockup - CPU#8 stuck for 171s! [kworker/8:12:19399] [ 607.118440] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 607.118457] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 607.206244] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 607.214493] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 607.223784] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 607.229952] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 607.236900] pc : __sbq_wake_up+0x1c8/0x1f0 [ 607.240983] lr : sbitmap_queue_clear+0x74/0xc0 [ 607.245414] sp : ffff80000d79b280 [ 607.248715] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 607.255838] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 607.262961] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 607.270082] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 607.277205] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 607.284327] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 607.291449] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 607.298570] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 607.305692] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 607.312814] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 607.319936] Call trace: [ 607.322371] __sbq_wake_up+0x1c8/0x1f0 [ 607.326107] sbitmap_queue_clear+0x74/0xc0 [ 607.330190] blk_mq_put_tag+0x44/0x70 [ 607.333840] __blk_mq_free_request+0x88/0xf0 [ 607.338097] blk_mq_free_request+0xc0/0x160 [ 607.342267] __blk_mq_end_request+0x78/0x150 [ 607.346524] scsi_end_request+0x114/0x260 [ 607.350520] scsi_io_completion+0x64/0x174 [ 607.354603] scsi_finish_command+0xe0/0x120 [ 607.358773] scsi_complete+0x8c/0x1c4 [ 607.362423] blk_complete_reqs+0x5c/0x74 [ 607.366333] blk_done_softirq+0x38/0x4c [ 607.370155] __do_softirq+0x168/0x418 [ 607.373804] __irq_exit_rcu+0x120/0x170 [ 607.377627] irq_exit_rcu+0x18/0x24 [ 607.381103] el1_interrupt+0x38/0x70 [ 607.384665] el1h_64_irq_handler+0x18/0x24 [ 607.388748] el1h_64_irq+0x68/0x6c [ 607.392136] _raw_spin_unlock_irqrestore+0x20/0x50 [ 607.396914] __wake_up_common_lock+0x84/0xac [ 607.401171] __wake_up+0x20/0x30 [ 607.404387] __sbq_wake_up+0x13c/0x1f0 [ 607.408123] sbitmap_queue_clear+0x74/0xc0 [ 607.412206] blk_mq_put_tag+0x44/0x70 [ 607.415856] __blk_mq_free_request+0x88/0xf0 [ 607.420113] blk_mq_free_request+0xc0/0x160 [ 607.424283] bfq_insert_request.isra.0+0x204/0x270 [ 607.429061] bfq_insert_requests+0x70/0x90 [ 607.433144] blk_mq_sched_insert_requests+0xac/0x250 [ 607.438097] blk_mq_flush_plug_list+0x110/0x2cc [ 607.442614] __blk_flush_plug+0x100/0x140 [ 607.446611] blk_finish_plug+0x40/0x25c [ 607.450433] __iomap_dio_rw+0x1b4/0x500 [ 607.454256] iomap_dio_rw+0x18/0x50 [ 607.457731] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 607.463083] xfs_file_write_iter+0xf4/0x15c [xfs] [ 607.467825] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 607.472869] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 607.477999] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 607.483129] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 607.488515] process_one_work+0x1ec/0x470 [ 607.492511] worker_thread+0x74/0x410 [ 607.496160] kthread+0xec/0x100 [ 607.499288] ret_from_fork+0x10/0x20 [ 619.220723] watchdog: BUG: soft lockup - CPU#68 stuck for 182s! [swapper/68:0] [ 619.227931] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 619.227948] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 619.315735] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 619.323550] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 619.332841] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 619.339789] pc : __sbq_wake_up+0x20/0x1f0 [ 619.343785] lr : sbitmap_queue_clear+0x74/0xc0 [ 619.348217] sp : ffff8000088438b0 [ 619.351518] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 619.358640] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 619.365762] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 619.372884] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 619.380007] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 619.387129] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 619.394251] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 619.401373] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 619.408494] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 619.415616] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 619.422739] Call trace: [ 619.425172] __sbq_wake_up+0x20/0x1f0 [ 619.428821] sbitmap_queue_clear+0x74/0xc0 [ 619.432905] blk_mq_put_tag+0x44/0x70 [ 619.436554] __blk_mq_free_request+0x88/0xf0 [ 619.440811] blk_mq_free_request+0xc0/0x160 [ 619.444981] __blk_mq_end_request+0x78/0x150 [ 619.449238] scsi_end_request+0x114/0x260 [ 619.453234] scsi_io_completion+0x64/0x174 [ 619.457317] scsi_finish_command+0xe0/0x120 [ 619.461487] scsi_complete+0x8c/0x1c4 [ 619.465136] blk_complete_reqs+0x5c/0x74 [ 619.469046] blk_done_softirq+0x38/0x4c [ 619.472869] __do_softirq+0x168/0x418 [ 619.476517] __irq_exit_rcu+0x120/0x170 [ 619.480340] irq_exit_rcu+0x18/0x24 [ 619.483815] el1_interrupt+0x38/0x70 [ 619.487378] el1h_64_irq_handler+0x18/0x24 [ 619.491461] el1h_64_irq+0x68/0x6c [ 619.494849] cpuidle_enter_state+0xc4/0x4e0 [ 619.499019] cpuidle_enter+0x40/0x60 [ 619.502582] cpuidle_idle_call+0x118/0x1b0 [ 619.506665] do_idle+0xac/0x100 [ 619.509794] cpu_startup_entry+0x30/0x34 [ 619.513703] secondary_start_kernel+0xd8/0xf0 [ 619.518047] __secondary_switched+0xb0/0xb4 [ 631.110649] watchdog: BUG: soft lockup - CPU#8 stuck for 193s! [kworker/8:12:19399] [ 631.118291] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 631.118308] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 631.206095] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 631.214345] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 631.223636] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 631.229804] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 631.236752] pc : __sbq_wake_up+0x28/0x1f0 [ 631.240749] lr : sbitmap_queue_clear+0x74/0xc0 [ 631.245180] sp : ffff80000d79b280 [ 631.248481] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 631.255603] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 631.262725] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 631.269847] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 631.276969] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 631.284091] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 631.291213] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 631.298334] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 631.305458] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 631.312580] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000036 [ 631.319702] Call trace: [ 631.322135] __sbq_wake_up+0x28/0x1f0 [ 631.325785] sbitmap_queue_clear+0x74/0xc0 [ 631.329869] blk_mq_put_tag+0x44/0x70 [ 631.333518] __blk_mq_free_request+0x88/0xf0 [ 631.337775] blk_mq_free_request+0xc0/0x160 [ 631.341945] __blk_mq_end_request+0x78/0x150 [ 631.346203] scsi_end_request+0x114/0x260 [ 631.350201] scsi_io_completion+0x64/0x174 [ 631.354284] scsi_finish_command+0xe0/0x120 [ 631.358455] scsi_complete+0x8c/0x1c4 [ 631.362104] blk_complete_reqs+0x5c/0x74 [ 631.366013] blk_done_softirq+0x38/0x4c [ 631.369836] __do_softirq+0x168/0x418 [ 631.373486] __irq_exit_rcu+0x120/0x170 [ 631.377309] irq_exit_rcu+0x18/0x24 [ 631.380784] el1_interrupt+0x38/0x70 [ 631.384346] el1h_64_irq_handler+0x18/0x24 [ 631.388430] el1h_64_irq+0x68/0x6c [ 631.391819] _raw_spin_unlock_irqrestore+0x20/0x50 [ 631.396597] __wake_up_common_lock+0x84/0xac [ 631.400854] __wake_up+0x20/0x30 [ 631.404070] __sbq_wake_up+0x13c/0x1f0 [ 631.407806] sbitmap_queue_clear+0x74/0xc0 [ 631.411889] blk_mq_put_tag+0x44/0x70 [ 631.415539] __blk_mq_free_request+0x88/0xf0 [ 631.419797] blk_mq_free_request+0xc0/0x160 [ 631.423967] bfq_insert_request.isra.0+0x204/0x270 [ 631.428745] bfq_insert_requests+0x70/0x90 [ 631.432828] blk_mq_sched_insert_requests+0xac/0x250 [ 631.437780] blk_mq_flush_plug_list+0x110/0x2cc [ 631.442298] __blk_flush_plug+0x100/0x140 [ 631.446294] blk_finish_plug+0x40/0x25c [ 631.450117] __iomap_dio_rw+0x1b4/0x500 [ 631.453940] iomap_dio_rw+0x18/0x50 [ 631.457416] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 631.462767] xfs_file_write_iter+0xf4/0x15c [xfs] [ 631.467510] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 631.472554] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 631.477684] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 631.482813] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 631.488200] process_one_work+0x1ec/0x470 [ 631.492196] worker_thread+0x74/0x410 [ 631.495845] kthread+0xec/0x100 [ 631.498973] ret_from_fork+0x10/0x20 [ 643.220573] watchdog: BUG: soft lockup - CPU#68 stuck for 205s! [swapper/68:0] [ 643.227782] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 643.227798] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 643.315585] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 643.323401] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 643.332691] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 643.339639] pc : __sbq_wake_up+0x28/0x1f0 [ 643.343636] lr : sbitmap_queue_clear+0x74/0xc0 [ 643.348066] sp : ffff8000088438b0 [ 643.351368] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 643.358490] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 643.365613] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 643.372735] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 643.379856] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 643.386978] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 643.394100] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 643.401222] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 643.408344] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 643.415466] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000036 [ 643.422588] Call trace: [ 643.425022] __sbq_wake_up+0x28/0x1f0 [ 643.428671] sbitmap_queue_clear+0x74/0xc0 [ 643.432754] blk_mq_put_tag+0x44/0x70 [ 643.436404] __blk_mq_free_request+0x88/0xf0 [ 643.440660] blk_mq_free_request+0xc0/0x160 [ 643.444831] __blk_mq_end_request+0x78/0x150 [ 643.449088] scsi_end_request+0x114/0x260 [ 643.453085] scsi_io_completion+0x64/0x174 [ 643.457169] scsi_finish_command+0xe0/0x120 [ 643.461339] scsi_complete+0x8c/0x1c4 [ 643.464988] blk_complete_reqs+0x5c/0x74 [ 643.468899] blk_done_softirq+0x38/0x4c [ 643.472722] __do_softirq+0x168/0x418 [ 643.476370] __irq_exit_rcu+0x120/0x170 [ 643.480193] irq_exit_rcu+0x18/0x24 [ 643.483669] el1_interrupt+0x38/0x70 [ 643.487231] el1h_64_irq_handler+0x18/0x24 [ 643.491314] el1h_64_irq+0x68/0x6c [ 643.494702] cpuidle_enter_state+0xc4/0x4e0 [ 643.498872] cpuidle_enter+0x40/0x60 [ 643.502434] cpuidle_idle_call+0x118/0x1b0 [ 643.506518] do_idle+0xac/0x100 [ 643.509647] cpu_startup_entry+0x30/0x34 [ 643.513556] secondary_start_kernel+0xd8/0xf0 [ 643.517900] __secondary_switched+0xb0/0xb4 [ 655.110499] watchdog: BUG: soft lockup - CPU#8 stuck for 216s! [kworker/8:12:19399] [ 655.118141] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 655.118158] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 655.205945] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 655.214195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 655.223486] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 655.229653] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 655.236601] pc : __sbq_wake_up+0x7c/0x1f0 [ 655.240598] lr : sbitmap_queue_clear+0x74/0xc0 [ 655.245029] sp : ffff80000d79b2d0 [ 655.248330] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 655.255452] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 655.262574] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 655.269696] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 655.276818] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 655.283940] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 655.291064] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 655.298185] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 655.305308] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 655.312430] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 655.319552] Call trace: [ 655.321985] __sbq_wake_up+0x7c/0x1f0 [ 655.325634] blk_mq_put_tag+0x44/0x70 [ 655.329284] __blk_mq_free_request+0x88/0xf0 [ 655.333541] blk_mq_free_request+0xc0/0x160 [ 655.337712] __blk_mq_end_request+0x78/0x150 [ 655.341969] scsi_end_request+0x114/0x260 [ 655.345966] scsi_io_completion+0x64/0x174 [ 655.350049] scsi_finish_command+0xe0/0x120 [ 655.354220] scsi_complete+0x8c/0x1c4 [ 655.357869] blk_complete_reqs+0x5c/0x74 [ 655.361778] blk_done_softirq+0x38/0x4c [ 655.365601] __do_softirq+0x168/0x418 [ 655.369250] __irq_exit_rcu+0x120/0x170 [ 655.373072] irq_exit_rcu+0x18/0x24 [ 655.376548] el1_interrupt+0x38/0x70 [ 655.380110] el1h_64_irq_handler+0x18/0x24 [ 655.384194] el1h_64_irq+0x68/0x6c [ 655.387582] _raw_spin_unlock_irqrestore+0x20/0x50 [ 655.392360] __wake_up_common_lock+0x84/0xac [ 655.396617] __wake_up+0x20/0x30 [ 655.399833] __sbq_wake_up+0x13c/0x1f0 [ 655.403569] sbitmap_queue_clear+0x74/0xc0 [ 655.407653] blk_mq_put_tag+0x44/0x70 [ 655.411303] __blk_mq_free_request+0x88/0xf0 [ 655.415559] blk_mq_free_request+0xc0/0x160 [ 655.419729] bfq_insert_request.isra.0+0x204/0x270 [ 655.424507] bfq_insert_requests+0x70/0x90 [ 655.428591] blk_mq_sched_insert_requests+0xac/0x250 [ 655.433543] blk_mq_flush_plug_list+0x110/0x2cc [ 655.438061] __blk_flush_plug+0x100/0x140 [ 655.442057] blk_finish_plug+0x40/0x25c [ 655.445880] __iomap_dio_rw+0x1b4/0x500 [ 655.449702] iomap_dio_rw+0x18/0x50 [ 655.453178] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 655.458529] xfs_file_write_iter+0xf4/0x15c [xfs] [ 655.463272] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 655.468316] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 655.473447] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 655.478576] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 655.483962] process_one_work+0x1ec/0x470 [ 655.487958] worker_thread+0x74/0x410 [ 655.491607] kthread+0xec/0x100 [ 655.494736] ret_from_fork+0x10/0x20 [ 665.250436] rcu: INFO: rcu_preempt self-detected stall on CPU [ 665.256169] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=11962 [ 665.265292] (t=24035 jiffies g=36445 q=7224 ncpus=128) [ 665.270504] Task dump for CPU 8: [ 665.273718] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 665.283619] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 665.289789] Call trace: [ 665.292223] __switch_to+0xd4/0x100 [ 665.295699] 0xffff6c0200000080 [ 665.298829] Task dump for CPU 68: [ 665.302130] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 665.312030] Call trace: [ 665.314464] dump_backtrace+0xe8/0x130 [ 665.318201] show_stack+0x20/0x60 [ 665.321503] sched_show_task+0x160/0x190 [ 665.325413] dump_cpu_task+0x4c/0x78 [ 665.328977] rcu_dump_cpu_stacks+0xc4/0x150 [ 665.333148] print_cpu_stall+0x180/0x210 [ 665.337058] check_cpu_stall+0x20c/0x2b4 [ 665.340967] rcu_pending+0x4c/0x190 [ 665.344443] rcu_sched_clock_irq+0x80/0x204 [ 665.348613] update_process_times+0x7c/0xc0 [ 665.352783] tick_sched_handle+0x38/0x74 [ 665.356694] tick_sched_timer+0x60/0xc0 [ 665.360517] __hrtimer_run_queues+0x170/0x300 [ 665.364861] hrtimer_interrupt+0xfc/0x260 [ 665.368857] arch_timer_handler_phys+0x34/0x50 [ 665.373288] handle_percpu_devid_irq+0xac/0x240 [ 665.377805] generic_handle_domain_irq+0x34/0x50 [ 665.382410] gic_handle_irq+0x5c/0xe0 [ 665.386059] call_on_irq_stack+0x2c/0x38 [ 665.389969] do_interrupt_handler+0xdc/0xe0 [ 665.394140] el1_interrupt+0x34/0x70 [ 665.397703] el1h_64_irq_handler+0x18/0x24 [ 665.401786] el1h_64_irq+0x68/0x6c [ 665.405174] __sbq_wake_up+0x84/0x1f0 [ 665.408823] sbitmap_queue_clear+0x74/0xc0 [ 665.412907] blk_mq_put_tag+0x44/0x70 [ 665.416556] __blk_mq_free_request+0x88/0xf0 [ 665.420813] blk_mq_free_request+0xc0/0x160 [ 665.424983] __blk_mq_end_request+0x78/0x150 [ 665.429240] scsi_end_request+0x114/0x260 [ 665.433236] scsi_io_completion+0x64/0x174 [ 665.437319] scsi_finish_command+0xe0/0x120 [ 665.441490] scsi_complete+0x8c/0x1c4 [ 665.445138] blk_complete_reqs+0x5c/0x74 [ 665.449048] blk_done_softirq+0x38/0x4c [ 665.452871] __do_softirq+0x168/0x418 [ 665.456519] __irq_exit_rcu+0x120/0x170 [ 665.460342] irq_exit_rcu+0x18/0x24 [ 665.463817] el1_interrupt+0x38/0x70 [ 665.467380] el1h_64_irq_handler+0x18/0x24 [ 665.471463] el1h_64_irq+0x68/0x6c [ 665.474851] cpuidle_enter_state+0xc4/0x4e0 [ 665.479022] cpuidle_enter+0x40/0x60 [ 665.482585] cpuidle_idle_call+0x118/0x1b0 [ 665.486668] do_idle+0xac/0x100 [ 665.489797] cpu_startup_entry+0x30/0x34 [ 665.493708] secondary_start_kernel+0xd8/0xf0 [ 665.498052] __secondary_switched+0xb0/0xb4 [ 671.730457] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 22077 jiffies s: 2093 root: 0x11/. [ 671.742111] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 671.751321] Task dump for CPU 8: [ 671.754536] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 671.764439] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 671.770611] Call trace: [ 671.773044] __switch_to+0xd4/0x100 [ 671.776521] 0xffff6c0200000080 [ 671.779651] Task dump for CPU 68: [ 671.782954] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 671.792860] Call trace: [ 671.795294] __switch_to+0xd4/0x100 [ 671.798770] 0x0 [ 679.110349] watchdog: BUG: soft lockup - CPU#8 stuck for 238s! [kworker/8:12:19399] [ 679.117991] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 679.118008] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 679.205795] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 679.214044] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 679.223335] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 679.229503] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 679.236450] pc : __sbq_wake_up+0x7c/0x1f0 [ 679.240447] lr : sbitmap_queue_clear+0x74/0xc0 [ 679.244878] sp : ffff80000d79b2d0 [ 679.248179] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 679.255301] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 679.262424] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 679.269546] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 679.276668] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 679.283790] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 679.290912] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 679.298033] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 679.305155] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 679.312277] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 679.319399] Call trace: [ 679.321832] __sbq_wake_up+0x7c/0x1f0 [ 679.325482] blk_mq_put_tag+0x44/0x70 [ 679.329131] __blk_mq_free_request+0x88/0xf0 [ 679.333388] blk_mq_free_request+0xc0/0x160 [ 679.337558] __blk_mq_end_request+0x78/0x150 [ 679.341815] scsi_end_request+0x114/0x260 [ 679.345812] scsi_io_completion+0x64/0x174 [ 679.349895] scsi_finish_command+0xe0/0x120 [ 679.354065] scsi_complete+0x8c/0x1c4 [ 679.357714] blk_complete_reqs+0x5c/0x74 [ 679.361624] blk_done_softirq+0x38/0x4c [ 679.365446] __do_softirq+0x168/0x418 [ 679.369095] __irq_exit_rcu+0x120/0x170 [ 679.372918] irq_exit_rcu+0x18/0x24 [ 679.376394] el1_interrupt+0x38/0x70 [ 679.379956] el1h_64_irq_handler+0x18/0x24 [ 679.384039] el1h_64_irq+0x68/0x6c [ 679.387428] _raw_spin_unlock_irqrestore+0x20/0x50 [ 679.392205] __wake_up_common_lock+0x84/0xac [ 679.396462] __wake_up+0x20/0x30 [ 679.399678] __sbq_wake_up+0x13c/0x1f0 [ 679.403414] sbitmap_queue_clear+0x74/0xc0 [ 679.407497] blk_mq_put_tag+0x44/0x70 [ 679.411147] __blk_mq_free_request+0x88/0xf0 [ 679.415404] blk_mq_free_request+0xc0/0x160 [ 679.419574] bfq_insert_request.isra.0+0x204/0x270 [ 679.424352] bfq_insert_requests+0x70/0x90 [ 679.428435] blk_mq_sched_insert_requests+0xac/0x250 [ 679.433387] blk_mq_flush_plug_list+0x110/0x2cc [ 679.437905] __blk_flush_plug+0x100/0x140 [ 679.441901] blk_finish_plug+0x40/0x25c [ 679.445724] __iomap_dio_rw+0x1b4/0x500 [ 679.449546] iomap_dio_rw+0x18/0x50 [ 679.453022] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 679.458373] xfs_file_write_iter+0xf4/0x15c [xfs] [ 679.463116] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 679.468160] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 679.473290] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 679.478420] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 679.483806] process_one_work+0x1ec/0x470 [ 679.487803] worker_thread+0x74/0x410 [ 679.491451] kthread+0xec/0x100 [ 679.494580] ret_from_fork+0x10/0x20 [ 691.220273] watchdog: BUG: soft lockup - CPU#68 stuck for 249s! [swapper/68:0] [ 691.227482] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 691.227498] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 691.315285] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 691.323101] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 691.332392] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 691.339339] pc : __sbq_wake_up+0x58/0x1f0 [ 691.343335] lr : sbitmap_queue_clear+0x74/0xc0 [ 691.347766] sp : ffff8000088438b0 [ 691.351067] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 691.358189] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 691.365313] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 691.372435] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 691.379557] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 691.386679] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 691.393801] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 691.400923] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 691.408045] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 691.415166] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 691.422288] Call trace: [ 691.424721] __sbq_wake_up+0x58/0x1f0 [ 691.428370] sbitmap_queue_clear+0x74/0xc0 [ 691.432454] blk_mq_put_tag+0x44/0x70 [ 691.436103] __blk_mq_free_request+0x88/0xf0 [ 691.440360] blk_mq_free_request+0xc0/0x160 [ 691.444530] __blk_mq_end_request+0x78/0x150 [ 691.448786] scsi_end_request+0x114/0x260 [ 691.452783] scsi_io_completion+0x64/0x174 [ 691.456866] scsi_finish_command+0xe0/0x120 [ 691.461036] scsi_complete+0x8c/0x1c4 [ 691.464685] blk_complete_reqs+0x5c/0x74 [ 691.468595] blk_done_softirq+0x38/0x4c [ 691.472418] __do_softirq+0x168/0x418 [ 691.476067] __irq_exit_rcu+0x120/0x170 [ 691.479890] irq_exit_rcu+0x18/0x24 [ 691.483367] el1_interrupt+0x38/0x70 [ 691.486929] el1h_64_irq_handler+0x18/0x24 [ 691.491012] el1h_64_irq+0x68/0x6c [ 691.494401] cpuidle_enter_state+0xc4/0x4e0 [ 691.498571] cpuidle_enter+0x40/0x60 [ 691.502133] cpuidle_idle_call+0x118/0x1b0 [ 691.506217] do_idle+0xac/0x100 [ 691.509346] cpu_startup_entry+0x30/0x34 [ 691.513256] secondary_start_kernel+0xd8/0xf0 [ 691.517600] __secondary_switched+0xb0/0xb4 [ 703.110199] watchdog: BUG: soft lockup - CPU#8 stuck for 260s! [kworker/8:12:19399] [ 703.117841] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 703.117858] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 703.205645] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 703.213896] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 703.223187] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 703.229355] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 703.236303] pc : __sbq_wake_up+0x1c8/0x1f0 [ 703.240386] lr : sbitmap_queue_clear+0x74/0xc0 [ 703.244817] sp : ffff80000d79b280 [ 703.248118] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 703.255240] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 703.262362] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 703.269484] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 703.276606] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 703.283728] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 703.290849] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 703.297971] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 703.305093] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 703.312215] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 703.319337] Call trace: [ 703.321769] __sbq_wake_up+0x1c8/0x1f0 [ 703.325505] sbitmap_queue_clear+0x74/0xc0 [ 703.329589] blk_mq_put_tag+0x44/0x70 [ 703.333239] __blk_mq_free_request+0x88/0xf0 [ 703.337495] blk_mq_free_request+0xc0/0x160 [ 703.341665] __blk_mq_end_request+0x78/0x150 [ 703.345922] scsi_end_request+0x114/0x260 [ 703.349918] scsi_io_completion+0x64/0x174 [ 703.354001] scsi_finish_command+0xe0/0x120 [ 703.358172] scsi_complete+0x8c/0x1c4 [ 703.361821] blk_complete_reqs+0x5c/0x74 [ 703.365730] blk_done_softirq+0x38/0x4c [ 703.369553] __do_softirq+0x168/0x418 [ 703.373202] __irq_exit_rcu+0x120/0x170 [ 703.377025] irq_exit_rcu+0x18/0x24 [ 703.380500] el1_interrupt+0x38/0x70 [ 703.384063] el1h_64_irq_handler+0x18/0x24 [ 703.388146] el1h_64_irq+0x68/0x6c [ 703.391535] _raw_spin_unlock_irqrestore+0x20/0x50 [ 703.396313] __wake_up_common_lock+0x84/0xac [ 703.400571] __wake_up+0x20/0x30 [ 703.403786] __sbq_wake_up+0x13c/0x1f0 [ 703.407522] sbitmap_queue_clear+0x74/0xc0 [ 703.411605] blk_mq_put_tag+0x44/0x70 [ 703.415255] __blk_mq_free_request+0x88/0xf0 [ 703.419512] blk_mq_free_request+0xc0/0x160 [ 703.423682] bfq_insert_request.isra.0+0x204/0x270 [ 703.428460] bfq_insert_requests+0x70/0x90 [ 703.432543] blk_mq_sched_insert_requests+0xac/0x250 [ 703.437496] blk_mq_flush_plug_list+0x110/0x2cc [ 703.442013] __blk_flush_plug+0x100/0x140 [ 703.446010] blk_finish_plug+0x40/0x25c [ 703.449832] __iomap_dio_rw+0x1b4/0x500 [ 703.453655] iomap_dio_rw+0x18/0x50 [ 703.457130] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 703.462481] xfs_file_write_iter+0xf4/0x15c [xfs] [ 703.467224] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 703.472268] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 703.477398] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 703.482527] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 703.487913] process_one_work+0x1ec/0x470 [ 703.491909] worker_thread+0x74/0x410 [ 703.495558] kthread+0xec/0x100 [ 703.498687] ret_from_fork+0x10/0x20 [ 715.220123] watchdog: BUG: soft lockup - CPU#68 stuck for 272s! [swapper/68:0] [ 715.227331] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 715.227348] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 715.315135] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 715.322950] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 715.332241] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 715.339189] pc : __sbq_wake_up+0x1c8/0x1f0 [ 715.343272] lr : sbitmap_queue_clear+0x74/0xc0 [ 715.347702] sp : ffff8000088438b0 [ 715.351004] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 715.358125] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 715.365247] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 715.372370] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 715.379491] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 715.386615] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 715.393737] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 715.400859] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 715.407981] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 715.415103] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 715.422225] Call trace: [ 715.424658] __sbq_wake_up+0x1c8/0x1f0 [ 715.428394] sbitmap_queue_clear+0x74/0xc0 [ 715.432477] blk_mq_put_tag+0x44/0x70 [ 715.436126] __blk_mq_free_request+0x88/0xf0 [ 715.440383] blk_mq_free_request+0xc0/0x160 [ 715.444553] __blk_mq_end_request+0x78/0x150 [ 715.448810] scsi_end_request+0x114/0x260 [ 715.452806] scsi_io_completion+0x64/0x174 [ 715.456889] scsi_finish_command+0xe0/0x120 [ 715.461060] scsi_complete+0x8c/0x1c4 [ 715.464709] blk_complete_reqs+0x5c/0x74 [ 715.468618] blk_done_softirq+0x38/0x4c [ 715.472441] __do_softirq+0x168/0x418 [ 715.476089] __irq_exit_rcu+0x120/0x170 [ 715.479912] irq_exit_rcu+0x18/0x24 [ 715.483388] el1_interrupt+0x38/0x70 [ 715.486950] el1h_64_irq_handler+0x18/0x24 [ 715.491033] el1h_64_irq+0x68/0x6c [ 715.494422] cpuidle_enter_state+0xc4/0x4e0 [ 715.498592] cpuidle_enter+0x40/0x60 [ 715.502154] cpuidle_idle_call+0x118/0x1b0 [ 715.506240] do_idle+0xac/0x100 [ 715.509369] cpu_startup_entry+0x30/0x34 [ 715.513278] secondary_start_kernel+0xd8/0xf0 [ 715.517622] __secondary_switched+0xb0/0xb4 [ 727.110048] watchdog: BUG: soft lockup - CPU#8 stuck for 283s! [kworker/8:12:19399] [ 727.117691] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 727.117708] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 727.205495] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 727.213744] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 727.223036] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 727.229204] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 727.236152] pc : __sbq_wake_up+0x84/0x1f0 [ 727.240149] lr : sbitmap_queue_clear+0x74/0xc0 [ 727.244580] sp : ffff80000d79b280 [ 727.247880] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 727.255003] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 727.262124] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 727.269247] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 727.276370] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 727.283492] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 727.290613] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 727.297735] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 727.304857] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 727.311979] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 727.319101] Call trace: [ 727.321533] __sbq_wake_up+0x84/0x1f0 [ 727.325183] sbitmap_queue_clear+0x74/0xc0 [ 727.329267] blk_mq_put_tag+0x44/0x70 [ 727.332916] __blk_mq_free_request+0x88/0xf0 [ 727.337173] blk_mq_free_request+0xc0/0x160 [ 727.341343] __blk_mq_end_request+0x78/0x150 [ 727.345600] scsi_end_request+0x114/0x260 [ 727.349597] scsi_io_completion+0x64/0x174 [ 727.353680] scsi_finish_command+0xe0/0x120 [ 727.357850] scsi_complete+0x8c/0x1c4 [ 727.361499] blk_complete_reqs+0x5c/0x74 [ 727.365409] blk_done_softirq+0x38/0x4c [ 727.369231] __do_softirq+0x168/0x418 [ 727.372880] __irq_exit_rcu+0x120/0x170 [ 727.376703] irq_exit_rcu+0x18/0x24 [ 727.380179] el1_interrupt+0x38/0x70 [ 727.383741] el1h_64_irq_handler+0x18/0x24 [ 727.387824] el1h_64_irq+0x68/0x6c [ 727.391213] _raw_spin_unlock_irqrestore+0x20/0x50 [ 727.395990] __wake_up_common_lock+0x84/0xac [ 727.400247] __wake_up+0x20/0x30 [ 727.403463] __sbq_wake_up+0x13c/0x1f0 [ 727.407199] sbitmap_queue_clear+0x74/0xc0 [ 727.411282] blk_mq_put_tag+0x44/0x70 [ 727.414932] __blk_mq_free_request+0x88/0xf0 [ 727.419188] blk_mq_free_request+0xc0/0x160 [ 727.423358] bfq_insert_request.isra.0+0x204/0x270 [ 727.428136] bfq_insert_requests+0x70/0x90 [ 727.432220] blk_mq_sched_insert_requests+0xac/0x250 [ 727.437172] blk_mq_flush_plug_list+0x110/0x2cc [ 727.441689] __blk_flush_plug+0x100/0x140 [ 727.445686] blk_finish_plug+0x40/0x25c [ 727.449508] __iomap_dio_rw+0x1b4/0x500 [ 727.453331] iomap_dio_rw+0x18/0x50 [ 727.456806] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 727.462158] xfs_file_write_iter+0xf4/0x15c [xfs] [ 727.466900] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 727.471944] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 727.477074] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 727.482203] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 727.487589] process_one_work+0x1ec/0x470 [ 727.491585] worker_thread+0x74/0x410 [ 727.495234] kthread+0xec/0x100 [ 727.498363] ret_from_fork+0x10/0x20 [ 738.290024] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 28733 jiffies s: 2093 root: 0x11/. [ 738.301679] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 738.310888] Task dump for CPU 8: [ 738.314103] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 738.324006] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 738.330176] Call trace: [ 738.332609] __switch_to+0xd4/0x100 [ 738.336087] 0xffff6c0200000080 [ 738.339215] Task dump for CPU 68: [ 738.342518] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 738.352419] Call trace: [ 738.354853] __switch_to+0xd4/0x100 [ 738.358329] 0x0 [ 739.219973] watchdog: BUG: soft lockup - CPU#68 stuck for 294s! [swapper/68:0] [ 739.227181] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 739.227197] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 739.314984] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 739.322800] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 739.332090] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 739.339038] pc : __sbq_wake_up+0x58/0x1f0 [ 739.343034] lr : sbitmap_queue_clear+0x74/0xc0 [ 739.347465] sp : ffff8000088438b0 [ 739.350765] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 739.357887] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 739.365009] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 739.372131] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 739.379253] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 739.386375] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 739.393497] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 739.400620] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 739.407742] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 739.414863] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 739.421985] Call trace: [ 739.424419] __sbq_wake_up+0x58/0x1f0 [ 739.428068] sbitmap_queue_clear+0x74/0xc0 [ 739.432151] blk_mq_put_tag+0x44/0x70 [ 739.435801] __blk_mq_free_request+0x88/0xf0 [ 739.440058] blk_mq_free_request+0xc0/0x160 [ 739.444228] __blk_mq_end_request+0x78/0x150 [ 739.448484] scsi_end_request+0x114/0x260 [ 739.452480] scsi_io_completion+0x64/0x174 [ 739.456564] scsi_finish_command+0xe0/0x120 [ 739.460734] scsi_complete+0x8c/0x1c4 [ 739.464383] blk_complete_reqs+0x5c/0x74 [ 739.468293] blk_done_softirq+0x38/0x4c [ 739.472115] __do_softirq+0x168/0x418 [ 739.475764] __irq_exit_rcu+0x120/0x170 [ 739.479587] irq_exit_rcu+0x18/0x24 [ 739.483062] el1_interrupt+0x38/0x70 [ 739.486624] el1h_64_irq_handler+0x18/0x24 [ 739.490707] el1h_64_irq+0x68/0x6c [ 739.494096] cpuidle_enter_state+0xc4/0x4e0 [ 739.498268] cpuidle_enter+0x40/0x60 [ 739.501830] cpuidle_idle_call+0x118/0x1b0 [ 739.505914] do_idle+0xac/0x100 [ 739.509043] cpu_startup_entry+0x30/0x34 [ 739.512952] secondary_start_kernel+0xd8/0xf0 [ 739.517297] __secondary_switched+0xb0/0xb4 [ 751.109898] watchdog: BUG: soft lockup - CPU#8 stuck for 305s! [kworker/8:12:19399] [ 751.117540] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 751.117557] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 751.205344] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 751.213594] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 751.222885] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 751.229052] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 751.236000] pc : __sbq_wake_up+0x1c8/0x1f0 [ 751.240084] lr : sbitmap_queue_clear+0x74/0xc0 [ 751.244514] sp : ffff80000d79b280 [ 751.247815] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 751.254937] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 751.262059] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 751.269181] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 751.276303] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 751.283425] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 751.290547] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 751.297669] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 751.304790] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 751.311912] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 751.319034] Call trace: [ 751.321467] __sbq_wake_up+0x1c8/0x1f0 [ 751.325204] sbitmap_queue_clear+0x74/0xc0 [ 751.329287] blk_mq_put_tag+0x44/0x70 [ 751.332937] __blk_mq_free_request+0x88/0xf0 [ 751.337194] blk_mq_free_request+0xc0/0x160 [ 751.341364] __blk_mq_end_request+0x78/0x150 [ 751.345620] scsi_end_request+0x114/0x260 [ 751.349617] scsi_io_completion+0x64/0x174 [ 751.353700] scsi_finish_command+0xe0/0x120 [ 751.357870] scsi_complete+0x8c/0x1c4 [ 751.361519] blk_complete_reqs+0x5c/0x74 [ 751.365429] blk_done_softirq+0x38/0x4c [ 751.369252] __do_softirq+0x168/0x418 [ 751.372901] __irq_exit_rcu+0x120/0x170 [ 751.376723] irq_exit_rcu+0x18/0x24 [ 751.380199] el1_interrupt+0x38/0x70 [ 751.383761] el1h_64_irq_handler+0x18/0x24 [ 751.387844] el1h_64_irq+0x68/0x6c [ 751.391233] _raw_spin_unlock_irqrestore+0x20/0x50 [ 751.396010] __wake_up_common_lock+0x84/0xac [ 751.400267] __wake_up+0x20/0x30 [ 751.403483] __sbq_wake_up+0x13c/0x1f0 [ 751.407219] sbitmap_queue_clear+0x74/0xc0 [ 751.411303] blk_mq_put_tag+0x44/0x70 [ 751.414952] __blk_mq_free_request+0x88/0xf0 [ 751.419209] blk_mq_free_request+0xc0/0x160 [ 751.423379] bfq_insert_request.isra.0+0x204/0x270 [ 751.428157] bfq_insert_requests+0x70/0x90 [ 751.432240] blk_mq_sched_insert_requests+0xac/0x250 [ 751.437192] blk_mq_flush_plug_list+0x110/0x2cc [ 751.441710] __blk_flush_plug+0x100/0x140 [ 751.445706] blk_finish_plug+0x40/0x25c [ 751.449529] __iomap_dio_rw+0x1b4/0x500 [ 751.453352] iomap_dio_rw+0x18/0x50 [ 751.456827] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 751.462178] xfs_file_write_iter+0xf4/0x15c [xfs] [ 751.466920] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 751.471964] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 751.477094] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 751.482224] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 751.487610] process_one_work+0x1ec/0x470 [ 751.491606] worker_thread+0x74/0x410 [ 751.495255] kthread+0xec/0x100 [ 751.498383] ret_from_fork+0x10/0x20 [ 763.219822] watchdog: BUG: soft lockup - CPU#68 stuck for 316s! [swapper/68:0] [ 763.227030] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 763.227047] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 763.314834] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 763.322649] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 763.331940] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 763.338888] pc : __sbq_wake_up+0x84/0x1f0 [ 763.342884] lr : sbitmap_queue_clear+0x74/0xc0 [ 763.347315] sp : ffff8000088438b0 [ 763.350616] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 763.357738] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 763.364860] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 763.371982] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 763.379104] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 763.386225] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 763.393347] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 763.400469] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 763.407591] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 763.414713] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 763.421835] Call trace: [ 763.424267] __sbq_wake_up+0x84/0x1f0 [ 763.427916] sbitmap_queue_clear+0x74/0xc0 [ 763.431999] blk_mq_put_tag+0x44/0x70 [ 763.435649] __blk_mq_free_request+0x88/0xf0 [ 763.439906] blk_mq_free_request+0xc0/0x160 [ 763.444076] __blk_mq_end_request+0x78/0x150 [ 763.448332] scsi_end_request+0x114/0x260 [ 763.452329] scsi_io_completion+0x64/0x174 [ 763.456412] scsi_finish_command+0xe0/0x120 [ 763.460582] scsi_complete+0x8c/0x1c4 [ 763.464231] blk_complete_reqs+0x5c/0x74 [ 763.468141] blk_done_softirq+0x38/0x4c [ 763.471963] __do_softirq+0x168/0x418 [ 763.475612] __irq_exit_rcu+0x120/0x170 [ 763.479435] irq_exit_rcu+0x18/0x24 [ 763.482910] el1_interrupt+0x38/0x70 [ 763.486472] el1h_64_irq_handler+0x18/0x24 [ 763.490556] el1h_64_irq+0x68/0x6c [ 763.493944] cpuidle_enter_state+0xc4/0x4e0 [ 763.498114] cpuidle_enter+0x40/0x60 [ 763.501676] cpuidle_idle_call+0x118/0x1b0 [ 763.505759] do_idle+0xac/0x100 [ 763.508888] cpu_startup_entry+0x30/0x34 [ 763.512798] secondary_start_kernel+0xd8/0xf0 [ 763.517144] __secondary_switched+0xb0/0xb4 [ 775.109747] watchdog: BUG: soft lockup - CPU#8 stuck for 327s! [kworker/8:12:19399] [ 775.117390] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 775.117406] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 775.205194] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 775.213443] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 775.222734] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 775.228902] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 775.235849] pc : __sbq_wake_up+0x58/0x1f0 [ 775.239846] lr : sbitmap_queue_clear+0x74/0xc0 [ 775.244276] sp : ffff80000d79b280 [ 775.247577] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 775.254700] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 775.261821] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 775.268943] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 775.276065] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 775.283187] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 775.290309] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 775.297431] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 775.304552] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 775.311674] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 775.318796] Call trace: [ 775.321229] __sbq_wake_up+0x58/0x1f0 [ 775.324878] sbitmap_queue_clear+0x74/0xc0 [ 775.328961] blk_mq_put_tag+0x44/0x70 [ 775.332611] __blk_mq_free_request+0x88/0xf0 [ 775.336867] blk_mq_free_request+0xc0/0x160 [ 775.341037] __blk_mq_end_request+0x78/0x150 [ 775.345294] scsi_end_request+0x114/0x260 [ 775.349290] scsi_io_completion+0x64/0x174 [ 775.353373] scsi_finish_command+0xe0/0x120 [ 775.357544] scsi_complete+0x8c/0x1c4 [ 775.361193] blk_complete_reqs+0x5c/0x74 [ 775.365102] blk_done_softirq+0x38/0x4c [ 775.368925] __do_softirq+0x168/0x418 [ 775.372573] __irq_exit_rcu+0x120/0x170 [ 775.376396] irq_exit_rcu+0x18/0x24 [ 775.379872] el1_interrupt+0x38/0x70 [ 775.383434] el1h_64_irq_handler+0x18/0x24 [ 775.387517] el1h_64_irq+0x68/0x6c [ 775.390906] _raw_spin_unlock_irqrestore+0x20/0x50 [ 775.395684] __wake_up_common_lock+0x84/0xac [ 775.399941] __wake_up+0x20/0x30 [ 775.403157] __sbq_wake_up+0x13c/0x1f0 [ 775.406892] sbitmap_queue_clear+0x74/0xc0 [ 775.410976] blk_mq_put_tag+0x44/0x70 [ 775.414626] __blk_mq_free_request+0x88/0xf0 [ 775.418882] blk_mq_free_request+0xc0/0x160 [ 775.423053] bfq_insert_request.isra.0+0x204/0x270 [ 775.427831] bfq_insert_requests+0x70/0x90 [ 775.431915] blk_mq_sched_insert_requests+0xac/0x250 [ 775.436866] blk_mq_flush_plug_list+0x110/0x2cc [ 775.441384] __blk_flush_plug+0x100/0x140 [ 775.445380] blk_finish_plug+0x40/0x25c [ 775.449203] __iomap_dio_rw+0x1b4/0x500 [ 775.453026] iomap_dio_rw+0x18/0x50 [ 775.456501] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 775.461852] xfs_file_write_iter+0xf4/0x15c [xfs] [ 775.466594] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 775.471638] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 775.476768] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 775.481898] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 775.487284] process_one_work+0x1ec/0x470 [ 775.491280] worker_thread+0x74/0x410 [ 775.494929] kthread+0xec/0x100 [ 775.498058] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 04:10:00 2022] [ 787.219671] watchdog: BUG: soft lockup - CPU#68 stuck for 339s! [swapper/68:0] [ 787.226879] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 787.226896] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 787.314682] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 787.322498] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 787.331789] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 787.338737] pc : __sbq_wake_up+0x58/0x1f0 [ 787.342733] lr : sbitmap_queue_clear+0x74/0xc0 [ 787.347164] sp : ffff8000088438b0 [ 787.350465] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 787.357587] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 787.364709] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 787.371830] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 787.378952] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 787.386074] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 787.393196] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 787.400318] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 787.407440] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 787.414561] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 787.421683] Call trace: [ 787.424116] __sbq_wake_up+0x58/0x1f0 [ 787.427765] sbitmap_queue_clear+0x74/0xc0 [ 787.431848] blk_mq_put_tag+0x44/0x70 [ 787.435498] __blk_mq_free_request+0x88/0xf0 [ 787.439755] blk_mq_free_request+0xc0/0x160 [ 787.443924] __blk_mq_end_request+0x78/0x150 [ 787.448181] scsi_end_request+0x114/0x260 [ 787.452177] scsi_io_completion+0x64/0x174 [ 787.456261] scsi_finish_command+0xe0/0x120 [ 787.460431] scsi_complete+0x8c/0x1c4 [ 787.464081] blk_complete_reqs+0x5c/0x74 [ 787.467990] blk_done_softirq+0x38/0x4c [ 787.471813] __do_softirq+0x168/0x418 [ 787.475462] __irq_exit_rcu+0x120/0x170 [ 787.479285] irq_exit_rcu+0x18/0x24 [ 787.482760] el1_interrupt+0x38/0x70 [ 787.486322] el1h_64_irq_handler+0x18/0x24 [ 787.490405] el1h_64_irq+0x68/0x6c [ 787.493794] cpuidle_enter_state+0xc4/0x4e0 [ 787.497964] cpuidle_enter+0x40/0x60 [ 787.501526] cpuidle_idle_call+0x118/0x1b0 [ 787.505610] do_idle+0xac/0x100 [ 787.508739] cpu_startup_entry+0x30/0x34 [ 787.512648] secondary_start_kernel+0xd8/0xf0 [ 787.516992] __secondary_switched+0xb0/0xb4 [ 799.109595] watchdog: BUG: soft lockup - CPU#8 stuck for 350s! [kworker/8:12:19399] [ 799.117238] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 799.117254] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 799.205042] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 799.213291] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 799.222582] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 799.228750] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 799.235699] pc : __sbq_wake_up+0x38/0x1f0 [ 799.239695] lr : sbitmap_queue_clear+0x74/0xc0 [ 799.244126] sp : ffff80000d79b280 [ 799.247427] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 799.254549] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 799.261671] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 799.268794] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 799.275915] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 799.283037] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 799.290160] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 799.297282] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 799.304403] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 799.311525] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000036 [ 799.318647] Call trace: [ 799.321080] __sbq_wake_up+0x38/0x1f0 [ 799.324730] sbitmap_queue_clear+0x74/0xc0 [ 799.328813] blk_mq_put_tag+0x44/0x70 [ 799.332463] __blk_mq_free_request+0x88/0xf0 [ 799.336720] blk_mq_free_request+0xc0/0x160 [ 799.340890] __blk_mq_end_request+0x78/0x150 [ 799.345147] scsi_end_request+0x114/0x260 [ 799.349143] scsi_io_completion+0x64/0x174 [ 799.353226] scsi_finish_command+0xe0/0x120 [ 799.357397] scsi_complete+0x8c/0x1c4 [ 799.361046] blk_complete_reqs+0x5c/0x74 [ 799.364955] blk_done_softirq+0x38/0x4c [ 799.368778] __do_softirq+0x168/0x418 [ 799.372427] __irq_exit_rcu+0x120/0x170 [ 799.376250] irq_exit_rcu+0x18/0x24 [ 799.379725] el1_interrupt+0x38/0x70 [ 799.383288] el1h_64_irq_handler+0x18/0x24 [ 799.387371] el1h_64_irq+0x68/0x6c [ 799.390759] _raw_spin_unlock_irqrestore+0x20/0x50 [ 799.395537] __wake_up_common_lock+0x84/0xac [ 799.399794] __wake_up+0x20/0x30 [ 799.403010] __sbq_wake_up+0x13c/0x1f0 [ 799.406746] sbitmap_queue_clear+0x74/0xc0 [ 799.410829] blk_mq_put_tag+0x44/0x70 [ 799.414479] __blk_mq_free_request+0x88/0xf0 [ 799.418735] blk_mq_free_request+0xc0/0x160 [ 799.422905] bfq_insert_request.isra.0+0x204/0x270 [ 799.427684] bfq_insert_requests+0x70/0x90 [ 799.431767] blk_mq_sched_insert_requests+0xac/0x250 [ 799.436719] blk_mq_flush_plug_list+0x110/0x2cc [ 799.441237] __blk_flush_plug+0x100/0x140 [ 799.445233] blk_finish_plug+0x40/0x25c [ 799.449056] __iomap_dio_rw+0x1b4/0x500 [ 799.452879] iomap_dio_rw+0x18/0x50 [ 799.456354] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 799.461705] xfs_file_write_iter+0xf4/0x15c [xfs] [ 799.466448] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 799.471492] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 799.476622] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 799.481752] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 799.487138] process_one_work+0x1ec/0x470 [ 799.491134] worker_thread+0x74/0x410 [ 799.494783] kthread+0xec/0x100 [ 799.497912] ret_from_fork+0x10/0x20 [ 804.849614] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 35389 jiffies s: 2093 root: 0x11/. [ 804.861267] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 804.870477] Task dump for CPU 8: [ 804.873692] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 804.883594] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 804.889764] Call trace: [ 804.892197] __switch_to+0xd4/0x100 [ 804.895674] 0xffff6c0200000080 [ 804.898803] Task dump for CPU 68: [ 804.902106] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 804.912007] Call trace: [ 804.914440] __switch_to+0xd4/0x100 [ 804.917916] 0x0 [ 811.219519] watchdog: BUG: soft lockup - CPU#68 stuck for 361s! [swapper/68:0] [ 811.226727] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 811.226744] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 811.314530] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 811.322346] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 811.331637] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 811.338584] pc : __sbq_wake_up+0x20/0x1f0 [ 811.342581] lr : sbitmap_queue_clear+0x74/0xc0 [ 811.347011] sp : ffff8000088438b0 [ 811.350313] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 811.357435] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 811.364557] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 811.371678] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 811.378800] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 811.385922] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 811.393044] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 811.400166] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 811.407287] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 811.414409] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 811.421531] Call trace: [ 811.423964] __sbq_wake_up+0x20/0x1f0 [ 811.427613] sbitmap_queue_clear+0x74/0xc0 [ 811.431696] blk_mq_put_tag+0x44/0x70 [ 811.435346] __blk_mq_free_request+0x88/0xf0 [ 811.439603] blk_mq_free_request+0xc0/0x160 [ 811.443773] __blk_mq_end_request+0x78/0x150 [ 811.448029] scsi_end_request+0x114/0x260 [ 811.452026] scsi_io_completion+0x64/0x174 [ 811.456109] scsi_finish_command+0xe0/0x120 [ 811.460279] scsi_complete+0x8c/0x1c4 [ 811.463928] blk_complete_reqs+0x5c/0x74 [ 811.467838] blk_done_softirq+0x38/0x4c [ 811.471661] __do_softirq+0x168/0x418 [ 811.475310] __irq_exit_rcu+0x120/0x170 [ 811.479132] irq_exit_rcu+0x18/0x24 [ 811.482608] el1_interrupt+0x38/0x70 [ 811.486170] el1h_64_irq_handler+0x18/0x24 [ 811.490253] el1h_64_irq+0x68/0x6c [ 811.493641] cpuidle_enter_state+0xc4/0x4e0 [ 811.497811] cpuidle_enter+0x40/0x60 [ 811.501373] cpuidle_idle_call+0x118/0x1b0 [ 811.505457] do_idle+0xac/0x100 [ 811.508586] cpu_startup_entry+0x30/0x34 [ 811.512495] secondary_start_kernel+0xd8/0xf0 [ 811.516839] __secondary_switched+0xb0/0xb4 [ 823.109444] watchdog: BUG: soft lockup - CPU#8 stuck for 372s! [kworker/8:12:19399] [ 823.117087] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 823.117103] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 823.204890] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 823.213140] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 823.222431] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 823.228598] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 823.235546] pc : __sbq_wake_up+0x58/0x1f0 [ 823.239542] lr : sbitmap_queue_clear+0x74/0xc0 [ 823.243973] sp : ffff80000d79b280 [ 823.247274] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 823.254396] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 823.261518] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 823.268640] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 823.275762] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 823.282885] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 823.290007] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 823.297129] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 823.304251] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 823.311374] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 823.318496] Call trace: [ 823.320929] __sbq_wake_up+0x58/0x1f0 [ 823.324578] sbitmap_queue_clear+0x74/0xc0 [ 823.328662] blk_mq_put_tag+0x44/0x70 [ 823.332311] __blk_mq_free_request+0x88/0xf0 [ 823.336568] blk_mq_free_request+0xc0/0x160 [ 823.340738] __blk_mq_end_request+0x78/0x150 [ 823.344995] scsi_end_request+0x114/0x260 [ 823.348991] scsi_io_completion+0x64/0x174 [ 823.353074] scsi_finish_command+0xe0/0x120 [ 823.357245] scsi_complete+0x8c/0x1c4 [ 823.360894] blk_complete_reqs+0x5c/0x74 [ 823.364803] blk_done_softirq+0x38/0x4c [ 823.368626] __do_softirq+0x168/0x418 [ 823.372275] __irq_exit_rcu+0x120/0x170 [ 823.376098] irq_exit_rcu+0x18/0x24 [ 823.379573] el1_interrupt+0x38/0x70 [ 823.383135] el1h_64_irq_handler+0x18/0x24 [ 823.387219] el1h_64_irq+0x68/0x6c [ 823.390607] _raw_spin_unlock_irqrestore+0x20/0x50 [ 823.395385] __wake_up_common_lock+0x84/0xac [ 823.399642] __wake_up+0x20/0x30 [ 823.402857] __sbq_wake_up+0x13c/0x1f0 [ 823.406593] sbitmap_queue_clear+0x74/0xc0 [ 823.410677] blk_mq_put_tag+0x44/0x70 [ 823.414327] __blk_mq_free_request+0x88/0xf0 [ 823.418583] blk_mq_free_request+0xc0/0x160 [ 823.422753] bfq_insert_request.isra.0+0x204/0x270 [ 823.427531] bfq_insert_requests+0x70/0x90 [ 823.431615] blk_mq_sched_insert_requests+0xac/0x250 [ 823.436567] blk_mq_flush_plug_list+0x110/0x2cc [ 823.441085] __blk_flush_plug+0x100/0x140 [ 823.445081] blk_finish_plug+0x40/0x25c [ 823.448904] __iomap_dio_rw+0x1b4/0x500 [ 823.452727] iomap_dio_rw+0x18/0x50 [ 823.456202] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 823.461553] xfs_file_write_iter+0xf4/0x15c [xfs] [ 823.466295] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 823.471339] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 823.476469] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 823.481598] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 823.486984] process_one_work+0x1ec/0x470 [ 823.490981] worker_thread+0x74/0x410 [ 823.494630] kthread+0xec/0x100 [ 823.497758] ret_from_fork+0x10/0x20 [ 835.219367] watchdog: BUG: soft lockup - CPU#68 stuck for 383s! [swapper/68:0] [ 835.226575] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 835.226592] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 835.314379] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 835.322195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 835.331486] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 835.338433] pc : __sbq_wake_up+0x7c/0x1f0 [ 835.342429] lr : sbitmap_queue_clear+0x74/0xc0 [ 835.346860] sp : ffff800008843900 [ 835.350161] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 835.357283] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 835.364405] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 835.371527] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 835.378648] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 835.385771] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 835.392893] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 835.400014] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 835.407136] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 835.414258] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 835.421380] Call trace: [ 835.423813] __sbq_wake_up+0x7c/0x1f0 [ 835.427462] blk_mq_put_tag+0x44/0x70 [ 835.431112] __blk_mq_free_request+0x88/0xf0 [ 835.435369] blk_mq_free_request+0xc0/0x160 [ 835.439538] __blk_mq_end_request+0x78/0x150 [ 835.443795] scsi_end_request+0x114/0x260 [ 835.447792] scsi_io_completion+0x64/0x174 [ 835.451875] scsi_finish_command+0xe0/0x120 [ 835.456045] scsi_complete+0x8c/0x1c4 [ 835.459695] blk_complete_reqs+0x5c/0x74 [ 835.463605] blk_done_softirq+0x38/0x4c [ 835.467427] __do_softirq+0x168/0x418 [ 835.471076] __irq_exit_rcu+0x120/0x170 [ 835.474899] irq_exit_rcu+0x18/0x24 [ 835.478375] el1_interrupt+0x38/0x70 [ 835.481937] el1h_64_irq_handler+0x18/0x24 [ 835.486020] el1h_64_irq+0x68/0x6c [ 835.489408] cpuidle_enter_state+0xc4/0x4e0 [ 835.493579] cpuidle_enter+0x40/0x60 [ 835.497141] cpuidle_idle_call+0x118/0x1b0 [ 835.501224] do_idle+0xac/0x100 [ 835.504353] cpu_startup_entry+0x30/0x34 [ 835.508262] secondary_start_kernel+0xd8/0xf0 [ 835.512606] __secondary_switched+0xb0/0xb4 [ 845.529301] rcu: INFO: rcu_preempt self-detected stall on CPU [ 845.535035] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=20923 [ 845.544070] (t=42064 jiffies g=36445 q=12786 ncpus=128) [ 845.549368] Task dump for CPU 8: [ 845.552582] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 845.562483] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 845.568650] Call trace: [ 845.571083] dump_backtrace+0xe8/0x130 [ 845.574819] show_stack+0x20/0x60 [ 845.578121] sched_show_task+0x160/0x190 [ 845.582031] dump_cpu_task+0x4c/0x78 [ 845.585594] rcu_dump_cpu_stacks+0xc4/0x150 [ 845.589765] print_cpu_stall+0x180/0x210 [ 845.593674] check_cpu_stall+0x20c/0x2b4 [ 845.597584] rcu_pending+0x4c/0x190 [ 845.601059] rcu_sched_clock_irq+0x80/0x204 [ 845.605229] update_process_times+0x7c/0xc0 [ 845.609399] tick_sched_handle+0x38/0x74 [ 845.613310] tick_sched_timer+0x60/0xc0 [ 845.617133] __hrtimer_run_queues+0x170/0x300 [ 845.621476] hrtimer_interrupt+0xfc/0x260 [ 845.625473] arch_timer_handler_phys+0x34/0x50 [ 845.629904] handle_percpu_devid_irq+0xac/0x240 [ 845.634421] generic_handle_domain_irq+0x34/0x50 [ 845.639026] gic_handle_irq+0x5c/0xe0 [ 845.642675] call_on_irq_stack+0x2c/0x38 [ 845.646585] do_interrupt_handler+0xdc/0xe0 [ 845.650756] el1_interrupt+0x34/0x70 [ 845.654318] el1h_64_irq_handler+0x18/0x24 [ 845.658401] el1h_64_irq+0x68/0x6c [ 845.661790] __sbq_wake_up+0x84/0x1f0 [ 845.665439] sbitmap_queue_clear+0x74/0xc0 [ 845.669523] blk_mq_put_tag+0x44/0x70 [ 845.673173] __blk_mq_free_request+0x88/0xf0 [ 845.677429] blk_mq_free_request+0xc0/0x160 [ 845.681599] __blk_mq_end_request+0x78/0x150 [ 845.685856] scsi_end_request+0x114/0x260 [ 845.689852] scsi_io_completion+0x64/0x174 [ 845.693935] scsi_finish_command+0xe0/0x120 [ 845.698106] scsi_complete+0x8c/0x1c4 [ 845.701755] blk_complete_reqs+0x5c/0x74 [ 845.705665] blk_done_softirq+0x38/0x4c [ 845.709487] __do_softirq+0x168/0x418 [ 845.713136] __irq_exit_rcu+0x120/0x170 [ 845.716959] irq_exit_rcu+0x18/0x24 [ 845.720434] el1_interrupt+0x38/0x70 [ 845.723997] el1h_64_irq_handler+0x18/0x24 [ 845.728080] el1h_64_irq+0x68/0x6c [ 845.731468] _raw_spin_unlock_irqrestore+0x20/0x50 [ 845.736247] __wake_up_common_lock+0x84/0xac [ 845.740504] __wake_up+0x20/0x30 [ 845.743720] __sbq_wake_up+0x13c/0x1f0 [ 845.747456] sbitmap_queue_clear+0x74/0xc0 [ 845.751540] blk_mq_put_tag+0x44/0x70 [ 845.755189] __blk_mq_free_request+0x88/0xf0 [ 845.759446] blk_mq_free_request+0xc0/0x160 [ 845.763616] bfq_insert_request.isra.0+0x204/0x270 [ 845.768394] bfq_insert_requests+0x70/0x90 [ 845.772478] blk_mq_sched_insert_requests+0xac/0x250 [ 845.777430] blk_mq_flush_plug_list+0x110/0x2cc [ 845.781947] __blk_flush_plug+0x100/0x140 [ 845.785944] blk_finish_plug+0x40/0x25c [ 845.789766] __iomap_dio_rw+0x1b4/0x500 [ 845.793589] iomap_dio_rw+0x18/0x50 [ 845.797064] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 845.802416] xfs_file_write_iter+0xf4/0x15c [xfs] [ 845.807159] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 845.812203] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 845.817333] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 845.822462] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 845.827850] process_one_work+0x1ec/0x470 [ 845.831846] worker_thread+0x74/0x410 [ 845.835495] kthread+0xec/0x100 [ 845.838624] ret_from_fork+0x10/0x20 [ 845.842188] Task dump for CPU 68: [ 845.845489] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 845.855389] Call trace: [ 845.857822] __switch_to+0xd4/0x100 [ 845.861299] 0x0 [ 859.219214] watchdog: BUG: soft lockup - CPU#68 stuck for 406s! [swapper/68:0] [ 859.226422] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 859.226439] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 859.314225] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 859.322041] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 859.331332] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 859.338279] pc : __sbq_wake_up+0x58/0x1f0 [ 859.342276] lr : sbitmap_queue_clear+0x74/0xc0 [ 859.346706] sp : ffff8000088438b0 [ 859.350008] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 859.357131] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 859.364253] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 859.371374] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 859.378496] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 859.385618] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 859.392740] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 859.399862] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 859.406984] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 859.414105] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 859.421227] Call trace: [ 859.423661] __sbq_wake_up+0x58/0x1f0 [ 859.427310] sbitmap_queue_clear+0x74/0xc0 [ 859.431393] blk_mq_put_tag+0x44/0x70 [ 859.435043] __blk_mq_free_request+0x88/0xf0 [ 859.439300] blk_mq_free_request+0xc0/0x160 [ 859.443470] __blk_mq_end_request+0x78/0x150 [ 859.447727] scsi_end_request+0x114/0x260 [ 859.451723] scsi_io_completion+0x64/0x174 [ 859.455806] scsi_finish_command+0xe0/0x120 [ 859.459976] scsi_complete+0x8c/0x1c4 [ 859.463626] blk_complete_reqs+0x5c/0x74 [ 859.467535] blk_done_softirq+0x38/0x4c [ 859.471358] __do_softirq+0x168/0x418 [ 859.475007] __irq_exit_rcu+0x120/0x170 [ 859.478830] irq_exit_rcu+0x18/0x24 [ 859.482305] el1_interrupt+0x38/0x70 [ 859.485867] el1h_64_irq_handler+0x18/0x24 [ 859.489950] el1h_64_irq+0x68/0x6c [ 859.493338] cpuidle_enter_state+0xc4/0x4e0 [ 859.497509] cpuidle_enter+0x40/0x60 [ 859.501072] cpuidle_idle_call+0x118/0x1b0 [ 859.505156] do_idle+0xac/0x100 [ 859.508285] cpu_startup_entry+0x30/0x34 [ 859.512194] secondary_start_kernel+0xd8/0xf0 [ 859.516538] __secondary_switched+0xb0/0xb4 [ 871.109138] watchdog: BUG: soft lockup - CPU#8 stuck for 417s! [kworker/8:12:19399] [ 871.116781] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 871.116798] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 871.204586] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 871.212835] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 871.222127] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 871.228294] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 871.235242] pc : __sbq_wake_up+0x4/0x1f0 [ 871.239152] lr : sbitmap_queue_clear+0x74/0xc0 [ 871.243582] sp : ffff80000d79b2d0 [ 871.246884] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 871.254006] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 871.261128] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 871.268250] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 871.275372] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 871.282494] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 871.289616] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 871.296738] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 871.303859] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 871.310981] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 871.318103] Call trace: [ 871.320536] __sbq_wake_up+0x4/0x1f0 [ 871.324099] blk_mq_put_tag+0x44/0x70 [ 871.327748] __blk_mq_free_request+0x88/0xf0 [ 871.332005] blk_mq_free_request+0xc0/0x160 [ 871.336175] __blk_mq_end_request+0x78/0x150 [ 871.340432] scsi_end_request+0x114/0x260 [ 871.344428] scsi_io_completion+0x64/0x174 [ 871.348511] scsi_finish_command+0xe0/0x120 [ 871.352682] scsi_complete+0x8c/0x1c4 [ 871.356331] blk_complete_reqs+0x5c/0x74 [ 871.360241] blk_done_softirq+0x38/0x4c [ 871.364063] __do_softirq+0x168/0x418 [ 871.367712] __irq_exit_rcu+0x120/0x170 [ 871.371535] irq_exit_rcu+0x18/0x24 [ 871.375010] el1_interrupt+0x38/0x70 [ 871.378573] el1h_64_irq_handler+0x18/0x24 [ 871.382656] el1h_64_irq+0x68/0x6c [ 871.386044] _raw_spin_unlock_irqrestore+0x20/0x50 [ 871.390822] __wake_up_common_lock+0x84/0xac [ 871.395079] __wake_up+0x20/0x30 [ 871.398295] __sbq_wake_up+0x13c/0x1f0 [ 871.402031] sbitmap_queue_clear+0x74/0xc0 [ 871.406114] blk_mq_put_tag+0x44/0x70 [ 871.409187] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { [ 871.409764] __blk_mq_free_request+0x88/0xf0 [ 871.409765] 8-.... [ 871.416886] blk_mq_free_request+0xc0/0x160 [ 871.421147] 68-.... [ 871.423229] bfq_insert_request.isra.0+0x204/0x270 [ 871.427398] } 42046 jiffies s: 2093 root: 0x11/. [ 871.429570] bfq_insert_requests+0x70/0x90 [ 871.429572] blk_mq_sched_insert_requests+0xac/0x250 [ 871.434348] rcu: blocking rcu_node structures (internal RCU debug): [ 871.439038] blk_mq_flush_plug_list+0x110/0x2cc [ 871.443122] l=1:0-15:0x100/. [ 871.448071] __blk_flush_plug+0x100/0x140 [ 871.454325] l=1:64-79:0x10/. [ 871.458839] blk_finish_plug+0x40/0x25c [ 871.461795] [ 871.465788] __iomap_dio_rw+0x1b4/0x500 [ 871.468741] Task dump for CPU 8: [ 871.472563] iomap_dio_rw+0x18/0x50 [ 871.472564] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 871.474042] task:kworker/8:12 state:R [ 871.477864] xfs_file_write_iter+0xf4/0x15c [xfs] [ 871.481079] running task [ 871.484552] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 871.489851] stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 871.493758] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 871.498449] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 871.501488] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 871.506526] [ 871.512431] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 871.517557] Call trace: [ 871.523724] process_one_work+0x1ec/0x470 [ 871.523725] worker_thread+0x74/0x410 [ 871.528849] __switch_to+0xd4/0x100 [ 871.530327] kthread+0xec/0x100 [ 871.535713] 0xffff6c0200000080 [ 871.538146] ret_from_fork+0x10/0x20 [ 871.542144] Task dump for CPU 68: [ 871.562385] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 871.572287] Call trace: [ 871.574720] __switch_to+0xd4/0x100 [ 871.578197] 0x0 [ 883.219061] watchdog: BUG: soft lockup - CPU#68 stuck for 428s! [swapper/68:0] [ 883.226269] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 883.226286] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 883.314072] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 883.321888] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 883.331179] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 883.338126] pc : __sbq_wake_up+0x4/0x1f0 [ 883.342036] lr : sbitmap_queue_clear+0x74/0xc0 [ 883.346467] sp : ffff800008843900 [ 883.349768] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 883.356890] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 883.364012] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 883.371134] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 883.378256] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 883.385378] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 883.392500] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 883.399622] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 883.406744] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 883.413865] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 883.420987] Call trace: [ 883.423420] __sbq_wake_up+0x4/0x1f0 [ 883.426983] blk_mq_put_tag+0x44/0x70 [ 883.430633] __blk_mq_free_request+0x88/0xf0 [ 883.434890] blk_mq_free_request+0xc0/0x160 [ 883.439061] __blk_mq_end_request+0x78/0x150 [ 883.443317] scsi_end_request+0x114/0x260 [ 883.447314] scsi_io_completion+0x64/0x174 [ 883.451397] scsi_finish_command+0xe0/0x120 [ 883.455567] scsi_complete+0x8c/0x1c4 [ 883.459216] blk_complete_reqs+0x5c/0x74 [ 883.463127] blk_done_softirq+0x38/0x4c [ 883.466950] __do_softirq+0x168/0x418 [ 883.470599] __irq_exit_rcu+0x120/0x170 [ 883.474421] irq_exit_rcu+0x18/0x24 [ 883.477897] el1_interrupt+0x38/0x70 [ 883.481459] el1h_64_irq_handler+0x18/0x24 [ 883.485542] el1h_64_irq+0x68/0x6c [ 883.488931] cpuidle_enter_state+0xc4/0x4e0 [ 883.493101] cpuidle_enter+0x40/0x60 [ 883.496663] cpuidle_idle_call+0x118/0x1b0 [ 883.500747] do_idle+0xac/0x100 [ 883.503876] cpu_startup_entry+0x30/0x34 [ 883.507785] secondary_start_kernel+0xd8/0xf0 [ 883.512129] __secondary_switched+0xb0/0xb4 [ 895.108985] watchdog: BUG: soft lockup - CPU#8 stuck for 439s! [kworker/8:12:19399] [ 895.116628] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 895.116644] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 895.204432] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 895.212681] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 895.221972] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 895.228140] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 895.235087] pc : __sbq_wake_up+0x58/0x1f0 [ 895.239084] lr : sbitmap_queue_clear+0x74/0xc0 [ 895.243514] sp : ffff80000d79b280 [ 895.246816] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 895.253938] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 895.261060] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 895.268182] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 895.275304] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 895.282426] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 895.289548] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 895.296670] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 895.303791] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 895.310913] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 895.318036] Call trace: [ 895.320469] __sbq_wake_up+0x58/0x1f0 [ 895.324118] sbitmap_queue_clear+0x74/0xc0 [ 895.328202] blk_mq_put_tag+0x44/0x70 [ 895.331851] __blk_mq_free_request+0x88/0xf0 [ 895.336108] blk_mq_free_request+0xc0/0x160 [ 895.340278] __blk_mq_end_request+0x78/0x150 [ 895.344535] scsi_end_request+0x114/0x260 [ 895.348531] scsi_io_completion+0x64/0x174 [ 895.352614] scsi_finish_command+0xe0/0x120 [ 895.356784] scsi_complete+0x8c/0x1c4 [ 895.360433] blk_complete_reqs+0x5c/0x74 [ 895.364343] blk_done_softirq+0x38/0x4c [ 895.368166] __do_softirq+0x168/0x418 [ 895.371815] __irq_exit_rcu+0x120/0x170 [ 895.375637] irq_exit_rcu+0x18/0x24 [ 895.379113] el1_interrupt+0x38/0x70 [ 895.382675] el1h_64_irq_handler+0x18/0x24 [ 895.386758] el1h_64_irq+0x68/0x6c [ 895.390149] _raw_spin_unlock_irqrestore+0x20/0x50 [ 895.394926] __wake_up_common_lock+0x84/0xac [ 895.399183] __wake_up+0x20/0x30 [ 895.402399] __sbq_wake_up+0x13c/0x1f0 [ 895.406135] sbitmap_queue_clear+0x74/0xc0 [ 895.410218] blk_mq_put_tag+0x44/0x70 [ 895.413868] __blk_mq_free_request+0x88/0xf0 [ 895.418125] blk_mq_free_request+0xc0/0x160 [ 895.422295] bfq_insert_request.isra.0+0x204/0x270 [ 895.427073] bfq_insert_requests+0x70/0x90 [ 895.431156] blk_mq_sched_insert_requests+0xac/0x250 [ 895.436109] blk_mq_flush_plug_list+0x110/0x2cc [ 895.440626] __blk_flush_plug+0x100/0x140 [ 895.444623] blk_finish_plug+0x40/0x25c [ 895.448445] __iomap_dio_rw+0x1b4/0x500 [ 895.452268] iomap_dio_rw+0x18/0x50 [ 895.455744] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 895.461094] xfs_file_write_iter+0xf4/0x15c [xfs] [ 895.465837] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 895.470881] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 895.476010] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 895.481140] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 895.486526] process_one_work+0x1ec/0x470 [ 895.490522] worker_thread+0x74/0x410 [ 895.494170] kthread+0xec/0x100 [ 895.497299] ret_from_fork+0x10/0x20 [ 907.218907] watchdog: BUG: soft lockup - CPU#68 stuck for 450s! [swapper/68:0] [ 907.226115] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 907.226132] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 907.313919] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 907.321735] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 907.331025] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 907.337973] pc : __sbq_wake_up+0x58/0x1f0 [ 907.341969] lr : sbitmap_queue_clear+0x74/0xc0 [ 907.346400] sp : ffff8000088438b0 [ 907.349701] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 907.356823] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 907.363945] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 907.371066] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 907.378188] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 907.385310] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 907.392433] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 907.399555] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 907.406677] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 907.413799] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 907.420921] Call trace: [ 907.423354] __sbq_wake_up+0x58/0x1f0 [ 907.427003] sbitmap_queue_clear+0x74/0xc0 [ 907.431086] blk_mq_put_tag+0x44/0x70 [ 907.434736] __blk_mq_free_request+0x88/0xf0 [ 907.438993] blk_mq_free_request+0xc0/0x160 [ 907.443163] __blk_mq_end_request+0x78/0x150 [ 907.447419] scsi_end_request+0x114/0x260 [ 907.451415] scsi_io_completion+0x64/0x174 [ 907.455499] scsi_finish_command+0xe0/0x120 [ 907.459669] scsi_complete+0x8c/0x1c4 [ 907.463318] blk_complete_reqs+0x5c/0x74 [ 907.467228] blk_done_softirq+0x38/0x4c [ 907.471050] __do_softirq+0x168/0x418 [ 907.474699] __irq_exit_rcu+0x120/0x170 [ 907.478522] irq_exit_rcu+0x18/0x24 [ 907.481997] el1_interrupt+0x38/0x70 [ 907.485559] el1h_64_irq_handler+0x18/0x24 [ 907.489642] el1h_64_irq+0x68/0x6c [ 907.493030] cpuidle_enter_state+0xc4/0x4e0 [ 907.497200] cpuidle_enter+0x40/0x60 [ 907.500763] cpuidle_idle_call+0x118/0x1b0 [ 907.504846] do_idle+0xac/0x100 [ 907.507976] cpu_startup_entry+0x30/0x34 [ 907.511885] secondary_start_kernel+0xd8/0xf0 [ 907.516229] __secondary_switched+0xb0/0xb4 [ 919.108831] watchdog: BUG: soft lockup - CPU#8 stuck for 461s! [kworker/8:12:19399] [ 919.116473] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 919.116490] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 919.204277] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 919.212527] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 919.221818] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 919.227985] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 919.234933] pc : __sbq_wake_up+0x7c/0x1f0 [ 919.238930] lr : sbitmap_queue_clear+0x74/0xc0 [ 919.243361] sp : ffff80000d79b2d0 [ 919.246662] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 919.253784] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 919.260906] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 919.268028] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 919.275150] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 919.282272] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 919.289394] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 919.296516] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 919.303638] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 919.310760] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 919.317883] Call trace: [ 919.320316] __sbq_wake_up+0x7c/0x1f0 [ 919.323966] blk_mq_put_tag+0x44/0x70 [ 919.327615] __blk_mq_free_request+0x88/0xf0 [ 919.331872] blk_mq_free_request+0xc0/0x160 [ 919.336042] __blk_mq_end_request+0x78/0x150 [ 919.340299] scsi_end_request+0x114/0x260 [ 919.344295] scsi_io_completion+0x64/0x174 [ 919.348378] scsi_finish_command+0xe0/0x120 [ 919.352549] scsi_complete+0x8c/0x1c4 [ 919.356198] blk_complete_reqs+0x5c/0x74 [ 919.360107] blk_done_softirq+0x38/0x4c [ 919.363930] __do_softirq+0x168/0x418 [ 919.367579] __irq_exit_rcu+0x120/0x170 [ 919.371402] irq_exit_rcu+0x18/0x24 [ 919.374877] el1_interrupt+0x38/0x70 [ 919.378439] el1h_64_irq_handler+0x18/0x24 [ 919.382522] el1h_64_irq+0x68/0x6c [ 919.385911] _raw_spin_unlock_irqrestore+0x20/0x50 [ 919.390689] __wake_up_common_lock+0x84/0xac [ 919.394946] __wake_up+0x20/0x30 [ 919.398162] __sbq_wake_up+0x13c/0x1f0 [ 919.401900] sbitmap_queue_clear+0x74/0xc0 [ 919.405983] blk_mq_put_tag+0x44/0x70 [ 919.409633] __blk_mq_free_request+0x88/0xf0 [ 919.413890] blk_mq_free_request+0xc0/0x160 [ 919.418059] bfq_insert_request.isra.0+0x204/0x270 [ 919.422837] bfq_insert_requests+0x70/0x90 [ 919.426921] blk_mq_sched_insert_requests+0xac/0x250 [ 919.431873] blk_mq_flush_plug_list+0x110/0x2cc [ 919.436391] __blk_flush_plug+0x100/0x140 [ 919.440387] blk_finish_plug+0x40/0x25c [ 919.444210] __iomap_dio_rw+0x1b4/0x500 [ 919.448033] iomap_dio_rw+0x18/0x50 [ 919.451508] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 919.456860] xfs_file_write_iter+0xf4/0x15c [xfs] [ 919.461602] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 919.466646] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 919.471776] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 919.476905] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 919.482291] process_one_work+0x1ec/0x470 [ 919.486287] worker_thread+0x74/0x410 [ 919.489936] kthread+0xec/0x100 [ 919.493065] ret_from_fork+0x10/0x20 [ 931.218753] watchdog: BUG: soft lockup - CPU#68 stuck for 473s! [swapper/68:0] [ 931.225961] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 931.225978] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 931.313765] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 931.321581] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 931.330872] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 931.337819] pc : __sbq_wake_up+0x1c8/0x1f0 [ 931.341902] lr : sbitmap_queue_clear+0x74/0xc0 [ 931.346333] sp : ffff8000088438b0 [ 931.349633] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 931.356755] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 931.363877] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 931.370999] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 931.378121] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 931.385243] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 931.392364] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 931.399486] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 931.406609] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 931.413730] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 931.420852] Call trace: [ 931.423285] __sbq_wake_up+0x1c8/0x1f0 [ 931.427021] sbitmap_queue_clear+0x74/0xc0 [ 931.431105] blk_mq_put_tag+0x44/0x70 [ 931.434755] __blk_mq_free_request+0x88/0xf0 [ 931.439011] blk_mq_free_request+0xc0/0x160 [ 931.443181] __blk_mq_end_request+0x78/0x150 [ 931.447438] scsi_end_request+0x114/0x260 [ 931.451434] scsi_io_completion+0x64/0x174 [ 931.455518] scsi_finish_command+0xe0/0x120 [ 931.459688] scsi_complete+0x8c/0x1c4 [ 931.463337] blk_complete_reqs+0x5c/0x74 [ 931.467246] blk_done_softirq+0x38/0x4c [ 931.471069] __do_softirq+0x168/0x418 [ 931.474718] __irq_exit_rcu+0x120/0x170 [ 931.478540] irq_exit_rcu+0x18/0x24 [ 931.482016] el1_interrupt+0x38/0x70 [ 931.485578] el1h_64_irq_handler+0x18/0x24 [ 931.489661] el1h_64_irq+0x68/0x6c [ 931.493050] cpuidle_enter_state+0xc4/0x4e0 [ 931.497220] cpuidle_enter+0x40/0x60 [ 931.500782] cpuidle_idle_call+0x118/0x1b0 [ 931.504866] do_idle+0xac/0x100 [ 931.507995] cpu_startup_entry+0x30/0x34 [ 931.511904] secondary_start_kernel+0xd8/0xf0 [ 931.516248] __secondary_switched+0xb0/0xb4 [ 937.968750] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 48701 jiffies s: 2093 root: 0x11/. [ 937.980404] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 937.989613] Task dump for CPU 8: [ 937.992827] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 938.002729] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 938.008899] Call trace: [ 938.011333] __switch_to+0xd4/0x100 [ 938.014809] 0xffff6c0200000080 [ 938.017938] Task dump for CPU 68: [ 938.021241] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 938.031142] Call trace: [ 938.033575] __switch_to+0xd4/0x100 [ 938.037051] 0x0 [ 943.108677] watchdog: BUG: soft lockup - CPU#8 stuck for 484s! [kworker/8:12:19399] [ 943.116319] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 943.116336] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 943.204123] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 943.212372] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 943.221663] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 943.227831] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 943.234778] pc : __sbq_wake_up+0x84/0x1f0 [ 943.238775] lr : sbitmap_queue_clear+0x74/0xc0 [ 943.243205] sp : ffff80000d79b280 [ 943.246507] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 943.253629] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 943.260751] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 943.267873] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 943.274995] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 943.282117] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 943.289239] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 943.296360] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 943.303482] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 943.310604] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 943.317726] Call trace: [ 943.320159] __sbq_wake_up+0x84/0x1f0 [ 943.323809] sbitmap_queue_clear+0x74/0xc0 [ 943.327892] blk_mq_put_tag+0x44/0x70 [ 943.331542] __blk_mq_free_request+0x88/0xf0 [ 943.335798] blk_mq_free_request+0xc0/0x160 [ 943.339968] __blk_mq_end_request+0x78/0x150 [ 943.344225] scsi_end_request+0x114/0x260 [ 943.348221] scsi_io_completion+0x64/0x174 [ 943.352304] scsi_finish_command+0xe0/0x120 [ 943.356475] scsi_complete+0x8c/0x1c4 [ 943.360124] blk_complete_reqs+0x5c/0x74 [ 943.364033] blk_done_softirq+0x38/0x4c [ 943.367856] __do_softirq+0x168/0x418 [ 943.371505] __irq_exit_rcu+0x120/0x170 [ 943.375328] irq_exit_rcu+0x18/0x24 [ 943.378803] el1_interrupt+0x38/0x70 [ 943.382366] el1h_64_irq_handler+0x18/0x24 [ 943.386450] el1h_64_irq+0x68/0x6c [ 943.389839] _raw_spin_unlock_irqrestore+0x20/0x50 [ 943.394616] __wake_up_common_lock+0x84/0xac [ 943.398874] __wake_up+0x20/0x30 [ 943.402089] __sbq_wake_up+0x13c/0x1f0 [ 943.405825] sbitmap_queue_clear+0x74/0xc0 [ 943.409908] blk_mq_put_tag+0x44/0x70 [ 943.413558] __blk_mq_free_request+0x88/0xf0 [ 943.417815] blk_mq_free_request+0xc0/0x160 [ 943.421984] bfq_insert_request.isra.0+0x204/0x270 [ 943.426763] bfq_insert_requests+0x70/0x90 [ 943.430846] blk_mq_sched_insert_requests+0xac/0x250 [ 943.435798] blk_mq_flush_plug_list+0x110/0x2cc [ 943.440316] __blk_flush_plug+0x100/0x140 [ 943.444313] blk_finish_plug+0x40/0x25c [ 943.448136] __iomap_dio_rw+0x1b4/0x500 [ 943.451958] iomap_dio_rw+0x18/0x50 [ 943.455434] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 943.460785] xfs_file_write_iter+0xf4/0x15c [xfs] [ 943.465527] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 943.470570] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 943.475700] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 943.480830] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 943.486215] process_one_work+0x1ec/0x470 [ 943.490212] worker_thread+0x74/0x410 [ 943.493860] kthread+0xec/0x100 [ 943.496989] ret_from_fork+0x10/0x20 [ 955.218598] watchdog: BUG: soft lockup - CPU#68 stuck for 495s! [swapper/68:0] [ 955.225806] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 955.225822] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 955.313609] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 955.321425] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 955.330716] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 955.337663] pc : __sbq_wake_up+0x58/0x1f0 [ 955.341660] lr : sbitmap_queue_clear+0x74/0xc0 [ 955.346090] sp : ffff8000088438b0 [ 955.349391] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 955.356513] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 955.363637] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 955.370759] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 955.377881] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 955.385003] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 955.392124] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 955.399246] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 955.406368] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 955.413490] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 955.420612] Call trace: [ 955.423046] __sbq_wake_up+0x58/0x1f0 [ 955.426695] sbitmap_queue_clear+0x74/0xc0 [ 955.430778] blk_mq_put_tag+0x44/0x70 [ 955.434428] __blk_mq_free_request+0x88/0xf0 [ 955.438685] blk_mq_free_request+0xc0/0x160 [ 955.442855] __blk_mq_end_request+0x78/0x150 [ 955.447111] scsi_end_request+0x114/0x260 [ 955.451108] scsi_io_completion+0x64/0x174 [ 955.455191] scsi_finish_command+0xe0/0x120 [ 955.459361] scsi_complete+0x8c/0x1c4 [ 955.463010] blk_complete_reqs+0x5c/0x74 [ 955.466920] blk_done_softirq+0x38/0x4c [ 955.470743] __do_softirq+0x168/0x418 [ 955.474393] __irq_exit_rcu+0x120/0x170 [ 955.478215] irq_exit_rcu+0x18/0x24 [ 955.481692] el1_interrupt+0x38/0x70 [ 955.485254] el1h_64_irq_handler+0x18/0x24 [ 955.489337] el1h_64_irq+0x68/0x6c [ 955.492726] cpuidle_enter_state+0xc4/0x4e0 [ 955.496896] cpuidle_enter+0x40/0x60 [ 955.500458] cpuidle_idle_call+0x118/0x1b0 [ 955.504542] do_idle+0xac/0x100 [ 955.507671] cpu_startup_entry+0x30/0x34 [ 955.511580] secondary_start_kernel+0xd8/0xf0 [ 955.515924] __secondary_switched+0xb0/0xb4 [ 967.108517] watchdog: BUG: soft lockup - CPU#8 stuck for 506s! [kworker/8:12:19399] [ 967.116160] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 967.116177] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 967.203963] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 967.212213] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 967.221504] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 967.227671] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 967.234619] pc : __sbq_wake_up+0x58/0x1f0 [ 967.238615] lr : sbitmap_queue_clear+0x74/0xc0 [ 967.243046] sp : ffff80000d79b280 [ 967.246347] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 967.253470] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 967.260592] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 967.267714] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 967.274836] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 967.281959] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 967.289081] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 967.296202] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 967.303324] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 967.310446] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 967.317568] Call trace: [ 967.320001] __sbq_wake_up+0x58/0x1f0 [ 967.323650] sbitmap_queue_clear+0x74/0xc0 [ 967.327733] blk_mq_put_tag+0x44/0x70 [ 967.331383] __blk_mq_free_request+0x88/0xf0 [ 967.335640] blk_mq_free_request+0xc0/0x160 [ 967.339810] __blk_mq_end_request+0x78/0x150 [ 967.344066] scsi_end_request+0x114/0x260 [ 967.348063] scsi_io_completion+0x64/0x174 [ 967.352146] scsi_finish_command+0xe0/0x120 [ 967.356316] scsi_complete+0x8c/0x1c4 [ 967.359966] blk_complete_reqs+0x5c/0x74 [ 967.363875] blk_done_softirq+0x38/0x4c [ 967.367698] __do_softirq+0x168/0x418 [ 967.371347] __irq_exit_rcu+0x120/0x170 [ 967.375169] irq_exit_rcu+0x18/0x24 [ 967.378646] el1_interrupt+0x38/0x70 [ 967.382208] el1h_64_irq_handler+0x18/0x24 [ 967.386291] el1h_64_irq+0x68/0x6c [ 967.389680] _raw_spin_unlock_irqrestore+0x20/0x50 [ 967.394457] __wake_up_common_lock+0x84/0xac [ 967.398715] __wake_up+0x20/0x30 [ 967.401930] __sbq_wake_up+0x13c/0x1f0 [ 967.405666] sbitmap_queue_clear+0x74/0xc0 [ 967.409751] blk_mq_put_tag+0x44/0x70 [ 967.413400] __blk_mq_free_request+0x88/0xf0 [ 967.417657] blk_mq_free_request+0xc0/0x160 [ 967.421827] bfq_insert_request.isra.0+0x204/0x270 [ 967.426605] bfq_insert_requests+0x70/0x90 [ 967.430689] blk_mq_sched_insert_requests+0xac/0x250 [ 967.435640] blk_mq_flush_plug_list+0x110/0x2cc [ 967.440158] __blk_flush_plug+0x100/0x140 [ 967.444154] blk_finish_plug+0x40/0x25c [ 967.447977] __iomap_dio_rw+0x1b4/0x500 [ 967.451800] iomap_dio_rw+0x18/0x50 [ 967.455275] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 967.460626] xfs_file_write_iter+0xf4/0x15c [xfs] [ 967.465368] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 967.470412] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 967.475541] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 967.480672] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 967.486057] process_one_work+0x1ec/0x470 [ 967.490054] worker_thread+0x74/0x410 [ 967.493702] kthread+0xec/0x100 [ 967.496831] ret_from_fork+0x10/0x20 [ 979.218435] watchdog: BUG: soft lockup - CPU#68 stuck for 517s! [swapper/68:0] [ 979.225643] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 979.225660] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 979.313446] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 979.321262] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 979.330553] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 979.337501] pc : __sbq_wake_up+0x20/0x1f0 [ 979.341497] lr : sbitmap_queue_clear+0x74/0xc0 [ 979.345928] sp : ffff8000088438b0 [ 979.349229] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 979.356351] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 979.363473] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 979.370594] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 979.377717] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 979.384839] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 979.391960] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 979.399082] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 979.406204] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 979.413326] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 979.420448] Call trace: [ 979.422881] __sbq_wake_up+0x20/0x1f0 [ 979.426530] sbitmap_queue_clear+0x74/0xc0 [ 979.430614] blk_mq_put_tag+0x44/0x70 [ 979.434263] __blk_mq_free_request+0x88/0xf0 [ 979.438520] blk_mq_free_request+0xc0/0x160 [ 979.442690] __blk_mq_end_request+0x78/0x150 [ 979.446947] scsi_end_request+0x114/0x260 [ 979.450943] scsi_io_completion+0x64/0x174 [ 979.455026] scsi_finish_command+0xe0/0x120 [ 979.459197] scsi_complete+0x8c/0x1c4 [ 979.462846] blk_complete_reqs+0x5c/0x74 [ 979.466756] blk_done_softirq+0x38/0x4c [ 979.470578] __do_softirq+0x168/0x418 [ 979.474227] __irq_exit_rcu+0x120/0x170 [ 979.478050] irq_exit_rcu+0x18/0x24 [ 979.481525] el1_interrupt+0x38/0x70 [ 979.485087] el1h_64_irq_handler+0x18/0x24 [ 979.489171] el1h_64_irq+0x68/0x6c [ 979.492559] cpuidle_enter_state+0xc4/0x4e0 [ 979.496729] cpuidle_enter+0x40/0x60 [ 979.500291] cpuidle_idle_call+0x118/0x1b0 [ 979.504375] do_idle+0xac/0x100 [ 979.507504] cpu_startup_entry+0x30/0x34 [ 979.511413] secondary_start_kernel+0xd8/0xf0 [ 979.515757] __secondary_switched+0xb0/0xb4 [ 991.108355] watchdog: BUG: soft lockup - CPU#8 stuck for 529s! [kworker/8:12:19399] [ 991.115997] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 991.116014] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 991.203803] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 991.212052] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 991.221343] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 991.227511] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 991.234458] pc : __sbq_wake_up+0x4/0x1f0 [ 991.238368] lr : sbitmap_queue_clear+0x74/0xc0 [ 991.242799] sp : ffff80000d79b2d0 [ 991.246100] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 991.253222] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 991.260344] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 991.267467] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 991.274588] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 991.281710] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 991.288832] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 991.295954] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 991.303076] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 991.310197] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 991.317319] Call trace: [ 991.319752] __sbq_wake_up+0x4/0x1f0 [ 991.323314] blk_mq_put_tag+0x44/0x70 [ 991.326964] __blk_mq_free_request+0x88/0xf0 [ 991.331221] blk_mq_free_request+0xc0/0x160 [ 991.335390] __blk_mq_end_request+0x78/0x150 [ 991.339647] scsi_end_request+0x114/0x260 [ 991.343644] scsi_io_completion+0x64/0x174 [ 991.347727] scsi_finish_command+0xe0/0x120 [ 991.351897] scsi_complete+0x8c/0x1c4 [ 991.355546] blk_complete_reqs+0x5c/0x74 [ 991.359456] blk_done_softirq+0x38/0x4c [ 991.363278] __do_softirq+0x168/0x418 [ 991.366927] __irq_exit_rcu+0x120/0x170 [ 991.370750] irq_exit_rcu+0x18/0x24 [ 991.374226] el1_interrupt+0x38/0x70 [ 991.377788] el1h_64_irq_handler+0x18/0x24 [ 991.381872] el1h_64_irq+0x68/0x6c [ 991.385261] _raw_spin_unlock_irqrestore+0x20/0x50 [ 991.390039] __wake_up_common_lock+0x84/0xac [ 991.394296] __wake_up+0x20/0x30 [ 991.397512] __sbq_wake_up+0x13c/0x1f0 [ 991.401248] sbitmap_queue_clear+0x74/0xc0 [ 991.405331] blk_mq_put_tag+0x44/0x70 [ 991.408981] __blk_mq_free_request+0x88/0xf0 [ 991.413237] blk_mq_free_request+0xc0/0x160 [ 991.417407] bfq_insert_request.isra.0+0x204/0x270 [ 991.422185] bfq_insert_requests+0x70/0x90 [ 991.426269] blk_mq_sched_insert_requests+0xac/0x250 [ 991.431221] blk_mq_flush_plug_list+0x110/0x2cc [ 991.435739] __blk_flush_plug+0x100/0x140 [ 991.439735] blk_finish_plug+0x40/0x25c [ 991.443558] __iomap_dio_rw+0x1b4/0x500 [ 991.447381] iomap_dio_rw+0x18/0x50 [ 991.450857] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 991.456207] xfs_file_write_iter+0xf4/0x15c [xfs] [ 991.460949] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 991.465993] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 991.471123] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 991.476253] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 991.481638] process_one_work+0x1ec/0x470 [ 991.485635] worker_thread+0x74/0x410 [ 991.489283] kthread+0xec/0x100 [ 991.492412] ret_from_fork+0x10/0x20 [ 1003.218273] watchdog: BUG: soft lockup - CPU#68 stuck for 540s! [swapper/68:0] [ 1003.225481] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1003.225498] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1003.313285] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1003.321101] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1003.330392] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1003.337339] pc : __sbq_wake_up+0x84/0x1f0 [ 1003.341336] lr : sbitmap_queue_clear+0x74/0xc0 [ 1003.345766] sp : ffff8000088438b0 [ 1003.349067] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1003.356189] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1003.363311] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1003.370433] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1003.377555] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1003.384677] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1003.391799] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1003.398920] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1003.406042] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1003.413164] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1003.420286] Call trace: [ 1003.422719] __sbq_wake_up+0x84/0x1f0 [ 1003.426368] sbitmap_queue_clear+0x74/0xc0 [ 1003.430451] blk_mq_put_tag+0x44/0x70 [ 1003.434101] __blk_mq_free_request+0x88/0xf0 [ 1003.438358] blk_mq_free_request+0xc0/0x160 [ 1003.442528] __blk_mq_end_request+0x78/0x150 [ 1003.446784] scsi_end_request+0x114/0x260 [ 1003.450781] scsi_io_completion+0x64/0x174 [ 1003.454864] scsi_finish_command+0xe0/0x120 [ 1003.459034] scsi_complete+0x8c/0x1c4 [ 1003.462683] blk_complete_reqs+0x5c/0x74 [ 1003.466593] blk_done_softirq+0x38/0x4c [ 1003.470415] __do_softirq+0x168/0x418 [ 1003.474064] __irq_exit_rcu+0x120/0x170 [ 1003.477887] irq_exit_rcu+0x18/0x24 [ 1003.481362] el1_interrupt+0x38/0x70 [ 1003.484924] el1h_64_irq_handler+0x18/0x24 [ 1003.489007] el1h_64_irq+0x68/0x6c [ 1003.492396] cpuidle_enter_state+0xc4/0x4e0 [ 1003.496566] cpuidle_enter+0x40/0x60 [ 1003.500128] cpuidle_idle_call+0x118/0x1b0 [ 1003.504213] do_idle+0xac/0x100 [ 1003.507341] cpu_startup_entry+0x30/0x34 [ 1003.511250] secondary_start_kernel+0xd8/0xf0 [ 1003.515594] __secondary_switched+0xb0/0xb4 [ 1004.528277] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 55357 jiffies s: 2093 root: 0x11/. [ 1004.539935] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1004.549145] Task dump for CPU 8: [ 1004.552360] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1004.562262] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1004.568433] Call trace: [ 1004.570866] __switch_to+0xd4/0x100 [ 1004.574344] 0xffff6c0200000080 [ 1004.577473] Task dump for CPU 68: [ 1004.580776] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1004.590677] Call trace: [ 1004.593110] __switch_to+0xd4/0x100 [ 1004.596586] 0x0 [ 1015.108192] watchdog: BUG: soft lockup - CPU#8 stuck for 551s! [kworker/8:12:19399] [ 1015.115835] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1015.115851] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1015.203639] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1015.211888] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1015.221179] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1015.227347] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1015.234294] pc : __sbq_wake_up+0x4/0x1f0 [ 1015.238204] lr : sbitmap_queue_clear+0x74/0xc0 [ 1015.242635] sp : ffff80000d79b2d0 [ 1015.245936] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1015.253058] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1015.260181] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1015.267303] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1015.274425] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1015.281547] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1015.288669] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1015.295790] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1015.302912] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1015.310034] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 1015.317156] Call trace: [ 1015.319589] __sbq_wake_up+0x4/0x1f0 [ 1015.323152] blk_mq_put_tag+0x44/0x70 [ 1015.326802] __blk_mq_free_request+0x88/0xf0 [ 1015.331058] blk_mq_free_request+0xc0/0x160 [ 1015.335228] __blk_mq_end_request+0x78/0x150 [ 1015.339485] scsi_end_request+0x114/0x260 [ 1015.343481] scsi_io_completion+0x64/0x174 [ 1015.347564] scsi_finish_command+0xe0/0x120 [ 1015.351735] scsi_complete+0x8c/0x1c4 [ 1015.355384] blk_complete_reqs+0x5c/0x74 [ 1015.359293] blk_done_softirq+0x38/0x4c [ 1015.363116] __do_softirq+0x168/0x418 [ 1015.366765] __irq_exit_rcu+0x120/0x170 [ 1015.370588] irq_exit_rcu+0x18/0x24 [ 1015.374063] el1_interrupt+0x38/0x70 [ 1015.377625] el1h_64_irq_handler+0x18/0x24 [ 1015.381709] el1h_64_irq+0x68/0x6c [ 1015.385097] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1015.389875] __wake_up_common_lock+0x84/0xac [ 1015.394132] __wake_up+0x20/0x30 [ 1015.397348] __sbq_wake_up+0x13c/0x1f0 [ 1015.401084] sbitmap_queue_clear+0x74/0xc0 [ 1015.405168] blk_mq_put_tag+0x44/0x70 [ 1015.408817] __blk_mq_free_request+0x88/0xf0 [ 1015.413074] blk_mq_free_request+0xc0/0x160 [ 1015.417244] bfq_insert_request.isra.0+0x204/0x270 [ 1015.422022] bfq_insert_requests+0x70/0x90 [ 1015.426106] blk_mq_sched_insert_requests+0xac/0x250 [ 1015.431057] blk_mq_flush_plug_list+0x110/0x2cc [ 1015.435575] __blk_flush_plug+0x100/0x140 [ 1015.439571] blk_finish_plug+0x40/0x25c [ 1015.443395] __iomap_dio_rw+0x1b4/0x500 [ 1015.447217] iomap_dio_rw+0x18/0x50 [ 1015.450693] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1015.456044] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1015.460786] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1015.465830] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1015.470960] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1015.476089] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1015.481475] process_one_work+0x1ec/0x470 [ 1015.485472] worker_thread+0x74/0x410 [ 1015.489122] kthread+0xec/0x100 [ 1015.492251] ret_from_fork+0x10/0x20 [ 1025.888121] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1025.893854] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=29886 [ 1025.902888] (t=60100 jiffies g=36445 q=18081 ncpus=128) [ 1025.908187] Task dump for CPU 8: [ 1025.911401] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1025.921302] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1025.927469] Call trace: [ 1025.929903] dump_backtrace+0xe8/0x130 [ 1025.933639] show_stack+0x20/0x60 [ 1025.936941] sched_show_task+0x160/0x190 [ 1025.940850] dump_cpu_task+0x4c/0x78 [ 1025.944413] rcu_dump_cpu_stacks+0xc4/0x150 [ 1025.948584] print_cpu_stall+0x180/0x210 [ 1025.952494] check_cpu_stall+0x20c/0x2b4 [ 1025.956403] rcu_pending+0x4c/0x190 [ 1025.959879] rcu_sched_clock_irq+0x80/0x204 [ 1025.964049] update_process_times+0x7c/0xc0 [ 1025.968218] tick_sched_handle+0x38/0x74 [ 1025.972129] tick_sched_timer+0x60/0xc0 [ 1025.975953] __hrtimer_run_queues+0x170/0x300 [ 1025.980297] hrtimer_interrupt+0xfc/0x260 [ 1025.984293] arch_timer_handler_phys+0x34/0x50 [ 1025.988724] handle_percpu_devid_irq+0xac/0x240 [ 1025.993242] generic_handle_domain_irq+0x34/0x50 [ 1025.997847] gic_handle_irq+0x5c/0xe0 [ 1026.001496] call_on_irq_stack+0x2c/0x38 [ 1026.005405] do_interrupt_handler+0xdc/0xe0 [ 1026.009576] el1_interrupt+0x34/0x70 [ 1026.013138] el1h_64_irq_handler+0x18/0x24 [ 1026.017221] el1h_64_irq+0x68/0x6c [ 1026.020610] __sbq_wake_up+0x58/0x1f0 [ 1026.024259] sbitmap_queue_clear+0x74/0xc0 [ 1026.028343] blk_mq_put_tag+0x44/0x70 [ 1026.031992] __blk_mq_free_request+0x88/0xf0 [ 1026.036249] blk_mq_free_request+0xc0/0x160 [ 1026.040419] __blk_mq_end_request+0x78/0x150 [ 1026.044676] scsi_end_request+0x114/0x260 [ 1026.048672] scsi_io_completion+0x64/0x174 [ 1026.052755] scsi_finish_command+0xe0/0x120 [ 1026.056926] scsi_complete+0x8c/0x1c4 [ 1026.060576] blk_complete_reqs+0x5c/0x74 [ 1026.064485] blk_done_softirq+0x38/0x4c [ 1026.068308] __do_softirq+0x168/0x418 [ 1026.071957] __irq_exit_rcu+0x120/0x170 [ 1026.075779] irq_exit_rcu+0x18/0x24 [ 1026.079255] el1_interrupt+0x38/0x70 [ 1026.082817] el1h_64_irq_handler+0x18/0x24 [ 1026.086900] el1h_64_irq+0x68/0x6c [ 1026.090289] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1026.095067] __wake_up_common_lock+0x84/0xac [ 1026.099324] __wake_up+0x20/0x30 [ 1026.102540] __sbq_wake_up+0x13c/0x1f0 [ 1026.106276] sbitmap_queue_clear+0x74/0xc0 [ 1026.110359] blk_mq_put_tag+0x44/0x70 [ 1026.114009] __blk_mq_free_request+0x88/0xf0 [ 1026.118266] blk_mq_free_request+0xc0/0x160 [ 1026.122436] bfq_insert_request.isra.0+0x204/0x270 [ 1026.127214] bfq_insert_requests+0x70/0x90 [ 1026.131297] blk_mq_sched_insert_requests+0xac/0x250 [ 1026.136249] blk_mq_flush_plug_list+0x110/0x2cc [ 1026.140766] __blk_flush_plug+0x100/0x140 [ 1026.144762] blk_finish_plug+0x40/0x25c [ 1026.148585] __iomap_dio_rw+0x1b4/0x500 [ 1026.152408] iomap_dio_rw+0x18/0x50 [ 1026.155885] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1026.161236] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1026.165978] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1026.171022] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1026.176152] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1026.181282] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1026.186668] process_one_work+0x1ec/0x470 [ 1026.190664] worker_thread+0x74/0x410 [ 1026.194314] kthread+0xec/0x100 [ 1026.197442] ret_from_fork+0x10/0x20 [ 1026.201005] Task dump for CPU 68: [ 1026.204307] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1026.214207] Call trace: [ 1026.216640] __switch_to+0xd4/0x100 [ 1026.220116] 0x0 [ 1027.218112] watchdog: BUG: soft lockup - CPU#68 stuck for 562s! [swapper/68:0] [ 1027.225320] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1027.225337] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1027.313124] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1027.320941] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1027.330232] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1027.337180] pc : __sbq_wake_up+0x48/0x1f0 [ 1027.341176] lr : sbitmap_queue_clear+0x74/0xc0 [ 1027.345607] sp : ffff8000088438b0 [ 1027.348908] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1027.356030] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1027.363152] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1027.370274] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1027.377395] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1027.384517] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1027.391639] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1027.398761] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1027.405883] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1027.413004] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 1027.420126] Call trace: [ 1027.422560] __sbq_wake_up+0x48/0x1f0 [ 1027.426209] sbitmap_queue_clear+0x74/0xc0 [ 1027.430292] blk_mq_put_tag+0x44/0x70 [ 1027.433942] __blk_mq_free_request+0x88/0xf0 [ 1027.438199] blk_mq_free_request+0xc0/0x160 [ 1027.442368] __blk_mq_end_request+0x78/0x150 [ 1027.446625] scsi_end_request+0x114/0x260 [ 1027.450622] scsi_io_completion+0x64/0x174 [ 1027.454705] scsi_finish_command+0xe0/0x120 [ 1027.458876] scsi_complete+0x8c/0x1c4 [ 1027.462526] blk_complete_reqs+0x5c/0x74 [ 1027.466436] blk_done_softirq+0x38/0x4c [ 1027.470259] __do_softirq+0x168/0x418 [ 1027.473908] __irq_exit_rcu+0x120/0x170 [ 1027.477730] irq_exit_rcu+0x18/0x24 [ 1027.481206] el1_interrupt+0x38/0x70 [ 1027.484768] el1h_64_irq_handler+0x18/0x24 [ 1027.488851] el1h_64_irq+0x68/0x6c [ 1027.492240] cpuidle_enter_state+0xc4/0x4e0 [ 1027.496410] cpuidle_enter+0x40/0x60 [ 1027.499972] cpuidle_idle_call+0x118/0x1b0 [ 1027.504056] do_idle+0xac/0x100 [ 1027.507184] cpu_startup_entry+0x30/0x34 [ 1027.511093] secondary_start_kernel+0xd8/0xf0 [ 1027.515437] __secondary_switched+0xb0/0xb4 [ 1051.107955] watchdog: BUG: soft lockup - CPU#8 stuck for 584s! [kworker/8:12:19399] [ 1051.115597] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1051.115614] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1051.203401] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1051.211651] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1051.217954] watchdog: BUG: soft lockup - CPU#68 stuck for 585s! [swapper/68:0] [ 1051.220942] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1051.228149] Modules linked in: nvme_loop [ 1051.234315] [ 1051.234315] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1051.238224] nvme_fabrics [ 1051.239702] pc : __sbq_wake_up+0x3c/0x1f0 [ 1051.246648] nvmet [ 1051.249255] lr : sbitmap_queue_clear+0x74/0xc0 [ 1051.253250] nls_utf8 [ 1051.255249] sp : ffff80000d79b280 [ 1051.259679] hfsplus [ 1051.261938] x29: ffff80000d79b280 [ 1051.265239] hfs [ 1051.267411] x28: ffffd5becb6aec50 [ 1051.270712] isofs [ 1051.272537] x27: ffffd5beca4f7000 [ 1051.275925] nvme_core [ 1051.277925] [ 1051.281314] nvme_common [ 1051.283660] x26: ffffd5becaf68a98 [ 1051.285138] loop [ 1051.287658] x25: ffffd5becb6a60c0 [ 1051.290958] tls [ 1051.292870] x24: fffffbffef180dc0 [ 1051.296258] rfkill [ 1051.298084] [ 1051.301471] sunrpc [ 1051.303557] x23: ffff07ff81d381b0 [ 1051.305034] vfat [ 1051.307120] x22: ffff80000d79b30c [ 1051.310421] fat [ 1051.312333] x21: ffff07ff84e93c00 [ 1051.315721] acpi_ipmi [ 1051.317546] [ 1051.320933] ipmi_ssif [ 1051.323279] x20: 0000000000000024 [ 1051.324757] cppc_cpufreq [ 1051.327103] x19: ffff07ff91ccdb10 [ 1051.330404] ipmi_devintf [ 1051.333010] x18: ffffffffffffffff [ 1051.336398] igb [ 1051.339004] [ 1051.342392] ipmi_msghandler [ 1051.344217] x17: ffff32506c412000 [ 1051.345695] arm_cmn [ 1051.348561] x16: ffff800008044000 [ 1051.351862] arm_dmc620_pmu [ 1051.354034] x15: 0000000000004000 [ 1051.357422] arm_spe_pmu [ 1051.360202] [ 1051.363589] arm_dsu_pmu [ 1051.366109] x14: 010320bb28ec0000 [ 1051.367587] acpi_tad [ 1051.370107] x13: 0000000000000087 [ 1051.373409] fuse [ 1051.375668] x12: 071c71c71c71c71c [ 1051.379056] zram [ 1051.380968] [ 1051.384355] xfs [ 1051.386267] x11: 0000000000000040 [ 1051.387745] crct10dif_ce [ 1051.389570] x10: ffff07ff801a9c70 [ 1051.392871] polyval_ce [ 1051.395477] x9 : ffffd5bec9545fd8 [ 1051.398865] ast [ 1051.401298] [ 1051.404685] polyval_generic [ 1051.406510] x8 : ffff07ff80404b70 [ 1051.407988] drm_vram_helper [ 1051.410855] x7 : ffff07ff84e93dc8 [ 1051.414156] ghash_ce [ 1051.417022] x6 : 0000000000000008 [ 1051.420410] drm_ttm_helper [ 1051.422670] [ 1051.426057] sbsa_gwdt [ 1051.428837] x5 : ffff80000d043790 [ 1051.430315] ttm [ 1051.432661] x4 : ffff07ff84e93dc0 [ 1051.435962] xgene_hwmon [ 1051.437787] x3 : 0000000000000008 [ 1051.441174] Unloaded tainted modules: [ 1051.443694] [ 1051.447081] nvmet():1 [ 1051.450729] x2 : 0000000000000007 [ 1051.452207] nvme_fabrics():1 [ 1051.454553] x1 : ffff80000d79b30c [ 1051.457854] nvme_loop():1 [ 1051.460808] x0 : 0000000000000038 [ 1051.464195] nvmet():1 [ 1051.466888] [ 1051.470276] nvme_fabrics():1 [ 1051.472622] Call trace: [ 1051.474100] nvme_loop():1 [ 1051.477053] __sbq_wake_up+0x3c/0x1f0 [ 1051.479486] nvmet():1 [ 1051.482179] sbitmap_queue_clear+0x74/0xc0 [ 1051.485827] nvme_fabrics():1 [ 1051.488173] blk_mq_put_tag+0x44/0x70 [ 1051.492255] nvme_loop():1 [ 1051.495208] __blk_mq_free_request+0x88/0xf0 [ 1051.498856] nvmet():1 [ 1051.501550] blk_mq_free_request+0xc0/0x160 [ 1051.505805] nvme_fabrics():1 [ 1051.508151] __blk_mq_end_request+0x78/0x150 [ 1051.512320] nvme_loop():1 [ 1051.515273] scsi_end_request+0x114/0x260 [ 1051.519529] nvmet():1 [ 1051.522223] scsi_io_completion+0x64/0x174 [ 1051.526219] nvme_fabrics():1 [ 1051.528565] scsi_finish_command+0xe0/0x120 [ 1051.532647] nvme_loop():1 [ 1051.535601] scsi_complete+0x8c/0x1c4 [ 1051.539770] nvmet():1 [ 1051.542463] blk_complete_reqs+0x5c/0x74 [ 1051.546111] nvme_fabrics():1 [ 1051.548457] blk_done_softirq+0x38/0x4c [ 1051.552366] nvme_loop():1 [ 1051.555319] __do_softirq+0x168/0x418 [ 1051.559141] nvmet():1 [ 1051.561834] __irq_exit_rcu+0x120/0x170 [ 1051.565482] nvme_fabrics():1 [ 1051.567828] irq_exit_rcu+0x18/0x24 [ 1051.571650] nvme_loop():1 [ 1051.574603] el1_interrupt+0x38/0x70 [ 1051.578078] nvmet():1 [ 1051.580771] el1h_64_irq_handler+0x18/0x24 [ 1051.584332] nvme_fabrics():1 [ 1051.586678] el1h_64_irq+0x68/0x6c [ 1051.590760] nvme_loop():1 [ 1051.593713] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1051.597101] nvmet():1 [ 1051.599794] __wake_up_common_lock+0x84/0xac [ 1051.604570] nvme_fabrics():1 [ 1051.606916] __wake_up+0x20/0x30 [ 1051.611172] nvme_loop():1 [ 1051.614126] __sbq_wake_up+0x13c/0x1f0 [ 1051.617342] nvmet():1 [ 1051.620035] sbitmap_queue_clear+0x74/0xc0 [ 1051.623770] nvme_fabrics():1 [ 1051.626116] blk_mq_put_tag+0x44/0x70 [ 1051.630198] nvme_loop():1 [ 1051.633151] __blk_mq_free_request+0x88/0xf0 [ 1051.636799] null_blk():1 [ 1051.639492] blk_mq_free_request+0xc0/0x160 [ 1051.643748] null_blk():1 [ 1051.646354] bfq_insert_request.isra.0+0x204/0x270 [ 1051.650523] null_blk():1 [ 1051.653129] bfq_insert_requests+0x70/0x90 [ 1051.657906] null_blk():1 [ 1051.660512] blk_mq_sched_insert_requests+0xac/0x250 [ 1051.664594] null_blk():1 [ 1051.667201] blk_mq_flush_plug_list+0x110/0x2cc [ 1051.672151] null_blk():1 [ 1051.674757] __blk_flush_plug+0x100/0x140 [ 1051.679273] null_blk():1 [ 1051.681880] blk_finish_plug+0x40/0x25c [ 1051.685875] scsi_debug():1 [ 1051.688481] __iomap_dio_rw+0x1b4/0x500 [ 1051.692303] scsi_debug():1 [ 1051.695083] iomap_dio_rw+0x18/0x50 [ 1051.698904] ipmi_si():1 [ 1051.701684] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1051.705159] ipmi_si():1 [ 1051.707678] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1051.712976] [last unloaded: nvmet] [ 1051.715496] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1051.720185] [ 1051.723660] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1051.728697] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1051.730175] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1051.735299] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1051.743114] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1051.748238] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1051.757528] process_one_work+0x1ec/0x470 [ 1051.762912] pc : __sbq_wake_up+0x58/0x1f0 [ 1051.769859] worker_thread+0x74/0x410 [ 1051.773854] lr : sbitmap_queue_clear+0x74/0xc0 [ 1051.777849] kthread+0xec/0x100 [ 1051.781497] sp : ffff8000088438b0 [ 1051.785927] ret_from_fork+0x10/0x20 [ 1051.789055] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1051.803039] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1051.810161] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1051.817283] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1051.824405] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1051.831527] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1051.838648] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1051.845770] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1051.852892] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1051.860014] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1051.867136] Call trace: [ 1051.869569] __sbq_wake_up+0x58/0x1f0 [ 1051.873218] sbitmap_queue_clear+0x74/0xc0 [ 1051.877301] blk_mq_put_tag+0x44/0x70 [ 1051.880951] __blk_mq_free_request+0x88/0xf0 [ 1051.885207] blk_mq_free_request+0xc0/0x160 [ 1051.889377] __blk_mq_end_request+0x78/0x150 [ 1051.893634] scsi_end_request+0x114/0x260 [ 1051.897630] scsi_io_completion+0x64/0x174 [ 1051.901713] scsi_finish_command+0xe0/0x120 [ 1051.905885] scsi_complete+0x8c/0x1c4 [ 1051.909534] blk_complete_reqs+0x5c/0x74 [ 1051.913443] blk_done_softirq+0x38/0x4c [ 1051.917266] __do_softirq+0x168/0x418 [ 1051.920915] __irq_exit_rcu+0x120/0x170 [ 1051.924737] irq_exit_rcu+0x18/0x24 [ 1051.928213] el1_interrupt+0x38/0x70 [ 1051.931775] el1h_64_irq_handler+0x18/0x24 [ 1051.935858] el1h_64_irq+0x68/0x6c [ 1051.939246] cpuidle_enter_state+0xc4/0x4e0 [ 1051.943417] cpuidle_enter+0x40/0x60 [ 1051.946979] cpuidle_idle_call+0x118/0x1b0 [ 1051.951063] do_idle+0xac/0x100 [ 1051.954192] cpu_startup_entry+0x30/0x34 [ 1051.958101] secondary_start_kernel+0xd8/0xf0 [ 1051.962445] __secondary_switched+0xb0/0xb4 [ 1071.087833] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 62013 jiffies s: 2093 root: 0x11/. [ 1071.099486] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1071.108695] Task dump for CPU 8: [ 1071.111910] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1071.121812] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1071.127982] Call trace: [ 1071.130415] __switch_to+0xd4/0x100 [ 1071.133892] 0xffff6c0200000080 [ 1071.137021] Task dump for CPU 68: [ 1071.140323] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1071.150225] Call trace: [ 1071.152658] __switch_to+0xd4/0x100 [ 1071.156134] 0x0 [ 1075.107796] watchdog: BUG: soft lockup - CPU#8 stuck for 607s! [kworker/8:12:19399] [ 1075.115438] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1075.115455] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1075.203242] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1075.211491] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1075.217795] watchdog: BUG: soft lockup - CPU#68 stuck for 607s! [swapper/68:0] [ 1075.220782] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1075.227989] Modules linked in: nvme_loop [ 1075.234155] [ 1075.234156] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1075.238064] nvme_fabrics [ 1075.239542] pc : __sbq_wake_up+0x20/0x1f0 [ 1075.246489] nvmet [ 1075.249095] lr : sbitmap_queue_clear+0x74/0xc0 [ 1075.253090] nls_utf8 [ 1075.255089] sp : ffff80000d79b280 [ 1075.259518] hfsplus [ 1075.261777] x29: ffff80000d79b280 [ 1075.265078] hfs [ 1075.267251] x28: ffffd5becb6aec50 [ 1075.270551] isofs [ 1075.272377] x27: ffffd5beca4f7000 [ 1075.275764] nvme_core [ 1075.277763] [ 1075.281150] nvme_common [ 1075.283496] x26: ffffd5becaf68a98 [ 1075.284974] loop [ 1075.287494] x25: ffffd5becb6a60c0 [ 1075.290794] tls [ 1075.292706] x24: fffffbffef180dc0 [ 1075.296094] rfkill [ 1075.297919] [ 1075.301307] sunrpc [ 1075.303392] x23: ffff07ff81d381b0 [ 1075.304870] vfat [ 1075.306955] x22: 0000000000000024 [ 1075.310256] fat [ 1075.312168] x21: 0000000000000008 [ 1075.315556] acpi_ipmi [ 1075.317381] [ 1075.320768] ipmi_ssif [ 1075.323114] x20: 0000000000000024 [ 1075.324592] cppc_cpufreq [ 1075.326938] x19: ffff07ff91ccdb10 [ 1075.330239] ipmi_devintf [ 1075.332846] x18: ffffffffffffffff [ 1075.336233] igb [ 1075.338840] [ 1075.342228] ipmi_msghandler [ 1075.344054] x17: ffff32506c412000 [ 1075.345533] arm_cmn [ 1075.348400] x16: ffff800008044000 [ 1075.351701] arm_dmc620_pmu [ 1075.353873] x15: 0000000000004000 [ 1075.357261] arm_spe_pmu [ 1075.360041] [ 1075.363428] arm_dsu_pmu [ 1075.365948] x14: 010320bb28ec0000 [ 1075.367426] acpi_tad [ 1075.369945] x13: 0000000000000087 [ 1075.373246] fuse [ 1075.375505] x12: 071c71c71c71c71c [ 1075.378893] zram [ 1075.380804] [ 1075.384192] xfs [ 1075.386104] x11: 0000000000000040 [ 1075.387582] crct10dif_ce [ 1075.389407] x10: ffff07ff801a9c70 [ 1075.392708] polyval_ce [ 1075.395314] x9 : ffffd5bec9545fd8 [ 1075.398702] ast [ 1075.401135] [ 1075.404522] polyval_generic [ 1075.406347] x8 : ffff07ff80404b70 [ 1075.407825] drm_vram_helper [ 1075.410694] x7 : ffff07ff84e93dc8 [ 1075.413994] ghash_ce [ 1075.416861] x6 : 0000000000000008 [ 1075.420249] drm_ttm_helper [ 1075.422508] [ 1075.425895] sbsa_gwdt [ 1075.428675] x5 : ffff80000d043790 [ 1075.430153] ttm [ 1075.432499] x4 : ffff07ff84e93dc0 [ 1075.435800] xgene_hwmon [ 1075.437625] x3 : 0000000000000008 [ 1075.441012] Unloaded tainted modules: [ 1075.443532] [ 1075.446920] nvmet():1 [ 1075.450568] x2 : 0000000000000007 [ 1075.452045] nvme_fabrics():1 [ 1075.454391] x1 : ffff80000d79b30c [ 1075.457692] nvme_loop():1 [ 1075.460646] x0 : 0000000000000001 [ 1075.464033] nvmet():1 [ 1075.466726] [ 1075.470114] nvme_fabrics():1 [ 1075.472460] Call trace: [ 1075.473938] nvme_loop():1 [ 1075.476891] __sbq_wake_up+0x20/0x1f0 [ 1075.479324] nvmet():1 [ 1075.482017] sbitmap_queue_clear+0x74/0xc0 [ 1075.485665] nvme_fabrics():1 [ 1075.488011] blk_mq_put_tag+0x44/0x70 [ 1075.492093] nvme_loop():1 [ 1075.495047] __blk_mq_free_request+0x88/0xf0 [ 1075.498695] nvmet():1 [ 1075.501388] blk_mq_free_request+0xc0/0x160 [ 1075.505643] nvme_fabrics():1 [ 1075.507989] __blk_mq_end_request+0x78/0x150 [ 1075.512158] nvme_loop():1 [ 1075.515112] scsi_end_request+0x114/0x260 [ 1075.519367] nvmet():1 [ 1075.522060] scsi_io_completion+0x64/0x174 [ 1075.526056] nvme_fabrics():1 [ 1075.528402] scsi_finish_command+0xe0/0x120 [ 1075.532484] nvme_loop():1 [ 1075.535437] scsi_complete+0x8c/0x1c4 [ 1075.539606] nvmet():1 [ 1075.542299] blk_complete_reqs+0x5c/0x74 [ 1075.545947] nvme_fabrics():1 [ 1075.548293] blk_done_softirq+0x38/0x4c [ 1075.552202] nvme_loop():1 [ 1075.555155] __do_softirq+0x168/0x418 [ 1075.558977] nvmet():1 [ 1075.561670] __irq_exit_rcu+0x120/0x170 [ 1075.565318] nvme_fabrics():1 [ 1075.567664] irq_exit_rcu+0x18/0x24 [ 1075.571486] nvme_loop():1 [ 1075.574439] el1_interrupt+0x38/0x70 [ 1075.577914] nvmet():1 [ 1075.580607] el1h_64_irq_handler+0x18/0x24 [ 1075.584168] nvme_fabrics():1 [ 1075.586514] el1h_64_irq+0x68/0x6c [ 1075.590596] nvme_loop():1 [ 1075.593549] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1075.596937] nvmet():1 [ 1075.599630] __wake_up_common_lock+0x84/0xac [ 1075.604407] nvme_fabrics():1 [ 1075.606753] __wake_up+0x20/0x30 [ 1075.611009] nvme_loop():1 [ 1075.613962] __sbq_wake_up+0x13c/0x1f0 [ 1075.617176] nvmet():1 [ 1075.619869] sbitmap_queue_clear+0x74/0xc0 [ 1075.623604] nvme_fabrics():1 [ 1075.625950] blk_mq_put_tag+0x44/0x70 [ 1075.630032] nvme_loop():1 [ 1075.632986] __blk_mq_free_request+0x88/0xf0 [ 1075.636634] null_blk():1 [ 1075.639327] blk_mq_free_request+0xc0/0x160 [ 1075.643583] null_blk():1 [ 1075.646189] bfq_insert_request.isra.0+0x204/0x270 [ 1075.650358] null_blk():1 [ 1075.652964] bfq_insert_requests+0x70/0x90 [ 1075.657741] null_blk():1 [ 1075.660348] blk_mq_sched_insert_requests+0xac/0x250 [ 1075.664430] null_blk():1 [ 1075.667036] blk_mq_flush_plug_list+0x110/0x2cc [ 1075.671986] null_blk():1 [ 1075.674592] __blk_flush_plug+0x100/0x140 [ 1075.679108] null_blk():1 [ 1075.681715] blk_finish_plug+0x40/0x25c [ 1075.685710] scsi_debug():1 [ 1075.688316] __iomap_dio_rw+0x1b4/0x500 [ 1075.692138] scsi_debug():1 [ 1075.694918] iomap_dio_rw+0x18/0x50 [ 1075.698739] ipmi_si():1 [ 1075.701519] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1075.704994] ipmi_si():1 [ 1075.707513] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1075.712810] [last unloaded: nvmet] [ 1075.715331] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1075.720020] [ 1075.723495] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1075.728532] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1075.730009] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1075.735133] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1075.742948] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1075.748071] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1075.757362] process_one_work+0x1ec/0x470 [ 1075.762746] pc : __sbq_wake_up+0x20/0x1f0 [ 1075.769693] worker_thread+0x74/0x410 [ 1075.773688] lr : sbitmap_queue_clear+0x74/0xc0 [ 1075.777683] kthread+0xec/0x100 [ 1075.781331] sp : ffff8000088438b0 [ 1075.785761] ret_from_fork+0x10/0x20 [ 1075.788888] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1075.802872] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1075.809994] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1075.817116] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1075.824238] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1075.831360] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1075.838482] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1075.845604] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1075.852726] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1075.859849] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1075.866971] Call trace: [ 1075.869404] __sbq_wake_up+0x20/0x1f0 [ 1075.873053] sbitmap_queue_clear+0x74/0xc0 [ 1075.877137] blk_mq_put_tag+0x44/0x70 [ 1075.880787] __blk_mq_free_request+0x88/0xf0 [ 1075.885043] blk_mq_free_request+0xc0/0x160 [ 1075.889213] __blk_mq_end_request+0x78/0x150 [ 1075.893470] scsi_end_request+0x114/0x260 [ 1075.897466] scsi_io_completion+0x64/0x174 [ 1075.901549] scsi_finish_command+0xe0/0x120 [ 1075.905720] scsi_complete+0x8c/0x1c4 [ 1075.909369] blk_complete_reqs+0x5c/0x74 [ 1075.913278] blk_done_softirq+0x38/0x4c [ 1075.917101] __do_softirq+0x168/0x418 [ 1075.920750] __irq_exit_rcu+0x120/0x170 [ 1075.924573] irq_exit_rcu+0x18/0x24 [ 1075.928048] el1_interrupt+0x38/0x70 [ 1075.931611] el1h_64_irq_handler+0x18/0x24 [ 1075.935694] el1h_64_irq+0x68/0x6c [ 1075.939082] cpuidle_enter_state+0xc4/0x4e0 [ 1075.943252] cpuidle_enter+0x40/0x60 [ 1075.946814] cpuidle_idle_call+0x118/0x1b0 [ 1075.950898] do_idle+0xac/0x100 [ 1075.954027] cpu_startup_entry+0x30/0x34 [ 1075.957938] secondary_start_kernel+0xd8/0xf0 [ 1075.962282] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:15:00 2022] [ 1099.107637] watchdog: BUG: soft lockup - CPU#8 stuck for 629s! [kworker/8:12:19399] [ 1099.115280] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1099.115296] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1099.203084] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1099.211333] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1099.217637] watchdog: BUG: soft lockup - CPU#68 stuck for 629s! [swapper/68:0] [ 1099.220624] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1099.227831] Modules linked in: nvme_loop [ 1099.233997] [ 1099.233998] nvme_fabrics [ 1099.237907] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1099.239384] nvmet [ 1099.241991] pc : __sbq_wake_up+0x20/0x1f0 [ 1099.248937] nls_utf8 [ 1099.250936] lr : sbitmap_queue_clear+0x74/0xc0 [ 1099.254931] hfsplus [ 1099.257190] sp : ffff80000d79b280 [ 1099.261620] hfs [ 1099.263792] x29: ffff80000d79b280 [ 1099.267093] isofs [ 1099.268918] x28: ffffd5becb6aec50 [ 1099.272219] nvme_core [ 1099.274217] x27: ffffd5beca4f7000 [ 1099.277605] nvme_common [ 1099.279951] [ 1099.283339] loop [ 1099.285860] x26: ffffd5becaf68a98 [ 1099.287338] tls [ 1099.289250] x25: ffffd5becb6a60c0 [ 1099.292551] rfkill [ 1099.294376] x24: fffffbffef180dc0 [ 1099.297763] sunrpc [ 1099.299849] [ 1099.303236] vfat [ 1099.305322] x23: ffff07ff81d381b0 [ 1099.306800] fat [ 1099.308711] x22: 0000000000000024 [ 1099.312012] acpi_ipmi [ 1099.313837] x21: 0000000000000008 [ 1099.317225] ipmi_ssif [ 1099.319571] [ 1099.322958] cppc_cpufreq [ 1099.325305] x20: 0000000000000024 [ 1099.326783] ipmi_devintf [ 1099.329389] x19: ffff07ff91ccdb10 [ 1099.332690] igb [ 1099.335296] x18: ffffffffffffffff [ 1099.338684] ipmi_msghandler [ 1099.340509] [ 1099.343896] arm_cmn [ 1099.346763] x17: ffff32506c412000 [ 1099.348241] arm_dmc620_pmu [ 1099.350413] x16: ffff800008044000 [ 1099.353714] arm_spe_pmu [ 1099.356494] x15: 0000000000004000 [ 1099.359882] arm_dsu_pmu [ 1099.362401] [ 1099.365789] acpi_tad [ 1099.368309] x14: 010320bb28ec0000 [ 1099.369787] fuse [ 1099.372046] x13: 0000000000000087 [ 1099.375346] zram [ 1099.377258] x12: 071c71c71c71c71c [ 1099.380646] xfs [ 1099.382558] [ 1099.385945] crct10dif_ce [ 1099.387770] x11: 0000000000000040 [ 1099.389248] polyval_ce [ 1099.391855] x10: ffff07ff801a9c70 [ 1099.395155] ast [ 1099.397588] x9 : ffffd5bec9545fd8 [ 1099.400976] polyval_generic [ 1099.402801] [ 1099.406188] drm_vram_helper [ 1099.409055] x8 : ffff07ff80404b70 [ 1099.410533] ghash_ce [ 1099.413400] x7 : ffff07ff84e93dc8 [ 1099.416701] drm_ttm_helper [ 1099.418960] x6 : 0000000000000008 [ 1099.422348] sbsa_gwdt [ 1099.425128] [ 1099.428516] ttm [ 1099.430862] x5 : ffff80000d043790 [ 1099.432340] xgene_hwmon [ 1099.434166] x4 : ffff07ff84e93dc0 [ 1099.437467] Unloaded tainted modules: [ 1099.439986] x3 : 0000000000000008 [ 1099.443374] nvmet():1 [ 1099.447022] [ 1099.450410] nvme_fabrics():1 [ 1099.452755] x2 : 0000000000000007 [ 1099.454233] nvme_loop():1 [ 1099.457187] x1 : ffff80000d79b30c [ 1099.460488] nvmet():1 [ 1099.463181] x0 : 0000000000000001 [ 1099.466568] nvme_fabrics():1 [ 1099.468914] [ 1099.472302] nvme_loop():1 [ 1099.475255] Call trace: [ 1099.476733] nvmet():1 [ 1099.479426] __sbq_wake_up+0x20/0x1f0 [ 1099.481859] nvme_fabrics():1 [ 1099.484205] sbitmap_queue_clear+0x74/0xc0 [ 1099.487853] nvme_loop():1 [ 1099.490807] blk_mq_put_tag+0x44/0x70 [ 1099.494889] nvmet():1 [ 1099.497583] __blk_mq_free_request+0x88/0xf0 [ 1099.501231] nvme_fabrics():1 [ 1099.503577] blk_mq_free_request+0xc0/0x160 [ 1099.507833] nvme_loop():1 [ 1099.510787] __blk_mq_end_request+0x78/0x150 [ 1099.514956] nvmet():1 [ 1099.517649] scsi_end_request+0x114/0x260 [ 1099.521905] nvme_fabrics():1 [ 1099.524250] scsi_io_completion+0x64/0x174 [ 1099.528246] nvme_loop():1 [ 1099.531199] scsi_finish_command+0xe0/0x120 [ 1099.535281] nvmet():1 [ 1099.537974] scsi_complete+0x8c/0x1c4 [ 1099.542143] nvme_fabrics():1 [ 1099.544489] blk_complete_reqs+0x5c/0x74 [ 1099.548137] nvme_loop():1 [ 1099.551091] blk_done_softirq+0x38/0x4c [ 1099.554999] nvmet():1 [ 1099.557692] __do_softirq+0x168/0x418 [ 1099.561514] nvme_fabrics():1 [ 1099.563860] __irq_exit_rcu+0x120/0x170 [ 1099.567508] nvme_loop():1 [ 1099.570461] irq_exit_rcu+0x18/0x24 [ 1099.574283] nvmet():1 [ 1099.576976] el1_interrupt+0x38/0x70 [ 1099.580450] nvme_fabrics():1 [ 1099.582796] el1h_64_irq_handler+0x18/0x24 [ 1099.586357] nvme_loop():1 [ 1099.589311] el1h_64_irq+0x68/0x6c [ 1099.593393] nvmet():1 [ 1099.596086] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1099.599474] nvme_fabrics():1 [ 1099.601820] __wake_up_common_lock+0x84/0xac [ 1099.606596] nvme_loop():1 [ 1099.609550] __wake_up+0x20/0x30 [ 1099.613805] nvmet():1 [ 1099.616498] __sbq_wake_up+0x13c/0x1f0 [ 1099.619712] nvme_fabrics():1 [ 1099.622058] sbitmap_queue_clear+0x74/0xc0 [ 1099.625793] nvme_loop():1 [ 1099.628747] blk_mq_put_tag+0x44/0x70 [ 1099.632829] null_blk():1 [ 1099.635524] __blk_mq_free_request+0x88/0xf0 [ 1099.639172] null_blk():1 [ 1099.641779] blk_mq_free_request+0xc0/0x160 [ 1099.646034] null_blk():1 [ 1099.648641] bfq_insert_request.isra.0+0x204/0x270 [ 1099.652810] null_blk():1 [ 1099.655416] bfq_insert_requests+0x70/0x90 [ 1099.660192] null_blk():1 [ 1099.662799] blk_mq_sched_insert_requests+0xac/0x250 [ 1099.666881] null_blk():1 [ 1099.669487] blk_mq_flush_plug_list+0x110/0x2cc [ 1099.674437] null_blk():1 [ 1099.677043] __blk_flush_plug+0x100/0x140 [ 1099.681559] scsi_debug():1 [ 1099.684166] blk_finish_plug+0x40/0x25c [ 1099.688161] scsi_debug():1 [ 1099.690941] __iomap_dio_rw+0x1b4/0x500 [ 1099.694762] ipmi_si():1 [ 1099.697542] iomap_dio_rw+0x18/0x50 [ 1099.701364] ipmi_si():1 [ 1099.703884] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1099.707358] [last unloaded: nvmet] [ 1099.709878] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1099.715175] [ 1099.718650] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1099.723339] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1099.724817] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1099.729854] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1099.737669] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1099.742792] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1099.752083] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1099.757206] pc : __sbq_wake_up+0x4/0x1f0 [ 1099.764153] process_one_work+0x1ec/0x470 [ 1099.769537] lr : sbitmap_queue_clear+0x74/0xc0 [ 1099.773445] worker_thread+0x74/0x410 [ 1099.777441] sp : ffff800008843900 [ 1099.781870] kthread+0xec/0x100 [ 1099.785518] x29: ffff800008843900 [ 1099.788819] ret_from_fork+0x10/0x20 [ 1099.791946] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1099.804106] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1099.811228] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1099.818350] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1099.825472] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1099.832593] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1099.839715] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1099.846837] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1099.853959] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1099.861081] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1099.868203] Call trace: [ 1099.870635] __sbq_wake_up+0x4/0x1f0 [ 1099.874198] blk_mq_put_tag+0x44/0x70 [ 1099.877847] __blk_mq_free_request+0x88/0xf0 [ 1099.882104] blk_mq_free_request+0xc0/0x160 [ 1099.886274] __blk_mq_end_request+0x78/0x150 [ 1099.890531] scsi_end_request+0x114/0x260 [ 1099.894527] scsi_io_completion+0x64/0x174 [ 1099.898610] scsi_finish_command+0xe0/0x120 [ 1099.902781] scsi_complete+0x8c/0x1c4 [ 1099.906430] blk_complete_reqs+0x5c/0x74 [ 1099.910339] blk_done_softirq+0x38/0x4c [ 1099.914162] __do_softirq+0x168/0x418 [ 1099.917811] __irq_exit_rcu+0x120/0x170 [ 1099.921633] irq_exit_rcu+0x18/0x24 [ 1099.925109] el1_interrupt+0x38/0x70 [ 1099.928672] el1h_64_irq_handler+0x18/0x24 [ 1099.932756] el1h_64_irq+0x68/0x6c [ 1099.936144] cpuidle_enter_state+0xc4/0x4e0 [ 1099.940314] cpuidle_enter+0x40/0x60 [ 1099.943877] cpuidle_idle_call+0x118/0x1b0 [ 1099.947960] do_idle+0xac/0x100 [ 1099.951089] cpu_startup_entry+0x30/0x34 [ 1099.954998] secondary_start_kernel+0xd8/0xf0 [ 1099.959342] __secondary_switched+0xb0/0xb4 [ 1123.107479] watchdog: BUG: soft lockup - CPU#8 stuck for 651s! [kworker/8:12:19399] [ 1123.115122] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1123.115139] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1123.202926] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1123.211175] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1123.217478] watchdog: BUG: soft lockup - CPU#68 stuck for 652s! [swapper/68:0] [ 1123.220466] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1123.227673] Modules linked in: nvme_loop [ 1123.233839] [ 1123.233840] nvme_fabrics [ 1123.237749] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1123.239227] nvmet [ 1123.241833] pc : __sbq_wake_up+0x58/0x1f0 [ 1123.248779] nls_utf8 [ 1123.250778] lr : sbitmap_queue_clear+0x74/0xc0 [ 1123.254773] hfsplus [ 1123.257032] sp : ffff80000d79b280 [ 1123.261463] hfs [ 1123.263635] x29: ffff80000d79b280 [ 1123.266937] isofs [ 1123.268762] x28: ffffd5becb6aec50 [ 1123.272063] nvme_core [ 1123.274062] x27: ffffd5beca4f7000 [ 1123.277450] nvme_common [ 1123.279796] [ 1123.283183] loop [ 1123.285703] x26: ffffd5becaf68a98 [ 1123.287180] tls [ 1123.289092] x25: ffffd5becb6a60c0 [ 1123.292393] rfkill [ 1123.294218] x24: fffffbffef180dc0 [ 1123.297606] sunrpc [ 1123.299692] [ 1123.303079] vfat [ 1123.305165] x23: ffff07ff81d381b0 [ 1123.306643] fat [ 1123.308555] x22: ffff80000d79b30c [ 1123.311855] acpi_ipmi [ 1123.313681] x21: ffff07ff84e93c00 [ 1123.317068] ipmi_ssif [ 1123.319414] [ 1123.322802] cppc_cpufreq [ 1123.325147] x20: 00000000000001c0 [ 1123.326625] ipmi_devintf [ 1123.329232] x19: ffff07ff91ccdb10 [ 1123.332533] igb [ 1123.335140] x18: ffffffffffffffff [ 1123.338528] ipmi_msghandler [ 1123.340353] [ 1123.343741] arm_cmn [ 1123.346607] x17: ffff32506c412000 [ 1123.348085] arm_dmc620_pmu [ 1123.350258] x16: ffff800008044000 [ 1123.353558] arm_spe_pmu [ 1123.356339] x15: 0000000000004000 [ 1123.359726] arm_dsu_pmu [ 1123.362246] [ 1123.365633] acpi_tad [ 1123.368153] x14: 010320bb28ec0000 [ 1123.369631] fuse [ 1123.371890] x13: 0000000000000087 [ 1123.375191] zram [ 1123.377103] x12: 071c71c71c71c71c [ 1123.380491] xfs [ 1123.382403] [ 1123.385790] crct10dif_ce [ 1123.387615] x11: 0000000000000040 [ 1123.389093] polyval_ce [ 1123.391699] x10: ffff07ff801a9c70 [ 1123.395000] ast [ 1123.397433] x9 : ffffd5bec9545fd8 [ 1123.400820] polyval_generic [ 1123.402646] [ 1123.406034] drm_vram_helper [ 1123.408902] x8 : ffff07ff80404b70 [ 1123.410380] ghash_ce [ 1123.413247] x7 : ffff07ff84e93dc8 [ 1123.416547] drm_ttm_helper [ 1123.418806] x6 : 0000000000000008 [ 1123.422194] sbsa_gwdt [ 1123.424974] [ 1123.428362] ttm [ 1123.430707] x5 : ffff80000d043790 [ 1123.432185] xgene_hwmon [ 1123.434011] x4 : ffff07ff84e93dc0 [ 1123.437312] Unloaded tainted modules: [ 1123.439832] x3 : 0000000000000008 [ 1123.443221] nvmet():1 [ 1123.446869] [ 1123.450257] nvme_fabrics():1 [ 1123.452602] x2 : 0000000000000007 [ 1123.454080] nvme_loop():1 [ 1123.457035] x1 : ffff80000d79b30c [ 1123.460336] nvmet():1 [ 1123.463029] x0 : ffff07ff84e93dd0 [ 1123.466417] nvme_fabrics():1 [ 1123.468763] [ 1123.472150] nvme_loop():1 [ 1123.475104] Call trace: [ 1123.476582] nvmet():1 [ 1123.479275] __sbq_wake_up+0x58/0x1f0 [ 1123.481708] nvme_fabrics():1 [ 1123.484054] sbitmap_queue_clear+0x74/0xc0 [ 1123.487702] nvme_loop():1 [ 1123.490655] blk_mq_put_tag+0x44/0x70 [ 1123.494737] nvmet():1 [ 1123.497431] __blk_mq_free_request+0x88/0xf0 [ 1123.501080] nvme_fabrics():1 [ 1123.503426] blk_mq_free_request+0xc0/0x160 [ 1123.507681] nvme_loop():1 [ 1123.510635] __blk_mq_end_request+0x78/0x150 [ 1123.514804] nvmet():1 [ 1123.517497] scsi_end_request+0x114/0x260 [ 1123.521753] nvme_fabrics():1 [ 1123.524099] scsi_io_completion+0x64/0x174 [ 1123.528094] nvme_loop():1 [ 1123.531048] scsi_finish_command+0xe0/0x120 [ 1123.535130] nvmet():1 [ 1123.537823] scsi_complete+0x8c/0x1c4 [ 1123.541992] nvme_fabrics():1 [ 1123.544338] blk_complete_reqs+0x5c/0x74 [ 1123.547986] nvme_loop():1 [ 1123.550939] blk_done_softirq+0x38/0x4c [ 1123.554848] nvmet():1 [ 1123.557541] __do_softirq+0x168/0x418 [ 1123.561362] nvme_fabrics():1 [ 1123.563708] __irq_exit_rcu+0x120/0x170 [ 1123.567356] nvme_loop():1 [ 1123.570310] irq_exit_rcu+0x18/0x24 [ 1123.574131] nvmet():1 [ 1123.576825] el1_interrupt+0x38/0x70 [ 1123.580299] nvme_fabrics():1 [ 1123.582646] el1h_64_irq_handler+0x18/0x24 [ 1123.586207] nvme_loop():1 [ 1123.589161] el1h_64_irq+0x68/0x6c [ 1123.593243] nvmet():1 [ 1123.595936] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1123.599323] nvme_fabrics():1 [ 1123.601669] __wake_up_common_lock+0x84/0xac [ 1123.606446] nvme_loop():1 [ 1123.609399] __wake_up+0x20/0x30 [ 1123.613655] nvmet():1 [ 1123.616348] __sbq_wake_up+0x13c/0x1f0 [ 1123.619562] nvme_fabrics():1 [ 1123.621908] sbitmap_queue_clear+0x74/0xc0 [ 1123.625643] nvme_loop():1 [ 1123.628596] blk_mq_put_tag+0x44/0x70 [ 1123.632678] null_blk():1 [ 1123.635371] __blk_mq_free_request+0x88/0xf0 [ 1123.639019] null_blk():1 [ 1123.641626] blk_mq_free_request+0xc0/0x160 [ 1123.645881] null_blk():1 [ 1123.648488] bfq_insert_request.isra.0+0x204/0x270 [ 1123.652656] null_blk():1 [ 1123.655263] bfq_insert_requests+0x70/0x90 [ 1123.660039] null_blk():1 [ 1123.662646] blk_mq_sched_insert_requests+0xac/0x250 [ 1123.666728] null_blk():1 [ 1123.669334] blk_mq_flush_plug_list+0x110/0x2cc [ 1123.674284] null_blk():1 [ 1123.676890] __blk_flush_plug+0x100/0x140 [ 1123.681406] scsi_debug():1 [ 1123.684013] blk_finish_plug+0x40/0x25c [ 1123.688008] scsi_debug():1 [ 1123.690788] __iomap_dio_rw+0x1b4/0x500 [ 1123.694609] ipmi_si():1 [ 1123.697389] iomap_dio_rw+0x18/0x50 [ 1123.701211] ipmi_si():1 [ 1123.703730] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1123.707205] [last unloaded: nvmet] [ 1123.709724] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1123.715022] [ 1123.718496] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1123.723186] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1123.724664] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1123.729700] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1123.737515] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1123.742639] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1123.751930] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1123.757055] pc : __sbq_wake_up+0x58/0x1f0 [ 1123.764002] process_one_work+0x1ec/0x470 [ 1123.769386] lr : sbitmap_queue_clear+0x74/0xc0 [ 1123.773381] worker_thread+0x74/0x410 [ 1123.777376] sp : ffff8000088438b0 [ 1123.781806] kthread+0xec/0x100 [ 1123.785454] x29: ffff8000088438b0 [ 1123.788755] ret_from_fork+0x10/0x20 [ 1123.791882] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1123.804041] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1123.811163] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1123.818285] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1123.825407] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1123.832529] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1123.839651] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1123.846772] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1123.853894] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1123.861016] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1123.868138] Call trace: [ 1123.870571] __sbq_wake_up+0x58/0x1f0 [ 1123.874220] sbitmap_queue_clear+0x74/0xc0 [ 1123.878303] blk_mq_put_tag+0x44/0x70 [ 1123.881952] __blk_mq_free_request+0x88/0xf0 [ 1123.886209] blk_mq_free_request+0xc0/0x160 [ 1123.890379] __blk_mq_end_request+0x78/0x150 [ 1123.894636] scsi_end_request+0x114/0x260 [ 1123.898633] scsi_io_completion+0x64/0x174 [ 1123.902716] scsi_finish_command+0xe0/0x120 [ 1123.906886] scsi_complete+0x8c/0x1c4 [ 1123.910535] blk_complete_reqs+0x5c/0x74 [ 1123.914445] blk_done_softirq+0x38/0x4c [ 1123.918267] __do_softirq+0x168/0x418 [ 1123.921916] __irq_exit_rcu+0x120/0x170 [ 1123.925739] irq_exit_rcu+0x18/0x24 [ 1123.929215] el1_interrupt+0x38/0x70 [ 1123.932777] el1h_64_irq_handler+0x18/0x24 [ 1123.936860] el1h_64_irq+0x68/0x6c [ 1123.940249] cpuidle_enter_state+0xc4/0x4e0 [ 1123.944419] cpuidle_enter+0x40/0x60 [ 1123.947981] cpuidle_idle_call+0x118/0x1b0 [ 1123.952064] do_idle+0xac/0x100 [ 1123.955193] cpu_startup_entry+0x30/0x34 [ 1123.959102] secondary_start_kernel+0xd8/0xf0 [ 1123.963446] __secondary_switched+0xb0/0xb4 [ 1137.647394] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 68669 jiffies s: 2093 root: 0x11/. [ 1137.659046] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1137.668254] Task dump for CPU 8: [ 1137.671468] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1137.681370] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1137.687541] Call trace: [ 1137.689974] __switch_to+0xd4/0x100 [ 1137.693452] 0xffff6c0200000080 [ 1137.696580] Task dump for CPU 68: [ 1137.699883] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1137.709785] Call trace: [ 1137.712218] __switch_to+0xd4/0x100 [ 1137.715694] 0x0 [ 1147.107319] watchdog: BUG: soft lockup - CPU#8 stuck for 674s! [kworker/8:12:19399] [ 1147.114962] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1147.114978] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1147.202766] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1147.211016] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1147.217318] watchdog: BUG: soft lockup - CPU#68 stuck for 674s! [swapper/68:0] [ 1147.220307] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1147.227514] Modules linked in: nvme_loop [ 1147.233680] [ 1147.233680] nvme_fabrics [ 1147.237589] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1147.239067] nvmet [ 1147.241674] pc : __sbq_wake_up+0x58/0x1f0 [ 1147.248620] nls_utf8 [ 1147.250619] lr : sbitmap_queue_clear+0x74/0xc0 [ 1147.254614] hfsplus [ 1147.256873] sp : ffff80000d79b280 [ 1147.261303] hfs [ 1147.263475] x29: ffff80000d79b280 [ 1147.266776] isofs [ 1147.268602] x28: ffffd5becb6aec50 [ 1147.271903] nvme_core [ 1147.273901] x27: ffffd5beca4f7000 [ 1147.277289] nvme_common [ 1147.279635] [ 1147.283023] loop [ 1147.285544] x26: ffffd5becaf68a98 [ 1147.287021] tls [ 1147.288933] x25: ffffd5becb6a60c0 [ 1147.292234] rfkill [ 1147.294060] x24: fffffbffef180dc0 [ 1147.297447] sunrpc [ 1147.299533] [ 1147.302920] vfat [ 1147.305006] x23: ffff07ff81d381b0 [ 1147.306484] fat [ 1147.308396] x22: ffff80000d79b30c [ 1147.311696] acpi_ipmi [ 1147.313521] x21: ffff07ff84e93c00 [ 1147.316909] ipmi_ssif [ 1147.319255] [ 1147.322642] cppc_cpufreq [ 1147.324988] x20: 00000000000001c0 [ 1147.326466] ipmi_devintf [ 1147.329073] x19: ffff07ff91ccdb10 [ 1147.332373] igb [ 1147.334980] x18: ffffffffffffffff [ 1147.338367] ipmi_msghandler [ 1147.340193] [ 1147.343580] arm_cmn [ 1147.346448] x17: ffff32506c412000 [ 1147.347926] arm_dmc620_pmu [ 1147.350098] x16: ffff800008044000 [ 1147.353399] arm_spe_pmu [ 1147.356179] x15: 0000000000004000 [ 1147.359566] arm_dsu_pmu [ 1147.362086] [ 1147.365473] acpi_tad [ 1147.367993] x14: 010320bb28ec0000 [ 1147.369471] fuse [ 1147.371730] x13: 0000000000000087 [ 1147.375030] zram [ 1147.376942] x12: 071c71c71c71c71c [ 1147.380330] xfs [ 1147.382242] [ 1147.385630] crct10dif_ce [ 1147.387455] x11: 0000000000000040 [ 1147.388933] polyval_ce [ 1147.391540] x10: ffff07ff801a9c70 [ 1147.394841] ast [ 1147.397274] x9 : ffffd5bec9545fd8 [ 1147.400661] polyval_generic [ 1147.402486] [ 1147.405874] drm_vram_helper [ 1147.408741] x8 : ffff07ff80404b70 [ 1147.410221] ghash_ce [ 1147.413087] x7 : ffff07ff84e93dc8 [ 1147.416388] drm_ttm_helper [ 1147.418647] x6 : 0000000000000008 [ 1147.422035] sbsa_gwdt [ 1147.424817] [ 1147.428204] ttm [ 1147.430551] x5 : ffff80000d043790 [ 1147.432028] xgene_hwmon [ 1147.433854] x4 : ffff07ff84e93dc0 [ 1147.437154] Unloaded tainted modules: [ 1147.439674] x3 : 0000000000000008 [ 1147.443061] nvmet():1 [ 1147.446709] [ 1147.450097] nvme_fabrics():1 [ 1147.452443] x2 : 0000000000000007 [ 1147.453921] nvme_loop():1 [ 1147.456874] x1 : ffff80000d79b30c [ 1147.460175] nvmet():1 [ 1147.462868] x0 : ffff07ff84e93dd0 [ 1147.466256] nvme_fabrics():1 [ 1147.468602] [ 1147.471989] nvme_loop():1 [ 1147.474943] Call trace: [ 1147.476421] nvmet():1 [ 1147.479114] __sbq_wake_up+0x58/0x1f0 [ 1147.481546] nvme_fabrics():1 [ 1147.483892] sbitmap_queue_clear+0x74/0xc0 [ 1147.487540] nvme_loop():1 [ 1147.490494] blk_mq_put_tag+0x44/0x70 [ 1147.494576] nvmet():1 [ 1147.497269] __blk_mq_free_request+0x88/0xf0 [ 1147.500917] nvme_fabrics():1 [ 1147.503263] blk_mq_free_request+0xc0/0x160 [ 1147.507519] nvme_loop():1 [ 1147.510472] __blk_mq_end_request+0x78/0x150 [ 1147.514641] nvmet():1 [ 1147.517334] scsi_end_request+0x114/0x260 [ 1147.521590] nvme_fabrics():1 [ 1147.523936] scsi_io_completion+0x64/0x174 [ 1147.527932] nvme_loop():1 [ 1147.530886] scsi_finish_command+0xe0/0x120 [ 1147.534968] nvmet():1 [ 1147.537661] scsi_complete+0x8c/0x1c4 [ 1147.541830] nvme_fabrics():1 [ 1147.544176] blk_complete_reqs+0x5c/0x74 [ 1147.547824] nvme_loop():1 [ 1147.550777] blk_done_softirq+0x38/0x4c [ 1147.554686] nvmet():1 [ 1147.557379] __do_softirq+0x168/0x418 [ 1147.561201] nvme_fabrics():1 [ 1147.563547] __irq_exit_rcu+0x120/0x170 [ 1147.567195] nvme_loop():1 [ 1147.570149] irq_exit_rcu+0x18/0x24 [ 1147.573970] nvmet():1 [ 1147.576663] el1_interrupt+0x38/0x70 [ 1147.580138] nvme_fabrics():1 [ 1147.582484] el1h_64_irq_handler+0x18/0x24 [ 1147.586045] nvme_loop():1 [ 1147.588998] el1h_64_irq+0x68/0x6c [ 1147.593080] nvmet():1 [ 1147.595774] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1147.599162] nvme_fabrics():1 [ 1147.601508] __wake_up_common_lock+0x84/0xac [ 1147.606284] nvme_loop():1 [ 1147.609237] __wake_up+0x20/0x30 [ 1147.613493] nvmet():1 [ 1147.616187] __sbq_wake_up+0x13c/0x1f0 [ 1147.619401] nvme_fabrics():1 [ 1147.621747] sbitmap_queue_clear+0x74/0xc0 [ 1147.625482] nvme_loop():1 [ 1147.628435] blk_mq_put_tag+0x44/0x70 [ 1147.632517] null_blk():1 [ 1147.635210] __blk_mq_free_request+0x88/0xf0 [ 1147.638858] null_blk():1 [ 1147.641464] blk_mq_free_request+0xc0/0x160 [ 1147.645720] null_blk():1 [ 1147.648326] bfq_insert_request.isra.0+0x204/0x270 [ 1147.652495] null_blk():1 [ 1147.655102] bfq_insert_requests+0x70/0x90 [ 1147.659878] null_blk():1 [ 1147.662484] blk_mq_sched_insert_requests+0xac/0x250 [ 1147.666566] null_blk():1 [ 1147.669173] blk_mq_flush_plug_list+0x110/0x2cc [ 1147.674123] null_blk():1 [ 1147.676730] __blk_flush_plug+0x100/0x140 [ 1147.681246] scsi_debug():1 [ 1147.683852] blk_finish_plug+0x40/0x25c [ 1147.687847] scsi_debug():1 [ 1147.690627] __iomap_dio_rw+0x1b4/0x500 [ 1147.694449] ipmi_si():1 [ 1147.697229] iomap_dio_rw+0x18/0x50 [ 1147.701050] ipmi_si():1 [ 1147.703570] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1147.707044] [last unloaded: nvmet] [ 1147.709564] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1147.714861] [ 1147.718335] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1147.723025] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1147.724503] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1147.729540] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1147.737355] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1147.742479] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1147.751769] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1147.756893] pc : __sbq_wake_up+0x84/0x1f0 [ 1147.763839] process_one_work+0x1ec/0x470 [ 1147.769223] lr : sbitmap_queue_clear+0x74/0xc0 [ 1147.773218] worker_thread+0x74/0x410 [ 1147.777214] sp : ffff8000088438b0 [ 1147.781643] kthread+0xec/0x100 [ 1147.785291] x29: ffff8000088438b0 [ 1147.788592] ret_from_fork+0x10/0x20 [ 1147.791719] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1147.803879] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1147.811001] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1147.818123] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1147.825245] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1147.832366] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1147.839488] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1147.846610] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1147.853733] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1147.860856] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1147.867978] Call trace: [ 1147.870411] __sbq_wake_up+0x84/0x1f0 [ 1147.874060] sbitmap_queue_clear+0x74/0xc0 [ 1147.878143] blk_mq_put_tag+0x44/0x70 [ 1147.881793] __blk_mq_free_request+0x88/0xf0 [ 1147.886050] blk_mq_free_request+0xc0/0x160 [ 1147.890219] __blk_mq_end_request+0x78/0x150 [ 1147.894476] scsi_end_request+0x114/0x260 [ 1147.898472] scsi_io_completion+0x64/0x174 [ 1147.902556] scsi_finish_command+0xe0/0x120 [ 1147.906726] scsi_complete+0x8c/0x1c4 [ 1147.910375] blk_complete_reqs+0x5c/0x74 [ 1147.914285] blk_done_softirq+0x38/0x4c [ 1147.918107] __do_softirq+0x168/0x418 [ 1147.921756] __irq_exit_rcu+0x120/0x170 [ 1147.925578] irq_exit_rcu+0x18/0x24 [ 1147.929054] el1_interrupt+0x38/0x70 [ 1147.932616] el1h_64_irq_handler+0x18/0x24 [ 1147.936699] el1h_64_irq+0x68/0x6c [ 1147.940088] cpuidle_enter_state+0xc4/0x4e0 [ 1147.944258] cpuidle_enter+0x40/0x60 [ 1147.947821] cpuidle_idle_call+0x118/0x1b0 [ 1147.951904] do_idle+0xac/0x100 [ 1147.955033] cpu_startup_entry+0x30/0x34 [ 1147.958944] secondary_start_kernel+0xd8/0xf0 [ 1147.963288] __secondary_switched+0xb0/0xb4 [ 1171.107147] watchdog: BUG: soft lockup - CPU#8 stuck for 696s! [kworker/8:12:19399] [ 1171.114790] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1171.114807] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1171.202594] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1171.210844] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1171.217146] watchdog: BUG: soft lockup - CPU#68 stuck for 696s! [swapper/68:0] [ 1171.220135] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1171.227343] Modules linked in: nvme_loop [ 1171.233509] [ 1171.233509] nvme_fabrics nvmet [ 1171.237419] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1171.238896] nls_utf8 [ 1171.242024] pc : __sbq_wake_up+0x58/0x1f0 [ 1171.248971] hfsplus [ 1171.251230] lr : sbitmap_queue_clear+0x74/0xc0 [ 1171.255226] hfs [ 1171.257398] sp : ffff80000d79b280 [ 1171.261827] isofs [ 1171.263653] x29: ffff80000d79b280 [ 1171.266954] nvme_core [ 1171.268953] x28: ffffd5becb6aec50 [ 1171.272254] nvme_common [ 1171.274599] x27: ffffd5beca4f7000 [ 1171.277987] loop [ 1171.280507] [ 1171.283894] tls [ 1171.285806] x26: ffffd5becaf68a98 [ 1171.287284] rfkill [ 1171.289109] x25: ffffd5becb6a60c0 [ 1171.292410] sunrpc [ 1171.294495] x24: fffffbffef180dc0 [ 1171.297883] vfat [ 1171.299968] [ 1171.303356] fat [ 1171.305268] x23: ffff07ff81d381b0 [ 1171.306746] acpi_ipmi [ 1171.308571] x22: ffff80000d79b30c [ 1171.311871] ipmi_ssif [ 1171.314217] x21: ffff07ff84e93c00 [ 1171.317605] cppc_cpufreq [ 1171.319951] [ 1171.323338] ipmi_devintf [ 1171.325945] x20: 00000000000001c0 [ 1171.327423] igb [ 1171.330029] x19: ffff07ff91ccdb10 [ 1171.333330] ipmi_msghandler [ 1171.335155] x18: ffffffffffffffff [ 1171.338543] arm_cmn [ 1171.341411] [ 1171.344798] arm_dmc620_pmu [ 1171.346971] x17: ffff32506c412000 [ 1171.348448] arm_spe_pmu [ 1171.351229] x16: ffff800008044000 [ 1171.354530] arm_dsu_pmu [ 1171.357049] x15: 0000000000004000 [ 1171.360437] acpi_tad [ 1171.362956] [ 1171.366344] fuse [ 1171.368603] x14: 010320bb28ec0000 [ 1171.370081] zram [ 1171.371993] x13: 0000000000000087 [ 1171.375294] xfs [ 1171.377206] x12: 071c71c71c71c71c [ 1171.380593] crct10dif_ce [ 1171.382419] [ 1171.385806] polyval_ce [ 1171.388412] x11: 0000000000000040 [ 1171.389890] ast [ 1171.392323] x10: ffff07ff801a9c70 [ 1171.395624] polyval_generic [ 1171.397449] x9 : ffffd5bec9545fd8 [ 1171.400836] drm_vram_helper [ 1171.403703] [ 1171.407091] ghash_ce [ 1171.409957] x8 : ffff07ff80404b70 [ 1171.411436] drm_ttm_helper [ 1171.413696] x7 : ffff07ff84e93dc8 [ 1171.416997] sbsa_gwdt [ 1171.419777] x6 : 0000000000000008 [ 1171.423164] ttm [ 1171.425510] [ 1171.428898] xgene_hwmon [ 1171.430723] x5 : ffff80000d043790 [ 1171.432201] Unloaded tainted modules: [ 1171.434720] x4 : ffff07ff84e93dc0 [ 1171.438021] nvmet():1 [ 1171.441669] x3 : 0000000000000008 [ 1171.445057] nvme_fabrics():1 [ 1171.447402] [ 1171.450790] nvme_loop():1 [ 1171.453744] x2 : 0000000000000007 [ 1171.455222] nvmet():1 [ 1171.457915] x1 : ffff80000d79b30c [ 1171.461215] nvme_fabrics():1 [ 1171.463561] x0 : ffff07ff84e93dd0 [ 1171.466949] nvme_loop():1 [ 1171.469902] [ 1171.473290] nvmet():1 [ 1171.475983] Call trace: [ 1171.477461] nvme_fabrics():1 [ 1171.479807] __sbq_wake_up+0x58/0x1f0 [ 1171.482240] nvme_loop():1 [ 1171.485194] sbitmap_queue_clear+0x74/0xc0 [ 1171.488842] nvmet():1 [ 1171.491535] blk_mq_put_tag+0x44/0x70 [ 1171.495617] nvme_fabrics():1 [ 1171.497963] __blk_mq_free_request+0x88/0xf0 [ 1171.501611] nvme_loop():1 [ 1171.504564] blk_mq_free_request+0xc0/0x160 [ 1171.508820] nvmet():1 [ 1171.511513] __blk_mq_end_request+0x78/0x150 [ 1171.515682] nvme_fabrics():1 [ 1171.518028] scsi_end_request+0x114/0x260 [ 1171.522283] nvme_loop():1 [ 1171.525237] scsi_io_completion+0x64/0x174 [ 1171.529232] nvmet():1 [ 1171.531925] scsi_finish_command+0xe0/0x120 [ 1171.536007] nvme_fabrics():1 [ 1171.538353] scsi_complete+0x8c/0x1c4 [ 1171.542522] nvme_loop():1 [ 1171.545476] blk_complete_reqs+0x5c/0x74 [ 1171.549124] nvmet():1 [ 1171.551817] blk_done_softirq+0x38/0x4c [ 1171.555725] nvme_fabrics():1 [ 1171.558071] __do_softirq+0x168/0x418 [ 1171.561893] nvme_loop():1 [ 1171.564846] __irq_exit_rcu+0x120/0x170 [ 1171.568494] nvmet():1 [ 1171.571187] irq_exit_rcu+0x18/0x24 [ 1171.575009] nvme_fabrics():1 [ 1171.577355] el1_interrupt+0x38/0x70 [ 1171.580830] nvme_loop():1 [ 1171.583783] el1h_64_irq_handler+0x18/0x24 [ 1171.587345] nvmet():1 [ 1171.590038] el1h_64_irq+0x68/0x6c [ 1171.594122] nvme_fabrics():1 [ 1171.596468] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1171.599855] nvme_loop():1 [ 1171.602809] __wake_up_common_lock+0x84/0xac [ 1171.607585] nvmet():1 [ 1171.610278] __wake_up+0x20/0x30 [ 1171.614534] nvme_fabrics():1 [ 1171.616880] __sbq_wake_up+0x13c/0x1f0 [ 1171.620094] nvme_loop():1 [ 1171.623047] sbitmap_queue_clear+0x74/0xc0 [ 1171.626782] null_blk():1 [ 1171.629475] blk_mq_put_tag+0x44/0x70 [ 1171.633557] null_blk():1 [ 1171.636164] __blk_mq_free_request+0x88/0xf0 [ 1171.639812] null_blk():1 [ 1171.642418] blk_mq_free_request+0xc0/0x160 [ 1171.646674] null_blk():1 [ 1171.649280] bfq_insert_request.isra.0+0x204/0x270 [ 1171.653449] null_blk():1 [ 1171.656055] bfq_insert_requests+0x70/0x90 [ 1171.660832] null_blk():1 [ 1171.663439] blk_mq_sched_insert_requests+0xac/0x250 [ 1171.667521] null_blk():1 [ 1171.670127] blk_mq_flush_plug_list+0x110/0x2cc [ 1171.675077] scsi_debug():1 [ 1171.677684] __blk_flush_plug+0x100/0x140 [ 1171.682200] scsi_debug():1 [ 1171.684980] blk_finish_plug+0x40/0x25c [ 1171.688975] ipmi_si():1 [ 1171.691755] __iomap_dio_rw+0x1b4/0x500 [ 1171.695577] ipmi_si():1 [ 1171.698096] iomap_dio_rw+0x18/0x50 [ 1171.701918] [last unloaded: nvmet] [ 1171.704438] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1171.707912] [ 1171.711386] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1171.716684] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1171.718162] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1171.722851] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1171.730666] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1171.735703] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1171.744993] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1171.750117] pc : __sbq_wake_up+0x58/0x1f0 [ 1171.757063] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1171.762187] lr : sbitmap_queue_clear+0x74/0xc0 [ 1171.766182] process_one_work+0x1ec/0x470 [ 1171.771566] sp : ffff8000088438b0 [ 1171.775996] worker_thread+0x74/0x410 [ 1171.779991] x29: ffff8000088438b0 [ 1171.783292] kthread+0xec/0x100 [ 1171.786940] x28: ffffd5becb6aec50 [ 1171.790241] ret_from_fork+0x10/0x20 [ 1171.793368] x27: ffffd5beca4f7000 [ 1171.803704] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1171.810826] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1171.817948] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1171.825070] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1171.832192] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1171.839313] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1171.846435] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1171.853557] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1171.860679] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1171.867801] Call trace: [ 1171.870233] __sbq_wake_up+0x58/0x1f0 [ 1171.873883] sbitmap_queue_clear+0x74/0xc0 [ 1171.877966] blk_mq_put_tag+0x44/0x70 [ 1171.881615] __blk_mq_free_request+0x88/0xf0 [ 1171.885872] blk_mq_free_request+0xc0/0x160 [ 1171.890042] __blk_mq_end_request+0x78/0x150 [ 1171.894299] scsi_end_request+0x114/0x260 [ 1171.898295] scsi_io_completion+0x64/0x174 [ 1171.902378] scsi_finish_command+0xe0/0x120 [ 1171.906548] scsi_complete+0x8c/0x1c4 [ 1171.910197] blk_complete_reqs+0x5c/0x74 [ 1171.914107] blk_done_softirq+0x38/0x4c [ 1171.917930] __do_softirq+0x168/0x418 [ 1171.921578] __irq_exit_rcu+0x120/0x170 [ 1171.925401] irq_exit_rcu+0x18/0x24 [ 1171.928876] el1_interrupt+0x38/0x70 [ 1171.932439] el1h_64_irq_handler+0x18/0x24 [ 1171.936522] el1h_64_irq+0x68/0x6c [ 1171.939910] cpuidle_enter_state+0xc4/0x4e0 [ 1171.944080] cpuidle_enter+0x40/0x60 [ 1171.947643] cpuidle_idle_call+0x118/0x1b0 [ 1171.951726] do_idle+0xac/0x100 [ 1171.954855] cpu_startup_entry+0x30/0x34 [ 1171.958764] secondary_start_kernel+0xd8/0xf0 [ 1171.963108] __secondary_switched+0xb0/0xb4 [ 1195.106975] watchdog: BUG: soft lockup - CPU#8 stuck for 719s! [kworker/8:12:19399] [ 1195.114618] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1195.114635] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1195.202422] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1195.210671] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1195.216974] watchdog: BUG: soft lockup - CPU#68 stuck for 719s! [swapper/68:0] [ 1195.219962] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1195.227169] Modules linked in: nvme_loop [ 1195.233335] [ 1195.233336] nvme_fabrics [ 1195.237244] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1195.238722] nvmet [ 1195.241329] pc : __sbq_wake_up+0x58/0x1f0 [ 1195.248275] nls_utf8 [ 1195.250274] lr : sbitmap_queue_clear+0x74/0xc0 [ 1195.254269] hfsplus [ 1195.256529] sp : ffff80000d79b280 [ 1195.260958] hfs [ 1195.263131] x29: ffff80000d79b280 [ 1195.266431] isofs [ 1195.268256] x28: ffffd5becb6aec50 [ 1195.271557] nvme_core [ 1195.273556] x27: ffffd5beca4f7000 [ 1195.276944] nvme_common [ 1195.279290] [ 1195.282677] loop [ 1195.285197] x26: ffffd5becaf68a98 [ 1195.286675] tls [ 1195.288587] x25: ffffd5becb6a60c0 [ 1195.291888] rfkill [ 1195.293713] x24: fffffbffef180dc0 [ 1195.297100] sunrpc [ 1195.299186] [ 1195.302574] vfat [ 1195.304660] x23: ffff07ff81d381b0 [ 1195.306138] fat [ 1195.308052] x22: ffff80000d79b30c [ 1195.311353] acpi_ipmi [ 1195.313178] x21: ffff07ff84e93c00 [ 1195.316565] ipmi_ssif [ 1195.318911] [ 1195.322299] cppc_cpufreq [ 1195.324645] x20: 00000000000001c0 [ 1195.326122] ipmi_devintf [ 1195.328729] x19: ffff07ff91ccdb10 [ 1195.332030] igb [ 1195.334636] x18: ffffffffffffffff [ 1195.338023] ipmi_msghandler [ 1195.339849] [ 1195.343236] arm_cmn [ 1195.346103] x17: ffff32506c412000 [ 1195.347581] arm_dmc620_pmu [ 1195.349753] x16: ffff800008044000 [ 1195.353054] arm_spe_pmu [ 1195.355834] x15: 0000000000004000 [ 1195.359221] arm_dsu_pmu [ 1195.361741] [ 1195.365128] acpi_tad [ 1195.367648] x14: 010320bb28ec0000 [ 1195.369126] fuse [ 1195.371385] x13: 0000000000000087 [ 1195.374686] zram [ 1195.376597] x12: 071c71c71c71c71c [ 1195.379985] xfs [ 1195.381897] [ 1195.385285] crct10dif_ce [ 1195.387110] x11: 0000000000000040 [ 1195.388588] polyval_ce [ 1195.391195] x10: ffff07ff801a9c70 [ 1195.394496] ast [ 1195.396928] x9 : ffffd5bec9545fd8 [ 1195.400316] polyval_generic [ 1195.402141] [ 1195.405530] drm_vram_helper [ 1195.408396] x8 : ffff07ff80404b70 [ 1195.409874] ghash_ce [ 1195.412741] x7 : ffff07ff84e93dc8 [ 1195.416042] drm_ttm_helper [ 1195.418301] x6 : 0000000000000008 [ 1195.421688] sbsa_gwdt [ 1195.424468] [ 1195.427856] ttm [ 1195.430202] x5 : ffff80000d043790 [ 1195.431680] xgene_hwmon [ 1195.433505] x4 : ffff07ff84e93dc0 [ 1195.436806] Unloaded tainted modules: [ 1195.439325] x3 : 0000000000000008 [ 1195.442713] nvmet():1 [ 1195.446361] [ 1195.449748] nvme_fabrics():1 [ 1195.452094] x2 : 0000000000000007 [ 1195.453572] nvme_loop():1 [ 1195.456526] x1 : ffff80000d79b30c [ 1195.459826] nvmet():1 [ 1195.462519] x0 : ffff07ff84e93dd0 [ 1195.465907] nvme_fabrics():1 [ 1195.468253] [ 1195.471640] nvme_loop():1 [ 1195.474595] Call trace: [ 1195.476073] nvmet():1 [ 1195.478767] __sbq_wake_up+0x58/0x1f0 [ 1195.481199] nvme_fabrics():1 [ 1195.483545] sbitmap_queue_clear+0x74/0xc0 [ 1195.487193] nvme_loop():1 [ 1195.490147] blk_mq_put_tag+0x44/0x70 [ 1195.494229] nvmet():1 [ 1195.496922] __blk_mq_free_request+0x88/0xf0 [ 1195.500570] nvme_fabrics():1 [ 1195.502916] blk_mq_free_request+0xc0/0x160 [ 1195.507172] nvme_loop():1 [ 1195.510125] __blk_mq_end_request+0x78/0x150 [ 1195.514294] nvmet():1 [ 1195.516987] scsi_end_request+0x114/0x260 [ 1195.521243] nvme_fabrics():1 [ 1195.523589] scsi_io_completion+0x64/0x174 [ 1195.527584] nvme_loop():1 [ 1195.530538] scsi_finish_command+0xe0/0x120 [ 1195.534620] nvmet():1 [ 1195.537313] scsi_complete+0x8c/0x1c4 [ 1195.541482] nvme_fabrics():1 [ 1195.543828] blk_complete_reqs+0x5c/0x74 [ 1195.547476] nvme_loop():1 [ 1195.550429] blk_done_softirq+0x38/0x4c [ 1195.554338] nvmet():1 [ 1195.557031] __do_softirq+0x168/0x418 [ 1195.560852] nvme_fabrics():1 [ 1195.563198] __irq_exit_rcu+0x120/0x170 [ 1195.566846] nvme_loop():1 [ 1195.569800] irq_exit_rcu+0x18/0x24 [ 1195.573622] nvmet():1 [ 1195.576315] el1_interrupt+0x38/0x70 [ 1195.579789] nvme_fabrics():1 [ 1195.582135] el1h_64_irq_handler+0x18/0x24 [ 1195.585696] nvme_loop():1 [ 1195.588650] el1h_64_irq+0x68/0x6c [ 1195.592732] nvmet():1 [ 1195.595425] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1195.598813] nvme_fabrics():1 [ 1195.601159] __wake_up_common_lock+0x84/0xac [ 1195.605935] nvme_loop():1 [ 1195.608889] __wake_up+0x20/0x30 [ 1195.613144] nvmet():1 [ 1195.615838] __sbq_wake_up+0x13c/0x1f0 [ 1195.619052] nvme_fabrics():1 [ 1195.621397] sbitmap_queue_clear+0x74/0xc0 [ 1195.625132] nvme_loop():1 [ 1195.628086] blk_mq_put_tag+0x44/0x70 [ 1195.632169] null_blk():1 [ 1195.634863] __blk_mq_free_request+0x88/0xf0 [ 1195.638511] null_blk():1 [ 1195.641117] blk_mq_free_request+0xc0/0x160 [ 1195.645372] null_blk():1 [ 1195.647979] bfq_insert_request.isra.0+0x204/0x270 [ 1195.652148] null_blk():1 [ 1195.654755] bfq_insert_requests+0x70/0x90 [ 1195.659531] null_blk():1 [ 1195.662138] blk_mq_sched_insert_requests+0xac/0x250 [ 1195.666222] null_blk():1 [ 1195.668828] blk_mq_flush_plug_list+0x110/0x2cc [ 1195.673778] null_blk():1 [ 1195.676385] __blk_flush_plug+0x100/0x140 [ 1195.680901] scsi_debug():1 [ 1195.683507] blk_finish_plug+0x40/0x25c [ 1195.687502] scsi_debug():1 [ 1195.690282] __iomap_dio_rw+0x1b4/0x500 [ 1195.694104] ipmi_si():1 [ 1195.696884] iomap_dio_rw+0x18/0x50 [ 1195.700705] ipmi_si():1 [ 1195.703225] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1195.706699] [last unloaded: nvmet] [ 1195.709219] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1195.714517] [ 1195.717991] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1195.722681] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1195.724159] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1195.729195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1195.737010] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1195.742134] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1195.751424] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1195.756548] pc : __sbq_wake_up+0x4/0x1f0 [ 1195.763494] process_one_work+0x1ec/0x470 [ 1195.768878] lr : sbitmap_queue_clear+0x74/0xc0 [ 1195.772787] worker_thread+0x74/0x410 [ 1195.776782] sp : ffff800008843900 [ 1195.781211] kthread+0xec/0x100 [ 1195.784859] x29: ffff800008843900 [ 1195.788160] ret_from_fork+0x10/0x20 [ 1195.791287] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1195.803447] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1195.810569] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1195.817691] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1195.824812] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1195.831934] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1195.839058] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1195.846179] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1195.853301] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1195.860423] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1195.867545] Call trace: [ 1195.869978] __sbq_wake_up+0x4/0x1f0 [ 1195.873540] blk_mq_put_tag+0x44/0x70 [ 1195.877190] __blk_mq_free_request+0x88/0xf0 [ 1195.881446] blk_mq_free_request+0xc0/0x160 [ 1195.885616] __blk_mq_end_request+0x78/0x150 [ 1195.889873] scsi_end_request+0x114/0x260 [ 1195.893869] scsi_io_completion+0x64/0x174 [ 1195.897952] scsi_finish_command+0xe0/0x120 [ 1195.902123] scsi_complete+0x8c/0x1c4 [ 1195.905772] blk_complete_reqs+0x5c/0x74 [ 1195.909681] blk_done_softirq+0x38/0x4c [ 1195.913504] __do_softirq+0x168/0x418 [ 1195.917153] __irq_exit_rcu+0x120/0x170 [ 1195.920975] irq_exit_rcu+0x18/0x24 [ 1195.924451] el1_interrupt+0x38/0x70 [ 1195.928013] el1h_64_irq_handler+0x18/0x24 [ 1195.932096] el1h_64_irq+0x68/0x6c [ 1195.935484] cpuidle_enter_state+0xc4/0x4e0 [ 1195.939654] cpuidle_enter+0x40/0x60 [ 1195.943216] cpuidle_idle_call+0x118/0x1b0 [ 1195.947300] do_idle+0xac/0x100 [ 1195.950429] cpu_startup_entry+0x30/0x34 [ 1195.954338] secondary_start_kernel+0xd8/0xf0 [ 1195.958682] __secondary_switched+0xb0/0xb4 [ 1204.206920] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 75325 jiffies s: 2093 root: 0x11/. [ 1204.218572] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1204.227781] Task dump for CPU 8: [ 1204.230996] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1204.240897] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1204.247067] Call trace: [ 1204.249500] __switch_to+0xd4/0x100 [ 1204.252977] 0xffff6c0200000080 [ 1204.256106] Task dump for CPU 68: [ 1204.259409] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1204.269310] Call trace: [ 1204.271743] __switch_to+0xd4/0x100 [ 1204.275219] 0x0 [ 1206.246896] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1206.252628] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=38750 [ 1206.261662] (t=78136 jiffies g=36445 q=23238 ncpus=128) [ 1206.266961] Task dump for CPU 8: [ 1206.270175] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1206.280075] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1206.286243] Call trace: [ 1206.288676] dump_backtrace+0xe8/0x130 [ 1206.292412] show_stack+0x20/0x60 [ 1206.295714] sched_show_task+0x160/0x190 [ 1206.299624] dump_cpu_task+0x4c/0x78 [ 1206.303186] rcu_dump_cpu_stacks+0xc4/0x150 [ 1206.307357] print_cpu_stall+0x180/0x210 [ 1206.311267] check_cpu_stall+0x20c/0x2b4 [ 1206.315176] rcu_pending+0x4c/0x190 [ 1206.318652] rcu_sched_clock_irq+0x80/0x204 [ 1206.322822] update_process_times+0x7c/0xc0 [ 1206.326992] tick_sched_handle+0x38/0x74 [ 1206.330904] tick_sched_timer+0x60/0xc0 [ 1206.334727] __hrtimer_run_queues+0x170/0x300 [ 1206.339071] hrtimer_interrupt+0xfc/0x260 [ 1206.343067] arch_timer_handler_phys+0x34/0x50 [ 1206.347498] handle_percpu_devid_irq+0xac/0x240 [ 1206.352015] generic_handle_domain_irq+0x34/0x50 [ 1206.356620] gic_handle_irq+0x5c/0xe0 [ 1206.360269] call_on_irq_stack+0x2c/0x38 [ 1206.364179] do_interrupt_handler+0xdc/0xe0 [ 1206.368349] el1_interrupt+0x34/0x70 [ 1206.371912] el1h_64_irq_handler+0x18/0x24 [ 1206.375995] el1h_64_irq+0x68/0x6c [ 1206.379385] __sbq_wake_up+0x40/0x1f0 [ 1206.383035] sbitmap_queue_clear+0x74/0xc0 [ 1206.387118] blk_mq_put_tag+0x44/0x70 [ 1206.390767] __blk_mq_free_request+0x88/0xf0 [ 1206.395024] blk_mq_free_request+0xc0/0x160 [ 1206.399194] __blk_mq_end_request+0x78/0x150 [ 1206.403451] scsi_end_request+0x114/0x260 [ 1206.407447] scsi_io_completion+0x64/0x174 [ 1206.411530] scsi_finish_command+0xe0/0x120 [ 1206.415701] scsi_complete+0x8c/0x1c4 [ 1206.419350] blk_complete_reqs+0x5c/0x74 [ 1206.423259] blk_done_softirq+0x38/0x4c [ 1206.427082] __do_softirq+0x168/0x418 [ 1206.430730] __irq_exit_rcu+0x120/0x170 [ 1206.434553] irq_exit_rcu+0x18/0x24 [ 1206.438029] el1_interrupt+0x38/0x70 [ 1206.441591] el1h_64_irq_handler+0x18/0x24 [ 1206.445675] el1h_64_irq+0x68/0x6c [ 1206.449064] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1206.453841] __wake_up_common_lock+0x84/0xac [ 1206.458098] __wake_up+0x20/0x30 [ 1206.461315] __sbq_wake_up+0x13c/0x1f0 [ 1206.465051] sbitmap_queue_clear+0x74/0xc0 [ 1206.469135] blk_mq_put_tag+0x44/0x70 [ 1206.472784] __blk_mq_free_request+0x88/0xf0 [ 1206.477041] blk_mq_free_request+0xc0/0x160 [ 1206.481211] bfq_insert_request.isra.0+0x204/0x270 [ 1206.485989] bfq_insert_requests+0x70/0x90 [ 1206.490073] blk_mq_sched_insert_requests+0xac/0x250 [ 1206.495025] blk_mq_flush_plug_list+0x110/0x2cc [ 1206.499542] __blk_flush_plug+0x100/0x140 [ 1206.503539] blk_finish_plug+0x40/0x25c [ 1206.507361] __iomap_dio_rw+0x1b4/0x500 [ 1206.511184] iomap_dio_rw+0x18/0x50 [ 1206.514660] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1206.520011] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1206.524753] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1206.529797] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1206.534927] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1206.540057] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1206.545443] process_one_work+0x1ec/0x470 [ 1206.549439] worker_thread+0x74/0x410 [ 1206.553088] kthread+0xec/0x100 [ 1206.556217] ret_from_fork+0x10/0x20 [ 1206.559780] Task dump for CPU 68: [ 1206.563081] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1206.572981] Call trace: [ 1206.575415] __switch_to+0xd4/0x100 [ 1206.578891] 0x0 [ 1219.216802] watchdog: BUG: soft lockup - CPU#68 stuck for 741s! [swapper/68:0] [ 1219.224011] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1219.224027] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1219.311814] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1219.319629] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1219.328920] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1219.335868] pc : __sbq_wake_up+0x84/0x1f0 [ 1219.339865] lr : sbitmap_queue_clear+0x74/0xc0 [ 1219.344295] sp : ffff8000088438b0 [ 1219.347596] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1219.354718] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1219.361840] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1219.368963] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1219.376085] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1219.383207] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1219.390329] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1219.397451] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1219.404572] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1219.411694] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1219.418816] Call trace: [ 1219.421249] __sbq_wake_up+0x84/0x1f0 [ 1219.424898] sbitmap_queue_clear+0x74/0xc0 [ 1219.428982] blk_mq_put_tag+0x44/0x70 [ 1219.432631] __blk_mq_free_request+0x88/0xf0 [ 1219.436888] blk_mq_free_request+0xc0/0x160 [ 1219.441058] __blk_mq_end_request+0x78/0x150 [ 1219.445314] scsi_end_request+0x114/0x260 [ 1219.449311] scsi_io_completion+0x64/0x174 [ 1219.453394] scsi_finish_command+0xe0/0x120 [ 1219.457564] scsi_complete+0x8c/0x1c4 [ 1219.461213] blk_complete_reqs+0x5c/0x74 [ 1219.465123] blk_done_softirq+0x38/0x4c [ 1219.468946] __do_softirq+0x168/0x418 [ 1219.472594] __irq_exit_rcu+0x120/0x170 [ 1219.476417] irq_exit_rcu+0x18/0x24 [ 1219.479892] el1_interrupt+0x38/0x70 [ 1219.483454] el1h_64_irq_handler+0x18/0x24 [ 1219.487538] el1h_64_irq+0x68/0x6c [ 1219.490926] cpuidle_enter_state+0xc4/0x4e0 [ 1219.495096] cpuidle_enter+0x40/0x60 [ 1219.498658] cpuidle_idle_call+0x118/0x1b0 [ 1219.502743] do_idle+0xac/0x100 [ 1219.505872] cpu_startup_entry+0x30/0x34 [ 1219.509781] secondary_start_kernel+0xd8/0xf0 [ 1219.514125] __secondary_switched+0xb0/0xb4 [ 1231.106717] watchdog: BUG: soft lockup - CPU#8 stuck for 752s! [kworker/8:12:19399] [ 1231.114360] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1231.114377] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1231.202164] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1231.210414] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1231.219705] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1231.225872] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1231.232820] pc : __sbq_wake_up+0x58/0x1f0 [ 1231.236817] lr : sbitmap_queue_clear+0x74/0xc0 [ 1231.241247] sp : ffff80000d79b280 [ 1231.244548] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1231.251671] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1231.258793] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1231.265915] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1231.273037] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1231.280160] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1231.287282] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1231.294404] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1231.301525] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1231.308647] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1231.315769] Call trace: [ 1231.318202] __sbq_wake_up+0x58/0x1f0 [ 1231.321852] sbitmap_queue_clear+0x74/0xc0 [ 1231.325936] blk_mq_put_tag+0x44/0x70 [ 1231.329585] __blk_mq_free_request+0x88/0xf0 [ 1231.333842] blk_mq_free_request+0xc0/0x160 [ 1231.338012] __blk_mq_end_request+0x78/0x150 [ 1231.342269] scsi_end_request+0x114/0x260 [ 1231.346265] scsi_io_completion+0x64/0x174 [ 1231.350348] scsi_finish_command+0xe0/0x120 [ 1231.354519] scsi_complete+0x8c/0x1c4 [ 1231.358168] blk_complete_reqs+0x5c/0x74 [ 1231.362077] blk_done_softirq+0x38/0x4c [ 1231.365900] __do_softirq+0x168/0x418 [ 1231.369549] __irq_exit_rcu+0x120/0x170 [ 1231.373372] irq_exit_rcu+0x18/0x24 [ 1231.376847] el1_interrupt+0x38/0x70 [ 1231.380410] el1h_64_irq_handler+0x18/0x24 [ 1231.384493] el1h_64_irq+0x68/0x6c [ 1231.387882] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1231.392660] __wake_up_common_lock+0x84/0xac [ 1231.396918] __wake_up+0x20/0x30 [ 1231.400133] __sbq_wake_up+0x13c/0x1f0 [ 1231.403869] sbitmap_queue_clear+0x74/0xc0 [ 1231.407953] blk_mq_put_tag+0x44/0x70 [ 1231.411602] __blk_mq_free_request+0x88/0xf0 [ 1231.415859] blk_mq_free_request+0xc0/0x160 [ 1231.420029] bfq_insert_request.isra.0+0x204/0x270 [ 1231.424807] bfq_insert_requests+0x70/0x90 [ 1231.428891] blk_mq_sched_insert_requests+0xac/0x250 [ 1231.433844] blk_mq_flush_plug_list+0x110/0x2cc [ 1231.438362] __blk_flush_plug+0x100/0x140 [ 1231.442358] blk_finish_plug+0x40/0x25c [ 1231.446181] __iomap_dio_rw+0x1b4/0x500 [ 1231.450003] iomap_dio_rw+0x18/0x50 [ 1231.453479] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1231.458829] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1231.463572] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1231.468616] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1231.473745] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1231.478875] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1231.484261] process_one_work+0x1ec/0x470 [ 1231.488256] worker_thread+0x74/0x410 [ 1231.491905] kthread+0xec/0x100 [ 1231.495034] ret_from_fork+0x10/0x20 [ 1243.216630] watchdog: BUG: soft lockup - CPU#68 stuck for 763s! [swapper/68:0] [ 1243.223838] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1243.223855] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1243.311642] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1243.319459] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1243.328750] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1243.335698] pc : __sbq_wake_up+0x20/0x1f0 [ 1243.339694] lr : sbitmap_queue_clear+0x74/0xc0 [ 1243.344125] sp : ffff8000088438b0 [ 1243.347426] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1243.354548] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1243.361670] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1243.368792] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1243.375913] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1243.383035] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1243.390157] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1243.397279] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1243.404401] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1243.411522] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1243.418644] Call trace: [ 1243.421077] __sbq_wake_up+0x20/0x1f0 [ 1243.424726] sbitmap_queue_clear+0x74/0xc0 [ 1243.428809] blk_mq_put_tag+0x44/0x70 [ 1243.432459] __blk_mq_free_request+0x88/0xf0 [ 1243.436716] blk_mq_free_request+0xc0/0x160 [ 1243.440886] __blk_mq_end_request+0x78/0x150 [ 1243.445142] scsi_end_request+0x114/0x260 [ 1243.449138] scsi_io_completion+0x64/0x174 [ 1243.453222] scsi_finish_command+0xe0/0x120 [ 1243.457392] scsi_complete+0x8c/0x1c4 [ 1243.461042] blk_complete_reqs+0x5c/0x74 [ 1243.464952] blk_done_softirq+0x38/0x4c [ 1243.468774] __do_softirq+0x168/0x418 [ 1243.472423] __irq_exit_rcu+0x120/0x170 [ 1243.476246] irq_exit_rcu+0x18/0x24 [ 1243.479721] el1_interrupt+0x38/0x70 [ 1243.483283] el1h_64_irq_handler+0x18/0x24 [ 1243.487366] el1h_64_irq+0x68/0x6c [ 1243.490755] cpuidle_enter_state+0xc4/0x4e0 [ 1243.494925] cpuidle_enter+0x40/0x60 [ 1243.498488] cpuidle_idle_call+0x118/0x1b0 [ 1243.502571] do_idle+0xac/0x100 [ 1243.505700] cpu_startup_entry+0x30/0x34 [ 1243.509609] secondary_start_kernel+0xd8/0xf0 [ 1243.513953] __secondary_switched+0xb0/0xb4 [ 1255.106545] watchdog: BUG: soft lockup - CPU#8 stuck for 774s! [kworker/8:12:19399] [ 1255.114188] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1255.114205] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1255.201992] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1255.210243] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1255.219534] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1255.225701] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1255.232650] pc : __sbq_wake_up+0x18/0x1f0 [ 1255.236647] lr : sbitmap_queue_clear+0x74/0xc0 [ 1255.241078] sp : ffff80000d79b280 [ 1255.244379] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1255.251501] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1255.258623] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1255.265745] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1255.272867] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1255.279989] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1255.287112] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1255.294234] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1255.301356] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1255.308478] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 1255.315599] Call trace: [ 1255.318032] __sbq_wake_up+0x18/0x1f0 [ 1255.321681] sbitmap_queue_clear+0x74/0xc0 [ 1255.325765] blk_mq_put_tag+0x44/0x70 [ 1255.329415] __blk_mq_free_request+0x88/0xf0 [ 1255.333671] blk_mq_free_request+0xc0/0x160 [ 1255.337841] __blk_mq_end_request+0x78/0x150 [ 1255.342098] scsi_end_request+0x114/0x260 [ 1255.346095] scsi_io_completion+0x64/0x174 [ 1255.350179] scsi_finish_command+0xe0/0x120 [ 1255.354349] scsi_complete+0x8c/0x1c4 [ 1255.357998] blk_complete_reqs+0x5c/0x74 [ 1255.361908] blk_done_softirq+0x38/0x4c [ 1255.365731] __do_softirq+0x168/0x418 [ 1255.369380] __irq_exit_rcu+0x120/0x170 [ 1255.373202] irq_exit_rcu+0x18/0x24 [ 1255.376678] el1_interrupt+0x38/0x70 [ 1255.380240] el1h_64_irq_handler+0x18/0x24 [ 1255.384323] el1h_64_irq+0x68/0x6c [ 1255.387711] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1255.392489] __wake_up_common_lock+0x84/0xac [ 1255.396746] __wake_up+0x20/0x30 [ 1255.399962] __sbq_wake_up+0x13c/0x1f0 [ 1255.403698] sbitmap_queue_clear+0x74/0xc0 [ 1255.407781] blk_mq_put_tag+0x44/0x70 [ 1255.411431] __blk_mq_free_request+0x88/0xf0 [ 1255.415687] blk_mq_free_request+0xc0/0x160 [ 1255.419857] bfq_insert_request.isra.0+0x204/0x270 [ 1255.424635] bfq_insert_requests+0x70/0x90 [ 1255.428719] blk_mq_sched_insert_requests+0xac/0x250 [ 1255.433671] blk_mq_flush_plug_list+0x110/0x2cc [ 1255.438188] __blk_flush_plug+0x100/0x140 [ 1255.442185] blk_finish_plug+0x40/0x25c [ 1255.446007] __iomap_dio_rw+0x1b4/0x500 [ 1255.449830] iomap_dio_rw+0x18/0x50 [ 1255.453306] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1255.458656] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1255.463398] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1255.468442] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1255.473572] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1255.478702] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1255.484087] process_one_work+0x1ec/0x470 [ 1255.488084] worker_thread+0x74/0x410 [ 1255.491732] kthread+0xec/0x100 [ 1255.494861] ret_from_fork+0x10/0x20 [ 1267.216458] watchdog: BUG: soft lockup - CPU#68 stuck for 786s! [swapper/68:0] [ 1267.223667] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1267.223683] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1267.311470] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1267.319286] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1267.328577] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1267.335524] pc : __sbq_wake_up+0x58/0x1f0 [ 1267.339521] lr : sbitmap_queue_clear+0x74/0xc0 [ 1267.343951] sp : ffff8000088438b0 [ 1267.347252] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1267.354374] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1267.361496] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1267.368618] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1267.375740] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1267.382861] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1267.389983] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1267.397105] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1267.404227] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1267.411349] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1267.418471] Call trace: [ 1267.420904] __sbq_wake_up+0x58/0x1f0 [ 1267.424553] sbitmap_queue_clear+0x74/0xc0 [ 1267.428636] blk_mq_put_tag+0x44/0x70 [ 1267.432286] __blk_mq_free_request+0x88/0xf0 [ 1267.436543] blk_mq_free_request+0xc0/0x160 [ 1267.440714] __blk_mq_end_request+0x78/0x150 [ 1267.444971] scsi_end_request+0x114/0x260 [ 1267.448967] scsi_io_completion+0x64/0x174 [ 1267.453050] scsi_finish_command+0xe0/0x120 [ 1267.457221] scsi_complete+0x8c/0x1c4 [ 1267.460870] blk_complete_reqs+0x5c/0x74 [ 1267.464779] blk_done_softirq+0x38/0x4c [ 1267.468602] __do_softirq+0x168/0x418 [ 1267.472251] __irq_exit_rcu+0x120/0x170 [ 1267.476074] irq_exit_rcu+0x18/0x24 [ 1267.479549] el1_interrupt+0x38/0x70 [ 1267.483112] el1h_64_irq_handler+0x18/0x24 [ 1267.487195] el1h_64_irq+0x68/0x6c [ 1267.490583] cpuidle_enter_state+0xc4/0x4e0 [ 1267.494753] cpuidle_enter+0x40/0x60 [ 1267.498316] cpuidle_idle_call+0x118/0x1b0 [ 1267.502399] do_idle+0xac/0x100 [ 1267.505528] cpu_startup_entry+0x30/0x34 [ 1267.509437] secondary_start_kernel+0xd8/0xf0 [ 1267.513781] __secondary_switched+0xb0/0xb4 [ 1270.766443] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 81981 jiffies s: 2093 root: 0x11/. [ 1270.778095] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1270.787304] Task dump for CPU 8: [ 1270.790519] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1270.800421] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1270.806591] Call trace: [ 1270.809025] __switch_to+0xd4/0x100 [ 1270.812501] 0xffff6c0200000080 [ 1270.815629] Task dump for CPU 68: [ 1270.818932] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1270.828833] Call trace: [ 1270.831266] __switch_to+0xd4/0x100 [ 1270.834742] 0x0 [ 1279.106373] watchdog: BUG: soft lockup - CPU#8 stuck for 797s! [kworker/8:12:19399] [ 1279.114015] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1279.114032] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1279.201819] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1279.210069] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1279.219360] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1279.225527] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1279.232475] pc : __sbq_wake_up+0x58/0x1f0 [ 1279.236472] lr : sbitmap_queue_clear+0x74/0xc0 [ 1279.240903] sp : ffff80000d79b280 [ 1279.244204] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1279.251326] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1279.258448] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1279.265570] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1279.272692] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1279.279814] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1279.286936] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1279.294058] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1279.301180] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1279.308302] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1279.315424] Call trace: [ 1279.317857] __sbq_wake_up+0x58/0x1f0 [ 1279.321506] sbitmap_queue_clear+0x74/0xc0 [ 1279.325589] blk_mq_put_tag+0x44/0x70 [ 1279.329239] __blk_mq_free_request+0x88/0xf0 [ 1279.333496] blk_mq_free_request+0xc0/0x160 [ 1279.337666] __blk_mq_end_request+0x78/0x150 [ 1279.341922] scsi_end_request+0x114/0x260 [ 1279.345919] scsi_io_completion+0x64/0x174 [ 1279.350002] scsi_finish_command+0xe0/0x120 [ 1279.354172] scsi_complete+0x8c/0x1c4 [ 1279.357821] blk_complete_reqs+0x5c/0x74 [ 1279.361731] blk_done_softirq+0x38/0x4c [ 1279.365553] __do_softirq+0x168/0x418 [ 1279.369202] __irq_exit_rcu+0x120/0x170 [ 1279.373025] irq_exit_rcu+0x18/0x24 [ 1279.376500] el1_interrupt+0x38/0x70 [ 1279.380062] el1h_64_irq_handler+0x18/0x24 [ 1279.384145] el1h_64_irq+0x68/0x6c [ 1279.387534] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1279.392312] __wake_up_common_lock+0x84/0xac [ 1279.396569] __wake_up+0x20/0x30 [ 1279.399785] __sbq_wake_up+0x13c/0x1f0 [ 1279.403521] sbitmap_queue_clear+0x74/0xc0 [ 1279.407604] blk_mq_put_tag+0x44/0x70 [ 1279.411254] __blk_mq_free_request+0x88/0xf0 [ 1279.415510] blk_mq_free_request+0xc0/0x160 [ 1279.419680] bfq_insert_request.isra.0+0x204/0x270 [ 1279.424458] bfq_insert_requests+0x70/0x90 [ 1279.428542] blk_mq_sched_insert_requests+0xac/0x250 [ 1279.433494] blk_mq_flush_plug_list+0x110/0x2cc [ 1279.438011] __blk_flush_plug+0x100/0x140 [ 1279.442008] blk_finish_plug+0x40/0x25c [ 1279.445830] __iomap_dio_rw+0x1b4/0x500 [ 1279.449653] iomap_dio_rw+0x18/0x50 [ 1279.453128] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1279.458479] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1279.463221] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1279.468265] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1279.473395] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1279.478525] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1279.483911] process_one_work+0x1ec/0x470 [ 1279.487908] worker_thread+0x74/0x410 [ 1279.491557] kthread+0xec/0x100 [ 1279.494685] ret_from_fork+0x10/0x20 [ 1291.216286] watchdog: BUG: soft lockup - CPU#68 stuck for 808s! [swapper/68:0] [ 1291.223495] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1291.223511] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1291.311299] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1291.319115] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1291.328406] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1291.335353] pc : __sbq_wake_up+0x58/0x1f0 [ 1291.339350] lr : sbitmap_queue_clear+0x74/0xc0 [ 1291.343781] sp : ffff8000088438b0 [ 1291.347082] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1291.354203] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1291.361325] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1291.368447] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1291.375569] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1291.382691] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1291.389813] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1291.396934] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1291.404056] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1291.411178] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1291.418300] Call trace: [ 1291.420733] __sbq_wake_up+0x58/0x1f0 [ 1291.424383] sbitmap_queue_clear+0x74/0xc0 [ 1291.428466] blk_mq_put_tag+0x44/0x70 [ 1291.432116] __blk_mq_free_request+0x88/0xf0 [ 1291.436373] blk_mq_free_request+0xc0/0x160 [ 1291.440543] __blk_mq_end_request+0x78/0x150 [ 1291.444799] scsi_end_request+0x114/0x260 [ 1291.448796] scsi_io_completion+0x64/0x174 [ 1291.452879] scsi_finish_command+0xe0/0x120 [ 1291.457049] scsi_complete+0x8c/0x1c4 [ 1291.460698] blk_complete_reqs+0x5c/0x74 [ 1291.464608] blk_done_softirq+0x38/0x4c [ 1291.468431] __do_softirq+0x168/0x418 [ 1291.472081] __irq_exit_rcu+0x120/0x170 [ 1291.475903] irq_exit_rcu+0x18/0x24 [ 1291.479379] el1_interrupt+0x38/0x70 [ 1291.482941] el1h_64_irq_handler+0x18/0x24 [ 1291.487024] el1h_64_irq+0x68/0x6c [ 1291.490413] cpuidle_enter_state+0xc4/0x4e0 [ 1291.494583] cpuidle_enter+0x40/0x60 [ 1291.498145] cpuidle_idle_call+0x118/0x1b0 [ 1291.502230] do_idle+0xac/0x100 [ 1291.505359] cpu_startup_entry+0x30/0x34 [ 1291.509269] secondary_start_kernel+0xd8/0xf0 [ 1291.513613] __secondary_switched+0xb0/0xb4 [ 1303.106201] watchdog: BUG: soft lockup - CPU#8 stuck for 819s! [kworker/8:12:19399] [ 1303.113843] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1303.113861] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1303.201649] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1303.209899] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1303.219191] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1303.225358] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1303.232306] pc : __sbq_wake_up+0x84/0x1f0 [ 1303.236303] lr : sbitmap_queue_clear+0x74/0xc0 [ 1303.240733] sp : ffff80000d79b280 [ 1303.244035] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1303.251157] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1303.258279] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1303.265401] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1303.272522] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1303.279644] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1303.286767] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1303.293888] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1303.301010] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1303.308133] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1303.315254] Call trace: [ 1303.317687] __sbq_wake_up+0x84/0x1f0 [ 1303.321336] sbitmap_queue_clear+0x74/0xc0 [ 1303.325420] blk_mq_put_tag+0x44/0x70 [ 1303.329069] __blk_mq_free_request+0x88/0xf0 [ 1303.333326] blk_mq_free_request+0xc0/0x160 [ 1303.337496] __blk_mq_end_request+0x78/0x150 [ 1303.341753] scsi_end_request+0x114/0x260 [ 1303.345750] scsi_io_completion+0x64/0x174 [ 1303.349833] scsi_finish_command+0xe0/0x120 [ 1303.354003] scsi_complete+0x8c/0x1c4 [ 1303.357652] blk_complete_reqs+0x5c/0x74 [ 1303.361562] blk_done_softirq+0x38/0x4c [ 1303.365384] __do_softirq+0x168/0x418 [ 1303.369034] __irq_exit_rcu+0x120/0x170 [ 1303.372856] irq_exit_rcu+0x18/0x24 [ 1303.376332] el1_interrupt+0x38/0x70 [ 1303.379894] el1h_64_irq_handler+0x18/0x24 [ 1303.383978] el1h_64_irq+0x68/0x6c [ 1303.387367] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1303.392144] __wake_up_common_lock+0x84/0xac [ 1303.396402] __wake_up+0x20/0x30 [ 1303.399618] __sbq_wake_up+0x13c/0x1f0 [ 1303.403354] sbitmap_queue_clear+0x74/0xc0 [ 1303.407438] blk_mq_put_tag+0x44/0x70 [ 1303.411087] __blk_mq_free_request+0x88/0xf0 [ 1303.415344] blk_mq_free_request+0xc0/0x160 [ 1303.419514] bfq_insert_request.isra.0+0x204/0x270 [ 1303.424292] bfq_insert_requests+0x70/0x90 [ 1303.428376] blk_mq_sched_insert_requests+0xac/0x250 [ 1303.433329] blk_mq_flush_plug_list+0x110/0x2cc [ 1303.437847] __blk_flush_plug+0x100/0x140 [ 1303.441843] blk_finish_plug+0x40/0x25c [ 1303.445666] __iomap_dio_rw+0x1b4/0x500 [ 1303.449488] iomap_dio_rw+0x18/0x50 [ 1303.452964] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1303.458314] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1303.463057] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1303.468101] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1303.473231] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1303.478360] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1303.483747] process_one_work+0x1ec/0x470 [ 1303.487743] worker_thread+0x74/0x410 [ 1303.491391] kthread+0xec/0x100 [ 1303.494520] ret_from_fork+0x10/0x20 [ 1315.216114] watchdog: BUG: soft lockup - CPU#68 stuck for 830s! [swapper/68:0] [ 1315.223322] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1315.223339] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1315.311125] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1315.318941] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1315.328232] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1315.335179] pc : __sbq_wake_up+0x7c/0x1f0 [ 1315.339176] lr : sbitmap_queue_clear+0x74/0xc0 [ 1315.343606] sp : ffff800008843900 [ 1315.346908] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1315.354031] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1315.361153] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1315.368275] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1315.375397] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1315.382519] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1315.389640] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1315.396762] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1315.403884] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1315.411006] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 1315.418128] Call trace: [ 1315.420561] __sbq_wake_up+0x7c/0x1f0 [ 1315.424210] blk_mq_put_tag+0x44/0x70 [ 1315.427860] __blk_mq_free_request+0x88/0xf0 [ 1315.432116] blk_mq_free_request+0xc0/0x160 [ 1315.436286] __blk_mq_end_request+0x78/0x150 [ 1315.440543] scsi_end_request+0x114/0x260 [ 1315.444539] scsi_io_completion+0x64/0x174 [ 1315.448622] scsi_finish_command+0xe0/0x120 [ 1315.452793] scsi_complete+0x8c/0x1c4 [ 1315.456442] blk_complete_reqs+0x5c/0x74 [ 1315.460351] blk_done_softirq+0x38/0x4c [ 1315.464174] __do_softirq+0x168/0x418 [ 1315.467823] __irq_exit_rcu+0x120/0x170 [ 1315.471645] irq_exit_rcu+0x18/0x24 [ 1315.475121] el1_interrupt+0x38/0x70 [ 1315.478683] el1h_64_irq_handler+0x18/0x24 [ 1315.482766] el1h_64_irq+0x68/0x6c [ 1315.486154] cpuidle_enter_state+0xc4/0x4e0 [ 1315.490324] cpuidle_enter+0x40/0x60 [ 1315.493886] cpuidle_idle_call+0x118/0x1b0 [ 1315.497970] do_idle+0xac/0x100 [ 1315.501099] cpu_startup_entry+0x30/0x34 [ 1315.505008] secondary_start_kernel+0xd8/0xf0 [ 1315.509352] __secondary_switched+0xb0/0xb4 [ 1327.106028] watchdog: BUG: soft lockup - CPU#8 stuck for 841s! [kworker/8:12:19399] [ 1327.113671] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1327.113688] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1327.201475] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1327.209725] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1327.219015] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1327.225183] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1327.232130] pc : __sbq_wake_up+0x84/0x1f0 [ 1327.236127] lr : sbitmap_queue_clear+0x74/0xc0 [ 1327.240558] sp : ffff80000d79b280 [ 1327.243859] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1327.250981] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1327.258103] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1327.265225] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1327.272347] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1327.279469] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1327.286591] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1327.293713] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1327.300835] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1327.307956] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1327.315078] Call trace: [ 1327.317512] __sbq_wake_up+0x84/0x1f0 [ 1327.321161] sbitmap_queue_clear+0x74/0xc0 [ 1327.325245] blk_mq_put_tag+0x44/0x70 [ 1327.328895] __blk_mq_free_request+0x88/0xf0 [ 1327.333151] blk_mq_free_request+0xc0/0x160 [ 1327.337321] __blk_mq_end_request+0x78/0x150 [ 1327.341578] scsi_end_request+0x114/0x260 [ 1327.345574] scsi_io_completion+0x64/0x174 [ 1327.349657] scsi_finish_command+0xe0/0x120 [ 1327.353828] scsi_complete+0x8c/0x1c4 [ 1327.357477] blk_complete_reqs+0x5c/0x74 [ 1327.361386] blk_done_softirq+0x38/0x4c [ 1327.365209] __do_softirq+0x168/0x418 [ 1327.368858] __irq_exit_rcu+0x120/0x170 [ 1327.372681] irq_exit_rcu+0x18/0x24 [ 1327.376156] el1_interrupt+0x38/0x70 [ 1327.379718] el1h_64_irq_handler+0x18/0x24 [ 1327.383802] el1h_64_irq+0x68/0x6c [ 1327.387191] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1327.391968] __wake_up_common_lock+0x84/0xac [ 1327.396225] __wake_up+0x20/0x30 [ 1327.399441] __sbq_wake_up+0x13c/0x1f0 [ 1327.403178] sbitmap_queue_clear+0x74/0xc0 [ 1327.407261] blk_mq_put_tag+0x44/0x70 [ 1327.410911] __blk_mq_free_request+0x88/0xf0 [ 1327.415167] blk_mq_free_request+0xc0/0x160 [ 1327.419337] bfq_insert_request.isra.0+0x204/0x270 [ 1327.424115] bfq_insert_requests+0x70/0x90 [ 1327.428199] blk_mq_sched_insert_requests+0xac/0x250 [ 1327.433150] blk_mq_flush_plug_list+0x110/0x2cc [ 1327.437668] __blk_flush_plug+0x100/0x140 [ 1327.441664] blk_finish_plug+0x40/0x25c [ 1327.445487] __iomap_dio_rw+0x1b4/0x500 [ 1327.449310] iomap_dio_rw+0x18/0x50 [ 1327.452785] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1327.458136] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1327.462878] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1327.467922] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1327.473052] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1327.478182] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1327.483568] process_one_work+0x1ec/0x470 [ 1327.487564] worker_thread+0x74/0x410 [ 1327.491213] kthread+0xec/0x100 [ 1327.494341] ret_from_fork+0x10/0x20 [ 1337.325965] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 88637 jiffies s: 2093 root: 0x11/. [ 1337.337617] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1337.346825] Task dump for CPU 8: [ 1337.350040] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1337.359942] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1337.366111] Call trace: [ 1337.368544] __switch_to+0xd4/0x100 [ 1337.372021] 0xffff6c0200000080 [ 1337.375149] Task dump for CPU 68: [ 1337.378452] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1337.388353] Call trace: [ 1337.390786] __switch_to+0xd4/0x100 [ 1337.394262] 0x0 [ 1339.215941] watchdog: BUG: soft lockup - CPU#68 stuck for 853s! [swapper/68:0] [ 1339.223150] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1339.223166] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1339.310952] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1339.318769] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1339.328060] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1339.335008] pc : __sbq_wake_up+0x20/0x1f0 [ 1339.339004] lr : sbitmap_queue_clear+0x74/0xc0 [ 1339.343434] sp : ffff8000088438b0 [ 1339.346736] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1339.353857] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1339.360980] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1339.368102] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1339.375224] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1339.382346] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1339.389468] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1339.396589] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1339.403711] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1339.410833] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1339.417955] Call trace: [ 1339.420388] __sbq_wake_up+0x20/0x1f0 [ 1339.424037] sbitmap_queue_clear+0x74/0xc0 [ 1339.428121] blk_mq_put_tag+0x44/0x70 [ 1339.431771] __blk_mq_free_request+0x88/0xf0 [ 1339.436027] blk_mq_free_request+0xc0/0x160 [ 1339.440197] __blk_mq_end_request+0x78/0x150 [ 1339.444454] scsi_end_request+0x114/0x260 [ 1339.448451] scsi_io_completion+0x64/0x174 [ 1339.452534] scsi_finish_command+0xe0/0x120 [ 1339.456704] scsi_complete+0x8c/0x1c4 [ 1339.460353] blk_complete_reqs+0x5c/0x74 [ 1339.464264] blk_done_softirq+0x38/0x4c [ 1339.468087] __do_softirq+0x168/0x418 [ 1339.471735] __irq_exit_rcu+0x120/0x170 [ 1339.475558] irq_exit_rcu+0x18/0x24 [ 1339.479033] el1_interrupt+0x38/0x70 [ 1339.482595] el1h_64_irq_handler+0x18/0x24 [ 1339.486679] el1h_64_irq+0x68/0x6c [ 1339.490067] cpuidle_enter_state+0xc4/0x4e0 [ 1339.494237] cpuidle_enter+0x40/0x60 [ 1339.497799] cpuidle_idle_call+0x118/0x1b0 [ 1339.501883] do_idle+0xac/0x100 [ 1339.505012] cpu_startup_entry+0x30/0x34 [ 1339.508921] secondary_start_kernel+0xd8/0xf0 [ 1339.513265] __secondary_switched+0xb0/0xb4 [ 1351.105856] watchdog: BUG: soft lockup - CPU#8 stuck for 864s! [kworker/8:12:19399] [ 1351.113499] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1351.113516] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1351.201303] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1351.209552] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1351.218843] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1351.225011] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1351.231958] pc : __sbq_wake_up+0x20/0x1f0 [ 1351.235955] lr : sbitmap_queue_clear+0x74/0xc0 [ 1351.240386] sp : ffff80000d79b280 [ 1351.243687] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1351.250810] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1351.257932] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1351.265054] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1351.272176] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1351.279298] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1351.286419] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1351.293541] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1351.300663] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1351.307785] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 1351.314907] Call trace: [ 1351.317340] __sbq_wake_up+0x20/0x1f0 [ 1351.320989] sbitmap_queue_clear+0x74/0xc0 [ 1351.325072] blk_mq_put_tag+0x44/0x70 [ 1351.328722] __blk_mq_free_request+0x88/0xf0 [ 1351.332978] blk_mq_free_request+0xc0/0x160 [ 1351.337148] __blk_mq_end_request+0x78/0x150 [ 1351.341405] scsi_end_request+0x114/0x260 [ 1351.345402] scsi_io_completion+0x64/0x174 [ 1351.349486] scsi_finish_command+0xe0/0x120 [ 1351.353657] scsi_complete+0x8c/0x1c4 [ 1351.357306] blk_complete_reqs+0x5c/0x74 [ 1351.361215] blk_done_softirq+0x38/0x4c [ 1351.365038] __do_softirq+0x168/0x418 [ 1351.368686] __irq_exit_rcu+0x120/0x170 [ 1351.372509] irq_exit_rcu+0x18/0x24 [ 1351.375984] el1_interrupt+0x38/0x70 [ 1351.379547] el1h_64_irq_handler+0x18/0x24 [ 1351.383630] el1h_64_irq+0x68/0x6c [ 1351.387018] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1351.391796] __wake_up_common_lock+0x84/0xac [ 1351.396053] __wake_up+0x20/0x30 [ 1351.399269] __sbq_wake_up+0x13c/0x1f0 [ 1351.403005] sbitmap_queue_clear+0x74/0xc0 [ 1351.407088] blk_mq_put_tag+0x44/0x70 [ 1351.410738] __blk_mq_free_request+0x88/0xf0 [ 1351.414995] blk_mq_free_request+0xc0/0x160 [ 1351.419165] bfq_insert_request.isra.0+0x204/0x270 [ 1351.423943] bfq_insert_requests+0x70/0x90 [ 1351.428026] blk_mq_sched_insert_requests+0xac/0x250 [ 1351.432978] blk_mq_flush_plug_list+0x110/0x2cc [ 1351.437496] __blk_flush_plug+0x100/0x140 [ 1351.441493] blk_finish_plug+0x40/0x25c [ 1351.445315] __iomap_dio_rw+0x1b4/0x500 [ 1351.449138] iomap_dio_rw+0x18/0x50 [ 1351.452613] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1351.457964] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1351.462706] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1351.467750] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1351.472880] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1351.478010] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1351.483396] process_one_work+0x1ec/0x470 [ 1351.487392] worker_thread+0x74/0x410 [ 1351.491041] kthread+0xec/0x100 [ 1351.494170] ret_from_fork+0x10/0x20 [ 1363.215769] watchdog: BUG: soft lockup - CPU#68 stuck for 875s! [swapper/68:0] [ 1363.222977] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1363.222993] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1363.310780] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1363.318596] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1363.327887] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1363.334834] pc : __sbq_wake_up+0x58/0x1f0 [ 1363.338831] lr : sbitmap_queue_clear+0x74/0xc0 [ 1363.343261] sp : ffff8000088438b0 [ 1363.346563] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1363.353685] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1363.360806] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1363.367928] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1363.375050] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1363.382172] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1363.389294] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1363.396415] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1363.403537] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1363.410659] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1363.417781] Call trace: [ 1363.420214] __sbq_wake_up+0x58/0x1f0 [ 1363.423864] sbitmap_queue_clear+0x74/0xc0 [ 1363.427947] blk_mq_put_tag+0x44/0x70 [ 1363.431597] __blk_mq_free_request+0x88/0xf0 [ 1363.435853] blk_mq_free_request+0xc0/0x160 [ 1363.440023] __blk_mq_end_request+0x78/0x150 [ 1363.444280] scsi_end_request+0x114/0x260 [ 1363.448276] scsi_io_completion+0x64/0x174 [ 1363.452359] scsi_finish_command+0xe0/0x120 [ 1363.456530] scsi_complete+0x8c/0x1c4 [ 1363.460179] blk_complete_reqs+0x5c/0x74 [ 1363.464088] blk_done_softirq+0x38/0x4c [ 1363.467911] __do_softirq+0x168/0x418 [ 1363.471560] __irq_exit_rcu+0x120/0x170 [ 1363.475383] irq_exit_rcu+0x18/0x24 [ 1363.478858] el1_interrupt+0x38/0x70 [ 1363.482421] el1h_64_irq_handler+0x18/0x24 [ 1363.486504] el1h_64_irq+0x68/0x6c [ 1363.489892] cpuidle_enter_state+0xc4/0x4e0 [ 1363.494062] cpuidle_enter+0x40/0x60 [ 1363.497624] cpuidle_idle_call+0x118/0x1b0 [ 1363.501708] do_idle+0xac/0x100 [ 1363.504837] cpu_startup_entry+0x30/0x34 [ 1363.508746] secondary_start_kernel+0xd8/0xf0 [ 1363.513090] __secondary_switched+0xb0/0xb4 [ 1375.105684] watchdog: BUG: soft lockup - CPU#8 stuck for 886s! [kworker/8:12:19399] [ 1375.113326] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1375.113343] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1375.201130] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1375.209380] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1375.218671] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1375.224838] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1375.231786] pc : __sbq_wake_up+0x40/0x1f0 [ 1375.235783] lr : sbitmap_queue_clear+0x74/0xc0 [ 1375.240213] sp : ffff80000d79b280 [ 1375.243514] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1375.250637] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1375.257759] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1375.264880] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1375.272003] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1375.279125] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1375.286247] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1375.293368] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1375.300490] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1375.307612] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 1375.314734] Call trace: [ 1375.317168] __sbq_wake_up+0x40/0x1f0 [ 1375.320817] sbitmap_queue_clear+0x74/0xc0 [ 1375.324900] blk_mq_put_tag+0x44/0x70 [ 1375.328550] __blk_mq_free_request+0x88/0xf0 [ 1375.332807] blk_mq_free_request+0xc0/0x160 [ 1375.336977] __blk_mq_end_request+0x78/0x150 [ 1375.341233] scsi_end_request+0x114/0x260 [ 1375.345230] scsi_io_completion+0x64/0x174 [ 1375.349313] scsi_finish_command+0xe0/0x120 [ 1375.353483] scsi_complete+0x8c/0x1c4 [ 1375.357132] blk_complete_reqs+0x5c/0x74 [ 1375.361042] blk_done_softirq+0x38/0x4c [ 1375.364864] __do_softirq+0x168/0x418 [ 1375.368513] __irq_exit_rcu+0x120/0x170 [ 1375.372336] irq_exit_rcu+0x18/0x24 [ 1375.375812] el1_interrupt+0x38/0x70 [ 1375.379374] el1h_64_irq_handler+0x18/0x24 [ 1375.383457] el1h_64_irq+0x68/0x6c [ 1375.386846] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1375.391623] __wake_up_common_lock+0x84/0xac [ 1375.395880] __wake_up+0x20/0x30 [ 1375.399096] __sbq_wake_up+0x13c/0x1f0 [ 1375.402832] sbitmap_queue_clear+0x74/0xc0 [ 1375.406915] blk_mq_put_tag+0x44/0x70 [ 1375.410565] __blk_mq_free_request+0x88/0xf0 [ 1375.414822] blk_mq_free_request+0xc0/0x160 [ 1375.418992] bfq_insert_request.isra.0+0x204/0x270 [ 1375.423770] bfq_insert_requests+0x70/0x90 [ 1375.427853] blk_mq_sched_insert_requests+0xac/0x250 [ 1375.432805] blk_mq_flush_plug_list+0x110/0x2cc [ 1375.437323] __blk_flush_plug+0x100/0x140 [ 1375.441320] blk_finish_plug+0x40/0x25c [ 1375.445142] __iomap_dio_rw+0x1b4/0x500 [ 1375.448965] iomap_dio_rw+0x18/0x50 [ 1375.452441] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1375.457791] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1375.462533] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1375.467577] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1375.472707] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1375.477837] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1375.483223] process_one_work+0x1ec/0x470 [ 1375.487219] worker_thread+0x74/0x410 [ 1375.490868] kthread+0xec/0x100 [ 1375.493997] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 04:20:00 2022] [ 1386.605602] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1386.611335] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=47713 [ 1386.620455] (t=96171 jiffies g=36445 q=28588 ncpus=128) [ 1386.625755] Task dump for CPU 8: [ 1386.628971] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1386.638871] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1386.645040] Call trace: [ 1386.647472] __switch_to+0xd4/0x100 [ 1386.650948] 0xffff6c0200000080 [ 1386.654077] Task dump for CPU 68: [ 1386.657378] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1386.667279] Call trace: [ 1386.669712] dump_backtrace+0xe8/0x130 [ 1386.673448] show_stack+0x20/0x60 [ 1386.676750] sched_show_task+0x160/0x190 [ 1386.680659] dump_cpu_task+0x4c/0x78 [ 1386.684222] rcu_dump_cpu_stacks+0xc4/0x150 [ 1386.688393] print_cpu_stall+0x180/0x210 [ 1386.692303] check_cpu_stall+0x20c/0x2b4 [ 1386.696212] rcu_pending+0x4c/0x190 [ 1386.699688] rcu_sched_clock_irq+0x80/0x204 [ 1386.703858] update_process_times+0x7c/0xc0 [ 1386.708028] tick_sched_handle+0x38/0x74 [ 1386.711938] tick_sched_timer+0x60/0xc0 [ 1386.715761] __hrtimer_run_queues+0x170/0x300 [ 1386.720104] hrtimer_interrupt+0xfc/0x260 [ 1386.724101] arch_timer_handler_phys+0x34/0x50 [ 1386.728532] handle_percpu_devid_irq+0xac/0x240 [ 1386.733049] generic_handle_domain_irq+0x34/0x50 [ 1386.737654] gic_handle_irq+0x5c/0xe0 [ 1386.741304] call_on_irq_stack+0x2c/0x38 [ 1386.745214] do_interrupt_handler+0xdc/0xe0 [ 1386.749384] el1_interrupt+0x34/0x70 [ 1386.752947] el1h_64_irq_handler+0x18/0x24 [ 1386.757030] el1h_64_irq+0x68/0x6c [ 1386.760418] __sbq_wake_up+0x20/0x1f0 [ 1386.764068] sbitmap_queue_clear+0x74/0xc0 [ 1386.768152] blk_mq_put_tag+0x44/0x70 [ 1386.771801] __blk_mq_free_request+0x88/0xf0 [ 1386.776058] blk_mq_free_request+0xc0/0x160 [ 1386.780228] __blk_mq_end_request+0x78/0x150 [ 1386.784485] scsi_end_request+0x114/0x260 [ 1386.788481] scsi_io_completion+0x64/0x174 [ 1386.792564] scsi_finish_command+0xe0/0x120 [ 1386.796734] scsi_complete+0x8c/0x1c4 [ 1386.800383] blk_complete_reqs+0x5c/0x74 [ 1386.804293] blk_done_softirq+0x38/0x4c [ 1386.808116] __do_softirq+0x168/0x418 [ 1386.811764] __irq_exit_rcu+0x120/0x170 [ 1386.815587] irq_exit_rcu+0x18/0x24 [ 1386.819062] el1_interrupt+0x38/0x70 [ 1386.822625] el1h_64_irq_handler+0x18/0x24 [ 1386.826708] el1h_64_irq+0x68/0x6c [ 1386.830096] cpuidle_enter_state+0xc4/0x4e0 [ 1386.834266] cpuidle_enter+0x40/0x60 [ 1386.837828] cpuidle_idle_call+0x118/0x1b0 [ 1386.841912] do_idle+0xac/0x100 [ 1386.845041] cpu_startup_entry+0x30/0x34 [ 1386.848950] secondary_start_kernel+0xd8/0xf0 [ 1386.853294] __secondary_switched+0xb0/0xb4 [ 1399.105511] watchdog: BUG: soft lockup - CPU#8 stuck for 909s! [kworker/8:12:19399] [ 1399.113154] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1399.113171] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1399.200959] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1399.209208] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1399.218500] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1399.224668] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1399.231615] pc : __sbq_wake_up+0x58/0x1f0 [ 1399.235612] lr : sbitmap_queue_clear+0x74/0xc0 [ 1399.240043] sp : ffff80000d79b280 [ 1399.243344] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1399.250466] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1399.257588] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1399.264710] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1399.271832] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1399.278954] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1399.286075] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1399.293197] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1399.300319] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1399.307441] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1399.314563] Call trace: [ 1399.316996] __sbq_wake_up+0x58/0x1f0 [ 1399.320645] sbitmap_queue_clear+0x74/0xc0 [ 1399.324729] blk_mq_put_tag+0x44/0x70 [ 1399.328378] __blk_mq_free_request+0x88/0xf0 [ 1399.332635] blk_mq_free_request+0xc0/0x160 [ 1399.336805] __blk_mq_end_request+0x78/0x150 [ 1399.341062] scsi_end_request+0x114/0x260 [ 1399.345058] scsi_io_completion+0x64/0x174 [ 1399.349141] scsi_finish_command+0xe0/0x120 [ 1399.353311] scsi_complete+0x8c/0x1c4 [ 1399.356961] blk_complete_reqs+0x5c/0x74 [ 1399.360870] blk_done_softirq+0x38/0x4c [ 1399.364693] __do_softirq+0x168/0x418 [ 1399.368341] __irq_exit_rcu+0x120/0x170 [ 1399.372164] irq_exit_rcu+0x18/0x24 [ 1399.375639] el1_interrupt+0x38/0x70 [ 1399.379202] el1h_64_irq_handler+0x18/0x24 [ 1399.383286] el1h_64_irq+0x68/0x6c [ 1399.386675] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1399.391453] __wake_up_common_lock+0x84/0xac [ 1399.395710] __wake_up+0x20/0x30 [ 1399.398926] __sbq_wake_up+0x13c/0x1f0 [ 1399.402662] sbitmap_queue_clear+0x74/0xc0 [ 1399.406745] blk_mq_put_tag+0x44/0x70 [ 1399.410395] __blk_mq_free_request+0x88/0xf0 [ 1399.414652] blk_mq_free_request+0xc0/0x160 [ 1399.418822] bfq_insert_request.isra.0+0x204/0x270 [ 1399.423600] bfq_insert_requests+0x70/0x90 [ 1399.427683] blk_mq_sched_insert_requests+0xac/0x250 [ 1399.432636] blk_mq_flush_plug_list+0x110/0x2cc [ 1399.437154] __blk_flush_plug+0x100/0x140 [ 1399.441150] blk_finish_plug+0x40/0x25c [ 1399.444973] __iomap_dio_rw+0x1b4/0x500 [ 1399.448796] iomap_dio_rw+0x18/0x50 [ 1399.452272] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1399.457622] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1399.462364] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1399.467408] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1399.472538] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1399.477668] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1399.483054] process_one_work+0x1ec/0x470 [ 1399.487050] worker_thread+0x74/0x410 [ 1399.490699] kthread+0xec/0x100 [ 1399.493827] ret_from_fork+0x10/0x20 [ 1403.885487] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 95293 jiffies s: 2093 root: 0x11/. [ 1403.897138] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1403.906347] Task dump for CPU 8: [ 1403.909561] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1403.919463] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1403.925633] Call trace: [ 1403.928066] __switch_to+0xd4/0x100 [ 1403.931543] 0xffff6c0200000080 [ 1403.934671] Task dump for CPU 68: [ 1403.937973] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1403.947874] Call trace: [ 1403.950307] __switch_to+0xd4/0x100 [ 1403.953783] 0x0 [ 1411.215430] watchdog: BUG: soft lockup - CPU#68 stuck for 920s! [swapper/68:0] [ 1411.222639] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1411.222656] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1411.310445] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1411.318260] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1411.327551] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1411.334499] pc : __sbq_wake_up+0x4/0x1f0 [ 1411.338408] lr : sbitmap_queue_clear+0x74/0xc0 [ 1411.342839] sp : ffff800008843900 [ 1411.346140] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1411.353262] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1411.360384] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1411.367506] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1411.374628] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1411.381750] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1411.388871] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1411.395993] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1411.403115] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1411.410237] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1411.417359] Call trace: [ 1411.419792] __sbq_wake_up+0x4/0x1f0 [ 1411.423354] blk_mq_put_tag+0x44/0x70 [ 1411.427005] __blk_mq_free_request+0x88/0xf0 [ 1411.431262] blk_mq_free_request+0xc0/0x160 [ 1411.435432] __blk_mq_end_request+0x78/0x150 [ 1411.439688] scsi_end_request+0x114/0x260 [ 1411.443685] scsi_io_completion+0x64/0x174 [ 1411.447768] scsi_finish_command+0xe0/0x120 [ 1411.451938] scsi_complete+0x8c/0x1c4 [ 1411.455587] blk_complete_reqs+0x5c/0x74 [ 1411.459497] blk_done_softirq+0x38/0x4c [ 1411.463319] __do_softirq+0x168/0x418 [ 1411.466968] __irq_exit_rcu+0x120/0x170 [ 1411.470792] irq_exit_rcu+0x18/0x24 [ 1411.474267] el1_interrupt+0x38/0x70 [ 1411.477829] el1h_64_irq_handler+0x18/0x24 [ 1411.481912] el1h_64_irq+0x68/0x6c [ 1411.485300] cpuidle_enter_state+0xc4/0x4e0 [ 1411.489471] cpuidle_enter+0x40/0x60 [ 1411.493033] cpuidle_idle_call+0x118/0x1b0 [ 1411.497116] do_idle+0xac/0x100 [ 1411.500245] cpu_startup_entry+0x30/0x34 [ 1411.504155] secondary_start_kernel+0xd8/0xf0 [ 1411.508500] __secondary_switched+0xb0/0xb4 [ 1423.105354] watchdog: BUG: soft lockup - CPU#8 stuck for 931s! [kworker/8:12:19399] [ 1423.112996] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1423.113013] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1423.200800] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1423.209050] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1423.218341] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1423.224509] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1423.231458] pc : __sbq_wake_up+0x1c8/0x1f0 [ 1423.235541] lr : sbitmap_queue_clear+0x74/0xc0 [ 1423.239972] sp : ffff80000d79b280 [ 1423.243274] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1423.250396] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1423.257518] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1423.264640] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1423.271762] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1423.278884] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1423.286005] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1423.293127] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1423.300249] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1423.307371] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 1423.314493] Call trace: [ 1423.316926] __sbq_wake_up+0x1c8/0x1f0 [ 1423.320662] sbitmap_queue_clear+0x74/0xc0 [ 1423.324746] blk_mq_put_tag+0x44/0x70 [ 1423.328395] __blk_mq_free_request+0x88/0xf0 [ 1423.332652] blk_mq_free_request+0xc0/0x160 [ 1423.336822] __blk_mq_end_request+0x78/0x150 [ 1423.341079] scsi_end_request+0x114/0x260 [ 1423.345075] scsi_io_completion+0x64/0x174 [ 1423.349158] scsi_finish_command+0xe0/0x120 [ 1423.353328] scsi_complete+0x8c/0x1c4 [ 1423.356977] blk_complete_reqs+0x5c/0x74 [ 1423.360887] blk_done_softirq+0x38/0x4c [ 1423.364710] __do_softirq+0x168/0x418 [ 1423.368359] __irq_exit_rcu+0x120/0x170 [ 1423.372181] irq_exit_rcu+0x18/0x24 [ 1423.375657] el1_interrupt+0x38/0x70 [ 1423.379219] el1h_64_irq_handler+0x18/0x24 [ 1423.383302] el1h_64_irq+0x68/0x6c [ 1423.386691] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1423.391468] __wake_up_common_lock+0x84/0xac [ 1423.395726] __wake_up+0x20/0x30 [ 1423.398942] __sbq_wake_up+0x13c/0x1f0 [ 1423.402678] sbitmap_queue_clear+0x74/0xc0 [ 1423.406761] blk_mq_put_tag+0x44/0x70 [ 1423.410412] __blk_mq_free_request+0x88/0xf0 [ 1423.414668] blk_mq_free_request+0xc0/0x160 [ 1423.418838] bfq_insert_request.isra.0+0x204/0x270 [ 1423.423616] bfq_insert_requests+0x70/0x90 [ 1423.427699] blk_mq_sched_insert_requests+0xac/0x250 [ 1423.432651] blk_mq_flush_plug_list+0x110/0x2cc [ 1423.437169] __blk_flush_plug+0x100/0x140 [ 1423.441165] blk_finish_plug+0x40/0x25c [ 1423.444988] __iomap_dio_rw+0x1b4/0x500 [ 1423.448810] iomap_dio_rw+0x18/0x50 [ 1423.452286] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1423.457636] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1423.462379] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1423.467422] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1423.472552] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1423.477682] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1423.483068] process_one_work+0x1ec/0x470 [ 1423.487064] worker_thread+0x74/0x410 [ 1423.490713] kthread+0xec/0x100 [ 1423.493842] ret_from_fork+0x10/0x20 [ 1435.215276] watchdog: BUG: soft lockup - CPU#68 stuck for 942s! [swapper/68:0] [ 1435.222485] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1435.222501] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1435.310289] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1435.318105] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1435.327396] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1435.334343] pc : __sbq_wake_up+0x20/0x1f0 [ 1435.338340] lr : sbitmap_queue_clear+0x74/0xc0 [ 1435.342770] sp : ffff8000088438b0 [ 1435.346071] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1435.353194] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1435.360316] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1435.367437] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1435.374560] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1435.381681] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1435.388803] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1435.395925] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1435.403047] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1435.410169] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1435.417291] Call trace: [ 1435.419724] __sbq_wake_up+0x20/0x1f0 [ 1435.423373] sbitmap_queue_clear+0x74/0xc0 [ 1435.427457] blk_mq_put_tag+0x44/0x70 [ 1435.431106] __blk_mq_free_request+0x88/0xf0 [ 1435.435363] blk_mq_free_request+0xc0/0x160 [ 1435.439533] __blk_mq_end_request+0x78/0x150 [ 1435.443790] scsi_end_request+0x114/0x260 [ 1435.447786] scsi_io_completion+0x64/0x174 [ 1435.451869] scsi_finish_command+0xe0/0x120 [ 1435.456039] scsi_complete+0x8c/0x1c4 [ 1435.459688] blk_complete_reqs+0x5c/0x74 [ 1435.463598] blk_done_softirq+0x38/0x4c [ 1435.467421] __do_softirq+0x168/0x418 [ 1435.471070] __irq_exit_rcu+0x120/0x170 [ 1435.474892] irq_exit_rcu+0x18/0x24 [ 1435.478368] el1_interrupt+0x38/0x70 [ 1435.481930] el1h_64_irq_handler+0x18/0x24 [ 1435.486013] el1h_64_irq+0x68/0x6c [ 1435.489402] cpuidle_enter_state+0xc4/0x4e0 [ 1435.493572] cpuidle_enter+0x40/0x60 [ 1435.497134] cpuidle_idle_call+0x118/0x1b0 [ 1435.501218] do_idle+0xac/0x100 [ 1435.504347] cpu_startup_entry+0x30/0x34 [ 1435.508256] secondary_start_kernel+0xd8/0xf0 [ 1435.512600] __secondary_switched+0xb0/0xb4 [ 1447.105200] watchdog: BUG: soft lockup - CPU#8 stuck for 953s! [kworker/8:12:19399] [ 1447.112843] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1447.112859] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1447.200646] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1447.208896] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1447.218187] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1447.224355] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1447.231302] pc : sbitmap_queue_clear+0x74/0xc0 [ 1447.235733] lr : sbitmap_queue_clear+0x74/0xc0 [ 1447.240163] sp : ffff80000d79b2d0 [ 1447.243465] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1447.250587] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1447.257709] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1447.264831] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1447.271952] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1447.279074] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1447.286197] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1447.293319] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1447.300440] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1447.307562] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 1447.314684] Call trace: [ 1447.317117] sbitmap_queue_clear+0x74/0xc0 [ 1447.321200] blk_mq_put_tag+0x44/0x70 [ 1447.324850] __blk_mq_free_request+0x88/0xf0 [ 1447.329107] blk_mq_free_request+0xc0/0x160 [ 1447.333277] __blk_mq_end_request+0x78/0x150 [ 1447.337534] scsi_end_request+0x114/0x260 [ 1447.341530] scsi_io_completion+0x64/0x174 [ 1447.345613] scsi_finish_command+0xe0/0x120 [ 1447.349784] scsi_complete+0x8c/0x1c4 [ 1447.353433] blk_complete_reqs+0x5c/0x74 [ 1447.357342] blk_done_softirq+0x38/0x4c [ 1447.361165] __do_softirq+0x168/0x418 [ 1447.364814] __irq_exit_rcu+0x120/0x170 [ 1447.368637] irq_exit_rcu+0x18/0x24 [ 1447.372112] el1_interrupt+0x38/0x70 [ 1447.375674] el1h_64_irq_handler+0x18/0x24 [ 1447.379758] el1h_64_irq+0x68/0x6c [ 1447.383146] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1447.387924] __wake_up_common_lock+0x84/0xac [ 1447.392181] __wake_up+0x20/0x30 [ 1447.395397] __sbq_wake_up+0x13c/0x1f0 [ 1447.399132] sbitmap_queue_clear+0x74/0xc0 [ 1447.403216] blk_mq_put_tag+0x44/0x70 [ 1447.406865] __blk_mq_free_request+0x88/0xf0 [ 1447.411122] blk_mq_free_request+0xc0/0x160 [ 1447.415292] bfq_insert_request.isra.0+0x204/0x270 [ 1447.420070] bfq_insert_requests+0x70/0x90 [ 1447.424154] blk_mq_sched_insert_requests+0xac/0x250 [ 1447.429106] blk_mq_flush_plug_list+0x110/0x2cc [ 1447.433623] __blk_flush_plug+0x100/0x140 [ 1447.437620] blk_finish_plug+0x40/0x25c [ 1447.441442] __iomap_dio_rw+0x1b4/0x500 [ 1447.445265] iomap_dio_rw+0x18/0x50 [ 1447.448741] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1447.454091] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1447.458833] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1447.463877] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1447.469008] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1447.474137] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1447.479523] process_one_work+0x1ec/0x470 [ 1447.483519] worker_thread+0x74/0x410 [ 1447.487168] kthread+0xec/0x100 [ 1447.490297] ret_from_fork+0x10/0x20 [ 1459.215123] watchdog: BUG: soft lockup - CPU#68 stuck for 964s! [swapper/68:0] [ 1459.222331] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1459.222347] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1459.310134] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1459.317949] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1459.327240] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1459.334188] pc : __sbq_wake_up+0x58/0x1f0 [ 1459.338184] lr : sbitmap_queue_clear+0x74/0xc0 [ 1459.342615] sp : ffff8000088438b0 [ 1459.345917] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1459.353039] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1459.360161] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1459.367283] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1459.374405] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1459.381528] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1459.388649] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1459.395771] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1459.402893] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1459.410015] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1459.417137] Call trace: [ 1459.419570] __sbq_wake_up+0x58/0x1f0 [ 1459.423219] sbitmap_queue_clear+0x74/0xc0 [ 1459.427303] blk_mq_put_tag+0x44/0x70 [ 1459.430952] __blk_mq_free_request+0x88/0xf0 [ 1459.435209] blk_mq_free_request+0xc0/0x160 [ 1459.439379] __blk_mq_end_request+0x78/0x150 [ 1459.443636] scsi_end_request+0x114/0x260 [ 1459.447632] scsi_io_completion+0x64/0x174 [ 1459.451715] scsi_finish_command+0xe0/0x120 [ 1459.455886] scsi_complete+0x8c/0x1c4 [ 1459.459535] blk_complete_reqs+0x5c/0x74 [ 1459.463444] blk_done_softirq+0x38/0x4c [ 1459.467267] __do_softirq+0x168/0x418 [ 1459.470915] __irq_exit_rcu+0x120/0x170 [ 1459.474738] irq_exit_rcu+0x18/0x24 [ 1459.478213] el1_interrupt+0x38/0x70 [ 1459.481776] el1h_64_irq_handler+0x18/0x24 [ 1459.485859] el1h_64_irq+0x68/0x6c [ 1459.489247] cpuidle_enter_state+0xc4/0x4e0 [ 1459.493417] cpuidle_enter+0x40/0x60 [ 1459.496980] cpuidle_idle_call+0x118/0x1b0 [ 1459.501064] do_idle+0xac/0x100 [ 1459.504193] cpu_startup_entry+0x30/0x34 [ 1459.508102] secondary_start_kernel+0xd8/0xf0 [ 1459.512446] __secondary_switched+0xb0/0xb4 [ 1470.445060] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 101949 jiffies s: 2093 root: 0x11/. [ 1470.456798] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1470.466007] Task dump for CPU 8: [ 1470.469222] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1470.479123] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1470.485293] Call trace: [ 1470.487726] __switch_to+0xd4/0x100 [ 1470.491202] 0xffff6c0200000080 [ 1470.494330] Task dump for CPU 68: [ 1470.497633] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1470.507534] Call trace: [ 1470.509967] __switch_to+0xd4/0x100 [ 1470.513444] 0x0 [ 1471.105046] watchdog: BUG: soft lockup - CPU#8 stuck for 976s! [kworker/8:12:19399] [ 1471.112689] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1471.112706] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1471.200492] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1471.208742] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1471.218033] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1471.224200] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1471.231148] pc : __sbq_wake_up+0x84/0x1f0 [ 1471.235144] lr : sbitmap_queue_clear+0x74/0xc0 [ 1471.239574] sp : ffff80000d79b280 [ 1471.242876] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1471.249997] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1471.257119] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1471.264241] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1471.271364] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1471.278485] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1471.285607] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1471.292729] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1471.299851] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1471.306973] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1471.314095] Call trace: [ 1471.316528] __sbq_wake_up+0x84/0x1f0 [ 1471.320177] sbitmap_queue_clear+0x74/0xc0 [ 1471.324261] blk_mq_put_tag+0x44/0x70 [ 1471.327911] __blk_mq_free_request+0x88/0xf0 [ 1471.332169] blk_mq_free_request+0xc0/0x160 [ 1471.336339] __blk_mq_end_request+0x78/0x150 [ 1471.340596] scsi_end_request+0x114/0x260 [ 1471.344592] scsi_io_completion+0x64/0x174 [ 1471.348675] scsi_finish_command+0xe0/0x120 [ 1471.352845] scsi_complete+0x8c/0x1c4 [ 1471.356494] blk_complete_reqs+0x5c/0x74 [ 1471.360404] blk_done_softirq+0x38/0x4c [ 1471.364227] __do_softirq+0x168/0x418 [ 1471.367875] __irq_exit_rcu+0x120/0x170 [ 1471.371698] irq_exit_rcu+0x18/0x24 [ 1471.375173] el1_interrupt+0x38/0x70 [ 1471.378736] el1h_64_irq_handler+0x18/0x24 [ 1471.382819] el1h_64_irq+0x68/0x6c [ 1471.386207] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1471.390985] __wake_up_common_lock+0x84/0xac [ 1471.395242] __wake_up+0x20/0x30 [ 1471.398458] __sbq_wake_up+0x13c/0x1f0 [ 1471.402194] sbitmap_queue_clear+0x74/0xc0 [ 1471.406278] blk_mq_put_tag+0x44/0x70 [ 1471.409927] __blk_mq_free_request+0x88/0xf0 [ 1471.414184] blk_mq_free_request+0xc0/0x160 [ 1471.418354] bfq_insert_request.isra.0+0x204/0x270 [ 1471.423132] bfq_insert_requests+0x70/0x90 [ 1471.427216] blk_mq_sched_insert_requests+0xac/0x250 [ 1471.432167] blk_mq_flush_plug_list+0x110/0x2cc [ 1471.436685] __blk_flush_plug+0x100/0x140 [ 1471.440681] blk_finish_plug+0x40/0x25c [ 1471.444504] __iomap_dio_rw+0x1b4/0x500 [ 1471.448327] iomap_dio_rw+0x18/0x50 [ 1471.451802] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1471.457153] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1471.461895] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1471.466939] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1471.472069] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1471.477199] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1471.482585] process_one_work+0x1ec/0x470 [ 1471.486581] worker_thread+0x74/0x410 [ 1471.490230] kthread+0xec/0x100 [ 1471.493359] ret_from_fork+0x10/0x20 [ 1483.214969] watchdog: BUG: soft lockup - CPU#68 stuck for 987s! [swapper/68:0] [ 1483.222177] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1483.222193] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1483.309980] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1483.317796] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1483.327086] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1483.334034] pc : __sbq_wake_up+0x4/0x1f0 [ 1483.337944] lr : sbitmap_queue_clear+0x74/0xc0 [ 1483.342374] sp : ffff800008843900 [ 1483.345675] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1483.352797] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1483.359919] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1483.367041] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1483.374162] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1483.381284] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1483.388406] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1483.395528] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1483.402650] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1483.409772] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1483.416894] Call trace: [ 1483.419327] __sbq_wake_up+0x4/0x1f0 [ 1483.422890] blk_mq_put_tag+0x44/0x70 [ 1483.426540] __blk_mq_free_request+0x88/0xf0 [ 1483.430796] blk_mq_free_request+0xc0/0x160 [ 1483.434966] __blk_mq_end_request+0x78/0x150 [ 1483.439223] scsi_end_request+0x114/0x260 [ 1483.443220] scsi_io_completion+0x64/0x174 [ 1483.447303] scsi_finish_command+0xe0/0x120 [ 1483.451473] scsi_complete+0x8c/0x1c4 [ 1483.455122] blk_complete_reqs+0x5c/0x74 [ 1483.459032] blk_done_softirq+0x38/0x4c [ 1483.462855] __do_softirq+0x168/0x418 [ 1483.466504] __irq_exit_rcu+0x120/0x170 [ 1483.470326] irq_exit_rcu+0x18/0x24 [ 1483.473802] el1_interrupt+0x38/0x70 [ 1483.477364] el1h_64_irq_handler+0x18/0x24 [ 1483.481448] el1h_64_irq+0x68/0x6c [ 1483.484836] cpuidle_enter_state+0xc4/0x4e0 [ 1483.489006] cpuidle_enter+0x40/0x60 [ 1483.492569] cpuidle_idle_call+0x118/0x1b0 [ 1483.496652] do_idle+0xac/0x100 [ 1483.499781] cpu_startup_entry+0x30/0x34 [ 1483.503690] secondary_start_kernel+0xd8/0xf0 [ 1483.508034] __secondary_switched+0xb0/0xb4 [ 1495.104892] watchdog: BUG: soft lockup - CPU#8 stuck for 998s! [kworker/8:12:19399] [ 1495.112535] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1495.112551] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1495.200339] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1495.208588] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1495.217879] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1495.224047] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1495.230994] pc : __sbq_wake_up+0x4/0x1f0 [ 1495.234904] lr : sbitmap_queue_clear+0x74/0xc0 [ 1495.239335] sp : ffff80000d79b2d0 [ 1495.242637] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1495.249759] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1495.256881] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1495.264003] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1495.271125] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1495.278247] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1495.285369] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1495.292490] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1495.299612] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1495.306734] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 1495.313856] Call trace: [ 1495.316289] __sbq_wake_up+0x4/0x1f0 [ 1495.319851] blk_mq_put_tag+0x44/0x70 [ 1495.323500] __blk_mq_free_request+0x88/0xf0 [ 1495.327757] blk_mq_free_request+0xc0/0x160 [ 1495.331927] __blk_mq_end_request+0x78/0x150 [ 1495.336184] scsi_end_request+0x114/0x260 [ 1495.340180] scsi_io_completion+0x64/0x174 [ 1495.344263] scsi_finish_command+0xe0/0x120 [ 1495.348434] scsi_complete+0x8c/0x1c4 [ 1495.352083] blk_complete_reqs+0x5c/0x74 [ 1495.355992] blk_done_softirq+0x38/0x4c [ 1495.359815] __do_softirq+0x168/0x418 [ 1495.363465] __irq_exit_rcu+0x120/0x170 [ 1495.367287] irq_exit_rcu+0x18/0x24 [ 1495.370763] el1_interrupt+0x38/0x70 [ 1495.374325] el1h_64_irq_handler+0x18/0x24 [ 1495.378408] el1h_64_irq+0x68/0x6c [ 1495.381797] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1495.386574] __wake_up_common_lock+0x84/0xac [ 1495.390832] __wake_up+0x20/0x30 [ 1495.394047] __sbq_wake_up+0x13c/0x1f0 [ 1495.397783] sbitmap_queue_clear+0x74/0xc0 [ 1495.401866] blk_mq_put_tag+0x44/0x70 [ 1495.405516] __blk_mq_free_request+0x88/0xf0 [ 1495.409772] blk_mq_free_request+0xc0/0x160 [ 1495.413942] bfq_insert_request.isra.0+0x204/0x270 [ 1495.418720] bfq_insert_requests+0x70/0x90 [ 1495.422804] blk_mq_sched_insert_requests+0xac/0x250 [ 1495.427756] blk_mq_flush_plug_list+0x110/0x2cc [ 1495.432274] __blk_flush_plug+0x100/0x140 [ 1495.436270] blk_finish_plug+0x40/0x25c [ 1495.440093] __iomap_dio_rw+0x1b4/0x500 [ 1495.443916] iomap_dio_rw+0x18/0x50 [ 1495.447391] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1495.452742] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1495.457484] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1495.462528] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1495.467658] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1495.472788] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1495.478174] process_one_work+0x1ec/0x470 [ 1495.482170] worker_thread+0x74/0x410 [ 1495.485819] kthread+0xec/0x100 [ 1495.488948] ret_from_fork+0x10/0x20 [ 1507.214815] watchdog: BUG: soft lockup - CPU#68 stuck for 1009s! [swapper/68:0] [ 1507.222110] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1507.222126] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1507.309913] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1507.317728] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1507.327019] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1507.333967] pc : __sbq_wake_up+0x20/0x1f0 [ 1507.337964] lr : sbitmap_queue_clear+0x74/0xc0 [ 1507.342395] sp : ffff8000088438b0 [ 1507.345696] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1507.352818] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1507.359940] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1507.367062] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1507.374185] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1507.381307] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1507.388429] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1507.395551] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1507.402673] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1507.409795] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1507.416917] Call trace: [ 1507.419350] __sbq_wake_up+0x20/0x1f0 [ 1507.422999] sbitmap_queue_clear+0x74/0xc0 [ 1507.427082] blk_mq_put_tag+0x44/0x70 [ 1507.430732] __blk_mq_free_request+0x88/0xf0 [ 1507.434989] blk_mq_free_request+0xc0/0x160 [ 1507.439158] __blk_mq_end_request+0x78/0x150 [ 1507.443415] scsi_end_request+0x114/0x260 [ 1507.447411] scsi_io_completion+0x64/0x174 [ 1507.451495] scsi_finish_command+0xe0/0x120 [ 1507.455665] scsi_complete+0x8c/0x1c4 [ 1507.459314] blk_complete_reqs+0x5c/0x74 [ 1507.463224] blk_done_softirq+0x38/0x4c [ 1507.467047] __do_softirq+0x168/0x418 [ 1507.470696] __irq_exit_rcu+0x120/0x170 [ 1507.474519] irq_exit_rcu+0x18/0x24 [ 1507.477995] el1_interrupt+0x38/0x70 [ 1507.481557] el1h_64_irq_handler+0x18/0x24 [ 1507.485640] el1h_64_irq+0x68/0x6c [ 1507.489028] cpuidle_enter_state+0xc4/0x4e0 [ 1507.493198] cpuidle_enter+0x40/0x60 [ 1507.496760] cpuidle_idle_call+0x118/0x1b0 [ 1507.500844] do_idle+0xac/0x100 [ 1507.503973] cpu_startup_entry+0x30/0x34 [ 1507.507882] secondary_start_kernel+0xd8/0xf0 [ 1507.512226] __secondary_switched+0xb0/0xb4 [ 1519.104739] watchdog: BUG: soft lockup - CPU#8 stuck for 1020s! [kworker/8:12:19399] [ 1519.112468] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1519.112485] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1519.200272] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1519.208522] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1519.217813] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1519.223981] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1519.230929] pc : __sbq_wake_up+0x40/0x1f0 [ 1519.234925] lr : sbitmap_queue_clear+0x74/0xc0 [ 1519.239355] sp : ffff80000d79b280 [ 1519.242657] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1519.249779] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1519.256902] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1519.264024] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1519.271146] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1519.278268] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1519.285390] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1519.292512] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1519.299634] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1519.306755] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 1519.313877] Call trace: [ 1519.316310] __sbq_wake_up+0x40/0x1f0 [ 1519.319960] sbitmap_queue_clear+0x74/0xc0 [ 1519.324043] blk_mq_put_tag+0x44/0x70 [ 1519.327693] __blk_mq_free_request+0x88/0xf0 [ 1519.331950] blk_mq_free_request+0xc0/0x160 [ 1519.336119] __blk_mq_end_request+0x78/0x150 [ 1519.340376] scsi_end_request+0x114/0x260 [ 1519.344372] scsi_io_completion+0x64/0x174 [ 1519.348456] scsi_finish_command+0xe0/0x120 [ 1519.352626] scsi_complete+0x8c/0x1c4 [ 1519.356275] blk_complete_reqs+0x5c/0x74 [ 1519.360185] blk_done_softirq+0x38/0x4c [ 1519.364007] __do_softirq+0x168/0x418 [ 1519.367656] __irq_exit_rcu+0x120/0x170 [ 1519.371479] irq_exit_rcu+0x18/0x24 [ 1519.374954] el1_interrupt+0x38/0x70 [ 1519.378516] el1h_64_irq_handler+0x18/0x24 [ 1519.382599] el1h_64_irq+0x68/0x6c [ 1519.385988] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1519.390765] __wake_up_common_lock+0x84/0xac [ 1519.395023] __wake_up+0x20/0x30 [ 1519.398238] __sbq_wake_up+0x13c/0x1f0 [ 1519.401974] sbitmap_queue_clear+0x74/0xc0 [ 1519.406058] blk_mq_put_tag+0x44/0x70 [ 1519.409707] __blk_mq_free_request+0x88/0xf0 [ 1519.413964] blk_mq_free_request+0xc0/0x160 [ 1519.418134] bfq_insert_request.isra.0+0x204/0x270 [ 1519.422912] bfq_insert_requests+0x70/0x90 [ 1519.426996] blk_mq_sched_insert_requests+0xac/0x250 [ 1519.431950] blk_mq_flush_plug_list+0x110/0x2cc [ 1519.436467] __blk_flush_plug+0x100/0x140 [ 1519.440464] blk_finish_plug+0x40/0x25c [ 1519.444286] __iomap_dio_rw+0x1b4/0x500 [ 1519.448109] iomap_dio_rw+0x18/0x50 [ 1519.451585] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1519.456935] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1519.461678] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1519.466722] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1519.471852] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1519.476981] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1519.482367] process_one_work+0x1ec/0x470 [ 1519.486363] worker_thread+0x74/0x410 [ 1519.490012] kthread+0xec/0x100 [ 1519.493141] ret_from_fork+0x10/0x20 [ 1531.214661] watchdog: BUG: soft lockup - CPU#68 stuck for 1032s! [swapper/68:0] [ 1531.221956] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1531.221972] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1531.309759] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1531.317574] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1531.326865] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1531.333813] pc : __sbq_wake_up+0x84/0x1f0 [ 1531.337809] lr : sbitmap_queue_clear+0x74/0xc0 [ 1531.342239] sp : ffff8000088438b0 [ 1531.345541] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1531.352663] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1531.359784] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1531.366906] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1531.374028] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1531.381150] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1531.388272] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1531.395394] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1531.402515] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1531.409638] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1531.416760] Call trace: [ 1531.419192] __sbq_wake_up+0x84/0x1f0 [ 1531.422841] sbitmap_queue_clear+0x74/0xc0 [ 1531.426924] blk_mq_put_tag+0x44/0x70 [ 1531.430574] __blk_mq_free_request+0x88/0xf0 [ 1531.434831] blk_mq_free_request+0xc0/0x160 [ 1531.439002] __blk_mq_end_request+0x78/0x150 [ 1531.443259] scsi_end_request+0x114/0x260 [ 1531.447255] scsi_io_completion+0x64/0x174 [ 1531.451338] scsi_finish_command+0xe0/0x120 [ 1531.455508] scsi_complete+0x8c/0x1c4 [ 1531.459157] blk_complete_reqs+0x5c/0x74 [ 1531.463067] blk_done_softirq+0x38/0x4c [ 1531.466890] __do_softirq+0x168/0x418 [ 1531.470538] __irq_exit_rcu+0x120/0x170 [ 1531.474361] irq_exit_rcu+0x18/0x24 [ 1531.477836] el1_interrupt+0x38/0x70 [ 1531.481399] el1h_64_irq_handler+0x18/0x24 [ 1531.485482] el1h_64_irq+0x68/0x6c [ 1531.488870] cpuidle_enter_state+0xc4/0x4e0 [ 1531.493040] cpuidle_enter+0x40/0x60 [ 1531.496602] cpuidle_idle_call+0x118/0x1b0 [ 1531.500686] do_idle+0xac/0x100 [ 1531.503815] cpu_startup_entry+0x30/0x34 [ 1531.507724] secondary_start_kernel+0xd8/0xf0 [ 1531.512068] __secondary_switched+0xb0/0xb4 [ 1537.004633] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 108605 jiffies s: 2093 root: 0x11/. [ 1537.016371] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1537.025580] Task dump for CPU 8: [ 1537.028794] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1537.038696] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1537.044866] Call trace: [ 1537.047299] __switch_to+0xd4/0x100 [ 1537.050775] 0xffff6c0200000080 [ 1537.053904] Task dump for CPU 68: [ 1537.057207] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1537.067108] Call trace: [ 1537.069541] __switch_to+0xd4/0x100 [ 1537.073017] 0x0 [ 1543.104582] watchdog: BUG: soft lockup - CPU#8 stuck for 1043s! [kworker/8:12:19399] [ 1543.112311] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1543.112328] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1543.200115] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1543.208365] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1543.217656] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1543.223823] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1543.230771] pc : __sbq_wake_up+0x7c/0x1f0 [ 1543.234768] lr : sbitmap_queue_clear+0x74/0xc0 [ 1543.239199] sp : ffff80000d79b2d0 [ 1543.242500] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1543.249622] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1543.256744] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1543.263866] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1543.270988] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1543.278110] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1543.285232] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1543.292354] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1543.299476] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1543.306599] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 1543.313721] Call trace: [ 1543.316154] __sbq_wake_up+0x7c/0x1f0 [ 1543.319803] blk_mq_put_tag+0x44/0x70 [ 1543.323453] __blk_mq_free_request+0x88/0xf0 [ 1543.327710] blk_mq_free_request+0xc0/0x160 [ 1543.331880] __blk_mq_end_request+0x78/0x150 [ 1543.336136] scsi_end_request+0x114/0x260 [ 1543.340133] scsi_io_completion+0x64/0x174 [ 1543.344216] scsi_finish_command+0xe0/0x120 [ 1543.348386] scsi_complete+0x8c/0x1c4 [ 1543.352035] blk_complete_reqs+0x5c/0x74 [ 1543.355945] blk_done_softirq+0x38/0x4c [ 1543.359767] __do_softirq+0x168/0x418 [ 1543.363417] __irq_exit_rcu+0x120/0x170 [ 1543.367240] irq_exit_rcu+0x18/0x24 [ 1543.370716] el1_interrupt+0x38/0x70 [ 1543.374278] el1h_64_irq_handler+0x18/0x24 [ 1543.378362] el1h_64_irq+0x68/0x6c [ 1543.381750] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1543.386528] __wake_up_common_lock+0x84/0xac [ 1543.390786] __wake_up+0x20/0x30 [ 1543.394001] __sbq_wake_up+0x13c/0x1f0 [ 1543.397737] sbitmap_queue_clear+0x74/0xc0 [ 1543.401821] blk_mq_put_tag+0x44/0x70 [ 1543.405470] __blk_mq_free_request+0x88/0xf0 [ 1543.409727] blk_mq_free_request+0xc0/0x160 [ 1543.413897] bfq_insert_request.isra.0+0x204/0x270 [ 1543.418675] bfq_insert_requests+0x70/0x90 [ 1543.422759] blk_mq_sched_insert_requests+0xac/0x250 [ 1543.427711] blk_mq_flush_plug_list+0x110/0x2cc [ 1543.432229] __blk_flush_plug+0x100/0x140 [ 1543.436225] blk_finish_plug+0x40/0x25c [ 1543.440048] __iomap_dio_rw+0x1b4/0x500 [ 1543.443870] iomap_dio_rw+0x18/0x50 [ 1543.447346] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1543.452696] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1543.457439] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1543.462482] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1543.467612] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1543.472742] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1543.478128] process_one_work+0x1ec/0x470 [ 1543.482124] worker_thread+0x74/0x410 [ 1543.485773] kthread+0xec/0x100 [ 1543.488902] ret_from_fork+0x10/0x20 [ 1555.214502] watchdog: BUG: soft lockup - CPU#68 stuck for 1054s! [swapper/68:0] [ 1555.221797] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1555.221813] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1555.309601] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1555.317417] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1555.326707] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1555.333655] pc : __sbq_wake_up+0x40/0x1f0 [ 1555.337651] lr : sbitmap_queue_clear+0x74/0xc0 [ 1555.342082] sp : ffff8000088438b0 [ 1555.345384] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1555.352505] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1555.359627] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1555.366749] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1555.373871] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1555.380993] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1555.388114] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1555.395237] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1555.402359] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1555.409480] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 1555.416602] Call trace: [ 1555.419036] __sbq_wake_up+0x40/0x1f0 [ 1555.422685] sbitmap_queue_clear+0x74/0xc0 [ 1555.426768] blk_mq_put_tag+0x44/0x70 [ 1555.430418] __blk_mq_free_request+0x88/0xf0 [ 1555.434675] blk_mq_free_request+0xc0/0x160 [ 1555.438845] __blk_mq_end_request+0x78/0x150 [ 1555.443102] scsi_end_request+0x114/0x260 [ 1555.447098] scsi_io_completion+0x64/0x174 [ 1555.451182] scsi_finish_command+0xe0/0x120 [ 1555.455352] scsi_complete+0x8c/0x1c4 [ 1555.459001] blk_complete_reqs+0x5c/0x74 [ 1555.462911] blk_done_softirq+0x38/0x4c [ 1555.466734] __do_softirq+0x168/0x418 [ 1555.470382] __irq_exit_rcu+0x120/0x170 [ 1555.474205] irq_exit_rcu+0x18/0x24 [ 1555.477681] el1_interrupt+0x38/0x70 [ 1555.481243] el1h_64_irq_handler+0x18/0x24 [ 1555.485326] el1h_64_irq+0x68/0x6c [ 1555.488714] cpuidle_enter_state+0xc4/0x4e0 [ 1555.492885] cpuidle_enter+0x40/0x60 [ 1555.496447] cpuidle_idle_call+0x118/0x1b0 [ 1555.500530] do_idle+0xac/0x100 [ 1555.503659] cpu_startup_entry+0x30/0x34 [ 1555.507568] secondary_start_kernel+0xd8/0xf0 [ 1555.511912] __secondary_switched+0xb0/0xb4 [ 1566.884424] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1566.890158] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=56675 [ 1566.899279] (t=114199 jiffies g=36445 q=33763 ncpus=128) [ 1566.904664] Task dump for CPU 8: [ 1566.907879] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1566.917779] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1566.923947] Call trace: [ 1566.926379] __switch_to+0xd4/0x100 [ 1566.929856] 0xffff6c0200000080 [ 1566.932984] Task dump for CPU 68: [ 1566.936285] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1566.946185] Call trace: [ 1566.948618] dump_backtrace+0xe8/0x130 [ 1566.952354] show_stack+0x20/0x60 [ 1566.955656] sched_show_task+0x160/0x190 [ 1566.959565] dump_cpu_task+0x4c/0x78 [ 1566.963128] rcu_dump_cpu_stacks+0xc4/0x150 [ 1566.967298] print_cpu_stall+0x180/0x210 [ 1566.971208] check_cpu_stall+0x20c/0x2b4 [ 1566.975117] rcu_pending+0x4c/0x190 [ 1566.978593] rcu_sched_clock_irq+0x80/0x204 [ 1566.982763] update_process_times+0x7c/0xc0 [ 1566.986933] tick_sched_handle+0x38/0x74 [ 1566.990843] tick_sched_timer+0x60/0xc0 [ 1566.994666] __hrtimer_run_queues+0x170/0x300 [ 1566.999009] hrtimer_interrupt+0xfc/0x260 [ 1567.003006] arch_timer_handler_phys+0x34/0x50 [ 1567.007437] handle_percpu_devid_irq+0xac/0x240 [ 1567.011954] generic_handle_domain_irq+0x34/0x50 [ 1567.016559] gic_handle_irq+0x5c/0xe0 [ 1567.020208] call_on_irq_stack+0x2c/0x38 [ 1567.024118] do_interrupt_handler+0xdc/0xe0 [ 1567.028288] el1_interrupt+0x34/0x70 [ 1567.031850] el1h_64_irq_handler+0x18/0x24 [ 1567.035934] el1h_64_irq+0x68/0x6c [ 1567.039322] __sbq_wake_up+0x1c8/0x1f0 [ 1567.043058] sbitmap_queue_clear+0x74/0xc0 [ 1567.047141] blk_mq_put_tag+0x44/0x70 [ 1567.050791] __blk_mq_free_request+0x88/0xf0 [ 1567.055048] blk_mq_free_request+0xc0/0x160 [ 1567.059217] __blk_mq_end_request+0x78/0x150 [ 1567.063474] scsi_end_request+0x114/0x260 [ 1567.067470] scsi_io_completion+0x64/0x174 [ 1567.071554] scsi_finish_command+0xe0/0x120 [ 1567.075724] scsi_complete+0x8c/0x1c4 [ 1567.079373] blk_complete_reqs+0x5c/0x74 [ 1567.083283] blk_done_softirq+0x38/0x4c [ 1567.087105] __do_softirq+0x168/0x418 [ 1567.090754] __irq_exit_rcu+0x120/0x170 [ 1567.094577] irq_exit_rcu+0x18/0x24 [ 1567.098052] el1_interrupt+0x38/0x70 [ 1567.101615] el1h_64_irq_handler+0x18/0x24 [ 1567.104422] watchdog: BUG: soft lockup - CPU#8 stuck for 1065s! [kworker/8:12:19399] [ 1567.105698] el1h_64_irq+0x68/0x6c [ 1567.113426] Modules linked in: nvme_loop [ 1567.116813] cpuidle_enter_state+0xc4/0x4e0 [ 1567.116814] nvme_fabrics nvmet nls_utf8 [ 1567.124893] cpuidle_enter+0x40/0x60 [ 1567.128802] hfsplus hfs isofs [ 1567.132363] cpuidle_idle_call+0x118/0x1b0 [ 1567.135405] nvme_core nvme_common loop [ 1567.139487] do_idle+0xac/0x100 [ 1567.143310] tls rfkill [ 1567.146438] cpu_startup_entry+0x30/0x34 [ 1567.148871] sunrpc vfat fat [ 1567.152780] secondary_start_kernel+0xd8/0xf0 [ 1567.155648] acpi_ipmi ipmi_ssif cppc_cpufreq [ 1567.159991] __secondary_switched+0xb0/0xb4 [ 1567.164335] ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1567.168512] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1567.242324] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1567.250574] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1567.259865] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1567.266032] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1567.272980] pc : __sbq_wake_up+0x4/0x1f0 [ 1567.276889] lr : sbitmap_queue_clear+0x74/0xc0 [ 1567.281320] sp : ffff80000d79b2d0 [ 1567.284621] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1567.291744] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1567.298865] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 1567.305988] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1567.313110] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1567.320232] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1567.327354] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1567.334476] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1567.341598] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1567.348720] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 1567.355842] Call trace: [ 1567.358276] __sbq_wake_up+0x4/0x1f0 [ 1567.361838] blk_mq_put_tag+0x44/0x70 [ 1567.365488] __blk_mq_free_request+0x88/0xf0 [ 1567.369745] blk_mq_free_request+0xc0/0x160 [ 1567.373915] __blk_mq_end_request+0x78/0x150 [ 1567.378173] scsi_end_request+0x114/0x260 [ 1567.382169] scsi_io_completion+0x64/0x174 [ 1567.386252] scsi_finish_command+0xe0/0x120 [ 1567.390423] scsi_complete+0x8c/0x1c4 [ 1567.394072] blk_complete_reqs+0x5c/0x74 [ 1567.397981] blk_done_softirq+0x38/0x4c [ 1567.401804] __do_softirq+0x168/0x418 [ 1567.405453] __irq_exit_rcu+0x120/0x170 [ 1567.409276] irq_exit_rcu+0x18/0x24 [ 1567.412751] el1_interrupt+0x38/0x70 [ 1567.416313] el1h_64_irq_handler+0x18/0x24 [ 1567.420397] el1h_64_irq+0x68/0x6c [ 1567.423785] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1567.428563] __wake_up_common_lock+0x84/0xac [ 1567.432820] __wake_up+0x20/0x30 [ 1567.436036] __sbq_wake_up+0x13c/0x1f0 [ 1567.439773] sbitmap_queue_clear+0x74/0xc0 [ 1567.443856] blk_mq_put_tag+0x44/0x70 [ 1567.447506] __blk_mq_free_request+0x88/0xf0 [ 1567.451763] blk_mq_free_request+0xc0/0x160 [ 1567.455932] bfq_insert_request.isra.0+0x204/0x270 [ 1567.460711] bfq_insert_requests+0x70/0x90 [ 1567.464794] blk_mq_sched_insert_requests+0xac/0x250 [ 1567.469746] blk_mq_flush_plug_list+0x110/0x2cc [ 1567.474264] __blk_flush_plug+0x100/0x140 [ 1567.478260] blk_finish_plug+0x40/0x25c [ 1567.482083] __iomap_dio_rw+0x1b4/0x500 [ 1567.485905] iomap_dio_rw+0x18/0x50 [ 1567.489381] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1567.494731] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1567.499474] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1567.504517] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1567.509647] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1567.514777] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1567.520163] process_one_work+0x1ec/0x470 [ 1567.524159] worker_thread+0x74/0x410 [ 1567.527808] kthread+0xec/0x100 [ 1567.530937] ret_from_fork+0x10/0x20 [ 1591.104262] watchdog: BUG: soft lockup - CPU#8 stuck for 1087s! [kworker/8:12:19399] [ 1591.111991] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1591.112008] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1591.199795] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1591.208044] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1591.214262] watchdog: BUG: soft lockup - CPU#68 stuck for 1087s! [swapper/68:0] [ 1591.217335] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1591.224629] Modules linked in: nvme_loop [ 1591.230795] [ 1591.230796] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1591.234704] nvme_fabrics [ 1591.236182] pc : __sbq_wake_up+0x84/0x1f0 [ 1591.243128] nvmet [ 1591.245735] lr : sbitmap_queue_clear+0x74/0xc0 [ 1591.249730] nls_utf8 [ 1591.251729] sp : ffff80000d79b280 [ 1591.256158] hfsplus [ 1591.258418] x29: ffff80000d79b280 [ 1591.261718] hfs [ 1591.263891] x28: ffffd5becb6aec50 [ 1591.267191] isofs [ 1591.269017] x27: ffffd5beca4f7000 [ 1591.272405] nvme_core [ 1591.274403] [ 1591.277791] nvme_common [ 1591.280137] x26: ffffd5becaf68a98 [ 1591.281615] loop [ 1591.284135] x25: ffffd5becb6a60c0 [ 1591.287435] tls [ 1591.289347] x24: fffffbffef180dc0 [ 1591.292735] rfkill [ 1591.294560] [ 1591.297948] sunrpc [ 1591.300033] x23: ffff07ff81d381b0 [ 1591.301511] vfat [ 1591.303596] x22: ffff80000d79b30c [ 1591.306897] fat [ 1591.308810] x21: ffff07ff84e93c00 [ 1591.312197] acpi_ipmi [ 1591.314022] [ 1591.317411] ipmi_ssif [ 1591.319757] x20: 00000000000001c0 [ 1591.321235] cppc_cpufreq [ 1591.323581] x19: ffff07ff91ccdb10 [ 1591.326882] ipmi_devintf [ 1591.329488] x18: ffffffffffffffff [ 1591.332876] igb [ 1591.335482] [ 1591.338870] ipmi_msghandler [ 1591.340695] x17: ffff32506c412000 [ 1591.342173] arm_cmn [ 1591.345039] x16: ffff800008044000 [ 1591.348340] arm_dmc620_pmu [ 1591.350513] x15: 0000000000004000 [ 1591.353901] arm_spe_pmu [ 1591.356680] [ 1591.360068] arm_dsu_pmu [ 1591.362588] x14: 010320bb28ec0000 [ 1591.364065] acpi_tad [ 1591.366585] x13: 0000000000000087 [ 1591.369886] fuse [ 1591.372145] x12: 071c71c71c71c71c [ 1591.375533] zram [ 1591.377445] [ 1591.380833] xfs [ 1591.382745] x11: 0000000000000040 [ 1591.384223] crct10dif_ce [ 1591.386049] x10: ffff07ff801a9c70 [ 1591.389350] polyval_ce [ 1591.391956] x9 : ffffd5bec9545fd8 [ 1591.395343] ast [ 1591.397776] [ 1591.401164] polyval_generic [ 1591.402989] x8 : ffff07ff80404b70 [ 1591.404467] drm_vram_helper [ 1591.407333] x7 : ffff07ff84e93dc8 [ 1591.410634] ghash_ce [ 1591.413502] x6 : 0000000000000008 [ 1591.416889] drm_ttm_helper [ 1591.419148] [ 1591.422536] sbsa_gwdt [ 1591.425316] x5 : ffff80000d043790 [ 1591.426794] ttm [ 1591.429140] x4 : ffff07ff84e93dc0 [ 1591.432440] xgene_hwmon [ 1591.434265] x3 : 0000000000000008 [ 1591.437653] Unloaded tainted modules: [ 1591.440172] [ 1591.443560] nvmet():1 [ 1591.447208] x2 : 0000000000000007 [ 1591.448686] nvme_fabrics():1 [ 1591.451032] x1 : ffff80000d79b30c [ 1591.454333] nvme_loop():1 [ 1591.457286] x0 : ffff07ff84e93dd0 [ 1591.460674] nvmet():1 [ 1591.463367] [ 1591.466754] nvme_fabrics():1 [ 1591.469100] Call trace: [ 1591.470578] nvme_loop():1 [ 1591.473532] __sbq_wake_up+0x84/0x1f0 [ 1591.475964] nvmet():1 [ 1591.478658] sbitmap_queue_clear+0x74/0xc0 [ 1591.482306] nvme_fabrics():1 [ 1591.484652] blk_mq_put_tag+0x44/0x70 [ 1591.488734] nvme_loop():1 [ 1591.491688] __blk_mq_free_request+0x88/0xf0 [ 1591.495336] nvmet():1 [ 1591.498029] blk_mq_free_request+0xc0/0x160 [ 1591.502284] nvme_fabrics():1 [ 1591.504631] __blk_mq_end_request+0x78/0x150 [ 1591.508800] nvme_loop():1 [ 1591.511753] scsi_end_request+0x114/0x260 [ 1591.516009] nvmet():1 [ 1591.518702] scsi_io_completion+0x64/0x174 [ 1591.522697] nvme_fabrics():1 [ 1591.525043] scsi_finish_command+0xe0/0x120 [ 1591.529125] nvme_loop():1 [ 1591.532079] scsi_complete+0x8c/0x1c4 [ 1591.536247] nvmet():1 [ 1591.538941] blk_complete_reqs+0x5c/0x74 [ 1591.542588] nvme_fabrics():1 [ 1591.544934] blk_done_softirq+0x38/0x4c [ 1591.548843] nvme_loop():1 [ 1591.551796] __do_softirq+0x168/0x418 [ 1591.555618] nvmet():1 [ 1591.558311] __irq_exit_rcu+0x120/0x170 [ 1591.561959] nvme_fabrics():1 [ 1591.564305] irq_exit_rcu+0x18/0x24 [ 1591.568127] nvme_loop():1 [ 1591.571080] el1_interrupt+0x38/0x70 [ 1591.574555] nvmet():1 [ 1591.577248] el1h_64_irq_handler+0x18/0x24 [ 1591.580809] nvme_fabrics():1 [ 1591.583155] el1h_64_irq+0x68/0x6c [ 1591.587237] nvme_loop():1 [ 1591.590190] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1591.593578] nvmet():1 [ 1591.596271] __wake_up_common_lock+0x84/0xac [ 1591.601047] nvme_fabrics():1 [ 1591.603393] __wake_up+0x20/0x30 [ 1591.607649] nvme_loop():1 [ 1591.610603] __sbq_wake_up+0x13c/0x1f0 [ 1591.613817] nvmet():1 [ 1591.616511] sbitmap_queue_clear+0x74/0xc0 [ 1591.620246] nvme_fabrics():1 [ 1591.622592] blk_mq_put_tag+0x44/0x70 [ 1591.626674] nvme_loop():1 [ 1591.629628] __blk_mq_free_request+0x88/0xf0 [ 1591.633275] null_blk():1 [ 1591.635969] blk_mq_free_request+0xc0/0x160 [ 1591.640224] null_blk():1 [ 1591.642831] bfq_insert_request.isra.0+0x204/0x270 [ 1591.646999] null_blk():1 [ 1591.649606] bfq_insert_requests+0x70/0x90 [ 1591.654382] null_blk():1 [ 1591.656989] blk_mq_sched_insert_requests+0xac/0x250 [ 1591.661071] null_blk():1 [ 1591.663677] blk_mq_flush_plug_list+0x110/0x2cc [ 1591.668627] null_blk():1 [ 1591.671233] __blk_flush_plug+0x100/0x140 [ 1591.675749] null_blk():1 [ 1591.678357] blk_finish_plug+0x40/0x25c [ 1591.682352] scsi_debug():1 [ 1591.684958] __iomap_dio_rw+0x1b4/0x500 [ 1591.688780] scsi_debug():1 [ 1591.691560] iomap_dio_rw+0x18/0x50 [ 1591.695382] ipmi_si():1 [ 1591.698162] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1591.701636] ipmi_si():1 [ 1591.704156] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1591.709453] [last unloaded: nvmet] [ 1591.711973] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1591.716662] [ 1591.720137] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1591.725174] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1591.726652] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1591.731777] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1591.739592] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1591.744716] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1591.754006] process_one_work+0x1ec/0x470 [ 1591.759390] pc : __sbq_wake_up+0x8/0x1f0 [ 1591.766337] worker_thread+0x74/0x410 [ 1591.770332] lr : sbitmap_queue_clear+0x74/0xc0 [ 1591.774241] kthread+0xec/0x100 [ 1591.777889] sp : ffff8000088438b0 [ 1591.782318] ret_from_fork+0x10/0x20 [ 1591.785445] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1591.799429] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1591.806551] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1591.813672] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1591.820794] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1591.827917] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1591.835040] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1591.842162] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1591.849284] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1591.856406] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1591.863527] Call trace: [ 1591.865960] __sbq_wake_up+0x8/0x1f0 [ 1591.869522] blk_mq_put_tag+0x44/0x70 [ 1591.873172] __blk_mq_free_request+0x88/0xf0 [ 1591.877429] blk_mq_free_request+0xc0/0x160 [ 1591.881599] __blk_mq_end_request+0x78/0x150 [ 1591.885856] scsi_end_request+0x114/0x260 [ 1591.889853] scsi_io_completion+0x64/0x174 [ 1591.893936] scsi_finish_command+0xe0/0x120 [ 1591.898107] scsi_complete+0x8c/0x1c4 [ 1591.901756] blk_complete_reqs+0x5c/0x74 [ 1591.905665] blk_done_softirq+0x38/0x4c [ 1591.909488] __do_softirq+0x168/0x418 [ 1591.913137] __irq_exit_rcu+0x120/0x170 [ 1591.916960] irq_exit_rcu+0x18/0x24 [ 1591.920435] el1_interrupt+0x38/0x70 [ 1591.923998] el1h_64_irq_handler+0x18/0x24 [ 1591.928081] el1h_64_irq+0x68/0x6c [ 1591.931469] cpuidle_enter_state+0xc4/0x4e0 [ 1591.935639] cpuidle_enter+0x40/0x60 [ 1591.939201] cpuidle_idle_call+0x118/0x1b0 [ 1591.943285] do_idle+0xac/0x100 [ 1591.946413] cpu_startup_entry+0x30/0x34 [ 1591.950323] secondary_start_kernel+0xd8/0xf0 [ 1591.954668] __secondary_switched+0xb0/0xb4 [ 1603.564189] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 115261 jiffies s: 2093 root: 0x11/. [ 1603.575927] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1603.585136] Task dump for CPU 8: [ 1603.588350] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1603.598252] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1603.604422] Call trace: [ 1603.606855] __switch_to+0xd4/0x100 [ 1603.610332] 0xffff6c0200000080 [ 1603.613460] Task dump for CPU 68: [ 1603.616762] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1603.626664] Call trace: [ 1603.629097] __switch_to+0xd4/0x100 [ 1603.632573] 0x0 [ 1615.104102] watchdog: BUG: soft lockup - CPU#8 stuck for 1110s! [kworker/8:12:19399] [ 1615.111832] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1615.111849] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1615.199636] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1615.207886] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1615.214102] watchdog: BUG: soft lockup - CPU#68 stuck for 1110s! [swapper/68:0] [ 1615.217177] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1615.224471] Modules linked in: nvme_loop [ 1615.230637] [ 1615.230637] nvme_fabrics [ 1615.234546] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1615.236024] nvmet [ 1615.238631] pc : __sbq_wake_up+0x20/0x1f0 [ 1615.245578] nls_utf8 [ 1615.247577] lr : sbitmap_queue_clear+0x74/0xc0 [ 1615.251572] hfsplus [ 1615.253832] sp : ffff80000d79b280 [ 1615.258261] hfs [ 1615.260434] x29: ffff80000d79b280 [ 1615.263735] isofs [ 1615.265560] x28: ffffd5becb6aec50 [ 1615.268861] nvme_core [ 1615.270860] x27: ffffd5beca4f7000 [ 1615.274248] nvme_common [ 1615.276593] [ 1615.279981] loop [ 1615.282500] x26: ffffd5becaf68a98 [ 1615.283978] tls [ 1615.285891] x25: ffffd5becb6a60c0 [ 1615.289192] rfkill [ 1615.291017] x24: fffffbffef180dc0 [ 1615.294404] sunrpc [ 1615.296490] [ 1615.299877] vfat [ 1615.301963] x23: ffff07ff81d381b0 [ 1615.303441] fat [ 1615.305353] x22: 0000000000000024 [ 1615.308653] acpi_ipmi [ 1615.310478] x21: 0000000000000008 [ 1615.313866] ipmi_ssif [ 1615.316212] [ 1615.319599] cppc_cpufreq [ 1615.321945] x20: 0000000000000024 [ 1615.323424] ipmi_devintf [ 1615.326031] x19: ffff07ff91ccdb10 [ 1615.329332] igb [ 1615.331938] x18: ffffffffffffffff [ 1615.335326] ipmi_msghandler [ 1615.337153] [ 1615.340540] arm_cmn [ 1615.343407] x17: ffff32506c412000 [ 1615.344885] arm_dmc620_pmu [ 1615.347057] x16: ffff800008044000 [ 1615.350358] arm_spe_pmu [ 1615.353138] x15: 0000000000004000 [ 1615.356525] arm_dsu_pmu [ 1615.359045] [ 1615.362432] acpi_tad [ 1615.364952] x14: 010320bb28ec0000 [ 1615.366430] fuse [ 1615.368689] x13: 0000000000000087 [ 1615.371990] zram [ 1615.373902] x12: 071c71c71c71c71c [ 1615.377289] xfs [ 1615.379201] [ 1615.382589] crct10dif_ce [ 1615.384414] x11: 0000000000000040 [ 1615.385891] polyval_ce [ 1615.388498] x10: ffff07ff801a9c70 [ 1615.391799] ast [ 1615.394231] x9 : ffffd5bec9545fd8 [ 1615.397619] polyval_generic [ 1615.399446] [ 1615.402833] drm_vram_helper [ 1615.405700] x8 : ffff07ff80404b70 [ 1615.407178] ghash_ce [ 1615.410045] x7 : ffff07ff84e93dc8 [ 1615.413346] drm_ttm_helper [ 1615.415605] x6 : 0000000000000008 [ 1615.418993] sbsa_gwdt [ 1615.421772] [ 1615.425161] ttm [ 1615.427506] x5 : ffff80000d043790 [ 1615.428984] xgene_hwmon [ 1615.430809] x4 : ffff07ff84e93dc0 [ 1615.434110] Unloaded tainted modules: [ 1615.436630] x3 : 0000000000000008 [ 1615.440017] nvmet():1 [ 1615.443665] [ 1615.447053] nvme_fabrics():1 [ 1615.449399] x2 : 0000000000000007 [ 1615.450877] nvme_loop():1 [ 1615.453830] x1 : ffff80000d79b30c [ 1615.457131] nvmet():1 [ 1615.459825] x0 : 0000000000000001 [ 1615.463212] nvme_fabrics():1 [ 1615.465558] [ 1615.468946] nvme_loop():1 [ 1615.471899] Call trace: [ 1615.473377] nvmet():1 [ 1615.476071] __sbq_wake_up+0x20/0x1f0 [ 1615.478503] nvme_fabrics():1 [ 1615.480849] sbitmap_queue_clear+0x74/0xc0 [ 1615.484497] nvme_loop():1 [ 1615.487451] blk_mq_put_tag+0x44/0x70 [ 1615.491533] nvmet():1 [ 1615.494227] __blk_mq_free_request+0x88/0xf0 [ 1615.497876] nvme_fabrics():1 [ 1615.500222] blk_mq_free_request+0xc0/0x160 [ 1615.504478] nvme_loop():1 [ 1615.507431] __blk_mq_end_request+0x78/0x150 [ 1615.511600] nvmet():1 [ 1615.514295] scsi_end_request+0x114/0x260 [ 1615.518550] nvme_fabrics():1 [ 1615.520896] scsi_io_completion+0x64/0x174 [ 1615.524892] nvme_loop():1 [ 1615.527845] scsi_finish_command+0xe0/0x120 [ 1615.531927] nvmet():1 [ 1615.534620] scsi_complete+0x8c/0x1c4 [ 1615.538789] nvme_fabrics():1 [ 1615.541136] blk_complete_reqs+0x5c/0x74 [ 1615.544784] nvme_loop():1 [ 1615.547739] blk_done_softirq+0x38/0x4c [ 1615.551648] nvmet():1 [ 1615.554341] __do_softirq+0x168/0x418 [ 1615.558162] nvme_fabrics():1 [ 1615.560508] __irq_exit_rcu+0x120/0x170 [ 1615.564156] nvme_loop():1 [ 1615.567110] irq_exit_rcu+0x18/0x24 [ 1615.570931] nvmet():1 [ 1615.573625] el1_interrupt+0x38/0x70 [ 1615.577099] nvme_fabrics():1 [ 1615.579445] el1h_64_irq_handler+0x18/0x24 [ 1615.583006] nvme_loop():1 [ 1615.585960] el1h_64_irq+0x68/0x6c [ 1615.590042] nvmet():1 [ 1615.592735] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1615.596123] nvme_fabrics():1 [ 1615.598469] __wake_up_common_lock+0x84/0xac [ 1615.603245] nvme_loop():1 [ 1615.606199] __wake_up+0x20/0x30 [ 1615.610454] nvmet():1 [ 1615.613148] __sbq_wake_up+0x13c/0x1f0 [ 1615.616362] nvme_fabrics():1 [ 1615.618708] sbitmap_queue_clear+0x74/0xc0 [ 1615.622443] nvme_loop():1 [ 1615.625397] blk_mq_put_tag+0x44/0x70 [ 1615.629479] null_blk():1 [ 1615.632172] __blk_mq_free_request+0x88/0xf0 [ 1615.635820] null_blk():1 [ 1615.638426] blk_mq_free_request+0xc0/0x160 [ 1615.642682] null_blk():1 [ 1615.645288] bfq_insert_request.isra.0+0x204/0x270 [ 1615.649457] null_blk():1 [ 1615.652063] bfq_insert_requests+0x70/0x90 [ 1615.656840] null_blk():1 [ 1615.659446] blk_mq_sched_insert_requests+0xac/0x250 [ 1615.663528] null_blk():1 [ 1615.666135] blk_mq_flush_plug_list+0x110/0x2cc [ 1615.671085] null_blk():1 [ 1615.673691] __blk_flush_plug+0x100/0x140 [ 1615.678207] scsi_debug():1 [ 1615.680814] blk_finish_plug+0x40/0x25c [ 1615.684809] scsi_debug():1 [ 1615.687589] __iomap_dio_rw+0x1b4/0x500 [ 1615.691411] ipmi_si():1 [ 1615.694191] iomap_dio_rw+0x18/0x50 [ 1615.698013] ipmi_si():1 [ 1615.700532] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1615.704007] [last unloaded: nvmet] [ 1615.706526] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1615.711824] [ 1615.715299] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1615.719989] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1615.721466] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1615.726503] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1615.734318] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1615.739442] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1615.748732] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1615.753856] pc : __sbq_wake_up+0x8/0x1f0 [ 1615.760802] process_one_work+0x1ec/0x470 [ 1615.766186] lr : sbitmap_queue_clear+0x74/0xc0 [ 1615.770095] worker_thread+0x74/0x410 [ 1615.774090] sp : ffff8000088438b0 [ 1615.778519] kthread+0xec/0x100 [ 1615.782167] x29: ffff800008843900 [ 1615.785468] ret_from_fork+0x10/0x20 [ 1615.788595] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1615.800755] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1615.807877] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1615.814998] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1615.822120] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1615.829242] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1615.836364] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1615.843486] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1615.850607] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1615.857729] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1615.864851] Call trace: [ 1615.867284] __sbq_wake_up+0x8/0x1f0 [ 1615.870847] blk_mq_put_tag+0x44/0x70 [ 1615.874496] __blk_mq_free_request+0x88/0xf0 [ 1615.878753] blk_mq_free_request+0xc0/0x160 [ 1615.882923] __blk_mq_end_request+0x78/0x150 [ 1615.887180] scsi_end_request+0x114/0x260 [ 1615.891176] scsi_io_completion+0x64/0x174 [ 1615.895260] scsi_finish_command+0xe0/0x120 [ 1615.899430] scsi_complete+0x8c/0x1c4 [ 1615.903079] blk_complete_reqs+0x5c/0x74 [ 1615.906989] blk_done_softirq+0x38/0x4c [ 1615.910811] __do_softirq+0x168/0x418 [ 1615.914460] __irq_exit_rcu+0x120/0x170 [ 1615.918282] irq_exit_rcu+0x18/0x24 [ 1615.921757] el1_interrupt+0x38/0x70 [ 1615.925320] el1h_64_irq_handler+0x18/0x24 [ 1615.929403] el1h_64_irq+0x68/0x6c [ 1615.932791] cpuidle_enter_state+0xc4/0x4e0 [ 1615.936961] cpuidle_enter+0x40/0x60 [ 1615.940524] cpuidle_idle_call+0x118/0x1b0 [ 1615.944607] do_idle+0xac/0x100 [ 1615.947736] cpu_startup_entry+0x30/0x34 [ 1615.951645] secondary_start_kernel+0xd8/0xf0 [ 1615.955989] __secondary_switched+0xb0/0xb4 [ 1639.103942] watchdog: BUG: soft lockup - CPU#8 stuck for 1132s! [kworker/8:12:19399] [ 1639.111672] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1639.111689] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1639.199476] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1639.207725] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1639.213942] watchdog: BUG: soft lockup - CPU#68 stuck for 1132s! [swapper/68:0] [ 1639.217016] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1639.224310] Modules linked in: nvme_loop [ 1639.230478] [ 1639.230478] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1639.234387] nvme_fabrics [ 1639.235864] pc : __sbq_wake_up+0x58/0x1f0 [ 1639.242811] nvmet [ 1639.245417] lr : sbitmap_queue_clear+0x74/0xc0 [ 1639.249412] nls_utf8 [ 1639.251411] sp : ffff80000d79b280 [ 1639.255841] hfsplus [ 1639.258100] x29: ffff80000d79b280 [ 1639.261401] hfs [ 1639.263573] x28: ffffd5becb6aec50 [ 1639.266874] isofs [ 1639.268699] x27: ffffd5beca4f7000 [ 1639.272087] nvme_core [ 1639.274087] [ 1639.277474] nvme_common [ 1639.279821] x26: ffffd5becaf68a98 [ 1639.281299] loop [ 1639.283818] x25: ffffd5becb6a60c0 [ 1639.287119] tls [ 1639.289031] x24: fffffbffef180dc0 [ 1639.292419] rfkill [ 1639.294244] [ 1639.297631] sunrpc [ 1639.299717] x23: ffff07ff81d381b0 [ 1639.301194] vfat [ 1639.303280] x22: ffff80000d79b30c [ 1639.306581] fat [ 1639.308493] x21: ffff07ff84e93c00 [ 1639.311880] acpi_ipmi [ 1639.313705] [ 1639.317093] ipmi_ssif [ 1639.319439] x20: 00000000000001c0 [ 1639.320917] cppc_cpufreq [ 1639.323263] x19: ffff07ff91ccdb10 [ 1639.326564] ipmi_devintf [ 1639.329170] x18: ffffffffffffffff [ 1639.332558] igb [ 1639.335164] [ 1639.338552] ipmi_msghandler [ 1639.340377] x17: ffff32506c412000 [ 1639.341855] arm_cmn [ 1639.344721] x16: ffff800008044000 [ 1639.348022] arm_dmc620_pmu [ 1639.350196] x15: 0000000000004000 [ 1639.353584] arm_spe_pmu [ 1639.356364] [ 1639.359751] arm_dsu_pmu [ 1639.362271] x14: 010320bb28ec0000 [ 1639.363748] acpi_tad [ 1639.366268] x13: 0000000000000087 [ 1639.369569] fuse [ 1639.371828] x12: 071c71c71c71c71c [ 1639.375215] zram [ 1639.377127] [ 1639.380515] xfs [ 1639.382427] x11: 0000000000000040 [ 1639.383905] crct10dif_ce [ 1639.385730] x10: ffff07ff801a9c70 [ 1639.389030] polyval_ce [ 1639.391637] x9 : ffffd5bec9545fd8 [ 1639.395024] ast [ 1639.397457] [ 1639.400845] polyval_generic [ 1639.402670] x8 : ffff07ff80404b70 [ 1639.404148] drm_vram_helper [ 1639.407014] x7 : ffff07ff84e93dc8 [ 1639.410315] ghash_ce [ 1639.413182] x6 : 0000000000000008 [ 1639.416569] drm_ttm_helper [ 1639.418829] [ 1639.422216] sbsa_gwdt [ 1639.424996] x5 : ffff80000d043790 [ 1639.426474] ttm [ 1639.428820] x4 : ffff07ff84e93dc0 [ 1639.432122] xgene_hwmon [ 1639.433947] x3 : 0000000000000008 [ 1639.437335] Unloaded tainted modules: [ 1639.439854] [ 1639.443242] nvmet():1 [ 1639.446890] x2 : 0000000000000007 [ 1639.448368] nvme_fabrics():1 [ 1639.450714] x1 : ffff80000d79b30c [ 1639.454015] nvme_loop():1 [ 1639.456968] x0 : ffff07ff84e93dd0 [ 1639.460356] nvmet():1 [ 1639.463049] [ 1639.466437] nvme_fabrics():1 [ 1639.468783] Call trace: [ 1639.470261] nvme_loop():1 [ 1639.473214] __sbq_wake_up+0x58/0x1f0 [ 1639.475647] nvmet():1 [ 1639.478340] sbitmap_queue_clear+0x74/0xc0 [ 1639.481988] nvme_fabrics():1 [ 1639.484334] blk_mq_put_tag+0x44/0x70 [ 1639.488416] nvme_loop():1 [ 1639.491370] __blk_mq_free_request+0x88/0xf0 [ 1639.495018] nvmet():1 [ 1639.497711] blk_mq_free_request+0xc0/0x160 [ 1639.501966] nvme_fabrics():1 [ 1639.504313] __blk_mq_end_request+0x78/0x150 [ 1639.508483] nvme_loop():1 [ 1639.511437] scsi_end_request+0x114/0x260 [ 1639.515692] nvmet():1 [ 1639.518385] scsi_io_completion+0x64/0x174 [ 1639.522381] nvme_fabrics():1 [ 1639.524727] scsi_finish_command+0xe0/0x120 [ 1639.528809] nvme_loop():1 [ 1639.531762] scsi_complete+0x8c/0x1c4 [ 1639.535931] nvmet():1 [ 1639.538624] blk_complete_reqs+0x5c/0x74 [ 1639.542272] nvme_fabrics():1 [ 1639.544618] blk_done_softirq+0x38/0x4c [ 1639.548526] nvme_loop():1 [ 1639.551480] __do_softirq+0x168/0x418 [ 1639.555302] nvmet():1 [ 1639.557995] __irq_exit_rcu+0x120/0x170 [ 1639.561643] nvme_fabrics():1 [ 1639.563989] irq_exit_rcu+0x18/0x24 [ 1639.567810] nvme_loop():1 [ 1639.570764] el1_interrupt+0x38/0x70 [ 1639.574238] nvmet():1 [ 1639.576931] el1h_64_irq_handler+0x18/0x24 [ 1639.580492] nvme_fabrics():1 [ 1639.582839] el1h_64_irq+0x68/0x6c [ 1639.586921] nvme_loop():1 [ 1639.589874] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1639.593262] nvmet():1 [ 1639.595955] __wake_up_common_lock+0x84/0xac [ 1639.600732] nvme_fabrics():1 [ 1639.603078] __wake_up+0x20/0x30 [ 1639.607334] nvme_loop():1 [ 1639.610288] __sbq_wake_up+0x13c/0x1f0 [ 1639.613502] nvmet():1 [ 1639.616195] sbitmap_queue_clear+0x74/0xc0 [ 1639.619930] nvme_fabrics():1 [ 1639.622276] blk_mq_put_tag+0x44/0x70 [ 1639.626358] nvme_loop():1 [ 1639.629311] __blk_mq_free_request+0x88/0xf0 [ 1639.632959] null_blk():1 [ 1639.635652] blk_mq_free_request+0xc0/0x160 [ 1639.639908] null_blk():1 [ 1639.642514] bfq_insert_request.isra.0+0x204/0x270 [ 1639.646683] null_blk():1 [ 1639.649289] bfq_insert_requests+0x70/0x90 [ 1639.654066] null_blk():1 [ 1639.656672] blk_mq_sched_insert_requests+0xac/0x250 [ 1639.660754] null_blk():1 [ 1639.663361] blk_mq_flush_plug_list+0x110/0x2cc [ 1639.668311] null_blk():1 [ 1639.670917] __blk_flush_plug+0x100/0x140 [ 1639.675433] null_blk():1 [ 1639.678039] blk_finish_plug+0x40/0x25c [ 1639.682034] scsi_debug():1 [ 1639.684641] __iomap_dio_rw+0x1b4/0x500 [ 1639.688462] scsi_debug():1 [ 1639.691243] iomap_dio_rw+0x18/0x50 [ 1639.695064] ipmi_si():1 [ 1639.697844] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1639.701319] ipmi_si():1 [ 1639.703838] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1639.709137] [last unloaded: nvmet] [ 1639.711657] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1639.716346] [ 1639.719821] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1639.724860] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1639.726338] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1639.731461] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1639.739276] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1639.744400] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1639.753690] process_one_work+0x1ec/0x470 [ 1639.759074] pc : __sbq_wake_up+0x4/0x1f0 [ 1639.766021] worker_thread+0x74/0x410 [ 1639.770016] lr : sbitmap_queue_clear+0x74/0xc0 [ 1639.773925] kthread+0xec/0x100 [ 1639.777573] sp : ffff800008843900 [ 1639.782003] ret_from_fork+0x10/0x20 [ 1639.785130] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1639.799114] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1639.806236] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1639.813358] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1639.820479] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1639.827601] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1639.834723] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1639.841845] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1639.848967] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1639.856089] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 1639.863211] Call trace: [ 1639.865643] __sbq_wake_up+0x4/0x1f0 [ 1639.869205] blk_mq_put_tag+0x44/0x70 [ 1639.872855] __blk_mq_free_request+0x88/0xf0 [ 1639.877112] blk_mq_free_request+0xc0/0x160 [ 1639.881282] __blk_mq_end_request+0x78/0x150 [ 1639.885538] scsi_end_request+0x114/0x260 [ 1639.889535] scsi_io_completion+0x64/0x174 [ 1639.893618] scsi_finish_command+0xe0/0x120 [ 1639.897788] scsi_complete+0x8c/0x1c4 [ 1639.901437] blk_complete_reqs+0x5c/0x74 [ 1639.905347] blk_done_softirq+0x38/0x4c [ 1639.909170] __do_softirq+0x168/0x418 [ 1639.912819] __irq_exit_rcu+0x120/0x170 [ 1639.916641] irq_exit_rcu+0x18/0x24 [ 1639.920116] el1_interrupt+0x38/0x70 [ 1639.923678] el1h_64_irq_handler+0x18/0x24 [ 1639.927762] el1h_64_irq+0x68/0x6c [ 1639.931150] cpuidle_enter_state+0xc4/0x4e0 [ 1639.935321] cpuidle_enter+0x40/0x60 [ 1639.938883] cpuidle_idle_call+0x118/0x1b0 [ 1639.942967] do_idle+0xac/0x100 [ 1639.946095] cpu_startup_entry+0x30/0x34 [ 1639.950005] secondary_start_kernel+0xd8/0xf0 [ 1639.954349] __secondary_switched+0xb0/0xb4 [ 1663.103782] watchdog: BUG: soft lockup - CPU#8 stuck for 1154s! [kworker/8:12:19399] [ 1663.111512] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1663.111528] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1663.199316] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1663.207565] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1663.213782] watchdog: BUG: soft lockup - CPU#68 stuck for 1154s! [swapper/68:0] [ 1663.216856] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1663.224150] Modules linked in: nvme_loop [ 1663.230316] [ 1663.230316] nvme_fabrics [ 1663.234225] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1663.235703] nvmet [ 1663.238309] pc : __sbq_wake_up+0x58/0x1f0 [ 1663.245255] nls_utf8 [ 1663.247254] lr : sbitmap_queue_clear+0x74/0xc0 [ 1663.251249] hfsplus [ 1663.253509] sp : ffff80000d79b280 [ 1663.257938] hfs [ 1663.260111] x29: ffff80000d79b280 [ 1663.263411] isofs [ 1663.265236] x28: ffffd5becb6aec50 [ 1663.268537] nvme_core [ 1663.270536] x27: ffffd5beca4f7000 [ 1663.273924] nvme_common [ 1663.276271] [ 1663.279659] loop [ 1663.282178] x26: ffffd5becaf68a98 [ 1663.283656] tls [ 1663.285568] x25: ffffd5becb6a60c0 [ 1663.288869] rfkill [ 1663.290694] x24: fffffbffef180dc0 [ 1663.294081] sunrpc [ 1663.296167] [ 1663.299554] vfat [ 1663.301640] x23: ffff07ff81d381b0 [ 1663.303118] fat [ 1663.305030] x22: ffff80000d79b30c [ 1663.308330] acpi_ipmi [ 1663.310155] x21: ffff07ff84e93c00 [ 1663.313543] ipmi_ssif [ 1663.315889] [ 1663.319277] cppc_cpufreq [ 1663.321623] x20: 00000000000001c0 [ 1663.323102] ipmi_devintf [ 1663.325709] x19: ffff07ff91ccdb10 [ 1663.329010] igb [ 1663.331616] x18: ffffffffffffffff [ 1663.335004] ipmi_msghandler [ 1663.336830] [ 1663.340218] arm_cmn [ 1663.343084] x17: ffff32506c412000 [ 1663.344562] arm_dmc620_pmu [ 1663.346735] x16: ffff800008044000 [ 1663.350035] arm_spe_pmu [ 1663.352815] x15: 0000000000004000 [ 1663.356203] arm_dsu_pmu [ 1663.358722] [ 1663.362111] acpi_tad [ 1663.364631] x14: 010320bb28ec0000 [ 1663.366108] fuse [ 1663.368368] x13: 0000000000000087 [ 1663.371668] zram [ 1663.373580] x12: 071c71c71c71c71c [ 1663.376968] xfs [ 1663.378880] [ 1663.382267] crct10dif_ce [ 1663.384092] x11: 0000000000000040 [ 1663.385570] polyval_ce [ 1663.388177] x10: ffff07ff801a9c70 [ 1663.391477] ast [ 1663.393910] x9 : ffffd5bec9545fd8 [ 1663.397298] polyval_generic [ 1663.399123] [ 1663.402510] drm_vram_helper [ 1663.405377] x8 : ffff07ff80404b70 [ 1663.406855] ghash_ce [ 1663.409722] x7 : ffff07ff84e93dc8 [ 1663.413025] drm_ttm_helper [ 1663.415286] x6 : 0000000000000008 [ 1663.418673] sbsa_gwdt [ 1663.421453] [ 1663.424841] ttm [ 1663.427187] x5 : ffff80000d043790 [ 1663.428665] xgene_hwmon [ 1663.430490] x4 : ffff07ff84e93dc0 [ 1663.433790] Unloaded tainted modules: [ 1663.436310] x3 : 0000000000000008 [ 1663.439698] nvmet():1 [ 1663.443346] [ 1663.446733] nvme_fabrics():1 [ 1663.449079] x2 : 0000000000000007 [ 1663.450557] nvme_loop():1 [ 1663.453510] x1 : ffff80000d79b30c [ 1663.456811] nvmet():1 [ 1663.459504] x0 : ffff07ff84e93dd0 [ 1663.462892] nvme_fabrics():1 [ 1663.465238] [ 1663.468625] nvme_loop():1 [ 1663.471581] Call trace: [ 1663.473058] nvmet():1 [ 1663.475752] __sbq_wake_up+0x58/0x1f0 [ 1663.478184] nvme_fabrics():1 [ 1663.480530] sbitmap_queue_clear+0x74/0xc0 [ 1663.484178] nvme_loop():1 [ 1663.487132] blk_mq_put_tag+0x44/0x70 [ 1663.491214] nvmet():1 [ 1663.493907] __blk_mq_free_request+0x88/0xf0 [ 1663.497555] nvme_fabrics():1 [ 1663.499901] blk_mq_free_request+0xc0/0x160 [ 1663.504156] nvme_loop():1 [ 1663.507110] __blk_mq_end_request+0x78/0x150 [ 1663.511279] nvmet():1 [ 1663.513972] scsi_end_request+0x114/0x260 [ 1663.518228] nvme_fabrics():1 [ 1663.520575] scsi_io_completion+0x64/0x174 [ 1663.524570] nvme_loop():1 [ 1663.527524] scsi_finish_command+0xe0/0x120 [ 1663.531607] nvmet():1 [ 1663.534301] scsi_complete+0x8c/0x1c4 [ 1663.538470] nvme_fabrics():1 [ 1663.540816] blk_complete_reqs+0x5c/0x74 [ 1663.544464] nvme_loop():1 [ 1663.547418] blk_done_softirq+0x38/0x4c [ 1663.551327] nvmet():1 [ 1663.554021] __do_softirq+0x168/0x418 [ 1663.557843] nvme_fabrics():1 [ 1663.560189] __irq_exit_rcu+0x120/0x170 [ 1663.563837] nvme_loop():1 [ 1663.566790] irq_exit_rcu+0x18/0x24 [ 1663.570612] nvmet():1 [ 1663.573305] el1_interrupt+0x38/0x70 [ 1663.576779] nvme_fabrics():1 [ 1663.579126] el1h_64_irq_handler+0x18/0x24 [ 1663.582689] nvme_loop():1 [ 1663.585642] el1h_64_irq+0x68/0x6c [ 1663.589724] nvmet():1 [ 1663.592417] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1663.595805] nvme_fabrics():1 [ 1663.598151] __wake_up_common_lock+0x84/0xac [ 1663.602927] nvme_loop():1 [ 1663.605881] __wake_up+0x20/0x30 [ 1663.610136] nvmet():1 [ 1663.612830] __sbq_wake_up+0x13c/0x1f0 [ 1663.616043] nvme_fabrics():1 [ 1663.618389] sbitmap_queue_clear+0x74/0xc0 [ 1663.622124] nvme_loop():1 [ 1663.625078] blk_mq_put_tag+0x44/0x70 [ 1663.629160] null_blk():1 [ 1663.631853] __blk_mq_free_request+0x88/0xf0 [ 1663.635501] null_blk():1 [ 1663.638107] blk_mq_free_request+0xc0/0x160 [ 1663.642363] null_blk():1 [ 1663.644970] bfq_insert_request.isra.0+0x204/0x270 [ 1663.649138] null_blk():1 [ 1663.651745] bfq_insert_requests+0x70/0x90 [ 1663.656521] null_blk():1 [ 1663.659129] blk_mq_sched_insert_requests+0xac/0x250 [ 1663.663211] null_blk():1 [ 1663.665818] blk_mq_flush_plug_list+0x110/0x2cc [ 1663.670768] null_blk():1 [ 1663.673374] __blk_flush_plug+0x100/0x140 [ 1663.677891] scsi_debug():1 [ 1663.680497] blk_finish_plug+0x40/0x25c [ 1663.684492] scsi_debug():1 [ 1663.687272] __iomap_dio_rw+0x1b4/0x500 [ 1663.691094] ipmi_si():1 [ 1663.693874] iomap_dio_rw+0x18/0x50 [ 1663.697696] ipmi_si():1 [ 1663.700216] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1663.703690] [last unloaded: nvmet] [ 1663.706210] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1663.711507] [ 1663.714981] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1663.719671] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1663.721149] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1663.726186] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1663.734000] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1663.739124] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1663.748414] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1663.753538] pc : __sbq_wake_up+0x58/0x1f0 [ 1663.760485] process_one_work+0x1ec/0x470 [ 1663.765869] lr : sbitmap_queue_clear+0x74/0xc0 [ 1663.769864] worker_thread+0x74/0x410 [ 1663.773859] sp : ffff8000088438b0 [ 1663.778288] kthread+0xec/0x100 [ 1663.781936] x29: ffff8000088438b0 [ 1663.785237] ret_from_fork+0x10/0x20 [ 1663.788364] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1663.800525] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1663.807647] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1663.814769] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1663.821891] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1663.829012] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1663.836134] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1663.843256] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1663.850378] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1663.857500] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1663.864622] Call trace: [ 1663.867055] __sbq_wake_up+0x58/0x1f0 [ 1663.870704] sbitmap_queue_clear+0x74/0xc0 [ 1663.874787] blk_mq_put_tag+0x44/0x70 [ 1663.878437] __blk_mq_free_request+0x88/0xf0 [ 1663.882693] blk_mq_free_request+0xc0/0x160 [ 1663.886863] __blk_mq_end_request+0x78/0x150 [ 1663.891120] scsi_end_request+0x114/0x260 [ 1663.895116] scsi_io_completion+0x64/0x174 [ 1663.899199] scsi_finish_command+0xe0/0x120 [ 1663.903370] scsi_complete+0x8c/0x1c4 [ 1663.907019] blk_complete_reqs+0x5c/0x74 [ 1663.910928] blk_done_softirq+0x38/0x4c [ 1663.914751] __do_softirq+0x168/0x418 [ 1663.918400] __irq_exit_rcu+0x120/0x170 [ 1663.922222] irq_exit_rcu+0x18/0x24 [ 1663.925698] el1_interrupt+0x38/0x70 [ 1663.929260] el1h_64_irq_handler+0x18/0x24 [ 1663.933343] el1h_64_irq+0x68/0x6c [ 1663.936731] cpuidle_enter_state+0xc4/0x4e0 [ 1663.940901] cpuidle_enter+0x40/0x60 [ 1663.944463] cpuidle_idle_call+0x118/0x1b0 [ 1663.948547] do_idle+0xac/0x100 [ 1663.951676] cpu_startup_entry+0x30/0x34 [ 1663.955585] secondary_start_kernel+0xd8/0xf0 [ 1663.959929] __secondary_switched+0xb0/0xb4 [ 1670.123745] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 121917 jiffies s: 2093 root: 0x11/. [ 1670.135483] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1670.144691] Task dump for CPU 8: [ 1670.147906] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1670.157808] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1670.163977] Call trace: [ 1670.166410] __switch_to+0xd4/0x100 [ 1670.169887] 0xffff6c0200000080 [ 1670.173015] Task dump for CPU 68: [ 1670.176318] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1670.186219] Call trace: [ 1670.188651] __switch_to+0xd4/0x100 [ 1670.192128] 0x0 [-- MARK -- Wed Sep 28 04:25:00 2022] [ 1687.103621] watchdog: BUG: soft lockup - CPU#8 stuck for 1177s! [kworker/8:12:19399] [ 1687.111350] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1687.111366] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1687.199154] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1687.207403] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1687.213620] watchdog: BUG: soft lockup - CPU#68 stuck for 1177s! [swapper/68:0] [ 1687.216694] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1687.223988] Modules linked in: nvme_loop [ 1687.230154] [ 1687.230155] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1687.234063] nvme_fabrics [ 1687.235541] pc : __sbq_wake_up+0x40/0x1f0 [ 1687.242487] nvmet [ 1687.245094] lr : sbitmap_queue_clear+0x74/0xc0 [ 1687.249089] nls_utf8 [ 1687.251088] sp : ffff80000d79b280 [ 1687.255517] hfsplus [ 1687.257777] x29: ffff80000d79b280 [ 1687.261078] hfs [ 1687.263251] x28: ffffd5becb6aec50 [ 1687.266551] isofs [ 1687.268377] x27: ffffd5beca4f7000 [ 1687.271765] nvme_core [ 1687.273764] [ 1687.277151] nvme_common [ 1687.279497] x26: ffffd5becaf68a98 [ 1687.280975] loop [ 1687.283495] x25: ffffd5becb6a60c0 [ 1687.286795] tls [ 1687.288707] x24: fffffbffef180dc0 [ 1687.292095] rfkill [ 1687.293920] [ 1687.297308] sunrpc [ 1687.299394] x23: ffff07ff81d381b0 [ 1687.300871] vfat [ 1687.302957] x22: ffff80000d79b30c [ 1687.306258] fat [ 1687.308170] x21: ffff07ff84e93c00 [ 1687.311557] acpi_ipmi [ 1687.313382] [ 1687.316770] ipmi_ssif [ 1687.319116] x20: 0000000000000024 [ 1687.320594] cppc_cpufreq [ 1687.322940] x19: ffff07ff91ccdb10 [ 1687.326240] ipmi_devintf [ 1687.328847] x18: ffffffffffffffff [ 1687.332234] igb [ 1687.334841] [ 1687.338229] ipmi_msghandler [ 1687.340055] x17: ffff32506c412000 [ 1687.341533] arm_cmn [ 1687.344400] x16: ffff800008044000 [ 1687.347702] arm_dmc620_pmu [ 1687.349875] x15: 0000000000004000 [ 1687.353262] arm_spe_pmu [ 1687.356042] [ 1687.359430] arm_dsu_pmu [ 1687.361950] x14: 010320bb28ec0000 [ 1687.363428] acpi_tad [ 1687.365947] x13: 0000000000000087 [ 1687.369248] fuse [ 1687.371507] x12: 071c71c71c71c71c [ 1687.374895] zram [ 1687.376806] [ 1687.380194] xfs [ 1687.382106] x11: 0000000000000040 [ 1687.383584] crct10dif_ce [ 1687.385409] x10: ffff07ff801a9c70 [ 1687.388710] polyval_ce [ 1687.391316] x9 : ffffd5bec9545fd8 [ 1687.394704] ast [ 1687.397136] [ 1687.400524] polyval_generic [ 1687.402349] x8 : ffff07ff80404b70 [ 1687.403827] drm_vram_helper [ 1687.406694] x7 : ffff07ff84e93dc8 [ 1687.409994] ghash_ce [ 1687.412861] x6 : 0000000000000008 [ 1687.416249] drm_ttm_helper [ 1687.418508] [ 1687.421896] sbsa_gwdt [ 1687.424676] x5 : ffff80000d043790 [ 1687.426154] ttm [ 1687.428500] x4 : ffff07ff84e93dc0 [ 1687.431801] xgene_hwmon [ 1687.433626] x3 : 0000000000000008 [ 1687.437015] Unloaded tainted modules: [ 1687.439535] [ 1687.442922] nvmet():1 [ 1687.446570] x2 : 0000000000000007 [ 1687.448048] nvme_fabrics():1 [ 1687.450394] x1 : ffff80000d79b30c [ 1687.453695] nvme_loop():1 [ 1687.456648] x0 : 0000000000000038 [ 1687.460036] nvmet():1 [ 1687.462729] [ 1687.466118] nvme_fabrics():1 [ 1687.468464] Call trace: [ 1687.469941] nvme_loop():1 [ 1687.472895] __sbq_wake_up+0x40/0x1f0 [ 1687.475328] nvmet():1 [ 1687.478021] sbitmap_queue_clear+0x74/0xc0 [ 1687.481669] nvme_fabrics():1 [ 1687.484015] blk_mq_put_tag+0x44/0x70 [ 1687.488097] nvme_loop():1 [ 1687.491050] __blk_mq_free_request+0x88/0xf0 [ 1687.494698] nvmet():1 [ 1687.497392] blk_mq_free_request+0xc0/0x160 [ 1687.501648] nvme_fabrics():1 [ 1687.503994] __blk_mq_end_request+0x78/0x150 [ 1687.508162] nvme_loop():1 [ 1687.511116] scsi_end_request+0x114/0x260 [ 1687.515372] nvmet():1 [ 1687.518065] scsi_io_completion+0x64/0x174 [ 1687.522060] nvme_fabrics():1 [ 1687.524406] scsi_finish_command+0xe0/0x120 [ 1687.528488] nvme_loop():1 [ 1687.531441] scsi_complete+0x8c/0x1c4 [ 1687.535610] nvmet():1 [ 1687.538304] blk_complete_reqs+0x5c/0x74 [ 1687.541952] nvme_fabrics():1 [ 1687.544298] blk_done_softirq+0x38/0x4c [ 1687.548206] nvme_loop():1 [ 1687.551160] __do_softirq+0x168/0x418 [ 1687.554981] nvmet():1 [ 1687.557674] __irq_exit_rcu+0x120/0x170 [ 1687.561322] nvme_fabrics():1 [ 1687.563668] irq_exit_rcu+0x18/0x24 [ 1687.567490] nvme_loop():1 [ 1687.570443] el1_interrupt+0x38/0x70 [ 1687.573918] nvmet():1 [ 1687.576613] el1h_64_irq_handler+0x18/0x24 [ 1687.580174] nvme_fabrics():1 [ 1687.582520] el1h_64_irq+0x68/0x6c [ 1687.586602] nvme_loop():1 [ 1687.589555] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1687.592943] nvmet():1 [ 1687.595636] __wake_up_common_lock+0x84/0xac [ 1687.600412] nvme_fabrics():1 [ 1687.602758] __wake_up+0x20/0x30 [ 1687.607014] nvme_loop():1 [ 1687.609967] __sbq_wake_up+0x13c/0x1f0 [ 1687.613181] nvmet():1 [ 1687.615875] sbitmap_queue_clear+0x74/0xc0 [ 1687.619610] nvme_fabrics():1 [ 1687.621956] blk_mq_put_tag+0x44/0x70 [ 1687.626038] nvme_loop():1 [ 1687.628991] __blk_mq_free_request+0x88/0xf0 [ 1687.632639] null_blk():1 [ 1687.635333] blk_mq_free_request+0xc0/0x160 [ 1687.639588] null_blk():1 [ 1687.642194] bfq_insert_request.isra.0+0x204/0x270 [ 1687.646363] null_blk():1 [ 1687.648970] bfq_insert_requests+0x70/0x90 [ 1687.653746] null_blk():1 [ 1687.656352] blk_mq_sched_insert_requests+0xac/0x250 [ 1687.660434] null_blk():1 [ 1687.663041] blk_mq_flush_plug_list+0x110/0x2cc [ 1687.667991] null_blk():1 [ 1687.670597] __blk_flush_plug+0x100/0x140 [ 1687.675113] null_blk():1 [ 1687.677720] blk_finish_plug+0x40/0x25c [ 1687.681715] scsi_debug():1 [ 1687.684322] __iomap_dio_rw+0x1b4/0x500 [ 1687.688143] scsi_debug():1 [ 1687.690923] iomap_dio_rw+0x18/0x50 [ 1687.694745] ipmi_si():1 [ 1687.697525] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1687.700999] ipmi_si():1 [ 1687.703519] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1687.708816] [last unloaded: nvmet] [ 1687.711335] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1687.716025] [ 1687.719499] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1687.724536] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1687.726014] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1687.731138] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1687.738952] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1687.744076] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1687.753366] process_one_work+0x1ec/0x470 [ 1687.758751] pc : __sbq_wake_up+0x58/0x1f0 [ 1687.765697] worker_thread+0x74/0x410 [ 1687.769692] lr : sbitmap_queue_clear+0x74/0xc0 [ 1687.773688] kthread+0xec/0x100 [ 1687.777336] sp : ffff8000088438b0 [ 1687.781765] ret_from_fork+0x10/0x20 [ 1687.784892] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1687.798876] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1687.805997] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1687.813119] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1687.820241] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1687.827363] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1687.834485] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1687.841607] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1687.848728] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1687.855850] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1687.862972] Call trace: [ 1687.865405] __sbq_wake_up+0x58/0x1f0 [ 1687.869054] sbitmap_queue_clear+0x74/0xc0 [ 1687.873138] blk_mq_put_tag+0x44/0x70 [ 1687.876787] __blk_mq_free_request+0x88/0xf0 [ 1687.881044] blk_mq_free_request+0xc0/0x160 [ 1687.885214] __blk_mq_end_request+0x78/0x150 [ 1687.889470] scsi_end_request+0x114/0x260 [ 1687.893467] scsi_io_completion+0x64/0x174 [ 1687.897550] scsi_finish_command+0xe0/0x120 [ 1687.901720] scsi_complete+0x8c/0x1c4 [ 1687.905370] blk_complete_reqs+0x5c/0x74 [ 1687.909280] blk_done_softirq+0x38/0x4c [ 1687.913103] __do_softirq+0x168/0x418 [ 1687.916751] __irq_exit_rcu+0x120/0x170 [ 1687.920573] irq_exit_rcu+0x18/0x24 [ 1687.924049] el1_interrupt+0x38/0x70 [ 1687.927611] el1h_64_irq_handler+0x18/0x24 [ 1687.931694] el1h_64_irq+0x68/0x6c [ 1687.935082] cpuidle_enter_state+0xc4/0x4e0 [ 1687.939252] cpuidle_enter+0x40/0x60 [ 1687.942814] cpuidle_idle_call+0x118/0x1b0 [ 1687.946898] do_idle+0xac/0x100 [ 1687.950027] cpu_startup_entry+0x30/0x34 [ 1687.953936] secondary_start_kernel+0xd8/0xf0 [ 1687.958280] __secondary_switched+0xb0/0xb4 [ 1711.103459] watchdog: BUG: soft lockup - CPU#8 stuck for 1199s! [kworker/8:12:19399] [ 1711.111188] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1711.111205] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1711.198992] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1711.207242] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1711.213458] watchdog: BUG: soft lockup - CPU#68 stuck for 1199s! [swapper/68:0] [ 1711.216533] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1711.223827] Modules linked in: nvme_loop [ 1711.229993] [ 1711.229994] nvme_fabrics [ 1711.233902] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1711.235380] nvmet [ 1711.237986] pc : __sbq_wake_up+0x7c/0x1f0 [ 1711.244933] nls_utf8 [ 1711.246932] lr : sbitmap_queue_clear+0x74/0xc0 [ 1711.250929] hfsplus [ 1711.253189] sp : ffff80000d79b2d0 [ 1711.257619] hfs [ 1711.259791] x29: ffff80000d79b2d0 [ 1711.263092] isofs [ 1711.264918] x28: ffffd5becb6aec50 [ 1711.268219] nvme_core [ 1711.270218] x27: ffffd5beca4f7000 [ 1711.273605] nvme_common [ 1711.275951] [ 1711.279339] loop [ 1711.281858] x26: ffffd5becaf68a98 [ 1711.283336] tls [ 1711.285249] x25: ffffd5becb6a60c0 [ 1711.288549] rfkill [ 1711.290374] x24: fffffbffef180dc0 [ 1711.293762] sunrpc [ 1711.295848] [ 1711.299236] vfat [ 1711.301321] x23: ffff07ff81d381b0 [ 1711.302799] fat [ 1711.304711] x22: 0000000000000024 [ 1711.308012] acpi_ipmi [ 1711.309837] x21: 0000000000000008 [ 1711.313224] ipmi_ssif [ 1711.315570] [ 1711.318958] cppc_cpufreq [ 1711.321304] x20: 0000000000000024 [ 1711.322781] ipmi_devintf [ 1711.325388] x19: ffff07ff91ccdb10 [ 1711.328689] igb [ 1711.331295] x18: ffffffffffffffff [ 1711.334683] ipmi_msghandler [ 1711.336508] [ 1711.339896] arm_cmn [ 1711.342763] x17: ffff32506c412000 [ 1711.344241] arm_dmc620_pmu [ 1711.346414] x16: ffff800008044000 [ 1711.349714] arm_spe_pmu [ 1711.352494] x15: 0000000000004000 [ 1711.355882] arm_dsu_pmu [ 1711.358402] [ 1711.361790] acpi_tad [ 1711.364309] x14: 010320bb28ec0000 [ 1711.365787] fuse [ 1711.368047] x13: 0000000000000087 [ 1711.371347] zram [ 1711.373259] x12: 071c71c71c71c71c [ 1711.376647] xfs [ 1711.378559] [ 1711.381947] crct10dif_ce [ 1711.383772] x11: 0000000000000040 [ 1711.385250] polyval_ce [ 1711.387856] x10: ffff07ff801a9c70 [ 1711.391157] ast [ 1711.393590] x9 : ffffd5bec9545fd8 [ 1711.396977] polyval_generic [ 1711.398802] [ 1711.402190] drm_vram_helper [ 1711.405057] x8 : ffff07ff80404b70 [ 1711.406535] ghash_ce [ 1711.409402] x7 : ffff07ff84e93dc8 [ 1711.412703] drm_ttm_helper [ 1711.414962] x6 : 0000000000000008 [ 1711.418350] sbsa_gwdt [ 1711.421130] [ 1711.424519] ttm [ 1711.426865] x5 : ffff80000d043790 [ 1711.428343] xgene_hwmon [ 1711.430168] x4 : ffff07ff84e93dc0 [ 1711.433469] Unloaded tainted modules: [ 1711.435988] x3 : 0000000000000008 [ 1711.439376] nvmet():1 [ 1711.443024] [ 1711.446411] nvme_fabrics():1 [ 1711.448757] x2 : 0000000000000007 [ 1711.450235] nvme_loop():1 [ 1711.453188] x1 : 0000000000000000 [ 1711.456489] nvmet():1 [ 1711.459182] x0 : 0000000000000001 [ 1711.462570] nvme_fabrics():1 [ 1711.464916] [ 1711.468303] nvme_loop():1 [ 1711.471257] Call trace: [ 1711.472735] nvmet():1 [ 1711.475428] __sbq_wake_up+0x7c/0x1f0 [ 1711.477861] nvme_fabrics():1 [ 1711.480207] blk_mq_put_tag+0x44/0x70 [ 1711.483855] nvme_loop():1 [ 1711.486808] __blk_mq_free_request+0x88/0xf0 [ 1711.490456] nvmet():1 [ 1711.493149] blk_mq_free_request+0xc0/0x160 [ 1711.497405] nvme_fabrics():1 [ 1711.499751] __blk_mq_end_request+0x78/0x150 [ 1711.503920] nvme_loop():1 [ 1711.506873] scsi_end_request+0x114/0x260 [ 1711.511129] nvmet():1 [ 1711.513822] scsi_io_completion+0x64/0x174 [ 1711.517817] nvme_fabrics():1 [ 1711.520163] scsi_finish_command+0xe0/0x120 [ 1711.524245] nvme_loop():1 [ 1711.527199] scsi_complete+0x8c/0x1c4 [ 1711.531367] nvmet():1 [ 1711.534061] blk_complete_reqs+0x5c/0x74 [ 1711.537709] nvme_fabrics():1 [ 1711.540055] blk_done_softirq+0x38/0x4c [ 1711.543963] nvme_loop():1 [ 1711.546917] __do_softirq+0x168/0x418 [ 1711.550739] nvmet():1 [ 1711.553432] __irq_exit_rcu+0x120/0x170 [ 1711.557080] nvme_fabrics():1 [ 1711.559426] irq_exit_rcu+0x18/0x24 [ 1711.563249] nvme_loop():1 [ 1711.566203] el1_interrupt+0x38/0x70 [ 1711.569677] nvmet():1 [ 1711.572370] el1h_64_irq_handler+0x18/0x24 [ 1711.575932] nvme_fabrics():1 [ 1711.578277] el1h_64_irq+0x68/0x6c [ 1711.582360] nvme_loop():1 [ 1711.585313] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1711.588701] nvmet():1 [ 1711.591394] __wake_up_common_lock+0x84/0xac [ 1711.596170] nvme_fabrics():1 [ 1711.598516] __wake_up+0x20/0x30 [ 1711.602772] nvme_loop():1 [ 1711.605726] __sbq_wake_up+0x13c/0x1f0 [ 1711.608941] nvmet():1 [ 1711.611635] sbitmap_queue_clear+0x74/0xc0 [ 1711.615369] nvme_fabrics():1 [ 1711.617715] blk_mq_put_tag+0x44/0x70 [ 1711.621797] nvme_loop():1 [ 1711.624751] __blk_mq_free_request+0x88/0xf0 [ 1711.628399] null_blk():1 [ 1711.631093] blk_mq_free_request+0xc0/0x160 [ 1711.635348] null_blk():1 [ 1711.637954] bfq_insert_request.isra.0+0x204/0x270 [ 1711.642123] null_blk():1 [ 1711.644730] bfq_insert_requests+0x70/0x90 [ 1711.649508] null_blk():1 [ 1711.652114] blk_mq_sched_insert_requests+0xac/0x250 [ 1711.656196] null_blk():1 [ 1711.658802] blk_mq_flush_plug_list+0x110/0x2cc [ 1711.663752] null_blk():1 [ 1711.666359] __blk_flush_plug+0x100/0x140 [ 1711.670875] null_blk():1 [ 1711.673481] blk_finish_plug+0x40/0x25c [ 1711.677476] scsi_debug():1 [ 1711.680083] __iomap_dio_rw+0x1b4/0x500 [ 1711.683904] scsi_debug():1 [ 1711.686684] iomap_dio_rw+0x18/0x50 [ 1711.690506] ipmi_si():1 [ 1711.693286] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1711.696760] ipmi_si():1 [ 1711.699280] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1711.704577] [last unloaded: nvmet] [ 1711.707096] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1711.711786] [ 1711.715260] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1711.720297] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1711.721775] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1711.726899] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1711.734713] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1711.739837] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1711.749128] process_one_work+0x1ec/0x470 [ 1711.754512] pc : __sbq_wake_up+0x20/0x1f0 [ 1711.761458] worker_thread+0x74/0x410 [ 1711.765453] lr : sbitmap_queue_clear+0x74/0xc0 [ 1711.769449] kthread+0xec/0x100 [ 1711.773097] sp : ffff8000088438b0 [ 1711.777526] ret_from_fork+0x10/0x20 [ 1711.780653] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1711.794637] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1711.801759] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1711.808881] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1711.816003] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1711.823125] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1711.830246] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1711.837368] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1711.844490] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1711.851612] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1711.858734] Call trace: [ 1711.861166] __sbq_wake_up+0x20/0x1f0 [ 1711.864815] sbitmap_queue_clear+0x74/0xc0 [ 1711.868899] blk_mq_put_tag+0x44/0x70 [ 1711.872548] __blk_mq_free_request+0x88/0xf0 [ 1711.876805] blk_mq_free_request+0xc0/0x160 [ 1711.880975] __blk_mq_end_request+0x78/0x150 [ 1711.885232] scsi_end_request+0x114/0x260 [ 1711.889228] scsi_io_completion+0x64/0x174 [ 1711.893311] scsi_finish_command+0xe0/0x120 [ 1711.897481] scsi_complete+0x8c/0x1c4 [ 1711.901130] blk_complete_reqs+0x5c/0x74 [ 1711.905040] blk_done_softirq+0x38/0x4c [ 1711.908863] __do_softirq+0x168/0x418 [ 1711.912512] __irq_exit_rcu+0x120/0x170 [ 1711.916334] irq_exit_rcu+0x18/0x24 [ 1711.919810] el1_interrupt+0x38/0x70 [ 1711.923372] el1h_64_irq_handler+0x18/0x24 [ 1711.927455] el1h_64_irq+0x68/0x6c [ 1711.930843] cpuidle_enter_state+0xc4/0x4e0 [ 1711.935014] cpuidle_enter+0x40/0x60 [ 1711.938576] cpuidle_idle_call+0x118/0x1b0 [ 1711.942660] do_idle+0xac/0x100 [ 1711.945788] cpu_startup_entry+0x30/0x34 [ 1711.949697] secondary_start_kernel+0xd8/0xf0 [ 1711.954041] __secondary_switched+0xb0/0xb4 [ 1735.103297] watchdog: BUG: soft lockup - CPU#8 stuck for 1221s! [kworker/8:12:19399] [ 1735.111026] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1735.111043] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1735.198830] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1735.207079] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1735.213296] watchdog: BUG: soft lockup - CPU#68 stuck for 1222s! [swapper/68:0] [ 1735.216370] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1735.223664] Modules linked in: nvme_loop [ 1735.229830] [ 1735.229830] nvme_fabrics [ 1735.233739] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1735.235217] nvmet [ 1735.237824] pc : __sbq_wake_up+0x84/0x1f0 [ 1735.244770] nls_utf8 [ 1735.246769] lr : sbitmap_queue_clear+0x74/0xc0 [ 1735.250764] hfsplus [ 1735.253024] sp : ffff80000d79b280 [ 1735.257453] hfs [ 1735.259626] x29: ffff80000d79b280 [ 1735.262927] isofs [ 1735.264752] x28: ffffd5becb6aec50 [ 1735.268053] nvme_core [ 1735.270052] x27: ffffd5beca4f7000 [ 1735.273439] nvme_common [ 1735.275785] [ 1735.279173] loop [ 1735.281693] x26: ffffd5becaf68a98 [ 1735.283171] tls [ 1735.285082] x25: ffffd5becb6a60c0 [ 1735.288383] rfkill [ 1735.290208] x24: fffffbffef180dc0 [ 1735.293596] sunrpc [ 1735.295682] [ 1735.299069] vfat [ 1735.301155] x23: ffff07ff81d381b0 [ 1735.302633] fat [ 1735.304546] x22: ffff80000d79b30c [ 1735.307847] acpi_ipmi [ 1735.309672] x21: ffff07ff84e93c00 [ 1735.313059] ipmi_ssif [ 1735.315405] [ 1735.318793] cppc_cpufreq [ 1735.321139] x20: 00000000000001c0 [ 1735.322616] ipmi_devintf [ 1735.325223] x19: ffff07ff91ccdb10 [ 1735.328524] igb [ 1735.331130] x18: ffffffffffffffff [ 1735.334517] ipmi_msghandler [ 1735.336342] [ 1735.339730] arm_cmn [ 1735.342597] x17: ffff32506c412000 [ 1735.344075] arm_dmc620_pmu [ 1735.346247] x16: ffff800008044000 [ 1735.349548] arm_spe_pmu [ 1735.352329] x15: 0000000000004000 [ 1735.355717] arm_dsu_pmu [ 1735.358237] [ 1735.361625] acpi_tad [ 1735.364144] x14: 010320bb28ec0000 [ 1735.365622] fuse [ 1735.367881] x13: 0000000000000087 [ 1735.371182] zram [ 1735.373094] x12: 071c71c71c71c71c [ 1735.376482] xfs [ 1735.378394] [ 1735.381781] crct10dif_ce [ 1735.383606] x11: 0000000000000040 [ 1735.385084] polyval_ce [ 1735.387691] x10: ffff07ff801a9c70 [ 1735.390991] ast [ 1735.393424] x9 : ffffd5bec9545fd8 [ 1735.396812] polyval_generic [ 1735.398637] [ 1735.402024] drm_vram_helper [ 1735.404891] x8 : ffff07ff80404b70 [ 1735.406369] ghash_ce [ 1735.409236] x7 : ffff07ff84e93dc8 [ 1735.412536] drm_ttm_helper [ 1735.414795] x6 : 0000000000000008 [ 1735.418183] sbsa_gwdt [ 1735.420963] [ 1735.424351] ttm [ 1735.426696] x5 : ffff80000d043790 [ 1735.428174] xgene_hwmon [ 1735.429999] x4 : ffff07ff84e93dc0 [ 1735.433300] Unloaded tainted modules: [ 1735.435820] x3 : 0000000000000008 [ 1735.439207] nvmet():1 [ 1735.442855] [ 1735.446243] nvme_fabrics():1 [ 1735.448589] x2 : 0000000000000007 [ 1735.450067] nvme_loop():1 [ 1735.453020] x1 : ffff80000d79b30c [ 1735.456321] nvmet():1 [ 1735.459014] x0 : ffff07ff84e93dd0 [ 1735.462402] nvme_fabrics():1 [ 1735.464748] [ 1735.468136] nvme_loop():1 [ 1735.471089] Call trace: [ 1735.472567] nvmet():1 [ 1735.475261] __sbq_wake_up+0x84/0x1f0 [ 1735.477693] nvme_fabrics():1 [ 1735.480040] sbitmap_queue_clear+0x74/0xc0 [ 1735.483688] nvme_loop():1 [ 1735.486641] blk_mq_put_tag+0x44/0x70 [ 1735.490723] nvmet():1 [ 1735.493417] __blk_mq_free_request+0x88/0xf0 [ 1735.497065] nvme_fabrics():1 [ 1735.499411] blk_mq_free_request+0xc0/0x160 [ 1735.503666] nvme_loop():1 [ 1735.506620] __blk_mq_end_request+0x78/0x150 [ 1735.510788] nvmet():1 [ 1735.513482] scsi_end_request+0x114/0x260 [ 1735.517737] nvme_fabrics():1 [ 1735.520084] scsi_io_completion+0x64/0x174 [ 1735.524079] nvme_loop():1 [ 1735.527033] scsi_finish_command+0xe0/0x120 [ 1735.531115] nvmet():1 [ 1735.533808] scsi_complete+0x8c/0x1c4 [ 1735.537977] nvme_fabrics():1 [ 1735.540323] blk_complete_reqs+0x5c/0x74 [ 1735.543971] nvme_loop():1 [ 1735.546924] blk_done_softirq+0x38/0x4c [ 1735.550833] nvmet():1 [ 1735.553526] __do_softirq+0x168/0x418 [ 1735.557348] nvme_fabrics():1 [ 1735.559694] __irq_exit_rcu+0x120/0x170 [ 1735.563342] nvme_loop():1 [ 1735.566296] irq_exit_rcu+0x18/0x24 [ 1735.570117] nvmet():1 [ 1735.572811] el1_interrupt+0x38/0x70 [ 1735.576285] nvme_fabrics():1 [ 1735.578631] el1h_64_irq_handler+0x18/0x24 [ 1735.582192] nvme_loop():1 [ 1735.585146] el1h_64_irq+0x68/0x6c [ 1735.589228] nvmet():1 [ 1735.591923] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1735.595310] nvme_fabrics():1 [ 1735.597656] __wake_up_common_lock+0x84/0xac [ 1735.602432] nvme_loop():1 [ 1735.605386] __wake_up+0x20/0x30 [ 1735.609642] nvmet():1 [ 1735.612335] __sbq_wake_up+0x13c/0x1f0 [ 1735.615549] nvme_fabrics():1 [ 1735.617895] sbitmap_queue_clear+0x74/0xc0 [ 1735.621630] nvme_loop():1 [ 1735.624584] blk_mq_put_tag+0x44/0x70 [ 1735.628666] null_blk():1 [ 1735.631359] __blk_mq_free_request+0x88/0xf0 [ 1735.635007] null_blk():1 [ 1735.637614] blk_mq_free_request+0xc0/0x160 [ 1735.641869] null_blk():1 [ 1735.644476] bfq_insert_request.isra.0+0x204/0x270 [ 1735.648644] null_blk():1 [ 1735.651251] bfq_insert_requests+0x70/0x90 [ 1735.656027] null_blk():1 [ 1735.658633] blk_mq_sched_insert_requests+0xac/0x250 [ 1735.662716] null_blk():1 [ 1735.665322] blk_mq_flush_plug_list+0x110/0x2cc [ 1735.670272] null_blk():1 [ 1735.672878] __blk_flush_plug+0x100/0x140 [ 1735.677394] scsi_debug():1 [ 1735.680001] blk_finish_plug+0x40/0x25c [ 1735.683996] scsi_debug():1 [ 1735.686776] __iomap_dio_rw+0x1b4/0x500 [ 1735.690597] ipmi_si():1 [ 1735.693377] iomap_dio_rw+0x18/0x50 [ 1735.697199] ipmi_si():1 [ 1735.699718] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1735.703193] [last unloaded: nvmet] [ 1735.705712] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1735.711010] [ 1735.714484] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1735.719174] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1735.720652] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1735.725689] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1735.733503] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1735.738627] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1735.747919] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1735.753043] pc : __sbq_wake_up+0x84/0x1f0 [ 1735.759989] process_one_work+0x1ec/0x470 [ 1735.765373] lr : sbitmap_queue_clear+0x74/0xc0 [ 1735.769369] worker_thread+0x74/0x410 [ 1735.773364] sp : ffff8000088438b0 [ 1735.777793] kthread+0xec/0x100 [ 1735.781441] x29: ffff8000088438b0 [ 1735.784742] ret_from_fork+0x10/0x20 [ 1735.787869] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1735.800029] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1735.807152] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1735.814274] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1735.821395] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1735.828517] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1735.835639] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1735.842761] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1735.849883] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1735.857005] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1735.864126] Call trace: [ 1735.866559] __sbq_wake_up+0x84/0x1f0 [ 1735.870208] sbitmap_queue_clear+0x74/0xc0 [ 1735.874291] blk_mq_put_tag+0x44/0x70 [ 1735.877941] __blk_mq_free_request+0x88/0xf0 [ 1735.882198] blk_mq_free_request+0xc0/0x160 [ 1735.886367] __blk_mq_end_request+0x78/0x150 [ 1735.890624] scsi_end_request+0x114/0x260 [ 1735.894620] scsi_io_completion+0x64/0x174 [ 1735.898704] scsi_finish_command+0xe0/0x120 [ 1735.902874] scsi_complete+0x8c/0x1c4 [ 1735.906523] blk_complete_reqs+0x5c/0x74 [ 1735.910433] blk_done_softirq+0x38/0x4c [ 1735.914255] __do_softirq+0x168/0x418 [ 1735.917904] __irq_exit_rcu+0x120/0x170 [ 1735.921727] irq_exit_rcu+0x18/0x24 [ 1735.925202] el1_interrupt+0x38/0x70 [ 1735.928764] el1h_64_irq_handler+0x18/0x24 [ 1735.932847] el1h_64_irq+0x68/0x6c [ 1735.936235] cpuidle_enter_state+0xc4/0x4e0 [ 1735.940406] cpuidle_enter+0x40/0x60 [ 1735.943968] cpuidle_idle_call+0x118/0x1b0 [ 1735.948052] do_idle+0xac/0x100 [ 1735.951182] cpu_startup_entry+0x30/0x34 [ 1735.955091] secondary_start_kernel+0xd8/0xf0 [ 1735.959435] __secondary_switched+0xb0/0xb4 [ 1736.683296] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 128573 jiffies s: 2093 root: 0x11/. [ 1736.695034] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1736.704242] Task dump for CPU 8: [ 1736.707456] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1736.717358] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1736.723529] Call trace: [ 1736.725962] __switch_to+0xd4/0x100 [ 1736.729438] 0xffff6c0200000080 [ 1736.732566] Task dump for CPU 68: [ 1736.735869] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1736.745770] Call trace: [ 1736.748203] __switch_to+0xd4/0x100 [ 1736.751679] 0x0 [ 1747.193216] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1747.198949] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=65546 [ 1747.208070] (t=132231 jiffies g=36445 q=38924 ncpus=128) [ 1747.213455] Task dump for CPU 8: [ 1747.216669] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1747.226570] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1747.232737] Call trace: [ 1747.235171] __switch_to+0xd4/0x100 [ 1747.238647] 0xffff6c0200000080 [ 1747.241776] Task dump for CPU 68: [ 1747.245077] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1747.254976] Call trace: [ 1747.257409] dump_backtrace+0xe8/0x130 [ 1747.261145] show_stack+0x20/0x60 [ 1747.264447] sched_show_task+0x160/0x190 [ 1747.268356] dump_cpu_task+0x4c/0x78 [ 1747.271919] rcu_dump_cpu_stacks+0xc4/0x150 [ 1747.276090] print_cpu_stall+0x180/0x210 [ 1747.279999] check_cpu_stall+0x20c/0x2b4 [ 1747.283909] rcu_pending+0x4c/0x190 [ 1747.287384] rcu_sched_clock_irq+0x80/0x204 [ 1747.291554] update_process_times+0x7c/0xc0 [ 1747.295724] tick_sched_handle+0x38/0x74 [ 1747.299634] tick_sched_timer+0x60/0xc0 [ 1747.303458] __hrtimer_run_queues+0x170/0x300 [ 1747.307801] hrtimer_interrupt+0xfc/0x260 [ 1747.311797] arch_timer_handler_phys+0x34/0x50 [ 1747.316228] handle_percpu_devid_irq+0xac/0x240 [ 1747.320745] generic_handle_domain_irq+0x34/0x50 [ 1747.325350] gic_handle_irq+0x5c/0xe0 [ 1747.329000] call_on_irq_stack+0x2c/0x38 [ 1747.332909] do_interrupt_handler+0xdc/0xe0 [ 1747.337080] el1_interrupt+0x34/0x70 [ 1747.340643] el1h_64_irq_handler+0x18/0x24 [ 1747.344726] el1h_64_irq+0x68/0x6c [ 1747.348115] __sbq_wake_up+0x40/0x1f0 [ 1747.351764] sbitmap_queue_clear+0x74/0xc0 [ 1747.355847] blk_mq_put_tag+0x44/0x70 [ 1747.359496] __blk_mq_free_request+0x88/0xf0 [ 1747.363753] blk_mq_free_request+0xc0/0x160 [ 1747.367923] __blk_mq_end_request+0x78/0x150 [ 1747.372181] scsi_end_request+0x114/0x260 [ 1747.376179] scsi_io_completion+0x64/0x174 [ 1747.380263] scsi_finish_command+0xe0/0x120 [ 1747.384434] scsi_complete+0x8c/0x1c4 [ 1747.388083] blk_complete_reqs+0x5c/0x74 [ 1747.391992] blk_done_softirq+0x38/0x4c [ 1747.395815] __do_softirq+0x168/0x418 [ 1747.399464] __irq_exit_rcu+0x120/0x170 [ 1747.403287] irq_exit_rcu+0x18/0x24 [ 1747.406762] el1_interrupt+0x38/0x70 [ 1747.410325] el1h_64_irq_handler+0x18/0x24 [ 1747.414408] el1h_64_irq+0x68/0x6c [ 1747.417796] cpuidle_enter_state+0xc4/0x4e0 [ 1747.421966] cpuidle_enter+0x40/0x60 [ 1747.425529] cpuidle_idle_call+0x118/0x1b0 [ 1747.429612] do_idle+0xac/0x100 [ 1747.432741] cpu_startup_entry+0x30/0x34 [ 1747.436650] secondary_start_kernel+0xd8/0xf0 [ 1747.440994] __secondary_switched+0xb0/0xb4 [ 1759.103135] watchdog: BUG: soft lockup - CPU#8 stuck for 1244s! [kworker/8:12:19399] [ 1759.110864] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1759.110881] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1759.198668] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1759.206917] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1759.216208] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1759.222376] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1759.229323] pc : __sbq_wake_up+0x58/0x1f0 [ 1759.233320] lr : sbitmap_queue_clear+0x74/0xc0 [ 1759.237751] sp : ffff80000d79b280 [ 1759.241052] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1759.248175] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1759.255297] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1759.262420] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1759.269541] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1759.276664] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1759.283786] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1759.290908] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1759.298030] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1759.305152] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1759.312274] Call trace: [ 1759.314707] __sbq_wake_up+0x58/0x1f0 [ 1759.318356] sbitmap_queue_clear+0x74/0xc0 [ 1759.322440] blk_mq_put_tag+0x44/0x70 [ 1759.326090] __blk_mq_free_request+0x88/0xf0 [ 1759.330346] blk_mq_free_request+0xc0/0x160 [ 1759.334516] __blk_mq_end_request+0x78/0x150 [ 1759.338773] scsi_end_request+0x114/0x260 [ 1759.342770] scsi_io_completion+0x64/0x174 [ 1759.346853] scsi_finish_command+0xe0/0x120 [ 1759.351023] scsi_complete+0x8c/0x1c4 [ 1759.354672] blk_complete_reqs+0x5c/0x74 [ 1759.358582] blk_done_softirq+0x38/0x4c [ 1759.362405] __do_softirq+0x168/0x418 [ 1759.366054] __irq_exit_rcu+0x120/0x170 [ 1759.369876] irq_exit_rcu+0x18/0x24 [ 1759.373352] el1_interrupt+0x38/0x70 [ 1759.376914] el1h_64_irq_handler+0x18/0x24 [ 1759.380997] el1h_64_irq+0x68/0x6c [ 1759.384386] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1759.389163] __wake_up_common_lock+0x84/0xac [ 1759.393421] __wake_up+0x20/0x30 [ 1759.396636] __sbq_wake_up+0x13c/0x1f0 [ 1759.400372] sbitmap_queue_clear+0x74/0xc0 [ 1759.404456] blk_mq_put_tag+0x44/0x70 [ 1759.408105] __blk_mq_free_request+0x88/0xf0 [ 1759.412362] blk_mq_free_request+0xc0/0x160 [ 1759.416532] bfq_insert_request.isra.0+0x204/0x270 [ 1759.421312] bfq_insert_requests+0x70/0x90 [ 1759.425395] blk_mq_sched_insert_requests+0xac/0x250 [ 1759.430347] blk_mq_flush_plug_list+0x110/0x2cc [ 1759.434865] __blk_flush_plug+0x100/0x140 [ 1759.438861] blk_finish_plug+0x40/0x25c [ 1759.442684] __iomap_dio_rw+0x1b4/0x500 [ 1759.446506] iomap_dio_rw+0x18/0x50 [ 1759.449982] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1759.455332] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1759.460074] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1759.465118] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1759.470249] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1759.475378] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1759.480765] process_one_work+0x1ec/0x470 [ 1759.484761] worker_thread+0x74/0x410 [ 1759.488410] kthread+0xec/0x100 [ 1759.491539] ret_from_fork+0x10/0x20 [ 1771.213053] watchdog: BUG: soft lockup - CPU#68 stuck for 1255s! [swapper/68:0] [ 1771.220349] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1771.220365] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1771.308152] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1771.315968] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1771.325259] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1771.332206] pc : __sbq_wake_up+0x84/0x1f0 [ 1771.336202] lr : sbitmap_queue_clear+0x74/0xc0 [ 1771.340633] sp : ffff8000088438b0 [ 1771.343934] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1771.351056] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1771.358178] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1771.365300] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1771.372421] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1771.379543] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1771.386665] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1771.393787] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1771.400909] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1771.408031] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1771.415153] Call trace: [ 1771.417586] __sbq_wake_up+0x84/0x1f0 [ 1771.421236] sbitmap_queue_clear+0x74/0xc0 [ 1771.425320] blk_mq_put_tag+0x44/0x70 [ 1771.428969] __blk_mq_free_request+0x88/0xf0 [ 1771.433226] blk_mq_free_request+0xc0/0x160 [ 1771.437396] __blk_mq_end_request+0x78/0x150 [ 1771.441652] scsi_end_request+0x114/0x260 [ 1771.445649] scsi_io_completion+0x64/0x174 [ 1771.449732] scsi_finish_command+0xe0/0x120 [ 1771.453902] scsi_complete+0x8c/0x1c4 [ 1771.457551] blk_complete_reqs+0x5c/0x74 [ 1771.461461] blk_done_softirq+0x38/0x4c [ 1771.465283] __do_softirq+0x168/0x418 [ 1771.468933] __irq_exit_rcu+0x120/0x170 [ 1771.472755] irq_exit_rcu+0x18/0x24 [ 1771.476231] el1_interrupt+0x38/0x70 [ 1771.479793] el1h_64_irq_handler+0x18/0x24 [ 1771.483876] el1h_64_irq+0x68/0x6c [ 1771.487264] cpuidle_enter_state+0xc4/0x4e0 [ 1771.491434] cpuidle_enter+0x40/0x60 [ 1771.494996] cpuidle_idle_call+0x118/0x1b0 [ 1771.499080] do_idle+0xac/0x100 [ 1771.502209] cpu_startup_entry+0x30/0x34 [ 1771.506118] secondary_start_kernel+0xd8/0xf0 [ 1771.510462] __secondary_switched+0xb0/0xb4 [ 1783.102973] watchdog: BUG: soft lockup - CPU#8 stuck for 1266s! [kworker/8:12:19399] [ 1783.110702] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1783.110719] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1783.198506] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1783.206756] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1783.216047] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1783.222215] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1783.229162] pc : __sbq_wake_up+0x84/0x1f0 [ 1783.233159] lr : sbitmap_queue_clear+0x74/0xc0 [ 1783.237589] sp : ffff80000d79b280 [ 1783.240891] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1783.248013] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1783.255135] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1783.262257] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1783.269379] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1783.276501] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1783.283623] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1783.290745] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1783.297867] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1783.304989] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1783.312111] Call trace: [ 1783.314544] __sbq_wake_up+0x84/0x1f0 [ 1783.318193] sbitmap_queue_clear+0x74/0xc0 [ 1783.322277] blk_mq_put_tag+0x44/0x70 [ 1783.325927] __blk_mq_free_request+0x88/0xf0 [ 1783.330183] blk_mq_free_request+0xc0/0x160 [ 1783.334353] __blk_mq_end_request+0x78/0x150 [ 1783.338610] scsi_end_request+0x114/0x260 [ 1783.342606] scsi_io_completion+0x64/0x174 [ 1783.346689] scsi_finish_command+0xe0/0x120 [ 1783.350860] scsi_complete+0x8c/0x1c4 [ 1783.354509] blk_complete_reqs+0x5c/0x74 [ 1783.358418] blk_done_softirq+0x38/0x4c [ 1783.362241] __do_softirq+0x168/0x418 [ 1783.365890] __irq_exit_rcu+0x120/0x170 [ 1783.369712] irq_exit_rcu+0x18/0x24 [ 1783.373188] el1_interrupt+0x38/0x70 [ 1783.376750] el1h_64_irq_handler+0x18/0x24 [ 1783.380833] el1h_64_irq+0x68/0x6c [ 1783.384222] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1783.388999] __wake_up_common_lock+0x84/0xac [ 1783.393257] __wake_up+0x20/0x30 [ 1783.396472] __sbq_wake_up+0x13c/0x1f0 [ 1783.400208] sbitmap_queue_clear+0x74/0xc0 [ 1783.404291] blk_mq_put_tag+0x44/0x70 [ 1783.407942] __blk_mq_free_request+0x88/0xf0 [ 1783.412199] blk_mq_free_request+0xc0/0x160 [ 1783.416368] bfq_insert_request.isra.0+0x204/0x270 [ 1783.421147] bfq_insert_requests+0x70/0x90 [ 1783.425230] blk_mq_sched_insert_requests+0xac/0x250 [ 1783.430182] blk_mq_flush_plug_list+0x110/0x2cc [ 1783.434700] __blk_flush_plug+0x100/0x140 [ 1783.438696] blk_finish_plug+0x40/0x25c [ 1783.442518] __iomap_dio_rw+0x1b4/0x500 [ 1783.446341] iomap_dio_rw+0x18/0x50 [ 1783.449817] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1783.455167] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1783.459909] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1783.464953] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1783.470083] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1783.475213] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1783.480599] process_one_work+0x1ec/0x470 [ 1783.484595] worker_thread+0x74/0x410 [ 1783.488244] kthread+0xec/0x100 [ 1783.491373] ret_from_fork+0x10/0x20 [ 1795.212892] watchdog: BUG: soft lockup - CPU#68 stuck for 1277s! [swapper/68:0] [ 1795.220187] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1795.220203] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1795.307990] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1795.315805] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1795.325096] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1795.332044] pc : __sbq_wake_up+0x74/0x1f0 [ 1795.336040] lr : sbitmap_queue_clear+0x74/0xc0 [ 1795.340470] sp : ffff8000088438b0 [ 1795.343772] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1795.350894] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1795.358015] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1795.365137] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1795.372259] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1795.379381] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1795.386503] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1795.393625] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1795.400746] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1795.407868] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 1795.414990] Call trace: [ 1795.417423] __sbq_wake_up+0x74/0x1f0 [ 1795.421072] sbitmap_queue_clear+0x74/0xc0 [ 1795.425155] blk_mq_put_tag+0x44/0x70 [ 1795.428805] __blk_mq_free_request+0x88/0xf0 [ 1795.433062] blk_mq_free_request+0xc0/0x160 [ 1795.437232] __blk_mq_end_request+0x78/0x150 [ 1795.441488] scsi_end_request+0x114/0x260 [ 1795.445485] scsi_io_completion+0x64/0x174 [ 1795.449568] scsi_finish_command+0xe0/0x120 [ 1795.453738] scsi_complete+0x8c/0x1c4 [ 1795.457387] blk_complete_reqs+0x5c/0x74 [ 1795.461297] blk_done_softirq+0x38/0x4c [ 1795.465119] __do_softirq+0x168/0x418 [ 1795.468768] __irq_exit_rcu+0x120/0x170 [ 1795.472591] irq_exit_rcu+0x18/0x24 [ 1795.476066] el1_interrupt+0x38/0x70 [ 1795.479628] el1h_64_irq_handler+0x18/0x24 [ 1795.483712] el1h_64_irq+0x68/0x6c [ 1795.487100] cpuidle_enter_state+0xc4/0x4e0 [ 1795.491270] cpuidle_enter+0x40/0x60 [ 1795.494832] cpuidle_idle_call+0x118/0x1b0 [ 1795.498916] do_idle+0xac/0x100 [ 1795.502045] cpu_startup_entry+0x30/0x34 [ 1795.505954] secondary_start_kernel+0xd8/0xf0 [ 1795.510298] __secondary_switched+0xb0/0xb4 [ 1803.242848] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 135229 jiffies s: 2093 root: 0x11/. [ 1803.254586] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1803.263795] Task dump for CPU 8: [ 1803.267009] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1803.276911] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1803.283080] Call trace: [ 1803.285513] __switch_to+0xd4/0x100 [ 1803.288990] 0xffff6c0200000080 [ 1803.292118] Task dump for CPU 68: [ 1803.295421] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1803.305322] Call trace: [ 1803.307755] __switch_to+0xd4/0x100 [ 1803.311231] 0x0 [ 1807.102811] watchdog: BUG: soft lockup - CPU#8 stuck for 1288s! [kworker/8:12:19399] [ 1807.110541] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1807.110558] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1807.198345] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1807.206595] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1807.215886] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1807.222053] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1807.229001] pc : __sbq_wake_up+0x58/0x1f0 [ 1807.232998] lr : sbitmap_queue_clear+0x74/0xc0 [ 1807.237429] sp : ffff80000d79b280 [ 1807.240730] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1807.247852] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1807.254974] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1807.262096] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1807.269218] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1807.276340] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1807.283461] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1807.290583] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1807.297705] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1807.304827] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1807.311949] Call trace: [ 1807.314382] __sbq_wake_up+0x58/0x1f0 [ 1807.318032] sbitmap_queue_clear+0x74/0xc0 [ 1807.322116] blk_mq_put_tag+0x44/0x70 [ 1807.325766] __blk_mq_free_request+0x88/0xf0 [ 1807.330023] blk_mq_free_request+0xc0/0x160 [ 1807.334192] __blk_mq_end_request+0x78/0x150 [ 1807.338449] scsi_end_request+0x114/0x260 [ 1807.342446] scsi_io_completion+0x64/0x174 [ 1807.346529] scsi_finish_command+0xe0/0x120 [ 1807.350699] scsi_complete+0x8c/0x1c4 [ 1807.354348] blk_complete_reqs+0x5c/0x74 [ 1807.358258] blk_done_softirq+0x38/0x4c [ 1807.362080] __do_softirq+0x168/0x418 [ 1807.365729] __irq_exit_rcu+0x120/0x170 [ 1807.369552] irq_exit_rcu+0x18/0x24 [ 1807.373028] el1_interrupt+0x38/0x70 [ 1807.376590] el1h_64_irq_handler+0x18/0x24 [ 1807.380673] el1h_64_irq+0x68/0x6c [ 1807.384062] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1807.388839] __wake_up_common_lock+0x84/0xac [ 1807.393096] __wake_up+0x20/0x30 [ 1807.396312] __sbq_wake_up+0x13c/0x1f0 [ 1807.400048] sbitmap_queue_clear+0x74/0xc0 [ 1807.404131] blk_mq_put_tag+0x44/0x70 [ 1807.407781] __blk_mq_free_request+0x88/0xf0 [ 1807.412038] blk_mq_free_request+0xc0/0x160 [ 1807.416208] bfq_insert_request.isra.0+0x204/0x270 [ 1807.420986] bfq_insert_requests+0x70/0x90 [ 1807.425069] blk_mq_sched_insert_requests+0xac/0x250 [ 1807.430022] blk_mq_flush_plug_list+0x110/0x2cc [ 1807.434539] __blk_flush_plug+0x100/0x140 [ 1807.438536] blk_finish_plug+0x40/0x25c [ 1807.442359] __iomap_dio_rw+0x1b4/0x500 [ 1807.446182] iomap_dio_rw+0x18/0x50 [ 1807.449657] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1807.455007] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1807.459750] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1807.464794] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1807.469924] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1807.475053] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1807.480440] process_one_work+0x1ec/0x470 [ 1807.484436] worker_thread+0x74/0x410 [ 1807.488085] kthread+0xec/0x100 [ 1807.491213] ret_from_fork+0x10/0x20 [ 1819.212730] watchdog: BUG: soft lockup - CPU#68 stuck for 1300s! [swapper/68:0] [ 1819.220025] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1819.220042] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1819.307828] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1819.315644] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1819.324935] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1819.331883] pc : __sbq_wake_up+0x1c8/0x1f0 [ 1819.335967] lr : sbitmap_queue_clear+0x74/0xc0 [ 1819.340398] sp : ffff8000088438b0 [ 1819.343698] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1819.350820] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1819.357943] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1819.365064] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1819.372186] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1819.379309] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1819.386431] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1819.393554] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1819.400675] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1819.407797] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 1819.414919] Call trace: [ 1819.417352] __sbq_wake_up+0x1c8/0x1f0 [ 1819.421088] sbitmap_queue_clear+0x74/0xc0 [ 1819.425172] blk_mq_put_tag+0x44/0x70 [ 1819.428821] __blk_mq_free_request+0x88/0xf0 [ 1819.433078] blk_mq_free_request+0xc0/0x160 [ 1819.437248] __blk_mq_end_request+0x78/0x150 [ 1819.441505] scsi_end_request+0x114/0x260 [ 1819.445501] scsi_io_completion+0x64/0x174 [ 1819.449584] scsi_finish_command+0xe0/0x120 [ 1819.453756] scsi_complete+0x8c/0x1c4 [ 1819.457405] blk_complete_reqs+0x5c/0x74 [ 1819.461315] blk_done_softirq+0x38/0x4c [ 1819.465137] __do_softirq+0x168/0x418 [ 1819.468786] __irq_exit_rcu+0x120/0x170 [ 1819.472610] irq_exit_rcu+0x18/0x24 [ 1819.476085] el1_interrupt+0x38/0x70 [ 1819.479649] el1h_64_irq_handler+0x18/0x24 [ 1819.483732] el1h_64_irq+0x68/0x6c [ 1819.487120] cpuidle_enter_state+0xc4/0x4e0 [ 1819.491291] cpuidle_enter+0x40/0x60 [ 1819.494853] cpuidle_idle_call+0x118/0x1b0 [ 1819.498936] do_idle+0xac/0x100 [ 1819.502065] cpu_startup_entry+0x30/0x34 [ 1819.505974] secondary_start_kernel+0xd8/0xf0 [ 1819.510318] __secondary_switched+0xb0/0xb4 [ 1831.102650] watchdog: BUG: soft lockup - CPU#8 stuck for 1311s! [kworker/8:12:19399] [ 1831.110379] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1831.110395] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1831.198183] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1831.206432] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1831.215723] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1831.221891] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1831.228839] pc : __sbq_wake_up+0x84/0x1f0 [ 1831.232835] lr : sbitmap_queue_clear+0x74/0xc0 [ 1831.237266] sp : ffff80000d79b280 [ 1831.240567] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1831.247689] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1831.254811] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1831.261933] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1831.269056] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1831.276178] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1831.283300] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1831.290421] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1831.297544] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1831.304665] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1831.311787] Call trace: [ 1831.314220] __sbq_wake_up+0x84/0x1f0 [ 1831.317870] sbitmap_queue_clear+0x74/0xc0 [ 1831.321953] blk_mq_put_tag+0x44/0x70 [ 1831.325602] __blk_mq_free_request+0x88/0xf0 [ 1831.329859] blk_mq_free_request+0xc0/0x160 [ 1831.334029] __blk_mq_end_request+0x78/0x150 [ 1831.338286] scsi_end_request+0x114/0x260 [ 1831.342282] scsi_io_completion+0x64/0x174 [ 1831.346365] scsi_finish_command+0xe0/0x120 [ 1831.350536] scsi_complete+0x8c/0x1c4 [ 1831.354185] blk_complete_reqs+0x5c/0x74 [ 1831.358094] blk_done_softirq+0x38/0x4c [ 1831.361917] __do_softirq+0x168/0x418 [ 1831.365565] __irq_exit_rcu+0x120/0x170 [ 1831.369388] irq_exit_rcu+0x18/0x24 [ 1831.372863] el1_interrupt+0x38/0x70 [ 1831.376426] el1h_64_irq_handler+0x18/0x24 [ 1831.380509] el1h_64_irq+0x68/0x6c [ 1831.383898] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1831.388675] __wake_up_common_lock+0x84/0xac [ 1831.392932] __wake_up+0x20/0x30 [ 1831.396148] __sbq_wake_up+0x13c/0x1f0 [ 1831.399884] sbitmap_queue_clear+0x74/0xc0 [ 1831.403967] blk_mq_put_tag+0x44/0x70 [ 1831.407617] __blk_mq_free_request+0x88/0xf0 [ 1831.411874] blk_mq_free_request+0xc0/0x160 [ 1831.416044] bfq_insert_request.isra.0+0x204/0x270 [ 1831.420822] bfq_insert_requests+0x70/0x90 [ 1831.424905] blk_mq_sched_insert_requests+0xac/0x250 [ 1831.429858] blk_mq_flush_plug_list+0x110/0x2cc [ 1831.434375] __blk_flush_plug+0x100/0x140 [ 1831.438372] blk_finish_plug+0x40/0x25c [ 1831.442194] __iomap_dio_rw+0x1b4/0x500 [ 1831.446017] iomap_dio_rw+0x18/0x50 [ 1831.449492] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1831.454843] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1831.459585] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1831.464629] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1831.469759] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1831.474889] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1831.480275] process_one_work+0x1ec/0x470 [ 1831.484271] worker_thread+0x74/0x410 [ 1831.487920] kthread+0xec/0x100 [ 1831.491049] ret_from_fork+0x10/0x20 [ 1843.212568] watchdog: BUG: soft lockup - CPU#68 stuck for 1322s! [swapper/68:0] [ 1843.219863] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1843.219879] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1843.307666] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1843.315482] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1843.324773] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1843.331721] pc : __sbq_wake_up+0x20/0x1f0 [ 1843.335717] lr : sbitmap_queue_clear+0x74/0xc0 [ 1843.340147] sp : ffff8000088438b0 [ 1843.343449] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1843.350570] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1843.357692] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1843.364814] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1843.371936] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1843.379058] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1843.386180] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1843.393301] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1843.400423] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1843.407545] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1843.414667] Call trace: [ 1843.417100] __sbq_wake_up+0x20/0x1f0 [ 1843.420749] sbitmap_queue_clear+0x74/0xc0 [ 1843.424832] blk_mq_put_tag+0x44/0x70 [ 1843.428482] __blk_mq_free_request+0x88/0xf0 [ 1843.432739] blk_mq_free_request+0xc0/0x160 [ 1843.436909] __blk_mq_end_request+0x78/0x150 [ 1843.441165] scsi_end_request+0x114/0x260 [ 1843.445162] scsi_io_completion+0x64/0x174 [ 1843.449245] scsi_finish_command+0xe0/0x120 [ 1843.453415] scsi_complete+0x8c/0x1c4 [ 1843.457064] blk_complete_reqs+0x5c/0x74 [ 1843.460974] blk_done_softirq+0x38/0x4c [ 1843.464796] __do_softirq+0x168/0x418 [ 1843.468445] __irq_exit_rcu+0x120/0x170 [ 1843.472268] irq_exit_rcu+0x18/0x24 [ 1843.475743] el1_interrupt+0x38/0x70 [ 1843.479305] el1h_64_irq_handler+0x18/0x24 [ 1843.483388] el1h_64_irq+0x68/0x6c [ 1843.486776] cpuidle_enter_state+0xc4/0x4e0 [ 1843.490946] cpuidle_enter+0x40/0x60 [ 1843.494509] cpuidle_idle_call+0x118/0x1b0 [ 1843.498592] do_idle+0xac/0x100 [ 1843.501721] cpu_startup_entry+0x30/0x34 [ 1843.505630] secondary_start_kernel+0xd8/0xf0 [ 1843.509974] __secondary_switched+0xb0/0xb4 [ 1855.102488] watchdog: BUG: soft lockup - CPU#8 stuck for 1333s! [kworker/8:12:19399] [ 1855.110217] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1855.110234] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1855.198020] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1855.206270] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1855.215561] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1855.221728] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1855.228676] pc : __sbq_wake_up+0x84/0x1f0 [ 1855.232673] lr : sbitmap_queue_clear+0x74/0xc0 [ 1855.237103] sp : ffff80000d79b280 [ 1855.240404] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1855.247526] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1855.254648] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1855.261769] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1855.268892] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1855.276014] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1855.283136] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1855.290257] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1855.297379] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1855.304501] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1855.311623] Call trace: [ 1855.314055] __sbq_wake_up+0x84/0x1f0 [ 1855.317705] sbitmap_queue_clear+0x74/0xc0 [ 1855.321788] blk_mq_put_tag+0x44/0x70 [ 1855.325438] __blk_mq_free_request+0x88/0xf0 [ 1855.329695] blk_mq_free_request+0xc0/0x160 [ 1855.333864] __blk_mq_end_request+0x78/0x150 [ 1855.338121] scsi_end_request+0x114/0x260 [ 1855.342117] scsi_io_completion+0x64/0x174 [ 1855.346201] scsi_finish_command+0xe0/0x120 [ 1855.350371] scsi_complete+0x8c/0x1c4 [ 1855.354020] blk_complete_reqs+0x5c/0x74 [ 1855.357930] blk_done_softirq+0x38/0x4c [ 1855.361752] __do_softirq+0x168/0x418 [ 1855.365401] __irq_exit_rcu+0x120/0x170 [ 1855.369224] irq_exit_rcu+0x18/0x24 [ 1855.372699] el1_interrupt+0x38/0x70 [ 1855.376262] el1h_64_irq_handler+0x18/0x24 [ 1855.380345] el1h_64_irq+0x68/0x6c [ 1855.383734] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1855.388512] __wake_up_common_lock+0x84/0xac [ 1855.392769] __wake_up+0x20/0x30 [ 1855.395984] __sbq_wake_up+0x13c/0x1f0 [ 1855.399721] sbitmap_queue_clear+0x74/0xc0 [ 1855.403804] blk_mq_put_tag+0x44/0x70 [ 1855.407454] __blk_mq_free_request+0x88/0xf0 [ 1855.411710] blk_mq_free_request+0xc0/0x160 [ 1855.415880] bfq_insert_request.isra.0+0x204/0x270 [ 1855.420658] bfq_insert_requests+0x70/0x90 [ 1855.424742] blk_mq_sched_insert_requests+0xac/0x250 [ 1855.429694] blk_mq_flush_plug_list+0x110/0x2cc [ 1855.434212] __blk_flush_plug+0x100/0x140 [ 1855.438208] blk_finish_plug+0x40/0x25c [ 1855.442031] __iomap_dio_rw+0x1b4/0x500 [ 1855.445854] iomap_dio_rw+0x18/0x50 [ 1855.449329] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1855.454679] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1855.459422] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1855.464466] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1855.469596] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1855.474725] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1855.480111] process_one_work+0x1ec/0x470 [ 1855.484108] worker_thread+0x74/0x410 [ 1855.487756] kthread+0xec/0x100 [ 1855.490885] ret_from_fork+0x10/0x20 [ 1867.212406] watchdog: BUG: soft lockup - CPU#68 stuck for 1344s! [swapper/68:0] [ 1867.219701] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1867.219718] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1867.307504] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1867.315320] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1867.324611] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1867.331558] pc : __sbq_wake_up+0x20/0x1f0 [ 1867.335555] lr : sbitmap_queue_clear+0x74/0xc0 [ 1867.339986] sp : ffff8000088438b0 [ 1867.343287] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1867.350409] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1867.357531] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1867.364653] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1867.371775] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1867.378897] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1867.386018] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1867.393140] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1867.400262] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1867.407384] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1867.414507] Call trace: [ 1867.416940] __sbq_wake_up+0x20/0x1f0 [ 1867.420589] sbitmap_queue_clear+0x74/0xc0 [ 1867.424673] blk_mq_put_tag+0x44/0x70 [ 1867.428323] __blk_mq_free_request+0x88/0xf0 [ 1867.432579] blk_mq_free_request+0xc0/0x160 [ 1867.436749] __blk_mq_end_request+0x78/0x150 [ 1867.441006] scsi_end_request+0x114/0x260 [ 1867.445002] scsi_io_completion+0x64/0x174 [ 1867.449086] scsi_finish_command+0xe0/0x120 [ 1867.453256] scsi_complete+0x8c/0x1c4 [ 1867.456905] blk_complete_reqs+0x5c/0x74 [ 1867.460815] blk_done_softirq+0x38/0x4c [ 1867.464638] __do_softirq+0x168/0x418 [ 1867.468287] __irq_exit_rcu+0x120/0x170 [ 1867.472110] irq_exit_rcu+0x18/0x24 [ 1867.475586] el1_interrupt+0x38/0x70 [ 1867.479149] el1h_64_irq_handler+0x18/0x24 [ 1867.483232] el1h_64_irq+0x68/0x6c [ 1867.486620] cpuidle_enter_state+0xc4/0x4e0 [ 1867.490790] cpuidle_enter+0x40/0x60 [ 1867.494353] cpuidle_idle_call+0x118/0x1b0 [ 1867.498436] do_idle+0xac/0x100 [ 1867.501565] cpu_startup_entry+0x30/0x34 [ 1867.505474] secondary_start_kernel+0xd8/0xf0 [ 1867.509818] __secondary_switched+0xb0/0xb4 [ 1869.802398] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 141885 jiffies s: 2093 root: 0x11/. [ 1869.814136] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1869.823344] Task dump for CPU 8: [ 1869.826559] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1869.836461] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1869.842630] Call trace: [ 1869.845063] __switch_to+0xd4/0x100 [ 1869.848540] 0xffff6c0200000080 [ 1869.851668] Task dump for CPU 68: [ 1869.854971] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1869.864872] Call trace: [ 1869.867305] __switch_to+0xd4/0x100 [ 1869.870782] 0x0 [ 1879.102326] watchdog: BUG: soft lockup - CPU#8 stuck for 1356s! [kworker/8:12:19399] [ 1879.110055] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1879.110072] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1879.197859] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1879.206108] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1879.215399] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1879.221567] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1879.228514] pc : __sbq_wake_up+0x58/0x1f0 [ 1879.232511] lr : sbitmap_queue_clear+0x74/0xc0 [ 1879.236942] sp : ffff80000d79b280 [ 1879.240244] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1879.247366] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1879.254489] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1879.261611] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1879.268733] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1879.275855] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1879.282976] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1879.290098] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1879.297220] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1879.304342] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1879.311464] Call trace: [ 1879.313897] __sbq_wake_up+0x58/0x1f0 [ 1879.317547] sbitmap_queue_clear+0x74/0xc0 [ 1879.321630] blk_mq_put_tag+0x44/0x70 [ 1879.325279] __blk_mq_free_request+0x88/0xf0 [ 1879.329536] blk_mq_free_request+0xc0/0x160 [ 1879.333707] __blk_mq_end_request+0x78/0x150 [ 1879.337965] scsi_end_request+0x114/0x260 [ 1879.341961] scsi_io_completion+0x64/0x174 [ 1879.346044] scsi_finish_command+0xe0/0x120 [ 1879.350215] scsi_complete+0x8c/0x1c4 [ 1879.353864] blk_complete_reqs+0x5c/0x74 [ 1879.357773] blk_done_softirq+0x38/0x4c [ 1879.361596] __do_softirq+0x168/0x418 [ 1879.365245] __irq_exit_rcu+0x120/0x170 [ 1879.369068] irq_exit_rcu+0x18/0x24 [ 1879.372543] el1_interrupt+0x38/0x70 [ 1879.376106] el1h_64_irq_handler+0x18/0x24 [ 1879.380189] el1h_64_irq+0x68/0x6c [ 1879.383578] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1879.388355] __wake_up_common_lock+0x84/0xac [ 1879.392612] __wake_up+0x20/0x30 [ 1879.395828] __sbq_wake_up+0x13c/0x1f0 [ 1879.399564] sbitmap_queue_clear+0x74/0xc0 [ 1879.403648] blk_mq_put_tag+0x44/0x70 [ 1879.407297] __blk_mq_free_request+0x88/0xf0 [ 1879.411554] blk_mq_free_request+0xc0/0x160 [ 1879.415724] bfq_insert_request.isra.0+0x204/0x270 [ 1879.420502] bfq_insert_requests+0x70/0x90 [ 1879.424585] blk_mq_sched_insert_requests+0xac/0x250 [ 1879.429537] blk_mq_flush_plug_list+0x110/0x2cc [ 1879.434055] __blk_flush_plug+0x100/0x140 [ 1879.438051] blk_finish_plug+0x40/0x25c [ 1879.441874] __iomap_dio_rw+0x1b4/0x500 [ 1879.445697] iomap_dio_rw+0x18/0x50 [ 1879.449173] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1879.454524] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1879.459266] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1879.464310] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1879.469441] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1879.474571] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1879.479957] process_one_work+0x1ec/0x470 [ 1879.483953] worker_thread+0x74/0x410 [ 1879.487602] kthread+0xec/0x100 [ 1879.490731] ret_from_fork+0x10/0x20 [ 1891.212244] watchdog: BUG: soft lockup - CPU#68 stuck for 1367s! [swapper/68:0] [ 1891.219539] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1891.219556] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1891.307342] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1891.315158] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1891.324449] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1891.331397] pc : __sbq_wake_up+0x40/0x1f0 [ 1891.335393] lr : sbitmap_queue_clear+0x74/0xc0 [ 1891.339824] sp : ffff8000088438b0 [ 1891.343125] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1891.350247] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1891.357369] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1891.364491] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1891.371613] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1891.378735] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1891.385856] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1891.392978] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1891.400100] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1891.407222] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 1891.414344] Call trace: [ 1891.416777] __sbq_wake_up+0x40/0x1f0 [ 1891.420426] sbitmap_queue_clear+0x74/0xc0 [ 1891.424510] blk_mq_put_tag+0x44/0x70 [ 1891.428159] __blk_mq_free_request+0x88/0xf0 [ 1891.432416] blk_mq_free_request+0xc0/0x160 [ 1891.436586] __blk_mq_end_request+0x78/0x150 [ 1891.440844] scsi_end_request+0x114/0x260 [ 1891.444840] scsi_io_completion+0x64/0x174 [ 1891.448923] scsi_finish_command+0xe0/0x120 [ 1891.453094] scsi_complete+0x8c/0x1c4 [ 1891.456743] blk_complete_reqs+0x5c/0x74 [ 1891.460652] blk_done_softirq+0x38/0x4c [ 1891.464475] __do_softirq+0x168/0x418 [ 1891.468125] __irq_exit_rcu+0x120/0x170 [ 1891.471947] irq_exit_rcu+0x18/0x24 [ 1891.475423] el1_interrupt+0x38/0x70 [ 1891.478985] el1h_64_irq_handler+0x18/0x24 [ 1891.483068] el1h_64_irq+0x68/0x6c [ 1891.486457] cpuidle_enter_state+0xc4/0x4e0 [ 1891.490627] cpuidle_enter+0x40/0x60 [ 1891.494189] cpuidle_idle_call+0x118/0x1b0 [ 1891.498273] do_idle+0xac/0x100 [ 1891.501401] cpu_startup_entry+0x30/0x34 [ 1891.505311] secondary_start_kernel+0xd8/0xf0 [ 1891.509655] __secondary_switched+0xb0/0xb4 [ 1903.102164] watchdog: BUG: soft lockup - CPU#8 stuck for 1378s! [kworker/8:12:19399] [ 1903.109893] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1903.109910] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1903.197697] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1903.205947] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1903.215238] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1903.221406] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1903.228354] pc : __sbq_wake_up+0x58/0x1f0 [ 1903.232350] lr : sbitmap_queue_clear+0x74/0xc0 [ 1903.236781] sp : ffff80000d79b280 [ 1903.240082] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1903.247204] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1903.254326] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1903.261448] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1903.268570] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1903.275692] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1903.282814] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1903.289936] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1903.297058] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1903.304180] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1903.311301] Call trace: [ 1903.313735] __sbq_wake_up+0x58/0x1f0 [ 1903.317384] sbitmap_queue_clear+0x74/0xc0 [ 1903.321467] blk_mq_put_tag+0x44/0x70 [ 1903.325117] __blk_mq_free_request+0x88/0xf0 [ 1903.329374] blk_mq_free_request+0xc0/0x160 [ 1903.333544] __blk_mq_end_request+0x78/0x150 [ 1903.337800] scsi_end_request+0x114/0x260 [ 1903.341797] scsi_io_completion+0x64/0x174 [ 1903.345880] scsi_finish_command+0xe0/0x120 [ 1903.350051] scsi_complete+0x8c/0x1c4 [ 1903.353700] blk_complete_reqs+0x5c/0x74 [ 1903.357609] blk_done_softirq+0x38/0x4c [ 1903.361432] __do_softirq+0x168/0x418 [ 1903.365081] __irq_exit_rcu+0x120/0x170 [ 1903.368904] irq_exit_rcu+0x18/0x24 [ 1903.372379] el1_interrupt+0x38/0x70 [ 1903.375942] el1h_64_irq_handler+0x18/0x24 [ 1903.380026] el1h_64_irq+0x68/0x6c [ 1903.383415] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1903.388192] __wake_up_common_lock+0x84/0xac [ 1903.392450] __wake_up+0x20/0x30 [ 1903.395665] __sbq_wake_up+0x13c/0x1f0 [ 1903.399401] sbitmap_queue_clear+0x74/0xc0 [ 1903.403485] blk_mq_put_tag+0x44/0x70 [ 1903.407134] __blk_mq_free_request+0x88/0xf0 [ 1903.411391] blk_mq_free_request+0xc0/0x160 [ 1903.415561] bfq_insert_request.isra.0+0x204/0x270 [ 1903.420339] bfq_insert_requests+0x70/0x90 [ 1903.424423] blk_mq_sched_insert_requests+0xac/0x250 [ 1903.429376] blk_mq_flush_plug_list+0x110/0x2cc [ 1903.433894] __blk_flush_plug+0x100/0x140 [ 1903.437890] blk_finish_plug+0x40/0x25c [ 1903.441713] __iomap_dio_rw+0x1b4/0x500 [ 1903.445536] iomap_dio_rw+0x18/0x50 [ 1903.449012] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1903.454362] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1903.459105] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1903.464148] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1903.469278] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1903.474408] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1903.479793] process_one_work+0x1ec/0x470 [ 1903.483790] worker_thread+0x74/0x410 [ 1903.487438] kthread+0xec/0x100 [ 1903.490567] ret_from_fork+0x10/0x20 [ 1915.212083] watchdog: BUG: soft lockup - CPU#68 stuck for 1389s! [swapper/68:0] [ 1915.219378] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1915.219394] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1915.307181] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1915.314997] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1915.324288] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1915.331235] pc : __sbq_wake_up+0x20/0x1f0 [ 1915.335232] lr : sbitmap_queue_clear+0x74/0xc0 [ 1915.339662] sp : ffff8000088438b0 [ 1915.342963] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1915.350084] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1915.357206] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1915.364328] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1915.371450] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1915.378572] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1915.385694] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1915.392815] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1915.399937] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1915.407059] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1915.414181] Call trace: [ 1915.416614] __sbq_wake_up+0x20/0x1f0 [ 1915.420263] sbitmap_queue_clear+0x74/0xc0 [ 1915.424346] blk_mq_put_tag+0x44/0x70 [ 1915.427996] __blk_mq_free_request+0x88/0xf0 [ 1915.432253] blk_mq_free_request+0xc0/0x160 [ 1915.436423] __blk_mq_end_request+0x78/0x150 [ 1915.440679] scsi_end_request+0x114/0x260 [ 1915.444676] scsi_io_completion+0x64/0x174 [ 1915.448759] scsi_finish_command+0xe0/0x120 [ 1915.452929] scsi_complete+0x8c/0x1c4 [ 1915.456578] blk_complete_reqs+0x5c/0x74 [ 1915.460488] blk_done_softirq+0x38/0x4c [ 1915.464311] __do_softirq+0x168/0x418 [ 1915.467960] __irq_exit_rcu+0x120/0x170 [ 1915.471783] irq_exit_rcu+0x18/0x24 [ 1915.475258] el1_interrupt+0x38/0x70 [ 1915.478820] el1h_64_irq_handler+0x18/0x24 [ 1915.482904] el1h_64_irq+0x68/0x6c [ 1915.486292] cpuidle_enter_state+0xc4/0x4e0 [ 1915.490462] cpuidle_enter+0x40/0x60 [ 1915.494024] cpuidle_idle_call+0x118/0x1b0 [ 1915.498108] do_idle+0xac/0x100 [ 1915.501237] cpu_startup_entry+0x30/0x34 [ 1915.505146] secondary_start_kernel+0xd8/0xf0 [ 1915.509490] __secondary_switched+0xb0/0xb4 [ 1927.102002] watchdog: BUG: soft lockup - CPU#8 stuck for 1400s! [kworker/8:12:19399] [ 1927.109731] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1927.109748] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1927.197535] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1927.205785] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1927.215076] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1927.221243] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1927.228192] pc : __sbq_wake_up+0x58/0x1f0 [ 1927.232188] lr : sbitmap_queue_clear+0x74/0xc0 [ 1927.236619] sp : ffff80000d79b280 [ 1927.239920] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1927.247042] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1927.254165] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1927.261287] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1927.268410] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1927.275532] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1927.282653] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1927.289775] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1927.296897] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1927.304019] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1927.311141] Call trace: [ 1927.313576] __sbq_wake_up+0x58/0x1f0 [ 1927.317225] sbitmap_queue_clear+0x74/0xc0 [ 1927.321308] blk_mq_put_tag+0x44/0x70 [ 1927.324957] __blk_mq_free_request+0x88/0xf0 [ 1927.329214] blk_mq_free_request+0xc0/0x160 [ 1927.333384] __blk_mq_end_request+0x78/0x150 [ 1927.337641] scsi_end_request+0x114/0x260 [ 1927.341639] scsi_io_completion+0x64/0x174 [ 1927.345722] scsi_finish_command+0xe0/0x120 [ 1927.349892] scsi_complete+0x8c/0x1c4 [ 1927.353541] blk_complete_reqs+0x5c/0x74 [ 1927.357451] blk_done_softirq+0x38/0x4c [ 1927.361273] __do_softirq+0x168/0x418 [ 1927.364922] __irq_exit_rcu+0x120/0x170 [ 1927.368745] irq_exit_rcu+0x18/0x24 [ 1927.372220] el1_interrupt+0x38/0x70 [ 1927.375782] el1h_64_irq_handler+0x18/0x24 [ 1927.379865] el1h_64_irq+0x68/0x6c [ 1927.383254] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1927.388032] __wake_up_common_lock+0x84/0xac [ 1927.392289] __wake_up+0x20/0x30 [ 1927.395505] __sbq_wake_up+0x13c/0x1f0 [ 1927.399240] sbitmap_queue_clear+0x74/0xc0 [ 1927.403323] blk_mq_put_tag+0x44/0x70 [ 1927.406973] __blk_mq_free_request+0x88/0xf0 [ 1927.411230] blk_mq_free_request+0xc0/0x160 [ 1927.415400] bfq_insert_request.isra.0+0x204/0x270 [ 1927.420178] bfq_insert_requests+0x70/0x90 [ 1927.424261] blk_mq_sched_insert_requests+0xac/0x250 [ 1927.429213] blk_mq_flush_plug_list+0x110/0x2cc [ 1927.432000] rcu: INFO: rcu_preempt self-detected stall on CPU [ 1927.433731] __blk_flush_plug+0x100/0x140 [ 1927.439462] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=74503 [ 1927.443457] blk_finish_plug+0x40/0x25c [ 1927.452576] (t=150254 jiffies g=36445 q=44185 ncpus=128) [ 1927.456396] __iomap_dio_rw+0x1b4/0x500 [ 1927.461780] Task dump for CPU 8: [ 1927.465601] iomap_dio_rw+0x18/0x50 [ 1927.468815] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1927.472290] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1927.482189] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1927.487487] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1927.493653] Call trace: [ 1927.498343] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1927.500777] __switch_to+0xd4/0x100 [ 1927.505814] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1927.509288] 0xffff6c0200000080 [ 1927.514412] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1927.517540] Task dump for CPU 68: [ 1927.522664] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1927.525965] task:swapper/68 state:R running task [ 1927.531349] process_one_work+0x1ec/0x470 [ 1927.536821] stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1927.540816] worker_thread+0x74/0x410 [ 1927.546721] Call trace: [ 1927.550369] kthread+0xec/0x100 [ 1927.552802] dump_backtrace+0xe8/0x130 [ 1927.555929] ret_from_fork+0x10/0x20 [ 1927.559664] show_stack+0x20/0x60 [ 1927.566526] sched_show_task+0x160/0x190 [ 1927.570435] dump_cpu_task+0x4c/0x78 [ 1927.573998] rcu_dump_cpu_stacks+0xc4/0x150 [ 1927.578169] print_cpu_stall+0x180/0x210 [ 1927.582078] check_cpu_stall+0x20c/0x2b4 [ 1927.585987] rcu_pending+0x4c/0x190 [ 1927.589463] rcu_sched_clock_irq+0x80/0x204 [ 1927.593632] update_process_times+0x7c/0xc0 [ 1927.597802] tick_sched_handle+0x38/0x74 [ 1927.601712] tick_sched_timer+0x60/0xc0 [ 1927.605536] __hrtimer_run_queues+0x170/0x300 [ 1927.609879] hrtimer_interrupt+0xfc/0x260 [ 1927.613875] arch_timer_handler_phys+0x34/0x50 [ 1927.618306] handle_percpu_devid_irq+0xac/0x240 [ 1927.622823] generic_handle_domain_irq+0x34/0x50 [ 1927.627428] gic_handle_irq+0x5c/0xe0 [ 1927.631078] call_on_irq_stack+0x2c/0x38 [ 1927.634987] do_interrupt_handler+0xdc/0xe0 [ 1927.639157] el1_interrupt+0x34/0x70 [ 1927.642720] el1h_64_irq_handler+0x18/0x24 [ 1927.646803] el1h_64_irq+0x68/0x6c [ 1927.650192] __sbq_wake_up+0x40/0x1f0 [ 1927.653841] sbitmap_queue_clear+0x74/0xc0 [ 1927.657924] blk_mq_put_tag+0x44/0x70 [ 1927.661574] __blk_mq_free_request+0x88/0xf0 [ 1927.665832] blk_mq_free_request+0xc0/0x160 [ 1927.670002] __blk_mq_end_request+0x78/0x150 [ 1927.674259] scsi_end_request+0x114/0x260 [ 1927.678255] scsi_io_completion+0x64/0x174 [ 1927.682338] scsi_finish_command+0xe0/0x120 [ 1927.686509] scsi_complete+0x8c/0x1c4 [ 1927.690158] blk_complete_reqs+0x5c/0x74 [ 1927.694067] blk_done_softirq+0x38/0x4c [ 1927.697890] __do_softirq+0x168/0x418 [ 1927.701539] __irq_exit_rcu+0x120/0x170 [ 1927.705363] irq_exit_rcu+0x18/0x24 [ 1927.708838] el1_interrupt+0x38/0x70 [ 1927.712402] el1h_64_irq_handler+0x18/0x24 [ 1927.716485] el1h_64_irq+0x68/0x6c [ 1927.719873] cpuidle_enter_state+0xc4/0x4e0 [ 1927.724044] cpuidle_enter+0x40/0x60 [ 1927.727606] cpuidle_idle_call+0x118/0x1b0 [ 1927.731690] do_idle+0xac/0x100 [ 1927.734818] cpu_startup_entry+0x30/0x34 [ 1927.738727] secondary_start_kernel+0xd8/0xf0 [ 1927.743071] __secondary_switched+0xb0/0xb4 [ 1936.361948] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 148541 jiffies s: 2093 root: 0x11/. [ 1936.373686] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 1936.382895] Task dump for CPU 8: [ 1936.386109] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 1936.396011] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1936.402180] Call trace: [ 1936.404613] __switch_to+0xd4/0x100 [ 1936.408090] 0xffff6c0200000080 [ 1936.411218] Task dump for CPU 68: [ 1936.414520] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 1936.424422] Call trace: [ 1936.426855] __switch_to+0xd4/0x100 [ 1936.430331] 0x0 [ 1951.101836] watchdog: BUG: soft lockup - CPU#8 stuck for 1423s! [kworker/8:12:19399] [ 1951.109565] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1951.109582] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1951.197369] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1951.205619] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1951.214910] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1951.221077] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1951.228025] pc : __sbq_wake_up+0x58/0x1f0 [ 1951.232021] lr : sbitmap_queue_clear+0x74/0xc0 [ 1951.236452] sp : ffff80000d79b280 [ 1951.239754] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1951.246876] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1951.253998] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1951.261120] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1951.268242] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1951.275366] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1951.282488] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1951.289610] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1951.296731] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1951.303853] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1951.310975] Call trace: [ 1951.313408] __sbq_wake_up+0x58/0x1f0 [ 1951.317058] sbitmap_queue_clear+0x74/0xc0 [ 1951.321141] blk_mq_put_tag+0x44/0x70 [ 1951.324792] __blk_mq_free_request+0x88/0xf0 [ 1951.329049] blk_mq_free_request+0xc0/0x160 [ 1951.333219] __blk_mq_end_request+0x78/0x150 [ 1951.337476] scsi_end_request+0x114/0x260 [ 1951.341472] scsi_io_completion+0x64/0x174 [ 1951.345555] scsi_finish_command+0xe0/0x120 [ 1951.349726] scsi_complete+0x8c/0x1c4 [ 1951.353375] blk_complete_reqs+0x5c/0x74 [ 1951.357284] blk_done_softirq+0x38/0x4c [ 1951.361107] __do_softirq+0x168/0x418 [ 1951.364756] __irq_exit_rcu+0x120/0x170 [ 1951.368578] irq_exit_rcu+0x18/0x24 [ 1951.372054] el1_interrupt+0x38/0x70 [ 1951.375616] el1h_64_irq_handler+0x18/0x24 [ 1951.379699] el1h_64_irq+0x68/0x6c [ 1951.383087] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1951.387865] __wake_up_common_lock+0x84/0xac [ 1951.392122] __wake_up+0x20/0x30 [ 1951.395338] __sbq_wake_up+0x13c/0x1f0 [ 1951.399074] sbitmap_queue_clear+0x74/0xc0 [ 1951.403157] blk_mq_put_tag+0x44/0x70 [ 1951.406807] __blk_mq_free_request+0x88/0xf0 [ 1951.411064] blk_mq_free_request+0xc0/0x160 [ 1951.415234] bfq_insert_request.isra.0+0x204/0x270 [ 1951.420012] bfq_insert_requests+0x70/0x90 [ 1951.424095] blk_mq_sched_insert_requests+0xac/0x250 [ 1951.429047] blk_mq_flush_plug_list+0x110/0x2cc [ 1951.433565] __blk_flush_plug+0x100/0x140 [ 1951.437562] blk_finish_plug+0x40/0x25c [ 1951.441384] __iomap_dio_rw+0x1b4/0x500 [ 1951.445207] iomap_dio_rw+0x18/0x50 [ 1951.448682] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1951.454033] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1951.458776] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1951.463820] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1951.468950] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1951.474079] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1951.479465] process_one_work+0x1ec/0x470 [ 1951.483461] worker_thread+0x74/0x410 [ 1951.487110] kthread+0xec/0x100 [ 1951.490239] ret_from_fork+0x10/0x20 [ 1955.211808] watchdog: BUG: soft lockup - CPU#68 stuck for 1426s! [swapper/68:0] [ 1955.219103] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1955.219120] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1955.306906] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1955.314722] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1955.324013] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1955.330960] pc : __sbq_wake_up+0x20/0x1f0 [ 1955.334957] lr : sbitmap_queue_clear+0x74/0xc0 [ 1955.339387] sp : ffff8000088438b0 [ 1955.342688] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1955.349811] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1955.356932] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 1955.364054] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1955.371176] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1955.378298] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1955.385420] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1955.392541] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1955.399663] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1955.406785] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 1955.413907] Call trace: [ 1955.416340] __sbq_wake_up+0x20/0x1f0 [ 1955.419989] sbitmap_queue_clear+0x74/0xc0 [ 1955.424072] blk_mq_put_tag+0x44/0x70 [ 1955.427722] __blk_mq_free_request+0x88/0xf0 [ 1955.431980] blk_mq_free_request+0xc0/0x160 [ 1955.436150] __blk_mq_end_request+0x78/0x150 [ 1955.440406] scsi_end_request+0x114/0x260 [ 1955.444403] scsi_io_completion+0x64/0x174 [ 1955.448486] scsi_finish_command+0xe0/0x120 [ 1955.452656] scsi_complete+0x8c/0x1c4 [ 1955.456305] blk_complete_reqs+0x5c/0x74 [ 1955.460215] blk_done_softirq+0x38/0x4c [ 1955.464037] __do_softirq+0x168/0x418 [ 1955.467686] __irq_exit_rcu+0x120/0x170 [ 1955.471509] irq_exit_rcu+0x18/0x24 [ 1955.474984] el1_interrupt+0x38/0x70 [ 1955.478547] el1h_64_irq_handler+0x18/0x24 [ 1955.482630] el1h_64_irq+0x68/0x6c [ 1955.486018] cpuidle_enter_state+0xc4/0x4e0 [ 1955.490188] cpuidle_enter+0x40/0x60 [ 1955.493750] cpuidle_idle_call+0x118/0x1b0 [ 1955.497834] do_idle+0xac/0x100 [ 1955.500963] cpu_startup_entry+0x30/0x34 [ 1955.504872] secondary_start_kernel+0xd8/0xf0 [ 1955.509217] __secondary_switched+0xb0/0xb4 [ 1975.101671] watchdog: BUG: soft lockup - CPU#8 stuck for 1445s! [kworker/8:12:19399] [ 1975.109400] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1975.109417] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1975.197203] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1975.205453] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1975.214744] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1975.220911] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1975.227859] pc : __sbq_wake_up+0x84/0x1f0 [ 1975.231856] lr : sbitmap_queue_clear+0x74/0xc0 [ 1975.236286] sp : ffff80000d79b280 [ 1975.239587] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1975.246710] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1975.253832] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1975.260954] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1975.268075] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1975.275197] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1975.282319] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1975.289441] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1975.296563] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1975.303684] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 1975.310806] Call trace: [ 1975.313239] __sbq_wake_up+0x84/0x1f0 [ 1975.316888] sbitmap_queue_clear+0x74/0xc0 [ 1975.320971] blk_mq_put_tag+0x44/0x70 [ 1975.324621] __blk_mq_free_request+0x88/0xf0 [ 1975.328878] blk_mq_free_request+0xc0/0x160 [ 1975.333047] __blk_mq_end_request+0x78/0x150 [ 1975.337304] scsi_end_request+0x114/0x260 [ 1975.341300] scsi_io_completion+0x64/0x174 [ 1975.345384] scsi_finish_command+0xe0/0x120 [ 1975.349554] scsi_complete+0x8c/0x1c4 [ 1975.353203] blk_complete_reqs+0x5c/0x74 [ 1975.357113] blk_done_softirq+0x38/0x4c [ 1975.360935] __do_softirq+0x168/0x418 [ 1975.364584] __irq_exit_rcu+0x120/0x170 [ 1975.368407] irq_exit_rcu+0x18/0x24 [ 1975.371883] el1_interrupt+0x38/0x70 [ 1975.375446] el1h_64_irq_handler+0x18/0x24 [ 1975.379530] el1h_64_irq+0x68/0x6c [ 1975.382919] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1975.387696] __wake_up_common_lock+0x84/0xac [ 1975.391953] __wake_up+0x20/0x30 [ 1975.395169] __sbq_wake_up+0x13c/0x1f0 [ 1975.398905] sbitmap_queue_clear+0x74/0xc0 [ 1975.402989] blk_mq_put_tag+0x44/0x70 [ 1975.406638] __blk_mq_free_request+0x88/0xf0 [ 1975.410895] blk_mq_free_request+0xc0/0x160 [ 1975.415065] bfq_insert_request.isra.0+0x204/0x270 [ 1975.419843] bfq_insert_requests+0x70/0x90 [ 1975.423927] blk_mq_sched_insert_requests+0xac/0x250 [ 1975.428878] blk_mq_flush_plug_list+0x110/0x2cc [ 1975.433396] __blk_flush_plug+0x100/0x140 [ 1975.437392] blk_finish_plug+0x40/0x25c [ 1975.441215] __iomap_dio_rw+0x1b4/0x500 [ 1975.445038] iomap_dio_rw+0x18/0x50 [ 1975.448513] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1975.453864] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1975.458606] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1975.463650] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1975.468780] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1975.473909] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1975.479296] process_one_work+0x1ec/0x470 [ 1975.483292] worker_thread+0x74/0x410 [ 1975.486941] kthread+0xec/0x100 [ 1975.490069] ret_from_fork+0x10/0x20 [ 1979.211642] watchdog: BUG: soft lockup - CPU#68 stuck for 1449s! [swapper/68:0] [ 1979.218938] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1979.218954] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1979.306741] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 1979.314557] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1979.323848] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1979.330795] pc : __sbq_wake_up+0x58/0x1f0 [ 1979.334792] lr : sbitmap_queue_clear+0x74/0xc0 [ 1979.339223] sp : ffff8000088438b0 [ 1979.342524] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1979.349646] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 1979.356768] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 1979.363890] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 1979.371012] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 1979.378133] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 1979.385255] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 1979.392377] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1979.399499] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1979.406621] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 1979.413743] Call trace: [ 1979.416175] __sbq_wake_up+0x58/0x1f0 [ 1979.419825] sbitmap_queue_clear+0x74/0xc0 [ 1979.423910] blk_mq_put_tag+0x44/0x70 [ 1979.427559] __blk_mq_free_request+0x88/0xf0 [ 1979.431816] blk_mq_free_request+0xc0/0x160 [ 1979.435986] __blk_mq_end_request+0x78/0x150 [ 1979.440242] scsi_end_request+0x114/0x260 [ 1979.444239] scsi_io_completion+0x64/0x174 [ 1979.448322] scsi_finish_command+0xe0/0x120 [ 1979.452492] scsi_complete+0x8c/0x1c4 [ 1979.456141] blk_complete_reqs+0x5c/0x74 [ 1979.460051] blk_done_softirq+0x38/0x4c [ 1979.463874] __do_softirq+0x168/0x418 [ 1979.467522] __irq_exit_rcu+0x120/0x170 [ 1979.471344] irq_exit_rcu+0x18/0x24 [ 1979.474820] el1_interrupt+0x38/0x70 [ 1979.478382] el1h_64_irq_handler+0x18/0x24 [ 1979.482465] el1h_64_irq+0x68/0x6c [ 1979.485853] cpuidle_enter_state+0xc4/0x4e0 [ 1979.490024] cpuidle_enter+0x40/0x60 [ 1979.493586] cpuidle_idle_call+0x118/0x1b0 [ 1979.497670] do_idle+0xac/0x100 [ 1979.500799] cpu_startup_entry+0x30/0x34 [ 1979.504708] secondary_start_kernel+0xd8/0xf0 [ 1979.509053] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:30:00 2022] [ 1999.101505] watchdog: BUG: soft lockup - CPU#8 stuck for 1467s! [kworker/8:12:19399] [ 1999.109234] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 1999.109251] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 1999.197038] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 1999.205288] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 1999.214579] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 1999.220746] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 1999.227694] pc : __sbq_wake_up+0x48/0x1f0 [ 1999.231690] lr : sbitmap_queue_clear+0x74/0xc0 [ 1999.236121] sp : ffff80000d79b280 [ 1999.239422] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 1999.246544] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 1999.253667] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 1999.260789] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 1999.267911] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 1999.275033] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 1999.282155] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 1999.289277] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 1999.296399] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 1999.303521] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 1999.310642] Call trace: [ 1999.313075] __sbq_wake_up+0x48/0x1f0 [ 1999.316724] sbitmap_queue_clear+0x74/0xc0 [ 1999.320807] blk_mq_put_tag+0x44/0x70 [ 1999.324457] __blk_mq_free_request+0x88/0xf0 [ 1999.328714] blk_mq_free_request+0xc0/0x160 [ 1999.332884] __blk_mq_end_request+0x78/0x150 [ 1999.337140] scsi_end_request+0x114/0x260 [ 1999.341137] scsi_io_completion+0x64/0x174 [ 1999.345220] scsi_finish_command+0xe0/0x120 [ 1999.349390] scsi_complete+0x8c/0x1c4 [ 1999.353039] blk_complete_reqs+0x5c/0x74 [ 1999.356949] blk_done_softirq+0x38/0x4c [ 1999.360771] __do_softirq+0x168/0x418 [ 1999.364420] __irq_exit_rcu+0x120/0x170 [ 1999.368243] irq_exit_rcu+0x18/0x24 [ 1999.371719] el1_interrupt+0x38/0x70 [ 1999.375281] el1h_64_irq_handler+0x18/0x24 [ 1999.379364] el1h_64_irq+0x68/0x6c [ 1999.382753] _raw_spin_unlock_irqrestore+0x20/0x50 [ 1999.387531] __wake_up_common_lock+0x84/0xac [ 1999.391788] __wake_up+0x20/0x30 [ 1999.395004] __sbq_wake_up+0x13c/0x1f0 [ 1999.398739] sbitmap_queue_clear+0x74/0xc0 [ 1999.402823] blk_mq_put_tag+0x44/0x70 [ 1999.406473] __blk_mq_free_request+0x88/0xf0 [ 1999.410729] blk_mq_free_request+0xc0/0x160 [ 1999.414899] bfq_insert_request.isra.0+0x204/0x270 [ 1999.419677] bfq_insert_requests+0x70/0x90 [ 1999.423760] blk_mq_sched_insert_requests+0xac/0x250 [ 1999.428712] blk_mq_flush_plug_list+0x110/0x2cc [ 1999.433229] __blk_flush_plug+0x100/0x140 [ 1999.437226] blk_finish_plug+0x40/0x25c [ 1999.441049] __iomap_dio_rw+0x1b4/0x500 [ 1999.444872] iomap_dio_rw+0x18/0x50 [ 1999.448347] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 1999.453698] xfs_file_write_iter+0xf4/0x15c [xfs] [ 1999.458440] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 1999.463484] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 1999.468614] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 1999.473744] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 1999.479130] process_one_work+0x1ec/0x470 [ 1999.483126] worker_thread+0x74/0x410 [ 1999.486775] kthread+0xec/0x100 [ 1999.489903] ret_from_fork+0x10/0x20 [ 2002.921489] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 155197 jiffies s: 2093 root: 0x11/. [ 2002.933226] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2002.942435] Task dump for CPU 8: [ 2002.945649] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2002.955550] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2002.961720] Call trace: [ 2002.964153] __switch_to+0xd4/0x100 [ 2002.967629] 0xffff6c0200000080 [ 2002.970758] Task dump for CPU 68: [ 2002.974061] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2002.983962] Call trace: [ 2002.986395] __switch_to+0xd4/0x100 [ 2002.989871] 0x0 [ 2003.211477] watchdog: BUG: soft lockup - CPU#68 stuck for 1471s! [swapper/68:0] [ 2003.218772] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2003.218789] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2003.306575] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2003.314391] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2003.323682] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2003.330629] pc : __sbq_wake_up+0x58/0x1f0 [ 2003.334626] lr : sbitmap_queue_clear+0x74/0xc0 [ 2003.339056] sp : ffff8000088438b0 [ 2003.342357] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2003.349478] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2003.356600] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2003.363722] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2003.370844] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2003.377966] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2003.385088] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2003.392209] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2003.399331] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2003.406453] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2003.413575] Call trace: [ 2003.416008] __sbq_wake_up+0x58/0x1f0 [ 2003.419657] sbitmap_queue_clear+0x74/0xc0 [ 2003.423741] blk_mq_put_tag+0x44/0x70 [ 2003.427390] __blk_mq_free_request+0x88/0xf0 [ 2003.431647] blk_mq_free_request+0xc0/0x160 [ 2003.435817] __blk_mq_end_request+0x78/0x150 [ 2003.440074] scsi_end_request+0x114/0x260 [ 2003.444070] scsi_io_completion+0x64/0x174 [ 2003.448153] scsi_finish_command+0xe0/0x120 [ 2003.452324] scsi_complete+0x8c/0x1c4 [ 2003.455973] blk_complete_reqs+0x5c/0x74 [ 2003.459882] blk_done_softirq+0x38/0x4c [ 2003.463705] __do_softirq+0x168/0x418 [ 2003.467353] __irq_exit_rcu+0x120/0x170 [ 2003.471176] irq_exit_rcu+0x18/0x24 [ 2003.474652] el1_interrupt+0x38/0x70 [ 2003.478214] el1h_64_irq_handler+0x18/0x24 [ 2003.482297] el1h_64_irq+0x68/0x6c [ 2003.485685] cpuidle_enter_state+0xc4/0x4e0 [ 2003.489855] cpuidle_enter+0x40/0x60 [ 2003.493417] cpuidle_idle_call+0x118/0x1b0 [ 2003.497501] do_idle+0xac/0x100 [ 2003.500630] cpu_startup_entry+0x30/0x34 [ 2003.504539] secondary_start_kernel+0xd8/0xf0 [ 2003.508883] __secondary_switched+0xb0/0xb4 [ 2023.101340] watchdog: BUG: soft lockup - CPU#8 stuck for 1490s! [kworker/8:12:19399] [ 2023.109069] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2023.109086] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2023.196873] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2023.205123] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2023.214413] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2023.220581] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2023.227529] pc : __sbq_wake_up+0x20/0x1f0 [ 2023.231525] lr : sbitmap_queue_clear+0x74/0xc0 [ 2023.235956] sp : ffff80000d79b280 [ 2023.239258] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2023.246379] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2023.253502] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2023.260624] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2023.267746] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2023.274869] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2023.281991] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2023.289113] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2023.296235] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2023.303357] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 2023.310479] Call trace: [ 2023.312912] __sbq_wake_up+0x20/0x1f0 [ 2023.316561] sbitmap_queue_clear+0x74/0xc0 [ 2023.320645] blk_mq_put_tag+0x44/0x70 [ 2023.324294] __blk_mq_free_request+0x88/0xf0 [ 2023.328551] blk_mq_free_request+0xc0/0x160 [ 2023.332721] __blk_mq_end_request+0x78/0x150 [ 2023.336978] scsi_end_request+0x114/0x260 [ 2023.340974] scsi_io_completion+0x64/0x174 [ 2023.345057] scsi_finish_command+0xe0/0x120 [ 2023.349228] scsi_complete+0x8c/0x1c4 [ 2023.352877] blk_complete_reqs+0x5c/0x74 [ 2023.356788] blk_done_softirq+0x38/0x4c [ 2023.360610] __do_softirq+0x168/0x418 [ 2023.364259] __irq_exit_rcu+0x120/0x170 [ 2023.368083] irq_exit_rcu+0x18/0x24 [ 2023.371559] el1_interrupt+0x38/0x70 [ 2023.375121] el1h_64_irq_handler+0x18/0x24 [ 2023.379204] el1h_64_irq+0x68/0x6c [ 2023.382593] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2023.387370] __wake_up_common_lock+0x84/0xac [ 2023.391627] __wake_up+0x20/0x30 [ 2023.394843] __sbq_wake_up+0x13c/0x1f0 [ 2023.398579] sbitmap_queue_clear+0x74/0xc0 [ 2023.402662] blk_mq_put_tag+0x44/0x70 [ 2023.406312] __blk_mq_free_request+0x88/0xf0 [ 2023.410569] blk_mq_free_request+0xc0/0x160 [ 2023.414739] bfq_insert_request.isra.0+0x204/0x270 [ 2023.419517] bfq_insert_requests+0x70/0x90 [ 2023.423600] blk_mq_sched_insert_requests+0xac/0x250 [ 2023.428553] blk_mq_flush_plug_list+0x110/0x2cc [ 2023.433070] __blk_flush_plug+0x100/0x140 [ 2023.437067] blk_finish_plug+0x40/0x25c [ 2023.440890] __iomap_dio_rw+0x1b4/0x500 [ 2023.444713] iomap_dio_rw+0x18/0x50 [ 2023.448188] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2023.453539] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2023.458281] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2023.463325] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2023.468455] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2023.473585] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2023.478971] process_one_work+0x1ec/0x470 [ 2023.482967] worker_thread+0x74/0x410 [ 2023.486616] kthread+0xec/0x100 [ 2023.489744] ret_from_fork+0x10/0x20 [ 2027.211311] watchdog: BUG: soft lockup - CPU#68 stuck for 1493s! [swapper/68:0] [ 2027.218606] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2027.218623] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2027.306410] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2027.314225] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2027.323517] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2027.330466] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2027.334549] lr : sbitmap_queue_clear+0x74/0xc0 [ 2027.338979] sp : ffff8000088438b0 [ 2027.342281] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2027.349402] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2027.356524] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2027.363646] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2027.370768] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2027.377890] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2027.385012] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2027.392134] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2027.399256] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2027.406378] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2027.413499] Call trace: [ 2027.415932] __sbq_wake_up+0x1c8/0x1f0 [ 2027.419668] sbitmap_queue_clear+0x74/0xc0 [ 2027.423752] blk_mq_put_tag+0x44/0x70 [ 2027.427401] __blk_mq_free_request+0x88/0xf0 [ 2027.431658] blk_mq_free_request+0xc0/0x160 [ 2027.435828] __blk_mq_end_request+0x78/0x150 [ 2027.440086] scsi_end_request+0x114/0x260 [ 2027.444083] scsi_io_completion+0x64/0x174 [ 2027.448166] scsi_finish_command+0xe0/0x120 [ 2027.452337] scsi_complete+0x8c/0x1c4 [ 2027.455987] blk_complete_reqs+0x5c/0x74 [ 2027.459896] blk_done_softirq+0x38/0x4c [ 2027.463719] __do_softirq+0x168/0x418 [ 2027.467368] __irq_exit_rcu+0x120/0x170 [ 2027.471190] irq_exit_rcu+0x18/0x24 [ 2027.474666] el1_interrupt+0x38/0x70 [ 2027.478228] el1h_64_irq_handler+0x18/0x24 [ 2027.482311] el1h_64_irq+0x68/0x6c [ 2027.485700] cpuidle_enter_state+0xc4/0x4e0 [ 2027.489870] cpuidle_enter+0x40/0x60 [ 2027.493432] cpuidle_idle_call+0x118/0x1b0 [ 2027.497516] do_idle+0xac/0x100 [ 2027.500645] cpu_startup_entry+0x30/0x34 [ 2027.504554] secondary_start_kernel+0xd8/0xf0 [ 2027.508898] __secondary_switched+0xb0/0xb4 [ 2047.101174] watchdog: BUG: soft lockup - CPU#8 stuck for 1512s! [kworker/8:12:19399] [ 2047.108903] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2047.108920] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2047.196708] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2047.204958] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2047.214249] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2047.220416] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2047.227365] pc : __sbq_wake_up+0x58/0x1f0 [ 2047.231362] lr : sbitmap_queue_clear+0x74/0xc0 [ 2047.235793] sp : ffff80000d79b280 [ 2047.239094] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2047.246217] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2047.253339] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2047.260461] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2047.267583] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2047.274705] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2047.281826] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2047.288948] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2047.296071] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2047.303193] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2047.310315] Call trace: [ 2047.312748] __sbq_wake_up+0x58/0x1f0 [ 2047.316398] sbitmap_queue_clear+0x74/0xc0 [ 2047.320481] blk_mq_put_tag+0x44/0x70 [ 2047.324131] __blk_mq_free_request+0x88/0xf0 [ 2047.328388] blk_mq_free_request+0xc0/0x160 [ 2047.332557] __blk_mq_end_request+0x78/0x150 [ 2047.336814] scsi_end_request+0x114/0x260 [ 2047.340810] scsi_io_completion+0x64/0x174 [ 2047.344894] scsi_finish_command+0xe0/0x120 [ 2047.349064] scsi_complete+0x8c/0x1c4 [ 2047.352713] blk_complete_reqs+0x5c/0x74 [ 2047.356623] blk_done_softirq+0x38/0x4c [ 2047.360446] __do_softirq+0x168/0x418 [ 2047.364094] __irq_exit_rcu+0x120/0x170 [ 2047.367917] irq_exit_rcu+0x18/0x24 [ 2047.371393] el1_interrupt+0x38/0x70 [ 2047.374955] el1h_64_irq_handler+0x18/0x24 [ 2047.379038] el1h_64_irq+0x68/0x6c [ 2047.382427] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2047.387205] __wake_up_common_lock+0x84/0xac [ 2047.391462] __wake_up+0x20/0x30 [ 2047.394677] __sbq_wake_up+0x13c/0x1f0 [ 2047.398413] sbitmap_queue_clear+0x74/0xc0 [ 2047.402497] blk_mq_put_tag+0x44/0x70 [ 2047.406146] __blk_mq_free_request+0x88/0xf0 [ 2047.410403] blk_mq_free_request+0xc0/0x160 [ 2047.414573] bfq_insert_request.isra.0+0x204/0x270 [ 2047.419351] bfq_insert_requests+0x70/0x90 [ 2047.423436] blk_mq_sched_insert_requests+0xac/0x250 [ 2047.428388] blk_mq_flush_plug_list+0x110/0x2cc [ 2047.432905] __blk_flush_plug+0x100/0x140 [ 2047.436902] blk_finish_plug+0x40/0x25c [ 2047.440724] __iomap_dio_rw+0x1b4/0x500 [ 2047.444547] iomap_dio_rw+0x18/0x50 [ 2047.448023] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2047.453373] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2047.458115] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2047.463159] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2047.468289] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2047.473420] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2047.478805] process_one_work+0x1ec/0x470 [ 2047.482802] worker_thread+0x74/0x410 [ 2047.486451] kthread+0xec/0x100 [ 2047.489581] ret_from_fork+0x10/0x20 [ 2051.211146] watchdog: BUG: soft lockup - CPU#68 stuck for 1516s! [swapper/68:0] [ 2051.218441] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2051.218457] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2051.306244] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2051.314060] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2051.323351] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2051.330298] pc : __sbq_wake_up+0x84/0x1f0 [ 2051.334295] lr : sbitmap_queue_clear+0x74/0xc0 [ 2051.338725] sp : ffff8000088438b0 [ 2051.342026] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2051.349148] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2051.356270] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2051.363392] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2051.370513] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2051.377635] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2051.384758] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2051.391879] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2051.399001] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2051.406123] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2051.413245] Call trace: [ 2051.415678] __sbq_wake_up+0x84/0x1f0 [ 2051.419327] sbitmap_queue_clear+0x74/0xc0 [ 2051.423410] blk_mq_put_tag+0x44/0x70 [ 2051.427060] __blk_mq_free_request+0x88/0xf0 [ 2051.431316] blk_mq_free_request+0xc0/0x160 [ 2051.435486] __blk_mq_end_request+0x78/0x150 [ 2051.439743] scsi_end_request+0x114/0x260 [ 2051.443739] scsi_io_completion+0x64/0x174 [ 2051.447822] scsi_finish_command+0xe0/0x120 [ 2051.451993] scsi_complete+0x8c/0x1c4 [ 2051.455642] blk_complete_reqs+0x5c/0x74 [ 2051.459552] blk_done_softirq+0x38/0x4c [ 2051.463374] __do_softirq+0x168/0x418 [ 2051.467023] __irq_exit_rcu+0x120/0x170 [ 2051.470846] irq_exit_rcu+0x18/0x24 [ 2051.474321] el1_interrupt+0x38/0x70 [ 2051.477883] el1h_64_irq_handler+0x18/0x24 [ 2051.481966] el1h_64_irq+0x68/0x6c [ 2051.485354] cpuidle_enter_state+0xc4/0x4e0 [ 2051.489524] cpuidle_enter+0x40/0x60 [ 2051.493086] cpuidle_idle_call+0x118/0x1b0 [ 2051.497170] do_idle+0xac/0x100 [ 2051.500299] cpu_startup_entry+0x30/0x34 [ 2051.504208] secondary_start_kernel+0xd8/0xf0 [ 2051.508552] __secondary_switched+0xb0/0xb4 [ 2069.481030] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 161853 jiffies s: 2093 root: 0x11/. [ 2069.492768] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2069.501977] Task dump for CPU 8: [ 2069.505192] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2069.515093] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2069.521263] Call trace: [ 2069.523696] __switch_to+0xd4/0x100 [ 2069.527172] 0xffff6c0200000080 [ 2069.530301] Task dump for CPU 68: [ 2069.533604] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2069.543505] Call trace: [ 2069.545938] __switch_to+0xd4/0x100 [ 2069.549414] 0x0 [ 2071.101009] watchdog: BUG: soft lockup - CPU#8 stuck for 1534s! [kworker/8:12:19399] [ 2071.108738] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2071.108755] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2071.196542] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2071.204792] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2071.214083] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2071.220250] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2071.227198] pc : __sbq_wake_up+0xa4/0x1f0 [ 2071.231194] lr : sbitmap_queue_clear+0x74/0xc0 [ 2071.235625] sp : ffff80000d79b280 [ 2071.238926] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2071.246049] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2071.253172] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2071.260294] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2071.267416] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2071.274538] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2071.281660] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2071.288782] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2071.295903] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2071.303025] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2071.310147] Call trace: [ 2071.312580] __sbq_wake_up+0xa4/0x1f0 [ 2071.316230] sbitmap_queue_clear+0x74/0xc0 [ 2071.320314] blk_mq_put_tag+0x44/0x70 [ 2071.323964] __blk_mq_free_request+0x88/0xf0 [ 2071.328221] blk_mq_free_request+0xc0/0x160 [ 2071.332391] __blk_mq_end_request+0x78/0x150 [ 2071.336648] scsi_end_request+0x114/0x260 [ 2071.340644] scsi_io_completion+0x64/0x174 [ 2071.344727] scsi_finish_command+0xe0/0x120 [ 2071.348897] scsi_complete+0x8c/0x1c4 [ 2071.352547] blk_complete_reqs+0x5c/0x74 [ 2071.356456] blk_done_softirq+0x38/0x4c [ 2071.360279] __do_softirq+0x168/0x418 [ 2071.363928] __irq_exit_rcu+0x120/0x170 [ 2071.367751] irq_exit_rcu+0x18/0x24 [ 2071.371226] el1_interrupt+0x38/0x70 [ 2071.374788] el1h_64_irq_handler+0x18/0x24 [ 2071.378871] el1h_64_irq+0x68/0x6c [ 2071.382260] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2071.387038] __wake_up_common_lock+0x84/0xac [ 2071.391295] __wake_up+0x20/0x30 [ 2071.394511] __sbq_wake_up+0x13c/0x1f0 [ 2071.398247] sbitmap_queue_clear+0x74/0xc0 [ 2071.402331] blk_mq_put_tag+0x44/0x70 [ 2071.405981] __blk_mq_free_request+0x88/0xf0 [ 2071.410237] blk_mq_free_request+0xc0/0x160 [ 2071.414409] bfq_insert_request.isra.0+0x204/0x270 [ 2071.419187] bfq_insert_requests+0x70/0x90 [ 2071.423271] blk_mq_sched_insert_requests+0xac/0x250 [ 2071.428223] blk_mq_flush_plug_list+0x110/0x2cc [ 2071.432740] __blk_flush_plug+0x100/0x140 [ 2071.436737] blk_finish_plug+0x40/0x25c [ 2071.440560] __iomap_dio_rw+0x1b4/0x500 [ 2071.444382] iomap_dio_rw+0x18/0x50 [ 2071.447858] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2071.453208] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2071.457951] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2071.462995] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2071.468125] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2071.473255] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2071.478640] process_one_work+0x1ec/0x470 [ 2071.482637] worker_thread+0x74/0x410 [ 2071.486286] kthread+0xec/0x100 [ 2071.489414] ret_from_fork+0x10/0x20 [ 2075.210980] watchdog: BUG: soft lockup - CPU#68 stuck for 1538s! [swapper/68:0] [ 2075.218275] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2075.218292] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2075.306078] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2075.313894] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2075.323185] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2075.330132] pc : __sbq_wake_up+0x58/0x1f0 [ 2075.334129] lr : sbitmap_queue_clear+0x74/0xc0 [ 2075.338559] sp : ffff8000088438b0 [ 2075.341860] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2075.348982] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2075.356104] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2075.363226] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2075.370347] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2075.377469] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2075.384591] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2075.391713] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2075.398835] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2075.405958] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2075.413080] Call trace: [ 2075.415513] __sbq_wake_up+0x58/0x1f0 [ 2075.419163] sbitmap_queue_clear+0x74/0xc0 [ 2075.423246] blk_mq_put_tag+0x44/0x70 [ 2075.426896] __blk_mq_free_request+0x88/0xf0 [ 2075.431152] blk_mq_free_request+0xc0/0x160 [ 2075.435322] __blk_mq_end_request+0x78/0x150 [ 2075.439579] scsi_end_request+0x114/0x260 [ 2075.443575] scsi_io_completion+0x64/0x174 [ 2075.447658] scsi_finish_command+0xe0/0x120 [ 2075.451829] scsi_complete+0x8c/0x1c4 [ 2075.455478] blk_complete_reqs+0x5c/0x74 [ 2075.459387] blk_done_softirq+0x38/0x4c [ 2075.463210] __do_softirq+0x168/0x418 [ 2075.466859] __irq_exit_rcu+0x120/0x170 [ 2075.470681] irq_exit_rcu+0x18/0x24 [ 2075.474157] el1_interrupt+0x38/0x70 [ 2075.477719] el1h_64_irq_handler+0x18/0x24 [ 2075.481802] el1h_64_irq+0x68/0x6c [ 2075.485191] cpuidle_enter_state+0xc4/0x4e0 [ 2075.489361] cpuidle_enter+0x40/0x60 [ 2075.492923] cpuidle_idle_call+0x118/0x1b0 [ 2075.497007] do_idle+0xac/0x100 [ 2075.500136] cpu_startup_entry+0x30/0x34 [ 2075.504047] secondary_start_kernel+0xd8/0xf0 [ 2075.508391] __secondary_switched+0xb0/0xb4 [ 2095.100843] watchdog: BUG: soft lockup - CPU#8 stuck for 1557s! [kworker/8:12:19399] [ 2095.108573] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2095.108589] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2095.196376] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2095.204626] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2095.213917] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2095.220084] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2095.227032] pc : __sbq_wake_up+0x84/0x1f0 [ 2095.231029] lr : sbitmap_queue_clear+0x74/0xc0 [ 2095.235459] sp : ffff80000d79b280 [ 2095.238761] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2095.245883] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2095.253005] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2095.260127] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2095.267249] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2095.274371] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2095.281493] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2095.288615] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2095.295736] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2095.302858] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2095.309980] Call trace: [ 2095.312413] __sbq_wake_up+0x84/0x1f0 [ 2095.316062] sbitmap_queue_clear+0x74/0xc0 [ 2095.320146] blk_mq_put_tag+0x44/0x70 [ 2095.323795] __blk_mq_free_request+0x88/0xf0 [ 2095.328052] blk_mq_free_request+0xc0/0x160 [ 2095.332222] __blk_mq_end_request+0x78/0x150 [ 2095.336478] scsi_end_request+0x114/0x260 [ 2095.340475] scsi_io_completion+0x64/0x174 [ 2095.344558] scsi_finish_command+0xe0/0x120 [ 2095.348728] scsi_complete+0x8c/0x1c4 [ 2095.352377] blk_complete_reqs+0x5c/0x74 [ 2095.356287] blk_done_softirq+0x38/0x4c [ 2095.360109] __do_softirq+0x168/0x418 [ 2095.363759] __irq_exit_rcu+0x120/0x170 [ 2095.367581] irq_exit_rcu+0x18/0x24 [ 2095.371057] el1_interrupt+0x38/0x70 [ 2095.374619] el1h_64_irq_handler+0x18/0x24 [ 2095.378702] el1h_64_irq+0x68/0x6c [ 2095.382091] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2095.386869] __wake_up_common_lock+0x84/0xac [ 2095.391126] __wake_up+0x20/0x30 [ 2095.394342] __sbq_wake_up+0x13c/0x1f0 [ 2095.398078] sbitmap_queue_clear+0x74/0xc0 [ 2095.402161] blk_mq_put_tag+0x44/0x70 [ 2095.405810] __blk_mq_free_request+0x88/0xf0 [ 2095.410067] blk_mq_free_request+0xc0/0x160 [ 2095.414237] bfq_insert_request.isra.0+0x204/0x270 [ 2095.419015] bfq_insert_requests+0x70/0x90 [ 2095.423098] blk_mq_sched_insert_requests+0xac/0x250 [ 2095.428050] blk_mq_flush_plug_list+0x110/0x2cc [ 2095.432568] __blk_flush_plug+0x100/0x140 [ 2095.436565] blk_finish_plug+0x40/0x25c [ 2095.440387] __iomap_dio_rw+0x1b4/0x500 [ 2095.444210] iomap_dio_rw+0x18/0x50 [ 2095.447686] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2095.453037] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2095.457779] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2095.462823] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2095.467953] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2095.473083] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2095.478469] process_one_work+0x1ec/0x470 [ 2095.482465] worker_thread+0x74/0x410 [ 2095.486114] kthread+0xec/0x100 [ 2095.489243] ret_from_fork+0x10/0x20 [ 2099.210815] watchdog: BUG: soft lockup - CPU#68 stuck for 1561s! [swapper/68:0] [ 2099.218110] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2099.218126] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2099.305913] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2099.313728] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2099.323020] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2099.329969] pc : __sbq_wake_up+0x24/0x1f0 [ 2099.333965] lr : sbitmap_queue_clear+0x74/0xc0 [ 2099.338396] sp : ffff8000088438b0 [ 2099.341696] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2099.348818] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2099.355941] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2099.363062] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2099.370184] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2099.377306] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2099.384428] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2099.391550] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2099.398672] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2099.405793] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2099.412915] Call trace: [ 2099.415348] __sbq_wake_up+0x24/0x1f0 [ 2099.418997] sbitmap_queue_clear+0x74/0xc0 [ 2099.423080] blk_mq_put_tag+0x44/0x70 [ 2099.426730] __blk_mq_free_request+0x88/0xf0 [ 2099.430986] blk_mq_free_request+0xc0/0x160 [ 2099.435156] __blk_mq_end_request+0x78/0x150 [ 2099.439413] scsi_end_request+0x114/0x260 [ 2099.443409] scsi_io_completion+0x64/0x174 [ 2099.447492] scsi_finish_command+0xe0/0x120 [ 2099.451663] scsi_complete+0x8c/0x1c4 [ 2099.455312] blk_complete_reqs+0x5c/0x74 [ 2099.459221] blk_done_softirq+0x38/0x4c [ 2099.463044] __do_softirq+0x168/0x418 [ 2099.466693] __irq_exit_rcu+0x120/0x170 [ 2099.470516] irq_exit_rcu+0x18/0x24 [ 2099.473991] el1_interrupt+0x38/0x70 [ 2099.477554] el1h_64_irq_handler+0x18/0x24 [ 2099.481637] el1h_64_irq+0x68/0x6c [ 2099.485025] cpuidle_enter_state+0xc4/0x4e0 [ 2099.489195] cpuidle_enter+0x40/0x60 [ 2099.492758] cpuidle_idle_call+0x118/0x1b0 [ 2099.496841] do_idle+0xac/0x100 [ 2099.499970] cpu_startup_entry+0x30/0x34 [ 2099.503879] secondary_start_kernel+0xd8/0xf0 [ 2099.508223] __secondary_switched+0xb0/0xb4 [ 2107.750757] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2107.756489] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=83463 [ 2107.765609] (t=168286 jiffies g=36445 q=49285 ncpus=128) [ 2107.770994] Task dump for CPU 8: [ 2107.774209] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2107.784111] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2107.790278] Call trace: [ 2107.792711] __switch_to+0xd4/0x100 [ 2107.796187] 0xffff6c0200000080 [ 2107.799316] Task dump for CPU 68: [ 2107.802617] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2107.812517] Call trace: [ 2107.814952] dump_backtrace+0xe8/0x130 [ 2107.818688] show_stack+0x20/0x60 [ 2107.821989] sched_show_task+0x160/0x190 [ 2107.825899] dump_cpu_task+0x4c/0x78 [ 2107.829463] rcu_dump_cpu_stacks+0xc4/0x150 [ 2107.833634] print_cpu_stall+0x180/0x210 [ 2107.837543] check_cpu_stall+0x20c/0x2b4 [ 2107.841453] rcu_pending+0x4c/0x190 [ 2107.844928] rcu_sched_clock_irq+0x80/0x204 [ 2107.849098] update_process_times+0x7c/0xc0 [ 2107.853268] tick_sched_handle+0x38/0x74 [ 2107.857178] tick_sched_timer+0x60/0xc0 [ 2107.861001] __hrtimer_run_queues+0x170/0x300 [ 2107.865345] hrtimer_interrupt+0xfc/0x260 [ 2107.869341] arch_timer_handler_phys+0x34/0x50 [ 2107.873771] handle_percpu_devid_irq+0xac/0x240 [ 2107.878289] generic_handle_domain_irq+0x34/0x50 [ 2107.882894] gic_handle_irq+0x5c/0xe0 [ 2107.886543] call_on_irq_stack+0x2c/0x38 [ 2107.890452] do_interrupt_handler+0xdc/0xe0 [ 2107.894623] el1_interrupt+0x34/0x70 [ 2107.898185] el1h_64_irq_handler+0x18/0x24 [ 2107.902268] el1h_64_irq+0x68/0x6c [ 2107.905657] __sbq_wake_up+0x1c8/0x1f0 [ 2107.909393] sbitmap_queue_clear+0x74/0xc0 [ 2107.913476] blk_mq_put_tag+0x44/0x70 [ 2107.917125] __blk_mq_free_request+0x88/0xf0 [ 2107.921382] blk_mq_free_request+0xc0/0x160 [ 2107.925552] __blk_mq_end_request+0x78/0x150 [ 2107.929809] scsi_end_request+0x114/0x260 [ 2107.933805] scsi_io_completion+0x64/0x174 [ 2107.937888] scsi_finish_command+0xe0/0x120 [ 2107.942058] scsi_complete+0x8c/0x1c4 [ 2107.945708] blk_complete_reqs+0x5c/0x74 [ 2107.949617] blk_done_softirq+0x38/0x4c [ 2107.953440] __do_softirq+0x168/0x418 [ 2107.957088] __irq_exit_rcu+0x120/0x170 [ 2107.960911] irq_exit_rcu+0x18/0x24 [ 2107.964386] el1_interrupt+0x38/0x70 [ 2107.967949] el1h_64_irq_handler+0x18/0x24 [ 2107.972032] el1h_64_irq+0x68/0x6c [ 2107.975420] cpuidle_enter_state+0xc4/0x4e0 [ 2107.979590] cpuidle_enter+0x40/0x60 [ 2107.983153] cpuidle_idle_call+0x118/0x1b0 [ 2107.987236] do_idle+0xac/0x100 [ 2107.990365] cpu_startup_entry+0x30/0x34 [ 2107.994274] secondary_start_kernel+0xd8/0xf0 [ 2107.998618] __secondary_switched+0xb0/0xb4 [ 2119.100678] watchdog: BUG: soft lockup - CPU#8 stuck for 1579s! [kworker/8:12:19399] [ 2119.108407] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2119.108423] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2119.196211] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2119.204460] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2119.213751] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2119.219919] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2119.226866] pc : __sbq_wake_up+0x58/0x1f0 [ 2119.230863] lr : sbitmap_queue_clear+0x74/0xc0 [ 2119.235293] sp : ffff80000d79b280 [ 2119.238595] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2119.245717] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2119.252839] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2119.259961] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2119.267083] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2119.274205] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2119.281327] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2119.288449] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2119.295570] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2119.302692] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2119.309814] Call trace: [ 2119.312247] __sbq_wake_up+0x58/0x1f0 [ 2119.315897] sbitmap_queue_clear+0x74/0xc0 [ 2119.319981] blk_mq_put_tag+0x44/0x70 [ 2119.323630] __blk_mq_free_request+0x88/0xf0 [ 2119.327887] blk_mq_free_request+0xc0/0x160 [ 2119.332057] __blk_mq_end_request+0x78/0x150 [ 2119.336314] scsi_end_request+0x114/0x260 [ 2119.340310] scsi_io_completion+0x64/0x174 [ 2119.344394] scsi_finish_command+0xe0/0x120 [ 2119.348564] scsi_complete+0x8c/0x1c4 [ 2119.352213] blk_complete_reqs+0x5c/0x74 [ 2119.356122] blk_done_softirq+0x38/0x4c [ 2119.359945] __do_softirq+0x168/0x418 [ 2119.363594] __irq_exit_rcu+0x120/0x170 [ 2119.367417] irq_exit_rcu+0x18/0x24 [ 2119.370892] el1_interrupt+0x38/0x70 [ 2119.374455] el1h_64_irq_handler+0x18/0x24 [ 2119.378539] el1h_64_irq+0x68/0x6c [ 2119.381928] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2119.386706] __wake_up_common_lock+0x84/0xac [ 2119.390963] __wake_up+0x20/0x30 [ 2119.394178] __sbq_wake_up+0x13c/0x1f0 [ 2119.397914] sbitmap_queue_clear+0x74/0xc0 [ 2119.401997] blk_mq_put_tag+0x44/0x70 [ 2119.405647] __blk_mq_free_request+0x88/0xf0 [ 2119.409904] blk_mq_free_request+0xc0/0x160 [ 2119.414074] bfq_insert_request.isra.0+0x204/0x270 [ 2119.418852] bfq_insert_requests+0x70/0x90 [ 2119.422937] blk_mq_sched_insert_requests+0xac/0x250 [ 2119.427889] blk_mq_flush_plug_list+0x110/0x2cc [ 2119.432406] __blk_flush_plug+0x100/0x140 [ 2119.436403] blk_finish_plug+0x40/0x25c [ 2119.440225] __iomap_dio_rw+0x1b4/0x500 [ 2119.444049] iomap_dio_rw+0x18/0x50 [ 2119.447524] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2119.452875] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2119.457617] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2119.462661] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2119.467790] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2119.472920] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2119.478306] process_one_work+0x1ec/0x470 [ 2119.482302] worker_thread+0x74/0x410 [ 2119.485951] kthread+0xec/0x100 [ 2119.489079] ret_from_fork+0x10/0x20 [ 2135.210567] watchdog: BUG: soft lockup - CPU#68 stuck for 1594s! [swapper/68:0] [ 2135.217862] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2135.217878] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2135.305665] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2135.313480] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2135.322771] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2135.329720] pc : __sbq_wake_up+0x58/0x1f0 [ 2135.333716] lr : sbitmap_queue_clear+0x74/0xc0 [ 2135.338146] sp : ffff8000088438b0 [ 2135.341448] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2135.348570] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2135.355692] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2135.362814] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2135.369936] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2135.377058] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2135.384180] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2135.391302] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2135.398423] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2135.405545] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2135.412667] Call trace: [ 2135.415100] __sbq_wake_up+0x58/0x1f0 [ 2135.418749] sbitmap_queue_clear+0x74/0xc0 [ 2135.422832] blk_mq_put_tag+0x44/0x70 [ 2135.426482] __blk_mq_free_request+0x88/0xf0 [ 2135.430739] blk_mq_free_request+0xc0/0x160 [ 2135.434909] __blk_mq_end_request+0x78/0x150 [ 2135.439165] scsi_end_request+0x114/0x260 [ 2135.443162] scsi_io_completion+0x64/0x174 [ 2135.447245] scsi_finish_command+0xe0/0x120 [ 2135.451415] scsi_complete+0x8c/0x1c4 [ 2135.455064] blk_complete_reqs+0x5c/0x74 [ 2135.458974] blk_done_softirq+0x38/0x4c [ 2135.462797] __do_softirq+0x168/0x418 [ 2135.466446] __irq_exit_rcu+0x120/0x170 [ 2135.470268] irq_exit_rcu+0x18/0x24 [ 2135.473744] el1_interrupt+0x38/0x70 [ 2135.477306] el1h_64_irq_handler+0x18/0x24 [ 2135.481389] el1h_64_irq+0x68/0x6c [ 2135.484777] cpuidle_enter_state+0xc4/0x4e0 [ 2135.488947] cpuidle_enter+0x40/0x60 [ 2135.492509] cpuidle_idle_call+0x118/0x1b0 [ 2135.496593] do_idle+0xac/0x100 [ 2135.499722] cpu_startup_entry+0x30/0x34 [ 2135.503631] secondary_start_kernel+0xd8/0xf0 [ 2135.507975] __secondary_switched+0xb0/0xb4 [ 2136.040585] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 168509 jiffies s: 2093 root: 0x11/. [ 2136.052323] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2136.061532] Task dump for CPU 8: [ 2136.064746] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2136.074648] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2136.080817] Call trace: [ 2136.083250] __switch_to+0xd4/0x100 [ 2136.086727] 0xffff6c0200000080 [ 2136.089856] Task dump for CPU 68: [ 2136.093158] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2136.103059] Call trace: [ 2136.105493] __switch_to+0xd4/0x100 [ 2136.108969] 0x0 [ 2143.100512] watchdog: BUG: soft lockup - CPU#8 stuck for 1601s! [kworker/8:12:19399] [ 2143.108241] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2143.108258] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2143.196045] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2143.204294] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2143.213585] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2143.219753] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2143.226700] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2143.230784] lr : sbitmap_queue_clear+0x74/0xc0 [ 2143.235215] sp : ffff80000d79b280 [ 2143.238516] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2143.245638] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2143.252760] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2143.259882] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2143.267004] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2143.274126] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2143.281248] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2143.288369] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2143.295491] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2143.302613] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2143.309735] Call trace: [ 2143.312168] __sbq_wake_up+0x1c8/0x1f0 [ 2143.315904] sbitmap_queue_clear+0x74/0xc0 [ 2143.319987] blk_mq_put_tag+0x44/0x70 [ 2143.323637] __blk_mq_free_request+0x88/0xf0 [ 2143.327893] blk_mq_free_request+0xc0/0x160 [ 2143.332063] __blk_mq_end_request+0x78/0x150 [ 2143.336321] scsi_end_request+0x114/0x260 [ 2143.340317] scsi_io_completion+0x64/0x174 [ 2143.344400] scsi_finish_command+0xe0/0x120 [ 2143.348571] scsi_complete+0x8c/0x1c4 [ 2143.352220] blk_complete_reqs+0x5c/0x74 [ 2143.356129] blk_done_softirq+0x38/0x4c [ 2143.359952] __do_softirq+0x168/0x418 [ 2143.363601] __irq_exit_rcu+0x120/0x170 [ 2143.367423] irq_exit_rcu+0x18/0x24 [ 2143.370899] el1_interrupt+0x38/0x70 [ 2143.374462] el1h_64_irq_handler+0x18/0x24 [ 2143.378545] el1h_64_irq+0x68/0x6c [ 2143.381934] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2143.386712] __wake_up_common_lock+0x84/0xac [ 2143.390969] __wake_up+0x20/0x30 [ 2143.394185] __sbq_wake_up+0x13c/0x1f0 [ 2143.397921] sbitmap_queue_clear+0x74/0xc0 [ 2143.402005] blk_mq_put_tag+0x44/0x70 [ 2143.405654] __blk_mq_free_request+0x88/0xf0 [ 2143.409911] blk_mq_free_request+0xc0/0x160 [ 2143.414081] bfq_insert_request.isra.0+0x204/0x270 [ 2143.418859] bfq_insert_requests+0x70/0x90 [ 2143.422943] blk_mq_sched_insert_requests+0xac/0x250 [ 2143.427895] blk_mq_flush_plug_list+0x110/0x2cc [ 2143.432413] __blk_flush_plug+0x100/0x140 [ 2143.436409] blk_finish_plug+0x40/0x25c [ 2143.440232] __iomap_dio_rw+0x1b4/0x500 [ 2143.444055] iomap_dio_rw+0x18/0x50 [ 2143.447530] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2143.452881] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2143.457623] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2143.462666] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2143.467796] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2143.472926] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2143.478312] process_one_work+0x1ec/0x470 [ 2143.482309] worker_thread+0x74/0x410 [ 2143.485957] kthread+0xec/0x100 [ 2143.489087] ret_from_fork+0x10/0x20 [ 2159.210401] watchdog: BUG: soft lockup - CPU#68 stuck for 1616s! [swapper/68:0] [ 2159.217696] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2159.217712] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2159.305499] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2159.313314] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2159.322605] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2159.329553] pc : __sbq_wake_up+0x58/0x1f0 [ 2159.333549] lr : sbitmap_queue_clear+0x74/0xc0 [ 2159.337979] sp : ffff8000088438b0 [ 2159.341281] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2159.348403] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2159.355524] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2159.362646] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2159.369768] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2159.376891] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2159.384014] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2159.391136] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2159.398258] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2159.405380] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2159.412502] Call trace: [ 2159.414935] __sbq_wake_up+0x58/0x1f0 [ 2159.418584] sbitmap_queue_clear+0x74/0xc0 [ 2159.422667] blk_mq_put_tag+0x44/0x70 [ 2159.426317] __blk_mq_free_request+0x88/0xf0 [ 2159.430573] blk_mq_free_request+0xc0/0x160 [ 2159.434743] __blk_mq_end_request+0x78/0x150 [ 2159.439000] scsi_end_request+0x114/0x260 [ 2159.442998] scsi_io_completion+0x64/0x174 [ 2159.447080] scsi_finish_command+0xe0/0x120 [ 2159.451251] scsi_complete+0x8c/0x1c4 [ 2159.454900] blk_complete_reqs+0x5c/0x74 [ 2159.458809] blk_done_softirq+0x38/0x4c [ 2159.462632] __do_softirq+0x168/0x418 [ 2159.466280] __irq_exit_rcu+0x120/0x170 [ 2159.470103] irq_exit_rcu+0x18/0x24 [ 2159.473578] el1_interrupt+0x38/0x70 [ 2159.477141] el1h_64_irq_handler+0x18/0x24 [ 2159.481224] el1h_64_irq+0x68/0x6c [ 2159.484612] cpuidle_enter_state+0xc4/0x4e0 [ 2159.488782] cpuidle_enter+0x40/0x60 [ 2159.492344] cpuidle_idle_call+0x118/0x1b0 [ 2159.496428] do_idle+0xac/0x100 [ 2159.499556] cpu_startup_entry+0x30/0x34 [ 2159.503465] secondary_start_kernel+0xd8/0xf0 [ 2159.507810] __secondary_switched+0xb0/0xb4 [ 2167.100346] watchdog: BUG: soft lockup - CPU#8 stuck for 1624s! [kworker/8:12:19399] [ 2167.108076] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2167.108092] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2167.195880] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2167.204130] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2167.213420] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2167.219588] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2167.226535] pc : __sbq_wake_up+0x7c/0x1f0 [ 2167.230532] lr : sbitmap_queue_clear+0x74/0xc0 [ 2167.234963] sp : ffff80000d79b2d0 [ 2167.238264] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2167.245386] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2167.252508] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2167.259629] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2167.266751] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2167.273873] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2167.280995] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2167.288117] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2167.295239] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2167.302361] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 2167.309483] Call trace: [ 2167.311916] __sbq_wake_up+0x7c/0x1f0 [ 2167.315565] blk_mq_put_tag+0x44/0x70 [ 2167.319215] __blk_mq_free_request+0x88/0xf0 [ 2167.323471] blk_mq_free_request+0xc0/0x160 [ 2167.327641] __blk_mq_end_request+0x78/0x150 [ 2167.331898] scsi_end_request+0x114/0x260 [ 2167.335894] scsi_io_completion+0x64/0x174 [ 2167.339978] scsi_finish_command+0xe0/0x120 [ 2167.344148] scsi_complete+0x8c/0x1c4 [ 2167.347797] blk_complete_reqs+0x5c/0x74 [ 2167.351706] blk_done_softirq+0x38/0x4c [ 2167.355529] __do_softirq+0x168/0x418 [ 2167.359178] __irq_exit_rcu+0x120/0x170 [ 2167.363000] irq_exit_rcu+0x18/0x24 [ 2167.366476] el1_interrupt+0x38/0x70 [ 2167.370038] el1h_64_irq_handler+0x18/0x24 [ 2167.374121] el1h_64_irq+0x68/0x6c [ 2167.377510] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2167.382288] __wake_up_common_lock+0x84/0xac [ 2167.386545] __wake_up+0x20/0x30 [ 2167.389761] __sbq_wake_up+0x13c/0x1f0 [ 2167.393498] sbitmap_queue_clear+0x74/0xc0 [ 2167.397581] blk_mq_put_tag+0x44/0x70 [ 2167.401231] __blk_mq_free_request+0x88/0xf0 [ 2167.405488] blk_mq_free_request+0xc0/0x160 [ 2167.409658] bfq_insert_request.isra.0+0x204/0x270 [ 2167.414436] bfq_insert_requests+0x70/0x90 [ 2167.418519] blk_mq_sched_insert_requests+0xac/0x250 [ 2167.423472] blk_mq_flush_plug_list+0x110/0x2cc [ 2167.427989] __blk_flush_plug+0x100/0x140 [ 2167.431986] blk_finish_plug+0x40/0x25c [ 2167.435808] __iomap_dio_rw+0x1b4/0x500 [ 2167.439631] iomap_dio_rw+0x18/0x50 [ 2167.443106] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2167.448457] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2167.453200] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2167.458243] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2167.463373] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2167.468503] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2167.473889] process_one_work+0x1ec/0x470 [ 2167.477885] worker_thread+0x74/0x410 [ 2167.481533] kthread+0xec/0x100 [ 2167.484663] ret_from_fork+0x10/0x20 [ 2183.210236] watchdog: BUG: soft lockup - CPU#68 stuck for 1639s! [swapper/68:0] [ 2183.217531] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2183.217547] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2183.305334] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2183.313150] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2183.322440] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2183.329388] pc : __sbq_wake_up+0x58/0x1f0 [ 2183.333384] lr : sbitmap_queue_clear+0x74/0xc0 [ 2183.337814] sp : ffff8000088438b0 [ 2183.341115] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2183.348237] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2183.355359] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2183.362481] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2183.369603] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2183.376724] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2183.383847] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2183.390969] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2183.398091] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2183.405213] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2183.412335] Call trace: [ 2183.414767] __sbq_wake_up+0x58/0x1f0 [ 2183.418417] sbitmap_queue_clear+0x74/0xc0 [ 2183.422500] blk_mq_put_tag+0x44/0x70 [ 2183.426150] __blk_mq_free_request+0x88/0xf0 [ 2183.430406] blk_mq_free_request+0xc0/0x160 [ 2183.434576] __blk_mq_end_request+0x78/0x150 [ 2183.438833] scsi_end_request+0x114/0x260 [ 2183.442829] scsi_io_completion+0x64/0x174 [ 2183.446913] scsi_finish_command+0xe0/0x120 [ 2183.451083] scsi_complete+0x8c/0x1c4 [ 2183.454732] blk_complete_reqs+0x5c/0x74 [ 2183.458641] blk_done_softirq+0x38/0x4c [ 2183.462464] __do_softirq+0x168/0x418 [ 2183.466113] __irq_exit_rcu+0x120/0x170 [ 2183.469935] irq_exit_rcu+0x18/0x24 [ 2183.473411] el1_interrupt+0x38/0x70 [ 2183.476973] el1h_64_irq_handler+0x18/0x24 [ 2183.481056] el1h_64_irq+0x68/0x6c [ 2183.484444] cpuidle_enter_state+0xc4/0x4e0 [ 2183.488615] cpuidle_enter+0x40/0x60 [ 2183.492177] cpuidle_idle_call+0x118/0x1b0 [ 2183.496261] do_idle+0xac/0x100 [ 2183.499389] cpu_startup_entry+0x30/0x34 [ 2183.503298] secondary_start_kernel+0xd8/0xf0 [ 2183.507643] __secondary_switched+0xb0/0xb4 [ 2191.100182] watchdog: BUG: soft lockup - CPU#8 stuck for 1646s! [kworker/8:12:19399] [ 2191.107911] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2191.107928] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2191.195715] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2191.203965] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2191.213255] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2191.219423] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2191.226371] pc : __sbq_wake_up+0x4/0x1f0 [ 2191.230281] lr : sbitmap_queue_clear+0x74/0xc0 [ 2191.234711] sp : ffff80000d79b2d0 [ 2191.238013] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2191.245135] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2191.252258] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2191.259379] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2191.266501] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2191.273623] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2191.280745] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2191.287867] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2191.294989] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2191.302110] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 2191.309232] Call trace: [ 2191.311665] __sbq_wake_up+0x4/0x1f0 [ 2191.315228] blk_mq_put_tag+0x44/0x70 [ 2191.318877] __blk_mq_free_request+0x88/0xf0 [ 2191.323134] blk_mq_free_request+0xc0/0x160 [ 2191.327304] __blk_mq_end_request+0x78/0x150 [ 2191.331561] scsi_end_request+0x114/0x260 [ 2191.335557] scsi_io_completion+0x64/0x174 [ 2191.339642] scsi_finish_command+0xe0/0x120 [ 2191.343812] scsi_complete+0x8c/0x1c4 [ 2191.347461] blk_complete_reqs+0x5c/0x74 [ 2191.351371] blk_done_softirq+0x38/0x4c [ 2191.355193] __do_softirq+0x168/0x418 [ 2191.358842] __irq_exit_rcu+0x120/0x170 [ 2191.362665] irq_exit_rcu+0x18/0x24 [ 2191.366141] el1_interrupt+0x38/0x70 [ 2191.369703] el1h_64_irq_handler+0x18/0x24 [ 2191.373786] el1h_64_irq+0x68/0x6c [ 2191.377175] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2191.381952] __wake_up_common_lock+0x84/0xac [ 2191.386209] __wake_up+0x20/0x30 [ 2191.389425] __sbq_wake_up+0x13c/0x1f0 [ 2191.393161] sbitmap_queue_clear+0x74/0xc0 [ 2191.397246] blk_mq_put_tag+0x44/0x70 [ 2191.400895] __blk_mq_free_request+0x88/0xf0 [ 2191.405152] blk_mq_free_request+0xc0/0x160 [ 2191.409322] bfq_insert_request.isra.0+0x204/0x270 [ 2191.414100] bfq_insert_requests+0x70/0x90 [ 2191.418184] blk_mq_sched_insert_requests+0xac/0x250 [ 2191.423136] blk_mq_flush_plug_list+0x110/0x2cc [ 2191.427653] __blk_flush_plug+0x100/0x140 [ 2191.431650] blk_finish_plug+0x40/0x25c [ 2191.435472] __iomap_dio_rw+0x1b4/0x500 [ 2191.439295] iomap_dio_rw+0x18/0x50 [ 2191.442771] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2191.448121] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2191.452863] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2191.457907] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2191.463037] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2191.468167] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2191.473553] process_one_work+0x1ec/0x470 [ 2191.477549] worker_thread+0x74/0x410 [ 2191.481198] kthread+0xec/0x100 [ 2191.484326] ret_from_fork+0x10/0x20 [ 2202.600114] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 175165 jiffies s: 2093 root: 0x11/. [ 2202.611851] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2202.621060] Task dump for CPU 8: [ 2202.624274] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2202.634176] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2202.640345] Call trace: [ 2202.642778] __switch_to+0xd4/0x100 [ 2202.646255] 0xffff6c0200000080 [ 2202.649384] Task dump for CPU 68: [ 2202.652686] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2202.662587] Call trace: [ 2202.665020] __switch_to+0xd4/0x100 [ 2202.668496] 0x0 [ 2207.210072] watchdog: BUG: soft lockup - CPU#68 stuck for 1661s! [swapper/68:0] [ 2207.217367] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2207.217383] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2207.305170] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2207.312986] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2207.322277] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2207.329225] pc : __sbq_wake_up+0x3c/0x1f0 [ 2207.333222] lr : sbitmap_queue_clear+0x74/0xc0 [ 2207.337652] sp : ffff8000088438b0 [ 2207.340953] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2207.348075] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2207.355196] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2207.362318] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2207.369440] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2207.376562] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2207.383684] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2207.390805] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2207.397927] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2207.405049] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2207.412171] Call trace: [ 2207.414604] __sbq_wake_up+0x3c/0x1f0 [ 2207.418253] sbitmap_queue_clear+0x74/0xc0 [ 2207.422337] blk_mq_put_tag+0x44/0x70 [ 2207.425987] __blk_mq_free_request+0x88/0xf0 [ 2207.430243] blk_mq_free_request+0xc0/0x160 [ 2207.434413] __blk_mq_end_request+0x78/0x150 [ 2207.438670] scsi_end_request+0x114/0x260 [ 2207.442666] scsi_io_completion+0x64/0x174 [ 2207.446749] scsi_finish_command+0xe0/0x120 [ 2207.450920] scsi_complete+0x8c/0x1c4 [ 2207.454569] blk_complete_reqs+0x5c/0x74 [ 2207.458478] blk_done_softirq+0x38/0x4c [ 2207.462301] __do_softirq+0x168/0x418 [ 2207.465950] __irq_exit_rcu+0x120/0x170 [ 2207.469773] irq_exit_rcu+0x18/0x24 [ 2207.473248] el1_interrupt+0x38/0x70 [ 2207.476810] el1h_64_irq_handler+0x18/0x24 [ 2207.480894] el1h_64_irq+0x68/0x6c [ 2207.484282] cpuidle_enter_state+0xc4/0x4e0 [ 2207.488452] cpuidle_enter+0x40/0x60 [ 2207.492014] cpuidle_idle_call+0x118/0x1b0 [ 2207.496099] do_idle+0xac/0x100 [ 2207.499227] cpu_startup_entry+0x30/0x34 [ 2207.503136] secondary_start_kernel+0xd8/0xf0 [ 2207.507480] __secondary_switched+0xb0/0xb4 [ 2215.100018] watchdog: BUG: soft lockup - CPU#8 stuck for 1668s! [kworker/8:12:19399] [ 2215.107748] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2215.107764] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2215.195552] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2215.203801] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2215.213092] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2215.219260] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2215.226208] pc : __sbq_wake_up+0x58/0x1f0 [ 2215.230205] lr : sbitmap_queue_clear+0x74/0xc0 [ 2215.234636] sp : ffff80000d79b280 [ 2215.237938] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2215.245060] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2215.252183] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2215.259305] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2215.266428] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2215.273549] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2215.280671] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2215.287793] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2215.294915] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2215.302037] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2215.309159] Call trace: [ 2215.311592] __sbq_wake_up+0x58/0x1f0 [ 2215.315241] sbitmap_queue_clear+0x74/0xc0 [ 2215.319325] blk_mq_put_tag+0x44/0x70 [ 2215.322975] __blk_mq_free_request+0x88/0xf0 [ 2215.327231] blk_mq_free_request+0xc0/0x160 [ 2215.331401] __blk_mq_end_request+0x78/0x150 [ 2215.335658] scsi_end_request+0x114/0x260 [ 2215.339654] scsi_io_completion+0x64/0x174 [ 2215.343737] scsi_finish_command+0xe0/0x120 [ 2215.347908] scsi_complete+0x8c/0x1c4 [ 2215.351557] blk_complete_reqs+0x5c/0x74 [ 2215.355466] blk_done_softirq+0x38/0x4c [ 2215.359289] __do_softirq+0x168/0x418 [ 2215.362938] __irq_exit_rcu+0x120/0x170 [ 2215.366761] irq_exit_rcu+0x18/0x24 [ 2215.370236] el1_interrupt+0x38/0x70 [ 2215.373799] el1h_64_irq_handler+0x18/0x24 [ 2215.377882] el1h_64_irq+0x68/0x6c [ 2215.381270] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2215.386048] __wake_up_common_lock+0x84/0xac [ 2215.390305] __wake_up+0x20/0x30 [ 2215.393521] __sbq_wake_up+0x13c/0x1f0 [ 2215.397257] sbitmap_queue_clear+0x74/0xc0 [ 2215.401340] blk_mq_put_tag+0x44/0x70 [ 2215.404990] __blk_mq_free_request+0x88/0xf0 [ 2215.409247] blk_mq_free_request+0xc0/0x160 [ 2215.413416] bfq_insert_request.isra.0+0x204/0x270 [ 2215.418194] bfq_insert_requests+0x70/0x90 [ 2215.422278] blk_mq_sched_insert_requests+0xac/0x250 [ 2215.427229] blk_mq_flush_plug_list+0x110/0x2cc [ 2215.431747] __blk_flush_plug+0x100/0x140 [ 2215.435743] blk_finish_plug+0x40/0x25c [ 2215.439566] __iomap_dio_rw+0x1b4/0x500 [ 2215.443389] iomap_dio_rw+0x18/0x50 [ 2215.446864] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2215.452215] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2215.456957] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2215.462002] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2215.467132] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2215.472262] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2215.477648] process_one_work+0x1ec/0x470 [ 2215.481644] worker_thread+0x74/0x410 [ 2215.485293] kthread+0xec/0x100 [ 2215.488422] ret_from_fork+0x10/0x20 [ 2231.209909] watchdog: BUG: soft lockup - CPU#68 stuck for 1683s! [swapper/68:0] [ 2231.217204] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2231.217220] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2231.305007] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2231.312822] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2231.322113] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2231.329061] pc : __sbq_wake_up+0x58/0x1f0 [ 2231.333058] lr : sbitmap_queue_clear+0x74/0xc0 [ 2231.337488] sp : ffff8000088438b0 [ 2231.340789] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2231.347911] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2231.355033] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2231.362155] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2231.369277] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2231.376399] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2231.383520] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2231.390642] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2231.397764] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2231.404886] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2231.412008] Call trace: [ 2231.414441] __sbq_wake_up+0x58/0x1f0 [ 2231.418091] sbitmap_queue_clear+0x74/0xc0 [ 2231.422174] blk_mq_put_tag+0x44/0x70 [ 2231.425823] __blk_mq_free_request+0x88/0xf0 [ 2231.430080] blk_mq_free_request+0xc0/0x160 [ 2231.434250] __blk_mq_end_request+0x78/0x150 [ 2231.438507] scsi_end_request+0x114/0x260 [ 2231.442503] scsi_io_completion+0x64/0x174 [ 2231.446586] scsi_finish_command+0xe0/0x120 [ 2231.450756] scsi_complete+0x8c/0x1c4 [ 2231.454406] blk_complete_reqs+0x5c/0x74 [ 2231.458315] blk_done_softirq+0x38/0x4c [ 2231.462138] __do_softirq+0x168/0x418 [ 2231.465786] __irq_exit_rcu+0x120/0x170 [ 2231.469609] irq_exit_rcu+0x18/0x24 [ 2231.473084] el1_interrupt+0x38/0x70 [ 2231.476647] el1h_64_irq_handler+0x18/0x24 [ 2231.480730] el1h_64_irq+0x68/0x6c [ 2231.484118] cpuidle_enter_state+0xc4/0x4e0 [ 2231.488288] cpuidle_enter+0x40/0x60 [ 2231.491851] cpuidle_idle_call+0x118/0x1b0 [ 2231.495934] do_idle+0xac/0x100 [ 2231.499063] cpu_startup_entry+0x30/0x34 [ 2231.502972] secondary_start_kernel+0xd8/0xf0 [ 2231.507316] __secondary_switched+0xb0/0xb4 [ 2239.099855] watchdog: BUG: soft lockup - CPU#8 stuck for 1691s! [kworker/8:12:19399] [ 2239.107584] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2239.107601] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2239.195388] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2239.203638] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2239.212929] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2239.219096] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2239.226044] pc : __sbq_wake_up+0x58/0x1f0 [ 2239.230041] lr : sbitmap_queue_clear+0x74/0xc0 [ 2239.234471] sp : ffff80000d79b280 [ 2239.237773] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2239.244895] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2239.252017] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2239.259139] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2239.266261] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2239.273384] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2239.280506] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2239.287628] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2239.294749] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2239.301871] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2239.308993] Call trace: [ 2239.311426] __sbq_wake_up+0x58/0x1f0 [ 2239.315075] sbitmap_queue_clear+0x74/0xc0 [ 2239.319159] blk_mq_put_tag+0x44/0x70 [ 2239.322808] __blk_mq_free_request+0x88/0xf0 [ 2239.327065] blk_mq_free_request+0xc0/0x160 [ 2239.331235] __blk_mq_end_request+0x78/0x150 [ 2239.335491] scsi_end_request+0x114/0x260 [ 2239.339488] scsi_io_completion+0x64/0x174 [ 2239.343571] scsi_finish_command+0xe0/0x120 [ 2239.347741] scsi_complete+0x8c/0x1c4 [ 2239.351391] blk_complete_reqs+0x5c/0x74 [ 2239.355302] blk_done_softirq+0x38/0x4c [ 2239.359124] __do_softirq+0x168/0x418 [ 2239.362773] __irq_exit_rcu+0x120/0x170 [ 2239.366596] irq_exit_rcu+0x18/0x24 [ 2239.370071] el1_interrupt+0x38/0x70 [ 2239.373634] el1h_64_irq_handler+0x18/0x24 [ 2239.377717] el1h_64_irq+0x68/0x6c [ 2239.381105] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2239.385884] __wake_up_common_lock+0x84/0xac [ 2239.390141] __wake_up+0x20/0x30 [ 2239.393356] __sbq_wake_up+0x13c/0x1f0 [ 2239.397092] sbitmap_queue_clear+0x74/0xc0 [ 2239.401175] blk_mq_put_tag+0x44/0x70 [ 2239.404825] __blk_mq_free_request+0x88/0xf0 [ 2239.409082] blk_mq_free_request+0xc0/0x160 [ 2239.413252] bfq_insert_request.isra.0+0x204/0x270 [ 2239.418030] bfq_insert_requests+0x70/0x90 [ 2239.422113] blk_mq_sched_insert_requests+0xac/0x250 [ 2239.427065] blk_mq_flush_plug_list+0x110/0x2cc [ 2239.431583] __blk_flush_plug+0x100/0x140 [ 2239.435579] blk_finish_plug+0x40/0x25c [ 2239.439402] __iomap_dio_rw+0x1b4/0x500 [ 2239.443224] iomap_dio_rw+0x18/0x50 [ 2239.446700] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2239.452051] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2239.456793] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2239.461837] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2239.466967] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2239.472096] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2239.477483] process_one_work+0x1ec/0x470 [ 2239.481479] worker_thread+0x74/0x410 [ 2239.485128] kthread+0xec/0x100 [ 2239.488256] ret_from_fork+0x10/0x20 [ 2255.209745] watchdog: BUG: soft lockup - CPU#68 stuck for 1706s! [swapper/68:0] [ 2255.217041] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2255.217057] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2255.304844] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2255.312659] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2255.321950] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2255.328898] pc : __sbq_wake_up+0x20/0x1f0 [ 2255.332896] lr : sbitmap_queue_clear+0x74/0xc0 [ 2255.337328] sp : ffff8000088438b0 [ 2255.340629] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2255.347751] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2255.354873] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2255.361994] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2255.369116] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2255.376238] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2255.383360] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2255.390482] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2255.397604] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2255.404727] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 2255.411849] Call trace: [ 2255.414282] __sbq_wake_up+0x20/0x1f0 [ 2255.417931] sbitmap_queue_clear+0x74/0xc0 [ 2255.422014] blk_mq_put_tag+0x44/0x70 [ 2255.425664] __blk_mq_free_request+0x88/0xf0 [ 2255.429921] blk_mq_free_request+0xc0/0x160 [ 2255.434090] __blk_mq_end_request+0x78/0x150 [ 2255.438347] scsi_end_request+0x114/0x260 [ 2255.442344] scsi_io_completion+0x64/0x174 [ 2255.446427] scsi_finish_command+0xe0/0x120 [ 2255.450597] scsi_complete+0x8c/0x1c4 [ 2255.454246] blk_complete_reqs+0x5c/0x74 [ 2255.458156] blk_done_softirq+0x38/0x4c [ 2255.461978] __do_softirq+0x168/0x418 [ 2255.465627] __irq_exit_rcu+0x120/0x170 [ 2255.469450] irq_exit_rcu+0x18/0x24 [ 2255.472926] el1_interrupt+0x38/0x70 [ 2255.476488] el1h_64_irq_handler+0x18/0x24 [ 2255.480571] el1h_64_irq+0x68/0x6c [ 2255.483959] cpuidle_enter_state+0xc4/0x4e0 [ 2255.488129] cpuidle_enter+0x40/0x60 [ 2255.491692] cpuidle_idle_call+0x118/0x1b0 [ 2255.495775] do_idle+0xac/0x100 [ 2255.498904] cpu_startup_entry+0x30/0x34 [ 2255.502813] secondary_start_kernel+0xd8/0xf0 [ 2255.507157] __secondary_switched+0xb0/0xb4 [ 2263.099692] watchdog: BUG: soft lockup - CPU#8 stuck for 1713s! [kworker/8:12:19399] [ 2263.107421] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2263.107437] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2263.195225] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2263.203474] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2263.212765] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2263.218933] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2263.225880] pc : __sbq_wake_up+0x58/0x1f0 [ 2263.229877] lr : sbitmap_queue_clear+0x74/0xc0 [ 2263.234308] sp : ffff80000d79b280 [ 2263.237609] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2263.244731] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2263.251853] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2263.258974] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2263.266097] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2263.273219] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2263.280340] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2263.287462] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2263.294584] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2263.301706] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2263.308828] Call trace: [ 2263.311261] __sbq_wake_up+0x58/0x1f0 [ 2263.314910] sbitmap_queue_clear+0x74/0xc0 [ 2263.318993] blk_mq_put_tag+0x44/0x70 [ 2263.322643] __blk_mq_free_request+0x88/0xf0 [ 2263.326900] blk_mq_free_request+0xc0/0x160 [ 2263.331069] __blk_mq_end_request+0x78/0x150 [ 2263.335327] scsi_end_request+0x114/0x260 [ 2263.339323] scsi_io_completion+0x64/0x174 [ 2263.343406] scsi_finish_command+0xe0/0x120 [ 2263.347577] scsi_complete+0x8c/0x1c4 [ 2263.351226] blk_complete_reqs+0x5c/0x74 [ 2263.355135] blk_done_softirq+0x38/0x4c [ 2263.358958] __do_softirq+0x168/0x418 [ 2263.362607] __irq_exit_rcu+0x120/0x170 [ 2263.366429] irq_exit_rcu+0x18/0x24 [ 2263.369905] el1_interrupt+0x38/0x70 [ 2263.373467] el1h_64_irq_handler+0x18/0x24 [ 2263.377550] el1h_64_irq+0x68/0x6c [ 2263.380939] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2263.385717] __wake_up_common_lock+0x84/0xac [ 2263.389974] __wake_up+0x20/0x30 [ 2263.393189] __sbq_wake_up+0x13c/0x1f0 [ 2263.396925] sbitmap_queue_clear+0x74/0xc0 [ 2263.401009] blk_mq_put_tag+0x44/0x70 [ 2263.404659] __blk_mq_free_request+0x88/0xf0 [ 2263.408915] blk_mq_free_request+0xc0/0x160 [ 2263.413085] bfq_insert_request.isra.0+0x204/0x270 [ 2263.417863] bfq_insert_requests+0x70/0x90 [ 2263.421948] blk_mq_sched_insert_requests+0xac/0x250 [ 2263.426901] blk_mq_flush_plug_list+0x110/0x2cc [ 2263.431418] __blk_flush_plug+0x100/0x140 [ 2263.435415] blk_finish_plug+0x40/0x25c [ 2263.439237] __iomap_dio_rw+0x1b4/0x500 [ 2263.443060] iomap_dio_rw+0x18/0x50 [ 2263.446535] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2263.451886] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2263.456628] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2263.461672] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2263.466802] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2263.471931] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2263.477317] process_one_work+0x1ec/0x470 [ 2263.481313] worker_thread+0x74/0x410 [ 2263.484962] kthread+0xec/0x100 [ 2263.488092] ret_from_fork+0x10/0x20 [ 2269.159661] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 181821 jiffies s: 2093 root: 0x11/. [ 2269.171399] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2269.180608] Task dump for CPU 8: [ 2269.183822] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2269.193724] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2269.199894] Call trace: [ 2269.202327] __switch_to+0xd4/0x100 [ 2269.205803] 0xffff6c0200000080 [ 2269.208931] Task dump for CPU 68: [ 2269.212234] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2269.222136] Call trace: [ 2269.224569] __switch_to+0xd4/0x100 [ 2269.228045] 0x0 [ 2279.209582] watchdog: BUG: soft lockup - CPU#68 stuck for 1728s! [swapper/68:0] [ 2279.216877] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2279.216894] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2279.304680] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2279.312496] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2279.321787] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2279.328735] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2279.332818] lr : sbitmap_queue_clear+0x74/0xc0 [ 2279.337248] sp : ffff8000088438b0 [ 2279.340550] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2279.347672] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2279.354794] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2279.361915] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2279.369037] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2279.376159] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2279.383281] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2279.390403] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2279.397524] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2279.404646] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2279.411768] Call trace: [ 2279.414201] __sbq_wake_up+0x1c8/0x1f0 [ 2279.417937] sbitmap_queue_clear+0x74/0xc0 [ 2279.422021] blk_mq_put_tag+0x44/0x70 [ 2279.425670] __blk_mq_free_request+0x88/0xf0 [ 2279.429927] blk_mq_free_request+0xc0/0x160 [ 2279.434097] __blk_mq_end_request+0x78/0x150 [ 2279.438355] scsi_end_request+0x114/0x260 [ 2279.442352] scsi_io_completion+0x64/0x174 [ 2279.446435] scsi_finish_command+0xe0/0x120 [ 2279.450605] scsi_complete+0x8c/0x1c4 [ 2279.454254] blk_complete_reqs+0x5c/0x74 [ 2279.458164] blk_done_softirq+0x38/0x4c [ 2279.461987] __do_softirq+0x168/0x418 [ 2279.465636] __irq_exit_rcu+0x120/0x170 [ 2279.469458] irq_exit_rcu+0x18/0x24 [ 2279.472934] el1_interrupt+0x38/0x70 [ 2279.476496] el1h_64_irq_handler+0x18/0x24 [ 2279.480579] el1h_64_irq+0x68/0x6c [ 2279.483968] cpuidle_enter_state+0xc4/0x4e0 [ 2279.488138] cpuidle_enter+0x40/0x60 [ 2279.491700] cpuidle_idle_call+0x118/0x1b0 [ 2279.495786] do_idle+0xac/0x100 [ 2279.498914] cpu_startup_entry+0x30/0x34 [ 2279.502824] secondary_start_kernel+0xd8/0xf0 [ 2279.507168] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:35:00 2022] [ 2287.099528] watchdog: BUG: soft lockup - CPU#8 stuck for 1736s! [kworker/8:12:19399] [ 2287.107258] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2287.107274] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2287.195061] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2287.203311] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2287.212602] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2287.218770] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2287.225719] pc : __sbq_wake_up+0x4/0x1f0 [ 2287.229629] lr : sbitmap_queue_clear+0x74/0xc0 [ 2287.234059] sp : ffff80000d79b2d0 [ 2287.237361] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2287.244482] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2287.251604] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2287.258726] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2287.265849] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2287.272971] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2287.280093] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2287.287215] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2287.294337] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2287.301459] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 2287.308581] Call trace: [ 2287.311014] __sbq_wake_up+0x4/0x1f0 [ 2287.314576] blk_mq_put_tag+0x44/0x70 [ 2287.318226] __blk_mq_free_request+0x88/0xf0 [ 2287.322482] blk_mq_free_request+0xc0/0x160 [ 2287.326652] __blk_mq_end_request+0x78/0x150 [ 2287.330909] scsi_end_request+0x114/0x260 [ 2287.334906] scsi_io_completion+0x64/0x174 [ 2287.338989] scsi_finish_command+0xe0/0x120 [ 2287.343159] scsi_complete+0x8c/0x1c4 [ 2287.346808] blk_complete_reqs+0x5c/0x74 [ 2287.350718] blk_done_softirq+0x38/0x4c [ 2287.354541] __do_softirq+0x168/0x418 [ 2287.358190] __irq_exit_rcu+0x120/0x170 [ 2287.362012] irq_exit_rcu+0x18/0x24 [ 2287.365488] el1_interrupt+0x38/0x70 [ 2287.369050] el1h_64_irq_handler+0x18/0x24 [ 2287.373133] el1h_64_irq+0x68/0x6c [ 2287.376522] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2287.381299] __wake_up_common_lock+0x84/0xac [ 2287.385557] __wake_up+0x20/0x30 [ 2287.388772] __sbq_wake_up+0x13c/0x1f0 [ 2287.392508] sbitmap_queue_clear+0x74/0xc0 [ 2287.396592] blk_mq_put_tag+0x44/0x70 [ 2287.400242] __blk_mq_free_request+0x88/0xf0 [ 2287.404499] blk_mq_free_request+0xc0/0x160 [ 2287.408669] bfq_insert_request.isra.0+0x204/0x270 [ 2287.413448] bfq_insert_requests+0x70/0x90 [ 2287.417531] blk_mq_sched_insert_requests+0xac/0x250 [ 2287.422483] blk_mq_flush_plug_list+0x110/0x2cc [ 2287.427001] __blk_flush_plug+0x100/0x140 [ 2287.430997] blk_finish_plug+0x40/0x25c [ 2287.434820] __iomap_dio_rw+0x1b4/0x500 [ 2287.438642] iomap_dio_rw+0x18/0x50 [ 2287.442118] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2287.447468] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2287.452210] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2287.457254] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2287.462385] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2287.467514] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2287.472901] process_one_work+0x1ec/0x470 [ 2287.476897] worker_thread+0x74/0x410 [ 2287.480546] kthread+0xec/0x100 [ 2287.483675] ret_from_fork+0x10/0x20 [ 2287.989523] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2287.995255] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=92421 [ 2288.004289] (t=186311 jiffies g=36445 q=54526 ncpus=128) [ 2288.009675] Task dump for CPU 8: [ 2288.012889] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2288.022789] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2288.028957] Call trace: [ 2288.031390] dump_backtrace+0xe8/0x130 [ 2288.035126] show_stack+0x20/0x60 [ 2288.038428] sched_show_task+0x160/0x190 [ 2288.042338] dump_cpu_task+0x4c/0x78 [ 2288.045901] rcu_dump_cpu_stacks+0xc4/0x150 [ 2288.050072] print_cpu_stall+0x180/0x210 [ 2288.053982] check_cpu_stall+0x20c/0x2b4 [ 2288.057891] rcu_pending+0x4c/0x190 [ 2288.061367] rcu_sched_clock_irq+0x80/0x204 [ 2288.065539] update_process_times+0x7c/0xc0 [ 2288.069709] tick_sched_handle+0x38/0x74 [ 2288.073619] tick_sched_timer+0x60/0xc0 [ 2288.077442] __hrtimer_run_queues+0x170/0x300 [ 2288.081786] hrtimer_interrupt+0xfc/0x260 [ 2288.085782] arch_timer_handler_phys+0x34/0x50 [ 2288.090213] handle_percpu_devid_irq+0xac/0x240 [ 2288.094731] generic_handle_domain_irq+0x34/0x50 [ 2288.099336] gic_handle_irq+0x5c/0xe0 [ 2288.102985] call_on_irq_stack+0x2c/0x38 [ 2288.106895] do_interrupt_handler+0xdc/0xe0 [ 2288.111065] el1_interrupt+0x34/0x70 [ 2288.114628] el1h_64_irq_handler+0x18/0x24 [ 2288.118711] el1h_64_irq+0x68/0x6c [ 2288.122099] __sbq_wake_up+0x74/0x1f0 [ 2288.125749] sbitmap_queue_clear+0x74/0xc0 [ 2288.129833] blk_mq_put_tag+0x44/0x70 [ 2288.133482] __blk_mq_free_request+0x88/0xf0 [ 2288.137739] blk_mq_free_request+0xc0/0x160 [ 2288.141909] __blk_mq_end_request+0x78/0x150 [ 2288.146165] scsi_end_request+0x114/0x260 [ 2288.150162] scsi_io_completion+0x64/0x174 [ 2288.154245] scsi_finish_command+0xe0/0x120 [ 2288.158415] scsi_complete+0x8c/0x1c4 [ 2288.162065] blk_complete_reqs+0x5c/0x74 [ 2288.165976] blk_done_softirq+0x38/0x4c [ 2288.169798] __do_softirq+0x168/0x418 [ 2288.173447] __irq_exit_rcu+0x120/0x170 [ 2288.177269] irq_exit_rcu+0x18/0x24 [ 2288.180745] el1_interrupt+0x38/0x70 [ 2288.184307] el1h_64_irq_handler+0x18/0x24 [ 2288.188390] el1h_64_irq+0x68/0x6c [ 2288.191779] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2288.196557] __wake_up_common_lock+0x84/0xac [ 2288.200814] __wake_up+0x20/0x30 [ 2288.204030] __sbq_wake_up+0x13c/0x1f0 [ 2288.207766] sbitmap_queue_clear+0x74/0xc0 [ 2288.211849] blk_mq_put_tag+0x44/0x70 [ 2288.215498] __blk_mq_free_request+0x88/0xf0 [ 2288.219755] blk_mq_free_request+0xc0/0x160 [ 2288.223925] bfq_insert_request.isra.0+0x204/0x270 [ 2288.228703] bfq_insert_requests+0x70/0x90 [ 2288.232787] blk_mq_sched_insert_requests+0xac/0x250 [ 2288.237738] blk_mq_flush_plug_list+0x110/0x2cc [ 2288.242256] __blk_flush_plug+0x100/0x140 [ 2288.246252] blk_finish_plug+0x40/0x25c [ 2288.250075] __iomap_dio_rw+0x1b4/0x500 [ 2288.253898] iomap_dio_rw+0x18/0x50 [ 2288.257374] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2288.262725] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2288.267467] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2288.272511] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2288.277641] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2288.282771] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2288.288157] process_one_work+0x1ec/0x470 [ 2288.292153] worker_thread+0x74/0x410 [ 2288.295802] kthread+0xec/0x100 [ 2288.298931] ret_from_fork+0x10/0x20 [ 2288.302494] Task dump for CPU 68: [ 2288.305795] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2288.315695] Call trace: [ 2288.318128] __switch_to+0xd4/0x100 [ 2288.321605] 0x0 [ 2303.209419] watchdog: BUG: soft lockup - CPU#68 stuck for 1751s! [swapper/68:0] [ 2303.216714] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2303.216730] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2303.304517] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2303.312332] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2303.321623] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2303.328571] pc : __sbq_wake_up+0x28/0x1f0 [ 2303.332567] lr : sbitmap_queue_clear+0x74/0xc0 [ 2303.336998] sp : ffff8000088438b0 [ 2303.340299] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2303.347421] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2303.354542] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2303.361664] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2303.368786] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2303.375908] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2303.383030] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2303.390151] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2303.397273] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2303.404395] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2303.411517] Call trace: [ 2303.413950] __sbq_wake_up+0x28/0x1f0 [ 2303.417599] sbitmap_queue_clear+0x74/0xc0 [ 2303.421683] blk_mq_put_tag+0x44/0x70 [ 2303.425333] __blk_mq_free_request+0x88/0xf0 [ 2303.429589] blk_mq_free_request+0xc0/0x160 [ 2303.433759] __blk_mq_end_request+0x78/0x150 [ 2303.438016] scsi_end_request+0x114/0x260 [ 2303.442012] scsi_io_completion+0x64/0x174 [ 2303.446095] scsi_finish_command+0xe0/0x120 [ 2303.450266] scsi_complete+0x8c/0x1c4 [ 2303.453915] blk_complete_reqs+0x5c/0x74 [ 2303.457825] blk_done_softirq+0x38/0x4c [ 2303.461648] __do_softirq+0x168/0x418 [ 2303.465297] __irq_exit_rcu+0x120/0x170 [ 2303.469119] irq_exit_rcu+0x18/0x24 [ 2303.472595] el1_interrupt+0x38/0x70 [ 2303.476157] el1h_64_irq_handler+0x18/0x24 [ 2303.480240] el1h_64_irq+0x68/0x6c [ 2303.483628] cpuidle_enter_state+0xc4/0x4e0 [ 2303.487798] cpuidle_enter+0x40/0x60 [ 2303.491362] cpuidle_idle_call+0x118/0x1b0 [ 2303.495446] do_idle+0xac/0x100 [ 2303.498574] cpu_startup_entry+0x30/0x34 [ 2303.502483] secondary_start_kernel+0xd8/0xf0 [ 2303.506827] __secondary_switched+0xb0/0xb4 [ 2315.099338] watchdog: BUG: soft lockup - CPU#8 stuck for 1762s! [kworker/8:12:19399] [ 2315.107067] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2315.107084] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2315.194872] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2315.203121] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2315.212412] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2315.218580] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2315.225527] pc : sbitmap_queue_clear+0x6c/0xc0 [ 2315.229958] lr : sbitmap_queue_clear+0x74/0xc0 [ 2315.234389] sp : ffff80000d79b2d0 [ 2315.237690] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2315.244812] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2315.251935] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2315.259057] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2315.266179] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2315.273300] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2315.280422] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2315.287544] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2315.294666] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2315.301788] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 2315.308910] Call trace: [ 2315.311343] sbitmap_queue_clear+0x6c/0xc0 [ 2315.315427] blk_mq_put_tag+0x44/0x70 [ 2315.319076] __blk_mq_free_request+0x88/0xf0 [ 2315.323333] blk_mq_free_request+0xc0/0x160 [ 2315.327503] __blk_mq_end_request+0x78/0x150 [ 2315.331760] scsi_end_request+0x114/0x260 [ 2315.335756] scsi_io_completion+0x64/0x174 [ 2315.339839] scsi_finish_command+0xe0/0x120 [ 2315.344010] scsi_complete+0x8c/0x1c4 [ 2315.347659] blk_complete_reqs+0x5c/0x74 [ 2315.351568] blk_done_softirq+0x38/0x4c [ 2315.355391] __do_softirq+0x168/0x418 [ 2315.359040] __irq_exit_rcu+0x120/0x170 [ 2315.362863] irq_exit_rcu+0x18/0x24 [ 2315.366338] el1_interrupt+0x38/0x70 [ 2315.369901] el1h_64_irq_handler+0x18/0x24 [ 2315.373984] el1h_64_irq+0x68/0x6c [ 2315.377372] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2315.382150] __wake_up_common_lock+0x84/0xac [ 2315.386407] __wake_up+0x20/0x30 [ 2315.389623] __sbq_wake_up+0x13c/0x1f0 [ 2315.393359] sbitmap_queue_clear+0x74/0xc0 [ 2315.397442] blk_mq_put_tag+0x44/0x70 [ 2315.401092] __blk_mq_free_request+0x88/0xf0 [ 2315.405349] blk_mq_free_request+0xc0/0x160 [ 2315.409519] bfq_insert_request.isra.0+0x204/0x270 [ 2315.414297] bfq_insert_requests+0x70/0x90 [ 2315.418380] blk_mq_sched_insert_requests+0xac/0x250 [ 2315.423332] blk_mq_flush_plug_list+0x110/0x2cc [ 2315.427850] __blk_flush_plug+0x100/0x140 [ 2315.431846] blk_finish_plug+0x40/0x25c [ 2315.435669] __iomap_dio_rw+0x1b4/0x500 [ 2315.439492] iomap_dio_rw+0x18/0x50 [ 2315.442968] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2315.448318] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2315.453061] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2315.458104] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2315.463235] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2315.468364] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2315.473751] process_one_work+0x1ec/0x470 [ 2315.477747] worker_thread+0x74/0x410 [ 2315.481395] kthread+0xec/0x100 [ 2315.484524] ret_from_fork+0x10/0x20 [ 2327.209255] watchdog: BUG: soft lockup - CPU#68 stuck for 1773s! [swapper/68:0] [ 2327.216551] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2327.216567] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2327.304354] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2327.312169] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2327.321460] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2327.328408] pc : __sbq_wake_up+0x40/0x1f0 [ 2327.332404] lr : sbitmap_queue_clear+0x74/0xc0 [ 2327.336835] sp : ffff8000088438b0 [ 2327.340136] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2327.347258] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2327.354380] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2327.361502] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2327.368624] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2327.375746] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2327.382867] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2327.389989] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2327.397111] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2327.404233] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2327.411355] Call trace: [ 2327.413787] __sbq_wake_up+0x40/0x1f0 [ 2327.417437] sbitmap_queue_clear+0x74/0xc0 [ 2327.421520] blk_mq_put_tag+0x44/0x70 [ 2327.425169] __blk_mq_free_request+0x88/0xf0 [ 2327.429426] blk_mq_free_request+0xc0/0x160 [ 2327.433596] __blk_mq_end_request+0x78/0x150 [ 2327.437853] scsi_end_request+0x114/0x260 [ 2327.441849] scsi_io_completion+0x64/0x174 [ 2327.445932] scsi_finish_command+0xe0/0x120 [ 2327.450102] scsi_complete+0x8c/0x1c4 [ 2327.453751] blk_complete_reqs+0x5c/0x74 [ 2327.457661] blk_done_softirq+0x38/0x4c [ 2327.461484] __do_softirq+0x168/0x418 [ 2327.465133] __irq_exit_rcu+0x120/0x170 [ 2327.468956] irq_exit_rcu+0x18/0x24 [ 2327.472431] el1_interrupt+0x38/0x70 [ 2327.475993] el1h_64_irq_handler+0x18/0x24 [ 2327.480076] el1h_64_irq+0x68/0x6c [ 2327.483464] cpuidle_enter_state+0xc4/0x4e0 [ 2327.487634] cpuidle_enter+0x40/0x60 [ 2327.491197] cpuidle_idle_call+0x118/0x1b0 [ 2327.495281] do_idle+0xac/0x100 [ 2327.498410] cpu_startup_entry+0x30/0x34 [ 2327.502319] secondary_start_kernel+0xd8/0xf0 [ 2327.506663] __secondary_switched+0xb0/0xb4 [ 2335.719207] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 188477 jiffies s: 2093 root: 0x11/. [ 2335.730945] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2335.740154] Task dump for CPU 8: [ 2335.743368] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2335.753270] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2335.759440] Call trace: [ 2335.761872] __switch_to+0xd4/0x100 [ 2335.765349] 0xffff6c0200000080 [ 2335.768477] Task dump for CPU 68: [ 2335.771780] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2335.781681] Call trace: [ 2335.784114] __switch_to+0xd4/0x100 [ 2335.787590] 0x0 [ 2339.099174] watchdog: BUG: soft lockup - CPU#8 stuck for 1784s! [kworker/8:12:19399] [ 2339.106904] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2339.106921] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2339.194708] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2339.202958] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2339.212249] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2339.218416] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2339.225364] pc : __sbq_wake_up+0x58/0x1f0 [ 2339.229361] lr : sbitmap_queue_clear+0x74/0xc0 [ 2339.233792] sp : ffff80000d79b280 [ 2339.237093] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2339.244215] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2339.251337] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2339.258459] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2339.265581] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2339.272703] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2339.279825] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2339.286947] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2339.294069] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2339.301191] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2339.308313] Call trace: [ 2339.310746] __sbq_wake_up+0x58/0x1f0 [ 2339.314395] sbitmap_queue_clear+0x74/0xc0 [ 2339.318479] blk_mq_put_tag+0x44/0x70 [ 2339.322129] __blk_mq_free_request+0x88/0xf0 [ 2339.326385] blk_mq_free_request+0xc0/0x160 [ 2339.330555] __blk_mq_end_request+0x78/0x150 [ 2339.334812] scsi_end_request+0x114/0x260 [ 2339.338808] scsi_io_completion+0x64/0x174 [ 2339.342891] scsi_finish_command+0xe0/0x120 [ 2339.347062] scsi_complete+0x8c/0x1c4 [ 2339.350711] blk_complete_reqs+0x5c/0x74 [ 2339.354620] blk_done_softirq+0x38/0x4c [ 2339.358443] __do_softirq+0x168/0x418 [ 2339.362092] __irq_exit_rcu+0x120/0x170 [ 2339.365915] irq_exit_rcu+0x18/0x24 [ 2339.369390] el1_interrupt+0x38/0x70 [ 2339.372952] el1h_64_irq_handler+0x18/0x24 [ 2339.377037] el1h_64_irq+0x68/0x6c [ 2339.380426] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2339.385204] __wake_up_common_lock+0x84/0xac [ 2339.389461] __wake_up+0x20/0x30 [ 2339.392677] __sbq_wake_up+0x13c/0x1f0 [ 2339.396413] sbitmap_queue_clear+0x74/0xc0 [ 2339.400497] blk_mq_put_tag+0x44/0x70 [ 2339.404146] __blk_mq_free_request+0x88/0xf0 [ 2339.408403] blk_mq_free_request+0xc0/0x160 [ 2339.412573] bfq_insert_request.isra.0+0x204/0x270 [ 2339.417351] bfq_insert_requests+0x70/0x90 [ 2339.421436] blk_mq_sched_insert_requests+0xac/0x250 [ 2339.426389] blk_mq_flush_plug_list+0x110/0x2cc [ 2339.430907] __blk_flush_plug+0x100/0x140 [ 2339.434903] blk_finish_plug+0x40/0x25c [ 2339.438726] __iomap_dio_rw+0x1b4/0x500 [ 2339.442549] iomap_dio_rw+0x18/0x50 [ 2339.446024] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2339.451375] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2339.456117] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2339.461161] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2339.466291] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2339.471421] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2339.476806] process_one_work+0x1ec/0x470 [ 2339.480803] worker_thread+0x74/0x410 [ 2339.484451] kthread+0xec/0x100 [ 2339.487580] ret_from_fork+0x10/0x20 [ 2351.209092] watchdog: BUG: soft lockup - CPU#68 stuck for 1795s! [swapper/68:0] [ 2351.216387] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2351.216404] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2351.304191] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2351.312006] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2351.321297] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2351.328245] pc : __sbq_wake_up+0x58/0x1f0 [ 2351.332241] lr : sbitmap_queue_clear+0x74/0xc0 [ 2351.336671] sp : ffff8000088438b0 [ 2351.339972] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2351.347094] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2351.354216] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2351.361338] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2351.368459] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2351.375581] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2351.382703] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2351.389825] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2351.396947] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2351.404069] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2351.411191] Call trace: [ 2351.413625] __sbq_wake_up+0x58/0x1f0 [ 2351.417274] sbitmap_queue_clear+0x74/0xc0 [ 2351.421358] blk_mq_put_tag+0x44/0x70 [ 2351.425008] __blk_mq_free_request+0x88/0xf0 [ 2351.429265] blk_mq_free_request+0xc0/0x160 [ 2351.433435] __blk_mq_end_request+0x78/0x150 [ 2351.437692] scsi_end_request+0x114/0x260 [ 2351.441688] scsi_io_completion+0x64/0x174 [ 2351.445771] scsi_finish_command+0xe0/0x120 [ 2351.449942] scsi_complete+0x8c/0x1c4 [ 2351.453592] blk_complete_reqs+0x5c/0x74 [ 2351.457501] blk_done_softirq+0x38/0x4c [ 2351.461324] __do_softirq+0x168/0x418 [ 2351.464972] __irq_exit_rcu+0x120/0x170 [ 2351.468795] irq_exit_rcu+0x18/0x24 [ 2351.472270] el1_interrupt+0x38/0x70 [ 2351.475833] el1h_64_irq_handler+0x18/0x24 [ 2351.479916] el1h_64_irq+0x68/0x6c [ 2351.483304] cpuidle_enter_state+0xc4/0x4e0 [ 2351.487476] cpuidle_enter+0x40/0x60 [ 2351.491039] cpuidle_idle_call+0x118/0x1b0 [ 2351.495123] do_idle+0xac/0x100 [ 2351.498252] cpu_startup_entry+0x30/0x34 [ 2351.502161] secondary_start_kernel+0xd8/0xf0 [ 2351.506505] __secondary_switched+0xb0/0xb4 [ 2363.099011] watchdog: BUG: soft lockup - CPU#8 stuck for 1806s! [kworker/8:12:19399] [ 2363.106740] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2363.106757] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2363.194543] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2363.202793] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2363.212084] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2363.218251] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2363.225199] pc : __sbq_wake_up+0x58/0x1f0 [ 2363.229196] lr : sbitmap_queue_clear+0x74/0xc0 [ 2363.233626] sp : ffff80000d79b280 [ 2363.236928] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2363.244049] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2363.251171] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2363.258293] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2363.265415] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2363.272537] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2363.279659] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2363.286781] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2363.293903] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2363.301025] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2363.308146] Call trace: [ 2363.310580] __sbq_wake_up+0x58/0x1f0 [ 2363.314229] sbitmap_queue_clear+0x74/0xc0 [ 2363.318312] blk_mq_put_tag+0x44/0x70 [ 2363.321962] __blk_mq_free_request+0x88/0xf0 [ 2363.326219] blk_mq_free_request+0xc0/0x160 [ 2363.330389] __blk_mq_end_request+0x78/0x150 [ 2363.334646] scsi_end_request+0x114/0x260 [ 2363.338642] scsi_io_completion+0x64/0x174 [ 2363.342725] scsi_finish_command+0xe0/0x120 [ 2363.346896] scsi_complete+0x8c/0x1c4 [ 2363.350545] blk_complete_reqs+0x5c/0x74 [ 2363.354454] blk_done_softirq+0x38/0x4c [ 2363.358278] __do_softirq+0x168/0x418 [ 2363.361927] __irq_exit_rcu+0x120/0x170 [ 2363.365749] irq_exit_rcu+0x18/0x24 [ 2363.369225] el1_interrupt+0x38/0x70 [ 2363.372787] el1h_64_irq_handler+0x18/0x24 [ 2363.376870] el1h_64_irq+0x68/0x6c [ 2363.380259] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2363.385036] __wake_up_common_lock+0x84/0xac [ 2363.389294] __wake_up+0x20/0x30 [ 2363.392509] __sbq_wake_up+0x13c/0x1f0 [ 2363.396246] sbitmap_queue_clear+0x74/0xc0 [ 2363.400329] blk_mq_put_tag+0x44/0x70 [ 2363.403978] __blk_mq_free_request+0x88/0xf0 [ 2363.408235] blk_mq_free_request+0xc0/0x160 [ 2363.412405] bfq_insert_request.isra.0+0x204/0x270 [ 2363.417183] bfq_insert_requests+0x70/0x90 [ 2363.421266] blk_mq_sched_insert_requests+0xac/0x250 [ 2363.426218] blk_mq_flush_plug_list+0x110/0x2cc [ 2363.430735] __blk_flush_plug+0x100/0x140 [ 2363.434732] blk_finish_plug+0x40/0x25c [ 2363.438555] __iomap_dio_rw+0x1b4/0x500 [ 2363.442379] iomap_dio_rw+0x18/0x50 [ 2363.445855] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2363.451206] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2363.455948] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2363.460993] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2363.466123] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2363.471253] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2363.476639] process_one_work+0x1ec/0x470 [ 2363.480635] worker_thread+0x74/0x410 [ 2363.484283] kthread+0xec/0x100 [ 2363.487412] ret_from_fork+0x10/0x20 [ 2375.208928] watchdog: BUG: soft lockup - CPU#68 stuck for 1818s! [swapper/68:0] [ 2375.216223] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2375.216240] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2375.304026] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2375.311842] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2375.321133] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2375.328080] pc : __sbq_wake_up+0x8/0x1f0 [ 2375.331990] lr : sbitmap_queue_clear+0x74/0xc0 [ 2375.336420] sp : ffff8000088438b0 [ 2375.339722] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2375.346843] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2375.353966] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2375.361088] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2375.368210] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2375.375332] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2375.382454] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2375.389575] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2375.396697] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2375.403819] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 2375.410941] Call trace: [ 2375.413374] __sbq_wake_up+0x8/0x1f0 [ 2375.416936] blk_mq_put_tag+0x44/0x70 [ 2375.420586] __blk_mq_free_request+0x88/0xf0 [ 2375.424843] blk_mq_free_request+0xc0/0x160 [ 2375.429013] __blk_mq_end_request+0x78/0x150 [ 2375.433270] scsi_end_request+0x114/0x260 [ 2375.437266] scsi_io_completion+0x64/0x174 [ 2375.441349] scsi_finish_command+0xe0/0x120 [ 2375.445520] scsi_complete+0x8c/0x1c4 [ 2375.449169] blk_complete_reqs+0x5c/0x74 [ 2375.453078] blk_done_softirq+0x38/0x4c [ 2375.456901] __do_softirq+0x168/0x418 [ 2375.460549] __irq_exit_rcu+0x120/0x170 [ 2375.464372] irq_exit_rcu+0x18/0x24 [ 2375.467848] el1_interrupt+0x38/0x70 [ 2375.471410] el1h_64_irq_handler+0x18/0x24 [ 2375.475494] el1h_64_irq+0x68/0x6c [ 2375.478882] cpuidle_enter_state+0xc4/0x4e0 [ 2375.483052] cpuidle_enter+0x40/0x60 [ 2375.486614] cpuidle_idle_call+0x118/0x1b0 [ 2375.490697] do_idle+0xac/0x100 [ 2375.493826] cpu_startup_entry+0x30/0x34 [ 2375.497736] secondary_start_kernel+0xd8/0xf0 [ 2375.502080] __secondary_switched+0xb0/0xb4 [ 2387.098847] watchdog: BUG: soft lockup - CPU#8 stuck for 1829s! [kworker/8:12:19399] [ 2387.106577] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2387.106593] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2387.194381] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2387.202630] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2387.211921] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2387.218089] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2387.225036] pc : __sbq_wake_up+0x7c/0x1f0 [ 2387.229033] lr : sbitmap_queue_clear+0x74/0xc0 [ 2387.233464] sp : ffff80000d79b2d0 [ 2387.236765] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2387.243888] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2387.251010] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2387.258132] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2387.265253] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2387.272375] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2387.279497] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2387.286619] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2387.293741] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2387.300862] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 2387.307984] Call trace: [ 2387.310417] __sbq_wake_up+0x7c/0x1f0 [ 2387.314067] blk_mq_put_tag+0x44/0x70 [ 2387.317716] __blk_mq_free_request+0x88/0xf0 [ 2387.321973] blk_mq_free_request+0xc0/0x160 [ 2387.326143] __blk_mq_end_request+0x78/0x150 [ 2387.330399] scsi_end_request+0x114/0x260 [ 2387.334396] scsi_io_completion+0x64/0x174 [ 2387.338479] scsi_finish_command+0xe0/0x120 [ 2387.342649] scsi_complete+0x8c/0x1c4 [ 2387.346298] blk_complete_reqs+0x5c/0x74 [ 2387.350208] blk_done_softirq+0x38/0x4c [ 2387.354031] __do_softirq+0x168/0x418 [ 2387.357680] __irq_exit_rcu+0x120/0x170 [ 2387.361502] irq_exit_rcu+0x18/0x24 [ 2387.364978] el1_interrupt+0x38/0x70 [ 2387.368540] el1h_64_irq_handler+0x18/0x24 [ 2387.372623] el1h_64_irq+0x68/0x6c [ 2387.376012] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2387.380789] __wake_up_common_lock+0x84/0xac [ 2387.385047] __wake_up+0x20/0x30 [ 2387.388263] __sbq_wake_up+0x13c/0x1f0 [ 2387.391999] sbitmap_queue_clear+0x74/0xc0 [ 2387.396082] blk_mq_put_tag+0x44/0x70 [ 2387.399732] __blk_mq_free_request+0x88/0xf0 [ 2387.403989] blk_mq_free_request+0xc0/0x160 [ 2387.408159] bfq_insert_request.isra.0+0x204/0x270 [ 2387.412938] bfq_insert_requests+0x70/0x90 [ 2387.417022] blk_mq_sched_insert_requests+0xac/0x250 [ 2387.421974] blk_mq_flush_plug_list+0x110/0x2cc [ 2387.426491] __blk_flush_plug+0x100/0x140 [ 2387.430488] blk_finish_plug+0x40/0x25c [ 2387.434310] __iomap_dio_rw+0x1b4/0x500 [ 2387.438134] iomap_dio_rw+0x18/0x50 [ 2387.441609] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2387.446959] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2387.451702] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2387.456746] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2387.461876] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2387.467006] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2387.472392] process_one_work+0x1ec/0x470 [ 2387.476387] worker_thread+0x74/0x410 [ 2387.480036] kthread+0xec/0x100 [ 2387.483165] ret_from_fork+0x10/0x20 [ 2399.208765] watchdog: BUG: soft lockup - CPU#68 stuck for 1840s! [swapper/68:0] [ 2399.216060] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2399.216076] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2399.303864] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2399.311680] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2399.320971] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2399.327918] pc : __sbq_wake_up+0x58/0x1f0 [ 2399.331915] lr : sbitmap_queue_clear+0x74/0xc0 [ 2399.336345] sp : ffff8000088438b0 [ 2399.339646] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2399.346768] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2399.353890] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2399.361012] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2399.368134] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2399.375256] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2399.382378] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2399.389500] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2399.396622] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2399.403744] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2399.410865] Call trace: [ 2399.413298] __sbq_wake_up+0x58/0x1f0 [ 2399.416947] sbitmap_queue_clear+0x74/0xc0 [ 2399.421031] blk_mq_put_tag+0x44/0x70 [ 2399.424680] __blk_mq_free_request+0x88/0xf0 [ 2399.428937] blk_mq_free_request+0xc0/0x160 [ 2399.433107] __blk_mq_end_request+0x78/0x150 [ 2399.437366] scsi_end_request+0x114/0x260 [ 2399.441362] scsi_io_completion+0x64/0x174 [ 2399.445445] scsi_finish_command+0xe0/0x120 [ 2399.449615] scsi_complete+0x8c/0x1c4 [ 2399.453264] blk_complete_reqs+0x5c/0x74 [ 2399.457174] blk_done_softirq+0x38/0x4c [ 2399.460997] __do_softirq+0x168/0x418 [ 2399.464645] __irq_exit_rcu+0x120/0x170 [ 2399.468468] irq_exit_rcu+0x18/0x24 [ 2399.471943] el1_interrupt+0x38/0x70 [ 2399.475505] el1h_64_irq_handler+0x18/0x24 [ 2399.479590] el1h_64_irq+0x68/0x6c [ 2399.482978] cpuidle_enter_state+0xc4/0x4e0 [ 2399.487149] cpuidle_enter+0x40/0x60 [ 2399.490711] cpuidle_idle_call+0x118/0x1b0 [ 2399.494795] do_idle+0xac/0x100 [ 2399.497923] cpu_startup_entry+0x30/0x34 [ 2399.501833] secondary_start_kernel+0xd8/0xf0 [ 2399.506177] __secondary_switched+0xb0/0xb4 [ 2402.278754] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 195133 jiffies s: 2093 root: 0x11/. [ 2402.290493] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2402.299701] Task dump for CPU 8: [ 2402.302916] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2402.312818] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2402.318987] Call trace: [ 2402.321420] __switch_to+0xd4/0x100 [ 2402.324897] 0xffff6c0200000080 [ 2402.328028] Task dump for CPU 68: [ 2402.331330] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2402.341232] Call trace: [ 2402.343665] __switch_to+0xd4/0x100 [ 2402.347141] 0x0 [ 2411.098684] watchdog: BUG: soft lockup - CPU#8 stuck for 1851s! [kworker/8:12:19399] [ 2411.106413] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2411.106430] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2411.194218] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2411.202468] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2411.211759] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2411.217928] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2411.224875] pc : __sbq_wake_up+0x84/0x1f0 [ 2411.228872] lr : sbitmap_queue_clear+0x74/0xc0 [ 2411.233302] sp : ffff80000d79b280 [ 2411.236604] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2411.243726] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2411.250848] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2411.257970] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2411.265093] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2411.272215] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2411.279336] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2411.286458] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2411.293580] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2411.300702] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2411.307824] Call trace: [ 2411.310257] __sbq_wake_up+0x84/0x1f0 [ 2411.313907] sbitmap_queue_clear+0x74/0xc0 [ 2411.317990] blk_mq_put_tag+0x44/0x70 [ 2411.321640] __blk_mq_free_request+0x88/0xf0 [ 2411.325897] blk_mq_free_request+0xc0/0x160 [ 2411.330066] __blk_mq_end_request+0x78/0x150 [ 2411.334323] scsi_end_request+0x114/0x260 [ 2411.338319] scsi_io_completion+0x64/0x174 [ 2411.342403] scsi_finish_command+0xe0/0x120 [ 2411.346573] scsi_complete+0x8c/0x1c4 [ 2411.350222] blk_complete_reqs+0x5c/0x74 [ 2411.354131] blk_done_softirq+0x38/0x4c [ 2411.357954] __do_softirq+0x168/0x418 [ 2411.361603] __irq_exit_rcu+0x120/0x170 [ 2411.365426] irq_exit_rcu+0x18/0x24 [ 2411.368901] el1_interrupt+0x38/0x70 [ 2411.372464] el1h_64_irq_handler+0x18/0x24 [ 2411.376547] el1h_64_irq+0x68/0x6c [ 2411.379936] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2411.384714] __wake_up_common_lock+0x84/0xac [ 2411.388971] __wake_up+0x20/0x30 [ 2411.392186] __sbq_wake_up+0x13c/0x1f0 [ 2411.395922] sbitmap_queue_clear+0x74/0xc0 [ 2411.400007] blk_mq_put_tag+0x44/0x70 [ 2411.403656] __blk_mq_free_request+0x88/0xf0 [ 2411.407913] blk_mq_free_request+0xc0/0x160 [ 2411.412083] bfq_insert_request.isra.0+0x204/0x270 [ 2411.416862] bfq_insert_requests+0x70/0x90 [ 2411.420946] blk_mq_sched_insert_requests+0xac/0x250 [ 2411.425898] blk_mq_flush_plug_list+0x110/0x2cc [ 2411.430415] __blk_flush_plug+0x100/0x140 [ 2411.434412] blk_finish_plug+0x40/0x25c [ 2411.438235] __iomap_dio_rw+0x1b4/0x500 [ 2411.442058] iomap_dio_rw+0x18/0x50 [ 2411.445534] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2411.450885] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2411.455627] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2411.460672] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2411.465802] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2411.470932] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2411.476317] process_one_work+0x1ec/0x470 [ 2411.480314] worker_thread+0x74/0x410 [ 2411.483963] kthread+0xec/0x100 [ 2411.487092] ret_from_fork+0x10/0x20 [ 2423.208602] watchdog: BUG: soft lockup - CPU#68 stuck for 1862s! [swapper/68:0] [ 2423.215897] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2423.215914] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2423.303701] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2423.311517] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2423.320808] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2423.327755] pc : __sbq_wake_up+0x7c/0x1f0 [ 2423.331752] lr : sbitmap_queue_clear+0x74/0xc0 [ 2423.336182] sp : ffff800008843900 [ 2423.339483] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2423.346605] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2423.353728] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2423.360849] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2423.367971] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2423.375093] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2423.382215] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2423.389337] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2423.396459] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2423.403580] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 2423.410702] Call trace: [ 2423.413135] __sbq_wake_up+0x7c/0x1f0 [ 2423.416785] blk_mq_put_tag+0x44/0x70 [ 2423.420434] __blk_mq_free_request+0x88/0xf0 [ 2423.424691] blk_mq_free_request+0xc0/0x160 [ 2423.428861] __blk_mq_end_request+0x78/0x150 [ 2423.433117] scsi_end_request+0x114/0x260 [ 2423.437114] scsi_io_completion+0x64/0x174 [ 2423.441197] scsi_finish_command+0xe0/0x120 [ 2423.445368] scsi_complete+0x8c/0x1c4 [ 2423.449017] blk_complete_reqs+0x5c/0x74 [ 2423.452926] blk_done_softirq+0x38/0x4c [ 2423.456749] __do_softirq+0x168/0x418 [ 2423.460398] __irq_exit_rcu+0x120/0x170 [ 2423.464220] irq_exit_rcu+0x18/0x24 [ 2423.467696] el1_interrupt+0x38/0x70 [ 2423.471258] el1h_64_irq_handler+0x18/0x24 [ 2423.475341] el1h_64_irq+0x68/0x6c [ 2423.478729] cpuidle_enter_state+0xc4/0x4e0 [ 2423.482899] cpuidle_enter+0x40/0x60 [ 2423.486461] cpuidle_idle_call+0x118/0x1b0 [ 2423.490545] do_idle+0xac/0x100 [ 2423.493674] cpu_startup_entry+0x30/0x34 [ 2423.497583] secondary_start_kernel+0xd8/0xf0 [ 2423.501927] __secondary_switched+0xb0/0xb4 [ 2435.098521] watchdog: BUG: soft lockup - CPU#8 stuck for 1873s! [kworker/8:12:19399] [ 2435.106250] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2435.106267] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2435.194055] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2435.202305] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2435.211596] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2435.217763] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2435.224711] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2435.228795] lr : sbitmap_queue_clear+0x74/0xc0 [ 2435.233225] sp : ffff80000d79b280 [ 2435.236527] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2435.243649] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2435.250772] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2435.257894] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2435.265017] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2435.272139] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2435.279260] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2435.286382] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2435.293504] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2435.300626] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2435.307748] Call trace: [ 2435.310180] __sbq_wake_up+0x1c8/0x1f0 [ 2435.313916] sbitmap_queue_clear+0x74/0xc0 [ 2435.318000] blk_mq_put_tag+0x44/0x70 [ 2435.321650] __blk_mq_free_request+0x88/0xf0 [ 2435.325906] blk_mq_free_request+0xc0/0x160 [ 2435.330076] __blk_mq_end_request+0x78/0x150 [ 2435.334333] scsi_end_request+0x114/0x260 [ 2435.338330] scsi_io_completion+0x64/0x174 [ 2435.342413] scsi_finish_command+0xe0/0x120 [ 2435.346583] scsi_complete+0x8c/0x1c4 [ 2435.350232] blk_complete_reqs+0x5c/0x74 [ 2435.354142] blk_done_softirq+0x38/0x4c [ 2435.357964] __do_softirq+0x168/0x418 [ 2435.361613] __irq_exit_rcu+0x120/0x170 [ 2435.365436] irq_exit_rcu+0x18/0x24 [ 2435.368911] el1_interrupt+0x38/0x70 [ 2435.372474] el1h_64_irq_handler+0x18/0x24 [ 2435.376557] el1h_64_irq+0x68/0x6c [ 2435.379945] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2435.384723] __wake_up_common_lock+0x84/0xac [ 2435.388980] __wake_up+0x20/0x30 [ 2435.392196] __sbq_wake_up+0x13c/0x1f0 [ 2435.395932] sbitmap_queue_clear+0x74/0xc0 [ 2435.400015] blk_mq_put_tag+0x44/0x70 [ 2435.403665] __blk_mq_free_request+0x88/0xf0 [ 2435.407922] blk_mq_free_request+0xc0/0x160 [ 2435.412092] bfq_insert_request.isra.0+0x204/0x270 [ 2435.416870] bfq_insert_requests+0x70/0x90 [ 2435.420954] blk_mq_sched_insert_requests+0xac/0x250 [ 2435.425906] blk_mq_flush_plug_list+0x110/0x2cc [ 2435.430424] __blk_flush_plug+0x100/0x140 [ 2435.434420] blk_finish_plug+0x40/0x25c [ 2435.438243] __iomap_dio_rw+0x1b4/0x500 [ 2435.442066] iomap_dio_rw+0x18/0x50 [ 2435.445542] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2435.450892] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2435.455635] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2435.460679] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2435.465809] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2435.470939] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2435.476326] process_one_work+0x1ec/0x470 [ 2435.480322] worker_thread+0x74/0x410 [ 2435.483971] kthread+0xec/0x100 [ 2435.487100] ret_from_fork+0x10/0x20 [ 2447.208439] watchdog: BUG: soft lockup - CPU#68 stuck for 1885s! [swapper/68:0] [ 2447.215735] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2447.215751] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2447.303539] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2447.311354] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2447.320645] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2447.327593] pc : __sbq_wake_up+0x58/0x1f0 [ 2447.331590] lr : sbitmap_queue_clear+0x74/0xc0 [ 2447.336020] sp : ffff8000088438b0 [ 2447.339321] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2447.346444] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2447.353566] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2447.360688] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2447.367810] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2447.374932] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2447.382055] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2447.389177] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2447.396298] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2447.403420] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2447.410542] Call trace: [ 2447.412975] __sbq_wake_up+0x58/0x1f0 [ 2447.416624] sbitmap_queue_clear+0x74/0xc0 [ 2447.420708] blk_mq_put_tag+0x44/0x70 [ 2447.424357] __blk_mq_free_request+0x88/0xf0 [ 2447.428614] blk_mq_free_request+0xc0/0x160 [ 2447.432785] __blk_mq_end_request+0x78/0x150 [ 2447.437042] scsi_end_request+0x114/0x260 [ 2447.441038] scsi_io_completion+0x64/0x174 [ 2447.445122] scsi_finish_command+0xe0/0x120 [ 2447.449292] scsi_complete+0x8c/0x1c4 [ 2447.452941] blk_complete_reqs+0x5c/0x74 [ 2447.456851] blk_done_softirq+0x38/0x4c [ 2447.460673] __do_softirq+0x168/0x418 [ 2447.464322] __irq_exit_rcu+0x120/0x170 [ 2447.468145] irq_exit_rcu+0x18/0x24 [ 2447.471621] el1_interrupt+0x38/0x70 [ 2447.475183] el1h_64_irq_handler+0x18/0x24 [ 2447.479266] el1h_64_irq+0x68/0x6c [ 2447.482655] cpuidle_enter_state+0xc4/0x4e0 [ 2447.486824] cpuidle_enter+0x40/0x60 [ 2447.490387] cpuidle_idle_call+0x118/0x1b0 [ 2447.494470] do_idle+0xac/0x100 [ 2447.497599] cpu_startup_entry+0x30/0x34 [ 2447.501508] secondary_start_kernel+0xd8/0xf0 [ 2447.505853] __secondary_switched+0xb0/0xb4 [ 2459.098360] watchdog: BUG: soft lockup - CPU#8 stuck for 1896s! [kworker/8:12:19399] [ 2459.106090] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2459.106107] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2459.193896] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2459.202145] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2459.211436] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2459.217604] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2459.224551] pc : __sbq_wake_up+0x58/0x1f0 [ 2459.228548] lr : sbitmap_queue_clear+0x74/0xc0 [ 2459.232979] sp : ffff80000d79b280 [ 2459.236280] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2459.243403] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2459.250525] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2459.257646] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2459.264769] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2459.271891] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2459.279013] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2459.286135] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2459.293257] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2459.300379] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2459.307501] Call trace: [ 2459.309934] __sbq_wake_up+0x58/0x1f0 [ 2459.313583] sbitmap_queue_clear+0x74/0xc0 [ 2459.317667] blk_mq_put_tag+0x44/0x70 [ 2459.321317] __blk_mq_free_request+0x88/0xf0 [ 2459.325573] blk_mq_free_request+0xc0/0x160 [ 2459.329743] __blk_mq_end_request+0x78/0x150 [ 2459.334000] scsi_end_request+0x114/0x260 [ 2459.337997] scsi_io_completion+0x64/0x174 [ 2459.342080] scsi_finish_command+0xe0/0x120 [ 2459.346250] scsi_complete+0x8c/0x1c4 [ 2459.349899] blk_complete_reqs+0x5c/0x74 [ 2459.353809] blk_done_softirq+0x38/0x4c [ 2459.357631] __do_softirq+0x168/0x418 [ 2459.361280] __irq_exit_rcu+0x120/0x170 [ 2459.365103] irq_exit_rcu+0x18/0x24 [ 2459.368579] el1_interrupt+0x38/0x70 [ 2459.372141] el1h_64_irq_handler+0x18/0x24 [ 2459.376224] el1h_64_irq+0x68/0x6c [ 2459.379613] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2459.384390] __wake_up_common_lock+0x84/0xac [ 2459.388648] __wake_up+0x20/0x30 [ 2459.391863] __sbq_wake_up+0x13c/0x1f0 [ 2459.395599] sbitmap_queue_clear+0x74/0xc0 [ 2459.399682] blk_mq_put_tag+0x44/0x70 [ 2459.403332] __blk_mq_free_request+0x88/0xf0 [ 2459.407588] blk_mq_free_request+0xc0/0x160 [ 2459.411758] bfq_insert_request.isra.0+0x204/0x270 [ 2459.416536] bfq_insert_requests+0x70/0x90 [ 2459.420620] blk_mq_sched_insert_requests+0xac/0x250 [ 2459.425572] blk_mq_flush_plug_list+0x110/0x2cc [ 2459.430089] __blk_flush_plug+0x100/0x140 [ 2459.434086] blk_finish_plug+0x40/0x25c [ 2459.437909] __iomap_dio_rw+0x1b4/0x500 [ 2459.441732] iomap_dio_rw+0x18/0x50 [ 2459.445207] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2459.450558] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2459.455300] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2459.460344] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2459.465474] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2459.470604] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2459.475990] process_one_work+0x1ec/0x470 [ 2459.479987] worker_thread+0x74/0x410 [ 2459.483635] kthread+0xec/0x100 [ 2459.486764] ret_from_fork+0x10/0x20 [ 2468.348299] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2468.354031] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=101382 [ 2468.363153] (t=204347 jiffies g=36445 q=59803 ncpus=128) [ 2468.368538] Task dump for CPU 8: [ 2468.371753] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2468.381652] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2468.387820] Call trace: [ 2468.390253] dump_backtrace+0xe8/0x130 [ 2468.393989] show_stack+0x20/0x60 [ 2468.397291] sched_show_task+0x160/0x190 [ 2468.401201] dump_cpu_task+0x4c/0x78 [ 2468.404764] rcu_dump_cpu_stacks+0xc4/0x150 [ 2468.408935] print_cpu_stall+0x180/0x210 [ 2468.412844] check_cpu_stall+0x20c/0x2b4 [ 2468.416754] rcu_pending+0x4c/0x190 [ 2468.420229] rcu_sched_clock_irq+0x80/0x204 [ 2468.424400] update_process_times+0x7c/0xc0 [ 2468.428569] tick_sched_handle+0x38/0x74 [ 2468.432479] tick_sched_timer+0x60/0xc0 [ 2468.436303] __hrtimer_run_queues+0x170/0x300 [ 2468.440646] hrtimer_interrupt+0xfc/0x260 [ 2468.444643] arch_timer_handler_phys+0x34/0x50 [ 2468.449073] handle_percpu_devid_irq+0xac/0x240 [ 2468.453591] generic_handle_domain_irq+0x34/0x50 [ 2468.458196] gic_handle_irq+0x5c/0xe0 [ 2468.461845] call_on_irq_stack+0x2c/0x38 [ 2468.465755] do_interrupt_handler+0xdc/0xe0 [ 2468.469926] el1_interrupt+0x34/0x70 [ 2468.473488] el1h_64_irq_handler+0x18/0x24 [ 2468.477571] el1h_64_irq+0x68/0x6c [ 2468.480960] __sbq_wake_up+0x40/0x1f0 [ 2468.484609] sbitmap_queue_clear+0x74/0xc0 [ 2468.488693] blk_mq_put_tag+0x44/0x70 [ 2468.492343] __blk_mq_free_request+0x88/0xf0 [ 2468.496599] blk_mq_free_request+0xc0/0x160 [ 2468.500769] __blk_mq_end_request+0x78/0x150 [ 2468.505026] scsi_end_request+0x114/0x260 [ 2468.509022] scsi_io_completion+0x64/0x174 [ 2468.513106] scsi_finish_command+0xe0/0x120 [ 2468.517276] scsi_complete+0x8c/0x1c4 [ 2468.520925] blk_complete_reqs+0x5c/0x74 [ 2468.524835] blk_done_softirq+0x38/0x4c [ 2468.528657] __do_softirq+0x168/0x418 [ 2468.532307] __irq_exit_rcu+0x120/0x170 [ 2468.536130] irq_exit_rcu+0x18/0x24 [ 2468.539605] el1_interrupt+0x38/0x70 [ 2468.543167] el1h_64_irq_handler+0x18/0x24 [ 2468.547251] el1h_64_irq+0x68/0x6c [ 2468.550639] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2468.555417] __wake_up_common_lock+0x84/0xac [ 2468.559675] __wake_up+0x20/0x30 [ 2468.562891] __sbq_wake_up+0x13c/0x1f0 [ 2468.566627] sbitmap_queue_clear+0x74/0xc0 [ 2468.570711] blk_mq_put_tag+0x44/0x70 [ 2468.574361] __blk_mq_free_request+0x88/0xf0 [ 2468.578618] blk_mq_free_request+0xc0/0x160 [ 2468.582787] bfq_insert_request.isra.0+0x204/0x270 [ 2468.587566] bfq_insert_requests+0x70/0x90 [ 2468.591649] blk_mq_sched_insert_requests+0xac/0x250 [ 2468.596601] blk_mq_flush_plug_list+0x110/0x2cc [ 2468.601119] __blk_flush_plug+0x100/0x140 [ 2468.605115] blk_finish_plug+0x40/0x25c [ 2468.608938] __iomap_dio_rw+0x1b4/0x500 [ 2468.612760] iomap_dio_rw+0x18/0x50 [ 2468.616236] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2468.621587] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2468.626330] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2468.631374] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2468.636504] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2468.641634] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2468.647020] process_one_work+0x1ec/0x470 [ 2468.651016] worker_thread+0x74/0x410 [ 2468.654665] kthread+0xec/0x100 [ 2468.657794] ret_from_fork+0x10/0x20 [ 2468.661357] Task dump for CPU 68: [ 2468.664658] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2468.674559] Call trace: [ 2468.676992] __switch_to+0xd4/0x100 [ 2468.680468] 0x0 [ 2468.838305] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 201789 jiffies s: 2093 root: 0x11/. [ 2468.850044] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2468.859252] Task dump for CPU 8: [ 2468.862467] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2468.872369] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2468.878539] Call trace: [ 2468.880972] __switch_to+0xd4/0x100 [ 2468.884448] 0xffff6c0200000080 [ 2468.887576] Task dump for CPU 68: [ 2468.890879] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2468.900781] Call trace: [ 2468.903214] __switch_to+0xd4/0x100 [ 2468.906690] 0x0 [ 2471.208279] watchdog: BUG: soft lockup - CPU#68 stuck for 1907s! [swapper/68:0] [ 2471.215575] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2471.215591] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2471.303379] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2471.311195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2471.320486] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2471.327433] pc : __sbq_wake_up+0x58/0x1f0 [ 2471.331430] lr : sbitmap_queue_clear+0x74/0xc0 [ 2471.335861] sp : ffff8000088438b0 [ 2471.339162] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2471.346284] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2471.353406] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2471.360528] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2471.367650] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2471.374772] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2471.381894] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2471.389016] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2471.396137] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2471.403259] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2471.410381] Call trace: [ 2471.412814] __sbq_wake_up+0x58/0x1f0 [ 2471.416463] sbitmap_queue_clear+0x74/0xc0 [ 2471.420546] blk_mq_put_tag+0x44/0x70 [ 2471.424196] __blk_mq_free_request+0x88/0xf0 [ 2471.428453] blk_mq_free_request+0xc0/0x160 [ 2471.432623] __blk_mq_end_request+0x78/0x150 [ 2471.436879] scsi_end_request+0x114/0x260 [ 2471.440876] scsi_io_completion+0x64/0x174 [ 2471.444959] scsi_finish_command+0xe0/0x120 [ 2471.449129] scsi_complete+0x8c/0x1c4 [ 2471.452778] blk_complete_reqs+0x5c/0x74 [ 2471.456688] blk_done_softirq+0x38/0x4c [ 2471.460510] __do_softirq+0x168/0x418 [ 2471.464159] __irq_exit_rcu+0x120/0x170 [ 2471.467982] irq_exit_rcu+0x18/0x24 [ 2471.471457] el1_interrupt+0x38/0x70 [ 2471.475020] el1h_64_irq_handler+0x18/0x24 [ 2471.479103] el1h_64_irq+0x68/0x6c [ 2471.482491] cpuidle_enter_state+0xc4/0x4e0 [ 2471.486661] cpuidle_enter+0x40/0x60 [ 2471.490223] cpuidle_idle_call+0x118/0x1b0 [ 2471.494307] do_idle+0xac/0x100 [ 2471.497436] cpu_startup_entry+0x30/0x34 [ 2471.501345] secondary_start_kernel+0xd8/0xf0 [ 2471.505689] __secondary_switched+0xb0/0xb4 [ 2495.098120] watchdog: BUG: soft lockup - CPU#8 stuck for 1929s! [kworker/8:12:19399] [ 2495.105850] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2495.105868] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2495.193656] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2495.201906] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2495.208119] watchdog: BUG: soft lockup - CPU#68 stuck for 1929s! [swapper/68:0] [ 2495.211197] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2495.218491] Modules linked in: nvme_loop [ 2495.224657] [ 2495.224658] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2495.228566] nvme_fabrics [ 2495.230044] pc : __sbq_wake_up+0x58/0x1f0 [ 2495.236990] nvmet [ 2495.239597] lr : sbitmap_queue_clear+0x74/0xc0 [ 2495.243592] nls_utf8 [ 2495.245591] sp : ffff80000d79b280 [ 2495.250020] hfsplus [ 2495.252279] x29: ffff80000d79b280 [ 2495.255580] hfs [ 2495.257752] x28: ffffd5becb6aec50 [ 2495.261053] isofs [ 2495.262878] x27: ffffd5beca4f7000 [ 2495.266265] nvme_core [ 2495.268264] [ 2495.271652] nvme_common [ 2495.273998] x26: ffffd5becaf68a98 [ 2495.275476] loop [ 2495.277995] x25: ffffd5becb6a60c0 [ 2495.281296] tls [ 2495.283208] x24: fffffbffef180dc0 [ 2495.286595] rfkill [ 2495.288420] [ 2495.291808] sunrpc [ 2495.293893] x23: ffff07ff81d381b0 [ 2495.295371] vfat [ 2495.297457] x22: ffff80000d79b30c [ 2495.300758] fat [ 2495.302669] x21: ffff07ff84e93c00 [ 2495.306057] acpi_ipmi [ 2495.307882] [ 2495.311270] ipmi_ssif [ 2495.313616] x20: 00000000000001c0 [ 2495.315093] cppc_cpufreq [ 2495.317439] x19: ffff07ff91ccdb10 [ 2495.320740] ipmi_devintf [ 2495.323346] x18: ffffffffffffffff [ 2495.326734] igb [ 2495.329340] [ 2495.332730] ipmi_msghandler [ 2495.334555] x17: ffff32506c412000 [ 2495.336032] arm_cmn [ 2495.338899] x16: ffff800008044000 [ 2495.342200] arm_dmc620_pmu [ 2495.344372] x15: 0000000000004000 [ 2495.347760] arm_spe_pmu [ 2495.350540] [ 2495.353927] arm_dsu_pmu [ 2495.356447] x14: 010320bb28ec0000 [ 2495.357925] acpi_tad [ 2495.360444] x13: 0000000000000087 [ 2495.363745] fuse [ 2495.366004] x12: 071c71c71c71c71c [ 2495.369392] zram [ 2495.371304] [ 2495.374692] xfs [ 2495.376604] x11: 0000000000000040 [ 2495.378081] crct10dif_ce [ 2495.379907] x10: ffff07ff801a9c70 [ 2495.383207] polyval_ce [ 2495.385814] x9 : ffffd5bec9545fd8 [ 2495.389201] ast [ 2495.391634] [ 2495.395022] polyval_generic [ 2495.396847] x8 : ffff07ff80404b70 [ 2495.398324] drm_vram_helper [ 2495.401191] x7 : ffff07ff84e93dc8 [ 2495.404492] ghash_ce [ 2495.407359] x6 : 0000000000000008 [ 2495.410746] drm_ttm_helper [ 2495.413005] [ 2495.416393] sbsa_gwdt [ 2495.419173] x5 : ffff80000d043790 [ 2495.420651] ttm [ 2495.422997] x4 : ffff07ff84e93dc0 [ 2495.426297] xgene_hwmon [ 2495.428122] x3 : 0000000000000008 [ 2495.431510] Unloaded tainted modules: [ 2495.434030] [ 2495.437417] nvmet():1 [ 2495.441065] x2 : 0000000000000007 [ 2495.442543] nvme_fabrics():1 [ 2495.444889] x1 : ffff80000d79b30c [ 2495.448190] nvme_loop():1 [ 2495.451143] x0 : ffff07ff84e93dd0 [ 2495.454531] nvmet():1 [ 2495.457224] [ 2495.460612] nvme_fabrics():1 [ 2495.462958] Call trace: [ 2495.464435] nvme_loop():1 [ 2495.467389] __sbq_wake_up+0x58/0x1f0 [ 2495.469822] nvmet():1 [ 2495.472515] sbitmap_queue_clear+0x74/0xc0 [ 2495.476165] nvme_fabrics():1 [ 2495.478511] blk_mq_put_tag+0x44/0x70 [ 2495.482593] nvme_loop():1 [ 2495.485546] __blk_mq_free_request+0x88/0xf0 [ 2495.489194] nvmet():1 [ 2495.491887] blk_mq_free_request+0xc0/0x160 [ 2495.496143] nvme_fabrics():1 [ 2495.498489] __blk_mq_end_request+0x78/0x150 [ 2495.502658] nvme_loop():1 [ 2495.505611] scsi_end_request+0x114/0x260 [ 2495.509867] nvmet():1 [ 2495.512560] scsi_io_completion+0x64/0x174 [ 2495.516555] nvme_fabrics():1 [ 2495.518901] scsi_finish_command+0xe0/0x120 [ 2495.522983] nvme_loop():1 [ 2495.525937] scsi_complete+0x8c/0x1c4 [ 2495.530106] nvmet():1 [ 2495.532799] blk_complete_reqs+0x5c/0x74 [ 2495.536447] nvme_fabrics():1 [ 2495.538793] blk_done_softirq+0x38/0x4c [ 2495.542701] nvme_loop():1 [ 2495.545655] __do_softirq+0x168/0x418 [ 2495.549476] nvmet():1 [ 2495.552169] __irq_exit_rcu+0x120/0x170 [ 2495.555817] nvme_fabrics():1 [ 2495.558163] irq_exit_rcu+0x18/0x24 [ 2495.561985] nvme_loop():1 [ 2495.564938] el1_interrupt+0x38/0x70 [ 2495.568413] nvmet():1 [ 2495.571106] el1h_64_irq_handler+0x18/0x24 [ 2495.574667] nvme_fabrics():1 [ 2495.577013] el1h_64_irq+0x68/0x6c [ 2495.581095] nvme_loop():1 [ 2495.584049] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2495.587436] nvmet():1 [ 2495.590129] __wake_up_common_lock+0x84/0xac [ 2495.594906] nvme_fabrics():1 [ 2495.597252] __wake_up+0x20/0x30 [ 2495.601507] nvme_loop():1 [ 2495.604461] __sbq_wake_up+0x13c/0x1f0 [ 2495.607675] nvmet():1 [ 2495.610368] sbitmap_queue_clear+0x74/0xc0 [ 2495.614103] nvme_fabrics():1 [ 2495.616449] blk_mq_put_tag+0x44/0x70 [ 2495.620531] nvme_loop():1 [ 2495.623484] __blk_mq_free_request+0x88/0xf0 [ 2495.627132] null_blk():1 [ 2495.629825] blk_mq_free_request+0xc0/0x160 [ 2495.634081] null_blk():1 [ 2495.636687] bfq_insert_request.isra.0+0x204/0x270 [ 2495.640856] null_blk():1 [ 2495.643462] bfq_insert_requests+0x70/0x90 [ 2495.648239] null_blk():1 [ 2495.650846] blk_mq_sched_insert_requests+0xac/0x250 [ 2495.654928] null_blk():1 [ 2495.657534] blk_mq_flush_plug_list+0x110/0x2cc [ 2495.662484] null_blk():1 [ 2495.665090] __blk_flush_plug+0x100/0x140 [ 2495.669606] null_blk():1 [ 2495.672213] blk_finish_plug+0x40/0x25c [ 2495.676208] scsi_debug():1 [ 2495.678814] __iomap_dio_rw+0x1b4/0x500 [ 2495.682636] scsi_debug():1 [ 2495.685416] iomap_dio_rw+0x18/0x50 [ 2495.689238] ipmi_si():1 [ 2495.692018] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2495.695493] ipmi_si():1 [ 2495.698012] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2495.703310] [last unloaded: nvmet] [ 2495.705829] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2495.710519] [ 2495.713994] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2495.719031] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2495.720510] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2495.725634] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2495.733448] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2495.738572] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2495.747863] process_one_work+0x1ec/0x470 [ 2495.753247] pc : __sbq_wake_up+0x84/0x1f0 [ 2495.760194] worker_thread+0x74/0x410 [ 2495.764189] lr : sbitmap_queue_clear+0x74/0xc0 [ 2495.768184] kthread+0xec/0x100 [ 2495.771832] sp : ffff8000088438b0 [ 2495.776261] ret_from_fork+0x10/0x20 [ 2495.779388] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2495.793372] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2495.800494] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2495.807616] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2495.814738] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2495.821860] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2495.828982] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2495.836104] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2495.843226] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2495.850347] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2495.857469] Call trace: [ 2495.859902] __sbq_wake_up+0x84/0x1f0 [ 2495.863551] sbitmap_queue_clear+0x74/0xc0 [ 2495.867634] blk_mq_put_tag+0x44/0x70 [ 2495.871284] __blk_mq_free_request+0x88/0xf0 [ 2495.875540] blk_mq_free_request+0xc0/0x160 [ 2495.879710] __blk_mq_end_request+0x78/0x150 [ 2495.883967] scsi_end_request+0x114/0x260 [ 2495.887963] scsi_io_completion+0x64/0x174 [ 2495.892046] scsi_finish_command+0xe0/0x120 [ 2495.896217] scsi_complete+0x8c/0x1c4 [ 2495.899866] blk_complete_reqs+0x5c/0x74 [ 2495.903775] blk_done_softirq+0x38/0x4c [ 2495.907598] __do_softirq+0x168/0x418 [ 2495.911247] __irq_exit_rcu+0x120/0x170 [ 2495.915070] irq_exit_rcu+0x18/0x24 [ 2495.918545] el1_interrupt+0x38/0x70 [ 2495.922108] el1h_64_irq_handler+0x18/0x24 [ 2495.926191] el1h_64_irq+0x68/0x6c [ 2495.929579] cpuidle_enter_state+0xc4/0x4e0 [ 2495.933749] cpuidle_enter+0x40/0x60 [ 2495.937312] cpuidle_idle_call+0x118/0x1b0 [ 2495.941397] do_idle+0xac/0x100 [ 2495.944526] cpu_startup_entry+0x30/0x34 [ 2495.948435] secondary_start_kernel+0xd8/0xf0 [ 2495.952779] __secondary_switched+0xb0/0xb4 [ 2519.097960] watchdog: BUG: soft lockup - CPU#8 stuck for 1952s! [kworker/8:12:19399] [ 2519.105690] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2519.105707] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2519.193495] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2519.201744] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2519.207959] watchdog: BUG: soft lockup - CPU#68 stuck for 1952s! [swapper/68:0] [ 2519.211035] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2519.218329] Modules linked in: [ 2519.224495] [ 2519.224496] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2519.227536] nvme_loop [ 2519.229014] pc : __sbq_wake_up+0x4/0x1f0 [ 2519.235960] nvme_fabrics [ 2519.238307] lr : sbitmap_queue_clear+0x74/0xc0 [ 2519.242217] nvmet [ 2519.244823] sp : ffff80000d79b2d0 [ 2519.249253] nls_utf8 [ 2519.251253] x29: ffff80000d79b2d0 [ 2519.254553] hfsplus [ 2519.256812] x28: ffffd5becb6aec50 [ 2519.260113] hfs [ 2519.262286] x27: ffffd5beca4f7000 [ 2519.265673] isofs [ 2519.267498] [ 2519.270886] nvme_core [ 2519.272884] x26: ffffd5becaf68a98 [ 2519.274362] nvme_common [ 2519.276708] x25: ffffd5becb6a60c0 [ 2519.280009] loop [ 2519.282529] x24: fffffbffef180dc0 [ 2519.285916] tls [ 2519.287828] [ 2519.291216] rfkill [ 2519.293041] x23: ffff07ff81d381b0 [ 2519.294518] sunrpc [ 2519.296604] x22: 0000000000000024 [ 2519.299905] vfat [ 2519.301990] x21: 0000000000000008 [ 2519.305378] fat [ 2519.307290] [ 2519.310677] acpi_ipmi [ 2519.312502] x20: 0000000000000024 [ 2519.313980] ipmi_ssif [ 2519.316326] x19: ffff07ff91ccdb10 [ 2519.319627] cppc_cpufreq [ 2519.321973] x18: ffffffffffffffff [ 2519.325360] ipmi_devintf [ 2519.327967] [ 2519.331354] igb [ 2519.333961] x17: ffff32506c412000 [ 2519.335438] ipmi_msghandler [ 2519.337263] x16: ffff800008044000 [ 2519.340564] arm_cmn [ 2519.343431] x15: 0000000000004000 [ 2519.346819] arm_dmc620_pmu [ 2519.348991] [ 2519.352378] arm_spe_pmu [ 2519.355158] x14: 010320bb28ec0000 [ 2519.356636] arm_dsu_pmu [ 2519.359156] x13: 0000000000000087 [ 2519.362457] acpi_tad [ 2519.364976] x12: 071c71c71c71c71c [ 2519.368364] fuse [ 2519.370623] [ 2519.374010] zram [ 2519.375922] x11: 0000000000000040 [ 2519.377401] xfs [ 2519.379313] x10: ffff07ff801a9c70 [ 2519.382613] crct10dif_ce [ 2519.384438] x9 : ffffd5bec9545fd8 [ 2519.387826] polyval_ce [ 2519.390432] [ 2519.393820] ast [ 2519.396253] x8 : ffff07ff80404b70 [ 2519.397730] polyval_generic [ 2519.399555] x7 : ffff07ff84e93dc8 [ 2519.402856] drm_vram_helper [ 2519.405723] x6 : 0000000000000008 [ 2519.409111] ghash_ce [ 2519.411977] [ 2519.415365] drm_ttm_helper [ 2519.417624] x5 : ffff80000d043790 [ 2519.419102] sbsa_gwdt [ 2519.421882] x4 : ffff07ff84e93dc0 [ 2519.425183] ttm [ 2519.427529] x3 : 0000000000000008 [ 2519.430917] xgene_hwmon [ 2519.432742] [ 2519.436129] Unloaded tainted modules: [ 2519.438649] x2 : 0000000000000007 [ 2519.440127] nvmet():1 [ 2519.443775] x1 : ffff80000d79b30c [ 2519.447075] nvme_fabrics():1 [ 2519.449421] x0 : ffff07ff91ccdb10 [ 2519.452810] nvme_loop():1 [ 2519.455764] [ 2519.459151] nvmet():1 [ 2519.461844] Call trace: [ 2519.463322] nvme_fabrics():1 [ 2519.465669] __sbq_wake_up+0x4/0x1f0 [ 2519.468101] nvme_loop():1 [ 2519.471055] blk_mq_put_tag+0x44/0x70 [ 2519.474616] nvmet():1 [ 2519.477310] __blk_mq_free_request+0x88/0xf0 [ 2519.480958] nvme_fabrics():1 [ 2519.483304] blk_mq_free_request+0xc0/0x160 [ 2519.487559] nvme_loop():1 [ 2519.490513] __blk_mq_end_request+0x78/0x150 [ 2519.494681] nvmet():1 [ 2519.497375] scsi_end_request+0x114/0x260 [ 2519.501630] nvme_fabrics():1 [ 2519.503977] scsi_io_completion+0x64/0x174 [ 2519.507972] nvme_loop():1 [ 2519.510926] scsi_finish_command+0xe0/0x120 [ 2519.515008] nvmet():1 [ 2519.517701] scsi_complete+0x8c/0x1c4 [ 2519.521870] nvme_fabrics():1 [ 2519.524216] blk_complete_reqs+0x5c/0x74 [ 2519.527864] nvme_loop():1 [ 2519.530818] blk_done_softirq+0x38/0x4c [ 2519.534728] nvmet():1 [ 2519.537422] __do_softirq+0x168/0x418 [ 2519.541244] nvme_fabrics():1 [ 2519.543590] __irq_exit_rcu+0x120/0x170 [ 2519.547238] nvme_loop():1 [ 2519.550191] irq_exit_rcu+0x18/0x24 [ 2519.554013] nvmet():1 [ 2519.556706] el1_interrupt+0x38/0x70 [ 2519.560180] nvme_fabrics():1 [ 2519.562526] el1h_64_irq_handler+0x18/0x24 [ 2519.566088] nvme_loop():1 [ 2519.569041] el1h_64_irq+0x68/0x6c [ 2519.573123] nvmet():1 [ 2519.575817] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2519.579205] nvme_fabrics():1 [ 2519.581551] __wake_up_common_lock+0x84/0xac [ 2519.586328] nvme_loop():1 [ 2519.589281] __wake_up+0x20/0x30 [ 2519.593537] nvmet():1 [ 2519.596230] __sbq_wake_up+0x13c/0x1f0 [ 2519.599444] nvme_fabrics():1 [ 2519.601790] sbitmap_queue_clear+0x74/0xc0 [ 2519.605525] nvme_loop():1 [ 2519.608478] blk_mq_put_tag+0x44/0x70 [ 2519.612560] nvmet():1 [ 2519.615253] __blk_mq_free_request+0x88/0xf0 [ 2519.618901] nvme_fabrics():1 [ 2519.621247] blk_mq_free_request+0xc0/0x160 [ 2519.625503] nvme_loop():1 [ 2519.628456] bfq_insert_request.isra.0+0x204/0x270 [ 2519.632625] null_blk():1 [ 2519.635319] bfq_insert_requests+0x70/0x90 [ 2519.640097] null_blk():1 [ 2519.642703] blk_mq_sched_insert_requests+0xac/0x250 [ 2519.646785] null_blk():1 [ 2519.649391] blk_mq_flush_plug_list+0x110/0x2cc [ 2519.654341] null_blk():1 [ 2519.656948] __blk_flush_plug+0x100/0x140 [ 2519.661464] null_blk():1 [ 2519.664070] blk_finish_plug+0x40/0x25c [ 2519.668065] null_blk():1 [ 2519.670671] __iomap_dio_rw+0x1b4/0x500 [ 2519.674493] null_blk():1 [ 2519.677099] iomap_dio_rw+0x18/0x50 [ 2519.680922] scsi_debug():1 [ 2519.683528] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2519.687002] scsi_debug():1 [ 2519.689782] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2519.695079] ipmi_si():1 [ 2519.697859] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2519.702549] ipmi_si():1 [ 2519.705069] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2519.710105] [last unloaded: nvmet] [ 2519.712625] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2519.717749] [ 2519.721223] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2519.726347] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2519.727825] process_one_work+0x1ec/0x470 [ 2519.733209] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2519.741024] worker_thread+0x74/0x410 [ 2519.745019] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2519.754309] kthread+0xec/0x100 [ 2519.757957] pc : __sbq_wake_up+0x58/0x1f0 [ 2519.764904] ret_from_fork+0x10/0x20 [ 2519.768031] lr : sbitmap_queue_clear+0x74/0xc0 [ 2519.780017] sp : ffff8000088438b0 [ 2519.783318] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2519.790440] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2519.797562] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2519.804684] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2519.811806] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2519.818928] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2519.826050] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2519.833171] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2519.840293] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2519.847415] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2519.854537] Call trace: [ 2519.856970] __sbq_wake_up+0x58/0x1f0 [ 2519.860619] sbitmap_queue_clear+0x74/0xc0 [ 2519.864702] blk_mq_put_tag+0x44/0x70 [ 2519.868352] __blk_mq_free_request+0x88/0xf0 [ 2519.872609] blk_mq_free_request+0xc0/0x160 [ 2519.876778] __blk_mq_end_request+0x78/0x150 [ 2519.881035] scsi_end_request+0x114/0x260 [ 2519.885032] scsi_io_completion+0x64/0x174 [ 2519.889115] scsi_finish_command+0xe0/0x120 [ 2519.893285] scsi_complete+0x8c/0x1c4 [ 2519.896934] blk_complete_reqs+0x5c/0x74 [ 2519.900844] blk_done_softirq+0x38/0x4c [ 2519.904666] __do_softirq+0x168/0x418 [ 2519.908315] __irq_exit_rcu+0x120/0x170 [ 2519.912138] irq_exit_rcu+0x18/0x24 [ 2519.915613] el1_interrupt+0x38/0x70 [ 2519.919176] el1h_64_irq_handler+0x18/0x24 [ 2519.923259] el1h_64_irq+0x68/0x6c [ 2519.926648] cpuidle_enter_state+0xc4/0x4e0 [ 2519.930817] cpuidle_enter+0x40/0x60 [ 2519.934380] cpuidle_idle_call+0x118/0x1b0 [ 2519.938463] do_idle+0xac/0x100 [ 2519.941592] cpu_startup_entry+0x30/0x34 [ 2519.945501] secondary_start_kernel+0xd8/0xf0 [ 2519.949845] __secondary_switched+0xb0/0xb4 [ 2535.397861] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 208445 jiffies s: 2093 root: 0x11/. [ 2535.409600] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2535.418809] Task dump for CPU 8: [ 2535.422023] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2535.431925] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2535.438095] Call trace: [ 2535.440528] __switch_to+0xd4/0x100 [ 2535.444004] 0xffff6c0200000080 [ 2535.447132] Task dump for CPU 68: [ 2535.450435] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2535.460337] Call trace: [ 2535.462770] __switch_to+0xd4/0x100 [ 2535.466247] 0x0 [ 2543.097800] watchdog: BUG: soft lockup - CPU#8 stuck for 1974s! [kworker/8:12:19399] [ 2543.105530] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2543.105547] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2543.193335] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2543.201584] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2543.207799] watchdog: BUG: soft lockup - CPU#68 stuck for 1974s! [swapper/68:0] [ 2543.210875] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2543.218169] Modules linked in: nvme_loop [ 2543.224335] [ 2543.224335] nvme_fabrics [ 2543.228244] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2543.229722] nvmet [ 2543.232330] pc : __sbq_wake_up+0x84/0x1f0 [ 2543.239277] nls_utf8 [ 2543.241276] lr : sbitmap_queue_clear+0x74/0xc0 [ 2543.245271] hfsplus [ 2543.247530] sp : ffff80000d79b280 [ 2543.251960] hfs [ 2543.254132] x29: ffff80000d79b280 [ 2543.257433] isofs [ 2543.259258] x28: ffffd5becb6aec50 [ 2543.262558] nvme_core [ 2543.264557] x27: ffffd5beca4f7000 [ 2543.267945] nvme_common [ 2543.270291] [ 2543.273678] loop [ 2543.276198] x26: ffffd5becaf68a98 [ 2543.277675] tls [ 2543.279587] x25: ffffd5becb6a60c0 [ 2543.282888] rfkill [ 2543.284713] x24: fffffbffef180dc0 [ 2543.288101] sunrpc [ 2543.290186] [ 2543.293574] vfat [ 2543.295660] x23: ffff07ff81d381b0 [ 2543.297138] fat [ 2543.299050] x22: ffff80000d79b30c [ 2543.302351] acpi_ipmi [ 2543.304176] x21: ffff07ff84e93c00 [ 2543.307564] ipmi_ssif [ 2543.309910] [ 2543.313297] cppc_cpufreq [ 2543.315643] x20: 00000000000001c0 [ 2543.317121] ipmi_devintf [ 2543.319727] x19: ffff07ff91ccdb10 [ 2543.323028] igb [ 2543.325634] x18: ffffffffffffffff [ 2543.329022] ipmi_msghandler [ 2543.330847] [ 2543.334235] arm_cmn [ 2543.337101] x17: ffff32506c412000 [ 2543.338579] arm_dmc620_pmu [ 2543.340751] x16: ffff800008044000 [ 2543.344052] arm_spe_pmu [ 2543.346832] x15: 0000000000004000 [ 2543.350220] arm_dsu_pmu [ 2543.352741] [ 2543.356129] acpi_tad [ 2543.358649] x14: 010320bb28ec0000 [ 2543.360126] fuse [ 2543.362385] x13: 0000000000000087 [ 2543.365686] zram [ 2543.367598] x12: 071c71c71c71c71c [ 2543.370986] xfs [ 2543.372898] [ 2543.376285] crct10dif_ce [ 2543.378111] x11: 0000000000000040 [ 2543.379589] polyval_ce [ 2543.382196] x10: ffff07ff801a9c70 [ 2543.385496] ast [ 2543.387929] x9 : ffffd5bec9545fd8 [ 2543.391317] polyval_generic [ 2543.393142] [ 2543.396529] drm_vram_helper [ 2543.399396] x8 : ffff07ff80404b70 [ 2543.400874] ghash_ce [ 2543.403741] x7 : ffff07ff84e93dc8 [ 2543.407041] drm_ttm_helper [ 2543.409300] x6 : 0000000000000008 [ 2543.412688] sbsa_gwdt [ 2543.415468] [ 2543.418856] ttm [ 2543.421201] x5 : ffff80000d043790 [ 2543.422679] xgene_hwmon [ 2543.424504] x4 : ffff07ff84e93dc0 [ 2543.427805] Unloaded tainted modules: [ 2543.430325] x3 : 0000000000000008 [ 2543.433713] nvmet():1 [ 2543.437361] [ 2543.440748] nvme_fabrics():1 [ 2543.443094] x2 : 0000000000000007 [ 2543.444572] nvme_loop():1 [ 2543.447526] x1 : ffff80000d79b30c [ 2543.450826] nvmet():1 [ 2543.453520] x0 : ffff07ff84e93dd0 [ 2543.456907] nvme_fabrics():1 [ 2543.459253] [ 2543.462641] nvme_loop():1 [ 2543.465594] Call trace: [ 2543.467072] nvmet():1 [ 2543.469765] __sbq_wake_up+0x84/0x1f0 [ 2543.472198] nvme_fabrics():1 [ 2543.474544] sbitmap_queue_clear+0x74/0xc0 [ 2543.478192] nvme_loop():1 [ 2543.481145] blk_mq_put_tag+0x44/0x70 [ 2543.485227] nvmet():1 [ 2543.487921] __blk_mq_free_request+0x88/0xf0 [ 2543.491570] nvme_fabrics():1 [ 2543.493916] blk_mq_free_request+0xc0/0x160 [ 2543.498172] nvme_loop():1 [ 2543.501125] __blk_mq_end_request+0x78/0x150 [ 2543.505294] nvmet():1 [ 2543.507987] scsi_end_request+0x114/0x260 [ 2543.512243] nvme_fabrics():1 [ 2543.514589] scsi_io_completion+0x64/0x174 [ 2543.518584] nvme_loop():1 [ 2543.521537] scsi_finish_command+0xe0/0x120 [ 2543.525620] nvmet():1 [ 2543.528314] scsi_complete+0x8c/0x1c4 [ 2543.532483] nvme_fabrics():1 [ 2543.534829] blk_complete_reqs+0x5c/0x74 [ 2543.538477] nvme_loop():1 [ 2543.541431] blk_done_softirq+0x38/0x4c [ 2543.545339] nvmet():1 [ 2543.548033] __do_softirq+0x168/0x418 [ 2543.551854] nvme_fabrics():1 [ 2543.554200] __irq_exit_rcu+0x120/0x170 [ 2543.557848] nvme_loop():1 [ 2543.560802] irq_exit_rcu+0x18/0x24 [ 2543.564624] nvmet():1 [ 2543.567317] el1_interrupt+0x38/0x70 [ 2543.570792] nvme_fabrics():1 [ 2543.573138] el1h_64_irq_handler+0x18/0x24 [ 2543.576699] nvme_loop():1 [ 2543.579652] el1h_64_irq+0x68/0x6c [ 2543.583734] nvmet():1 [ 2543.586428] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2543.589815] nvme_fabrics():1 [ 2543.592161] __wake_up_common_lock+0x84/0xac [ 2543.596938] nvme_loop():1 [ 2543.599891] __wake_up+0x20/0x30 [ 2543.604147] nvmet():1 [ 2543.606840] __sbq_wake_up+0x13c/0x1f0 [ 2543.610054] nvme_fabrics():1 [ 2543.612400] sbitmap_queue_clear+0x74/0xc0 [ 2543.616135] nvme_loop():1 [ 2543.619088] blk_mq_put_tag+0x44/0x70 [ 2543.623170] null_blk():1 [ 2543.625863] __blk_mq_free_request+0x88/0xf0 [ 2543.629511] null_blk():1 [ 2543.632117] blk_mq_free_request+0xc0/0x160 [ 2543.636373] null_blk():1 [ 2543.638979] bfq_insert_request.isra.0+0x204/0x270 [ 2543.643148] null_blk():1 [ 2543.645755] bfq_insert_requests+0x70/0x90 [ 2543.650531] null_blk():1 [ 2543.653137] blk_mq_sched_insert_requests+0xac/0x250 [ 2543.657219] null_blk():1 [ 2543.659826] blk_mq_flush_plug_list+0x110/0x2cc [ 2543.664776] null_blk():1 [ 2543.667382] __blk_flush_plug+0x100/0x140 [ 2543.671898] scsi_debug():1 [ 2543.674505] blk_finish_plug+0x40/0x25c [ 2543.678500] scsi_debug():1 [ 2543.681280] __iomap_dio_rw+0x1b4/0x500 [ 2543.685101] ipmi_si():1 [ 2543.687881] iomap_dio_rw+0x18/0x50 [ 2543.691703] ipmi_si():1 [ 2543.694222] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2543.697697] [last unloaded: nvmet] [ 2543.700216] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2543.705515] [ 2543.708989] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2543.713679] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2543.715156] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2543.720193] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2543.728008] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2543.733132] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2543.742422] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2543.747546] pc : __sbq_wake_up+0xa4/0x1f0 [ 2543.754492] process_one_work+0x1ec/0x470 [ 2543.759876] lr : sbitmap_queue_clear+0x74/0xc0 [ 2543.763872] worker_thread+0x74/0x410 [ 2543.767867] sp : ffff8000088438b0 [ 2543.772297] kthread+0xec/0x100 [ 2543.775944] x29: ffff8000088438b0 [ 2543.779245] ret_from_fork+0x10/0x20 [ 2543.782372] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2543.794532] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2543.801654] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2543.808776] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2543.815898] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2543.823020] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2543.830141] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2543.837263] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2543.844385] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2543.851507] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2543.858629] Call trace: [ 2543.861062] __sbq_wake_up+0xa4/0x1f0 [ 2543.864711] sbitmap_queue_clear+0x74/0xc0 [ 2543.868795] blk_mq_put_tag+0x44/0x70 [ 2543.872444] __blk_mq_free_request+0x88/0xf0 [ 2543.876701] blk_mq_free_request+0xc0/0x160 [ 2543.880871] __blk_mq_end_request+0x78/0x150 [ 2543.885128] scsi_end_request+0x114/0x260 [ 2543.889124] scsi_io_completion+0x64/0x174 [ 2543.893207] scsi_finish_command+0xe0/0x120 [ 2543.897377] scsi_complete+0x8c/0x1c4 [ 2543.901026] blk_complete_reqs+0x5c/0x74 [ 2543.904936] blk_done_softirq+0x38/0x4c [ 2543.908759] __do_softirq+0x168/0x418 [ 2543.912407] __irq_exit_rcu+0x120/0x170 [ 2543.916230] irq_exit_rcu+0x18/0x24 [ 2543.919705] el1_interrupt+0x38/0x70 [ 2543.923268] el1h_64_irq_handler+0x18/0x24 [ 2543.927351] el1h_64_irq+0x68/0x6c [ 2543.930739] cpuidle_enter_state+0xc4/0x4e0 [ 2543.934909] cpuidle_enter+0x40/0x60 [ 2543.938472] cpuidle_idle_call+0x118/0x1b0 [ 2543.942555] do_idle+0xac/0x100 [ 2543.945684] cpu_startup_entry+0x30/0x34 [ 2543.949593] secondary_start_kernel+0xd8/0xf0 [ 2543.953937] __secondary_switched+0xb0/0xb4 [ 2567.097640] watchdog: BUG: soft lockup - CPU#8 stuck for 1996s! [kworker/8:12:19399] [ 2567.105370] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2567.105387] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2567.193174] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2567.201424] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2567.207639] watchdog: BUG: soft lockup - CPU#68 stuck for 1996s! [swapper/68:0] [ 2567.210716] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2567.218010] Modules linked in: nvme_loop [ 2567.224176] [ 2567.224176] nvme_fabrics [ 2567.228085] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2567.229563] nvmet [ 2567.232170] pc : __sbq_wake_up+0x58/0x1f0 [ 2567.239116] nls_utf8 [ 2567.241115] lr : sbitmap_queue_clear+0x74/0xc0 [ 2567.245110] hfsplus [ 2567.247369] sp : ffff80000d79b280 [ 2567.251798] hfs [ 2567.253971] x29: ffff80000d79b280 [ 2567.257271] isofs [ 2567.259097] x28: ffffd5becb6aec50 [ 2567.262397] nvme_core [ 2567.264396] x27: ffffd5beca4f7000 [ 2567.267784] nvme_common [ 2567.270130] [ 2567.273517] loop [ 2567.276037] x26: ffffd5becaf68a98 [ 2567.277514] tls [ 2567.279426] x25: ffffd5becb6a60c0 [ 2567.282727] rfkill [ 2567.284552] x24: fffffbffef180dc0 [ 2567.287940] sunrpc [ 2567.290026] [ 2567.293413] vfat [ 2567.295499] x23: ffff07ff81d381b0 [ 2567.296977] fat [ 2567.298889] x22: ffff80000d79b30c [ 2567.302189] acpi_ipmi [ 2567.304014] x21: ffff07ff84e93c00 [ 2567.307402] ipmi_ssif [ 2567.309748] [ 2567.313136] cppc_cpufreq [ 2567.315481] x20: 00000000000001c0 [ 2567.316959] ipmi_devintf [ 2567.319566] x19: ffff07ff91ccdb10 [ 2567.322866] igb [ 2567.325473] x18: ffffffffffffffff [ 2567.328860] ipmi_msghandler [ 2567.330685] [ 2567.334073] arm_cmn [ 2567.336940] x17: ffff32506c412000 [ 2567.338418] arm_dmc620_pmu [ 2567.340590] x16: ffff800008044000 [ 2567.343891] arm_spe_pmu [ 2567.346671] x15: 0000000000004000 [ 2567.350058] arm_dsu_pmu [ 2567.352578] [ 2567.355965] acpi_tad [ 2567.358485] x14: 010320bb28ec0000 [ 2567.359963] fuse [ 2567.362222] x13: 0000000000000087 [ 2567.365524] zram [ 2567.367436] x12: 071c71c71c71c71c [ 2567.370824] xfs [ 2567.372738] [ 2567.376125] crct10dif_ce [ 2567.377950] x11: 0000000000000040 [ 2567.379428] polyval_ce [ 2567.382034] x10: ffff07ff801a9c70 [ 2567.385335] ast [ 2567.387768] x9 : ffffd5bec9545fd8 [ 2567.391156] polyval_generic [ 2567.392981] [ 2567.396369] drm_vram_helper [ 2567.399235] x8 : ffff07ff80404b70 [ 2567.400713] ghash_ce [ 2567.403580] x7 : ffff07ff84e93dc8 [ 2567.406881] drm_ttm_helper [ 2567.409140] x6 : 0000000000000008 [ 2567.412527] sbsa_gwdt [ 2567.415307] [ 2567.418695] ttm [ 2567.421041] x5 : ffff80000d043790 [ 2567.422519] xgene_hwmon [ 2567.424344] x4 : ffff07ff84e93dc0 [ 2567.427645] Unloaded tainted modules: [ 2567.430164] x3 : 0000000000000008 [ 2567.433552] nvmet():1 [ 2567.437200] [ 2567.440587] nvme_fabrics():1 [ 2567.442933] x2 : 0000000000000007 [ 2567.444411] nvme_loop():1 [ 2567.447365] x1 : ffff80000d79b30c [ 2567.450665] nvmet():1 [ 2567.453358] x0 : ffff07ff84e93dd0 [ 2567.456746] nvme_fabrics():1 [ 2567.459092] [ 2567.462480] nvme_loop():1 [ 2567.465433] Call trace: [ 2567.466911] nvmet():1 [ 2567.469605] __sbq_wake_up+0x58/0x1f0 [ 2567.472037] nvme_fabrics():1 [ 2567.474383] sbitmap_queue_clear+0x74/0xc0 [ 2567.478031] nvme_loop():1 [ 2567.480985] blk_mq_put_tag+0x44/0x70 [ 2567.485067] nvmet():1 [ 2567.487760] __blk_mq_free_request+0x88/0xf0 [ 2567.491408] nvme_fabrics():1 [ 2567.493754] blk_mq_free_request+0xc0/0x160 [ 2567.498009] nvme_loop():1 [ 2567.500963] __blk_mq_end_request+0x78/0x150 [ 2567.505132] nvmet():1 [ 2567.507825] scsi_end_request+0x114/0x260 [ 2567.512081] nvme_fabrics():1 [ 2567.514426] scsi_io_completion+0x64/0x174 [ 2567.518422] nvme_loop():1 [ 2567.521375] scsi_finish_command+0xe0/0x120 [ 2567.525457] nvmet():1 [ 2567.528150] scsi_complete+0x8c/0x1c4 [ 2567.532319] nvme_fabrics():1 [ 2567.534665] blk_complete_reqs+0x5c/0x74 [ 2567.538313] nvme_loop():1 [ 2567.541267] blk_done_softirq+0x38/0x4c [ 2567.545175] nvmet():1 [ 2567.547868] __do_softirq+0x168/0x418 [ 2567.551690] nvme_fabrics():1 [ 2567.554036] __irq_exit_rcu+0x120/0x170 [ 2567.557684] nvme_loop():1 [ 2567.560638] irq_exit_rcu+0x18/0x24 [ 2567.564459] nvmet():1 [ 2567.567152] el1_interrupt+0x38/0x70 [ 2567.570627] nvme_fabrics():1 [ 2567.572973] el1h_64_irq_handler+0x18/0x24 [ 2567.576534] nvme_loop():1 [ 2567.579488] el1h_64_irq+0x68/0x6c [ 2567.583570] nvmet():1 [ 2567.586263] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2567.589650] nvme_fabrics():1 [ 2567.591996] __wake_up_common_lock+0x84/0xac [ 2567.596773] nvme_loop():1 [ 2567.599726] __wake_up+0x20/0x30 [ 2567.603982] nvmet():1 [ 2567.606675] __sbq_wake_up+0x13c/0x1f0 [ 2567.609889] nvme_fabrics():1 [ 2567.612235] sbitmap_queue_clear+0x74/0xc0 [ 2567.615970] nvme_loop():1 [ 2567.618924] blk_mq_put_tag+0x44/0x70 [ 2567.623007] null_blk():1 [ 2567.625700] __blk_mq_free_request+0x88/0xf0 [ 2567.629348] null_blk():1 [ 2567.631954] blk_mq_free_request+0xc0/0x160 [ 2567.636210] null_blk():1 [ 2567.638816] bfq_insert_request.isra.0+0x204/0x270 [ 2567.642985] null_blk():1 [ 2567.645591] bfq_insert_requests+0x70/0x90 [ 2567.650368] null_blk():1 [ 2567.652974] blk_mq_sched_insert_requests+0xac/0x250 [ 2567.657056] null_blk():1 [ 2567.659662] blk_mq_flush_plug_list+0x110/0x2cc [ 2567.664613] null_blk():1 [ 2567.667220] __blk_flush_plug+0x100/0x140 [ 2567.671737] scsi_debug():1 [ 2567.674343] blk_finish_plug+0x40/0x25c [ 2567.678339] scsi_debug():1 [ 2567.681118] __iomap_dio_rw+0x1b4/0x500 [ 2567.684940] ipmi_si():1 [ 2567.687720] iomap_dio_rw+0x18/0x50 [ 2567.691542] ipmi_si():1 [ 2567.694061] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2567.697536] [last unloaded: nvmet] [ 2567.700055] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2567.705352] [ 2567.708827] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2567.713516] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2567.714994] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2567.720031] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2567.727846] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2567.732969] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2567.742260] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2567.747384] pc : __sbq_wake_up+0x84/0x1f0 [ 2567.754330] process_one_work+0x1ec/0x470 [ 2567.759714] lr : sbitmap_queue_clear+0x74/0xc0 [ 2567.763709] worker_thread+0x74/0x410 [ 2567.767705] sp : ffff8000088438b0 [ 2567.772134] kthread+0xec/0x100 [ 2567.775782] x29: ffff8000088438b0 [ 2567.779083] ret_from_fork+0x10/0x20 [ 2567.782210] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2567.794369] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2567.801491] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2567.808613] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2567.815735] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2567.822857] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2567.829979] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2567.837100] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2567.844223] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2567.851345] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2567.858466] Call trace: [ 2567.860899] __sbq_wake_up+0x84/0x1f0 [ 2567.864548] sbitmap_queue_clear+0x74/0xc0 [ 2567.868631] blk_mq_put_tag+0x44/0x70 [ 2567.872281] __blk_mq_free_request+0x88/0xf0 [ 2567.876538] blk_mq_free_request+0xc0/0x160 [ 2567.880708] __blk_mq_end_request+0x78/0x150 [ 2567.884964] scsi_end_request+0x114/0x260 [ 2567.888961] scsi_io_completion+0x64/0x174 [ 2567.893044] scsi_finish_command+0xe0/0x120 [ 2567.897214] scsi_complete+0x8c/0x1c4 [ 2567.900863] blk_complete_reqs+0x5c/0x74 [ 2567.904773] blk_done_softirq+0x38/0x4c [ 2567.908595] __do_softirq+0x168/0x418 [ 2567.912244] __irq_exit_rcu+0x120/0x170 [ 2567.916067] irq_exit_rcu+0x18/0x24 [ 2567.919542] el1_interrupt+0x38/0x70 [ 2567.923104] el1h_64_irq_handler+0x18/0x24 [ 2567.927187] el1h_64_irq+0x68/0x6c [ 2567.930575] cpuidle_enter_state+0xc4/0x4e0 [ 2567.934745] cpuidle_enter+0x40/0x60 [ 2567.938308] cpuidle_idle_call+0x118/0x1b0 [ 2567.942392] do_idle+0xac/0x100 [ 2567.945521] cpu_startup_entry+0x30/0x34 [ 2567.949430] secondary_start_kernel+0xd8/0xf0 [ 2567.953774] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:40:00 2022] [ 2591.097480] watchdog: BUG: soft lockup - CPU#8 stuck for 2019s! [kworker/8:12:19399] [ 2591.105210] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2591.105227] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2591.193015] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2591.201265] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2591.207479] watchdog: BUG: soft lockup - CPU#68 stuck for 2019s! [swapper/68:0] [ 2591.210556] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2591.217849] Modules linked in: nvme_loop [ 2591.224015] [ 2591.224016] nvme_fabrics [ 2591.227925] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2591.229403] nvmet [ 2591.232009] pc : __sbq_wake_up+0x58/0x1f0 [ 2591.238956] nls_utf8 [ 2591.240954] lr : sbitmap_queue_clear+0x74/0xc0 [ 2591.244950] hfsplus [ 2591.247209] sp : ffff80000d79b280 [ 2591.251638] hfs [ 2591.253810] x29: ffff80000d79b280 [ 2591.257111] isofs [ 2591.258936] x28: ffffd5becb6aec50 [ 2591.262237] nvme_core [ 2591.264235] x27: ffffd5beca4f7000 [ 2591.267623] nvme_common [ 2591.269971] [ 2591.273358] loop [ 2591.275878] x26: ffffd5becaf68a98 [ 2591.277356] tls [ 2591.279268] x25: ffffd5becb6a60c0 [ 2591.282569] rfkill [ 2591.284394] x24: fffffbffef180dc0 [ 2591.287782] sunrpc [ 2591.289867] [ 2591.293255] vfat [ 2591.295340] x23: ffff07ff81d381b0 [ 2591.296818] fat [ 2591.298730] x22: ffff80000d79b30c [ 2591.302031] acpi_ipmi [ 2591.303856] x21: ffff07ff84e93c00 [ 2591.307244] ipmi_ssif [ 2591.309590] [ 2591.312977] cppc_cpufreq [ 2591.315324] x20: 00000000000001c0 [ 2591.316803] ipmi_devintf [ 2591.319409] x19: ffff07ff91ccdb10 [ 2591.322710] igb [ 2591.325316] x18: ffffffffffffffff [ 2591.328704] ipmi_msghandler [ 2591.330529] [ 2591.333917] arm_cmn [ 2591.336784] x17: ffff32506c412000 [ 2591.338262] arm_dmc620_pmu [ 2591.340435] x16: ffff800008044000 [ 2591.343736] arm_spe_pmu [ 2591.346515] x15: 0000000000004000 [ 2591.349903] arm_dsu_pmu [ 2591.352423] [ 2591.355810] acpi_tad [ 2591.358330] x14: 010320bb28ec0000 [ 2591.359807] fuse [ 2591.362067] x13: 0000000000000087 [ 2591.365367] zram [ 2591.367279] x12: 071c71c71c71c71c [ 2591.370667] xfs [ 2591.372579] [ 2591.375966] crct10dif_ce [ 2591.377791] x11: 0000000000000040 [ 2591.379269] polyval_ce [ 2591.381876] x10: ffff07ff801a9c70 [ 2591.385176] ast [ 2591.387609] x9 : ffffd5bec9545fd8 [ 2591.390997] polyval_generic [ 2591.392822] [ 2591.396209] drm_vram_helper [ 2591.399076] x8 : ffff07ff80404b70 [ 2591.400554] ghash_ce [ 2591.403421] x7 : ffff07ff84e93dc8 [ 2591.406722] drm_ttm_helper [ 2591.408983] x6 : 0000000000000008 [ 2591.412370] sbsa_gwdt [ 2591.415150] [ 2591.418538] ttm [ 2591.420884] x5 : ffff80000d043790 [ 2591.422361] xgene_hwmon [ 2591.424187] x4 : ffff07ff84e93dc0 [ 2591.427487] Unloaded tainted modules: [ 2591.430007] x3 : 0000000000000008 [ 2591.433394] nvmet():1 [ 2591.437042] [ 2591.440430] nvme_fabrics():1 [ 2591.442776] x2 : 0000000000000007 [ 2591.444254] nvme_loop():1 [ 2591.447207] x1 : ffff80000d79b30c [ 2591.450508] nvmet():1 [ 2591.453201] x0 : ffff07ff84e93dd0 [ 2591.456589] nvme_fabrics():1 [ 2591.458935] [ 2591.462323] nvme_loop():1 [ 2591.465278] Call trace: [ 2591.466756] nvmet():1 [ 2591.469449] __sbq_wake_up+0x58/0x1f0 [ 2591.471882] nvme_fabrics():1 [ 2591.474228] sbitmap_queue_clear+0x74/0xc0 [ 2591.477877] nvme_loop():1 [ 2591.480831] blk_mq_put_tag+0x44/0x70 [ 2591.484913] nvmet():1 [ 2591.487606] __blk_mq_free_request+0x88/0xf0 [ 2591.491254] nvme_fabrics():1 [ 2591.493600] blk_mq_free_request+0xc0/0x160 [ 2591.497855] nvme_loop():1 [ 2591.500809] __blk_mq_end_request+0x78/0x150 [ 2591.504978] nvmet():1 [ 2591.507671] scsi_end_request+0x114/0x260 [ 2591.511927] nvme_fabrics():1 [ 2591.514272] scsi_io_completion+0x64/0x174 [ 2591.518268] nvme_loop():1 [ 2591.521221] scsi_finish_command+0xe0/0x120 [ 2591.525303] nvmet():1 [ 2591.527996] scsi_complete+0x8c/0x1c4 [ 2591.532165] nvme_fabrics():1 [ 2591.534511] blk_complete_reqs+0x5c/0x74 [ 2591.538159] nvme_loop():1 [ 2591.541113] blk_done_softirq+0x38/0x4c [ 2591.545021] nvmet():1 [ 2591.547714] __do_softirq+0x168/0x418 [ 2591.551536] nvme_fabrics():1 [ 2591.553882] __irq_exit_rcu+0x120/0x170 [ 2591.557530] nvme_loop():1 [ 2591.560483] irq_exit_rcu+0x18/0x24 [ 2591.564305] nvmet():1 [ 2591.566998] el1_interrupt+0x38/0x70 [ 2591.570472] nvme_fabrics():1 [ 2591.572818] el1h_64_irq_handler+0x18/0x24 [ 2591.576379] nvme_loop():1 [ 2591.579333] el1h_64_irq+0x68/0x6c [ 2591.583415] nvmet():1 [ 2591.586108] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2591.589496] nvme_fabrics():1 [ 2591.591842] __wake_up_common_lock+0x84/0xac [ 2591.596618] nvme_loop():1 [ 2591.599572] __wake_up+0x20/0x30 [ 2591.603827] nvmet():1 [ 2591.606521] __sbq_wake_up+0x13c/0x1f0 [ 2591.609735] nvme_fabrics():1 [ 2591.612081] sbitmap_queue_clear+0x74/0xc0 [ 2591.615816] nvme_loop():1 [ 2591.618769] blk_mq_put_tag+0x44/0x70 [ 2591.622851] null_blk():1 [ 2591.625544] __blk_mq_free_request+0x88/0xf0 [ 2591.629192] null_blk():1 [ 2591.631799] blk_mq_free_request+0xc0/0x160 [ 2591.636054] null_blk():1 [ 2591.638661] bfq_insert_request.isra.0+0x204/0x270 [ 2591.642829] null_blk():1 [ 2591.645436] bfq_insert_requests+0x70/0x90 [ 2591.650212] null_blk():1 [ 2591.652819] blk_mq_sched_insert_requests+0xac/0x250 [ 2591.656901] null_blk():1 [ 2591.659507] blk_mq_flush_plug_list+0x110/0x2cc [ 2591.664457] null_blk():1 [ 2591.667064] __blk_flush_plug+0x100/0x140 [ 2591.671580] scsi_debug():1 [ 2591.674186] blk_finish_plug+0x40/0x25c [ 2591.678181] scsi_debug():1 [ 2591.680961] __iomap_dio_rw+0x1b4/0x500 [ 2591.684783] ipmi_si():1 [ 2591.687563] iomap_dio_rw+0x18/0x50 [ 2591.691384] ipmi_si():1 [ 2591.693904] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2591.697378] [last unloaded: nvmet] [ 2591.699898] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2591.705195] [ 2591.708670] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2591.713359] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2591.714837] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2591.719874] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2591.727689] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2591.732812] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2591.742103] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2591.747226] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2591.754174] process_one_work+0x1ec/0x470 [ 2591.759558] lr : sbitmap_queue_clear+0x74/0xc0 [ 2591.763641] worker_thread+0x74/0x410 [ 2591.767636] sp : ffff8000088438b0 [ 2591.772065] kthread+0xec/0x100 [ 2591.775713] x29: ffff8000088438b0 [ 2591.779014] ret_from_fork+0x10/0x20 [ 2591.782141] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2591.794301] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2591.801423] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2591.808545] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2591.815667] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2591.822789] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2591.829911] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2591.837033] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2591.844154] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2591.851276] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2591.858398] Call trace: [ 2591.860831] __sbq_wake_up+0x1c8/0x1f0 [ 2591.864567] sbitmap_queue_clear+0x74/0xc0 [ 2591.868650] blk_mq_put_tag+0x44/0x70 [ 2591.872300] __blk_mq_free_request+0x88/0xf0 [ 2591.876557] blk_mq_free_request+0xc0/0x160 [ 2591.880727] __blk_mq_end_request+0x78/0x150 [ 2591.884983] scsi_end_request+0x114/0x260 [ 2591.888980] scsi_io_completion+0x64/0x174 [ 2591.893063] scsi_finish_command+0xe0/0x120 [ 2591.897233] scsi_complete+0x8c/0x1c4 [ 2591.900882] blk_complete_reqs+0x5c/0x74 [ 2591.904792] blk_done_softirq+0x38/0x4c [ 2591.908614] __do_softirq+0x168/0x418 [ 2591.912263] __irq_exit_rcu+0x120/0x170 [ 2591.916086] irq_exit_rcu+0x18/0x24 [ 2591.919561] el1_interrupt+0x38/0x70 [ 2591.923123] el1h_64_irq_handler+0x18/0x24 [ 2591.927207] el1h_64_irq+0x68/0x6c [ 2591.930595] cpuidle_enter_state+0xc4/0x4e0 [ 2591.934765] cpuidle_enter+0x40/0x60 [ 2591.938327] cpuidle_idle_call+0x118/0x1b0 [ 2591.942411] do_idle+0xac/0x100 [ 2591.945540] cpu_startup_entry+0x30/0x34 [ 2591.949449] secondary_start_kernel+0xd8/0xf0 [ 2591.953793] __secondary_switched+0xb0/0xb4 [ 2601.957417] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 215101 jiffies s: 2093 root: 0x11/. [ 2601.969155] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2601.978364] Task dump for CPU 8: [ 2601.981578] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2601.991482] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2601.997651] Call trace: [ 2602.000084] __switch_to+0xd4/0x100 [ 2602.003561] 0xffff6c0200000080 [ 2602.006690] Task dump for CPU 68: [ 2602.009993] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2602.019894] Call trace: [ 2602.022327] __switch_to+0xd4/0x100 [ 2602.025803] 0x0 [ 2615.097320] watchdog: BUG: soft lockup - CPU#8 stuck for 2041s! [kworker/8:12:19399] [ 2615.105050] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2615.105067] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2615.192856] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2615.201106] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2615.207319] watchdog: BUG: soft lockup - CPU#68 stuck for 2041s! [swapper/68:0] [ 2615.210397] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2615.217691] Modules linked in: nvme_loop [ 2615.223856] [ 2615.223857] nvme_fabrics [ 2615.227765] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2615.229243] nvmet [ 2615.231850] pc : __sbq_wake_up+0x58/0x1f0 [ 2615.238796] nls_utf8 [ 2615.240795] lr : sbitmap_queue_clear+0x74/0xc0 [ 2615.244790] hfsplus [ 2615.247050] sp : ffff80000d79b280 [ 2615.251479] hfs [ 2615.253651] x29: ffff80000d79b280 [ 2615.256952] isofs [ 2615.258777] x28: ffffd5becb6aec50 [ 2615.262078] nvme_core [ 2615.264076] x27: ffffd5beca4f7000 [ 2615.267464] nvme_common [ 2615.269810] [ 2615.273198] loop [ 2615.275717] x26: ffffd5becaf68a98 [ 2615.277195] tls [ 2615.279107] x25: ffffd5becb6a60c0 [ 2615.282408] rfkill [ 2615.284233] x24: fffffbffef180dc0 [ 2615.287620] sunrpc [ 2615.289706] [ 2615.293093] vfat [ 2615.295179] x23: ffff07ff81d381b0 [ 2615.296657] fat [ 2615.298569] x22: ffff80000d79b30c [ 2615.301870] acpi_ipmi [ 2615.303695] x21: ffff07ff84e93c00 [ 2615.307083] ipmi_ssif [ 2615.309428] [ 2615.312816] cppc_cpufreq [ 2615.315162] x20: 00000000000001c0 [ 2615.316640] ipmi_devintf [ 2615.319246] x19: ffff07ff91ccdb10 [ 2615.322547] igb [ 2615.325153] x18: ffffffffffffffff [ 2615.328541] ipmi_msghandler [ 2615.330366] [ 2615.333754] arm_cmn [ 2615.336621] x17: ffff32506c412000 [ 2615.338099] arm_dmc620_pmu [ 2615.340271] x16: ffff800008044000 [ 2615.343572] arm_spe_pmu [ 2615.346352] x15: 0000000000004000 [ 2615.349739] arm_dsu_pmu [ 2615.352259] [ 2615.355646] acpi_tad [ 2615.358166] x14: 010320bb28ec0000 [ 2615.359644] fuse [ 2615.361903] x13: 0000000000000087 [ 2615.365204] zram [ 2615.367116] x12: 071c71c71c71c71c [ 2615.370504] xfs [ 2615.372415] [ 2615.375803] crct10dif_ce [ 2615.377628] x11: 0000000000000040 [ 2615.379106] polyval_ce [ 2615.381712] x10: ffff07ff801a9c70 [ 2615.385013] ast [ 2615.387446] x9 : ffffd5bec9545fd8 [ 2615.390833] polyval_generic [ 2615.392658] [ 2615.396046] drm_vram_helper [ 2615.398913] x8 : ffff07ff80404b70 [ 2615.400391] ghash_ce [ 2615.403257] x7 : ffff07ff84e93dc8 [ 2615.406558] drm_ttm_helper [ 2615.408817] x6 : 0000000000000008 [ 2615.412205] sbsa_gwdt [ 2615.414985] [ 2615.418372] ttm [ 2615.420718] x5 : ffff80000d043790 [ 2615.422196] xgene_hwmon [ 2615.424021] x4 : ffff07ff84e93dc0 [ 2615.427322] Unloaded tainted modules: [ 2615.429841] x3 : 0000000000000008 [ 2615.433229] nvmet():1 [ 2615.436877] [ 2615.440265] nvme_fabrics():1 [ 2615.442610] x2 : 0000000000000007 [ 2615.444088] nvme_loop():1 [ 2615.447042] x1 : ffff80000d79b30c [ 2615.450343] nvmet():1 [ 2615.453036] x0 : ffff07ff84e93dd0 [ 2615.456423] nvme_fabrics():1 [ 2615.458769] [ 2615.462157] nvme_loop():1 [ 2615.465111] Call trace: [ 2615.466588] nvmet():1 [ 2615.469281] __sbq_wake_up+0x58/0x1f0 [ 2615.471714] nvme_fabrics():1 [ 2615.474060] sbitmap_queue_clear+0x74/0xc0 [ 2615.477708] nvme_loop():1 [ 2615.480661] blk_mq_put_tag+0x44/0x70 [ 2615.484743] nvmet():1 [ 2615.487437] __blk_mq_free_request+0x88/0xf0 [ 2615.491085] nvme_fabrics():1 [ 2615.493431] blk_mq_free_request+0xc0/0x160 [ 2615.497686] nvme_loop():1 [ 2615.500640] __blk_mq_end_request+0x78/0x150 [ 2615.504809] nvmet():1 [ 2615.507504] scsi_end_request+0x114/0x260 [ 2615.511759] nvme_fabrics():1 [ 2615.514105] scsi_io_completion+0x64/0x174 [ 2615.518101] nvme_loop():1 [ 2615.521054] scsi_finish_command+0xe0/0x120 [ 2615.525136] nvmet():1 [ 2615.527829] scsi_complete+0x8c/0x1c4 [ 2615.531998] nvme_fabrics():1 [ 2615.534344] blk_complete_reqs+0x5c/0x74 [ 2615.537992] nvme_loop():1 [ 2615.540946] blk_done_softirq+0x38/0x4c [ 2615.544854] nvmet():1 [ 2615.547548] __do_softirq+0x168/0x418 [ 2615.551369] nvme_fabrics():1 [ 2615.553715] __irq_exit_rcu+0x120/0x170 [ 2615.557363] nvme_loop():1 [ 2615.560317] irq_exit_rcu+0x18/0x24 [ 2615.564138] nvmet():1 [ 2615.566831] el1_interrupt+0x38/0x70 [ 2615.570306] nvme_fabrics():1 [ 2615.572652] el1h_64_irq_handler+0x18/0x24 [ 2615.576213] nvme_loop():1 [ 2615.579166] el1h_64_irq+0x68/0x6c [ 2615.583248] nvmet():1 [ 2615.585942] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2615.589329] nvme_fabrics():1 [ 2615.591675] __wake_up_common_lock+0x84/0xac [ 2615.596452] nvme_loop():1 [ 2615.599405] __wake_up+0x20/0x30 [ 2615.603661] nvmet():1 [ 2615.606354] __sbq_wake_up+0x13c/0x1f0 [ 2615.609568] nvme_fabrics():1 [ 2615.611914] sbitmap_queue_clear+0x74/0xc0 [ 2615.615648] nvme_loop():1 [ 2615.618602] blk_mq_put_tag+0x44/0x70 [ 2615.622684] null_blk():1 [ 2615.625377] __blk_mq_free_request+0x88/0xf0 [ 2615.629025] null_blk():1 [ 2615.631631] blk_mq_free_request+0xc0/0x160 [ 2615.635887] null_blk():1 [ 2615.638494] bfq_insert_request.isra.0+0x204/0x270 [ 2615.642662] null_blk():1 [ 2615.645269] bfq_insert_requests+0x70/0x90 [ 2615.650045] null_blk():1 [ 2615.652651] blk_mq_sched_insert_requests+0xac/0x250 [ 2615.656733] null_blk():1 [ 2615.659340] blk_mq_flush_plug_list+0x110/0x2cc [ 2615.664290] null_blk():1 [ 2615.666896] __blk_flush_plug+0x100/0x140 [ 2615.671412] scsi_debug():1 [ 2615.674018] blk_finish_plug+0x40/0x25c [ 2615.678014] scsi_debug():1 [ 2615.680794] __iomap_dio_rw+0x1b4/0x500 [ 2615.684615] ipmi_si():1 [ 2615.687395] iomap_dio_rw+0x18/0x50 [ 2615.691217] ipmi_si():1 [ 2615.693737] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2615.697211] [last unloaded: nvmet] [ 2615.699731] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2615.705028] [ 2615.708503] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2615.713193] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2615.714671] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2615.719707] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2615.727522] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2615.732646] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2615.741936] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2615.747060] pc : __sbq_wake_up+0x38/0x1f0 [ 2615.754006] process_one_work+0x1ec/0x470 [ 2615.759390] lr : sbitmap_queue_clear+0x74/0xc0 [ 2615.763386] worker_thread+0x74/0x410 [ 2615.767381] sp : ffff8000088438b0 [ 2615.771811] kthread+0xec/0x100 [ 2615.775459] x29: ffff8000088438b0 [ 2615.778759] ret_from_fork+0x10/0x20 [ 2615.781886] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2615.794046] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2615.801168] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2615.808290] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2615.815412] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2615.822534] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2615.829656] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2615.836778] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2615.843900] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2615.851022] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2615.858146] Call trace: [ 2615.860579] __sbq_wake_up+0x38/0x1f0 [ 2615.864228] sbitmap_queue_clear+0x74/0xc0 [ 2615.868311] blk_mq_put_tag+0x44/0x70 [ 2615.871961] __blk_mq_free_request+0x88/0xf0 [ 2615.876218] blk_mq_free_request+0xc0/0x160 [ 2615.880388] __blk_mq_end_request+0x78/0x150 [ 2615.884644] scsi_end_request+0x114/0x260 [ 2615.888641] scsi_io_completion+0x64/0x174 [ 2615.892724] scsi_finish_command+0xe0/0x120 [ 2615.896894] scsi_complete+0x8c/0x1c4 [ 2615.900543] blk_complete_reqs+0x5c/0x74 [ 2615.904453] blk_done_softirq+0x38/0x4c [ 2615.908275] __do_softirq+0x168/0x418 [ 2615.911924] __irq_exit_rcu+0x120/0x170 [ 2615.915747] irq_exit_rcu+0x18/0x24 [ 2615.919223] el1_interrupt+0x38/0x70 [ 2615.922785] el1h_64_irq_handler+0x18/0x24 [ 2615.926868] el1h_64_irq+0x68/0x6c [ 2615.930256] cpuidle_enter_state+0xc4/0x4e0 [ 2615.934426] cpuidle_enter+0x40/0x60 [ 2615.937988] cpuidle_idle_call+0x118/0x1b0 [ 2615.942072] do_idle+0xac/0x100 [ 2615.945201] cpu_startup_entry+0x30/0x34 [ 2615.949110] secondary_start_kernel+0xd8/0xf0 [ 2615.953454] __secondary_switched+0xb0/0xb4 [ 2639.097160] watchdog: BUG: soft lockup - CPU#8 stuck for 2063s! [kworker/8:12:19399] [ 2639.104890] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2639.104907] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2639.192693] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2639.200943] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2639.207159] watchdog: BUG: soft lockup - CPU#68 stuck for 2063s! [swapper/68:0] [ 2639.210234] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2639.217528] Modules linked in: nvme_loop [ 2639.223694] [ 2639.223694] nvme_fabrics [ 2639.227603] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2639.229081] nvmet [ 2639.231687] pc : __sbq_wake_up+0x84/0x1f0 [ 2639.238634] nls_utf8 [ 2639.240633] lr : sbitmap_queue_clear+0x74/0xc0 [ 2639.244629] hfsplus [ 2639.246889] sp : ffff80000d79b280 [ 2639.251319] hfs [ 2639.253491] x29: ffff80000d79b280 [ 2639.256792] isofs [ 2639.258618] x28: ffffd5becb6aec50 [ 2639.261918] nvme_core [ 2639.263917] x27: ffffd5beca4f7000 [ 2639.267305] nvme_common [ 2639.269650] [ 2639.273038] loop [ 2639.275558] x26: ffffd5becaf68a98 [ 2639.277036] tls [ 2639.278947] x25: ffffd5becb6a60c0 [ 2639.282248] rfkill [ 2639.284073] x24: fffffbffef180dc0 [ 2639.287461] sunrpc [ 2639.289546] [ 2639.292934] vfat [ 2639.295019] x23: ffff07ff81d381b0 [ 2639.296497] fat [ 2639.298409] x22: ffff80000d79b30c [ 2639.301710] acpi_ipmi [ 2639.303535] x21: ffff07ff84e93c00 [ 2639.306923] ipmi_ssif [ 2639.309268] [ 2639.312656] cppc_cpufreq [ 2639.315002] x20: 00000000000001c0 [ 2639.316480] ipmi_devintf [ 2639.319086] x19: ffff07ff91ccdb10 [ 2639.322387] igb [ 2639.324993] x18: ffffffffffffffff [ 2639.328381] ipmi_msghandler [ 2639.330206] [ 2639.333593] arm_cmn [ 2639.336460] x17: ffff32506c412000 [ 2639.337938] arm_dmc620_pmu [ 2639.340110] x16: ffff800008044000 [ 2639.343411] arm_spe_pmu [ 2639.346191] x15: 0000000000004000 [ 2639.349579] arm_dsu_pmu [ 2639.352098] [ 2639.355486] acpi_tad [ 2639.358006] x14: 010320bb28ec0000 [ 2639.359484] fuse [ 2639.361743] x13: 0000000000000087 [ 2639.365043] zram [ 2639.366955] x12: 071c71c71c71c71c [ 2639.370343] xfs [ 2639.372255] [ 2639.375642] crct10dif_ce [ 2639.377467] x11: 0000000000000040 [ 2639.378945] polyval_ce [ 2639.381552] x10: ffff07ff801a9c70 [ 2639.384852] ast [ 2639.387285] x9 : ffffd5bec9545fd8 [ 2639.390673] polyval_generic [ 2639.392498] [ 2639.395885] drm_vram_helper [ 2639.398752] x8 : ffff07ff80404b70 [ 2639.400230] ghash_ce [ 2639.403098] x7 : ffff07ff84e93dc8 [ 2639.406399] drm_ttm_helper [ 2639.408658] x6 : 0000000000000008 [ 2639.412045] sbsa_gwdt [ 2639.414825] [ 2639.418213] ttm [ 2639.420559] x5 : ffff80000d043790 [ 2639.422037] xgene_hwmon [ 2639.423862] x4 : ffff07ff84e93dc0 [ 2639.427162] Unloaded tainted modules: [ 2639.429682] x3 : 0000000000000008 [ 2639.433070] nvmet():1 [ 2639.436717] [ 2639.440105] nvme_fabrics():1 [ 2639.442451] x2 : 0000000000000007 [ 2639.443929] nvme_loop():1 [ 2639.446884] x1 : ffff80000d79b30c [ 2639.450185] nvmet():1 [ 2639.452878] x0 : ffff07ff84e93dd0 [ 2639.456265] nvme_fabrics():1 [ 2639.458611] [ 2639.461999] nvme_loop():1 [ 2639.464952] Call trace: [ 2639.466430] nvmet():1 [ 2639.469124] __sbq_wake_up+0x84/0x1f0 [ 2639.471558] nvme_fabrics():1 [ 2639.473904] sbitmap_queue_clear+0x74/0xc0 [ 2639.477552] nvme_loop():1 [ 2639.480506] blk_mq_put_tag+0x44/0x70 [ 2639.484588] nvmet():1 [ 2639.487282] __blk_mq_free_request+0x88/0xf0 [ 2639.490930] nvme_fabrics():1 [ 2639.493276] blk_mq_free_request+0xc0/0x160 [ 2639.497532] nvme_loop():1 [ 2639.500486] __blk_mq_end_request+0x78/0x150 [ 2639.504654] nvmet():1 [ 2639.507348] scsi_end_request+0x114/0x260 [ 2639.511603] nvme_fabrics():1 [ 2639.513949] scsi_io_completion+0x64/0x174 [ 2639.517944] nvme_loop():1 [ 2639.520898] scsi_finish_command+0xe0/0x120 [ 2639.524980] nvmet():1 [ 2639.527673] scsi_complete+0x8c/0x1c4 [ 2639.531842] nvme_fabrics():1 [ 2639.534188] blk_complete_reqs+0x5c/0x74 [ 2639.537836] nvme_loop():1 [ 2639.540789] blk_done_softirq+0x38/0x4c [ 2639.544698] nvmet():1 [ 2639.547391] __do_softirq+0x168/0x418 [ 2639.551212] nvme_fabrics():1 [ 2639.553558] __irq_exit_rcu+0x120/0x170 [ 2639.557206] nvme_loop():1 [ 2639.560160] irq_exit_rcu+0x18/0x24 [ 2639.563981] nvmet():1 [ 2639.566675] el1_interrupt+0x38/0x70 [ 2639.570149] nvme_fabrics():1 [ 2639.572495] el1h_64_irq_handler+0x18/0x24 [ 2639.576056] nvme_loop():1 [ 2639.579010] el1h_64_irq+0x68/0x6c [ 2639.583092] nvmet():1 [ 2639.585785] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2639.589172] nvme_fabrics():1 [ 2639.591518] __wake_up_common_lock+0x84/0xac [ 2639.596295] nvme_loop():1 [ 2639.599248] __wake_up+0x20/0x30 [ 2639.603504] nvmet():1 [ 2639.606197] __sbq_wake_up+0x13c/0x1f0 [ 2639.609411] nvme_fabrics():1 [ 2639.611757] sbitmap_queue_clear+0x74/0xc0 [ 2639.615492] nvme_loop():1 [ 2639.618445] blk_mq_put_tag+0x44/0x70 [ 2639.622528] null_blk():1 [ 2639.625222] __blk_mq_free_request+0x88/0xf0 [ 2639.628870] null_blk():1 [ 2639.631477] blk_mq_free_request+0xc0/0x160 [ 2639.635733] null_blk():1 [ 2639.638339] bfq_insert_request.isra.0+0x204/0x270 [ 2639.642508] null_blk():1 [ 2639.645114] bfq_insert_requests+0x70/0x90 [ 2639.649890] null_blk():1 [ 2639.652497] blk_mq_sched_insert_requests+0xac/0x250 [ 2639.656579] null_blk():1 [ 2639.659186] blk_mq_flush_plug_list+0x110/0x2cc [ 2639.664136] null_blk():1 [ 2639.666742] __blk_flush_plug+0x100/0x140 [ 2639.671258] scsi_debug():1 [ 2639.673864] blk_finish_plug+0x40/0x25c [ 2639.677860] scsi_debug():1 [ 2639.680640] __iomap_dio_rw+0x1b4/0x500 [ 2639.684461] ipmi_si():1 [ 2639.687241] iomap_dio_rw+0x18/0x50 [ 2639.691063] ipmi_si():1 [ 2639.693582] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2639.697057] [last unloaded: nvmet] [ 2639.699576] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2639.704873] [ 2639.708348] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2639.713038] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2639.714515] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2639.719552] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2639.727367] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2639.732490] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2639.741781] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2639.746904] pc : __sbq_wake_up+0x58/0x1f0 [ 2639.753851] process_one_work+0x1ec/0x470 [ 2639.759235] lr : sbitmap_queue_clear+0x74/0xc0 [ 2639.763230] worker_thread+0x74/0x410 [ 2639.767226] sp : ffff8000088438b0 [ 2639.771655] kthread+0xec/0x100 [ 2639.775303] x29: ffff8000088438b0 [ 2639.778604] ret_from_fork+0x10/0x20 [ 2639.781731] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2639.793891] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2639.801012] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2639.808134] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2639.815256] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2639.822378] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2639.829500] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2639.836621] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2639.843743] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2639.850865] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2639.857987] Call trace: [ 2639.860420] __sbq_wake_up+0x58/0x1f0 [ 2639.864069] sbitmap_queue_clear+0x74/0xc0 [ 2639.868152] blk_mq_put_tag+0x44/0x70 [ 2639.871801] __blk_mq_free_request+0x88/0xf0 [ 2639.876058] blk_mq_free_request+0xc0/0x160 [ 2639.880228] __blk_mq_end_request+0x78/0x150 [ 2639.884485] scsi_end_request+0x114/0x260 [ 2639.888481] scsi_io_completion+0x64/0x174 [ 2639.892564] scsi_finish_command+0xe0/0x120 [ 2639.896735] scsi_complete+0x8c/0x1c4 [ 2639.900384] blk_complete_reqs+0x5c/0x74 [ 2639.904293] blk_done_softirq+0x38/0x4c [ 2639.908116] __do_softirq+0x168/0x418 [ 2639.911765] __irq_exit_rcu+0x120/0x170 [ 2639.915588] irq_exit_rcu+0x18/0x24 [ 2639.919063] el1_interrupt+0x38/0x70 [ 2639.922625] el1h_64_irq_handler+0x18/0x24 [ 2639.926708] el1h_64_irq+0x68/0x6c [ 2639.930097] cpuidle_enter_state+0xc4/0x4e0 [ 2639.934266] cpuidle_enter+0x40/0x60 [ 2639.937829] cpuidle_idle_call+0x118/0x1b0 [ 2639.941912] do_idle+0xac/0x100 [ 2639.945041] cpu_startup_entry+0x30/0x34 [ 2639.948950] secondary_start_kernel+0xd8/0xf0 [ 2639.953294] __secondary_switched+0xb0/0xb4 [ 2648.707097] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2648.712829] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=110247 [ 2648.722036] (t=222382 jiffies g=36445 q=65125 ncpus=128) [ 2648.727421] Task dump for CPU 8: [ 2648.730636] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2648.740536] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2648.746704] Call trace: [ 2648.749138] __switch_to+0xd4/0x100 [ 2648.752614] 0xffff6c0200000080 [ 2648.755743] Task dump for CPU 68: [ 2648.759044] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2648.768944] Call trace: [ 2648.771377] dump_backtrace+0xe8/0x130 [ 2648.775113] show_stack+0x20/0x60 [ 2648.778414] sched_show_task+0x160/0x190 [ 2648.782324] dump_cpu_task+0x4c/0x78 [ 2648.785887] rcu_dump_cpu_stacks+0xc4/0x150 [ 2648.790059] print_cpu_stall+0x180/0x210 [ 2648.793969] check_cpu_stall+0x20c/0x2b4 [ 2648.797878] rcu_pending+0x4c/0x190 [ 2648.801354] rcu_sched_clock_irq+0x80/0x204 [ 2648.805524] update_process_times+0x7c/0xc0 [ 2648.809694] tick_sched_handle+0x38/0x74 [ 2648.813604] tick_sched_timer+0x60/0xc0 [ 2648.817427] __hrtimer_run_queues+0x170/0x300 [ 2648.821771] hrtimer_interrupt+0xfc/0x260 [ 2648.825767] arch_timer_handler_phys+0x34/0x50 [ 2648.830198] handle_percpu_devid_irq+0xac/0x240 [ 2648.834716] generic_handle_domain_irq+0x34/0x50 [ 2648.839320] gic_handle_irq+0x5c/0xe0 [ 2648.842969] call_on_irq_stack+0x2c/0x38 [ 2648.846879] do_interrupt_handler+0xdc/0xe0 [ 2648.851049] el1_interrupt+0x34/0x70 [ 2648.854612] el1h_64_irq_handler+0x18/0x24 [ 2648.858695] el1h_64_irq+0x68/0x6c [ 2648.862083] __sbq_wake_up+0x20/0x1f0 [ 2648.865732] sbitmap_queue_clear+0x74/0xc0 [ 2648.869815] blk_mq_put_tag+0x44/0x70 [ 2648.873465] __blk_mq_free_request+0x88/0xf0 [ 2648.877721] blk_mq_free_request+0xc0/0x160 [ 2648.881891] __blk_mq_end_request+0x78/0x150 [ 2648.886148] scsi_end_request+0x114/0x260 [ 2648.890145] scsi_io_completion+0x64/0x174 [ 2648.894228] scsi_finish_command+0xe0/0x120 [ 2648.898398] scsi_complete+0x8c/0x1c4 [ 2648.902047] blk_complete_reqs+0x5c/0x74 [ 2648.905957] blk_done_softirq+0x38/0x4c [ 2648.909779] __do_softirq+0x168/0x418 [ 2648.913428] __irq_exit_rcu+0x120/0x170 [ 2648.917251] irq_exit_rcu+0x18/0x24 [ 2648.920727] el1_interrupt+0x38/0x70 [ 2648.924289] el1h_64_irq_handler+0x18/0x24 [ 2648.928373] el1h_64_irq+0x68/0x6c [ 2648.931761] cpuidle_enter_state+0xc4/0x4e0 [ 2648.935932] cpuidle_enter+0x40/0x60 [ 2648.939494] cpuidle_idle_call+0x118/0x1b0 [ 2648.943578] do_idle+0xac/0x100 [ 2648.946706] cpu_startup_entry+0x30/0x34 [ 2648.950615] secondary_start_kernel+0xd8/0xf0 [ 2648.954959] __secondary_switched+0xb0/0xb4 [ 2663.097000] watchdog: BUG: soft lockup - CPU#8 stuck for 2086s! [kworker/8:12:19399] [ 2663.104730] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2663.104747] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2663.192536] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2663.200785] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2663.210076] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2663.216244] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2663.223191] pc : __sbq_wake_up+0x20/0x1f0 [ 2663.227188] lr : sbitmap_queue_clear+0x74/0xc0 [ 2663.231618] sp : ffff80000d79b280 [ 2663.234920] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2663.242042] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2663.249165] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2663.256287] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2663.263409] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2663.270531] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2663.277653] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2663.284775] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2663.291897] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2663.299019] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 2663.306141] Call trace: [ 2663.308574] __sbq_wake_up+0x20/0x1f0 [ 2663.312223] sbitmap_queue_clear+0x74/0xc0 [ 2663.316307] blk_mq_put_tag+0x44/0x70 [ 2663.319956] __blk_mq_free_request+0x88/0xf0 [ 2663.324213] blk_mq_free_request+0xc0/0x160 [ 2663.328383] __blk_mq_end_request+0x78/0x150 [ 2663.332640] scsi_end_request+0x114/0x260 [ 2663.336636] scsi_io_completion+0x64/0x174 [ 2663.340720] scsi_finish_command+0xe0/0x120 [ 2663.344890] scsi_complete+0x8c/0x1c4 [ 2663.348539] blk_complete_reqs+0x5c/0x74 [ 2663.352449] blk_done_softirq+0x38/0x4c [ 2663.356272] __do_softirq+0x168/0x418 [ 2663.359921] __irq_exit_rcu+0x120/0x170 [ 2663.363744] irq_exit_rcu+0x18/0x24 [ 2663.367219] el1_interrupt+0x38/0x70 [ 2663.370781] el1h_64_irq_handler+0x18/0x24 [ 2663.374864] el1h_64_irq+0x68/0x6c [ 2663.378253] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2663.383031] __wake_up_common_lock+0x84/0xac [ 2663.387289] __wake_up+0x20/0x30 [ 2663.390505] __sbq_wake_up+0x13c/0x1f0 [ 2663.394241] sbitmap_queue_clear+0x74/0xc0 [ 2663.398325] blk_mq_put_tag+0x44/0x70 [ 2663.401974] __blk_mq_free_request+0x88/0xf0 [ 2663.406231] blk_mq_free_request+0xc0/0x160 [ 2663.410401] bfq_insert_request.isra.0+0x204/0x270 [ 2663.415179] bfq_insert_requests+0x70/0x90 [ 2663.419263] blk_mq_sched_insert_requests+0xac/0x250 [ 2663.424215] blk_mq_flush_plug_list+0x110/0x2cc [ 2663.428733] __blk_flush_plug+0x100/0x140 [ 2663.432731] blk_finish_plug+0x40/0x25c [ 2663.436554] __iomap_dio_rw+0x1b4/0x500 [ 2663.440377] iomap_dio_rw+0x18/0x50 [ 2663.443852] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2663.449203] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2663.453945] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2663.458989] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2663.464119] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2663.469248] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2663.474635] process_one_work+0x1ec/0x470 [ 2663.478631] worker_thread+0x74/0x410 [ 2663.482281] kthread+0xec/0x100 [ 2663.485410] ret_from_fork+0x10/0x20 [ 2668.516974] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 221757 jiffies s: 2093 root: 0x11/. [ 2668.528712] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2668.537921] Task dump for CPU 8: [ 2668.541136] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2668.551037] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2668.557207] Call trace: [ 2668.559640] __switch_to+0xd4/0x100 [ 2668.563116] 0xffff6c0200000080 [ 2668.566245] Task dump for CPU 68: [ 2668.569548] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2668.579449] Call trace: [ 2668.581883] __switch_to+0xd4/0x100 [ 2668.585359] 0x0 [ 2675.206919] watchdog: BUG: soft lockup - CPU#68 stuck for 2097s! [swapper/68:0] [ 2675.214215] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2675.214232] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2675.302019] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2675.309834] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2675.319126] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2675.326073] pc : __sbq_wake_up+0x84/0x1f0 [ 2675.330069] lr : sbitmap_queue_clear+0x74/0xc0 [ 2675.334500] sp : ffff8000088438b0 [ 2675.337801] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2675.344923] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2675.352045] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2675.359167] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2675.366289] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2675.373411] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2675.380533] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2675.387655] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2675.394777] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2675.401899] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2675.409021] Call trace: [ 2675.411454] __sbq_wake_up+0x84/0x1f0 [ 2675.415103] sbitmap_queue_clear+0x74/0xc0 [ 2675.419186] blk_mq_put_tag+0x44/0x70 [ 2675.422835] __blk_mq_free_request+0x88/0xf0 [ 2675.427092] blk_mq_free_request+0xc0/0x160 [ 2675.431262] __blk_mq_end_request+0x78/0x150 [ 2675.435519] scsi_end_request+0x114/0x260 [ 2675.439515] scsi_io_completion+0x64/0x174 [ 2675.443598] scsi_finish_command+0xe0/0x120 [ 2675.447769] scsi_complete+0x8c/0x1c4 [ 2675.451418] blk_complete_reqs+0x5c/0x74 [ 2675.455327] blk_done_softirq+0x38/0x4c [ 2675.459150] __do_softirq+0x168/0x418 [ 2675.462798] __irq_exit_rcu+0x120/0x170 [ 2675.466621] irq_exit_rcu+0x18/0x24 [ 2675.470097] el1_interrupt+0x38/0x70 [ 2675.473659] el1h_64_irq_handler+0x18/0x24 [ 2675.477742] el1h_64_irq+0x68/0x6c [ 2675.481130] cpuidle_enter_state+0xc4/0x4e0 [ 2675.485300] cpuidle_enter+0x40/0x60 [ 2675.488863] cpuidle_idle_call+0x118/0x1b0 [ 2675.492946] do_idle+0xac/0x100 [ 2675.496075] cpu_startup_entry+0x30/0x34 [ 2675.499984] secondary_start_kernel+0xd8/0xf0 [ 2675.504329] __secondary_switched+0xb0/0xb4 [ 2687.096840] watchdog: BUG: soft lockup - CPU#8 stuck for 2108s! [kworker/8:12:19399] [ 2687.104569] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2687.104587] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2687.192374] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2687.200624] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2687.209914] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2687.216082] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2687.223030] pc : __sbq_wake_up+0x84/0x1f0 [ 2687.227028] lr : sbitmap_queue_clear+0x74/0xc0 [ 2687.231460] sp : ffff80000d79b280 [ 2687.234761] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2687.241884] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2687.249006] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2687.256128] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2687.263250] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2687.270372] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2687.277494] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2687.284616] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2687.291738] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2687.298860] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2687.305982] Call trace: [ 2687.308415] __sbq_wake_up+0x84/0x1f0 [ 2687.312064] sbitmap_queue_clear+0x74/0xc0 [ 2687.316148] blk_mq_put_tag+0x44/0x70 [ 2687.319797] __blk_mq_free_request+0x88/0xf0 [ 2687.324054] blk_mq_free_request+0xc0/0x160 [ 2687.328224] __blk_mq_end_request+0x78/0x150 [ 2687.332481] scsi_end_request+0x114/0x260 [ 2687.336477] scsi_io_completion+0x64/0x174 [ 2687.340560] scsi_finish_command+0xe0/0x120 [ 2687.344731] scsi_complete+0x8c/0x1c4 [ 2687.348381] blk_complete_reqs+0x5c/0x74 [ 2687.352290] blk_done_softirq+0x38/0x4c [ 2687.356113] __do_softirq+0x168/0x418 [ 2687.359761] __irq_exit_rcu+0x120/0x170 [ 2687.363584] irq_exit_rcu+0x18/0x24 [ 2687.367059] el1_interrupt+0x38/0x70 [ 2687.370622] el1h_64_irq_handler+0x18/0x24 [ 2687.374705] el1h_64_irq+0x68/0x6c [ 2687.378094] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2687.382872] __wake_up_common_lock+0x84/0xac [ 2687.387129] __wake_up+0x20/0x30 [ 2687.390344] __sbq_wake_up+0x13c/0x1f0 [ 2687.394080] sbitmap_queue_clear+0x74/0xc0 [ 2687.398164] blk_mq_put_tag+0x44/0x70 [ 2687.401814] __blk_mq_free_request+0x88/0xf0 [ 2687.406070] blk_mq_free_request+0xc0/0x160 [ 2687.410240] bfq_insert_request.isra.0+0x204/0x270 [ 2687.415018] bfq_insert_requests+0x70/0x90 [ 2687.419102] blk_mq_sched_insert_requests+0xac/0x250 [ 2687.424054] blk_mq_flush_plug_list+0x110/0x2cc [ 2687.428571] __blk_flush_plug+0x100/0x140 [ 2687.432568] blk_finish_plug+0x40/0x25c [ 2687.436391] __iomap_dio_rw+0x1b4/0x500 [ 2687.440214] iomap_dio_rw+0x18/0x50 [ 2687.443689] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2687.449039] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2687.453782] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2687.458826] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2687.463955] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2687.469085] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2687.474471] process_one_work+0x1ec/0x470 [ 2687.478468] worker_thread+0x74/0x410 [ 2687.482117] kthread+0xec/0x100 [ 2687.485245] ret_from_fork+0x10/0x20 [ 2699.206760] watchdog: BUG: soft lockup - CPU#68 stuck for 2119s! [swapper/68:0] [ 2699.214055] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2699.214072] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2699.301859] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2699.309674] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2699.318965] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2699.325913] pc : __sbq_wake_up+0x58/0x1f0 [ 2699.329910] lr : sbitmap_queue_clear+0x74/0xc0 [ 2699.334341] sp : ffff8000088438b0 [ 2699.337642] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2699.344764] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2699.351886] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2699.359008] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2699.366130] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2699.373252] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2699.380374] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2699.387496] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2699.394618] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2699.401740] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2699.408861] Call trace: [ 2699.411294] __sbq_wake_up+0x58/0x1f0 [ 2699.414943] sbitmap_queue_clear+0x74/0xc0 [ 2699.419027] blk_mq_put_tag+0x44/0x70 [ 2699.422676] __blk_mq_free_request+0x88/0xf0 [ 2699.426933] blk_mq_free_request+0xc0/0x160 [ 2699.431103] __blk_mq_end_request+0x78/0x150 [ 2699.435360] scsi_end_request+0x114/0x260 [ 2699.439356] scsi_io_completion+0x64/0x174 [ 2699.443439] scsi_finish_command+0xe0/0x120 [ 2699.447610] scsi_complete+0x8c/0x1c4 [ 2699.451259] blk_complete_reqs+0x5c/0x74 [ 2699.455168] blk_done_softirq+0x38/0x4c [ 2699.458991] __do_softirq+0x168/0x418 [ 2699.462640] __irq_exit_rcu+0x120/0x170 [ 2699.466463] irq_exit_rcu+0x18/0x24 [ 2699.469938] el1_interrupt+0x38/0x70 [ 2699.473500] el1h_64_irq_handler+0x18/0x24 [ 2699.477583] el1h_64_irq+0x68/0x6c [ 2699.480972] cpuidle_enter_state+0xc4/0x4e0 [ 2699.485142] cpuidle_enter+0x40/0x60 [ 2699.488704] cpuidle_idle_call+0x118/0x1b0 [ 2699.492788] do_idle+0xac/0x100 [ 2699.495917] cpu_startup_entry+0x30/0x34 [ 2699.499826] secondary_start_kernel+0xd8/0xf0 [ 2699.504170] __secondary_switched+0xb0/0xb4 [ 2711.096681] watchdog: BUG: soft lockup - CPU#8 stuck for 2130s! [kworker/8:12:19399] [ 2711.104410] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2711.104427] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2711.192215] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2711.200465] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2711.209755] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2711.215923] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2711.222871] pc : __sbq_wake_up+0x58/0x1f0 [ 2711.226868] lr : sbitmap_queue_clear+0x74/0xc0 [ 2711.231298] sp : ffff80000d79b280 [ 2711.234600] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2711.241722] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2711.248844] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2711.255967] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2711.263089] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2711.270211] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2711.277333] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2711.284455] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2711.291577] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2711.298699] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2711.305821] Call trace: [ 2711.308253] __sbq_wake_up+0x58/0x1f0 [ 2711.311903] sbitmap_queue_clear+0x74/0xc0 [ 2711.315987] blk_mq_put_tag+0x44/0x70 [ 2711.319637] __blk_mq_free_request+0x88/0xf0 [ 2711.323894] blk_mq_free_request+0xc0/0x160 [ 2711.328063] __blk_mq_end_request+0x78/0x150 [ 2711.332321] scsi_end_request+0x114/0x260 [ 2711.336317] scsi_io_completion+0x64/0x174 [ 2711.340400] scsi_finish_command+0xe0/0x120 [ 2711.344570] scsi_complete+0x8c/0x1c4 [ 2711.348220] blk_complete_reqs+0x5c/0x74 [ 2711.352129] blk_done_softirq+0x38/0x4c [ 2711.355952] __do_softirq+0x168/0x418 [ 2711.359601] __irq_exit_rcu+0x120/0x170 [ 2711.363423] irq_exit_rcu+0x18/0x24 [ 2711.366899] el1_interrupt+0x38/0x70 [ 2711.370461] el1h_64_irq_handler+0x18/0x24 [ 2711.374544] el1h_64_irq+0x68/0x6c [ 2711.377933] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2711.382711] __wake_up_common_lock+0x84/0xac [ 2711.386968] __wake_up+0x20/0x30 [ 2711.390184] __sbq_wake_up+0x13c/0x1f0 [ 2711.393920] sbitmap_queue_clear+0x74/0xc0 [ 2711.398003] blk_mq_put_tag+0x44/0x70 [ 2711.401653] __blk_mq_free_request+0x88/0xf0 [ 2711.405910] blk_mq_free_request+0xc0/0x160 [ 2711.410080] bfq_insert_request.isra.0+0x204/0x270 [ 2711.414858] bfq_insert_requests+0x70/0x90 [ 2711.418941] blk_mq_sched_insert_requests+0xac/0x250 [ 2711.423893] blk_mq_flush_plug_list+0x110/0x2cc [ 2711.428411] __blk_flush_plug+0x100/0x140 [ 2711.432407] blk_finish_plug+0x40/0x25c [ 2711.436230] __iomap_dio_rw+0x1b4/0x500 [ 2711.440055] iomap_dio_rw+0x18/0x50 [ 2711.443530] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2711.448881] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2711.453623] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2711.458668] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2711.463798] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2711.468928] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2711.474314] process_one_work+0x1ec/0x470 [ 2711.478310] worker_thread+0x74/0x410 [ 2711.481959] kthread+0xec/0x100 [ 2711.485088] ret_from_fork+0x10/0x20 [ 2723.206601] watchdog: BUG: soft lockup - CPU#68 stuck for 2142s! [swapper/68:0] [ 2723.213897] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2723.213914] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2723.301701] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2723.309517] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2723.318808] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2723.325755] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2723.329839] lr : sbitmap_queue_clear+0x74/0xc0 [ 2723.334269] sp : ffff8000088438b0 [ 2723.337570] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2723.344692] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2723.351815] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2723.358936] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2723.366059] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2723.373181] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2723.380304] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2723.387425] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2723.394547] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2723.401669] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2723.408791] Call trace: [ 2723.411224] __sbq_wake_up+0x1c8/0x1f0 [ 2723.414960] sbitmap_queue_clear+0x74/0xc0 [ 2723.419043] blk_mq_put_tag+0x44/0x70 [ 2723.422693] __blk_mq_free_request+0x88/0xf0 [ 2723.426949] blk_mq_free_request+0xc0/0x160 [ 2723.431119] __blk_mq_end_request+0x78/0x150 [ 2723.435376] scsi_end_request+0x114/0x260 [ 2723.439372] scsi_io_completion+0x64/0x174 [ 2723.443455] scsi_finish_command+0xe0/0x120 [ 2723.447626] scsi_complete+0x8c/0x1c4 [ 2723.451275] blk_complete_reqs+0x5c/0x74 [ 2723.455185] blk_done_softirq+0x38/0x4c [ 2723.459008] __do_softirq+0x168/0x418 [ 2723.462657] __irq_exit_rcu+0x120/0x170 [ 2723.466479] irq_exit_rcu+0x18/0x24 [ 2723.469955] el1_interrupt+0x38/0x70 [ 2723.473517] el1h_64_irq_handler+0x18/0x24 [ 2723.477600] el1h_64_irq+0x68/0x6c [ 2723.480989] cpuidle_enter_state+0xc4/0x4e0 [ 2723.485159] cpuidle_enter+0x40/0x60 [ 2723.488722] cpuidle_idle_call+0x118/0x1b0 [ 2723.492805] do_idle+0xac/0x100 [ 2723.495935] cpu_startup_entry+0x30/0x34 [ 2723.499844] secondary_start_kernel+0xd8/0xf0 [ 2723.504188] __secondary_switched+0xb0/0xb4 [ 2735.076533] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 228413 jiffies s: 2093 root: 0x11/. [ 2735.088272] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2735.096523] watchdog: BUG: soft lockup - CPU#8 stuck for 2153s! [kworker/8:12:19399] [ 2735.097481] Task dump for CPU 8: [ 2735.105207] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 [ 2735.108423] task:kworker/8:12 state:R [ 2735.108424] hfsplus hfs [ 2735.114764] running task [ 2735.118673] isofs nvme_core nvme_common loop tls rfkill sunrpc [ 2735.121195] stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2735.124235] vfat [ 2735.130142] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2735.136045] fat [ 2735.137960] [ 2735.144124] acpi_ipmi [ 2735.145949] Call trace: [ 2735.147427] ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler [ 2735.149775] __switch_to+0xd4/0x100 [ 2735.152208] arm_cmn [ 2735.158549] 0xffff6c0200000080 [ 2735.162021] arm_dmc620_pmu [ 2735.164194] Task dump for CPU 68: [ 2735.167321] arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram [ 2735.170103] task:swapper/68 state:R [ 2735.173403] xfs [ 2735.178616] running task [ 2735.182523] crct10dif_ce [ 2735.184348] stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2735.187388] polyval_ce ast polyval_generic drm_vram_helper ghash_ce [ 2735.189997] Call trace: [ 2735.195901] drm_ttm_helper [ 2735.202242] __switch_to+0xd4/0x100 [ 2735.204673] sbsa_gwdt [ 2735.207455] 0x0 [ 2735.210927] ttm xgene_hwmon [ 2735.215099] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2735.271811] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2735.280061] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2735.289352] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2735.295520] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2735.302467] pc : __sbq_wake_up+0x58/0x1f0 [ 2735.306464] lr : sbitmap_queue_clear+0x74/0xc0 [ 2735.310895] sp : ffff80000d79b280 [ 2735.314196] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2735.321319] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2735.328441] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2735.335563] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2735.342685] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2735.349808] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2735.356930] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2735.364053] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2735.371175] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2735.378297] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2735.385419] Call trace: [ 2735.387852] __sbq_wake_up+0x58/0x1f0 [ 2735.391502] sbitmap_queue_clear+0x74/0xc0 [ 2735.395585] blk_mq_put_tag+0x44/0x70 [ 2735.399234] __blk_mq_free_request+0x88/0xf0 [ 2735.403492] blk_mq_free_request+0xc0/0x160 [ 2735.407662] __blk_mq_end_request+0x78/0x150 [ 2735.411919] scsi_end_request+0x114/0x260 [ 2735.415916] scsi_io_completion+0x64/0x174 [ 2735.419999] scsi_finish_command+0xe0/0x120 [ 2735.424169] scsi_complete+0x8c/0x1c4 [ 2735.427818] blk_complete_reqs+0x5c/0x74 [ 2735.431728] blk_done_softirq+0x38/0x4c [ 2735.435551] __do_softirq+0x168/0x418 [ 2735.439200] __irq_exit_rcu+0x120/0x170 [ 2735.443022] irq_exit_rcu+0x18/0x24 [ 2735.446498] el1_interrupt+0x38/0x70 [ 2735.450060] el1h_64_irq_handler+0x18/0x24 [ 2735.454143] el1h_64_irq+0x68/0x6c [ 2735.457532] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2735.462309] __wake_up_common_lock+0x84/0xac [ 2735.466566] __wake_up+0x20/0x30 [ 2735.469782] __sbq_wake_up+0x13c/0x1f0 [ 2735.473518] sbitmap_queue_clear+0x74/0xc0 [ 2735.477601] blk_mq_put_tag+0x44/0x70 [ 2735.481251] __blk_mq_free_request+0x88/0xf0 [ 2735.485507] blk_mq_free_request+0xc0/0x160 [ 2735.489677] bfq_insert_request.isra.0+0x204/0x270 [ 2735.494455] bfq_insert_requests+0x70/0x90 [ 2735.498539] blk_mq_sched_insert_requests+0xac/0x250 [ 2735.503491] blk_mq_flush_plug_list+0x110/0x2cc [ 2735.508008] __blk_flush_plug+0x100/0x140 [ 2735.512005] blk_finish_plug+0x40/0x25c [ 2735.515828] __iomap_dio_rw+0x1b4/0x500 [ 2735.519651] iomap_dio_rw+0x18/0x50 [ 2735.523126] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2735.528477] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2735.533220] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2735.538264] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2735.543393] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2735.548523] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2735.553909] process_one_work+0x1ec/0x470 [ 2735.557906] worker_thread+0x74/0x410 [ 2735.561555] kthread+0xec/0x100 [ 2735.564684] ret_from_fork+0x10/0x20 [ 2747.206443] watchdog: BUG: soft lockup - CPU#68 stuck for 2164s! [swapper/68:0] [ 2747.213739] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2747.213756] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2747.301543] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2747.309359] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2747.318650] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2747.325597] pc : __sbq_wake_up+0x24/0x1f0 [ 2747.329594] lr : sbitmap_queue_clear+0x74/0xc0 [ 2747.334024] sp : ffff8000088438b0 [ 2747.337325] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2747.344447] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2747.351569] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2747.358691] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2747.365813] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2747.372934] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2747.380056] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2747.387178] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2747.394300] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2747.401422] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 2747.408544] Call trace: [ 2747.410977] __sbq_wake_up+0x24/0x1f0 [ 2747.414626] sbitmap_queue_clear+0x74/0xc0 [ 2747.418709] blk_mq_put_tag+0x44/0x70 [ 2747.422359] __blk_mq_free_request+0x88/0xf0 [ 2747.426615] blk_mq_free_request+0xc0/0x160 [ 2747.430785] __blk_mq_end_request+0x78/0x150 [ 2747.435043] scsi_end_request+0x114/0x260 [ 2747.439040] scsi_io_completion+0x64/0x174 [ 2747.443123] scsi_finish_command+0xe0/0x120 [ 2747.447293] scsi_complete+0x8c/0x1c4 [ 2747.450942] blk_complete_reqs+0x5c/0x74 [ 2747.454852] blk_done_softirq+0x38/0x4c [ 2747.458675] __do_softirq+0x168/0x418 [ 2747.462324] __irq_exit_rcu+0x120/0x170 [ 2747.466146] irq_exit_rcu+0x18/0x24 [ 2747.469622] el1_interrupt+0x38/0x70 [ 2747.473184] el1h_64_irq_handler+0x18/0x24 [ 2747.477267] el1h_64_irq+0x68/0x6c [ 2747.480655] cpuidle_enter_state+0xc4/0x4e0 [ 2747.484825] cpuidle_enter+0x40/0x60 [ 2747.488388] cpuidle_idle_call+0x118/0x1b0 [ 2747.492472] do_idle+0xac/0x100 [ 2747.495600] cpu_startup_entry+0x30/0x34 [ 2747.499509] secondary_start_kernel+0xd8/0xf0 [ 2747.503854] __secondary_switched+0xb0/0xb4 [ 2759.096365] watchdog: BUG: soft lockup - CPU#8 stuck for 2175s! [kworker/8:12:19399] [ 2759.104094] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2759.104111] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2759.191899] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2759.200149] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2759.209440] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2759.215608] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2759.222556] pc : __sbq_wake_up+0x84/0x1f0 [ 2759.226553] lr : sbitmap_queue_clear+0x74/0xc0 [ 2759.230984] sp : ffff80000d79b280 [ 2759.234285] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2759.241407] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2759.248529] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2759.255651] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2759.262773] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2759.269895] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2759.277017] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2759.284139] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2759.291261] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2759.298383] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2759.305505] Call trace: [ 2759.307938] __sbq_wake_up+0x84/0x1f0 [ 2759.311587] sbitmap_queue_clear+0x74/0xc0 [ 2759.315670] blk_mq_put_tag+0x44/0x70 [ 2759.319319] __blk_mq_free_request+0x88/0xf0 [ 2759.323576] blk_mq_free_request+0xc0/0x160 [ 2759.327746] __blk_mq_end_request+0x78/0x150 [ 2759.332003] scsi_end_request+0x114/0x260 [ 2759.335999] scsi_io_completion+0x64/0x174 [ 2759.340083] scsi_finish_command+0xe0/0x120 [ 2759.344253] scsi_complete+0x8c/0x1c4 [ 2759.347902] blk_complete_reqs+0x5c/0x74 [ 2759.351812] blk_done_softirq+0x38/0x4c [ 2759.355634] __do_softirq+0x168/0x418 [ 2759.359283] __irq_exit_rcu+0x120/0x170 [ 2759.363106] irq_exit_rcu+0x18/0x24 [ 2759.366581] el1_interrupt+0x38/0x70 [ 2759.370143] el1h_64_irq_handler+0x18/0x24 [ 2759.374226] el1h_64_irq+0x68/0x6c [ 2759.377615] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2759.382393] __wake_up_common_lock+0x84/0xac [ 2759.386650] __wake_up+0x20/0x30 [ 2759.389865] __sbq_wake_up+0x13c/0x1f0 [ 2759.393602] sbitmap_queue_clear+0x74/0xc0 [ 2759.397686] blk_mq_put_tag+0x44/0x70 [ 2759.401335] __blk_mq_free_request+0x88/0xf0 [ 2759.405592] blk_mq_free_request+0xc0/0x160 [ 2759.409762] bfq_insert_request.isra.0+0x204/0x270 [ 2759.414540] bfq_insert_requests+0x70/0x90 [ 2759.418624] blk_mq_sched_insert_requests+0xac/0x250 [ 2759.423576] blk_mq_flush_plug_list+0x110/0x2cc [ 2759.428094] __blk_flush_plug+0x100/0x140 [ 2759.432091] blk_finish_plug+0x40/0x25c [ 2759.435913] __iomap_dio_rw+0x1b4/0x500 [ 2759.439736] iomap_dio_rw+0x18/0x50 [ 2759.443212] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2759.448562] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2759.453304] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2759.458348] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2759.463478] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2759.468608] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2759.473994] process_one_work+0x1ec/0x470 [ 2759.477991] worker_thread+0x74/0x410 [ 2759.481639] kthread+0xec/0x100 [ 2759.484768] ret_from_fork+0x10/0x20 [ 2771.206285] watchdog: BUG: soft lockup - CPU#68 stuck for 2186s! [swapper/68:0] [ 2771.213581] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2771.213598] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2771.301386] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2771.309202] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2771.318493] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2771.325441] pc : __sbq_wake_up+0x84/0x1f0 [ 2771.329437] lr : sbitmap_queue_clear+0x74/0xc0 [ 2771.333868] sp : ffff8000088438b0 [ 2771.337169] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2771.344292] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2771.351414] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2771.358536] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2771.365658] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2771.372781] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2771.379903] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2771.387024] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2771.394147] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2771.401269] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2771.408390] Call trace: [ 2771.410823] __sbq_wake_up+0x84/0x1f0 [ 2771.414472] sbitmap_queue_clear+0x74/0xc0 [ 2771.418556] blk_mq_put_tag+0x44/0x70 [ 2771.422205] __blk_mq_free_request+0x88/0xf0 [ 2771.426462] blk_mq_free_request+0xc0/0x160 [ 2771.430632] __blk_mq_end_request+0x78/0x150 [ 2771.434889] scsi_end_request+0x114/0x260 [ 2771.438886] scsi_io_completion+0x64/0x174 [ 2771.442969] scsi_finish_command+0xe0/0x120 [ 2771.447139] scsi_complete+0x8c/0x1c4 [ 2771.450788] blk_complete_reqs+0x5c/0x74 [ 2771.454698] blk_done_softirq+0x38/0x4c [ 2771.458520] __do_softirq+0x168/0x418 [ 2771.462169] __irq_exit_rcu+0x120/0x170 [ 2771.465992] irq_exit_rcu+0x18/0x24 [ 2771.469467] el1_interrupt+0x38/0x70 [ 2771.473029] el1h_64_irq_handler+0x18/0x24 [ 2771.477112] el1h_64_irq+0x68/0x6c [ 2771.480501] cpuidle_enter_state+0xc4/0x4e0 [ 2771.484671] cpuidle_enter+0x40/0x60 [ 2771.488233] cpuidle_idle_call+0x118/0x1b0 [ 2771.492316] do_idle+0xac/0x100 [ 2771.495445] cpu_startup_entry+0x30/0x34 [ 2771.499354] secondary_start_kernel+0xd8/0xf0 [ 2771.503698] __secondary_switched+0xb0/0xb4 [ 2783.096207] watchdog: BUG: soft lockup - CPU#8 stuck for 2197s! [kworker/8:12:19399] [ 2783.103936] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2783.103953] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2783.191741] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2783.199991] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2783.209282] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2783.215450] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2783.222398] pc : __sbq_wake_up+0x38/0x1f0 [ 2783.226395] lr : sbitmap_queue_clear+0x74/0xc0 [ 2783.230826] sp : ffff80000d79b280 [ 2783.234127] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2783.241249] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2783.248372] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2783.255494] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2783.262616] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2783.269739] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2783.276862] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2783.283984] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2783.291106] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2783.298228] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 2783.305350] Call trace: [ 2783.307783] __sbq_wake_up+0x38/0x1f0 [ 2783.311433] sbitmap_queue_clear+0x74/0xc0 [ 2783.315516] blk_mq_put_tag+0x44/0x70 [ 2783.319166] __blk_mq_free_request+0x88/0xf0 [ 2783.323422] blk_mq_free_request+0xc0/0x160 [ 2783.327592] __blk_mq_end_request+0x78/0x150 [ 2783.331850] scsi_end_request+0x114/0x260 [ 2783.335846] scsi_io_completion+0x64/0x174 [ 2783.339929] scsi_finish_command+0xe0/0x120 [ 2783.344099] scsi_complete+0x8c/0x1c4 [ 2783.347749] blk_complete_reqs+0x5c/0x74 [ 2783.351658] blk_done_softirq+0x38/0x4c [ 2783.355481] __do_softirq+0x168/0x418 [ 2783.359130] __irq_exit_rcu+0x120/0x170 [ 2783.362952] irq_exit_rcu+0x18/0x24 [ 2783.366428] el1_interrupt+0x38/0x70 [ 2783.369990] el1h_64_irq_handler+0x18/0x24 [ 2783.374074] el1h_64_irq+0x68/0x6c [ 2783.377462] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2783.382240] __wake_up_common_lock+0x84/0xac [ 2783.386497] __wake_up+0x20/0x30 [ 2783.389713] __sbq_wake_up+0x13c/0x1f0 [ 2783.393449] sbitmap_queue_clear+0x74/0xc0 [ 2783.397532] blk_mq_put_tag+0x44/0x70 [ 2783.401182] __blk_mq_free_request+0x88/0xf0 [ 2783.405439] blk_mq_free_request+0xc0/0x160 [ 2783.409609] bfq_insert_request.isra.0+0x204/0x270 [ 2783.414387] bfq_insert_requests+0x70/0x90 [ 2783.418470] blk_mq_sched_insert_requests+0xac/0x250 [ 2783.423422] blk_mq_flush_plug_list+0x110/0x2cc [ 2783.427940] __blk_flush_plug+0x100/0x140 [ 2783.431936] blk_finish_plug+0x40/0x25c [ 2783.435760] __iomap_dio_rw+0x1b4/0x500 [ 2783.439583] iomap_dio_rw+0x18/0x50 [ 2783.443058] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2783.448409] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2783.453152] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2783.458196] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2783.463326] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2783.468456] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2783.473842] process_one_work+0x1ec/0x470 [ 2783.477838] worker_thread+0x74/0x410 [ 2783.481487] kthread+0xec/0x100 [ 2783.484615] ret_from_fork+0x10/0x20 [ 2795.206127] watchdog: BUG: soft lockup - CPU#68 stuck for 2209s! [swapper/68:0] [ 2795.213422] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2795.213439] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2795.301226] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2795.309042] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2795.318333] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2795.325280] pc : __sbq_wake_up+0x9c/0x1f0 [ 2795.329277] lr : sbitmap_queue_clear+0x74/0xc0 [ 2795.333707] sp : ffff8000088438b0 [ 2795.337008] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2795.344130] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2795.351252] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2795.358374] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2795.365496] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2795.372618] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2795.379740] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2795.386862] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2795.393984] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2795.401107] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000007 [ 2795.408229] Call trace: [ 2795.410662] __sbq_wake_up+0x9c/0x1f0 [ 2795.414312] sbitmap_queue_clear+0x74/0xc0 [ 2795.418395] blk_mq_put_tag+0x44/0x70 [ 2795.422044] __blk_mq_free_request+0x88/0xf0 [ 2795.426301] blk_mq_free_request+0xc0/0x160 [ 2795.430471] __blk_mq_end_request+0x78/0x150 [ 2795.434728] scsi_end_request+0x114/0x260 [ 2795.438725] scsi_io_completion+0x64/0x174 [ 2795.442808] scsi_finish_command+0xe0/0x120 [ 2795.446978] scsi_complete+0x8c/0x1c4 [ 2795.450627] blk_complete_reqs+0x5c/0x74 [ 2795.454536] blk_done_softirq+0x38/0x4c [ 2795.458359] __do_softirq+0x168/0x418 [ 2795.462008] __irq_exit_rcu+0x120/0x170 [ 2795.465830] irq_exit_rcu+0x18/0x24 [ 2795.469305] el1_interrupt+0x38/0x70 [ 2795.472868] el1h_64_irq_handler+0x18/0x24 [ 2795.476951] el1h_64_irq+0x68/0x6c [ 2795.480339] cpuidle_enter_state+0xc4/0x4e0 [ 2795.484509] cpuidle_enter+0x40/0x60 [ 2795.488072] cpuidle_idle_call+0x118/0x1b0 [ 2795.492155] do_idle+0xac/0x100 [ 2795.495284] cpu_startup_entry+0x30/0x34 [ 2795.499194] secondary_start_kernel+0xd8/0xf0 [ 2795.503538] __secondary_switched+0xb0/0xb4 [ 2801.636094] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 235069 jiffies s: 2093 root: 0x11/. [ 2801.647833] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2801.657042] Task dump for CPU 8: [ 2801.660256] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2801.670158] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2801.676329] Call trace: [ 2801.678762] __switch_to+0xd4/0x100 [ 2801.682239] 0xffff6c0200000080 [ 2801.685367] Task dump for CPU 68: [ 2801.688670] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2801.698571] Call trace: [ 2801.701004] __switch_to+0xd4/0x100 [ 2801.704480] 0x0 [ 2807.096049] watchdog: BUG: soft lockup - CPU#8 stuck for 2220s! [kworker/8:12:19399] [ 2807.103778] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2807.103794] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2807.191582] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2807.199832] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2807.209123] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2807.215290] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2807.222238] pc : __sbq_wake_up+0x58/0x1f0 [ 2807.226235] lr : sbitmap_queue_clear+0x74/0xc0 [ 2807.230665] sp : ffff80000d79b280 [ 2807.233967] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2807.241089] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2807.248211] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2807.255333] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2807.262455] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2807.269576] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2807.276698] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2807.283820] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2807.290942] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2807.298064] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2807.305186] Call trace: [ 2807.307619] __sbq_wake_up+0x58/0x1f0 [ 2807.311268] sbitmap_queue_clear+0x74/0xc0 [ 2807.315352] blk_mq_put_tag+0x44/0x70 [ 2807.319001] __blk_mq_free_request+0x88/0xf0 [ 2807.323258] blk_mq_free_request+0xc0/0x160 [ 2807.327428] __blk_mq_end_request+0x78/0x150 [ 2807.331685] scsi_end_request+0x114/0x260 [ 2807.335681] scsi_io_completion+0x64/0x174 [ 2807.339764] scsi_finish_command+0xe0/0x120 [ 2807.343935] scsi_complete+0x8c/0x1c4 [ 2807.347584] blk_complete_reqs+0x5c/0x74 [ 2807.351494] blk_done_softirq+0x38/0x4c [ 2807.355317] __do_softirq+0x168/0x418 [ 2807.358966] __irq_exit_rcu+0x120/0x170 [ 2807.362788] irq_exit_rcu+0x18/0x24 [ 2807.366264] el1_interrupt+0x38/0x70 [ 2807.369826] el1h_64_irq_handler+0x18/0x24 [ 2807.373909] el1h_64_irq+0x68/0x6c [ 2807.377298] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2807.382076] __wake_up_common_lock+0x84/0xac [ 2807.386333] __wake_up+0x20/0x30 [ 2807.389548] __sbq_wake_up+0x13c/0x1f0 [ 2807.393284] sbitmap_queue_clear+0x74/0xc0 [ 2807.397368] blk_mq_put_tag+0x44/0x70 [ 2807.401017] __blk_mq_free_request+0x88/0xf0 [ 2807.405274] blk_mq_free_request+0xc0/0x160 [ 2807.409444] bfq_insert_request.isra.0+0x204/0x270 [ 2807.414222] bfq_insert_requests+0x70/0x90 [ 2807.418306] blk_mq_sched_insert_requests+0xac/0x250 [ 2807.423258] blk_mq_flush_plug_list+0x110/0x2cc [ 2807.427776] __blk_flush_plug+0x100/0x140 [ 2807.431772] blk_finish_plug+0x40/0x25c [ 2807.435595] __iomap_dio_rw+0x1b4/0x500 [ 2807.439417] iomap_dio_rw+0x18/0x50 [ 2807.442893] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2807.448244] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2807.452986] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2807.458030] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2807.463160] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2807.468290] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2807.473676] process_one_work+0x1ec/0x470 [ 2807.477672] worker_thread+0x74/0x410 [ 2807.481321] kthread+0xec/0x100 [ 2807.484450] ret_from_fork+0x10/0x20 [ 2819.205969] watchdog: BUG: soft lockup - CPU#68 stuck for 2231s! [swapper/68:0] [ 2819.213264] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2819.213280] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2819.301068] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2819.308883] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2819.318174] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2819.325122] pc : __sbq_wake_up+0x84/0x1f0 [ 2819.329119] lr : sbitmap_queue_clear+0x74/0xc0 [ 2819.333549] sp : ffff8000088438b0 [ 2819.336851] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2819.343973] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2819.351095] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2819.358217] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2819.365339] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2819.372461] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2819.379583] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2819.386705] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2819.393827] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2819.400949] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2819.408071] Call trace: [ 2819.410504] __sbq_wake_up+0x84/0x1f0 [ 2819.414153] sbitmap_queue_clear+0x74/0xc0 [ 2819.418236] blk_mq_put_tag+0x44/0x70 [ 2819.421886] __blk_mq_free_request+0x88/0xf0 [ 2819.426142] blk_mq_free_request+0xc0/0x160 [ 2819.430312] __blk_mq_end_request+0x78/0x150 [ 2819.434570] scsi_end_request+0x114/0x260 [ 2819.438566] scsi_io_completion+0x64/0x174 [ 2819.442649] scsi_finish_command+0xe0/0x120 [ 2819.446819] scsi_complete+0x8c/0x1c4 [ 2819.450468] blk_complete_reqs+0x5c/0x74 [ 2819.454378] blk_done_softirq+0x38/0x4c [ 2819.458201] __do_softirq+0x168/0x418 [ 2819.461850] __irq_exit_rcu+0x120/0x170 [ 2819.465672] irq_exit_rcu+0x18/0x24 [ 2819.469147] el1_interrupt+0x38/0x70 [ 2819.472710] el1h_64_irq_handler+0x18/0x24 [ 2819.476793] el1h_64_irq+0x68/0x6c [ 2819.480181] cpuidle_enter_state+0xc4/0x4e0 [ 2819.484351] cpuidle_enter+0x40/0x60 [ 2819.487914] cpuidle_idle_call+0x118/0x1b0 [ 2819.491997] do_idle+0xac/0x100 [ 2819.495126] cpu_startup_entry+0x30/0x34 [ 2819.499035] secondary_start_kernel+0xd8/0xf0 [ 2819.503379] __secondary_switched+0xb0/0xb4 [ 2828.955905] rcu: INFO: rcu_preempt self-detected stall on CPU [ 2828.961638] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=119206 [ 2828.970760] (t=240408 jiffies g=36445 q=70489 ncpus=128) [ 2828.976145] Task dump for CPU 8: [ 2828.979359] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2828.989259] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2828.995427] Call trace: [ 2828.997860] dump_backtrace+0xe8/0x130 [ 2829.001597] show_stack+0x20/0x60 [ 2829.004898] sched_show_task+0x160/0x190 [ 2829.008808] dump_cpu_task+0x4c/0x78 [ 2829.012371] rcu_dump_cpu_stacks+0xc4/0x150 [ 2829.016542] print_cpu_stall+0x180/0x210 [ 2829.020451] check_cpu_stall+0x20c/0x2b4 [ 2829.024362] rcu_pending+0x4c/0x190 [ 2829.027837] rcu_sched_clock_irq+0x80/0x204 [ 2829.032007] update_process_times+0x7c/0xc0 [ 2829.036177] tick_sched_handle+0x38/0x74 [ 2829.040088] tick_sched_timer+0x60/0xc0 [ 2829.043911] __hrtimer_run_queues+0x170/0x300 [ 2829.048255] hrtimer_interrupt+0xfc/0x260 [ 2829.052251] arch_timer_handler_phys+0x34/0x50 [ 2829.056683] handle_percpu_devid_irq+0xac/0x240 [ 2829.061201] generic_handle_domain_irq+0x34/0x50 [ 2829.065806] gic_handle_irq+0x5c/0xe0 [ 2829.069455] call_on_irq_stack+0x2c/0x38 [ 2829.073364] do_interrupt_handler+0xdc/0xe0 [ 2829.077535] el1_interrupt+0x34/0x70 [ 2829.081097] el1h_64_irq_handler+0x18/0x24 [ 2829.085180] el1h_64_irq+0x68/0x6c [ 2829.088569] __sbq_wake_up+0x84/0x1f0 [ 2829.092218] sbitmap_queue_clear+0x74/0xc0 [ 2829.096302] blk_mq_put_tag+0x44/0x70 [ 2829.099952] __blk_mq_free_request+0x88/0xf0 [ 2829.104208] blk_mq_free_request+0xc0/0x160 [ 2829.108378] __blk_mq_end_request+0x78/0x150 [ 2829.112636] scsi_end_request+0x114/0x260 [ 2829.116632] scsi_io_completion+0x64/0x174 [ 2829.120715] scsi_finish_command+0xe0/0x120 [ 2829.124885] scsi_complete+0x8c/0x1c4 [ 2829.128534] blk_complete_reqs+0x5c/0x74 [ 2829.132444] blk_done_softirq+0x38/0x4c [ 2829.136267] __do_softirq+0x168/0x418 [ 2829.139915] __irq_exit_rcu+0x120/0x170 [ 2829.143738] irq_exit_rcu+0x18/0x24 [ 2829.147213] el1_interrupt+0x38/0x70 [ 2829.150776] el1h_64_irq_handler+0x18/0x24 [ 2829.154859] el1h_64_irq+0x68/0x6c [ 2829.158248] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2829.163026] __wake_up_common_lock+0x84/0xac [ 2829.167283] __wake_up+0x20/0x30 [ 2829.170499] __sbq_wake_up+0x13c/0x1f0 [ 2829.174235] sbitmap_queue_clear+0x74/0xc0 [ 2829.178318] blk_mq_put_tag+0x44/0x70 [ 2829.181968] __blk_mq_free_request+0x88/0xf0 [ 2829.186225] blk_mq_free_request+0xc0/0x160 [ 2829.190395] bfq_insert_request.isra.0+0x204/0x270 [ 2829.195173] bfq_insert_requests+0x70/0x90 [ 2829.199256] blk_mq_sched_insert_requests+0xac/0x250 [ 2829.204208] blk_mq_flush_plug_list+0x110/0x2cc [ 2829.208726] __blk_flush_plug+0x100/0x140 [ 2829.212722] blk_finish_plug+0x40/0x25c [ 2829.216545] __iomap_dio_rw+0x1b4/0x500 [ 2829.220368] iomap_dio_rw+0x18/0x50 [ 2829.223843] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2829.229194] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2829.233937] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2829.238981] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2829.244110] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2829.249240] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2829.254627] process_one_work+0x1ec/0x470 [ 2829.258623] worker_thread+0x74/0x410 [ 2829.262272] kthread+0xec/0x100 [ 2829.265401] ret_from_fork+0x10/0x20 [ 2829.268964] Task dump for CPU 68: [ 2829.272266] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2829.282166] Call trace: [ 2829.284599] __switch_to+0xd4/0x100 [ 2829.288075] 0x0 [ 2843.205811] watchdog: BUG: soft lockup - CPU#68 stuck for 2253s! [swapper/68:0] [ 2843.213106] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2843.213122] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2843.300910] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2843.308725] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2843.318016] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2843.324964] pc : __sbq_wake_up+0x58/0x1f0 [ 2843.328960] lr : sbitmap_queue_clear+0x74/0xc0 [ 2843.333390] sp : ffff8000088438b0 [ 2843.336692] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2843.343814] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2843.350936] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2843.358058] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2843.365180] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2843.372302] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2843.379425] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2843.386547] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2843.393668] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2843.400791] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2843.407914] Call trace: [ 2843.410347] __sbq_wake_up+0x58/0x1f0 [ 2843.413997] sbitmap_queue_clear+0x74/0xc0 [ 2843.418080] blk_mq_put_tag+0x44/0x70 [ 2843.421730] __blk_mq_free_request+0x88/0xf0 [ 2843.425986] blk_mq_free_request+0xc0/0x160 [ 2843.430156] __blk_mq_end_request+0x78/0x150 [ 2843.434413] scsi_end_request+0x114/0x260 [ 2843.438409] scsi_io_completion+0x64/0x174 [ 2843.442492] scsi_finish_command+0xe0/0x120 [ 2843.446663] scsi_complete+0x8c/0x1c4 [ 2843.450312] blk_complete_reqs+0x5c/0x74 [ 2843.454222] blk_done_softirq+0x38/0x4c [ 2843.458044] __do_softirq+0x168/0x418 [ 2843.461693] __irq_exit_rcu+0x120/0x170 [ 2843.465516] irq_exit_rcu+0x18/0x24 [ 2843.468991] el1_interrupt+0x38/0x70 [ 2843.472554] el1h_64_irq_handler+0x18/0x24 [ 2843.476637] el1h_64_irq+0x68/0x6c [ 2843.480025] cpuidle_enter_state+0xc4/0x4e0 [ 2843.484195] cpuidle_enter+0x40/0x60 [ 2843.487758] cpuidle_idle_call+0x118/0x1b0 [ 2843.491841] do_idle+0xac/0x100 [ 2843.494970] cpu_startup_entry+0x30/0x34 [ 2843.498879] secondary_start_kernel+0xd8/0xf0 [ 2843.503223] __secondary_switched+0xb0/0xb4 [ 2855.095732] watchdog: BUG: soft lockup - CPU#8 stuck for 2265s! [kworker/8:12:19399] [ 2855.103462] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2855.103479] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2855.191267] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2855.199517] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2855.208808] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2855.214975] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2855.221923] pc : __sbq_wake_up+0x20/0x1f0 [ 2855.225920] lr : sbitmap_queue_clear+0x74/0xc0 [ 2855.230350] sp : ffff80000d79b280 [ 2855.233652] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2855.240774] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2855.247897] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2855.255018] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2855.262141] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2855.269263] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2855.276384] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2855.283506] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2855.290628] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2855.297750] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 2855.304872] Call trace: [ 2855.307304] __sbq_wake_up+0x20/0x1f0 [ 2855.310954] sbitmap_queue_clear+0x74/0xc0 [ 2855.315038] blk_mq_put_tag+0x44/0x70 [ 2855.318687] __blk_mq_free_request+0x88/0xf0 [ 2855.322944] blk_mq_free_request+0xc0/0x160 [ 2855.327114] __blk_mq_end_request+0x78/0x150 [ 2855.331371] scsi_end_request+0x114/0x260 [ 2855.335367] scsi_io_completion+0x64/0x174 [ 2855.339450] scsi_finish_command+0xe0/0x120 [ 2855.343620] scsi_complete+0x8c/0x1c4 [ 2855.347270] blk_complete_reqs+0x5c/0x74 [ 2855.351179] blk_done_softirq+0x38/0x4c [ 2855.355002] __do_softirq+0x168/0x418 [ 2855.358650] __irq_exit_rcu+0x120/0x170 [ 2855.362473] irq_exit_rcu+0x18/0x24 [ 2855.365948] el1_interrupt+0x38/0x70 [ 2855.369511] el1h_64_irq_handler+0x18/0x24 [ 2855.373594] el1h_64_irq+0x68/0x6c [ 2855.376983] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2855.381761] __wake_up_common_lock+0x84/0xac [ 2855.386018] __wake_up+0x20/0x30 [ 2855.389233] __sbq_wake_up+0x13c/0x1f0 [ 2855.392969] sbitmap_queue_clear+0x74/0xc0 [ 2855.397053] blk_mq_put_tag+0x44/0x70 [ 2855.400702] __blk_mq_free_request+0x88/0xf0 [ 2855.404959] blk_mq_free_request+0xc0/0x160 [ 2855.409129] bfq_insert_request.isra.0+0x204/0x270 [ 2855.413907] bfq_insert_requests+0x70/0x90 [ 2855.417991] blk_mq_sched_insert_requests+0xac/0x250 [ 2855.422943] blk_mq_flush_plug_list+0x110/0x2cc [ 2855.427461] __blk_flush_plug+0x100/0x140 [ 2855.431458] blk_finish_plug+0x40/0x25c [ 2855.435280] __iomap_dio_rw+0x1b4/0x500 [ 2855.439103] iomap_dio_rw+0x18/0x50 [ 2855.442578] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2855.447929] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2855.452671] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2855.457715] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2855.462846] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2855.467975] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2855.473362] process_one_work+0x1ec/0x470 [ 2855.477358] worker_thread+0x74/0x410 [ 2855.481007] kthread+0xec/0x100 [ 2855.484135] ret_from_fork+0x10/0x20 [ 2867.205652] watchdog: BUG: soft lockup - CPU#68 stuck for 2276s! [swapper/68:0] [ 2867.212947] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2867.212964] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2867.300751] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2867.308567] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2867.317858] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2867.324806] pc : __sbq_wake_up+0x84/0x1f0 [ 2867.328802] lr : sbitmap_queue_clear+0x74/0xc0 [ 2867.333233] sp : ffff8000088438b0 [ 2867.336534] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2867.343656] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2867.350778] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2867.357900] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2867.365022] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2867.372144] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2867.379266] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2867.386388] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2867.393510] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2867.400632] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2867.407753] Call trace: [ 2867.410186] __sbq_wake_up+0x84/0x1f0 [ 2867.413835] sbitmap_queue_clear+0x74/0xc0 [ 2867.417918] blk_mq_put_tag+0x44/0x70 [ 2867.421568] __blk_mq_free_request+0x88/0xf0 [ 2867.425825] blk_mq_free_request+0xc0/0x160 [ 2867.429994] __blk_mq_end_request+0x78/0x150 [ 2867.434251] scsi_end_request+0x114/0x260 [ 2867.438248] scsi_io_completion+0x64/0x174 [ 2867.442331] scsi_finish_command+0xe0/0x120 [ 2867.446501] scsi_complete+0x8c/0x1c4 [ 2867.450150] blk_complete_reqs+0x5c/0x74 [ 2867.454060] blk_done_softirq+0x38/0x4c [ 2867.457882] __do_softirq+0x168/0x418 [ 2867.461531] __irq_exit_rcu+0x120/0x170 [ 2867.465354] irq_exit_rcu+0x18/0x24 [ 2867.468829] el1_interrupt+0x38/0x70 [ 2867.472391] el1h_64_irq_handler+0x18/0x24 [ 2867.476474] el1h_64_irq+0x68/0x6c [ 2867.479863] cpuidle_enter_state+0xc4/0x4e0 [ 2867.484033] cpuidle_enter+0x40/0x60 [ 2867.487596] cpuidle_idle_call+0x118/0x1b0 [ 2867.491679] do_idle+0xac/0x100 [ 2867.494808] cpu_startup_entry+0x30/0x34 [ 2867.498717] secondary_start_kernel+0xd8/0xf0 [ 2867.503061] __secondary_switched+0xb0/0xb4 [ 2868.195655] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 241725 jiffies s: 2093 root: 0x11/. [ 2868.207394] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2868.216604] Task dump for CPU 8: [ 2868.219818] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2868.229720] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2868.235889] Call trace: [ 2868.238322] __switch_to+0xd4/0x100 [ 2868.241799] 0xffff6c0200000080 [ 2868.244928] Task dump for CPU 68: [ 2868.248230] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2868.258132] Call trace: [ 2868.260565] __switch_to+0xd4/0x100 [ 2868.264041] 0x0 [ 2879.095574] watchdog: BUG: soft lockup - CPU#8 stuck for 2287s! [kworker/8:12:19399] [ 2879.103304] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2879.103321] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2879.191108] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2879.199359] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2879.208650] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2879.214817] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2879.221765] pc : __sbq_wake_up+0x58/0x1f0 [ 2879.225762] lr : sbitmap_queue_clear+0x74/0xc0 [ 2879.230192] sp : ffff80000d79b280 [ 2879.233493] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2879.240616] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2879.247737] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2879.254860] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2879.261982] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2879.269104] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2879.276226] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2879.283348] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2879.290470] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2879.297592] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2879.304714] Call trace: [ 2879.307147] __sbq_wake_up+0x58/0x1f0 [ 2879.310796] sbitmap_queue_clear+0x74/0xc0 [ 2879.314880] blk_mq_put_tag+0x44/0x70 [ 2879.318530] __blk_mq_free_request+0x88/0xf0 [ 2879.322786] blk_mq_free_request+0xc0/0x160 [ 2879.326956] __blk_mq_end_request+0x78/0x150 [ 2879.331214] scsi_end_request+0x114/0x260 [ 2879.335210] scsi_io_completion+0x64/0x174 [ 2879.339294] scsi_finish_command+0xe0/0x120 [ 2879.343464] scsi_complete+0x8c/0x1c4 [ 2879.347113] blk_complete_reqs+0x5c/0x74 [ 2879.351023] blk_done_softirq+0x38/0x4c [ 2879.354845] __do_softirq+0x168/0x418 [ 2879.358494] __irq_exit_rcu+0x120/0x170 [ 2879.362317] irq_exit_rcu+0x18/0x24 [ 2879.365792] el1_interrupt+0x38/0x70 [ 2879.369355] el1h_64_irq_handler+0x18/0x24 [ 2879.373438] el1h_64_irq+0x68/0x6c [ 2879.376826] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2879.381604] __wake_up_common_lock+0x84/0xac [ 2879.385861] __wake_up+0x20/0x30 [ 2879.389077] __sbq_wake_up+0x13c/0x1f0 [ 2879.392813] sbitmap_queue_clear+0x74/0xc0 [ 2879.396896] blk_mq_put_tag+0x44/0x70 [ 2879.400545] __blk_mq_free_request+0x88/0xf0 [ 2879.404802] blk_mq_free_request+0xc0/0x160 [ 2879.408972] bfq_insert_request.isra.0+0x204/0x270 [ 2879.413750] bfq_insert_requests+0x70/0x90 [ 2879.417834] blk_mq_sched_insert_requests+0xac/0x250 [ 2879.422786] blk_mq_flush_plug_list+0x110/0x2cc [ 2879.427303] __blk_flush_plug+0x100/0x140 [ 2879.431299] blk_finish_plug+0x40/0x25c [ 2879.435122] __iomap_dio_rw+0x1b4/0x500 [ 2879.438945] iomap_dio_rw+0x18/0x50 [ 2879.442420] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2879.447771] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2879.452515] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2879.457559] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2879.462689] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2879.467818] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2879.473204] process_one_work+0x1ec/0x470 [ 2879.477200] worker_thread+0x74/0x410 [ 2879.480850] kthread+0xec/0x100 [ 2879.483978] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 04:45:00 2022] [ 2891.205494] watchdog: BUG: soft lockup - CPU#68 stuck for 2298s! [swapper/68:0] [ 2891.212790] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2891.212807] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2891.300595] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2891.308412] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2891.317703] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2891.324651] pc : __sbq_wake_up+0x58/0x1f0 [ 2891.328647] lr : sbitmap_queue_clear+0x74/0xc0 [ 2891.333078] sp : ffff8000088438b0 [ 2891.336379] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2891.343502] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2891.350623] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2891.357745] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2891.364868] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2891.371989] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2891.379111] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2891.386233] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2891.393355] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2891.400477] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2891.407599] Call trace: [ 2891.410031] __sbq_wake_up+0x58/0x1f0 [ 2891.413681] sbitmap_queue_clear+0x74/0xc0 [ 2891.417765] blk_mq_put_tag+0x44/0x70 [ 2891.421414] __blk_mq_free_request+0x88/0xf0 [ 2891.425671] blk_mq_free_request+0xc0/0x160 [ 2891.429841] __blk_mq_end_request+0x78/0x150 [ 2891.434098] scsi_end_request+0x114/0x260 [ 2891.438094] scsi_io_completion+0x64/0x174 [ 2891.442179] scsi_finish_command+0xe0/0x120 [ 2891.446349] scsi_complete+0x8c/0x1c4 [ 2891.449998] blk_complete_reqs+0x5c/0x74 [ 2891.453909] blk_done_softirq+0x38/0x4c [ 2891.457731] __do_softirq+0x168/0x418 [ 2891.461380] __irq_exit_rcu+0x120/0x170 [ 2891.465203] irq_exit_rcu+0x18/0x24 [ 2891.468678] el1_interrupt+0x38/0x70 [ 2891.472241] el1h_64_irq_handler+0x18/0x24 [ 2891.476324] el1h_64_irq+0x68/0x6c [ 2891.479713] cpuidle_enter_state+0xc4/0x4e0 [ 2891.483882] cpuidle_enter+0x40/0x60 [ 2891.487445] cpuidle_idle_call+0x118/0x1b0 [ 2891.491528] do_idle+0xac/0x100 [ 2891.494657] cpu_startup_entry+0x30/0x34 [ 2891.498567] secondary_start_kernel+0xd8/0xf0 [ 2891.502911] __secondary_switched+0xb0/0xb4 [ 2903.095416] watchdog: BUG: soft lockup - CPU#8 stuck for 2309s! [kworker/8:12:19399] [ 2903.103145] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2903.103163] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2903.190951] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2903.199201] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2903.208492] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2903.214659] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2903.221607] pc : __sbq_wake_up+0x58/0x1f0 [ 2903.225603] lr : sbitmap_queue_clear+0x74/0xc0 [ 2903.230034] sp : ffff80000d79b280 [ 2903.233335] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2903.240457] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2903.247579] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2903.254702] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2903.261823] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2903.268946] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2903.276069] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2903.283191] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2903.290313] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2903.297435] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2903.304557] Call trace: [ 2903.306989] __sbq_wake_up+0x58/0x1f0 [ 2903.310639] sbitmap_queue_clear+0x74/0xc0 [ 2903.314723] blk_mq_put_tag+0x44/0x70 [ 2903.318372] __blk_mq_free_request+0x88/0xf0 [ 2903.322629] blk_mq_free_request+0xc0/0x160 [ 2903.326799] __blk_mq_end_request+0x78/0x150 [ 2903.331056] scsi_end_request+0x114/0x260 [ 2903.335052] scsi_io_completion+0x64/0x174 [ 2903.339135] scsi_finish_command+0xe0/0x120 [ 2903.343306] scsi_complete+0x8c/0x1c4 [ 2903.346955] blk_complete_reqs+0x5c/0x74 [ 2903.350865] blk_done_softirq+0x38/0x4c [ 2903.354687] __do_softirq+0x168/0x418 [ 2903.358336] __irq_exit_rcu+0x120/0x170 [ 2903.362158] irq_exit_rcu+0x18/0x24 [ 2903.365634] el1_interrupt+0x38/0x70 [ 2903.369196] el1h_64_irq_handler+0x18/0x24 [ 2903.373279] el1h_64_irq+0x68/0x6c [ 2903.376668] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2903.381446] __wake_up_common_lock+0x84/0xac [ 2903.385703] __wake_up+0x20/0x30 [ 2903.388919] __sbq_wake_up+0x13c/0x1f0 [ 2903.392655] sbitmap_queue_clear+0x74/0xc0 [ 2903.396738] blk_mq_put_tag+0x44/0x70 [ 2903.400388] __blk_mq_free_request+0x88/0xf0 [ 2903.404645] blk_mq_free_request+0xc0/0x160 [ 2903.408815] bfq_insert_request.isra.0+0x204/0x270 [ 2903.413593] bfq_insert_requests+0x70/0x90 [ 2903.417676] blk_mq_sched_insert_requests+0xac/0x250 [ 2903.422629] blk_mq_flush_plug_list+0x110/0x2cc [ 2903.427146] __blk_flush_plug+0x100/0x140 [ 2903.431143] blk_finish_plug+0x40/0x25c [ 2903.434965] __iomap_dio_rw+0x1b4/0x500 [ 2903.438788] iomap_dio_rw+0x18/0x50 [ 2903.442263] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2903.447614] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2903.452357] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2903.457401] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2903.462531] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2903.467660] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2903.473047] process_one_work+0x1ec/0x470 [ 2903.477043] worker_thread+0x74/0x410 [ 2903.480692] kthread+0xec/0x100 [ 2903.483821] ret_from_fork+0x10/0x20 [ 2915.205336] watchdog: BUG: soft lockup - CPU#68 stuck for 2320s! [swapper/68:0] [ 2915.212631] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2915.212648] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2915.300435] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2915.308251] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2915.317542] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2915.324489] pc : __sbq_wake_up+0x20/0x1f0 [ 2915.328486] lr : sbitmap_queue_clear+0x74/0xc0 [ 2915.332917] sp : ffff8000088438b0 [ 2915.336218] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2915.343340] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2915.350462] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 2915.357583] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2915.364705] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2915.371827] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2915.378949] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2915.386071] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2915.393193] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2915.400316] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 2915.407438] Call trace: [ 2915.409871] __sbq_wake_up+0x20/0x1f0 [ 2915.413520] sbitmap_queue_clear+0x74/0xc0 [ 2915.417603] blk_mq_put_tag+0x44/0x70 [ 2915.421252] __blk_mq_free_request+0x88/0xf0 [ 2915.425509] blk_mq_free_request+0xc0/0x160 [ 2915.429679] __blk_mq_end_request+0x78/0x150 [ 2915.433936] scsi_end_request+0x114/0x260 [ 2915.437932] scsi_io_completion+0x64/0x174 [ 2915.442015] scsi_finish_command+0xe0/0x120 [ 2915.446185] scsi_complete+0x8c/0x1c4 [ 2915.449835] blk_complete_reqs+0x5c/0x74 [ 2915.453744] blk_done_softirq+0x38/0x4c [ 2915.457567] __do_softirq+0x168/0x418 [ 2915.461216] __irq_exit_rcu+0x120/0x170 [ 2915.465038] irq_exit_rcu+0x18/0x24 [ 2915.468514] el1_interrupt+0x38/0x70 [ 2915.472076] el1h_64_irq_handler+0x18/0x24 [ 2915.476159] el1h_64_irq+0x68/0x6c [ 2915.479548] cpuidle_enter_state+0xc4/0x4e0 [ 2915.483718] cpuidle_enter+0x40/0x60 [ 2915.487281] cpuidle_idle_call+0x118/0x1b0 [ 2915.491364] do_idle+0xac/0x100 [ 2915.494493] cpu_startup_entry+0x30/0x34 [ 2915.498404] secondary_start_kernel+0xd8/0xf0 [ 2915.502748] __secondary_switched+0xb0/0xb4 [ 2927.095258] watchdog: BUG: soft lockup - CPU#8 stuck for 2332s! [kworker/8:12:19399] [ 2927.102987] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2927.103005] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2927.190792] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2927.199043] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2927.208334] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2927.214501] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2927.221449] pc : __sbq_wake_up+0x1c8/0x1f0 [ 2927.225532] lr : sbitmap_queue_clear+0x74/0xc0 [ 2927.229963] sp : ffff80000d79b280 [ 2927.233264] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2927.240386] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2927.247509] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2927.254631] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2927.261752] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2927.268875] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2927.275996] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2927.283118] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2927.290240] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2927.297362] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 2927.304484] Call trace: [ 2927.306917] __sbq_wake_up+0x1c8/0x1f0 [ 2927.310653] sbitmap_queue_clear+0x74/0xc0 [ 2927.314737] blk_mq_put_tag+0x44/0x70 [ 2927.318387] __blk_mq_free_request+0x88/0xf0 [ 2927.322643] blk_mq_free_request+0xc0/0x160 [ 2927.326815] __blk_mq_end_request+0x78/0x150 [ 2927.331072] scsi_end_request+0x114/0x260 [ 2927.335068] scsi_io_completion+0x64/0x174 [ 2927.339151] scsi_finish_command+0xe0/0x120 [ 2927.343322] scsi_complete+0x8c/0x1c4 [ 2927.346971] blk_complete_reqs+0x5c/0x74 [ 2927.350881] blk_done_softirq+0x38/0x4c [ 2927.354703] __do_softirq+0x168/0x418 [ 2927.358352] __irq_exit_rcu+0x120/0x170 [ 2927.362175] irq_exit_rcu+0x18/0x24 [ 2927.365651] el1_interrupt+0x38/0x70 [ 2927.369213] el1h_64_irq_handler+0x18/0x24 [ 2927.373296] el1h_64_irq+0x68/0x6c [ 2927.376685] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2927.381463] __wake_up_common_lock+0x84/0xac [ 2927.385720] __wake_up+0x20/0x30 [ 2927.388935] __sbq_wake_up+0x13c/0x1f0 [ 2927.392672] sbitmap_queue_clear+0x74/0xc0 [ 2927.396755] blk_mq_put_tag+0x44/0x70 [ 2927.400405] __blk_mq_free_request+0x88/0xf0 [ 2927.404661] blk_mq_free_request+0xc0/0x160 [ 2927.408831] bfq_insert_request.isra.0+0x204/0x270 [ 2927.413609] bfq_insert_requests+0x70/0x90 [ 2927.417693] blk_mq_sched_insert_requests+0xac/0x250 [ 2927.422645] blk_mq_flush_plug_list+0x110/0x2cc [ 2927.427163] __blk_flush_plug+0x100/0x140 [ 2927.431159] blk_finish_plug+0x40/0x25c [ 2927.434982] __iomap_dio_rw+0x1b4/0x500 [ 2927.438805] iomap_dio_rw+0x18/0x50 [ 2927.442280] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2927.447630] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2927.452373] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2927.457417] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2927.462547] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2927.467677] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2927.473063] process_one_work+0x1ec/0x470 [ 2927.477059] worker_thread+0x74/0x410 [ 2927.480708] kthread+0xec/0x100 [ 2927.483836] ret_from_fork+0x10/0x20 [ 2934.755217] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 248381 jiffies s: 2093 root: 0x11/. [ 2934.766956] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 2934.776164] Task dump for CPU 8: [ 2934.779379] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 2934.789280] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2934.795450] Call trace: [ 2934.797883] __switch_to+0xd4/0x100 [ 2934.801359] 0xffff6c0200000080 [ 2934.804488] Task dump for CPU 68: [ 2934.807791] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 2934.817692] Call trace: [ 2934.820125] __switch_to+0xd4/0x100 [ 2934.823601] 0x0 [ 2939.205178] watchdog: BUG: soft lockup - CPU#68 stuck for 2343s! [swapper/68:0] [ 2939.212473] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2939.212490] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2939.300277] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2939.308093] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2939.317384] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2939.324332] pc : __sbq_wake_up+0x58/0x1f0 [ 2939.328329] lr : sbitmap_queue_clear+0x74/0xc0 [ 2939.332759] sp : ffff8000088438b0 [ 2939.336060] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2939.343182] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2939.350304] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2939.357426] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2939.364548] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2939.371670] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2939.378792] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2939.385914] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2939.393036] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2939.400158] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2939.407280] Call trace: [ 2939.409713] __sbq_wake_up+0x58/0x1f0 [ 2939.413362] sbitmap_queue_clear+0x74/0xc0 [ 2939.417445] blk_mq_put_tag+0x44/0x70 [ 2939.421095] __blk_mq_free_request+0x88/0xf0 [ 2939.425353] blk_mq_free_request+0xc0/0x160 [ 2939.429522] __blk_mq_end_request+0x78/0x150 [ 2939.433779] scsi_end_request+0x114/0x260 [ 2939.437776] scsi_io_completion+0x64/0x174 [ 2939.441859] scsi_finish_command+0xe0/0x120 [ 2939.446029] scsi_complete+0x8c/0x1c4 [ 2939.449678] blk_complete_reqs+0x5c/0x74 [ 2939.453588] blk_done_softirq+0x38/0x4c [ 2939.457410] __do_softirq+0x168/0x418 [ 2939.461059] __irq_exit_rcu+0x120/0x170 [ 2939.464882] irq_exit_rcu+0x18/0x24 [ 2939.468357] el1_interrupt+0x38/0x70 [ 2939.471919] el1h_64_irq_handler+0x18/0x24 [ 2939.476002] el1h_64_irq+0x68/0x6c [ 2939.479391] cpuidle_enter_state+0xc4/0x4e0 [ 2939.483561] cpuidle_enter+0x40/0x60 [ 2939.487123] cpuidle_idle_call+0x118/0x1b0 [ 2939.491207] do_idle+0xac/0x100 [ 2939.494336] cpu_startup_entry+0x30/0x34 [ 2939.498245] secondary_start_kernel+0xd8/0xf0 [ 2939.502589] __secondary_switched+0xb0/0xb4 [ 2951.095100] watchdog: BUG: soft lockup - CPU#8 stuck for 2354s! [kworker/8:12:19399] [ 2951.102829] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2951.102846] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2951.190634] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2951.198884] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2951.208174] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2951.214342] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2951.221289] pc : __sbq_wake_up+0x20/0x1f0 [ 2951.225286] lr : sbitmap_queue_clear+0x74/0xc0 [ 2951.229717] sp : ffff80000d79b280 [ 2951.233018] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2951.240140] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2951.247263] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 2951.254385] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2951.261506] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2951.268629] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2951.275751] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2951.282872] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2951.289994] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2951.297116] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 2951.304238] Call trace: [ 2951.306671] __sbq_wake_up+0x20/0x1f0 [ 2951.310320] sbitmap_queue_clear+0x74/0xc0 [ 2951.314404] blk_mq_put_tag+0x44/0x70 [ 2951.318054] __blk_mq_free_request+0x88/0xf0 [ 2951.322310] blk_mq_free_request+0xc0/0x160 [ 2951.326480] __blk_mq_end_request+0x78/0x150 [ 2951.330737] scsi_end_request+0x114/0x260 [ 2951.334733] scsi_io_completion+0x64/0x174 [ 2951.338816] scsi_finish_command+0xe0/0x120 [ 2951.342987] scsi_complete+0x8c/0x1c4 [ 2951.346636] blk_complete_reqs+0x5c/0x74 [ 2951.350545] blk_done_softirq+0x38/0x4c [ 2951.354369] __do_softirq+0x168/0x418 [ 2951.358018] __irq_exit_rcu+0x120/0x170 [ 2951.361841] irq_exit_rcu+0x18/0x24 [ 2951.365316] el1_interrupt+0x38/0x70 [ 2951.368878] el1h_64_irq_handler+0x18/0x24 [ 2951.372962] el1h_64_irq+0x68/0x6c [ 2951.376350] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2951.381128] __wake_up_common_lock+0x84/0xac [ 2951.385386] __wake_up+0x20/0x30 [ 2951.388601] __sbq_wake_up+0x13c/0x1f0 [ 2951.392337] sbitmap_queue_clear+0x74/0xc0 [ 2951.396421] blk_mq_put_tag+0x44/0x70 [ 2951.400070] __blk_mq_free_request+0x88/0xf0 [ 2951.404327] blk_mq_free_request+0xc0/0x160 [ 2951.408497] bfq_insert_request.isra.0+0x204/0x270 [ 2951.413275] bfq_insert_requests+0x70/0x90 [ 2951.417359] blk_mq_sched_insert_requests+0xac/0x250 [ 2951.422311] blk_mq_flush_plug_list+0x110/0x2cc [ 2951.426829] __blk_flush_plug+0x100/0x140 [ 2951.430826] blk_finish_plug+0x40/0x25c [ 2951.434649] __iomap_dio_rw+0x1b4/0x500 [ 2951.438471] iomap_dio_rw+0x18/0x50 [ 2951.441948] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2951.447299] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2951.452041] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2951.457086] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2951.462216] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2951.467346] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2951.472732] process_one_work+0x1ec/0x470 [ 2951.476728] worker_thread+0x74/0x410 [ 2951.480378] kthread+0xec/0x100 [ 2951.483506] ret_from_fork+0x10/0x20 [ 2963.205020] watchdog: BUG: soft lockup - CPU#68 stuck for 2365s! [swapper/68:0] [ 2963.212315] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2963.212332] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2963.300120] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2963.307935] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2963.317226] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2963.324174] pc : __sbq_wake_up+0x58/0x1f0 [ 2963.328170] lr : sbitmap_queue_clear+0x74/0xc0 [ 2963.332601] sp : ffff8000088438b0 [ 2963.335903] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2963.343025] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2963.350147] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2963.357268] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2963.364390] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2963.371513] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2963.378634] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2963.385756] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2963.392878] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2963.400000] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2963.407122] Call trace: [ 2963.409554] __sbq_wake_up+0x58/0x1f0 [ 2963.413204] sbitmap_queue_clear+0x74/0xc0 [ 2963.417287] blk_mq_put_tag+0x44/0x70 [ 2963.420937] __blk_mq_free_request+0x88/0xf0 [ 2963.425193] blk_mq_free_request+0xc0/0x160 [ 2963.429363] __blk_mq_end_request+0x78/0x150 [ 2963.433620] scsi_end_request+0x114/0x260 [ 2963.437616] scsi_io_completion+0x64/0x174 [ 2963.441699] scsi_finish_command+0xe0/0x120 [ 2963.445870] scsi_complete+0x8c/0x1c4 [ 2963.449519] blk_complete_reqs+0x5c/0x74 [ 2963.453428] blk_done_softirq+0x38/0x4c [ 2963.457251] __do_softirq+0x168/0x418 [ 2963.460900] __irq_exit_rcu+0x120/0x170 [ 2963.464722] irq_exit_rcu+0x18/0x24 [ 2963.468198] el1_interrupt+0x38/0x70 [ 2963.471761] el1h_64_irq_handler+0x18/0x24 [ 2963.475844] el1h_64_irq+0x68/0x6c [ 2963.479233] cpuidle_enter_state+0xc4/0x4e0 [ 2963.483403] cpuidle_enter+0x40/0x60 [ 2963.486966] cpuidle_idle_call+0x118/0x1b0 [ 2963.491050] do_idle+0xac/0x100 [ 2963.494179] cpu_startup_entry+0x30/0x34 [ 2963.498088] secondary_start_kernel+0xd8/0xf0 [ 2963.502432] __secondary_switched+0xb0/0xb4 [ 2975.094941] watchdog: BUG: soft lockup - CPU#8 stuck for 2376s! [kworker/8:12:19399] [ 2975.102671] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2975.102688] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2975.190475] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2975.198725] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2975.208016] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2975.214183] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2975.221131] pc : __sbq_wake_up+0x58/0x1f0 [ 2975.225128] lr : sbitmap_queue_clear+0x74/0xc0 [ 2975.229559] sp : ffff80000d79b280 [ 2975.232860] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2975.239982] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2975.247104] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2975.254226] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2975.261348] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2975.268470] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2975.275591] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2975.282714] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2975.289835] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2975.296957] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2975.304079] Call trace: [ 2975.306512] __sbq_wake_up+0x58/0x1f0 [ 2975.310162] sbitmap_queue_clear+0x74/0xc0 [ 2975.314245] blk_mq_put_tag+0x44/0x70 [ 2975.317895] __blk_mq_free_request+0x88/0xf0 [ 2975.322152] blk_mq_free_request+0xc0/0x160 [ 2975.326322] __blk_mq_end_request+0x78/0x150 [ 2975.330579] scsi_end_request+0x114/0x260 [ 2975.334575] scsi_io_completion+0x64/0x174 [ 2975.338659] scsi_finish_command+0xe0/0x120 [ 2975.342829] scsi_complete+0x8c/0x1c4 [ 2975.346478] blk_complete_reqs+0x5c/0x74 [ 2975.350388] blk_done_softirq+0x38/0x4c [ 2975.354211] __do_softirq+0x168/0x418 [ 2975.357860] __irq_exit_rcu+0x120/0x170 [ 2975.361683] irq_exit_rcu+0x18/0x24 [ 2975.365158] el1_interrupt+0x38/0x70 [ 2975.368720] el1h_64_irq_handler+0x18/0x24 [ 2975.372803] el1h_64_irq+0x68/0x6c [ 2975.376192] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2975.380970] __wake_up_common_lock+0x84/0xac [ 2975.385227] __wake_up+0x20/0x30 [ 2975.388443] __sbq_wake_up+0x13c/0x1f0 [ 2975.392179] sbitmap_queue_clear+0x74/0xc0 [ 2975.396262] blk_mq_put_tag+0x44/0x70 [ 2975.399912] __blk_mq_free_request+0x88/0xf0 [ 2975.404168] blk_mq_free_request+0xc0/0x160 [ 2975.408338] bfq_insert_request.isra.0+0x204/0x270 [ 2975.413116] bfq_insert_requests+0x70/0x90 [ 2975.417200] blk_mq_sched_insert_requests+0xac/0x250 [ 2975.422152] blk_mq_flush_plug_list+0x110/0x2cc [ 2975.426670] __blk_flush_plug+0x100/0x140 [ 2975.430666] blk_finish_plug+0x40/0x25c [ 2975.434489] __iomap_dio_rw+0x1b4/0x500 [ 2975.438312] iomap_dio_rw+0x18/0x50 [ 2975.441787] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2975.447137] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2975.451880] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2975.456924] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2975.462053] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2975.467183] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2975.472569] process_one_work+0x1ec/0x470 [ 2975.476565] worker_thread+0x74/0x410 [ 2975.480215] kthread+0xec/0x100 [ 2975.483343] ret_from_fork+0x10/0x20 [ 2987.204862] watchdog: BUG: soft lockup - CPU#68 stuck for 2388s! [swapper/68:0] [ 2987.212157] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2987.212173] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2987.299961] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 2987.307777] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2987.317068] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2987.324015] pc : __sbq_wake_up+0x58/0x1f0 [ 2987.328011] lr : sbitmap_queue_clear+0x74/0xc0 [ 2987.332442] sp : ffff8000088438b0 [ 2987.335743] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2987.342864] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 2987.349988] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 2987.357110] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 2987.364232] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 2987.371353] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 2987.378475] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 2987.385597] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2987.392719] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2987.399841] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 2987.406963] Call trace: [ 2987.409396] __sbq_wake_up+0x58/0x1f0 [ 2987.413045] sbitmap_queue_clear+0x74/0xc0 [ 2987.417128] blk_mq_put_tag+0x44/0x70 [ 2987.420778] __blk_mq_free_request+0x88/0xf0 [ 2987.425034] blk_mq_free_request+0xc0/0x160 [ 2987.429204] __blk_mq_end_request+0x78/0x150 [ 2987.433461] scsi_end_request+0x114/0x260 [ 2987.437458] scsi_io_completion+0x64/0x174 [ 2987.441541] scsi_finish_command+0xe0/0x120 [ 2987.445711] scsi_complete+0x8c/0x1c4 [ 2987.449360] blk_complete_reqs+0x5c/0x74 [ 2987.453270] blk_done_softirq+0x38/0x4c [ 2987.457093] __do_softirq+0x168/0x418 [ 2987.460742] __irq_exit_rcu+0x120/0x170 [ 2987.464564] irq_exit_rcu+0x18/0x24 [ 2987.468040] el1_interrupt+0x38/0x70 [ 2987.471603] el1h_64_irq_handler+0x18/0x24 [ 2987.475687] el1h_64_irq+0x68/0x6c [ 2987.479075] cpuidle_enter_state+0xc4/0x4e0 [ 2987.483245] cpuidle_enter+0x40/0x60 [ 2987.486808] cpuidle_idle_call+0x118/0x1b0 [ 2987.490891] do_idle+0xac/0x100 [ 2987.494020] cpu_startup_entry+0x30/0x34 [ 2987.497929] secondary_start_kernel+0xd8/0xf0 [ 2987.502273] __secondary_switched+0xb0/0xb4 [ 2999.094783] watchdog: BUG: soft lockup - CPU#8 stuck for 2399s! [kworker/8:12:19399] [ 2999.102512] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 2999.102530] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 2999.190318] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 2999.198567] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 2999.207858] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 2999.214026] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2999.220973] pc : __sbq_wake_up+0x84/0x1f0 [ 2999.224970] lr : sbitmap_queue_clear+0x74/0xc0 [ 2999.229400] sp : ffff80000d79b280 [ 2999.232702] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 2999.239824] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 2999.246946] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 2999.254068] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 2999.261191] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 2999.268313] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 2999.275434] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 2999.282556] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 2999.289678] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 2999.296800] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 2999.303922] Call trace: [ 2999.306354] __sbq_wake_up+0x84/0x1f0 [ 2999.310003] sbitmap_queue_clear+0x74/0xc0 [ 2999.314087] blk_mq_put_tag+0x44/0x70 [ 2999.317736] __blk_mq_free_request+0x88/0xf0 [ 2999.321993] blk_mq_free_request+0xc0/0x160 [ 2999.326163] __blk_mq_end_request+0x78/0x150 [ 2999.330420] scsi_end_request+0x114/0x260 [ 2999.334416] scsi_io_completion+0x64/0x174 [ 2999.338499] scsi_finish_command+0xe0/0x120 [ 2999.342670] scsi_complete+0x8c/0x1c4 [ 2999.346319] blk_complete_reqs+0x5c/0x74 [ 2999.350228] blk_done_softirq+0x38/0x4c [ 2999.354051] __do_softirq+0x168/0x418 [ 2999.357701] __irq_exit_rcu+0x120/0x170 [ 2999.361524] irq_exit_rcu+0x18/0x24 [ 2999.364999] el1_interrupt+0x38/0x70 [ 2999.368562] el1h_64_irq_handler+0x18/0x24 [ 2999.372645] el1h_64_irq+0x68/0x6c [ 2999.376033] _raw_spin_unlock_irqrestore+0x20/0x50 [ 2999.380811] __wake_up_common_lock+0x84/0xac [ 2999.385069] __wake_up+0x20/0x30 [ 2999.388284] __sbq_wake_up+0x13c/0x1f0 [ 2999.392020] sbitmap_queue_clear+0x74/0xc0 [ 2999.396104] blk_mq_put_tag+0x44/0x70 [ 2999.399754] __blk_mq_free_request+0x88/0xf0 [ 2999.404011] blk_mq_free_request+0xc0/0x160 [ 2999.408181] bfq_insert_request.isra.0+0x204/0x270 [ 2999.412959] bfq_insert_requests+0x70/0x90 [ 2999.417042] blk_mq_sched_insert_requests+0xac/0x250 [ 2999.421994] blk_mq_flush_plug_list+0x110/0x2cc [ 2999.426512] __blk_flush_plug+0x100/0x140 [ 2999.430509] blk_finish_plug+0x40/0x25c [ 2999.434331] __iomap_dio_rw+0x1b4/0x500 [ 2999.438154] iomap_dio_rw+0x18/0x50 [ 2999.441629] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 2999.446980] xfs_file_write_iter+0xf4/0x15c [xfs] [ 2999.451723] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 2999.456767] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 2999.461896] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 2999.467026] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 2999.472412] process_one_work+0x1ec/0x470 [ 2999.476408] worker_thread+0x74/0x410 [ 2999.480057] kthread+0xec/0x100 [ 2999.483186] ret_from_fork+0x10/0x20 [ 3001.314778] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 255037 jiffies s: 2093 root: 0x11/. [ 3001.326516] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3001.335725] Task dump for CPU 8: [ 3001.338940] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3001.348843] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3001.355012] Call trace: [ 3001.357445] __switch_to+0xd4/0x100 [ 3001.360922] 0xffff6c0200000080 [ 3001.364051] Task dump for CPU 68: [ 3001.367354] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3001.377256] Call trace: [ 3001.379689] __switch_to+0xd4/0x100 [ 3001.383165] 0x0 [ 3009.314717] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3009.320449] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=128169 [ 3009.329656] (t=258443 jiffies g=36445 q=75636 ncpus=128) [ 3009.335041] Task dump for CPU 8: [ 3009.338256] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3009.348156] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3009.354324] Call trace: [ 3009.356757] __switch_to+0xd4/0x100 [ 3009.360233] 0xffff6c0200000080 [ 3009.363362] Task dump for CPU 68: [ 3009.366663] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3009.376563] Call trace: [ 3009.378997] dump_backtrace+0xe8/0x130 [ 3009.382733] show_stack+0x20/0x60 [ 3009.386034] sched_show_task+0x160/0x190 [ 3009.389944] dump_cpu_task+0x4c/0x78 [ 3009.393507] rcu_dump_cpu_stacks+0xc4/0x150 [ 3009.397678] print_cpu_stall+0x180/0x210 [ 3009.401588] check_cpu_stall+0x20c/0x2b4 [ 3009.405497] rcu_pending+0x4c/0x190 [ 3009.408973] rcu_sched_clock_irq+0x80/0x204 [ 3009.413143] update_process_times+0x7c/0xc0 [ 3009.417313] tick_sched_handle+0x38/0x74 [ 3009.421223] tick_sched_timer+0x60/0xc0 [ 3009.425046] __hrtimer_run_queues+0x170/0x300 [ 3009.429390] hrtimer_interrupt+0xfc/0x260 [ 3009.433386] arch_timer_handler_phys+0x34/0x50 [ 3009.437816] handle_percpu_devid_irq+0xac/0x240 [ 3009.442334] generic_handle_domain_irq+0x34/0x50 [ 3009.446939] gic_handle_irq+0x5c/0xe0 [ 3009.450588] call_on_irq_stack+0x2c/0x38 [ 3009.454497] do_interrupt_handler+0xdc/0xe0 [ 3009.458668] el1_interrupt+0x34/0x70 [ 3009.462230] el1h_64_irq_handler+0x18/0x24 [ 3009.466313] el1h_64_irq+0x68/0x6c [ 3009.469701] __sbq_wake_up+0x4/0x1f0 [ 3009.473263] blk_mq_put_tag+0x44/0x70 [ 3009.476913] __blk_mq_free_request+0x88/0xf0 [ 3009.481170] blk_mq_free_request+0xc0/0x160 [ 3009.485340] __blk_mq_end_request+0x78/0x150 [ 3009.489597] scsi_end_request+0x114/0x260 [ 3009.493593] scsi_io_completion+0x64/0x174 [ 3009.497676] scsi_finish_command+0xe0/0x120 [ 3009.501847] scsi_complete+0x8c/0x1c4 [ 3009.505496] blk_complete_reqs+0x5c/0x74 [ 3009.509405] blk_done_softirq+0x38/0x4c [ 3009.513228] __do_softirq+0x168/0x418 [ 3009.516877] __irq_exit_rcu+0x120/0x170 [ 3009.520700] irq_exit_rcu+0x18/0x24 [ 3009.524175] el1_interrupt+0x38/0x70 [ 3009.527737] el1h_64_irq_handler+0x18/0x24 [ 3009.531821] el1h_64_irq+0x68/0x6c [ 3009.535209] cpuidle_enter_state+0xc4/0x4e0 [ 3009.539380] cpuidle_enter+0x40/0x60 [ 3009.542942] cpuidle_idle_call+0x118/0x1b0 [ 3009.547026] do_idle+0xac/0x100 [ 3009.550154] cpu_startup_entry+0x30/0x34 [ 3009.554063] secondary_start_kernel+0xd8/0xf0 [ 3009.558408] __secondary_switched+0xb0/0xb4 [ 3023.094625] watchdog: BUG: soft lockup - CPU#8 stuck for 2421s! [kworker/8:12:19399] [ 3023.102355] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3023.102372] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3023.190159] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3023.198409] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3023.207700] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3023.213868] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3023.220815] pc : __sbq_wake_up+0x8/0x1f0 [ 3023.224725] lr : sbitmap_queue_clear+0x74/0xc0 [ 3023.229156] sp : ffff80000d79b280 [ 3023.232458] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3023.239580] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3023.246703] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3023.253824] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3023.260947] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3023.268069] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3023.275191] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3023.282312] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3023.289434] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3023.296556] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 3023.303678] Call trace: [ 3023.306111] __sbq_wake_up+0x8/0x1f0 [ 3023.309673] blk_mq_put_tag+0x44/0x70 [ 3023.313323] __blk_mq_free_request+0x88/0xf0 [ 3023.317580] blk_mq_free_request+0xc0/0x160 [ 3023.321750] __blk_mq_end_request+0x78/0x150 [ 3023.326008] scsi_end_request+0x114/0x260 [ 3023.330004] scsi_io_completion+0x64/0x174 [ 3023.334087] scsi_finish_command+0xe0/0x120 [ 3023.338258] scsi_complete+0x8c/0x1c4 [ 3023.341907] blk_complete_reqs+0x5c/0x74 [ 3023.345816] blk_done_softirq+0x38/0x4c [ 3023.349639] __do_softirq+0x168/0x418 [ 3023.353288] __irq_exit_rcu+0x120/0x170 [ 3023.357110] irq_exit_rcu+0x18/0x24 [ 3023.360586] el1_interrupt+0x38/0x70 [ 3023.364148] el1h_64_irq_handler+0x18/0x24 [ 3023.368231] el1h_64_irq+0x68/0x6c [ 3023.371619] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3023.376397] __wake_up_common_lock+0x84/0xac [ 3023.380654] __wake_up+0x20/0x30 [ 3023.383870] __sbq_wake_up+0x13c/0x1f0 [ 3023.387606] sbitmap_queue_clear+0x74/0xc0 [ 3023.391690] blk_mq_put_tag+0x44/0x70 [ 3023.395339] __blk_mq_free_request+0x88/0xf0 [ 3023.399596] blk_mq_free_request+0xc0/0x160 [ 3023.403766] bfq_insert_request.isra.0+0x204/0x270 [ 3023.408544] bfq_insert_requests+0x70/0x90 [ 3023.412628] blk_mq_sched_insert_requests+0xac/0x250 [ 3023.417579] blk_mq_flush_plug_list+0x110/0x2cc [ 3023.422097] __blk_flush_plug+0x100/0x140 [ 3023.426094] blk_finish_plug+0x40/0x25c [ 3023.429916] __iomap_dio_rw+0x1b4/0x500 [ 3023.433739] iomap_dio_rw+0x18/0x50 [ 3023.437215] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3023.442565] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3023.447307] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3023.452351] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3023.457481] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3023.462611] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3023.467997] process_one_work+0x1ec/0x470 [ 3023.471994] worker_thread+0x74/0x410 [ 3023.475643] kthread+0xec/0x100 [ 3023.478772] ret_from_fork+0x10/0x20 [ 3035.204545] watchdog: BUG: soft lockup - CPU#68 stuck for 2432s! [swapper/68:0] [ 3035.211840] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3035.211858] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3035.299645] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3035.307460] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3035.316751] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3035.323699] pc : __sbq_wake_up+0x7c/0x1f0 [ 3035.327696] lr : sbitmap_queue_clear+0x74/0xc0 [ 3035.332126] sp : ffff800008843900 [ 3035.335428] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3035.342550] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3035.349672] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3035.356795] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3035.363916] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3035.371038] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3035.378160] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3035.385282] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3035.392404] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3035.399526] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3035.406648] Call trace: [ 3035.409081] __sbq_wake_up+0x7c/0x1f0 [ 3035.412730] blk_mq_put_tag+0x44/0x70 [ 3035.416379] __blk_mq_free_request+0x88/0xf0 [ 3035.420636] blk_mq_free_request+0xc0/0x160 [ 3035.424806] __blk_mq_end_request+0x78/0x150 [ 3035.429063] scsi_end_request+0x114/0x260 [ 3035.433059] scsi_io_completion+0x64/0x174 [ 3035.437142] scsi_finish_command+0xe0/0x120 [ 3035.441313] scsi_complete+0x8c/0x1c4 [ 3035.444962] blk_complete_reqs+0x5c/0x74 [ 3035.448871] blk_done_softirq+0x38/0x4c [ 3035.452694] __do_softirq+0x168/0x418 [ 3035.456342] __irq_exit_rcu+0x120/0x170 [ 3035.460165] irq_exit_rcu+0x18/0x24 [ 3035.463640] el1_interrupt+0x38/0x70 [ 3035.467203] el1h_64_irq_handler+0x18/0x24 [ 3035.471286] el1h_64_irq+0x68/0x6c [ 3035.474674] cpuidle_enter_state+0xc4/0x4e0 [ 3035.478844] cpuidle_enter+0x40/0x60 [ 3035.482406] cpuidle_idle_call+0x118/0x1b0 [ 3035.486490] do_idle+0xac/0x100 [ 3035.489619] cpu_startup_entry+0x30/0x34 [ 3035.493527] secondary_start_kernel+0xd8/0xf0 [ 3035.497872] __secondary_switched+0xb0/0xb4 [ 3047.094467] watchdog: BUG: soft lockup - CPU#8 stuck for 2443s! [kworker/8:12:19399] [ 3047.102196] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3047.102213] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3047.190001] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3047.198251] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3047.207542] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3047.213709] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3047.220657] pc : __sbq_wake_up+0x7c/0x1f0 [ 3047.224654] lr : sbitmap_queue_clear+0x74/0xc0 [ 3047.229085] sp : ffff80000d79b2d0 [ 3047.232386] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3047.239508] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3047.246631] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3047.253754] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3047.260877] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3047.267999] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3047.275121] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3047.282243] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3047.289366] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3047.296488] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3047.303610] Call trace: [ 3047.306043] __sbq_wake_up+0x7c/0x1f0 [ 3047.309693] blk_mq_put_tag+0x44/0x70 [ 3047.313342] __blk_mq_free_request+0x88/0xf0 [ 3047.317599] blk_mq_free_request+0xc0/0x160 [ 3047.321769] __blk_mq_end_request+0x78/0x150 [ 3047.326026] scsi_end_request+0x114/0x260 [ 3047.330022] scsi_io_completion+0x64/0x174 [ 3047.334106] scsi_finish_command+0xe0/0x120 [ 3047.338276] scsi_complete+0x8c/0x1c4 [ 3047.341925] blk_complete_reqs+0x5c/0x74 [ 3047.345835] blk_done_softirq+0x38/0x4c [ 3047.349658] __do_softirq+0x168/0x418 [ 3047.353306] __irq_exit_rcu+0x120/0x170 [ 3047.357129] irq_exit_rcu+0x18/0x24 [ 3047.360604] el1_interrupt+0x38/0x70 [ 3047.364167] el1h_64_irq_handler+0x18/0x24 [ 3047.368250] el1h_64_irq+0x68/0x6c [ 3047.371638] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3047.376416] __wake_up_common_lock+0x84/0xac [ 3047.380673] __wake_up+0x20/0x30 [ 3047.383889] __sbq_wake_up+0x13c/0x1f0 [ 3047.387625] sbitmap_queue_clear+0x74/0xc0 [ 3047.391708] blk_mq_put_tag+0x44/0x70 [ 3047.395358] __blk_mq_free_request+0x88/0xf0 [ 3047.399614] blk_mq_free_request+0xc0/0x160 [ 3047.403784] bfq_insert_request.isra.0+0x204/0x270 [ 3047.408562] bfq_insert_requests+0x70/0x90 [ 3047.412646] blk_mq_sched_insert_requests+0xac/0x250 [ 3047.417598] blk_mq_flush_plug_list+0x110/0x2cc [ 3047.422115] __blk_flush_plug+0x100/0x140 [ 3047.426111] blk_finish_plug+0x40/0x25c [ 3047.429934] __iomap_dio_rw+0x1b4/0x500 [ 3047.433757] iomap_dio_rw+0x18/0x50 [ 3047.437232] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3047.442583] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3047.447326] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3047.452370] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3047.457499] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3047.462629] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3047.468014] process_one_work+0x1ec/0x470 [ 3047.472011] worker_thread+0x74/0x410 [ 3047.475659] kthread+0xec/0x100 [ 3047.478788] ret_from_fork+0x10/0x20 [ 3059.204387] watchdog: BUG: soft lockup - CPU#68 stuck for 2455s! [swapper/68:0] [ 3059.211682] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3059.211699] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3059.299486] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3059.307302] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3059.316593] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3059.323540] pc : __sbq_wake_up+0x58/0x1f0 [ 3059.327537] lr : sbitmap_queue_clear+0x74/0xc0 [ 3059.331968] sp : ffff8000088438b0 [ 3059.335269] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3059.342391] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3059.349513] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3059.356635] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3059.363757] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3059.370879] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3059.378000] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3059.385122] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3059.392244] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3059.399366] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3059.406488] Call trace: [ 3059.408921] __sbq_wake_up+0x58/0x1f0 [ 3059.412570] sbitmap_queue_clear+0x74/0xc0 [ 3059.416654] blk_mq_put_tag+0x44/0x70 [ 3059.420303] __blk_mq_free_request+0x88/0xf0 [ 3059.424560] blk_mq_free_request+0xc0/0x160 [ 3059.428730] __blk_mq_end_request+0x78/0x150 [ 3059.432987] scsi_end_request+0x114/0x260 [ 3059.436983] scsi_io_completion+0x64/0x174 [ 3059.441066] scsi_finish_command+0xe0/0x120 [ 3059.445237] scsi_complete+0x8c/0x1c4 [ 3059.448885] blk_complete_reqs+0x5c/0x74 [ 3059.452795] blk_done_softirq+0x38/0x4c [ 3059.456618] __do_softirq+0x168/0x418 [ 3059.460267] __irq_exit_rcu+0x120/0x170 [ 3059.464089] irq_exit_rcu+0x18/0x24 [ 3059.467565] el1_interrupt+0x38/0x70 [ 3059.471127] el1h_64_irq_handler+0x18/0x24 [ 3059.475210] el1h_64_irq+0x68/0x6c [ 3059.478599] cpuidle_enter_state+0xc4/0x4e0 [ 3059.482769] cpuidle_enter+0x40/0x60 [ 3059.486331] cpuidle_idle_call+0x118/0x1b0 [ 3059.490415] do_idle+0xac/0x100 [ 3059.493544] cpu_startup_entry+0x30/0x34 [ 3059.497453] secondary_start_kernel+0xd8/0xf0 [ 3059.501797] __secondary_switched+0xb0/0xb4 [ 3067.874340] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 261693 jiffies s: 2093 root: 0x11/. [ 3067.886079] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3067.895287] Task dump for CPU 8: [ 3067.898502] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3067.908404] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3067.914573] Call trace: [ 3067.917006] __switch_to+0xd4/0x100 [ 3067.920483] 0xffff6c0200000080 [ 3067.923611] Task dump for CPU 68: [ 3067.926914] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3067.936815] Call trace: [ 3067.939248] __switch_to+0xd4/0x100 [ 3067.942725] 0x0 [ 3071.094309] watchdog: BUG: soft lockup - CPU#8 stuck for 2466s! [kworker/8:12:19399] [ 3071.102038] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3071.102056] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3071.189844] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3071.198095] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3071.207386] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3071.213553] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3071.220501] pc : __sbq_wake_up+0x58/0x1f0 [ 3071.224497] lr : sbitmap_queue_clear+0x74/0xc0 [ 3071.228928] sp : ffff80000d79b280 [ 3071.232230] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3071.239352] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3071.246474] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3071.253596] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3071.260718] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3071.267840] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3071.274962] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3071.282084] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3071.289206] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3071.296329] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3071.303451] Call trace: [ 3071.305884] __sbq_wake_up+0x58/0x1f0 [ 3071.309534] sbitmap_queue_clear+0x74/0xc0 [ 3071.313617] blk_mq_put_tag+0x44/0x70 [ 3071.317267] __blk_mq_free_request+0x88/0xf0 [ 3071.321523] blk_mq_free_request+0xc0/0x160 [ 3071.325693] __blk_mq_end_request+0x78/0x150 [ 3071.329950] scsi_end_request+0x114/0x260 [ 3071.333946] scsi_io_completion+0x64/0x174 [ 3071.338030] scsi_finish_command+0xe0/0x120 [ 3071.342200] scsi_complete+0x8c/0x1c4 [ 3071.345849] blk_complete_reqs+0x5c/0x74 [ 3071.349758] blk_done_softirq+0x38/0x4c [ 3071.353581] __do_softirq+0x168/0x418 [ 3071.357230] __irq_exit_rcu+0x120/0x170 [ 3071.361053] irq_exit_rcu+0x18/0x24 [ 3071.364528] el1_interrupt+0x38/0x70 [ 3071.368091] el1h_64_irq_handler+0x18/0x24 [ 3071.372174] el1h_64_irq+0x68/0x6c [ 3071.375562] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3071.380339] __wake_up_common_lock+0x84/0xac [ 3071.384597] __wake_up+0x20/0x30 [ 3071.387813] __sbq_wake_up+0x13c/0x1f0 [ 3071.391549] sbitmap_queue_clear+0x74/0xc0 [ 3071.395632] blk_mq_put_tag+0x44/0x70 [ 3071.399281] __blk_mq_free_request+0x88/0xf0 [ 3071.403538] blk_mq_free_request+0xc0/0x160 [ 3071.407708] bfq_insert_request.isra.0+0x204/0x270 [ 3071.412486] bfq_insert_requests+0x70/0x90 [ 3071.416570] blk_mq_sched_insert_requests+0xac/0x250 [ 3071.421521] blk_mq_flush_plug_list+0x110/0x2cc [ 3071.426038] __blk_flush_plug+0x100/0x140 [ 3071.430035] blk_finish_plug+0x40/0x25c [ 3071.433858] __iomap_dio_rw+0x1b4/0x500 [ 3071.437680] iomap_dio_rw+0x18/0x50 [ 3071.441156] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3071.446506] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3071.451248] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3071.456292] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3071.461422] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3071.466552] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3071.471939] process_one_work+0x1ec/0x470 [ 3071.475935] worker_thread+0x74/0x410 [ 3071.479584] kthread+0xec/0x100 [ 3071.482713] ret_from_fork+0x10/0x20 [ 3083.204229] watchdog: BUG: soft lockup - CPU#68 stuck for 2477s! [swapper/68:0] [ 3083.211524] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3083.211541] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3083.299329] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3083.307144] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3083.316435] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3083.323383] pc : __sbq_wake_up+0x1c8/0x1f0 [ 3083.327466] lr : sbitmap_queue_clear+0x74/0xc0 [ 3083.331897] sp : ffff8000088438b0 [ 3083.335198] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3083.342319] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3083.349442] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3083.356564] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3083.363685] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3083.370807] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3083.377929] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3083.385051] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3083.392173] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3083.399295] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 3083.406417] Call trace: [ 3083.408850] __sbq_wake_up+0x1c8/0x1f0 [ 3083.412586] sbitmap_queue_clear+0x74/0xc0 [ 3083.416669] blk_mq_put_tag+0x44/0x70 [ 3083.420318] __blk_mq_free_request+0x88/0xf0 [ 3083.424575] blk_mq_free_request+0xc0/0x160 [ 3083.428745] __blk_mq_end_request+0x78/0x150 [ 3083.433002] scsi_end_request+0x114/0x260 [ 3083.436998] scsi_io_completion+0x64/0x174 [ 3083.441081] scsi_finish_command+0xe0/0x120 [ 3083.445251] scsi_complete+0x8c/0x1c4 [ 3083.448900] blk_complete_reqs+0x5c/0x74 [ 3083.452810] blk_done_softirq+0x38/0x4c [ 3083.456633] __do_softirq+0x168/0x418 [ 3083.460281] __irq_exit_rcu+0x120/0x170 [ 3083.464104] irq_exit_rcu+0x18/0x24 [ 3083.467579] el1_interrupt+0x38/0x70 [ 3083.471142] el1h_64_irq_handler+0x18/0x24 [ 3083.475225] el1h_64_irq+0x68/0x6c [ 3083.478613] cpuidle_enter_state+0xc4/0x4e0 [ 3083.482783] cpuidle_enter+0x40/0x60 [ 3083.486345] cpuidle_idle_call+0x118/0x1b0 [ 3083.490429] do_idle+0xac/0x100 [ 3083.493558] cpu_startup_entry+0x30/0x34 [ 3083.497467] secondary_start_kernel+0xd8/0xf0 [ 3083.501811] __secondary_switched+0xb0/0xb4 [ 3095.094150] watchdog: BUG: soft lockup - CPU#8 stuck for 2488s! [kworker/8:12:19399] [ 3095.101880] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3095.101897] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3095.189685] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3095.197934] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3095.207225] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3095.213393] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3095.220341] pc : __sbq_wake_up+0x84/0x1f0 [ 3095.224337] lr : sbitmap_queue_clear+0x74/0xc0 [ 3095.228768] sp : ffff80000d79b280 [ 3095.232069] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3095.239191] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3095.246313] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3095.253435] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3095.260558] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3095.267681] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3095.274803] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3095.281925] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3095.289047] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3095.296168] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3095.303290] Call trace: [ 3095.305723] __sbq_wake_up+0x84/0x1f0 [ 3095.309372] sbitmap_queue_clear+0x74/0xc0 [ 3095.313455] blk_mq_put_tag+0x44/0x70 [ 3095.317106] __blk_mq_free_request+0x88/0xf0 [ 3095.321363] blk_mq_free_request+0xc0/0x160 [ 3095.325533] __blk_mq_end_request+0x78/0x150 [ 3095.329789] scsi_end_request+0x114/0x260 [ 3095.333786] scsi_io_completion+0x64/0x174 [ 3095.337869] scsi_finish_command+0xe0/0x120 [ 3095.342039] scsi_complete+0x8c/0x1c4 [ 3095.345688] blk_complete_reqs+0x5c/0x74 [ 3095.349598] blk_done_softirq+0x38/0x4c [ 3095.353420] __do_softirq+0x168/0x418 [ 3095.357069] __irq_exit_rcu+0x120/0x170 [ 3095.360892] irq_exit_rcu+0x18/0x24 [ 3095.364367] el1_interrupt+0x38/0x70 [ 3095.367929] el1h_64_irq_handler+0x18/0x24 [ 3095.372013] el1h_64_irq+0x68/0x6c [ 3095.375401] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3095.380179] __wake_up_common_lock+0x84/0xac [ 3095.384436] __wake_up+0x20/0x30 [ 3095.387651] __sbq_wake_up+0x13c/0x1f0 [ 3095.391387] sbitmap_queue_clear+0x74/0xc0 [ 3095.395470] blk_mq_put_tag+0x44/0x70 [ 3095.399120] __blk_mq_free_request+0x88/0xf0 [ 3095.403377] blk_mq_free_request+0xc0/0x160 [ 3095.407547] bfq_insert_request.isra.0+0x204/0x270 [ 3095.412325] bfq_insert_requests+0x70/0x90 [ 3095.416408] blk_mq_sched_insert_requests+0xac/0x250 [ 3095.421360] blk_mq_flush_plug_list+0x110/0x2cc [ 3095.425877] __blk_flush_plug+0x100/0x140 [ 3095.429873] blk_finish_plug+0x40/0x25c [ 3095.433696] __iomap_dio_rw+0x1b4/0x500 [ 3095.437519] iomap_dio_rw+0x18/0x50 [ 3095.440994] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3095.446345] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3095.451088] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3095.456132] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3095.461262] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3095.466391] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3095.471778] process_one_work+0x1ec/0x470 [ 3095.475774] worker_thread+0x74/0x410 [ 3095.479422] kthread+0xec/0x100 [ 3095.482551] ret_from_fork+0x10/0x20 [ 3107.204071] watchdog: BUG: soft lockup - CPU#68 stuck for 2499s! [swapper/68:0] [ 3107.211366] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3107.211383] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3107.299171] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3107.306986] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3107.316277] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3107.323225] pc : __sbq_wake_up+0x1c8/0x1f0 [ 3107.327308] lr : sbitmap_queue_clear+0x74/0xc0 [ 3107.331739] sp : ffff8000088438b0 [ 3107.335040] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3107.342162] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3107.349284] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3107.356405] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3107.363527] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3107.370650] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3107.377772] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3107.384893] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3107.392015] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3107.399137] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 3107.406259] Call trace: [ 3107.408694] __sbq_wake_up+0x1c8/0x1f0 [ 3107.412430] sbitmap_queue_clear+0x74/0xc0 [ 3107.416513] blk_mq_put_tag+0x44/0x70 [ 3107.420163] __blk_mq_free_request+0x88/0xf0 [ 3107.424420] blk_mq_free_request+0xc0/0x160 [ 3107.428589] __blk_mq_end_request+0x78/0x150 [ 3107.432846] scsi_end_request+0x114/0x260 [ 3107.436843] scsi_io_completion+0x64/0x174 [ 3107.440926] scsi_finish_command+0xe0/0x120 [ 3107.445096] scsi_complete+0x8c/0x1c4 [ 3107.448745] blk_complete_reqs+0x5c/0x74 [ 3107.452655] blk_done_softirq+0x38/0x4c [ 3107.456477] __do_softirq+0x168/0x418 [ 3107.460126] __irq_exit_rcu+0x120/0x170 [ 3107.463949] irq_exit_rcu+0x18/0x24 [ 3107.467425] el1_interrupt+0x38/0x70 [ 3107.470987] el1h_64_irq_handler+0x18/0x24 [ 3107.475070] el1h_64_irq+0x68/0x6c [ 3107.478458] cpuidle_enter_state+0xc4/0x4e0 [ 3107.482628] cpuidle_enter+0x40/0x60 [ 3107.486190] cpuidle_idle_call+0x118/0x1b0 [ 3107.490274] do_idle+0xac/0x100 [ 3107.493403] cpu_startup_entry+0x30/0x34 [ 3107.497312] secondary_start_kernel+0xd8/0xf0 [ 3107.501656] __secondary_switched+0xb0/0xb4 [ 3119.093992] watchdog: BUG: soft lockup - CPU#8 stuck for 2510s! [kworker/8:12:19399] [ 3119.101721] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3119.101739] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3119.189527] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3119.197776] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3119.207068] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3119.213235] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3119.220183] pc : __sbq_wake_up+0x24/0x1f0 [ 3119.224180] lr : sbitmap_queue_clear+0x74/0xc0 [ 3119.228610] sp : ffff80000d79b280 [ 3119.231912] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3119.239034] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3119.246156] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3119.253278] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3119.260400] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3119.267522] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3119.274644] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3119.281766] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3119.288888] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3119.296010] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 3119.303132] Call trace: [ 3119.305565] __sbq_wake_up+0x24/0x1f0 [ 3119.309215] sbitmap_queue_clear+0x74/0xc0 [ 3119.313298] blk_mq_put_tag+0x44/0x70 [ 3119.316948] __blk_mq_free_request+0x88/0xf0 [ 3119.321205] blk_mq_free_request+0xc0/0x160 [ 3119.325374] __blk_mq_end_request+0x78/0x150 [ 3119.329632] scsi_end_request+0x114/0x260 [ 3119.333628] scsi_io_completion+0x64/0x174 [ 3119.337711] scsi_finish_command+0xe0/0x120 [ 3119.341882] scsi_complete+0x8c/0x1c4 [ 3119.345531] blk_complete_reqs+0x5c/0x74 [ 3119.349440] blk_done_softirq+0x38/0x4c [ 3119.353263] __do_softirq+0x168/0x418 [ 3119.356912] __irq_exit_rcu+0x120/0x170 [ 3119.360735] irq_exit_rcu+0x18/0x24 [ 3119.364210] el1_interrupt+0x38/0x70 [ 3119.367772] el1h_64_irq_handler+0x18/0x24 [ 3119.371855] el1h_64_irq+0x68/0x6c [ 3119.375244] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3119.380022] __wake_up_common_lock+0x84/0xac [ 3119.384279] __wake_up+0x20/0x30 [ 3119.387495] __sbq_wake_up+0x13c/0x1f0 [ 3119.391231] sbitmap_queue_clear+0x74/0xc0 [ 3119.395315] blk_mq_put_tag+0x44/0x70 [ 3119.398965] __blk_mq_free_request+0x88/0xf0 [ 3119.403221] blk_mq_free_request+0xc0/0x160 [ 3119.407391] bfq_insert_request.isra.0+0x204/0x270 [ 3119.412169] bfq_insert_requests+0x70/0x90 [ 3119.416253] blk_mq_sched_insert_requests+0xac/0x250 [ 3119.421205] blk_mq_flush_plug_list+0x110/0x2cc [ 3119.425723] __blk_flush_plug+0x100/0x140 [ 3119.429719] blk_finish_plug+0x40/0x25c [ 3119.433542] __iomap_dio_rw+0x1b4/0x500 [ 3119.437365] iomap_dio_rw+0x18/0x50 [ 3119.440841] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3119.446192] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3119.450934] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3119.455978] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3119.461108] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3119.466238] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3119.471624] process_one_work+0x1ec/0x470 [ 3119.475620] worker_thread+0x74/0x410 [ 3119.479269] kthread+0xec/0x100 [ 3119.482398] ret_from_fork+0x10/0x20 [ 3131.203913] watchdog: BUG: soft lockup - CPU#68 stuck for 2522s! [swapper/68:0] [ 3131.211208] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3131.211225] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3131.299013] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3131.306828] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3131.316119] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3131.323067] pc : __sbq_wake_up+0x58/0x1f0 [ 3131.327063] lr : sbitmap_queue_clear+0x74/0xc0 [ 3131.331494] sp : ffff8000088438b0 [ 3131.334795] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3131.341917] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3131.349039] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3131.356161] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3131.363283] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3131.370405] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3131.377527] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3131.384648] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3131.391770] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3131.398892] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3131.406014] Call trace: [ 3131.408447] __sbq_wake_up+0x58/0x1f0 [ 3131.412096] sbitmap_queue_clear+0x74/0xc0 [ 3131.416179] blk_mq_put_tag+0x44/0x70 [ 3131.419828] __blk_mq_free_request+0x88/0xf0 [ 3131.424085] blk_mq_free_request+0xc0/0x160 [ 3131.428255] __blk_mq_end_request+0x78/0x150 [ 3131.432512] scsi_end_request+0x114/0x260 [ 3131.436509] scsi_io_completion+0x64/0x174 [ 3131.440592] scsi_finish_command+0xe0/0x120 [ 3131.444762] scsi_complete+0x8c/0x1c4 [ 3131.448412] blk_complete_reqs+0x5c/0x74 [ 3131.452321] blk_done_softirq+0x38/0x4c [ 3131.456144] __do_softirq+0x168/0x418 [ 3131.459792] __irq_exit_rcu+0x120/0x170 [ 3131.463615] irq_exit_rcu+0x18/0x24 [ 3131.467091] el1_interrupt+0x38/0x70 [ 3131.470653] el1h_64_irq_handler+0x18/0x24 [ 3131.474736] el1h_64_irq+0x68/0x6c [ 3131.478124] cpuidle_enter_state+0xc4/0x4e0 [ 3131.482295] cpuidle_enter+0x40/0x60 [ 3131.485857] cpuidle_idle_call+0x118/0x1b0 [ 3131.489941] do_idle+0xac/0x100 [ 3131.493069] cpu_startup_entry+0x30/0x34 [ 3131.496978] secondary_start_kernel+0xd8/0xf0 [ 3131.501322] __secondary_switched+0xb0/0xb4 [ 3134.433901] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 268349 jiffies s: 2093 root: 0x11/. [ 3134.445640] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3134.454848] Task dump for CPU 8: [ 3134.458063] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3134.467965] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3134.474135] Call trace: [ 3134.476568] __switch_to+0xd4/0x100 [ 3134.480045] 0xffff6c0200000080 [ 3134.483173] Task dump for CPU 68: [ 3134.486475] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3134.496377] Call trace: [ 3134.498810] __switch_to+0xd4/0x100 [ 3134.502286] 0x0 [ 3143.093834] watchdog: BUG: soft lockup - CPU#8 stuck for 2533s! [kworker/8:12:19399] [ 3143.101563] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3143.101580] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3143.189368] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3143.197618] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3143.206909] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3143.213076] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3143.220024] pc : __sbq_wake_up+0x58/0x1f0 [ 3143.224021] lr : sbitmap_queue_clear+0x74/0xc0 [ 3143.228452] sp : ffff80000d79b280 [ 3143.231753] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3143.238875] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3143.245997] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3143.253119] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3143.260242] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3143.267363] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3143.274485] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3143.281607] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3143.288729] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3143.295851] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3143.302973] Call trace: [ 3143.305406] __sbq_wake_up+0x58/0x1f0 [ 3143.309055] sbitmap_queue_clear+0x74/0xc0 [ 3143.313139] blk_mq_put_tag+0x44/0x70 [ 3143.316789] __blk_mq_free_request+0x88/0xf0 [ 3143.321045] blk_mq_free_request+0xc0/0x160 [ 3143.325215] __blk_mq_end_request+0x78/0x150 [ 3143.329472] scsi_end_request+0x114/0x260 [ 3143.333469] scsi_io_completion+0x64/0x174 [ 3143.337552] scsi_finish_command+0xe0/0x120 [ 3143.341722] scsi_complete+0x8c/0x1c4 [ 3143.345371] blk_complete_reqs+0x5c/0x74 [ 3143.349281] blk_done_softirq+0x38/0x4c [ 3143.353104] __do_softirq+0x168/0x418 [ 3143.356753] __irq_exit_rcu+0x120/0x170 [ 3143.360575] irq_exit_rcu+0x18/0x24 [ 3143.364051] el1_interrupt+0x38/0x70 [ 3143.367613] el1h_64_irq_handler+0x18/0x24 [ 3143.371696] el1h_64_irq+0x68/0x6c [ 3143.375085] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3143.379863] __wake_up_common_lock+0x84/0xac [ 3143.384120] __wake_up+0x20/0x30 [ 3143.387336] __sbq_wake_up+0x13c/0x1f0 [ 3143.391072] sbitmap_queue_clear+0x74/0xc0 [ 3143.395155] blk_mq_put_tag+0x44/0x70 [ 3143.398804] __blk_mq_free_request+0x88/0xf0 [ 3143.403061] blk_mq_free_request+0xc0/0x160 [ 3143.407231] bfq_insert_request.isra.0+0x204/0x270 [ 3143.412009] bfq_insert_requests+0x70/0x90 [ 3143.416093] blk_mq_sched_insert_requests+0xac/0x250 [ 3143.421044] blk_mq_flush_plug_list+0x110/0x2cc [ 3143.425562] __blk_flush_plug+0x100/0x140 [ 3143.429559] blk_finish_plug+0x40/0x25c [ 3143.433382] __iomap_dio_rw+0x1b4/0x500 [ 3143.437204] iomap_dio_rw+0x18/0x50 [ 3143.440680] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3143.446030] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3143.450773] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3143.455816] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3143.460947] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3143.466076] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3143.471463] process_one_work+0x1ec/0x470 [ 3143.475459] worker_thread+0x74/0x410 [ 3143.479108] kthread+0xec/0x100 [ 3143.482237] ret_from_fork+0x10/0x20 [ 3155.203754] watchdog: BUG: soft lockup - CPU#68 stuck for 2544s! [swapper/68:0] [ 3155.211050] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3155.211067] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3155.298854] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3155.306669] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3155.315960] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3155.322909] pc : __sbq_wake_up+0x58/0x1f0 [ 3155.326907] lr : sbitmap_queue_clear+0x74/0xc0 [ 3155.331337] sp : ffff8000088438b0 [ 3155.334638] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3155.341760] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3155.348883] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3155.356005] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3155.363126] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3155.370248] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3155.377370] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3155.384492] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3155.391614] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3155.398736] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3155.405858] Call trace: [ 3155.408291] __sbq_wake_up+0x58/0x1f0 [ 3155.411940] sbitmap_queue_clear+0x74/0xc0 [ 3155.416023] blk_mq_put_tag+0x44/0x70 [ 3155.419672] __blk_mq_free_request+0x88/0xf0 [ 3155.423929] blk_mq_free_request+0xc0/0x160 [ 3155.428099] __blk_mq_end_request+0x78/0x150 [ 3155.432356] scsi_end_request+0x114/0x260 [ 3155.436352] scsi_io_completion+0x64/0x174 [ 3155.440435] scsi_finish_command+0xe0/0x120 [ 3155.444606] scsi_complete+0x8c/0x1c4 [ 3155.448255] blk_complete_reqs+0x5c/0x74 [ 3155.452164] blk_done_softirq+0x38/0x4c [ 3155.455987] __do_softirq+0x168/0x418 [ 3155.459636] __irq_exit_rcu+0x120/0x170 [ 3155.463459] irq_exit_rcu+0x18/0x24 [ 3155.466934] el1_interrupt+0x38/0x70 [ 3155.470496] el1h_64_irq_handler+0x18/0x24 [ 3155.474580] el1h_64_irq+0x68/0x6c [ 3155.477968] cpuidle_enter_state+0xc4/0x4e0 [ 3155.482138] cpuidle_enter+0x40/0x60 [ 3155.485700] cpuidle_idle_call+0x118/0x1b0 [ 3155.489784] do_idle+0xac/0x100 [ 3155.492913] cpu_startup_entry+0x30/0x34 [ 3155.496822] secondary_start_kernel+0xd8/0xf0 [ 3155.501166] __secondary_switched+0xb0/0xb4 [ 3167.093676] watchdog: BUG: soft lockup - CPU#8 stuck for 2555s! [kworker/8:12:19399] [ 3167.101405] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3167.101423] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3167.189211] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3167.197461] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3167.206752] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3167.212920] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3167.219867] pc : __sbq_wake_up+0x40/0x1f0 [ 3167.223864] lr : sbitmap_queue_clear+0x74/0xc0 [ 3167.228295] sp : ffff80000d79b280 [ 3167.231596] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3167.238719] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3167.245840] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3167.252963] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3167.260086] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3167.267209] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3167.274330] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3167.281452] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3167.288574] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3167.295696] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 3167.302818] Call trace: [ 3167.305250] __sbq_wake_up+0x40/0x1f0 [ 3167.308900] sbitmap_queue_clear+0x74/0xc0 [ 3167.312984] blk_mq_put_tag+0x44/0x70 [ 3167.316633] __blk_mq_free_request+0x88/0xf0 [ 3167.320890] blk_mq_free_request+0xc0/0x160 [ 3167.325060] __blk_mq_end_request+0x78/0x150 [ 3167.329317] scsi_end_request+0x114/0x260 [ 3167.333313] scsi_io_completion+0x64/0x174 [ 3167.337396] scsi_finish_command+0xe0/0x120 [ 3167.341567] scsi_complete+0x8c/0x1c4 [ 3167.345215] blk_complete_reqs+0x5c/0x74 [ 3167.349125] blk_done_softirq+0x38/0x4c [ 3167.352948] __do_softirq+0x168/0x418 [ 3167.356596] __irq_exit_rcu+0x120/0x170 [ 3167.360419] irq_exit_rcu+0x18/0x24 [ 3167.363894] el1_interrupt+0x38/0x70 [ 3167.367457] el1h_64_irq_handler+0x18/0x24 [ 3167.371540] el1h_64_irq+0x68/0x6c [ 3167.374928] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3167.379706] __wake_up_common_lock+0x84/0xac [ 3167.383963] __wake_up+0x20/0x30 [ 3167.387179] __sbq_wake_up+0x13c/0x1f0 [ 3167.390915] sbitmap_queue_clear+0x74/0xc0 [ 3167.394998] blk_mq_put_tag+0x44/0x70 [ 3167.398648] __blk_mq_free_request+0x88/0xf0 [ 3167.402905] blk_mq_free_request+0xc0/0x160 [ 3167.407075] bfq_insert_request.isra.0+0x204/0x270 [ 3167.411853] bfq_insert_requests+0x70/0x90 [ 3167.415936] blk_mq_sched_insert_requests+0xac/0x250 [ 3167.420889] blk_mq_flush_plug_list+0x110/0x2cc [ 3167.425406] __blk_flush_plug+0x100/0x140 [ 3167.429402] blk_finish_plug+0x40/0x25c [ 3167.433225] __iomap_dio_rw+0x1b4/0x500 [ 3167.437048] iomap_dio_rw+0x18/0x50 [ 3167.440523] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3167.445874] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3167.450616] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3167.455661] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3167.460791] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3167.465920] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3167.471307] process_one_work+0x1ec/0x470 [ 3167.475303] worker_thread+0x74/0x410 [ 3167.478952] kthread+0xec/0x100 [ 3167.482081] ret_from_fork+0x10/0x20 [ 3179.203596] watchdog: BUG: soft lockup - CPU#68 stuck for 2566s! [swapper/68:0] [ 3179.210891] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3179.210908] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3179.298695] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3179.306511] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3179.315802] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3179.322749] pc : __sbq_wake_up+0x1c8/0x1f0 [ 3179.326833] lr : sbitmap_queue_clear+0x74/0xc0 [ 3179.331263] sp : ffff8000088438b0 [ 3179.334565] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3179.341687] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3179.348809] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3179.355931] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3179.363053] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3179.370175] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3179.377297] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3179.384419] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3179.391540] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3179.398662] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 3179.405784] Call trace: [ 3179.408217] __sbq_wake_up+0x1c8/0x1f0 [ 3179.411953] sbitmap_queue_clear+0x74/0xc0 [ 3179.416036] blk_mq_put_tag+0x44/0x70 [ 3179.419686] __blk_mq_free_request+0x88/0xf0 [ 3179.423943] blk_mq_free_request+0xc0/0x160 [ 3179.428113] __blk_mq_end_request+0x78/0x150 [ 3179.432371] scsi_end_request+0x114/0x260 [ 3179.436368] scsi_io_completion+0x64/0x174 [ 3179.440451] scsi_finish_command+0xe0/0x120 [ 3179.444621] scsi_complete+0x8c/0x1c4 [ 3179.448270] blk_complete_reqs+0x5c/0x74 [ 3179.452180] blk_done_softirq+0x38/0x4c [ 3179.456002] __do_softirq+0x168/0x418 [ 3179.459651] __irq_exit_rcu+0x120/0x170 [ 3179.463474] irq_exit_rcu+0x18/0x24 [ 3179.466949] el1_interrupt+0x38/0x70 [ 3179.470512] el1h_64_irq_handler+0x18/0x24 [ 3179.474596] el1h_64_irq+0x68/0x6c [ 3179.477985] cpuidle_enter_state+0xc4/0x4e0 [ 3179.482155] cpuidle_enter+0x40/0x60 [ 3179.485717] cpuidle_idle_call+0x118/0x1b0 [ 3179.489801] do_idle+0xac/0x100 [ 3179.492929] cpu_startup_entry+0x30/0x34 [ 3179.496838] secondary_start_kernel+0xd8/0xf0 [ 3179.501182] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:50:00 2022] [ 3189.563528] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3189.569261] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=137129 [ 3189.578383] (t=276469 jiffies g=36445 q=81069 ncpus=128) [ 3189.583768] Task dump for CPU 8: [ 3189.586982] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3189.596882] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3189.603050] Call trace: [ 3189.605483] dump_backtrace+0xe8/0x130 [ 3189.609220] show_stack+0x20/0x60 [ 3189.612521] sched_show_task+0x160/0x190 [ 3189.616431] dump_cpu_task+0x4c/0x78 [ 3189.619994] rcu_dump_cpu_stacks+0xc4/0x150 [ 3189.624165] print_cpu_stall+0x180/0x210 [ 3189.628074] check_cpu_stall+0x20c/0x2b4 [ 3189.631985] rcu_pending+0x4c/0x190 [ 3189.635461] rcu_sched_clock_irq+0x80/0x204 [ 3189.639631] update_process_times+0x7c/0xc0 [ 3189.643801] tick_sched_handle+0x38/0x74 [ 3189.647712] tick_sched_timer+0x60/0xc0 [ 3189.651535] __hrtimer_run_queues+0x170/0x300 [ 3189.655878] hrtimer_interrupt+0xfc/0x260 [ 3189.659875] arch_timer_handler_phys+0x34/0x50 [ 3189.664306] handle_percpu_devid_irq+0xac/0x240 [ 3189.668823] generic_handle_domain_irq+0x34/0x50 [ 3189.673428] gic_handle_irq+0x5c/0xe0 [ 3189.677077] call_on_irq_stack+0x2c/0x38 [ 3189.680987] do_interrupt_handler+0xdc/0xe0 [ 3189.685157] el1_interrupt+0x34/0x70 [ 3189.688720] el1h_64_irq_handler+0x18/0x24 [ 3189.692803] el1h_64_irq+0x68/0x6c [ 3189.696192] __sbq_wake_up+0x58/0x1f0 [ 3189.699842] sbitmap_queue_clear+0x74/0xc0 [ 3189.703925] blk_mq_put_tag+0x44/0x70 [ 3189.707575] __blk_mq_free_request+0x88/0xf0 [ 3189.711832] blk_mq_free_request+0xc0/0x160 [ 3189.716001] __blk_mq_end_request+0x78/0x150 [ 3189.720259] scsi_end_request+0x114/0x260 [ 3189.724255] scsi_io_completion+0x64/0x174 [ 3189.728338] scsi_finish_command+0xe0/0x120 [ 3189.732509] scsi_complete+0x8c/0x1c4 [ 3189.736158] blk_complete_reqs+0x5c/0x74 [ 3189.740067] blk_done_softirq+0x38/0x4c [ 3189.743890] __do_softirq+0x168/0x418 [ 3189.747539] __irq_exit_rcu+0x120/0x170 [ 3189.751362] irq_exit_rcu+0x18/0x24 [ 3189.754837] el1_interrupt+0x38/0x70 [ 3189.758399] el1h_64_irq_handler+0x18/0x24 [ 3189.762482] el1h_64_irq+0x68/0x6c [ 3189.765871] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3189.770649] __wake_up_common_lock+0x84/0xac [ 3189.774906] __wake_up+0x20/0x30 [ 3189.778121] __sbq_wake_up+0x13c/0x1f0 [ 3189.781857] sbitmap_queue_clear+0x74/0xc0 [ 3189.785940] blk_mq_put_tag+0x44/0x70 [ 3189.789590] __blk_mq_free_request+0x88/0xf0 [ 3189.793847] blk_mq_free_request+0xc0/0x160 [ 3189.798017] bfq_insert_request.isra.0+0x204/0x270 [ 3189.802795] bfq_insert_requests+0x70/0x90 [ 3189.806878] blk_mq_sched_insert_requests+0xac/0x250 [ 3189.811830] blk_mq_flush_plug_list+0x110/0x2cc [ 3189.816347] __blk_flush_plug+0x100/0x140 [ 3189.820343] blk_finish_plug+0x40/0x25c [ 3189.824166] __iomap_dio_rw+0x1b4/0x500 [ 3189.827989] iomap_dio_rw+0x18/0x50 [ 3189.831464] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3189.836815] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3189.841558] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3189.846601] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3189.851731] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3189.856861] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3189.862247] process_one_work+0x1ec/0x470 [ 3189.866243] worker_thread+0x74/0x410 [ 3189.869892] kthread+0xec/0x100 [ 3189.873021] ret_from_fork+0x10/0x20 [ 3189.876584] Task dump for CPU 68: [ 3189.879885] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3189.889785] Call trace: [ 3189.892218] __switch_to+0xd4/0x100 [ 3189.895694] 0x0 [ 3200.993463] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 275005 jiffies s: 2093 root: 0x11/. [ 3201.005201] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3201.014410] Task dump for CPU 8: [ 3201.017625] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3201.027527] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3201.033697] Call trace: [ 3201.036130] __switch_to+0xd4/0x100 [ 3201.039606] 0xffff6c0200000080 [ 3201.042735] Task dump for CPU 68: [ 3201.046037] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3201.055939] Call trace: [ 3201.058372] __switch_to+0xd4/0x100 [ 3201.061848] 0x0 [ 3203.203438] watchdog: BUG: soft lockup - CPU#68 stuck for 2589s! [swapper/68:0] [ 3203.210733] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3203.210750] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3203.298538] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3203.306353] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3203.315644] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3203.322592] pc : __sbq_wake_up+0x58/0x1f0 [ 3203.326588] lr : sbitmap_queue_clear+0x74/0xc0 [ 3203.331018] sp : ffff8000088438b0 [ 3203.334320] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3203.341442] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3203.348564] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3203.355686] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3203.362808] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3203.369930] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3203.377052] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3203.384174] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3203.391296] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3203.398418] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3203.405540] Call trace: [ 3203.407974] __sbq_wake_up+0x58/0x1f0 [ 3203.411623] sbitmap_queue_clear+0x74/0xc0 [ 3203.415707] blk_mq_put_tag+0x44/0x70 [ 3203.419356] __blk_mq_free_request+0x88/0xf0 [ 3203.423613] blk_mq_free_request+0xc0/0x160 [ 3203.427783] __blk_mq_end_request+0x78/0x150 [ 3203.432040] scsi_end_request+0x114/0x260 [ 3203.436036] scsi_io_completion+0x64/0x174 [ 3203.440119] scsi_finish_command+0xe0/0x120 [ 3203.444290] scsi_complete+0x8c/0x1c4 [ 3203.447939] blk_complete_reqs+0x5c/0x74 [ 3203.451848] blk_done_softirq+0x38/0x4c [ 3203.455671] __do_softirq+0x168/0x418 [ 3203.459319] __irq_exit_rcu+0x120/0x170 [ 3203.463142] irq_exit_rcu+0x18/0x24 [ 3203.466617] el1_interrupt+0x38/0x70 [ 3203.470180] el1h_64_irq_handler+0x18/0x24 [ 3203.474263] el1h_64_irq+0x68/0x6c [ 3203.477653] cpuidle_enter_state+0xc4/0x4e0 [ 3203.481823] cpuidle_enter+0x40/0x60 [ 3203.485385] cpuidle_idle_call+0x118/0x1b0 [ 3203.489469] do_idle+0xac/0x100 [ 3203.492598] cpu_startup_entry+0x30/0x34 [ 3203.496507] secondary_start_kernel+0xd8/0xf0 [ 3203.500851] __secondary_switched+0xb0/0xb4 [ 3215.093360] watchdog: BUG: soft lockup - CPU#8 stuck for 2600s! [kworker/8:12:19399] [ 3215.101089] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3215.101107] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3215.188895] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3215.197145] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3215.206435] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3215.212603] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3215.219550] pc : __sbq_wake_up+0x58/0x1f0 [ 3215.223547] lr : sbitmap_queue_clear+0x74/0xc0 [ 3215.227978] sp : ffff80000d79b280 [ 3215.231279] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3215.238402] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3215.245524] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3215.252646] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3215.259768] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3215.266890] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3215.274011] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3215.281133] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3215.288255] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3215.295377] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3215.302499] Call trace: [ 3215.304933] __sbq_wake_up+0x58/0x1f0 [ 3215.308582] sbitmap_queue_clear+0x74/0xc0 [ 3215.312665] blk_mq_put_tag+0x44/0x70 [ 3215.316315] __blk_mq_free_request+0x88/0xf0 [ 3215.320572] blk_mq_free_request+0xc0/0x160 [ 3215.324742] __blk_mq_end_request+0x78/0x150 [ 3215.328999] scsi_end_request+0x114/0x260 [ 3215.332995] scsi_io_completion+0x64/0x174 [ 3215.337078] scsi_finish_command+0xe0/0x120 [ 3215.341249] scsi_complete+0x8c/0x1c4 [ 3215.344898] blk_complete_reqs+0x5c/0x74 [ 3215.348807] blk_done_softirq+0x38/0x4c [ 3215.352630] __do_softirq+0x168/0x418 [ 3215.356279] __irq_exit_rcu+0x120/0x170 [ 3215.360102] irq_exit_rcu+0x18/0x24 [ 3215.363577] el1_interrupt+0x38/0x70 [ 3215.367139] el1h_64_irq_handler+0x18/0x24 [ 3215.371222] el1h_64_irq+0x68/0x6c [ 3215.374611] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3215.379389] __wake_up_common_lock+0x84/0xac [ 3215.383646] __wake_up+0x20/0x30 [ 3215.386862] __sbq_wake_up+0x13c/0x1f0 [ 3215.390598] sbitmap_queue_clear+0x74/0xc0 [ 3215.394682] blk_mq_put_tag+0x44/0x70 [ 3215.398331] __blk_mq_free_request+0x88/0xf0 [ 3215.402588] blk_mq_free_request+0xc0/0x160 [ 3215.406758] bfq_insert_request.isra.0+0x204/0x270 [ 3215.411536] bfq_insert_requests+0x70/0x90 [ 3215.415620] blk_mq_sched_insert_requests+0xac/0x250 [ 3215.420572] blk_mq_flush_plug_list+0x110/0x2cc [ 3215.425089] __blk_flush_plug+0x100/0x140 [ 3215.429086] blk_finish_plug+0x40/0x25c [ 3215.432908] __iomap_dio_rw+0x1b4/0x500 [ 3215.436731] iomap_dio_rw+0x18/0x50 [ 3215.440207] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3215.445558] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3215.450301] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3215.455345] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3215.460475] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3215.465605] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3215.470991] process_one_work+0x1ec/0x470 [ 3215.474987] worker_thread+0x74/0x410 [ 3215.478636] kthread+0xec/0x100 [ 3215.481766] ret_from_fork+0x10/0x20 [ 3227.203279] watchdog: BUG: soft lockup - CPU#68 stuck for 2611s! [swapper/68:0] [ 3227.210574] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3227.210591] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3227.298379] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3227.306195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3227.315486] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3227.322433] pc : __sbq_wake_up+0x84/0x1f0 [ 3227.326430] lr : sbitmap_queue_clear+0x74/0xc0 [ 3227.330860] sp : ffff8000088438b0 [ 3227.334162] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3227.341283] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3227.348406] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3227.355528] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3227.362650] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3227.369772] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3227.376894] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3227.384016] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3227.391138] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3227.398260] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3227.405382] Call trace: [ 3227.407815] __sbq_wake_up+0x84/0x1f0 [ 3227.411464] sbitmap_queue_clear+0x74/0xc0 [ 3227.415547] blk_mq_put_tag+0x44/0x70 [ 3227.419197] __blk_mq_free_request+0x88/0xf0 [ 3227.423453] blk_mq_free_request+0xc0/0x160 [ 3227.427623] __blk_mq_end_request+0x78/0x150 [ 3227.431880] scsi_end_request+0x114/0x260 [ 3227.435876] scsi_io_completion+0x64/0x174 [ 3227.439959] scsi_finish_command+0xe0/0x120 [ 3227.444130] scsi_complete+0x8c/0x1c4 [ 3227.447779] blk_complete_reqs+0x5c/0x74 [ 3227.451688] blk_done_softirq+0x38/0x4c [ 3227.455511] __do_softirq+0x168/0x418 [ 3227.459160] __irq_exit_rcu+0x120/0x170 [ 3227.462983] irq_exit_rcu+0x18/0x24 [ 3227.466458] el1_interrupt+0x38/0x70 [ 3227.470020] el1h_64_irq_handler+0x18/0x24 [ 3227.474104] el1h_64_irq+0x68/0x6c [ 3227.477492] cpuidle_enter_state+0xc4/0x4e0 [ 3227.481662] cpuidle_enter+0x40/0x60 [ 3227.485224] cpuidle_idle_call+0x118/0x1b0 [ 3227.489308] do_idle+0xac/0x100 [ 3227.492437] cpu_startup_entry+0x30/0x34 [ 3227.496346] secondary_start_kernel+0xd8/0xf0 [ 3227.500690] __secondary_switched+0xb0/0xb4 [ 3239.093200] watchdog: BUG: soft lockup - CPU#8 stuck for 2622s! [kworker/8:12:19399] [ 3239.100929] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3239.100947] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3239.188734] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3239.196984] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3239.206275] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3239.212442] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3239.219390] pc : __sbq_wake_up+0x58/0x1f0 [ 3239.223387] lr : sbitmap_queue_clear+0x74/0xc0 [ 3239.227818] sp : ffff80000d79b280 [ 3239.231119] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3239.238241] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3239.245363] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3239.252485] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3239.259606] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3239.266728] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3239.273850] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3239.280972] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3239.288094] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3239.295216] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3239.302337] Call trace: [ 3239.304770] __sbq_wake_up+0x58/0x1f0 [ 3239.308419] sbitmap_queue_clear+0x74/0xc0 [ 3239.312503] blk_mq_put_tag+0x44/0x70 [ 3239.316152] __blk_mq_free_request+0x88/0xf0 [ 3239.320409] blk_mq_free_request+0xc0/0x160 [ 3239.324579] __blk_mq_end_request+0x78/0x150 [ 3239.328837] scsi_end_request+0x114/0x260 [ 3239.332834] scsi_io_completion+0x64/0x174 [ 3239.336917] scsi_finish_command+0xe0/0x120 [ 3239.341087] scsi_complete+0x8c/0x1c4 [ 3239.344736] blk_complete_reqs+0x5c/0x74 [ 3239.348646] blk_done_softirq+0x38/0x4c [ 3239.352469] __do_softirq+0x168/0x418 [ 3239.356118] __irq_exit_rcu+0x120/0x170 [ 3239.359940] irq_exit_rcu+0x18/0x24 [ 3239.363416] el1_interrupt+0x38/0x70 [ 3239.366978] el1h_64_irq_handler+0x18/0x24 [ 3239.371061] el1h_64_irq+0x68/0x6c [ 3239.374450] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3239.379227] __wake_up_common_lock+0x84/0xac [ 3239.383485] __wake_up+0x20/0x30 [ 3239.386701] __sbq_wake_up+0x13c/0x1f0 [ 3239.390437] sbitmap_queue_clear+0x74/0xc0 [ 3239.394520] blk_mq_put_tag+0x44/0x70 [ 3239.398169] __blk_mq_free_request+0x88/0xf0 [ 3239.402426] blk_mq_free_request+0xc0/0x160 [ 3239.406596] bfq_insert_request.isra.0+0x204/0x270 [ 3239.411374] bfq_insert_requests+0x70/0x90 [ 3239.415458] blk_mq_sched_insert_requests+0xac/0x250 [ 3239.420409] blk_mq_flush_plug_list+0x110/0x2cc [ 3239.424927] __blk_flush_plug+0x100/0x140 [ 3239.428923] blk_finish_plug+0x40/0x25c [ 3239.432746] __iomap_dio_rw+0x1b4/0x500 [ 3239.436569] iomap_dio_rw+0x18/0x50 [ 3239.440044] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3239.445395] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3239.450137] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3239.455181] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3239.460311] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3239.465441] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3239.470827] process_one_work+0x1ec/0x470 [ 3239.474823] worker_thread+0x74/0x410 [ 3239.478472] kthread+0xec/0x100 [ 3239.481600] ret_from_fork+0x10/0x20 [ 3251.203120] watchdog: BUG: soft lockup - CPU#68 stuck for 2633s! [swapper/68:0] [ 3251.210415] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3251.210432] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3251.298220] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3251.306036] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3251.315327] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3251.322274] pc : __sbq_wake_up+0x58/0x1f0 [ 3251.326271] lr : sbitmap_queue_clear+0x74/0xc0 [ 3251.330701] sp : ffff8000088438b0 [ 3251.334002] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3251.341124] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3251.348247] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3251.355368] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3251.362490] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3251.369613] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3251.376736] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3251.383857] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3251.390979] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3251.398101] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3251.405223] Call trace: [ 3251.407655] __sbq_wake_up+0x58/0x1f0 [ 3251.411305] sbitmap_queue_clear+0x74/0xc0 [ 3251.415388] blk_mq_put_tag+0x44/0x70 [ 3251.419038] __blk_mq_free_request+0x88/0xf0 [ 3251.423295] blk_mq_free_request+0xc0/0x160 [ 3251.427464] __blk_mq_end_request+0x78/0x150 [ 3251.431721] scsi_end_request+0x114/0x260 [ 3251.435718] scsi_io_completion+0x64/0x174 [ 3251.439801] scsi_finish_command+0xe0/0x120 [ 3251.443971] scsi_complete+0x8c/0x1c4 [ 3251.447622] blk_complete_reqs+0x5c/0x74 [ 3251.451531] blk_done_softirq+0x38/0x4c [ 3251.455354] __do_softirq+0x168/0x418 [ 3251.459003] __irq_exit_rcu+0x120/0x170 [ 3251.462826] irq_exit_rcu+0x18/0x24 [ 3251.466301] el1_interrupt+0x38/0x70 [ 3251.469863] el1h_64_irq_handler+0x18/0x24 [ 3251.473947] el1h_64_irq+0x68/0x6c [ 3251.477335] cpuidle_enter_state+0xc4/0x4e0 [ 3251.481506] cpuidle_enter+0x40/0x60 [ 3251.485068] cpuidle_idle_call+0x118/0x1b0 [ 3251.489152] do_idle+0xac/0x100 [ 3251.492280] cpu_startup_entry+0x30/0x34 [ 3251.496189] secondary_start_kernel+0xd8/0xf0 [ 3251.500533] __secondary_switched+0xb0/0xb4 [ 3263.093041] watchdog: BUG: soft lockup - CPU#8 stuck for 2644s! [kworker/8:12:19399] [ 3263.100770] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3263.100787] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3263.188575] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3263.196825] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3263.206116] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3263.212283] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3263.219231] pc : __sbq_wake_up+0x7c/0x1f0 [ 3263.223228] lr : sbitmap_queue_clear+0x74/0xc0 [ 3263.227659] sp : ffff80000d79b2d0 [ 3263.230960] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3263.238082] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3263.245204] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3263.252326] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3263.259448] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3263.266571] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3263.273693] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3263.280815] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3263.287937] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3263.295059] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3263.302180] Call trace: [ 3263.304614] __sbq_wake_up+0x7c/0x1f0 [ 3263.308263] blk_mq_put_tag+0x44/0x70 [ 3263.311912] __blk_mq_free_request+0x88/0xf0 [ 3263.316169] blk_mq_free_request+0xc0/0x160 [ 3263.320341] __blk_mq_end_request+0x78/0x150 [ 3263.324598] scsi_end_request+0x114/0x260 [ 3263.328594] scsi_io_completion+0x64/0x174 [ 3263.332677] scsi_finish_command+0xe0/0x120 [ 3263.336848] scsi_complete+0x8c/0x1c4 [ 3263.340497] blk_complete_reqs+0x5c/0x74 [ 3263.344406] blk_done_softirq+0x38/0x4c [ 3263.348229] __do_softirq+0x168/0x418 [ 3263.351878] __irq_exit_rcu+0x120/0x170 [ 3263.355701] irq_exit_rcu+0x18/0x24 [ 3263.359176] el1_interrupt+0x38/0x70 [ 3263.362738] el1h_64_irq_handler+0x18/0x24 [ 3263.366821] el1h_64_irq+0x68/0x6c [ 3263.370210] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3263.374988] __wake_up_common_lock+0x84/0xac [ 3263.379245] __wake_up+0x20/0x30 [ 3263.382461] __sbq_wake_up+0x13c/0x1f0 [ 3263.386197] sbitmap_queue_clear+0x74/0xc0 [ 3263.390280] blk_mq_put_tag+0x44/0x70 [ 3263.393930] __blk_mq_free_request+0x88/0xf0 [ 3263.398186] blk_mq_free_request+0xc0/0x160 [ 3263.402356] bfq_insert_request.isra.0+0x204/0x270 [ 3263.407134] bfq_insert_requests+0x70/0x90 [ 3263.411217] blk_mq_sched_insert_requests+0xac/0x250 [ 3263.416170] blk_mq_flush_plug_list+0x110/0x2cc [ 3263.420688] __blk_flush_plug+0x100/0x140 [ 3263.424684] blk_finish_plug+0x40/0x25c [ 3263.428507] __iomap_dio_rw+0x1b4/0x500 [ 3263.432330] iomap_dio_rw+0x18/0x50 [ 3263.435805] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3263.441156] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3263.445898] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3263.450942] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3263.456072] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3263.461202] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3263.466588] process_one_work+0x1ec/0x470 [ 3263.470585] worker_thread+0x74/0x410 [ 3263.474234] kthread+0xec/0x100 [ 3263.477362] ret_from_fork+0x10/0x20 [ 3267.553021] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 281661 jiffies s: 2093 root: 0x11/. [ 3267.564759] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3267.573968] Task dump for CPU 8: [ 3267.577183] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3267.587084] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3267.593254] Call trace: [ 3267.595686] __switch_to+0xd4/0x100 [ 3267.599163] 0xffff6c0200000080 [ 3267.602292] Task dump for CPU 68: [ 3267.605594] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3267.615496] Call trace: [ 3267.617929] __switch_to+0xd4/0x100 [ 3267.621406] 0x0 [ 3275.202960] watchdog: BUG: soft lockup - CPU#68 stuck for 2656s! [swapper/68:0] [ 3275.210256] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3275.210273] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3275.298061] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3275.305876] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3275.315167] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3275.322115] pc : __sbq_wake_up+0x7c/0x1f0 [ 3275.326113] lr : sbitmap_queue_clear+0x74/0xc0 [ 3275.330544] sp : ffff800008843900 [ 3275.333845] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3275.340967] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3275.348088] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3275.355210] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3275.362332] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3275.369454] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3275.376577] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3275.383699] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3275.390820] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3275.397943] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3275.405065] Call trace: [ 3275.407498] __sbq_wake_up+0x7c/0x1f0 [ 3275.411147] blk_mq_put_tag+0x44/0x70 [ 3275.414797] __blk_mq_free_request+0x88/0xf0 [ 3275.419053] blk_mq_free_request+0xc0/0x160 [ 3275.423223] __blk_mq_end_request+0x78/0x150 [ 3275.427480] scsi_end_request+0x114/0x260 [ 3275.431476] scsi_io_completion+0x64/0x174 [ 3275.435559] scsi_finish_command+0xe0/0x120 [ 3275.439730] scsi_complete+0x8c/0x1c4 [ 3275.443379] blk_complete_reqs+0x5c/0x74 [ 3275.447288] blk_done_softirq+0x38/0x4c [ 3275.451111] __do_softirq+0x168/0x418 [ 3275.454760] __irq_exit_rcu+0x120/0x170 [ 3275.458582] irq_exit_rcu+0x18/0x24 [ 3275.462058] el1_interrupt+0x38/0x70 [ 3275.465620] el1h_64_irq_handler+0x18/0x24 [ 3275.469704] el1h_64_irq+0x68/0x6c [ 3275.473092] cpuidle_enter_state+0xc4/0x4e0 [ 3275.477262] cpuidle_enter+0x40/0x60 [ 3275.480825] cpuidle_idle_call+0x118/0x1b0 [ 3275.484908] do_idle+0xac/0x100 [ 3275.488037] cpu_startup_entry+0x30/0x34 [ 3275.491946] secondary_start_kernel+0xd8/0xf0 [ 3275.496290] __secondary_switched+0xb0/0xb4 [ 3287.092881] watchdog: BUG: soft lockup - CPU#8 stuck for 2667s! [kworker/8:12:19399] [ 3287.100610] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3287.100628] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3287.188415] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3287.196665] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3287.205956] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3287.212123] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3287.219071] pc : __sbq_wake_up+0x84/0x1f0 [ 3287.223068] lr : sbitmap_queue_clear+0x74/0xc0 [ 3287.227498] sp : ffff80000d79b280 [ 3287.230800] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3287.237922] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3287.245044] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3287.252166] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3287.259290] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3287.266411] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3287.273534] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3287.280656] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3287.287778] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3287.294900] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3287.302022] Call trace: [ 3287.304455] __sbq_wake_up+0x84/0x1f0 [ 3287.308105] sbitmap_queue_clear+0x74/0xc0 [ 3287.312188] blk_mq_put_tag+0x44/0x70 [ 3287.315838] __blk_mq_free_request+0x88/0xf0 [ 3287.320095] blk_mq_free_request+0xc0/0x160 [ 3287.324264] __blk_mq_end_request+0x78/0x150 [ 3287.328522] scsi_end_request+0x114/0x260 [ 3287.332518] scsi_io_completion+0x64/0x174 [ 3287.336602] scsi_finish_command+0xe0/0x120 [ 3287.340773] scsi_complete+0x8c/0x1c4 [ 3287.344422] blk_complete_reqs+0x5c/0x74 [ 3287.348331] blk_done_softirq+0x38/0x4c [ 3287.352154] __do_softirq+0x168/0x418 [ 3287.355803] __irq_exit_rcu+0x120/0x170 [ 3287.359626] irq_exit_rcu+0x18/0x24 [ 3287.363101] el1_interrupt+0x38/0x70 [ 3287.366663] el1h_64_irq_handler+0x18/0x24 [ 3287.370746] el1h_64_irq+0x68/0x6c [ 3287.374135] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3287.378913] __wake_up_common_lock+0x84/0xac [ 3287.383170] __wake_up+0x20/0x30 [ 3287.386385] __sbq_wake_up+0x13c/0x1f0 [ 3287.390122] sbitmap_queue_clear+0x74/0xc0 [ 3287.394205] blk_mq_put_tag+0x44/0x70 [ 3287.397855] __blk_mq_free_request+0x88/0xf0 [ 3287.402111] blk_mq_free_request+0xc0/0x160 [ 3287.406281] bfq_insert_request.isra.0+0x204/0x270 [ 3287.411059] bfq_insert_requests+0x70/0x90 [ 3287.415143] blk_mq_sched_insert_requests+0xac/0x250 [ 3287.420095] blk_mq_flush_plug_list+0x110/0x2cc [ 3287.424612] __blk_flush_plug+0x100/0x140 [ 3287.428609] blk_finish_plug+0x40/0x25c [ 3287.432432] __iomap_dio_rw+0x1b4/0x500 [ 3287.436255] iomap_dio_rw+0x18/0x50 [ 3287.439730] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3287.445080] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3287.449823] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3287.454867] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3287.459997] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3287.465126] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3287.470513] process_one_work+0x1ec/0x470 [ 3287.474509] worker_thread+0x74/0x410 [ 3287.478158] kthread+0xec/0x100 [ 3287.481286] ret_from_fork+0x10/0x20 [ 3299.202801] watchdog: BUG: soft lockup - CPU#68 stuck for 2678s! [swapper/68:0] [ 3299.210096] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3299.210113] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3299.297900] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3299.305716] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3299.315007] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3299.321954] pc : __sbq_wake_up+0x8/0x1f0 [ 3299.325864] lr : sbitmap_queue_clear+0x74/0xc0 [ 3299.330295] sp : ffff8000088438b0 [ 3299.333596] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3299.340719] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3299.347842] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3299.354964] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3299.362086] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3299.369207] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3299.376329] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3299.383452] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3299.390573] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3299.397695] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 3299.404817] Call trace: [ 3299.407250] __sbq_wake_up+0x8/0x1f0 [ 3299.410812] blk_mq_put_tag+0x44/0x70 [ 3299.414462] __blk_mq_free_request+0x88/0xf0 [ 3299.418719] blk_mq_free_request+0xc0/0x160 [ 3299.422890] __blk_mq_end_request+0x78/0x150 [ 3299.427147] scsi_end_request+0x114/0x260 [ 3299.431143] scsi_io_completion+0x64/0x174 [ 3299.435227] scsi_finish_command+0xe0/0x120 [ 3299.439397] scsi_complete+0x8c/0x1c4 [ 3299.443046] blk_complete_reqs+0x5c/0x74 [ 3299.446956] blk_done_softirq+0x38/0x4c [ 3299.450778] __do_softirq+0x168/0x418 [ 3299.454427] __irq_exit_rcu+0x120/0x170 [ 3299.458250] irq_exit_rcu+0x18/0x24 [ 3299.461725] el1_interrupt+0x38/0x70 [ 3299.465288] el1h_64_irq_handler+0x18/0x24 [ 3299.469371] el1h_64_irq+0x68/0x6c [ 3299.472759] cpuidle_enter_state+0xc4/0x4e0 [ 3299.476929] cpuidle_enter+0x40/0x60 [ 3299.480491] cpuidle_idle_call+0x118/0x1b0 [ 3299.484575] do_idle+0xac/0x100 [ 3299.487704] cpu_startup_entry+0x30/0x34 [ 3299.491613] secondary_start_kernel+0xd8/0xf0 [ 3299.495958] __secondary_switched+0xb0/0xb4 [ 3311.092722] watchdog: BUG: soft lockup - CPU#8 stuck for 2689s! [kworker/8:12:19399] [ 3311.100451] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3311.100468] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3311.188257] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3311.196507] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3311.205798] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3311.211966] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3311.218913] pc : __sbq_wake_up+0xa4/0x1f0 [ 3311.222910] lr : sbitmap_queue_clear+0x74/0xc0 [ 3311.227341] sp : ffff80000d79b280 [ 3311.230642] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3311.237764] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3311.244887] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3311.252009] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3311.259131] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3311.266253] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3311.273375] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3311.280497] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3311.287619] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3311.294741] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 3311.301863] Call trace: [ 3311.304296] __sbq_wake_up+0xa4/0x1f0 [ 3311.307946] sbitmap_queue_clear+0x74/0xc0 [ 3311.312029] blk_mq_put_tag+0x44/0x70 [ 3311.315679] __blk_mq_free_request+0x88/0xf0 [ 3311.319936] blk_mq_free_request+0xc0/0x160 [ 3311.324106] __blk_mq_end_request+0x78/0x150 [ 3311.328363] scsi_end_request+0x114/0x260 [ 3311.332359] scsi_io_completion+0x64/0x174 [ 3311.336442] scsi_finish_command+0xe0/0x120 [ 3311.340613] scsi_complete+0x8c/0x1c4 [ 3311.344262] blk_complete_reqs+0x5c/0x74 [ 3311.348171] blk_done_softirq+0x38/0x4c [ 3311.351994] __do_softirq+0x168/0x418 [ 3311.355643] __irq_exit_rcu+0x120/0x170 [ 3311.359466] irq_exit_rcu+0x18/0x24 [ 3311.362941] el1_interrupt+0x38/0x70 [ 3311.366504] el1h_64_irq_handler+0x18/0x24 [ 3311.370587] el1h_64_irq+0x68/0x6c [ 3311.373975] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3311.378753] __wake_up_common_lock+0x84/0xac [ 3311.383010] __wake_up+0x20/0x30 [ 3311.386226] __sbq_wake_up+0x13c/0x1f0 [ 3311.389962] sbitmap_queue_clear+0x74/0xc0 [ 3311.394045] blk_mq_put_tag+0x44/0x70 [ 3311.397694] __blk_mq_free_request+0x88/0xf0 [ 3311.401951] blk_mq_free_request+0xc0/0x160 [ 3311.406121] bfq_insert_request.isra.0+0x204/0x270 [ 3311.410899] bfq_insert_requests+0x70/0x90 [ 3311.414983] blk_mq_sched_insert_requests+0xac/0x250 [ 3311.419934] blk_mq_flush_plug_list+0x110/0x2cc [ 3311.424452] __blk_flush_plug+0x100/0x140 [ 3311.428448] blk_finish_plug+0x40/0x25c [ 3311.432272] __iomap_dio_rw+0x1b4/0x500 [ 3311.436095] iomap_dio_rw+0x18/0x50 [ 3311.439571] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3311.444921] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3311.449664] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3311.454708] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3311.459838] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3311.464968] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3311.470354] process_one_work+0x1ec/0x470 [ 3311.474350] worker_thread+0x74/0x410 [ 3311.477998] kthread+0xec/0x100 [ 3311.481127] ret_from_fork+0x10/0x20 [ 3323.202641] watchdog: BUG: soft lockup - CPU#68 stuck for 2700s! [swapper/68:0] [ 3323.209936] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3323.209954] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3323.297742] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3323.305557] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3323.314849] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3323.321796] pc : __sbq_wake_up+0x58/0x1f0 [ 3323.325793] lr : sbitmap_queue_clear+0x74/0xc0 [ 3323.330223] sp : ffff8000088438b0 [ 3323.333526] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3323.340648] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3323.347770] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3323.354892] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3323.362014] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3323.369136] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3323.376258] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3323.383380] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3323.390501] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3323.397623] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3323.404745] Call trace: [ 3323.407178] __sbq_wake_up+0x58/0x1f0 [ 3323.410828] sbitmap_queue_clear+0x74/0xc0 [ 3323.414911] blk_mq_put_tag+0x44/0x70 [ 3323.418561] __blk_mq_free_request+0x88/0xf0 [ 3323.422817] blk_mq_free_request+0xc0/0x160 [ 3323.426988] __blk_mq_end_request+0x78/0x150 [ 3323.431245] scsi_end_request+0x114/0x260 [ 3323.435241] scsi_io_completion+0x64/0x174 [ 3323.439325] scsi_finish_command+0xe0/0x120 [ 3323.443495] scsi_complete+0x8c/0x1c4 [ 3323.447144] blk_complete_reqs+0x5c/0x74 [ 3323.451053] blk_done_softirq+0x38/0x4c [ 3323.454876] __do_softirq+0x168/0x418 [ 3323.458525] __irq_exit_rcu+0x120/0x170 [ 3323.462347] irq_exit_rcu+0x18/0x24 [ 3323.465823] el1_interrupt+0x38/0x70 [ 3323.469386] el1h_64_irq_handler+0x18/0x24 [ 3323.473469] el1h_64_irq+0x68/0x6c [ 3323.476857] cpuidle_enter_state+0xc4/0x4e0 [ 3323.481027] cpuidle_enter+0x40/0x60 [ 3323.484590] cpuidle_idle_call+0x118/0x1b0 [ 3323.488673] do_idle+0xac/0x100 [ 3323.491802] cpu_startup_entry+0x30/0x34 [ 3323.495711] secondary_start_kernel+0xd8/0xf0 [ 3323.500055] __secondary_switched+0xb0/0xb4 [ 3334.112579] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 288317 jiffies s: 2093 root: 0x11/. [ 3334.124317] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3334.133525] Task dump for CPU 8: [ 3334.136740] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3334.146641] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3334.152811] Call trace: [ 3334.155244] __switch_to+0xd4/0x100 [ 3334.158720] 0xffff6c0200000080 [ 3334.161849] Task dump for CPU 68: [ 3334.165151] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3334.175053] Call trace: [ 3334.177486] __switch_to+0xd4/0x100 [ 3334.180962] 0x0 [ 3335.092562] watchdog: BUG: soft lockup - CPU#8 stuck for 2712s! [kworker/8:12:19399] [ 3335.100292] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3335.100309] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3335.188097] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3335.196347] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3335.205637] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3335.211805] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3335.218753] pc : __sbq_wake_up+0x84/0x1f0 [ 3335.222749] lr : sbitmap_queue_clear+0x74/0xc0 [ 3335.227180] sp : ffff80000d79b280 [ 3335.230481] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3335.237603] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3335.244726] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3335.251848] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3335.258970] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3335.266092] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3335.273214] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3335.280335] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3335.287457] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3335.294580] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3335.301702] Call trace: [ 3335.304135] __sbq_wake_up+0x84/0x1f0 [ 3335.307784] sbitmap_queue_clear+0x74/0xc0 [ 3335.311868] blk_mq_put_tag+0x44/0x70 [ 3335.315517] __blk_mq_free_request+0x88/0xf0 [ 3335.319774] blk_mq_free_request+0xc0/0x160 [ 3335.323944] __blk_mq_end_request+0x78/0x150 [ 3335.328201] scsi_end_request+0x114/0x260 [ 3335.332197] scsi_io_completion+0x64/0x174 [ 3335.336280] scsi_finish_command+0xe0/0x120 [ 3335.340451] scsi_complete+0x8c/0x1c4 [ 3335.344100] blk_complete_reqs+0x5c/0x74 [ 3335.348011] blk_done_softirq+0x38/0x4c [ 3335.351834] __do_softirq+0x168/0x418 [ 3335.355483] __irq_exit_rcu+0x120/0x170 [ 3335.359305] irq_exit_rcu+0x18/0x24 [ 3335.362781] el1_interrupt+0x38/0x70 [ 3335.366343] el1h_64_irq_handler+0x18/0x24 [ 3335.370426] el1h_64_irq+0x68/0x6c [ 3335.373815] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3335.378593] __wake_up_common_lock+0x84/0xac [ 3335.382850] __wake_up+0x20/0x30 [ 3335.386065] __sbq_wake_up+0x13c/0x1f0 [ 3335.389801] sbitmap_queue_clear+0x74/0xc0 [ 3335.393885] blk_mq_put_tag+0x44/0x70 [ 3335.397534] __blk_mq_free_request+0x88/0xf0 [ 3335.401791] blk_mq_free_request+0xc0/0x160 [ 3335.405961] bfq_insert_request.isra.0+0x204/0x270 [ 3335.410739] bfq_insert_requests+0x70/0x90 [ 3335.414823] blk_mq_sched_insert_requests+0xac/0x250 [ 3335.419774] blk_mq_flush_plug_list+0x110/0x2cc [ 3335.424292] __blk_flush_plug+0x100/0x140 [ 3335.428288] blk_finish_plug+0x40/0x25c [ 3335.432111] __iomap_dio_rw+0x1b4/0x500 [ 3335.435934] iomap_dio_rw+0x18/0x50 [ 3335.439409] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3335.444760] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3335.449502] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3335.454546] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3335.459676] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3335.464806] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3335.470192] process_one_work+0x1ec/0x470 [ 3335.474188] worker_thread+0x74/0x410 [ 3335.477837] kthread+0xec/0x100 [ 3335.480966] ret_from_fork+0x10/0x20 [ 3347.202482] watchdog: BUG: soft lockup - CPU#68 stuck for 2723s! [swapper/68:0] [ 3347.209777] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3347.209794] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3347.297582] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3347.305397] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3347.314688] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3347.321636] pc : __sbq_wake_up+0x58/0x1f0 [ 3347.325632] lr : sbitmap_queue_clear+0x74/0xc0 [ 3347.330063] sp : ffff8000088438b0 [ 3347.333364] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3347.340486] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3347.347608] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3347.354730] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3347.361852] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3347.368973] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3347.376095] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3347.383218] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3347.390340] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3347.397461] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3347.404583] Call trace: [ 3347.407016] __sbq_wake_up+0x58/0x1f0 [ 3347.410665] sbitmap_queue_clear+0x74/0xc0 [ 3347.414749] blk_mq_put_tag+0x44/0x70 [ 3347.418398] __blk_mq_free_request+0x88/0xf0 [ 3347.422655] blk_mq_free_request+0xc0/0x160 [ 3347.426825] __blk_mq_end_request+0x78/0x150 [ 3347.431082] scsi_end_request+0x114/0x260 [ 3347.435078] scsi_io_completion+0x64/0x174 [ 3347.439161] scsi_finish_command+0xe0/0x120 [ 3347.443331] scsi_complete+0x8c/0x1c4 [ 3347.446980] blk_complete_reqs+0x5c/0x74 [ 3347.450890] blk_done_softirq+0x38/0x4c [ 3347.454713] __do_softirq+0x168/0x418 [ 3347.458361] __irq_exit_rcu+0x120/0x170 [ 3347.462184] irq_exit_rcu+0x18/0x24 [ 3347.465659] el1_interrupt+0x38/0x70 [ 3347.469222] el1h_64_irq_handler+0x18/0x24 [ 3347.473306] el1h_64_irq+0x68/0x6c [ 3347.476694] cpuidle_enter_state+0xc4/0x4e0 [ 3347.480865] cpuidle_enter+0x40/0x60 [ 3347.484427] cpuidle_idle_call+0x118/0x1b0 [ 3347.488511] do_idle+0xac/0x100 [ 3347.491640] cpu_startup_entry+0x30/0x34 [ 3347.495549] secondary_start_kernel+0xd8/0xf0 [ 3347.499893] __secondary_switched+0xb0/0xb4 [ 3359.092403] watchdog: BUG: soft lockup - CPU#8 stuck for 2734s! [kworker/8:12:19399] [ 3359.100133] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3359.100150] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3359.187938] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3359.196187] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3359.205478] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3359.211646] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3359.218593] pc : __sbq_wake_up+0x20/0x1f0 [ 3359.222590] lr : sbitmap_queue_clear+0x74/0xc0 [ 3359.227021] sp : ffff80000d79b280 [ 3359.230322] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3359.237445] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3359.244567] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3359.251689] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3359.258811] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3359.265933] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3359.273054] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3359.280176] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3359.287298] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3359.294420] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 3359.301542] Call trace: [ 3359.303975] __sbq_wake_up+0x20/0x1f0 [ 3359.307625] sbitmap_queue_clear+0x74/0xc0 [ 3359.311708] blk_mq_put_tag+0x44/0x70 [ 3359.315358] __blk_mq_free_request+0x88/0xf0 [ 3359.319615] blk_mq_free_request+0xc0/0x160 [ 3359.323785] __blk_mq_end_request+0x78/0x150 [ 3359.328041] scsi_end_request+0x114/0x260 [ 3359.332038] scsi_io_completion+0x64/0x174 [ 3359.336121] scsi_finish_command+0xe0/0x120 [ 3359.340291] scsi_complete+0x8c/0x1c4 [ 3359.343940] blk_complete_reqs+0x5c/0x74 [ 3359.347850] blk_done_softirq+0x38/0x4c [ 3359.351672] __do_softirq+0x168/0x418 [ 3359.355321] __irq_exit_rcu+0x120/0x170 [ 3359.359144] irq_exit_rcu+0x18/0x24 [ 3359.362620] el1_interrupt+0x38/0x70 [ 3359.366182] el1h_64_irq_handler+0x18/0x24 [ 3359.370265] el1h_64_irq+0x68/0x6c [ 3359.373654] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3359.378431] __wake_up_common_lock+0x84/0xac [ 3359.382688] __wake_up+0x20/0x30 [ 3359.385905] __sbq_wake_up+0x13c/0x1f0 [ 3359.389641] sbitmap_queue_clear+0x74/0xc0 [ 3359.393724] blk_mq_put_tag+0x44/0x70 [ 3359.397374] __blk_mq_free_request+0x88/0xf0 [ 3359.401630] blk_mq_free_request+0xc0/0x160 [ 3359.405800] bfq_insert_request.isra.0+0x204/0x270 [ 3359.410578] bfq_insert_requests+0x70/0x90 [ 3359.414662] blk_mq_sched_insert_requests+0xac/0x250 [ 3359.419614] blk_mq_flush_plug_list+0x110/0x2cc [ 3359.424132] __blk_flush_plug+0x100/0x140 [ 3359.428128] blk_finish_plug+0x40/0x25c [ 3359.431951] __iomap_dio_rw+0x1b4/0x500 [ 3359.435774] iomap_dio_rw+0x18/0x50 [ 3359.439251] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3359.444602] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3359.449344] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3359.454388] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3359.459518] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3359.464648] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3359.470033] process_one_work+0x1ec/0x470 [ 3359.474030] worker_thread+0x74/0x410 [ 3359.477679] kthread+0xec/0x100 [ 3359.480808] ret_from_fork+0x10/0x20 [ 3369.922331] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3369.928064] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=146092 [ 3369.937186] (t=294505 jiffies g=36445 q=86302 ncpus=128) [ 3369.942571] Task dump for CPU 8: [ 3369.945785] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3369.955685] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3369.961852] Call trace: [ 3369.964285] dump_backtrace+0xe8/0x130 [ 3369.968021] show_stack+0x20/0x60 [ 3369.971324] sched_show_task+0x160/0x190 [ 3369.975233] dump_cpu_task+0x4c/0x78 [ 3369.978796] rcu_dump_cpu_stacks+0xc4/0x150 [ 3369.982967] print_cpu_stall+0x180/0x210 [ 3369.986876] check_cpu_stall+0x20c/0x2b4 [ 3369.990786] rcu_pending+0x4c/0x190 [ 3369.994261] rcu_sched_clock_irq+0x80/0x204 [ 3369.998431] update_process_times+0x7c/0xc0 [ 3370.002601] tick_sched_handle+0x38/0x74 [ 3370.006511] tick_sched_timer+0x60/0xc0 [ 3370.010334] __hrtimer_run_queues+0x170/0x300 [ 3370.014678] hrtimer_interrupt+0xfc/0x260 [ 3370.018674] arch_timer_handler_phys+0x34/0x50 [ 3370.023105] handle_percpu_devid_irq+0xac/0x240 [ 3370.027623] generic_handle_domain_irq+0x34/0x50 [ 3370.032228] gic_handle_irq+0x5c/0xe0 [ 3370.035879] call_on_irq_stack+0x2c/0x38 [ 3370.039789] do_interrupt_handler+0xdc/0xe0 [ 3370.043959] el1_interrupt+0x34/0x70 [ 3370.047521] el1h_64_irq_handler+0x18/0x24 [ 3370.051604] el1h_64_irq+0x68/0x6c [ 3370.054993] __sbq_wake_up+0x58/0x1f0 [ 3370.058644] sbitmap_queue_clear+0x74/0xc0 [ 3370.062727] blk_mq_put_tag+0x44/0x70 [ 3370.066376] __blk_mq_free_request+0x88/0xf0 [ 3370.070633] blk_mq_free_request+0xc0/0x160 [ 3370.074803] __blk_mq_end_request+0x78/0x150 [ 3370.079060] scsi_end_request+0x114/0x260 [ 3370.083057] scsi_io_completion+0x64/0x174 [ 3370.087140] scsi_finish_command+0xe0/0x120 [ 3370.091310] scsi_complete+0x8c/0x1c4 [ 3370.094959] blk_complete_reqs+0x5c/0x74 [ 3370.098869] blk_done_softirq+0x38/0x4c [ 3370.102691] __do_softirq+0x168/0x418 [ 3370.106340] __irq_exit_rcu+0x120/0x170 [ 3370.110163] irq_exit_rcu+0x18/0x24 [ 3370.113639] el1_interrupt+0x38/0x70 [ 3370.117201] el1h_64_irq_handler+0x18/0x24 [ 3370.121284] el1h_64_irq+0x68/0x6c [ 3370.124673] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3370.129451] __wake_up_common_lock+0x84/0xac [ 3370.133708] __wake_up+0x20/0x30 [ 3370.136924] __sbq_wake_up+0x13c/0x1f0 [ 3370.140661] sbitmap_queue_clear+0x74/0xc0 [ 3370.144744] blk_mq_put_tag+0x44/0x70 [ 3370.148393] __blk_mq_free_request+0x88/0xf0 [ 3370.152650] blk_mq_free_request+0xc0/0x160 [ 3370.156820] bfq_insert_request.isra.0+0x204/0x270 [ 3370.161598] bfq_insert_requests+0x70/0x90 [ 3370.165682] blk_mq_sched_insert_requests+0xac/0x250 [ 3370.170634] blk_mq_flush_plug_list+0x110/0x2cc [ 3370.175151] __blk_flush_plug+0x100/0x140 [ 3370.179148] blk_finish_plug+0x40/0x25c [ 3370.182971] __iomap_dio_rw+0x1b4/0x500 [ 3370.186794] iomap_dio_rw+0x18/0x50 [ 3370.190269] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3370.195620] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3370.200363] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3370.205406] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3370.210536] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3370.215666] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3370.221052] process_one_work+0x1ec/0x470 [ 3370.225048] worker_thread+0x74/0x410 [ 3370.228697] kthread+0xec/0x100 [ 3370.231826] ret_from_fork+0x10/0x20 [ 3370.235389] Task dump for CPU 68: [ 3370.238690] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3370.248590] Call trace: [ 3370.251023] __switch_to+0xd4/0x100 [ 3370.254499] 0x0 [ 3371.202323] watchdog: BUG: soft lockup - CPU#68 stuck for 2745s! [swapper/68:0] [ 3371.209619] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3371.209636] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3371.297423] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3371.305240] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3371.314531] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3371.321479] pc : __sbq_wake_up+0x20/0x1f0 [ 3371.325476] lr : sbitmap_queue_clear+0x74/0xc0 [ 3371.329906] sp : ffff8000088438b0 [ 3371.333207] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3371.340329] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3371.347450] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3371.354572] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3371.361695] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3371.368816] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3371.375938] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3371.383060] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3371.390182] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3371.397304] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 3371.404426] Call trace: [ 3371.406859] __sbq_wake_up+0x20/0x1f0 [ 3371.410509] sbitmap_queue_clear+0x74/0xc0 [ 3371.414592] blk_mq_put_tag+0x44/0x70 [ 3371.418242] __blk_mq_free_request+0x88/0xf0 [ 3371.422498] blk_mq_free_request+0xc0/0x160 [ 3371.426668] __blk_mq_end_request+0x78/0x150 [ 3371.430925] scsi_end_request+0x114/0x260 [ 3371.434922] scsi_io_completion+0x64/0x174 [ 3371.439006] scsi_finish_command+0xe0/0x120 [ 3371.443176] scsi_complete+0x8c/0x1c4 [ 3371.446825] blk_complete_reqs+0x5c/0x74 [ 3371.450736] blk_done_softirq+0x38/0x4c [ 3371.454558] __do_softirq+0x168/0x418 [ 3371.458207] __irq_exit_rcu+0x120/0x170 [ 3371.462030] irq_exit_rcu+0x18/0x24 [ 3371.465505] el1_interrupt+0x38/0x70 [ 3371.469068] el1h_64_irq_handler+0x18/0x24 [ 3371.473151] el1h_64_irq+0x68/0x6c [ 3371.476539] cpuidle_enter_state+0xc4/0x4e0 [ 3371.480709] cpuidle_enter+0x40/0x60 [ 3371.484272] cpuidle_idle_call+0x118/0x1b0 [ 3371.488356] do_idle+0xac/0x100 [ 3371.491485] cpu_startup_entry+0x30/0x34 [ 3371.495394] secondary_start_kernel+0xd8/0xf0 [ 3371.499738] __secondary_switched+0xb0/0xb4 [ 3395.092164] watchdog: BUG: soft lockup - CPU#8 stuck for 2767s! [kworker/8:12:19399] [ 3395.099893] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3395.099911] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3395.187699] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3395.195949] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3395.202163] watchdog: BUG: soft lockup - CPU#68 stuck for 2768s! [swapper/68:0] [ 3395.205241] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3395.212534] Modules linked in: nvme_loop [ 3395.218700] [ 3395.218701] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3395.222609] nvme_fabrics [ 3395.224087] pc : __sbq_wake_up+0x8/0x1f0 [ 3395.231034] nvmet [ 3395.233640] lr : sbitmap_queue_clear+0x74/0xc0 [ 3395.237548] nls_utf8 [ 3395.239547] sp : ffff80000d79b280 [ 3395.243977] hfsplus [ 3395.246236] x29: ffff80000d79b2d0 [ 3395.249537] hfs [ 3395.251709] x28: ffffd5becb6aec50 [ 3395.255010] isofs [ 3395.256835] x27: ffffd5beca4f7000 [ 3395.260223] nvme_core [ 3395.262222] [ 3395.265609] nvme_common [ 3395.267955] x26: ffffd5becaf68a98 [ 3395.269433] loop [ 3395.271952] x25: ffffd5becb6a60c0 [ 3395.275253] tls [ 3395.277165] x24: fffffbffef180dc0 [ 3395.280553] rfkill [ 3395.282378] [ 3395.285765] sunrpc [ 3395.287851] x23: ffff07ff81d381b0 [ 3395.289329] vfat [ 3395.291414] x22: 0000000000000024 [ 3395.294715] fat [ 3395.296627] x21: 0000000000000008 [ 3395.300014] acpi_ipmi [ 3395.301839] [ 3395.305227] ipmi_ssif [ 3395.307573] x20: 0000000000000024 [ 3395.309051] cppc_cpufreq [ 3395.311397] x19: ffff07ff91ccdb10 [ 3395.314697] ipmi_devintf [ 3395.317304] x18: ffffffffffffffff [ 3395.320691] igb [ 3395.323298] [ 3395.326685] ipmi_msghandler [ 3395.328510] x17: ffff32506c412000 [ 3395.329988] arm_cmn [ 3395.332855] x16: ffff800008044000 [ 3395.336156] arm_dmc620_pmu [ 3395.338328] x15: 0000000000004000 [ 3395.341717] arm_spe_pmu [ 3395.344497] [ 3395.347885] arm_dsu_pmu [ 3395.350404] x14: 010320bb28ec0000 [ 3395.351882] acpi_tad [ 3395.354402] x13: 0000000000000087 [ 3395.357702] fuse [ 3395.359962] x12: 071c71c71c71c71c [ 3395.363349] zram [ 3395.365261] [ 3395.368649] xfs [ 3395.370560] x11: 0000000000000040 [ 3395.372038] crct10dif_ce [ 3395.373863] x10: ffff07ff801a9c70 [ 3395.377164] polyval_ce [ 3395.379770] x9 : ffffd5bec9545fd8 [ 3395.383158] ast [ 3395.385591] [ 3395.388979] polyval_generic [ 3395.390804] x8 : ffff07ff80404b70 [ 3395.392282] drm_vram_helper [ 3395.395149] x7 : ffff07ff84e93dc8 [ 3395.398449] ghash_ce [ 3395.401316] x6 : 0000000000000008 [ 3395.404704] drm_ttm_helper [ 3395.406963] [ 3395.410350] sbsa_gwdt [ 3395.413130] x5 : ffff80000d043790 [ 3395.414608] ttm [ 3395.416956] x4 : ffff07ff84e93dc0 [ 3395.420256] xgene_hwmon [ 3395.422081] x3 : 0000000000000008 [ 3395.425469] Unloaded tainted modules: [ 3395.427989] [ 3395.431376] nvmet():1 [ 3395.435024] x2 : 0000000000000007 [ 3395.436502] nvme_fabrics():1 [ 3395.438848] x1 : ffff80000d79b30c [ 3395.442149] nvme_loop():1 [ 3395.445102] x0 : ffff07ff91ccdb10 [ 3395.448490] nvmet():1 [ 3395.451183] [ 3395.454571] nvme_fabrics():1 [ 3395.456916] Call trace: [ 3395.458394] nvme_loop():1 [ 3395.461348] __sbq_wake_up+0x8/0x1f0 [ 3395.463781] nvmet():1 [ 3395.466474] blk_mq_put_tag+0x44/0x70 [ 3395.470035] nvme_fabrics():1 [ 3395.472381] __blk_mq_free_request+0x88/0xf0 [ 3395.476029] nvme_loop():1 [ 3395.478982] blk_mq_free_request+0xc0/0x160 [ 3395.483238] nvmet():1 [ 3395.485931] __blk_mq_end_request+0x78/0x150 [ 3395.490100] nvme_fabrics():1 [ 3395.492446] scsi_end_request+0x114/0x260 [ 3395.496701] nvme_loop():1 [ 3395.499655] scsi_io_completion+0x64/0x174 [ 3395.503651] nvmet():1 [ 3395.506345] scsi_finish_command+0xe0/0x120 [ 3395.510428] nvme_fabrics():1 [ 3395.512773] scsi_complete+0x8c/0x1c4 [ 3395.516942] nvme_loop():1 [ 3395.519896] blk_complete_reqs+0x5c/0x74 [ 3395.523544] nvmet():1 [ 3395.526237] blk_done_softirq+0x38/0x4c [ 3395.530145] nvme_fabrics():1 [ 3395.532491] __do_softirq+0x168/0x418 [ 3395.536313] nvme_loop():1 [ 3395.539266] __irq_exit_rcu+0x120/0x170 [ 3395.542914] nvmet():1 [ 3395.545607] irq_exit_rcu+0x18/0x24 [ 3395.549429] nvme_fabrics():1 [ 3395.551775] el1_interrupt+0x38/0x70 [ 3395.555249] nvme_loop():1 [ 3395.558203] el1h_64_irq_handler+0x18/0x24 [ 3395.561764] nvmet():1 [ 3395.564457] el1h_64_irq+0x68/0x6c [ 3395.568539] nvme_fabrics():1 [ 3395.570885] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3395.574273] nvme_loop():1 [ 3395.577226] __wake_up_common_lock+0x84/0xac [ 3395.582003] nvmet():1 [ 3395.584696] __wake_up+0x20/0x30 [ 3395.588951] nvme_fabrics():1 [ 3395.591297] __sbq_wake_up+0x13c/0x1f0 [ 3395.594511] nvme_loop():1 [ 3395.597465] sbitmap_queue_clear+0x74/0xc0 [ 3395.601200] nvmet():1 [ 3395.603893] blk_mq_put_tag+0x44/0x70 [ 3395.607975] nvme_fabrics():1 [ 3395.610321] __blk_mq_free_request+0x88/0xf0 [ 3395.613969] nvme_loop():1 [ 3395.616922] blk_mq_free_request+0xc0/0x160 [ 3395.621178] null_blk():1 [ 3395.623871] bfq_insert_request.isra.0+0x204/0x270 [ 3395.628040] null_blk():1 [ 3395.630646] bfq_insert_requests+0x70/0x90 [ 3395.635423] null_blk():1 [ 3395.638031] blk_mq_sched_insert_requests+0xac/0x250 [ 3395.642113] null_blk():1 [ 3395.644720] blk_mq_flush_plug_list+0x110/0x2cc [ 3395.649670] null_blk():1 [ 3395.652276] __blk_flush_plug+0x100/0x140 [ 3395.656792] null_blk():1 [ 3395.659398] blk_finish_plug+0x40/0x25c [ 3395.663393] null_blk():1 [ 3395.666000] __iomap_dio_rw+0x1b4/0x500 [ 3395.669821] scsi_debug():1 [ 3395.672429] iomap_dio_rw+0x18/0x50 [ 3395.676251] scsi_debug():1 [ 3395.679031] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3395.682505] ipmi_si():1 [ 3395.685285] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3395.690582] ipmi_si():1 [ 3395.693102] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3395.697792] [last unloaded: nvmet] [ 3395.700311] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3395.705348] [ 3395.708823] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3395.713947] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3395.715425] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3395.720548] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3395.728363] process_one_work+0x1ec/0x470 [ 3395.733747] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3395.743038] worker_thread+0x74/0x410 [ 3395.747033] pc : __sbq_wake_up+0x84/0x1f0 [ 3395.753980] kthread+0xec/0x100 [ 3395.757628] lr : sbitmap_queue_clear+0x74/0xc0 [ 3395.761623] ret_from_fork+0x10/0x20 [ 3395.764750] sp : ffff8000088438b0 [ 3395.776041] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3395.783163] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3395.790285] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3395.797407] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3395.804528] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3395.811651] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3395.818773] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3395.825894] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3395.833016] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3395.840138] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3395.847260] Call trace: [ 3395.849693] __sbq_wake_up+0x84/0x1f0 [ 3395.853342] sbitmap_queue_clear+0x74/0xc0 [ 3395.857425] blk_mq_put_tag+0x44/0x70 [ 3395.861075] __blk_mq_free_request+0x88/0xf0 [ 3395.865331] blk_mq_free_request+0xc0/0x160 [ 3395.869501] __blk_mq_end_request+0x78/0x150 [ 3395.873758] scsi_end_request+0x114/0x260 [ 3395.877754] scsi_io_completion+0x64/0x174 [ 3395.881837] scsi_finish_command+0xe0/0x120 [ 3395.886008] scsi_complete+0x8c/0x1c4 [ 3395.889657] blk_complete_reqs+0x5c/0x74 [ 3395.893566] blk_done_softirq+0x38/0x4c [ 3395.897389] __do_softirq+0x168/0x418 [ 3395.901037] __irq_exit_rcu+0x120/0x170 [ 3395.904860] irq_exit_rcu+0x18/0x24 [ 3395.908335] el1_interrupt+0x38/0x70 [ 3395.911898] el1h_64_irq_handler+0x18/0x24 [ 3395.915981] el1h_64_irq+0x68/0x6c [ 3395.919370] cpuidle_enter_state+0xc4/0x4e0 [ 3395.923539] cpuidle_enter+0x40/0x60 [ 3395.927102] cpuidle_idle_call+0x118/0x1b0 [ 3395.931186] do_idle+0xac/0x100 [ 3395.934315] cpu_startup_entry+0x30/0x34 [ 3395.938224] secondary_start_kernel+0xd8/0xf0 [ 3395.942568] __secondary_switched+0xb0/0xb4 [ 3400.672137] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 294973 jiffies s: 2093 root: 0x11/. [ 3400.683875] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3400.693084] Task dump for CPU 8: [ 3400.696298] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3400.706201] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3400.712371] Call trace: [ 3400.714804] __switch_to+0xd4/0x100 [ 3400.718280] 0xffff6c0200000080 [ 3400.721409] Task dump for CPU 68: [ 3400.724712] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3400.734613] Call trace: [ 3400.737046] __switch_to+0xd4/0x100 [ 3400.740522] 0x0 [ 3419.092005] watchdog: BUG: soft lockup - CPU#8 stuck for 2790s! [kworker/8:12:19399] [ 3419.099734] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3419.099751] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3419.187539] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3419.195789] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3419.202004] watchdog: BUG: soft lockup - CPU#68 stuck for 2790s! [swapper/68:0] [ 3419.205080] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3419.212373] Modules linked in: nvme_loop [ 3419.218539] [ 3419.218540] nvme_fabrics nvmet [ 3419.222449] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3419.223927] nls_utf8 [ 3419.227055] pc : __sbq_wake_up+0x84/0x1f0 [ 3419.234001] hfsplus [ 3419.236261] lr : sbitmap_queue_clear+0x74/0xc0 [ 3419.240256] hfs [ 3419.242428] sp : ffff80000d79b280 [ 3419.246857] isofs [ 3419.248682] x29: ffff80000d79b280 [ 3419.251983] nvme_core [ 3419.253982] x28: ffffd5becb6aec50 [ 3419.257283] nvme_common [ 3419.259629] x27: ffffd5beca4f7000 [ 3419.263016] loop [ 3419.265536] [ 3419.268923] tls [ 3419.270835] x26: ffffd5becaf68a98 [ 3419.272313] rfkill [ 3419.274138] x25: ffffd5becb6a60c0 [ 3419.277439] sunrpc [ 3419.279524] x24: fffffbffef180dc0 [ 3419.282912] vfat [ 3419.284997] [ 3419.288385] fat [ 3419.290297] x23: ffff07ff81d381b0 [ 3419.291775] acpi_ipmi [ 3419.293600] x22: ffff80000d79b30c [ 3419.296901] ipmi_ssif [ 3419.299246] x21: ffff07ff84e93c00 [ 3419.302634] cppc_cpufreq [ 3419.304980] [ 3419.308367] ipmi_devintf [ 3419.310974] x20: 00000000000001c0 [ 3419.312452] igb [ 3419.315058] x19: ffff07ff91ccdb10 [ 3419.318359] ipmi_msghandler [ 3419.320184] x18: ffffffffffffffff [ 3419.323571] arm_cmn [ 3419.326438] [ 3419.329826] arm_dmc620_pmu [ 3419.331998] x17: ffff32506c412000 [ 3419.333476] arm_spe_pmu [ 3419.336256] x16: ffff800008044000 [ 3419.339557] arm_dsu_pmu [ 3419.342076] x15: 0000000000004000 [ 3419.345464] acpi_tad [ 3419.347983] [ 3419.351371] fuse [ 3419.353630] x14: 010320bb28ec0000 [ 3419.355108] zram [ 3419.357020] x13: 0000000000000087 [ 3419.360321] xfs [ 3419.362233] x12: 071c71c71c71c71c [ 3419.365620] crct10dif_ce [ 3419.367445] [ 3419.370833] polyval_ce [ 3419.373441] x11: 0000000000000040 [ 3419.374919] ast [ 3419.377352] x10: ffff07ff801a9c70 [ 3419.380653] polyval_generic [ 3419.382478] x9 : ffffd5bec9545fd8 [ 3419.385866] drm_vram_helper [ 3419.388732] [ 3419.392120] ghash_ce [ 3419.394987] x8 : ffff07ff80404b70 [ 3419.396464] drm_ttm_helper [ 3419.398724] x7 : ffff07ff84e93dc8 [ 3419.402024] sbsa_gwdt [ 3419.404806] x6 : 0000000000000008 [ 3419.408194] ttm [ 3419.410539] [ 3419.413927] xgene_hwmon [ 3419.415752] x5 : ffff80000d043790 [ 3419.417230] Unloaded tainted modules: [ 3419.419751] x4 : ffff07ff84e93dc0 [ 3419.423051] nvmet():1 [ 3419.426699] x3 : 0000000000000008 [ 3419.430087] nvme_fabrics():1 [ 3419.432433] [ 3419.435820] nvme_loop():1 [ 3419.438774] x2 : 0000000000000007 [ 3419.440252] nvmet():1 [ 3419.442945] x1 : ffff80000d79b30c [ 3419.446246] nvme_fabrics():1 [ 3419.448592] x0 : ffff07ff84e93dd0 [ 3419.451979] nvme_loop():1 [ 3419.454933] [ 3419.458320] nvmet():1 [ 3419.461013] Call trace: [ 3419.462491] nvme_fabrics():1 [ 3419.464837] __sbq_wake_up+0x84/0x1f0 [ 3419.467270] nvme_loop():1 [ 3419.470223] sbitmap_queue_clear+0x74/0xc0 [ 3419.473871] nvmet():1 [ 3419.476564] blk_mq_put_tag+0x44/0x70 [ 3419.480647] nvme_fabrics():1 [ 3419.482992] __blk_mq_free_request+0x88/0xf0 [ 3419.486640] nvme_loop():1 [ 3419.489594] blk_mq_free_request+0xc0/0x160 [ 3419.493850] nvmet():1 [ 3419.496543] __blk_mq_end_request+0x78/0x150 [ 3419.500712] nvme_fabrics():1 [ 3419.503057] scsi_end_request+0x114/0x260 [ 3419.507313] nvme_loop():1 [ 3419.510267] scsi_io_completion+0x64/0x174 [ 3419.514262] nvmet():1 [ 3419.516956] scsi_finish_command+0xe0/0x120 [ 3419.521038] nvme_fabrics():1 [ 3419.523383] scsi_complete+0x8c/0x1c4 [ 3419.527552] nvme_loop():1 [ 3419.530506] blk_complete_reqs+0x5c/0x74 [ 3419.534154] nvmet():1 [ 3419.536847] blk_done_softirq+0x38/0x4c [ 3419.540755] nvme_fabrics():1 [ 3419.543101] __do_softirq+0x168/0x418 [ 3419.546923] nvme_loop():1 [ 3419.549876] __irq_exit_rcu+0x120/0x170 [ 3419.553524] nvmet():1 [ 3419.556218] irq_exit_rcu+0x18/0x24 [ 3419.560039] nvme_fabrics():1 [ 3419.562386] el1_interrupt+0x38/0x70 [ 3419.565860] nvme_loop():1 [ 3419.568813] el1h_64_irq_handler+0x18/0x24 [ 3419.572375] nvmet():1 [ 3419.575068] el1h_64_irq+0x68/0x6c [ 3419.579150] nvme_fabrics():1 [ 3419.581496] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3419.584883] nvme_loop():1 [ 3419.587837] __wake_up_common_lock+0x84/0xac [ 3419.592613] nvmet():1 [ 3419.595306] __wake_up+0x20/0x30 [ 3419.599562] nvme_fabrics():1 [ 3419.601908] __sbq_wake_up+0x13c/0x1f0 [ 3419.605122] nvme_loop():1 [ 3419.608075] sbitmap_queue_clear+0x74/0xc0 [ 3419.611810] null_blk():1 [ 3419.614503] blk_mq_put_tag+0x44/0x70 [ 3419.618585] null_blk():1 [ 3419.621192] __blk_mq_free_request+0x88/0xf0 [ 3419.624840] null_blk():1 [ 3419.627446] blk_mq_free_request+0xc0/0x160 [ 3419.631702] null_blk():1 [ 3419.634308] bfq_insert_request.isra.0+0x204/0x270 [ 3419.638477] null_blk():1 [ 3419.641083] bfq_insert_requests+0x70/0x90 [ 3419.645860] null_blk():1 [ 3419.648466] blk_mq_sched_insert_requests+0xac/0x250 [ 3419.652548] null_blk():1 [ 3419.655154] blk_mq_flush_plug_list+0x110/0x2cc [ 3419.660104] scsi_debug():1 [ 3419.662711] __blk_flush_plug+0x100/0x140 [ 3419.667227] scsi_debug():1 [ 3419.670007] blk_finish_plug+0x40/0x25c [ 3419.674002] ipmi_si():1 [ 3419.676782] __iomap_dio_rw+0x1b4/0x500 [ 3419.680603] ipmi_si():1 [ 3419.683123] iomap_dio_rw+0x18/0x50 [ 3419.686945] [last unloaded: nvmet] [ 3419.689464] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3419.692938] [ 3419.696413] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3419.701710] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3419.703188] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3419.707878] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3419.715692] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3419.720729] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3419.730019] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3419.735143] pc : __sbq_wake_up+0x84/0x1f0 [ 3419.742090] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3419.747213] lr : sbitmap_queue_clear+0x74/0xc0 [ 3419.751209] process_one_work+0x1ec/0x470 [ 3419.756593] sp : ffff8000088438b0 [ 3419.761022] worker_thread+0x74/0x410 [ 3419.765018] x29: ffff8000088438b0 [ 3419.768318] kthread+0xec/0x100 [ 3419.771966] x28: ffffd5becb6aec50 [ 3419.775267] ret_from_fork+0x10/0x20 [ 3419.778394] x27: ffffd5beca4f7000 [ 3419.788731] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3419.795853] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3419.802974] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3419.810096] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3419.817218] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3419.824340] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3419.831462] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3419.838583] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3419.845705] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3419.852827] Call trace: [ 3419.855260] __sbq_wake_up+0x84/0x1f0 [ 3419.858910] sbitmap_queue_clear+0x74/0xc0 [ 3419.862993] blk_mq_put_tag+0x44/0x70 [ 3419.866643] __blk_mq_free_request+0x88/0xf0 [ 3419.870899] blk_mq_free_request+0xc0/0x160 [ 3419.875069] __blk_mq_end_request+0x78/0x150 [ 3419.879327] scsi_end_request+0x114/0x260 [ 3419.883324] scsi_io_completion+0x64/0x174 [ 3419.887407] scsi_finish_command+0xe0/0x120 [ 3419.891577] scsi_complete+0x8c/0x1c4 [ 3419.895227] blk_complete_reqs+0x5c/0x74 [ 3419.899136] blk_done_softirq+0x38/0x4c [ 3419.902959] __do_softirq+0x168/0x418 [ 3419.906607] __irq_exit_rcu+0x120/0x170 [ 3419.910430] irq_exit_rcu+0x18/0x24 [ 3419.913906] el1_interrupt+0x38/0x70 [ 3419.917468] el1h_64_irq_handler+0x18/0x24 [ 3419.921551] el1h_64_irq+0x68/0x6c [ 3419.924939] cpuidle_enter_state+0xc4/0x4e0 [ 3419.929110] cpuidle_enter+0x40/0x60 [ 3419.932672] cpuidle_idle_call+0x118/0x1b0 [ 3419.936756] do_idle+0xac/0x100 [ 3419.939885] cpu_startup_entry+0x30/0x34 [ 3419.943794] secondary_start_kernel+0xd8/0xf0 [ 3419.948138] __secondary_switched+0xb0/0xb4 [ 3443.091845] watchdog: BUG: soft lockup - CPU#8 stuck for 2812s! [kworker/8:12:19399] [ 3443.099574] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3443.099591] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3443.187379] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3443.195629] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3443.201844] watchdog: BUG: soft lockup - CPU#68 stuck for 2812s! [swapper/68:0] [ 3443.204921] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3443.212215] Modules linked in: nvme_loop [ 3443.218382] [ 3443.218382] nvme_fabrics [ 3443.222291] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3443.223769] nvmet [ 3443.226375] pc : __sbq_wake_up+0x58/0x1f0 [ 3443.233322] nls_utf8 [ 3443.235321] lr : sbitmap_queue_clear+0x74/0xc0 [ 3443.239316] hfsplus [ 3443.241575] sp : ffff80000d79b280 [ 3443.246004] hfs [ 3443.248176] x29: ffff80000d79b280 [ 3443.251477] isofs [ 3443.253303] x28: ffffd5becb6aec50 [ 3443.256604] nvme_core [ 3443.258602] x27: ffffd5beca4f7000 [ 3443.261990] nvme_common [ 3443.264336] [ 3443.267723] loop [ 3443.270243] x26: ffffd5becaf68a98 [ 3443.271721] tls [ 3443.273633] x25: ffffd5becb6a60c0 [ 3443.276933] rfkill [ 3443.278758] x24: fffffbffef180dc0 [ 3443.282146] sunrpc [ 3443.284232] [ 3443.287620] vfat [ 3443.289705] x23: ffff07ff81d381b0 [ 3443.291183] fat [ 3443.293095] x22: ffff80000d79b30c [ 3443.296396] acpi_ipmi [ 3443.298221] x21: ffff07ff84e93c00 [ 3443.301608] ipmi_ssif [ 3443.303954] [ 3443.307342] cppc_cpufreq [ 3443.309688] x20: 00000000000001c0 [ 3443.311166] ipmi_devintf [ 3443.313772] x19: ffff07ff91ccdb10 [ 3443.317073] igb [ 3443.319679] x18: ffffffffffffffff [ 3443.323067] ipmi_msghandler [ 3443.324892] [ 3443.328279] arm_cmn [ 3443.331146] x17: ffff32506c412000 [ 3443.332624] arm_dmc620_pmu [ 3443.334796] x16: ffff800008044000 [ 3443.338097] arm_spe_pmu [ 3443.340877] x15: 0000000000004000 [ 3443.344264] arm_dsu_pmu [ 3443.346784] [ 3443.350172] acpi_tad [ 3443.352691] x14: 010320bb28ec0000 [ 3443.354169] fuse [ 3443.356429] x13: 0000000000000087 [ 3443.359731] zram [ 3443.361644] x12: 071c71c71c71c71c [ 3443.365031] xfs [ 3443.366943] [ 3443.370331] crct10dif_ce [ 3443.372158] x11: 0000000000000040 [ 3443.373635] polyval_ce [ 3443.376242] x10: ffff07ff801a9c70 [ 3443.379543] ast [ 3443.381976] x9 : ffffd5bec9545fd8 [ 3443.385363] polyval_generic [ 3443.387189] [ 3443.390576] drm_vram_helper [ 3443.393443] x8 : ffff07ff80404b70 [ 3443.394921] ghash_ce [ 3443.397787] x7 : ffff07ff84e93dc8 [ 3443.401088] drm_ttm_helper [ 3443.403347] x6 : 0000000000000008 [ 3443.406735] sbsa_gwdt [ 3443.409515] [ 3443.412903] ttm [ 3443.415249] x5 : ffff80000d043790 [ 3443.416727] xgene_hwmon [ 3443.418552] x4 : ffff07ff84e93dc0 [ 3443.421853] Unloaded tainted modules: [ 3443.424372] x3 : 0000000000000008 [ 3443.427760] nvmet():1 [ 3443.431408] [ 3443.434795] nvme_fabrics():1 [ 3443.437141] x2 : 0000000000000007 [ 3443.438619] nvme_loop():1 [ 3443.441572] x1 : ffff80000d79b30c [ 3443.444873] nvmet():1 [ 3443.447566] x0 : ffff07ff84e93dd0 [ 3443.450954] nvme_fabrics():1 [ 3443.453300] [ 3443.456687] nvme_loop():1 [ 3443.459641] Call trace: [ 3443.461119] nvmet():1 [ 3443.463812] __sbq_wake_up+0x58/0x1f0 [ 3443.466245] nvme_fabrics():1 [ 3443.468591] sbitmap_queue_clear+0x74/0xc0 [ 3443.472239] nvme_loop():1 [ 3443.475192] blk_mq_put_tag+0x44/0x70 [ 3443.479274] nvmet():1 [ 3443.481967] __blk_mq_free_request+0x88/0xf0 [ 3443.485615] nvme_fabrics():1 [ 3443.487961] blk_mq_free_request+0xc0/0x160 [ 3443.492217] nvme_loop():1 [ 3443.495170] __blk_mq_end_request+0x78/0x150 [ 3443.499339] nvmet():1 [ 3443.502033] scsi_end_request+0x114/0x260 [ 3443.506288] nvme_fabrics():1 [ 3443.508634] scsi_io_completion+0x64/0x174 [ 3443.512630] nvme_loop():1 [ 3443.515583] scsi_finish_command+0xe0/0x120 [ 3443.519665] nvmet():1 [ 3443.522359] scsi_complete+0x8c/0x1c4 [ 3443.526528] nvme_fabrics():1 [ 3443.528875] blk_complete_reqs+0x5c/0x74 [ 3443.532523] nvme_loop():1 [ 3443.535477] blk_done_softirq+0x38/0x4c [ 3443.539385] nvmet():1 [ 3443.542078] __do_softirq+0x168/0x418 [ 3443.545900] nvme_fabrics():1 [ 3443.548246] __irq_exit_rcu+0x120/0x170 [ 3443.551894] nvme_loop():1 [ 3443.554848] irq_exit_rcu+0x18/0x24 [ 3443.558669] nvmet():1 [ 3443.561363] el1_interrupt+0x38/0x70 [ 3443.564837] nvme_fabrics():1 [ 3443.567183] el1h_64_irq_handler+0x18/0x24 [ 3443.570744] nvme_loop():1 [ 3443.573698] el1h_64_irq+0x68/0x6c [ 3443.577780] nvmet():1 [ 3443.580473] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3443.583860] nvme_fabrics():1 [ 3443.586206] __wake_up_common_lock+0x84/0xac [ 3443.590983] nvme_loop():1 [ 3443.593936] __wake_up+0x20/0x30 [ 3443.598192] nvmet():1 [ 3443.600885] __sbq_wake_up+0x13c/0x1f0 [ 3443.604099] nvme_fabrics():1 [ 3443.606445] sbitmap_queue_clear+0x74/0xc0 [ 3443.610180] nvme_loop():1 [ 3443.613133] blk_mq_put_tag+0x44/0x70 [ 3443.617215] null_blk():1 [ 3443.619908] __blk_mq_free_request+0x88/0xf0 [ 3443.623556] null_blk():1 [ 3443.626163] blk_mq_free_request+0xc0/0x160 [ 3443.630419] null_blk():1 [ 3443.633025] bfq_insert_request.isra.0+0x204/0x270 [ 3443.637194] null_blk():1 [ 3443.639800] bfq_insert_requests+0x70/0x90 [ 3443.644577] null_blk():1 [ 3443.647183] blk_mq_sched_insert_requests+0xac/0x250 [ 3443.651265] null_blk():1 [ 3443.653871] blk_mq_flush_plug_list+0x110/0x2cc [ 3443.658822] null_blk():1 [ 3443.661428] __blk_flush_plug+0x100/0x140 [ 3443.665946] scsi_debug():1 [ 3443.668552] blk_finish_plug+0x40/0x25c [ 3443.672548] scsi_debug():1 [ 3443.675328] __iomap_dio_rw+0x1b4/0x500 [ 3443.679149] ipmi_si():1 [ 3443.681930] iomap_dio_rw+0x18/0x50 [ 3443.685751] ipmi_si():1 [ 3443.688271] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3443.691746] [last unloaded: nvmet] [ 3443.694266] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3443.699563] [ 3443.703037] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3443.707727] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3443.709205] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3443.714242] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3443.722056] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3443.727180] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3443.736471] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3443.741595] pc : __sbq_wake_up+0x40/0x1f0 [ 3443.748541] process_one_work+0x1ec/0x470 [ 3443.753926] lr : sbitmap_queue_clear+0x74/0xc0 [ 3443.757922] worker_thread+0x74/0x410 [ 3443.761917] sp : ffff8000088438b0 [ 3443.766347] kthread+0xec/0x100 [ 3443.769995] x29: ffff8000088438b0 [ 3443.773295] ret_from_fork+0x10/0x20 [ 3443.776423] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3443.788582] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3443.795704] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3443.802826] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3443.809948] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3443.817070] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3443.824191] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3443.831313] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3443.838435] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3443.845557] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 3443.852679] Call trace: [ 3443.855113] __sbq_wake_up+0x40/0x1f0 [ 3443.858762] sbitmap_queue_clear+0x74/0xc0 [ 3443.862845] blk_mq_put_tag+0x44/0x70 [ 3443.866495] __blk_mq_free_request+0x88/0xf0 [ 3443.870752] blk_mq_free_request+0xc0/0x160 [ 3443.874921] __blk_mq_end_request+0x78/0x150 [ 3443.879178] scsi_end_request+0x114/0x260 [ 3443.883174] scsi_io_completion+0x64/0x174 [ 3443.887258] scsi_finish_command+0xe0/0x120 [ 3443.891429] scsi_complete+0x8c/0x1c4 [ 3443.895078] blk_complete_reqs+0x5c/0x74 [ 3443.898988] blk_done_softirq+0x38/0x4c [ 3443.902810] __do_softirq+0x168/0x418 [ 3443.906459] __irq_exit_rcu+0x120/0x170 [ 3443.910282] irq_exit_rcu+0x18/0x24 [ 3443.913758] el1_interrupt+0x38/0x70 [ 3443.917320] el1h_64_irq_handler+0x18/0x24 [ 3443.921404] el1h_64_irq+0x68/0x6c [ 3443.924792] cpuidle_enter_state+0xc4/0x4e0 [ 3443.928963] cpuidle_enter+0x40/0x60 [ 3443.932525] cpuidle_idle_call+0x118/0x1b0 [ 3443.936608] do_idle+0xac/0x100 [ 3443.939737] cpu_startup_entry+0x30/0x34 [ 3443.943646] secondary_start_kernel+0xd8/0xf0 [ 3443.947990] __secondary_switched+0xb0/0xb4 [ 3467.091686] watchdog: BUG: soft lockup - CPU#8 stuck for 2834s! [kworker/8:12:19399] [ 3467.099415] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3467.099433] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3467.187222] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3467.195472] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3467.201685] watchdog: BUG: soft lockup - CPU#68 stuck for 2835s! [swapper/68:0] [ 3467.204763] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3467.212057] Modules linked in: [ 3467.218222] [ 3467.218223] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3467.221263] nvme_loop [ 3467.222741] pc : __sbq_wake_up+0x58/0x1f0 [ 3467.229688] nvme_fabrics [ 3467.232034] lr : sbitmap_queue_clear+0x74/0xc0 [ 3467.236029] nvmet [ 3467.238635] sp : ffff80000d79b280 [ 3467.241695] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 301629 jiffies s: 2093 root: 0x11/. [ 3467.241709] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3467.241719] Task dump for CPU 8: [ 3467.241720] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3467.241727] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3467.241735] Call trace: [ 3467.241736] __switch_to+0xd4/0x100 [ 3467.241743] 0xffff6c0200000080 [ 3467.241747] Task dump for CPU 68: [ 3467.241749] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3467.241755] Call trace: [ 3467.241756] __switch_to+0xd4/0x100 [ 3467.241762] 0x0 [ 3467.243064] nls_utf8 [ 3467.245064] x29: ffff80000d79b280 [ 3467.248365] hfsplus [ 3467.260086] x28: ffffd5becb6aec50 [ 3467.269290] hfs [ 3467.272504] x27: ffffd5beca4f7000 [ 3467.282402] isofs [ 3467.288568] [ 3467.291001] nvme_core [ 3467.294475] x26: ffffd5becaf68a98 [ 3467.297603] nvme_common [ 3467.300904] x25: ffffd5becb6a60c0 [ 3467.310802] loop [ 3467.313235] x24: fffffbffef180dc0 [ 3467.316709] tls [ 3467.318534] [ 3467.320794] rfkill [ 3467.324094] x23: ffff07ff81d381b0 [ 3467.326267] sunrpc [ 3467.329654] x22: ffff80000d79b30c [ 3467.331479] vfat [ 3467.334867] x21: ffff07ff84e93c00 [ 3467.336866] fat [ 3467.338344] [ 3467.340689] acpi_ipmi [ 3467.343990] x20: 00000000000001c0 [ 3467.346510] ipmi_ssif [ 3467.349897] x19: ffff07ff91ccdb10 [ 3467.351809] cppc_cpufreq [ 3467.355197] x18: ffffffffffffffff [ 3467.357022] ipmi_devintf [ 3467.358500] [ 3467.360585] igb [ 3467.363886] x17: ffff32506c412000 [ 3467.365971] ipmi_msghandler [ 3467.369359] x16: ffff800008044000 [ 3467.371271] arm_cmn [ 3467.374658] x15: 0000000000004000 [ 3467.376484] arm_dmc620_pmu [ 3467.377961] [ 3467.380307] arm_spe_pmu [ 3467.383608] x14: 010320bb28ec0000 [ 3467.385954] arm_dsu_pmu [ 3467.389342] x13: 0000000000000087 [ 3467.391948] acpi_tad [ 3467.395335] x12: 071c71c71c71c71c [ 3467.397942] fuse [ 3467.399420] [ 3467.401245] zram [ 3467.404546] x11: 0000000000000040 [ 3467.407413] xfs [ 3467.410800] x10: ffff07ff801a9c70 [ 3467.412973] crct10dif_ce [ 3467.416360] x9 : ffffd5bec9545fd8 [ 3467.419140] polyval_ce [ 3467.420618] [ 3467.423138] ast [ 3467.426439] x8 : ffff07ff80404b70 [ 3467.428958] polyval_generic [ 3467.432346] x7 : ffff07ff84e93dc8 [ 3467.434605] drm_vram_helper [ 3467.437993] x6 : 0000000000000008 [ 3467.439904] ghash_ce [ 3467.441382] [ 3467.443294] drm_ttm_helper [ 3467.446595] x5 : ffff80000d043790 [ 3467.448420] sbsa_gwdt [ 3467.451808] x4 : ffff07ff84e93dc0 [ 3467.454414] ttm [ 3467.457802] x3 : 0000000000000008 [ 3467.460234] xgene_hwmon [ 3467.461712] [ 3467.463537] Unloaded tainted modules: [ 3467.466838] x2 : 0000000000000007 [ 3467.469705] nvmet():1 [ 3467.473092] x1 : ffff80000d79b30c [ 3467.475959] nvme_fabrics():1 [ 3467.479347] x0 : ffff07ff84e93dd0 [ 3467.481606] nvme_loop():1 [ 3467.483084] [ 3467.485863] nvmet():1 [ 3467.489165] Call trace: [ 3467.491510] nvme_fabrics():1 [ 3467.494898] __sbq_wake_up+0x58/0x1f0 [ 3467.496723] nvme_loop():1 [ 3467.500111] sbitmap_queue_clear+0x74/0xc0 [ 3467.502630] nvmet():1 [ 3467.504108] blk_mq_put_tag+0x44/0x70 [ 3467.507756] nvme_fabrics():1 [ 3467.511057] __blk_mq_free_request+0x88/0xf0 [ 3467.513402] nvme_loop():1 [ 3467.516790] blk_mq_free_request+0xc0/0x160 [ 3467.519744] nvmet():1 [ 3467.523131] __blk_mq_end_request+0x78/0x150 [ 3467.525824] nvme_fabrics():1 [ 3467.527302] scsi_end_request+0x114/0x260 [ 3467.529648] nvme_loop():1 [ 3467.532081] scsi_io_completion+0x64/0x174 [ 3467.535034] nvmet():1 [ 3467.538682] scsi_finish_command+0xe0/0x120 [ 3467.541375] nvme_fabrics():1 [ 3467.545458] scsi_complete+0x8c/0x1c4 [ 3467.547803] nvme_loop():1 [ 3467.551451] blk_complete_reqs+0x5c/0x74 [ 3467.554405] nvmet():1 [ 3467.558661] blk_done_softirq+0x38/0x4c [ 3467.561354] nvme_fabrics():1 [ 3467.565522] __do_softirq+0x168/0x418 [ 3467.567868] nvme_loop():1 [ 3467.572124] __irq_exit_rcu+0x120/0x170 [ 3467.575078] nvmet():1 [ 3467.579073] irq_exit_rcu+0x18/0x24 [ 3467.581766] nvme_fabrics():1 [ 3467.585848] el1_interrupt+0x38/0x70 [ 3467.588194] nvme_loop():1 [ 3467.592363] el1h_64_irq_handler+0x18/0x24 [ 3467.595316] nvmet():1 [ 3467.598964] el1h_64_irq+0x68/0x6c [ 3467.601657] nvme_fabrics():1 [ 3467.605566] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3467.607912] nvme_loop():1 [ 3467.611733] __wake_up_common_lock+0x84/0xac [ 3467.614687] nvmet():1 [ 3467.618335] __wake_up+0x20/0x30 [ 3467.621028] nvme_fabrics():1 [ 3467.624851] __sbq_wake_up+0x13c/0x1f0 [ 3467.627197] nvme_loop():1 [ 3467.630671] sbitmap_queue_clear+0x74/0xc0 [ 3467.633626] nvmet():1 [ 3467.637188] blk_mq_put_tag+0x44/0x70 [ 3467.639881] nvme_fabrics():1 [ 3467.643963] __blk_mq_free_request+0x88/0xf0 [ 3467.646309] nvme_loop():1 [ 3467.649696] blk_mq_free_request+0xc0/0x160 [ 3467.652650] null_blk():1 [ 3467.657426] bfq_insert_request.isra.0+0x204/0x270 [ 3467.660120] null_blk():1 [ 3467.664375] bfq_insert_requests+0x70/0x90 [ 3467.666721] null_blk():1 [ 3467.669935] blk_mq_sched_insert_requests+0xac/0x250 [ 3467.672889] null_blk():1 [ 3467.676623] blk_mq_flush_plug_list+0x110/0x2cc [ 3467.679317] null_blk():1 [ 3467.683399] __blk_flush_plug+0x100/0x140 [ 3467.685744] null_blk():1 [ 3467.689392] blk_finish_plug+0x40/0x25c [ 3467.692346] null_blk():1 [ 3467.696602] __iomap_dio_rw+0x1b4/0x500 [ 3467.699295] scsi_debug():1 [ 3467.703464] iomap_dio_rw+0x18/0x50 [ 3467.706070] scsi_debug():1 [ 3467.710846] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3467.713453] ipmi_si():1 [ 3467.717535] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3467.720141] ipmi_si():1 [ 3467.725091] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3467.727697] [last unloaded: nvmet] [ 3467.732213] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3467.734820] [ 3467.738815] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3467.741421] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3467.745243] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3467.747849] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3467.751671] process_one_work+0x1ec/0x470 [ 3467.754451] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3467.757925] worker_thread+0x74/0x410 [ 3467.760706] pc : __sbq_wake_up+0x58/0x1f0 [ 3467.766003] kthread+0xec/0x100 [ 3467.768522] lr : sbitmap_queue_clear+0x74/0xc0 [ 3467.773212] ret_from_fork+0x10/0x20 [ 3467.775732] sp : ffff8000088438b0 [ 3467.851460] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3467.858582] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3467.865704] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3467.872826] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3467.879948] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3467.887070] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3467.894191] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3467.901313] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3467.908435] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3467.915557] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3467.922679] Call trace: [ 3467.925112] __sbq_wake_up+0x58/0x1f0 [ 3467.928761] sbitmap_queue_clear+0x74/0xc0 [ 3467.932844] blk_mq_put_tag+0x44/0x70 [ 3467.936494] __blk_mq_free_request+0x88/0xf0 [ 3467.940750] blk_mq_free_request+0xc0/0x160 [ 3467.944920] __blk_mq_end_request+0x78/0x150 [ 3467.949177] scsi_end_request+0x114/0x260 [ 3467.953173] scsi_io_completion+0x64/0x174 [ 3467.957257] scsi_finish_command+0xe0/0x120 [ 3467.961428] scsi_complete+0x8c/0x1c4 [ 3467.965078] blk_complete_reqs+0x5c/0x74 [ 3467.968987] blk_done_softirq+0x38/0x4c [ 3467.972810] __do_softirq+0x168/0x418 [ 3467.976459] __irq_exit_rcu+0x120/0x170 [ 3467.980282] irq_exit_rcu+0x18/0x24 [ 3467.983757] el1_interrupt+0x38/0x70 [ 3467.987319] el1h_64_irq_handler+0x18/0x24 [ 3467.991402] el1h_64_irq+0x68/0x6c [ 3467.994791] cpuidle_enter_state+0xc4/0x4e0 [ 3467.998961] cpuidle_enter+0x40/0x60 [ 3468.002524] cpuidle_idle_call+0x118/0x1b0 [ 3468.006608] do_idle+0xac/0x100 [ 3468.009737] cpu_startup_entry+0x30/0x34 [ 3468.013646] secondary_start_kernel+0xd8/0xf0 [ 3468.017990] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 04:55:00 2022] [ 3491.091526] watchdog: BUG: soft lockup - CPU#8 stuck for 2857s! [kworker/8:12:19399] [ 3491.099256] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3491.099273] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3491.187062] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3491.195312] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3491.201526] watchdog: BUG: soft lockup - CPU#68 stuck for 2857s! [swapper/68:0] [ 3491.204603] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3491.211896] Modules linked in: [ 3491.218062] [ 3491.218063] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3491.221103] nvme_loop [ 3491.222581] pc : __sbq_wake_up+0x84/0x1f0 [ 3491.229527] nvme_fabrics [ 3491.231873] lr : sbitmap_queue_clear+0x74/0xc0 [ 3491.235869] nvmet [ 3491.238475] sp : ffff80000d79b280 [ 3491.242904] nls_utf8 [ 3491.244903] x29: ffff80000d79b280 [ 3491.248204] hfsplus [ 3491.250463] x28: ffffd5becb6aec50 [ 3491.253764] hfs [ 3491.255936] x27: ffffd5beca4f7000 [ 3491.259324] isofs [ 3491.261149] [ 3491.264537] nvme_core [ 3491.266535] x26: ffffd5becaf68a98 [ 3491.268013] nvme_common [ 3491.270360] x25: ffffd5becb6a60c0 [ 3491.273660] loop [ 3491.276180] x24: fffffbffef180dc0 [ 3491.279567] tls [ 3491.281479] [ 3491.284867] rfkill [ 3491.286692] x23: ffff07ff81d381b0 [ 3491.288170] sunrpc [ 3491.290255] x22: ffff80000d79b30c [ 3491.293556] vfat [ 3491.295642] x21: ffff07ff84e93c00 [ 3491.299029] fat [ 3491.300941] [ 3491.304329] acpi_ipmi [ 3491.306154] x20: 00000000000001c0 [ 3491.307632] ipmi_ssif [ 3491.309980] x19: ffff07ff91ccdb10 [ 3491.313280] cppc_cpufreq [ 3491.315626] x18: ffffffffffffffff [ 3491.319014] ipmi_devintf [ 3491.321620] [ 3491.325008] igb [ 3491.327614] x17: ffff32506c412000 [ 3491.329092] ipmi_msghandler [ 3491.330917] x16: ffff800008044000 [ 3491.334218] arm_cmn [ 3491.337085] x15: 0000000000004000 [ 3491.340472] arm_dmc620_pmu [ 3491.342644] [ 3491.346033] arm_spe_pmu [ 3491.348814] x14: 010320bb28ec0000 [ 3491.350292] arm_dsu_pmu [ 3491.352811] x13: 0000000000000087 [ 3491.356112] acpi_tad [ 3491.358632] x12: 071c71c71c71c71c [ 3491.362021] fuse [ 3491.364280] [ 3491.367668] zram [ 3491.369579] x11: 0000000000000040 [ 3491.371057] xfs [ 3491.372969] x10: ffff07ff801a9c70 [ 3491.376270] crct10dif_ce [ 3491.378095] x9 : ffffd5bec9545fd8 [ 3491.381483] polyval_ce [ 3491.384089] [ 3491.387477] ast [ 3491.389909] x8 : ffff07ff80404b70 [ 3491.391387] polyval_generic [ 3491.393212] x7 : ffff07ff84e93dc8 [ 3491.396513] drm_vram_helper [ 3491.399380] x6 : 0000000000000008 [ 3491.402767] ghash_ce [ 3491.405635] [ 3491.409023] drm_ttm_helper [ 3491.411282] x5 : ffff80000d043790 [ 3491.412760] sbsa_gwdt [ 3491.415540] x4 : ffff07ff84e93dc0 [ 3491.418840] ttm [ 3491.421186] x3 : 0000000000000008 [ 3491.424574] xgene_hwmon [ 3491.426399] [ 3491.429787] Unloaded tainted modules: [ 3491.432306] x2 : 0000000000000007 [ 3491.433784] nvmet():1 [ 3491.437432] x1 : ffff80000d79b30c [ 3491.440734] nvme_fabrics():1 [ 3491.443080] x0 : ffff07ff84e93dd0 [ 3491.446468] nvme_loop():1 [ 3491.449421] [ 3491.452809] nvmet():1 [ 3491.455502] Call trace: [ 3491.456980] nvme_fabrics():1 [ 3491.459326] __sbq_wake_up+0x84/0x1f0 [ 3491.461759] nvme_loop():1 [ 3491.464712] sbitmap_queue_clear+0x74/0xc0 [ 3491.468360] nvmet():1 [ 3491.471053] blk_mq_put_tag+0x44/0x70 [ 3491.475135] nvme_fabrics():1 [ 3491.477481] __blk_mq_free_request+0x88/0xf0 [ 3491.481129] nvme_loop():1 [ 3491.484083] blk_mq_free_request+0xc0/0x160 [ 3491.488338] nvmet():1 [ 3491.491031] __blk_mq_end_request+0x78/0x150 [ 3491.495200] nvme_fabrics():1 [ 3491.497546] scsi_end_request+0x114/0x260 [ 3491.501802] nvme_loop():1 [ 3491.504755] scsi_io_completion+0x64/0x174 [ 3491.508751] nvmet():1 [ 3491.511444] scsi_finish_command+0xe0/0x120 [ 3491.515526] nvme_fabrics():1 [ 3491.517872] scsi_complete+0x8c/0x1c4 [ 3491.522040] nvme_loop():1 [ 3491.524994] blk_complete_reqs+0x5c/0x74 [ 3491.528642] nvmet():1 [ 3491.531335] blk_done_softirq+0x38/0x4c [ 3491.535243] nvme_fabrics():1 [ 3491.537589] __do_softirq+0x168/0x418 [ 3491.541411] nvme_loop():1 [ 3491.544365] __irq_exit_rcu+0x120/0x170 [ 3491.548013] nvmet():1 [ 3491.550706] irq_exit_rcu+0x18/0x24 [ 3491.554527] nvme_fabrics():1 [ 3491.556873] el1_interrupt+0x38/0x70 [ 3491.560348] nvme_loop():1 [ 3491.563301] el1h_64_irq_handler+0x18/0x24 [ 3491.566862] nvmet():1 [ 3491.569555] el1h_64_irq+0x68/0x6c [ 3491.573637] nvme_fabrics():1 [ 3491.575983] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3491.579371] nvme_loop():1 [ 3491.582324] __wake_up_common_lock+0x84/0xac [ 3491.587101] nvmet():1 [ 3491.589794] __wake_up+0x20/0x30 [ 3491.594050] nvme_fabrics():1 [ 3491.596396] __sbq_wake_up+0x13c/0x1f0 [ 3491.599610] nvme_loop():1 [ 3491.602563] sbitmap_queue_clear+0x74/0xc0 [ 3491.606298] nvmet():1 [ 3491.608991] blk_mq_put_tag+0x44/0x70 [ 3491.613073] nvme_fabrics():1 [ 3491.615419] __blk_mq_free_request+0x88/0xf0 [ 3491.619067] nvme_loop():1 [ 3491.622021] blk_mq_free_request+0xc0/0x160 [ 3491.626276] null_blk():1 [ 3491.628969] bfq_insert_request.isra.0+0x204/0x270 [ 3491.633139] null_blk():1 [ 3491.635747] bfq_insert_requests+0x70/0x90 [ 3491.640523] null_blk():1 [ 3491.643129] blk_mq_sched_insert_requests+0xac/0x250 [ 3491.647211] null_blk():1 [ 3491.649818] blk_mq_flush_plug_list+0x110/0x2cc [ 3491.654768] null_blk():1 [ 3491.657374] __blk_flush_plug+0x100/0x140 [ 3491.661890] null_blk():1 [ 3491.664497] blk_finish_plug+0x40/0x25c [ 3491.668494] null_blk():1 [ 3491.671100] __iomap_dio_rw+0x1b4/0x500 [ 3491.674922] scsi_debug():1 [ 3491.677529] iomap_dio_rw+0x18/0x50 [ 3491.681350] scsi_debug():1 [ 3491.684131] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3491.687607] ipmi_si():1 [ 3491.690387] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3491.695684] ipmi_si():1 [ 3491.698203] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3491.702893] [last unloaded: nvmet] [ 3491.705413] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3491.710449] [ 3491.713924] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3491.719048] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3491.720525] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3491.725649] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3491.733464] process_one_work+0x1ec/0x470 [ 3491.738848] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3491.748139] worker_thread+0x74/0x410 [ 3491.752134] pc : __sbq_wake_up+0x58/0x1f0 [ 3491.759081] kthread+0xec/0x100 [ 3491.762728] lr : sbitmap_queue_clear+0x74/0xc0 [ 3491.766724] ret_from_fork+0x10/0x20 [ 3491.769851] sp : ffff8000088438b0 [ 3491.781143] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3491.788265] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3491.795387] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3491.802508] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3491.809630] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3491.816753] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3491.823875] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3491.830997] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3491.838119] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3491.845241] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3491.852362] Call trace: [ 3491.854795] __sbq_wake_up+0x58/0x1f0 [ 3491.858444] sbitmap_queue_clear+0x74/0xc0 [ 3491.862527] blk_mq_put_tag+0x44/0x70 [ 3491.866177] __blk_mq_free_request+0x88/0xf0 [ 3491.870434] blk_mq_free_request+0xc0/0x160 [ 3491.874604] __blk_mq_end_request+0x78/0x150 [ 3491.878860] scsi_end_request+0x114/0x260 [ 3491.882857] scsi_io_completion+0x64/0x174 [ 3491.886940] scsi_finish_command+0xe0/0x120 [ 3491.891110] scsi_complete+0x8c/0x1c4 [ 3491.894759] blk_complete_reqs+0x5c/0x74 [ 3491.898669] blk_done_softirq+0x38/0x4c [ 3491.902491] __do_softirq+0x168/0x418 [ 3491.906140] __irq_exit_rcu+0x120/0x170 [ 3491.909963] irq_exit_rcu+0x18/0x24 [ 3491.913438] el1_interrupt+0x38/0x70 [ 3491.917000] el1h_64_irq_handler+0x18/0x24 [ 3491.921083] el1h_64_irq+0x68/0x6c [ 3491.924472] cpuidle_enter_state+0xc4/0x4e0 [ 3491.928642] cpuidle_enter+0x40/0x60 [ 3491.932204] cpuidle_idle_call+0x118/0x1b0 [ 3491.936288] do_idle+0xac/0x100 [ 3491.939417] cpu_startup_entry+0x30/0x34 [ 3491.943326] secondary_start_kernel+0xd8/0xf0 [ 3491.947670] __secondary_switched+0xb0/0xb4 [ 3515.091367] watchdog: BUG: soft lockup - CPU#8 stuck for 2879s! [kworker/8:12:19399] [ 3515.099096] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3515.099113] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3515.186902] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3515.195151] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3515.201366] watchdog: BUG: soft lockup - CPU#68 stuck for 2879s! [swapper/68:0] [ 3515.204442] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3515.211736] Modules linked in: nvme_loop [ 3515.217902] [ 3515.217902] nvme_fabrics nvmet [ 3515.221811] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3515.223289] nls_utf8 [ 3515.226416] pc : __sbq_wake_up+0x40/0x1f0 [ 3515.233363] hfsplus [ 3515.235622] lr : sbitmap_queue_clear+0x74/0xc0 [ 3515.239617] hfs [ 3515.241790] sp : ffff80000d79b280 [ 3515.246219] isofs [ 3515.248044] x29: ffff80000d79b280 [ 3515.251345] nvme_core [ 3515.253344] x28: ffffd5becb6aec50 [ 3515.256646] nvme_common [ 3515.258992] x27: ffffd5beca4f7000 [ 3515.262380] loop [ 3515.264899] [ 3515.268287] tls [ 3515.270199] x26: ffffd5becaf68a98 [ 3515.271677] rfkill [ 3515.273502] x25: ffffd5becb6a60c0 [ 3515.276802] sunrpc [ 3515.278888] x24: fffffbffef180dc0 [ 3515.282276] vfat [ 3515.284361] [ 3515.287749] fat [ 3515.289661] x23: ffff07ff81d381b0 [ 3515.291139] acpi_ipmi [ 3515.292964] x22: ffff80000d79b30c [ 3515.296265] ipmi_ssif [ 3515.298611] x21: ffff07ff84e93c00 [ 3515.301998] cppc_cpufreq [ 3515.304344] [ 3515.307732] ipmi_devintf [ 3515.310338] x20: 0000000000000024 [ 3515.311816] igb [ 3515.314422] x19: ffff07ff91ccdb10 [ 3515.317723] ipmi_msghandler [ 3515.319548] x18: ffffffffffffffff [ 3515.322936] arm_cmn [ 3515.325802] [ 3515.329190] arm_dmc620_pmu [ 3515.331362] x17: ffff32506c412000 [ 3515.332840] arm_spe_pmu [ 3515.335622] x16: ffff800008044000 [ 3515.338922] arm_dsu_pmu [ 3515.341442] x15: 0000000000004000 [ 3515.344831] acpi_tad [ 3515.347351] [ 3515.350738] fuse [ 3515.352997] x14: 010320bb28ec0000 [ 3515.354475] zram [ 3515.356388] x13: 0000000000000087 [ 3515.359689] xfs [ 3515.361601] x12: 071c71c71c71c71c [ 3515.364989] crct10dif_ce [ 3515.366814] [ 3515.370202] polyval_ce [ 3515.372808] x11: 0000000000000040 [ 3515.374286] ast [ 3515.376718] x10: ffff07ff801a9c70 [ 3515.380019] polyval_generic [ 3515.381844] x9 : ffffd5bec9545fd8 [ 3515.385232] drm_vram_helper [ 3515.388100] [ 3515.391488] ghash_ce [ 3515.394355] x8 : ffff07ff80404b70 [ 3515.395833] drm_ttm_helper [ 3515.398094] x7 : ffff07ff84e93dc8 [ 3515.401395] sbsa_gwdt [ 3515.404175] x6 : 0000000000000008 [ 3515.407563] ttm [ 3515.409909] [ 3515.413296] xgene_hwmon [ 3515.415121] x5 : ffff80000d043790 [ 3515.416599] Unloaded tainted modules: [ 3515.419119] x4 : ffff07ff84e93dc0 [ 3515.422419] nvmet():1 [ 3515.426067] x3 : 0000000000000008 [ 3515.429455] nvme_fabrics():1 [ 3515.431801] [ 3515.435188] nvme_loop():1 [ 3515.438142] x2 : 0000000000000007 [ 3515.439620] nvmet():1 [ 3515.442313] x1 : ffff80000d79b30c [ 3515.445614] nvme_fabrics():1 [ 3515.447959] x0 : 0000000000000038 [ 3515.451347] nvme_loop():1 [ 3515.454301] [ 3515.457688] nvmet():1 [ 3515.460381] Call trace: [ 3515.461859] nvme_fabrics():1 [ 3515.464205] __sbq_wake_up+0x40/0x1f0 [ 3515.466638] nvme_loop():1 [ 3515.469591] sbitmap_queue_clear+0x74/0xc0 [ 3515.473239] nvmet():1 [ 3515.475933] blk_mq_put_tag+0x44/0x70 [ 3515.480016] nvme_fabrics():1 [ 3515.482363] __blk_mq_free_request+0x88/0xf0 [ 3515.486011] nvme_loop():1 [ 3515.488964] blk_mq_free_request+0xc0/0x160 [ 3515.493220] nvmet():1 [ 3515.495913] __blk_mq_end_request+0x78/0x150 [ 3515.500082] nvme_fabrics():1 [ 3515.502428] scsi_end_request+0x114/0x260 [ 3515.506683] nvme_loop():1 [ 3515.509637] scsi_io_completion+0x64/0x174 [ 3515.513632] nvmet():1 [ 3515.516327] scsi_finish_command+0xe0/0x120 [ 3515.520409] nvme_fabrics():1 [ 3515.522756] scsi_complete+0x8c/0x1c4 [ 3515.526925] nvme_loop():1 [ 3515.529878] blk_complete_reqs+0x5c/0x74 [ 3515.533526] nvmet():1 [ 3515.536219] blk_done_softirq+0x38/0x4c [ 3515.540128] nvme_fabrics():1 [ 3515.542474] __do_softirq+0x168/0x418 [ 3515.546295] nvme_loop():1 [ 3515.549249] __irq_exit_rcu+0x120/0x170 [ 3515.552897] nvmet():1 [ 3515.555590] irq_exit_rcu+0x18/0x24 [ 3515.559411] nvme_fabrics():1 [ 3515.561757] el1_interrupt+0x38/0x70 [ 3515.565232] nvme_loop():1 [ 3515.568185] el1h_64_irq_handler+0x18/0x24 [ 3515.571746] nvmet():1 [ 3515.574440] el1h_64_irq+0x68/0x6c [ 3515.578522] nvme_fabrics():1 [ 3515.580868] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3515.584255] nvme_loop():1 [ 3515.587209] __wake_up_common_lock+0x84/0xac [ 3515.591985] nvmet():1 [ 3515.594678] __wake_up+0x20/0x30 [ 3515.598934] nvme_fabrics():1 [ 3515.601281] __sbq_wake_up+0x13c/0x1f0 [ 3515.604495] nvme_loop():1 [ 3515.607449] sbitmap_queue_clear+0x74/0xc0 [ 3515.611184] null_blk():1 [ 3515.613877] blk_mq_put_tag+0x44/0x70 [ 3515.617959] null_blk():1 [ 3515.620565] __blk_mq_free_request+0x88/0xf0 [ 3515.624213] null_blk():1 [ 3515.626819] blk_mq_free_request+0xc0/0x160 [ 3515.631075] null_blk():1 [ 3515.633681] bfq_insert_request.isra.0+0x204/0x270 [ 3515.637850] null_blk():1 [ 3515.640457] bfq_insert_requests+0x70/0x90 [ 3515.645233] null_blk():1 [ 3515.647839] blk_mq_sched_insert_requests+0xac/0x250 [ 3515.651921] null_blk():1 [ 3515.654528] blk_mq_flush_plug_list+0x110/0x2cc [ 3515.659478] scsi_debug():1 [ 3515.662084] __blk_flush_plug+0x100/0x140 [ 3515.666602] scsi_debug():1 [ 3515.669382] blk_finish_plug+0x40/0x25c [ 3515.673377] ipmi_si():1 [ 3515.676157] __iomap_dio_rw+0x1b4/0x500 [ 3515.679979] ipmi_si():1 [ 3515.682499] iomap_dio_rw+0x18/0x50 [ 3515.686320] [last unloaded: nvmet] [ 3515.688840] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3515.692314] [ 3515.695788] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3515.701086] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3515.702564] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3515.707255] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3515.715069] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3515.720106] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3515.729397] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3515.734520] pc : __sbq_wake_up+0x58/0x1f0 [ 3515.741467] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3515.746591] lr : sbitmap_queue_clear+0x74/0xc0 [ 3515.750586] process_one_work+0x1ec/0x470 [ 3515.755970] sp : ffff8000088438b0 [ 3515.760399] worker_thread+0x74/0x410 [ 3515.764394] x29: ffff8000088438b0 [ 3515.767695] kthread+0xec/0x100 [ 3515.771343] x28: ffffd5becb6aec50 [ 3515.774644] ret_from_fork+0x10/0x20 [ 3515.777771] x27: ffffd5beca4f7000 [ 3515.788107] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3515.795230] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3515.802353] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3515.809475] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3515.816597] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3515.823719] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3515.830840] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3515.837962] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3515.845084] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3515.852206] Call trace: [ 3515.854639] __sbq_wake_up+0x58/0x1f0 [ 3515.858288] sbitmap_queue_clear+0x74/0xc0 [ 3515.862371] blk_mq_put_tag+0x44/0x70 [ 3515.866021] __blk_mq_free_request+0x88/0xf0 [ 3515.870278] blk_mq_free_request+0xc0/0x160 [ 3515.874448] __blk_mq_end_request+0x78/0x150 [ 3515.878704] scsi_end_request+0x114/0x260 [ 3515.882701] scsi_io_completion+0x64/0x174 [ 3515.886784] scsi_finish_command+0xe0/0x120 [ 3515.890954] scsi_complete+0x8c/0x1c4 [ 3515.894603] blk_complete_reqs+0x5c/0x74 [ 3515.898513] blk_done_softirq+0x38/0x4c [ 3515.902335] __do_softirq+0x168/0x418 [ 3515.905984] __irq_exit_rcu+0x120/0x170 [ 3515.909806] irq_exit_rcu+0x18/0x24 [ 3515.913282] el1_interrupt+0x38/0x70 [ 3515.916844] el1h_64_irq_handler+0x18/0x24 [ 3515.920927] el1h_64_irq+0x68/0x6c [ 3515.924315] cpuidle_enter_state+0xc4/0x4e0 [ 3515.928485] cpuidle_enter+0x40/0x60 [ 3515.932047] cpuidle_idle_call+0x118/0x1b0 [ 3515.936131] do_idle+0xac/0x100 [ 3515.939261] cpu_startup_entry+0x30/0x34 [ 3515.943170] secondary_start_kernel+0xd8/0xf0 [ 3515.947514] __secondary_switched+0xb0/0xb4 [ 3533.791253] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 308285 jiffies s: 2093 root: 0x11/. [ 3533.802991] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3533.812200] Task dump for CPU 8: [ 3533.815414] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3533.825316] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3533.831485] Call trace: [ 3533.833918] __switch_to+0xd4/0x100 [ 3533.837395] 0xffff6c0200000080 [ 3533.840523] Task dump for CPU 68: [ 3533.843825] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3533.853727] Call trace: [ 3533.856160] __switch_to+0xd4/0x100 [ 3533.859636] 0x0 [ 3539.091207] watchdog: BUG: soft lockup - CPU#8 stuck for 2902s! [kworker/8:12:19399] [ 3539.098937] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3539.098954] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3539.186742] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3539.194991] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3539.201207] watchdog: BUG: soft lockup - CPU#68 stuck for 2902s! [swapper/68:0] [ 3539.204282] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3539.211576] Modules linked in: [ 3539.217742] [ 3539.217743] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3539.220783] nvme_loop [ 3539.222261] pc : __sbq_wake_up+0x58/0x1f0 [ 3539.229207] nvme_fabrics [ 3539.231553] lr : sbitmap_queue_clear+0x74/0xc0 [ 3539.235549] nvmet [ 3539.238155] sp : ffff80000d79b280 [ 3539.242584] nls_utf8 [ 3539.244583] x29: ffff80000d79b280 [ 3539.247884] hfsplus [ 3539.250143] x28: ffffd5becb6aec50 [ 3539.253443] hfs [ 3539.255616] x27: ffffd5beca4f7000 [ 3539.259003] isofs [ 3539.260828] [ 3539.264216] nvme_core [ 3539.266215] x26: ffffd5becaf68a98 [ 3539.267693] nvme_common [ 3539.270039] x25: ffffd5becb6a60c0 [ 3539.273340] loop [ 3539.275859] x24: fffffbffef180dc0 [ 3539.279247] tls [ 3539.281159] [ 3539.284546] rfkill [ 3539.286371] x23: ffff07ff81d381b0 [ 3539.287849] sunrpc [ 3539.289935] x22: ffff80000d79b30c [ 3539.293238] vfat [ 3539.295323] x21: ffff07ff84e93c00 [ 3539.298711] fat [ 3539.300623] [ 3539.304010] acpi_ipmi [ 3539.305835] x20: 00000000000001c0 [ 3539.307313] ipmi_ssif [ 3539.309659] x19: ffff07ff91ccdb10 [ 3539.312960] cppc_cpufreq [ 3539.315306] x18: ffffffffffffffff [ 3539.318693] ipmi_devintf [ 3539.321300] [ 3539.324687] igb [ 3539.327294] x17: ffff32506c412000 [ 3539.328771] ipmi_msghandler [ 3539.330597] x16: ffff800008044000 [ 3539.333897] arm_cmn [ 3539.336764] x15: 0000000000004000 [ 3539.340152] arm_dmc620_pmu [ 3539.342324] [ 3539.345711] arm_spe_pmu [ 3539.348491] x14: 010320bb28ec0000 [ 3539.349969] arm_dsu_pmu [ 3539.352489] x13: 0000000000000087 [ 3539.355790] acpi_tad [ 3539.358310] x12: 071c71c71c71c71c [ 3539.361697] fuse [ 3539.363956] [ 3539.367344] zram [ 3539.369257] x11: 0000000000000040 [ 3539.370735] xfs [ 3539.372648] x10: ffff07ff801a9c70 [ 3539.375949] crct10dif_ce [ 3539.377774] x9 : ffffd5bec9545fd8 [ 3539.381162] polyval_ce [ 3539.383768] [ 3539.387155] ast [ 3539.389588] x8 : ffff07ff80404b70 [ 3539.391066] polyval_generic [ 3539.392891] x7 : ffff07ff84e93dc8 [ 3539.396192] drm_vram_helper [ 3539.399059] x6 : 0000000000000008 [ 3539.402446] ghash_ce [ 3539.405313] [ 3539.408701] drm_ttm_helper [ 3539.410960] x5 : ffff80000d043790 [ 3539.412438] sbsa_gwdt [ 3539.415217] x4 : ffff07ff84e93dc0 [ 3539.418518] ttm [ 3539.420864] x3 : 0000000000000008 [ 3539.424252] xgene_hwmon [ 3539.426077] [ 3539.429464] Unloaded tainted modules: [ 3539.431984] x2 : 0000000000000007 [ 3539.433462] nvmet():1 [ 3539.437110] x1 : ffff80000d79b30c [ 3539.440411] nvme_fabrics():1 [ 3539.442756] x0 : ffff07ff84e93dd0 [ 3539.446144] nvme_loop():1 [ 3539.449098] [ 3539.452485] nvmet():1 [ 3539.455178] Call trace: [ 3539.456656] nvme_fabrics():1 [ 3539.459002] __sbq_wake_up+0x58/0x1f0 [ 3539.461435] nvme_loop():1 [ 3539.464388] sbitmap_queue_clear+0x74/0xc0 [ 3539.468037] nvmet():1 [ 3539.470731] blk_mq_put_tag+0x44/0x70 [ 3539.474813] nvme_fabrics():1 [ 3539.477159] __blk_mq_free_request+0x88/0xf0 [ 3539.480807] nvme_loop():1 [ 3539.483760] blk_mq_free_request+0xc0/0x160 [ 3539.488016] nvmet():1 [ 3539.490709] __blk_mq_end_request+0x78/0x150 [ 3539.494878] nvme_fabrics():1 [ 3539.497224] scsi_end_request+0x114/0x260 [ 3539.501480] nvme_loop():1 [ 3539.504433] scsi_io_completion+0x64/0x174 [ 3539.508428] nvmet():1 [ 3539.511122] scsi_finish_command+0xe0/0x120 [ 3539.515203] nvme_fabrics():1 [ 3539.517549] scsi_complete+0x8c/0x1c4 [ 3539.521718] nvme_loop():1 [ 3539.524672] blk_complete_reqs+0x5c/0x74 [ 3539.528320] nvmet():1 [ 3539.531013] blk_done_softirq+0x38/0x4c [ 3539.534921] nvme_fabrics():1 [ 3539.537267] __do_softirq+0x168/0x418 [ 3539.541089] nvme_loop():1 [ 3539.544042] __irq_exit_rcu+0x120/0x170 [ 3539.547690] nvmet():1 [ 3539.550384] irq_exit_rcu+0x18/0x24 [ 3539.554205] nvme_fabrics():1 [ 3539.556551] el1_interrupt+0x38/0x70 [ 3539.560025] nvme_loop():1 [ 3539.562979] el1h_64_irq_handler+0x18/0x24 [ 3539.566540] nvmet():1 [ 3539.569233] el1h_64_irq+0x68/0x6c [ 3539.573316] nvme_fabrics():1 [ 3539.575663] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3539.579050] nvme_loop():1 [ 3539.582004] __wake_up_common_lock+0x84/0xac [ 3539.586780] nvmet():1 [ 3539.589473] __wake_up+0x20/0x30 [ 3539.593729] nvme_fabrics():1 [ 3539.596075] __sbq_wake_up+0x13c/0x1f0 [ 3539.599289] nvme_loop():1 [ 3539.602243] sbitmap_queue_clear+0x74/0xc0 [ 3539.605977] nvmet():1 [ 3539.608670] blk_mq_put_tag+0x44/0x70 [ 3539.612752] nvme_fabrics():1 [ 3539.615098] __blk_mq_free_request+0x88/0xf0 [ 3539.618746] nvme_loop():1 [ 3539.621700] blk_mq_free_request+0xc0/0x160 [ 3539.625956] null_blk():1 [ 3539.628649] bfq_insert_request.isra.0+0x204/0x270 [ 3539.632818] null_blk():1 [ 3539.635424] bfq_insert_requests+0x70/0x90 [ 3539.640200] null_blk():1 [ 3539.642807] blk_mq_sched_insert_requests+0xac/0x250 [ 3539.646889] null_blk():1 [ 3539.649495] blk_mq_flush_plug_list+0x110/0x2cc [ 3539.654445] null_blk():1 [ 3539.657051] __blk_flush_plug+0x100/0x140 [ 3539.661567] null_blk():1 [ 3539.664174] blk_finish_plug+0x40/0x25c [ 3539.668169] null_blk():1 [ 3539.670775] __iomap_dio_rw+0x1b4/0x500 [ 3539.674597] scsi_debug():1 [ 3539.677204] iomap_dio_rw+0x18/0x50 [ 3539.681025] scsi_debug():1 [ 3539.683805] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3539.687280] ipmi_si():1 [ 3539.690060] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3539.695357] ipmi_si():1 [ 3539.697877] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3539.702566] [last unloaded: nvmet] [ 3539.705086] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3539.710122] [ 3539.713597] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3539.718721] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3539.720199] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3539.725323] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3539.733137] process_one_work+0x1ec/0x470 [ 3539.738521] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3539.747812] worker_thread+0x74/0x410 [ 3539.751807] pc : __sbq_wake_up+0x84/0x1f0 [ 3539.758754] kthread+0xec/0x100 [ 3539.762402] lr : sbitmap_queue_clear+0x74/0xc0 [ 3539.766397] ret_from_fork+0x10/0x20 [ 3539.769524] sp : ffff8000088438b0 [ 3539.780815] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3539.787937] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3539.795060] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3539.802181] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3539.809303] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3539.816425] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3539.823547] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3539.830669] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3539.837791] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3539.844912] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3539.852034] Call trace: [ 3539.854468] __sbq_wake_up+0x84/0x1f0 [ 3539.858117] sbitmap_queue_clear+0x74/0xc0 [ 3539.862200] blk_mq_put_tag+0x44/0x70 [ 3539.865849] __blk_mq_free_request+0x88/0xf0 [ 3539.870106] blk_mq_free_request+0xc0/0x160 [ 3539.874276] __blk_mq_end_request+0x78/0x150 [ 3539.878533] scsi_end_request+0x114/0x260 [ 3539.882531] scsi_io_completion+0x64/0x174 [ 3539.886614] scsi_finish_command+0xe0/0x120 [ 3539.890784] scsi_complete+0x8c/0x1c4 [ 3539.894433] blk_complete_reqs+0x5c/0x74 [ 3539.898343] blk_done_softirq+0x38/0x4c [ 3539.902165] __do_softirq+0x168/0x418 [ 3539.905814] __irq_exit_rcu+0x120/0x170 [ 3539.909637] irq_exit_rcu+0x18/0x24 [ 3539.913112] el1_interrupt+0x38/0x70 [ 3539.916674] el1h_64_irq_handler+0x18/0x24 [ 3539.920757] el1h_64_irq+0x68/0x6c [ 3539.924146] cpuidle_enter_state+0xc4/0x4e0 [ 3539.928315] cpuidle_enter+0x40/0x60 [ 3539.931878] cpuidle_idle_call+0x118/0x1b0 [ 3539.935961] do_idle+0xac/0x100 [ 3539.939090] cpu_startup_entry+0x30/0x34 [ 3539.942999] secondary_start_kernel+0xd8/0xf0 [ 3539.947343] __secondary_switched+0xb0/0xb4 [ 3550.281134] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3550.286866] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=154954 [ 3550.296074] (t=312540 jiffies g=36445 q=91653 ncpus=128) [ 3550.301459] Task dump for CPU 8: [ 3550.304674] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3550.314574] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3550.320742] Call trace: [ 3550.323176] __switch_to+0xd4/0x100 [ 3550.326652] 0xffff6c0200000080 [ 3550.329781] Task dump for CPU 68: [ 3550.333082] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3550.342982] Call trace: [ 3550.345415] dump_backtrace+0xe8/0x130 [ 3550.349151] show_stack+0x20/0x60 [ 3550.352453] sched_show_task+0x160/0x190 [ 3550.356362] dump_cpu_task+0x4c/0x78 [ 3550.359925] rcu_dump_cpu_stacks+0xc4/0x150 [ 3550.364096] print_cpu_stall+0x180/0x210 [ 3550.368005] check_cpu_stall+0x20c/0x2b4 [ 3550.371915] rcu_pending+0x4c/0x190 [ 3550.375391] rcu_sched_clock_irq+0x80/0x204 [ 3550.379561] update_process_times+0x7c/0xc0 [ 3550.383731] tick_sched_handle+0x38/0x74 [ 3550.387641] tick_sched_timer+0x60/0xc0 [ 3550.391465] __hrtimer_run_queues+0x170/0x300 [ 3550.395808] hrtimer_interrupt+0xfc/0x260 [ 3550.399804] arch_timer_handler_phys+0x34/0x50 [ 3550.404235] handle_percpu_devid_irq+0xac/0x240 [ 3550.408753] generic_handle_domain_irq+0x34/0x50 [ 3550.413357] gic_handle_irq+0x5c/0xe0 [ 3550.417006] call_on_irq_stack+0x2c/0x38 [ 3550.420915] do_interrupt_handler+0xdc/0xe0 [ 3550.425086] el1_interrupt+0x34/0x70 [ 3550.428649] el1h_64_irq_handler+0x18/0x24 [ 3550.432732] el1h_64_irq+0x68/0x6c [ 3550.436121] __sbq_wake_up+0x84/0x1f0 [ 3550.439770] sbitmap_queue_clear+0x74/0xc0 [ 3550.443853] blk_mq_put_tag+0x44/0x70 [ 3550.447503] __blk_mq_free_request+0x88/0xf0 [ 3550.451759] blk_mq_free_request+0xc0/0x160 [ 3550.455929] __blk_mq_end_request+0x78/0x150 [ 3550.460186] scsi_end_request+0x114/0x260 [ 3550.464182] scsi_io_completion+0x64/0x174 [ 3550.468265] scsi_finish_command+0xe0/0x120 [ 3550.472436] scsi_complete+0x8c/0x1c4 [ 3550.476085] blk_complete_reqs+0x5c/0x74 [ 3550.479996] blk_done_softirq+0x38/0x4c [ 3550.483818] __do_softirq+0x168/0x418 [ 3550.487467] __irq_exit_rcu+0x120/0x170 [ 3550.491290] irq_exit_rcu+0x18/0x24 [ 3550.494765] el1_interrupt+0x38/0x70 [ 3550.498327] el1h_64_irq_handler+0x18/0x24 [ 3550.502410] el1h_64_irq+0x68/0x6c [ 3550.505798] cpuidle_enter_state+0xc4/0x4e0 [ 3550.509968] cpuidle_enter+0x40/0x60 [ 3550.513531] cpuidle_idle_call+0x118/0x1b0 [ 3550.517614] do_idle+0xac/0x100 [ 3550.520743] cpu_startup_entry+0x30/0x34 [ 3550.524652] secondary_start_kernel+0xd8/0xf0 [ 3550.528996] __secondary_switched+0xb0/0xb4 [ 3563.091048] watchdog: BUG: soft lockup - CPU#8 stuck for 2924s! [kworker/8:12:19399] [ 3563.098778] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3563.098795] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3563.186583] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3563.194832] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3563.204123] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3563.210291] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3563.217238] pc : __sbq_wake_up+0x7c/0x1f0 [ 3563.221235] lr : sbitmap_queue_clear+0x74/0xc0 [ 3563.225666] sp : ffff80000d79b2d0 [ 3563.228967] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3563.236090] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3563.243212] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3563.250334] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3563.257456] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3563.264579] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3563.271701] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3563.278822] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3563.285944] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3563.293066] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3563.300188] Call trace: [ 3563.302621] __sbq_wake_up+0x7c/0x1f0 [ 3563.306271] blk_mq_put_tag+0x44/0x70 [ 3563.309920] __blk_mq_free_request+0x88/0xf0 [ 3563.314177] blk_mq_free_request+0xc0/0x160 [ 3563.318347] __blk_mq_end_request+0x78/0x150 [ 3563.322604] scsi_end_request+0x114/0x260 [ 3563.326600] scsi_io_completion+0x64/0x174 [ 3563.330683] scsi_finish_command+0xe0/0x120 [ 3563.334854] scsi_complete+0x8c/0x1c4 [ 3563.338503] blk_complete_reqs+0x5c/0x74 [ 3563.342412] blk_done_softirq+0x38/0x4c [ 3563.346235] __do_softirq+0x168/0x418 [ 3563.349883] __irq_exit_rcu+0x120/0x170 [ 3563.353706] irq_exit_rcu+0x18/0x24 [ 3563.357181] el1_interrupt+0x38/0x70 [ 3563.360744] el1h_64_irq_handler+0x18/0x24 [ 3563.364827] el1h_64_irq+0x68/0x6c [ 3563.368215] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3563.372993] __wake_up_common_lock+0x84/0xac [ 3563.377250] __wake_up+0x20/0x30 [ 3563.380466] __sbq_wake_up+0x13c/0x1f0 [ 3563.384202] sbitmap_queue_clear+0x74/0xc0 [ 3563.388285] blk_mq_put_tag+0x44/0x70 [ 3563.391935] __blk_mq_free_request+0x88/0xf0 [ 3563.396192] blk_mq_free_request+0xc0/0x160 [ 3563.400362] bfq_insert_request.isra.0+0x204/0x270 [ 3563.405140] bfq_insert_requests+0x70/0x90 [ 3563.409223] blk_mq_sched_insert_requests+0xac/0x250 [ 3563.414175] blk_mq_flush_plug_list+0x110/0x2cc [ 3563.418692] __blk_flush_plug+0x100/0x140 [ 3563.422688] blk_finish_plug+0x40/0x25c [ 3563.426511] __iomap_dio_rw+0x1b4/0x500 [ 3563.430334] iomap_dio_rw+0x18/0x50 [ 3563.433809] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3563.439160] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3563.443902] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3563.448946] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3563.454076] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3563.459206] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3563.464592] process_one_work+0x1ec/0x470 [ 3563.468588] worker_thread+0x74/0x410 [ 3563.472237] kthread+0xec/0x100 [ 3563.475366] ret_from_fork+0x10/0x20 [ 3575.200968] watchdog: BUG: soft lockup - CPU#68 stuck for 2935s! [swapper/68:0] [ 3575.208263] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3575.208280] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3575.296069] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3575.303884] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3575.313176] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3575.320125] pc : __sbq_wake_up+0x84/0x1f0 [ 3575.324121] lr : sbitmap_queue_clear+0x74/0xc0 [ 3575.328552] sp : ffff8000088438b0 [ 3575.331853] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3575.338975] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3575.346097] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3575.353218] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3575.360340] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3575.367462] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3575.374584] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3575.381707] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3575.388829] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3575.395951] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3575.403073] Call trace: [ 3575.405506] __sbq_wake_up+0x84/0x1f0 [ 3575.409155] sbitmap_queue_clear+0x74/0xc0 [ 3575.413239] blk_mq_put_tag+0x44/0x70 [ 3575.416888] __blk_mq_free_request+0x88/0xf0 [ 3575.421145] blk_mq_free_request+0xc0/0x160 [ 3575.425315] __blk_mq_end_request+0x78/0x150 [ 3575.429572] scsi_end_request+0x114/0x260 [ 3575.433568] scsi_io_completion+0x64/0x174 [ 3575.437651] scsi_finish_command+0xe0/0x120 [ 3575.441821] scsi_complete+0x8c/0x1c4 [ 3575.445471] blk_complete_reqs+0x5c/0x74 [ 3575.449380] blk_done_softirq+0x38/0x4c [ 3575.453203] __do_softirq+0x168/0x418 [ 3575.456851] __irq_exit_rcu+0x120/0x170 [ 3575.460674] irq_exit_rcu+0x18/0x24 [ 3575.464149] el1_interrupt+0x38/0x70 [ 3575.467713] el1h_64_irq_handler+0x18/0x24 [ 3575.471796] el1h_64_irq+0x68/0x6c [ 3575.475185] cpuidle_enter_state+0xc4/0x4e0 [ 3575.479355] cpuidle_enter+0x40/0x60 [ 3575.482917] cpuidle_idle_call+0x118/0x1b0 [ 3575.487000] do_idle+0xac/0x100 [ 3575.490129] cpu_startup_entry+0x30/0x34 [ 3575.494038] secondary_start_kernel+0xd8/0xf0 [ 3575.498382] __secondary_switched+0xb0/0xb4 [ 3587.090889] watchdog: BUG: soft lockup - CPU#8 stuck for 2946s! [kworker/8:12:19399] [ 3587.098618] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3587.098635] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3587.186423] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3587.194673] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3587.203964] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3587.210132] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3587.217079] pc : __sbq_wake_up+0x84/0x1f0 [ 3587.221076] lr : sbitmap_queue_clear+0x74/0xc0 [ 3587.225507] sp : ffff80000d79b280 [ 3587.228808] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3587.235931] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3587.243053] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3587.250175] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3587.257297] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3587.264419] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3587.271541] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3587.278663] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3587.285785] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3587.292907] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3587.300029] Call trace: [ 3587.302461] __sbq_wake_up+0x84/0x1f0 [ 3587.306110] sbitmap_queue_clear+0x74/0xc0 [ 3587.310194] blk_mq_put_tag+0x44/0x70 [ 3587.313843] __blk_mq_free_request+0x88/0xf0 [ 3587.318100] blk_mq_free_request+0xc0/0x160 [ 3587.322270] __blk_mq_end_request+0x78/0x150 [ 3587.326526] scsi_end_request+0x114/0x260 [ 3587.330523] scsi_io_completion+0x64/0x174 [ 3587.334606] scsi_finish_command+0xe0/0x120 [ 3587.338776] scsi_complete+0x8c/0x1c4 [ 3587.342425] blk_complete_reqs+0x5c/0x74 [ 3587.346335] blk_done_softirq+0x38/0x4c [ 3587.350158] __do_softirq+0x168/0x418 [ 3587.353807] __irq_exit_rcu+0x120/0x170 [ 3587.357629] irq_exit_rcu+0x18/0x24 [ 3587.361104] el1_interrupt+0x38/0x70 [ 3587.364667] el1h_64_irq_handler+0x18/0x24 [ 3587.368750] el1h_64_irq+0x68/0x6c [ 3587.372138] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3587.376916] __wake_up_common_lock+0x84/0xac [ 3587.381173] __wake_up+0x20/0x30 [ 3587.384389] __sbq_wake_up+0x13c/0x1f0 [ 3587.388125] sbitmap_queue_clear+0x74/0xc0 [ 3587.392208] blk_mq_put_tag+0x44/0x70 [ 3587.395858] __blk_mq_free_request+0x88/0xf0 [ 3587.400115] blk_mq_free_request+0xc0/0x160 [ 3587.404284] bfq_insert_request.isra.0+0x204/0x270 [ 3587.409063] bfq_insert_requests+0x70/0x90 [ 3587.413146] blk_mq_sched_insert_requests+0xac/0x250 [ 3587.418098] blk_mq_flush_plug_list+0x110/0x2cc [ 3587.422616] __blk_flush_plug+0x100/0x140 [ 3587.426612] blk_finish_plug+0x40/0x25c [ 3587.430434] __iomap_dio_rw+0x1b4/0x500 [ 3587.434258] iomap_dio_rw+0x18/0x50 [ 3587.437734] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3587.443085] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3587.447827] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3587.452871] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3587.458001] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3587.463131] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3587.468517] process_one_work+0x1ec/0x470 [ 3587.472513] worker_thread+0x74/0x410 [ 3587.476161] kthread+0xec/0x100 [ 3587.479290] ret_from_fork+0x10/0x20 [ 3599.200808] watchdog: BUG: soft lockup - CPU#68 stuck for 2958s! [swapper/68:0] [ 3599.208103] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3599.208120] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3599.295908] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3599.303724] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3599.313015] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3599.319963] pc : __sbq_wake_up+0x58/0x1f0 [ 3599.323959] lr : sbitmap_queue_clear+0x74/0xc0 [ 3599.328390] sp : ffff8000088438b0 [ 3599.331692] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3599.338814] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3599.345936] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3599.353058] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3599.360180] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3599.367302] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3599.374424] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3599.381546] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3599.388668] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3599.395790] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3599.402911] Call trace: [ 3599.405345] __sbq_wake_up+0x58/0x1f0 [ 3599.408994] sbitmap_queue_clear+0x74/0xc0 [ 3599.413077] blk_mq_put_tag+0x44/0x70 [ 3599.416727] __blk_mq_free_request+0x88/0xf0 [ 3599.420983] blk_mq_free_request+0xc0/0x160 [ 3599.425153] __blk_mq_end_request+0x78/0x150 [ 3599.429410] scsi_end_request+0x114/0x260 [ 3599.433406] scsi_io_completion+0x64/0x174 [ 3599.437489] scsi_finish_command+0xe0/0x120 [ 3599.441660] scsi_complete+0x8c/0x1c4 [ 3599.445309] blk_complete_reqs+0x5c/0x74 [ 3599.449218] blk_done_softirq+0x38/0x4c [ 3599.453041] __do_softirq+0x168/0x418 [ 3599.456690] __irq_exit_rcu+0x120/0x170 [ 3599.460513] irq_exit_rcu+0x18/0x24 [ 3599.463989] el1_interrupt+0x38/0x70 [ 3599.467551] el1h_64_irq_handler+0x18/0x24 [ 3599.471634] el1h_64_irq+0x68/0x6c [ 3599.475023] cpuidle_enter_state+0xc4/0x4e0 [ 3599.479193] cpuidle_enter+0x40/0x60 [ 3599.482755] cpuidle_idle_call+0x118/0x1b0 [ 3599.486839] do_idle+0xac/0x100 [ 3599.489968] cpu_startup_entry+0x30/0x34 [ 3599.493877] secondary_start_kernel+0xd8/0xf0 [ 3599.498221] __secondary_switched+0xb0/0xb4 [ 3600.350804] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 314941 jiffies s: 2093 root: 0x11/. [ 3600.362533] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3600.371741] Task dump for CPU 8: [ 3600.374957] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3600.384859] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3600.391028] Call trace: [ 3600.393461] __switch_to+0xd4/0x100 [ 3600.396938] 0xffff6c0200000080 [ 3600.400067] Task dump for CPU 68: [ 3600.403370] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3600.413271] Call trace: [ 3600.415704] __switch_to+0xd4/0x100 [ 3600.419180] 0x0 [ 3611.090729] watchdog: BUG: soft lockup - CPU#8 stuck for 2969s! [kworker/8:12:19399] [ 3611.098459] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3611.098476] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3611.186264] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3611.194514] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3611.203806] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3611.209973] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3611.216921] pc : __sbq_wake_up+0x58/0x1f0 [ 3611.220917] lr : sbitmap_queue_clear+0x74/0xc0 [ 3611.225348] sp : ffff80000d79b280 [ 3611.228650] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3611.235772] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3611.242894] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3611.250016] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3611.257138] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3611.264260] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3611.271382] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3611.278504] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3611.285626] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3611.292748] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3611.299870] Call trace: [ 3611.302303] __sbq_wake_up+0x58/0x1f0 [ 3611.305952] sbitmap_queue_clear+0x74/0xc0 [ 3611.310036] blk_mq_put_tag+0x44/0x70 [ 3611.313687] __blk_mq_free_request+0x88/0xf0 [ 3611.317943] blk_mq_free_request+0xc0/0x160 [ 3611.322113] __blk_mq_end_request+0x78/0x150 [ 3611.326370] scsi_end_request+0x114/0x260 [ 3611.330366] scsi_io_completion+0x64/0x174 [ 3611.334449] scsi_finish_command+0xe0/0x120 [ 3611.338620] scsi_complete+0x8c/0x1c4 [ 3611.342269] blk_complete_reqs+0x5c/0x74 [ 3611.346180] blk_done_softirq+0x38/0x4c [ 3611.350003] __do_softirq+0x168/0x418 [ 3611.353652] __irq_exit_rcu+0x120/0x170 [ 3611.357475] irq_exit_rcu+0x18/0x24 [ 3611.360950] el1_interrupt+0x38/0x70 [ 3611.364512] el1h_64_irq_handler+0x18/0x24 [ 3611.368595] el1h_64_irq+0x68/0x6c [ 3611.371984] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3611.376761] __wake_up_common_lock+0x84/0xac [ 3611.381019] __wake_up+0x20/0x30 [ 3611.384234] __sbq_wake_up+0x13c/0x1f0 [ 3611.387970] sbitmap_queue_clear+0x74/0xc0 [ 3611.392053] blk_mq_put_tag+0x44/0x70 [ 3611.395703] __blk_mq_free_request+0x88/0xf0 [ 3611.399960] blk_mq_free_request+0xc0/0x160 [ 3611.404130] bfq_insert_request.isra.0+0x204/0x270 [ 3611.408908] bfq_insert_requests+0x70/0x90 [ 3611.412991] blk_mq_sched_insert_requests+0xac/0x250 [ 3611.417943] blk_mq_flush_plug_list+0x110/0x2cc [ 3611.422461] __blk_flush_plug+0x100/0x140 [ 3611.426458] blk_finish_plug+0x40/0x25c [ 3611.430280] __iomap_dio_rw+0x1b4/0x500 [ 3611.434103] iomap_dio_rw+0x18/0x50 [ 3611.437578] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3611.442929] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3611.447672] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3611.452716] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3611.457846] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3611.462975] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3611.468362] process_one_work+0x1ec/0x470 [ 3611.472358] worker_thread+0x74/0x410 [ 3611.476007] kthread+0xec/0x100 [ 3611.479136] ret_from_fork+0x10/0x20 [ 3623.200649] watchdog: BUG: soft lockup - CPU#68 stuck for 2980s! [swapper/68:0] [ 3623.207944] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3623.207961] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3623.295749] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3623.303565] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3623.312856] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3623.319804] pc : __sbq_wake_up+0x58/0x1f0 [ 3623.323801] lr : sbitmap_queue_clear+0x74/0xc0 [ 3623.328231] sp : ffff8000088438b0 [ 3623.331532] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3623.338654] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3623.345776] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3623.352898] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3623.360020] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3623.367142] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3623.374264] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3623.381386] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3623.388508] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3623.395629] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3623.402751] Call trace: [ 3623.405184] __sbq_wake_up+0x58/0x1f0 [ 3623.408834] sbitmap_queue_clear+0x74/0xc0 [ 3623.412917] blk_mq_put_tag+0x44/0x70 [ 3623.416567] __blk_mq_free_request+0x88/0xf0 [ 3623.420823] blk_mq_free_request+0xc0/0x160 [ 3623.424994] __blk_mq_end_request+0x78/0x150 [ 3623.429251] scsi_end_request+0x114/0x260 [ 3623.433247] scsi_io_completion+0x64/0x174 [ 3623.437330] scsi_finish_command+0xe0/0x120 [ 3623.441500] scsi_complete+0x8c/0x1c4 [ 3623.445149] blk_complete_reqs+0x5c/0x74 [ 3623.449059] blk_done_softirq+0x38/0x4c [ 3623.452882] __do_softirq+0x168/0x418 [ 3623.456530] __irq_exit_rcu+0x120/0x170 [ 3623.460353] irq_exit_rcu+0x18/0x24 [ 3623.463828] el1_interrupt+0x38/0x70 [ 3623.467391] el1h_64_irq_handler+0x18/0x24 [ 3623.471474] el1h_64_irq+0x68/0x6c [ 3623.474862] cpuidle_enter_state+0xc4/0x4e0 [ 3623.479032] cpuidle_enter+0x40/0x60 [ 3623.482595] cpuidle_idle_call+0x118/0x1b0 [ 3623.486678] do_idle+0xac/0x100 [ 3623.489807] cpu_startup_entry+0x30/0x34 [ 3623.493716] secondary_start_kernel+0xd8/0xf0 [ 3623.498060] __secondary_switched+0xb0/0xb4 [ 3635.090570] watchdog: BUG: soft lockup - CPU#8 stuck for 2991s! [kworker/8:12:19399] [ 3635.098299] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3635.098316] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3635.186104] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3635.194354] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3635.203645] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3635.209812] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3635.216760] pc : __sbq_wake_up+0x40/0x1f0 [ 3635.220757] lr : sbitmap_queue_clear+0x74/0xc0 [ 3635.225187] sp : ffff80000d79b280 [ 3635.228489] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3635.235611] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3635.242733] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3635.249855] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3635.256978] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3635.264100] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3635.271221] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3635.278343] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3635.285465] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3635.292587] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 3635.299709] Call trace: [ 3635.302141] __sbq_wake_up+0x40/0x1f0 [ 3635.305791] sbitmap_queue_clear+0x74/0xc0 [ 3635.309875] blk_mq_put_tag+0x44/0x70 [ 3635.313524] __blk_mq_free_request+0x88/0xf0 [ 3635.317781] blk_mq_free_request+0xc0/0x160 [ 3635.321951] __blk_mq_end_request+0x78/0x150 [ 3635.326208] scsi_end_request+0x114/0x260 [ 3635.330204] scsi_io_completion+0x64/0x174 [ 3635.334288] scsi_finish_command+0xe0/0x120 [ 3635.338458] scsi_complete+0x8c/0x1c4 [ 3635.342107] blk_complete_reqs+0x5c/0x74 [ 3635.346017] blk_done_softirq+0x38/0x4c [ 3635.349839] __do_softirq+0x168/0x418 [ 3635.353489] __irq_exit_rcu+0x120/0x170 [ 3635.357312] irq_exit_rcu+0x18/0x24 [ 3635.360788] el1_interrupt+0x38/0x70 [ 3635.364350] el1h_64_irq_handler+0x18/0x24 [ 3635.368433] el1h_64_irq+0x68/0x6c [ 3635.371822] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3635.376600] __wake_up_common_lock+0x84/0xac [ 3635.380857] __wake_up+0x20/0x30 [ 3635.384072] __sbq_wake_up+0x13c/0x1f0 [ 3635.387808] sbitmap_queue_clear+0x74/0xc0 [ 3635.391892] blk_mq_put_tag+0x44/0x70 [ 3635.395542] __blk_mq_free_request+0x88/0xf0 [ 3635.399798] blk_mq_free_request+0xc0/0x160 [ 3635.403968] bfq_insert_request.isra.0+0x204/0x270 [ 3635.408746] bfq_insert_requests+0x70/0x90 [ 3635.412830] blk_mq_sched_insert_requests+0xac/0x250 [ 3635.417782] blk_mq_flush_plug_list+0x110/0x2cc [ 3635.422300] __blk_flush_plug+0x100/0x140 [ 3635.426296] blk_finish_plug+0x40/0x25c [ 3635.430119] __iomap_dio_rw+0x1b4/0x500 [ 3635.433942] iomap_dio_rw+0x18/0x50 [ 3635.437417] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3635.442768] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3635.447511] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3635.452555] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3635.457685] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3635.462815] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3635.468201] process_one_work+0x1ec/0x470 [ 3635.472198] worker_thread+0x74/0x410 [ 3635.475846] kthread+0xec/0x100 [ 3635.478975] ret_from_fork+0x10/0x20 [ 3647.200489] watchdog: BUG: soft lockup - CPU#68 stuck for 3002s! [swapper/68:0] [ 3647.207786] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3647.207803] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3647.295590] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3647.303408] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3647.312699] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3647.319646] pc : __sbq_wake_up+0x40/0x1f0 [ 3647.323643] lr : sbitmap_queue_clear+0x74/0xc0 [ 3647.328073] sp : ffff8000088438b0 [ 3647.331374] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3647.338496] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3647.345618] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3647.352740] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3647.359862] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3647.366983] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3647.374105] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3647.381227] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3647.388349] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3647.395471] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 3647.402592] Call trace: [ 3647.405025] __sbq_wake_up+0x40/0x1f0 [ 3647.408674] sbitmap_queue_clear+0x74/0xc0 [ 3647.412758] blk_mq_put_tag+0x44/0x70 [ 3647.416408] __blk_mq_free_request+0x88/0xf0 [ 3647.420664] blk_mq_free_request+0xc0/0x160 [ 3647.424834] __blk_mq_end_request+0x78/0x150 [ 3647.429092] scsi_end_request+0x114/0x260 [ 3647.433089] scsi_io_completion+0x64/0x174 [ 3647.437172] scsi_finish_command+0xe0/0x120 [ 3647.441342] scsi_complete+0x8c/0x1c4 [ 3647.444991] blk_complete_reqs+0x5c/0x74 [ 3647.448901] blk_done_softirq+0x38/0x4c [ 3647.452724] __do_softirq+0x168/0x418 [ 3647.456372] __irq_exit_rcu+0x120/0x170 [ 3647.460195] irq_exit_rcu+0x18/0x24 [ 3647.463670] el1_interrupt+0x38/0x70 [ 3647.467233] el1h_64_irq_handler+0x18/0x24 [ 3647.471316] el1h_64_irq+0x68/0x6c [ 3647.474704] cpuidle_enter_state+0xc4/0x4e0 [ 3647.478874] cpuidle_enter+0x40/0x60 [ 3647.482436] cpuidle_idle_call+0x118/0x1b0 [ 3647.486520] do_idle+0xac/0x100 [ 3647.489648] cpu_startup_entry+0x30/0x34 [ 3647.493557] secondary_start_kernel+0xd8/0xf0 [ 3647.497901] __secondary_switched+0xb0/0xb4 [ 3659.090410] watchdog: BUG: soft lockup - CPU#8 stuck for 3013s! [kworker/8:12:19399] [ 3659.098140] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3659.098157] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3659.185944] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3659.194195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3659.203486] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3659.209653] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3659.216601] pc : __sbq_wake_up+0x40/0x1f0 [ 3659.220597] lr : sbitmap_queue_clear+0x74/0xc0 [ 3659.225028] sp : ffff80000d79b280 [ 3659.228330] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3659.235452] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3659.242574] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3659.249697] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3659.256819] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3659.263941] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3659.271063] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3659.278185] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3659.285306] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3659.292428] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 3659.299550] Call trace: [ 3659.301983] __sbq_wake_up+0x40/0x1f0 [ 3659.305632] sbitmap_queue_clear+0x74/0xc0 [ 3659.309716] blk_mq_put_tag+0x44/0x70 [ 3659.313366] __blk_mq_free_request+0x88/0xf0 [ 3659.317622] blk_mq_free_request+0xc0/0x160 [ 3659.321792] __blk_mq_end_request+0x78/0x150 [ 3659.326049] scsi_end_request+0x114/0x260 [ 3659.330045] scsi_io_completion+0x64/0x174 [ 3659.334128] scsi_finish_command+0xe0/0x120 [ 3659.338299] scsi_complete+0x8c/0x1c4 [ 3659.341948] blk_complete_reqs+0x5c/0x74 [ 3659.345858] blk_done_softirq+0x38/0x4c [ 3659.349681] __do_softirq+0x168/0x418 [ 3659.353330] __irq_exit_rcu+0x120/0x170 [ 3659.357152] irq_exit_rcu+0x18/0x24 [ 3659.360628] el1_interrupt+0x38/0x70 [ 3659.364190] el1h_64_irq_handler+0x18/0x24 [ 3659.368273] el1h_64_irq+0x68/0x6c [ 3659.371662] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3659.376440] __wake_up_common_lock+0x84/0xac [ 3659.380697] __wake_up+0x20/0x30 [ 3659.383913] __sbq_wake_up+0x13c/0x1f0 [ 3659.387649] sbitmap_queue_clear+0x74/0xc0 [ 3659.391732] blk_mq_put_tag+0x44/0x70 [ 3659.395381] __blk_mq_free_request+0x88/0xf0 [ 3659.399638] blk_mq_free_request+0xc0/0x160 [ 3659.403810] bfq_insert_request.isra.0+0x204/0x270 [ 3659.408588] bfq_insert_requests+0x70/0x90 [ 3659.412671] blk_mq_sched_insert_requests+0xac/0x250 [ 3659.417623] blk_mq_flush_plug_list+0x110/0x2cc [ 3659.422141] __blk_flush_plug+0x100/0x140 [ 3659.426137] blk_finish_plug+0x40/0x25c [ 3659.429960] __iomap_dio_rw+0x1b4/0x500 [ 3659.433782] iomap_dio_rw+0x18/0x50 [ 3659.437258] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3659.442609] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3659.447351] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3659.452394] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3659.457524] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3659.462654] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3659.468041] process_one_work+0x1ec/0x470 [ 3659.472037] worker_thread+0x74/0x410 [ 3659.475686] kthread+0xec/0x100 [ 3659.478815] ret_from_fork+0x10/0x20 [ 3666.910368] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 321597 jiffies s: 2093 root: 0x11/. [ 3666.922107] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3666.931316] Task dump for CPU 8: [ 3666.934530] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3666.944432] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3666.950602] Call trace: [ 3666.953035] __switch_to+0xd4/0x100 [ 3666.956512] 0xffff6c0200000080 [ 3666.959642] Task dump for CPU 68: [ 3666.962945] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3666.972846] Call trace: [ 3666.975279] __switch_to+0xd4/0x100 [ 3666.978755] 0x0 [ 3671.200330] watchdog: BUG: soft lockup - CPU#68 stuck for 3025s! [swapper/68:0] [ 3671.207625] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3671.207642] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3671.295430] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3671.303245] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3671.312536] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3671.319484] pc : __sbq_wake_up+0x40/0x1f0 [ 3671.323480] lr : sbitmap_queue_clear+0x74/0xc0 [ 3671.327911] sp : ffff8000088438b0 [ 3671.331212] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3671.338334] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3671.345456] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3671.352578] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3671.359700] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3671.366822] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3671.373943] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3671.381066] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3671.388188] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3671.395309] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 3671.402431] Call trace: [ 3671.404864] __sbq_wake_up+0x40/0x1f0 [ 3671.408514] sbitmap_queue_clear+0x74/0xc0 [ 3671.412597] blk_mq_put_tag+0x44/0x70 [ 3671.416247] __blk_mq_free_request+0x88/0xf0 [ 3671.420504] blk_mq_free_request+0xc0/0x160 [ 3671.424673] __blk_mq_end_request+0x78/0x150 [ 3671.428930] scsi_end_request+0x114/0x260 [ 3671.432927] scsi_io_completion+0x64/0x174 [ 3671.437010] scsi_finish_command+0xe0/0x120 [ 3671.441180] scsi_complete+0x8c/0x1c4 [ 3671.444829] blk_complete_reqs+0x5c/0x74 [ 3671.448739] blk_done_softirq+0x38/0x4c [ 3671.452561] __do_softirq+0x168/0x418 [ 3671.456210] __irq_exit_rcu+0x120/0x170 [ 3671.460033] irq_exit_rcu+0x18/0x24 [ 3671.463509] el1_interrupt+0x38/0x70 [ 3671.467071] el1h_64_irq_handler+0x18/0x24 [ 3671.471154] el1h_64_irq+0x68/0x6c [ 3671.474544] cpuidle_enter_state+0xc4/0x4e0 [ 3671.478714] cpuidle_enter+0x40/0x60 [ 3671.482276] cpuidle_idle_call+0x118/0x1b0 [ 3671.486360] do_idle+0xac/0x100 [ 3671.489489] cpu_startup_entry+0x30/0x34 [ 3671.493400] secondary_start_kernel+0xd8/0xf0 [ 3671.497746] __secondary_switched+0xb0/0xb4 [ 3683.090251] watchdog: BUG: soft lockup - CPU#8 stuck for 3036s! [kworker/8:12:19399] [ 3683.097980] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3683.097998] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3683.185786] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3683.194036] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3683.203327] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3683.209494] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3683.216442] pc : __sbq_wake_up+0x58/0x1f0 [ 3683.220439] lr : sbitmap_queue_clear+0x74/0xc0 [ 3683.224869] sp : ffff80000d79b280 [ 3683.228171] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3683.235293] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3683.242415] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3683.249537] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3683.256660] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3683.263782] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3683.270904] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3683.278025] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3683.285148] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3683.292271] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3683.299393] Call trace: [ 3683.301826] __sbq_wake_up+0x58/0x1f0 [ 3683.305476] sbitmap_queue_clear+0x74/0xc0 [ 3683.309559] blk_mq_put_tag+0x44/0x70 [ 3683.313209] __blk_mq_free_request+0x88/0xf0 [ 3683.317465] blk_mq_free_request+0xc0/0x160 [ 3683.321635] __blk_mq_end_request+0x78/0x150 [ 3683.325893] scsi_end_request+0x114/0x260 [ 3683.329889] scsi_io_completion+0x64/0x174 [ 3683.333972] scsi_finish_command+0xe0/0x120 [ 3683.338142] scsi_complete+0x8c/0x1c4 [ 3683.341792] blk_complete_reqs+0x5c/0x74 [ 3683.345701] blk_done_softirq+0x38/0x4c [ 3683.349524] __do_softirq+0x168/0x418 [ 3683.353173] __irq_exit_rcu+0x120/0x170 [ 3683.356995] irq_exit_rcu+0x18/0x24 [ 3683.360471] el1_interrupt+0x38/0x70 [ 3683.364034] el1h_64_irq_handler+0x18/0x24 [ 3683.368117] el1h_64_irq+0x68/0x6c [ 3683.371506] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3683.376284] __wake_up_common_lock+0x84/0xac [ 3683.380541] __wake_up+0x20/0x30 [ 3683.383756] __sbq_wake_up+0x13c/0x1f0 [ 3683.387493] sbitmap_queue_clear+0x74/0xc0 [ 3683.391577] blk_mq_put_tag+0x44/0x70 [ 3683.395227] __blk_mq_free_request+0x88/0xf0 [ 3683.399483] blk_mq_free_request+0xc0/0x160 [ 3683.403653] bfq_insert_request.isra.0+0x204/0x270 [ 3683.408431] bfq_insert_requests+0x70/0x90 [ 3683.412515] blk_mq_sched_insert_requests+0xac/0x250 [ 3683.417467] blk_mq_flush_plug_list+0x110/0x2cc [ 3683.421985] __blk_flush_plug+0x100/0x140 [ 3683.425981] blk_finish_plug+0x40/0x25c [ 3683.429804] __iomap_dio_rw+0x1b4/0x500 [ 3683.433627] iomap_dio_rw+0x18/0x50 [ 3683.437102] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3683.442453] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3683.447195] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3683.452239] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3683.457369] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3683.462499] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3683.467885] process_one_work+0x1ec/0x470 [ 3683.471882] worker_thread+0x74/0x410 [ 3683.475531] kthread+0xec/0x100 [ 3683.478659] ret_from_fork+0x10/0x20 [ 3695.200171] watchdog: BUG: soft lockup - CPU#68 stuck for 3047s! [swapper/68:0] [ 3695.207466] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3695.207483] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3695.295271] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3695.303086] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3695.312377] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3695.319326] pc : __sbq_wake_up+0x58/0x1f0 [ 3695.323323] lr : sbitmap_queue_clear+0x74/0xc0 [ 3695.327753] sp : ffff8000088438b0 [ 3695.331054] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3695.338176] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3695.345298] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3695.352420] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3695.359542] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3695.366664] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3695.373785] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3695.380907] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3695.388029] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3695.395151] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3695.402273] Call trace: [ 3695.404706] __sbq_wake_up+0x58/0x1f0 [ 3695.408355] sbitmap_queue_clear+0x74/0xc0 [ 3695.412439] blk_mq_put_tag+0x44/0x70 [ 3695.416088] __blk_mq_free_request+0x88/0xf0 [ 3695.420345] blk_mq_free_request+0xc0/0x160 [ 3695.424515] __blk_mq_end_request+0x78/0x150 [ 3695.428772] scsi_end_request+0x114/0x260 [ 3695.432768] scsi_io_completion+0x64/0x174 [ 3695.436852] scsi_finish_command+0xe0/0x120 [ 3695.441022] scsi_complete+0x8c/0x1c4 [ 3695.444671] blk_complete_reqs+0x5c/0x74 [ 3695.448581] blk_done_softirq+0x38/0x4c [ 3695.452403] __do_softirq+0x168/0x418 [ 3695.456052] __irq_exit_rcu+0x120/0x170 [ 3695.459875] irq_exit_rcu+0x18/0x24 [ 3695.463350] el1_interrupt+0x38/0x70 [ 3695.466913] el1h_64_irq_handler+0x18/0x24 [ 3695.470996] el1h_64_irq+0x68/0x6c [ 3695.474384] cpuidle_enter_state+0xc4/0x4e0 [ 3695.478554] cpuidle_enter+0x40/0x60 [ 3695.482117] cpuidle_idle_call+0x118/0x1b0 [ 3695.486200] do_idle+0xac/0x100 [ 3695.489329] cpu_startup_entry+0x30/0x34 [ 3695.493238] secondary_start_kernel+0xd8/0xf0 [ 3695.497582] __secondary_switched+0xb0/0xb4 [ 3707.090092] watchdog: BUG: soft lockup - CPU#8 stuck for 3058s! [kworker/8:12:19399] [ 3707.097821] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3707.097838] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3707.185627] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3707.193876] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3707.203167] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3707.209335] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3707.216282] pc : __sbq_wake_up+0x40/0x1f0 [ 3707.220279] lr : sbitmap_queue_clear+0x74/0xc0 [ 3707.224710] sp : ffff80000d79b280 [ 3707.228011] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3707.235134] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3707.242256] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3707.249378] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3707.256501] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3707.263623] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3707.270745] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3707.277866] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3707.284988] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3707.292110] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000038 [ 3707.299232] Call trace: [ 3707.301665] __sbq_wake_up+0x40/0x1f0 [ 3707.305314] sbitmap_queue_clear+0x74/0xc0 [ 3707.309397] blk_mq_put_tag+0x44/0x70 [ 3707.313046] __blk_mq_free_request+0x88/0xf0 [ 3707.317303] blk_mq_free_request+0xc0/0x160 [ 3707.321473] __blk_mq_end_request+0x78/0x150 [ 3707.325730] scsi_end_request+0x114/0x260 [ 3707.329727] scsi_io_completion+0x64/0x174 [ 3707.333810] scsi_finish_command+0xe0/0x120 [ 3707.337980] scsi_complete+0x8c/0x1c4 [ 3707.341629] blk_complete_reqs+0x5c/0x74 [ 3707.345539] blk_done_softirq+0x38/0x4c [ 3707.349361] __do_softirq+0x168/0x418 [ 3707.353010] __irq_exit_rcu+0x120/0x170 [ 3707.356833] irq_exit_rcu+0x18/0x24 [ 3707.360309] el1_interrupt+0x38/0x70 [ 3707.363871] el1h_64_irq_handler+0x18/0x24 [ 3707.367954] el1h_64_irq+0x68/0x6c [ 3707.371343] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3707.376120] __wake_up_common_lock+0x84/0xac [ 3707.380377] __wake_up+0x20/0x30 [ 3707.383593] __sbq_wake_up+0x13c/0x1f0 [ 3707.387329] sbitmap_queue_clear+0x74/0xc0 [ 3707.391412] blk_mq_put_tag+0x44/0x70 [ 3707.395062] __blk_mq_free_request+0x88/0xf0 [ 3707.399318] blk_mq_free_request+0xc0/0x160 [ 3707.403488] bfq_insert_request.isra.0+0x204/0x270 [ 3707.408266] bfq_insert_requests+0x70/0x90 [ 3707.412350] blk_mq_sched_insert_requests+0xac/0x250 [ 3707.417302] blk_mq_flush_plug_list+0x110/0x2cc [ 3707.421819] __blk_flush_plug+0x100/0x140 [ 3707.425816] blk_finish_plug+0x40/0x25c [ 3707.429638] __iomap_dio_rw+0x1b4/0x500 [ 3707.433462] iomap_dio_rw+0x18/0x50 [ 3707.436937] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3707.442288] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3707.447031] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3707.452075] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3707.457204] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3707.462334] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3707.467720] process_one_work+0x1ec/0x470 [ 3707.471717] worker_thread+0x74/0x410 [ 3707.475366] kthread+0xec/0x100 [ 3707.478494] ret_from_fork+0x10/0x20 [ 3719.200011] watchdog: BUG: soft lockup - CPU#68 stuck for 3069s! [swapper/68:0] [ 3719.207307] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3719.207324] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3719.295111] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3719.302927] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3719.312218] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3719.319165] pc : __sbq_wake_up+0x38/0x1f0 [ 3719.323162] lr : sbitmap_queue_clear+0x74/0xc0 [ 3719.327592] sp : ffff8000088438b0 [ 3719.330894] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3719.338016] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3719.345138] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3719.352260] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3719.359382] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3719.366503] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3719.373625] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3719.380747] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3719.387869] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3719.394991] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000038 [ 3719.402113] Call trace: [ 3719.404545] __sbq_wake_up+0x38/0x1f0 [ 3719.408194] sbitmap_queue_clear+0x74/0xc0 [ 3719.412277] blk_mq_put_tag+0x44/0x70 [ 3719.415927] __blk_mq_free_request+0x88/0xf0 [ 3719.420184] blk_mq_free_request+0xc0/0x160 [ 3719.424354] __blk_mq_end_request+0x78/0x150 [ 3719.428610] scsi_end_request+0x114/0x260 [ 3719.432607] scsi_io_completion+0x64/0x174 [ 3719.436690] scsi_finish_command+0xe0/0x120 [ 3719.440860] scsi_complete+0x8c/0x1c4 [ 3719.444509] blk_complete_reqs+0x5c/0x74 [ 3719.448419] blk_done_softirq+0x38/0x4c [ 3719.452241] __do_softirq+0x168/0x418 [ 3719.455890] __irq_exit_rcu+0x120/0x170 [ 3719.459713] irq_exit_rcu+0x18/0x24 [ 3719.463188] el1_interrupt+0x38/0x70 [ 3719.466750] el1h_64_irq_handler+0x18/0x24 [ 3719.470833] el1h_64_irq+0x68/0x6c [ 3719.474222] cpuidle_enter_state+0xc4/0x4e0 [ 3719.478392] cpuidle_enter+0x40/0x60 [ 3719.481954] cpuidle_idle_call+0x118/0x1b0 [ 3719.486038] do_idle+0xac/0x100 [ 3719.489167] cpu_startup_entry+0x30/0x34 [ 3719.493076] secondary_start_kernel+0xd8/0xf0 [ 3719.497420] __secondary_switched+0xb0/0xb4 [ 3730.519936] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3730.525669] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=163913 [ 3730.534791] (t=330565 jiffies g=36445 q=97168 ncpus=128) [ 3730.540176] Task dump for CPU 8: [ 3730.543391] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3730.553291] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3730.559458] Call trace: [ 3730.561891] dump_backtrace+0xe8/0x130 [ 3730.565627] show_stack+0x20/0x60 [ 3730.568929] sched_show_task+0x160/0x190 [ 3730.572839] dump_cpu_task+0x4c/0x78 [ 3730.576402] rcu_dump_cpu_stacks+0xc4/0x150 [ 3730.580573] print_cpu_stall+0x180/0x210 [ 3730.584483] check_cpu_stall+0x20c/0x2b4 [ 3730.588392] rcu_pending+0x4c/0x190 [ 3730.591867] rcu_sched_clock_irq+0x80/0x204 [ 3730.596037] update_process_times+0x7c/0xc0 [ 3730.600207] tick_sched_handle+0x38/0x74 [ 3730.604118] tick_sched_timer+0x60/0xc0 [ 3730.607941] __hrtimer_run_queues+0x170/0x300 [ 3730.612285] hrtimer_interrupt+0xfc/0x260 [ 3730.616281] arch_timer_handler_phys+0x34/0x50 [ 3730.620712] handle_percpu_devid_irq+0xac/0x240 [ 3730.625229] generic_handle_domain_irq+0x34/0x50 [ 3730.629834] gic_handle_irq+0x5c/0xe0 [ 3730.633484] call_on_irq_stack+0x2c/0x38 [ 3730.637394] do_interrupt_handler+0xdc/0xe0 [ 3730.641565] el1_interrupt+0x34/0x70 [ 3730.645127] el1h_64_irq_handler+0x18/0x24 [ 3730.649210] el1h_64_irq+0x68/0x6c [ 3730.652599] __sbq_wake_up+0x58/0x1f0 [ 3730.656248] sbitmap_queue_clear+0x74/0xc0 [ 3730.660333] blk_mq_put_tag+0x44/0x70 [ 3730.663984] __blk_mq_free_request+0x88/0xf0 [ 3730.668240] blk_mq_free_request+0xc0/0x160 [ 3730.672410] __blk_mq_end_request+0x78/0x150 [ 3730.676667] scsi_end_request+0x114/0x260 [ 3730.680663] scsi_io_completion+0x64/0x174 [ 3730.684746] scsi_finish_command+0xe0/0x120 [ 3730.688918] scsi_complete+0x8c/0x1c4 [ 3730.692567] blk_complete_reqs+0x5c/0x74 [ 3730.696476] blk_done_softirq+0x38/0x4c [ 3730.700300] __do_softirq+0x168/0x418 [ 3730.703948] __irq_exit_rcu+0x120/0x170 [ 3730.707771] irq_exit_rcu+0x18/0x24 [ 3730.711246] el1_interrupt+0x38/0x70 [ 3730.714809] el1h_64_irq_handler+0x18/0x24 [ 3730.718892] el1h_64_irq+0x68/0x6c [ 3730.722280] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3730.727058] __wake_up_common_lock+0x84/0xac [ 3730.731315] __wake_up+0x20/0x30 [ 3730.734531] __sbq_wake_up+0x13c/0x1f0 [ 3730.738267] sbitmap_queue_clear+0x74/0xc0 [ 3730.742351] blk_mq_put_tag+0x44/0x70 [ 3730.746001] __blk_mq_free_request+0x88/0xf0 [ 3730.750258] blk_mq_free_request+0xc0/0x160 [ 3730.754428] bfq_insert_request.isra.0+0x204/0x270 [ 3730.759206] bfq_insert_requests+0x70/0x90 [ 3730.763289] blk_mq_sched_insert_requests+0xac/0x250 [ 3730.768242] blk_mq_flush_plug_list+0x110/0x2cc [ 3730.772759] __blk_flush_plug+0x100/0x140 [ 3730.776756] blk_finish_plug+0x40/0x25c [ 3730.780578] __iomap_dio_rw+0x1b4/0x500 [ 3730.784401] iomap_dio_rw+0x18/0x50 [ 3730.787876] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3730.793228] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3730.797970] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3730.803014] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3730.808144] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3730.813273] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3730.818660] process_one_work+0x1ec/0x470 [ 3730.822656] worker_thread+0x74/0x410 [ 3730.826305] kthread+0xec/0x100 [ 3730.829434] ret_from_fork+0x10/0x20 [ 3730.832997] Task dump for CPU 68: [ 3730.836299] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3730.846199] Call trace: [ 3730.848632] __switch_to+0xd4/0x100 [ 3730.852108] 0x0 [ 3733.469943] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 328253 jiffies s: 2093 root: 0x11/. [ 3733.481681] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3733.490890] Task dump for CPU 8: [ 3733.494104] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3733.504006] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3733.510176] Call trace: [ 3733.512609] __switch_to+0xd4/0x100 [ 3733.516086] 0xffff6c0200000080 [ 3733.519214] Task dump for CPU 68: [ 3733.522517] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3733.532418] Call trace: [ 3733.534851] __switch_to+0xd4/0x100 [ 3733.538327] 0x0 [ 3743.199858] watchdog: BUG: soft lockup - CPU#68 stuck for 3092s! [swapper/68:0] [ 3743.207154] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3743.207171] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3743.294959] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3743.302775] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3743.312066] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3743.319013] pc : __sbq_wake_up+0x58/0x1f0 [ 3743.323010] lr : sbitmap_queue_clear+0x74/0xc0 [ 3743.327440] sp : ffff8000088438b0 [ 3743.330741] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3743.337863] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3743.344985] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3743.352107] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3743.359229] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3743.366350] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3743.373473] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3743.380595] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3743.387716] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3743.394838] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3743.401960] Call trace: [ 3743.404393] __sbq_wake_up+0x58/0x1f0 [ 3743.408042] sbitmap_queue_clear+0x74/0xc0 [ 3743.412125] blk_mq_put_tag+0x44/0x70 [ 3743.415775] __blk_mq_free_request+0x88/0xf0 [ 3743.420032] blk_mq_free_request+0xc0/0x160 [ 3743.424202] __blk_mq_end_request+0x78/0x150 [ 3743.428458] scsi_end_request+0x114/0x260 [ 3743.432455] scsi_io_completion+0x64/0x174 [ 3743.436538] scsi_finish_command+0xe0/0x120 [ 3743.440708] scsi_complete+0x8c/0x1c4 [ 3743.444357] blk_complete_reqs+0x5c/0x74 [ 3743.448267] blk_done_softirq+0x38/0x4c [ 3743.452089] __do_softirq+0x168/0x418 [ 3743.455738] __irq_exit_rcu+0x120/0x170 [ 3743.459561] irq_exit_rcu+0x18/0x24 [ 3743.463037] el1_interrupt+0x38/0x70 [ 3743.466599] el1h_64_irq_handler+0x18/0x24 [ 3743.470683] el1h_64_irq+0x68/0x6c [ 3743.474071] cpuidle_enter_state+0xc4/0x4e0 [ 3743.478241] cpuidle_enter+0x40/0x60 [ 3743.481803] cpuidle_idle_call+0x118/0x1b0 [ 3743.485887] do_idle+0xac/0x100 [ 3743.489016] cpu_startup_entry+0x30/0x34 [ 3743.492925] secondary_start_kernel+0xd8/0xf0 [ 3743.497269] __secondary_switched+0xb0/0xb4 [ 3755.089785] watchdog: BUG: soft lockup - CPU#8 stuck for 3103s! [kworker/8:12:19399] [ 3755.097515] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3755.097533] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3755.185321] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3755.193571] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3755.202862] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3755.209030] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3755.215977] pc : __sbq_wake_up+0x40/0x1f0 [ 3755.219974] lr : sbitmap_queue_clear+0x74/0xc0 [ 3755.224405] sp : ffff80000d79b280 [ 3755.227706] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3755.234828] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3755.241950] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3755.249072] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3755.256194] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3755.263315] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3755.270437] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3755.277559] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3755.284681] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3755.291803] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 3755.298925] Call trace: [ 3755.301358] __sbq_wake_up+0x40/0x1f0 [ 3755.305007] sbitmap_queue_clear+0x74/0xc0 [ 3755.309091] blk_mq_put_tag+0x44/0x70 [ 3755.312741] __blk_mq_free_request+0x88/0xf0 [ 3755.316997] blk_mq_free_request+0xc0/0x160 [ 3755.321167] __blk_mq_end_request+0x78/0x150 [ 3755.325425] scsi_end_request+0x114/0x260 [ 3755.329421] scsi_io_completion+0x64/0x174 [ 3755.333504] scsi_finish_command+0xe0/0x120 [ 3755.337674] scsi_complete+0x8c/0x1c4 [ 3755.341324] blk_complete_reqs+0x5c/0x74 [ 3755.345233] blk_done_softirq+0x38/0x4c [ 3755.349056] __do_softirq+0x168/0x418 [ 3755.352705] __irq_exit_rcu+0x120/0x170 [ 3755.356527] irq_exit_rcu+0x18/0x24 [ 3755.360003] el1_interrupt+0x38/0x70 [ 3755.363565] el1h_64_irq_handler+0x18/0x24 [ 3755.367649] el1h_64_irq+0x68/0x6c [ 3755.371039] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3755.375817] __wake_up_common_lock+0x84/0xac [ 3755.380074] __wake_up+0x20/0x30 [ 3755.383290] __sbq_wake_up+0x13c/0x1f0 [ 3755.387026] sbitmap_queue_clear+0x74/0xc0 [ 3755.391109] blk_mq_put_tag+0x44/0x70 [ 3755.394759] __blk_mq_free_request+0x88/0xf0 [ 3755.399015] blk_mq_free_request+0xc0/0x160 [ 3755.403185] bfq_insert_request.isra.0+0x204/0x270 [ 3755.407963] bfq_insert_requests+0x70/0x90 [ 3755.412047] blk_mq_sched_insert_requests+0xac/0x250 [ 3755.416999] blk_mq_flush_plug_list+0x110/0x2cc [ 3755.421517] __blk_flush_plug+0x100/0x140 [ 3755.425513] blk_finish_plug+0x40/0x25c [ 3755.429336] __iomap_dio_rw+0x1b4/0x500 [ 3755.433159] iomap_dio_rw+0x18/0x50 [ 3755.436635] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3755.441985] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3755.446727] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3755.451771] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3755.456901] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3755.462031] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3755.467417] process_one_work+0x1ec/0x470 [ 3755.471413] worker_thread+0x74/0x410 [ 3755.475062] kthread+0xec/0x100 [ 3755.478190] ret_from_fork+0x10/0x20 [ 3767.199711] watchdog: BUG: soft lockup - CPU#68 stuck for 3114s! [swapper/68:0] [ 3767.207006] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3767.207024] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3767.294812] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3767.302628] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3767.311918] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3767.318866] pc : __sbq_wake_up+0x84/0x1f0 [ 3767.322862] lr : sbitmap_queue_clear+0x74/0xc0 [ 3767.327293] sp : ffff8000088438b0 [ 3767.330594] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3767.337716] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3767.344838] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3767.351960] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3767.359082] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3767.366204] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3767.373326] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3767.380448] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3767.387570] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3767.394692] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3767.401813] Call trace: [ 3767.404246] __sbq_wake_up+0x84/0x1f0 [ 3767.407895] sbitmap_queue_clear+0x74/0xc0 [ 3767.411980] blk_mq_put_tag+0x44/0x70 [ 3767.415630] __blk_mq_free_request+0x88/0xf0 [ 3767.419886] blk_mq_free_request+0xc0/0x160 [ 3767.424056] __blk_mq_end_request+0x78/0x150 [ 3767.428313] scsi_end_request+0x114/0x260 [ 3767.432309] scsi_io_completion+0x64/0x174 [ 3767.436392] scsi_finish_command+0xe0/0x120 [ 3767.440563] scsi_complete+0x8c/0x1c4 [ 3767.444212] blk_complete_reqs+0x5c/0x74 [ 3767.448121] blk_done_softirq+0x38/0x4c [ 3767.451944] __do_softirq+0x168/0x418 [ 3767.455593] __irq_exit_rcu+0x120/0x170 [ 3767.459416] irq_exit_rcu+0x18/0x24 [ 3767.462891] el1_interrupt+0x38/0x70 [ 3767.466453] el1h_64_irq_handler+0x18/0x24 [ 3767.470537] el1h_64_irq+0x68/0x6c [ 3767.473925] cpuidle_enter_state+0xc4/0x4e0 [ 3767.478095] cpuidle_enter+0x40/0x60 [ 3767.481657] cpuidle_idle_call+0x118/0x1b0 [ 3767.485742] do_idle+0xac/0x100 [ 3767.488870] cpu_startup_entry+0x30/0x34 [ 3767.492779] secondary_start_kernel+0xd8/0xf0 [ 3767.497123] __secondary_switched+0xb0/0xb4 [ 3779.089639] watchdog: BUG: soft lockup - CPU#8 stuck for 3125s! [kworker/8:12:19399] [ 3779.097368] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3779.097385] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3779.185173] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3779.193423] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3779.202714] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3779.208881] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3779.215829] pc : __sbq_wake_up+0x4/0x1f0 [ 3779.219739] lr : sbitmap_queue_clear+0x74/0xc0 [ 3779.224170] sp : ffff80000d79b2d0 [ 3779.227471] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3779.234594] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3779.241716] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3779.248838] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3779.255960] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3779.263082] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3779.270204] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3779.277326] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3779.284448] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3779.291569] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 3779.298691] Call trace: [ 3779.301124] __sbq_wake_up+0x4/0x1f0 [ 3779.304687] blk_mq_put_tag+0x44/0x70 [ 3779.308336] __blk_mq_free_request+0x88/0xf0 [ 3779.312593] blk_mq_free_request+0xc0/0x160 [ 3779.316763] __blk_mq_end_request+0x78/0x150 [ 3779.321020] scsi_end_request+0x114/0x260 [ 3779.325016] scsi_io_completion+0x64/0x174 [ 3779.329099] scsi_finish_command+0xe0/0x120 [ 3779.333269] scsi_complete+0x8c/0x1c4 [ 3779.336919] blk_complete_reqs+0x5c/0x74 [ 3779.340828] blk_done_softirq+0x38/0x4c [ 3779.344651] __do_softirq+0x168/0x418 [ 3779.348300] __irq_exit_rcu+0x120/0x170 [ 3779.352123] irq_exit_rcu+0x18/0x24 [ 3779.355598] el1_interrupt+0x38/0x70 [ 3779.359160] el1h_64_irq_handler+0x18/0x24 [ 3779.363243] el1h_64_irq+0x68/0x6c [ 3779.366632] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3779.371410] __wake_up_common_lock+0x84/0xac [ 3779.375667] __wake_up+0x20/0x30 [ 3779.378882] __sbq_wake_up+0x13c/0x1f0 [ 3779.382619] sbitmap_queue_clear+0x74/0xc0 [ 3779.386702] blk_mq_put_tag+0x44/0x70 [ 3779.390352] __blk_mq_free_request+0x88/0xf0 [ 3779.394609] blk_mq_free_request+0xc0/0x160 [ 3779.398779] bfq_insert_request.isra.0+0x204/0x270 [ 3779.403557] bfq_insert_requests+0x70/0x90 [ 3779.407640] blk_mq_sched_insert_requests+0xac/0x250 [ 3779.412593] blk_mq_flush_plug_list+0x110/0x2cc [ 3779.417110] __blk_flush_plug+0x100/0x140 [ 3779.421107] blk_finish_plug+0x40/0x25c [ 3779.424930] __iomap_dio_rw+0x1b4/0x500 [ 3779.428753] iomap_dio_rw+0x18/0x50 [ 3779.432229] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3779.437580] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3779.442322] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3779.447366] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3779.452496] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3779.457626] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3779.463012] process_one_work+0x1ec/0x470 [ 3779.467008] worker_thread+0x74/0x410 [ 3779.470657] kthread+0xec/0x100 [ 3779.473786] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 05:00:00 2022] [ 3791.199564] watchdog: BUG: soft lockup - CPU#68 stuck for 3136s! [swapper/68:0] [ 3791.206860] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3791.206877] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3791.294665] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3791.302482] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3791.311773] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3791.318720] pc : __sbq_wake_up+0x40/0x1f0 [ 3791.322716] lr : sbitmap_queue_clear+0x74/0xc0 [ 3791.327147] sp : ffff8000088438b0 [ 3791.330448] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3791.337569] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3791.344691] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3791.351813] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3791.358935] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3791.366057] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3791.373178] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3791.380300] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3791.387422] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3791.394544] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 3791.401666] Call trace: [ 3791.404099] __sbq_wake_up+0x40/0x1f0 [ 3791.407748] sbitmap_queue_clear+0x74/0xc0 [ 3791.411832] blk_mq_put_tag+0x44/0x70 [ 3791.415481] __blk_mq_free_request+0x88/0xf0 [ 3791.419738] blk_mq_free_request+0xc0/0x160 [ 3791.423909] __blk_mq_end_request+0x78/0x150 [ 3791.428165] scsi_end_request+0x114/0x260 [ 3791.432162] scsi_io_completion+0x64/0x174 [ 3791.436245] scsi_finish_command+0xe0/0x120 [ 3791.440415] scsi_complete+0x8c/0x1c4 [ 3791.444064] blk_complete_reqs+0x5c/0x74 [ 3791.447974] blk_done_softirq+0x38/0x4c [ 3791.451796] __do_softirq+0x168/0x418 [ 3791.455445] __irq_exit_rcu+0x120/0x170 [ 3791.459268] irq_exit_rcu+0x18/0x24 [ 3791.462744] el1_interrupt+0x38/0x70 [ 3791.466306] el1h_64_irq_handler+0x18/0x24 [ 3791.470389] el1h_64_irq+0x68/0x6c [ 3791.473777] cpuidle_enter_state+0xc4/0x4e0 [ 3791.477947] cpuidle_enter+0x40/0x60 [ 3791.481509] cpuidle_idle_call+0x118/0x1b0 [ 3791.485593] do_idle+0xac/0x100 [ 3791.488722] cpu_startup_entry+0x30/0x34 [ 3791.492631] secondary_start_kernel+0xd8/0xf0 [ 3791.496975] __secondary_switched+0xb0/0xb4 [ 3800.029521] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 334909 jiffies s: 2093 root: 0x11/. [ 3800.041259] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3800.050467] Task dump for CPU 8: [ 3800.053682] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3800.063584] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3800.069754] Call trace: [ 3800.072187] __switch_to+0xd4/0x100 [ 3800.075663] 0xffff6c0200000080 [ 3800.078793] Task dump for CPU 68: [ 3800.082097] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3800.091998] Call trace: [ 3800.094431] __switch_to+0xd4/0x100 [ 3800.097908] 0x0 [ 3803.089492] watchdog: BUG: soft lockup - CPU#8 stuck for 3147s! [kworker/8:12:19399] [ 3803.097221] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3803.097239] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3803.185027] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3803.193277] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3803.202568] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3803.208736] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3803.215683] pc : __sbq_wake_up+0x4/0x1f0 [ 3803.219593] lr : sbitmap_queue_clear+0x74/0xc0 [ 3803.224024] sp : ffff80000d79b2d0 [ 3803.227325] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3803.234448] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3803.241571] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3803.248693] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3803.255816] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3803.262938] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3803.270059] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3803.277181] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3803.284303] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3803.291425] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 3803.298548] Call trace: [ 3803.300981] __sbq_wake_up+0x4/0x1f0 [ 3803.304543] blk_mq_put_tag+0x44/0x70 [ 3803.308193] __blk_mq_free_request+0x88/0xf0 [ 3803.312450] blk_mq_free_request+0xc0/0x160 [ 3803.316620] __blk_mq_end_request+0x78/0x150 [ 3803.320877] scsi_end_request+0x114/0x260 [ 3803.324873] scsi_io_completion+0x64/0x174 [ 3803.328956] scsi_finish_command+0xe0/0x120 [ 3803.333127] scsi_complete+0x8c/0x1c4 [ 3803.336776] blk_complete_reqs+0x5c/0x74 [ 3803.340685] blk_done_softirq+0x38/0x4c [ 3803.344508] __do_softirq+0x168/0x418 [ 3803.348157] __irq_exit_rcu+0x120/0x170 [ 3803.351980] irq_exit_rcu+0x18/0x24 [ 3803.355455] el1_interrupt+0x38/0x70 [ 3803.359018] el1h_64_irq_handler+0x18/0x24 [ 3803.363101] el1h_64_irq+0x68/0x6c [ 3803.366489] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3803.371267] __wake_up_common_lock+0x84/0xac [ 3803.375524] __wake_up+0x20/0x30 [ 3803.378740] __sbq_wake_up+0x13c/0x1f0 [ 3803.382476] sbitmap_queue_clear+0x74/0xc0 [ 3803.386559] blk_mq_put_tag+0x44/0x70 [ 3803.390209] __blk_mq_free_request+0x88/0xf0 [ 3803.394466] blk_mq_free_request+0xc0/0x160 [ 3803.398635] bfq_insert_request.isra.0+0x204/0x270 [ 3803.403413] bfq_insert_requests+0x70/0x90 [ 3803.407497] blk_mq_sched_insert_requests+0xac/0x250 [ 3803.412449] blk_mq_flush_plug_list+0x110/0x2cc [ 3803.416968] __blk_flush_plug+0x100/0x140 [ 3803.420964] blk_finish_plug+0x40/0x25c [ 3803.424787] __iomap_dio_rw+0x1b4/0x500 [ 3803.428610] iomap_dio_rw+0x18/0x50 [ 3803.432085] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3803.437436] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3803.442179] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3803.447222] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3803.452353] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3803.457483] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3803.462869] process_one_work+0x1ec/0x470 [ 3803.466865] worker_thread+0x74/0x410 [ 3803.470514] kthread+0xec/0x100 [ 3803.473643] ret_from_fork+0x10/0x20 [ 3815.199418] watchdog: BUG: soft lockup - CPU#68 stuck for 3159s! [swapper/68:0] [ 3815.206713] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3815.206730] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3815.294517] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3815.302333] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3815.311624] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3815.318572] pc : __sbq_wake_up+0x20/0x1f0 [ 3815.322568] lr : sbitmap_queue_clear+0x74/0xc0 [ 3815.326999] sp : ffff8000088438b0 [ 3815.330300] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3815.337423] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3815.344545] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3815.351667] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3815.358789] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3815.365911] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3815.373033] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3815.380155] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3815.387277] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3815.394398] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 3815.401520] Call trace: [ 3815.403954] __sbq_wake_up+0x20/0x1f0 [ 3815.407603] sbitmap_queue_clear+0x74/0xc0 [ 3815.411688] blk_mq_put_tag+0x44/0x70 [ 3815.415337] __blk_mq_free_request+0x88/0xf0 [ 3815.419594] blk_mq_free_request+0xc0/0x160 [ 3815.423764] __blk_mq_end_request+0x78/0x150 [ 3815.428021] scsi_end_request+0x114/0x260 [ 3815.432017] scsi_io_completion+0x64/0x174 [ 3815.436101] scsi_finish_command+0xe0/0x120 [ 3815.440272] scsi_complete+0x8c/0x1c4 [ 3815.443921] blk_complete_reqs+0x5c/0x74 [ 3815.447831] blk_done_softirq+0x38/0x4c [ 3815.451654] __do_softirq+0x168/0x418 [ 3815.455302] __irq_exit_rcu+0x120/0x170 [ 3815.459126] irq_exit_rcu+0x18/0x24 [ 3815.462602] el1_interrupt+0x38/0x70 [ 3815.466164] el1h_64_irq_handler+0x18/0x24 [ 3815.470247] el1h_64_irq+0x68/0x6c [ 3815.473636] cpuidle_enter_state+0xc4/0x4e0 [ 3815.477807] cpuidle_enter+0x40/0x60 [ 3815.481369] cpuidle_idle_call+0x118/0x1b0 [ 3815.485453] do_idle+0xac/0x100 [ 3815.488582] cpu_startup_entry+0x30/0x34 [ 3815.492491] secondary_start_kernel+0xd8/0xf0 [ 3815.496835] __secondary_switched+0xb0/0xb4 [ 3827.089345] watchdog: BUG: soft lockup - CPU#8 stuck for 3170s! [kworker/8:12:19399] [ 3827.097074] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3827.097091] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3827.184879] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3827.193128] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3827.202420] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3827.208587] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3827.215535] pc : __sbq_wake_up+0x58/0x1f0 [ 3827.219532] lr : sbitmap_queue_clear+0x74/0xc0 [ 3827.223963] sp : ffff80000d79b280 [ 3827.227264] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3827.234386] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3827.241509] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3827.248631] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3827.255753] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3827.262875] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3827.269996] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3827.277118] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3827.284240] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3827.291362] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 3827.298484] Call trace: [ 3827.300917] __sbq_wake_up+0x58/0x1f0 [ 3827.304566] sbitmap_queue_clear+0x74/0xc0 [ 3827.308650] blk_mq_put_tag+0x44/0x70 [ 3827.312300] __blk_mq_free_request+0x88/0xf0 [ 3827.316557] blk_mq_free_request+0xc0/0x160 [ 3827.320727] __blk_mq_end_request+0x78/0x150 [ 3827.324984] scsi_end_request+0x114/0x260 [ 3827.328980] scsi_io_completion+0x64/0x174 [ 3827.333063] scsi_finish_command+0xe0/0x120 [ 3827.337234] scsi_complete+0x8c/0x1c4 [ 3827.340883] blk_complete_reqs+0x5c/0x74 [ 3827.344792] blk_done_softirq+0x38/0x4c [ 3827.348615] __do_softirq+0x168/0x418 [ 3827.352264] __irq_exit_rcu+0x120/0x170 [ 3827.356087] irq_exit_rcu+0x18/0x24 [ 3827.359562] el1_interrupt+0x38/0x70 [ 3827.363124] el1h_64_irq_handler+0x18/0x24 [ 3827.367207] el1h_64_irq+0x68/0x6c [ 3827.370596] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3827.375374] __wake_up_common_lock+0x84/0xac [ 3827.379631] __wake_up+0x20/0x30 [ 3827.382847] __sbq_wake_up+0x13c/0x1f0 [ 3827.386582] sbitmap_queue_clear+0x74/0xc0 [ 3827.390666] blk_mq_put_tag+0x44/0x70 [ 3827.394316] __blk_mq_free_request+0x88/0xf0 [ 3827.398572] blk_mq_free_request+0xc0/0x160 [ 3827.402743] bfq_insert_request.isra.0+0x204/0x270 [ 3827.407521] bfq_insert_requests+0x70/0x90 [ 3827.411605] blk_mq_sched_insert_requests+0xac/0x250 [ 3827.416557] blk_mq_flush_plug_list+0x110/0x2cc [ 3827.421075] __blk_flush_plug+0x100/0x140 [ 3827.425071] blk_finish_plug+0x40/0x25c [ 3827.428895] __iomap_dio_rw+0x1b4/0x500 [ 3827.432717] iomap_dio_rw+0x18/0x50 [ 3827.436193] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3827.441543] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3827.446285] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3827.451329] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3827.456460] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3827.461589] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3827.466975] process_one_work+0x1ec/0x470 [ 3827.470972] worker_thread+0x74/0x410 [ 3827.474621] kthread+0xec/0x100 [ 3827.477750] ret_from_fork+0x10/0x20 [ 3839.199271] watchdog: BUG: soft lockup - CPU#68 stuck for 3181s! [swapper/68:0] [ 3839.206566] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3839.206583] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3839.294371] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3839.302186] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3839.311477] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3839.318425] pc : __sbq_wake_up+0x7c/0x1f0 [ 3839.322422] lr : sbitmap_queue_clear+0x74/0xc0 [ 3839.326852] sp : ffff800008843900 [ 3839.330153] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3839.337275] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3839.344396] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3839.351519] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3839.358641] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3839.365762] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3839.372885] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3839.380007] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3839.387128] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3839.394250] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 3839.401372] Call trace: [ 3839.403805] __sbq_wake_up+0x7c/0x1f0 [ 3839.407454] blk_mq_put_tag+0x44/0x70 [ 3839.411105] __blk_mq_free_request+0x88/0xf0 [ 3839.415361] blk_mq_free_request+0xc0/0x160 [ 3839.419531] __blk_mq_end_request+0x78/0x150 [ 3839.423788] scsi_end_request+0x114/0x260 [ 3839.427784] scsi_io_completion+0x64/0x174 [ 3839.431867] scsi_finish_command+0xe0/0x120 [ 3839.436038] scsi_complete+0x8c/0x1c4 [ 3839.439687] blk_complete_reqs+0x5c/0x74 [ 3839.443596] blk_done_softirq+0x38/0x4c [ 3839.447419] __do_softirq+0x168/0x418 [ 3839.451067] __irq_exit_rcu+0x120/0x170 [ 3839.454890] irq_exit_rcu+0x18/0x24 [ 3839.458366] el1_interrupt+0x38/0x70 [ 3839.461928] el1h_64_irq_handler+0x18/0x24 [ 3839.466012] el1h_64_irq+0x68/0x6c [ 3839.469400] cpuidle_enter_state+0xc4/0x4e0 [ 3839.473571] cpuidle_enter+0x40/0x60 [ 3839.477133] cpuidle_idle_call+0x118/0x1b0 [ 3839.481217] do_idle+0xac/0x100 [ 3839.484346] cpu_startup_entry+0x30/0x34 [ 3839.488255] secondary_start_kernel+0xd8/0xf0 [ 3839.492599] __secondary_switched+0xb0/0xb4 [ 3851.089198] watchdog: BUG: soft lockup - CPU#8 stuck for 3192s! [kworker/8:12:19399] [ 3851.096928] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3851.096945] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3851.184733] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3851.192983] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3851.202274] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3851.208441] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3851.215389] pc : __sbq_wake_up+0x1c8/0x1f0 [ 3851.219472] lr : sbitmap_queue_clear+0x74/0xc0 [ 3851.223903] sp : ffff80000d79b280 [ 3851.227204] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3851.234326] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3851.241449] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 3851.248571] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3851.255694] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3851.262815] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3851.269937] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3851.277059] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3851.284181] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3851.291303] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 3851.298425] Call trace: [ 3851.300858] __sbq_wake_up+0x1c8/0x1f0 [ 3851.304596] sbitmap_queue_clear+0x74/0xc0 [ 3851.308679] blk_mq_put_tag+0x44/0x70 [ 3851.312329] __blk_mq_free_request+0x88/0xf0 [ 3851.316585] blk_mq_free_request+0xc0/0x160 [ 3851.320755] __blk_mq_end_request+0x78/0x150 [ 3851.325012] scsi_end_request+0x114/0x260 [ 3851.329008] scsi_io_completion+0x64/0x174 [ 3851.333091] scsi_finish_command+0xe0/0x120 [ 3851.337262] scsi_complete+0x8c/0x1c4 [ 3851.340911] blk_complete_reqs+0x5c/0x74 [ 3851.344820] blk_done_softirq+0x38/0x4c [ 3851.348643] __do_softirq+0x168/0x418 [ 3851.352292] __irq_exit_rcu+0x120/0x170 [ 3851.356114] irq_exit_rcu+0x18/0x24 [ 3851.359590] el1_interrupt+0x38/0x70 [ 3851.363152] el1h_64_irq_handler+0x18/0x24 [ 3851.367236] el1h_64_irq+0x68/0x6c [ 3851.370624] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3851.375402] __wake_up_common_lock+0x84/0xac [ 3851.379659] __wake_up+0x20/0x30 [ 3851.382875] __sbq_wake_up+0x13c/0x1f0 [ 3851.386611] sbitmap_queue_clear+0x74/0xc0 [ 3851.390694] blk_mq_put_tag+0x44/0x70 [ 3851.394344] __blk_mq_free_request+0x88/0xf0 [ 3851.398601] blk_mq_free_request+0xc0/0x160 [ 3851.402770] bfq_insert_request.isra.0+0x204/0x270 [ 3851.407548] bfq_insert_requests+0x70/0x90 [ 3851.411632] blk_mq_sched_insert_requests+0xac/0x250 [ 3851.416584] blk_mq_flush_plug_list+0x110/0x2cc [ 3851.421102] __blk_flush_plug+0x100/0x140 [ 3851.425098] blk_finish_plug+0x40/0x25c [ 3851.428920] __iomap_dio_rw+0x1b4/0x500 [ 3851.432743] iomap_dio_rw+0x18/0x50 [ 3851.436219] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3851.441570] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3851.446312] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3851.451356] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3851.456486] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3851.461616] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3851.467002] process_one_work+0x1ec/0x470 [ 3851.470998] worker_thread+0x74/0x410 [ 3851.474647] kthread+0xec/0x100 [ 3851.477776] ret_from_fork+0x10/0x20 [ 3863.199124] watchdog: BUG: soft lockup - CPU#68 stuck for 3203s! [swapper/68:0] [ 3863.206419] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3863.206436] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3863.294223] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3863.302039] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3863.311330] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3863.318278] pc : __sbq_wake_up+0x58/0x1f0 [ 3863.322275] lr : sbitmap_queue_clear+0x74/0xc0 [ 3863.326705] sp : ffff8000088438b0 [ 3863.330008] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3863.337130] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3863.344251] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3863.351373] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3863.358495] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3863.365617] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3863.372739] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3863.379861] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3863.386982] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3863.394104] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3863.401228] Call trace: [ 3863.403660] __sbq_wake_up+0x58/0x1f0 [ 3863.407310] sbitmap_queue_clear+0x74/0xc0 [ 3863.411393] blk_mq_put_tag+0x44/0x70 [ 3863.415043] __blk_mq_free_request+0x88/0xf0 [ 3863.419299] blk_mq_free_request+0xc0/0x160 [ 3863.423469] __blk_mq_end_request+0x78/0x150 [ 3863.427726] scsi_end_request+0x114/0x260 [ 3863.431722] scsi_io_completion+0x64/0x174 [ 3863.435805] scsi_finish_command+0xe0/0x120 [ 3863.439976] scsi_complete+0x8c/0x1c4 [ 3863.443625] blk_complete_reqs+0x5c/0x74 [ 3863.447534] blk_done_softirq+0x38/0x4c [ 3863.451357] __do_softirq+0x168/0x418 [ 3863.455006] __irq_exit_rcu+0x120/0x170 [ 3863.458829] irq_exit_rcu+0x18/0x24 [ 3863.462304] el1_interrupt+0x38/0x70 [ 3863.465868] el1h_64_irq_handler+0x18/0x24 [ 3863.469951] el1h_64_irq+0x68/0x6c [ 3863.473339] cpuidle_enter_state+0xc4/0x4e0 [ 3863.477509] cpuidle_enter+0x40/0x60 [ 3863.481072] cpuidle_idle_call+0x118/0x1b0 [ 3863.485155] do_idle+0xac/0x100 [ 3863.488284] cpu_startup_entry+0x30/0x34 [ 3863.492193] secondary_start_kernel+0xd8/0xf0 [ 3863.496537] __secondary_switched+0xb0/0xb4 [ 3866.589113] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 341565 jiffies s: 2093 root: 0x11/. [ 3866.600851] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3866.610060] Task dump for CPU 8: [ 3866.613275] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3866.623176] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3866.629346] Call trace: [ 3866.631779] __switch_to+0xd4/0x100 [ 3866.635256] 0xffff6c0200000080 [ 3866.638384] Task dump for CPU 68: [ 3866.641687] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3866.651589] Call trace: [ 3866.654022] __switch_to+0xd4/0x100 [ 3866.657498] 0x0 [ 3875.089051] watchdog: BUG: soft lockup - CPU#8 stuck for 3214s! [kworker/8:12:19399] [ 3875.096781] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3875.096798] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3875.184586] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3875.192836] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3875.202127] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3875.208294] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3875.215242] pc : __sbq_wake_up+0x8/0x1f0 [ 3875.219151] lr : sbitmap_queue_clear+0x74/0xc0 [ 3875.223582] sp : ffff80000d79b280 [ 3875.226883] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3875.234006] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3875.241128] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3875.248250] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3875.255372] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3875.262495] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3875.269617] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3875.276739] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3875.283861] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3875.290983] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 3875.298105] Call trace: [ 3875.300538] __sbq_wake_up+0x8/0x1f0 [ 3875.304100] blk_mq_put_tag+0x44/0x70 [ 3875.307750] __blk_mq_free_request+0x88/0xf0 [ 3875.312006] blk_mq_free_request+0xc0/0x160 [ 3875.316176] __blk_mq_end_request+0x78/0x150 [ 3875.320433] scsi_end_request+0x114/0x260 [ 3875.324429] scsi_io_completion+0x64/0x174 [ 3875.328512] scsi_finish_command+0xe0/0x120 [ 3875.332683] scsi_complete+0x8c/0x1c4 [ 3875.336332] blk_complete_reqs+0x5c/0x74 [ 3875.340241] blk_done_softirq+0x38/0x4c [ 3875.344064] __do_softirq+0x168/0x418 [ 3875.347714] __irq_exit_rcu+0x120/0x170 [ 3875.351536] irq_exit_rcu+0x18/0x24 [ 3875.355012] el1_interrupt+0x38/0x70 [ 3875.358574] el1h_64_irq_handler+0x18/0x24 [ 3875.362658] el1h_64_irq+0x68/0x6c [ 3875.366047] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3875.370825] __wake_up_common_lock+0x84/0xac [ 3875.375082] __wake_up+0x20/0x30 [ 3875.378298] __sbq_wake_up+0x13c/0x1f0 [ 3875.382034] sbitmap_queue_clear+0x74/0xc0 [ 3875.386117] blk_mq_put_tag+0x44/0x70 [ 3875.389766] __blk_mq_free_request+0x88/0xf0 [ 3875.394023] blk_mq_free_request+0xc0/0x160 [ 3875.398193] bfq_insert_request.isra.0+0x204/0x270 [ 3875.402971] bfq_insert_requests+0x70/0x90 [ 3875.407054] blk_mq_sched_insert_requests+0xac/0x250 [ 3875.412006] blk_mq_flush_plug_list+0x110/0x2cc [ 3875.416524] __blk_flush_plug+0x100/0x140 [ 3875.420520] blk_finish_plug+0x40/0x25c [ 3875.424343] __iomap_dio_rw+0x1b4/0x500 [ 3875.428166] iomap_dio_rw+0x18/0x50 [ 3875.431641] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3875.436992] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3875.441735] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3875.446778] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3875.451909] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3875.457038] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3875.462425] process_one_work+0x1ec/0x470 [ 3875.466421] worker_thread+0x74/0x410 [ 3875.470070] kthread+0xec/0x100 [ 3875.473199] ret_from_fork+0x10/0x20 [ 3887.198977] watchdog: BUG: soft lockup - CPU#68 stuck for 3226s! [swapper/68:0] [ 3887.206272] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3887.206289] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3887.294077] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3887.301893] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3887.311183] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3887.318131] pc : __sbq_wake_up+0x20/0x1f0 [ 3887.322127] lr : sbitmap_queue_clear+0x74/0xc0 [ 3887.326558] sp : ffff8000088438b0 [ 3887.329859] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3887.336981] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3887.344103] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3887.351224] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3887.358346] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3887.365468] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3887.372590] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3887.379712] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3887.386834] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3887.393955] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 3887.401077] Call trace: [ 3887.403511] __sbq_wake_up+0x20/0x1f0 [ 3887.407160] sbitmap_queue_clear+0x74/0xc0 [ 3887.411243] blk_mq_put_tag+0x44/0x70 [ 3887.414893] __blk_mq_free_request+0x88/0xf0 [ 3887.419150] blk_mq_free_request+0xc0/0x160 [ 3887.423319] __blk_mq_end_request+0x78/0x150 [ 3887.427576] scsi_end_request+0x114/0x260 [ 3887.431573] scsi_io_completion+0x64/0x174 [ 3887.435656] scsi_finish_command+0xe0/0x120 [ 3887.439826] scsi_complete+0x8c/0x1c4 [ 3887.443475] blk_complete_reqs+0x5c/0x74 [ 3887.447385] blk_done_softirq+0x38/0x4c [ 3887.451207] __do_softirq+0x168/0x418 [ 3887.454857] __irq_exit_rcu+0x120/0x170 [ 3887.458680] irq_exit_rcu+0x18/0x24 [ 3887.462156] el1_interrupt+0x38/0x70 [ 3887.465718] el1h_64_irq_handler+0x18/0x24 [ 3887.469801] el1h_64_irq+0x68/0x6c [ 3887.473190] cpuidle_enter_state+0xc4/0x4e0 [ 3887.477360] cpuidle_enter+0x40/0x60 [ 3887.480922] cpuidle_idle_call+0x118/0x1b0 [ 3887.485006] do_idle+0xac/0x100 [ 3887.488135] cpu_startup_entry+0x30/0x34 [ 3887.492044] secondary_start_kernel+0xd8/0xf0 [ 3887.496388] __secondary_switched+0xb0/0xb4 [ 3899.088904] watchdog: BUG: soft lockup - CPU#8 stuck for 3237s! [kworker/8:12:19399] [ 3899.096634] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3899.096651] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3899.184439] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3899.192689] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3899.201980] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3899.208148] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3899.215096] pc : __sbq_wake_up+0x4/0x1f0 [ 3899.219006] lr : sbitmap_queue_clear+0x74/0xc0 [ 3899.223437] sp : ffff80000d79b2d0 [ 3899.226738] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3899.233861] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 3899.240983] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 3899.248105] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 3899.255227] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 3899.262349] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 3899.269472] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 3899.276593] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3899.283715] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3899.290837] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 3899.297959] Call trace: [ 3899.300392] __sbq_wake_up+0x4/0x1f0 [ 3899.303954] blk_mq_put_tag+0x44/0x70 [ 3899.307604] __blk_mq_free_request+0x88/0xf0 [ 3899.311860] blk_mq_free_request+0xc0/0x160 [ 3899.316030] __blk_mq_end_request+0x78/0x150 [ 3899.320287] scsi_end_request+0x114/0x260 [ 3899.324283] scsi_io_completion+0x64/0x174 [ 3899.328366] scsi_finish_command+0xe0/0x120 [ 3899.332537] scsi_complete+0x8c/0x1c4 [ 3899.336186] blk_complete_reqs+0x5c/0x74 [ 3899.340095] blk_done_softirq+0x38/0x4c [ 3899.343918] __do_softirq+0x168/0x418 [ 3899.347566] __irq_exit_rcu+0x120/0x170 [ 3899.351389] irq_exit_rcu+0x18/0x24 [ 3899.354865] el1_interrupt+0x38/0x70 [ 3899.358427] el1h_64_irq_handler+0x18/0x24 [ 3899.362510] el1h_64_irq+0x68/0x6c [ 3899.365899] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3899.370676] __wake_up_common_lock+0x84/0xac [ 3899.374934] __wake_up+0x20/0x30 [ 3899.378149] __sbq_wake_up+0x13c/0x1f0 [ 3899.381885] sbitmap_queue_clear+0x74/0xc0 [ 3899.385969] blk_mq_put_tag+0x44/0x70 [ 3899.389619] __blk_mq_free_request+0x88/0xf0 [ 3899.393875] blk_mq_free_request+0xc0/0x160 [ 3899.398045] bfq_insert_request.isra.0+0x204/0x270 [ 3899.402824] bfq_insert_requests+0x70/0x90 [ 3899.406908] blk_mq_sched_insert_requests+0xac/0x250 [ 3899.411860] blk_mq_flush_plug_list+0x110/0x2cc [ 3899.416379] __blk_flush_plug+0x100/0x140 [ 3899.420375] blk_finish_plug+0x40/0x25c [ 3899.424198] __iomap_dio_rw+0x1b4/0x500 [ 3899.428021] iomap_dio_rw+0x18/0x50 [ 3899.431496] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3899.436847] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3899.441589] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3899.446633] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3899.451763] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3899.456893] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3899.462279] process_one_work+0x1ec/0x470 [ 3899.466275] worker_thread+0x74/0x410 [ 3899.469924] kthread+0xec/0x100 [ 3899.473053] ret_from_fork+0x10/0x20 [ 3910.878832] rcu: INFO: rcu_preempt self-detected stall on CPU [ 3910.884565] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=172874 [ 3910.893686] (t=348601 jiffies g=36445 q=101770 ncpus=128) [ 3910.899158] Task dump for CPU 8: [ 3910.902372] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3910.912272] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3910.918440] Call trace: [ 3910.920873] dump_backtrace+0xe8/0x130 [ 3910.924609] show_stack+0x20/0x60 [ 3910.927912] sched_show_task+0x160/0x190 [ 3910.931822] dump_cpu_task+0x4c/0x78 [ 3910.935384] rcu_dump_cpu_stacks+0xc4/0x150 [ 3910.939556] print_cpu_stall+0x180/0x210 [ 3910.943465] check_cpu_stall+0x20c/0x2b4 [ 3910.947375] rcu_pending+0x4c/0x190 [ 3910.950850] rcu_sched_clock_irq+0x80/0x204 [ 3910.955020] update_process_times+0x7c/0xc0 [ 3910.959190] tick_sched_handle+0x38/0x74 [ 3910.963100] tick_sched_timer+0x60/0xc0 [ 3910.966923] __hrtimer_run_queues+0x170/0x300 [ 3910.971267] hrtimer_interrupt+0xfc/0x260 [ 3910.975263] arch_timer_handler_phys+0x34/0x50 [ 3910.979694] handle_percpu_devid_irq+0xac/0x240 [ 3910.984212] generic_handle_domain_irq+0x34/0x50 [ 3910.988817] gic_handle_irq+0x5c/0xe0 [ 3910.992467] call_on_irq_stack+0x2c/0x38 [ 3910.996377] do_interrupt_handler+0xdc/0xe0 [ 3911.000547] el1_interrupt+0x34/0x70 [ 3911.004110] el1h_64_irq_handler+0x18/0x24 [ 3911.008194] el1h_64_irq+0x68/0x6c [ 3911.011583] __sbq_wake_up+0x58/0x1f0 [ 3911.015232] sbitmap_queue_clear+0x74/0xc0 [ 3911.019316] blk_mq_put_tag+0x44/0x70 [ 3911.022966] __blk_mq_free_request+0x88/0xf0 [ 3911.027222] blk_mq_free_request+0xc0/0x160 [ 3911.031392] __blk_mq_end_request+0x78/0x150 [ 3911.035649] scsi_end_request+0x114/0x260 [ 3911.039646] scsi_io_completion+0x64/0x174 [ 3911.043729] scsi_finish_command+0xe0/0x120 [ 3911.047900] scsi_complete+0x8c/0x1c4 [ 3911.051549] blk_complete_reqs+0x5c/0x74 [ 3911.055458] blk_done_softirq+0x38/0x4c [ 3911.059281] __do_softirq+0x168/0x418 [ 3911.062930] __irq_exit_rcu+0x120/0x170 [ 3911.066753] irq_exit_rcu+0x18/0x24 [ 3911.070228] el1_interrupt+0x38/0x70 [ 3911.073791] el1h_64_irq_handler+0x18/0x24 [ 3911.077874] el1h_64_irq+0x68/0x6c [ 3911.081262] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3911.086040] __wake_up_common_lock+0x84/0xac [ 3911.090298] __wake_up+0x20/0x30 [ 3911.093513] __sbq_wake_up+0x13c/0x1f0 [ 3911.097250] sbitmap_queue_clear+0x74/0xc0 [ 3911.101333] blk_mq_put_tag+0x44/0x70 [ 3911.104983] __blk_mq_free_request+0x88/0xf0 [ 3911.109240] blk_mq_free_request+0xc0/0x160 [ 3911.113410] bfq_insert_request.isra.0+0x204/0x270 [ 3911.118188] bfq_insert_requests+0x70/0x90 [ 3911.122271] blk_mq_sched_insert_requests+0xac/0x250 [ 3911.127223] blk_mq_flush_plug_list+0x110/0x2cc [ 3911.131741] __blk_flush_plug+0x100/0x140 [ 3911.135737] blk_finish_plug+0x40/0x25c [ 3911.139560] __iomap_dio_rw+0x1b4/0x500 [ 3911.143383] iomap_dio_rw+0x18/0x50 [ 3911.146858] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3911.152210] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3911.156952] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3911.161996] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3911.167126] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3911.172256] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3911.177643] process_one_work+0x1ec/0x470 [ 3911.181639] worker_thread+0x74/0x410 [ 3911.185287] kthread+0xec/0x100 [ 3911.188416] ret_from_fork+0x10/0x20 [ 3911.191979] Task dump for CPU 68: [ 3911.195281] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3911.198830] watchdog: BUG: soft lockup - CPU#68 stuck for 3248s! [swapper/68:0] [ 3911.205181] Call trace: [ 3911.212474] Modules linked in: nvme_loop nvme_fabrics [ 3911.214907] __switch_to+0xd4/0x100 [ 3911.219946] nvmet nls_utf8 [ 3911.223420] 0x0 [ 3911.226201] hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3911.228041] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3911.310968] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3911.318784] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3911.328074] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3911.335022] pc : __sbq_wake_up+0x4/0x1f0 [ 3911.338932] lr : sbitmap_queue_clear+0x74/0xc0 [ 3911.343363] sp : ffff800008843900 [ 3911.346664] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3911.353786] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3911.360907] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3911.368030] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3911.375152] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3911.382274] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3911.389395] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3911.396517] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3911.403639] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3911.410761] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 3911.417883] Call trace: [ 3911.420317] __sbq_wake_up+0x4/0x1f0 [ 3911.423879] blk_mq_put_tag+0x44/0x70 [ 3911.427529] __blk_mq_free_request+0x88/0xf0 [ 3911.431786] blk_mq_free_request+0xc0/0x160 [ 3911.435955] __blk_mq_end_request+0x78/0x150 [ 3911.440212] scsi_end_request+0x114/0x260 [ 3911.444209] scsi_io_completion+0x64/0x174 [ 3911.448293] scsi_finish_command+0xe0/0x120 [ 3911.452464] scsi_complete+0x8c/0x1c4 [ 3911.456113] blk_complete_reqs+0x5c/0x74 [ 3911.460022] blk_done_softirq+0x38/0x4c [ 3911.463845] __do_softirq+0x168/0x418 [ 3911.467493] __irq_exit_rcu+0x120/0x170 [ 3911.471316] irq_exit_rcu+0x18/0x24 [ 3911.474792] el1_interrupt+0x38/0x70 [ 3911.478354] el1h_64_irq_handler+0x18/0x24 [ 3911.482437] el1h_64_irq+0x68/0x6c [ 3911.485826] cpuidle_enter_state+0xc4/0x4e0 [ 3911.489995] cpuidle_enter+0x40/0x60 [ 3911.493558] cpuidle_idle_call+0x118/0x1b0 [ 3911.497641] do_idle+0xac/0x100 [ 3911.500770] cpu_startup_entry+0x30/0x34 [ 3911.504680] secondary_start_kernel+0xd8/0xf0 [ 3911.509024] __secondary_switched+0xb0/0xb4 [ 3933.148706] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 348221 jiffies s: 2093 root: 0x11/. [ 3933.160445] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3933.169653] Task dump for CPU 8: [ 3933.172867] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3933.182769] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3933.188938] Call trace: [ 3933.191371] __switch_to+0xd4/0x100 [ 3933.194847] 0xffff6c0200000080 [ 3933.197976] Task dump for CPU 68: [ 3933.201279] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3933.211181] Call trace: [ 3933.213614] __switch_to+0xd4/0x100 [ 3933.217090] 0x0 [ 3935.088684] watchdog: BUG: soft lockup - CPU#8 stuck for 3270s! [kworker/8:12:19399] [ 3935.096414] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3935.096432] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3935.184220] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3935.192469] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3935.198683] watchdog: BUG: soft lockup - CPU#68 stuck for 3270s! [swapper/68:0] [ 3935.201760] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3935.209054] Modules linked in: nvme_loop [ 3935.215220] [ 3935.215220] nvme_fabrics [ 3935.219130] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3935.220607] nvmet [ 3935.223214] pc : __sbq_wake_up+0x1c8/0x1f0 [ 3935.230160] nls_utf8 [ 3935.232159] lr : sbitmap_queue_clear+0x74/0xc0 [ 3935.236241] hfsplus [ 3935.238500] sp : ffff80000d79b280 [ 3935.242929] hfs [ 3935.245102] x29: ffff80000d79b280 [ 3935.248402] isofs [ 3935.250228] x28: ffffd5becb6aec50 [ 3935.253529] nvme_core [ 3935.255527] x27: ffffd5beca4f7000 [ 3935.258915] nvme_common [ 3935.261261] [ 3935.264648] loop [ 3935.267168] x26: ffffd5becaf68a98 [ 3935.268646] tls [ 3935.270558] x25: ffffd5becb6a60c0 [ 3935.273859] rfkill [ 3935.275684] x24: fffffbffef180dc0 [ 3935.279071] sunrpc [ 3935.281157] [ 3935.284544] vfat [ 3935.286630] x23: ffff07ff81d381b0 [ 3935.288108] fat [ 3935.290020] x22: ffff80000d79b30c [ 3935.293320] acpi_ipmi [ 3935.295145] x21: ffff07ff84e93c00 [ 3935.298533] ipmi_ssif [ 3935.300879] [ 3935.304266] cppc_cpufreq [ 3935.306613] x20: 00000000000001c0 [ 3935.308092] ipmi_devintf [ 3935.310698] x19: ffff07ff91ccdb10 [ 3935.313999] igb [ 3935.316605] x18: ffffffffffffffff [ 3935.319993] ipmi_msghandler [ 3935.321818] [ 3935.325205] arm_cmn [ 3935.328072] x17: ffff32506c412000 [ 3935.329550] arm_dmc620_pmu [ 3935.331722] x16: ffff800008044000 [ 3935.335023] arm_spe_pmu [ 3935.337803] x15: 0000000000004000 [ 3935.341191] arm_dsu_pmu [ 3935.343710] [ 3935.347098] acpi_tad [ 3935.349617] x14: 010320bb28ec0000 [ 3935.351095] fuse [ 3935.353354] x13: 0000000000000087 [ 3935.356655] zram [ 3935.358567] x12: 071c71c71c71c71c [ 3935.361954] xfs [ 3935.363866] [ 3935.367254] crct10dif_ce [ 3935.369079] x11: 0000000000000040 [ 3935.370557] polyval_ce [ 3935.373164] x10: ffff07ff801a9c70 [ 3935.376465] ast [ 3935.378898] x9 : ffffd5bec9545fd8 [ 3935.382285] polyval_generic [ 3935.384110] [ 3935.387498] drm_vram_helper [ 3935.390365] x8 : ffff07ff80404b70 [ 3935.391843] ghash_ce [ 3935.394709] x7 : ffff07ff84e93dc8 [ 3935.398010] drm_ttm_helper [ 3935.400271] x6 : 0000000000000008 [ 3935.403658] sbsa_gwdt [ 3935.406438] [ 3935.409826] ttm [ 3935.412172] x5 : ffff80000d043790 [ 3935.413650] xgene_hwmon [ 3935.415475] x4 : ffff07ff84e93dc0 [ 3935.418776] Unloaded tainted modules: [ 3935.421295] x3 : 0000000000000008 [ 3935.424683] nvmet():1 [ 3935.428331] [ 3935.431718] nvme_fabrics():1 [ 3935.434066] x2 : 0000000000000007 [ 3935.435544] nvme_loop():1 [ 3935.438497] x1 : 0000000000000000 [ 3935.441798] nvmet():1 [ 3935.444491] x0 : 0000000000000007 [ 3935.447879] nvme_fabrics():1 [ 3935.450224] [ 3935.453612] nvme_loop():1 [ 3935.456566] Call trace: [ 3935.458043] nvmet():1 [ 3935.460737] __sbq_wake_up+0x1c8/0x1f0 [ 3935.463169] nvme_fabrics():1 [ 3935.465515] sbitmap_queue_clear+0x74/0xc0 [ 3935.469250] nvme_loop():1 [ 3935.472204] blk_mq_put_tag+0x44/0x70 [ 3935.476286] nvmet():1 [ 3935.478979] __blk_mq_free_request+0x88/0xf0 [ 3935.482627] nvme_fabrics():1 [ 3935.484973] blk_mq_free_request+0xc0/0x160 [ 3935.489228] nvme_loop():1 [ 3935.492182] __blk_mq_end_request+0x78/0x150 [ 3935.496351] nvmet():1 [ 3935.499044] scsi_end_request+0x114/0x260 [ 3935.503299] nvme_fabrics():1 [ 3935.505645] scsi_io_completion+0x64/0x174 [ 3935.509641] nvme_loop():1 [ 3935.512594] scsi_finish_command+0xe0/0x120 [ 3935.516676] nvmet():1 [ 3935.519369] scsi_complete+0x8c/0x1c4 [ 3935.523538] nvme_fabrics():1 [ 3935.525884] blk_complete_reqs+0x5c/0x74 [ 3935.529532] nvme_loop():1 [ 3935.532485] blk_done_softirq+0x38/0x4c [ 3935.536394] nvmet():1 [ 3935.539087] __do_softirq+0x168/0x418 [ 3935.542909] nvme_fabrics():1 [ 3935.545255] __irq_exit_rcu+0x120/0x170 [ 3935.548903] nvme_loop():1 [ 3935.551856] irq_exit_rcu+0x18/0x24 [ 3935.555678] nvmet():1 [ 3935.558371] el1_interrupt+0x38/0x70 [ 3935.561845] nvme_fabrics():1 [ 3935.564191] el1h_64_irq_handler+0x18/0x24 [ 3935.567752] nvme_loop():1 [ 3935.570706] el1h_64_irq+0x68/0x6c [ 3935.574788] nvmet():1 [ 3935.577481] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3935.580869] nvme_fabrics():1 [ 3935.583215] __wake_up_common_lock+0x84/0xac [ 3935.587991] nvme_loop():1 [ 3935.590945] __wake_up+0x20/0x30 [ 3935.595201] nvmet():1 [ 3935.597894] __sbq_wake_up+0x13c/0x1f0 [ 3935.601108] nvme_fabrics():1 [ 3935.603455] sbitmap_queue_clear+0x74/0xc0 [ 3935.607190] nvme_loop():1 [ 3935.610143] blk_mq_put_tag+0x44/0x70 [ 3935.614225] null_blk():1 [ 3935.616918] __blk_mq_free_request+0x88/0xf0 [ 3935.620566] null_blk():1 [ 3935.623172] blk_mq_free_request+0xc0/0x160 [ 3935.627428] null_blk():1 [ 3935.630034] bfq_insert_request.isra.0+0x204/0x270 [ 3935.634203] null_blk():1 [ 3935.636810] bfq_insert_requests+0x70/0x90 [ 3935.641588] null_blk():1 [ 3935.644194] blk_mq_sched_insert_requests+0xac/0x250 [ 3935.648276] null_blk():1 [ 3935.650882] blk_mq_flush_plug_list+0x110/0x2cc [ 3935.655832] null_blk():1 [ 3935.658439] __blk_flush_plug+0x100/0x140 [ 3935.662955] scsi_debug():1 [ 3935.665561] blk_finish_plug+0x40/0x25c [ 3935.669556] scsi_debug():1 [ 3935.672336] __iomap_dio_rw+0x1b4/0x500 [ 3935.676158] ipmi_si():1 [ 3935.678938] iomap_dio_rw+0x18/0x50 [ 3935.682759] ipmi_si():1 [ 3935.685279] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3935.688753] [last unloaded: nvmet] [ 3935.691273] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3935.696570] [ 3935.700045] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3935.704735] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3935.706213] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3935.711250] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3935.719065] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3935.724188] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3935.733479] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3935.738602] pc : __sbq_wake_up+0x58/0x1f0 [ 3935.745549] process_one_work+0x1ec/0x470 [ 3935.750933] lr : sbitmap_queue_clear+0x74/0xc0 [ 3935.754928] worker_thread+0x74/0x410 [ 3935.758923] sp : ffff8000088438b0 [ 3935.763353] kthread+0xec/0x100 [ 3935.767001] x29: ffff8000088438b0 [ 3935.770301] ret_from_fork+0x10/0x20 [ 3935.773428] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3935.785588] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3935.792710] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3935.799832] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3935.806954] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3935.814076] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3935.821197] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3935.828319] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3935.835441] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3935.842563] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3935.849685] Call trace: [ 3935.852118] __sbq_wake_up+0x58/0x1f0 [ 3935.855767] sbitmap_queue_clear+0x74/0xc0 [ 3935.859850] blk_mq_put_tag+0x44/0x70 [ 3935.863501] __blk_mq_free_request+0x88/0xf0 [ 3935.867758] blk_mq_free_request+0xc0/0x160 [ 3935.871927] __blk_mq_end_request+0x78/0x150 [ 3935.876184] scsi_end_request+0x114/0x260 [ 3935.880180] scsi_io_completion+0x64/0x174 [ 3935.884264] scsi_finish_command+0xe0/0x120 [ 3935.888434] scsi_complete+0x8c/0x1c4 [ 3935.892083] blk_complete_reqs+0x5c/0x74 [ 3935.895993] blk_done_softirq+0x38/0x4c [ 3935.899815] __do_softirq+0x168/0x418 [ 3935.903464] __irq_exit_rcu+0x120/0x170 [ 3935.907286] irq_exit_rcu+0x18/0x24 [ 3935.910762] el1_interrupt+0x38/0x70 [ 3935.914324] el1h_64_irq_handler+0x18/0x24 [ 3935.918407] el1h_64_irq+0x68/0x6c [ 3935.921796] cpuidle_enter_state+0xc4/0x4e0 [ 3935.925966] cpuidle_enter+0x40/0x60 [ 3935.929528] cpuidle_idle_call+0x118/0x1b0 [ 3935.933612] do_idle+0xac/0x100 [ 3935.936741] cpu_startup_entry+0x30/0x34 [ 3935.940649] secondary_start_kernel+0xd8/0xf0 [ 3935.944994] __secondary_switched+0xb0/0xb4 [ 3959.088537] watchdog: BUG: soft lockup - CPU#8 stuck for 3293s! [kworker/8:12:19399] [ 3959.096267] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3959.096284] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3959.184071] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3959.192321] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3959.198536] watchdog: BUG: soft lockup - CPU#68 stuck for 3293s! [swapper/68:0] [ 3959.201612] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3959.208906] Modules linked in: nvme_loop [ 3959.215072] [ 3959.215073] nvme_fabrics [ 3959.218981] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3959.220459] nvmet [ 3959.223065] pc : __sbq_wake_up+0x58/0x1f0 [ 3959.230012] nls_utf8 [ 3959.232011] lr : sbitmap_queue_clear+0x74/0xc0 [ 3959.236008] hfsplus [ 3959.238267] sp : ffff80000d79b280 [ 3959.242696] hfs [ 3959.244869] x29: ffff80000d79b280 [ 3959.248169] isofs [ 3959.249995] x28: ffffd5becb6aec50 [ 3959.253295] nvme_core [ 3959.255294] x27: ffffd5beca4f7000 [ 3959.258682] nvme_common [ 3959.261028] [ 3959.264416] loop [ 3959.266936] x26: ffffd5becaf68a98 [ 3959.268414] tls [ 3959.270328] x25: ffffd5becb6a60c0 [ 3959.273628] rfkill [ 3959.275453] x24: fffffbffef180dc0 [ 3959.278841] sunrpc [ 3959.280926] [ 3959.284314] vfat [ 3959.286400] x23: ffff07ff81d381b0 [ 3959.287877] fat [ 3959.289789] x22: ffff80000d79b30c [ 3959.293090] acpi_ipmi [ 3959.294916] x21: ffff07ff84e93c00 [ 3959.298304] ipmi_ssif [ 3959.300650] [ 3959.304037] cppc_cpufreq [ 3959.306383] x20: 00000000000001c0 [ 3959.307861] ipmi_devintf [ 3959.310467] x19: ffff07ff91ccdb10 [ 3959.313768] igb [ 3959.316375] x18: ffffffffffffffff [ 3959.319762] ipmi_msghandler [ 3959.321587] [ 3959.324975] arm_cmn [ 3959.327841] x17: ffff32506c412000 [ 3959.329319] arm_dmc620_pmu [ 3959.331492] x16: ffff800008044000 [ 3959.334792] arm_spe_pmu [ 3959.337572] x15: 0000000000004000 [ 3959.340960] arm_dsu_pmu [ 3959.343479] [ 3959.346867] acpi_tad [ 3959.349387] x14: 010320bb28ec0000 [ 3959.350864] fuse [ 3959.353124] x13: 0000000000000087 [ 3959.356424] zram [ 3959.358336] x12: 071c71c71c71c71c [ 3959.361724] xfs [ 3959.363636] [ 3959.367023] crct10dif_ce [ 3959.368848] x11: 0000000000000040 [ 3959.370326] polyval_ce [ 3959.372932] x10: ffff07ff801a9c70 [ 3959.376233] ast [ 3959.378666] x9 : ffffd5bec9545fd8 [ 3959.382054] polyval_generic [ 3959.383879] [ 3959.387266] drm_vram_helper [ 3959.390133] x8 : ffff07ff80404b70 [ 3959.391611] ghash_ce [ 3959.394478] x7 : ffff07ff84e93dc8 [ 3959.397778] drm_ttm_helper [ 3959.400037] x6 : 0000000000000008 [ 3959.403425] sbsa_gwdt [ 3959.406205] [ 3959.409593] ttm [ 3959.411938] x5 : ffff80000d043790 [ 3959.413416] xgene_hwmon [ 3959.415241] x4 : ffff07ff84e93dc0 [ 3959.418542] Unloaded tainted modules: [ 3959.421062] x3 : 0000000000000008 [ 3959.424449] nvmet():1 [ 3959.428097] [ 3959.431485] nvme_fabrics():1 [ 3959.433831] x2 : 0000000000000007 [ 3959.435309] nvme_loop():1 [ 3959.438262] x1 : ffff80000d79b30c [ 3959.441563] nvmet():1 [ 3959.444256] x0 : ffff07ff84e93dd0 [ 3959.447644] nvme_fabrics():1 [ 3959.449989] [ 3959.453377] nvme_loop():1 [ 3959.456331] Call trace: [ 3959.457808] nvmet():1 [ 3959.460502] __sbq_wake_up+0x58/0x1f0 [ 3959.462936] nvme_fabrics():1 [ 3959.465282] sbitmap_queue_clear+0x74/0xc0 [ 3959.468930] nvme_loop():1 [ 3959.471883] blk_mq_put_tag+0x44/0x70 [ 3959.475965] nvmet():1 [ 3959.478659] __blk_mq_free_request+0x88/0xf0 [ 3959.482307] nvme_fabrics():1 [ 3959.484652] blk_mq_free_request+0xc0/0x160 [ 3959.488908] nvme_loop():1 [ 3959.491862] __blk_mq_end_request+0x78/0x150 [ 3959.496030] nvmet():1 [ 3959.498724] scsi_end_request+0x114/0x260 [ 3959.502980] nvme_fabrics():1 [ 3959.505326] scsi_io_completion+0x64/0x174 [ 3959.509321] nvme_loop():1 [ 3959.512274] scsi_finish_command+0xe0/0x120 [ 3959.516356] nvmet():1 [ 3959.519050] scsi_complete+0x8c/0x1c4 [ 3959.523218] nvme_fabrics():1 [ 3959.525565] blk_complete_reqs+0x5c/0x74 [ 3959.529213] nvme_loop():1 [ 3959.532166] blk_done_softirq+0x38/0x4c [ 3959.536075] nvmet():1 [ 3959.538768] __do_softirq+0x168/0x418 [ 3959.542589] nvme_fabrics():1 [ 3959.544935] __irq_exit_rcu+0x120/0x170 [ 3959.548583] nvme_loop():1 [ 3959.551537] irq_exit_rcu+0x18/0x24 [ 3959.555358] nvmet():1 [ 3959.558052] el1_interrupt+0x38/0x70 [ 3959.561526] nvme_fabrics():1 [ 3959.563872] el1h_64_irq_handler+0x18/0x24 [ 3959.567434] nvme_loop():1 [ 3959.570387] el1h_64_irq+0x68/0x6c [ 3959.574469] nvmet():1 [ 3959.577163] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3959.580550] nvme_fabrics():1 [ 3959.582896] __wake_up_common_lock+0x84/0xac [ 3959.587673] nvme_loop():1 [ 3959.590626] __wake_up+0x20/0x30 [ 3959.594882] nvmet():1 [ 3959.597575] __sbq_wake_up+0x13c/0x1f0 [ 3959.600789] nvme_fabrics():1 [ 3959.603135] sbitmap_queue_clear+0x74/0xc0 [ 3959.606870] nvme_loop():1 [ 3959.609823] blk_mq_put_tag+0x44/0x70 [ 3959.613905] null_blk():1 [ 3959.616599] __blk_mq_free_request+0x88/0xf0 [ 3959.620246] null_blk():1 [ 3959.622853] blk_mq_free_request+0xc0/0x160 [ 3959.627108] null_blk():1 [ 3959.629716] bfq_insert_request.isra.0+0x204/0x270 [ 3959.633885] null_blk():1 [ 3959.636492] bfq_insert_requests+0x70/0x90 [ 3959.641268] null_blk():1 [ 3959.643874] blk_mq_sched_insert_requests+0xac/0x250 [ 3959.647956] null_blk():1 [ 3959.650563] blk_mq_flush_plug_list+0x110/0x2cc [ 3959.655513] null_blk():1 [ 3959.658120] __blk_flush_plug+0x100/0x140 [ 3959.662636] scsi_debug():1 [ 3959.665242] blk_finish_plug+0x40/0x25c [ 3959.669237] scsi_debug():1 [ 3959.672017] __iomap_dio_rw+0x1b4/0x500 [ 3959.675839] ipmi_si():1 [ 3959.678620] iomap_dio_rw+0x18/0x50 [ 3959.682442] ipmi_si():1 [ 3959.684962] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3959.688436] [last unloaded: nvmet] [ 3959.690956] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3959.696253] [ 3959.699728] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3959.704417] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3959.705895] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3959.710932] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3959.718747] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3959.723870] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3959.733161] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3959.738284] pc : __sbq_wake_up+0x84/0x1f0 [ 3959.745231] process_one_work+0x1ec/0x470 [ 3959.750615] lr : sbitmap_queue_clear+0x74/0xc0 [ 3959.754610] worker_thread+0x74/0x410 [ 3959.758605] sp : ffff8000088438b0 [ 3959.763035] kthread+0xec/0x100 [ 3959.766683] x29: ffff8000088438b0 [ 3959.769984] ret_from_fork+0x10/0x20 [ 3959.773111] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3959.785272] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3959.792394] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 3959.799516] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3959.806638] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3959.813760] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3959.820882] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3959.828003] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3959.835125] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3959.842247] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 3959.849369] Call trace: [ 3959.851803] __sbq_wake_up+0x84/0x1f0 [ 3959.855452] sbitmap_queue_clear+0x74/0xc0 [ 3959.859535] blk_mq_put_tag+0x44/0x70 [ 3959.863185] __blk_mq_free_request+0x88/0xf0 [ 3959.867442] blk_mq_free_request+0xc0/0x160 [ 3959.871612] __blk_mq_end_request+0x78/0x150 [ 3959.875868] scsi_end_request+0x114/0x260 [ 3959.879865] scsi_io_completion+0x64/0x174 [ 3959.883948] scsi_finish_command+0xe0/0x120 [ 3959.888118] scsi_complete+0x8c/0x1c4 [ 3959.891767] blk_complete_reqs+0x5c/0x74 [ 3959.895677] blk_done_softirq+0x38/0x4c [ 3959.899499] __do_softirq+0x168/0x418 [ 3959.903148] __irq_exit_rcu+0x120/0x170 [ 3959.906971] irq_exit_rcu+0x18/0x24 [ 3959.910446] el1_interrupt+0x38/0x70 [ 3959.914008] el1h_64_irq_handler+0x18/0x24 [ 3959.918091] el1h_64_irq+0x68/0x6c [ 3959.921479] cpuidle_enter_state+0xc4/0x4e0 [ 3959.925649] cpuidle_enter+0x40/0x60 [ 3959.929212] cpuidle_idle_call+0x118/0x1b0 [ 3959.933295] do_idle+0xac/0x100 [ 3959.936425] cpu_startup_entry+0x30/0x34 [ 3959.940334] secondary_start_kernel+0xd8/0xf0 [ 3959.944678] __secondary_switched+0xb0/0xb4 [ 3983.088390] watchdog: BUG: soft lockup - CPU#8 stuck for 3315s! [kworker/8:12:19399] [ 3983.096120] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 3983.096137] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 3983.183925] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 3983.192175] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3983.198390] watchdog: BUG: soft lockup - CPU#68 stuck for 3315s! [swapper/68:0] [ 3983.201465] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3983.208759] Modules linked in: nvme_loop [ 3983.214925] [ 3983.214925] nvme_fabrics [ 3983.218835] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3983.220313] nvmet [ 3983.222920] pc : __sbq_wake_up+0x58/0x1f0 [ 3983.229866] nls_utf8 [ 3983.231865] lr : sbitmap_queue_clear+0x74/0xc0 [ 3983.235860] hfsplus [ 3983.238119] sp : ffff80000d79b280 [ 3983.242549] hfs [ 3983.244722] x29: ffff80000d79b280 [ 3983.248022] isofs [ 3983.249847] x28: ffffd5becb6aec50 [ 3983.253148] nvme_core [ 3983.255147] x27: ffffd5beca4f7000 [ 3983.258534] nvme_common [ 3983.260880] [ 3983.264268] loop [ 3983.266787] x26: ffffd5becaf68a98 [ 3983.268265] tls [ 3983.270178] x25: ffffd5becb6a60c0 [ 3983.273479] rfkill [ 3983.275304] x24: fffffbffef180dc0 [ 3983.278691] sunrpc [ 3983.280777] [ 3983.284164] vfat [ 3983.286250] x23: ffff07ff81d381b0 [ 3983.287728] fat [ 3983.289639] x22: ffff80000d79b30c [ 3983.292940] acpi_ipmi [ 3983.294765] x21: ffff07ff84e93c00 [ 3983.298153] ipmi_ssif [ 3983.300499] [ 3983.303886] cppc_cpufreq [ 3983.306232] x20: 00000000000001c0 [ 3983.307710] ipmi_devintf [ 3983.310317] x19: ffff07ff91ccdb10 [ 3983.313618] igb [ 3983.316224] x18: ffffffffffffffff [ 3983.319612] ipmi_msghandler [ 3983.321437] [ 3983.324824] arm_cmn [ 3983.327691] x17: ffff32506c412000 [ 3983.329169] arm_dmc620_pmu [ 3983.331343] x16: ffff800008044000 [ 3983.334644] arm_spe_pmu [ 3983.337424] x15: 0000000000004000 [ 3983.340812] arm_dsu_pmu [ 3983.343331] [ 3983.346719] acpi_tad [ 3983.349238] x14: 010320bb28ec0000 [ 3983.350716] fuse [ 3983.352975] x13: 0000000000000087 [ 3983.356276] zram [ 3983.358188] x12: 071c71c71c71c71c [ 3983.361575] xfs [ 3983.363487] [ 3983.366875] crct10dif_ce [ 3983.368700] x11: 0000000000000040 [ 3983.370178] polyval_ce [ 3983.372784] x10: ffff07ff801a9c70 [ 3983.376085] ast [ 3983.378518] x9 : ffffd5bec9545fd8 [ 3983.381905] polyval_generic [ 3983.383730] [ 3983.387118] drm_vram_helper [ 3983.389985] x8 : ffff07ff80404b70 [ 3983.391463] ghash_ce [ 3983.394329] x7 : ffff07ff84e93dc8 [ 3983.397630] drm_ttm_helper [ 3983.399889] x6 : 0000000000000008 [ 3983.403277] sbsa_gwdt [ 3983.406057] [ 3983.409444] ttm [ 3983.411790] x5 : ffff80000d043790 [ 3983.413268] xgene_hwmon [ 3983.415093] x4 : ffff07ff84e93dc0 [ 3983.418394] Unloaded tainted modules: [ 3983.420913] x3 : 0000000000000008 [ 3983.424301] nvmet():1 [ 3983.427949] [ 3983.431337] nvme_fabrics():1 [ 3983.433682] x2 : 0000000000000007 [ 3983.435160] nvme_loop():1 [ 3983.438114] x1 : ffff80000d79b30c [ 3983.441415] nvmet():1 [ 3983.444109] x0 : ffff07ff84e93dd0 [ 3983.447497] nvme_fabrics():1 [ 3983.449843] [ 3983.453230] nvme_loop():1 [ 3983.456184] Call trace: [ 3983.457662] nvmet():1 [ 3983.460355] __sbq_wake_up+0x58/0x1f0 [ 3983.462788] nvme_fabrics():1 [ 3983.465134] sbitmap_queue_clear+0x74/0xc0 [ 3983.468782] nvme_loop():1 [ 3983.471735] blk_mq_put_tag+0x44/0x70 [ 3983.475817] nvmet():1 [ 3983.478510] __blk_mq_free_request+0x88/0xf0 [ 3983.482158] nvme_fabrics():1 [ 3983.484504] blk_mq_free_request+0xc0/0x160 [ 3983.488760] nvme_loop():1 [ 3983.491713] __blk_mq_end_request+0x78/0x150 [ 3983.495882] nvmet():1 [ 3983.498575] scsi_end_request+0x114/0x260 [ 3983.502831] nvme_fabrics():1 [ 3983.505177] scsi_io_completion+0x64/0x174 [ 3983.509172] nvme_loop():1 [ 3983.512126] scsi_finish_command+0xe0/0x120 [ 3983.516208] nvmet():1 [ 3983.518901] scsi_complete+0x8c/0x1c4 [ 3983.523070] nvme_fabrics():1 [ 3983.525415] blk_complete_reqs+0x5c/0x74 [ 3983.529063] nvme_loop():1 [ 3983.532017] blk_done_softirq+0x38/0x4c [ 3983.535925] nvmet():1 [ 3983.538619] __do_softirq+0x168/0x418 [ 3983.542440] nvme_fabrics():1 [ 3983.544786] __irq_exit_rcu+0x120/0x170 [ 3983.548434] nvme_loop():1 [ 3983.551388] irq_exit_rcu+0x18/0x24 [ 3983.555209] nvmet():1 [ 3983.557902] el1_interrupt+0x38/0x70 [ 3983.561377] nvme_fabrics():1 [ 3983.563723] el1h_64_irq_handler+0x18/0x24 [ 3983.567284] nvme_loop():1 [ 3983.570237] el1h_64_irq+0x68/0x6c [ 3983.574319] nvmet():1 [ 3983.577013] _raw_spin_unlock_irqrestore+0x20/0x50 [ 3983.580402] nvme_fabrics():1 [ 3983.582748] __wake_up_common_lock+0x84/0xac [ 3983.587524] nvme_loop():1 [ 3983.590478] __wake_up+0x20/0x30 [ 3983.594733] nvmet():1 [ 3983.597426] __sbq_wake_up+0x13c/0x1f0 [ 3983.600640] nvme_fabrics():1 [ 3983.602986] sbitmap_queue_clear+0x74/0xc0 [ 3983.606721] nvme_loop():1 [ 3983.609675] blk_mq_put_tag+0x44/0x70 [ 3983.613757] null_blk():1 [ 3983.616450] __blk_mq_free_request+0x88/0xf0 [ 3983.620098] null_blk():1 [ 3983.622704] blk_mq_free_request+0xc0/0x160 [ 3983.626960] null_blk():1 [ 3983.629566] bfq_insert_request.isra.0+0x204/0x270 [ 3983.633736] null_blk():1 [ 3983.636343] bfq_insert_requests+0x70/0x90 [ 3983.641119] null_blk():1 [ 3983.643726] blk_mq_sched_insert_requests+0xac/0x250 [ 3983.647808] null_blk():1 [ 3983.650414] blk_mq_flush_plug_list+0x110/0x2cc [ 3983.655364] null_blk():1 [ 3983.657970] __blk_flush_plug+0x100/0x140 [ 3983.662486] scsi_debug():1 [ 3983.665093] blk_finish_plug+0x40/0x25c [ 3983.669088] scsi_debug():1 [ 3983.671868] __iomap_dio_rw+0x1b4/0x500 [ 3983.675689] ipmi_si():1 [ 3983.678469] iomap_dio_rw+0x18/0x50 [ 3983.682291] ipmi_si():1 [ 3983.684810] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 3983.688285] [last unloaded: nvmet] [ 3983.690804] xfs_file_write_iter+0xf4/0x15c [xfs] [ 3983.696102] [ 3983.699576] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 3983.704266] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 3983.705744] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 3983.710780] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 3983.718596] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 3983.723719] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3983.733010] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 3983.738135] pc : __sbq_wake_up+0x20/0x1f0 [ 3983.745081] process_one_work+0x1ec/0x470 [ 3983.750466] lr : sbitmap_queue_clear+0x74/0xc0 [ 3983.754461] worker_thread+0x74/0x410 [ 3983.758456] sp : ffff8000088438b0 [ 3983.762886] kthread+0xec/0x100 [ 3983.766534] x29: ffff8000088438b0 [ 3983.769835] ret_from_fork+0x10/0x20 [ 3983.772962] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 3983.785121] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 3983.792243] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 3983.799366] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 3983.806488] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 3983.813610] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 3983.820732] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 3983.827854] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 3983.834976] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 3983.842098] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 3983.849219] Call trace: [ 3983.851652] __sbq_wake_up+0x20/0x1f0 [ 3983.855301] sbitmap_queue_clear+0x74/0xc0 [ 3983.859384] blk_mq_put_tag+0x44/0x70 [ 3983.863034] __blk_mq_free_request+0x88/0xf0 [ 3983.867291] blk_mq_free_request+0xc0/0x160 [ 3983.871461] __blk_mq_end_request+0x78/0x150 [ 3983.875717] scsi_end_request+0x114/0x260 [ 3983.879714] scsi_io_completion+0x64/0x174 [ 3983.883797] scsi_finish_command+0xe0/0x120 [ 3983.887967] scsi_complete+0x8c/0x1c4 [ 3983.891616] blk_complete_reqs+0x5c/0x74 [ 3983.895526] blk_done_softirq+0x38/0x4c [ 3983.899348] __do_softirq+0x168/0x418 [ 3983.902997] __irq_exit_rcu+0x120/0x170 [ 3983.906820] irq_exit_rcu+0x18/0x24 [ 3983.910295] el1_interrupt+0x38/0x70 [ 3983.913857] el1h_64_irq_handler+0x18/0x24 [ 3983.917940] el1h_64_irq+0x68/0x6c [ 3983.921329] cpuidle_enter_state+0xc4/0x4e0 [ 3983.925498] cpuidle_enter+0x40/0x60 [ 3983.929061] cpuidle_idle_call+0x118/0x1b0 [ 3983.933144] do_idle+0xac/0x100 [ 3983.936273] cpu_startup_entry+0x30/0x34 [ 3983.940182] secondary_start_kernel+0xd8/0xf0 [ 3983.944526] __secondary_switched+0xb0/0xb4 [ 3999.708299] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 354877 jiffies s: 2093 root: 0x11/. [ 3999.720037] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 3999.729245] Task dump for CPU 8: [ 3999.732459] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 3999.742361] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 3999.748530] Call trace: [ 3999.750964] __switch_to+0xd4/0x100 [ 3999.754440] 0xffff6c0200000080 [ 3999.757568] Task dump for CPU 68: [ 3999.760872] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 3999.770773] Call trace: [ 3999.773206] __switch_to+0xd4/0x100 [ 3999.776682] 0x0 [ 4007.088243] watchdog: BUG: soft lockup - CPU#8 stuck for 3337s! [kworker/8:12:19399] [ 4007.095973] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4007.095990] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4007.183778] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4007.192028] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4007.198243] watchdog: BUG: soft lockup - CPU#68 stuck for 3337s! [swapper/68:0] [ 4007.201319] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4007.208613] Modules linked in: [ 4007.208614] [ 4007.208614] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4007.214780] nvme_loop [ 4007.217820] pc : __sbq_wake_up+0x58/0x1f0 [ 4007.219298] nvme_fabrics [ 4007.226244] lr : sbitmap_queue_clear+0x74/0xc0 [ 4007.228590] nvmet [ 4007.232586] sp : ffff80000d79b280 [ 4007.235192] nls_utf8 [ 4007.239622] x29: ffff80000d79b280 [ 4007.241621] hfsplus [ 4007.244922] x28: ffffd5becb6aec50 [ 4007.247181] hfs [ 4007.250481] x27: ffffd5beca4f7000 [ 4007.252654] isofs [ 4007.256041] [ 4007.257867] nvme_core [ 4007.261255] x26: ffffd5becaf68a98 [ 4007.263253] nvme_common [ 4007.264731] x25: ffffd5becb6a60c0 [ 4007.267077] loop [ 4007.270378] x24: fffffbffef180dc0 [ 4007.272897] tls [ 4007.276285] [ 4007.278197] rfkill [ 4007.281585] x23: ffff07ff81d381b0 [ 4007.283410] sunrpc [ 4007.284888] x22: ffff80000d79b30c [ 4007.286973] vfat [ 4007.290274] x21: ffff07ff84e93c00 [ 4007.292360] fat [ 4007.295748] [ 4007.297660] acpi_ipmi [ 4007.301048] x20: 00000000000001c0 [ 4007.302873] ipmi_ssif [ 4007.304351] x19: ffff07ff91ccdb10 [ 4007.306697] cppc_cpufreq [ 4007.309997] x18: ffffffffffffffff [ 4007.312343] ipmi_devintf [ 4007.315731] [ 4007.318337] igb [ 4007.321725] x17: ffff32506c412000 [ 4007.324331] ipmi_msghandler [ 4007.325809] x16: ffff800008044000 [ 4007.327634] arm_cmn [ 4007.330935] x15: 0000000000004000 [ 4007.333801] arm_dmc620_pmu [ 4007.337189] [ 4007.339361] arm_spe_pmu [ 4007.342750] x14: 010320bb28ec0000 [ 4007.345529] arm_dsu_pmu [ 4007.347007] x13: 0000000000000087 [ 4007.349527] acpi_tad [ 4007.352828] x12: 071c71c71c71c71c [ 4007.355348] fuse [ 4007.358735] [ 4007.360994] zram [ 4007.364382] x11: 0000000000000040 [ 4007.366294] xfs [ 4007.367772] x10: ffff07ff801a9c70 [ 4007.369684] crct10dif_ce [ 4007.372984] x9 : ffffd5bec9545fd8 [ 4007.374810] polyval_ce [ 4007.378197] [ 4007.380804] ast [ 4007.384191] x8 : ffff07ff80404b70 [ 4007.386624] polyval_generic [ 4007.388102] x7 : ffff07ff84e93dc8 [ 4007.389927] drm_vram_helper [ 4007.393228] x6 : 0000000000000008 [ 4007.396095] ghash_ce [ 4007.399482] [ 4007.402349] drm_ttm_helper [ 4007.405736] x5 : ffff80000d043790 [ 4007.407996] sbsa_gwdt [ 4007.409473] x4 : ffff07ff84e93dc0 [ 4007.412253] ttm [ 4007.415554] x3 : 0000000000000008 [ 4007.417900] xgene_hwmon [ 4007.421288] [ 4007.423113] Unloaded tainted modules: [ 4007.426502] x2 : 0000000000000007 [ 4007.429022] nvmet():1 [ 4007.430500] x1 : ffff80000d79b30c [ 4007.434148] nvme_fabrics():1 [ 4007.437449] x0 : ffff07ff84e93dd0 [ 4007.439794] nvme_loop():1 [ 4007.443182] [ 4007.446136] nvmet():1 [ 4007.449523] Call trace: [ 4007.452216] nvme_fabrics():1 [ 4007.453694] __sbq_wake_up+0x58/0x1f0 [ 4007.456040] nvme_loop():1 [ 4007.458473] sbitmap_queue_clear+0x74/0xc0 [ 4007.461426] nvmet():1 [ 4007.465074] blk_mq_put_tag+0x44/0x70 [ 4007.467767] nvme_fabrics():1 [ 4007.471849] __blk_mq_free_request+0x88/0xf0 [ 4007.474195] nvme_loop():1 [ 4007.477843] blk_mq_free_request+0xc0/0x160 [ 4007.480797] nvmet():1 [ 4007.485053] __blk_mq_end_request+0x78/0x150 [ 4007.487746] nvme_fabrics():1 [ 4007.491915] scsi_end_request+0x114/0x260 [ 4007.494262] nvme_loop():1 [ 4007.498518] scsi_io_completion+0x64/0x174 [ 4007.501471] nvmet():1 [ 4007.505466] scsi_finish_command+0xe0/0x120 [ 4007.508160] nvme_fabrics():1 [ 4007.512242] scsi_complete+0x8c/0x1c4 [ 4007.514587] nvme_loop():1 [ 4007.518756] blk_complete_reqs+0x5c/0x74 [ 4007.521710] nvmet():1 [ 4007.525358] blk_done_softirq+0x38/0x4c [ 4007.528051] nvme_fabrics():1 [ 4007.531959] __do_softirq+0x168/0x418 [ 4007.534305] nvme_loop():1 [ 4007.538127] __irq_exit_rcu+0x120/0x170 [ 4007.541080] nvmet():1 [ 4007.544728] irq_exit_rcu+0x18/0x24 [ 4007.547422] nvme_fabrics():1 [ 4007.551245] el1_interrupt+0x38/0x70 [ 4007.553591] nvme_loop():1 [ 4007.557066] el1h_64_irq_handler+0x18/0x24 [ 4007.560019] nvmet():1 [ 4007.563580] el1h_64_irq+0x68/0x6c [ 4007.566273] nvme_fabrics():1 [ 4007.570356] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4007.572701] nvme_loop():1 [ 4007.576089] __wake_up_common_lock+0x84/0xac [ 4007.579043] nvmet():1 [ 4007.583819] __wake_up+0x20/0x30 [ 4007.586512] nvme_fabrics():1 [ 4007.590768] __sbq_wake_up+0x13c/0x1f0 [ 4007.593114] nvme_loop():1 [ 4007.596329] sbitmap_queue_clear+0x74/0xc0 [ 4007.599283] nvmet():1 [ 4007.603018] blk_mq_put_tag+0x44/0x70 [ 4007.605711] nvme_fabrics():1 [ 4007.609793] __blk_mq_free_request+0x88/0xf0 [ 4007.612139] nvme_loop():1 [ 4007.615787] blk_mq_free_request+0xc0/0x160 [ 4007.618740] null_blk():1 [ 4007.622996] bfq_insert_request.isra.0+0x204/0x270 [ 4007.625689] null_blk():1 [ 4007.629858] bfq_insert_requests+0x70/0x90 [ 4007.632464] null_blk():1 [ 4007.637241] blk_mq_sched_insert_requests+0xac/0x250 [ 4007.639847] null_blk():1 [ 4007.643929] blk_mq_flush_plug_list+0x110/0x2cc [ 4007.646535] null_blk():1 [ 4007.651485] __blk_flush_plug+0x100/0x140 [ 4007.654092] null_blk():1 [ 4007.658608] blk_finish_plug+0x40/0x25c [ 4007.661214] null_blk():1 [ 4007.665209] __iomap_dio_rw+0x1b4/0x500 [ 4007.667816] scsi_debug():1 [ 4007.671637] iomap_dio_rw+0x18/0x50 [ 4007.674243] scsi_debug():1 [ 4007.678065] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4007.680845] ipmi_si():1 [ 4007.684319] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4007.687099] ipmi_si():1 [ 4007.692397] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4007.694917] [last unloaded: nvmet] [ 4007.699606] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4007.702126] [ 4007.707163] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4007.710637] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4007.715761] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4007.717239] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4007.722362] process_one_work+0x1ec/0x470 [ 4007.730177] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4007.735561] worker_thread+0x74/0x410 [ 4007.744851] pc : __sbq_wake_up+0x40/0x1f0 [ 4007.748847] kthread+0xec/0x100 [ 4007.755793] lr : sbitmap_queue_clear+0x74/0xc0 [ 4007.759441] ret_from_fork+0x10/0x20 [ 4007.763436] sp : ffff8000088438b0 [ 4007.777855] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4007.784977] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4007.792099] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4007.799221] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4007.806343] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4007.813465] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4007.820587] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4007.827708] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4007.834831] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4007.841953] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4007.849075] Call trace: [ 4007.851508] __sbq_wake_up+0x40/0x1f0 [ 4007.855157] sbitmap_queue_clear+0x74/0xc0 [ 4007.859241] blk_mq_put_tag+0x44/0x70 [ 4007.862891] __blk_mq_free_request+0x88/0xf0 [ 4007.867147] blk_mq_free_request+0xc0/0x160 [ 4007.871317] __blk_mq_end_request+0x78/0x150 [ 4007.875574] scsi_end_request+0x114/0x260 [ 4007.879570] scsi_io_completion+0x64/0x174 [ 4007.883653] scsi_finish_command+0xe0/0x120 [ 4007.887824] scsi_complete+0x8c/0x1c4 [ 4007.891473] blk_complete_reqs+0x5c/0x74 [ 4007.895382] blk_done_softirq+0x38/0x4c [ 4007.899205] __do_softirq+0x168/0x418 [ 4007.902854] __irq_exit_rcu+0x120/0x170 [ 4007.906676] irq_exit_rcu+0x18/0x24 [ 4007.910152] el1_interrupt+0x38/0x70 [ 4007.913714] el1h_64_irq_handler+0x18/0x24 [ 4007.917797] el1h_64_irq+0x68/0x6c [ 4007.921185] cpuidle_enter_state+0xc4/0x4e0 [ 4007.925355] cpuidle_enter+0x40/0x60 [ 4007.928918] cpuidle_idle_call+0x118/0x1b0 [ 4007.933002] do_idle+0xac/0x100 [ 4007.936130] cpu_startup_entry+0x30/0x34 [ 4007.940039] secondary_start_kernel+0xd8/0xf0 [ 4007.944383] __secondary_switched+0xb0/0xb4 [ 4031.088096] watchdog: BUG: soft lockup - CPU#8 stuck for 3360s! [kworker/8:12:19399] [ 4031.095826] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4031.095844] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4031.183632] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4031.191882] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4031.198096] watchdog: BUG: soft lockup - CPU#68 stuck for 3360s! [swapper/68:0] [ 4031.201173] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4031.208467] Modules linked in: nvme_loop [ 4031.214634] [ 4031.214634] nvme_fabrics nvmet [ 4031.218543] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4031.220021] nls_utf8 [ 4031.223148] pc : __sbq_wake_up+0x48/0x1f0 [ 4031.230095] hfsplus [ 4031.232354] lr : sbitmap_queue_clear+0x74/0xc0 [ 4031.236349] hfs [ 4031.238521] sp : ffff80000d79b280 [ 4031.242951] isofs [ 4031.244776] x29: ffff80000d79b280 [ 4031.248077] nvme_core [ 4031.250076] x28: ffffd5becb6aec50 [ 4031.253376] nvme_common [ 4031.255722] x27: ffffd5beca4f7000 [ 4031.259110] loop [ 4031.261629] [ 4031.265017] tls [ 4031.266930] x26: ffffd5becaf68a98 [ 4031.268408] rfkill [ 4031.270233] x25: ffffd5becb6a60c0 [ 4031.273534] sunrpc [ 4031.275620] x24: fffffbffef180dc0 [ 4031.279008] vfat [ 4031.281094] [ 4031.284481] fat [ 4031.286393] x23: ffff07ff81d381b0 [ 4031.287871] acpi_ipmi [ 4031.289696] x22: ffff80000d79b30c [ 4031.292997] ipmi_ssif [ 4031.295343] x21: ffff07ff84e93c00 [ 4031.298730] cppc_cpufreq [ 4031.301076] [ 4031.304464] ipmi_devintf [ 4031.307070] x20: 00000000000001c0 [ 4031.308548] igb [ 4031.311155] x19: ffff07ff91ccdb10 [ 4031.314455] ipmi_msghandler [ 4031.316280] x18: ffffffffffffffff [ 4031.319668] arm_cmn [ 4031.322535] [ 4031.325922] arm_dmc620_pmu [ 4031.328094] x17: ffff32506c412000 [ 4031.329572] arm_spe_pmu [ 4031.332352] x16: ffff800008044000 [ 4031.335653] arm_dsu_pmu [ 4031.338173] x15: 0000000000004000 [ 4031.341560] acpi_tad [ 4031.344080] [ 4031.347467] fuse [ 4031.349726] x14: 010320bb28ec0000 [ 4031.351204] zram [ 4031.353116] x13: 0000000000000087 [ 4031.356418] xfs [ 4031.358330] x12: 071c71c71c71c71c [ 4031.361717] crct10dif_ce [ 4031.363542] [ 4031.366930] polyval_ce [ 4031.369536] x11: 0000000000000040 [ 4031.371014] ast [ 4031.373449] x10: ffff07ff801a9c70 [ 4031.376749] polyval_generic [ 4031.378575] x9 : ffffd5bec9545fd8 [ 4031.381963] drm_vram_helper [ 4031.384829] [ 4031.388217] ghash_ce [ 4031.391084] x8 : ffff07ff80404b70 [ 4031.392561] drm_ttm_helper [ 4031.394821] x7 : ffff07ff84e93dc8 [ 4031.398121] sbsa_gwdt [ 4031.400901] x6 : 0000000000000008 [ 4031.404289] ttm [ 4031.406635] [ 4031.410022] xgene_hwmon [ 4031.411847] x5 : ffff80000d043790 [ 4031.413325] Unloaded tainted modules: [ 4031.415845] x4 : ffff07ff84e93dc0 [ 4031.419146] nvmet():1 [ 4031.422793] x3 : 0000000000000008 [ 4031.426181] nvme_fabrics():1 [ 4031.428527] [ 4031.431915] nvme_loop():1 [ 4031.434870] x2 : 0000000000000007 [ 4031.436348] nvmet():1 [ 4031.439042] x1 : ffff80000d79b30c [ 4031.442343] nvme_fabrics():1 [ 4031.444688] x0 : 0000000000000039 [ 4031.448076] nvme_loop():1 [ 4031.451030] [ 4031.454417] nvmet():1 [ 4031.457112] Call trace: [ 4031.458590] nvme_fabrics():1 [ 4031.460936] __sbq_wake_up+0x48/0x1f0 [ 4031.463368] nvme_loop():1 [ 4031.466322] sbitmap_queue_clear+0x74/0xc0 [ 4031.469971] nvmet():1 [ 4031.472664] blk_mq_put_tag+0x44/0x70 [ 4031.476746] nvme_fabrics():1 [ 4031.479092] __blk_mq_free_request+0x88/0xf0 [ 4031.482740] nvme_loop():1 [ 4031.485693] blk_mq_free_request+0xc0/0x160 [ 4031.489949] nvmet():1 [ 4031.492642] __blk_mq_end_request+0x78/0x150 [ 4031.496811] nvme_fabrics():1 [ 4031.499157] scsi_end_request+0x114/0x260 [ 4031.503413] nvme_loop():1 [ 4031.506366] scsi_io_completion+0x64/0x174 [ 4031.510362] nvmet():1 [ 4031.513055] scsi_finish_command+0xe0/0x120 [ 4031.517137] nvme_fabrics():1 [ 4031.519484] scsi_complete+0x8c/0x1c4 [ 4031.523653] nvme_loop():1 [ 4031.526606] blk_complete_reqs+0x5c/0x74 [ 4031.530254] nvmet():1 [ 4031.532947] blk_done_softirq+0x38/0x4c [ 4031.536856] nvme_fabrics():1 [ 4031.539201] __do_softirq+0x168/0x418 [ 4031.543023] nvme_loop():1 [ 4031.545977] __irq_exit_rcu+0x120/0x170 [ 4031.549625] nvmet():1 [ 4031.552319] irq_exit_rcu+0x18/0x24 [ 4031.556141] nvme_fabrics():1 [ 4031.558487] el1_interrupt+0x38/0x70 [ 4031.561961] nvme_loop():1 [ 4031.564915] el1h_64_irq_handler+0x18/0x24 [ 4031.568476] nvmet():1 [ 4031.571169] el1h_64_irq+0x68/0x6c [ 4031.575251] nvme_fabrics():1 [ 4031.577597] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4031.580985] nvme_loop():1 [ 4031.583938] __wake_up_common_lock+0x84/0xac [ 4031.588715] nvmet():1 [ 4031.591408] __wake_up+0x20/0x30 [ 4031.595663] nvme_fabrics():1 [ 4031.598009] __sbq_wake_up+0x13c/0x1f0 [ 4031.601223] nvme_loop():1 [ 4031.604177] sbitmap_queue_clear+0x74/0xc0 [ 4031.607912] null_blk():1 [ 4031.610605] blk_mq_put_tag+0x44/0x70 [ 4031.614687] null_blk():1 [ 4031.617294] __blk_mq_free_request+0x88/0xf0 [ 4031.620942] null_blk():1 [ 4031.623548] blk_mq_free_request+0xc0/0x160 [ 4031.627804] null_blk():1 [ 4031.630410] bfq_insert_request.isra.0+0x204/0x270 [ 4031.634579] null_blk():1 [ 4031.637185] bfq_insert_requests+0x70/0x90 [ 4031.641962] null_blk():1 [ 4031.644568] blk_mq_sched_insert_requests+0xac/0x250 [ 4031.648650] null_blk():1 [ 4031.651257] blk_mq_flush_plug_list+0x110/0x2cc [ 4031.656207] scsi_debug():1 [ 4031.658813] __blk_flush_plug+0x100/0x140 [ 4031.663329] scsi_debug():1 [ 4031.666109] blk_finish_plug+0x40/0x25c [ 4031.670104] ipmi_si():1 [ 4031.672884] __iomap_dio_rw+0x1b4/0x500 [ 4031.676706] ipmi_si():1 [ 4031.679225] iomap_dio_rw+0x18/0x50 [ 4031.683047] [last unloaded: nvmet] [ 4031.685566] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4031.689041] [ 4031.692515] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4031.697814] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4031.699292] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4031.703982] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4031.711796] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4031.716833] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4031.726123] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4031.731247] pc : __sbq_wake_up+0x20/0x1f0 [ 4031.738194] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4031.743318] lr : sbitmap_queue_clear+0x74/0xc0 [ 4031.747315] process_one_work+0x1ec/0x470 [ 4031.752699] sp : ffff8000088438b0 [ 4031.757129] worker_thread+0x74/0x410 [ 4031.761124] x29: ffff8000088438b0 [ 4031.764425] kthread+0xec/0x100 [ 4031.768073] x28: ffffd5becb6aec50 [ 4031.771374] ret_from_fork+0x10/0x20 [ 4031.774501] x27: ffffd5beca4f7000 [ 4031.784837] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4031.791960] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4031.799082] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4031.806204] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4031.813326] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4031.820448] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4031.827569] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4031.834691] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4031.841814] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4031.848937] Call trace: [ 4031.851370] __sbq_wake_up+0x20/0x1f0 [ 4031.855019] sbitmap_queue_clear+0x74/0xc0 [ 4031.859102] blk_mq_put_tag+0x44/0x70 [ 4031.862752] __blk_mq_free_request+0x88/0xf0 [ 4031.867008] blk_mq_free_request+0xc0/0x160 [ 4031.871178] __blk_mq_end_request+0x78/0x150 [ 4031.875436] scsi_end_request+0x114/0x260 [ 4031.879432] scsi_io_completion+0x64/0x174 [ 4031.883515] scsi_finish_command+0xe0/0x120 [ 4031.887686] scsi_complete+0x8c/0x1c4 [ 4031.891335] blk_complete_reqs+0x5c/0x74 [ 4031.895245] blk_done_softirq+0x38/0x4c [ 4031.899068] __do_softirq+0x168/0x418 [ 4031.902717] __irq_exit_rcu+0x120/0x170 [ 4031.906539] irq_exit_rcu+0x18/0x24 [ 4031.910015] el1_interrupt+0x38/0x70 [ 4031.913577] el1h_64_irq_handler+0x18/0x24 [ 4031.917660] el1h_64_irq+0x68/0x6c [ 4031.921048] cpuidle_enter_state+0xc4/0x4e0 [ 4031.925219] cpuidle_enter+0x40/0x60 [ 4031.928781] cpuidle_idle_call+0x118/0x1b0 [ 4031.932865] do_idle+0xac/0x100 [ 4031.935993] cpu_startup_entry+0x30/0x34 [ 4031.939902] secondary_start_kernel+0xd8/0xf0 [ 4031.944247] __secondary_switched+0xb0/0xb4 [ 4055.087950] watchdog: BUG: soft lockup - CPU#8 stuck for 3382s! [kworker/8:12:19399] [ 4055.095679] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4055.095696] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4055.183484] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4055.191734] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4055.197949] watchdog: BUG: soft lockup - CPU#68 stuck for 3382s! [swapper/68:0] [ 4055.201025] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4055.208318] Modules linked in: [ 4055.214484] [ 4055.214485] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4055.217525] nvme_loop [ 4055.219003] pc : __sbq_wake_up+0x58/0x1f0 [ 4055.225950] nvme_fabrics [ 4055.228295] lr : sbitmap_queue_clear+0x74/0xc0 [ 4055.232291] nvmet [ 4055.234899] sp : ffff80000d79b280 [ 4055.239328] nls_utf8 [ 4055.241327] x29: ffff80000d79b280 [ 4055.244627] hfsplus [ 4055.246886] x28: ffffd5becb6aec50 [ 4055.250187] hfs [ 4055.252360] x27: ffffd5beca4f7000 [ 4055.255747] isofs [ 4055.257572] [ 4055.260960] nvme_core [ 4055.262958] x26: ffffd5becaf68a98 [ 4055.264436] nvme_common [ 4055.266782] x25: ffffd5becb6a60c0 [ 4055.270083] loop [ 4055.272602] x24: fffffbffef180dc0 [ 4055.275990] tls [ 4055.277902] [ 4055.281290] rfkill [ 4055.283115] x23: ffff07ff81d381b0 [ 4055.284592] sunrpc [ 4055.286678] x22: ffff80000d79b30c [ 4055.289979] vfat [ 4055.292064] x21: ffff07ff84e93c00 [ 4055.295452] fat [ 4055.297364] [ 4055.300751] acpi_ipmi [ 4055.302576] x20: 00000000000001c0 [ 4055.304054] ipmi_ssif [ 4055.306401] x19: ffff07ff91ccdb10 [ 4055.309701] cppc_cpufreq [ 4055.312047] x18: ffffffffffffffff [ 4055.315435] ipmi_devintf [ 4055.318041] [ 4055.321429] igb [ 4055.324035] x17: ffff32506c412000 [ 4055.325513] ipmi_msghandler [ 4055.327338] x16: ffff800008044000 [ 4055.330639] arm_cmn [ 4055.333506] x15: 0000000000004000 [ 4055.336893] arm_dmc620_pmu [ 4055.339066] [ 4055.342454] arm_spe_pmu [ 4055.345233] x14: 010320bb28ec0000 [ 4055.346711] arm_dsu_pmu [ 4055.349231] x13: 0000000000000087 [ 4055.352532] acpi_tad [ 4055.355051] x12: 071c71c71c71c71c [ 4055.358439] fuse [ 4055.360698] [ 4055.364085] zram [ 4055.365997] x11: 0000000000000040 [ 4055.367475] xfs [ 4055.369387] x10: ffff07ff801a9c70 [ 4055.372688] crct10dif_ce [ 4055.374513] x9 : ffffd5bec9545fd8 [ 4055.377900] polyval_ce [ 4055.380507] [ 4055.383894] ast [ 4055.386327] x8 : ffff07ff80404b70 [ 4055.387805] polyval_generic [ 4055.389630] x7 : ffff07ff84e93dc8 [ 4055.392931] drm_vram_helper [ 4055.395798] x6 : 0000000000000008 [ 4055.399185] ghash_ce [ 4055.402052] [ 4055.405439] drm_ttm_helper [ 4055.407699] x5 : ffff80000d043790 [ 4055.409176] sbsa_gwdt [ 4055.411956] x4 : ffff07ff84e93dc0 [ 4055.415257] ttm [ 4055.417604] x3 : 0000000000000008 [ 4055.420991] xgene_hwmon [ 4055.422817] [ 4055.426206] Unloaded tainted modules: [ 4055.428725] x2 : 0000000000000007 [ 4055.430203] nvmet():1 [ 4055.433851] x1 : ffff80000d79b30c [ 4055.437152] nvme_fabrics():1 [ 4055.439498] x0 : ffff07ff84e93dd0 [ 4055.442886] nvme_loop():1 [ 4055.445839] [ 4055.449227] nvmet():1 [ 4055.451920] Call trace: [ 4055.453399] nvme_fabrics():1 [ 4055.455745] __sbq_wake_up+0x58/0x1f0 [ 4055.458178] nvme_loop():1 [ 4055.461131] sbitmap_queue_clear+0x74/0xc0 [ 4055.464780] nvmet():1 [ 4055.467473] blk_mq_put_tag+0x44/0x70 [ 4055.471555] nvme_fabrics():1 [ 4055.473901] __blk_mq_free_request+0x88/0xf0 [ 4055.477549] nvme_loop():1 [ 4055.480503] blk_mq_free_request+0xc0/0x160 [ 4055.484758] nvmet():1 [ 4055.487451] __blk_mq_end_request+0x78/0x150 [ 4055.491620] nvme_fabrics():1 [ 4055.493966] scsi_end_request+0x114/0x260 [ 4055.498222] nvme_loop():1 [ 4055.501175] scsi_io_completion+0x64/0x174 [ 4055.505170] nvmet():1 [ 4055.507864] scsi_finish_command+0xe0/0x120 [ 4055.511946] nvme_fabrics():1 [ 4055.514292] scsi_complete+0x8c/0x1c4 [ 4055.518460] nvme_loop():1 [ 4055.521414] blk_complete_reqs+0x5c/0x74 [ 4055.525062] nvmet():1 [ 4055.527755] blk_done_softirq+0x38/0x4c [ 4055.531663] nvme_fabrics():1 [ 4055.534009] __do_softirq+0x168/0x418 [ 4055.537831] nvme_loop():1 [ 4055.540785] __irq_exit_rcu+0x120/0x170 [ 4055.544433] nvmet():1 [ 4055.547126] irq_exit_rcu+0x18/0x24 [ 4055.550947] nvme_fabrics():1 [ 4055.553293] el1_interrupt+0x38/0x70 [ 4055.556768] nvme_loop():1 [ 4055.559721] el1h_64_irq_handler+0x18/0x24 [ 4055.563282] nvmet():1 [ 4055.565975] el1h_64_irq+0x68/0x6c [ 4055.570057] nvme_fabrics():1 [ 4055.572403] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4055.575791] nvme_loop():1 [ 4055.578744] __wake_up_common_lock+0x84/0xac [ 4055.583521] nvmet():1 [ 4055.586215] __wake_up+0x20/0x30 [ 4055.590470] nvme_fabrics():1 [ 4055.592816] __sbq_wake_up+0x13c/0x1f0 [ 4055.596030] nvme_loop():1 [ 4055.598984] sbitmap_queue_clear+0x74/0xc0 [ 4055.602718] nvmet():1 [ 4055.605412] blk_mq_put_tag+0x44/0x70 [ 4055.609494] nvme_fabrics():1 [ 4055.611839] __blk_mq_free_request+0x88/0xf0 [ 4055.615487] nvme_loop():1 [ 4055.618441] blk_mq_free_request+0xc0/0x160 [ 4055.622697] null_blk():1 [ 4055.625390] bfq_insert_request.isra.0+0x204/0x270 [ 4055.629559] null_blk():1 [ 4055.632165] bfq_insert_requests+0x70/0x90 [ 4055.636941] null_blk():1 [ 4055.639548] blk_mq_sched_insert_requests+0xac/0x250 [ 4055.643630] null_blk():1 [ 4055.646236] blk_mq_flush_plug_list+0x110/0x2cc [ 4055.651186] null_blk():1 [ 4055.653792] __blk_flush_plug+0x100/0x140 [ 4055.658308] null_blk():1 [ 4055.660915] blk_finish_plug+0x40/0x25c [ 4055.664910] null_blk():1 [ 4055.667516] __iomap_dio_rw+0x1b4/0x500 [ 4055.671340] scsi_debug():1 [ 4055.673946] iomap_dio_rw+0x18/0x50 [ 4055.677767] scsi_debug():1 [ 4055.680547] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4055.684022] ipmi_si():1 [ 4055.686802] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4055.692099] ipmi_si():1 [ 4055.694618] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4055.699308] [last unloaded: nvmet] [ 4055.701828] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4055.706865] [ 4055.710339] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4055.715463] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4055.716940] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4055.722064] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4055.729879] process_one_work+0x1ec/0x470 [ 4055.735263] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4055.744554] worker_thread+0x74/0x410 [ 4055.748549] pc : __sbq_wake_up+0x58/0x1f0 [ 4055.755496] kthread+0xec/0x100 [ 4055.759144] lr : sbitmap_queue_clear+0x74/0xc0 [ 4055.763139] ret_from_fork+0x10/0x20 [ 4055.766266] sp : ffff8000088438b0 [ 4055.777558] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4055.784680] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4055.791802] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4055.798924] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4055.806045] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4055.813167] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4055.820289] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4055.827412] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4055.834533] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4055.841655] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4055.848777] Call trace: [ 4055.851210] __sbq_wake_up+0x58/0x1f0 [ 4055.854859] sbitmap_queue_clear+0x74/0xc0 [ 4055.858942] blk_mq_put_tag+0x44/0x70 [ 4055.862592] __blk_mq_free_request+0x88/0xf0 [ 4055.866849] blk_mq_free_request+0xc0/0x160 [ 4055.871019] __blk_mq_end_request+0x78/0x150 [ 4055.875275] scsi_end_request+0x114/0x260 [ 4055.879272] scsi_io_completion+0x64/0x174 [ 4055.883355] scsi_finish_command+0xe0/0x120 [ 4055.887525] scsi_complete+0x8c/0x1c4 [ 4055.891174] blk_complete_reqs+0x5c/0x74 [ 4055.895084] blk_done_softirq+0x38/0x4c [ 4055.898906] __do_softirq+0x168/0x418 [ 4055.902555] __irq_exit_rcu+0x120/0x170 [ 4055.906378] irq_exit_rcu+0x18/0x24 [ 4055.909853] el1_interrupt+0x38/0x70 [ 4055.913415] el1h_64_irq_handler+0x18/0x24 [ 4055.917500] el1h_64_irq+0x68/0x6c [ 4055.920888] cpuidle_enter_state+0xc4/0x4e0 [ 4055.925058] cpuidle_enter+0x40/0x60 [ 4055.928620] cpuidle_idle_call+0x118/0x1b0 [ 4055.932703] do_idle+0xac/0x100 [ 4055.935832] cpu_startup_entry+0x30/0x34 [ 4055.939741] secondary_start_kernel+0xd8/0xf0 [ 4055.944085] __secondary_switched+0xb0/0xb4 [ 4066.267891] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 361533 jiffies s: 2093 root: 0x11/. [ 4066.279629] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4066.288838] Task dump for CPU 8: [ 4066.292052] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4066.301954] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4066.308123] Call trace: [ 4066.310556] __switch_to+0xd4/0x100 [ 4066.314033] 0xffff6c0200000080 [ 4066.317162] Task dump for CPU 68: [ 4066.320465] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4066.330367] Call trace: [ 4066.332800] __switch_to+0xd4/0x100 [ 4066.336276] 0x0 [ 4079.087803] watchdog: BUG: soft lockup - CPU#8 stuck for 3404s! [kworker/8:12:19399] [ 4079.095532] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4079.095550] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4079.183339] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4079.191589] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4079.197802] watchdog: BUG: soft lockup - CPU#68 stuck for 3405s! [swapper/68:0] [ 4079.200880] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4079.208173] Modules linked in: nvme_loop [ 4079.214339] [ 4079.214340] nvme_fabrics [ 4079.218249] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4079.219727] nvmet [ 4079.222333] pc : __sbq_wake_up+0x58/0x1f0 [ 4079.229280] nls_utf8 [ 4079.231278] lr : sbitmap_queue_clear+0x74/0xc0 [ 4079.235273] hfsplus [ 4079.237533] sp : ffff80000d79b280 [ 4079.241962] hfs [ 4079.244134] x29: ffff80000d79b280 [ 4079.247435] isofs [ 4079.249260] x28: ffffd5becb6aec50 [ 4079.252561] nvme_core [ 4079.254559] x27: ffffd5beca4f7000 [ 4079.257947] nvme_common [ 4079.260293] [ 4079.263680] loop [ 4079.266200] x26: ffffd5becaf68a98 [ 4079.267678] tls [ 4079.269590] x25: ffffd5becb6a60c0 [ 4079.272890] rfkill [ 4079.274716] x24: fffffbffef180dc0 [ 4079.278103] sunrpc [ 4079.280189] [ 4079.283576] vfat [ 4079.285662] x23: ffff07ff81d381b0 [ 4079.287140] fat [ 4079.289051] x22: ffff80000d79b30c [ 4079.292352] acpi_ipmi [ 4079.294177] x21: ffff07ff84e93c00 [ 4079.297565] ipmi_ssif [ 4079.299911] [ 4079.303298] cppc_cpufreq [ 4079.305644] x20: 00000000000001c0 [ 4079.307122] ipmi_devintf [ 4079.309728] x19: ffff07ff91ccdb10 [ 4079.313029] igb [ 4079.315635] x18: ffffffffffffffff [ 4079.319023] ipmi_msghandler [ 4079.320848] [ 4079.324236] arm_cmn [ 4079.327102] x17: ffff32506c412000 [ 4079.328581] arm_dmc620_pmu [ 4079.330755] x16: ffff800008044000 [ 4079.334056] arm_spe_pmu [ 4079.336836] x15: 0000000000004000 [ 4079.340224] arm_dsu_pmu [ 4079.342743] [ 4079.346131] acpi_tad [ 4079.348650] x14: 010320bb28ec0000 [ 4079.350128] fuse [ 4079.352389] x13: 0000000000000087 [ 4079.355690] zram [ 4079.357602] x12: 071c71c71c71c71c [ 4079.360989] xfs [ 4079.362902] [ 4079.366289] crct10dif_ce [ 4079.368114] x11: 0000000000000040 [ 4079.369592] polyval_ce [ 4079.372198] x10: ffff07ff801a9c70 [ 4079.375499] ast [ 4079.377932] x9 : ffffd5bec9545fd8 [ 4079.381319] polyval_generic [ 4079.383145] [ 4079.386532] drm_vram_helper [ 4079.389399] x8 : ffff07ff80404b70 [ 4079.390877] ghash_ce [ 4079.393743] x7 : ffff07ff84e93dc8 [ 4079.397044] drm_ttm_helper [ 4079.399303] x6 : 0000000000000008 [ 4079.402691] sbsa_gwdt [ 4079.405471] [ 4079.408858] ttm [ 4079.411204] x5 : ffff80000d043790 [ 4079.412682] xgene_hwmon [ 4079.414507] x4 : ffff07ff84e93dc0 [ 4079.417808] Unloaded tainted modules: [ 4079.420328] x3 : 0000000000000008 [ 4079.423715] nvmet():1 [ 4079.427363] [ 4079.430751] nvme_fabrics():1 [ 4079.433097] x2 : 0000000000000007 [ 4079.434574] nvme_loop():1 [ 4079.437528] x1 : ffff80000d79b30c [ 4079.440829] nvmet():1 [ 4079.443522] x0 : ffff07ff84e93dd0 [ 4079.446911] nvme_fabrics():1 [ 4079.449256] [ 4079.452645] nvme_loop():1 [ 4079.455598] Call trace: [ 4079.457076] nvmet():1 [ 4079.459769] __sbq_wake_up+0x58/0x1f0 [ 4079.462202] nvme_fabrics():1 [ 4079.464548] sbitmap_queue_clear+0x74/0xc0 [ 4079.468196] nvme_loop():1 [ 4079.471149] blk_mq_put_tag+0x44/0x70 [ 4079.475231] nvmet():1 [ 4079.477924] __blk_mq_free_request+0x88/0xf0 [ 4079.481572] nvme_fabrics():1 [ 4079.483919] blk_mq_free_request+0xc0/0x160 [ 4079.488176] nvme_loop():1 [ 4079.491129] __blk_mq_end_request+0x78/0x150 [ 4079.495298] nvmet():1 [ 4079.497991] scsi_end_request+0x114/0x260 [ 4079.502247] nvme_fabrics():1 [ 4079.504593] scsi_io_completion+0x64/0x174 [ 4079.508588] nvme_loop():1 [ 4079.511541] scsi_finish_command+0xe0/0x120 [ 4079.515623] nvmet():1 [ 4079.518316] scsi_complete+0x8c/0x1c4 [ 4079.522485] nvme_fabrics():1 [ 4079.524831] blk_complete_reqs+0x5c/0x74 [ 4079.528479] nvme_loop():1 [ 4079.531433] blk_done_softirq+0x38/0x4c [ 4079.535341] nvmet():1 [ 4079.538034] __do_softirq+0x168/0x418 [ 4079.541856] nvme_fabrics():1 [ 4079.544202] __irq_exit_rcu+0x120/0x170 [ 4079.547850] nvme_loop():1 [ 4079.550803] irq_exit_rcu+0x18/0x24 [ 4079.554625] nvmet():1 [ 4079.557318] el1_interrupt+0x38/0x70 [ 4079.560792] nvme_fabrics():1 [ 4079.563138] el1h_64_irq_handler+0x18/0x24 [ 4079.566700] nvme_loop():1 [ 4079.569653] el1h_64_irq+0x68/0x6c [ 4079.573735] nvmet():1 [ 4079.576428] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4079.579816] nvme_fabrics():1 [ 4079.582162] __wake_up_common_lock+0x84/0xac [ 4079.586939] nvme_loop():1 [ 4079.589892] __wake_up+0x20/0x30 [ 4079.594148] nvmet():1 [ 4079.596841] __sbq_wake_up+0x13c/0x1f0 [ 4079.600055] nvme_fabrics():1 [ 4079.602401] sbitmap_queue_clear+0x74/0xc0 [ 4079.606136] nvme_loop():1 [ 4079.609089] blk_mq_put_tag+0x44/0x70 [ 4079.613171] null_blk():1 [ 4079.615864] __blk_mq_free_request+0x88/0xf0 [ 4079.619512] null_blk():1 [ 4079.622119] blk_mq_free_request+0xc0/0x160 [ 4079.626374] null_blk():1 [ 4079.628981] bfq_insert_request.isra.0+0x204/0x270 [ 4079.633149] null_blk():1 [ 4079.635756] bfq_insert_requests+0x70/0x90 [ 4079.640532] null_blk():1 [ 4079.643139] blk_mq_sched_insert_requests+0xac/0x250 [ 4079.647221] null_blk():1 [ 4079.649827] blk_mq_flush_plug_list+0x110/0x2cc [ 4079.654777] null_blk():1 [ 4079.657383] __blk_flush_plug+0x100/0x140 [ 4079.661899] scsi_debug():1 [ 4079.664506] blk_finish_plug+0x40/0x25c [ 4079.668501] scsi_debug():1 [ 4079.671281] __iomap_dio_rw+0x1b4/0x500 [ 4079.675102] ipmi_si():1 [ 4079.677883] iomap_dio_rw+0x18/0x50 [ 4079.681705] ipmi_si():1 [ 4079.684224] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4079.687698] [last unloaded: nvmet] [ 4079.690218] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4079.695517] [ 4079.698993] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4079.703683] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4079.705160] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4079.710197] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4079.718012] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4079.723136] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4079.732426] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4079.737550] pc : __sbq_wake_up+0x58/0x1f0 [ 4079.744496] process_one_work+0x1ec/0x470 [ 4079.749880] lr : sbitmap_queue_clear+0x74/0xc0 [ 4079.753875] worker_thread+0x74/0x410 [ 4079.757871] sp : ffff8000088438b0 [ 4079.762300] kthread+0xec/0x100 [ 4079.765948] x29: ffff8000088438b0 [ 4079.769249] ret_from_fork+0x10/0x20 [ 4079.772376] x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4079.784536] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4079.791658] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4079.798780] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4079.805902] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4079.813024] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4079.820146] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4079.827268] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4079.834389] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4079.841511] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4079.848633] Call trace: [ 4079.851066] __sbq_wake_up+0x58/0x1f0 [ 4079.854715] sbitmap_queue_clear+0x74/0xc0 [ 4079.858800] blk_mq_put_tag+0x44/0x70 [ 4079.862450] __blk_mq_free_request+0x88/0xf0 [ 4079.866706] blk_mq_free_request+0xc0/0x160 [ 4079.870876] __blk_mq_end_request+0x78/0x150 [ 4079.875133] scsi_end_request+0x114/0x260 [ 4079.879129] scsi_io_completion+0x64/0x174 [ 4079.883212] scsi_finish_command+0xe0/0x120 [ 4079.887383] scsi_complete+0x8c/0x1c4 [ 4079.891032] blk_complete_reqs+0x5c/0x74 [ 4079.894941] blk_done_softirq+0x38/0x4c [ 4079.898764] __do_softirq+0x168/0x418 [ 4079.902413] __irq_exit_rcu+0x120/0x170 [ 4079.906236] irq_exit_rcu+0x18/0x24 [ 4079.909711] el1_interrupt+0x38/0x70 [ 4079.913275] el1h_64_irq_handler+0x18/0x24 [ 4079.917358] el1h_64_irq+0x68/0x6c [ 4079.920746] cpuidle_enter_state+0xc4/0x4e0 [ 4079.924916] cpuidle_enter+0x40/0x60 [ 4079.928480] cpuidle_idle_call+0x118/0x1b0 [ 4079.932563] do_idle+0xac/0x100 [ 4079.935692] cpu_startup_entry+0x30/0x34 [ 4079.939601] secondary_start_kernel+0xd8/0xf0 [ 4079.943945] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 05:05:00 2022] [ 4091.247729] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4091.253462] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=181737 [ 4091.262669] (t=366638 jiffies g=36445 q=107320 ncpus=128) [ 4091.268143] Task dump for CPU 8: [ 4091.271357] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4091.281257] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4091.287426] Call trace: [ 4091.289858] __switch_to+0xd4/0x100 [ 4091.293335] 0xffff6c0200000080 [ 4091.296464] Task dump for CPU 68: [ 4091.299765] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4091.309665] Call trace: [ 4091.312098] dump_backtrace+0xe8/0x130 [ 4091.315834] show_stack+0x20/0x60 [ 4091.319136] sched_show_task+0x160/0x190 [ 4091.323045] dump_cpu_task+0x4c/0x78 [ 4091.326608] rcu_dump_cpu_stacks+0xc4/0x150 [ 4091.330779] print_cpu_stall+0x180/0x210 [ 4091.334688] check_cpu_stall+0x20c/0x2b4 [ 4091.338598] rcu_pending+0x4c/0x190 [ 4091.342074] rcu_sched_clock_irq+0x80/0x204 [ 4091.346244] update_process_times+0x7c/0xc0 [ 4091.350414] tick_sched_handle+0x38/0x74 [ 4091.354324] tick_sched_timer+0x60/0xc0 [ 4091.358148] __hrtimer_run_queues+0x170/0x300 [ 4091.362491] hrtimer_interrupt+0xfc/0x260 [ 4091.366487] arch_timer_handler_phys+0x34/0x50 [ 4091.370918] handle_percpu_devid_irq+0xac/0x240 [ 4091.375436] generic_handle_domain_irq+0x34/0x50 [ 4091.380040] gic_handle_irq+0x5c/0xe0 [ 4091.383689] call_on_irq_stack+0x2c/0x38 [ 4091.387599] do_interrupt_handler+0xdc/0xe0 [ 4091.391770] el1_interrupt+0x34/0x70 [ 4091.395334] el1h_64_irq_handler+0x18/0x24 [ 4091.399417] el1h_64_irq+0x68/0x6c [ 4091.402805] __sbq_wake_up+0x8/0x1f0 [ 4091.406368] blk_mq_put_tag+0x44/0x70 [ 4091.410019] __blk_mq_free_request+0x88/0xf0 [ 4091.414276] blk_mq_free_request+0xc0/0x160 [ 4091.418446] __blk_mq_end_request+0x78/0x150 [ 4091.422702] scsi_end_request+0x114/0x260 [ 4091.426699] scsi_io_completion+0x64/0x174 [ 4091.430782] scsi_finish_command+0xe0/0x120 [ 4091.434952] scsi_complete+0x8c/0x1c4 [ 4091.438601] blk_complete_reqs+0x5c/0x74 [ 4091.442511] blk_done_softirq+0x38/0x4c [ 4091.446333] __do_softirq+0x168/0x418 [ 4091.449982] __irq_exit_rcu+0x120/0x170 [ 4091.453805] irq_exit_rcu+0x18/0x24 [ 4091.457280] el1_interrupt+0x38/0x70 [ 4091.460842] el1h_64_irq_handler+0x18/0x24 [ 4091.464926] el1h_64_irq+0x68/0x6c [ 4091.468315] cpuidle_enter_state+0xc4/0x4e0 [ 4091.472485] cpuidle_enter+0x40/0x60 [ 4091.476047] cpuidle_idle_call+0x118/0x1b0 [ 4091.480131] do_idle+0xac/0x100 [ 4091.483260] cpu_startup_entry+0x30/0x34 [ 4091.487169] secondary_start_kernel+0xd8/0xf0 [ 4091.491513] __secondary_switched+0xb0/0xb4 [ 4103.087656] watchdog: BUG: soft lockup - CPU#8 stuck for 3427s! [kworker/8:12:19399] [ 4103.095385] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4103.095403] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4103.183191] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4103.191441] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4103.200732] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4103.206899] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4103.213847] pc : __sbq_wake_up+0x40/0x1f0 [ 4103.217844] lr : sbitmap_queue_clear+0x74/0xc0 [ 4103.222275] sp : ffff80000d79b280 [ 4103.225576] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4103.232698] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4103.239821] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4103.246943] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4103.254066] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4103.261187] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4103.268310] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4103.275432] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4103.282554] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4103.289676] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 4103.296798] Call trace: [ 4103.299231] __sbq_wake_up+0x40/0x1f0 [ 4103.302880] sbitmap_queue_clear+0x74/0xc0 [ 4103.306964] blk_mq_put_tag+0x44/0x70 [ 4103.310614] __blk_mq_free_request+0x88/0xf0 [ 4103.314870] blk_mq_free_request+0xc0/0x160 [ 4103.319040] __blk_mq_end_request+0x78/0x150 [ 4103.323297] scsi_end_request+0x114/0x260 [ 4103.327293] scsi_io_completion+0x64/0x174 [ 4103.331376] scsi_finish_command+0xe0/0x120 [ 4103.335547] scsi_complete+0x8c/0x1c4 [ 4103.339196] blk_complete_reqs+0x5c/0x74 [ 4103.343105] blk_done_softirq+0x38/0x4c [ 4103.346928] __do_softirq+0x168/0x418 [ 4103.350577] __irq_exit_rcu+0x120/0x170 [ 4103.354400] irq_exit_rcu+0x18/0x24 [ 4103.357875] el1_interrupt+0x38/0x70 [ 4103.361438] el1h_64_irq_handler+0x18/0x24 [ 4103.365521] el1h_64_irq+0x68/0x6c [ 4103.368910] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4103.373687] __wake_up_common_lock+0x84/0xac [ 4103.377945] __wake_up+0x20/0x30 [ 4103.381160] __sbq_wake_up+0x13c/0x1f0 [ 4103.384896] sbitmap_queue_clear+0x74/0xc0 [ 4103.388979] blk_mq_put_tag+0x44/0x70 [ 4103.392629] __blk_mq_free_request+0x88/0xf0 [ 4103.396886] blk_mq_free_request+0xc0/0x160 [ 4103.401056] bfq_insert_request.isra.0+0x204/0x270 [ 4103.405834] bfq_insert_requests+0x70/0x90 [ 4103.409917] blk_mq_sched_insert_requests+0xac/0x250 [ 4103.414870] blk_mq_flush_plug_list+0x110/0x2cc [ 4103.419387] __blk_flush_plug+0x100/0x140 [ 4103.423383] blk_finish_plug+0x40/0x25c [ 4103.427206] __iomap_dio_rw+0x1b4/0x500 [ 4103.431029] iomap_dio_rw+0x18/0x50 [ 4103.434504] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4103.439855] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4103.444598] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4103.449641] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4103.454772] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4103.459901] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4103.465288] process_one_work+0x1ec/0x470 [ 4103.469284] worker_thread+0x74/0x410 [ 4103.472932] kthread+0xec/0x100 [ 4103.476061] ret_from_fork+0x10/0x20 [ 4119.197557] watchdog: BUG: soft lockup - CPU#68 stuck for 3442s! [swapper/68:0] [ 4119.204853] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4119.204870] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4119.292659] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4119.300475] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4119.309765] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4119.316714] pc : __sbq_wake_up+0x84/0x1f0 [ 4119.320710] lr : sbitmap_queue_clear+0x74/0xc0 [ 4119.325141] sp : ffff8000088438b0 [ 4119.328442] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4119.335564] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4119.342687] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4119.349809] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4119.356930] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4119.364052] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4119.371174] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4119.378296] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4119.385418] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4119.392539] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4119.399661] Call trace: [ 4119.402094] __sbq_wake_up+0x84/0x1f0 [ 4119.405743] sbitmap_queue_clear+0x74/0xc0 [ 4119.409826] blk_mq_put_tag+0x44/0x70 [ 4119.413476] __blk_mq_free_request+0x88/0xf0 [ 4119.417733] blk_mq_free_request+0xc0/0x160 [ 4119.421902] __blk_mq_end_request+0x78/0x150 [ 4119.426160] scsi_end_request+0x114/0x260 [ 4119.430156] scsi_io_completion+0x64/0x174 [ 4119.434240] scsi_finish_command+0xe0/0x120 [ 4119.438411] scsi_complete+0x8c/0x1c4 [ 4119.442060] blk_complete_reqs+0x5c/0x74 [ 4119.445969] blk_done_softirq+0x38/0x4c [ 4119.449792] __do_softirq+0x168/0x418 [ 4119.453440] __irq_exit_rcu+0x120/0x170 [ 4119.457263] irq_exit_rcu+0x18/0x24 [ 4119.460739] el1_interrupt+0x38/0x70 [ 4119.464301] el1h_64_irq_handler+0x18/0x24 [ 4119.468384] el1h_64_irq+0x68/0x6c [ 4119.471772] cpuidle_enter_state+0xc4/0x4e0 [ 4119.475943] cpuidle_enter+0x40/0x60 [ 4119.479505] cpuidle_idle_call+0x118/0x1b0 [ 4119.483588] do_idle+0xac/0x100 [ 4119.486717] cpu_startup_entry+0x30/0x34 [ 4119.490626] secondary_start_kernel+0xd8/0xf0 [ 4119.494970] __secondary_switched+0xb0/0xb4 [ 4127.087509] watchdog: BUG: soft lockup - CPU#8 stuck for 3449s! [kworker/8:12:19399] [ 4127.095239] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4127.095256] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4127.183045] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4127.191295] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4127.200586] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4127.206753] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4127.213701] pc : __sbq_wake_up+0x3c/0x1f0 [ 4127.217698] lr : sbitmap_queue_clear+0x74/0xc0 [ 4127.222129] sp : ffff80000d79b280 [ 4127.225430] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4127.232552] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4127.239675] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4127.246797] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4127.253919] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4127.261041] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4127.268163] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4127.275285] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4127.282407] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4127.289529] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 4127.296651] Call trace: [ 4127.299084] __sbq_wake_up+0x3c/0x1f0 [ 4127.302733] sbitmap_queue_clear+0x74/0xc0 [ 4127.306816] blk_mq_put_tag+0x44/0x70 [ 4127.310466] __blk_mq_free_request+0x88/0xf0 [ 4127.314723] blk_mq_free_request+0xc0/0x160 [ 4127.318894] __blk_mq_end_request+0x78/0x150 [ 4127.323152] scsi_end_request+0x114/0x260 [ 4127.327148] scsi_io_completion+0x64/0x174 [ 4127.331231] scsi_finish_command+0xe0/0x120 [ 4127.335401] scsi_complete+0x8c/0x1c4 [ 4127.339050] blk_complete_reqs+0x5c/0x74 [ 4127.342960] blk_done_softirq+0x38/0x4c [ 4127.346783] __do_softirq+0x168/0x418 [ 4127.350432] __irq_exit_rcu+0x120/0x170 [ 4127.354254] irq_exit_rcu+0x18/0x24 [ 4127.357730] el1_interrupt+0x38/0x70 [ 4127.361292] el1h_64_irq_handler+0x18/0x24 [ 4127.365375] el1h_64_irq+0x68/0x6c [ 4127.368764] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4127.373541] __wake_up_common_lock+0x84/0xac [ 4127.377799] __wake_up+0x20/0x30 [ 4127.381014] __sbq_wake_up+0x13c/0x1f0 [ 4127.384750] sbitmap_queue_clear+0x74/0xc0 [ 4127.388834] blk_mq_put_tag+0x44/0x70 [ 4127.392483] __blk_mq_free_request+0x88/0xf0 [ 4127.396740] blk_mq_free_request+0xc0/0x160 [ 4127.400910] bfq_insert_request.isra.0+0x204/0x270 [ 4127.405688] bfq_insert_requests+0x70/0x90 [ 4127.409772] blk_mq_sched_insert_requests+0xac/0x250 [ 4127.414724] blk_mq_flush_plug_list+0x110/0x2cc [ 4127.419242] __blk_flush_plug+0x100/0x140 [ 4127.423238] blk_finish_plug+0x40/0x25c [ 4127.427061] __iomap_dio_rw+0x1b4/0x500 [ 4127.430884] iomap_dio_rw+0x18/0x50 [ 4127.434359] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4127.439710] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4127.444453] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4127.449497] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4127.454627] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4127.459757] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4127.465143] process_one_work+0x1ec/0x470 [ 4127.469140] worker_thread+0x74/0x410 [ 4127.472789] kthread+0xec/0x100 [ 4127.475917] ret_from_fork+0x10/0x20 [ 4132.827484] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 368189 jiffies s: 2093 root: 0x11/. [ 4132.839224] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4132.848433] Task dump for CPU 8: [ 4132.851648] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4132.861549] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4132.867719] Call trace: [ 4132.870152] __switch_to+0xd4/0x100 [ 4132.873629] 0xffff6c0200000080 [ 4132.876758] Task dump for CPU 68: [ 4132.880060] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4132.889962] Call trace: [ 4132.892395] __switch_to+0xd4/0x100 [ 4132.895871] 0x0 [ 4143.197410] watchdog: BUG: soft lockup - CPU#68 stuck for 3464s! [swapper/68:0] [ 4143.204706] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4143.204723] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4143.292510] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4143.300326] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4143.309617] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4143.316564] pc : __sbq_wake_up+0x7c/0x1f0 [ 4143.320561] lr : sbitmap_queue_clear+0x74/0xc0 [ 4143.324991] sp : ffff800008843900 [ 4143.328293] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4143.335415] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4143.342536] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4143.349659] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4143.356781] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4143.363902] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4143.371024] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4143.378146] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4143.385268] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4143.392390] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 4143.399512] Call trace: [ 4143.401945] __sbq_wake_up+0x7c/0x1f0 [ 4143.405594] blk_mq_put_tag+0x44/0x70 [ 4143.409243] __blk_mq_free_request+0x88/0xf0 [ 4143.413500] blk_mq_free_request+0xc0/0x160 [ 4143.417670] __blk_mq_end_request+0x78/0x150 [ 4143.421927] scsi_end_request+0x114/0x260 [ 4143.425923] scsi_io_completion+0x64/0x174 [ 4143.430006] scsi_finish_command+0xe0/0x120 [ 4143.434177] scsi_complete+0x8c/0x1c4 [ 4143.437826] blk_complete_reqs+0x5c/0x74 [ 4143.441735] blk_done_softirq+0x38/0x4c [ 4143.445558] __do_softirq+0x168/0x418 [ 4143.449206] __irq_exit_rcu+0x120/0x170 [ 4143.453029] irq_exit_rcu+0x18/0x24 [ 4143.456505] el1_interrupt+0x38/0x70 [ 4143.460067] el1h_64_irq_handler+0x18/0x24 [ 4143.464150] el1h_64_irq+0x68/0x6c [ 4143.467538] cpuidle_enter_state+0xc4/0x4e0 [ 4143.471709] cpuidle_enter+0x40/0x60 [ 4143.475271] cpuidle_idle_call+0x118/0x1b0 [ 4143.479354] do_idle+0xac/0x100 [ 4143.482483] cpu_startup_entry+0x30/0x34 [ 4143.486392] secondary_start_kernel+0xd8/0xf0 [ 4143.490736] __secondary_switched+0xb0/0xb4 [ 4151.087362] watchdog: BUG: soft lockup - CPU#8 stuck for 3472s! [kworker/8:12:19399] [ 4151.095092] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4151.095108] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4151.182897] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4151.191147] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4151.200438] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4151.206605] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4151.213553] pc : __sbq_wake_up+0x58/0x1f0 [ 4151.217550] lr : sbitmap_queue_clear+0x74/0xc0 [ 4151.221981] sp : ffff80000d79b280 [ 4151.225282] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4151.232404] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4151.239526] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4151.246649] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4151.253771] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4151.260893] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4151.268016] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4151.275138] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4151.282260] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4151.289382] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4151.296504] Call trace: [ 4151.298937] __sbq_wake_up+0x58/0x1f0 [ 4151.302587] sbitmap_queue_clear+0x74/0xc0 [ 4151.306671] blk_mq_put_tag+0x44/0x70 [ 4151.310320] __blk_mq_free_request+0x88/0xf0 [ 4151.314577] blk_mq_free_request+0xc0/0x160 [ 4151.318747] __blk_mq_end_request+0x78/0x150 [ 4151.323003] scsi_end_request+0x114/0x260 [ 4151.327000] scsi_io_completion+0x64/0x174 [ 4151.331083] scsi_finish_command+0xe0/0x120 [ 4151.335253] scsi_complete+0x8c/0x1c4 [ 4151.338902] blk_complete_reqs+0x5c/0x74 [ 4151.342812] blk_done_softirq+0x38/0x4c [ 4151.346634] __do_softirq+0x168/0x418 [ 4151.350284] __irq_exit_rcu+0x120/0x170 [ 4151.354106] irq_exit_rcu+0x18/0x24 [ 4151.357582] el1_interrupt+0x38/0x70 [ 4151.361144] el1h_64_irq_handler+0x18/0x24 [ 4151.365228] el1h_64_irq+0x68/0x6c [ 4151.368616] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4151.373394] __wake_up_common_lock+0x84/0xac [ 4151.377652] __wake_up+0x20/0x30 [ 4151.380867] __sbq_wake_up+0x13c/0x1f0 [ 4151.384603] sbitmap_queue_clear+0x74/0xc0 [ 4151.388686] blk_mq_put_tag+0x44/0x70 [ 4151.392336] __blk_mq_free_request+0x88/0xf0 [ 4151.396592] blk_mq_free_request+0xc0/0x160 [ 4151.400762] bfq_insert_request.isra.0+0x204/0x270 [ 4151.405540] bfq_insert_requests+0x70/0x90 [ 4151.409624] blk_mq_sched_insert_requests+0xac/0x250 [ 4151.414575] blk_mq_flush_plug_list+0x110/0x2cc [ 4151.419093] __blk_flush_plug+0x100/0x140 [ 4151.423089] blk_finish_plug+0x40/0x25c [ 4151.426912] __iomap_dio_rw+0x1b4/0x500 [ 4151.430735] iomap_dio_rw+0x18/0x50 [ 4151.434210] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4151.439561] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4151.444303] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4151.449347] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4151.454477] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4151.459607] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4151.464994] process_one_work+0x1ec/0x470 [ 4151.468990] worker_thread+0x74/0x410 [ 4151.472639] kthread+0xec/0x100 [ 4151.475768] ret_from_fork+0x10/0x20 [ 4167.197264] watchdog: BUG: soft lockup - CPU#68 stuck for 3487s! [swapper/68:0] [ 4167.204560] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4167.204577] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4167.292365] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4167.300180] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4167.309471] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4167.316419] pc : __sbq_wake_up+0x84/0x1f0 [ 4167.320416] lr : sbitmap_queue_clear+0x74/0xc0 [ 4167.324846] sp : ffff8000088438b0 [ 4167.328147] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4167.335269] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4167.342391] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4167.349513] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4167.356635] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4167.363756] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4167.370878] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4167.378000] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4167.385122] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4167.392244] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4167.399365] Call trace: [ 4167.401798] __sbq_wake_up+0x84/0x1f0 [ 4167.405447] sbitmap_queue_clear+0x74/0xc0 [ 4167.409530] blk_mq_put_tag+0x44/0x70 [ 4167.413180] __blk_mq_free_request+0x88/0xf0 [ 4167.417437] blk_mq_free_request+0xc0/0x160 [ 4167.421607] __blk_mq_end_request+0x78/0x150 [ 4167.425863] scsi_end_request+0x114/0x260 [ 4167.429860] scsi_io_completion+0x64/0x174 [ 4167.433944] scsi_finish_command+0xe0/0x120 [ 4167.438114] scsi_complete+0x8c/0x1c4 [ 4167.441763] blk_complete_reqs+0x5c/0x74 [ 4167.445673] blk_done_softirq+0x38/0x4c [ 4167.449495] __do_softirq+0x168/0x418 [ 4167.453145] __irq_exit_rcu+0x120/0x170 [ 4167.456967] irq_exit_rcu+0x18/0x24 [ 4167.460443] el1_interrupt+0x38/0x70 [ 4167.464005] el1h_64_irq_handler+0x18/0x24 [ 4167.468088] el1h_64_irq+0x68/0x6c [ 4167.471477] cpuidle_enter_state+0xc4/0x4e0 [ 4167.475647] cpuidle_enter+0x40/0x60 [ 4167.479209] cpuidle_idle_call+0x118/0x1b0 [ 4167.483293] do_idle+0xac/0x100 [ 4167.486421] cpu_startup_entry+0x30/0x34 [ 4167.490330] secondary_start_kernel+0xd8/0xf0 [ 4167.494674] __secondary_switched+0xb0/0xb4 [ 4175.087215] watchdog: BUG: soft lockup - CPU#8 stuck for 3494s! [kworker/8:12:19399] [ 4175.094945] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4175.094962] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4175.182750] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4175.191000] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4175.200291] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4175.206458] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4175.213406] pc : __sbq_wake_up+0x18/0x1f0 [ 4175.217402] lr : sbitmap_queue_clear+0x74/0xc0 [ 4175.221833] sp : ffff80000d79b280 [ 4175.225134] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4175.232257] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4175.239378] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4175.246500] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4175.253623] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4175.260744] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4175.267866] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4175.274988] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4175.282110] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4175.289232] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 4175.296355] Call trace: [ 4175.298788] __sbq_wake_up+0x18/0x1f0 [ 4175.302438] sbitmap_queue_clear+0x74/0xc0 [ 4175.306521] blk_mq_put_tag+0x44/0x70 [ 4175.310171] __blk_mq_free_request+0x88/0xf0 [ 4175.314428] blk_mq_free_request+0xc0/0x160 [ 4175.318598] __blk_mq_end_request+0x78/0x150 [ 4175.322855] scsi_end_request+0x114/0x260 [ 4175.326852] scsi_io_completion+0x64/0x174 [ 4175.330935] scsi_finish_command+0xe0/0x120 [ 4175.335105] scsi_complete+0x8c/0x1c4 [ 4175.338754] blk_complete_reqs+0x5c/0x74 [ 4175.342664] blk_done_softirq+0x38/0x4c [ 4175.346486] __do_softirq+0x168/0x418 [ 4175.350135] __irq_exit_rcu+0x120/0x170 [ 4175.353958] irq_exit_rcu+0x18/0x24 [ 4175.357434] el1_interrupt+0x38/0x70 [ 4175.360996] el1h_64_irq_handler+0x18/0x24 [ 4175.365079] el1h_64_irq+0x68/0x6c [ 4175.368468] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4175.373245] __wake_up_common_lock+0x84/0xac [ 4175.377502] __wake_up+0x20/0x30 [ 4175.380718] __sbq_wake_up+0x13c/0x1f0 [ 4175.384454] sbitmap_queue_clear+0x74/0xc0 [ 4175.388537] blk_mq_put_tag+0x44/0x70 [ 4175.392187] __blk_mq_free_request+0x88/0xf0 [ 4175.396444] blk_mq_free_request+0xc0/0x160 [ 4175.400614] bfq_insert_request.isra.0+0x204/0x270 [ 4175.405392] bfq_insert_requests+0x70/0x90 [ 4175.409475] blk_mq_sched_insert_requests+0xac/0x250 [ 4175.414428] blk_mq_flush_plug_list+0x110/0x2cc [ 4175.418945] __blk_flush_plug+0x100/0x140 [ 4175.422941] blk_finish_plug+0x40/0x25c [ 4175.426764] __iomap_dio_rw+0x1b4/0x500 [ 4175.430586] iomap_dio_rw+0x18/0x50 [ 4175.434062] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4175.439412] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4175.444155] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4175.449199] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4175.454329] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4175.459459] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4175.464845] process_one_work+0x1ec/0x470 [ 4175.468842] worker_thread+0x74/0x410 [ 4175.472491] kthread+0xec/0x100 [ 4175.475619] ret_from_fork+0x10/0x20 [ 4191.197117] watchdog: BUG: soft lockup - CPU#68 stuck for 3509s! [swapper/68:0] [ 4191.204412] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4191.204430] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4191.292218] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4191.300034] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4191.309325] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4191.316272] pc : __sbq_wake_up+0x58/0x1f0 [ 4191.320269] lr : sbitmap_queue_clear+0x74/0xc0 [ 4191.324699] sp : ffff8000088438b0 [ 4191.328001] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4191.335123] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4191.342245] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4191.349367] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4191.356489] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4191.363611] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4191.370733] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4191.377855] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4191.384976] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4191.392098] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4191.399220] Call trace: [ 4191.401653] __sbq_wake_up+0x58/0x1f0 [ 4191.405302] sbitmap_queue_clear+0x74/0xc0 [ 4191.409386] blk_mq_put_tag+0x44/0x70 [ 4191.413036] __blk_mq_free_request+0x88/0xf0 [ 4191.417292] blk_mq_free_request+0xc0/0x160 [ 4191.421462] __blk_mq_end_request+0x78/0x150 [ 4191.425719] scsi_end_request+0x114/0x260 [ 4191.429715] scsi_io_completion+0x64/0x174 [ 4191.433798] scsi_finish_command+0xe0/0x120 [ 4191.437969] scsi_complete+0x8c/0x1c4 [ 4191.441618] blk_complete_reqs+0x5c/0x74 [ 4191.445527] blk_done_softirq+0x38/0x4c [ 4191.449350] __do_softirq+0x168/0x418 [ 4191.452999] __irq_exit_rcu+0x120/0x170 [ 4191.456822] irq_exit_rcu+0x18/0x24 [ 4191.460297] el1_interrupt+0x38/0x70 [ 4191.463860] el1h_64_irq_handler+0x18/0x24 [ 4191.467944] el1h_64_irq+0x68/0x6c [ 4191.471332] cpuidle_enter_state+0xc4/0x4e0 [ 4191.475502] cpuidle_enter+0x40/0x60 [ 4191.479065] cpuidle_idle_call+0x118/0x1b0 [ 4191.483148] do_idle+0xac/0x100 [ 4191.486277] cpu_startup_entry+0x30/0x34 [ 4191.490186] secondary_start_kernel+0xd8/0xf0 [ 4191.494530] __secondary_switched+0xb0/0xb4 [ 4199.087068] watchdog: BUG: soft lockup - CPU#8 stuck for 3516s! [kworker/8:12:19399] [ 4199.094798] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4199.094815] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4199.182603] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4199.190853] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4199.200144] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4199.206312] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4199.213259] pc : __sbq_wake_up+0x20/0x1f0 [ 4199.217255] lr : sbitmap_queue_clear+0x74/0xc0 [ 4199.221686] sp : ffff80000d79b280 [ 4199.224988] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4199.232110] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4199.239232] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4199.246355] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4199.253477] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4199.260599] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4199.267721] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4199.274843] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4199.281965] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4199.289087] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 4199.296209] Call trace: [ 4199.298642] __sbq_wake_up+0x20/0x1f0 [ 4199.302291] sbitmap_queue_clear+0x74/0xc0 [ 4199.306375] blk_mq_put_tag+0x44/0x70 [ 4199.310025] __blk_mq_free_request+0x88/0xf0 [ 4199.314281] blk_mq_free_request+0xc0/0x160 [ 4199.318451] __blk_mq_end_request+0x78/0x150 [ 4199.322708] scsi_end_request+0x114/0x260 [ 4199.326704] scsi_io_completion+0x64/0x174 [ 4199.330788] scsi_finish_command+0xe0/0x120 [ 4199.334958] scsi_complete+0x8c/0x1c4 [ 4199.338607] blk_complete_reqs+0x5c/0x74 [ 4199.342517] blk_done_softirq+0x38/0x4c [ 4199.346340] __do_softirq+0x168/0x418 [ 4199.349989] __irq_exit_rcu+0x120/0x170 [ 4199.353811] irq_exit_rcu+0x18/0x24 [ 4199.357287] el1_interrupt+0x38/0x70 [ 4199.360849] el1h_64_irq_handler+0x18/0x24 [ 4199.364932] el1h_64_irq+0x68/0x6c [ 4199.368321] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4199.373098] __wake_up_common_lock+0x84/0xac [ 4199.377356] __wake_up+0x20/0x30 [ 4199.380571] __sbq_wake_up+0x13c/0x1f0 [ 4199.384307] sbitmap_queue_clear+0x74/0xc0 [ 4199.387076] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { [ 4199.388391] blk_mq_put_tag+0x44/0x70 [ 4199.388391] 8-.... [ 4199.388393] __blk_mq_free_request+0x88/0xf0 [ 4199.395514] 68-.... [ 4199.399162] blk_mq_free_request+0xc0/0x160 [ 4199.399163] bfq_insert_request.isra.0+0x204/0x270 [ 4199.401251] } 374846 jiffies s: 2093 root: 0x11/. [ 4199.405506] bfq_insert_requests+0x70/0x90 [ 4199.407680] rcu: blocking rcu_node structures (internal RCU debug): [ 4199.411847] blk_mq_sched_insert_requests+0xac/0x250 [ 4199.416623] l=1:0-15:0x100/. [ 4199.421400] blk_mq_flush_plug_list+0x110/0x2cc [ 4199.425485] l=1:64-79:0x10/. [ 4199.431737] __blk_flush_plug+0x100/0x140 [ 4199.431738] blk_finish_plug+0x40/0x25c [ 4199.436689] [ 4199.439643] __iomap_dio_rw+0x1b4/0x500 [ 4199.439645] iomap_dio_rw+0x18/0x50 [ 4199.444162] Task dump for CPU 8: [ 4199.447115] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4199.451112] task:kworker/8:12 state:R [ 4199.454934] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4199.456412] running task [ 4199.460233] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4199.463709] stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4199.466923] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4199.472222] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4199.476129] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4199.480820] [ 4199.483859] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4199.488898] Call trace: [ 4199.494801] process_one_work+0x1ec/0x470 [ 4199.499926] __switch_to+0xd4/0x100 [ 4199.506090] worker_thread+0x74/0x410 [ 4199.511216] 0xffff6c0200000080 [ 4199.512692] kthread+0xec/0x100 [ 4199.518078] Task dump for CPU 68: [ 4199.520509] ret_from_fork+0x10/0x20 [ 4199.524504] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4199.554647] Call trace: [ 4199.557081] __switch_to+0xd4/0x100 [ 4199.560557] 0x0 [ 4215.196970] watchdog: BUG: soft lockup - CPU#68 stuck for 3531s! [swapper/68:0] [ 4215.204265] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4215.204283] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4215.292070] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4215.299886] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4215.309177] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4215.316124] pc : __sbq_wake_up+0x24/0x1f0 [ 4215.320121] lr : sbitmap_queue_clear+0x74/0xc0 [ 4215.324551] sp : ffff8000088438b0 [ 4215.327852] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4215.334974] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4215.342096] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4215.349218] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4215.356340] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4215.363462] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4215.370584] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4215.377706] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4215.384828] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4215.391949] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4215.399071] Call trace: [ 4215.401504] __sbq_wake_up+0x24/0x1f0 [ 4215.405154] sbitmap_queue_clear+0x74/0xc0 [ 4215.409237] blk_mq_put_tag+0x44/0x70 [ 4215.412887] __blk_mq_free_request+0x88/0xf0 [ 4215.417144] blk_mq_free_request+0xc0/0x160 [ 4215.421313] __blk_mq_end_request+0x78/0x150 [ 4215.425570] scsi_end_request+0x114/0x260 [ 4215.429567] scsi_io_completion+0x64/0x174 [ 4215.433651] scsi_finish_command+0xe0/0x120 [ 4215.437822] scsi_complete+0x8c/0x1c4 [ 4215.441471] blk_complete_reqs+0x5c/0x74 [ 4215.445380] blk_done_softirq+0x38/0x4c [ 4215.449203] __do_softirq+0x168/0x418 [ 4215.452851] __irq_exit_rcu+0x120/0x170 [ 4215.456674] irq_exit_rcu+0x18/0x24 [ 4215.460151] el1_interrupt+0x38/0x70 [ 4215.463713] el1h_64_irq_handler+0x18/0x24 [ 4215.467797] el1h_64_irq+0x68/0x6c [ 4215.471185] cpuidle_enter_state+0xc4/0x4e0 [ 4215.475355] cpuidle_enter+0x40/0x60 [ 4215.478917] cpuidle_idle_call+0x118/0x1b0 [ 4215.483001] do_idle+0xac/0x100 [ 4215.486130] cpu_startup_entry+0x30/0x34 [ 4215.490039] secondary_start_kernel+0xd8/0xf0 [ 4215.494383] __secondary_switched+0xb0/0xb4 [ 4223.086922] watchdog: BUG: soft lockup - CPU#8 stuck for 3539s! [kworker/8:12:19399] [ 4223.094651] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4223.094668] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4223.182456] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4223.190705] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4223.199996] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4223.206164] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4223.213112] pc : __sbq_wake_up+0x1c8/0x1f0 [ 4223.217195] lr : sbitmap_queue_clear+0x74/0xc0 [ 4223.221626] sp : ffff80000d79b280 [ 4223.224927] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4223.232050] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4223.239172] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4223.246294] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4223.253416] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4223.260538] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4223.267660] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4223.274782] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4223.281903] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4223.289025] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 4223.296147] Call trace: [ 4223.298580] __sbq_wake_up+0x1c8/0x1f0 [ 4223.302317] sbitmap_queue_clear+0x74/0xc0 [ 4223.306401] blk_mq_put_tag+0x44/0x70 [ 4223.310050] __blk_mq_free_request+0x88/0xf0 [ 4223.314307] blk_mq_free_request+0xc0/0x160 [ 4223.318477] __blk_mq_end_request+0x78/0x150 [ 4223.322734] scsi_end_request+0x114/0x260 [ 4223.326730] scsi_io_completion+0x64/0x174 [ 4223.330813] scsi_finish_command+0xe0/0x120 [ 4223.334984] scsi_complete+0x8c/0x1c4 [ 4223.338633] blk_complete_reqs+0x5c/0x74 [ 4223.342542] blk_done_softirq+0x38/0x4c [ 4223.346365] __do_softirq+0x168/0x418 [ 4223.350014] __irq_exit_rcu+0x120/0x170 [ 4223.353837] irq_exit_rcu+0x18/0x24 [ 4223.357312] el1_interrupt+0x38/0x70 [ 4223.360875] el1h_64_irq_handler+0x18/0x24 [ 4223.364958] el1h_64_irq+0x68/0x6c [ 4223.368346] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4223.373124] __wake_up_common_lock+0x84/0xac [ 4223.377382] __wake_up+0x20/0x30 [ 4223.380597] __sbq_wake_up+0x13c/0x1f0 [ 4223.384334] sbitmap_queue_clear+0x74/0xc0 [ 4223.388417] blk_mq_put_tag+0x44/0x70 [ 4223.392067] __blk_mq_free_request+0x88/0xf0 [ 4223.396324] blk_mq_free_request+0xc0/0x160 [ 4223.400494] bfq_insert_request.isra.0+0x204/0x270 [ 4223.405272] bfq_insert_requests+0x70/0x90 [ 4223.409355] blk_mq_sched_insert_requests+0xac/0x250 [ 4223.414307] blk_mq_flush_plug_list+0x110/0x2cc [ 4223.418825] __blk_flush_plug+0x100/0x140 [ 4223.422822] blk_finish_plug+0x40/0x25c [ 4223.426644] __iomap_dio_rw+0x1b4/0x500 [ 4223.430467] iomap_dio_rw+0x18/0x50 [ 4223.433942] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4223.439293] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4223.444036] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4223.449079] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4223.454209] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4223.459339] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4223.464725] process_one_work+0x1ec/0x470 [ 4223.468721] worker_thread+0x74/0x410 [ 4223.472370] kthread+0xec/0x100 [ 4223.475499] ret_from_fork+0x10/0x20 [ 4239.196823] watchdog: BUG: soft lockup - CPU#68 stuck for 3554s! [swapper/68:0] [ 4239.204118] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4239.204135] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4239.291924] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4239.299739] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4239.309030] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4239.315978] pc : __sbq_wake_up+0x84/0x1f0 [ 4239.319975] lr : sbitmap_queue_clear+0x74/0xc0 [ 4239.324406] sp : ffff8000088438b0 [ 4239.327707] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4239.334829] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4239.341951] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4239.349074] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4239.356195] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4239.363318] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4239.370440] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4239.377561] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4239.384683] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4239.391805] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4239.398927] Call trace: [ 4239.401360] __sbq_wake_up+0x84/0x1f0 [ 4239.405009] sbitmap_queue_clear+0x74/0xc0 [ 4239.409092] blk_mq_put_tag+0x44/0x70 [ 4239.412741] __blk_mq_free_request+0x88/0xf0 [ 4239.416998] blk_mq_free_request+0xc0/0x160 [ 4239.421168] __blk_mq_end_request+0x78/0x150 [ 4239.425425] scsi_end_request+0x114/0x260 [ 4239.429421] scsi_io_completion+0x64/0x174 [ 4239.433504] scsi_finish_command+0xe0/0x120 [ 4239.437675] scsi_complete+0x8c/0x1c4 [ 4239.441324] blk_complete_reqs+0x5c/0x74 [ 4239.445233] blk_done_softirq+0x38/0x4c [ 4239.449056] __do_softirq+0x168/0x418 [ 4239.452704] __irq_exit_rcu+0x120/0x170 [ 4239.456527] irq_exit_rcu+0x18/0x24 [ 4239.460002] el1_interrupt+0x38/0x70 [ 4239.463565] el1h_64_irq_handler+0x18/0x24 [ 4239.467648] el1h_64_irq+0x68/0x6c [ 4239.471036] cpuidle_enter_state+0xc4/0x4e0 [ 4239.475206] cpuidle_enter+0x40/0x60 [ 4239.478769] cpuidle_idle_call+0x118/0x1b0 [ 4239.482852] do_idle+0xac/0x100 [ 4239.485981] cpu_startup_entry+0x30/0x34 [ 4239.489890] secondary_start_kernel+0xd8/0xf0 [ 4239.494234] __secondary_switched+0xb0/0xb4 [ 4247.086774] watchdog: BUG: soft lockup - CPU#8 stuck for 3561s! [kworker/8:12:19399] [ 4247.094504] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4247.094521] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4247.182308] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4247.190558] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4247.199849] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4247.206017] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4247.212964] pc : __sbq_wake_up+0x18/0x1f0 [ 4247.216961] lr : sbitmap_queue_clear+0x74/0xc0 [ 4247.221392] sp : ffff80000d79b280 [ 4247.224693] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4247.231816] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4247.238938] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4247.246060] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4247.253182] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4247.260304] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4247.267425] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4247.274547] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4247.281669] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4247.288791] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 4247.295913] Call trace: [ 4247.298346] __sbq_wake_up+0x18/0x1f0 [ 4247.301995] sbitmap_queue_clear+0x74/0xc0 [ 4247.306079] blk_mq_put_tag+0x44/0x70 [ 4247.309728] __blk_mq_free_request+0x88/0xf0 [ 4247.313985] blk_mq_free_request+0xc0/0x160 [ 4247.318155] __blk_mq_end_request+0x78/0x150 [ 4247.322412] scsi_end_request+0x114/0x260 [ 4247.326408] scsi_io_completion+0x64/0x174 [ 4247.330491] scsi_finish_command+0xe0/0x120 [ 4247.334661] scsi_complete+0x8c/0x1c4 [ 4247.338311] blk_complete_reqs+0x5c/0x74 [ 4247.342220] blk_done_softirq+0x38/0x4c [ 4247.346043] __do_softirq+0x168/0x418 [ 4247.349692] __irq_exit_rcu+0x120/0x170 [ 4247.353515] irq_exit_rcu+0x18/0x24 [ 4247.356990] el1_interrupt+0x38/0x70 [ 4247.360552] el1h_64_irq_handler+0x18/0x24 [ 4247.364635] el1h_64_irq+0x68/0x6c [ 4247.368024] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4247.372801] __wake_up_common_lock+0x84/0xac [ 4247.377059] __wake_up+0x20/0x30 [ 4247.380275] __sbq_wake_up+0x13c/0x1f0 [ 4247.384011] sbitmap_queue_clear+0x74/0xc0 [ 4247.388094] blk_mq_put_tag+0x44/0x70 [ 4247.391744] __blk_mq_free_request+0x88/0xf0 [ 4247.396001] blk_mq_free_request+0xc0/0x160 [ 4247.400171] bfq_insert_request.isra.0+0x204/0x270 [ 4247.404949] bfq_insert_requests+0x70/0x90 [ 4247.409032] blk_mq_sched_insert_requests+0xac/0x250 [ 4247.413985] blk_mq_flush_plug_list+0x110/0x2cc [ 4247.418502] __blk_flush_plug+0x100/0x140 [ 4247.422499] blk_finish_plug+0x40/0x25c [ 4247.426322] __iomap_dio_rw+0x1b4/0x500 [ 4247.430144] iomap_dio_rw+0x18/0x50 [ 4247.433621] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4247.438972] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4247.443714] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4247.448758] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4247.453888] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4247.459018] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4247.464404] process_one_work+0x1ec/0x470 [ 4247.468400] worker_thread+0x74/0x410 [ 4247.472049] kthread+0xec/0x100 [ 4247.475178] ret_from_fork+0x10/0x20 [ 4263.196665] watchdog: BUG: soft lockup - CPU#68 stuck for 3576s! [swapper/68:0] [ 4263.203962] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4263.203979] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4263.291768] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4263.299584] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4263.308875] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4263.315822] pc : __sbq_wake_up+0x58/0x1f0 [ 4263.319818] lr : sbitmap_queue_clear+0x74/0xc0 [ 4263.324249] sp : ffff8000088438b0 [ 4263.327550] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4263.334672] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4263.341794] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4263.348916] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4263.356038] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4263.363160] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4263.370282] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4263.377403] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4263.384525] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4263.391648] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4263.398769] Call trace: [ 4263.401202] __sbq_wake_up+0x58/0x1f0 [ 4263.404851] sbitmap_queue_clear+0x74/0xc0 [ 4263.408935] blk_mq_put_tag+0x44/0x70 [ 4263.412584] __blk_mq_free_request+0x88/0xf0 [ 4263.416841] blk_mq_free_request+0xc0/0x160 [ 4263.421011] __blk_mq_end_request+0x78/0x150 [ 4263.425268] scsi_end_request+0x114/0x260 [ 4263.429265] scsi_io_completion+0x64/0x174 [ 4263.433349] scsi_finish_command+0xe0/0x120 [ 4263.437519] scsi_complete+0x8c/0x1c4 [ 4263.441168] blk_complete_reqs+0x5c/0x74 [ 4263.445078] blk_done_softirq+0x38/0x4c [ 4263.448901] __do_softirq+0x168/0x418 [ 4263.452550] __irq_exit_rcu+0x120/0x170 [ 4263.456372] irq_exit_rcu+0x18/0x24 [ 4263.459847] el1_interrupt+0x38/0x70 [ 4263.463409] el1h_64_irq_handler+0x18/0x24 [ 4263.467493] el1h_64_irq+0x68/0x6c [ 4263.470881] cpuidle_enter_state+0xc4/0x4e0 [ 4263.475051] cpuidle_enter+0x40/0x60 [ 4263.478613] cpuidle_idle_call+0x118/0x1b0 [ 4263.482698] do_idle+0xac/0x100 [ 4263.485826] cpu_startup_entry+0x30/0x34 [ 4263.489735] secondary_start_kernel+0xd8/0xf0 [ 4263.494079] __secondary_switched+0xb0/0xb4 [ 4265.946657] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 381501 jiffies s: 2093 root: 0x11/. [ 4265.958395] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4265.967603] Task dump for CPU 8: [ 4265.970818] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4265.980720] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4265.986889] Call trace: [ 4265.989322] __switch_to+0xd4/0x100 [ 4265.992799] 0xffff6c0200000080 [ 4265.995928] Task dump for CPU 68: [ 4265.999231] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4266.009132] Call trace: [ 4266.011565] __switch_to+0xd4/0x100 [ 4266.015041] 0x0 [ 4271.086612] watchdog: BUG: soft lockup - CPU#8 stuck for 3583s! [kworker/8:12:19399] [ 4271.094341] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4271.094358] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4271.182147] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4271.190397] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4271.199688] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4271.205856] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4271.212803] pc : __sbq_wake_up+0x1c8/0x1f0 [ 4271.216887] lr : sbitmap_queue_clear+0x74/0xc0 [ 4271.221318] sp : ffff80000d79b280 [ 4271.224619] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4271.231741] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4271.238864] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4271.245985] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4271.253108] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4271.260230] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4271.267353] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4271.274474] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4271.281596] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4271.288718] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 4271.295841] Call trace: [ 4271.298274] __sbq_wake_up+0x1c8/0x1f0 [ 4271.302010] sbitmap_queue_clear+0x74/0xc0 [ 4271.306094] blk_mq_put_tag+0x44/0x70 [ 4271.309743] __blk_mq_free_request+0x88/0xf0 [ 4271.314000] blk_mq_free_request+0xc0/0x160 [ 4271.318170] __blk_mq_end_request+0x78/0x150 [ 4271.322427] scsi_end_request+0x114/0x260 [ 4271.326423] scsi_io_completion+0x64/0x174 [ 4271.330506] scsi_finish_command+0xe0/0x120 [ 4271.334677] scsi_complete+0x8c/0x1c4 [ 4271.338326] blk_complete_reqs+0x5c/0x74 [ 4271.342236] blk_done_softirq+0x38/0x4c [ 4271.346058] __do_softirq+0x168/0x418 [ 4271.349707] __irq_exit_rcu+0x120/0x170 [ 4271.353530] irq_exit_rcu+0x18/0x24 [ 4271.357006] el1_interrupt+0x38/0x70 [ 4271.360568] el1h_64_irq_handler+0x18/0x24 [ 4271.364652] el1h_64_irq+0x68/0x6c [ 4271.368040] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4271.372818] __wake_up_common_lock+0x84/0xac [ 4271.377075] __wake_up+0x20/0x30 [ 4271.380291] __sbq_wake_up+0x13c/0x1f0 [ 4271.384026] sbitmap_queue_clear+0x74/0xc0 [ 4271.388110] blk_mq_put_tag+0x44/0x70 [ 4271.391760] __blk_mq_free_request+0x88/0xf0 [ 4271.396016] blk_mq_free_request+0xc0/0x160 [ 4271.400186] bfq_insert_request.isra.0+0x204/0x270 [ 4271.404964] bfq_insert_requests+0x70/0x90 [ 4271.409047] blk_mq_sched_insert_requests+0xac/0x250 [ 4271.413999] blk_mq_flush_plug_list+0x110/0x2cc [ 4271.418517] __blk_flush_plug+0x100/0x140 [ 4271.422513] blk_finish_plug+0x40/0x25c [ 4271.426336] __iomap_dio_rw+0x1b4/0x500 [ 4271.430159] iomap_dio_rw+0x18/0x50 [ 4271.433634] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4271.438985] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4271.443728] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4271.448772] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4271.453902] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4271.459032] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4271.464419] process_one_work+0x1ec/0x470 [ 4271.468416] worker_thread+0x74/0x410 [ 4271.472065] kthread+0xec/0x100 [ 4271.475194] ret_from_fork+0x10/0x20 [ 4271.476610] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4271.484488] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=190697 [ 4271.493694] (t=384660 jiffies g=36445 q=112786 ncpus=128) [ 4271.499167] Task dump for CPU 8: [ 4271.502381] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4271.512281] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4271.518449] Call trace: [ 4271.520882] __switch_to+0xd4/0x100 [ 4271.524358] 0xffff6c0200000080 [ 4271.527486] Task dump for CPU 68: [ 4271.530788] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4271.540688] Call trace: [ 4271.543120] dump_backtrace+0xe8/0x130 [ 4271.546856] show_stack+0x20/0x60 [ 4271.550158] sched_show_task+0x160/0x190 [ 4271.554067] dump_cpu_task+0x4c/0x78 [ 4271.557630] rcu_dump_cpu_stacks+0xc4/0x150 [ 4271.561801] print_cpu_stall+0x180/0x210 [ 4271.565711] check_cpu_stall+0x20c/0x2b4 [ 4271.569620] rcu_pending+0x4c/0x190 [ 4271.573096] rcu_sched_clock_irq+0x80/0x204 [ 4271.577266] update_process_times+0x7c/0xc0 [ 4271.581436] tick_sched_handle+0x38/0x74 [ 4271.585346] tick_sched_timer+0x60/0xc0 [ 4271.589169] __hrtimer_run_queues+0x170/0x300 [ 4271.593512] hrtimer_interrupt+0xfc/0x260 [ 4271.597509] arch_timer_handler_phys+0x34/0x50 [ 4271.601939] handle_percpu_devid_irq+0xac/0x240 [ 4271.606457] generic_handle_domain_irq+0x34/0x50 [ 4271.611062] gic_handle_irq+0x5c/0xe0 [ 4271.614711] call_on_irq_stack+0x2c/0x38 [ 4271.618621] do_interrupt_handler+0xdc/0xe0 [ 4271.622791] el1_interrupt+0x34/0x70 [ 4271.626353] el1h_64_irq_handler+0x18/0x24 [ 4271.630437] el1h_64_irq+0x68/0x6c [ 4271.633825] __sbq_wake_up+0x58/0x1f0 [ 4271.637474] sbitmap_queue_clear+0x74/0xc0 [ 4271.641557] blk_mq_put_tag+0x44/0x70 [ 4271.645207] __blk_mq_free_request+0x88/0xf0 [ 4271.649464] blk_mq_free_request+0xc0/0x160 [ 4271.653634] __blk_mq_end_request+0x78/0x150 [ 4271.657890] scsi_end_request+0x114/0x260 [ 4271.661887] scsi_io_completion+0x64/0x174 [ 4271.665970] scsi_finish_command+0xe0/0x120 [ 4271.670140] scsi_complete+0x8c/0x1c4 [ 4271.673789] blk_complete_reqs+0x5c/0x74 [ 4271.677699] blk_done_softirq+0x38/0x4c [ 4271.681521] __do_softirq+0x168/0x418 [ 4271.685170] __irq_exit_rcu+0x120/0x170 [ 4271.688993] irq_exit_rcu+0x18/0x24 [ 4271.692468] el1_interrupt+0x38/0x70 [ 4271.696030] el1h_64_irq_handler+0x18/0x24 [ 4271.700113] el1h_64_irq+0x68/0x6c [ 4271.703502] cpuidle_enter_state+0xc4/0x4e0 [ 4271.707672] cpuidle_enter+0x40/0x60 [ 4271.711234] cpuidle_idle_call+0x118/0x1b0 [ 4271.715318] do_idle+0xac/0x100 [ 4271.718447] cpu_startup_entry+0x30/0x34 [ 4271.722356] secondary_start_kernel+0xd8/0xf0 [ 4271.726700] __secondary_switched+0xb0/0xb4 [ 4295.086450] watchdog: BUG: soft lockup - CPU#8 stuck for 3606s! [kworker/8:12:19399] [ 4295.094180] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4295.094197] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4295.181985] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4295.190235] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4295.199526] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4295.205694] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4295.212641] pc : __sbq_wake_up+0x24/0x1f0 [ 4295.216638] lr : sbitmap_queue_clear+0x74/0xc0 [ 4295.221069] sp : ffff80000d79b280 [ 4295.224370] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4295.231492] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4295.238615] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4295.245738] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4295.252860] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4295.259982] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4295.267105] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4295.274227] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4295.281349] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4295.288471] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 4295.295593] Call trace: [ 4295.298026] __sbq_wake_up+0x24/0x1f0 [ 4295.301675] sbitmap_queue_clear+0x74/0xc0 [ 4295.305758] blk_mq_put_tag+0x44/0x70 [ 4295.309408] __blk_mq_free_request+0x88/0xf0 [ 4295.313665] blk_mq_free_request+0xc0/0x160 [ 4295.317835] __blk_mq_end_request+0x78/0x150 [ 4295.322092] scsi_end_request+0x114/0x260 [ 4295.326088] scsi_io_completion+0x64/0x174 [ 4295.330171] scsi_finish_command+0xe0/0x120 [ 4295.334342] scsi_complete+0x8c/0x1c4 [ 4295.337991] blk_complete_reqs+0x5c/0x74 [ 4295.341900] blk_done_softirq+0x38/0x4c [ 4295.345724] __do_softirq+0x168/0x418 [ 4295.349373] __irq_exit_rcu+0x120/0x170 [ 4295.353195] irq_exit_rcu+0x18/0x24 [ 4295.356671] el1_interrupt+0x38/0x70 [ 4295.360233] el1h_64_irq_handler+0x18/0x24 [ 4295.364316] el1h_64_irq+0x68/0x6c [ 4295.367705] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4295.372482] __wake_up_common_lock+0x84/0xac [ 4295.376740] __wake_up+0x20/0x30 [ 4295.379955] __sbq_wake_up+0x13c/0x1f0 [ 4295.383692] sbitmap_queue_clear+0x74/0xc0 [ 4295.387775] blk_mq_put_tag+0x44/0x70 [ 4295.391425] __blk_mq_free_request+0x88/0xf0 [ 4295.395682] blk_mq_free_request+0xc0/0x160 [ 4295.399851] bfq_insert_request.isra.0+0x204/0x270 [ 4295.404630] bfq_insert_requests+0x70/0x90 [ 4295.408713] blk_mq_sched_insert_requests+0xac/0x250 [ 4295.413665] blk_mq_flush_plug_list+0x110/0x2cc [ 4295.418183] __blk_flush_plug+0x100/0x140 [ 4295.422180] blk_finish_plug+0x40/0x25c [ 4295.426003] __iomap_dio_rw+0x1b4/0x500 [ 4295.429825] iomap_dio_rw+0x18/0x50 [ 4295.433301] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4295.438652] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4295.443394] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4295.448439] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4295.453568] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4295.458698] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4295.464084] process_one_work+0x1ec/0x470 [ 4295.468080] worker_thread+0x74/0x410 [ 4295.471729] kthread+0xec/0x100 [ 4295.474858] ret_from_fork+0x10/0x20 [ 4299.196422] watchdog: BUG: soft lockup - CPU#68 stuck for 3609s! [swapper/68:0] [ 4299.203718] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4299.203735] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4299.291523] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4299.299338] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4299.308629] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4299.315577] pc : __sbq_wake_up+0x58/0x1f0 [ 4299.319573] lr : sbitmap_queue_clear+0x74/0xc0 [ 4299.324004] sp : ffff8000088438b0 [ 4299.327305] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4299.334427] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4299.341550] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4299.348672] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4299.355793] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4299.362916] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4299.370039] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4299.377161] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4299.384283] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4299.391405] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4299.398527] Call trace: [ 4299.400960] __sbq_wake_up+0x58/0x1f0 [ 4299.404609] sbitmap_queue_clear+0x74/0xc0 [ 4299.408692] blk_mq_put_tag+0x44/0x70 [ 4299.412342] __blk_mq_free_request+0x88/0xf0 [ 4299.416599] blk_mq_free_request+0xc0/0x160 [ 4299.420769] __blk_mq_end_request+0x78/0x150 [ 4299.425025] scsi_end_request+0x114/0x260 [ 4299.429022] scsi_io_completion+0x64/0x174 [ 4299.433105] scsi_finish_command+0xe0/0x120 [ 4299.437275] scsi_complete+0x8c/0x1c4 [ 4299.440924] blk_complete_reqs+0x5c/0x74 [ 4299.444834] blk_done_softirq+0x38/0x4c [ 4299.448657] __do_softirq+0x168/0x418 [ 4299.452306] __irq_exit_rcu+0x120/0x170 [ 4299.456128] irq_exit_rcu+0x18/0x24 [ 4299.459604] el1_interrupt+0x38/0x70 [ 4299.463166] el1h_64_irq_handler+0x18/0x24 [ 4299.467249] el1h_64_irq+0x68/0x6c [ 4299.470638] cpuidle_enter_state+0xc4/0x4e0 [ 4299.474808] cpuidle_enter+0x40/0x60 [ 4299.478370] cpuidle_idle_call+0x118/0x1b0 [ 4299.482454] do_idle+0xac/0x100 [ 4299.485583] cpu_startup_entry+0x30/0x34 [ 4299.489492] secondary_start_kernel+0xd8/0xf0 [ 4299.493836] __secondary_switched+0xb0/0xb4 [ 4319.086288] watchdog: BUG: soft lockup - CPU#8 stuck for 3628s! [kworker/8:12:19399] [ 4319.094017] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4319.094034] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4319.181823] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4319.190073] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4319.199364] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4319.205531] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4319.212479] pc : __sbq_wake_up+0x58/0x1f0 [ 4319.216476] lr : sbitmap_queue_clear+0x74/0xc0 [ 4319.220907] sp : ffff80000d79b280 [ 4319.224208] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4319.231330] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4319.238453] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4319.245575] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4319.252698] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4319.259819] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4319.266941] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4319.274063] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4319.281185] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4319.288307] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4319.295429] Call trace: [ 4319.297862] __sbq_wake_up+0x58/0x1f0 [ 4319.301511] sbitmap_queue_clear+0x74/0xc0 [ 4319.305594] blk_mq_put_tag+0x44/0x70 [ 4319.309244] __blk_mq_free_request+0x88/0xf0 [ 4319.313501] blk_mq_free_request+0xc0/0x160 [ 4319.317671] __blk_mq_end_request+0x78/0x150 [ 4319.321927] scsi_end_request+0x114/0x260 [ 4319.325924] scsi_io_completion+0x64/0x174 [ 4319.330007] scsi_finish_command+0xe0/0x120 [ 4319.334177] scsi_complete+0x8c/0x1c4 [ 4319.337826] blk_complete_reqs+0x5c/0x74 [ 4319.341736] blk_done_softirq+0x38/0x4c [ 4319.345558] __do_softirq+0x168/0x418 [ 4319.349208] __irq_exit_rcu+0x120/0x170 [ 4319.353030] irq_exit_rcu+0x18/0x24 [ 4319.356506] el1_interrupt+0x38/0x70 [ 4319.360068] el1h_64_irq_handler+0x18/0x24 [ 4319.364151] el1h_64_irq+0x68/0x6c [ 4319.367539] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4319.372317] __wake_up_common_lock+0x84/0xac [ 4319.376574] __wake_up+0x20/0x30 [ 4319.379790] __sbq_wake_up+0x13c/0x1f0 [ 4319.383525] sbitmap_queue_clear+0x74/0xc0 [ 4319.387609] blk_mq_put_tag+0x44/0x70 [ 4319.391259] __blk_mq_free_request+0x88/0xf0 [ 4319.395515] blk_mq_free_request+0xc0/0x160 [ 4319.399685] bfq_insert_request.isra.0+0x204/0x270 [ 4319.404463] bfq_insert_requests+0x70/0x90 [ 4319.408547] blk_mq_sched_insert_requests+0xac/0x250 [ 4319.413499] blk_mq_flush_plug_list+0x110/0x2cc [ 4319.418016] __blk_flush_plug+0x100/0x140 [ 4319.422013] blk_finish_plug+0x40/0x25c [ 4319.425835] __iomap_dio_rw+0x1b4/0x500 [ 4319.429658] iomap_dio_rw+0x18/0x50 [ 4319.433134] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4319.438484] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4319.443226] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4319.448270] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4319.453400] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4319.458530] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4319.463916] process_one_work+0x1ec/0x470 [ 4319.467913] worker_thread+0x74/0x410 [ 4319.471561] kthread+0xec/0x100 [ 4319.474690] ret_from_fork+0x10/0x20 [ 4323.196260] watchdog: BUG: soft lockup - CPU#68 stuck for 3632s! [swapper/68:0] [ 4323.203556] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4323.203573] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4323.291361] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4323.299177] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4323.308468] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4323.315415] pc : __sbq_wake_up+0x38/0x1f0 [ 4323.319413] lr : sbitmap_queue_clear+0x74/0xc0 [ 4323.323844] sp : ffff8000088438b0 [ 4323.327145] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4323.334267] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4323.341389] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4323.348511] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4323.355634] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4323.362756] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4323.369877] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4323.376999] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4323.384121] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4323.391243] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4323.398365] Call trace: [ 4323.400798] __sbq_wake_up+0x38/0x1f0 [ 4323.404447] sbitmap_queue_clear+0x74/0xc0 [ 4323.408531] blk_mq_put_tag+0x44/0x70 [ 4323.412180] __blk_mq_free_request+0x88/0xf0 [ 4323.416437] blk_mq_free_request+0xc0/0x160 [ 4323.420607] __blk_mq_end_request+0x78/0x150 [ 4323.424864] scsi_end_request+0x114/0x260 [ 4323.428861] scsi_io_completion+0x64/0x174 [ 4323.432944] scsi_finish_command+0xe0/0x120 [ 4323.437115] scsi_complete+0x8c/0x1c4 [ 4323.440764] blk_complete_reqs+0x5c/0x74 [ 4323.444673] blk_done_softirq+0x38/0x4c [ 4323.448496] __do_softirq+0x168/0x418 [ 4323.452144] __irq_exit_rcu+0x120/0x170 [ 4323.455967] irq_exit_rcu+0x18/0x24 [ 4323.459442] el1_interrupt+0x38/0x70 [ 4323.463005] el1h_64_irq_handler+0x18/0x24 [ 4323.467088] el1h_64_irq+0x68/0x6c [ 4323.470476] cpuidle_enter_state+0xc4/0x4e0 [ 4323.474646] cpuidle_enter+0x40/0x60 [ 4323.478208] cpuidle_idle_call+0x118/0x1b0 [ 4323.482292] do_idle+0xac/0x100 [ 4323.485421] cpu_startup_entry+0x30/0x34 [ 4323.489331] secondary_start_kernel+0xd8/0xf0 [ 4323.493675] __secondary_switched+0xb0/0xb4 [ 4332.506207] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 388157 jiffies s: 2093 root: 0x11/. [ 4332.517946] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4332.527155] Task dump for CPU 8: [ 4332.530369] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4332.540270] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4332.546440] Call trace: [ 4332.548873] __switch_to+0xd4/0x100 [ 4332.552349] 0xffff6c0200000080 [ 4332.555478] Task dump for CPU 68: [ 4332.558780] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4332.568683] Call trace: [ 4332.571116] __switch_to+0xd4/0x100 [ 4332.574592] 0x0 [ 4343.086126] watchdog: BUG: soft lockup - CPU#8 stuck for 3650s! [kworker/8:12:19399] [ 4343.093856] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4343.093873] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4343.181660] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4343.189910] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4343.199201] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4343.205369] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4343.212316] pc : __sbq_wake_up+0x84/0x1f0 [ 4343.216313] lr : sbitmap_queue_clear+0x74/0xc0 [ 4343.220744] sp : ffff80000d79b280 [ 4343.224045] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4343.231168] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4343.238290] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4343.245412] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4343.252535] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4343.259656] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4343.266778] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4343.273901] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4343.281023] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4343.288144] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4343.295266] Call trace: [ 4343.297700] __sbq_wake_up+0x84/0x1f0 [ 4343.301349] sbitmap_queue_clear+0x74/0xc0 [ 4343.305432] blk_mq_put_tag+0x44/0x70 [ 4343.309082] __blk_mq_free_request+0x88/0xf0 [ 4343.313339] blk_mq_free_request+0xc0/0x160 [ 4343.317508] __blk_mq_end_request+0x78/0x150 [ 4343.321766] scsi_end_request+0x114/0x260 [ 4343.325762] scsi_io_completion+0x64/0x174 [ 4343.329845] scsi_finish_command+0xe0/0x120 [ 4343.334016] scsi_complete+0x8c/0x1c4 [ 4343.337665] blk_complete_reqs+0x5c/0x74 [ 4343.341574] blk_done_softirq+0x38/0x4c [ 4343.345397] __do_softirq+0x168/0x418 [ 4343.349045] __irq_exit_rcu+0x120/0x170 [ 4343.352868] irq_exit_rcu+0x18/0x24 [ 4343.356343] el1_interrupt+0x38/0x70 [ 4343.359906] el1h_64_irq_handler+0x18/0x24 [ 4343.363989] el1h_64_irq+0x68/0x6c [ 4343.367378] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4343.372155] __wake_up_common_lock+0x84/0xac [ 4343.376412] __wake_up+0x20/0x30 [ 4343.379628] __sbq_wake_up+0x13c/0x1f0 [ 4343.383364] sbitmap_queue_clear+0x74/0xc0 [ 4343.387448] blk_mq_put_tag+0x44/0x70 [ 4343.391098] __blk_mq_free_request+0x88/0xf0 [ 4343.395354] blk_mq_free_request+0xc0/0x160 [ 4343.399524] bfq_insert_request.isra.0+0x204/0x270 [ 4343.404302] bfq_insert_requests+0x70/0x90 [ 4343.408386] blk_mq_sched_insert_requests+0xac/0x250 [ 4343.413338] blk_mq_flush_plug_list+0x110/0x2cc [ 4343.417856] __blk_flush_plug+0x100/0x140 [ 4343.421853] blk_finish_plug+0x40/0x25c [ 4343.425675] __iomap_dio_rw+0x1b4/0x500 [ 4343.429498] iomap_dio_rw+0x18/0x50 [ 4343.432973] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4343.438324] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4343.443067] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4343.448111] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4343.453240] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4343.458370] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4343.463756] process_one_work+0x1ec/0x470 [ 4343.467752] worker_thread+0x74/0x410 [ 4343.471401] kthread+0xec/0x100 [ 4343.474530] ret_from_fork+0x10/0x20 [ 4347.196098] watchdog: BUG: soft lockup - CPU#68 stuck for 3654s! [swapper/68:0] [ 4347.203394] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4347.203411] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4347.291198] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4347.299013] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4347.308305] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4347.315254] pc : __sbq_wake_up+0x40/0x1f0 [ 4347.319250] lr : sbitmap_queue_clear+0x74/0xc0 [ 4347.323681] sp : ffff8000088438b0 [ 4347.326982] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4347.334104] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4347.341226] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4347.348347] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4347.355470] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4347.362592] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4347.369714] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4347.376836] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4347.383958] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4347.391080] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4347.398202] Call trace: [ 4347.400636] __sbq_wake_up+0x40/0x1f0 [ 4347.404285] sbitmap_queue_clear+0x74/0xc0 [ 4347.408368] blk_mq_put_tag+0x44/0x70 [ 4347.412018] __blk_mq_free_request+0x88/0xf0 [ 4347.416276] blk_mq_free_request+0xc0/0x160 [ 4347.420446] __blk_mq_end_request+0x78/0x150 [ 4347.424702] scsi_end_request+0x114/0x260 [ 4347.428699] scsi_io_completion+0x64/0x174 [ 4347.432782] scsi_finish_command+0xe0/0x120 [ 4347.436952] scsi_complete+0x8c/0x1c4 [ 4347.440601] blk_complete_reqs+0x5c/0x74 [ 4347.444511] blk_done_softirq+0x38/0x4c [ 4347.448334] __do_softirq+0x168/0x418 [ 4347.451982] __irq_exit_rcu+0x120/0x170 [ 4347.455805] irq_exit_rcu+0x18/0x24 [ 4347.459280] el1_interrupt+0x38/0x70 [ 4347.462842] el1h_64_irq_handler+0x18/0x24 [ 4347.466926] el1h_64_irq+0x68/0x6c [ 4347.470314] cpuidle_enter_state+0xc4/0x4e0 [ 4347.474484] cpuidle_enter+0x40/0x60 [ 4347.478047] cpuidle_idle_call+0x118/0x1b0 [ 4347.482131] do_idle+0xac/0x100 [ 4347.485259] cpu_startup_entry+0x30/0x34 [ 4347.489168] secondary_start_kernel+0xd8/0xf0 [ 4347.493512] __secondary_switched+0xb0/0xb4 [ 4367.085964] watchdog: BUG: soft lockup - CPU#8 stuck for 3673s! [kworker/8:12:19399] [ 4367.093694] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4367.093711] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4367.181499] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4367.189748] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4367.199039] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4367.205207] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4367.212154] pc : __sbq_wake_up+0x84/0x1f0 [ 4367.216151] lr : sbitmap_queue_clear+0x74/0xc0 [ 4367.220582] sp : ffff80000d79b280 [ 4367.223883] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4367.231005] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4367.238127] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4367.245250] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4367.252372] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4367.259494] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4367.266616] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4367.273738] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4367.280860] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4367.287982] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4367.295103] Call trace: [ 4367.297536] __sbq_wake_up+0x84/0x1f0 [ 4367.301185] sbitmap_queue_clear+0x74/0xc0 [ 4367.305269] blk_mq_put_tag+0x44/0x70 [ 4367.308918] __blk_mq_free_request+0x88/0xf0 [ 4367.313175] blk_mq_free_request+0xc0/0x160 [ 4367.317345] __blk_mq_end_request+0x78/0x150 [ 4367.321602] scsi_end_request+0x114/0x260 [ 4367.325599] scsi_io_completion+0x64/0x174 [ 4367.329682] scsi_finish_command+0xe0/0x120 [ 4367.333852] scsi_complete+0x8c/0x1c4 [ 4367.337501] blk_complete_reqs+0x5c/0x74 [ 4367.341411] blk_done_softirq+0x38/0x4c [ 4367.345233] __do_softirq+0x168/0x418 [ 4367.348882] __irq_exit_rcu+0x120/0x170 [ 4367.352705] irq_exit_rcu+0x18/0x24 [ 4367.356181] el1_interrupt+0x38/0x70 [ 4367.359743] el1h_64_irq_handler+0x18/0x24 [ 4367.363826] el1h_64_irq+0x68/0x6c [ 4367.367215] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4367.371992] __wake_up_common_lock+0x84/0xac [ 4367.376249] __wake_up+0x20/0x30 [ 4367.379465] __sbq_wake_up+0x13c/0x1f0 [ 4367.383201] sbitmap_queue_clear+0x74/0xc0 [ 4367.387285] blk_mq_put_tag+0x44/0x70 [ 4367.390935] __blk_mq_free_request+0x88/0xf0 [ 4367.395191] blk_mq_free_request+0xc0/0x160 [ 4367.399361] bfq_insert_request.isra.0+0x204/0x270 [ 4367.404139] bfq_insert_requests+0x70/0x90 [ 4367.408223] blk_mq_sched_insert_requests+0xac/0x250 [ 4367.413175] blk_mq_flush_plug_list+0x110/0x2cc [ 4367.417693] __blk_flush_plug+0x100/0x140 [ 4367.421689] blk_finish_plug+0x40/0x25c [ 4367.425512] __iomap_dio_rw+0x1b4/0x500 [ 4367.429335] iomap_dio_rw+0x18/0x50 [ 4367.432810] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4367.438161] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4367.442903] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4367.447947] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4367.453077] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4367.458207] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4367.463593] process_one_work+0x1ec/0x470 [ 4367.467589] worker_thread+0x74/0x410 [ 4367.471238] kthread+0xec/0x100 [ 4367.474367] ret_from_fork+0x10/0x20 [ 4371.195937] watchdog: BUG: soft lockup - CPU#68 stuck for 3676s! [swapper/68:0] [ 4371.203232] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4371.203249] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4371.291036] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4371.298852] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4371.308142] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4371.315090] pc : __sbq_wake_up+0x58/0x1f0 [ 4371.319086] lr : sbitmap_queue_clear+0x74/0xc0 [ 4371.323517] sp : ffff8000088438b0 [ 4371.326818] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4371.333940] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4371.341062] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4371.348184] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4371.355306] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4371.362428] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4371.369550] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4371.376671] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4371.383793] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4371.390915] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4371.398037] Call trace: [ 4371.400470] __sbq_wake_up+0x58/0x1f0 [ 4371.404119] sbitmap_queue_clear+0x74/0xc0 [ 4371.408203] blk_mq_put_tag+0x44/0x70 [ 4371.411853] __blk_mq_free_request+0x88/0xf0 [ 4371.416110] blk_mq_free_request+0xc0/0x160 [ 4371.420280] __blk_mq_end_request+0x78/0x150 [ 4371.424536] scsi_end_request+0x114/0x260 [ 4371.428533] scsi_io_completion+0x64/0x174 [ 4371.432616] scsi_finish_command+0xe0/0x120 [ 4371.436786] scsi_complete+0x8c/0x1c4 [ 4371.440435] blk_complete_reqs+0x5c/0x74 [ 4371.444345] blk_done_softirq+0x38/0x4c [ 4371.448167] __do_softirq+0x168/0x418 [ 4371.451817] __irq_exit_rcu+0x120/0x170 [ 4371.455639] irq_exit_rcu+0x18/0x24 [ 4371.459115] el1_interrupt+0x38/0x70 [ 4371.462677] el1h_64_irq_handler+0x18/0x24 [ 4371.466760] el1h_64_irq+0x68/0x6c [ 4371.470149] cpuidle_enter_state+0xc4/0x4e0 [ 4371.474319] cpuidle_enter+0x40/0x60 [ 4371.477882] cpuidle_idle_call+0x118/0x1b0 [ 4371.481966] do_idle+0xac/0x100 [ 4371.485094] cpu_startup_entry+0x30/0x34 [ 4371.489004] secondary_start_kernel+0xd8/0xf0 [ 4371.493348] __secondary_switched+0xb0/0xb4 [-- MARK -- Wed Sep 28 05:10:00 2022] [ 4391.085802] watchdog: BUG: soft lockup - CPU#8 stuck for 3695s! [kworker/8:12:19399] [ 4391.093532] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4391.093549] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4391.181337] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4391.189588] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4391.198879] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4391.205046] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4391.211994] pc : __sbq_wake_up+0x4/0x1f0 [ 4391.215903] lr : sbitmap_queue_clear+0x74/0xc0 [ 4391.220334] sp : ffff80000d79b2d0 [ 4391.223635] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4391.230758] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4391.237880] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4391.245003] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4391.252124] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4391.259247] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4391.266369] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4391.273490] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4391.280612] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4391.287734] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 4391.294856] Call trace: [ 4391.297289] __sbq_wake_up+0x4/0x1f0 [ 4391.300851] blk_mq_put_tag+0x44/0x70 [ 4391.304501] __blk_mq_free_request+0x88/0xf0 [ 4391.308758] blk_mq_free_request+0xc0/0x160 [ 4391.312928] __blk_mq_end_request+0x78/0x150 [ 4391.317184] scsi_end_request+0x114/0x260 [ 4391.321181] scsi_io_completion+0x64/0x174 [ 4391.325264] scsi_finish_command+0xe0/0x120 [ 4391.329434] scsi_complete+0x8c/0x1c4 [ 4391.333083] blk_complete_reqs+0x5c/0x74 [ 4391.336993] blk_done_softirq+0x38/0x4c [ 4391.340815] __do_softirq+0x168/0x418 [ 4391.344465] __irq_exit_rcu+0x120/0x170 [ 4391.348287] irq_exit_rcu+0x18/0x24 [ 4391.351763] el1_interrupt+0x38/0x70 [ 4391.355325] el1h_64_irq_handler+0x18/0x24 [ 4391.359408] el1h_64_irq+0x68/0x6c [ 4391.362797] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4391.367574] __wake_up_common_lock+0x84/0xac [ 4391.371831] __wake_up+0x20/0x30 [ 4391.375047] __sbq_wake_up+0x13c/0x1f0 [ 4391.378783] sbitmap_queue_clear+0x74/0xc0 [ 4391.382867] blk_mq_put_tag+0x44/0x70 [ 4391.386516] __blk_mq_free_request+0x88/0xf0 [ 4391.390773] blk_mq_free_request+0xc0/0x160 [ 4391.394943] bfq_insert_request.isra.0+0x204/0x270 [ 4391.399721] bfq_insert_requests+0x70/0x90 [ 4391.403805] blk_mq_sched_insert_requests+0xac/0x250 [ 4391.408756] blk_mq_flush_plug_list+0x110/0x2cc [ 4391.413274] __blk_flush_plug+0x100/0x140 [ 4391.417271] blk_finish_plug+0x40/0x25c [ 4391.421093] __iomap_dio_rw+0x1b4/0x500 [ 4391.424916] iomap_dio_rw+0x18/0x50 [ 4391.428392] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4391.433742] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4391.438484] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4391.443528] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4391.448658] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4391.453788] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4391.459174] process_one_work+0x1ec/0x470 [ 4391.463171] worker_thread+0x74/0x410 [ 4391.466819] kthread+0xec/0x100 [ 4391.469948] ret_from_fork+0x10/0x20 [ 4395.195775] watchdog: BUG: soft lockup - CPU#68 stuck for 3699s! [swapper/68:0] [ 4395.203070] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4395.203087] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4395.290875] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4395.298690] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4395.307981] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4395.314929] pc : __sbq_wake_up+0x58/0x1f0 [ 4395.318926] lr : sbitmap_queue_clear+0x74/0xc0 [ 4395.323356] sp : ffff8000088438b0 [ 4395.326657] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4395.333779] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4395.340902] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4395.348023] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4395.355145] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4395.362267] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4395.369389] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4395.376511] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4395.383633] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4395.390755] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4395.397877] Call trace: [ 4395.400309] __sbq_wake_up+0x58/0x1f0 [ 4395.403958] sbitmap_queue_clear+0x74/0xc0 [ 4395.408042] blk_mq_put_tag+0x44/0x70 [ 4395.411691] __blk_mq_free_request+0x88/0xf0 [ 4395.415948] blk_mq_free_request+0xc0/0x160 [ 4395.420118] __blk_mq_end_request+0x78/0x150 [ 4395.424375] scsi_end_request+0x114/0x260 [ 4395.428371] scsi_io_completion+0x64/0x174 [ 4395.432454] scsi_finish_command+0xe0/0x120 [ 4395.436624] scsi_complete+0x8c/0x1c4 [ 4395.440273] blk_complete_reqs+0x5c/0x74 [ 4395.444183] blk_done_softirq+0x38/0x4c [ 4395.448006] __do_softirq+0x168/0x418 [ 4395.451656] __irq_exit_rcu+0x120/0x170 [ 4395.455478] irq_exit_rcu+0x18/0x24 [ 4395.458954] el1_interrupt+0x38/0x70 [ 4395.462516] el1h_64_irq_handler+0x18/0x24 [ 4395.466599] el1h_64_irq+0x68/0x6c [ 4395.469988] cpuidle_enter_state+0xc4/0x4e0 [ 4395.474158] cpuidle_enter+0x40/0x60 [ 4395.477721] cpuidle_idle_call+0x118/0x1b0 [ 4395.481804] do_idle+0xac/0x100 [ 4395.484933] cpu_startup_entry+0x30/0x34 [ 4395.488842] secondary_start_kernel+0xd8/0xf0 [ 4395.493186] __secondary_switched+0xb0/0xb4 [ 4399.065758] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 394813 jiffies s: 2093 root: 0x11/. [ 4399.077497] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4399.086705] Task dump for CPU 8: [ 4399.089920] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4399.099822] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4399.105992] Call trace: [ 4399.108426] __switch_to+0xd4/0x100 [ 4399.111902] 0xffff6c0200000080 [ 4399.115031] Task dump for CPU 68: [ 4399.118334] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4399.128235] Call trace: [ 4399.130668] __switch_to+0xd4/0x100 [ 4399.134145] 0x0 [ 4415.085640] watchdog: BUG: soft lockup - CPU#8 stuck for 3717s! [kworker/8:12:19399] [ 4415.093371] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4415.093388] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4415.181177] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4415.189426] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4415.198717] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4415.204885] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4415.211833] pc : __sbq_wake_up+0x20/0x1f0 [ 4415.215829] lr : sbitmap_queue_clear+0x74/0xc0 [ 4415.220260] sp : ffff80000d79b280 [ 4415.223562] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4415.230684] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4415.237806] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4415.244929] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4415.252050] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4415.259172] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4415.266294] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4415.273416] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4415.280538] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4415.287660] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 4415.294782] Call trace: [ 4415.297215] __sbq_wake_up+0x20/0x1f0 [ 4415.300864] sbitmap_queue_clear+0x74/0xc0 [ 4415.304947] blk_mq_put_tag+0x44/0x70 [ 4415.308596] __blk_mq_free_request+0x88/0xf0 [ 4415.312853] blk_mq_free_request+0xc0/0x160 [ 4415.317023] __blk_mq_end_request+0x78/0x150 [ 4415.321280] scsi_end_request+0x114/0x260 [ 4415.325277] scsi_io_completion+0x64/0x174 [ 4415.329360] scsi_finish_command+0xe0/0x120 [ 4415.333530] scsi_complete+0x8c/0x1c4 [ 4415.337179] blk_complete_reqs+0x5c/0x74 [ 4415.341089] blk_done_softirq+0x38/0x4c [ 4415.344911] __do_softirq+0x168/0x418 [ 4415.348560] __irq_exit_rcu+0x120/0x170 [ 4415.352383] irq_exit_rcu+0x18/0x24 [ 4415.355859] el1_interrupt+0x38/0x70 [ 4415.359421] el1h_64_irq_handler+0x18/0x24 [ 4415.363504] el1h_64_irq+0x68/0x6c [ 4415.366893] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4415.371671] __wake_up_common_lock+0x84/0xac [ 4415.375928] __wake_up+0x20/0x30 [ 4415.379144] __sbq_wake_up+0x13c/0x1f0 [ 4415.382880] sbitmap_queue_clear+0x74/0xc0 [ 4415.386964] blk_mq_put_tag+0x44/0x70 [ 4415.390613] __blk_mq_free_request+0x88/0xf0 [ 4415.394870] blk_mq_free_request+0xc0/0x160 [ 4415.399040] bfq_insert_request.isra.0+0x204/0x270 [ 4415.403820] bfq_insert_requests+0x70/0x90 [ 4415.407903] blk_mq_sched_insert_requests+0xac/0x250 [ 4415.412856] blk_mq_flush_plug_list+0x110/0x2cc [ 4415.417373] __blk_flush_plug+0x100/0x140 [ 4415.421370] blk_finish_plug+0x40/0x25c [ 4415.425192] __iomap_dio_rw+0x1b4/0x500 [ 4415.429015] iomap_dio_rw+0x18/0x50 [ 4415.432490] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4415.437841] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4415.442584] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4415.447628] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4415.452759] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4415.457888] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4415.463275] process_one_work+0x1ec/0x470 [ 4415.467271] worker_thread+0x74/0x410 [ 4415.470920] kthread+0xec/0x100 [ 4415.474049] ret_from_fork+0x10/0x20 [ 4419.195613] watchdog: BUG: soft lockup - CPU#68 stuck for 3721s! [swapper/68:0] [ 4419.202908] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4419.202925] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4419.290714] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4419.298530] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4419.307821] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4419.314768] pc : __sbq_wake_up+0x40/0x1f0 [ 4419.318765] lr : sbitmap_queue_clear+0x74/0xc0 [ 4419.323195] sp : ffff8000088438b0 [ 4419.326497] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4419.333619] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4419.340741] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4419.347863] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4419.354985] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4419.362107] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4419.369229] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4419.376351] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4419.383472] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4419.390594] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4419.397716] Call trace: [ 4419.400149] __sbq_wake_up+0x40/0x1f0 [ 4419.403798] sbitmap_queue_clear+0x74/0xc0 [ 4419.407881] blk_mq_put_tag+0x44/0x70 [ 4419.411531] __blk_mq_free_request+0x88/0xf0 [ 4419.415788] blk_mq_free_request+0xc0/0x160 [ 4419.419958] __blk_mq_end_request+0x78/0x150 [ 4419.424215] scsi_end_request+0x114/0x260 [ 4419.428211] scsi_io_completion+0x64/0x174 [ 4419.432294] scsi_finish_command+0xe0/0x120 [ 4419.436465] scsi_complete+0x8c/0x1c4 [ 4419.440114] blk_complete_reqs+0x5c/0x74 [ 4419.444023] blk_done_softirq+0x38/0x4c [ 4419.447847] __do_softirq+0x168/0x418 [ 4419.451496] __irq_exit_rcu+0x120/0x170 [ 4419.455319] irq_exit_rcu+0x18/0x24 [ 4419.458794] el1_interrupt+0x38/0x70 [ 4419.462357] el1h_64_irq_handler+0x18/0x24 [ 4419.466440] el1h_64_irq+0x68/0x6c [ 4419.469828] cpuidle_enter_state+0xc4/0x4e0 [ 4419.473998] cpuidle_enter+0x40/0x60 [ 4419.477561] cpuidle_idle_call+0x118/0x1b0 [ 4419.481644] do_idle+0xac/0x100 [ 4419.484773] cpu_startup_entry+0x30/0x34 [ 4419.488682] secondary_start_kernel+0xd8/0xf0 [ 4419.493026] __secondary_switched+0xb0/0xb4 [ 4439.085478] watchdog: BUG: soft lockup - CPU#8 stuck for 3740s! [kworker/8:12:19399] [ 4439.093208] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4439.093225] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4439.181013] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4439.189263] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4439.198554] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4439.204721] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4439.211669] pc : __sbq_wake_up+0x84/0x1f0 [ 4439.215666] lr : sbitmap_queue_clear+0x74/0xc0 [ 4439.220097] sp : ffff80000d79b280 [ 4439.223398] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4439.230520] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4439.237642] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4439.244765] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4439.251887] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4439.259009] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4439.266131] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4439.273254] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4439.280375] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4439.287497] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4439.294619] Call trace: [ 4439.297052] __sbq_wake_up+0x84/0x1f0 [ 4439.300702] sbitmap_queue_clear+0x74/0xc0 [ 4439.304786] blk_mq_put_tag+0x44/0x70 [ 4439.308435] __blk_mq_free_request+0x88/0xf0 [ 4439.312692] blk_mq_free_request+0xc0/0x160 [ 4439.316862] __blk_mq_end_request+0x78/0x150 [ 4439.321119] scsi_end_request+0x114/0x260 [ 4439.325115] scsi_io_completion+0x64/0x174 [ 4439.329198] scsi_finish_command+0xe0/0x120 [ 4439.333368] scsi_complete+0x8c/0x1c4 [ 4439.337017] blk_complete_reqs+0x5c/0x74 [ 4439.340927] blk_done_softirq+0x38/0x4c [ 4439.344750] __do_softirq+0x168/0x418 [ 4439.348399] __irq_exit_rcu+0x120/0x170 [ 4439.352221] irq_exit_rcu+0x18/0x24 [ 4439.355697] el1_interrupt+0x38/0x70 [ 4439.359260] el1h_64_irq_handler+0x18/0x24 [ 4439.363343] el1h_64_irq+0x68/0x6c [ 4439.366732] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4439.371510] __wake_up_common_lock+0x84/0xac [ 4439.375767] __wake_up+0x20/0x30 [ 4439.378983] __sbq_wake_up+0x13c/0x1f0 [ 4439.382719] sbitmap_queue_clear+0x74/0xc0 [ 4439.386802] blk_mq_put_tag+0x44/0x70 [ 4439.390452] __blk_mq_free_request+0x88/0xf0 [ 4439.394709] blk_mq_free_request+0xc0/0x160 [ 4439.398879] bfq_insert_request.isra.0+0x204/0x270 [ 4439.403657] bfq_insert_requests+0x70/0x90 [ 4439.407740] blk_mq_sched_insert_requests+0xac/0x250 [ 4439.412692] blk_mq_flush_plug_list+0x110/0x2cc [ 4439.417210] __blk_flush_plug+0x100/0x140 [ 4439.421206] blk_finish_plug+0x40/0x25c [ 4439.425029] __iomap_dio_rw+0x1b4/0x500 [ 4439.428852] iomap_dio_rw+0x18/0x50 [ 4439.432327] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4439.437678] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4439.442421] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4439.447465] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4439.452595] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4439.457724] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4439.463111] process_one_work+0x1ec/0x470 [ 4439.467107] worker_thread+0x74/0x410 [ 4439.470756] kthread+0xec/0x100 [ 4439.473885] ret_from_fork+0x10/0x20 [ 4443.195451] watchdog: BUG: soft lockup - CPU#68 stuck for 3744s! [swapper/68:0] [ 4443.202746] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4443.202763] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4443.290551] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4443.298366] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4443.307657] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4443.314605] pc : __sbq_wake_up+0x58/0x1f0 [ 4443.318601] lr : sbitmap_queue_clear+0x74/0xc0 [ 4443.323031] sp : ffff8000088438b0 [ 4443.326333] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4443.333455] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4443.340577] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4443.347699] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4443.354821] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4443.361943] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4443.369065] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4443.376187] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4443.383308] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4443.390430] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4443.397552] Call trace: [ 4443.399985] __sbq_wake_up+0x58/0x1f0 [ 4443.403634] sbitmap_queue_clear+0x74/0xc0 [ 4443.407717] blk_mq_put_tag+0x44/0x70 [ 4443.411366] __blk_mq_free_request+0x88/0xf0 [ 4443.415623] blk_mq_free_request+0xc0/0x160 [ 4443.419793] __blk_mq_end_request+0x78/0x150 [ 4443.424050] scsi_end_request+0x114/0x260 [ 4443.428046] scsi_io_completion+0x64/0x174 [ 4443.432129] scsi_finish_command+0xe0/0x120 [ 4443.436300] scsi_complete+0x8c/0x1c4 [ 4443.439949] blk_complete_reqs+0x5c/0x74 [ 4443.443858] blk_done_softirq+0x38/0x4c [ 4443.447681] __do_softirq+0x168/0x418 [ 4443.451329] __irq_exit_rcu+0x120/0x170 [ 4443.455152] irq_exit_rcu+0x18/0x24 [ 4443.458628] el1_interrupt+0x38/0x70 [ 4443.462191] el1h_64_irq_handler+0x18/0x24 [ 4443.466275] el1h_64_irq+0x68/0x6c [ 4443.469663] cpuidle_enter_state+0xc4/0x4e0 [ 4443.473833] cpuidle_enter+0x40/0x60 [ 4443.477395] cpuidle_idle_call+0x118/0x1b0 [ 4443.481479] do_idle+0xac/0x100 [ 4443.484607] cpu_startup_entry+0x30/0x34 [ 4443.488516] secondary_start_kernel+0xd8/0xf0 [ 4443.492860] __secondary_switched+0xb0/0xb4 [ 4451.745394] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4451.751126] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=199656 [ 4451.760333] (t=402687 jiffies g=36445 q=118460 ncpus=128) [ 4451.765805] Task dump for CPU 8: [ 4451.769019] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4451.778919] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4451.785087] Call trace: [ 4451.787520] __switch_to+0xd4/0x100 [ 4451.790996] 0xffff6c0200000080 [ 4451.794125] Task dump for CPU 68: [ 4451.797426] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4451.807326] Call trace: [ 4451.809759] dump_backtrace+0xe8/0x130 [ 4451.813495] show_stack+0x20/0x60 [ 4451.816797] sched_show_task+0x160/0x190 [ 4451.820706] dump_cpu_task+0x4c/0x78 [ 4451.824268] rcu_dump_cpu_stacks+0xc4/0x150 [ 4451.828439] print_cpu_stall+0x180/0x210 [ 4451.832348] check_cpu_stall+0x20c/0x2b4 [ 4451.836258] rcu_pending+0x4c/0x190 [ 4451.839733] rcu_sched_clock_irq+0x80/0x204 [ 4451.843903] update_process_times+0x7c/0xc0 [ 4451.848073] tick_sched_handle+0x38/0x74 [ 4451.851983] tick_sched_timer+0x60/0xc0 [ 4451.855806] __hrtimer_run_queues+0x170/0x300 [ 4451.860150] hrtimer_interrupt+0xfc/0x260 [ 4451.864146] arch_timer_handler_phys+0x34/0x50 [ 4451.868576] handle_percpu_devid_irq+0xac/0x240 [ 4451.873094] generic_handle_domain_irq+0x34/0x50 [ 4451.877699] gic_handle_irq+0x5c/0xe0 [ 4451.881348] call_on_irq_stack+0x2c/0x38 [ 4451.885258] do_interrupt_handler+0xdc/0xe0 [ 4451.889428] el1_interrupt+0x34/0x70 [ 4451.892991] el1h_64_irq_handler+0x18/0x24 [ 4451.897076] el1h_64_irq+0x68/0x6c [ 4451.900464] __sbq_wake_up+0x58/0x1f0 [ 4451.904113] sbitmap_queue_clear+0x74/0xc0 [ 4451.908197] blk_mq_put_tag+0x44/0x70 [ 4451.911847] __blk_mq_free_request+0x88/0xf0 [ 4451.916103] blk_mq_free_request+0xc0/0x160 [ 4451.920274] __blk_mq_end_request+0x78/0x150 [ 4451.924530] scsi_end_request+0x114/0x260 [ 4451.928527] scsi_io_completion+0x64/0x174 [ 4451.932610] scsi_finish_command+0xe0/0x120 [ 4451.936780] scsi_complete+0x8c/0x1c4 [ 4451.940429] blk_complete_reqs+0x5c/0x74 [ 4451.944339] blk_done_softirq+0x38/0x4c [ 4451.948162] __do_softirq+0x168/0x418 [ 4451.951811] __irq_exit_rcu+0x120/0x170 [ 4451.955633] irq_exit_rcu+0x18/0x24 [ 4451.959109] el1_interrupt+0x38/0x70 [ 4451.962671] el1h_64_irq_handler+0x18/0x24 [ 4451.966754] el1h_64_irq+0x68/0x6c [ 4451.970143] cpuidle_enter_state+0xc4/0x4e0 [ 4451.974313] cpuidle_enter+0x40/0x60 [ 4451.977875] cpuidle_idle_call+0x118/0x1b0 [ 4451.981959] do_idle+0xac/0x100 [ 4451.985087] cpu_startup_entry+0x30/0x34 [ 4451.988996] secondary_start_kernel+0xd8/0xf0 [ 4451.993341] __secondary_switched+0xb0/0xb4 [ 4463.085317] watchdog: BUG: soft lockup - CPU#8 stuck for 3762s! [kworker/8:12:19399] [ 4463.093046] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4463.093063] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4463.180851] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4463.189101] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4463.198392] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4463.204559] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4463.211507] pc : __sbq_wake_up+0x1c8/0x1f0 [ 4463.215590] lr : sbitmap_queue_clear+0x74/0xc0 [ 4463.220021] sp : ffff80000d79b280 [ 4463.223323] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4463.230445] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4463.237567] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4463.244689] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4463.251811] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4463.258933] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4463.266055] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4463.273177] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4463.280300] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4463.287422] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 4463.294544] Call trace: [ 4463.296977] __sbq_wake_up+0x1c8/0x1f0 [ 4463.300713] sbitmap_queue_clear+0x74/0xc0 [ 4463.304796] blk_mq_put_tag+0x44/0x70 [ 4463.308446] __blk_mq_free_request+0x88/0xf0 [ 4463.312703] blk_mq_free_request+0xc0/0x160 [ 4463.316873] __blk_mq_end_request+0x78/0x150 [ 4463.321130] scsi_end_request+0x114/0x260 [ 4463.325127] scsi_io_completion+0x64/0x174 [ 4463.329210] scsi_finish_command+0xe0/0x120 [ 4463.333380] scsi_complete+0x8c/0x1c4 [ 4463.337029] blk_complete_reqs+0x5c/0x74 [ 4463.340939] blk_done_softirq+0x38/0x4c [ 4463.344761] __do_softirq+0x168/0x418 [ 4463.348410] __irq_exit_rcu+0x120/0x170 [ 4463.352233] irq_exit_rcu+0x18/0x24 [ 4463.355709] el1_interrupt+0x38/0x70 [ 4463.359271] el1h_64_irq_handler+0x18/0x24 [ 4463.363355] el1h_64_irq+0x68/0x6c [ 4463.366743] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4463.371521] __wake_up_common_lock+0x84/0xac [ 4463.375778] __wake_up+0x20/0x30 [ 4463.378994] __sbq_wake_up+0x13c/0x1f0 [ 4463.382730] sbitmap_queue_clear+0x74/0xc0 [ 4463.386813] blk_mq_put_tag+0x44/0x70 [ 4463.390463] __blk_mq_free_request+0x88/0xf0 [ 4463.394719] blk_mq_free_request+0xc0/0x160 [ 4463.398889] bfq_insert_request.isra.0+0x204/0x270 [ 4463.403667] bfq_insert_requests+0x70/0x90 [ 4463.407751] blk_mq_sched_insert_requests+0xac/0x250 [ 4463.412703] blk_mq_flush_plug_list+0x110/0x2cc [ 4463.417221] __blk_flush_plug+0x100/0x140 [ 4463.421217] blk_finish_plug+0x40/0x25c [ 4463.425040] __iomap_dio_rw+0x1b4/0x500 [ 4463.428863] iomap_dio_rw+0x18/0x50 [ 4463.432338] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4463.437689] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4463.442431] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4463.447475] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4463.452605] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4463.457735] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4463.463121] process_one_work+0x1ec/0x470 [ 4463.467117] worker_thread+0x74/0x410 [ 4463.470766] kthread+0xec/0x100 [ 4463.473895] ret_from_fork+0x10/0x20 [ 4465.625309] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 401469 jiffies s: 2093 root: 0x11/. [ 4465.637047] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4465.646256] Task dump for CPU 8: [ 4465.649470] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4465.659372] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4465.665541] Call trace: [ 4465.667974] __switch_to+0xd4/0x100 [ 4465.671451] 0xffff6c0200000080 [ 4465.674580] Task dump for CPU 68: [ 4465.677883] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4465.687785] Call trace: [ 4465.690218] __switch_to+0xd4/0x100 [ 4465.693694] 0x0 [ 4479.195208] watchdog: BUG: soft lockup - CPU#68 stuck for 3777s! [swapper/68:0] [ 4479.202503] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4479.202520] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4479.290307] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4479.298123] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4479.307414] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4479.314362] pc : __sbq_wake_up+0x84/0x1f0 [ 4479.318358] lr : sbitmap_queue_clear+0x74/0xc0 [ 4479.322788] sp : ffff8000088438b0 [ 4479.326090] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4479.333211] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4479.340333] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4479.347456] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4479.354578] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4479.361700] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4479.368822] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4479.375945] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4479.383066] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4479.390188] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4479.397310] Call trace: [ 4479.399743] __sbq_wake_up+0x84/0x1f0 [ 4479.403393] sbitmap_queue_clear+0x74/0xc0 [ 4479.407476] blk_mq_put_tag+0x44/0x70 [ 4479.411126] __blk_mq_free_request+0x88/0xf0 [ 4479.415382] blk_mq_free_request+0xc0/0x160 [ 4479.419552] __blk_mq_end_request+0x78/0x150 [ 4479.423809] scsi_end_request+0x114/0x260 [ 4479.427805] scsi_io_completion+0x64/0x174 [ 4479.431888] scsi_finish_command+0xe0/0x120 [ 4479.436059] scsi_complete+0x8c/0x1c4 [ 4479.439708] blk_complete_reqs+0x5c/0x74 [ 4479.443617] blk_done_softirq+0x38/0x4c [ 4479.447440] __do_softirq+0x168/0x418 [ 4479.451089] __irq_exit_rcu+0x120/0x170 [ 4479.454911] irq_exit_rcu+0x18/0x24 [ 4479.458387] el1_interrupt+0x38/0x70 [ 4479.461949] el1h_64_irq_handler+0x18/0x24 [ 4479.466032] el1h_64_irq+0x68/0x6c [ 4479.469420] cpuidle_enter_state+0xc4/0x4e0 [ 4479.473590] cpuidle_enter+0x40/0x60 [ 4479.477153] cpuidle_idle_call+0x118/0x1b0 [ 4479.481236] do_idle+0xac/0x100 [ 4479.484365] cpu_startup_entry+0x30/0x34 [ 4479.488274] secondary_start_kernel+0xd8/0xf0 [ 4479.492618] __secondary_switched+0xb0/0xb4 [ 4487.085155] watchdog: BUG: soft lockup - CPU#8 stuck for 3784s! [kworker/8:12:19399] [ 4487.092884] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4487.092901] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4487.180689] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4487.188939] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4487.198230] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4487.204397] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4487.211345] pc : __sbq_wake_up+0x7c/0x1f0 [ 4487.215342] lr : sbitmap_queue_clear+0x74/0xc0 [ 4487.219773] sp : ffff80000d79b2d0 [ 4487.223074] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4487.230196] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4487.237318] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4487.244440] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4487.251562] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4487.258684] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4487.265806] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4487.272928] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4487.280050] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4487.287171] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 4487.294293] Call trace: [ 4487.296726] __sbq_wake_up+0x7c/0x1f0 [ 4487.300376] blk_mq_put_tag+0x44/0x70 [ 4487.304025] __blk_mq_free_request+0x88/0xf0 [ 4487.308282] blk_mq_free_request+0xc0/0x160 [ 4487.312452] __blk_mq_end_request+0x78/0x150 [ 4487.316709] scsi_end_request+0x114/0x260 [ 4487.320705] scsi_io_completion+0x64/0x174 [ 4487.324788] scsi_finish_command+0xe0/0x120 [ 4487.328959] scsi_complete+0x8c/0x1c4 [ 4487.332608] blk_complete_reqs+0x5c/0x74 [ 4487.336518] blk_done_softirq+0x38/0x4c [ 4487.340340] __do_softirq+0x168/0x418 [ 4487.343989] __irq_exit_rcu+0x120/0x170 [ 4487.347811] irq_exit_rcu+0x18/0x24 [ 4487.351287] el1_interrupt+0x38/0x70 [ 4487.354849] el1h_64_irq_handler+0x18/0x24 [ 4487.358933] el1h_64_irq+0x68/0x6c [ 4487.362321] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4487.367098] __wake_up_common_lock+0x84/0xac [ 4487.371356] __wake_up+0x20/0x30 [ 4487.374571] __sbq_wake_up+0x13c/0x1f0 [ 4487.378307] sbitmap_queue_clear+0x74/0xc0 [ 4487.382390] blk_mq_put_tag+0x44/0x70 [ 4487.386040] __blk_mq_free_request+0x88/0xf0 [ 4487.390297] blk_mq_free_request+0xc0/0x160 [ 4487.394466] bfq_insert_request.isra.0+0x204/0x270 [ 4487.399245] bfq_insert_requests+0x70/0x90 [ 4487.403328] blk_mq_sched_insert_requests+0xac/0x250 [ 4487.408280] blk_mq_flush_plug_list+0x110/0x2cc [ 4487.412797] __blk_flush_plug+0x100/0x140 [ 4487.416793] blk_finish_plug+0x40/0x25c [ 4487.420616] __iomap_dio_rw+0x1b4/0x500 [ 4487.424439] iomap_dio_rw+0x18/0x50 [ 4487.427914] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4487.433265] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4487.438007] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4487.443051] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4487.448181] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4487.453311] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4487.458696] process_one_work+0x1ec/0x470 [ 4487.462693] worker_thread+0x74/0x410 [ 4487.466341] kthread+0xec/0x100 [ 4487.469470] ret_from_fork+0x10/0x20 [ 4503.195046] watchdog: BUG: soft lockup - CPU#68 stuck for 3799s! [swapper/68:0] [ 4503.202341] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4503.202358] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4503.290145] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4503.297961] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4503.307252] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4503.314199] pc : __sbq_wake_up+0x40/0x1f0 [ 4503.318196] lr : sbitmap_queue_clear+0x74/0xc0 [ 4503.322627] sp : ffff8000088438b0 [ 4503.325928] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4503.333050] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4503.340172] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4503.347293] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4503.354415] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4503.361537] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4503.368659] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4503.375782] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4503.382904] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4503.390025] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4503.397147] Call trace: [ 4503.399580] __sbq_wake_up+0x40/0x1f0 [ 4503.403229] sbitmap_queue_clear+0x74/0xc0 [ 4503.407312] blk_mq_put_tag+0x44/0x70 [ 4503.410962] __blk_mq_free_request+0x88/0xf0 [ 4503.415218] blk_mq_free_request+0xc0/0x160 [ 4503.419388] __blk_mq_end_request+0x78/0x150 [ 4503.423645] scsi_end_request+0x114/0x260 [ 4503.427641] scsi_io_completion+0x64/0x174 [ 4503.431724] scsi_finish_command+0xe0/0x120 [ 4503.435895] scsi_complete+0x8c/0x1c4 [ 4503.439544] blk_complete_reqs+0x5c/0x74 [ 4503.443453] blk_done_softirq+0x38/0x4c [ 4503.447276] __do_softirq+0x168/0x418 [ 4503.450925] __irq_exit_rcu+0x120/0x170 [ 4503.454747] irq_exit_rcu+0x18/0x24 [ 4503.458223] el1_interrupt+0x38/0x70 [ 4503.461785] el1h_64_irq_handler+0x18/0x24 [ 4503.465868] el1h_64_irq+0x68/0x6c [ 4503.469257] cpuidle_enter_state+0xc4/0x4e0 [ 4503.473427] cpuidle_enter+0x40/0x60 [ 4503.476989] cpuidle_idle_call+0x118/0x1b0 [ 4503.481073] do_idle+0xac/0x100 [ 4503.484201] cpu_startup_entry+0x30/0x34 [ 4503.488110] secondary_start_kernel+0xd8/0xf0 [ 4503.492455] __secondary_switched+0xb0/0xb4 [ 4511.084993] watchdog: BUG: soft lockup - CPU#8 stuck for 3807s! [kworker/8:12:19399] [ 4511.092722] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4511.092739] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4511.180526] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4511.188776] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4511.198067] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4511.204234] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4511.211182] pc : __sbq_wake_up+0x58/0x1f0 [ 4511.215178] lr : sbitmap_queue_clear+0x74/0xc0 [ 4511.219609] sp : ffff80000d79b280 [ 4511.222910] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4511.230032] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4511.237154] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4511.244277] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4511.251399] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4511.258521] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4511.265643] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4511.272765] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4511.279887] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4511.287009] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4511.294130] Call trace: [ 4511.296564] __sbq_wake_up+0x58/0x1f0 [ 4511.300213] sbitmap_queue_clear+0x74/0xc0 [ 4511.304296] blk_mq_put_tag+0x44/0x70 [ 4511.307947] __blk_mq_free_request+0x88/0xf0 [ 4511.312205] blk_mq_free_request+0xc0/0x160 [ 4511.316375] __blk_mq_end_request+0x78/0x150 [ 4511.320632] scsi_end_request+0x114/0x260 [ 4511.324628] scsi_io_completion+0x64/0x174 [ 4511.328711] scsi_finish_command+0xe0/0x120 [ 4511.332882] scsi_complete+0x8c/0x1c4 [ 4511.336531] blk_complete_reqs+0x5c/0x74 [ 4511.340440] blk_done_softirq+0x38/0x4c [ 4511.344263] __do_softirq+0x168/0x418 [ 4511.347912] __irq_exit_rcu+0x120/0x170 [ 4511.351735] irq_exit_rcu+0x18/0x24 [ 4511.355210] el1_interrupt+0x38/0x70 [ 4511.358773] el1h_64_irq_handler+0x18/0x24 [ 4511.362856] el1h_64_irq+0x68/0x6c [ 4511.366244] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4511.371022] __wake_up_common_lock+0x84/0xac [ 4511.375280] __wake_up+0x20/0x30 [ 4511.378495] __sbq_wake_up+0x13c/0x1f0 [ 4511.382232] sbitmap_queue_clear+0x74/0xc0 [ 4511.386315] blk_mq_put_tag+0x44/0x70 [ 4511.389964] __blk_mq_free_request+0x88/0xf0 [ 4511.394221] blk_mq_free_request+0xc0/0x160 [ 4511.398391] bfq_insert_request.isra.0+0x204/0x270 [ 4511.403169] bfq_insert_requests+0x70/0x90 [ 4511.407253] blk_mq_sched_insert_requests+0xac/0x250 [ 4511.412204] blk_mq_flush_plug_list+0x110/0x2cc [ 4511.416722] __blk_flush_plug+0x100/0x140 [ 4511.420718] blk_finish_plug+0x40/0x25c [ 4511.424541] __iomap_dio_rw+0x1b4/0x500 [ 4511.428364] iomap_dio_rw+0x18/0x50 [ 4511.431839] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4511.437190] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4511.441932] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4511.446976] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4511.452106] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4511.457236] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4511.462622] process_one_work+0x1ec/0x470 [ 4511.466618] worker_thread+0x74/0x410 [ 4511.470267] kthread+0xec/0x100 [ 4511.473396] ret_from_fork+0x10/0x20 [ 4527.194884] watchdog: BUG: soft lockup - CPU#68 stuck for 3822s! [swapper/68:0] [ 4527.202180] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4527.202197] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4527.289984] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4527.297800] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4527.307090] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4527.314038] pc : __sbq_wake_up+0x20/0x1f0 [ 4527.318034] lr : sbitmap_queue_clear+0x74/0xc0 [ 4527.322465] sp : ffff8000088438b0 [ 4527.325766] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4527.332888] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4527.340010] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4527.347132] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4527.354253] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4527.361375] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4527.368497] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4527.375619] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4527.382741] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4527.389863] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4527.396985] Call trace: [ 4527.399418] __sbq_wake_up+0x20/0x1f0 [ 4527.403067] sbitmap_queue_clear+0x74/0xc0 [ 4527.407150] blk_mq_put_tag+0x44/0x70 [ 4527.410799] __blk_mq_free_request+0x88/0xf0 [ 4527.415056] blk_mq_free_request+0xc0/0x160 [ 4527.419226] __blk_mq_end_request+0x78/0x150 [ 4527.423483] scsi_end_request+0x114/0x260 [ 4527.427479] scsi_io_completion+0x64/0x174 [ 4527.431562] scsi_finish_command+0xe0/0x120 [ 4527.435733] scsi_complete+0x8c/0x1c4 [ 4527.439382] blk_complete_reqs+0x5c/0x74 [ 4527.443291] blk_done_softirq+0x38/0x4c [ 4527.447114] __do_softirq+0x168/0x418 [ 4527.450763] __irq_exit_rcu+0x120/0x170 [ 4527.454586] irq_exit_rcu+0x18/0x24 [ 4527.458061] el1_interrupt+0x38/0x70 [ 4527.461623] el1h_64_irq_handler+0x18/0x24 [ 4527.465706] el1h_64_irq+0x68/0x6c [ 4527.469095] cpuidle_enter_state+0xc4/0x4e0 [ 4527.473265] cpuidle_enter+0x40/0x60 [ 4527.476828] cpuidle_idle_call+0x118/0x1b0 [ 4527.480911] do_idle+0xac/0x100 [ 4527.484040] cpu_startup_entry+0x30/0x34 [ 4527.487949] secondary_start_kernel+0xd8/0xf0 [ 4527.492293] __secondary_switched+0xb0/0xb4 [ 4532.184860] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 408125 jiffies s: 2093 root: 0x11/. [ 4532.196598] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4532.205806] Task dump for CPU 8: [ 4532.209021] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4532.218923] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4532.225093] Call trace: [ 4532.227526] __switch_to+0xd4/0x100 [ 4532.231002] 0xffff6c0200000080 [ 4532.234131] Task dump for CPU 68: [ 4532.237434] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4532.247335] Call trace: [ 4532.249768] __switch_to+0xd4/0x100 [ 4532.253244] 0x0 [ 4535.084831] watchdog: BUG: soft lockup - CPU#8 stuck for 3829s! [kworker/8:12:19399] [ 4535.092560] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4535.092577] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4535.180365] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4535.188614] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4535.197905] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4535.204073] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4535.211020] pc : __sbq_wake_up+0x84/0x1f0 [ 4535.215017] lr : sbitmap_queue_clear+0x74/0xc0 [ 4535.219448] sp : ffff80000d79b280 [ 4535.222751] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4535.229873] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4535.236996] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4535.244117] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4535.251240] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4535.258362] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4535.265484] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4535.272606] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4535.279728] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4535.286850] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4535.293972] Call trace: [ 4535.296405] __sbq_wake_up+0x84/0x1f0 [ 4535.300055] sbitmap_queue_clear+0x74/0xc0 [ 4535.304138] blk_mq_put_tag+0x44/0x70 [ 4535.307788] __blk_mq_free_request+0x88/0xf0 [ 4535.312045] blk_mq_free_request+0xc0/0x160 [ 4535.316215] __blk_mq_end_request+0x78/0x150 [ 4535.320472] scsi_end_request+0x114/0x260 [ 4535.324468] scsi_io_completion+0x64/0x174 [ 4535.328551] scsi_finish_command+0xe0/0x120 [ 4535.332722] scsi_complete+0x8c/0x1c4 [ 4535.336371] blk_complete_reqs+0x5c/0x74 [ 4535.340280] blk_done_softirq+0x38/0x4c [ 4535.344103] __do_softirq+0x168/0x418 [ 4535.347752] __irq_exit_rcu+0x120/0x170 [ 4535.351575] irq_exit_rcu+0x18/0x24 [ 4535.355050] el1_interrupt+0x38/0x70 [ 4535.358612] el1h_64_irq_handler+0x18/0x24 [ 4535.362695] el1h_64_irq+0x68/0x6c [ 4535.366084] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4535.370862] __wake_up_common_lock+0x84/0xac [ 4535.375119] __wake_up+0x20/0x30 [ 4535.378335] __sbq_wake_up+0x13c/0x1f0 [ 4535.382070] sbitmap_queue_clear+0x74/0xc0 [ 4535.386154] blk_mq_put_tag+0x44/0x70 [ 4535.389803] __blk_mq_free_request+0x88/0xf0 [ 4535.394060] blk_mq_free_request+0xc0/0x160 [ 4535.398230] bfq_insert_request.isra.0+0x204/0x270 [ 4535.403008] bfq_insert_requests+0x70/0x90 [ 4535.407091] blk_mq_sched_insert_requests+0xac/0x250 [ 4535.412043] blk_mq_flush_plug_list+0x110/0x2cc [ 4535.416561] __blk_flush_plug+0x100/0x140 [ 4535.420558] blk_finish_plug+0x40/0x25c [ 4535.424380] __iomap_dio_rw+0x1b4/0x500 [ 4535.428203] iomap_dio_rw+0x18/0x50 [ 4535.431679] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4535.437030] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4535.441772] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4535.446816] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4535.451947] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4535.457077] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4535.462463] process_one_work+0x1ec/0x470 [ 4535.466459] worker_thread+0x74/0x410 [ 4535.470108] kthread+0xec/0x100 [ 4535.473237] ret_from_fork+0x10/0x20 [ 4551.194722] watchdog: BUG: soft lockup - CPU#68 stuck for 3844s! [swapper/68:0] [ 4551.202017] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4551.202034] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4551.289821] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4551.297637] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4551.306928] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4551.313875] pc : __sbq_wake_up+0x58/0x1f0 [ 4551.317871] lr : sbitmap_queue_clear+0x74/0xc0 [ 4551.322302] sp : ffff8000088438b0 [ 4551.325603] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4551.332725] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4551.339847] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4551.346969] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4551.354091] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4551.361213] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4551.368335] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4551.375457] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4551.382579] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4551.389702] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4551.396824] Call trace: [ 4551.399257] __sbq_wake_up+0x58/0x1f0 [ 4551.402907] sbitmap_queue_clear+0x74/0xc0 [ 4551.406990] blk_mq_put_tag+0x44/0x70 [ 4551.410640] __blk_mq_free_request+0x88/0xf0 [ 4551.414896] blk_mq_free_request+0xc0/0x160 [ 4551.419066] __blk_mq_end_request+0x78/0x150 [ 4551.423323] scsi_end_request+0x114/0x260 [ 4551.427319] scsi_io_completion+0x64/0x174 [ 4551.431402] scsi_finish_command+0xe0/0x120 [ 4551.435573] scsi_complete+0x8c/0x1c4 [ 4551.439222] blk_complete_reqs+0x5c/0x74 [ 4551.443131] blk_done_softirq+0x38/0x4c [ 4551.446954] __do_softirq+0x168/0x418 [ 4551.450603] __irq_exit_rcu+0x120/0x170 [ 4551.454426] irq_exit_rcu+0x18/0x24 [ 4551.457901] el1_interrupt+0x38/0x70 [ 4551.461464] el1h_64_irq_handler+0x18/0x24 [ 4551.465547] el1h_64_irq+0x68/0x6c [ 4551.468935] cpuidle_enter_state+0xc4/0x4e0 [ 4551.473105] cpuidle_enter+0x40/0x60 [ 4551.476668] cpuidle_idle_call+0x118/0x1b0 [ 4551.480751] do_idle+0xac/0x100 [ 4551.483880] cpu_startup_entry+0x30/0x34 [ 4551.487791] secondary_start_kernel+0xd8/0xf0 [ 4551.492135] __secondary_switched+0xb0/0xb4 [ 4559.084669] watchdog: BUG: soft lockup - CPU#8 stuck for 3851s! [kworker/8:12:19399] [ 4559.092398] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4559.092416] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4559.180203] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4559.188453] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4559.197744] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4559.203911] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4559.210859] pc : __sbq_wake_up+0x7c/0x1f0 [ 4559.214855] lr : sbitmap_queue_clear+0x74/0xc0 [ 4559.219286] sp : ffff80000d79b2d0 [ 4559.222587] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4559.229709] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4559.236832] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4559.243954] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4559.251076] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4559.258198] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4559.265319] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4559.272441] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4559.279564] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4559.286685] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 4559.293807] Call trace: [ 4559.296240] __sbq_wake_up+0x7c/0x1f0 [ 4559.299889] blk_mq_put_tag+0x44/0x70 [ 4559.303539] __blk_mq_free_request+0x88/0xf0 [ 4559.307795] blk_mq_free_request+0xc0/0x160 [ 4559.311965] __blk_mq_end_request+0x78/0x150 [ 4559.316223] scsi_end_request+0x114/0x260 [ 4559.320219] scsi_io_completion+0x64/0x174 [ 4559.324302] scsi_finish_command+0xe0/0x120 [ 4559.328472] scsi_complete+0x8c/0x1c4 [ 4559.332121] blk_complete_reqs+0x5c/0x74 [ 4559.336031] blk_done_softirq+0x38/0x4c [ 4559.339854] __do_softirq+0x168/0x418 [ 4559.343503] __irq_exit_rcu+0x120/0x170 [ 4559.347325] irq_exit_rcu+0x18/0x24 [ 4559.350801] el1_interrupt+0x38/0x70 [ 4559.354363] el1h_64_irq_handler+0x18/0x24 [ 4559.358446] el1h_64_irq+0x68/0x6c [ 4559.361835] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4559.366612] __wake_up_common_lock+0x84/0xac [ 4559.370870] __wake_up+0x20/0x30 [ 4559.374086] __sbq_wake_up+0x13c/0x1f0 [ 4559.377822] sbitmap_queue_clear+0x74/0xc0 [ 4559.381906] blk_mq_put_tag+0x44/0x70 [ 4559.385555] __blk_mq_free_request+0x88/0xf0 [ 4559.389812] blk_mq_free_request+0xc0/0x160 [ 4559.393982] bfq_insert_request.isra.0+0x204/0x270 [ 4559.398760] bfq_insert_requests+0x70/0x90 [ 4559.402844] blk_mq_sched_insert_requests+0xac/0x250 [ 4559.407796] blk_mq_flush_plug_list+0x110/0x2cc [ 4559.412313] __blk_flush_plug+0x100/0x140 [ 4559.416309] blk_finish_plug+0x40/0x25c [ 4559.420132] __iomap_dio_rw+0x1b4/0x500 [ 4559.423955] iomap_dio_rw+0x18/0x50 [ 4559.427431] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4559.432781] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4559.437524] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4559.442568] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4559.447698] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4559.452827] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4559.458214] process_one_work+0x1ec/0x470 [ 4559.462210] worker_thread+0x74/0x410 [ 4559.465859] kthread+0xec/0x100 [ 4559.468988] ret_from_fork+0x10/0x20 [ 4575.194560] watchdog: BUG: soft lockup - CPU#68 stuck for 3866s! [swapper/68:0] [ 4575.201855] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4575.201872] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4575.289661] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4575.297476] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4575.306767] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4575.313716] pc : __sbq_wake_up+0x20/0x1f0 [ 4575.317712] lr : sbitmap_queue_clear+0x74/0xc0 [ 4575.322143] sp : ffff8000088438b0 [ 4575.325445] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4575.332566] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4575.339688] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4575.346810] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4575.353932] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4575.361054] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4575.368176] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4575.375297] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4575.382419] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4575.389541] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4575.396663] Call trace: [ 4575.399096] __sbq_wake_up+0x20/0x1f0 [ 4575.402745] sbitmap_queue_clear+0x74/0xc0 [ 4575.406828] blk_mq_put_tag+0x44/0x70 [ 4575.410478] __blk_mq_free_request+0x88/0xf0 [ 4575.414735] blk_mq_free_request+0xc0/0x160 [ 4575.418904] __blk_mq_end_request+0x78/0x150 [ 4575.423161] scsi_end_request+0x114/0x260 [ 4575.427158] scsi_io_completion+0x64/0x174 [ 4575.431241] scsi_finish_command+0xe0/0x120 [ 4575.435411] scsi_complete+0x8c/0x1c4 [ 4575.439060] blk_complete_reqs+0x5c/0x74 [ 4575.442970] blk_done_softirq+0x38/0x4c [ 4575.446792] __do_softirq+0x168/0x418 [ 4575.450441] __irq_exit_rcu+0x120/0x170 [ 4575.454264] irq_exit_rcu+0x18/0x24 [ 4575.457739] el1_interrupt+0x38/0x70 [ 4575.461301] el1h_64_irq_handler+0x18/0x24 [ 4575.465384] el1h_64_irq+0x68/0x6c [ 4575.468773] cpuidle_enter_state+0xc4/0x4e0 [ 4575.472943] cpuidle_enter+0x40/0x60 [ 4575.476506] cpuidle_idle_call+0x118/0x1b0 [ 4575.480590] do_idle+0xac/0x100 [ 4575.483719] cpu_startup_entry+0x30/0x34 [ 4575.487628] secondary_start_kernel+0xd8/0xf0 [ 4575.491972] __secondary_switched+0xb0/0xb4 [ 4583.084507] watchdog: BUG: soft lockup - CPU#8 stuck for 3874s! [kworker/8:12:19399] [ 4583.092236] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4583.092254] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4583.180044] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4583.188293] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4583.197584] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4583.203752] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4583.210699] pc : __sbq_wake_up+0x58/0x1f0 [ 4583.214696] lr : sbitmap_queue_clear+0x74/0xc0 [ 4583.219126] sp : ffff80000d79b280 [ 4583.222428] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4583.229550] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4583.236674] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4583.243796] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4583.250918] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4583.258040] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4583.265162] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4583.272284] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4583.279406] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4583.286528] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4583.293650] Call trace: [ 4583.296084] __sbq_wake_up+0x58/0x1f0 [ 4583.299733] sbitmap_queue_clear+0x74/0xc0 [ 4583.303817] blk_mq_put_tag+0x44/0x70 [ 4583.307466] __blk_mq_free_request+0x88/0xf0 [ 4583.311723] blk_mq_free_request+0xc0/0x160 [ 4583.315893] __blk_mq_end_request+0x78/0x150 [ 4583.320150] scsi_end_request+0x114/0x260 [ 4583.324146] scsi_io_completion+0x64/0x174 [ 4583.328229] scsi_finish_command+0xe0/0x120 [ 4583.332400] scsi_complete+0x8c/0x1c4 [ 4583.336049] blk_complete_reqs+0x5c/0x74 [ 4583.339958] blk_done_softirq+0x38/0x4c [ 4583.343781] __do_softirq+0x168/0x418 [ 4583.347429] __irq_exit_rcu+0x120/0x170 [ 4583.351252] irq_exit_rcu+0x18/0x24 [ 4583.354729] el1_interrupt+0x38/0x70 [ 4583.358291] el1h_64_irq_handler+0x18/0x24 [ 4583.362375] el1h_64_irq+0x68/0x6c [ 4583.365763] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4583.370541] __wake_up_common_lock+0x84/0xac [ 4583.374798] __wake_up+0x20/0x30 [ 4583.378014] __sbq_wake_up+0x13c/0x1f0 [ 4583.381750] sbitmap_queue_clear+0x74/0xc0 [ 4583.385833] blk_mq_put_tag+0x44/0x70 [ 4583.389483] __blk_mq_free_request+0x88/0xf0 [ 4583.393739] blk_mq_free_request+0xc0/0x160 [ 4583.397909] bfq_insert_request.isra.0+0x204/0x270 [ 4583.402687] bfq_insert_requests+0x70/0x90 [ 4583.406771] blk_mq_sched_insert_requests+0xac/0x250 [ 4583.411723] blk_mq_flush_plug_list+0x110/0x2cc [ 4583.416240] __blk_flush_plug+0x100/0x140 [ 4583.420237] blk_finish_plug+0x40/0x25c [ 4583.424060] __iomap_dio_rw+0x1b4/0x500 [ 4583.427883] iomap_dio_rw+0x18/0x50 [ 4583.431358] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4583.436709] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4583.441453] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4583.446496] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4583.451627] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4583.456756] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4583.462143] process_one_work+0x1ec/0x470 [ 4583.466139] worker_thread+0x74/0x410 [ 4583.469787] kthread+0xec/0x100 [ 4583.472916] ret_from_fork+0x10/0x20 [ 4598.744411] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 414781 jiffies s: 2093 root: 0x11/. [ 4598.756150] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4598.765359] Task dump for CPU 8: [ 4598.768573] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4598.778475] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4598.784645] Call trace: [ 4598.787078] __switch_to+0xd4/0x100 [ 4598.790555] 0xffff6c0200000080 [ 4598.793683] Task dump for CPU 68: [ 4598.796986] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4598.806889] Call trace: [ 4598.809323] __switch_to+0xd4/0x100 [ 4598.812799] 0x0 [ 4599.194398] watchdog: BUG: soft lockup - CPU#68 stuck for 3889s! [swapper/68:0] [ 4599.201693] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4599.201710] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4599.289498] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4599.297314] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4599.306605] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4599.313553] pc : __sbq_wake_up+0x4/0x1f0 [ 4599.317463] lr : sbitmap_queue_clear+0x74/0xc0 [ 4599.321893] sp : ffff800008843900 [ 4599.325194] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4599.332317] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4599.339439] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4599.346561] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4599.353683] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4599.360807] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4599.367929] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4599.375051] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4599.382174] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4599.389296] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 4599.396418] Call trace: [ 4599.398851] __sbq_wake_up+0x4/0x1f0 [ 4599.402413] blk_mq_put_tag+0x44/0x70 [ 4599.406063] __blk_mq_free_request+0x88/0xf0 [ 4599.410319] blk_mq_free_request+0xc0/0x160 [ 4599.414489] __blk_mq_end_request+0x78/0x150 [ 4599.418746] scsi_end_request+0x114/0x260 [ 4599.422742] scsi_io_completion+0x64/0x174 [ 4599.426825] scsi_finish_command+0xe0/0x120 [ 4599.430996] scsi_complete+0x8c/0x1c4 [ 4599.434645] blk_complete_reqs+0x5c/0x74 [ 4599.438554] blk_done_softirq+0x38/0x4c [ 4599.442377] __do_softirq+0x168/0x418 [ 4599.446025] __irq_exit_rcu+0x120/0x170 [ 4599.449848] irq_exit_rcu+0x18/0x24 [ 4599.453324] el1_interrupt+0x38/0x70 [ 4599.456886] el1h_64_irq_handler+0x18/0x24 [ 4599.460969] el1h_64_irq+0x68/0x6c [ 4599.464358] cpuidle_enter_state+0xc4/0x4e0 [ 4599.468528] cpuidle_enter+0x40/0x60 [ 4599.472090] cpuidle_idle_call+0x118/0x1b0 [ 4599.476174] do_idle+0xac/0x100 [ 4599.479303] cpu_startup_entry+0x30/0x34 [ 4599.483212] secondary_start_kernel+0xd8/0xf0 [ 4599.487556] __secondary_switched+0xb0/0xb4 [ 4607.084345] watchdog: BUG: soft lockup - CPU#8 stuck for 3896s! [kworker/8:12:19399] [ 4607.092074] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4607.092091] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4607.179879] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4607.188129] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4607.197420] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4607.203587] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4607.210535] pc : __sbq_wake_up+0x58/0x1f0 [ 4607.214531] lr : sbitmap_queue_clear+0x74/0xc0 [ 4607.218962] sp : ffff80000d79b280 [ 4607.222264] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4607.229387] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4607.236509] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4607.243631] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4607.250753] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4607.257875] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4607.264997] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4607.272120] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4607.279242] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4607.286363] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4607.293485] Call trace: [ 4607.295919] __sbq_wake_up+0x58/0x1f0 [ 4607.299568] sbitmap_queue_clear+0x74/0xc0 [ 4607.303652] blk_mq_put_tag+0x44/0x70 [ 4607.307303] __blk_mq_free_request+0x88/0xf0 [ 4607.311560] blk_mq_free_request+0xc0/0x160 [ 4607.315730] __blk_mq_end_request+0x78/0x150 [ 4607.319987] scsi_end_request+0x114/0x260 [ 4607.323983] scsi_io_completion+0x64/0x174 [ 4607.328066] scsi_finish_command+0xe0/0x120 [ 4607.332237] scsi_complete+0x8c/0x1c4 [ 4607.335886] blk_complete_reqs+0x5c/0x74 [ 4607.339795] blk_done_softirq+0x38/0x4c [ 4607.343618] __do_softirq+0x168/0x418 [ 4607.347266] __irq_exit_rcu+0x120/0x170 [ 4607.351089] irq_exit_rcu+0x18/0x24 [ 4607.354565] el1_interrupt+0x38/0x70 [ 4607.358127] el1h_64_irq_handler+0x18/0x24 [ 4607.362210] el1h_64_irq+0x68/0x6c [ 4607.365599] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4607.370376] __wake_up_common_lock+0x84/0xac [ 4607.374633] __wake_up+0x20/0x30 [ 4607.377849] __sbq_wake_up+0x13c/0x1f0 [ 4607.381585] sbitmap_queue_clear+0x74/0xc0 [ 4607.385668] blk_mq_put_tag+0x44/0x70 [ 4607.389318] __blk_mq_free_request+0x88/0xf0 [ 4607.393574] blk_mq_free_request+0xc0/0x160 [ 4607.397744] bfq_insert_request.isra.0+0x204/0x270 [ 4607.402522] bfq_insert_requests+0x70/0x90 [ 4607.406606] blk_mq_sched_insert_requests+0xac/0x250 [ 4607.411558] blk_mq_flush_plug_list+0x110/0x2cc [ 4607.416075] __blk_flush_plug+0x100/0x140 [ 4607.420072] blk_finish_plug+0x40/0x25c [ 4607.423894] __iomap_dio_rw+0x1b4/0x500 [ 4607.427717] iomap_dio_rw+0x18/0x50 [ 4607.431192] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4607.436543] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4607.441286] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4607.446330] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4607.451460] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4607.456589] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4607.461976] process_one_work+0x1ec/0x470 [ 4607.465972] worker_thread+0x74/0x410 [ 4607.469621] kthread+0xec/0x100 [ 4607.472749] ret_from_fork+0x10/0x20 [ 4623.194236] watchdog: BUG: soft lockup - CPU#68 stuck for 3911s! [swapper/68:0] [ 4623.201532] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4623.201549] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4623.289337] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4623.297153] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4623.306444] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4623.313391] pc : __sbq_wake_up+0x58/0x1f0 [ 4623.317388] lr : sbitmap_queue_clear+0x74/0xc0 [ 4623.321818] sp : ffff8000088438b0 [ 4623.325120] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4623.332242] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4623.339363] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4623.346485] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4623.353607] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4623.360729] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4623.367851] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4623.374973] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4623.382095] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4623.389217] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4623.396339] Call trace: [ 4623.398772] __sbq_wake_up+0x58/0x1f0 [ 4623.402421] sbitmap_queue_clear+0x74/0xc0 [ 4623.406505] blk_mq_put_tag+0x44/0x70 [ 4623.410154] __blk_mq_free_request+0x88/0xf0 [ 4623.414411] blk_mq_free_request+0xc0/0x160 [ 4623.418581] __blk_mq_end_request+0x78/0x150 [ 4623.422837] scsi_end_request+0x114/0x260 [ 4623.426834] scsi_io_completion+0x64/0x174 [ 4623.430917] scsi_finish_command+0xe0/0x120 [ 4623.435087] scsi_complete+0x8c/0x1c4 [ 4623.438736] blk_complete_reqs+0x5c/0x74 [ 4623.442646] blk_done_softirq+0x38/0x4c [ 4623.446469] __do_softirq+0x168/0x418 [ 4623.450118] __irq_exit_rcu+0x120/0x170 [ 4623.453940] irq_exit_rcu+0x18/0x24 [ 4623.457416] el1_interrupt+0x38/0x70 [ 4623.460978] el1h_64_irq_handler+0x18/0x24 [ 4623.465061] el1h_64_irq+0x68/0x6c [ 4623.468449] cpuidle_enter_state+0xc4/0x4e0 [ 4623.472620] cpuidle_enter+0x40/0x60 [ 4623.476182] cpuidle_idle_call+0x118/0x1b0 [ 4623.480266] do_idle+0xac/0x100 [ 4623.483394] cpu_startup_entry+0x30/0x34 [ 4623.487303] secondary_start_kernel+0xd8/0xf0 [ 4623.491647] __secondary_switched+0xb0/0xb4 [ 4631.084183] watchdog: BUG: soft lockup - CPU#8 stuck for 3919s! [kworker/8:12:19399] [ 4631.091912] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4631.091929] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4631.179718] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4631.187967] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4631.197259] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4631.203426] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4631.210374] pc : __sbq_wake_up+0x84/0x1f0 [ 4631.214371] lr : sbitmap_queue_clear+0x74/0xc0 [ 4631.218802] sp : ffff80000d79b280 [ 4631.222103] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4631.229225] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4631.236348] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4631.243469] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4631.250591] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4631.257713] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4631.264836] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4631.271957] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4631.279079] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4631.286201] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4631.293323] Call trace: [ 4631.295756] __sbq_wake_up+0x84/0x1f0 [ 4631.299405] sbitmap_queue_clear+0x74/0xc0 [ 4631.303489] blk_mq_put_tag+0x44/0x70 [ 4631.307138] __blk_mq_free_request+0x88/0xf0 [ 4631.311395] blk_mq_free_request+0xc0/0x160 [ 4631.315565] __blk_mq_end_request+0x78/0x150 [ 4631.319822] scsi_end_request+0x114/0x260 [ 4631.323819] scsi_io_completion+0x64/0x174 [ 4631.327902] scsi_finish_command+0xe0/0x120 [ 4631.332072] scsi_complete+0x8c/0x1c4 [ 4631.335721] blk_complete_reqs+0x5c/0x74 [ 4631.339631] blk_done_softirq+0x38/0x4c [ 4631.343453] __do_softirq+0x168/0x418 [ 4631.347102] __irq_exit_rcu+0x120/0x170 [ 4631.350925] irq_exit_rcu+0x18/0x24 [ 4631.354400] el1_interrupt+0x38/0x70 [ 4631.357962] el1h_64_irq_handler+0x18/0x24 [ 4631.362045] el1h_64_irq+0x68/0x6c [ 4631.365434] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4631.370212] __wake_up_common_lock+0x84/0xac [ 4631.374469] __wake_up+0x20/0x30 [ 4631.377684] __sbq_wake_up+0x13c/0x1f0 [ 4631.381420] sbitmap_queue_clear+0x74/0xc0 [ 4631.385504] blk_mq_put_tag+0x44/0x70 [ 4631.389154] __blk_mq_free_request+0x88/0xf0 [ 4631.393412] blk_mq_free_request+0xc0/0x160 [ 4631.397582] bfq_insert_request.isra.0+0x204/0x270 [ 4631.402360] bfq_insert_requests+0x70/0x90 [ 4631.406444] blk_mq_sched_insert_requests+0xac/0x250 [ 4631.411395] blk_mq_flush_plug_list+0x110/0x2cc [ 4631.415913] __blk_flush_plug+0x100/0x140 [ 4631.419909] blk_finish_plug+0x40/0x25c [ 4631.423732] __iomap_dio_rw+0x1b4/0x500 [ 4631.427555] iomap_dio_rw+0x18/0x50 [ 4631.431032] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4631.436383] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4631.441125] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4631.446169] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4631.451299] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4631.456429] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4631.461816] process_one_work+0x1ec/0x470 [ 4631.465812] worker_thread+0x74/0x410 [ 4631.469461] kthread+0xec/0x100 [ 4631.472590] ret_from_fork+0x10/0x20 [ 4631.994178] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4631.999911] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=208617 [ 4632.009117] (t=420712 jiffies g=36445 q=123936 ncpus=128) [ 4632.014589] Task dump for CPU 8: [ 4632.017804] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4632.027704] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4632.033872] Call trace: [ 4632.036304] __switch_to+0xd4/0x100 [ 4632.039781] 0xffff6c0200000080 [ 4632.042910] Task dump for CPU 68: [ 4632.046210] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4632.056110] Call trace: [ 4632.058543] dump_backtrace+0xe8/0x130 [ 4632.062279] show_stack+0x20/0x60 [ 4632.065581] sched_show_task+0x160/0x190 [ 4632.069491] dump_cpu_task+0x4c/0x78 [ 4632.073053] rcu_dump_cpu_stacks+0xc4/0x150 [ 4632.077224] print_cpu_stall+0x180/0x210 [ 4632.081133] check_cpu_stall+0x20c/0x2b4 [ 4632.085043] rcu_pending+0x4c/0x190 [ 4632.088518] rcu_sched_clock_irq+0x80/0x204 [ 4632.092688] update_process_times+0x7c/0xc0 [ 4632.096858] tick_sched_handle+0x38/0x74 [ 4632.100768] tick_sched_timer+0x60/0xc0 [ 4632.104591] __hrtimer_run_queues+0x170/0x300 [ 4632.108935] hrtimer_interrupt+0xfc/0x260 [ 4632.112931] arch_timer_handler_phys+0x34/0x50 [ 4632.117362] handle_percpu_devid_irq+0xac/0x240 [ 4632.121879] generic_handle_domain_irq+0x34/0x50 [ 4632.126484] gic_handle_irq+0x5c/0xe0 [ 4632.130133] call_on_irq_stack+0x2c/0x38 [ 4632.134043] do_interrupt_handler+0xdc/0xe0 [ 4632.138213] el1_interrupt+0x34/0x70 [ 4632.141775] el1h_64_irq_handler+0x18/0x24 [ 4632.145859] el1h_64_irq+0x68/0x6c [ 4632.149248] __sbq_wake_up+0x58/0x1f0 [ 4632.152898] sbitmap_queue_clear+0x74/0xc0 [ 4632.156981] blk_mq_put_tag+0x44/0x70 [ 4632.160631] __blk_mq_free_request+0x88/0xf0 [ 4632.164887] blk_mq_free_request+0xc0/0x160 [ 4632.169057] __blk_mq_end_request+0x78/0x150 [ 4632.173314] scsi_end_request+0x114/0x260 [ 4632.177310] scsi_io_completion+0x64/0x174 [ 4632.181393] scsi_finish_command+0xe0/0x120 [ 4632.185564] scsi_complete+0x8c/0x1c4 [ 4632.189213] blk_complete_reqs+0x5c/0x74 [ 4632.193122] blk_done_softirq+0x38/0x4c [ 4632.196945] __do_softirq+0x168/0x418 [ 4632.200594] __irq_exit_rcu+0x120/0x170 [ 4632.204417] irq_exit_rcu+0x18/0x24 [ 4632.207892] el1_interrupt+0x38/0x70 [ 4632.211455] el1h_64_irq_handler+0x18/0x24 [ 4632.215538] el1h_64_irq+0x68/0x6c [ 4632.218927] cpuidle_enter_state+0xc4/0x4e0 [ 4632.223097] cpuidle_enter+0x40/0x60 [ 4632.226659] cpuidle_idle_call+0x118/0x1b0 [ 4632.230743] do_idle+0xac/0x100 [ 4632.233872] cpu_startup_entry+0x30/0x34 [ 4632.237781] secondary_start_kernel+0xd8/0xf0 [ 4632.242125] __secondary_switched+0xb0/0xb4 [ 4655.084021] watchdog: BUG: soft lockup - CPU#8 stuck for 3941s! [kworker/8:12:19399] [ 4655.091751] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4655.091769] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4655.179556] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4655.187806] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4655.197097] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4655.203265] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4655.210213] pc : __sbq_wake_up+0x58/0x1f0 [ 4655.214209] lr : sbitmap_queue_clear+0x74/0xc0 [ 4655.218640] sp : ffff80000d79b280 [ 4655.221941] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4655.229063] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4655.236186] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4655.243308] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4655.250431] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4655.257552] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4655.264674] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4655.271796] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4655.278918] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4655.286040] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4655.293162] Call trace: [ 4655.295595] __sbq_wake_up+0x58/0x1f0 [ 4655.299244] sbitmap_queue_clear+0x74/0xc0 [ 4655.303328] blk_mq_put_tag+0x44/0x70 [ 4655.306978] __blk_mq_free_request+0x88/0xf0 [ 4655.311234] blk_mq_free_request+0xc0/0x160 [ 4655.315404] __blk_mq_end_request+0x78/0x150 [ 4655.319661] scsi_end_request+0x114/0x260 [ 4655.323657] scsi_io_completion+0x64/0x174 [ 4655.327740] scsi_finish_command+0xe0/0x120 [ 4655.331911] scsi_complete+0x8c/0x1c4 [ 4655.335560] blk_complete_reqs+0x5c/0x74 [ 4655.339469] blk_done_softirq+0x38/0x4c [ 4655.343292] __do_softirq+0x168/0x418 [ 4655.346941] __irq_exit_rcu+0x120/0x170 [ 4655.350764] irq_exit_rcu+0x18/0x24 [ 4655.354241] el1_interrupt+0x38/0x70 [ 4655.357803] el1h_64_irq_handler+0x18/0x24 [ 4655.361886] el1h_64_irq+0x68/0x6c [ 4655.365275] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4655.370054] __wake_up_common_lock+0x84/0xac [ 4655.374311] __wake_up+0x20/0x30 [ 4655.377526] __sbq_wake_up+0x13c/0x1f0 [ 4655.381262] sbitmap_queue_clear+0x74/0xc0 [ 4655.385346] blk_mq_put_tag+0x44/0x70 [ 4655.388996] __blk_mq_free_request+0x88/0xf0 [ 4655.393252] blk_mq_free_request+0xc0/0x160 [ 4655.397422] bfq_insert_request.isra.0+0x204/0x270 [ 4655.402200] bfq_insert_requests+0x70/0x90 [ 4655.406284] blk_mq_sched_insert_requests+0xac/0x250 [ 4655.411236] blk_mq_flush_plug_list+0x110/0x2cc [ 4655.415753] __blk_flush_plug+0x100/0x140 [ 4655.419750] blk_finish_plug+0x40/0x25c [ 4655.423572] __iomap_dio_rw+0x1b4/0x500 [ 4655.427395] iomap_dio_rw+0x18/0x50 [ 4655.430870] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4655.436221] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4655.440964] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4655.446008] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4655.451138] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4655.456267] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4655.461654] process_one_work+0x1ec/0x470 [ 4655.465650] worker_thread+0x74/0x410 [ 4655.469299] kthread+0xec/0x100 [ 4655.472428] ret_from_fork+0x10/0x20 [ 4659.193993] watchdog: BUG: soft lockup - CPU#68 stuck for 3945s! [swapper/68:0] [ 4659.201289] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4659.201305] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4659.289093] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4659.296909] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4659.306200] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4659.313147] pc : __sbq_wake_up+0x84/0x1f0 [ 4659.317144] lr : sbitmap_queue_clear+0x74/0xc0 [ 4659.321574] sp : ffff8000088438b0 [ 4659.324876] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4659.331998] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4659.339120] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4659.346242] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4659.353364] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4659.360486] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4659.367608] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4659.374730] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4659.381852] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4659.388974] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4659.396096] Call trace: [ 4659.398528] __sbq_wake_up+0x84/0x1f0 [ 4659.402178] sbitmap_queue_clear+0x74/0xc0 [ 4659.406262] blk_mq_put_tag+0x44/0x70 [ 4659.409911] __blk_mq_free_request+0x88/0xf0 [ 4659.414168] blk_mq_free_request+0xc0/0x160 [ 4659.418338] __blk_mq_end_request+0x78/0x150 [ 4659.422595] scsi_end_request+0x114/0x260 [ 4659.426591] scsi_io_completion+0x64/0x174 [ 4659.430674] scsi_finish_command+0xe0/0x120 [ 4659.434844] scsi_complete+0x8c/0x1c4 [ 4659.438494] blk_complete_reqs+0x5c/0x74 [ 4659.442403] blk_done_softirq+0x38/0x4c [ 4659.446226] __do_softirq+0x168/0x418 [ 4659.449874] __irq_exit_rcu+0x120/0x170 [ 4659.453697] irq_exit_rcu+0x18/0x24 [ 4659.457172] el1_interrupt+0x38/0x70 [ 4659.460735] el1h_64_irq_handler+0x18/0x24 [ 4659.464818] el1h_64_irq+0x68/0x6c [ 4659.468207] cpuidle_enter_state+0xc4/0x4e0 [ 4659.472377] cpuidle_enter+0x40/0x60 [ 4659.475939] cpuidle_idle_call+0x118/0x1b0 [ 4659.480023] do_idle+0xac/0x100 [ 4659.483151] cpu_startup_entry+0x30/0x34 [ 4659.487060] secondary_start_kernel+0xd8/0xf0 [ 4659.491405] __secondary_switched+0xb0/0xb4 [ 4665.303962] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 421437 jiffies s: 2093 root: 0x11/. [ 4665.315701] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4665.324910] Task dump for CPU 8: [ 4665.328125] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4665.338026] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4665.344196] Call trace: [ 4665.346629] __switch_to+0xd4/0x100 [ 4665.350106] 0xffff6c0200000080 [ 4665.353235] Task dump for CPU 68: [ 4665.356538] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4665.366439] Call trace: [ 4665.368872] __switch_to+0xd4/0x100 [ 4665.372349] 0x0 [ 4679.083859] watchdog: BUG: soft lockup - CPU#8 stuck for 3963s! [kworker/8:12:19399] [ 4679.091590] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4679.091608] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4679.179396] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4679.187646] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4679.196937] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4679.203104] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4679.210052] pc : __sbq_wake_up+0x20/0x1f0 [ 4679.214049] lr : sbitmap_queue_clear+0x74/0xc0 [ 4679.218480] sp : ffff80000d79b280 [ 4679.221781] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4679.228903] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4679.236026] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4679.243148] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4679.250270] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4679.257392] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4679.264514] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4679.271636] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4679.278758] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4679.285880] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 4679.293002] Call trace: [ 4679.295435] __sbq_wake_up+0x20/0x1f0 [ 4679.299084] sbitmap_queue_clear+0x74/0xc0 [ 4679.303167] blk_mq_put_tag+0x44/0x70 [ 4679.306817] __blk_mq_free_request+0x88/0xf0 [ 4679.311073] blk_mq_free_request+0xc0/0x160 [ 4679.315243] __blk_mq_end_request+0x78/0x150 [ 4679.319500] scsi_end_request+0x114/0x260 [ 4679.323496] scsi_io_completion+0x64/0x174 [ 4679.327579] scsi_finish_command+0xe0/0x120 [ 4679.331750] scsi_complete+0x8c/0x1c4 [ 4679.335399] blk_complete_reqs+0x5c/0x74 [ 4679.339308] blk_done_softirq+0x38/0x4c [ 4679.343131] __do_softirq+0x168/0x418 [ 4679.346780] __irq_exit_rcu+0x120/0x170 [ 4679.350603] irq_exit_rcu+0x18/0x24 [ 4679.354078] el1_interrupt+0x38/0x70 [ 4679.357640] el1h_64_irq_handler+0x18/0x24 [ 4679.361723] el1h_64_irq+0x68/0x6c [ 4679.365112] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4679.369890] __wake_up_common_lock+0x84/0xac [ 4679.374147] __wake_up+0x20/0x30 [ 4679.377363] __sbq_wake_up+0x13c/0x1f0 [ 4679.381099] sbitmap_queue_clear+0x74/0xc0 [ 4679.385183] blk_mq_put_tag+0x44/0x70 [ 4679.388832] __blk_mq_free_request+0x88/0xf0 [ 4679.393089] blk_mq_free_request+0xc0/0x160 [ 4679.397259] bfq_insert_request.isra.0+0x204/0x270 [ 4679.402039] bfq_insert_requests+0x70/0x90 [ 4679.406122] blk_mq_sched_insert_requests+0xac/0x250 [ 4679.411074] blk_mq_flush_plug_list+0x110/0x2cc [ 4679.415592] __blk_flush_plug+0x100/0x140 [ 4679.419588] blk_finish_plug+0x40/0x25c [ 4679.423412] __iomap_dio_rw+0x1b4/0x500 [ 4679.427234] iomap_dio_rw+0x18/0x50 [ 4679.430710] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4679.436061] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4679.440803] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4679.445847] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4679.450977] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4679.456107] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4679.461493] process_one_work+0x1ec/0x470 [ 4679.465489] worker_thread+0x74/0x410 [ 4679.469138] kthread+0xec/0x100 [ 4679.472267] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 05:15:00 2022] [ 4683.193832] watchdog: BUG: soft lockup - CPU#68 stuck for 3967s! [swapper/68:0] [ 4683.201127] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4683.201143] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4683.288932] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4683.296748] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4683.306039] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4683.312987] pc : __sbq_wake_up+0x58/0x1f0 [ 4683.316983] lr : sbitmap_queue_clear+0x74/0xc0 [ 4683.321414] sp : ffff8000088438b0 [ 4683.324714] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4683.331836] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4683.338958] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4683.346080] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4683.353202] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4683.360324] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4683.367446] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4683.374568] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4683.381690] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4683.388811] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4683.395933] Call trace: [ 4683.398366] __sbq_wake_up+0x58/0x1f0 [ 4683.402016] sbitmap_queue_clear+0x74/0xc0 [ 4683.406099] blk_mq_put_tag+0x44/0x70 [ 4683.409749] __blk_mq_free_request+0x88/0xf0 [ 4683.414006] blk_mq_free_request+0xc0/0x160 [ 4683.418175] __blk_mq_end_request+0x78/0x150 [ 4683.422432] scsi_end_request+0x114/0x260 [ 4683.426429] scsi_io_completion+0x64/0x174 [ 4683.430512] scsi_finish_command+0xe0/0x120 [ 4683.434683] scsi_complete+0x8c/0x1c4 [ 4683.438332] blk_complete_reqs+0x5c/0x74 [ 4683.442241] blk_done_softirq+0x38/0x4c [ 4683.446064] __do_softirq+0x168/0x418 [ 4683.449713] __irq_exit_rcu+0x120/0x170 [ 4683.453537] irq_exit_rcu+0x18/0x24 [ 4683.457013] el1_interrupt+0x38/0x70 [ 4683.460575] el1h_64_irq_handler+0x18/0x24 [ 4683.464658] el1h_64_irq+0x68/0x6c [ 4683.468046] cpuidle_enter_state+0xc4/0x4e0 [ 4683.472216] cpuidle_enter+0x40/0x60 [ 4683.475779] cpuidle_idle_call+0x118/0x1b0 [ 4683.479863] do_idle+0xac/0x100 [ 4683.482991] cpu_startup_entry+0x30/0x34 [ 4683.486900] secondary_start_kernel+0xd8/0xf0 [ 4683.491245] __secondary_switched+0xb0/0xb4 [ 4703.083697] watchdog: BUG: soft lockup - CPU#8 stuck for 3986s! [kworker/8:12:19399] [ 4703.091427] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4703.091444] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4703.179232] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4703.187482] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4703.196773] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4703.202940] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4703.209888] pc : __sbq_wake_up+0x4/0x1f0 [ 4703.213798] lr : sbitmap_queue_clear+0x74/0xc0 [ 4703.218229] sp : ffff80000d79b2d0 [ 4703.221530] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4703.228652] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4703.235774] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4703.242897] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4703.250019] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4703.257141] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4703.264264] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4703.271386] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4703.278508] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4703.285630] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 4703.292752] Call trace: [ 4703.295185] __sbq_wake_up+0x4/0x1f0 [ 4703.298747] blk_mq_put_tag+0x44/0x70 [ 4703.302397] __blk_mq_free_request+0x88/0xf0 [ 4703.306654] blk_mq_free_request+0xc0/0x160 [ 4703.310824] __blk_mq_end_request+0x78/0x150 [ 4703.315081] scsi_end_request+0x114/0x260 [ 4703.319078] scsi_io_completion+0x64/0x174 [ 4703.323161] scsi_finish_command+0xe0/0x120 [ 4703.327331] scsi_complete+0x8c/0x1c4 [ 4703.330980] blk_complete_reqs+0x5c/0x74 [ 4703.334890] blk_done_softirq+0x38/0x4c [ 4703.338712] __do_softirq+0x168/0x418 [ 4703.342362] __irq_exit_rcu+0x120/0x170 [ 4703.346185] irq_exit_rcu+0x18/0x24 [ 4703.349661] el1_interrupt+0x38/0x70 [ 4703.353223] el1h_64_irq_handler+0x18/0x24 [ 4703.357307] el1h_64_irq+0x68/0x6c [ 4703.360695] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4703.365473] __wake_up_common_lock+0x84/0xac [ 4703.369731] __wake_up+0x20/0x30 [ 4703.372946] __sbq_wake_up+0x13c/0x1f0 [ 4703.376682] sbitmap_queue_clear+0x74/0xc0 [ 4703.380766] blk_mq_put_tag+0x44/0x70 [ 4703.384415] __blk_mq_free_request+0x88/0xf0 [ 4703.388672] blk_mq_free_request+0xc0/0x160 [ 4703.392842] bfq_insert_request.isra.0+0x204/0x270 [ 4703.397620] bfq_insert_requests+0x70/0x90 [ 4703.401703] blk_mq_sched_insert_requests+0xac/0x250 [ 4703.406655] blk_mq_flush_plug_list+0x110/0x2cc [ 4703.411175] __blk_flush_plug+0x100/0x140 [ 4703.415171] blk_finish_plug+0x40/0x25c [ 4703.418994] __iomap_dio_rw+0x1b4/0x500 [ 4703.422816] iomap_dio_rw+0x18/0x50 [ 4703.426292] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4703.431643] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4703.436385] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4703.441429] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4703.446559] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4703.451689] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4703.457075] process_one_work+0x1ec/0x470 [ 4703.461071] worker_thread+0x74/0x410 [ 4703.464719] kthread+0xec/0x100 [ 4703.467848] ret_from_fork+0x10/0x20 [ 4707.193670] watchdog: BUG: soft lockup - CPU#68 stuck for 3989s! [swapper/68:0] [ 4707.200965] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4707.200982] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4707.288769] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4707.296585] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4707.305876] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4707.312824] pc : __sbq_wake_up+0x58/0x1f0 [ 4707.316820] lr : sbitmap_queue_clear+0x74/0xc0 [ 4707.321251] sp : ffff8000088438b0 [ 4707.324553] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4707.331675] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4707.338797] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4707.345919] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4707.353040] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4707.360163] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4707.367285] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4707.374408] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4707.381529] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4707.388651] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4707.395773] Call trace: [ 4707.398206] __sbq_wake_up+0x58/0x1f0 [ 4707.401855] sbitmap_queue_clear+0x74/0xc0 [ 4707.405939] blk_mq_put_tag+0x44/0x70 [ 4707.409588] __blk_mq_free_request+0x88/0xf0 [ 4707.413845] blk_mq_free_request+0xc0/0x160 [ 4707.418015] __blk_mq_end_request+0x78/0x150 [ 4707.422271] scsi_end_request+0x114/0x260 [ 4707.426268] scsi_io_completion+0x64/0x174 [ 4707.430351] scsi_finish_command+0xe0/0x120 [ 4707.434521] scsi_complete+0x8c/0x1c4 [ 4707.438170] blk_complete_reqs+0x5c/0x74 [ 4707.442080] blk_done_softirq+0x38/0x4c [ 4707.445903] __do_softirq+0x168/0x418 [ 4707.449551] __irq_exit_rcu+0x120/0x170 [ 4707.453374] irq_exit_rcu+0x18/0x24 [ 4707.456849] el1_interrupt+0x38/0x70 [ 4707.460411] el1h_64_irq_handler+0x18/0x24 [ 4707.464495] el1h_64_irq+0x68/0x6c [ 4707.467883] cpuidle_enter_state+0xc4/0x4e0 [ 4707.472053] cpuidle_enter+0x40/0x60 [ 4707.475615] cpuidle_idle_call+0x118/0x1b0 [ 4707.479699] do_idle+0xac/0x100 [ 4707.482828] cpu_startup_entry+0x30/0x34 [ 4707.486737] secondary_start_kernel+0xd8/0xf0 [ 4707.491081] __secondary_switched+0xb0/0xb4 [ 4727.083535] watchdog: BUG: soft lockup - CPU#8 stuck for 4008s! [kworker/8:12:19399] [ 4727.091265] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4727.091282] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4727.179070] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4727.187320] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4727.196610] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4727.202778] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4727.209726] pc : __sbq_wake_up+0x40/0x1f0 [ 4727.213722] lr : sbitmap_queue_clear+0x74/0xc0 [ 4727.218153] sp : ffff80000d79b280 [ 4727.221455] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4727.228577] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4727.235699] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4727.242821] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4727.249943] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4727.257065] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4727.264188] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4727.271310] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4727.278431] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4727.285553] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 4727.292675] Call trace: [ 4727.295108] __sbq_wake_up+0x40/0x1f0 [ 4727.298757] sbitmap_queue_clear+0x74/0xc0 [ 4727.302840] blk_mq_put_tag+0x44/0x70 [ 4727.306490] __blk_mq_free_request+0x88/0xf0 [ 4727.310747] blk_mq_free_request+0xc0/0x160 [ 4727.314917] __blk_mq_end_request+0x78/0x150 [ 4727.319173] scsi_end_request+0x114/0x260 [ 4727.323170] scsi_io_completion+0x64/0x174 [ 4727.327253] scsi_finish_command+0xe0/0x120 [ 4727.331423] scsi_complete+0x8c/0x1c4 [ 4727.335072] blk_complete_reqs+0x5c/0x74 [ 4727.338982] blk_done_softirq+0x38/0x4c [ 4727.342805] __do_softirq+0x168/0x418 [ 4727.346454] __irq_exit_rcu+0x120/0x170 [ 4727.350276] irq_exit_rcu+0x18/0x24 [ 4727.353752] el1_interrupt+0x38/0x70 [ 4727.357314] el1h_64_irq_handler+0x18/0x24 [ 4727.361397] el1h_64_irq+0x68/0x6c [ 4727.364786] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4727.369563] __wake_up_common_lock+0x84/0xac [ 4727.373821] __wake_up+0x20/0x30 [ 4727.377036] __sbq_wake_up+0x13c/0x1f0 [ 4727.380772] sbitmap_queue_clear+0x74/0xc0 [ 4727.384856] blk_mq_put_tag+0x44/0x70 [ 4727.388505] __blk_mq_free_request+0x88/0xf0 [ 4727.392762] blk_mq_free_request+0xc0/0x160 [ 4727.396934] bfq_insert_request.isra.0+0x204/0x270 [ 4727.401712] bfq_insert_requests+0x70/0x90 [ 4727.405795] blk_mq_sched_insert_requests+0xac/0x250 [ 4727.410747] blk_mq_flush_plug_list+0x110/0x2cc [ 4727.415265] __blk_flush_plug+0x100/0x140 [ 4727.419261] blk_finish_plug+0x40/0x25c [ 4727.423084] __iomap_dio_rw+0x1b4/0x500 [ 4727.426906] iomap_dio_rw+0x18/0x50 [ 4727.430382] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4727.435732] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4727.440475] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4727.445518] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4727.450649] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4727.455778] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4727.461164] process_one_work+0x1ec/0x470 [ 4727.465161] worker_thread+0x74/0x410 [ 4727.468810] kthread+0xec/0x100 [ 4727.471939] ret_from_fork+0x10/0x20 [ 4731.193508] watchdog: BUG: soft lockup - CPU#68 stuck for 4012s! [swapper/68:0] [ 4731.200803] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4731.200820] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4731.288608] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4731.296423] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4731.305714] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4731.312662] pc : __sbq_wake_up+0x58/0x1f0 [ 4731.316659] lr : sbitmap_queue_clear+0x74/0xc0 [ 4731.321089] sp : ffff8000088438b0 [ 4731.324390] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4731.331513] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4731.338635] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4731.345757] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4731.352879] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4731.360000] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4731.367122] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4731.374244] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4731.381366] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4731.388488] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4731.395610] Call trace: [ 4731.398042] __sbq_wake_up+0x58/0x1f0 [ 4731.401692] sbitmap_queue_clear+0x74/0xc0 [ 4731.405775] blk_mq_put_tag+0x44/0x70 [ 4731.409425] __blk_mq_free_request+0x88/0xf0 [ 4731.413681] blk_mq_free_request+0xc0/0x160 [ 4731.417851] __blk_mq_end_request+0x78/0x150 [ 4731.422108] scsi_end_request+0x114/0x260 [ 4731.426104] scsi_io_completion+0x64/0x174 [ 4731.430187] scsi_finish_command+0xe0/0x120 [ 4731.434358] scsi_complete+0x8c/0x1c4 [ 4731.438007] blk_complete_reqs+0x5c/0x74 [ 4731.441916] blk_done_softirq+0x38/0x4c [ 4731.445739] __do_softirq+0x168/0x418 [ 4731.449388] __irq_exit_rcu+0x120/0x170 [ 4731.453211] irq_exit_rcu+0x18/0x24 [ 4731.456686] el1_interrupt+0x38/0x70 [ 4731.460249] el1h_64_irq_handler+0x18/0x24 [ 4731.464332] el1h_64_irq+0x68/0x6c [ 4731.467720] cpuidle_enter_state+0xc4/0x4e0 [ 4731.471890] cpuidle_enter+0x40/0x60 [ 4731.475453] cpuidle_idle_call+0x118/0x1b0 [ 4731.479536] do_idle+0xac/0x100 [ 4731.482665] cpu_startup_entry+0x30/0x34 [ 4731.486574] secondary_start_kernel+0xd8/0xf0 [ 4731.490918] __secondary_switched+0xb0/0xb4 [ 4731.863513] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 428093 jiffies s: 2093 root: 0x11/. [ 4731.875252] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4731.884461] Task dump for CPU 8: [ 4731.887675] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4731.897577] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4731.903747] Call trace: [ 4731.906180] __switch_to+0xd4/0x100 [ 4731.909657] 0xffff6c0200000080 [ 4731.912785] Task dump for CPU 68: [ 4731.916088] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4731.925989] Call trace: [ 4731.928422] __switch_to+0xd4/0x100 [ 4731.931898] 0x0 [ 4751.083373] watchdog: BUG: soft lockup - CPU#8 stuck for 4030s! [kworker/8:12:19399] [ 4751.091103] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4751.091120] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4751.178907] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4751.187157] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4751.196448] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4751.202615] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4751.209563] pc : __sbq_wake_up+0x1c8/0x1f0 [ 4751.213646] lr : sbitmap_queue_clear+0x74/0xc0 [ 4751.218077] sp : ffff80000d79b280 [ 4751.221378] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4751.228501] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4751.235623] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4751.242746] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4751.249868] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4751.256990] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4751.264113] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4751.271235] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4751.278356] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4751.285478] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 4751.292600] Call trace: [ 4751.295033] __sbq_wake_up+0x1c8/0x1f0 [ 4751.298769] sbitmap_queue_clear+0x74/0xc0 [ 4751.302853] blk_mq_put_tag+0x44/0x70 [ 4751.306502] __blk_mq_free_request+0x88/0xf0 [ 4751.310759] blk_mq_free_request+0xc0/0x160 [ 4751.314931] __blk_mq_end_request+0x78/0x150 [ 4751.319187] scsi_end_request+0x114/0x260 [ 4751.323184] scsi_io_completion+0x64/0x174 [ 4751.327267] scsi_finish_command+0xe0/0x120 [ 4751.331437] scsi_complete+0x8c/0x1c4 [ 4751.335086] blk_complete_reqs+0x5c/0x74 [ 4751.338996] blk_done_softirq+0x38/0x4c [ 4751.342818] __do_softirq+0x168/0x418 [ 4751.346468] __irq_exit_rcu+0x120/0x170 [ 4751.350290] irq_exit_rcu+0x18/0x24 [ 4751.353766] el1_interrupt+0x38/0x70 [ 4751.357328] el1h_64_irq_handler+0x18/0x24 [ 4751.361411] el1h_64_irq+0x68/0x6c [ 4751.364800] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4751.369577] __wake_up_common_lock+0x84/0xac [ 4751.373835] __wake_up+0x20/0x30 [ 4751.377050] __sbq_wake_up+0x13c/0x1f0 [ 4751.380786] sbitmap_queue_clear+0x74/0xc0 [ 4751.384870] blk_mq_put_tag+0x44/0x70 [ 4751.388520] __blk_mq_free_request+0x88/0xf0 [ 4751.392777] blk_mq_free_request+0xc0/0x160 [ 4751.396947] bfq_insert_request.isra.0+0x204/0x270 [ 4751.401725] bfq_insert_requests+0x70/0x90 [ 4751.405809] blk_mq_sched_insert_requests+0xac/0x250 [ 4751.410760] blk_mq_flush_plug_list+0x110/0x2cc [ 4751.415278] __blk_flush_plug+0x100/0x140 [ 4751.419274] blk_finish_plug+0x40/0x25c [ 4751.423097] __iomap_dio_rw+0x1b4/0x500 [ 4751.426920] iomap_dio_rw+0x18/0x50 [ 4751.430395] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4751.435746] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4751.440488] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4751.445532] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4751.450662] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4751.455792] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4751.461178] process_one_work+0x1ec/0x470 [ 4751.465174] worker_thread+0x74/0x410 [ 4751.468823] kthread+0xec/0x100 [ 4751.471952] ret_from_fork+0x10/0x20 [ 4755.193346] watchdog: BUG: soft lockup - CPU#68 stuck for 4034s! [swapper/68:0] [ 4755.200641] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4755.200658] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4755.288446] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4755.296261] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4755.305552] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4755.312500] pc : __sbq_wake_up+0x58/0x1f0 [ 4755.316496] lr : sbitmap_queue_clear+0x74/0xc0 [ 4755.320927] sp : ffff8000088438b0 [ 4755.324228] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4755.331350] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4755.338473] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4755.345594] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4755.352717] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4755.359839] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4755.366962] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4755.374083] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4755.381205] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4755.388327] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4755.395449] Call trace: [ 4755.397883] __sbq_wake_up+0x58/0x1f0 [ 4755.401532] sbitmap_queue_clear+0x74/0xc0 [ 4755.405615] blk_mq_put_tag+0x44/0x70 [ 4755.409265] __blk_mq_free_request+0x88/0xf0 [ 4755.413522] blk_mq_free_request+0xc0/0x160 [ 4755.417692] __blk_mq_end_request+0x78/0x150 [ 4755.421948] scsi_end_request+0x114/0x260 [ 4755.425945] scsi_io_completion+0x64/0x174 [ 4755.430030] scsi_finish_command+0xe0/0x120 [ 4755.434200] scsi_complete+0x8c/0x1c4 [ 4755.437849] blk_complete_reqs+0x5c/0x74 [ 4755.441759] blk_done_softirq+0x38/0x4c [ 4755.445582] __do_softirq+0x168/0x418 [ 4755.449230] __irq_exit_rcu+0x120/0x170 [ 4755.453053] irq_exit_rcu+0x18/0x24 [ 4755.456528] el1_interrupt+0x38/0x70 [ 4755.460091] el1h_64_irq_handler+0x18/0x24 [ 4755.464174] el1h_64_irq+0x68/0x6c [ 4755.467562] cpuidle_enter_state+0xc4/0x4e0 [ 4755.471732] cpuidle_enter+0x40/0x60 [ 4755.475294] cpuidle_idle_call+0x118/0x1b0 [ 4755.479378] do_idle+0xac/0x100 [ 4755.482506] cpu_startup_entry+0x30/0x34 [ 4755.486415] secondary_start_kernel+0xd8/0xf0 [ 4755.490759] __secondary_switched+0xb0/0xb4 [ 4775.083212] watchdog: BUG: soft lockup - CPU#8 stuck for 4053s! [kworker/8:12:19399] [ 4775.090941] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4775.090958] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4775.178746] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4775.186995] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4775.196286] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4775.202454] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4775.209401] pc : __sbq_wake_up+0x7c/0x1f0 [ 4775.213398] lr : sbitmap_queue_clear+0x74/0xc0 [ 4775.217829] sp : ffff80000d79b2d0 [ 4775.221130] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4775.228253] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4775.235375] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4775.242497] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4775.249619] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4775.256741] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4775.263863] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4775.270985] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4775.278107] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4775.285229] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 4775.292351] Call trace: [ 4775.294785] __sbq_wake_up+0x7c/0x1f0 [ 4775.298434] blk_mq_put_tag+0x44/0x70 [ 4775.302084] __blk_mq_free_request+0x88/0xf0 [ 4775.306341] blk_mq_free_request+0xc0/0x160 [ 4775.310511] __blk_mq_end_request+0x78/0x150 [ 4775.314768] scsi_end_request+0x114/0x260 [ 4775.318765] scsi_io_completion+0x64/0x174 [ 4775.322848] scsi_finish_command+0xe0/0x120 [ 4775.327018] scsi_complete+0x8c/0x1c4 [ 4775.330667] blk_complete_reqs+0x5c/0x74 [ 4775.334577] blk_done_softirq+0x38/0x4c [ 4775.338399] __do_softirq+0x168/0x418 [ 4775.342048] __irq_exit_rcu+0x120/0x170 [ 4775.345871] irq_exit_rcu+0x18/0x24 [ 4775.349347] el1_interrupt+0x38/0x70 [ 4775.352909] el1h_64_irq_handler+0x18/0x24 [ 4775.356992] el1h_64_irq+0x68/0x6c [ 4775.360381] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4775.365159] __wake_up_common_lock+0x84/0xac [ 4775.369417] __wake_up+0x20/0x30 [ 4775.372633] __sbq_wake_up+0x13c/0x1f0 [ 4775.376369] sbitmap_queue_clear+0x74/0xc0 [ 4775.380452] blk_mq_put_tag+0x44/0x70 [ 4775.384102] __blk_mq_free_request+0x88/0xf0 [ 4775.388359] blk_mq_free_request+0xc0/0x160 [ 4775.392528] bfq_insert_request.isra.0+0x204/0x270 [ 4775.397306] bfq_insert_requests+0x70/0x90 [ 4775.401390] blk_mq_sched_insert_requests+0xac/0x250 [ 4775.406341] blk_mq_flush_plug_list+0x110/0x2cc [ 4775.410859] __blk_flush_plug+0x100/0x140 [ 4775.414856] blk_finish_plug+0x40/0x25c [ 4775.418678] __iomap_dio_rw+0x1b4/0x500 [ 4775.422501] iomap_dio_rw+0x18/0x50 [ 4775.425977] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4775.431328] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4775.436070] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4775.441114] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4775.446243] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4775.451373] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4775.456759] process_one_work+0x1ec/0x470 [ 4775.460755] worker_thread+0x74/0x410 [ 4775.464404] kthread+0xec/0x100 [ 4775.467532] ret_from_fork+0x10/0x20 [ 4779.193184] watchdog: BUG: soft lockup - CPU#68 stuck for 4056s! [swapper/68:0] [ 4779.200479] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4779.200496] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4779.288284] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4779.296100] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4779.305390] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4779.312338] pc : __sbq_wake_up+0x58/0x1f0 [ 4779.316335] lr : sbitmap_queue_clear+0x74/0xc0 [ 4779.320766] sp : ffff8000088438b0 [ 4779.324067] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4779.331189] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4779.338311] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4779.345433] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4779.352554] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4779.359676] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4779.366798] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4779.373920] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4779.381042] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4779.388164] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4779.395285] Call trace: [ 4779.397718] __sbq_wake_up+0x58/0x1f0 [ 4779.401367] sbitmap_queue_clear+0x74/0xc0 [ 4779.405450] blk_mq_put_tag+0x44/0x70 [ 4779.409100] __blk_mq_free_request+0x88/0xf0 [ 4779.413357] blk_mq_free_request+0xc0/0x160 [ 4779.417527] __blk_mq_end_request+0x78/0x150 [ 4779.421783] scsi_end_request+0x114/0x260 [ 4779.425780] scsi_io_completion+0x64/0x174 [ 4779.429863] scsi_finish_command+0xe0/0x120 [ 4779.434034] scsi_complete+0x8c/0x1c4 [ 4779.437683] blk_complete_reqs+0x5c/0x74 [ 4779.441592] blk_done_softirq+0x38/0x4c [ 4779.445415] __do_softirq+0x168/0x418 [ 4779.449064] __irq_exit_rcu+0x120/0x170 [ 4779.452887] irq_exit_rcu+0x18/0x24 [ 4779.456362] el1_interrupt+0x38/0x70 [ 4779.459924] el1h_64_irq_handler+0x18/0x24 [ 4779.464007] el1h_64_irq+0x68/0x6c [ 4779.467396] cpuidle_enter_state+0xc4/0x4e0 [ 4779.471566] cpuidle_enter+0x40/0x60 [ 4779.475128] cpuidle_idle_call+0x118/0x1b0 [ 4779.479212] do_idle+0xac/0x100 [ 4779.482341] cpu_startup_entry+0x30/0x34 [ 4779.486250] secondary_start_kernel+0xd8/0xf0 [ 4779.490595] __secondary_switched+0xb0/0xb4 [ 4798.423064] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 434749 jiffies s: 2093 root: 0x11/. [ 4798.434802] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4798.444012] Task dump for CPU 8: [ 4798.447226] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4798.457128] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4798.463298] Call trace: [ 4798.465731] __switch_to+0xd4/0x100 [ 4798.469207] 0xffff6c0200000080 [ 4798.472336] Task dump for CPU 68: [ 4798.475639] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4798.485540] Call trace: [ 4798.487973] __switch_to+0xd4/0x100 [ 4798.491450] 0x0 [ 4799.083050] watchdog: BUG: soft lockup - CPU#8 stuck for 4075s! [kworker/8:12:19399] [ 4799.090779] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4799.090796] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4799.178584] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4799.186833] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4799.196124] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4799.202292] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4799.209240] pc : __sbq_wake_up+0x84/0x1f0 [ 4799.213238] lr : sbitmap_queue_clear+0x74/0xc0 [ 4799.217669] sp : ffff80000d79b280 [ 4799.220971] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4799.228093] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4799.235215] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4799.242337] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4799.249460] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4799.256582] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4799.263704] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4799.270826] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4799.277948] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4799.285071] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4799.292193] Call trace: [ 4799.294626] __sbq_wake_up+0x84/0x1f0 [ 4799.298276] sbitmap_queue_clear+0x74/0xc0 [ 4799.302359] blk_mq_put_tag+0x44/0x70 [ 4799.306009] __blk_mq_free_request+0x88/0xf0 [ 4799.310266] blk_mq_free_request+0xc0/0x160 [ 4799.314436] __blk_mq_end_request+0x78/0x150 [ 4799.318693] scsi_end_request+0x114/0x260 [ 4799.322689] scsi_io_completion+0x64/0x174 [ 4799.326772] scsi_finish_command+0xe0/0x120 [ 4799.330943] scsi_complete+0x8c/0x1c4 [ 4799.334592] blk_complete_reqs+0x5c/0x74 [ 4799.338501] blk_done_softirq+0x38/0x4c [ 4799.342324] __do_softirq+0x168/0x418 [ 4799.345973] __irq_exit_rcu+0x120/0x170 [ 4799.349796] irq_exit_rcu+0x18/0x24 [ 4799.353271] el1_interrupt+0x38/0x70 [ 4799.356833] el1h_64_irq_handler+0x18/0x24 [ 4799.360917] el1h_64_irq+0x68/0x6c [ 4799.364305] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4799.369083] __wake_up_common_lock+0x84/0xac [ 4799.373340] __wake_up+0x20/0x30 [ 4799.376556] __sbq_wake_up+0x13c/0x1f0 [ 4799.380291] sbitmap_queue_clear+0x74/0xc0 [ 4799.384374] blk_mq_put_tag+0x44/0x70 [ 4799.388024] __blk_mq_free_request+0x88/0xf0 [ 4799.392281] blk_mq_free_request+0xc0/0x160 [ 4799.396451] bfq_insert_request.isra.0+0x204/0x270 [ 4799.401229] bfq_insert_requests+0x70/0x90 [ 4799.405312] blk_mq_sched_insert_requests+0xac/0x250 [ 4799.410265] blk_mq_flush_plug_list+0x110/0x2cc [ 4799.414782] __blk_flush_plug+0x100/0x140 [ 4799.418779] blk_finish_plug+0x40/0x25c [ 4799.422602] __iomap_dio_rw+0x1b4/0x500 [ 4799.426425] iomap_dio_rw+0x18/0x50 [ 4799.429900] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4799.435251] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4799.439993] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4799.445037] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4799.450167] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4799.455297] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4799.460683] process_one_work+0x1ec/0x470 [ 4799.464679] worker_thread+0x74/0x410 [ 4799.468328] kthread+0xec/0x100 [ 4799.471457] ret_from_fork+0x10/0x20 [ 4803.193022] watchdog: BUG: soft lockup - CPU#68 stuck for 4079s! [swapper/68:0] [ 4803.200317] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4803.200334] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4803.288121] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4803.295937] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4803.305228] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4803.312175] pc : __sbq_wake_up+0x20/0x1f0 [ 4803.316171] lr : sbitmap_queue_clear+0x74/0xc0 [ 4803.320602] sp : ffff8000088438b0 [ 4803.323903] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4803.331025] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4803.338147] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4803.345269] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4803.352391] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4803.359513] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4803.366635] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4803.373756] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4803.380878] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4803.388000] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4803.395122] Call trace: [ 4803.397555] __sbq_wake_up+0x20/0x1f0 [ 4803.401204] sbitmap_queue_clear+0x74/0xc0 [ 4803.405288] blk_mq_put_tag+0x44/0x70 [ 4803.408938] __blk_mq_free_request+0x88/0xf0 [ 4803.413194] blk_mq_free_request+0xc0/0x160 [ 4803.417364] __blk_mq_end_request+0x78/0x150 [ 4803.421621] scsi_end_request+0x114/0x260 [ 4803.425617] scsi_io_completion+0x64/0x174 [ 4803.429700] scsi_finish_command+0xe0/0x120 [ 4803.433871] scsi_complete+0x8c/0x1c4 [ 4803.437520] blk_complete_reqs+0x5c/0x74 [ 4803.441429] blk_done_softirq+0x38/0x4c [ 4803.445252] __do_softirq+0x168/0x418 [ 4803.448900] __irq_exit_rcu+0x120/0x170 [ 4803.452723] irq_exit_rcu+0x18/0x24 [ 4803.456199] el1_interrupt+0x38/0x70 [ 4803.459761] el1h_64_irq_handler+0x18/0x24 [ 4803.463844] el1h_64_irq+0x68/0x6c [ 4803.467233] cpuidle_enter_state+0xc4/0x4e0 [ 4803.471403] cpuidle_enter+0x40/0x60 [ 4803.474965] cpuidle_idle_call+0x118/0x1b0 [ 4803.479049] do_idle+0xac/0x100 [ 4803.482177] cpu_startup_entry+0x30/0x34 [ 4803.486086] secondary_start_kernel+0xd8/0xf0 [ 4803.490430] __secondary_switched+0xb0/0xb4 [ 4812.262961] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4812.268694] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=217577 [ 4812.277901] (t=438739 jiffies g=36445 q=129580 ncpus=128) [ 4812.283373] Task dump for CPU 8: [ 4812.286587] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4812.296488] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4812.302655] Call trace: [ 4812.305088] __switch_to+0xd4/0x100 [ 4812.308564] 0xffff6c0200000080 [ 4812.311693] Task dump for CPU 68: [ 4812.314994] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4812.324894] Call trace: [ 4812.327327] dump_backtrace+0xe8/0x130 [ 4812.331063] show_stack+0x20/0x60 [ 4812.334364] sched_show_task+0x160/0x190 [ 4812.338274] dump_cpu_task+0x4c/0x78 [ 4812.341837] rcu_dump_cpu_stacks+0xc4/0x150 [ 4812.346008] print_cpu_stall+0x180/0x210 [ 4812.349917] check_cpu_stall+0x20c/0x2b4 [ 4812.353827] rcu_pending+0x4c/0x190 [ 4812.357302] rcu_sched_clock_irq+0x80/0x204 [ 4812.361471] update_process_times+0x7c/0xc0 [ 4812.365641] tick_sched_handle+0x38/0x74 [ 4812.369551] tick_sched_timer+0x60/0xc0 [ 4812.373375] __hrtimer_run_queues+0x170/0x300 [ 4812.377718] hrtimer_interrupt+0xfc/0x260 [ 4812.381715] arch_timer_handler_phys+0x34/0x50 [ 4812.386146] handle_percpu_devid_irq+0xac/0x240 [ 4812.390664] generic_handle_domain_irq+0x34/0x50 [ 4812.395268] gic_handle_irq+0x5c/0xe0 [ 4812.398917] call_on_irq_stack+0x2c/0x38 [ 4812.402827] do_interrupt_handler+0xdc/0xe0 [ 4812.406998] el1_interrupt+0x34/0x70 [ 4812.410560] el1h_64_irq_handler+0x18/0x24 [ 4812.414643] el1h_64_irq+0x68/0x6c [ 4812.418032] __sbq_wake_up+0x58/0x1f0 [ 4812.421681] sbitmap_queue_clear+0x74/0xc0 [ 4812.425764] blk_mq_put_tag+0x44/0x70 [ 4812.429413] __blk_mq_free_request+0x88/0xf0 [ 4812.433670] blk_mq_free_request+0xc0/0x160 [ 4812.437840] __blk_mq_end_request+0x78/0x150 [ 4812.442097] scsi_end_request+0x114/0x260 [ 4812.446094] scsi_io_completion+0x64/0x174 [ 4812.450177] scsi_finish_command+0xe0/0x120 [ 4812.454347] scsi_complete+0x8c/0x1c4 [ 4812.457996] blk_complete_reqs+0x5c/0x74 [ 4812.461906] blk_done_softirq+0x38/0x4c [ 4812.465728] __do_softirq+0x168/0x418 [ 4812.469377] __irq_exit_rcu+0x120/0x170 [ 4812.473200] irq_exit_rcu+0x18/0x24 [ 4812.476675] el1_interrupt+0x38/0x70 [ 4812.480237] el1h_64_irq_handler+0x18/0x24 [ 4812.484320] el1h_64_irq+0x68/0x6c [ 4812.487709] cpuidle_enter_state+0xc4/0x4e0 [ 4812.491880] cpuidle_enter+0x40/0x60 [ 4812.495442] cpuidle_idle_call+0x118/0x1b0 [ 4812.499526] do_idle+0xac/0x100 [ 4812.502654] cpu_startup_entry+0x30/0x34 [ 4812.506563] secondary_start_kernel+0xd8/0xf0 [ 4812.510908] __secondary_switched+0xb0/0xb4 [ 4823.082888] watchdog: BUG: soft lockup - CPU#8 stuck for 4097s! [kworker/8:12:19399] [ 4823.090617] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4823.090634] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4823.178422] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4823.186672] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4823.195963] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4823.202131] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4823.209078] pc : __sbq_wake_up+0x84/0x1f0 [ 4823.213075] lr : sbitmap_queue_clear+0x74/0xc0 [ 4823.217505] sp : ffff80000d79b280 [ 4823.220806] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4823.227929] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4823.235050] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4823.242173] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4823.249295] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4823.256417] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4823.263539] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4823.270661] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4823.277783] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4823.284905] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4823.292027] Call trace: [ 4823.294460] __sbq_wake_up+0x84/0x1f0 [ 4823.298110] sbitmap_queue_clear+0x74/0xc0 [ 4823.302194] blk_mq_put_tag+0x44/0x70 [ 4823.305843] __blk_mq_free_request+0x88/0xf0 [ 4823.310100] blk_mq_free_request+0xc0/0x160 [ 4823.314270] __blk_mq_end_request+0x78/0x150 [ 4823.318527] scsi_end_request+0x114/0x260 [ 4823.322523] scsi_io_completion+0x64/0x174 [ 4823.326606] scsi_finish_command+0xe0/0x120 [ 4823.330776] scsi_complete+0x8c/0x1c4 [ 4823.334425] blk_complete_reqs+0x5c/0x74 [ 4823.338335] blk_done_softirq+0x38/0x4c [ 4823.342158] __do_softirq+0x168/0x418 [ 4823.345807] __irq_exit_rcu+0x120/0x170 [ 4823.349629] irq_exit_rcu+0x18/0x24 [ 4823.353105] el1_interrupt+0x38/0x70 [ 4823.356668] el1h_64_irq_handler+0x18/0x24 [ 4823.360752] el1h_64_irq+0x68/0x6c [ 4823.364141] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4823.368919] __wake_up_common_lock+0x84/0xac [ 4823.373176] __wake_up+0x20/0x30 [ 4823.376392] __sbq_wake_up+0x13c/0x1f0 [ 4823.380127] sbitmap_queue_clear+0x74/0xc0 [ 4823.384211] blk_mq_put_tag+0x44/0x70 [ 4823.387861] __blk_mq_free_request+0x88/0xf0 [ 4823.392117] blk_mq_free_request+0xc0/0x160 [ 4823.396287] bfq_insert_request.isra.0+0x204/0x270 [ 4823.401065] bfq_insert_requests+0x70/0x90 [ 4823.405149] blk_mq_sched_insert_requests+0xac/0x250 [ 4823.410101] blk_mq_flush_plug_list+0x110/0x2cc [ 4823.414619] __blk_flush_plug+0x100/0x140 [ 4823.418615] blk_finish_plug+0x40/0x25c [ 4823.422438] __iomap_dio_rw+0x1b4/0x500 [ 4823.426261] iomap_dio_rw+0x18/0x50 [ 4823.429736] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4823.435086] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4823.439829] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4823.444873] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4823.450003] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4823.455133] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4823.460519] process_one_work+0x1ec/0x470 [ 4823.464515] worker_thread+0x74/0x410 [ 4823.468164] kthread+0xec/0x100 [ 4823.471294] ret_from_fork+0x10/0x20 [ 4839.192779] watchdog: BUG: soft lockup - CPU#68 stuck for 4112s! [swapper/68:0] [ 4839.200075] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4839.200092] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4839.287880] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4839.295695] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4839.304986] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4839.311934] pc : __sbq_wake_up+0x20/0x1f0 [ 4839.315931] lr : sbitmap_queue_clear+0x74/0xc0 [ 4839.320362] sp : ffff8000088438b0 [ 4839.323663] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4839.330786] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4839.337908] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4839.345030] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4839.352152] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4839.359275] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4839.366396] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4839.373518] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4839.380641] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4839.387762] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4839.394884] Call trace: [ 4839.397319] __sbq_wake_up+0x20/0x1f0 [ 4839.400968] sbitmap_queue_clear+0x74/0xc0 [ 4839.405051] blk_mq_put_tag+0x44/0x70 [ 4839.408701] __blk_mq_free_request+0x88/0xf0 [ 4839.412958] blk_mq_free_request+0xc0/0x160 [ 4839.417128] __blk_mq_end_request+0x78/0x150 [ 4839.421385] scsi_end_request+0x114/0x260 [ 4839.425381] scsi_io_completion+0x64/0x174 [ 4839.429464] scsi_finish_command+0xe0/0x120 [ 4839.433635] scsi_complete+0x8c/0x1c4 [ 4839.437284] blk_complete_reqs+0x5c/0x74 [ 4839.441193] blk_done_softirq+0x38/0x4c [ 4839.445016] __do_softirq+0x168/0x418 [ 4839.448664] __irq_exit_rcu+0x120/0x170 [ 4839.452487] irq_exit_rcu+0x18/0x24 [ 4839.455963] el1_interrupt+0x38/0x70 [ 4839.459525] el1h_64_irq_handler+0x18/0x24 [ 4839.463608] el1h_64_irq+0x68/0x6c [ 4839.466996] cpuidle_enter_state+0xc4/0x4e0 [ 4839.471166] cpuidle_enter+0x40/0x60 [ 4839.474728] cpuidle_idle_call+0x118/0x1b0 [ 4839.478812] do_idle+0xac/0x100 [ 4839.481941] cpu_startup_entry+0x30/0x34 [ 4839.485850] secondary_start_kernel+0xd8/0xf0 [ 4839.490194] __secondary_switched+0xb0/0xb4 [ 4847.082726] watchdog: BUG: soft lockup - CPU#8 stuck for 4120s! [kworker/8:12:19399] [ 4847.090455] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4847.090472] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4847.178260] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4847.186509] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4847.195800] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4847.201968] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4847.208915] pc : __sbq_wake_up+0x58/0x1f0 [ 4847.212912] lr : sbitmap_queue_clear+0x74/0xc0 [ 4847.217342] sp : ffff80000d79b280 [ 4847.220643] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4847.227766] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4847.234889] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4847.242011] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4847.249134] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4847.256256] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4847.263378] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4847.270500] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4847.277622] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4847.284744] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4847.291866] Call trace: [ 4847.294300] __sbq_wake_up+0x58/0x1f0 [ 4847.297950] sbitmap_queue_clear+0x74/0xc0 [ 4847.302033] blk_mq_put_tag+0x44/0x70 [ 4847.305683] __blk_mq_free_request+0x88/0xf0 [ 4847.309940] blk_mq_free_request+0xc0/0x160 [ 4847.314110] __blk_mq_end_request+0x78/0x150 [ 4847.318367] scsi_end_request+0x114/0x260 [ 4847.322363] scsi_io_completion+0x64/0x174 [ 4847.326446] scsi_finish_command+0xe0/0x120 [ 4847.330617] scsi_complete+0x8c/0x1c4 [ 4847.334266] blk_complete_reqs+0x5c/0x74 [ 4847.338175] blk_done_softirq+0x38/0x4c [ 4847.341998] __do_softirq+0x168/0x418 [ 4847.345646] __irq_exit_rcu+0x120/0x170 [ 4847.349470] irq_exit_rcu+0x18/0x24 [ 4847.352946] el1_interrupt+0x38/0x70 [ 4847.356509] el1h_64_irq_handler+0x18/0x24 [ 4847.360592] el1h_64_irq+0x68/0x6c [ 4847.363980] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4847.368757] __wake_up_common_lock+0x84/0xac [ 4847.373015] __wake_up+0x20/0x30 [ 4847.376230] __sbq_wake_up+0x13c/0x1f0 [ 4847.379966] sbitmap_queue_clear+0x74/0xc0 [ 4847.384050] blk_mq_put_tag+0x44/0x70 [ 4847.387699] __blk_mq_free_request+0x88/0xf0 [ 4847.391956] blk_mq_free_request+0xc0/0x160 [ 4847.396126] bfq_insert_request.isra.0+0x204/0x270 [ 4847.400904] bfq_insert_requests+0x70/0x90 [ 4847.404987] blk_mq_sched_insert_requests+0xac/0x250 [ 4847.409939] blk_mq_flush_plug_list+0x110/0x2cc [ 4847.414456] __blk_flush_plug+0x100/0x140 [ 4847.418453] blk_finish_plug+0x40/0x25c [ 4847.422275] __iomap_dio_rw+0x1b4/0x500 [ 4847.426098] iomap_dio_rw+0x18/0x50 [ 4847.429574] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4847.434924] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4847.439666] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4847.444710] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4847.449840] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4847.454970] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4847.460356] process_one_work+0x1ec/0x470 [ 4847.464353] worker_thread+0x74/0x410 [ 4847.468002] kthread+0xec/0x100 [ 4847.471131] ret_from_fork+0x10/0x20 [ 4863.192617] watchdog: BUG: soft lockup - CPU#68 stuck for 4135s! [swapper/68:0] [ 4863.199912] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4863.199929] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4863.287717] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4863.295533] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4863.304824] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4863.311771] pc : __sbq_wake_up+0x40/0x1f0 [ 4863.315768] lr : sbitmap_queue_clear+0x74/0xc0 [ 4863.320199] sp : ffff8000088438b0 [ 4863.323500] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4863.330622] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4863.337744] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4863.344866] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4863.351988] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4863.359110] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4863.366232] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4863.373354] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4863.380476] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4863.387598] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 4863.394720] Call trace: [ 4863.397154] __sbq_wake_up+0x40/0x1f0 [ 4863.400803] sbitmap_queue_clear+0x74/0xc0 [ 4863.404888] blk_mq_put_tag+0x44/0x70 [ 4863.408538] __blk_mq_free_request+0x88/0xf0 [ 4863.412794] blk_mq_free_request+0xc0/0x160 [ 4863.416964] __blk_mq_end_request+0x78/0x150 [ 4863.421221] scsi_end_request+0x114/0x260 [ 4863.425217] scsi_io_completion+0x64/0x174 [ 4863.429300] scsi_finish_command+0xe0/0x120 [ 4863.433471] scsi_complete+0x8c/0x1c4 [ 4863.437121] blk_complete_reqs+0x5c/0x74 [ 4863.441031] blk_done_softirq+0x38/0x4c [ 4863.444854] __do_softirq+0x168/0x418 [ 4863.448502] __irq_exit_rcu+0x120/0x170 [ 4863.452325] irq_exit_rcu+0x18/0x24 [ 4863.455800] el1_interrupt+0x38/0x70 [ 4863.459363] el1h_64_irq_handler+0x18/0x24 [ 4863.463447] el1h_64_irq+0x68/0x6c [ 4863.466835] cpuidle_enter_state+0xc4/0x4e0 [ 4863.471006] cpuidle_enter+0x40/0x60 [ 4863.474568] cpuidle_idle_call+0x118/0x1b0 [ 4863.478652] do_idle+0xac/0x100 [ 4863.481780] cpu_startup_entry+0x30/0x34 [ 4863.485689] secondary_start_kernel+0xd8/0xf0 [ 4863.490033] __secondary_switched+0xb0/0xb4 [ 4864.982615] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 441405 jiffies s: 2093 root: 0x11/. [ 4864.994353] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4865.003562] Task dump for CPU 8: [ 4865.006776] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4865.016678] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4865.022848] Call trace: [ 4865.025281] __switch_to+0xd4/0x100 [ 4865.028758] 0xffff6c0200000080 [ 4865.031887] Task dump for CPU 68: [ 4865.035190] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4865.045092] Call trace: [ 4865.047525] __switch_to+0xd4/0x100 [ 4865.051001] 0x0 [ 4871.082564] watchdog: BUG: soft lockup - CPU#8 stuck for 4142s! [kworker/8:12:19399] [ 4871.090293] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4871.090310] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4871.178098] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4871.186348] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4871.195639] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4871.201806] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4871.208754] pc : __sbq_wake_up+0x58/0x1f0 [ 4871.212751] lr : sbitmap_queue_clear+0x74/0xc0 [ 4871.217181] sp : ffff80000d79b280 [ 4871.220482] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4871.227604] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4871.234727] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4871.241849] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4871.248971] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4871.256093] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4871.263214] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4871.270336] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4871.277458] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4871.284580] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4871.291702] Call trace: [ 4871.294136] __sbq_wake_up+0x58/0x1f0 [ 4871.297785] sbitmap_queue_clear+0x74/0xc0 [ 4871.301869] blk_mq_put_tag+0x44/0x70 [ 4871.305518] __blk_mq_free_request+0x88/0xf0 [ 4871.309775] blk_mq_free_request+0xc0/0x160 [ 4871.313945] __blk_mq_end_request+0x78/0x150 [ 4871.318202] scsi_end_request+0x114/0x260 [ 4871.322198] scsi_io_completion+0x64/0x174 [ 4871.326281] scsi_finish_command+0xe0/0x120 [ 4871.330452] scsi_complete+0x8c/0x1c4 [ 4871.334101] blk_complete_reqs+0x5c/0x74 [ 4871.338010] blk_done_softirq+0x38/0x4c [ 4871.341833] __do_softirq+0x168/0x418 [ 4871.345481] __irq_exit_rcu+0x120/0x170 [ 4871.349304] irq_exit_rcu+0x18/0x24 [ 4871.352779] el1_interrupt+0x38/0x70 [ 4871.356342] el1h_64_irq_handler+0x18/0x24 [ 4871.360425] el1h_64_irq+0x68/0x6c [ 4871.363813] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4871.368591] __wake_up_common_lock+0x84/0xac [ 4871.372848] __wake_up+0x20/0x30 [ 4871.376064] __sbq_wake_up+0x13c/0x1f0 [ 4871.379800] sbitmap_queue_clear+0x74/0xc0 [ 4871.383883] blk_mq_put_tag+0x44/0x70 [ 4871.387533] __blk_mq_free_request+0x88/0xf0 [ 4871.391790] blk_mq_free_request+0xc0/0x160 [ 4871.395960] bfq_insert_request.isra.0+0x204/0x270 [ 4871.400738] bfq_insert_requests+0x70/0x90 [ 4871.404821] blk_mq_sched_insert_requests+0xac/0x250 [ 4871.409773] blk_mq_flush_plug_list+0x110/0x2cc [ 4871.414291] __blk_flush_plug+0x100/0x140 [ 4871.418287] blk_finish_plug+0x40/0x25c [ 4871.422110] __iomap_dio_rw+0x1b4/0x500 [ 4871.425932] iomap_dio_rw+0x18/0x50 [ 4871.429408] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4871.434758] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4871.439501] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4871.444545] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4871.449675] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4871.454805] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4871.460191] process_one_work+0x1ec/0x470 [ 4871.464188] worker_thread+0x74/0x410 [ 4871.467836] kthread+0xec/0x100 [ 4871.470965] ret_from_fork+0x10/0x20 [ 4887.192455] watchdog: BUG: soft lockup - CPU#68 stuck for 4157s! [swapper/68:0] [ 4887.199750] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4887.199767] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4887.287555] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4887.295371] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4887.304662] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4887.311610] pc : __sbq_wake_up+0x58/0x1f0 [ 4887.315606] lr : sbitmap_queue_clear+0x74/0xc0 [ 4887.320037] sp : ffff8000088438b0 [ 4887.323338] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4887.330460] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4887.337582] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4887.344703] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4887.351827] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4887.358949] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4887.366070] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4887.373192] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4887.380314] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4887.387436] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4887.394558] Call trace: [ 4887.396990] __sbq_wake_up+0x58/0x1f0 [ 4887.400640] sbitmap_queue_clear+0x74/0xc0 [ 4887.404723] blk_mq_put_tag+0x44/0x70 [ 4887.408372] __blk_mq_free_request+0x88/0xf0 [ 4887.412629] blk_mq_free_request+0xc0/0x160 [ 4887.416799] __blk_mq_end_request+0x78/0x150 [ 4887.421056] scsi_end_request+0x114/0x260 [ 4887.425052] scsi_io_completion+0x64/0x174 [ 4887.429135] scsi_finish_command+0xe0/0x120 [ 4887.433305] scsi_complete+0x8c/0x1c4 [ 4887.436955] blk_complete_reqs+0x5c/0x74 [ 4887.440864] blk_done_softirq+0x38/0x4c [ 4887.444687] __do_softirq+0x168/0x418 [ 4887.448336] __irq_exit_rcu+0x120/0x170 [ 4887.452159] irq_exit_rcu+0x18/0x24 [ 4887.455634] el1_interrupt+0x38/0x70 [ 4887.459196] el1h_64_irq_handler+0x18/0x24 [ 4887.463279] el1h_64_irq+0x68/0x6c [ 4887.466667] cpuidle_enter_state+0xc4/0x4e0 [ 4887.470837] cpuidle_enter+0x40/0x60 [ 4887.474400] cpuidle_idle_call+0x118/0x1b0 [ 4887.478484] do_idle+0xac/0x100 [ 4887.481613] cpu_startup_entry+0x30/0x34 [ 4887.485522] secondary_start_kernel+0xd8/0xf0 [ 4887.489866] __secondary_switched+0xb0/0xb4 [ 4895.082402] watchdog: BUG: soft lockup - CPU#8 stuck for 4164s! [kworker/8:12:19399] [ 4895.090131] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4895.090149] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4895.177936] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4895.186186] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4895.195477] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4895.201644] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4895.208592] pc : __sbq_wake_up+0x58/0x1f0 [ 4895.212589] lr : sbitmap_queue_clear+0x74/0xc0 [ 4895.217019] sp : ffff80000d79b280 [ 4895.220320] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4895.227443] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4895.234565] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4895.241688] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4895.248810] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4895.255932] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4895.263054] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4895.270175] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4895.277297] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4895.284420] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4895.291541] Call trace: [ 4895.293974] __sbq_wake_up+0x58/0x1f0 [ 4895.297624] sbitmap_queue_clear+0x74/0xc0 [ 4895.301707] blk_mq_put_tag+0x44/0x70 [ 4895.305357] __blk_mq_free_request+0x88/0xf0 [ 4895.309614] blk_mq_free_request+0xc0/0x160 [ 4895.313784] __blk_mq_end_request+0x78/0x150 [ 4895.318041] scsi_end_request+0x114/0x260 [ 4895.322037] scsi_io_completion+0x64/0x174 [ 4895.326120] scsi_finish_command+0xe0/0x120 [ 4895.330291] scsi_complete+0x8c/0x1c4 [ 4895.333940] blk_complete_reqs+0x5c/0x74 [ 4895.337849] blk_done_softirq+0x38/0x4c [ 4895.341672] __do_softirq+0x168/0x418 [ 4895.345323] __irq_exit_rcu+0x120/0x170 [ 4895.349145] irq_exit_rcu+0x18/0x24 [ 4895.352621] el1_interrupt+0x38/0x70 [ 4895.356183] el1h_64_irq_handler+0x18/0x24 [ 4895.360267] el1h_64_irq+0x68/0x6c [ 4895.363655] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4895.368433] __wake_up_common_lock+0x84/0xac [ 4895.372690] __wake_up+0x20/0x30 [ 4895.375906] __sbq_wake_up+0x13c/0x1f0 [ 4895.379642] sbitmap_queue_clear+0x74/0xc0 [ 4895.383725] blk_mq_put_tag+0x44/0x70 [ 4895.387376] __blk_mq_free_request+0x88/0xf0 [ 4895.391632] blk_mq_free_request+0xc0/0x160 [ 4895.395802] bfq_insert_request.isra.0+0x204/0x270 [ 4895.400580] bfq_insert_requests+0x70/0x90 [ 4895.404664] blk_mq_sched_insert_requests+0xac/0x250 [ 4895.409616] blk_mq_flush_plug_list+0x110/0x2cc [ 4895.414134] __blk_flush_plug+0x100/0x140 [ 4895.418130] blk_finish_plug+0x40/0x25c [ 4895.421953] __iomap_dio_rw+0x1b4/0x500 [ 4895.425776] iomap_dio_rw+0x18/0x50 [ 4895.429251] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4895.434602] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4895.439344] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4895.444388] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4895.449518] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4895.454647] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4895.460034] process_one_work+0x1ec/0x470 [ 4895.464031] worker_thread+0x74/0x410 [ 4895.467680] kthread+0xec/0x100 [ 4895.470808] ret_from_fork+0x10/0x20 [ 4911.192293] watchdog: BUG: soft lockup - CPU#68 stuck for 4179s! [swapper/68:0] [ 4911.199588] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4911.199605] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4911.287393] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4911.295208] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4911.304499] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4911.311447] pc : __sbq_wake_up+0x84/0x1f0 [ 4911.315444] lr : sbitmap_queue_clear+0x74/0xc0 [ 4911.319874] sp : ffff8000088438b0 [ 4911.323175] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4911.330297] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4911.337419] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4911.344541] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4911.351663] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4911.358785] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4911.365907] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4911.373029] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4911.380151] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4911.387272] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4911.394394] Call trace: [ 4911.396827] __sbq_wake_up+0x84/0x1f0 [ 4911.400476] sbitmap_queue_clear+0x74/0xc0 [ 4911.404559] blk_mq_put_tag+0x44/0x70 [ 4911.408209] __blk_mq_free_request+0x88/0xf0 [ 4911.412466] blk_mq_free_request+0xc0/0x160 [ 4911.416635] __blk_mq_end_request+0x78/0x150 [ 4911.420892] scsi_end_request+0x114/0x260 [ 4911.424888] scsi_io_completion+0x64/0x174 [ 4911.428971] scsi_finish_command+0xe0/0x120 [ 4911.433142] scsi_complete+0x8c/0x1c4 [ 4911.436791] blk_complete_reqs+0x5c/0x74 [ 4911.440700] blk_done_softirq+0x38/0x4c [ 4911.444523] __do_softirq+0x168/0x418 [ 4911.448172] __irq_exit_rcu+0x120/0x170 [ 4911.451995] irq_exit_rcu+0x18/0x24 [ 4911.455470] el1_interrupt+0x38/0x70 [ 4911.459033] el1h_64_irq_handler+0x18/0x24 [ 4911.463116] el1h_64_irq+0x68/0x6c [ 4911.466504] cpuidle_enter_state+0xc4/0x4e0 [ 4911.470676] cpuidle_enter+0x40/0x60 [ 4911.474238] cpuidle_idle_call+0x118/0x1b0 [ 4911.478322] do_idle+0xac/0x100 [ 4911.481451] cpu_startup_entry+0x30/0x34 [ 4911.485360] secondary_start_kernel+0xd8/0xf0 [ 4911.489704] __secondary_switched+0xb0/0xb4 [ 4919.082240] watchdog: BUG: soft lockup - CPU#8 stuck for 4187s! [kworker/8:12:19399] [ 4919.089969] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4919.089986] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4919.177774] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4919.186024] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4919.195314] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4919.201482] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4919.208430] pc : __sbq_wake_up+0x4/0x1f0 [ 4919.212340] lr : sbitmap_queue_clear+0x74/0xc0 [ 4919.216770] sp : ffff80000d79b2d0 [ 4919.220072] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4919.227194] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4919.234316] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 4919.241438] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4919.248561] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4919.255683] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4919.262805] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4919.269927] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4919.277049] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4919.284171] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 4919.291293] Call trace: [ 4919.293726] __sbq_wake_up+0x4/0x1f0 [ 4919.297288] blk_mq_put_tag+0x44/0x70 [ 4919.300938] __blk_mq_free_request+0x88/0xf0 [ 4919.305195] blk_mq_free_request+0xc0/0x160 [ 4919.309364] __blk_mq_end_request+0x78/0x150 [ 4919.313621] scsi_end_request+0x114/0x260 [ 4919.317618] scsi_io_completion+0x64/0x174 [ 4919.321701] scsi_finish_command+0xe0/0x120 [ 4919.325871] scsi_complete+0x8c/0x1c4 [ 4919.329521] blk_complete_reqs+0x5c/0x74 [ 4919.333431] blk_done_softirq+0x38/0x4c [ 4919.337253] __do_softirq+0x168/0x418 [ 4919.340903] __irq_exit_rcu+0x120/0x170 [ 4919.344725] irq_exit_rcu+0x18/0x24 [ 4919.348201] el1_interrupt+0x38/0x70 [ 4919.351763] el1h_64_irq_handler+0x18/0x24 [ 4919.355846] el1h_64_irq+0x68/0x6c [ 4919.359235] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4919.364012] __wake_up_common_lock+0x84/0xac [ 4919.368269] __wake_up+0x20/0x30 [ 4919.371487] __sbq_wake_up+0x13c/0x1f0 [ 4919.375223] sbitmap_queue_clear+0x74/0xc0 [ 4919.379306] blk_mq_put_tag+0x44/0x70 [ 4919.382956] __blk_mq_free_request+0x88/0xf0 [ 4919.387213] blk_mq_free_request+0xc0/0x160 [ 4919.391382] bfq_insert_request.isra.0+0x204/0x270 [ 4919.396161] bfq_insert_requests+0x70/0x90 [ 4919.400244] blk_mq_sched_insert_requests+0xac/0x250 [ 4919.405196] blk_mq_flush_plug_list+0x110/0x2cc [ 4919.409714] __blk_flush_plug+0x100/0x140 [ 4919.413710] blk_finish_plug+0x40/0x25c [ 4919.417533] __iomap_dio_rw+0x1b4/0x500 [ 4919.421356] iomap_dio_rw+0x18/0x50 [ 4919.424831] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4919.430182] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4919.434924] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4919.439968] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4919.445099] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4919.450229] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4919.455616] process_one_work+0x1ec/0x470 [ 4919.459612] worker_thread+0x74/0x410 [ 4919.463260] kthread+0xec/0x100 [ 4919.466389] ret_from_fork+0x10/0x20 [ 4931.542166] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 448061 jiffies s: 2093 root: 0x11/. [ 4931.553904] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4931.563112] Task dump for CPU 8: [ 4931.566327] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4931.576229] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4931.582398] Call trace: [ 4931.584831] __switch_to+0xd4/0x100 [ 4931.588308] 0xffff6c0200000080 [ 4931.591437] Task dump for CPU 68: [ 4931.594739] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4931.604641] Call trace: [ 4931.607074] __switch_to+0xd4/0x100 [ 4931.610550] 0x0 [ 4935.192131] watchdog: BUG: soft lockup - CPU#68 stuck for 4202s! [swapper/68:0] [ 4935.199426] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4935.199443] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4935.287232] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4935.295047] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4935.304338] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4935.311286] pc : __sbq_wake_up+0x7c/0x1f0 [ 4935.315282] lr : sbitmap_queue_clear+0x74/0xc0 [ 4935.319712] sp : ffff800008843900 [ 4935.323014] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4935.330136] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4935.337258] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4935.344381] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4935.351502] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4935.358624] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4935.365746] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4935.372868] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4935.379990] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4935.387112] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 4935.394234] Call trace: [ 4935.396667] __sbq_wake_up+0x7c/0x1f0 [ 4935.400316] blk_mq_put_tag+0x44/0x70 [ 4935.403965] __blk_mq_free_request+0x88/0xf0 [ 4935.408222] blk_mq_free_request+0xc0/0x160 [ 4935.412393] __blk_mq_end_request+0x78/0x150 [ 4935.416649] scsi_end_request+0x114/0x260 [ 4935.420646] scsi_io_completion+0x64/0x174 [ 4935.424729] scsi_finish_command+0xe0/0x120 [ 4935.428899] scsi_complete+0x8c/0x1c4 [ 4935.432548] blk_complete_reqs+0x5c/0x74 [ 4935.436458] blk_done_softirq+0x38/0x4c [ 4935.440280] __do_softirq+0x168/0x418 [ 4935.443929] __irq_exit_rcu+0x120/0x170 [ 4935.447752] irq_exit_rcu+0x18/0x24 [ 4935.451228] el1_interrupt+0x38/0x70 [ 4935.454790] el1h_64_irq_handler+0x18/0x24 [ 4935.458873] el1h_64_irq+0x68/0x6c [ 4935.462261] cpuidle_enter_state+0xc4/0x4e0 [ 4935.466431] cpuidle_enter+0x40/0x60 [ 4935.469993] cpuidle_idle_call+0x118/0x1b0 [ 4935.474077] do_idle+0xac/0x100 [ 4935.477206] cpu_startup_entry+0x30/0x34 [ 4935.481115] secondary_start_kernel+0xd8/0xf0 [ 4935.485460] __secondary_switched+0xb0/0xb4 [ 4943.082078] watchdog: BUG: soft lockup - CPU#8 stuck for 4209s! [kworker/8:12:19399] [ 4943.089807] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4943.089824] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4943.177612] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4943.185862] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4943.195152] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4943.201320] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4943.208268] pc : __sbq_wake_up+0x58/0x1f0 [ 4943.212264] lr : sbitmap_queue_clear+0x74/0xc0 [ 4943.216695] sp : ffff80000d79b280 [ 4943.219997] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4943.227119] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4943.234241] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4943.241363] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4943.248485] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4943.255607] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4943.262729] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4943.269851] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4943.276973] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4943.284095] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4943.291217] Call trace: [ 4943.293650] __sbq_wake_up+0x58/0x1f0 [ 4943.297299] sbitmap_queue_clear+0x74/0xc0 [ 4943.301382] blk_mq_put_tag+0x44/0x70 [ 4943.305032] __blk_mq_free_request+0x88/0xf0 [ 4943.309289] blk_mq_free_request+0xc0/0x160 [ 4943.313459] __blk_mq_end_request+0x78/0x150 [ 4943.317715] scsi_end_request+0x114/0x260 [ 4943.321712] scsi_io_completion+0x64/0x174 [ 4943.325795] scsi_finish_command+0xe0/0x120 [ 4943.329965] scsi_complete+0x8c/0x1c4 [ 4943.333614] blk_complete_reqs+0x5c/0x74 [ 4943.337524] blk_done_softirq+0x38/0x4c [ 4943.341346] __do_softirq+0x168/0x418 [ 4943.344995] __irq_exit_rcu+0x120/0x170 [ 4943.348818] irq_exit_rcu+0x18/0x24 [ 4943.352293] el1_interrupt+0x38/0x70 [ 4943.355856] el1h_64_irq_handler+0x18/0x24 [ 4943.359939] el1h_64_irq+0x68/0x6c [ 4943.363328] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4943.368105] __wake_up_common_lock+0x84/0xac [ 4943.372363] __wake_up+0x20/0x30 [ 4943.375578] __sbq_wake_up+0x13c/0x1f0 [ 4943.379314] sbitmap_queue_clear+0x74/0xc0 [ 4943.383398] blk_mq_put_tag+0x44/0x70 [ 4943.387047] __blk_mq_free_request+0x88/0xf0 [ 4943.391304] blk_mq_free_request+0xc0/0x160 [ 4943.395474] bfq_insert_request.isra.0+0x204/0x270 [ 4943.400254] bfq_insert_requests+0x70/0x90 [ 4943.404337] blk_mq_sched_insert_requests+0xac/0x250 [ 4943.409289] blk_mq_flush_plug_list+0x110/0x2cc [ 4943.413806] __blk_flush_plug+0x100/0x140 [ 4943.417802] blk_finish_plug+0x40/0x25c [ 4943.421625] __iomap_dio_rw+0x1b4/0x500 [ 4943.425448] iomap_dio_rw+0x18/0x50 [ 4943.428923] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4943.434274] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4943.439017] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4943.444060] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4943.449191] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4943.454320] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4943.459706] process_one_work+0x1ec/0x470 [ 4943.463702] worker_thread+0x74/0x410 [ 4943.467351] kthread+0xec/0x100 [ 4943.470480] ret_from_fork+0x10/0x20 [ 4959.191969] watchdog: BUG: soft lockup - CPU#68 stuck for 4224s! [swapper/68:0] [ 4959.199265] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4959.199282] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4959.287070] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4959.294886] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4959.304177] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4959.311125] pc : __sbq_wake_up+0x58/0x1f0 [ 4959.315121] lr : sbitmap_queue_clear+0x74/0xc0 [ 4959.319551] sp : ffff8000088438b0 [ 4959.322853] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4959.329974] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4959.337097] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 4959.344219] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4959.351341] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4959.358462] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4959.365584] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4959.372706] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4959.379828] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4959.386950] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 4959.394071] Call trace: [ 4959.396504] __sbq_wake_up+0x58/0x1f0 [ 4959.400153] sbitmap_queue_clear+0x74/0xc0 [ 4959.404237] blk_mq_put_tag+0x44/0x70 [ 4959.407886] __blk_mq_free_request+0x88/0xf0 [ 4959.412143] blk_mq_free_request+0xc0/0x160 [ 4959.416313] __blk_mq_end_request+0x78/0x150 [ 4959.420570] scsi_end_request+0x114/0x260 [ 4959.424566] scsi_io_completion+0x64/0x174 [ 4959.428649] scsi_finish_command+0xe0/0x120 [ 4959.432820] scsi_complete+0x8c/0x1c4 [ 4959.436469] blk_complete_reqs+0x5c/0x74 [ 4959.440378] blk_done_softirq+0x38/0x4c [ 4959.444201] __do_softirq+0x168/0x418 [ 4959.447849] __irq_exit_rcu+0x120/0x170 [ 4959.451672] irq_exit_rcu+0x18/0x24 [ 4959.455148] el1_interrupt+0x38/0x70 [ 4959.458710] el1h_64_irq_handler+0x18/0x24 [ 4959.462793] el1h_64_irq+0x68/0x6c [ 4959.466181] cpuidle_enter_state+0xc4/0x4e0 [ 4959.470351] cpuidle_enter+0x40/0x60 [ 4959.473913] cpuidle_idle_call+0x118/0x1b0 [ 4959.477997] do_idle+0xac/0x100 [ 4959.481126] cpu_startup_entry+0x30/0x34 [ 4959.485035] secondary_start_kernel+0xd8/0xf0 [ 4959.489379] __secondary_switched+0xb0/0xb4 [ 4967.081916] watchdog: BUG: soft lockup - CPU#8 stuck for 4231s! [kworker/8:12:19399] [ 4967.089645] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4967.089663] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4967.177451] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4967.185701] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4967.194992] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4967.201159] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4967.208107] pc : __sbq_wake_up+0x84/0x1f0 [ 4967.212104] lr : sbitmap_queue_clear+0x74/0xc0 [ 4967.216534] sp : ffff80000d79b280 [ 4967.219836] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4967.226958] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4967.234080] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4967.241202] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4967.248324] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4967.255446] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4967.262569] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4967.269691] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4967.276813] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4967.283934] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4967.291056] Call trace: [ 4967.293489] __sbq_wake_up+0x84/0x1f0 [ 4967.297138] sbitmap_queue_clear+0x74/0xc0 [ 4967.301222] blk_mq_put_tag+0x44/0x70 [ 4967.304871] __blk_mq_free_request+0x88/0xf0 [ 4967.309128] blk_mq_free_request+0xc0/0x160 [ 4967.313298] __blk_mq_end_request+0x78/0x150 [ 4967.317555] scsi_end_request+0x114/0x260 [ 4967.321551] scsi_io_completion+0x64/0x174 [ 4967.325634] scsi_finish_command+0xe0/0x120 [ 4967.329804] scsi_complete+0x8c/0x1c4 [ 4967.333453] blk_complete_reqs+0x5c/0x74 [ 4967.337363] blk_done_softirq+0x38/0x4c [ 4967.341186] __do_softirq+0x168/0x418 [ 4967.344835] __irq_exit_rcu+0x120/0x170 [ 4967.348657] irq_exit_rcu+0x18/0x24 [ 4967.352133] el1_interrupt+0x38/0x70 [ 4967.355695] el1h_64_irq_handler+0x18/0x24 [ 4967.359778] el1h_64_irq+0x68/0x6c [ 4967.363167] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4967.367944] __wake_up_common_lock+0x84/0xac [ 4967.372202] __wake_up+0x20/0x30 [ 4967.375417] __sbq_wake_up+0x13c/0x1f0 [ 4967.379153] sbitmap_queue_clear+0x74/0xc0 [ 4967.383236] blk_mq_put_tag+0x44/0x70 [ 4967.386886] __blk_mq_free_request+0x88/0xf0 [ 4967.391143] blk_mq_free_request+0xc0/0x160 [ 4967.395313] bfq_insert_request.isra.0+0x204/0x270 [ 4967.400091] bfq_insert_requests+0x70/0x90 [ 4967.404174] blk_mq_sched_insert_requests+0xac/0x250 [ 4967.409126] blk_mq_flush_plug_list+0x110/0x2cc [ 4967.413644] __blk_flush_plug+0x100/0x140 [ 4967.417640] blk_finish_plug+0x40/0x25c [ 4967.421463] __iomap_dio_rw+0x1b4/0x500 [ 4967.425286] iomap_dio_rw+0x18/0x50 [ 4967.428761] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4967.434112] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4967.438855] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4967.443898] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4967.449029] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4967.454158] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4967.459545] process_one_work+0x1ec/0x470 [ 4967.463541] worker_thread+0x74/0x410 [ 4967.467190] kthread+0xec/0x100 [ 4967.470319] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 05:20:00 2022] [ 4983.191807] watchdog: BUG: soft lockup - CPU#68 stuck for 4246s! [swapper/68:0] [ 4983.199104] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4983.199121] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4983.286909] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 4983.294724] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4983.304015] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4983.310963] pc : __sbq_wake_up+0x20/0x1f0 [ 4983.314959] lr : sbitmap_queue_clear+0x74/0xc0 [ 4983.319390] sp : ffff8000088438b0 [ 4983.322691] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4983.329814] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 4983.336935] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 4983.344057] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 4983.351179] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 4983.358301] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 4983.365423] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 4983.372545] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4983.379667] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4983.386789] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 4983.393911] Call trace: [ 4983.396343] __sbq_wake_up+0x20/0x1f0 [ 4983.399992] sbitmap_queue_clear+0x74/0xc0 [ 4983.404075] blk_mq_put_tag+0x44/0x70 [ 4983.407725] __blk_mq_free_request+0x88/0xf0 [ 4983.411982] blk_mq_free_request+0xc0/0x160 [ 4983.416152] __blk_mq_end_request+0x78/0x150 [ 4983.420408] scsi_end_request+0x114/0x260 [ 4983.424405] scsi_io_completion+0x64/0x174 [ 4983.428488] scsi_finish_command+0xe0/0x120 [ 4983.432658] scsi_complete+0x8c/0x1c4 [ 4983.436307] blk_complete_reqs+0x5c/0x74 [ 4983.440217] blk_done_softirq+0x38/0x4c [ 4983.444039] __do_softirq+0x168/0x418 [ 4983.447688] __irq_exit_rcu+0x120/0x170 [ 4983.451511] irq_exit_rcu+0x18/0x24 [ 4983.454987] el1_interrupt+0x38/0x70 [ 4983.458549] el1h_64_irq_handler+0x18/0x24 [ 4983.462633] el1h_64_irq+0x68/0x6c [ 4983.466021] cpuidle_enter_state+0xc4/0x4e0 [ 4983.470191] cpuidle_enter+0x40/0x60 [ 4983.473753] cpuidle_idle_call+0x118/0x1b0 [ 4983.477836] do_idle+0xac/0x100 [ 4983.480965] cpu_startup_entry+0x30/0x34 [ 4983.484874] secondary_start_kernel+0xd8/0xf0 [ 4983.489218] __secondary_switched+0xb0/0xb4 [ 4991.081754] watchdog: BUG: soft lockup - CPU#8 stuck for 4254s! [kworker/8:12:19399] [ 4991.089483] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 4991.089501] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 4991.177288] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 4991.185538] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 4991.194829] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4991.200996] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 4991.207944] pc : __sbq_wake_up+0x58/0x1f0 [ 4991.211940] lr : sbitmap_queue_clear+0x74/0xc0 [ 4991.216371] sp : ffff80000d79b280 [ 4991.219672] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 4991.226795] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 4991.233917] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 4991.241039] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 4991.248161] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 4991.255283] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 4991.262405] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 4991.269527] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 4991.276649] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 4991.283771] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 4991.290892] Call trace: [ 4991.293325] __sbq_wake_up+0x58/0x1f0 [ 4991.296974] sbitmap_queue_clear+0x74/0xc0 [ 4991.301059] blk_mq_put_tag+0x44/0x70 [ 4991.304709] __blk_mq_free_request+0x88/0xf0 [ 4991.308965] blk_mq_free_request+0xc0/0x160 [ 4991.313135] __blk_mq_end_request+0x78/0x150 [ 4991.317393] scsi_end_request+0x114/0x260 [ 4991.321389] scsi_io_completion+0x64/0x174 [ 4991.325472] scsi_finish_command+0xe0/0x120 [ 4991.329642] scsi_complete+0x8c/0x1c4 [ 4991.333292] blk_complete_reqs+0x5c/0x74 [ 4991.337201] blk_done_softirq+0x38/0x4c [ 4991.341024] __do_softirq+0x168/0x418 [ 4991.344673] __irq_exit_rcu+0x120/0x170 [ 4991.348495] irq_exit_rcu+0x18/0x24 [ 4991.351971] el1_interrupt+0x38/0x70 [ 4991.355533] el1h_64_irq_handler+0x18/0x24 [ 4991.359616] el1h_64_irq+0x68/0x6c [ 4991.363005] _raw_spin_unlock_irqrestore+0x20/0x50 [ 4991.367782] __wake_up_common_lock+0x84/0xac [ 4991.372040] __wake_up+0x20/0x30 [ 4991.375255] __sbq_wake_up+0x13c/0x1f0 [ 4991.378991] sbitmap_queue_clear+0x74/0xc0 [ 4991.383075] blk_mq_put_tag+0x44/0x70 [ 4991.386724] __blk_mq_free_request+0x88/0xf0 [ 4991.390981] blk_mq_free_request+0xc0/0x160 [ 4991.395151] bfq_insert_request.isra.0+0x204/0x270 [ 4991.399929] bfq_insert_requests+0x70/0x90 [ 4991.404012] blk_mq_sched_insert_requests+0xac/0x250 [ 4991.408964] blk_mq_flush_plug_list+0x110/0x2cc [ 4991.413482] __blk_flush_plug+0x100/0x140 [ 4991.417478] blk_finish_plug+0x40/0x25c [ 4991.421302] __iomap_dio_rw+0x1b4/0x500 [ 4991.425125] iomap_dio_rw+0x18/0x50 [ 4991.428600] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 4991.433951] xfs_file_write_iter+0xf4/0x15c [xfs] [ 4991.438693] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 4991.443737] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 4991.448867] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 4991.453997] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 4991.459383] process_one_work+0x1ec/0x470 [ 4991.463379] worker_thread+0x74/0x410 [ 4991.467028] kthread+0xec/0x100 [ 4991.470157] ret_from_fork+0x10/0x20 [ 4992.521745] rcu: INFO: rcu_preempt self-detected stall on CPU [ 4992.527478] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=226538 [ 4992.536685] (t=456765 jiffies g=36445 q=135175 ncpus=128) [ 4992.542157] Task dump for CPU 8: [ 4992.545373] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4992.555273] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4992.561440] Call trace: [ 4992.563874] __switch_to+0xd4/0x100 [ 4992.567350] 0xffff6c0200000080 [ 4992.570478] Task dump for CPU 68: [ 4992.573779] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4992.583679] Call trace: [ 4992.586111] dump_backtrace+0xe8/0x130 [ 4992.589847] show_stack+0x20/0x60 [ 4992.593149] sched_show_task+0x160/0x190 [ 4992.597059] dump_cpu_task+0x4c/0x78 [ 4992.600621] rcu_dump_cpu_stacks+0xc4/0x150 [ 4992.604792] print_cpu_stall+0x180/0x210 [ 4992.608701] check_cpu_stall+0x20c/0x2b4 [ 4992.612611] rcu_pending+0x4c/0x190 [ 4992.616086] rcu_sched_clock_irq+0x80/0x204 [ 4992.620256] update_process_times+0x7c/0xc0 [ 4992.624426] tick_sched_handle+0x38/0x74 [ 4992.628336] tick_sched_timer+0x60/0xc0 [ 4992.632160] __hrtimer_run_queues+0x170/0x300 [ 4992.636505] hrtimer_interrupt+0xfc/0x260 [ 4992.640501] arch_timer_handler_phys+0x34/0x50 [ 4992.644931] handle_percpu_devid_irq+0xac/0x240 [ 4992.649449] generic_handle_domain_irq+0x34/0x50 [ 4992.654053] gic_handle_irq+0x5c/0xe0 [ 4992.657702] call_on_irq_stack+0x2c/0x38 [ 4992.661612] do_interrupt_handler+0xdc/0xe0 [ 4992.665782] el1_interrupt+0x34/0x70 [ 4992.669345] el1h_64_irq_handler+0x18/0x24 [ 4992.673428] el1h_64_irq+0x68/0x6c [ 4992.676816] __sbq_wake_up+0x58/0x1f0 [ 4992.680465] sbitmap_queue_clear+0x74/0xc0 [ 4992.684549] blk_mq_put_tag+0x44/0x70 [ 4992.688199] __blk_mq_free_request+0x88/0xf0 [ 4992.692455] blk_mq_free_request+0xc0/0x160 [ 4992.696625] __blk_mq_end_request+0x78/0x150 [ 4992.700882] scsi_end_request+0x114/0x260 [ 4992.704878] scsi_io_completion+0x64/0x174 [ 4992.708961] scsi_finish_command+0xe0/0x120 [ 4992.713132] scsi_complete+0x8c/0x1c4 [ 4992.716781] blk_complete_reqs+0x5c/0x74 [ 4992.720690] blk_done_softirq+0x38/0x4c [ 4992.724513] __do_softirq+0x168/0x418 [ 4992.728162] __irq_exit_rcu+0x120/0x170 [ 4992.731985] irq_exit_rcu+0x18/0x24 [ 4992.735460] el1_interrupt+0x38/0x70 [ 4992.739023] el1h_64_irq_handler+0x18/0x24 [ 4992.743106] el1h_64_irq+0x68/0x6c [ 4992.746494] cpuidle_enter_state+0xc4/0x4e0 [ 4992.750664] cpuidle_enter+0x40/0x60 [ 4992.754227] cpuidle_idle_call+0x118/0x1b0 [ 4992.758310] do_idle+0xac/0x100 [ 4992.761439] cpu_startup_entry+0x30/0x34 [ 4992.765348] secondary_start_kernel+0xd8/0xf0 [ 4992.769693] __secondary_switched+0xb0/0xb4 [ 4998.101717] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 454717 jiffies s: 2093 root: 0x11/. [ 4998.113456] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 4998.122665] Task dump for CPU 8: [ 4998.125879] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 4998.135781] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 4998.141951] Call trace: [ 4998.144384] __switch_to+0xd4/0x100 [ 4998.147860] 0xffff6c0200000080 [ 4998.150989] Task dump for CPU 68: [ 4998.154292] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 4998.164193] Call trace: [ 4998.166625] __switch_to+0xd4/0x100 [ 4998.170101] 0x0 [ 5015.081592] watchdog: BUG: soft lockup - CPU#8 stuck for 4276s! [kworker/8:12:19399] [ 5015.089322] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5015.089339] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5015.177126] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5015.185376] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5015.194667] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5015.200835] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5015.207783] pc : __sbq_wake_up+0x20/0x1f0 [ 5015.211779] lr : sbitmap_queue_clear+0x74/0xc0 [ 5015.216210] sp : ffff80000d79b280 [ 5015.219511] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5015.226634] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5015.233755] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5015.240878] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5015.248001] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5015.255123] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5015.262245] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5015.269367] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5015.276489] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5015.283610] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5015.290732] Call trace: [ 5015.293165] __sbq_wake_up+0x20/0x1f0 [ 5015.296815] sbitmap_queue_clear+0x74/0xc0 [ 5015.300898] blk_mq_put_tag+0x44/0x70 [ 5015.304548] __blk_mq_free_request+0x88/0xf0 [ 5015.308804] blk_mq_free_request+0xc0/0x160 [ 5015.312974] __blk_mq_end_request+0x78/0x150 [ 5015.317231] scsi_end_request+0x114/0x260 [ 5015.321228] scsi_io_completion+0x64/0x174 [ 5015.325311] scsi_finish_command+0xe0/0x120 [ 5015.329482] scsi_complete+0x8c/0x1c4 [ 5015.333131] blk_complete_reqs+0x5c/0x74 [ 5015.337041] blk_done_softirq+0x38/0x4c [ 5015.340863] __do_softirq+0x168/0x418 [ 5015.344512] __irq_exit_rcu+0x120/0x170 [ 5015.348334] irq_exit_rcu+0x18/0x24 [ 5015.351810] el1_interrupt+0x38/0x70 [ 5015.355372] el1h_64_irq_handler+0x18/0x24 [ 5015.359455] el1h_64_irq+0x68/0x6c [ 5015.362843] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5015.367621] __wake_up_common_lock+0x84/0xac [ 5015.371878] __wake_up+0x20/0x30 [ 5015.375094] __sbq_wake_up+0x13c/0x1f0 [ 5015.378830] sbitmap_queue_clear+0x74/0xc0 [ 5015.382913] blk_mq_put_tag+0x44/0x70 [ 5015.386563] __blk_mq_free_request+0x88/0xf0 [ 5015.390820] blk_mq_free_request+0xc0/0x160 [ 5015.394989] bfq_insert_request.isra.0+0x204/0x270 [ 5015.399768] bfq_insert_requests+0x70/0x90 [ 5015.403851] blk_mq_sched_insert_requests+0xac/0x250 [ 5015.408804] blk_mq_flush_plug_list+0x110/0x2cc [ 5015.413321] __blk_flush_plug+0x100/0x140 [ 5015.417317] blk_finish_plug+0x40/0x25c [ 5015.421141] __iomap_dio_rw+0x1b4/0x500 [ 5015.424963] iomap_dio_rw+0x18/0x50 [ 5015.428439] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5015.433789] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5015.438532] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5015.443575] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5015.448705] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5015.453835] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5015.459221] process_one_work+0x1ec/0x470 [ 5015.463217] worker_thread+0x74/0x410 [ 5015.466866] kthread+0xec/0x100 [ 5015.469995] ret_from_fork+0x10/0x20 [ 5019.191565] watchdog: BUG: soft lockup - CPU#68 stuck for 4280s! [swapper/68:0] [ 5019.198860] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5019.198877] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5019.286666] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5019.294481] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5019.303772] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5019.310720] pc : __sbq_wake_up+0x58/0x1f0 [ 5019.314716] lr : sbitmap_queue_clear+0x74/0xc0 [ 5019.319147] sp : ffff8000088438b0 [ 5019.322448] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5019.329570] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5019.336692] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5019.343814] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5019.350936] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5019.358058] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5019.365180] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5019.372302] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5019.379424] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5019.386546] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5019.393668] Call trace: [ 5019.396101] __sbq_wake_up+0x58/0x1f0 [ 5019.399750] sbitmap_queue_clear+0x74/0xc0 [ 5019.403833] blk_mq_put_tag+0x44/0x70 [ 5019.407483] __blk_mq_free_request+0x88/0xf0 [ 5019.411740] blk_mq_free_request+0xc0/0x160 [ 5019.415909] __blk_mq_end_request+0x78/0x150 [ 5019.420166] scsi_end_request+0x114/0x260 [ 5019.424163] scsi_io_completion+0x64/0x174 [ 5019.428247] scsi_finish_command+0xe0/0x120 [ 5019.432417] scsi_complete+0x8c/0x1c4 [ 5019.436066] blk_complete_reqs+0x5c/0x74 [ 5019.439976] blk_done_softirq+0x38/0x4c [ 5019.443798] __do_softirq+0x168/0x418 [ 5019.447447] __irq_exit_rcu+0x120/0x170 [ 5019.451270] irq_exit_rcu+0x18/0x24 [ 5019.454746] el1_interrupt+0x38/0x70 [ 5019.458308] el1h_64_irq_handler+0x18/0x24 [ 5019.462391] el1h_64_irq+0x68/0x6c [ 5019.465780] cpuidle_enter_state+0xc4/0x4e0 [ 5019.469950] cpuidle_enter+0x40/0x60 [ 5019.473512] cpuidle_idle_call+0x118/0x1b0 [ 5019.477596] do_idle+0xac/0x100 [ 5019.480725] cpu_startup_entry+0x30/0x34 [ 5019.484634] secondary_start_kernel+0xd8/0xf0 [ 5019.488978] __secondary_switched+0xb0/0xb4 [ 5039.081430] watchdog: BUG: soft lockup - CPU#8 stuck for 4299s! [kworker/8:12:19399] [ 5039.089159] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5039.089176] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5039.176964] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5039.185213] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5039.194504] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5039.200672] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5039.207620] pc : __sbq_wake_up+0x58/0x1f0 [ 5039.211616] lr : sbitmap_queue_clear+0x74/0xc0 [ 5039.216047] sp : ffff80000d79b280 [ 5039.219349] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5039.226471] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5039.233593] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5039.240716] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5039.247838] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5039.254960] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5039.262082] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5039.269204] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5039.276326] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5039.283448] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5039.290570] Call trace: [ 5039.293002] __sbq_wake_up+0x58/0x1f0 [ 5039.296651] sbitmap_queue_clear+0x74/0xc0 [ 5039.300735] blk_mq_put_tag+0x44/0x70 [ 5039.304385] __blk_mq_free_request+0x88/0xf0 [ 5039.308641] blk_mq_free_request+0xc0/0x160 [ 5039.312811] __blk_mq_end_request+0x78/0x150 [ 5039.317068] scsi_end_request+0x114/0x260 [ 5039.321064] scsi_io_completion+0x64/0x174 [ 5039.325147] scsi_finish_command+0xe0/0x120 [ 5039.329318] scsi_complete+0x8c/0x1c4 [ 5039.332967] blk_complete_reqs+0x5c/0x74 [ 5039.336876] blk_done_softirq+0x38/0x4c [ 5039.340700] __do_softirq+0x168/0x418 [ 5039.344349] __irq_exit_rcu+0x120/0x170 [ 5039.348171] irq_exit_rcu+0x18/0x24 [ 5039.351647] el1_interrupt+0x38/0x70 [ 5039.355209] el1h_64_irq_handler+0x18/0x24 [ 5039.359292] el1h_64_irq+0x68/0x6c [ 5039.362681] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5039.367458] __wake_up_common_lock+0x84/0xac [ 5039.371716] __wake_up+0x20/0x30 [ 5039.374932] __sbq_wake_up+0x13c/0x1f0 [ 5039.378668] sbitmap_queue_clear+0x74/0xc0 [ 5039.382751] blk_mq_put_tag+0x44/0x70 [ 5039.386400] __blk_mq_free_request+0x88/0xf0 [ 5039.390657] blk_mq_free_request+0xc0/0x160 [ 5039.394827] bfq_insert_request.isra.0+0x204/0x270 [ 5039.399605] bfq_insert_requests+0x70/0x90 [ 5039.403689] blk_mq_sched_insert_requests+0xac/0x250 [ 5039.408641] blk_mq_flush_plug_list+0x110/0x2cc [ 5039.413159] __blk_flush_plug+0x100/0x140 [ 5039.417155] blk_finish_plug+0x40/0x25c [ 5039.420978] __iomap_dio_rw+0x1b4/0x500 [ 5039.424802] iomap_dio_rw+0x18/0x50 [ 5039.428277] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5039.433628] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5039.438370] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5039.443414] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5039.448544] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5039.453674] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5039.459060] process_one_work+0x1ec/0x470 [ 5039.463056] worker_thread+0x74/0x410 [ 5039.466705] kthread+0xec/0x100 [ 5039.469834] ret_from_fork+0x10/0x20 [ 5043.191403] watchdog: BUG: soft lockup - CPU#68 stuck for 4302s! [swapper/68:0] [ 5043.198698] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5043.198715] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5043.286503] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5043.294318] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5043.303610] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5043.310557] pc : __sbq_wake_up+0x58/0x1f0 [ 5043.314554] lr : sbitmap_queue_clear+0x74/0xc0 [ 5043.318984] sp : ffff8000088438b0 [ 5043.322286] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5043.329408] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5043.336530] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5043.343652] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5043.350774] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5043.357896] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5043.365018] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5043.372140] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5043.379262] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5043.386385] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5043.393507] Call trace: [ 5043.395940] __sbq_wake_up+0x58/0x1f0 [ 5043.399589] sbitmap_queue_clear+0x74/0xc0 [ 5043.403672] blk_mq_put_tag+0x44/0x70 [ 5043.407322] __blk_mq_free_request+0x88/0xf0 [ 5043.411579] blk_mq_free_request+0xc0/0x160 [ 5043.415748] __blk_mq_end_request+0x78/0x150 [ 5043.420005] scsi_end_request+0x114/0x260 [ 5043.424001] scsi_io_completion+0x64/0x174 [ 5043.428085] scsi_finish_command+0xe0/0x120 [ 5043.432255] scsi_complete+0x8c/0x1c4 [ 5043.435904] blk_complete_reqs+0x5c/0x74 [ 5043.439814] blk_done_softirq+0x38/0x4c [ 5043.443636] __do_softirq+0x168/0x418 [ 5043.447285] __irq_exit_rcu+0x120/0x170 [ 5043.451107] irq_exit_rcu+0x18/0x24 [ 5043.454583] el1_interrupt+0x38/0x70 [ 5043.458145] el1h_64_irq_handler+0x18/0x24 [ 5043.462229] el1h_64_irq+0x68/0x6c [ 5043.465618] cpuidle_enter_state+0xc4/0x4e0 [ 5043.469788] cpuidle_enter+0x40/0x60 [ 5043.473351] cpuidle_idle_call+0x118/0x1b0 [ 5043.477434] do_idle+0xac/0x100 [ 5043.480563] cpu_startup_entry+0x30/0x34 [ 5043.484472] secondary_start_kernel+0xd8/0xf0 [ 5043.488816] __secondary_switched+0xb0/0xb4 [ 5063.081268] watchdog: BUG: soft lockup - CPU#8 stuck for 4321s! [kworker/8:12:19399] [ 5063.088998] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5063.089015] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5063.176802] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5063.185052] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5063.194344] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5063.200511] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5063.207459] pc : __sbq_wake_up+0x28/0x1f0 [ 5063.211455] lr : sbitmap_queue_clear+0x74/0xc0 [ 5063.215886] sp : ffff80000d79b280 [ 5063.219188] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5063.226310] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5063.233432] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5063.240555] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5063.247677] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5063.254800] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5063.261923] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5063.269045] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5063.276166] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5063.283288] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000039 [ 5063.290410] Call trace: [ 5063.292843] __sbq_wake_up+0x28/0x1f0 [ 5063.296493] sbitmap_queue_clear+0x74/0xc0 [ 5063.300576] blk_mq_put_tag+0x44/0x70 [ 5063.304226] __blk_mq_free_request+0x88/0xf0 [ 5063.308483] blk_mq_free_request+0xc0/0x160 [ 5063.312653] __blk_mq_end_request+0x78/0x150 [ 5063.316910] scsi_end_request+0x114/0x260 [ 5063.320907] scsi_io_completion+0x64/0x174 [ 5063.324990] scsi_finish_command+0xe0/0x120 [ 5063.329160] scsi_complete+0x8c/0x1c4 [ 5063.332809] blk_complete_reqs+0x5c/0x74 [ 5063.336719] blk_done_softirq+0x38/0x4c [ 5063.340541] __do_softirq+0x168/0x418 [ 5063.344190] __irq_exit_rcu+0x120/0x170 [ 5063.348012] irq_exit_rcu+0x18/0x24 [ 5063.351488] el1_interrupt+0x38/0x70 [ 5063.355050] el1h_64_irq_handler+0x18/0x24 [ 5063.359133] el1h_64_irq+0x68/0x6c [ 5063.362522] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5063.367300] __wake_up_common_lock+0x84/0xac [ 5063.371557] __wake_up+0x20/0x30 [ 5063.374772] __sbq_wake_up+0x13c/0x1f0 [ 5063.378509] sbitmap_queue_clear+0x74/0xc0 [ 5063.382592] blk_mq_put_tag+0x44/0x70 [ 5063.386242] __blk_mq_free_request+0x88/0xf0 [ 5063.390499] blk_mq_free_request+0xc0/0x160 [ 5063.394669] bfq_insert_request.isra.0+0x204/0x270 [ 5063.399447] bfq_insert_requests+0x70/0x90 [ 5063.403530] blk_mq_sched_insert_requests+0xac/0x250 [ 5063.408483] blk_mq_flush_plug_list+0x110/0x2cc [ 5063.413000] __blk_flush_plug+0x100/0x140 [ 5063.416997] blk_finish_plug+0x40/0x25c [ 5063.420819] __iomap_dio_rw+0x1b4/0x500 [ 5063.424642] iomap_dio_rw+0x18/0x50 [ 5063.428117] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5063.433468] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5063.438210] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5063.443254] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5063.448385] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5063.453514] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5063.458901] process_one_work+0x1ec/0x470 [ 5063.462897] worker_thread+0x74/0x410 [ 5063.466546] kthread+0xec/0x100 [ 5063.469675] ret_from_fork+0x10/0x20 [ 5064.661268] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 461373 jiffies s: 2093 root: 0x11/. [ 5064.673006] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5064.682215] Task dump for CPU 8: [ 5064.685429] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5064.695331] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5064.701501] Call trace: [ 5064.703934] __switch_to+0xd4/0x100 [ 5064.707410] 0xffff6c0200000080 [ 5064.710539] Task dump for CPU 68: [ 5064.713842] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5064.723743] Call trace: [ 5064.726176] __switch_to+0xd4/0x100 [ 5064.729653] 0x0 [ 5067.191241] watchdog: BUG: soft lockup - CPU#68 stuck for 4325s! [swapper/68:0] [ 5067.198536] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5067.198553] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5067.286341] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5067.294156] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5067.303447] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5067.310395] pc : __sbq_wake_up+0x7c/0x1f0 [ 5067.314392] lr : sbitmap_queue_clear+0x74/0xc0 [ 5067.318822] sp : ffff800008843900 [ 5067.322124] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5067.329246] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5067.336367] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5067.343489] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5067.350611] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5067.357733] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5067.364856] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5067.371978] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5067.379099] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5067.386221] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5067.393343] Call trace: [ 5067.395777] __sbq_wake_up+0x7c/0x1f0 [ 5067.399426] blk_mq_put_tag+0x44/0x70 [ 5067.403076] __blk_mq_free_request+0x88/0xf0 [ 5067.407333] blk_mq_free_request+0xc0/0x160 [ 5067.411503] __blk_mq_end_request+0x78/0x150 [ 5067.415759] scsi_end_request+0x114/0x260 [ 5067.419756] scsi_io_completion+0x64/0x174 [ 5067.423839] scsi_finish_command+0xe0/0x120 [ 5067.428009] scsi_complete+0x8c/0x1c4 [ 5067.431658] blk_complete_reqs+0x5c/0x74 [ 5067.435568] blk_done_softirq+0x38/0x4c [ 5067.439391] __do_softirq+0x168/0x418 [ 5067.443040] __irq_exit_rcu+0x120/0x170 [ 5067.446862] irq_exit_rcu+0x18/0x24 [ 5067.450338] el1_interrupt+0x38/0x70 [ 5067.453900] el1h_64_irq_handler+0x18/0x24 [ 5067.457983] el1h_64_irq+0x68/0x6c [ 5067.461372] cpuidle_enter_state+0xc4/0x4e0 [ 5067.465542] cpuidle_enter+0x40/0x60 [ 5067.469104] cpuidle_idle_call+0x118/0x1b0 [ 5067.473188] do_idle+0xac/0x100 [ 5067.476317] cpu_startup_entry+0x30/0x34 [ 5067.480226] secondary_start_kernel+0xd8/0xf0 [ 5067.484570] __secondary_switched+0xb0/0xb4 [ 5087.081107] watchdog: BUG: soft lockup - CPU#8 stuck for 4343s! [kworker/8:12:19399] [ 5087.088836] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5087.088853] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5087.176641] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5087.184890] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5087.194181] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5087.200349] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5087.207296] pc : __sbq_wake_up+0x20/0x1f0 [ 5087.211293] lr : sbitmap_queue_clear+0x74/0xc0 [ 5087.215724] sp : ffff80000d79b280 [ 5087.219026] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5087.226148] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5087.233270] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5087.240393] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5087.247514] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5087.254636] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5087.261758] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5087.268880] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5087.276004] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5087.283126] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5087.290248] Call trace: [ 5087.292681] __sbq_wake_up+0x20/0x1f0 [ 5087.296330] sbitmap_queue_clear+0x74/0xc0 [ 5087.300414] blk_mq_put_tag+0x44/0x70 [ 5087.304064] __blk_mq_free_request+0x88/0xf0 [ 5087.308320] blk_mq_free_request+0xc0/0x160 [ 5087.312490] __blk_mq_end_request+0x78/0x150 [ 5087.316748] scsi_end_request+0x114/0x260 [ 5087.320744] scsi_io_completion+0x64/0x174 [ 5087.324827] scsi_finish_command+0xe0/0x120 [ 5087.328997] scsi_complete+0x8c/0x1c4 [ 5087.332647] blk_complete_reqs+0x5c/0x74 [ 5087.336556] blk_done_softirq+0x38/0x4c [ 5087.340379] __do_softirq+0x168/0x418 [ 5087.344029] __irq_exit_rcu+0x120/0x170 [ 5087.347851] irq_exit_rcu+0x18/0x24 [ 5087.351327] el1_interrupt+0x38/0x70 [ 5087.354889] el1h_64_irq_handler+0x18/0x24 [ 5087.358972] el1h_64_irq+0x68/0x6c [ 5087.362361] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5087.367139] __wake_up_common_lock+0x84/0xac [ 5087.371396] __wake_up+0x20/0x30 [ 5087.374611] __sbq_wake_up+0x13c/0x1f0 [ 5087.378347] sbitmap_queue_clear+0x74/0xc0 [ 5087.382430] blk_mq_put_tag+0x44/0x70 [ 5087.386080] __blk_mq_free_request+0x88/0xf0 [ 5087.390338] blk_mq_free_request+0xc0/0x160 [ 5087.394508] bfq_insert_request.isra.0+0x204/0x270 [ 5087.399286] bfq_insert_requests+0x70/0x90 [ 5087.403369] blk_mq_sched_insert_requests+0xac/0x250 [ 5087.408321] blk_mq_flush_plug_list+0x110/0x2cc [ 5087.412839] __blk_flush_plug+0x100/0x140 [ 5087.416836] blk_finish_plug+0x40/0x25c [ 5087.420658] __iomap_dio_rw+0x1b4/0x500 [ 5087.424481] iomap_dio_rw+0x18/0x50 [ 5087.427956] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5087.433307] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5087.438050] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5087.443094] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5087.448223] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5087.453353] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5087.458739] process_one_work+0x1ec/0x470 [ 5087.462735] worker_thread+0x74/0x410 [ 5087.466384] kthread+0xec/0x100 [ 5087.469513] ret_from_fork+0x10/0x20 [ 5091.191079] watchdog: BUG: soft lockup - CPU#68 stuck for 4347s! [swapper/68:0] [ 5091.198374] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5091.198391] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5091.286178] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5091.293994] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5091.303284] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5091.310232] pc : __sbq_wake_up+0x1c8/0x1f0 [ 5091.314315] lr : sbitmap_queue_clear+0x74/0xc0 [ 5091.318746] sp : ffff8000088438b0 [ 5091.322047] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5091.329170] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5091.336292] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5091.343413] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5091.350536] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5091.357658] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5091.364780] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5091.371902] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5091.379024] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5091.386146] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 5091.393268] Call trace: [ 5091.395701] __sbq_wake_up+0x1c8/0x1f0 [ 5091.399437] sbitmap_queue_clear+0x74/0xc0 [ 5091.403520] blk_mq_put_tag+0x44/0x70 [ 5091.407170] __blk_mq_free_request+0x88/0xf0 [ 5091.411426] blk_mq_free_request+0xc0/0x160 [ 5091.415596] __blk_mq_end_request+0x78/0x150 [ 5091.419853] scsi_end_request+0x114/0x260 [ 5091.423849] scsi_io_completion+0x64/0x174 [ 5091.427933] scsi_finish_command+0xe0/0x120 [ 5091.432103] scsi_complete+0x8c/0x1c4 [ 5091.435752] blk_complete_reqs+0x5c/0x74 [ 5091.439662] blk_done_softirq+0x38/0x4c [ 5091.443486] __do_softirq+0x168/0x418 [ 5091.447134] __irq_exit_rcu+0x120/0x170 [ 5091.450957] irq_exit_rcu+0x18/0x24 [ 5091.454432] el1_interrupt+0x38/0x70 [ 5091.457995] el1h_64_irq_handler+0x18/0x24 [ 5091.462078] el1h_64_irq+0x68/0x6c [ 5091.465466] cpuidle_enter_state+0xc4/0x4e0 [ 5091.469636] cpuidle_enter+0x40/0x60 [ 5091.473199] cpuidle_idle_call+0x118/0x1b0 [ 5091.477282] do_idle+0xac/0x100 [ 5091.480411] cpu_startup_entry+0x30/0x34 [ 5091.484320] secondary_start_kernel+0xd8/0xf0 [ 5091.488664] __secondary_switched+0xb0/0xb4 [ 5111.080945] watchdog: BUG: soft lockup - CPU#8 stuck for 4366s! [kworker/8:12:19399] [ 5111.088674] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5111.088690] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5111.176478] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5111.184728] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5111.194019] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5111.200186] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5111.207134] pc : __sbq_wake_up+0x84/0x1f0 [ 5111.211131] lr : sbitmap_queue_clear+0x74/0xc0 [ 5111.215561] sp : ffff80000d79b280 [ 5111.218863] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5111.225986] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5111.233108] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5111.240230] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5111.247352] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5111.254474] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5111.261596] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5111.268718] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5111.275840] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5111.282961] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5111.290084] Call trace: [ 5111.292518] __sbq_wake_up+0x84/0x1f0 [ 5111.296167] sbitmap_queue_clear+0x74/0xc0 [ 5111.300250] blk_mq_put_tag+0x44/0x70 [ 5111.303900] __blk_mq_free_request+0x88/0xf0 [ 5111.308157] blk_mq_free_request+0xc0/0x160 [ 5111.312327] __blk_mq_end_request+0x78/0x150 [ 5111.316584] scsi_end_request+0x114/0x260 [ 5111.320581] scsi_io_completion+0x64/0x174 [ 5111.324664] scsi_finish_command+0xe0/0x120 [ 5111.328835] scsi_complete+0x8c/0x1c4 [ 5111.332484] blk_complete_reqs+0x5c/0x74 [ 5111.336393] blk_done_softirq+0x38/0x4c [ 5111.340216] __do_softirq+0x168/0x418 [ 5111.343865] __irq_exit_rcu+0x120/0x170 [ 5111.347687] irq_exit_rcu+0x18/0x24 [ 5111.351163] el1_interrupt+0x38/0x70 [ 5111.354725] el1h_64_irq_handler+0x18/0x24 [ 5111.358808] el1h_64_irq+0x68/0x6c [ 5111.362197] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5111.366974] __wake_up_common_lock+0x84/0xac [ 5111.371232] __wake_up+0x20/0x30 [ 5111.374448] __sbq_wake_up+0x13c/0x1f0 [ 5111.378184] sbitmap_queue_clear+0x74/0xc0 [ 5111.382267] blk_mq_put_tag+0x44/0x70 [ 5111.385917] __blk_mq_free_request+0x88/0xf0 [ 5111.390174] blk_mq_free_request+0xc0/0x160 [ 5111.394344] bfq_insert_request.isra.0+0x204/0x270 [ 5111.399122] bfq_insert_requests+0x70/0x90 [ 5111.403205] blk_mq_sched_insert_requests+0xac/0x250 [ 5111.408157] blk_mq_flush_plug_list+0x110/0x2cc [ 5111.412675] __blk_flush_plug+0x100/0x140 [ 5111.416671] blk_finish_plug+0x40/0x25c [ 5111.420494] __iomap_dio_rw+0x1b4/0x500 [ 5111.424317] iomap_dio_rw+0x18/0x50 [ 5111.427792] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5111.433143] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5111.437885] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5111.442929] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5111.448058] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5111.453188] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5111.458574] process_one_work+0x1ec/0x470 [ 5111.462571] worker_thread+0x74/0x410 [ 5111.466220] kthread+0xec/0x100 [ 5111.469349] ret_from_fork+0x10/0x20 [ 5115.190917] watchdog: BUG: soft lockup - CPU#68 stuck for 4369s! [swapper/68:0] [ 5115.198212] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5115.198229] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5115.286016] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5115.293832] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5115.303123] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5115.310071] pc : __sbq_wake_up+0x9c/0x1f0 [ 5115.314068] lr : sbitmap_queue_clear+0x74/0xc0 [ 5115.318498] sp : ffff8000088438b0 [ 5115.321800] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5115.328921] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5115.336043] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5115.343166] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5115.350287] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5115.357409] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5115.364531] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5115.371653] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5115.378775] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5115.385897] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000007 [ 5115.393018] Call trace: [ 5115.395451] __sbq_wake_up+0x9c/0x1f0 [ 5115.399100] sbitmap_queue_clear+0x74/0xc0 [ 5115.403183] blk_mq_put_tag+0x44/0x70 [ 5115.406833] __blk_mq_free_request+0x88/0xf0 [ 5115.411090] blk_mq_free_request+0xc0/0x160 [ 5115.415259] __blk_mq_end_request+0x78/0x150 [ 5115.419516] scsi_end_request+0x114/0x260 [ 5115.423513] scsi_io_completion+0x64/0x174 [ 5115.427596] scsi_finish_command+0xe0/0x120 [ 5115.431766] scsi_complete+0x8c/0x1c4 [ 5115.435415] blk_complete_reqs+0x5c/0x74 [ 5115.439325] blk_done_softirq+0x38/0x4c [ 5115.443147] __do_softirq+0x168/0x418 [ 5115.446796] __irq_exit_rcu+0x120/0x170 [ 5115.450618] irq_exit_rcu+0x18/0x24 [ 5115.454094] el1_interrupt+0x38/0x70 [ 5115.457656] el1h_64_irq_handler+0x18/0x24 [ 5115.461739] el1h_64_irq+0x68/0x6c [ 5115.465128] cpuidle_enter_state+0xc4/0x4e0 [ 5115.469298] cpuidle_enter+0x40/0x60 [ 5115.472860] cpuidle_idle_call+0x118/0x1b0 [ 5115.476945] do_idle+0xac/0x100 [ 5115.480074] cpu_startup_entry+0x30/0x34 [ 5115.483983] secondary_start_kernel+0xd8/0xf0 [ 5115.488327] __secondary_switched+0xb0/0xb4 [ 5131.220873] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 468029 jiffies s: 2093 root: 0x11/. [ 5131.232611] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5131.241819] Task dump for CPU 8: [ 5131.245034] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5131.254935] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5131.261105] Call trace: [ 5131.263538] __switch_to+0xd4/0x100 [ 5131.267015] 0xffff6c0200000080 [ 5131.270143] Task dump for CPU 68: [ 5131.273446] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5131.283347] Call trace: [ 5131.285780] __switch_to+0xd4/0x100 [ 5131.289257] 0x0 [ 5135.080783] watchdog: BUG: soft lockup - CPU#8 stuck for 4388s! [kworker/8:12:19399] [ 5135.088512] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5135.088529] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5135.176317] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5135.184567] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5135.193858] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5135.200025] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5135.206973] pc : __sbq_wake_up+0x4/0x1f0 [ 5135.210883] lr : sbitmap_queue_clear+0x74/0xc0 [ 5135.215314] sp : ffff80000d79b2d0 [ 5135.218615] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5135.225737] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5135.232860] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5135.239982] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5135.247105] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5135.254227] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5135.261349] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5135.268472] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5135.275593] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5135.282716] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff91ccdb10 [ 5135.289838] Call trace: [ 5135.292271] __sbq_wake_up+0x4/0x1f0 [ 5135.295833] blk_mq_put_tag+0x44/0x70 [ 5135.299483] __blk_mq_free_request+0x88/0xf0 [ 5135.303741] blk_mq_free_request+0xc0/0x160 [ 5135.307911] __blk_mq_end_request+0x78/0x150 [ 5135.312168] scsi_end_request+0x114/0x260 [ 5135.316164] scsi_io_completion+0x64/0x174 [ 5135.320247] scsi_finish_command+0xe0/0x120 [ 5135.324418] scsi_complete+0x8c/0x1c4 [ 5135.328067] blk_complete_reqs+0x5c/0x74 [ 5135.331976] blk_done_softirq+0x38/0x4c [ 5135.335799] __do_softirq+0x168/0x418 [ 5135.339448] __irq_exit_rcu+0x120/0x170 [ 5135.343271] irq_exit_rcu+0x18/0x24 [ 5135.346746] el1_interrupt+0x38/0x70 [ 5135.350309] el1h_64_irq_handler+0x18/0x24 [ 5135.354392] el1h_64_irq+0x68/0x6c [ 5135.357781] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5135.362558] __wake_up_common_lock+0x84/0xac [ 5135.366815] __wake_up+0x20/0x30 [ 5135.370031] __sbq_wake_up+0x13c/0x1f0 [ 5135.373767] sbitmap_queue_clear+0x74/0xc0 [ 5135.377850] blk_mq_put_tag+0x44/0x70 [ 5135.381500] __blk_mq_free_request+0x88/0xf0 [ 5135.385756] blk_mq_free_request+0xc0/0x160 [ 5135.389926] bfq_insert_request.isra.0+0x204/0x270 [ 5135.394704] bfq_insert_requests+0x70/0x90 [ 5135.398788] blk_mq_sched_insert_requests+0xac/0x250 [ 5135.403740] blk_mq_flush_plug_list+0x110/0x2cc [ 5135.408258] __blk_flush_plug+0x100/0x140 [ 5135.412254] blk_finish_plug+0x40/0x25c [ 5135.416077] __iomap_dio_rw+0x1b4/0x500 [ 5135.419900] iomap_dio_rw+0x18/0x50 [ 5135.423375] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5135.428726] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5135.433468] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5135.438512] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5135.443642] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5135.448772] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5135.454158] process_one_work+0x1ec/0x470 [ 5135.458155] worker_thread+0x74/0x410 [ 5135.461804] kthread+0xec/0x100 [ 5135.464932] ret_from_fork+0x10/0x20 [ 5139.190755] watchdog: BUG: soft lockup - CPU#68 stuck for 4392s! [swapper/68:0] [ 5139.198050] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5139.198067] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5139.285855] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5139.293671] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5139.302962] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5139.309910] pc : __sbq_wake_up+0x4/0x1f0 [ 5139.313819] lr : sbitmap_queue_clear+0x74/0xc0 [ 5139.318249] sp : ffff800008843900 [ 5139.321551] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5139.328673] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5139.335795] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5139.342918] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5139.350040] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5139.357162] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5139.364284] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5139.371406] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5139.378528] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5139.385650] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 5139.392772] Call trace: [ 5139.395204] __sbq_wake_up+0x4/0x1f0 [ 5139.398767] blk_mq_put_tag+0x44/0x70 [ 5139.402416] __blk_mq_free_request+0x88/0xf0 [ 5139.406673] blk_mq_free_request+0xc0/0x160 [ 5139.410843] __blk_mq_end_request+0x78/0x150 [ 5139.415100] scsi_end_request+0x114/0x260 [ 5139.419096] scsi_io_completion+0x64/0x174 [ 5139.423179] scsi_finish_command+0xe0/0x120 [ 5139.427349] scsi_complete+0x8c/0x1c4 [ 5139.430998] blk_complete_reqs+0x5c/0x74 [ 5139.434908] blk_done_softirq+0x38/0x4c [ 5139.438731] __do_softirq+0x168/0x418 [ 5139.442380] __irq_exit_rcu+0x120/0x170 [ 5139.446202] irq_exit_rcu+0x18/0x24 [ 5139.449678] el1_interrupt+0x38/0x70 [ 5139.453240] el1h_64_irq_handler+0x18/0x24 [ 5139.457323] el1h_64_irq+0x68/0x6c [ 5139.460712] cpuidle_enter_state+0xc4/0x4e0 [ 5139.464882] cpuidle_enter+0x40/0x60 [ 5139.468445] cpuidle_idle_call+0x118/0x1b0 [ 5139.472528] do_idle+0xac/0x100 [ 5139.475657] cpu_startup_entry+0x30/0x34 [ 5139.479567] secondary_start_kernel+0xd8/0xf0 [ 5139.483912] __secondary_switched+0xb0/0xb4 [ 5159.080621] watchdog: BUG: soft lockup - CPU#8 stuck for 4410s! [kworker/8:12:19399] [ 5159.088350] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5159.088368] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5159.176156] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5159.184406] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5159.193697] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5159.199864] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5159.206812] pc : __sbq_wake_up+0x20/0x1f0 [ 5159.210809] lr : sbitmap_queue_clear+0x74/0xc0 [ 5159.215239] sp : ffff80000d79b280 [ 5159.218540] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5159.225663] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5159.232784] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5159.239907] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5159.247029] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5159.254151] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5159.261273] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5159.268395] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5159.275516] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5159.282638] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5159.289760] Call trace: [ 5159.292193] __sbq_wake_up+0x20/0x1f0 [ 5159.295842] sbitmap_queue_clear+0x74/0xc0 [ 5159.299926] blk_mq_put_tag+0x44/0x70 [ 5159.303576] __blk_mq_free_request+0x88/0xf0 [ 5159.307832] blk_mq_free_request+0xc0/0x160 [ 5159.312002] __blk_mq_end_request+0x78/0x150 [ 5159.316260] scsi_end_request+0x114/0x260 [ 5159.320256] scsi_io_completion+0x64/0x174 [ 5159.324339] scsi_finish_command+0xe0/0x120 [ 5159.328509] scsi_complete+0x8c/0x1c4 [ 5159.332158] blk_complete_reqs+0x5c/0x74 [ 5159.336068] blk_done_softirq+0x38/0x4c [ 5159.339891] __do_softirq+0x168/0x418 [ 5159.343540] __irq_exit_rcu+0x120/0x170 [ 5159.347362] irq_exit_rcu+0x18/0x24 [ 5159.350838] el1_interrupt+0x38/0x70 [ 5159.354400] el1h_64_irq_handler+0x18/0x24 [ 5159.358484] el1h_64_irq+0x68/0x6c [ 5159.361874] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5159.366651] __wake_up_common_lock+0x84/0xac [ 5159.370908] __wake_up+0x20/0x30 [ 5159.374124] __sbq_wake_up+0x13c/0x1f0 [ 5159.377860] sbitmap_queue_clear+0x74/0xc0 [ 5159.381943] blk_mq_put_tag+0x44/0x70 [ 5159.385593] __blk_mq_free_request+0x88/0xf0 [ 5159.389850] blk_mq_free_request+0xc0/0x160 [ 5159.394020] bfq_insert_request.isra.0+0x204/0x270 [ 5159.398798] bfq_insert_requests+0x70/0x90 [ 5159.402881] blk_mq_sched_insert_requests+0xac/0x250 [ 5159.407833] blk_mq_flush_plug_list+0x110/0x2cc [ 5159.412351] __blk_flush_plug+0x100/0x140 [ 5159.416347] blk_finish_plug+0x40/0x25c [ 5159.420170] __iomap_dio_rw+0x1b4/0x500 [ 5159.423993] iomap_dio_rw+0x18/0x50 [ 5159.427468] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5159.432819] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5159.437561] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5159.442605] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5159.447735] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5159.452864] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5159.458251] process_one_work+0x1ec/0x470 [ 5159.462247] worker_thread+0x74/0x410 [ 5159.465896] kthread+0xec/0x100 [ 5159.469025] ret_from_fork+0x10/0x20 [ 5163.190593] watchdog: BUG: soft lockup - CPU#68 stuck for 4414s! [swapper/68:0] [ 5163.197889] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5163.197906] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5163.285693] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5163.293509] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5163.302800] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5163.309747] pc : __sbq_wake_up+0x58/0x1f0 [ 5163.313744] lr : sbitmap_queue_clear+0x74/0xc0 [ 5163.318174] sp : ffff8000088438b0 [ 5163.321476] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5163.328598] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5163.335720] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5163.342842] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5163.349964] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5163.357086] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5163.364208] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5163.371330] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5163.378451] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5163.385573] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5163.392695] Call trace: [ 5163.395128] __sbq_wake_up+0x58/0x1f0 [ 5163.398778] sbitmap_queue_clear+0x74/0xc0 [ 5163.402861] blk_mq_put_tag+0x44/0x70 [ 5163.406511] __blk_mq_free_request+0x88/0xf0 [ 5163.410768] blk_mq_free_request+0xc0/0x160 [ 5163.414940] __blk_mq_end_request+0x78/0x150 [ 5163.419196] scsi_end_request+0x114/0x260 [ 5163.423193] scsi_io_completion+0x64/0x174 [ 5163.427276] scsi_finish_command+0xe0/0x120 [ 5163.431446] scsi_complete+0x8c/0x1c4 [ 5163.435095] blk_complete_reqs+0x5c/0x74 [ 5163.439005] blk_done_softirq+0x38/0x4c [ 5163.442827] __do_softirq+0x168/0x418 [ 5163.446476] __irq_exit_rcu+0x120/0x170 [ 5163.450299] irq_exit_rcu+0x18/0x24 [ 5163.453775] el1_interrupt+0x38/0x70 [ 5163.457337] el1h_64_irq_handler+0x18/0x24 [ 5163.461420] el1h_64_irq+0x68/0x6c [ 5163.464808] cpuidle_enter_state+0xc4/0x4e0 [ 5163.468978] cpuidle_enter+0x40/0x60 [ 5163.472540] cpuidle_idle_call+0x118/0x1b0 [ 5163.476624] do_idle+0xac/0x100 [ 5163.479753] cpu_startup_entry+0x30/0x34 [ 5163.483662] secondary_start_kernel+0xd8/0xf0 [ 5163.488006] __secondary_switched+0xb0/0xb4 [ 5172.800529] rcu: INFO: rcu_preempt self-detected stall on CPU [ 5172.806263] rcu: 68-....: (1 GPs behind) idle=1abc/1/0x4000000000000004 softirq=2186/2187 fqs=235500 [ 5172.815469] (t=474793 jiffies g=36445 q=140694 ncpus=128) [ 5172.820941] Task dump for CPU 8: [ 5172.824156] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5172.834055] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5172.840223] Call trace: [ 5172.842656] __switch_to+0xd4/0x100 [ 5172.846133] 0xffff6c0200000080 [ 5172.849262] Task dump for CPU 68: [ 5172.852563] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5172.862463] Call trace: [ 5172.864896] dump_backtrace+0xe8/0x130 [ 5172.868632] show_stack+0x20/0x60 [ 5172.871934] sched_show_task+0x160/0x190 [ 5172.875843] dump_cpu_task+0x4c/0x78 [ 5172.879406] rcu_dump_cpu_stacks+0xc4/0x150 [ 5172.883576] print_cpu_stall+0x180/0x210 [ 5172.887486] check_cpu_stall+0x20c/0x2b4 [ 5172.891395] rcu_pending+0x4c/0x190 [ 5172.894871] rcu_sched_clock_irq+0x80/0x204 [ 5172.899041] update_process_times+0x7c/0xc0 [ 5172.903211] tick_sched_handle+0x38/0x74 [ 5172.907121] tick_sched_timer+0x60/0xc0 [ 5172.910944] __hrtimer_run_queues+0x170/0x300 [ 5172.915288] hrtimer_interrupt+0xfc/0x260 [ 5172.919284] arch_timer_handler_phys+0x34/0x50 [ 5172.923714] handle_percpu_devid_irq+0xac/0x240 [ 5172.928232] generic_handle_domain_irq+0x34/0x50 [ 5172.932836] gic_handle_irq+0x5c/0xe0 [ 5172.936485] call_on_irq_stack+0x2c/0x38 [ 5172.940395] do_interrupt_handler+0xdc/0xe0 [ 5172.944565] el1_interrupt+0x34/0x70 [ 5172.948128] el1h_64_irq_handler+0x18/0x24 [ 5172.952212] el1h_64_irq+0x68/0x6c [ 5172.955601] __sbq_wake_up+0x8/0x1f0 [ 5172.959163] blk_mq_put_tag+0x44/0x70 [ 5172.962813] __blk_mq_free_request+0x88/0xf0 [ 5172.967070] blk_mq_free_request+0xc0/0x160 [ 5172.971240] __blk_mq_end_request+0x78/0x150 [ 5172.975497] scsi_end_request+0x114/0x260 [ 5172.979493] scsi_io_completion+0x64/0x174 [ 5172.983576] scsi_finish_command+0xe0/0x120 [ 5172.987746] scsi_complete+0x8c/0x1c4 [ 5172.991396] blk_complete_reqs+0x5c/0x74 [ 5172.995305] blk_done_softirq+0x38/0x4c [ 5172.999128] __do_softirq+0x168/0x418 [ 5173.002778] __irq_exit_rcu+0x120/0x170 [ 5173.006601] irq_exit_rcu+0x18/0x24 [ 5173.010076] el1_interrupt+0x38/0x70 [ 5173.013639] el1h_64_irq_handler+0x18/0x24 [ 5173.017723] el1h_64_irq+0x68/0x6c [ 5173.021111] cpuidle_enter_state+0xc4/0x4e0 [ 5173.025282] cpuidle_enter+0x40/0x60 [ 5173.028844] cpuidle_idle_call+0x118/0x1b0 [ 5173.032928] do_idle+0xac/0x100 [ 5173.036056] cpu_startup_entry+0x30/0x34 [ 5173.039965] secondary_start_kernel+0xd8/0xf0 [ 5173.044309] __secondary_switched+0xb0/0xb4 [ 5183.080459] watchdog: BUG: soft lockup - CPU#8 stuck for 4433s! [kworker/8:12:19399] [ 5183.088188] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5183.088205] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5183.175992] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5183.184242] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5183.193533] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5183.199700] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5183.206648] pc : __sbq_wake_up+0x7c/0x1f0 [ 5183.210645] lr : sbitmap_queue_clear+0x74/0xc0 [ 5183.215075] sp : ffff80000d79b2d0 [ 5183.218376] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5183.225499] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5183.232621] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5183.239743] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5183.246865] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5183.253988] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5183.261110] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5183.268232] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5183.275354] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5183.282475] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5183.289598] Call trace: [ 5183.292031] __sbq_wake_up+0x7c/0x1f0 [ 5183.295680] blk_mq_put_tag+0x44/0x70 [ 5183.299330] __blk_mq_free_request+0x88/0xf0 [ 5183.303587] blk_mq_free_request+0xc0/0x160 [ 5183.307756] __blk_mq_end_request+0x78/0x150 [ 5183.312014] scsi_end_request+0x114/0x260 [ 5183.316010] scsi_io_completion+0x64/0x174 [ 5183.320093] scsi_finish_command+0xe0/0x120 [ 5183.324264] scsi_complete+0x8c/0x1c4 [ 5183.327913] blk_complete_reqs+0x5c/0x74 [ 5183.331822] blk_done_softirq+0x38/0x4c [ 5183.335645] __do_softirq+0x168/0x418 [ 5183.339294] __irq_exit_rcu+0x120/0x170 [ 5183.343117] irq_exit_rcu+0x18/0x24 [ 5183.346592] el1_interrupt+0x38/0x70 [ 5183.350154] el1h_64_irq_handler+0x18/0x24 [ 5183.354237] el1h_64_irq+0x68/0x6c [ 5183.357626] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5183.362404] __wake_up_common_lock+0x84/0xac [ 5183.366661] __wake_up+0x20/0x30 [ 5183.369876] __sbq_wake_up+0x13c/0x1f0 [ 5183.373612] sbitmap_queue_clear+0x74/0xc0 [ 5183.377695] blk_mq_put_tag+0x44/0x70 [ 5183.381345] __blk_mq_free_request+0x88/0xf0 [ 5183.385602] blk_mq_free_request+0xc0/0x160 [ 5183.389772] bfq_insert_request.isra.0+0x204/0x270 [ 5183.394550] bfq_insert_requests+0x70/0x90 [ 5183.398633] blk_mq_sched_insert_requests+0xac/0x250 [ 5183.403585] blk_mq_flush_plug_list+0x110/0x2cc [ 5183.408103] __blk_flush_plug+0x100/0x140 [ 5183.412099] blk_finish_plug+0x40/0x25c [ 5183.415922] __iomap_dio_rw+0x1b4/0x500 [ 5183.419744] iomap_dio_rw+0x18/0x50 [ 5183.423220] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5183.428570] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5183.433313] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5183.438357] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5183.443487] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5183.448616] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5183.454003] process_one_work+0x1ec/0x470 [ 5183.457999] worker_thread+0x74/0x410 [ 5183.461648] kthread+0xec/0x100 [ 5183.464777] ret_from_fork+0x10/0x20 [ 5197.780369] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 474685 jiffies s: 2093 root: 0x11/. [ 5197.792108] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5197.801317] Task dump for CPU 8: [ 5197.804532] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5197.814434] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5197.820604] Call trace: [ 5197.823037] __switch_to+0xd4/0x100 [ 5197.826513] 0xffff6c0200000080 [ 5197.829642] Task dump for CPU 68: [ 5197.832945] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5197.842846] Call trace: [ 5197.845279] __switch_to+0xd4/0x100 [ 5197.848755] 0x0 [ 5199.190350] watchdog: BUG: soft lockup - CPU#68 stuck for 4448s! [swapper/68:0] [ 5199.197646] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5199.197663] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5199.285452] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5199.293268] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5199.302559] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5199.309506] pc : __sbq_wake_up+0x7c/0x1f0 [ 5199.313503] lr : sbitmap_queue_clear+0x74/0xc0 [ 5199.317934] sp : ffff800008843900 [ 5199.321235] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5199.328357] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5199.335479] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5199.342602] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5199.349723] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5199.356845] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5199.363968] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5199.371090] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5199.378212] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5199.385334] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5199.392456] Call trace: [ 5199.394889] __sbq_wake_up+0x7c/0x1f0 [ 5199.398538] blk_mq_put_tag+0x44/0x70 [ 5199.402187] __blk_mq_free_request+0x88/0xf0 [ 5199.406444] blk_mq_free_request+0xc0/0x160 [ 5199.410614] __blk_mq_end_request+0x78/0x150 [ 5199.414871] scsi_end_request+0x114/0x260 [ 5199.418867] scsi_io_completion+0x64/0x174 [ 5199.422950] scsi_finish_command+0xe0/0x120 [ 5199.427121] scsi_complete+0x8c/0x1c4 [ 5199.430771] blk_complete_reqs+0x5c/0x74 [ 5199.434681] blk_done_softirq+0x38/0x4c [ 5199.438503] __do_softirq+0x168/0x418 [ 5199.442152] __irq_exit_rcu+0x120/0x170 [ 5199.445975] irq_exit_rcu+0x18/0x24 [ 5199.449450] el1_interrupt+0x38/0x70 [ 5199.453012] el1h_64_irq_handler+0x18/0x24 [ 5199.457095] el1h_64_irq+0x68/0x6c [ 5199.460484] cpuidle_enter_state+0xc4/0x4e0 [ 5199.464653] cpuidle_enter+0x40/0x60 [ 5199.468216] cpuidle_idle_call+0x118/0x1b0 [ 5199.472299] do_idle+0xac/0x100 [ 5199.475428] cpu_startup_entry+0x30/0x34 [ 5199.479337] secondary_start_kernel+0xd8/0xf0 [ 5199.483681] __secondary_switched+0xb0/0xb4 [ 5207.080297] watchdog: BUG: soft lockup - CPU#8 stuck for 4455s! [kworker/8:12:19399] [ 5207.088027] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5207.088044] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5207.175832] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5207.184081] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5207.193372] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5207.199540] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5207.206487] pc : __sbq_wake_up+0x58/0x1f0 [ 5207.210484] lr : sbitmap_queue_clear+0x74/0xc0 [ 5207.214915] sp : ffff80000d79b280 [ 5207.218216] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5207.225338] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5207.232460] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5207.239582] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5207.246704] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5207.253826] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5207.260948] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5207.268070] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5207.275192] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5207.282314] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5207.289436] Call trace: [ 5207.291869] __sbq_wake_up+0x58/0x1f0 [ 5207.295518] sbitmap_queue_clear+0x74/0xc0 [ 5207.299601] blk_mq_put_tag+0x44/0x70 [ 5207.303251] __blk_mq_free_request+0x88/0xf0 [ 5207.307507] blk_mq_free_request+0xc0/0x160 [ 5207.311677] __blk_mq_end_request+0x78/0x150 [ 5207.315935] scsi_end_request+0x114/0x260 [ 5207.319931] scsi_io_completion+0x64/0x174 [ 5207.324014] scsi_finish_command+0xe0/0x120 [ 5207.328184] scsi_complete+0x8c/0x1c4 [ 5207.331835] blk_complete_reqs+0x5c/0x74 [ 5207.335744] blk_done_softirq+0x38/0x4c [ 5207.339567] __do_softirq+0x168/0x418 [ 5207.343216] __irq_exit_rcu+0x120/0x170 [ 5207.347039] irq_exit_rcu+0x18/0x24 [ 5207.350514] el1_interrupt+0x38/0x70 [ 5207.354076] el1h_64_irq_handler+0x18/0x24 [ 5207.358159] el1h_64_irq+0x68/0x6c [ 5207.361548] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5207.366325] __wake_up_common_lock+0x84/0xac [ 5207.370582] __wake_up+0x20/0x30 [ 5207.373798] __sbq_wake_up+0x13c/0x1f0 [ 5207.377534] sbitmap_queue_clear+0x74/0xc0 [ 5207.381617] blk_mq_put_tag+0x44/0x70 [ 5207.385267] __blk_mq_free_request+0x88/0xf0 [ 5207.389524] blk_mq_free_request+0xc0/0x160 [ 5207.393694] bfq_insert_request.isra.0+0x204/0x270 [ 5207.398472] bfq_insert_requests+0x70/0x90 [ 5207.402555] blk_mq_sched_insert_requests+0xac/0x250 [ 5207.407507] blk_mq_flush_plug_list+0x110/0x2cc [ 5207.412025] __blk_flush_plug+0x100/0x140 [ 5207.416021] blk_finish_plug+0x40/0x25c [ 5207.419844] __iomap_dio_rw+0x1b4/0x500 [ 5207.423666] iomap_dio_rw+0x18/0x50 [ 5207.427142] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5207.432492] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5207.437235] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5207.442278] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5207.447408] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5207.452538] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5207.457924] process_one_work+0x1ec/0x470 [ 5207.461920] worker_thread+0x74/0x410 [ 5207.465569] kthread+0xec/0x100 [ 5207.468698] ret_from_fork+0x10/0x20 [ 5223.190188] watchdog: BUG: soft lockup - CPU#68 stuck for 4470s! [swapper/68:0] [ 5223.197483] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5223.197501] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5223.285289] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5223.293105] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5223.302396] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5223.309343] pc : __sbq_wake_up+0x7c/0x1f0 [ 5223.313340] lr : sbitmap_queue_clear+0x74/0xc0 [ 5223.317770] sp : ffff800008843900 [ 5223.321072] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5223.328194] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5223.335315] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5223.342437] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5223.349559] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5223.356681] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5223.363803] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5223.370924] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5223.378046] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5223.385168] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5223.392290] Call trace: [ 5223.394723] __sbq_wake_up+0x7c/0x1f0 [ 5223.398372] blk_mq_put_tag+0x44/0x70 [ 5223.402021] __blk_mq_free_request+0x88/0xf0 [ 5223.406278] blk_mq_free_request+0xc0/0x160 [ 5223.410449] __blk_mq_end_request+0x78/0x150 [ 5223.414706] scsi_end_request+0x114/0x260 [ 5223.418702] scsi_io_completion+0x64/0x174 [ 5223.422786] scsi_finish_command+0xe0/0x120 [ 5223.426956] scsi_complete+0x8c/0x1c4 [ 5223.430605] blk_complete_reqs+0x5c/0x74 [ 5223.434515] blk_done_softirq+0x38/0x4c [ 5223.438337] __do_softirq+0x168/0x418 [ 5223.441986] __irq_exit_rcu+0x120/0x170 [ 5223.445809] irq_exit_rcu+0x18/0x24 [ 5223.449285] el1_interrupt+0x38/0x70 [ 5223.452847] el1h_64_irq_handler+0x18/0x24 [ 5223.456930] el1h_64_irq+0x68/0x6c [ 5223.460319] cpuidle_enter_state+0xc4/0x4e0 [ 5223.464489] cpuidle_enter+0x40/0x60 [ 5223.468051] cpuidle_idle_call+0x118/0x1b0 [ 5223.472134] do_idle+0xac/0x100 [ 5223.475263] cpu_startup_entry+0x30/0x34 [ 5223.479172] secondary_start_kernel+0xd8/0xf0 [ 5223.483516] __secondary_switched+0xb0/0xb4 [ 5231.080135] watchdog: BUG: soft lockup - CPU#8 stuck for 4477s! [kworker/8:12:19399] [ 5231.087864] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5231.087881] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5231.175668] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5231.183918] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5231.193209] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5231.199376] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5231.206324] pc : __sbq_wake_up+0x84/0x1f0 [ 5231.210321] lr : sbitmap_queue_clear+0x74/0xc0 [ 5231.214752] sp : ffff80000d79b280 [ 5231.218053] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5231.225175] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5231.232297] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5231.239419] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5231.246541] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5231.253664] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5231.260786] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5231.267908] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5231.275030] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5231.282152] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5231.289274] Call trace: [ 5231.291707] __sbq_wake_up+0x84/0x1f0 [ 5231.295357] sbitmap_queue_clear+0x74/0xc0 [ 5231.299441] blk_mq_put_tag+0x44/0x70 [ 5231.303090] __blk_mq_free_request+0x88/0xf0 [ 5231.307347] blk_mq_free_request+0xc0/0x160 [ 5231.311517] __blk_mq_end_request+0x78/0x150 [ 5231.315774] scsi_end_request+0x114/0x260 [ 5231.319770] scsi_io_completion+0x64/0x174 [ 5231.323853] scsi_finish_command+0xe0/0x120 [ 5231.328024] scsi_complete+0x8c/0x1c4 [ 5231.331673] blk_complete_reqs+0x5c/0x74 [ 5231.335582] blk_done_softirq+0x38/0x4c [ 5231.339405] __do_softirq+0x168/0x418 [ 5231.343054] __irq_exit_rcu+0x120/0x170 [ 5231.346877] irq_exit_rcu+0x18/0x24 [ 5231.350352] el1_interrupt+0x38/0x70 [ 5231.353915] el1h_64_irq_handler+0x18/0x24 [ 5231.357998] el1h_64_irq+0x68/0x6c [ 5231.361386] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5231.366164] __wake_up_common_lock+0x84/0xac [ 5231.370421] __wake_up+0x20/0x30 [ 5231.373637] __sbq_wake_up+0x13c/0x1f0 [ 5231.377373] sbitmap_queue_clear+0x74/0xc0 [ 5231.381456] blk_mq_put_tag+0x44/0x70 [ 5231.385106] __blk_mq_free_request+0x88/0xf0 [ 5231.389363] blk_mq_free_request+0xc0/0x160 [ 5231.393533] bfq_insert_request.isra.0+0x204/0x270 [ 5231.398311] bfq_insert_requests+0x70/0x90 [ 5231.402395] blk_mq_sched_insert_requests+0xac/0x250 [ 5231.407346] blk_mq_flush_plug_list+0x110/0x2cc [ 5231.411864] __blk_flush_plug+0x100/0x140 [ 5231.415860] blk_finish_plug+0x40/0x25c [ 5231.419683] __iomap_dio_rw+0x1b4/0x500 [ 5231.423505] iomap_dio_rw+0x18/0x50 [ 5231.426981] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5231.432332] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5231.437074] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5231.442118] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5231.447248] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5231.452378] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5231.457764] process_one_work+0x1ec/0x470 [ 5231.461760] worker_thread+0x74/0x410 [ 5231.465409] kthread+0xec/0x100 [ 5231.468538] ret_from_fork+0x10/0x20 [ 5247.190026] watchdog: BUG: soft lockup - CPU#68 stuck for 4492s! [swapper/68:0] [ 5247.197322] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5247.197339] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5247.285127] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5247.292942] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5247.302233] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5247.309181] pc : __sbq_wake_up+0x58/0x1f0 [ 5247.313178] lr : sbitmap_queue_clear+0x74/0xc0 [ 5247.317608] sp : ffff8000088438b0 [ 5247.320909] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5247.328031] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5247.335153] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5247.342275] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5247.349397] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5247.356519] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5247.363641] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5247.370763] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5247.377885] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5247.385006] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5247.392128] Call trace: [ 5247.394561] __sbq_wake_up+0x58/0x1f0 [ 5247.398211] sbitmap_queue_clear+0x74/0xc0 [ 5247.402294] blk_mq_put_tag+0x44/0x70 [ 5247.405944] __blk_mq_free_request+0x88/0xf0 [ 5247.410201] blk_mq_free_request+0xc0/0x160 [ 5247.414371] __blk_mq_end_request+0x78/0x150 [ 5247.418628] scsi_end_request+0x114/0x260 [ 5247.422624] scsi_io_completion+0x64/0x174 [ 5247.426707] scsi_finish_command+0xe0/0x120 [ 5247.430878] scsi_complete+0x8c/0x1c4 [ 5247.434527] blk_complete_reqs+0x5c/0x74 [ 5247.438436] blk_done_softirq+0x38/0x4c [ 5247.442259] __do_softirq+0x168/0x418 [ 5247.445907] __irq_exit_rcu+0x120/0x170 [ 5247.449730] irq_exit_rcu+0x18/0x24 [ 5247.453205] el1_interrupt+0x38/0x70 [ 5247.456768] el1h_64_irq_handler+0x18/0x24 [ 5247.460851] el1h_64_irq+0x68/0x6c [ 5247.464240] cpuidle_enter_state+0xc4/0x4e0 [ 5247.468409] cpuidle_enter+0x40/0x60 [ 5247.471972] cpuidle_idle_call+0x118/0x1b0 [ 5247.476055] do_idle+0xac/0x100 [ 5247.479184] cpu_startup_entry+0x30/0x34 [ 5247.483094] secondary_start_kernel+0xd8/0xf0 [ 5247.487438] __secondary_switched+0xb0/0xb4 [ 5255.079973] watchdog: BUG: soft lockup - CPU#8 stuck for 4500s! [kworker/8:12:19399] [ 5255.087702] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5255.087719] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5255.175506] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5255.183756] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5255.193047] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5255.199215] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5255.206162] pc : __sbq_wake_up+0x20/0x1f0 [ 5255.210159] lr : sbitmap_queue_clear+0x74/0xc0 [ 5255.214590] sp : ffff80000d79b280 [ 5255.217893] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5255.225015] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5255.232137] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5255.239259] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5255.246381] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5255.253503] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5255.260626] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5255.267747] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5255.274870] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5255.281992] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5255.289114] Call trace: [ 5255.291547] __sbq_wake_up+0x20/0x1f0 [ 5255.295196] sbitmap_queue_clear+0x74/0xc0 [ 5255.299280] blk_mq_put_tag+0x44/0x70 [ 5255.302930] __blk_mq_free_request+0x88/0xf0 [ 5255.307186] blk_mq_free_request+0xc0/0x160 [ 5255.311357] __blk_mq_end_request+0x78/0x150 [ 5255.315613] scsi_end_request+0x114/0x260 [ 5255.319610] scsi_io_completion+0x64/0x174 [ 5255.323693] scsi_finish_command+0xe0/0x120 [ 5255.327863] scsi_complete+0x8c/0x1c4 [ 5255.331512] blk_complete_reqs+0x5c/0x74 [ 5255.335422] blk_done_softirq+0x38/0x4c [ 5255.339245] __do_softirq+0x168/0x418 [ 5255.342894] __irq_exit_rcu+0x120/0x170 [ 5255.346716] irq_exit_rcu+0x18/0x24 [ 5255.350192] el1_interrupt+0x38/0x70 [ 5255.353755] el1h_64_irq_handler+0x18/0x24 [ 5255.357838] el1h_64_irq+0x68/0x6c [ 5255.361226] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5255.366004] __wake_up_common_lock+0x84/0xac [ 5255.370261] __wake_up+0x20/0x30 [ 5255.373477] __sbq_wake_up+0x13c/0x1f0 [ 5255.377213] sbitmap_queue_clear+0x74/0xc0 [ 5255.381296] blk_mq_put_tag+0x44/0x70 [ 5255.384946] __blk_mq_free_request+0x88/0xf0 [ 5255.389203] blk_mq_free_request+0xc0/0x160 [ 5255.393373] bfq_insert_request.isra.0+0x204/0x270 [ 5255.398151] bfq_insert_requests+0x70/0x90 [ 5255.402234] blk_mq_sched_insert_requests+0xac/0x250 [ 5255.407187] blk_mq_flush_plug_list+0x110/0x2cc [ 5255.411704] __blk_flush_plug+0x100/0x140 [ 5255.415701] blk_finish_plug+0x40/0x25c [ 5255.419524] __iomap_dio_rw+0x1b4/0x500 [ 5255.423347] iomap_dio_rw+0x18/0x50 [ 5255.426822] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5255.432174] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5255.436916] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5255.441960] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5255.447090] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5255.452219] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5255.457606] process_one_work+0x1ec/0x470 [ 5255.461602] worker_thread+0x74/0x410 [ 5255.465251] kthread+0xec/0x100 [ 5255.468380] ret_from_fork+0x10/0x20 [ 5264.339921] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 481341 jiffies s: 2093 root: 0x11/. [ 5264.351660] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5264.360869] Task dump for CPU 8: [ 5264.364084] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5264.373985] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5264.380155] Call trace: [ 5264.382588] __switch_to+0xd4/0x100 [ 5264.386065] 0xffff6c0200000080 [ 5264.389193] Task dump for CPU 68: [ 5264.392496] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5264.402398] Call trace: [ 5264.404831] __switch_to+0xd4/0x100 [ 5264.408307] 0x0 [ 5271.189864] watchdog: BUG: soft lockup - CPU#68 stuck for 4515s! [swapper/68:0] [ 5271.197160] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5271.197177] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5271.284965] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5271.292781] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5271.302072] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5271.309019] pc : __sbq_wake_up+0x58/0x1f0 [ 5271.313016] lr : sbitmap_queue_clear+0x74/0xc0 [ 5271.317447] sp : ffff8000088438b0 [ 5271.320748] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5271.327871] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5271.334993] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5271.342114] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5271.349236] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5271.356358] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5271.363480] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5271.370602] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5271.377724] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5271.384848] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5271.391969] Call trace: [ 5271.394403] __sbq_wake_up+0x58/0x1f0 [ 5271.398052] sbitmap_queue_clear+0x74/0xc0 [ 5271.402135] blk_mq_put_tag+0x44/0x70 [ 5271.405785] __blk_mq_free_request+0x88/0xf0 [ 5271.410042] blk_mq_free_request+0xc0/0x160 [ 5271.414211] __blk_mq_end_request+0x78/0x150 [ 5271.418468] scsi_end_request+0x114/0x260 [ 5271.422465] scsi_io_completion+0x64/0x174 [ 5271.426548] scsi_finish_command+0xe0/0x120 [ 5271.430719] scsi_complete+0x8c/0x1c4 [ 5271.434368] blk_complete_reqs+0x5c/0x74 [ 5271.438277] blk_done_softirq+0x38/0x4c [ 5271.442100] __do_softirq+0x168/0x418 [ 5271.445748] __irq_exit_rcu+0x120/0x170 [ 5271.449571] irq_exit_rcu+0x18/0x24 [ 5271.453047] el1_interrupt+0x38/0x70 [ 5271.456609] el1h_64_irq_handler+0x18/0x24 [ 5271.460692] el1h_64_irq+0x68/0x6c [ 5271.464080] cpuidle_enter_state+0xc4/0x4e0 [ 5271.468250] cpuidle_enter+0x40/0x60 [ 5271.471813] cpuidle_idle_call+0x118/0x1b0 [ 5271.475896] do_idle+0xac/0x100 [ 5271.479026] cpu_startup_entry+0x30/0x34 [ 5271.482936] secondary_start_kernel+0xd8/0xf0 [ 5271.487280] __secondary_switched+0xb0/0xb4 [ 5279.079816] watchdog: BUG: soft lockup - CPU#8 stuck for 4522s! [kworker/8:12:19399] [ 5279.087545] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5279.087563] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5279.175351] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5279.183600] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5279.192892] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5279.199060] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5279.206008] pc : __sbq_wake_up+0x58/0x1f0 [ 5279.210005] lr : sbitmap_queue_clear+0x74/0xc0 [ 5279.214435] sp : ffff80000d79b280 [ 5279.217737] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5279.224859] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5279.231982] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5279.239104] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5279.246226] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5279.253348] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5279.260471] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5279.267593] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5279.274715] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5279.281837] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5279.288959] Call trace: [ 5279.291392] __sbq_wake_up+0x58/0x1f0 [ 5279.295041] sbitmap_queue_clear+0x74/0xc0 [ 5279.299125] blk_mq_put_tag+0x44/0x70 [ 5279.302774] __blk_mq_free_request+0x88/0xf0 [ 5279.307031] blk_mq_free_request+0xc0/0x160 [ 5279.311201] __blk_mq_end_request+0x78/0x150 [ 5279.315458] scsi_end_request+0x114/0x260 [ 5279.319454] scsi_io_completion+0x64/0x174 [ 5279.323537] scsi_finish_command+0xe0/0x120 [ 5279.327708] scsi_complete+0x8c/0x1c4 [ 5279.331357] blk_complete_reqs+0x5c/0x74 [ 5279.335266] blk_done_softirq+0x38/0x4c [ 5279.339089] __do_softirq+0x168/0x418 [ 5279.342740] __irq_exit_rcu+0x120/0x170 [ 5279.346562] irq_exit_rcu+0x18/0x24 [ 5279.350038] el1_interrupt+0x38/0x70 [ 5279.353600] el1h_64_irq_handler+0x18/0x24 [ 5279.357683] el1h_64_irq+0x68/0x6c [ 5279.361072] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5279.365849] __wake_up_common_lock+0x84/0xac [ 5279.370107] __wake_up+0x20/0x30 [ 5279.373322] __sbq_wake_up+0x13c/0x1f0 [ 5279.377059] sbitmap_queue_clear+0x74/0xc0 [ 5279.381142] blk_mq_put_tag+0x44/0x70 [ 5279.384792] __blk_mq_free_request+0x88/0xf0 [ 5279.389050] blk_mq_free_request+0xc0/0x160 [ 5279.393220] bfq_insert_request.isra.0+0x204/0x270 [ 5279.397998] bfq_insert_requests+0x70/0x90 [ 5279.402082] blk_mq_sched_insert_requests+0xac/0x250 [ 5279.407034] blk_mq_flush_plug_list+0x110/0x2cc [ 5279.411552] __blk_flush_plug+0x100/0x140 [ 5279.415548] blk_finish_plug+0x40/0x25c [ 5279.419371] __iomap_dio_rw+0x1b4/0x500 [ 5279.423193] iomap_dio_rw+0x18/0x50 [ 5279.426670] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5279.432020] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5279.436763] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5279.441807] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5279.446937] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5279.452066] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5279.457453] process_one_work+0x1ec/0x470 [ 5279.461449] worker_thread+0x74/0x410 [ 5279.465098] kthread+0xec/0x100 [ 5279.468227] ret_from_fork+0x10/0x20 [-- MARK -- Wed Sep 28 05:25:00 2022] [ 5295.189718] watchdog: BUG: soft lockup - CPU#68 stuck for 4537s! [swapper/68:0] [ 5295.197013] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5295.197030] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5295.284818] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5295.292633] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5295.301924] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5295.308872] pc : __sbq_wake_up+0x40/0x1f0 [ 5295.312868] lr : sbitmap_queue_clear+0x74/0xc0 [ 5295.317299] sp : ffff8000088438b0 [ 5295.320600] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5295.327722] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5295.334844] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5295.341966] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5295.349088] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5295.356210] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5295.363332] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5295.370454] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5295.377576] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5295.384698] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 5295.391820] Call trace: [ 5295.394252] __sbq_wake_up+0x40/0x1f0 [ 5295.397902] sbitmap_queue_clear+0x74/0xc0 [ 5295.401986] blk_mq_put_tag+0x44/0x70 [ 5295.405635] __blk_mq_free_request+0x88/0xf0 [ 5295.409892] blk_mq_free_request+0xc0/0x160 [ 5295.414062] __blk_mq_end_request+0x78/0x150 [ 5295.418319] scsi_end_request+0x114/0x260 [ 5295.422315] scsi_io_completion+0x64/0x174 [ 5295.426398] scsi_finish_command+0xe0/0x120 [ 5295.430568] scsi_complete+0x8c/0x1c4 [ 5295.434217] blk_complete_reqs+0x5c/0x74 [ 5295.438127] blk_done_softirq+0x38/0x4c [ 5295.441950] __do_softirq+0x168/0x418 [ 5295.445599] __irq_exit_rcu+0x120/0x170 [ 5295.449421] irq_exit_rcu+0x18/0x24 [ 5295.452898] el1_interrupt+0x38/0x70 [ 5295.456460] el1h_64_irq_handler+0x18/0x24 [ 5295.460543] el1h_64_irq+0x68/0x6c [ 5295.463932] cpuidle_enter_state+0xc4/0x4e0 [ 5295.468102] cpuidle_enter+0x40/0x60 [ 5295.471664] cpuidle_idle_call+0x118/0x1b0 [ 5295.475748] do_idle+0xac/0x100 [ 5295.478877] cpu_startup_entry+0x30/0x34 [ 5295.482786] secondary_start_kernel+0xd8/0xf0 [ 5295.487130] __secondary_switched+0xb0/0xb4 [ 5303.079670] watchdog: BUG: soft lockup - CPU#8 stuck for 4544s! [kworker/8:12:19399] [ 5303.087399] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5303.087416] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5303.175204] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5303.183454] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5303.192745] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5303.198912] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5303.205860] pc : __sbq_wake_up+0x20/0x1f0 [ 5303.209856] lr : sbitmap_queue_clear+0x74/0xc0 [ 5303.214287] sp : ffff80000d79b280 [ 5303.217589] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5303.224712] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5303.231835] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5303.238956] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5303.246079] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5303.253201] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5303.260323] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5303.267445] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5303.274567] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5303.281689] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5303.288812] Call trace: [ 5303.291245] __sbq_wake_up+0x20/0x1f0 [ 5303.294894] sbitmap_queue_clear+0x74/0xc0 [ 5303.298978] blk_mq_put_tag+0x44/0x70 [ 5303.302628] __blk_mq_free_request+0x88/0xf0 [ 5303.306885] blk_mq_free_request+0xc0/0x160 [ 5303.311055] __blk_mq_end_request+0x78/0x150 [ 5303.315312] scsi_end_request+0x114/0x260 [ 5303.319308] scsi_io_completion+0x64/0x174 [ 5303.323391] scsi_finish_command+0xe0/0x120 [ 5303.327562] scsi_complete+0x8c/0x1c4 [ 5303.331211] blk_complete_reqs+0x5c/0x74 [ 5303.335120] blk_done_softirq+0x38/0x4c [ 5303.338943] __do_softirq+0x168/0x418 [ 5303.342591] __irq_exit_rcu+0x120/0x170 [ 5303.346415] irq_exit_rcu+0x18/0x24 [ 5303.349891] el1_interrupt+0x38/0x70 [ 5303.353453] el1h_64_irq_handler+0x18/0x24 [ 5303.357536] el1h_64_irq+0x68/0x6c [ 5303.360925] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5303.365702] __wake_up_common_lock+0x84/0xac [ 5303.369960] __wake_up+0x20/0x30 [ 5303.373176] __sbq_wake_up+0x13c/0x1f0 [ 5303.376912] sbitmap_queue_clear+0x74/0xc0 [ 5303.380995] blk_mq_put_tag+0x44/0x70 [ 5303.384644] __blk_mq_free_request+0x88/0xf0 [ 5303.388901] blk_mq_free_request+0xc0/0x160 [ 5303.393071] bfq_insert_request.isra.0+0x204/0x270 [ 5303.397849] bfq_insert_requests+0x70/0x90 [ 5303.401933] blk_mq_sched_insert_requests+0xac/0x250 [ 5303.406884] blk_mq_flush_plug_list+0x110/0x2cc [ 5303.411401] __blk_flush_plug+0x100/0x140 [ 5303.415398] blk_finish_plug+0x40/0x25c [ 5303.419220] __iomap_dio_rw+0x1b4/0x500 [ 5303.423043] iomap_dio_rw+0x18/0x50 [ 5303.426519] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5303.431869] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5303.436611] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5303.441655] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5303.446784] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5303.451914] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5303.457300] process_one_work+0x1ec/0x470 [ 5303.461297] worker_thread+0x74/0x410 [ 5303.464947] kthread+0xec/0x100 [ 5303.468076] ret_from_fork+0x10/0x20 [ 5319.189572] watchdog: BUG: soft lockup - CPU#68 stuck for 4559s! [swapper/68:0] [ 5319.196867] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5319.196884] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5319.284672] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5319.292487] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5319.301778] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5319.308726] pc : __sbq_wake_up+0x40/0x1f0 [ 5319.312724] lr : sbitmap_queue_clear+0x74/0xc0 [ 5319.317155] sp : ffff8000088438b0 [ 5319.320456] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5319.327578] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5319.334701] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5319.341823] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5319.348944] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5319.356067] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5319.363189] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5319.370311] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5319.377433] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5319.384555] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 5319.391677] Call trace: [ 5319.394110] __sbq_wake_up+0x40/0x1f0 [ 5319.397759] sbitmap_queue_clear+0x74/0xc0 [ 5319.401842] blk_mq_put_tag+0x44/0x70 [ 5319.405492] __blk_mq_free_request+0x88/0xf0 [ 5319.409749] blk_mq_free_request+0xc0/0x160 [ 5319.413919] __blk_mq_end_request+0x78/0x150 [ 5319.418175] scsi_end_request+0x114/0x260 [ 5319.422172] scsi_io_completion+0x64/0x174 [ 5319.426255] scsi_finish_command+0xe0/0x120 [ 5319.430425] scsi_complete+0x8c/0x1c4 [ 5319.434074] blk_complete_reqs+0x5c/0x74 [ 5319.437984] blk_done_softirq+0x38/0x4c [ 5319.441806] __do_softirq+0x168/0x418 [ 5319.445456] __irq_exit_rcu+0x120/0x170 [ 5319.449278] irq_exit_rcu+0x18/0x24 [ 5319.452754] el1_interrupt+0x38/0x70 [ 5319.456316] el1h_64_irq_handler+0x18/0x24 [ 5319.460399] el1h_64_irq+0x68/0x6c [ 5319.463787] cpuidle_enter_state+0xc4/0x4e0 [ 5319.467958] cpuidle_enter+0x40/0x60 [ 5319.471520] cpuidle_idle_call+0x118/0x1b0 [ 5319.475606] do_idle+0xac/0x100 [ 5319.478734] cpu_startup_entry+0x30/0x34 [ 5319.482643] secondary_start_kernel+0xd8/0xf0 [ 5319.486987] __secondary_switched+0xb0/0xb4 [ 5327.079524] watchdog: BUG: soft lockup - CPU#8 stuck for 4567s! [kworker/8:12:19399] [ 5327.087253] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5327.087270] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5327.175058] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5327.183308] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5327.192599] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5327.198766] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5327.205714] pc : __sbq_wake_up+0x58/0x1f0 [ 5327.209711] lr : sbitmap_queue_clear+0x74/0xc0 [ 5327.214142] sp : ffff80000d79b280 [ 5327.217443] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5327.224565] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5327.231687] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5327.238809] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5327.245931] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5327.253053] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5327.260175] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5327.267297] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5327.274419] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5327.281541] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5327.288663] Call trace: [ 5327.291096] __sbq_wake_up+0x58/0x1f0 [ 5327.294745] sbitmap_queue_clear+0x74/0xc0 [ 5327.298829] blk_mq_put_tag+0x44/0x70 [ 5327.302478] __blk_mq_free_request+0x88/0xf0 [ 5327.306735] blk_mq_free_request+0xc0/0x160 [ 5327.310905] __blk_mq_end_request+0x78/0x150 [ 5327.315162] scsi_end_request+0x114/0x260 [ 5327.319158] scsi_io_completion+0x64/0x174 [ 5327.323242] scsi_finish_command+0xe0/0x120 [ 5327.327412] scsi_complete+0x8c/0x1c4 [ 5327.331061] blk_complete_reqs+0x5c/0x74 [ 5327.334971] blk_done_softirq+0x38/0x4c [ 5327.338793] __do_softirq+0x168/0x418 [ 5327.342442] __irq_exit_rcu+0x120/0x170 [ 5327.346265] irq_exit_rcu+0x18/0x24 [ 5327.349740] el1_interrupt+0x38/0x70 [ 5327.353303] el1h_64_irq_handler+0x18/0x24 [ 5327.357386] el1h_64_irq+0x68/0x6c [ 5327.360774] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5327.365552] __wake_up_common_lock+0x84/0xac [ 5327.369809] __wake_up+0x20/0x30 [ 5327.373025] __sbq_wake_up+0x13c/0x1f0 [ 5327.376761] sbitmap_queue_clear+0x74/0xc0 [ 5327.380845] blk_mq_put_tag+0x44/0x70 [ 5327.384495] __blk_mq_free_request+0x88/0xf0 [ 5327.388752] blk_mq_free_request+0xc0/0x160 [ 5327.392922] bfq_insert_request.isra.0+0x204/0x270 [ 5327.397700] bfq_insert_requests+0x70/0x90 [ 5327.401784] blk_mq_sched_insert_requests+0xac/0x250 [ 5327.406736] blk_mq_flush_plug_list+0x110/0x2cc [ 5327.411254] __blk_flush_plug+0x100/0x140 [ 5327.415251] blk_finish_plug+0x40/0x25c [ 5327.419073] __iomap_dio_rw+0x1b4/0x500 [ 5327.422897] iomap_dio_rw+0x18/0x50 [ 5327.426372] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5327.431723] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5327.436466] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5327.441509] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5327.446639] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5327.451769] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5327.457155] process_one_work+0x1ec/0x470 [ 5327.461151] worker_thread+0x74/0x410 [ 5327.464800] kthread+0xec/0x100 [ 5327.467929] ret_from_fork+0x10/0x20 [ 5330.899524] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 487997 jiffies s: 2093 root: 0x11/. [ 5330.911263] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5330.920472] Task dump for CPU 8: [ 5330.923686] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5330.933588] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5330.939758] Call trace: [ 5330.942191] __switch_to+0xd4/0x100 [ 5330.945668] 0xffff6c0200000080 [ 5330.948796] Task dump for CPU 68: [ 5330.952099] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5330.962001] Call trace: [ 5330.964434] __switch_to+0xd4/0x100 [ 5330.967910] 0x0 [ 5343.189426] watchdog: BUG: soft lockup - CPU#68 stuck for 4582s! [swapper/68:0] [ 5343.196722] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5343.196739] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5343.284527] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5343.292343] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5343.301634] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5343.308582] pc : __sbq_wake_up+0x58/0x1f0 [ 5343.312578] lr : sbitmap_queue_clear+0x74/0xc0 [ 5343.317009] sp : ffff8000088438b0 [ 5343.320311] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5343.327432] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5343.334555] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5343.341677] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5343.348798] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5343.355920] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5343.363042] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5343.370164] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5343.377286] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5343.384408] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5343.391530] Call trace: [ 5343.393963] __sbq_wake_up+0x58/0x1f0 [ 5343.397612] sbitmap_queue_clear+0x74/0xc0 [ 5343.401695] blk_mq_put_tag+0x44/0x70 [ 5343.405345] __blk_mq_free_request+0x88/0xf0 [ 5343.409602] blk_mq_free_request+0xc0/0x160 [ 5343.413772] __blk_mq_end_request+0x78/0x150 [ 5343.418028] scsi_end_request+0x114/0x260 [ 5343.422025] scsi_io_completion+0x64/0x174 [ 5343.426109] scsi_finish_command+0xe0/0x120 [ 5343.430280] scsi_complete+0x8c/0x1c4 [ 5343.433929] blk_complete_reqs+0x5c/0x74 [ 5343.437838] blk_done_softirq+0x38/0x4c [ 5343.441661] __do_softirq+0x168/0x418 [ 5343.445310] __irq_exit_rcu+0x120/0x170 [ 5343.449133] irq_exit_rcu+0x18/0x24 [ 5343.452608] el1_interrupt+0x38/0x70 [ 5343.456171] el1h_64_irq_handler+0x18/0x24 [ 5343.460254] el1h_64_irq+0x68/0x6c [ 5343.463642] cpuidle_enter_state+0xc4/0x4e0 [ 5343.467813] cpuidle_enter+0x40/0x60 [ 5343.471375] cpuidle_idle_call+0x118/0x1b0 [ 5343.475459] do_idle+0xac/0x100 [ 5343.478587] cpu_startup_entry+0x30/0x34 [ 5343.482496] secondary_start_kernel+0xd8/0xf0 [ 5343.486841] __secondary_switched+0xb0/0xb4 [ 5351.079378] watchdog: BUG: soft lockup - CPU#8 stuck for 4589s! [kworker/8:12:19399] [ 5351.087107] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5351.087124] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5351.174912] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5351.183162] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5351.192453] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5351.198620] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5351.205568] pc : __sbq_wake_up+0x58/0x1f0 [ 5351.209565] lr : sbitmap_queue_clear+0x74/0xc0 [ 5351.213996] sp : ffff80000d79b280 [ 5351.217299] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5351.224421] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5351.231543] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5351.238665] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5351.245788] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5351.252909] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5351.260031] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5351.267153] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5351.274275] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5351.281398] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5351.288520] Call trace: [ 5351.290953] __sbq_wake_up+0x58/0x1f0 [ 5351.294602] sbitmap_queue_clear+0x74/0xc0 [ 5351.298686] blk_mq_put_tag+0x44/0x70 [ 5351.302336] __blk_mq_free_request+0x88/0xf0 [ 5351.306593] blk_mq_free_request+0xc0/0x160 [ 5351.310764] __blk_mq_end_request+0x78/0x150 [ 5351.315021] scsi_end_request+0x114/0x260 [ 5351.319017] scsi_io_completion+0x64/0x174 [ 5351.323101] scsi_finish_command+0xe0/0x120 [ 5351.327271] scsi_complete+0x8c/0x1c4 [ 5351.330920] blk_complete_reqs+0x5c/0x74 [ 5351.334830] blk_done_softirq+0x38/0x4c [ 5351.338652] __do_softirq+0x168/0x418 [ 5351.342301] __irq_exit_rcu+0x120/0x170 [ 5351.346123] irq_exit_rcu+0x18/0x24 [ 5351.349599] el1_interrupt+0x38/0x70 [ 5351.353161] el1h_64_irq_handler+0x18/0x24 [ 5351.357244] el1h_64_irq+0x68/0x6c [ 5351.360633] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5351.365410] __wake_up_common_lock+0x84/0xac [ 5351.369668] __wake_up+0x20/0x30 [ 5351.372883] __sbq_wake_up+0x13c/0x1f0 [ 5351.376619] sbitmap_queue_clear+0x74/0xc0 [ 5351.380702] blk_mq_put_tag+0x44/0x70 [ 5351.384352] __blk_mq_free_request+0x88/0xf0 [ 5351.388609] blk_mq_free_request+0xc0/0x160 [ 5351.392779] bfq_insert_request.isra.0+0x204/0x270 [ 5351.397557] bfq_insert_requests+0x70/0x90 [ 5351.401640] blk_mq_sched_insert_requests+0xac/0x250 [ 5351.406593] blk_mq_flush_plug_list+0x110/0x2cc [ 5351.411111] __blk_flush_plug+0x100/0x140 [ 5351.415108] blk_finish_plug+0x40/0x25c [ 5351.418930] __iomap_dio_rw+0x1b4/0x500 [ 5351.422753] iomap_dio_rw+0x18/0x50 [ 5351.426228] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5351.431579] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5351.436322] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5351.441366] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5351.446496] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5351.451625] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5351.457012] process_one_work+0x1ec/0x470 [ 5351.461008] worker_thread+0x74/0x410 [ 5351.464657] kthread+0xec/0x100 [ 5351.467786] ret_from_fork+0x10/0x20 [ 5353.029366] rcu: INFO: rcu_preempt self-detected stall on CPU [ 5353.035099] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=244459 [ 5353.044221] (t=492817 jiffies g=36445 q=146217 ncpus=128) [ 5353.049693] Task dump for CPU 8: [ 5353.052907] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5353.062807] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5353.068975] Call trace: [ 5353.071408] dump_backtrace+0xe8/0x130 [ 5353.075145] show_stack+0x20/0x60 [ 5353.078447] sched_show_task+0x160/0x190 [ 5353.082357] dump_cpu_task+0x4c/0x78 [ 5353.085920] rcu_dump_cpu_stacks+0xc4/0x150 [ 5353.090092] print_cpu_stall+0x180/0x210 [ 5353.094001] check_cpu_stall+0x20c/0x2b4 [ 5353.097911] rcu_pending+0x4c/0x190 [ 5353.101386] rcu_sched_clock_irq+0x80/0x204 [ 5353.105556] update_process_times+0x7c/0xc0 [ 5353.109726] tick_sched_handle+0x38/0x74 [ 5353.113637] tick_sched_timer+0x60/0xc0 [ 5353.117460] __hrtimer_run_queues+0x170/0x300 [ 5353.121804] hrtimer_interrupt+0xfc/0x260 [ 5353.125800] arch_timer_handler_phys+0x34/0x50 [ 5353.130231] handle_percpu_devid_irq+0xac/0x240 [ 5353.134749] generic_handle_domain_irq+0x34/0x50 [ 5353.139354] gic_handle_irq+0x5c/0xe0 [ 5353.143003] call_on_irq_stack+0x2c/0x38 [ 5353.146913] do_interrupt_handler+0xdc/0xe0 [ 5353.151084] el1_interrupt+0x34/0x70 [ 5353.154646] el1h_64_irq_handler+0x18/0x24 [ 5353.158730] el1h_64_irq+0x68/0x6c [ 5353.162118] __sbq_wake_up+0x58/0x1f0 [ 5353.165768] sbitmap_queue_clear+0x74/0xc0 [ 5353.169852] blk_mq_put_tag+0x44/0x70 [ 5353.173501] __blk_mq_free_request+0x88/0xf0 [ 5353.177758] blk_mq_free_request+0xc0/0x160 [ 5353.181928] __blk_mq_end_request+0x78/0x150 [ 5353.186185] scsi_end_request+0x114/0x260 [ 5353.190183] scsi_io_completion+0x64/0x174 [ 5353.194266] scsi_finish_command+0xe0/0x120 [ 5353.198436] scsi_complete+0x8c/0x1c4 [ 5353.202086] blk_complete_reqs+0x5c/0x74 [ 5353.205995] blk_done_softirq+0x38/0x4c [ 5353.209818] __do_softirq+0x168/0x418 [ 5353.213467] __irq_exit_rcu+0x120/0x170 [ 5353.217290] irq_exit_rcu+0x18/0x24 [ 5353.220766] el1_interrupt+0x38/0x70 [ 5353.224328] el1h_64_irq_handler+0x18/0x24 [ 5353.228411] el1h_64_irq+0x68/0x6c [ 5353.231800] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5353.236578] __wake_up_common_lock+0x84/0xac [ 5353.240835] __wake_up+0x20/0x30 [ 5353.244051] __sbq_wake_up+0x13c/0x1f0 [ 5353.247787] sbitmap_queue_clear+0x74/0xc0 [ 5353.251871] blk_mq_put_tag+0x44/0x70 [ 5353.255520] __blk_mq_free_request+0x88/0xf0 [ 5353.259777] blk_mq_free_request+0xc0/0x160 [ 5353.263947] bfq_insert_request.isra.0+0x204/0x270 [ 5353.268725] bfq_insert_requests+0x70/0x90 [ 5353.272809] blk_mq_sched_insert_requests+0xac/0x250 [ 5353.277761] blk_mq_flush_plug_list+0x110/0x2cc [ 5353.282279] __blk_flush_plug+0x100/0x140 [ 5353.286276] blk_finish_plug+0x40/0x25c [ 5353.290100] __iomap_dio_rw+0x1b4/0x500 [ 5353.293923] iomap_dio_rw+0x18/0x50 [ 5353.297398] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5353.302750] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5353.307492] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5353.312536] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5353.317666] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5353.322796] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5353.328182] process_one_work+0x1ec/0x470 [ 5353.332178] worker_thread+0x74/0x410 [ 5353.335827] kthread+0xec/0x100 [ 5353.338956] ret_from_fork+0x10/0x20 [ 5353.342519] Task dump for CPU 68: [ 5353.345821] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5353.355721] Call trace: [ 5353.358154] __switch_to+0xd4/0x100 [ 5353.361630] 0x0 [ 5367.189280] watchdog: BUG: soft lockup - CPU#68 stuck for 4604s! [swapper/68:0] [ 5367.196575] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5367.196591] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5367.284379] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5367.292195] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5367.301486] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5367.308434] pc : __sbq_wake_up+0x40/0x1f0 [ 5367.312430] lr : sbitmap_queue_clear+0x74/0xc0 [ 5367.316861] sp : ffff8000088438b0 [ 5367.320162] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5367.327284] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5367.334406] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5367.341527] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5367.348649] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5367.355771] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5367.362893] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5367.370015] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5367.377137] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5367.384259] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000039 [ 5367.391381] Call trace: [ 5367.393814] __sbq_wake_up+0x40/0x1f0 [ 5367.397463] sbitmap_queue_clear+0x74/0xc0 [ 5367.401546] blk_mq_put_tag+0x44/0x70 [ 5367.405196] __blk_mq_free_request+0x88/0xf0 [ 5367.409453] blk_mq_free_request+0xc0/0x160 [ 5367.413622] __blk_mq_end_request+0x78/0x150 [ 5367.417879] scsi_end_request+0x114/0x260 [ 5367.421875] scsi_io_completion+0x64/0x174 [ 5367.425959] scsi_finish_command+0xe0/0x120 [ 5367.430129] scsi_complete+0x8c/0x1c4 [ 5367.433778] blk_complete_reqs+0x5c/0x74 [ 5367.437688] blk_done_softirq+0x38/0x4c [ 5367.441510] __do_softirq+0x168/0x418 [ 5367.445160] __irq_exit_rcu+0x120/0x170 [ 5367.448983] irq_exit_rcu+0x18/0x24 [ 5367.452459] el1_interrupt+0x38/0x70 [ 5367.456021] el1h_64_irq_handler+0x18/0x24 [ 5367.460104] el1h_64_irq+0x68/0x6c [ 5367.463493] cpuidle_enter_state+0xc4/0x4e0 [ 5367.467663] cpuidle_enter+0x40/0x60 [ 5367.471225] cpuidle_idle_call+0x118/0x1b0 [ 5367.475309] do_idle+0xac/0x100 [ 5367.478438] cpu_startup_entry+0x30/0x34 [ 5367.482347] secondary_start_kernel+0xd8/0xf0 [ 5367.486691] __secondary_switched+0xb0/0xb4 [ 5379.079207] watchdog: BUG: soft lockup - CPU#8 stuck for 4615s! [kworker/8:12:19399] [ 5379.086937] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5379.086954] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5379.174742] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5379.182992] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5379.192283] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5379.198450] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5379.205398] pc : __sbq_wake_up+0x18/0x1f0 [ 5379.209395] lr : sbitmap_queue_clear+0x74/0xc0 [ 5379.213826] sp : ffff80000d79b280 [ 5379.217127] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5379.224249] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5379.231371] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5379.238493] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5379.245615] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5379.252737] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5379.259859] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5379.266981] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5379.274103] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5379.281224] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5379.288346] Call trace: [ 5379.290780] __sbq_wake_up+0x18/0x1f0 [ 5379.294429] sbitmap_queue_clear+0x74/0xc0 [ 5379.298513] blk_mq_put_tag+0x44/0x70 [ 5379.302162] __blk_mq_free_request+0x88/0xf0 [ 5379.306419] blk_mq_free_request+0xc0/0x160 [ 5379.310589] __blk_mq_end_request+0x78/0x150 [ 5379.314846] scsi_end_request+0x114/0x260 [ 5379.318843] scsi_io_completion+0x64/0x174 [ 5379.322926] scsi_finish_command+0xe0/0x120 [ 5379.327096] scsi_complete+0x8c/0x1c4 [ 5379.330745] blk_complete_reqs+0x5c/0x74 [ 5379.334655] blk_done_softirq+0x38/0x4c [ 5379.338478] __do_softirq+0x168/0x418 [ 5379.342127] __irq_exit_rcu+0x120/0x170 [ 5379.345950] irq_exit_rcu+0x18/0x24 [ 5379.349425] el1_interrupt+0x38/0x70 [ 5379.352988] el1h_64_irq_handler+0x18/0x24 [ 5379.357071] el1h_64_irq+0x68/0x6c [ 5379.360460] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5379.365237] __wake_up_common_lock+0x84/0xac [ 5379.369495] __wake_up+0x20/0x30 [ 5379.372710] __sbq_wake_up+0x13c/0x1f0 [ 5379.376447] sbitmap_queue_clear+0x74/0xc0 [ 5379.380530] blk_mq_put_tag+0x44/0x70 [ 5379.384180] __blk_mq_free_request+0x88/0xf0 [ 5379.388438] blk_mq_free_request+0xc0/0x160 [ 5379.392608] bfq_insert_request.isra.0+0x204/0x270 [ 5379.397386] bfq_insert_requests+0x70/0x90 [ 5379.401470] blk_mq_sched_insert_requests+0xac/0x250 [ 5379.406423] blk_mq_flush_plug_list+0x110/0x2cc [ 5379.410941] __blk_flush_plug+0x100/0x140 [ 5379.414938] blk_finish_plug+0x40/0x25c [ 5379.418760] __iomap_dio_rw+0x1b4/0x500 [ 5379.422583] iomap_dio_rw+0x18/0x50 [ 5379.426059] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5379.431410] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5379.436152] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5379.441196] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5379.446326] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5379.451455] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5379.456841] process_one_work+0x1ec/0x470 [ 5379.460838] worker_thread+0x74/0x410 [ 5379.464486] kthread+0xec/0x100 [ 5379.467615] ret_from_fork+0x10/0x20 [ 5391.189134] watchdog: BUG: soft lockup - CPU#68 stuck for 4626s! [swapper/68:0] [ 5391.196429] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5391.196446] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5391.284233] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5391.292049] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5391.301340] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5391.308288] pc : __sbq_wake_up+0x4/0x1f0 [ 5391.312198] lr : sbitmap_queue_clear+0x74/0xc0 [ 5391.316629] sp : ffff800008843900 [ 5391.319930] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5391.327052] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5391.334174] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5391.341296] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5391.348419] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5391.355541] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5391.362663] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5391.369785] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5391.376908] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5391.384030] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff91ccdb10 [ 5391.391152] Call trace: [ 5391.393585] __sbq_wake_up+0x4/0x1f0 [ 5391.397148] blk_mq_put_tag+0x44/0x70 [ 5391.400797] __blk_mq_free_request+0x88/0xf0 [ 5391.405054] blk_mq_free_request+0xc0/0x160 [ 5391.409224] __blk_mq_end_request+0x78/0x150 [ 5391.413480] scsi_end_request+0x114/0x260 [ 5391.417477] scsi_io_completion+0x64/0x174 [ 5391.421560] scsi_finish_command+0xe0/0x120 [ 5391.425731] scsi_complete+0x8c/0x1c4 [ 5391.429381] blk_complete_reqs+0x5c/0x74 [ 5391.433290] blk_done_softirq+0x38/0x4c [ 5391.437113] __do_softirq+0x168/0x418 [ 5391.440762] __irq_exit_rcu+0x120/0x170 [ 5391.444585] irq_exit_rcu+0x18/0x24 [ 5391.448060] el1_interrupt+0x38/0x70 [ 5391.451623] el1h_64_irq_handler+0x18/0x24 [ 5391.455706] el1h_64_irq+0x68/0x6c [ 5391.459094] cpuidle_enter_state+0xc4/0x4e0 [ 5391.463264] cpuidle_enter+0x40/0x60 [ 5391.466827] cpuidle_idle_call+0x118/0x1b0 [ 5391.470910] do_idle+0xac/0x100 [ 5391.474039] cpu_startup_entry+0x30/0x34 [ 5391.477948] secondary_start_kernel+0xd8/0xf0 [ 5391.482293] __secondary_switched+0xb0/0xb4 [ 5397.459106] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 494653 jiffies s: 2093 root: 0x11/. [ 5397.470844] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5397.480052] Task dump for CPU 8: [ 5397.483266] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5397.493169] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5397.499338] Call trace: [ 5397.501771] __switch_to+0xd4/0x100 [ 5397.505248] 0xffff6c0200000080 [ 5397.508376] Task dump for CPU 68: [ 5397.511679] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5397.521581] Call trace: [ 5397.524014] __switch_to+0xd4/0x100 [ 5397.527490] 0x0 [ 5403.079061] watchdog: BUG: soft lockup - CPU#8 stuck for 4638s! [kworker/8:12:19399] [ 5403.086791] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5403.086808] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5403.174596] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5403.182845] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5403.192137] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5403.198304] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5403.205252] pc : __sbq_wake_up+0x18/0x1f0 [ 5403.209248] lr : sbitmap_queue_clear+0x74/0xc0 [ 5403.213679] sp : ffff80000d79b280 [ 5403.216981] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5403.224103] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5403.231225] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5403.238347] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5403.245470] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5403.252591] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5403.259714] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5403.266835] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5403.273957] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5403.281079] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : 0000000000000001 [ 5403.288201] Call trace: [ 5403.290634] __sbq_wake_up+0x18/0x1f0 [ 5403.294283] sbitmap_queue_clear+0x74/0xc0 [ 5403.298367] blk_mq_put_tag+0x44/0x70 [ 5403.302017] __blk_mq_free_request+0x88/0xf0 [ 5403.306273] blk_mq_free_request+0xc0/0x160 [ 5403.310444] __blk_mq_end_request+0x78/0x150 [ 5403.314701] scsi_end_request+0x114/0x260 [ 5403.318697] scsi_io_completion+0x64/0x174 [ 5403.322780] scsi_finish_command+0xe0/0x120 [ 5403.326951] scsi_complete+0x8c/0x1c4 [ 5403.330600] blk_complete_reqs+0x5c/0x74 [ 5403.334509] blk_done_softirq+0x38/0x4c [ 5403.338332] __do_softirq+0x168/0x418 [ 5403.341981] __irq_exit_rcu+0x120/0x170 [ 5403.345803] irq_exit_rcu+0x18/0x24 [ 5403.349279] el1_interrupt+0x38/0x70 [ 5403.352841] el1h_64_irq_handler+0x18/0x24 [ 5403.356924] el1h_64_irq+0x68/0x6c [ 5403.360313] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5403.365091] __wake_up_common_lock+0x84/0xac [ 5403.369348] __wake_up+0x20/0x30 [ 5403.372564] __sbq_wake_up+0x13c/0x1f0 [ 5403.376300] sbitmap_queue_clear+0x74/0xc0 [ 5403.380384] blk_mq_put_tag+0x44/0x70 [ 5403.384033] __blk_mq_free_request+0x88/0xf0 [ 5403.388290] blk_mq_free_request+0xc0/0x160 [ 5403.392460] bfq_insert_request.isra.0+0x204/0x270 [ 5403.397238] bfq_insert_requests+0x70/0x90 [ 5403.401322] blk_mq_sched_insert_requests+0xac/0x250 [ 5403.406273] blk_mq_flush_plug_list+0x110/0x2cc [ 5403.410791] __blk_flush_plug+0x100/0x140 [ 5403.414787] blk_finish_plug+0x40/0x25c [ 5403.418610] __iomap_dio_rw+0x1b4/0x500 [ 5403.422433] iomap_dio_rw+0x18/0x50 [ 5403.425909] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5403.431259] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5403.436002] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5403.441046] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5403.446176] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5403.451305] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5403.456692] process_one_work+0x1ec/0x470 [ 5403.460688] worker_thread+0x74/0x410 [ 5403.464337] kthread+0xec/0x100 [ 5403.467466] ret_from_fork+0x10/0x20 [ 5415.188988] watchdog: BUG: soft lockup - CPU#68 stuck for 4649s! [swapper/68:0] [ 5415.196283] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5415.196299] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5415.284087] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5415.291903] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5415.301194] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5415.308141] pc : __sbq_wake_up+0x1c8/0x1f0 [ 5415.312225] lr : sbitmap_queue_clear+0x74/0xc0 [ 5415.316656] sp : ffff8000088438b0 [ 5415.319957] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5415.327079] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5415.334201] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5415.341323] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5415.348444] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5415.355566] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5415.362688] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5415.369810] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5415.376932] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5415.384055] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000007 [ 5415.391176] Call trace: [ 5415.393610] __sbq_wake_up+0x1c8/0x1f0 [ 5415.397346] sbitmap_queue_clear+0x74/0xc0 [ 5415.401429] blk_mq_put_tag+0x44/0x70 [ 5415.405080] __blk_mq_free_request+0x88/0xf0 [ 5415.409337] blk_mq_free_request+0xc0/0x160 [ 5415.413507] __blk_mq_end_request+0x78/0x150 [ 5415.417764] scsi_end_request+0x114/0x260 [ 5415.421760] scsi_io_completion+0x64/0x174 [ 5415.425843] scsi_finish_command+0xe0/0x120 [ 5415.430014] scsi_complete+0x8c/0x1c4 [ 5415.433663] blk_complete_reqs+0x5c/0x74 [ 5415.437572] blk_done_softirq+0x38/0x4c [ 5415.441395] __do_softirq+0x168/0x418 [ 5415.445043] __irq_exit_rcu+0x120/0x170 [ 5415.448866] irq_exit_rcu+0x18/0x24 [ 5415.452342] el1_interrupt+0x38/0x70 [ 5415.455904] el1h_64_irq_handler+0x18/0x24 [ 5415.459987] el1h_64_irq+0x68/0x6c [ 5415.463376] cpuidle_enter_state+0xc4/0x4e0 [ 5415.467545] cpuidle_enter+0x40/0x60 [ 5415.471108] cpuidle_idle_call+0x118/0x1b0 [ 5415.475192] do_idle+0xac/0x100 [ 5415.478321] cpu_startup_entry+0x30/0x34 [ 5415.482230] secondary_start_kernel+0xd8/0xf0 [ 5415.486575] __secondary_switched+0xb0/0xb4 [ 5427.078915] watchdog: BUG: soft lockup - CPU#8 stuck for 4660s! [kworker/8:12:19399] [ 5427.086644] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5427.086661] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5427.174449] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5427.182699] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5427.191990] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5427.198158] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5427.205105] pc : __sbq_wake_up+0x58/0x1f0 [ 5427.209102] lr : sbitmap_queue_clear+0x74/0xc0 [ 5427.213533] sp : ffff80000d79b280 [ 5427.216834] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5427.223957] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5427.231079] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5427.238201] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5427.245323] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5427.252445] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5427.259568] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5427.266690] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5427.273812] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5427.280934] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5427.288056] Call trace: [ 5427.290489] __sbq_wake_up+0x58/0x1f0 [ 5427.294139] sbitmap_queue_clear+0x74/0xc0 [ 5427.298223] blk_mq_put_tag+0x44/0x70 [ 5427.301872] __blk_mq_free_request+0x88/0xf0 [ 5427.306129] blk_mq_free_request+0xc0/0x160 [ 5427.310299] __blk_mq_end_request+0x78/0x150 [ 5427.314556] scsi_end_request+0x114/0x260 [ 5427.318553] scsi_io_completion+0x64/0x174 [ 5427.322636] scsi_finish_command+0xe0/0x120 [ 5427.326806] scsi_complete+0x8c/0x1c4 [ 5427.330455] blk_complete_reqs+0x5c/0x74 [ 5427.334365] blk_done_softirq+0x38/0x4c [ 5427.338187] __do_softirq+0x168/0x418 [ 5427.341836] __irq_exit_rcu+0x120/0x170 [ 5427.345659] irq_exit_rcu+0x18/0x24 [ 5427.349135] el1_interrupt+0x38/0x70 [ 5427.352697] el1h_64_irq_handler+0x18/0x24 [ 5427.356780] el1h_64_irq+0x68/0x6c [ 5427.360169] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5427.364947] __wake_up_common_lock+0x84/0xac [ 5427.369204] __wake_up+0x20/0x30 [ 5427.372419] __sbq_wake_up+0x13c/0x1f0 [ 5427.376156] sbitmap_queue_clear+0x74/0xc0 [ 5427.380240] blk_mq_put_tag+0x44/0x70 [ 5427.383889] __blk_mq_free_request+0x88/0xf0 [ 5427.388146] blk_mq_free_request+0xc0/0x160 [ 5427.392316] bfq_insert_request.isra.0+0x204/0x270 [ 5427.397094] bfq_insert_requests+0x70/0x90 [ 5427.401178] blk_mq_sched_insert_requests+0xac/0x250 [ 5427.406130] blk_mq_flush_plug_list+0x110/0x2cc [ 5427.410647] __blk_flush_plug+0x100/0x140 [ 5427.414644] blk_finish_plug+0x40/0x25c [ 5427.418466] __iomap_dio_rw+0x1b4/0x500 [ 5427.422289] iomap_dio_rw+0x18/0x50 [ 5427.425765] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5427.431116] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5427.435858] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5427.440901] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5427.446031] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5427.451161] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5427.456547] process_one_work+0x1ec/0x470 [ 5427.460545] worker_thread+0x74/0x410 [ 5427.464193] kthread+0xec/0x100 [ 5427.467322] ret_from_fork+0x10/0x20 [ 5439.188841] watchdog: BUG: soft lockup - CPU#68 stuck for 4671s! [swapper/68:0] [ 5439.196137] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5439.196154] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5439.283941] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5439.291756] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5439.301047] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5439.307995] pc : __sbq_wake_up+0x20/0x1f0 [ 5439.311992] lr : sbitmap_queue_clear+0x74/0xc0 [ 5439.316423] sp : ffff8000088438b0 [ 5439.319724] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5439.326846] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5439.333967] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5439.341089] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5439.348212] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5439.355334] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5439.362456] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5439.369578] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5439.376700] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5439.383822] x2 : 0000000000000007 x1 : ffff80000884393c x0 : 0000000000000001 [ 5439.390944] Call trace: [ 5439.393376] __sbq_wake_up+0x20/0x1f0 [ 5439.397026] sbitmap_queue_clear+0x74/0xc0 [ 5439.401109] blk_mq_put_tag+0x44/0x70 [ 5439.404759] __blk_mq_free_request+0x88/0xf0 [ 5439.409016] blk_mq_free_request+0xc0/0x160 [ 5439.413185] __blk_mq_end_request+0x78/0x150 [ 5439.417442] scsi_end_request+0x114/0x260 [ 5439.421438] scsi_io_completion+0x64/0x174 [ 5439.425522] scsi_finish_command+0xe0/0x120 [ 5439.429692] scsi_complete+0x8c/0x1c4 [ 5439.433341] blk_complete_reqs+0x5c/0x74 [ 5439.437251] blk_done_softirq+0x38/0x4c [ 5439.441073] __do_softirq+0x168/0x418 [ 5439.444722] __irq_exit_rcu+0x120/0x170 [ 5439.448545] irq_exit_rcu+0x18/0x24 [ 5439.452020] el1_interrupt+0x38/0x70 [ 5439.455583] el1h_64_irq_handler+0x18/0x24 [ 5439.459666] el1h_64_irq+0x68/0x6c [ 5439.463054] cpuidle_enter_state+0xc4/0x4e0 [ 5439.467224] cpuidle_enter+0x40/0x60 [ 5439.470787] cpuidle_idle_call+0x118/0x1b0 [ 5439.474871] do_idle+0xac/0x100 [ 5439.478000] cpu_startup_entry+0x30/0x34 [ 5439.481909] secondary_start_kernel+0xd8/0xf0 [ 5439.486253] __secondary_switched+0xb0/0xb4 [ 5451.078769] watchdog: BUG: soft lockup - CPU#8 stuck for 4682s! [kworker/8:12:19399] [ 5451.086499] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5451.086515] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5451.174303] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5451.182552] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5451.191843] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5451.198011] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5451.204958] pc : __sbq_wake_up+0x58/0x1f0 [ 5451.208955] lr : sbitmap_queue_clear+0x74/0xc0 [ 5451.213386] sp : ffff80000d79b280 [ 5451.216687] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5451.223810] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5451.230932] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5451.238054] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5451.245177] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5451.252299] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5451.259421] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5451.266543] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5451.273665] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5451.280787] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5451.287909] Call trace: [ 5451.290342] __sbq_wake_up+0x58/0x1f0 [ 5451.293991] sbitmap_queue_clear+0x74/0xc0 [ 5451.298075] blk_mq_put_tag+0x44/0x70 [ 5451.301724] __blk_mq_free_request+0x88/0xf0 [ 5451.305981] blk_mq_free_request+0xc0/0x160 [ 5451.310151] __blk_mq_end_request+0x78/0x150 [ 5451.314408] scsi_end_request+0x114/0x260 [ 5451.318404] scsi_io_completion+0x64/0x174 [ 5451.322487] scsi_finish_command+0xe0/0x120 [ 5451.326657] scsi_complete+0x8c/0x1c4 [ 5451.330307] blk_complete_reqs+0x5c/0x74 [ 5451.334216] blk_done_softirq+0x38/0x4c [ 5451.338039] __do_softirq+0x168/0x418 [ 5451.341688] __irq_exit_rcu+0x120/0x170 [ 5451.345511] irq_exit_rcu+0x18/0x24 [ 5451.348986] el1_interrupt+0x38/0x70 [ 5451.352548] el1h_64_irq_handler+0x18/0x24 [ 5451.356631] el1h_64_irq+0x68/0x6c [ 5451.360020] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5451.364798] __wake_up_common_lock+0x84/0xac [ 5451.369055] __wake_up+0x20/0x30 [ 5451.372270] __sbq_wake_up+0x13c/0x1f0 [ 5451.376006] sbitmap_queue_clear+0x74/0xc0 [ 5451.380090] blk_mq_put_tag+0x44/0x70 [ 5451.383739] __blk_mq_free_request+0x88/0xf0 [ 5451.387996] blk_mq_free_request+0xc0/0x160 [ 5451.392166] bfq_insert_request.isra.0+0x204/0x270 [ 5451.396944] bfq_insert_requests+0x70/0x90 [ 5451.401028] blk_mq_sched_insert_requests+0xac/0x250 [ 5451.405980] blk_mq_flush_plug_list+0x110/0x2cc [ 5451.410499] __blk_flush_plug+0x100/0x140 [ 5451.414495] blk_finish_plug+0x40/0x25c [ 5451.418318] __iomap_dio_rw+0x1b4/0x500 [ 5451.422141] iomap_dio_rw+0x18/0x50 [ 5451.425616] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5451.430968] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5451.435710] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5451.440754] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5451.445883] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5451.451013] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5451.456399] process_one_work+0x1ec/0x470 [ 5451.460395] worker_thread+0x74/0x410 [ 5451.464045] kthread+0xec/0x100 [ 5451.467174] ret_from_fork+0x10/0x20 [ 5463.188695] watchdog: BUG: soft lockup - CPU#68 stuck for 4693s! [swapper/68:0] [ 5463.195990] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5463.196007] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5463.283795] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5463.291610] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5463.300901] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5463.307849] pc : sbitmap_queue_clear+0x74/0xc0 [ 5463.312280] lr : sbitmap_queue_clear+0x74/0xc0 [ 5463.316711] sp : ffff800008843900 [ 5463.320012] x29: ffff800008843900 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5463.327134] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5463.334256] x23: ffff07ff81d381b0 x22: 0000000000000000 x21: 0000000000000044 [ 5463.341379] x20: 0000000000000000 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5463.348501] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5463.355623] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5463.362745] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5463.369867] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5463.376989] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5463.384111] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5463.391233] Call trace: [ 5463.393667] sbitmap_queue_clear+0x74/0xc0 [ 5463.397751] blk_mq_put_tag+0x44/0x70 [ 5463.401401] __blk_mq_free_request+0x88/0xf0 [ 5463.405657] blk_mq_free_request+0xc0/0x160 [ 5463.409827] __blk_mq_end_request+0x78/0x150 [ 5463.414084] scsi_end_request+0x114/0x260 [ 5463.418080] scsi_io_completion+0x64/0x174 [ 5463.422163] scsi_finish_command+0xe0/0x120 [ 5463.426334] scsi_complete+0x8c/0x1c4 [ 5463.429983] blk_complete_reqs+0x5c/0x74 [ 5463.433892] blk_done_softirq+0x38/0x4c [ 5463.437715] __do_softirq+0x168/0x418 [ 5463.441364] __irq_exit_rcu+0x120/0x170 [ 5463.445186] irq_exit_rcu+0x18/0x24 [ 5463.448662] el1_interrupt+0x38/0x70 [ 5463.452224] el1h_64_irq_handler+0x18/0x24 [ 5463.456307] el1h_64_irq+0x68/0x6c [ 5463.459695] cpuidle_enter_state+0xc4/0x4e0 [ 5463.463866] cpuidle_enter+0x40/0x60 [ 5463.467428] cpuidle_idle_call+0x118/0x1b0 [ 5463.471512] do_idle+0xac/0x100 [ 5463.474640] cpu_startup_entry+0x30/0x34 [ 5463.478550] secondary_start_kernel+0xd8/0xf0 [ 5463.482894] __secondary_switched+0xb0/0xb4 [ 5464.018701] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 501309 jiffies s: 2093 root: 0x11/. [ 5464.030439] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5464.039648] Task dump for CPU 8: [ 5464.042862] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5464.052764] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5464.058934] Call trace: [ 5464.061367] __switch_to+0xd4/0x100 [ 5464.064844] 0xffff6c0200000080 [ 5464.067972] Task dump for CPU 68: [ 5464.071275] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5464.081177] Call trace: [ 5464.083610] __switch_to+0xd4/0x100 [ 5464.087086] 0x0 [ 5475.078623] watchdog: BUG: soft lockup - CPU#8 stuck for 4705s! [kworker/8:12:19399] [ 5475.086353] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5475.086370] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5475.174158] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5475.182407] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5475.191698] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5475.197866] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5475.204813] pc : __sbq_wake_up+0x7c/0x1f0 [ 5475.208810] lr : sbitmap_queue_clear+0x74/0xc0 [ 5475.213241] sp : ffff80000d79b2d0 [ 5475.216542] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5475.223665] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5475.230787] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5475.237910] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5475.245032] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5475.252154] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5475.259277] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5475.266399] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5475.273521] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5475.280643] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5475.287765] Call trace: [ 5475.290198] __sbq_wake_up+0x7c/0x1f0 [ 5475.293847] blk_mq_put_tag+0x44/0x70 [ 5475.297497] __blk_mq_free_request+0x88/0xf0 [ 5475.301754] blk_mq_free_request+0xc0/0x160 [ 5475.305924] __blk_mq_end_request+0x78/0x150 [ 5475.310181] scsi_end_request+0x114/0x260 [ 5475.314177] scsi_io_completion+0x64/0x174 [ 5475.318260] scsi_finish_command+0xe0/0x120 [ 5475.322431] scsi_complete+0x8c/0x1c4 [ 5475.326080] blk_complete_reqs+0x5c/0x74 [ 5475.329989] blk_done_softirq+0x38/0x4c [ 5475.333812] __do_softirq+0x168/0x418 [ 5475.337461] __irq_exit_rcu+0x120/0x170 [ 5475.341284] irq_exit_rcu+0x18/0x24 [ 5475.344759] el1_interrupt+0x38/0x70 [ 5475.348321] el1h_64_irq_handler+0x18/0x24 [ 5475.352405] el1h_64_irq+0x68/0x6c [ 5475.355793] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5475.360571] __wake_up_common_lock+0x84/0xac [ 5475.364829] __wake_up+0x20/0x30 [ 5475.368045] __sbq_wake_up+0x13c/0x1f0 [ 5475.371781] sbitmap_queue_clear+0x74/0xc0 [ 5475.375864] blk_mq_put_tag+0x44/0x70 [ 5475.379514] __blk_mq_free_request+0x88/0xf0 [ 5475.383770] blk_mq_free_request+0xc0/0x160 [ 5475.387940] bfq_insert_request.isra.0+0x204/0x270 [ 5475.392719] bfq_insert_requests+0x70/0x90 [ 5475.396802] blk_mq_sched_insert_requests+0xac/0x250 [ 5475.401754] blk_mq_flush_plug_list+0x110/0x2cc [ 5475.406272] __blk_flush_plug+0x100/0x140 [ 5475.410268] blk_finish_plug+0x40/0x25c [ 5475.414091] __iomap_dio_rw+0x1b4/0x500 [ 5475.417914] iomap_dio_rw+0x18/0x50 [ 5475.421389] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5475.426740] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5475.431483] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5475.436526] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5475.441656] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5475.446786] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5475.452172] process_one_work+0x1ec/0x470 [ 5475.456168] worker_thread+0x74/0x410 [ 5475.459817] kthread+0xec/0x100 [ 5475.462946] ret_from_fork+0x10/0x20 [ 5487.188549] watchdog: BUG: soft lockup - CPU#68 stuck for 4716s! [swapper/68:0] [ 5487.195845] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5487.195861] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5487.283649] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5487.291465] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5487.300755] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5487.307703] pc : __sbq_wake_up+0x58/0x1f0 [ 5487.311700] lr : sbitmap_queue_clear+0x74/0xc0 [ 5487.316130] sp : ffff8000088438b0 [ 5487.319432] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5487.326554] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5487.333676] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5487.340798] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5487.347920] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5487.355042] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5487.362164] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5487.369285] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5487.376407] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5487.383529] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5487.390651] Call trace: [ 5487.393084] __sbq_wake_up+0x58/0x1f0 [ 5487.396733] sbitmap_queue_clear+0x74/0xc0 [ 5487.400817] blk_mq_put_tag+0x44/0x70 [ 5487.404467] __blk_mq_free_request+0x88/0xf0 [ 5487.408723] blk_mq_free_request+0xc0/0x160 [ 5487.412893] __blk_mq_end_request+0x78/0x150 [ 5487.417150] scsi_end_request+0x114/0x260 [ 5487.421146] scsi_io_completion+0x64/0x174 [ 5487.425229] scsi_finish_command+0xe0/0x120 [ 5487.429400] scsi_complete+0x8c/0x1c4 [ 5487.433049] blk_complete_reqs+0x5c/0x74 [ 5487.436958] blk_done_softirq+0x38/0x4c [ 5487.440781] __do_softirq+0x168/0x418 [ 5487.444430] __irq_exit_rcu+0x120/0x170 [ 5487.448252] irq_exit_rcu+0x18/0x24 [ 5487.451728] el1_interrupt+0x38/0x70 [ 5487.455290] el1h_64_irq_handler+0x18/0x24 [ 5487.459373] el1h_64_irq+0x68/0x6c [ 5487.462762] cpuidle_enter_state+0xc4/0x4e0 [ 5487.466932] cpuidle_enter+0x40/0x60 [ 5487.470496] cpuidle_idle_call+0x118/0x1b0 [ 5487.474580] do_idle+0xac/0x100 [ 5487.477708] cpu_startup_entry+0x30/0x34 [ 5487.481618] secondary_start_kernel+0xd8/0xf0 [ 5487.485962] __secondary_switched+0xb0/0xb4 [ 5499.078477] watchdog: BUG: soft lockup - CPU#8 stuck for 4727s! [kworker/8:12:19399] [ 5499.086206] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5499.086224] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5499.174012] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5499.182262] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5499.191552] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5499.197720] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5499.204668] pc : __sbq_wake_up+0x58/0x1f0 [ 5499.208665] lr : sbitmap_queue_clear+0x74/0xc0 [ 5499.213096] sp : ffff80000d79b280 [ 5499.216397] x29: ffff80000d79b280 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5499.223519] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5499.230641] x23: ffff07ff81d381b0 x22: ffff80000d79b30c x21: ffff07ff84e93c00 [ 5499.237764] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5499.244886] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5499.252009] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5499.259131] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5499.266254] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5499.273376] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5499.280497] x2 : 0000000000000007 x1 : ffff80000d79b30c x0 : ffff07ff84e93dd0 [ 5499.287620] Call trace: [ 5499.290053] __sbq_wake_up+0x58/0x1f0 [ 5499.293702] sbitmap_queue_clear+0x74/0xc0 [ 5499.297786] blk_mq_put_tag+0x44/0x70 [ 5499.301437] __blk_mq_free_request+0x88/0xf0 [ 5499.305694] blk_mq_free_request+0xc0/0x160 [ 5499.309864] __blk_mq_end_request+0x78/0x150 [ 5499.314120] scsi_end_request+0x114/0x260 [ 5499.318117] scsi_io_completion+0x64/0x174 [ 5499.322200] scsi_finish_command+0xe0/0x120 [ 5499.326370] scsi_complete+0x8c/0x1c4 [ 5499.330019] blk_complete_reqs+0x5c/0x74 [ 5499.333929] blk_done_softirq+0x38/0x4c [ 5499.337751] __do_softirq+0x168/0x418 [ 5499.341400] __irq_exit_rcu+0x120/0x170 [ 5499.345223] irq_exit_rcu+0x18/0x24 [ 5499.348699] el1_interrupt+0x38/0x70 [ 5499.352261] el1h_64_irq_handler+0x18/0x24 [ 5499.356344] el1h_64_irq+0x68/0x6c [ 5499.359733] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5499.364510] __wake_up_common_lock+0x84/0xac [ 5499.368767] __wake_up+0x20/0x30 [ 5499.371983] __sbq_wake_up+0x13c/0x1f0 [ 5499.375719] sbitmap_queue_clear+0x74/0xc0 [ 5499.379803] blk_mq_put_tag+0x44/0x70 [ 5499.383452] __blk_mq_free_request+0x88/0xf0 [ 5499.387709] blk_mq_free_request+0xc0/0x160 [ 5499.391879] bfq_insert_request.isra.0+0x204/0x270 [ 5499.396657] bfq_insert_requests+0x70/0x90 [ 5499.400740] blk_mq_sched_insert_requests+0xac/0x250 [ 5499.405693] blk_mq_flush_plug_list+0x110/0x2cc [ 5499.410210] __blk_flush_plug+0x100/0x140 [ 5499.414207] blk_finish_plug+0x40/0x25c [ 5499.418030] __iomap_dio_rw+0x1b4/0x500 [ 5499.421853] iomap_dio_rw+0x18/0x50 [ 5499.425328] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5499.430678] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5499.435421] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5499.440465] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5499.445594] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5499.450724] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5499.456110] process_one_work+0x1ec/0x470 [ 5499.460107] worker_thread+0x74/0x410 [ 5499.463756] kthread+0xec/0x100 [ 5499.466884] ret_from_fork+0x10/0x20 [ 5511.188403] watchdog: BUG: soft lockup - CPU#68 stuck for 4738s! [swapper/68:0] [ 5511.195699] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5511.195715] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5511.283502] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5511.291318] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5511.300609] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5511.307557] pc : __sbq_wake_up+0x84/0x1f0 [ 5511.311554] lr : sbitmap_queue_clear+0x74/0xc0 [ 5511.315984] sp : ffff8000088438b0 [ 5511.319286] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5511.326407] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5511.333529] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5511.340651] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5511.347773] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5511.354895] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5511.362017] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5511.369138] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5511.376260] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5511.383382] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5511.390504] Call trace: [ 5511.392937] __sbq_wake_up+0x84/0x1f0 [ 5511.396586] sbitmap_queue_clear+0x74/0xc0 [ 5511.400669] blk_mq_put_tag+0x44/0x70 [ 5511.404318] __blk_mq_free_request+0x88/0xf0 [ 5511.408575] blk_mq_free_request+0xc0/0x160 [ 5511.412745] __blk_mq_end_request+0x78/0x150 [ 5511.417002] scsi_end_request+0x114/0x260 [ 5511.420998] scsi_io_completion+0x64/0x174 [ 5511.425081] scsi_finish_command+0xe0/0x120 [ 5511.429252] scsi_complete+0x8c/0x1c4 [ 5511.432901] blk_complete_reqs+0x5c/0x74 [ 5511.436810] blk_done_softirq+0x38/0x4c [ 5511.440633] __do_softirq+0x168/0x418 [ 5511.444281] __irq_exit_rcu+0x120/0x170 [ 5511.448104] irq_exit_rcu+0x18/0x24 [ 5511.451580] el1_interrupt+0x38/0x70 [ 5511.455142] el1h_64_irq_handler+0x18/0x24 [ 5511.459225] el1h_64_irq+0x68/0x6c [ 5511.462614] cpuidle_enter_state+0xc4/0x4e0 [ 5511.466784] cpuidle_enter+0x40/0x60 [ 5511.470346] cpuidle_idle_call+0x118/0x1b0 [ 5511.474430] do_idle+0xac/0x100 [ 5511.477559] cpu_startup_entry+0x30/0x34 [ 5511.481468] secondary_start_kernel+0xd8/0xf0 [ 5511.485812] __secondary_switched+0xb0/0xb4 [ 5523.078331] watchdog: BUG: soft lockup - CPU#8 stuck for 4749s! [kworker/8:12:19399] [ 5523.086060] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5523.086077] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5523.173865] CPU: 8 PID: 19399 Comm: kworker/8:12 Tainted: G L 6.0.0-rc7 #1 [ 5523.182115] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5523.191406] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5523.197573] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5523.204521] pc : __sbq_wake_up+0x7c/0x1f0 [ 5523.208518] lr : sbitmap_queue_clear+0x74/0xc0 [ 5523.212949] sp : ffff80000d79b2d0 [ 5523.216250] x29: ffff80000d79b2d0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5523.223372] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef180dc0 [ 5523.230495] x23: ffff07ff81d381b0 x22: 0000000000000024 x21: 0000000000000008 [ 5523.237617] x20: 0000000000000024 x19: ffff07ff91ccdb10 x18: ffffffffffffffff [ 5523.244740] x17: ffff32506c412000 x16: ffff800008044000 x15: 0000000000004000 [ 5523.251862] x14: 010320bb28ec0000 x13: 0000000000000087 x12: 071c71c71c71c71c [ 5523.258984] x11: 0000000000000040 x10: ffff07ff801a9c70 x9 : ffffd5bec9545fd8 [ 5523.266106] x8 : ffff07ff80404b70 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5523.273229] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5523.280350] x2 : 0000000000000007 x1 : 0000000000000000 x0 : 0000000000000001 [ 5523.287472] Call trace: [ 5523.289905] __sbq_wake_up+0x7c/0x1f0 [ 5523.293555] blk_mq_put_tag+0x44/0x70 [ 5523.297205] __blk_mq_free_request+0x88/0xf0 [ 5523.301461] blk_mq_free_request+0xc0/0x160 [ 5523.305631] __blk_mq_end_request+0x78/0x150 [ 5523.309888] scsi_end_request+0x114/0x260 [ 5523.313884] scsi_io_completion+0x64/0x174 [ 5523.317967] scsi_finish_command+0xe0/0x120 [ 5523.322138] scsi_complete+0x8c/0x1c4 [ 5523.325787] blk_complete_reqs+0x5c/0x74 [ 5523.329696] blk_done_softirq+0x38/0x4c [ 5523.333519] __do_softirq+0x168/0x418 [ 5523.337168] __irq_exit_rcu+0x120/0x170 [ 5523.340991] irq_exit_rcu+0x18/0x24 [ 5523.344466] el1_interrupt+0x38/0x70 [ 5523.348029] el1h_64_irq_handler+0x18/0x24 [ 5523.352112] el1h_64_irq+0x68/0x6c [ 5523.355500] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5523.360278] __wake_up_common_lock+0x84/0xac [ 5523.364536] __wake_up+0x20/0x30 [ 5523.367751] __sbq_wake_up+0x13c/0x1f0 [ 5523.371487] sbitmap_queue_clear+0x74/0xc0 [ 5523.375570] blk_mq_put_tag+0x44/0x70 [ 5523.379220] __blk_mq_free_request+0x88/0xf0 [ 5523.383477] blk_mq_free_request+0xc0/0x160 [ 5523.387647] bfq_insert_request.isra.0+0x204/0x270 [ 5523.392425] bfq_insert_requests+0x70/0x90 [ 5523.396508] blk_mq_sched_insert_requests+0xac/0x250 [ 5523.401460] blk_mq_flush_plug_list+0x110/0x2cc [ 5523.405978] __blk_flush_plug+0x100/0x140 [ 5523.409974] blk_finish_plug+0x40/0x25c [ 5523.413797] __iomap_dio_rw+0x1b4/0x500 [ 5523.417620] iomap_dio_rw+0x18/0x50 [ 5523.421095] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5523.426446] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5523.431189] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5523.436232] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5523.441362] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5523.446492] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5523.451879] process_one_work+0x1ec/0x470 [ 5523.455875] worker_thread+0x74/0x410 [ 5523.459524] kthread+0xec/0x100 [ 5523.462653] ret_from_fork+0x10/0x20 [ 5530.578295] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 8-.... 68-.... } 507965 jiffies s: 2093 root: 0x11/. [ 5530.590034] rcu: blocking rcu_node structures (internal RCU debug): l=1:0-15:0x100/. l=1:64-79:0x10/. [ 5530.599242] Task dump for CPU 8: [ 5530.602457] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5530.612359] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5530.618529] Call trace: [ 5530.620962] __switch_to+0xd4/0x100 [ 5530.624439] 0xffff6c0200000080 [ 5530.627567] Task dump for CPU 68: [ 5530.630869] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5530.640771] Call trace: [ 5530.643204] __switch_to+0xd4/0x100 [ 5530.646680] 0x0 [ 5533.388268] rcu: INFO: rcu_preempt self-detected stall on CPU [ 5533.394002] rcu: 8-....: (1 GPs behind) idle=c8b4/1/0x4000000000000000 softirq=6134/6135 fqs=253421 [ 5533.403123] (t=510853 jiffies g=36445 q=151623 ncpus=128) [ 5533.408595] Task dump for CPU 8: [ 5533.411810] task:kworker/8:12 state:R running task stack: 0 pid:19399 ppid: 2 flags:0x0000000a [ 5533.421710] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop] [ 5533.427878] Call trace: [ 5533.430311] dump_backtrace+0xe8/0x130 [ 5533.434048] show_stack+0x20/0x60 [ 5533.437349] sched_show_task+0x160/0x190 [ 5533.441259] dump_cpu_task+0x4c/0x78 [ 5533.444822] rcu_dump_cpu_stacks+0xc4/0x150 [ 5533.448993] print_cpu_stall+0x180/0x210 [ 5533.452903] check_cpu_stall+0x20c/0x2b4 [ 5533.456812] rcu_pending+0x4c/0x190 [ 5533.460288] rcu_sched_clock_irq+0x80/0x204 [ 5533.464458] update_process_times+0x7c/0xc0 [ 5533.468628] tick_sched_handle+0x38/0x74 [ 5533.472539] tick_sched_timer+0x60/0xc0 [ 5533.476362] __hrtimer_run_queues+0x170/0x300 [ 5533.480705] hrtimer_interrupt+0xfc/0x260 [ 5533.484702] arch_timer_handler_phys+0x34/0x50 [ 5533.489132] handle_percpu_devid_irq+0xac/0x240 [ 5533.493650] generic_handle_domain_irq+0x34/0x50 [ 5533.498255] gic_handle_irq+0x5c/0xe0 [ 5533.501904] call_on_irq_stack+0x2c/0x38 [ 5533.505814] do_interrupt_handler+0xdc/0xe0 [ 5533.509984] el1_interrupt+0x34/0x70 [ 5533.513547] el1h_64_irq_handler+0x18/0x24 [ 5533.517630] el1h_64_irq+0x68/0x6c [ 5533.521019] __sbq_wake_up+0x58/0x1f0 [ 5533.524668] sbitmap_queue_clear+0x74/0xc0 [ 5533.528752] blk_mq_put_tag+0x44/0x70 [ 5533.532402] __blk_mq_free_request+0x88/0xf0 [ 5533.536658] blk_mq_free_request+0xc0/0x160 [ 5533.540828] __blk_mq_end_request+0x78/0x150 [ 5533.545085] scsi_end_request+0x114/0x260 [ 5533.549081] scsi_io_completion+0x64/0x174 [ 5533.553164] scsi_finish_command+0xe0/0x120 [ 5533.557335] scsi_complete+0x8c/0x1c4 [ 5533.560984] blk_complete_reqs+0x5c/0x74 [ 5533.564894] blk_done_softirq+0x38/0x4c [ 5533.568716] __do_softirq+0x168/0x418 [ 5533.572365] __irq_exit_rcu+0x120/0x170 [ 5533.576188] irq_exit_rcu+0x18/0x24 [ 5533.579663] el1_interrupt+0x38/0x70 [ 5533.583226] el1h_64_irq_handler+0x18/0x24 [ 5533.587309] el1h_64_irq+0x68/0x6c [ 5533.590697] _raw_spin_unlock_irqrestore+0x20/0x50 [ 5533.595475] __wake_up_common_lock+0x84/0xac [ 5533.599733] __wake_up+0x20/0x30 [ 5533.602949] __sbq_wake_up+0x13c/0x1f0 [ 5533.606685] sbitmap_queue_clear+0x74/0xc0 [ 5533.610768] blk_mq_put_tag+0x44/0x70 [ 5533.614418] __blk_mq_free_request+0x88/0xf0 [ 5533.618675] blk_mq_free_request+0xc0/0x160 [ 5533.622844] bfq_insert_request.isra.0+0x204/0x270 [ 5533.627623] bfq_insert_requests+0x70/0x90 [ 5533.631706] blk_mq_sched_insert_requests+0xac/0x250 [ 5533.636658] blk_mq_flush_plug_list+0x110/0x2cc [ 5533.641176] __blk_flush_plug+0x100/0x140 [ 5533.645172] blk_finish_plug+0x40/0x25c [ 5533.648995] __iomap_dio_rw+0x1b4/0x500 [ 5533.652817] iomap_dio_rw+0x18/0x50 [ 5533.656293] xfs_file_dio_write_aligned+0xa4/0x180 [xfs] [ 5533.661644] xfs_file_write_iter+0xf4/0x15c [xfs] [ 5533.666387] nvmet_file_submit_bvec+0x94/0xd0 [nvmet] [ 5533.671430] nvmet_file_execute_io+0x13c/0x260 [nvmet] [ 5533.676561] nvmet_file_execute_rw+0x130/0x150 [nvmet] [ 5533.681690] nvme_loop_execute_work+0x24/0x30 [nvme_loop] [ 5533.687077] process_one_work+0x1ec/0x470 [ 5533.691073] worker_thread+0x74/0x410 [ 5533.694722] kthread+0xec/0x100 [ 5533.697851] ret_from_fork+0x10/0x20 [ 5533.701414] Task dump for CPU 68: [ 5533.704715] task:swapper/68 state:R running task stack: 0 pid: 0 ppid: 1 flags:0x0000000a [ 5533.714616] Call trace: [ 5533.717049] __switch_to+0xd4/0x100 [ 5533.720525] 0x0 [ 5535.188257] watchdog: BUG: soft lockup - CPU#68 stuck for 4761s! [swapper/68:0] [ 5535.195553] Modules linked in: nvme_loop nvme_fabrics nvmet nls_utf8 hfsplus hfs isofs nvme_core nvme_common loop tls rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif cppc_cpufreq ipmi_devintf igb ipmi_msghandler arm_cmn arm_dmc620_pmu arm_spe_pmu arm_dsu_pmu acpi_tad fuse zram xfs crct10dif_ce polyval_ce ast polyval_generic drm_vram_helper ghash_ce drm_ttm_helper sbsa_gwdt ttm xgene_hwmon [ 5535.195569] Unloaded tainted modules: nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 nvmet():1 nvme_fabrics():1 nvme_loop():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 null_blk():1 scsi_debug():1 scsi_debug():1 ipmi_si():1 ipmi_si():1 [last unloaded: nvmet] [ 5535.283356] CPU: 68 PID: 0 Comm: swapper/68 Tainted: G L 6.0.0-rc7 #1 [ 5535.291171] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F26a (SCP: 2.06.20220308) 04/25/2022 [ 5535.300462] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 5535.307410] pc : __sbq_wake_up+0x84/0x1f0 [ 5535.311407] lr : sbitmap_queue_clear+0x74/0xc0 [ 5535.315837] sp : ffff8000088438b0 [ 5535.319139] x29: ffff8000088438b0 x28: ffffd5becb6aec50 x27: ffffd5beca4f7000 [ 5535.326261] x26: ffffd5becaf68a98 x25: ffffd5becb6a60c0 x24: fffffbffef8c4dc0 [ 5535.333383] x23: ffff07ff81d381b0 x22: ffff80000884393c x21: ffff07ff84e93c00 [ 5535.340506] x20: 00000000000001c0 x19: ffff07ff91ccdb10 x18: 0000000000000014 [ 5535.347628] x17: ffff32506cb56000 x16: ffff800008224000 x15: 0000000000004000 [ 5535.354751] x14: 010320bb28ec0000 x13: 0000000000000068 x12: 071c71c71c71c71c [ 5535.361873] x11: 0000000000000040 x10: 0000000000000000 x9 : ffffd5bec9545fd8 [ 5535.368995] x8 : 0000000000000001 x7 : ffff07ff84e93dc8 x6 : 0000000000000008 [ 5535.376118] x5 : ffff80000d043790 x4 : ffff07ff84e93dc0 x3 : 0000000000000008 [ 5535.383241] x2 : 0000000000000007 x1 : ffff80000884393c x0 : ffff07ff84e93dd0 [ 5535.390363] Call trace: [ 5535.392796] __sbq_wake_up+0x84/0x1f0 [ 5535.396445] sbitmap_queue_clear+0x74/0xc0 [ 5535.400528] blk_mq_put_tag+0x44/0x70 [ 5535.404178] __blk_mq_free_request+0x88/0xf0 [ 5535.408435] blk_mq_free_request+0xc0/0x160 [ 5535.412606] __blk_mq_end_request+0x78/0x150 [ 5535.416863] scsi_end_request+0x114/0x260 [ 5535.420859] scsi_io_completion+0x64/0x174 [ 5535.424942] scsi_finish_command+0xe0/0x120 [ 5535.429113] scsi_complete+0x8c/0x1c4 [ 5535.432762] blk_complete_reqs+0x5c/0x74 [ 5535.436671] blk_done_softirq+0x38/0x4c [ 5535.440494] __do_softirq+0x168/0x418 [ 5535.444143] __irq_exit_rcu+0x120/0x170 [ 5535.447967] irq_exit_rcu+0x18/0x24 [ 5535.451442] el1_interrupt+0x38/0x70 [ 5535.455005] el1h_64_irq_handler+0x18/0x24 [ 5535.459088] el1h_64_irq+0x68/0x6c [ 5535.462476] cpuidle_enter_state+0xc4/0x4e0 [ 5535.466646] cpuidle_enter+0x40/0x60 [ 5535.470209] cpuidle_idle_call+0x118/0x1b0 [ 5535.474292] do_idle+0xac/0x100 [ 5535.477421] cpu_startup_entry+0x30/0x34 [ 5535.481330] secondary_start_kernel+0xd8/0xf0 [ 5535.485675] __secondary_switched+0xb0/0xb4