Untitled

[19128.711622] ------------[ cut here ]------------

[19128.711657] btree trans held srcu lock (delaying memory reclaim) for 11 seconds

[19128.711705] WARNING: CPU: 9 PID: 163059 at fs/bcachefs/btree_iter.c:3212 bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[19128.716973] Modules linked in: bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[19128.728720] CPU: 9 UID: 0 PID: 163059 Comm: umount Not tainted 6.15.0-rc5+ #8 NONE

[19128.730259] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[19128.731713] pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[19128.733103] pc : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[19128.734448] lr : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[19128.735796] sp : ffff8000c1d7b980

[19128.736469] x29: ffff8000c1d7b980 x28: 0000cef1eb3ee958 x27: ffff000305180000

[19128.737896] x26: ffff0004a6e40000 x25: 0000000000000001 x24: ffff0000e1018000

[19128.739339] x23: ffff8000c1d7bb00 x22: ffff0000e1018000 x21: 0000000000000003

[19128.740762] x20: ffff0004a6e40000 x19: ffff0000e1018000 x18: 00000000ffffffff

[19128.742188] x17: 000c6f1e0003b3b4 x16: 0000000000000018 x15: ffff0009fa76c571

[19128.743633] x14: ffff8000c1d7b4f0 x13: ffff0009fa76c57c x12: 616c636572207972

[19128.745063] x11: 6f6d656d20676e69 x10: 79616c656428206b x9 : ffffd0fc25028e28

[19128.746491] x8 : 0000000000000002 x7 : 0000000000000000 x6 : 000000000000000d

[19128.748115] x5 : ffff0009fa8ab908 x4 : 0000000000000000 x3 : 0000000000000000

[19128.749534] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0001642a8000

[19128.750953] Call trace:

[19128.751619] bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs] (P)

[19128.752834] bch2_trans_begin+0x5e8/0x760 [bcachefs]

[19128.753880] bch2_verify_accounting_clean+0x13c/0xb10 [bcachefs]

[19128.755303] bch2_fs_read_only+0x370/0x408 [bcachefs]

[19128.756350] __bch2_fs_stop+0x60/0x2a8 [bcachefs]

[19128.757317] bch2_put_super+0x1c/0x38 [bcachefs]

[19128.758257] generic_shutdown_super+0x80/0x120

[19128.759051] bch2_kill_sb+0x20/0x40 [bcachefs]

[19128.760113] deactivate_locked_super+0x4c/0x170

[19128.760859] deactivate_super+0x8c/0xb0

[19128.761490] cleanup_mnt+0xb4/0x150

[19128.762068] __cleanup_mnt+0x1c/0x30

[19128.762656] task_work_run+0x98/0x130

[19128.763438] do_notify_resume+0x154/0x180

[19128.764051] el0_svc+0xf4/0x120

[19128.764535] el0t_64_sync_handler+0x10c/0x140

[19128.765200] el0t_64_sync+0x198/0x1a0

[19128.765760] ---[ end trace 0000000000000000 ]---

[19285.714718] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): Using encoding defined by superblock: utf8-12.1.0

[19285.718328] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): starting version 1.27: fast_device_removal opts=metadata_replicas=2,data_replicas=2,metadata_replicas_required=2,compression=lz4,background_compression=zstd:9,metadata_target=nvme,foreground_target=hdd,background_target=hdd

allowing incompatible features above 1.26: snapshot_deletion_v2

[19285.718383] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): recovering from clean shutdown, journal seq 3922414

[19285.757662] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): accounting_read... done

[19290.639224] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): alloc_read... done

[19290.655970] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): snapshots_read... done

[19290.721905] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): going read-write

[19290.736807] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): journal_replay... done

[19290.870625] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): resume_logged_ops... done

[19290.992383] bcachefs (2ae4b8fc-f793-4673-89f6-f84aa8232367): delete_dead_inodes... done

[19311.501613] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.

[19311.501711] device-mapper: uevent: version 1.0.3

[19311.501946] device-mapper: ioctl: 4.49.0-ioctl (2025-01-17) initialised: [email protected]

[22446.862738] hrtimer: interrupt took 15232303 ns

[38272.771244] ------------[ cut here ]------------

[38272.771250] ------------[ cut here ]------------

[38272.771251] ------------[ cut here ]------------

[38272.771254] btree trans held srcu lock (delaying memory reclaim) for 12 seconds

[38272.771261] btree trans held srcu lock (delaying memory reclaim) for 12 seconds

[38272.771261] btree trans held srcu lock (delaying memory reclaim) for 12 seconds

[38272.771270] WARNING: CPU: 16 PID: 499027 at fs/bcachefs/btree_iter.c:3212 bch2_trans_put+0x220/0x240 [bcachefs]

[38272.771272] WARNING: CPU: 9 PID: 164475 at fs/bcachefs/btree_iter.c:3212 bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.771341] Modules linked in:

[38272.771286] WARNING: CPU: 14 PID: 409871 at fs/bcachefs/btree_iter.c:3212 bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.771342] dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[38272.771378] CPU: 16 UID: 0 PID: 499027 Comm: kworker/u81:3 Tainted: G W 6.15.0-rc5+ #8 NONE

[38272.771383] Tainted: [W]=WARN

[38272.771385] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[38272.771402] Modules linked in: dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod

[38272.771387] Workqueue: bcachefs bch2_write_point_do_index_updates [bcachefs]

[38272.771417] sg

[38272.771420] nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce

 

[38272.771454] pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[38272.771456] arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon

[38272.771460] pc : bch2_trans_put+0x220/0x240 [bcachefs]

[38272.771487] raid6_pq

[38272.771491] xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[38272.771503] CPU: 14 UID: 0 PID: 409871 Comm: kworker/14:0 Tainted: G W 6.15.0-rc5+ #8 NONE

[38272.771506] Tainted: [W]=WARN

[38272.771508] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[38272.771510] Workqueue: bcachefs_write_ref bch2_delete_dead_snapshots_work [bcachefs]

[38272.771519] lr : bch2_trans_put+0x220/0x240 [bcachefs]

[38272.771569] (bcachefs-delete-dead-snapshots/)

[38272.771570] sp : ffff8000a2323c70

 

[38272.771573] x29: ffff8000a2323c70

[38272.771575] pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[38272.771576] x28: 0000000000000003 x27: ffff0004a6e4efc0

[38272.771580] x26: ffff0004ebbedc80 x25: 0000000000000000 x24: 0000000000000000

[38272.771585] x23: ffff0004a6e4f260 x22: 0000000000000001 x21: ffff000608f11290

[38272.771589] x20: ffff0004a6e40000 x19: ffff0004a8b20000 x18: 00000000ffffffff

[38272.771592] x17: ffff0000c0e8d640 x16: 0000000000005000 x15: ffff0009fa76d769

[38272.771597] x14: ffff8000a23237e0 x13: ffff0009fa76d774 x12: 616c636572207972

[38272.771602] x11: 6f6d656d20676e69 x10: 79616c656428206b x9 : ffffd0fc24f0832c

[38272.771605] x8 : 0000000000000002

[38272.771579] pc : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.771609] x7 : 0000000000000000 x6 : 000000000000000d

[38272.771613] x5 : ffff0009fa98b908 x4 : ffffd0fc265f7940 x3 : ffff2f0dd4394000

[38272.771618] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0004ebbedc80

[38272.771622] Call trace:

[38272.771624] bch2_trans_put+0x220/0x240 [bcachefs] (P)

[38272.771637] lr : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.771691] sp : ffff800085e43880

[38272.771692] x29: ffff800085e43880 x28: ffff0004ab918490 x27: 0000000000000300

[38272.771698] x26: 0000000000000006 x25: 0000000000000006 x24: ffff0004ab918000

[38272.771676] bch2_data_update_index_update+0x48/0x80 [bcachefs]

 

[38272.771706] x23: 0000000000000000 x22: ffff0004ab918490 x21: 0000000000000003

[38272.771710] x20: ffff0004a6e40000 x19: ffff0004ab918000 x18: 00000000ffffffff

[38272.771714] x17: ffff0000c0e8d680 x16: 0000000000005000 x15: ffff0009fa76d7b9

[38272.771719] x14: ffff800085e433f0 x13: ffff0009fa76d7c4 x12: 616c636572207972

[38272.771722] x11: 6f6d656d20676e69 x10: 79616c656428206b x9 : ffffd0fc25028e28

[38272.771726] x8 : 0000000000000002 x7 : 0000000000000000 x6 : 000000000000000d

[38272.771732] x5 : ffff0009fa94b908 x4 : 0000000000000000 x3 : 0000000000000000

[38272.771736] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000296d83780

[38272.771741] Call trace:

[38272.771730] __bch2_write_index+0x1d8/0x2a0 [bcachefs]

[38272.771743] bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs] (P)

[38272.771781] bch2_write_point_do_index_updates+0xc0/0x208 [bcachefs]

[38272.771795] bch2_trans_begin+0x5e8/0x760 [bcachefs]

[38272.771831] process_one_work+0x178/0x3d0

[38272.771837] worker_thread+0x208/0x400

[38272.771840] kthread+0x120/0x220

[38272.771843] ret_from_fork+0x10/0x20

[38272.771849] ---[ end trace 0000000000000000 ]---

[38272.771845] delete_dead_snapshot_keys_range.constprop.0+0xc4/0x230 [bcachefs]

[38272.771901] delete_dead_snapshot_keys_v2+0x1fc/0x5f0 [bcachefs]

[38272.771953] __bch2_delete_dead_snapshots+0x2e0/0x1098 [bcachefs]

[38272.772004] bch2_delete_dead_snapshots_work+0x40/0x60 [bcachefs]

[38272.772054] process_one_work+0x178/0x3d0

[38272.772059] worker_thread+0x208/0x400

[38272.772061] kthread+0x120/0x220

[38272.772066] ret_from_fork+0x10/0x20

[38272.772070] ---[ end trace 0000000000000000 ]---

[38272.825534] Modules linked in: dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[38272.830637] CPU: 9 UID: 0 PID: 164475 Comm: bch-rebalance/2 Tainted: G W 6.15.0-rc5+ #8 NONE

[38272.831710] Tainted: [W]=WARN

[38272.831995] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[38272.832522] pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[38272.833166] pc : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.833672] lr : bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs]

[38272.834567] sp : ffff8000847b39e0

[38272.834879] x29: ffff8000847b39e0 x28: ffff8000847b3c90 x27: ffff0004a6e40000

[38272.835470] x26: 0000000000000000 x25: ffff0000c7a6b780 x24: 0000000000000001

[38272.836377] x23: dead000000000122 x22: dead000000000100 x21: ffff8000847b3dd0

[38272.836968] x20: ffff0004a6e40000 x19: ffff000493940000 x18: 00000000ffffffff

[38272.837621] x17: ffff0000c0e8d7c0 x16: 0000000000005000 x15: ffff0009fa76d809

[38272.838164] x14: ffff8000847b3550 x13: ffff0009fa76d814 x12: ffff0009fa719fa8

[38272.838770] x11: ffff0009fa66a000 x10: ffff0009fa71a000 x9 : ffffd0fc24f0832c

[38272.839369] x8 : 0000000000000001 x7 : 000000000002ffe8 x6 : c0000000ffffdfff

[38272.839973] x5 : ffff0009fa8ab908 x4 : ffffd0fc265f7940 x3 : ffff2f0dd42b4000

[38272.840670] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000c7a6b780

[38272.841414] Call trace:

[38272.841596] bch2_trans_srcu_unlock+0x14c/0x170 [bcachefs] (P)

[38272.842191] bch2_trans_unlock_long+0x60/0x80 [bcachefs]

[38272.842727] bch2_moving_ctxt_do_pending_writes+0x70/0x3c8 [bcachefs]

[38272.843248] bch2_move_ratelimit+0xd0/0x470 [bcachefs]

[38272.843682] do_rebalance+0x144/0x720 [bcachefs]

[38272.844218] bch2_rebalance_thread+0xf8/0x158 [bcachefs]

[38272.844699] kthread+0x120/0x220

[38272.844939] ret_from_fork+0x10/0x20

[38272.845204] ---[ end trace 0000000000000000 ]---

[43451.112246] netfs: FS-Cache loaded

[43451.168537] Key type ceph registered

[43451.168982] libceph: loaded (mon/osd proto 15/24)

[43451.226997] ceph: loaded (mds proto 32)

[43451.249662] libceph: mon0 (1)10.10.0.39:6789 session established

[43451.251621] libceph: client271012728 fsid f07d7c39-5e08-4e92-a0b0-f3c8a514583c

[43451.255610] libceph: mon0 (1)10.10.0.39:6789 session established

[43451.256292] libceph: client271012731 fsid f07d7c39-5e08-4e92-a0b0-f3c8a514583c

[46868.296056] Unable to handle kernel paging request at virtual address ffff000259440088

[46868.296078] Mem abort info:

[46868.296101] ESR = 0x0000000096000007

[46868.296104] EC = 0x25: DABT (current EL), IL = 32 bits

[46868.296107] SET = 0, FnV = 0

[46868.296109] EA = 0, S1PTW = 0

[46868.296111] FSC = 0x07: level 3 translation fault

[46868.296113] Data abort info:

[46868.296114] ISV = 0, ISS = 0x00000007, ISS2 = 0x00000000

[46868.296116] CM = 0, WnR = 0, TnD = 0, TagAccess = 0

[46868.296118] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0

[46868.296120] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000088298000

[46868.296123] [ffff000259440088] pgd=0000000000000000, p4d=1800000a3ffff403, pud=1800000a3e867403, pmd=1800000a3e79c403, pte=0068000299440f06

[46868.296164] Internal error: Oops: 0000000096000007 [#1] SMP

[46868.314197] Modules linked in: ceph libceph netfs dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[46868.322519] CPU: 2 UID: 0 PID: 726925 Comm: kworker/u80:11 Tainted: G W 6.15.0-rc5+ #8 NONE

[46868.323351] Tainted: [W]=WARN

[46868.323602] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[46868.324315] Workqueue: btree_update btree_interior_update_work [bcachefs]

[46868.325606] pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[46868.326422] pc : btree_node_seq_matches+0x10/0x40 [bcachefs]

[46868.327303] lr : btree_interior_update_work+0x198/0x9c0 [bcachefs]

[46868.328292] sp : ffff800083c0bc80

[46868.328580] x29: ffff800083c0bd20 x28: ffff0000c5689000 x27: 0000000000000000

[46868.329363] x26: ffffd0fc25b26aa8 x25: 0000000000000000 x24: ffff0004a6e40000

[46868.330370] x23: ffff000124453600 x22: ffff0000c56898c0 x21: ffff0000c5689900

[46868.331049] x20: ffff0000c5689058 x19: ffff000427a5c000 x18: 0000000000000000

[46868.332012] x17: ffff0000c0e8d9c0 x16: 0000000000005000 x15: 000000000001b62c

[46868.332890] x14: 0000000000000000 x13: 0000000000000020 x12: 0101010101010101

[46868.333778] x11: 7f7f7f7f7f7f7f7f x10: fefefeff64736063 x9 : ffffd0fc1ffe1e60

[46868.334610] x8 : ffff00016b964000 x7 : 0000000000000000 x6 : 000ffe01f87fffff

[46868.335419] x5 : 0000000000000012 x4 : 000000000000000c x3 : ffff0000c5689558

[46868.336761] x2 : 0000000015000000 x1 : 31734c23d7b33264 x0 : ffff000259440000

[46868.338088] Call trace:

[46868.338295] btree_node_seq_matches+0x10/0x40 [bcachefs] (P)

[46868.338838] process_one_work+0x178/0x3d0

[46868.339176] worker_thread+0x208/0x400

[46868.339489] kthread+0x120/0x220

[46868.339862] ret_from_fork+0x10/0x20

[46868.340350] Code: aa1e03e9 d503201f f9404c00 b40000a0 (f9404400)

[46868.340975] ---[ end trace 0000000000000000 ]---

[46884.203799] watchdog: CPU1: Watchdog detected hard LOCKUP on cpu 2

[46884.203832] Modules linked in: ceph libceph netfs dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[46884.203972] Sending NMI from CPU 1 to CPUs 2:

[46889.383698] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:

[46889.681400] rcu: 2-...0: (1 GPs behind) idle=152c/1/0x4000000000000000 softirq=694636/694637 fqs=1075

[46889.717861] rcu: (detected by 18, t=5335 jiffies, g=2193169, q=30712 ncpus=20)

[46913.189622] watchdog: BUG: soft lockup - CPU#2 stuck for 42s! [kworker/u80:11:726925]

[46913.189636] Modules linked in: ceph libceph netfs dm_mod bcachefs chacha_neon libchacha poly1305_neon lz4hc_compress lz4_compress binfmt_misc sd_mod sg nls_ascii nls_cp437 iscsi_tcp vfat fat libiscsi_tcp libiscsi scsi_transport_iscsi scsi_mod scsi_common aes_ce_blk aes_ce_cipher polyval_ce polyval_generic ghash_ce gf128mul evdev sha2_ce sha256_arm64 virtio_console sha1_ce arm_smccc_trng button efi_pstore configfs nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock efivarfs qemu_fw_cfg virtio_rng ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq xhci_pci xhci_hcd virtio_net usbcore net_failover failover virtio_blk usb_common

[46913.189718] CPU: 2 UID: 0 PID: 726925 Comm: kworker/u80:11 Tainted: G D W 6.15.0-rc5+ #8 NONE

[46913.189726] Tainted: [D]=DIE, [W]=WARN

[46913.189727] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[46913.189730] Workqueue: btree_update btree_interior_update_work [bcachefs]

[46913.189912] pstate: 40401005 (nZcv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[46913.189917] pc : handle_softirqs+0xa8/0x2f0

[46913.189924] lr : __do_softirq+0x1c/0x28

[46913.189928] sp : ffff800080013f40

[46913.189930] x29: ffff800080013f40 x28: ffff0004b7643780 x27: ffffd0fc265f7000

[46913.189936] x26: ffffd0fc25e701c0 x25: ffffd0fc266960c0 x24: ffffd0fc1ffdf3bc

[46913.189941] x23: 0000000040401005 x22: ffffd0fc24e43228 x21: 0000000000000008

[46913.189946] x20: ffffd0fc24e200a8 x19: ffff800083c0b800 x18: 00000000ffffffff

[46913.189952] x17: ffff2f0dd41d4000 x16: ffff800080010000 x15: 0000000000000001

[46913.189958] x14: ffff800083c0b6b8 x13: 000000000000003c x12: ffff800083c0b6cc

[46913.189963] x11: 0000000000000040 x10: ffff0000c01685f0 x9 : ffffd0fc24e20204

[46913.189970] x8 : ffff0000c1e81268 x7 : 747e520fbc5d4702 x6 : ffffd0fc2660a840

[46913.189975] x5 : ffff2f0dd41d4000 x4 : 0000000100b1d086 x3 : 0000000100b1d087

[46913.189980] x2 : 0000000000000100 x1 : ffff2f0dd41d4000 x0 : ffffd0fc2660a840

[46913.189985] Call trace:

[46913.189987] handle_softirqs+0xa8/0x2f0 (P)

[46913.189993] __do_softirq+0x1c/0x28

[46913.189997] ____do_softirq+0x18/0x30

[46913.190005] call_on_irq_stack+0x24/0x68

[46913.190012] do_softirq_own_stack+0x24/0x50

[46913.190019] __irq_exit_rcu+0x130/0x168

[46913.190023] irq_exit_rcu+0x18/0x30

[46913.190028] el1_interrupt+0x38/0x58

[46913.190034] el1h_64_irq_handler+0x18/0x28

[46913.190040] el1h_64_irq+0x6c/0x70

[46913.190044] die+0x120/0x2c0 (P)

[46913.190049] die_kernel_fault+0x4c4/0x4e8

[46913.190054] __do_kernel_fault+0x1c4/0x1d0

[46913.190059] do_bad_area+0x70/0x90

[46913.190062] do_translation_fault+0x50/0xa0

[46913.190068] do_mem_abort+0x48/0xa0

[46913.190072] el1_abort+0x40/0x68

[46913.190076] el1h_64_sync_handler+0xf0/0x120

[46913.190081] el1h_64_sync+0x6c/0x70

[46913.190085] btree_node_seq_matches+0x10/0x40 [bcachefs] (P)

[46913.190248] process_one_work+0x178/0x3d0

[46913.190253] worker_thread+0x208/0x400

[46913.190257] kthread+0x120/0x220

[46913.190263] ret_from_fork+0x10/0x20

[46913.190441] NMI backtrace for cpu 2

[46913.190443] CPU: 2 UID: 0 PID: 726925 Comm: kworker/u80:11 Tainted: G D W L 6.15.0-rc5+ #8 NONE

[46913.190450] Tainted: [D]=DIE, [W]=WARN, [L]=SOFTLOCKUP

[46913.190453] Hardware name: QEMU KVM Virtual Machine, BIOS 2025.02-6 04/08/2025

[46913.190455] Workqueue: btree_update btree_interior_update_work [bcachefs]

[46913.190611] pstate: 40401005 (nZcv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)

[46913.190615] pc : handle_softirqs+0xa8/0x2f0

[46913.190620] lr : __do_softirq+0x1c/0x28

[46913.190623] sp : ffff800080013f40

[46913.190624] x29: ffff800080013f40 x28: ffff0004b7643780 x27: ffffd0fc265f7000

[46913.190629] x26: ffffd0fc25e701c0 x25: ffffd0fc266960c0 x24: ffffd0fc1ffdf3bc

[46913.190634] x23: 0000000040401005 x22: ffffd0fc24e43228 x21: 0000000000000008

[46913.190638] x20: ffffd0fc24e200a8 x19: ffff800083c0b800 x18: 00000000ffffffff

[46913.190643] x17: ffff2f0dd41d4000 x16: ffff800080010000 x15: 0000000000000001

[46913.190648] x14: ffff800083c0b6b8 x13: 000000000000003c x12: ffff800083c0b6cc

[46913.190652] x11: 0000000000000040 x10: ffff0000c01685f0 x9 : ffffd0fc24e20204

[46913.190657] x8 : ffff0000c1e81268 x7 : 747e520fbc5d4702 x6 : ffffd0fc2660a840

[46913.190661] x5 : ffff2f0dd41d4000 x4 : 0000000100b1d086 x3 : 0000000100b1d087

[46913.190666] x2 : 0000000000000100 x1 : ffff2f0dd41d4000 x0 : ffffd0fc2660a840

[46913.190671] Call trace:

[46913.190672] handle_softirqs+0xa8/0x2f0 (P)

[46913.190676] __do_softirq+0x1c/0x28

[46913.190680] ____do_softirq+0x18/0x30

[46913.190686] call_on_irq_stack+0x24/0x68

[46913.190692] do_softirq_own_stack+0x24/0x50

[46913.190699] __irq_exit_rcu+0x130/0x168

[46913.190703] irq_exit_rcu+0x18/0x30

[46913.190707] el1_interrupt+0x38/0x58

[46913.190711] el1h_64_irq_handler+0x18/0x28

[46913.190716] el1h_64_irq+0x6c/0x70

[46913.190720] die+0x120/0x2c0 (P)

[46913.190725] die_kernel_fault+0x4c4/0x4e8

[46913.190729] __do_kernel_fault+0x1c4/0x1d0

[46913.190733] do_bad_area+0x70/0x90

[46913.190737] do_translation_fault+0x50/0xa0

[46913.190742] do_mem_abort+0x48/0xa0

[46913.190746] el1_abort+0x40/0x68

[46913.190750] el1h_64_sync_handler+0xf0/0x120

[46913.190755] el1h_64_sync+0x6c/0x70

[46913.190758] btree_node_seq_matches+0x10/0x40 [bcachefs] (P)

[46913.190909] process_one_work+0x178/0x3d0

[46913.190913] worker_thread+0x208/0x400

[46913.190916] kthread+0x120/0x220

[46913.190923] ret_from_fork+0x10/0x20

[47170.692080] Allocator stuck? Waited for 30 seconds

[47170.692113] Allocator debug:

[47170.692151] capacity87026953749

[47170.692153] reserved 7567561195

[47170.692154] hidden 113934336

[47170.692156] btree 718493696

[47170.692157] data 39669520744

[47170.692159] cached 5715473620

[47170.692160] reserved 0

[47170.692162] online_reserved 3433308

[47170.692164] nr_inodes 0

[47170.692165]

[47170.692166] freelist_wait empty

[47170.692168] open buckets allocated550

[47170.692169] open buckets total 1024

[47170.692170] open_buckets_wait waiting

[47170.692172] open_buckets_btree 480

[47170.692173] open_buckets_user 69

[47170.692174] btree reserve cache 29

 

[47170.692177] Dev 0:

[47170.692178] buckets sectors fragmented

[47170.692180] free 15213 0 0

[47170.692181] sb 3 6152 6136

[47170.692183] journal 799 3272704 0

[47170.692184] btree 52491 185235456 29767680

[47170.692186] user 32780 133787781 1636383018875

[47170.692187] cached 37 62247 89305

[47170.692189] parity 0 0 0

[47170.692191] stripe 0 0 0

[47170.692192] need_gc_gens 0 0 0

[47170.692193] need_discard 11316 0 0

[47170.692195] unstriped 0 0 0

[47170.692196] capacity 112639

[47170.692198]

[47170.692199] reserves:

[47170.692200] stripe 3526

[47170.692201] normal 1767

[47170.692203] copygc 8

[47170.692204] btree 4

[47170.692205] btree_copygc 0

[47170.692206] reclaim 0

[47170.692208] interior_updates 0

[47170.692209]

[47170.692210] open buckets 115

[47170.692211] buckets to invalidate 0

 

[47170.692214] Dev 1:

[47170.692215] buckets sectors fragmented

[47170.692217] free 15698 0 0

[47170.692218] sb 3 6152 6136

[47170.692222] journal 799 3272704 0

[47170.692225] btree 50447 177857536 28773376

[47170.692227] user 34035 138936152 6416681611432

[47170.692228] cached 39 64797 94947

[47170.692229] parity 0 0 0

[47170.692231] stripe 0 0 0

[47170.692232] need_gc_gens 0 0 0

[47170.692234] need_discard 11618 0 0

[47170.692236] unstriped 0 0 0

[47170.692238] capacity 112639

[47170.692239]

[47170.692240] reserves:

[47170.692241] stripe 3526

[47170.692244] normal 1767

[47170.692246] copygc 8

[47170.692248] btree 4

[47170.692252] btree_copygc 0

[47170.692255] reclaim 0

[47170.692258] interior_updates 0

[47170.692261]

[47170.692263] open buckets 125

[47170.692266] buckets to invalidate 0

 

[47170.692273] Dev 3:

[47170.692276] buckets sectors fragmented

[47170.692279] free 15759 0 0

[47170.692282] sb 3 6152 6136

[47170.692286] journal 799 3272704 0

[47170.692289] btree 48851 172660736 27432960

[47170.692293] user 34627 141336084 6305012486636

[47170.692296] cached 42 105932 12884967988

[47170.692300] parity 0 0 0

[47170.692303] stripe 0 0 0

[47170.692307] need_gc_gens 0 0 0

[47170.692310] need_discard 12558 0 0

[47170.692314] unstriped 0 0 0

[47170.692317] capacity 112639

[47170.692320]

[47170.692322] reserves:

[47170.692326] stripe 3526

[47170.692329] normal 1767

[47170.692333] copygc 8

[47170.692337] btree 4

[47170.692341] btree_copygc 0

[47170.692345] reclaim 0

[47170.692348] interior_updates 0

[47170.692351]

[47170.692356] open buckets 129

[47170.692362] buckets to invalidate 0

 

[47170.692370] Dev 4:

[47170.692404] buckets sectors fragmented

[47170.692408] free 16469 0 0

[47170.692412] sb 3 6152 6136

[47170.692416] journal 799 3272704 0

[47170.692421] btree 51615 182739968 28675072

[47170.692424] user 34978 142724029 2091649619011

[47170.692428] cached 21 36721 8589983887

[47170.692431] parity 0 0 0

[47170.692434] stripe 0 0 0

[47170.692439] need_gc_gens 0 0 0

[47170.692444] need_discard 8754 0 0

[47170.692449] unstriped 0 0 0

[47170.692453] capacity 112639

[47170.692457]

[47170.692462] reserves:

[47170.692466] stripe 3526

[47170.692470] normal 1767

[47170.692474] copygc 8

[47170.692477] btree 4

[47170.692482] btree_copygc 0

[47170.692486] reclaim 0

[47170.692490] interior_updates 0

[47170.692494]

[47170.692498] open buckets 113

[47170.692501] buckets to invalidate 0

 

[47170.692507] Dev 6:

[47170.692510] buckets sectors fragmented

[47170.692514] free 336318 0 0

[47170.692519] sb 1 4104 12280

[47170.692522] journal 1024 16777216 0

[47170.692526] btree 0 0 0

[47170.692531] user 504993 8243339754 358002734456342

[47170.692534] cached 91325 1159652713 132753480768151

[47170.692538] parity 0 0 0

[47170.692543] stripe 0 0 0

[47170.692548] need_gc_gens 0 0 0

[47170.692551] need_discard 4711 0 0

[47170.692553] unstriped 0 0 0

[47170.692555] capacity 938372

[47170.692557]

[47170.692559] reserves:

[47170.692560] stripe 29326

[47170.692562] normal 14664

[47170.692564] copygc 2

[47170.692566] btree 1

[47170.692568] btree_copygc 0

[47170.692570] reclaim 0

[47170.692572] interior_updates 0

[47170.692574]

[47170.692576] open buckets 12

[47170.692578] buckets to invalidate 0

 

[47170.692581] Dev 7:

[47170.692583] buckets sectors fragmented

[47170.692586] free 336374 0 0

[47170.692588] sb 1 4104 12280

[47170.692590] journal 1024 16777216 0

[47170.692592] btree 0 0 0

[47170.692593] user 504746 8239173640 358445116207096

[47170.692595] cached 91618 1162290540 133389138090644

[47170.692597] parity 0 0 0

[47170.692599] stripe 0 0 0

[47170.692600] need_gc_gens 0 0 0

[47170.692602] need_discard 4609 0 0

[47170.692604] unstriped 0 0 0

[47170.692606] capacity 938372

[47170.692607]

[47170.692608] reserves:

[47170.692609] stripe 29326

[47170.692611] normal 14664

[47170.692612] copygc 2

[47170.692613] btree 1

[47170.692615] btree_copygc 0

[47170.692616] reclaim 0

[47170.692617] interior_updates 0

[47170.692618]

[47170.692620] open buckets 12

[47170.692621] buckets to invalidate 0

 

[47170.692623] Dev 8:

[47170.692625] buckets sectors fragmented

[47170.692626] free 454953 0 0

[47170.692628] sb 1 4104 12280

[47170.692629] journal 1024 16777216 0

[47170.692631] btree 0 0 0

[47170.692632] user 338812 5513260218 15457625133894

[47170.692634] cached 138842 1688058017 163471336982367

[47170.692636] parity 0 0 0

[47170.692637] stripe 0 0 0

[47170.692639] need_gc_gens 0 0 0

[47170.692640] need_discard 4740 0 0

[47170.692642] unstriped 0 0 0

[47170.692643] capacity 938372

[47170.692645]

[47170.692646] reserves:

[47170.692647] stripe 29326

[47170.692648] normal 14664

[47170.692650] copygc 2

[47170.692651] btree 1

[47170.692652] btree_copygc 0

[47170.692654] reclaim 0

[47170.692655] interior_updates 0

[47170.692656]

[47170.692657] open buckets 21

[47170.692659] buckets to invalidate 0

 

[47170.692661] Dev 9:

[47170.692662] buckets sectors fragmented

[47170.692664] free 455544 0 0

[47170.692665] sb 1 4104 12280

[47170.692667] journal 1024 16777216 0

[47170.692668] btree 0 0 0

[47170.692669] user 338813 5512145518 15753979008402

[47170.692671] cached 138590 1689275142 164953095353594

[47170.692673] parity 0 0 0

[47170.692674] stripe 0 0 0

[47170.692676] need_gc_gens 0 0 0

[47170.692677] need_discard 4400 0 0

[47170.692678] unstriped 0 0 0

[47170.692680] capacity 938372

[47170.692681]

[47170.692682] reserves:

[47170.692683] stripe 29326

[47170.692685] normal 14664

[47170.692686] copygc 2

[47170.692687] btree 1

[47170.692688] btree_copygc 0

[47170.692689] reclaim 0

[47170.692691] interior_updates 0

[47170.692692]

[47170.692693] open buckets 21

[47170.692694] buckets to invalidate 0

 

[47170.692696] Dev 10:

[47170.692698] buckets sectors fragmented

[47170.692700] free 229288 0 0

[47170.692701] sb 1 4104 28664

[47170.692703] journal 512 16777216 0

[47170.692704] btree 0 0 0

[47170.692706] user 244319 7562377403 4419964815173

[47170.692708] cached 356 4553507 7111901

[47170.692709] parity 0 0 0

[47170.692711] stripe 0 0 0

[47170.692712] need_gc_gens 0 0 0

[47170.692714] need_discard 2390 0 0

[47170.692715] unstriped 0 0 0

[47170.692717] capacity 476866

[47170.692718]

[47170.692719] reserves:

[47170.692720] stripe 14904

[47170.692721] normal 7453

[47170.692723] copygc 2

[47170.692724] btree 1

[47170.692725] btree_copygc 0

[47170.692726] reclaim 0

[47170.692727] interior_updates 0

[47170.692729]

[47170.692730] open buckets 1

[47170.692731] buckets to invalidate 0

 

[47170.692733] Dev 11:

[47170.692735] buckets sectors fragmented

[47170.692736] free 334143 0 0

[47170.692738] sb 1 4104 28664

[47170.692739] journal 512 16777216 0

[47170.692740] btree 0 0 0

[47170.692742] user 140439 4042440165 6382880866843

[47170.692743] cached 0 0 0

[47170.692745] parity 0 0 0

[47170.692746] stripe 0 0 0

[47170.692748] need_gc_gens 0 0 0

[47170.692749] need_discard 1771 0 0

[47170.692751] unstriped 0 0 0

[47170.692752] capacity 476866

[47170.692753]

[47170.692754] reserves:

[47170.692756] stripe 14904

[47170.692757] normal 7453

[47170.692758] copygc 2

[47170.692760] btree 1

[47170.692761] btree_copygc 0

[47170.692762] reclaim 0

[47170.692763] interior_updates 0

[47170.692764]

[47170.692766] open buckets 0

[47170.692767] buckets to invalidate 0

 

[47170.692769] Copygc debug:

[47170.692770] running: 0

[47170.692771] copygc_wait:0

[47170.692773] copygc_wait_at:0

[47170.692774] Currently waiting for:0 B

[47170.692775] Currently waiting since:64.6 TiB

[47170.692776] Currently calculated wait:

[47170.692778] vdb1: 0 B

[47170.692779] vdc1: 0 B

[47170.692780] vdd1: 0 B

[47170.692781] vde1: 0 B

[47170.692782] sdf1: 0 B

[47170.692783] sda1: 0 B

[47170.692784] sdd1: 0 B

[47170.692786] sde1: 0 B

[47170.692787] sdc1: 0 B

[47170.692788] sdb1: 0 B

[47170.692789] [<0>] ret_from_fork+0x10/0x20

 

[47170.692792] Journal debug:

[47170.692793] flags: replay_done,running,may_skip_flush

[47170.692794] dirty journal entries: 1454/32768

[47170.692796] seq: 3981258

[47170.692797] seq_ondisk: 3981257

[47170.692800] last_seq: 3979805

[47170.692801] last_seq_ondisk: 3979805

[47170.692803] flushed_seq_ondisk: 3981257

[47170.692804] watermark: stripe

[47170.692806] each entry reserved: 321

[47170.692807] nr flush writes: 17027

[47170.692808] nr noflush writes: 41816

[47170.692810] average write size: 1.84 MiB

[47170.692811] free buf: 0

[47170.692812] nr direct reclaim: 0

[47170.692814] nr background reclaim: 32366504

[47170.692815] reclaim kicked: 0

[47170.692816] reclaim runs in: 52 ms

[47170.692817] blocked: 0

[47170.692819] current entry sectors: 4096

[47170.692820] current entry error: (No error)

[47170.692821] current entry: 155/261816

[47170.692823] unwritten entries:

[47170.692824] seq: 3981258

[47170.692825] refcount: 1

[47170.692827] io: journal_write_done+0x0/0x450 [bcachefs] r 1

[47170.692828] size: 56.0 B

[47170.692830] expires: 134 jiffies

[47170.692831] flags: need_flush_to_write_buffer

[47170.692833] last buf open

[47170.692834] space:

[47170.692835] discarded 4096:6881280

[47170.692837] clean ondisk 4096:16744448

[47170.692838] clean 4096:16744448

[47170.692840] total 4096:16777216

[47170.692841] dev 0:

[47170.692843] durability 1:

[47170.692844] nr 799

[47170.692845] bucket size 4096

[47170.692846] available 89:1659

[47170.692848] discard_idx 782

[47170.692849] dirty_ondisk 782 (seq 3979805)

[47170.692851] dirty_idx 782 (seq 3979805)

[47170.692852] cur_idx 691 (seq 3981257)

[47170.692854] dev 1:

[47170.692855] durability 1:

[47170.692856] nr 799

[47170.692857] bucket size 4096

[47170.692859] available 90:0

[47170.692860] discard_idx 537

[47170.692861] dirty_ondisk 537 (seq 3979806)

[47170.692863] dirty_idx 537 (seq 3979806)

[47170.692864] cur_idx 445 (seq 3981233)

[47170.692865] dev 3:

[47170.692866] durability 1:

[47170.692867] nr 799

[47170.692869] bucket size 4096

[47170.692870] available 90:0

[47170.692871] discard_idx 13

[47170.692873] dirty_ondisk 13 (seq 3979806)

[47170.692874] dirty_idx 13 (seq 3979806)

[47170.692876] cur_idx 720 (seq 3981233)

[47170.692877] dev 4:

[47170.692880] durability 1:

[47170.692881] nr 799

[47170.692882] bucket size 4096

[47170.692884] available 89:1659

[47170.692885] discard_idx 784

[47170.692886] dirty_ondisk 784 (seq 3979805)

[47170.692888] dirty_idx 784 (seq 3979805)

[47170.692889] cur_idx 693 (seq 3981257)

[47170.692890] dev 6:

[47170.692891] durability 1:

[47170.692893] nr 1024

[47170.692894] bucket size 16384

[47170.692895] available 128:989

[47170.692896] discard_idx 68

[47170.692898] dirty_ondisk 962 (seq 0)

[47170.692899] dirty_idx 962 (seq 0)

[47170.692901] cur_idx 962 (seq 0)

[47170.692902] dev 7:

[47170.692903] durability 1:

[47170.692904] nr 1024

[47170.692905] bucket size 16384

[47170.692907] available 128:1417

[47170.692908] discard_idx 71

[47170.692909] dirty_ondisk 965 (seq 0)

[47170.692911] dirty_idx 965 (seq 0)

[47170.692912] cur_idx 965 (seq 0)

[47170.692914] dev 8:

[47170.692915] durability 1:

[47170.692917] nr 1024

[47170.692919] bucket size 16384

[47170.692920] available 128:12

[47170.692922] discard_idx 123

[47170.692923] dirty_ondisk 1017 (seq 0)

[47170.692924] dirty_idx 1017 (seq 0)

[47170.692926] cur_idx 1017 (seq 0)

[47170.692927] dev 9:

[47170.692928] durability 1:

[47170.692930] nr 1024

[47170.692931] bucket size 16384

[47170.692932] available 128:19

[47170.692934] discard_idx 123

[47170.692935] dirty_ondisk 1017 (seq 0)

[47170.692936] dirty_idx 1017 (seq 0)

[47170.692937] cur_idx 1017 (seq 0)

[47170.692939] dev 10:

[47170.692940] durability 1:

[47170.692941] nr 512

[47170.692943] bucket size 32768

[47170.692944] available 308:24519

[47170.692945] discard_idx 0

[47170.692947] dirty_ondisk 202 (seq 0)

[47170.692949] dirty_idx 202 (seq 0)

[47170.692951] cur_idx 202 (seq 0)

[47170.692952] dev 11:

[47170.692953] durability 1:

[47170.692955] nr 512

[47170.692956] bucket size 32768

[47170.692957] available 210:17926

[47170.692959] discard_idx 0

[47170.692960] dirty_ondisk 300 (seq 0)

[47170.692961] dirty_idx 300 (seq 0)

[47170.692963] cur_idx 300 (seq 0)

[47170.692964] replicas want 2 need 2

[47176.835968] Allocator stuck? Waited for 30 seconds

[47176.836021] Allocator debug:

[47176.836055] capacity87026953749

[47176.836060] reserved 7567561195

[47176.836062] hidden 113934336

[47176.836063] btree 718493696

[47176.836065] data 39669520744

[47176.836066] cached 5715473620

[47176.836068] reserved 0

[47176.836069] online_reserved 3433308

[47176.836070] nr_inodes 0

[47176.836072]

[47176.836073] freelist_wait empty

[47176.836074] open buckets allocated550

[47176.836075] open buckets total 1024

[47176.836077] open_buckets_wait waiting

[47176.836078] open_buckets_btree 480

[47176.836079] open_buckets_user 69

[47176.836081] btree reserve cache 29

 

[47176.836083] Dev 0:

[47176.836085] buckets sectors fragmented

[47176.836086] free 15213 0 0

[47176.836088] sb 3 6152 6136

[47176.836091] journal 799 3272704 0

[47176.836092] btree 52491 185235456 29767680

[47176.836094] user 32780 133787781 1636383018875

[47176.836095] cached 37 62247 89305

[47176.836097] parity 0 0 0

[47176.836098] stripe 0 0 0

[47176.836100] need_gc_gens 0 0 0

[47176.836102] need_discard 11316 0 0

[47176.836103] unstriped 0 0 0

[47176.836105] capacity 112639

[47176.836106]

[47176.836107] reserves:

[47176.836108] stripe 3526

[47176.836110] normal 1767

[47176.836111] copygc 8

[47176.836112] btree 4

[47176.836113] btree_copygc 0

[47176.836115] reclaim 0

[47176.836116] interior_updates 0

[47176.836117]

[47176.836118] open buckets 115

[47176.836119] buckets to invalidate 0

 

[47176.836122] Dev 1:

[47176.836123] buckets sectors fragmented

[47176.836125] free 15698 0 0

[47176.836126] sb 3 6152 6136

[47176.836128] journal 799 3272704 0

[47176.836129] btree 50447 177857536 28773376

[47176.836131] user 34035 138936152 6416681611432

[47176.836133] cached 39 64797 94947

[47176.836134] parity 0 0 0

[47176.836136] stripe 0 0 0

[47176.836137] need_gc_gens 0 0 0

[47176.836139] need_discard 11618 0 0

[47176.836140] unstriped 0 0 0

[47176.836142] capacity 112639

[47176.836143]

[47176.836145] reserves:

[47176.836146] stripe 3526

[47176.836147] normal 1767

[47176.836149] copygc 8

[47176.836150] btree 4

[47176.836151] btree_copygc 0

[47176.836152] reclaim 0

[47176.836153] interior_updates 0

[47176.836155]

[47176.836156] open buckets 125

[47176.836157] buckets to invalidate 0

 

[47176.836160] Dev 3:

[47176.836161] buckets sectors fragmented

[47176.836163] free 15759 0 0

[47176.836164] sb 3 6152 6136

[47176.836166] journal 799 3272704 0

[47176.836167] btree 48851 172660736 27432960

[47176.836169] user 34627 141336084 6305012486636

[47176.836170] cached 42 105932 12884967988

[47176.836172] parity 0 0 0

[47176.836176] stripe 0 0 0

[47176.836178] need_gc_gens 0 0 0

[47176.836180] need_discard 12558 0 0

[47176.836181] unstriped 0 0 0

[47176.836183] capacity 112639

[47176.836185]

[47176.836188] reserves:

[47176.836189] stripe 3526

[47176.836191] normal 1767

[47176.836193] copygc 8

[47176.836196] btree 4

[47176.836198] btree_copygc 0

[47176.836202] reclaim 0

[47176.836205] interior_updates 0

[47176.836208]

[47176.836211] open buckets 129

[47176.836214] buckets to invalidate 0

 

[47176.836219] Dev 4:

[47176.836221] buckets sectors fragmented

[47176.836225] free 16469 0 0

[47176.836230] sb 3 6152 6136

[47176.836233] journal 799 3272704 0

[47176.836236] btree 51615 182739968 28675072

[47176.836241] user 34978 142724029 2091649619011

[47176.836244] cached 21 36721 8589983887

[47176.836248] parity 0 0 0

[47176.836252] stripe 0 0 0

[47176.836255] need_gc_gens 0 0 0

[47176.836258] need_discard 8754 0 0

[47176.836262] unstriped 0 0 0

[47176.836267] capacity 112639

[47176.836271]

[47176.836274] reserves:

[47176.836278] stripe 3526

[47176.836283] normal 1767

[47176.836287] copygc 8

[47176.836290] btree 4

[47176.836294] btree_copygc 0

[47176.836298] reclaim 0

[47176.836302] interior_updates 0

[47176.836306]

[47176.836310] open buckets 113

[47176.836314] buckets to invalidate 0

 

[47176.836321] Dev 6:

[47176.836325] buckets sectors fragmented

[47176.836329] free 336318 0 0

[47176.836333] sb 1 4104 12280

[47176.836338] journal 1024 16777216 0

[47176.836342] btree 0 0 0

[47176.836346] user 504993 8243339754 358002734456342

[47176.836350] cached 91325 1159652713 132753480768151

[47176.836353] parity 0 0 0

[47176.836357] stripe 0 0 0

[47176.836361] need_gc_gens 0 0 0

[47176.836391] need_discard 4711 0 0

[47176.836395] unstriped 0 0 0

[47176.836399] capacity 938372

[47176.836404]

[47176.836408] reserves:

[47176.836412] stripe 29326

[47176.836415] normal 14664

[47176.836420] copygc 2

[47176.836424] btree 1

[47176.836428] btree_copygc 0

[47176.836433] reclaim 0

[47176.836437] interior_updates 0

[47176.836442]

[47176.836446] open buckets 12

[47176.836450] buckets to invalidate 0

 

[47176.836457] Dev 7:

[47176.836460] buckets sectors fragmented

[47176.836466] free 336374 0 0

[47176.836468] sb 1 4104 12280

[47176.836470] journal 1024 16777216 0

[47176.836472] btree 0 0 0

[47176.836474] user 504746 8239173640 358445116207096

[47176.836477] cached 91618 1162290540 133389138090644

[47176.836480] parity 0 0 0

[47176.836483] stripe 0 0 0

[47176.836485] need_gc_gens 0 0 0

[47176.836487] need_discard 4609 0 0

[47176.836489] unstriped 0 0 0

[47176.836491] capacity 938372

[47176.836493]

[47176.836494] reserves:

[47176.836496] stripe 29326

[47176.836497] normal 14664

[47176.836499] copygc 2

[47176.836500] btree 1

[47176.836502] btree_copygc 0

[47176.836503] reclaim 0

[47176.836505] interior_updates 0

[47176.836506]

[47176.836507] open buckets 12

[47176.836509] buckets to invalidate 0

 

[47176.836511] Dev 8:

[47176.836513] buckets sectors fragmented

[47176.836514] free 454953 0 0

[47176.836516] sb 1 4104 12280

[47176.836517] journal 1024 16777216 0

[47176.836519] btree 0 0 0

[47176.836520] user 338812 5513260218 15457625133894

[47176.836522] cached 138842 1688058017 163471336982367

[47176.836523] parity 0 0 0

[47176.836525] stripe 0 0 0

[47176.836526] need_gc_gens 0 0 0

[47176.836527] need_discard 4740 0 0

[47176.836529] unstriped 0 0 0

[47176.836530] capacity 938372

[47176.836531]

[47176.836533] reserves:

[47176.836534] stripe 29326

[47176.836535] normal 14664

[47176.836537] copygc 2

[47176.836538] btree 1

[47176.836539] btree_copygc 0

[47176.836540] reclaim 0

[47176.836542] interior_updates 0

[47176.836543]

[47176.836544] open buckets 21

[47176.836545] buckets to invalidate 0

 

[47176.836548] Dev 9:

[47176.836549] buckets sectors fragmented

[47176.836550] free 455544 0 0

[47176.836552] sb 1 4104 12280

[47176.836553] journal 1024 16777216 0

[47176.836555] btree 0 0 0

[47176.836556] user 338813 5512145518 15753979008402

[47176.836558] cached 138590 1689275142 164953095353594

[47176.836559] parity 0 0 0

[47176.836561] stripe 0 0 0

[47176.836562] need_gc_gens 0 0 0

[47176.836564] need_discard 4400 0 0

[47176.836565] unstriped 0 0 0

[47176.836567] capacity 938372

[47176.836568]

[47176.836569] reserves:

[47176.836570] stripe 29326

[47176.836572] normal 14664

[47176.836573] copygc 2

[47176.836574] btree 1

[47176.836576] btree_copygc 0

[47176.836577] reclaim 0

[47176.836578] interior_updates 0

[47176.836579]

[47176.836580] open buckets 21

[47176.836582] buckets to invalidate 0

 

[47176.836584] Dev 10:

[47176.836585] buckets sectors fragmented

[47176.836587] free 229288 0 0

[47176.836588] sb 1 4104 28664

[47176.836590] journal 512 16777216 0

[47176.836591] btree 0 0 0

[47176.836593] user 244319 7562377403 4419964815173

[47176.836594] cached 356 4553507 7111901

[47176.836596] parity 0 0 0

[47176.836597] stripe 0 0 0

[47176.836599] need_gc_gens 0 0 0

[47176.836600] need_discard 2390 0 0

[47176.836602] unstriped 0 0 0

[47176.836603] capacity 476866

[47176.836605]

[47176.836606] reserves:

[47176.836607] stripe 14904

[47176.836608] normal 7453

[47176.836609] copygc 2

[47176.836611] btree 1

[47176.836612] btree_copygc 0

[47176.836613] reclaim 0

[47176.836614] interior_updates 0

[47176.836616]

[47176.836617] open buckets 1

[47176.836619] buckets to invalidate 0

 

[47176.836622] Dev 11:

[47176.836623] buckets sectors fragmented

[47176.836624] free 334143 0 0

[47176.836626] sb 1 4104 28664

[47176.836628] journal 512 16777216 0

[47176.836629] btree 0 0 0

[47176.836631] user 140439 4042440165 6382880866843

[47176.836632] cached 0 0 0

[47176.836634] parity 0 0 0

[47176.836636] stripe 0 0 0

[47176.836637] need_gc_gens 0 0 0

[47176.836639] need_discard 1771 0 0

[47176.836640] unstriped 0 0 0

[47176.836642] capacity 476866

[47176.836643]

[47176.836644] reserves:

[47176.836645] stripe 14904

[47176.836646] normal 7453

[47176.836648] copygc 2

[47176.836649] btree 1

[47176.836650] btree_copygc 0

[47176.836651] reclaim 0

[47176.836653] interior_updates 0

[47176.836654]

[47176.836655] open buckets 0

[47176.836656] buckets to invalidate 0

 

[47176.836659] Copygc debug:

[47176.836660] running: 0

[47176.836661] copygc_wait:0

[47176.836662] copygc_wait_at:0

[47176.836664] Currently waiting for:0 B

[47176.836665] Currently waiting since:64.6 TiB

[47176.836667] Currently calculated wait:

[47176.836669] vdb1: 0 B

[47176.836670] vdc1: 0 B

[47176.836671] vdd1: 0 B

[47176.836672] vde1: 0 B

[47176.836673] sdf1: 0 B

[47176.836674] sda1: 0 B

[47176.836676] sdd1: 0 B

[47176.836677] sde1: 0 B

[47176.836678] sdc1: 0 B

[47176.836679] sdb1: 0 B

[47176.836680] [<0>] ret_from_fork+0x10/0x20

 

[47176.836683] Journal debug:

[47176.836684] flags: replay_done,running,may_skip_flush

[47176.836685] dirty journal entries: 1460/32768

[47176.836687] seq: 3981264

[47176.836688] seq_ondisk: 3981263

[47176.836690] last_seq: 3979805

[47176.836691] last_seq_ondisk: 3979805

[47176.836692] flushed_seq_ondisk: 3981263

[47176.836693] watermark: stripe

[47176.836695] each entry reserved: 321

[47176.836697] nr flush writes: 17033

[47176.836698] nr noflush writes: 41816

[47176.836700] average write size: 1.84 MiB

[47176.836701] free buf: 0

[47176.836703] nr direct reclaim: 0

[47176.836704] nr background reclaim: 32366563

[47176.836706] reclaim kicked: 0

[47176.836707] reclaim runs in: 60 ms

[47176.836708] blocked: 0

[47176.836710] current entry sectors: 4096

[47176.836711] current entry error: (No error)

[47176.836712] current entry: 124/261816

[47176.836714] unwritten entries:

[47176.836715] seq: 3981264

[47176.836716] refcount: 1

[47176.836717] io: journal_write_done+0x0/0x450 [bcachefs] r 1

[47176.836719] size: 56.0 B

[47176.836720] expires: 162 jiffies

[47176.836722] flags: need_flush_to_write_buffer

[47176.836723] last buf open

[47176.836724] space:

[47176.836726] discarded 4096:6881280

[47176.836727] clean ondisk 4096:16744448

[47176.836729] clean 4096:16744448

[47176.836730] total 4096:16777216

[47176.836731] dev 0:

[47176.836733] durability 1:

[47176.836734] nr 799

[47176.836736] bucket size 4096

[47176.836737] available 89:1599

[47176.836738] discard_idx 782

[47176.836740] dirty_ondisk 782 (seq 3979805)

[47176.836741] dirty_idx 782 (seq 3979805)

[47176.836742] cur_idx 691 (seq 3981263)

[47176.836744] dev 1:

[47176.836745] durability 1:

[47176.836746] nr 799

[47176.836747] bucket size 4096

[47176.836748] available 90:0

[47176.836750] discard_idx 537

[47176.836751] dirty_ondisk 537 (seq 3979806)

[47176.836752] dirty_idx 537 (seq 3979806)

[47176.836754] cur_idx 445 (seq 3981233)

[47176.836755] dev 3:

[47176.836756] durability 1:

[47176.836757] nr 799

[47176.836759] bucket size 4096

[47176.836760] available 90:0

[47176.836761] discard_idx 13

[47176.836763] dirty_ondisk 13 (seq 3979806)

[47176.836764] dirty_idx 13 (seq 3979806)

[47176.836766] cur_idx 720 (seq 3981233)

[47176.836768] dev 4:

[47176.836769] durability 1:

[47176.836770] nr 799

[47176.836771] bucket size 4096

[47176.836772] available 89:1599

[47176.836774] discard_idx 784

[47176.836775] dirty_ondisk 784 (seq 3979805)

[47176.836776] dirty_idx 784 (seq 3979805)

[47176.836778] cur_idx 693 (seq 3981263)

[47176.836779] dev 6:

[47176.836780] durability 1:

[47176.836781] nr 1024

[47176.836783] bucket size 16384

[47176.836784] available 128:989

[47176.836785] discard_idx 68

[47176.836787] dirty_ondisk 962 (seq 0)

[47176.836788] dirty_idx 962 (seq 0)

[47176.836789] cur_idx 962 (seq 0)

[47176.836790] dev 7:

[47176.836792] durability 1:

[47176.836793] nr 1024

[47176.836795] bucket size 16384

[47176.836796] available 128:1417

[47176.836798] discard_idx 71

[47176.836799] dirty_ondisk 965 (seq 0)

[47176.836800] dirty_idx 965 (seq 0)

[47176.836802] cur_idx 965 (seq 0)

[47176.836803] dev 8:

[47176.836804] durability 1:

[47176.836805] nr 1024

[47176.836807] bucket size 16384

[47176.836808] available 128:12

[47176.836809] discard_idx 123

[47176.836810] dirty_ondisk 1017 (seq 0)

[47176.836812] dirty_idx 1017 (seq 0)

[47176.836813] cur_idx 1017 (seq 0)

[47176.836815] dev 9:

[47176.836816] durability 1:

[47176.836817] nr 1024

[47176.836818] bucket size 16384

[47176.836820] available 128:19

[47176.836821] discard_idx 123

[47176.836822] dirty_ondisk 1017 (seq 0)

[47176.836823] dirty_idx 1017 (seq 0)

[47176.836825] cur_idx 1017 (seq 0)

[47176.836826] dev 10:

[47176.836827] durability 1:

[47176.836828] nr 512

[47176.836830] bucket size 32768

[47176.836831] available 308:24519

[47176.836832] discard_idx 0

[47176.836834] dirty_ondisk 202 (seq 0)

[47176.836835] dirty_idx 202 (seq 0)

[47176.836836] cur_idx 202 (seq 0)

[47176.836838] dev 11:

[47176.836839] durability 1:

[47176.836840] nr 512

[47176.836841] bucket size 32768

[47176.836843] available 210:17926

[47176.836844] discard_idx 0

[47176.836845] dirty_ondisk 300 (seq 0)

[47176.836847] dirty_idx 300 (seq 0)

[47176.836848] cur_idx 300 (seq 0)

[47176.836850] replicas want 2 need 2

[47176.836960] Allocator stuck? Waited for 30 seconds

[47176.836966] Allocator debug:

[47176.836968] capacity87026953749

[47176.836969] reserved 7567561195

[47176.836971] hidden 113934336

[47176.836972] btree 718493696

[47176.836974] data 39669520744

[47176.836975] cached 5715473620

[47176.836977] reserved 0

[47176.836978] online_reserved 3433308

[47176.836981] nr_inodes 0

[47176.836982]

[47176.836984] freelist_wait empty

[47176.836985] open buckets allocated550

[47176.836986] open buckets total 1024

[47176.836988] open_buckets_wait waiting

[47176.836989] open_buckets_btree 480

[47176.836990] open_buckets_user 69

[47176.836991] btree reserve cache 29

 

[47176.836994] Dev 0:

[47176.836995] buckets sectors fragmented

[47176.836997] free 15213 0 0

[47176.836998] sb 3 6152 6136

[47176.837000] journal 799 3272704 0

[47176.837001] btree 52491 185235456 29767680

[47176.837003] user 32780 133787781 1636383018875

[47176.837004] cached 37 62247 89305

[47176.837006] parity 0 0 0

[47176.837066] stripe 0 0 0

[47176.837068] need_gc_gens 0 0 0

[47176.837070] need_discard 11316 0 0

[47176.837072] unstriped 0 0 0

[47176.837073] capacity 112639

[47176.837074]

[47176.837076] reserves:

[47176.837077] stripe 3526

[47176.837078] normal 1767

[47176.837079] copygc 8

[47176.837081] btree 4

[47176.837082] btree_copygc 0

[47176.837083] reclaim 0

[47176.837084] interior_updates 0

[47176.837094]

[47176.837095] open buckets 115

[47176.837097] buckets to invalidate 0

 

[47176.837099] Dev 1:

[47176.837100] buckets sectors fragmented

[47176.837102] free 15698 0 0

[47176.837103] sb 3 6152 6136

[47176.837105] journal 799 3272704 0

[47176.837106] btree 50447 177857536 28773376

[47176.837108] user 34035 138936152 6416681611432

[47176.837109] cached 39 64797 94947

[47176.837111] parity 0 0 0

[47176.837112] stripe 0 0 0

[47176.837114] need_gc_gens 0 0 0

[47176.837116] need_discard 11618 0 0

[47176.837118] unstriped 0 0 0

[47176.837119] capacity 112639

[47176.837121]

[47176.837122] reserves:

[47176.837123] stripe 3526

[47176.837124] normal 1767

[47176.837126] copygc 8

[47176.837127] btree 4

[47176.837128] btree_copygc 0

[47176.837130] reclaim 0

[47176.837131] interior_updates 0

[47176.837132]

[47176.837133] open buckets 125

[47176.837134] buckets to invalidate 0

 

[47176.837137] Dev 3:

[47176.837138] buckets sectors fragmented

[47176.837140] free 15759 0 0

[47176.837141] sb 3 6152 6136

[47176.837143] journal 799 3272704 0

[47176.837145] btree 48851 172660736 27432960

[47176.837148] user 34627 141336084 6305012486636

[47176.837149] cached 42 105932 12884967988

[47176.837151] parity 0 0 0

[47176.837152] stripe 0 0 0

[47176.837154] need_gc_gens 0 0 0

[47176.837155] need_discard 12558 0 0

[47176.837157] unstriped 0 0 0

[47176.837158] capacity 112639

[47176.837160]

[47176.837161] reserves:

[47176.837162] stripe 3526

[47176.837163] normal 1767

[47176.837165] copygc 8

[47176.837166] btree 4

[47176.837167] btree_copygc 0

[47176.837169] reclaim 0

[47176.837170] interior_updates 0

[47176.837172]

[47176.837173] open buckets 129

[47176.837175] buckets to invalidate 0

 

[47176.837177] Dev 4:

[47176.837179] buckets sectors fragmented

[47176.837180] free 16469 0 0

[47176.837182] sb 3 6152 6136

[47176.837183] journal 799 3272704 0

[47176.837185] btree 51615 182739968 28675072

[47176.837186] user 34978 142724029 2091649619011

[47176.837188] cached 21 36721 8589983887

[47176.837189] parity 0 0 0

[47176.837191] stripe 0 0 0

[47176.837192] need_gc_gens 0 0 0

[47176.837194] need_discard 8754 0 0

[47176.837195] unstriped 0 0 0

[47176.837196] capacity 112639

[47176.837198]

[47176.837199] reserves:

[47176.837200] stripe 3526

[47176.837202] normal 1767

[47176.837203] copygc 8

[47176.837205] btree 4

[47176.837206] btree_copygc 0

[47176.837207] reclaim 0

[47176.837209] interior_updates 0

[47176.837210]

[47176.837211] open buckets 113

[47176.837212] buckets to invalidate 0

 

[47176.837214] Dev 6:

[47176.837216] buckets sectors fragmented

[47176.837217] free 336318 0 0

[47176.837219] sb 1 4104 12280

[47176.837220] journal 1024 16777216 0

[47176.837222] btree 0 0 0

[47176.837223] user 504993 8243339754 358002734456342

[47176.837225] cached 91325 1159652713 132753480768151

[47176.837226] parity 0 0 0

[47176.837228] stripe 0 0 0

[47176.837229] need_gc_gens 0 0 0

[47176.837231] need_discard 4711 0 0

[47176.837233] unstriped 0 0 0

[47176.837234] capacity 938372

[47176.837236]

[47176.837237] reserves:

[47176.837239] stripe 29326

[47176.837240] normal 14664

[47176.837241] copygc 2

[47176.837242] btree 1

[47176.837245] btree_copygc 0

[47176.837247] reclaim 0

[47176.837250] interior_updates 0

[47176.837254]

[47176.837256] open buckets 12

[47176.837259] buckets to invalidate 0

 

[47176.837268] Dev 7:

[47176.837272] buckets sectors fragmented

[47176.837277] free 336374 0 0

[47176.837280] sb 1 4104 12280

[47176.837284] journal 1024 16777216 0

[47176.837288] btree 0 0 0

[47176.837292] user 504746 8239173640 358445116207096

[47176.837296] cached 91618 1162290540 133389138090644

[47176.837299] parity 0 0 0

[47176.837303] stripe 0 0 0

[47176.837307] need_gc_gens 0 0 0

[47176.837312] need_discard 4609 0 0

[47176.837315] unstriped 0 0 0

[47176.837328] capacity 938372

[47176.837332]

[47176.837335] reserves:

[47176.837339] stripe 29326

[47176.837342] normal 14664

[47176.837346] copygc 2

[47176.837350] btree 1

[47176.837354] btree_copygc 0

[47176.837358] reclaim 0

[47176.837362] interior_updates 0

[47176.837366]

[47176.837370] open buckets 12

[47176.837373] buckets to invalidate 0

 

[47176.837399] Dev 8:

[47176.837406] buckets sectors fragmented

[47176.837409] free 454953 0 0

[47176.837413] sb 1 4104 12280

[47176.837426] journal 1024 16777216 0

[47176.837430] btree 0 0 0

[47176.837434] user 338812 5513260218 15457625133894

[47176.837438] cached 138842 1688058017 163471336982367

[47176.837442] parity 0 0 0

[47176.837446] stripe 0 0 0

[47176.837450] need_gc_gens 0 0 0

[47176.837454] need_discard 4740 0 0

[47176.837458] unstriped 0 0 0

[47176.837462] capacity 938372

[47176.837466]

[47176.837469] reserves:

[47176.837472] stripe 29326

[47176.837476] normal 14664

[47176.837479] copygc 2

[47176.837482] btree 1

[47176.837484] btree_copygc 0

[47176.837485] reclaim 0

[47176.837486] interior_updates 0

[47176.837487]

[47176.837489] open buckets 21

[47176.837491] buckets to invalidate 0

 

[47176.837494] Dev 9:

[47176.837495] buckets sectors fragmented

[47176.837497] free 455544 0 0

[47176.837498] sb 1 4104 12280

[47176.837500] journal 1024 16777216 0

[47176.837501] btree 0 0 0

[47176.837503] user 338813 5512145518 15753979008402

[47176.837504] cached 138590 1689275142 164953095353594

[47176.837506] parity 0 0 0

[47176.837507] stripe 0 0 0

[47176.837509] need_gc_gens 0 0 0

[47176.837510] need_discard 4400 0 0

[47176.837512] unstriped 0 0 0

[47176.837513] capacity 938372

[47176.837514]

[47176.837515] reserves:

[47176.837517] stripe 29326

[47176.837518] normal 14664

[47176.837520] copygc 2

[47176.837521] btree 1

[47176.837523] btree_copygc 0

[47176.837524] reclaim 0

[47176.837525] interior_updates 0

[47176.837526]

[47176.837527] open buckets 21

[47176.837529] buckets to invalidate 0

 

[47176.837548] Dev 10:

[47176.837549] buckets sectors fragmented

[47176.837552] free 229288 0 0

[47176.837553] sb 1 4104 28664

[47176.837555] journal 512 16777216 0

[47176.837556] btree 0 0 0

[47176.837558] user 244319 7562377403 4419964815173

[47176.837559] cached 356 4553507 7111901

[47176.837561] parity 0 0 0

[47176.837562] stripe 0 0 0

[47176.837570] need_gc_gens 0 0 0

[47176.837572] need_discard 2390 0 0

[47176.837573] unstriped 0 0 0

[47176.837575] capacity 476866

[47176.837577]

[47176.837579] reserves:

[47176.837580] stripe 14904

[47176.837581] normal 7453

[47176.837583] copygc 2

[47176.837584] btree 1

[47176.837585] btree_copygc 0

[47176.837586] reclaim 0

[47176.837588] interior_updates 0

[47176.837589]

[47176.837590] open buckets 1

[47176.837591] buckets to invalidate 0

 

[47176.837593] Dev 11:

[47176.837595] buckets sectors fragmented

[47176.837596] free 334143 0 0

[47176.837598] sb 1 4104 28664

[47176.837599] journal 512 16777216 0

[47176.837601] btree 0 0 0

[47176.837602] user 140439 4042440165 6382880866843

[47176.837604] cached 0 0 0

[47176.837605] parity 0 0 0

[47176.837607] stripe 0 0 0

[47176.837608] need_gc_gens 0 0 0

[47176.837610] need_discard 1771 0 0

[47176.837612] unstriped 0 0 0

[47176.837613] capacity 476866

[47176.837615]

[47176.837616] reserves:

[47176.837617] stripe 14904

[47176.837618] normal 7453

[47176.837620] copygc 2

[47176.837621] btree 1

[47176.837622] btree_copygc 0

[47176.837623] reclaim 0

[47176.837625] interior_updates 0

[47176.837633]

[47176.837634] open buckets 0

[47176.837636] buckets to invalidate 0

 

[47176.837638] Copygc debug:

[47176.837640] running: 0

[47176.837642] copygc_wait:0

[47176.837643] copygc_wait_at:0

[47176.837644] Currently waiting for:0 B

[47176.837645] Currently waiting since:64.6 TiB

[47176.837647] Currently calculated wait:

[47176.837648] vdb1: 0 B

[47176.837649] vdc1: 0 B

[47176.837651] vdd1: 0 B

[47176.837652] vde1: 0 B

[47176.837653] sdf1: 0 B

[47176.837654] sda1: 0 B

[47176.837655] sdd1: 0 B

[47176.837656] sde1: 0 B

[47176.837657] sdc1: 0 B

[47176.837659] sdb1: 0 B

[47176.837660] [<0>] ret_from_fork+0x10/0x20

 

[47176.837662] Journal debug:

[47176.837663] flags: replay_done,running,may_skip_flush

[47176.837666] dirty journal entries: 1460/32768

[47176.837668] seq: 3981264

[47176.837670] seq_ondisk: 3981263

[47176.837671] last_seq: 3979805

[47176.837672] last_seq_ondisk: 3979805

[47176.837674] flushed_seq_ondisk: 3981263

[47176.837675] watermark: stripe

[47176.837676] each entry reserved: 321

[47176.837677] nr flush writes: 17033

[47176.837679] nr noflush writes: 41816

[47176.837680] average write size: 1.84 MiB

[47176.837681] free buf: 0

[47176.837683] nr direct reclaim: 0

[47176.837684] nr background reclaim: 32366563

[47176.837685] reclaim kicked: 0

[47176.837687] reclaim runs in: 60 ms

[47176.837688] blocked: 0

[47176.837689] current entry sectors: 4096

[47176.837691] current entry error: (No error)

[47176.837692] current entry: 124/261816

[47176.837693] unwritten entries:

[47176.837700] seq: 3981264

[47176.837701] refcount: 1

[47176.837703] io: journal_write_done+0x0/0x450 [bcachefs] r 1

[47176.837705] size: 56.0 B

[47176.837706] expires: 162 jiffies

[47176.837708] flags: need_flush_to_write_buffer

[47176.837709] last buf open

[47176.837710] space:

[47176.837712] discarded 4096:6881280

[47176.837713] clean ondisk 4096:16744448

[47176.837714] clean 4096:16744448

[47176.837716] total 4096:16777216

[47176.837717] dev 0:

[47176.837719] durability 1:

[47176.837720] nr 799

[47176.837721] bucket size 4096

[47176.837722] available 89:1599

[47176.837724] discard_idx 782

[47176.837725] dirty_ondisk 782 (seq 3979805)

[47176.837728] dirty_idx 782 (seq 3979805)

[47176.837729] cur_idx 691 (seq 3981263)

[47176.837731] dev 1:

[47176.837732] durability 1:

[47176.837734] nr 799

[47176.837735] bucket size 4096

[47176.837736] available 90:0

[47176.837737] discard_idx 537

[47176.837739] dirty_ondisk 537 (seq 3979806)

[47176.837740] dirty_idx 537 (seq 3979806)

[47176.837742] cur_idx 445 (seq 3981233)

[47176.837743] dev 3:

[47176.837744] durability 1:

[47176.837745] nr 799

[47176.837746] bucket size 4096

[47176.837748] available 90:0

[47176.837749] discard_idx 13

[47176.837750] dirty_ondisk 13 (seq 3979806)

[47176.837752] dirty_idx 13 (seq 3979806)

[47176.837753] cur_idx 720 (seq 3981233)

[47176.837755] dev 4:

[47176.837756] durability 1:

[47176.837758] nr 799

[47176.837760] bucket size 4096

[47176.837761] available 89:1599

[47176.837762] discard_idx 784

[47176.837764] dirty_ondisk 784 (seq 3979805)

[47176.837765] dirty_idx 784 (seq 3979805)

[47176.837766] cur_idx 693 (seq 3981263)

[47176.837768] dev 6:

[47176.837769] durability 1:

[47176.837770] nr 1024

[47176.837771] bucket size 16384

[47176.837772] available 128:989

[47176.837774] discard_idx 68

[47176.837775] dirty_ondisk 962 (seq 0)

[47176.837776] dirty_idx 962 (seq 0)

[47176.837778] cur_idx 962 (seq 0)

[47176.837780] dev 7:

[47176.837781] durability 1:

[47176.837783] nr 1024

[47176.837784] bucket size 16384

[47176.837785] available 128:1417

[47176.837787] discard_idx 71

[47176.837788] dirty_ondisk 965 (seq 0)

[47176.837789] dirty_idx 965 (seq 0)

[47176.837791] cur_idx 965 (seq 0)

[47176.837792] dev 8:

[47176.837793] durability 1:

[47176.837794] nr 1024

[47176.837795] bucket size 16384

[47176.837797] available 128:12

[47176.837799] discard_idx 123

[47176.837800] dirty_ondisk 1017 (seq 0)

[47176.837801] dirty_idx 1017 (seq 0)

[47176.837803] cur_idx 1017 (seq 0)

[47176.837804] dev 9:

[47176.837806] durability 1:

[47176.837807] nr 1024

[47176.837809] bucket size 16384

[47176.837811] available 128:19

[47176.837812] discard_idx 123

[47176.837813] dirty_ondisk 1017 (seq 0)

[47176.837815] dirty_idx 1017 (seq 0)

[47176.837816] cur_idx 1017 (seq 0)

[47176.837817] dev 10:

[47176.837818] durability 1:

[47176.837819] nr 512

[47176.837821] bucket size 32768

[47176.837822] available 308:24519

[47176.837823] discard_idx 0

[47176.837825] dirty_ondisk 202 (seq 0)

[47176.837826] dirty_idx 202 (seq 0)

[47176.837827] cur_idx 202 (seq 0)

[47176.837829] dev 11:

[47176.837830] durability 1:

[47176.837831] nr 512

[47176.837833] bucket size 32768

[47176.837834] available 210:17926

[47176.837836] discard_idx 0

[47176.837837] dirty_ondisk 300 (seq 0)

[47176.837839] dirty_idx 300 (seq 0)

[47176.837840] cur_idx 300 (seq 0)

[47176.837841] replicas want 2 need 2

[47383.688010] Allocator stuck? Waited for 30 seconds

[47383.688079] Allocator debug:

[47383.688118] capacity87026953749

[47383.688124] reserved 7567561195

[47383.688131] hidden 113934336

[47383.688134] btree 718493696

[47383.688137] data 39669520131

[47383.688139] cached 5715473615

[47383.688140] reserved 0

[47383.688144] online_reserved 5231630

[47383.688146] nr_inodes 0

[47383.688147]

[47383.688149] freelist_wait empty

[47383.688151] open buckets allocated768

[47383.688155] open buckets total 1024

[47383.688157] open_buckets_wait waiting

[47383.688160] open_buckets_btree 698

[47383.688165] open_buckets_user 69

[47383.688168] btree reserve cache 28

 

[47383.688174] Dev 0:

[47383.688178] buckets sectors fragmented

[47383.688182] free 15213 0 0

[47383.688187] sb 3 6152 6136

[47383.688191] journal 799 3272704 0

[47383.688196] btree 52491 185235456 29767680

[47383.688200] user 32780 133787781 1636383018875

[47383.688204] cached 37 62247 89305

[47383.688208] parity 0 0 0

[47383.688211] stripe 0 0 0

[47383.688216] need_gc_gens 0 0 0

[47383.688221] need_discard 11316 0 0

[47383.688225] unstriped 0 0 0

[47383.688230] capacity 112639

[47383.688234]

[47383.688237] reserves:

[47383.688239] stripe 3526

[47383.688243] normal 1767

[47383.688247] copygc 8

[47383.688251] btree 4

[47383.688254] btree_copygc 0

[47383.688258] reclaim 0

[47383.688262] interior_updates 0

[47383.688267]

[47383.688270] open buckets 169

[47383.688276] buckets to invalidate 0

 

[47383.688283] Dev 1:

[47383.688286] buckets sectors fragmented

[47383.688292] free 15698 0 0

[47383.688296] sb 3 6152 6136

[47383.688298] journal 799 3272704 0

[47383.688301] btree 50447 177857536 28773376

[47383.688304] user 34035 138936152 6416681611432

[47383.688306] cached 39 64797 94947

[47383.688309] parity 0 0 0

[47383.688312] stripe 0 0 0

[47383.688315] need_gc_gens 0 0 0

[47383.688318] need_discard 11618 0 0

[47383.688320] unstriped 0 0 0

[47383.688323] capacity 112639

[47383.688325]

[47383.688327] reserves:

[47383.688330] stripe 3526

[47383.688332] normal 1767

[47383.688335] copygc 8

[47383.688337] btree 4

[47383.688339] btree_copygc 0

[47383.688342] reclaim 0

[47383.688344] interior_updates 0

[47383.688347]

[47383.688350] open buckets 181

[47383.688351] buckets to invalidate 0

 

[47383.688355] Dev 3:

[47383.688357] buckets sectors fragmented

[47383.688359] free 15759 0 0

[47383.688361] sb 3 6152 6136

[47383.688363] journal 799 3272704 0

[47383.688365] btree 48851 172660736 27432960

[47383.688367] user 34627 141336084 6305012486636

[47383.688369] cached 42 105932 12884967988

[47383.688372] parity 0 0 0

[47383.688374] stripe 0 0 0

[47383.688377] need_gc_gens 0 0 0

[47383.688379] need_discard 12558 0 0

[47383.688380] unstriped 0 0 0

[47383.688382] capacity 112639

[47383.688384]

[47383.688386] reserves:

[47383.688388] stripe 3526

[47383.688389] normal 1767

[47383.688391] copygc 8

[47383.688392] btree 4

[47383.688394] btree_copygc 0

[47383.688395] reclaim 0

[47383.688397] interior_updates 0

[47383.688398]

[47383.688400] open buckets 186

[47383.688401] buckets to invalidate 0

 

[47383.688404] Dev 4:

[47383.688406] buckets sectors fragmented

[47383.688408] free 16469 0 0

[47383.688409] sb 3 6152 6136

[47383.688412] journal 799 3272704 0

[47383.688414] btree 51615 182739968 28675072

[47383.688415] user 34978 142724029 2091649619011

[47383.688417] cached 21 36721 8589983887

[47383.688419] parity 0 0 0

[47383.688420] stripe 0 0 0

[47383.688422] need_gc_gens 0 0 0

[47383.688424] need_discard 8754 0 0

[47383.688427] unstriped 0 0 0

[47383.688429] capacity 112639

[47383.688430]

[47383.688432] reserves:

[47383.688433] stripe 3526

[47383.688435] normal 1767

[47383.688437] copygc 8

[47383.688438] btree 4

[47383.688441] btree_copygc 0

[47383.688443] reclaim 0

[47383.688444] interior_updates 0

[47383.688446]

[47383.688447] open buckets 164

[47383.688448] buckets to invalidate 0

 

[47383.688451] Dev 6:

[47383.688453] buckets sectors fragmented

[47383.688454] free 336318 0 0

[47383.688456] sb 1 4104 12280

[47383.688458] journal 1024 16777216 0

[47383.688460] btree 0 0 0

[47383.688462] user 504993 8243339655 358002734456441

[47383.688464] cached 91325 1159652708 132753480768156

[47383.688466] parity 0 0 0

[47383.688469] stripe 0 0 0

[47383.688471] need_gc_gens 0 0 0

[47383.688473] need_discard 4711 0 0

[47383.688475] unstriped 0 0 0

[47383.688477] capacity 938372

[47383.688478]

[47383.688480] reserves:

[47383.688481] stripe 29326

[47383.688484] normal 14664

[47383.688485] copygc 2

[47383.688486] btree 1

[47383.688488] btree_copygc 0

[47383.688490] reclaim 0

[47383.688491] interior_updates 0

[47383.688493]

[47383.688495] open buckets 12

[47383.688496] buckets to invalidate 0

 

[47383.688498] Dev 7:

[47383.688500] buckets sectors fragmented

[47383.688502] free 336374 0 0

[47383.688504] sb 1 4104 12280

[47383.688507] journal 1024 16777216 0

[47383.688509] btree 0 0 0

[47383.688511] user 504746 8239173440 358445116207296

[47383.688512] cached 91618 1162290540 133389138090644

[47383.688514] parity 0 0 0

[47383.688516] stripe 0 0 0

[47383.688518] need_gc_gens 0 0 0

[47383.688521] need_discard 4609 0 0

[47383.688524] unstriped 0 0 0

[47383.688526] capacity 938372

[47383.688528]

[47383.688529] reserves:

[47383.688531] stripe 29326

[47383.688533] normal 14664

[47383.688534] copygc 2

[47383.688536] btree 1

[47383.688538] btree_copygc 0

[47383.688540] reclaim 0

[47383.688541] interior_updates 0

[47383.688543]

[47383.688545] open buckets 12

[47383.688546] buckets to invalidate 0

 

[47383.688550] Dev 8:

[47383.688551] buckets sectors fragmented

[47383.688553] free 454953 0 0

[47383.688555] sb 1 4104 12280

[47383.688557] journal 1024 16777216 0

[47383.688559] btree 0 0 0

[47383.688561] user 338812 5513259951 15453330166865

[47383.688563] cached 138842 1688058017 163471336982367

[47383.688565] parity 0 0 0

[47383.688567] stripe 0 0 0

[47383.688569] need_gc_gens 0 0 0

[47383.688573] need_discard 4740 0 0

[47383.688575] unstriped 0 0 0

[47383.688576] capacity 938372

[47383.688578]

[47383.688579] reserves:

[47383.688581] stripe 29326

[47383.688583] normal 14664

[47383.688585] copygc 2

[47383.688587] btree 1

[47383.688588] btree_copygc 0

[47383.688589] reclaim 0

[47383.688591] interior_updates 0

[47383.688592]

[47383.688595] open buckets 21

[47383.688596] buckets to invalidate 0

 

[47383.688600] Dev 9:

[47383.688601] buckets sectors fragmented

[47383.688603] free 455544 0 0

[47383.688605] sb 1 4104 12280

[47383.688607] journal 1024 16777216 0

[47383.688609] btree 0 0 0

[47383.688611] user 338813 5512145474 15749684041150

[47383.688612] cached 138590 1689275142 164953095353594

[47383.688614] parity 0 0 0

[47383.688615] stripe 0 0 0

[47383.688617] need_gc_gens 0 0 0

[47383.688619] need_discard 4400 0 0

[47383.688620] unstriped 0 0 0

[47383.688622] capacity 938372

[47383.688623]

[47383.688625] reserves:

[47383.688626] stripe 29326

[47383.688627] normal 14664

[47383.688628] copygc 2

[47383.688630] btree 1

[47383.688631] btree_copygc 0

[47383.688632] reclaim 0

[47383.688634] interior_updates 0

[47383.688635]

[47383.688636] open buckets 21

[47383.688638] buckets to invalidate 0

 

[47383.688641] Dev 10:

[47383.688643] buckets sectors fragmented

[47383.688645] free 229288 0 0

[47383.688646] sb 1 4104 28664

[47383.688648] journal 512 16777216 0

[47383.688649] btree 0 0 0

[47383.688651] user 244319 7562377403 4419964815173

[47383.688652] cached 356 4553507 7111901

[47383.688654] parity 0 0 0

[47383.688655] stripe 0 0 0

[47383.688657] need_gc_gens 0 0 0

[47383.688658] need_discard 2390 0 0

[47383.688660] unstriped 0 0 0

[47383.688661] capacity 476866

[47383.688662]

[47383.688664] reserves:

[47383.688666] stripe 14904

[47383.688667] normal 7453

[47383.688669] copygc 2

[47383.688670] btree 1

[47383.688672] btree_copygc 0

[47383.688673] reclaim 0

[47383.688675] interior_updates 0

[47383.688676]

[47383.688678] open buckets 1

[47383.688680] buckets to invalidate 0

 

[47383.688682] Dev 11:

[47383.688684] buckets sectors fragmented

[47383.688686] free 334143 0 0

[47383.688687] sb 1 4104 28664

[47383.688690] journal 512 16777216 0

[47383.688692] btree 0 0 0

[47383.688694] user 140439 4042440162 6382880866846

[47383.688696] cached 0 0 0

[47383.688697] parity 0 0 0

[47383.688699] stripe 0 0 0

[47383.688701] need_gc_gens 0 0 0

[47383.688703] need_discard 1771 0 0

[47383.688705] unstriped 0 0 0

[47383.688707] capacity 476866

[47383.688708]

[47383.688710] reserves:

[47383.688711] stripe 14904

[47383.688713] normal 7453

[47383.688715] copygc 2

[47383.688717] btree 1

[47383.688718] btree_copygc 0

[47383.688720] reclaim 0

[47383.688722] interior_updates 0

[47383.688723]

[47383.688725] open buckets 0

[47383.688726] buckets to invalidate 0

 

[47383.688729] Copygc debug:

[47383.688730] running: 0

[47383.688733] copygc_wait:0

[47383.688734] copygc_wait_at:0

[47383.688735] Currently waiting for:0 B

[47383.688737] Currently waiting since:64.6 TiB

[47383.688739] Currently calculated wait:

[47383.688741] vdb1: 0 B

[47383.688742] vdc1: 0 B

[47383.688744] vdd1: 0 B

[47383.688745] vde1: 0 B

[47383.688747] sdf1: 0 B

[47383.688749] sda1: 0 B

[47383.688750] sdd1: 0 B

[47383.688752] sde1: 0 B

[47383.688753] sdc1: 0 B

[47383.688755] sdb1: 0 B

[47383.688756] [<0>] ret_from_fork+0x10/0x20

 

[47383.688759] Journal debug:

[47383.688760] flags: replay_done,running,may_skip_flush

[47383.688762] dirty journal entries: 2342/32768

[47383.688764] seq: 3982146

[47383.688766] seq_ondisk: 3982145

[47383.688768] last_seq: 3979805

[47383.688769] last_seq_ondisk: 3979805

[47383.688771] flushed_seq_ondisk: 3982145

[47383.688773] watermark: stripe

[47383.688775] each entry reserved: 321

[47383.688777] nr flush writes: 17216

[47383.688779] nr noflush writes: 42516

[47383.688781] average write size: 1.84 MiB

[47383.688783] free buf: 2097152

[47383.688784] nr direct reclaim: 0

[47383.688786] nr background reclaim: 33005313

[47383.688787] reclaim kicked: 0

[47383.688788] reclaim runs in: 28 ms

[47383.688790] blocked: 0

[47383.688792] current entry sectors: 4096

[47383.688794] current entry error: (No error)

[47383.688796] current entry: closed

[47383.688798] unwritten entries:

[47383.688800] seq: 3982146

[47383.688801] io: journal_write_submit+0x0/0x2f8 [bcachefs] r 1

[47383.688803] size: 4.69 KiB

[47383.688805] expires: -8 jiffies

[47383.688807] flags: must_flush separate_flush write_started write_allocated

[47383.688809] last buf closed

[47383.688811] space:

[47383.688812] discarded 4096:5701632

[47383.688814] clean ondisk 4096:16171008

[47383.688816] clean 4096:16171008

[47383.688817] total 4096:16777216

[47383.688819] dev 0:

[47383.688821] durability 1:

[47383.688823] nr 799

[47383.688824] bucket size 4096

[47383.688826] available 0:1

[47383.688827] discard_idx 782

[47383.688829] dirty_ondisk 782 (seq 3979805)

[47383.688831] dirty_idx 782 (seq 3979805)

[47383.688833] cur_idx 780 (seq 3981542)

[47383.688834] dev 1:

[47383.688835] durability 1:

[47383.688837] nr 799

[47383.688838] bucket size 4096

[47383.688840] available 0:0

[47383.688841] discard_idx 537

[47383.688842] dirty_ondisk 537 (seq 3979806)

[47383.688844] dirty_idx 537 (seq 3979806)

[47383.688846] cur_idx 535 (seq 3981543)

[47383.688848] dev 3:

[47383.688849] durability 1:

[47383.688850] nr 799

[47383.688852] bucket size 4096

[47383.688853] available 0:0

[47383.688855] discard_idx 13

[47383.688857] dirty_ondisk 13 (seq 3979806)

[47383.688859] dirty_idx 13 (seq 3979806)

[47383.688861] cur_idx 11 (seq 3981543)

[47383.688862] dev 4:

[47383.688864] durability 1:

[47383.688865] nr 799

[47383.688866] bucket size 4096

[47383.688868] available 0:1

[47383.688870] discard_idx 784

[47383.688872] dirty_ondisk 784 (seq 3979805)

[47383.688874] dirty_idx 784 (seq 3979805)

[47383.688875] cur_idx 782 (seq 3981542)

[47383.688877] dev 6:

[47383.688878] durability 1:

[47383.688879] nr 1024

[47383.688881] bucket size 16384

[47383.688883] available 128:4

[47383.688884] discard_idx 104

[47383.688886] dirty_ondisk 963 (seq 3981556)

[47383.688887] dirty_idx 963 (seq 3981556)

[47383.688888] cur_idx 998 (seq 3982116)

[47383.688890] dev 7:

[47383.688892] durability 1:

[47383.688893] nr 1024

[47383.688895] bucket size 16384

[47383.688898] available 128:6

[47383.688900] discard_idx 107

[47383.688901] dirty_ondisk 966 (seq 3981555)

[47383.688903] dirty_idx 966 (seq 3981555)

[47383.688904] cur_idx 1001 (seq 3982115)

[47383.688906] dev 8:

[47383.688907] durability 1:

[47383.688909] nr 1024

[47383.688911] bucket size 16384

[47383.688913] available 128:3

[47383.688914] discard_idx 159

[47383.688915] dirty_ondisk 1018 (seq 3981554)

[47383.688917] dirty_idx 1018 (seq 3981554)

[47383.688920] cur_idx 29 (seq 3982111)

[47383.688921] dev 9:

[47383.688923] durability 1:

[47383.688925] nr 1024

[47383.688926] bucket size 16384

[47383.688928] available 128:3

[47383.688929] discard_idx 159

[47383.688931] dirty_ondisk 1018 (seq 3981555)

[47383.688933] dirty_idx 1018 (seq 3981555)

[47383.688935] cur_idx 29 (seq 3982112)

[47383.688938] dev 10:

[47383.688939] durability 1:

[47383.688941] nr 512

[47383.688943] bucket size 32768

[47383.688945] available 272:21329

[47383.688947] discard_idx 0

[47383.688948] dirty_ondisk 202 (seq 3981548)

[47383.688949] dirty_idx 202 (seq 3981548)

[47383.688951] cur_idx 238 (seq 3982146)

[47383.688953] dev 11:

[47383.688954] durability 1:

[47383.688955] nr 512

[47383.688957] bucket size 32768

[47383.688958] available 174:17233

[47383.688960] discard_idx 0

[47383.688962] dirty_ondisk 300 (seq 3981547)

[47383.688964] dirty_idx 300 (seq 3981547)

[47383.688965] cur_idx 336 (seq 3982146)

[47383.688967] replicas want 2 need 2

[47383.691933] Allocator stuck? Waited for 30 seconds

[47383.691944] Allocator debug:

[47383.691947] capacity87026953749

[47383.691950] reserved 7567561195

[47383.691951] hidden 113934336

[47383.691953] btree 718493696

[47383.691956] data 39669520131

[47383.691957] cached 5715473615

[47383.691960] reserved 0

[47383.691962] online_reserved 5231630

[47383.691964] nr_inodes 0

[47383.691965]

[47383.691967] freelist_wait empty

[47383.691968] open buckets allocated768

[47383.691970] open buckets total 1024

[47383.691971] open_buckets_wait waiting

[47383.691973] open_buckets_btree 698

[47383.691975] open_buckets_user 69

[47383.691976] btree reserve cache 28

 

[47383.691980] Dev 0:

[47383.691981] buckets sectors fragmented

[47383.691983] free 15213 0 0

[47383.691985] sb 3 6152 6136

[47383.691987] journal 799 3272704 0

[47383.691989] btree 52491 185235456 29767680

[47383.691990] user 32780 133787781 1636383018875

[47383.691992] cached 37 62247 89305

[47383.691993] parity 0 0 0

[47383.691995] stripe 0 0 0

[47383.691997] need_gc_gens 0 0 0

[47383.691998] need_discard 11316 0 0

[47383.692000] unstriped 0 0 0

[47383.692002] capacity 112639

[47383.692004]

[47383.692005] reserves:

[47383.692006] stripe 3526

[47383.692008] normal 1767

[47383.692010] copygc 8

[47383.692011] btree 4

[47383.692013] btree_copygc 0

[47383.692015] reclaim 0

[47383.692017] interior_updates 0

[47383.692018]

[47383.692019] open buckets 169

[47383.692021] buckets to invalidate 0

 

[47383.692023] Dev 1:

[47383.692025] buckets sectors fragmented

[47383.692027] free 15698 0 0

[47383.692028] sb 3 6152 6136

[47383.692030] journal 799 3272704 0

[47383.692031] btree 50447 177857536 28773376

[47383.692033] user 34035 138936152 6416681611432

[47383.692035] cached 39 64797 94947

[47383.692037] parity 0 0 0

[47383.692039] stripe 0 0 0

[47383.692041] need_gc_gens 0 0 0

[47383.692042] need_discard 11618 0 0

[47383.692074] unstriped 0 0 0

[47383.692076] capacity 112639

[47383.692077]

[47383.692078] reserves:

[47383.692080] stripe 3526

[47383.692081] normal 1767

[47383.692083] copygc 8

[47383.692084] btree 4

[47383.692086] btree_copygc 0

[47383.692087] reclaim 0

[47383.692089] interior_updates 0

[47383.692090]

[47383.692091] open buckets 181

[47383.692093] buckets to invalidate 0

 

[47383.692095] Dev 3:

[47383.692097] buckets sectors fragmented

[47383.692098] free 15759 0 0

[47383.692100] sb 3 6152 6136

[47383.692102] journal 799 3272704 0

[47383.692103] btree 48851 172660736 27432960

[47383.692105] user 34627 141336084 6305012486636

[47383.692107] cached 42 105932 12884967988

[47383.692108] parity 0 0 0

[47383.692110] stripe 0 0 0

[47383.692111] need_gc_gens 0 0 0

[47383.692113] need_discard 12558 0 0

[47383.692115] unstriped 0 0 0

[47383.692116] capacity 112639

[47383.692118]

[47383.692119] reserves:

[47383.692120] stripe 3526

[47383.692121] normal 1767

[47383.692123] copygc 8

[47383.692124] btree 4

[47383.692126] btree_copygc 0

[47383.692127] reclaim 0

[47383.692128] interior_updates 0

[47383.692129]

[47383.692131] open buckets 186

[47383.692132] buckets to invalidate 0

 

[47383.692135] Dev 4:

[47383.692136] buckets sectors fragmented

[47383.692138] free 16469 0 0

[47383.692151] sb 3 6152 6136

[47383.692153] journal 799 3272704 0

[47383.692154] btree 51615 182739968 28675072

[47383.692156] user 34978 142724029 2091649619011

[47383.692158] cached 21 36721 8589983887

[47383.692159] parity 0 0 0

[47383.692161] stripe 0 0 0

[47383.692162] need_gc_gens 0 0 0

[47383.692164] need_discard 8754 0 0

[47383.692165] unstriped 0 0 0

[47383.692167] capacity 112639

[47383.692169]

[47383.692170] reserves:

[47383.692171] stripe 3526

[47383.692172] normal 1767

[47383.692174] copygc 8

[47383.692175] btree 4

[47383.692176] btree_copygc 0

[47383.692178] reclaim 0

[47383.692179] interior_updates 0

[47383.692180]

[47383.692181] open buckets 164

[47383.692183] buckets to invalidate 0

 

[47383.692185] Dev 6:

[47383.692187] buckets sectors fragmented

[47383.692188] free 336318 0 0

[47383.692190] sb 1 4104 12280

[47383.692191] journal 1024 16777216 0

[47383.692193] btree 0 0 0

[47383.692194] user 504993 8243339655 358002734456441

[47383.692196] cached 91325 1159652708 132753480768156

[47383.692198] parity 0 0 0

[47383.692199] stripe 0 0 0

[47383.692201] need_gc_gens 0 0 0

[47383.692202] need_discard 4711 0 0

[47383.692204] unstriped 0 0 0

[47383.692205] capacity 938372

[47383.692207]

[47383.692208] reserves:

[47383.692209] stripe 29326

[47383.692211] normal 14664

[47383.692212] copygc 2

[47383.692213] btree 1

[47383.692214] btree_copygc 0

[47383.692216] reclaim 0

[47383.692217] interior_updates 0

[47383.692218]

[47383.692219] open buckets 12

[47383.692221] buckets to invalidate 0

 

[47383.692223] Dev 7:

[47383.692224] buckets sectors fragmented

[47383.692226] free 336374 0 0

[47383.692227] sb 1 4104 12280

[47383.692229] journal 1024 16777216 0

[47383.692231] btree 0 0 0

[47383.692232] user 504746 8239173440 358445116207296

[47383.692234] cached 91618 1162290540 133389138090644

[47383.692235] parity 0 0 0

[47383.692237] stripe 0 0 0

[47383.692238] need_gc_gens 0 0 0

[47383.692240] need_discard 4609 0 0

[47383.692241] unstriped 0 0 0

[47383.692243] capacity 938372

[47383.692244]

[47383.692245] reserves:

[47383.692246] stripe 29326

[47383.692248] normal 14664

[47383.692249] copygc 2

[47383.692250] btree 1

[47383.692251] btree_copygc 0

[47383.692253] reclaim 0

[47383.692254] interior_updates 0

[47383.692255]

[47383.692256] open buckets 12

[47383.692258] buckets to invalidate 0

 

[47383.692260] Dev 8:

[47383.692261] buckets sectors fragmented

[47383.692263] free 454953 0 0

[47383.692264] sb 1 4104 12280

[47383.692266] journal 1024 16777216 0

[47383.692267] btree 0 0 0

[47383.692269] user 338812 5513259951 15453330166865

[47383.692270] cached 138842 1688058017 163471336982367

[47383.692272] parity 0 0 0

[47383.692274] stripe 0 0 0

[47383.692275] need_gc_gens 0 0 0

[47383.692276] need_discard 4740 0 0

[47383.692278] unstriped 0 0 0

[47383.692280] capacity 938372

[47383.692281]

[47383.692282] reserves:

[47383.692283] stripe 29326

[47383.692285] normal 14664

[47383.692294] copygc 2

[47383.692295] btree 1

[47383.692296] btree_copygc 0

[47383.692298] reclaim 0

[47383.692299] interior_updates 0

[47383.692300]

[47383.692301] open buckets 21

[47383.692303] buckets to invalidate 0

 

[47383.692305] Dev 9:

[47383.692306] buckets sectors fragmented

[47383.692308] free 455544 0 0

[47383.692309] sb 1 4104 12280

[47383.692311] journal 1024 16777216 0

[47383.692313] btree 0 0 0

[47383.692314] user 338813 5512145474 15749684041150

[47383.692316] cached 138590 1689275142 164953095353594

[47383.692317] parity 0 0 0

[47383.692319] stripe 0 0 0

[47383.692320] need_gc_gens 0 0 0

[47383.692322] need_discard 4400 0 0

[47383.692323] unstriped 0 0 0

[47383.692325] capacity 938372

[47383.692326]

[47383.692327] reserves:

[47383.692329] stripe 29326

[47383.692330] normal 14664

[47383.692331] copygc 2

[47383.692333] btree 1

[47383.692334] btree_copygc 0

[47383.692335] reclaim 0

[47383.692336] interior_updates 0

[47383.692338]

[47383.692339] open buckets 21

[47383.692340] buckets to invalidate 0

 

[47383.692343] Dev 10:

[47383.692344] buckets sectors fragmented

[47383.692345] free 229288 0 0

[47383.692347] sb 1 4104 28664

[47383.692349] journal 512 16777216 0

[47383.692350] btree 0 0 0

[47383.692352] user 244319 7562377403 4419964815173

[47383.692353] cached 356 4553507 7111901

[47383.692355] parity 0 0 0

[47383.692356] stripe 0 0 0

[47383.692358] need_gc_gens 0 0 0

[47383.692359] need_discard 2390 0 0

[47383.692361] unstriped 0 0 0

[47383.692362] capacity 476866

[47383.692364]

[47383.692365] reserves:

[47383.692366] stripe 14904

[47383.692367] normal 7453

[47383.692368] copygc 2

[47383.692370] btree 1

[47383.692371] btree_copygc 0

[47383.692372] reclaim 0

[47383.692374] interior_updates 0

[47383.692375]

[47383.692376] open buckets 1

[47383.692377] buckets to invalidate 0

 

[47383.692380] Dev 11:

[47383.692381] buckets sectors fragmented

[47383.692382] free 334143 0 0

[47383.692384] sb 1 4104 28664

[47383.692386] journal 512 16777216 0

[47383.692387] btree 0 0 0

[47383.692389] user 140439 4042440162 6382880866846

[47383.692390] cached 0 0 0

[47383.692392] parity 0 0 0

[47383.692393] stripe 0 0 0

[47383.692395] need_gc_gens 0 0 0

[47383.692396] need_discard 1771 0 0

[47383.692398] unstriped 0 0 0

[47383.692399] capacity 476866

[47383.692401]

[47383.692402] reserves:

[47383.692403] stripe 14904

[47383.692404] normal 7453

[47383.692406] copygc 2

[47383.692407] btree 1

[47383.692408] btree_copygc 0

[47383.692409] reclaim 0

[47383.692411] interior_updates 0

[47383.692412]

[47383.692413] open buckets 0

[47383.692414] buckets to invalidate 0

 

[47383.692417] Copygc debug:

[47383.692418] running: 0

[47383.692419] copygc_wait:0

[47383.692421] copygc_wait_at:0

[47383.692423] Currently waiting for:0 B

[47383.692424] Currently waiting since:64.6 TiB

[47383.692426] Currently calculated wait:

[47383.692427] vdb1: 0 B

[47383.692428] vdc1: 0 B

[47383.692429] vdd1: 0 B

[47383.692430] vde1: 0 B

[47383.692432] sdf1: 0 B

[47383.692433] sda1: 0 B

[47383.692434] sdd1: 0 B

[47383.692435] sde1: 0 B

[47383.692436] sdc1: 0 B

[47383.692438] sdb1: 0 B

[47383.692439] [<0>] ret_from_fork+0x10/0x20

 

[47383.692442] Journal debug:

[47383.692443] flags: replay_done,running,may_skip_flush

[47383.692445] dirty journal entries: 2342/32768

[47383.692447] seq: 3982146

[47383.692448] seq_ondisk: 3982145

[47383.692449] last_seq: 3979805

[47383.692451] last_seq_ondisk: 3979805

[47383.692452] flushed_seq_ondisk: 3982145

[47383.692454] watermark: stripe

[47383.692455] each entry reserved: 321

[47383.692457] nr flush writes: 17216

[47383.692458] nr noflush writes: 42516

[47383.692459] average write size: 1.84 MiB

[47383.692461] free buf: 2097152

[47383.692462] nr direct reclaim: 0

[47383.692463] nr background reclaim: 33005313

[47383.692465] reclaim kicked: 0

[47383.692466] reclaim runs in: 24 ms

[47383.692468] blocked: 0

[47383.692469] current entry sectors: 4096

[47383.692471] current entry error: (No error)

[47383.692472] current entry: closed

[47383.692474] unwritten entries:

[47383.692475] seq: 3982146

[47383.692477] io: journal_write_done+0x0/0x450 [bcachefs] r 2

[47383.692478] size: 4.69 KiB

[47383.692480] expires: -9 jiffies

[47383.692481] flags: must_flush separate_flush write_started write_allocated

[47383.692483] last buf closed

[47383.692485] space:

[47383.692486] discarded 4096:5701632

[47383.692488] clean ondisk 4096:16171008

[47383.692489] clean 4096:16171008

[47383.692491] total 4096:16777216

[47383.692493] dev 0:

[47383.692494] durability 1:

[47383.692495] nr 799

[47383.692497] bucket size 4096

[47383.692498] available 0:1

[47383.692500] discard_idx 782

[47383.692501] dirty_ondisk 782 (seq 3979805)

[47383.692503] dirty_idx 782 (seq 3979805)

[47383.692504] cur_idx 780 (seq 3981542)

[47383.692506] dev 1:

[47383.692507] durability 1:

[47383.692508] nr 799

[47383.692510] bucket size 4096

[47383.692511] available 0:0

[47383.692513] discard_idx 537

[47383.692514] dirty_ondisk 537 (seq 3979806)

[47383.692516] dirty_idx 537 (seq 3979806)

[47383.692517] cur_idx 535 (seq 3981543)

[47383.692519] dev 3:

[47383.692520] durability 1:

[47383.692521] nr 799

[47383.692523] bucket size 4096

[47383.692524] available 0:0

[47383.692525] discard_idx 13

[47383.692527] dirty_ondisk 13 (seq 3979806)

[47383.692528] dirty_idx 13 (seq 3979806)

[47383.692530] cur_idx 11 (seq 3981543)

[47383.692531] dev 4:

[47383.692533] durability 1:

[47383.692534] nr 799

[47383.692535] bucket size 4096

[47383.692537] available 0:1

[47383.692539] discard_idx 784

[47383.692540] dirty_ondisk 784 (seq 3979805)

[47383.692546] dirty_idx 784 (seq 3979805)

[47383.692559] cur_idx 782 (seq 3981542)

[47383.692560] dev 6:

[47383.692562] durability 1:

[47383.692563] nr 1024

[47383.692564] bucket size 16384

[47383.692566] available 128:4

[47383.692567] discard_idx 104

[47383.692568] dirty_ondisk 963 (seq 3981556)

[47383.692570] dirty_idx 963 (seq 3981556)

[47383.692571] cur_idx 998 (seq 3982116)

[47383.692573] dev 7:

[47383.692574] durability 1:

[47383.692575] nr 1024

[47383.692577] bucket size 16384

[47383.692578] available 128:6

[47383.692579] discard_idx 107

[47383.692581] dirty_ondisk 966 (seq 3981555)

[47383.692582] dirty_idx 966 (seq 3981555)

[47383.692584] cur_idx 1001 (seq 3982115)

[47383.692585] dev 8:

[47383.692586] durability 1:

[47383.692588] nr 1024

[47383.692589] bucket size 16384

[47383.692591] available 128:3

[47383.692592] discard_idx 159

[47383.692593] dirty_ondisk 1018 (seq 3981554)

[47383.692595] dirty_idx 1018 (seq 3981554)

[47383.692596] cur_idx 29 (seq 3982111)

[47383.692598] dev 9:

[47383.692599] durability 1:

[47383.692600] nr 1024

[47383.692602] bucket size 16384

[47383.692603] available 128:3

[47383.692605] discard_idx 159

[47383.692606] dirty_ondisk 1018 (seq 3981555)

[47383.692608] dirty_idx 1018 (seq 3981555)

[47383.692609] cur_idx 29 (seq 3982112)

[47383.692610] dev 10:

[47383.692612] durability 1:

[47383.692613] nr 512

[47383.692614] bucket size 32768

[47383.692616] available 272:21329

[47383.692624] discard_idx 0

[47383.692626] dirty_ondisk 202 (seq 3981548)

[47383.692628] dirty_idx 202 (seq 3981548)

[47383.692629] cur_idx 238 (seq 3982146)

[47383.692630] dev 11:

[47383.692632] durability 1:

[47383.692633] nr 512

[47383.692634] bucket size 32768

[47383.692636] available 174:17233

[47383.692637] discard_idx 0

[47383.692638] dirty_ondisk 300 (seq 3981547)

[47383.692640] dirty_idx 300 (seq 3981547)

[47383.692641] cur_idx 336 (seq 3982146)

[47383.692643] replicas want 2 need 2