API tools faq
paste
Advertisement
ticpu

bch2 stuck kcompactd

ticpu
Mar 28th, 2024
56
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 6.39 KB | None | 0 0
raw download clone embed print report
  1. ❯ free -h
  2. total used free shared buff/cache available
  3. Mem: 61Gi 48Gi 7.5Gi 135Mi 6.9Gi 13Gi
  4. Swap: 23Gi 4.7Gi 19Gi
  5.  
  6. A new stack trace has appeared in dmesg since then:
  7. [44601.823201] INFO: task kcompactd0:222 blocked for more than 122 seconds.
  8. [44601.823203] Tainted: P W OE 6.8.0-1-bcachefs-git-11743-ga4145ce1e7bc #2
  9. [44601.823204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  10. [44601.823205] task:kcompactd0 state:D stack:0 pid:222 tgid:222 ppid:2 flags:0x00004000
  11. [44601.823207] Call Trace:
  12. [44601.823208] <TASK>
  13. [44601.823210] __schedule+0x3e5/0x14c0
  14. [44601.823216] schedule+0x27/0xe0
  15. [44601.823217] io_schedule+0x46/0x70
  16. [44601.823218] folio_wait_bit_common+0x13e/0x340
  17. [44601.823220] ? filemap_invalidate_unlock_two+0x40/0x40
  18. [44601.823222] migrate_pages_batch+0x5d4/0xc30
  19. [44601.823223] ? defer_compaction+0x80/0x80
  20. [44601.823225] ? isolate_freepages_block+0x440/0x440
  21. [44601.823226] ? set_ptes.constprop.0+0x30/0x30
  22. [44601.823227] ? isolate_freepages_block+0x440/0x440
  23. [44601.823228] migrate_pages+0xb6a/0xe30
  24. [44601.823229] ? defer_compaction+0x80/0x80
  25. [44601.823230] ? isolate_freepages_block+0x440/0x440
  26. [44601.823230] ? isolate_freepages_block+0x440/0x440
  27. [44601.823232] compact_zone+0x9e8/0x10c0
  28. [44601.823233] ? sched_clock_cpu+0xf/0x190
  29. [44601.823235] ? raw_spin_rq_lock_nested+0x1c/0x80
  30. [44601.823237] ? psi_group_change+0x213/0x3b0
  31. [44601.823238] compact_node+0xb7/0x130
  32. [44601.823240] kcompactd+0x351/0x430
  33. [44601.823241] ? do_wait_intr_irq+0x90/0x90
  34. [44601.823243] ? kcompactd_do_work+0x2f0/0x2f0
  35. [44601.823244] kthread+0xcf/0x100
  36. [44601.823246] ? kthread_complete_and_exit+0x20/0x20
  37. [44601.823246] ret_from_fork+0x31/0x50
  38. [44601.823248] ? kthread_complete_and_exit+0x20/0x20
  39. [44601.823249] ret_from_fork_asm+0x11/0x20
  40. [44601.823252] </TASK>
  41.  
  42. [44601.823357] INFO: task ollama:490122 blocked for more than 860 seconds.
  43. [44601.823357] Tainted: P W OE 6.8.0-1-bcachefs-git-11743-ga4145ce1e7bc #2
  44. [44601.823358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  45. [44601.823358] task:ollama state:D stack:0 pid:490122 tgid:490076 ppid:1 flags:0x00004006
  46. [44601.823359] Call Trace:
  47. [44601.823360] <TASK>
  48. [44601.823361] __schedule+0x3e5/0x14c0
  49. [44601.823362] schedule+0x27/0xe0
  50. [44601.823363] io_schedule+0x46/0x70
  51. [44601.823364] folio_wait_bit_common+0x13e/0x340
  52. [44601.823365] ? filemap_invalidate_unlock_two+0x40/0x40
  53. [44601.823366] migrate_pages_batch+0x5d4/0xc30
  54. [44601.823367] ? defer_compaction+0x80/0x80
  55. [44601.823368] ? isolate_freepages_block+0x440/0x440
  56. [44601.823369] ? isolate_freepages_block+0x440/0x440
  57. [44601.823370] migrate_pages+0x99a/0xe30
  58. [44601.823371] ? defer_compaction+0x80/0x80
  59. [44601.823371] ? isolate_freepages_block+0x440/0x440
  60. [44601.823372] ? isolate_freepages_block+0x440/0x440
  61. [44601.823373] compact_zone+0x9e8/0x10c0
  62. [44601.823375] compact_zone_order+0xc7/0x110
  63. [44601.823376] try_to_compact_pages+0xeb/0x2e0
  64. [44601.823378] __alloc_pages_direct_compact+0x92/0x290
  65. [44601.823379] __alloc_pages_slowpath.constprop.0+0x63c/0xd60
  66. [44601.823380] ? refill_stock+0x1a/0x30
  67. [44601.823382] __alloc_pages+0x329/0x360
  68. [44601.823383] alloc_pages_mpol+0xd9/0x1e0
  69. [44601.823385] folio_alloc+0x15/0x30
  70. [44601.823386] page_cache_ra_order+0x12b/0x2b0
  71. [44601.823388] filemap_fault+0x2c6/0xbb0
  72. [44601.823389] ? filemap_map_pages+0x474/0x550
  73. [44601.823393] bch2_page_fault+0x157/0x1d0 [bcachefs 13268a3f50f079785dddb9369fdbc8494a102770]
  74. [44601.823413] __do_fault+0x2f/0x120
  75. [44601.823414] do_fault+0x267/0x490
  76. [44601.823415] __handle_mm_fault+0x817/0xe20
  77. [44601.823416] handle_mm_fault+0x18d/0x320
  78. [44601.823417] __get_user_pages+0x15f/0x6e0
  79. [44601.823419] populate_vma_page_range+0x78/0xb0
  80. [44601.823420] __mm_populate+0x101/0x190
  81. [44601.823421] vm_mmap_pgoff+0x13d/0x1a0
  82. [44601.823424] ksys_mmap_pgoff+0x188/0x1f0
  83. [44601.823425] do_syscall_64+0x83/0x160
  84. [44601.823427] ? syscall_exit_to_user_mode+0x69/0x1f0
  85. [44601.823428] ? do_syscall_64+0x90/0x160
  86. [44601.823429] ? count_memcg_events.constprop.0+0x1a/0x30
  87. [44601.823430] ? handle_mm_fault+0xae/0x320
  88. [44601.823431] ? do_user_addr_fault+0x30b/0x6d0
  89. [44601.823433] ? exc_page_fault+0x7e/0x180
  90. [44601.823434] entry_SYSCALL_64_after_hwframe+0x46/0x4e
  91. [44601.823437] RIP: 0033:0x7f25d21e6a16
  92. [44601.823460] RSP: 002b:00007f2575ffed88 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
  93. [44601.823460] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f25d21e6a16
  94. [44601.823461] RDX: 0000000000000001 RSI: 000000044619d5a0 RDI: 0000000000000000
  95. [44601.823461] RBP: 0000000000008001 R08: 0000000000000009 R09: 0000000000000000
  96. [44601.823462] R10: 0000000000008001 R11: 0000000000000246 R12: 0000000000000000
  97. [44601.823462] R13: ffffffffffffffff R14: 0000000030582b00 R15: 0000000000000009
  98. [44601.823463] </TASK>
  99.  
  100. ❯ ls -l /proc/490122/fd
  101. total 0
  102. lr-x------ 1 ollama ollama 64 28 mar 10:44 0 -> /dev/null
  103. lrwx------ 1 ollama ollama 64 28 mar 10:44 1 -> 'socket:[1243862]'
  104. lrwx------ 1 ollama ollama 64 28 mar 10:44 2 -> 'socket:[1243862]'
  105. lrwx------ 1 ollama ollama 64 28 mar 10:44 3 -> 'socket:[1260661]'
  106. lrwx------ 1 ollama ollama 64 28 mar 10:44 4 -> 'anon_inode:[eventpoll]'
  107. lr-x------ 1 ollama ollama 64 28 mar 10:44 5 -> 'pipe:[1263889]'
  108. l-wx------ 1 ollama ollama 64 28 mar 10:44 6 -> 'pipe:[1263889]'
  109. lrwx------ 1 ollama ollama 64 28 mar 10:44 7 -> 'socket:[1296324]'
  110. lr-x------ 1 ollama ollama 64 28 mar 10:44 8 -> /mnt/bcachefs/var/lib/ollama/.ollama/models/blobs/sha256:83b45bda27326a4e5402e61f7ceb67f735729332ae2714f5fe857f117fb63445
  111. lr-x------ 1 ollama ollama 64 28 mar 10:44 9 -> /mnt/bcachefs/var/lib/ollama/.ollama/models/blobs/sha256:83b45bda27326a4e5402e61f7ceb67f735729332ae2714f5fe857f117fb63445
  112.  
  113. $ less /proc/490122/smaps
  114. 7f2065e62000-7f24ac000000 r--s 00000000 fe:17 269124205 /mnt/bcachefs/var/lib/ollama/.ollama/models/blobs/sha256:83b45bda27326a4e5402e61f7ceb67f735729332ae2714f5fe857f117fb63445
  115. Size: 17925752 kB
  116. KernelPageSize: 4 kB
  117. MMUPageSize: 4 kB
  118. Rss: 1412344 kB
  119. Pss: 1412344 kB
  120. Pss_Dirty: 0 kB
  121. Shared_Clean: 0 kB
  122. Shared_Dirty: 0 kB
  123. Private_Clean: 1412344 kB
  124. Private_Dirty: 0 kB
  125. Referenced: 46632 kB
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!