⬢ DragonFlyBSD Kernel Audit
DF-2660 / attempts_fghi.console.txt
← back to finding ↓ download raw
DF-2660 race attempts f/g/h/i -- console & telemetry (RECONSTRUCTED from the
verification session transcript; the guest-side /root/poc/df2660/run_*.log and
the serial boot.log were lost when the disk was reverted to the with-src
snapshot for the fix-validation build.  Content below is verbatim from the
tool outputs of the session; line numbers refer to the then-current boot.log.)

ATTEMPT F (vn0-2, label f, N=200, 6 parked full-speed hloops unleashed at
first 'insert rec' via syslog; see trigger history in git if needed)

run_f.log telemetry (guest, post-crash fetch):
  H2NOD_COUNT=3                       (real 3-master cluster, 3 sync threads)
  BACKLOG_SEEN at 1792043354 poll=0 -- UNLEASHING
  inserts_seen (grep -c 'insert rec par=' /var/log/messages) = 400
  HLOOP sweep completions: 0 (loops mid-sweep at crash)

serial (boot.log lines ~820-860), attempt f panic:
  replace rec ... slave 2 1.0000000000000418 mod=00000000000002a4   (deferred
  inode finals descending 0x418..0x400, then the PFS-root inode)
  replace rec ... slave 2 1.0000000000000400 mod=000000000000025c
  replace rec 0xfffff8011b861080 slave 2 1.951a27db32c18000 mod=...04b1

  Fatal trap 12: page fault while in kernel mode
  fault virtual address  = 0x118
  current process        = Idle          (kernel thread = xop worker)
  panic: page fault
  Trace:
  hammer2_inode_chain_and_parent.cold.9() at +0x15   0xffffffff802aad04
  hammer2_chain_inode_find() at +0x97                0xffffffff80974e57
  boot() called on cpu#4 ; Dumping 746 MB ... Dump complete

ATTEMPT G (hash-order forward sweeps, debug=0x8000)
serial (lines ~600-645): deferred passes START_SCAN IP=0x415..0x406, then
identical Fatal trap 12 / 0x118 / panic: page fault /
hammer2_inode_chain_and_parent.cold.9 +0x15 / hammer2_chain_inode_find+0x97,
Uptime: 2m44s, Dump complete.

ATTEMPT H (same as g): crash even earlier -- first deferred inode
(START_SCAN IP=0x400), "sync_slaves error 0 defer 0", then the identical
0x118 / chain_and_parent.cold.9 / chain_inode_find+0x97 trace, dump written.

ATTEMPT I (REVERSED hash-order sweeps, head-on meeting with the insertion
frontier): identical crash at Uptime: 42s --
  Fatal trap 12 ... fault virtual address = 0x118
  hammer2_inode_chain_and_parent.cold.9() at +0x15
  hammer2_chain_inode_find() at +0x97
  boot() called on cpu#2 ; Dumping 762 MB ... Dump complete

ATTEMPTS A-E (rig bring-up, documented for completeness):
  a: full-speed hammer; mounts formed (0/64/64) but m2/m3 subdirs were hidden
     under the mounted $M -> M2/M3 were mount(8) usage-error 64, NOT EBUSY
     merges -> no 3-master cluster; hammer exhausted the solo A column
     (ENOSPC).  (Mountpoint bug, later fixed with separate top-level dirs.)
  b: vn3 held by attempt a -> VN_ATTACH guard aborted cleanly.
  c: /dev/vn6 node did not exist -> VN_ATTACH guard aborted cleanly.
  d: 0.1s/pair throttle, real cluster (H2NOD threads verified), but
     inserts_seen=0 for 90s: the sync thread's ipcluster collect catches
     A/B root-mtid feed skew -> HAMMER2_ERROR_EINPROGRESS -> 60s 'h2coll'
     park per collect; the catch-up pass never ran under churn.
  e: mountpoint bug again (m2/m3), plus dmesg msgbuf saturated
     ("dmesg: sysctl kern.msgbuf: Cannot allocate memory"); backlog
     detection moved to /var/log/messages.