DF-2660 / attempts_fghi.console.txt
DF-2660 race attempts f/g/h/i -- console & telemetry (RECONSTRUCTED from the
verification session transcript; the guest-side /root/poc/df2660/run_*.log and
the serial boot.log were lost when the disk was reverted to the with-src
snapshot for the fix-validation build. Content below is verbatim from the
tool outputs of the session; line numbers refer to the then-current boot.log.)
ATTEMPT F (vn0-2, label f, N=200, 6 parked full-speed hloops unleashed at
first 'insert rec' via syslog; see trigger history in git if needed)
run_f.log telemetry (guest, post-crash fetch):
H2NOD_COUNT=3 (real 3-master cluster, 3 sync threads)
BACKLOG_SEEN at 1792043354 poll=0 -- UNLEASHING
inserts_seen (grep -c 'insert rec par=' /var/log/messages) = 400
HLOOP sweep completions: 0 (loops mid-sweep at crash)
serial (boot.log lines ~820-860), attempt f panic:
replace rec ... slave 2 1.0000000000000418 mod=00000000000002a4 (deferred
inode finals descending 0x418..0x400, then the PFS-root inode)
replace rec ... slave 2 1.0000000000000400 mod=000000000000025c
replace rec 0xfffff8011b861080 slave 2 1.951a27db32c18000 mod=...04b1
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x118
current process = Idle (kernel thread = xop worker)
panic: page fault
Trace:
hammer2_inode_chain_and_parent.cold.9() at +0x15 0xffffffff802aad04
hammer2_chain_inode_find() at +0x97 0xffffffff80974e57
boot() called on cpu#4 ; Dumping 746 MB ... Dump complete
ATTEMPT G (hash-order forward sweeps, debug=0x8000)
serial (lines ~600-645): deferred passes START_SCAN IP=0x415..0x406, then
identical Fatal trap 12 / 0x118 / panic: page fault /
hammer2_inode_chain_and_parent.cold.9 +0x15 / hammer2_chain_inode_find+0x97,
Uptime: 2m44s, Dump complete.
ATTEMPT H (same as g): crash even earlier -- first deferred inode
(START_SCAN IP=0x400), "sync_slaves error 0 defer 0", then the identical
0x118 / chain_and_parent.cold.9 / chain_inode_find+0x97 trace, dump written.
ATTEMPT I (REVERSED hash-order sweeps, head-on meeting with the insertion
frontier): identical crash at Uptime: 42s --
Fatal trap 12 ... fault virtual address = 0x118
hammer2_inode_chain_and_parent.cold.9() at +0x15
hammer2_chain_inode_find() at +0x97
boot() called on cpu#2 ; Dumping 762 MB ... Dump complete
ATTEMPTS A-E (rig bring-up, documented for completeness):
a: full-speed hammer; mounts formed (0/64/64) but m2/m3 subdirs were hidden
under the mounted $M -> M2/M3 were mount(8) usage-error 64, NOT EBUSY
merges -> no 3-master cluster; hammer exhausted the solo A column
(ENOSPC). (Mountpoint bug, later fixed with separate top-level dirs.)
b: vn3 held by attempt a -> VN_ATTACH guard aborted cleanly.
c: /dev/vn6 node did not exist -> VN_ATTACH guard aborted cleanly.
d: 0.1s/pair throttle, real cluster (H2NOD threads verified), but
inserts_seen=0 for 90s: the sync thread's ipcluster collect catches
A/B root-mtid feed skew -> HAMMER2_ERROR_EINPROGRESS -> 60s 'h2coll'
park per collect; the catch-up pass never ran under churn.
e: mountpoint bug again (m2/m3), plus dmesg msgbuf saturated
("dmesg: sysctl kern.msgbuf: Cannot allocate memory"); backlog
detection moved to /var/log/messages.