ZZ / Gemini-ZFS-Fix.md

When you have 128 GB of RAM, ZFS defaults to a 64 GB ARC max ceiling. If your Windows VM pins 96 GB via hugepages, your system is mathematically trying to commit 160 GB of RAM on a 128 GB physical machine. Since hugepages cannot be unpinned, the host kernel starves, leading directly to the brutal cascading page faults and segmentation faults you experienced. Technical Case Notes: - Root Cause 1 (Kernel Panic): An interface bug between Linux Kernel 6.18 code allocation profiling hooks (CONFIG_MEM_ALLOC_PROFILING) and the out-of-tree OpenZFS driver module. Background ZFS I/O workers drop pointers during slab memory cleanups, triggering __alloc_tagging_slab_free_hook page faults. - Root Cause 2 (Segfaults / Exit Code 139): Core memory allocation collision. Pinned Windows VM static hugepages (96 GB) combined with default OpenZFS ARC ceilings (50% of host RAM = 64 GB) exceeded physical RAM capacity (128 GB). This starved the host kernel during intensive parallel nixos-rebuild compiler routines, triggering segmentation faults across Python, Sphinx, and VS Code extraction tools. - Neutralize Kernel Allocation Profiling HooksForce the Linux kernel to bypass the unpatched memory tracking hooks that conflict with out-of-tree file system drivers.Action: Upgrade to a fixed minor kernel release (e.g., 6.18.48+) and append `alloc_tag_boot=off` directly to the boot loader command line parameters. ``` # Disable hugepages before recompiling kernel sudo sysctl -w vm.nr_hugepages=0 sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches sudo echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled ``` ZFS Notes - ZFS ARC Mem Use: `zarcstat` - ZFS ARC Summary: `zarcsummary` - ZFS ARC Size Floor: `cat /sys/module/zfs/parameters/zfs_arc_min` - ZFS ARC Size Ceiling: `cat /sys/module/zfs/parameters/zfs_arc_max` - ZFS ARC with alloc 50% of system mem by default, which could crash into a memory hugepage. - `cat /proc/spl/kstat/zfs/arcstats` ``` boot.kernelParams = [ # Balance Hugepage and ZFS ARC mem alloc limits directly via the kernel command line "hugepages=96" "hugepagesz=1G" "default_hugepagesz=1G" "transparent_hugepage=never" # Blocks the host kernel from modifying or merging memory blocks behind the back of OpenZFS and your VM hypervisor. "zfs.zfs_arc_min=4294967296" "zfs.zfs_arc_max=12884901888" "alloc_tag_boot=off" ] ```

[ 4934.380323 < 4287.009482>] BUG: unable to handle page fault for address: ffffff7fc0306250 [ 4934.380332 < 0.000009>] #PF: supervisor read access in kernel mode [ 4934.380335 < 0.000003>] #PF: error_code(0x0000) - not-present page [ 4934.380337 < 0.000002>] PGD 3d16067 P4D 3d16067 PUD 0 [ 4934.380341 < 0.000004>] Oops: Oops: 0000 [#1] SMP NOPTI [ 4934.380345 < 0.000004>] CPU: 30 UID: 0 PID: 3151 Comm: z_wr_int_2 Tainted: P O 6.18.48 #1-NixOS PREEMPT(lazy) [ 4934.380349 < 0.000004>] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE [ 4934.380351 < 0.000002>] Hardware name: Micro-Star International Co., Ltd. MS-7C35/MEG X570 ACE (MS-7C35), BIOS 1.S1 09/22/2025 [ 4934.380354 < 0.000003>] RIP: 0010:__alloc_tagging_slab_free_hook+0xac/0xe0 [ 4934.380361 < 0.000007>] Code: 2c 48 c1 ea 20 29 d0 d3 e8 0f b6 4f 2d 01 d0 d3 e8 48 c1 e0 04 4c 01 c8 48 83 f8 f8 74 28 48 8b 50 08 48 85 d2 74 1f 8b 4f 20 <4c> 8b 5a 20 48 f7 d9 65 49 01 0b 48 8b 52 20 65 48 ff 4a 08 48 c7 [ 4934.380364 < 0.000003>] RSP: 0018:ffffc916e86d7bc8 EFLAGS: 00010286 [ 4934.380367 < 0.000003>] RAX: ffff88a0e424c630 RBX: ffff88a1b659f368 RCX: 0000000000000178 [ 4934.380369 < 0.000002>] RDX: ffffff7fc0306230 RSI: ffffc916e86d7bf0 RDI: ffff889e5848bf00 [ 4934.380371 < 0.000002>] RBP: ffffc916e86d7c20 R08: fffff6fa51d96700 R09: ffff88a0e424c400 [ 4934.380373 < 0.000002>] R10: ffffc916e86d7bf8 R11: 8113070f005f00ff R12: ffff889e5848bf00 [ 4934.380374 < 0.000001>] R13: ffff88a04d210ec0 R14: ffff889fd7d6d300 R15: 0000000000000000 [ 4934.380376 < 0.000002>] FS: 0000000000000000(0000) GS:ffff88bdd023f000(0000) knlGS:0000000000000000 [ 4934.380379 < 0.000003>] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4934.380381 < 0.000002>] CR2: ffffff7fc0306250 CR3: 0000000179caa000 CR4: 0000000000350ef0 [ 4934.380383 < 0.000002>] Call Trace: [ 4934.380385 < 0.000002>] <TASK> [ 4934.380386 < 0.000001>] kmem_cache_free+0xc8/0x4b0 [ 4934.380392 < 0.000006>] ? dbuf_rele_and_unlock+0x118/0x560 [zfs] [ 4934.380550 < 0.000158>] ? spl_kmem_cache_free+0x13c/0x200 [spl] [ 4934.380559 < 0.000009>] ? dsl_pool_undirty_space.part.0+0x8f/0xb0 [zfs] [ 4934.380705 < 0.000146>] spl_kmem_cache_free+0x13c/0x200 [spl] [ 4934.380713 < 0.000008>] arc_write_done+0xb4/0x560 [zfs] [ 4934.380845 < 0.000132>] zio_done+0x53a/0x1030 [zfs] [ 4934.380979 < 0.000134>] ? spa_config_exit+0x4a/0xa0 [zfs] [ 4934.381111 < 0.000132>] zio_execute+0x8c/0x130 [zfs] [ 4934.381229 < 0.000118>] taskq_thread+0x363/0x740 [spl] [ 4934.381238 < 0.000009>] ? __pfx_default_wake_function+0x10/0x10 [ 4934.381242 < 0.000004>] ? __pfx_zio_execute+0x10/0x10 [zfs] [ 4934.381354 < 0.000112>] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 4934.381361 < 0.000007>] kthread+0xfb/0x260 [ 4934.381365 < 0.000004>] ? __pfx_kthread+0x10/0x10 [ 4934.381368 < 0.000003>] ret_from_fork+0x22d/0x260 [ 4934.381373 < 0.000005>] ? __pfx_kthread+0x10/0x10 [ 4934.381375 < 0.000002>] ret_from_fork_asm+0x1a/0x30 [ 4934.381379 < 0.000004>] </TASK>