ZZ / Gemini-ZFS-Fix.md

When you have 128 GB of RAM, ZFS defaults to a 64 GB ARC max ceiling. If your Windows VM pins 96 GB via hugepages, your system is mathematically trying to commit 160 GB of RAM on a 128 GB physical machine. Since hugepages cannot be unpinned, the host kernel starves, leading directly to the brutal cascading page faults and segmentation faults you experienced. Technical Case Notes: - Root Cause 1 (Kernel Panic): An interface bug between Linux Kernel 6.18 code allocation profiling hooks (CONFIG_MEM_ALLOC_PROFILING) and the out-of-tree OpenZFS driver module. Background ZFS I/O workers drop pointers during slab memory cleanups, triggering __alloc_tagging_slab_free_hook page faults. - Root Cause 2 (Segfaults / Exit Code 139): Core memory allocation collision. Pinned Windows VM static hugepages (96 GB) combined with default OpenZFS ARC ceilings (50% of host RAM = 64 GB) exceeded physical RAM capacity (128 GB). This starved the host kernel during intensive parallel nixos-rebuild compiler routines, triggering segmentation faults across Python, Sphinx, and VS Code extraction tools. - Neutralize Kernel Allocation Profiling HooksForce the Linux kernel to bypass the unpatched memory tracking hooks that conflict with out-of-tree file system drivers.Action: Upgrade to a fixed minor kernel release (e.g., 6.18.48+) and append `alloc_tag_boot=off` directly to the boot loader command line parameters. ``` # Disable hugepages before recompiling kernel sudo sysctl -w vm.nr_hugepages=0 sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches sudo echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled ``` ZFS Notes - ZFS ARC Mem Use: `zarcstat` - ZFS ARC Summary: `zarcsummary` - ZFS ARC Size Floor: `cat /sys/module/zfs/parameters/zfs_arc_min` - ZFS ARC Size Ceiling: `cat /sys/module/zfs/parameters/zfs_arc_max` - ZFS ARC with alloc 50% of system mem by default, which could crash into a memory hugepage. - `cat /proc/spl/kstat/zfs/arcstats` ``` boot.kernelParams = [ # Balance Hugepage and ZFS ARC mem alloc limits directly via the kernel command line "hugepages=96" "hugepagesz=1G" "default_hugepagesz=1G" "transparent_hugepage=never" # Blocks the host kernel from modifying or merging memory blocks behind the back of OpenZFS and your VM hypervisor. "zfs.zfs_arc_min=4294967296" "zfs.zfs_arc_max=12884901888" "alloc_tag_boot=off" ] ```

[ 647.370841 < 0.000005>] r8169 0000:27:00.0 enp39s0: entered allmulticast mode 15:13 [37/1420] [ 4934.380323 < 4287.009482>] BUG: unable to handle page fault for address: ffffff7fc0306250 [ 4934.380332 < 0.000009>] #PF: supervisor read access in kernel mode [ 4934.380335 < 0.000003>] #PF: error_code(0x0000) - not-present page [ 4934.380337 < 0.000002>] PGD 3d16067 P4D 3d16067 PUD 0 [ 4934.380341 < 0.000004>] Oops: Oops: 0000 [#1] SMP NOPTI [ 4934.380345 < 0.000004>] CPU: 30 UID: 0 PID: 3151 Comm: z_wr_int_2 Tainted: P O 6.18.48 #1-NixOS PREEMPT(lazy) [ 4934.380349 < 0.000004>] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE [ 4934.380351 < 0.000002>] Hardware name: Micro-Star International Co., Ltd. MS-7C35/MEG X570 ACE (MS-7C35), BIOS 1.S1 09/22/2025 [ 4934.380354 < 0.000003>] RIP: 0010:__alloc_tagging_slab_free_hook+0xac/0xe0 [ 4934.380361 < 0.000007>] Code: 2c 48 c1 ea 20 29 d0 d3 e8 0f b6 4f 2d 01 d0 d3 e8 48 c1 e0 04 4c 01 c8 48 83 f8 f8 74 28 48 8b 50 08 48 85 d2 74 1f 8b 4f 20 <4c> 8b 5a 20 48 f7 d9 65 49 01 0b 48 8b 52 20 65 48 ff 4a 08 48 c7 [ 4934.380364 < 0.000003>] RSP: 0018:ffffc916e86d7bc8 EFLAGS: 00010286 [ 4934.380367 < 0.000003>] RAX: ffff88a0e424c630 RBX: ffff88a1b659f368 RCX: 0000000000000178 [ 4934.380369 < 0.000002>] RDX: ffffff7fc0306230 RSI: ffffc916e86d7bf0 RDI: ffff889e5848bf00 [ 4934.380371 < 0.000002>] RBP: ffffc916e86d7c20 R08: fffff6fa51d96700 R09: ffff88a0e424c400 [ 4934.380373 < 0.000002>] R10: ffffc916e86d7bf8 R11: 8113070f005f00ff R12: ffff889e5848bf00 [ 4934.380374 < 0.000001>] R13: ffff88a04d210ec0 R14: ffff889fd7d6d300 R15: 0000000000000000 [ 4934.380376 < 0.000002>] FS: 0000000000000000(0000) GS:ffff88bdd023f000(0000) knlGS:0000000000000000 [ 4934.380379 < 0.000003>] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4934.380381 < 0.000002>] CR2: ffffff7fc0306250 CR3: 0000000179caa000 CR4: 0000000000350ef0 [ 4934.380383 < 0.000002>] Call Trace: [ 4934.380385 < 0.000002>] <TASK> [ 4934.380386 < 0.000001>] kmem_cache_free+0xc8/0x4b0 [ 4934.380392 < 0.000006>] ? dbuf_rele_and_unlock+0x118/0x560 [zfs] [ 4934.380550 < 0.000158>] ? spl_kmem_cache_free+0x13c/0x200 [spl] [ 4934.380559 < 0.000009>] ? dsl_pool_undirty_space.part.0+0x8f/0xb0 [zfs] [ 4934.380705 < 0.000146>] spl_kmem_cache_free+0x13c/0x200 [spl] [ 4934.380713 < 0.000008>] arc_write_done+0xb4/0x560 [zfs] [ 4934.380845 < 0.000132>] zio_done+0x53a/0x1030 [zfs] [ 4934.380979 < 0.000134>] ? spa_config_exit+0x4a/0xa0 [zfs] [ 4934.381111 < 0.000132>] zio_execute+0x8c/0x130 [zfs] [ 4934.381229 < 0.000118>] taskq_thread+0x363/0x740 [spl] [ 4934.381238 < 0.000009>] ? __pfx_default_wake_function+0x10/0x10 [ 4934.381242 < 0.000004>] ? __pfx_zio_execute+0x10/0x10 [zfs] [ 4934.381354 < 0.000112>] ? __pfx_taskq_thread+0x10/0x10 [spl] [ 4934.381361 < 0.000007>] kthread+0xfb/0x260 [ 4934.381365 < 0.000004>] ? __pfx_kthread+0x10/0x10 [ 4934.381368 < 0.000003>] ret_from_fork+0x22d/0x260 [ 4934.381373 < 0.000005>] ? __pfx_kthread+0x10/0x10 [ 4934.381375 < 0.000002>] ret_from_fork_asm+0x1a/0x30 [ 4934.381379 < 0.000004>] </TASK> [ 4934.381381 < 0.000002>] Modules linked in: vhost_net vhost vhost_iotlb macvtap macvlan tap vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd nf_conntrack_netlink xfrm_user xfrm_algo xt_set ip_set xt_addrtype overlay af_packet xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJEC T nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_tables nfnetlink wireguard libcurve25519 libchacha20poly1305 libchacha ip6_udp_tunnel udp_tunnel libpoly1305 cdc_ncm cdc_ether usbnet mii ipheth cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 nls_cp437 vfat fat btusb b trtl btintel btmtk btbcm bluetooth ecdh_generic rfkill ecc crc16 hid_cmedia gspca_ov534 gspca_main videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common r8169 igb realtek mdio_devres of_mdio fixed_phy fwnode_mdio ptp libphy pps_core dca i2c_algo_bit onboard_usb_dev mdio _bus tiny_power_button hid_logitech_hidpp snd_hda_codec_alc882 hid_apple snd_hda_codec_realtek_lib snd_hda_codec_generic snd_usb_audio snd_hda_intel snd_ump snd_usbmidi_lib snd_hda_codec snd_rawmidi [ 4934.381427 < 0.000046>] snd_seq_device snd_hda_core snd_intel_dspcfg snd_intel_sdw_acpi edac_mce_amd snd_hwdep intel_rapl_msr battery edac_core snd_pcm amd_atl wmi_bmof mxm_wmi intel_rapl_common sp5100_tco apple_mfi_fastcharge snd_timer polyval_clmulni watchdog ghash_clmulni_ intel snd aesni_intel soundcore k10temp rapl wmi rtc_cmos button ee1004 evdev input_leds mousedev joydev mac_hid sch_fq_codel kvm_amd ccp v4l2loopback(O) kvm xt_nat x_tables nf_nat nf_conntrack videodev br_netfilter nf_defrag_ipv6 nf_defrag_ipv4 atkbd mc bridge i2c_piix4 irqbypass l ed_class libps2 serio i2c_smbus i2c_dev vivaldi_fmap stp llc tun fuse loop veth configfs dmi_sysfs hid_logitech_dj hid_generic sd_mod usbhid uas usb_storage hid ahci libahci libata nvme nvme_core scsi_mod xhci_pci nvme_keyring nvme_auth scsi_common xhci_hcd hkdf zfs(PO) dm_mod spl(O ) efivarfs autofs4 [ 4934.381490 < 0.000063>] CR2: ffffff7fc0306250 [ 4934.381493 < 0.000003>] ---[ end trace 0000000000000000 ]--- [ 4934.495914 < 0.114421>] RIP: 0010:__alloc_tagging_slab_free_hook+0xac/0xe0 [ 4934.495928 < 0.000014>] Code: 2c 48 c1 ea 20 29 d0 d3 e8 0f b6 4f 2d 01 d0 d3 e8 48 c1 e0 04 4c 01 c8 48 83 f8 f8 74 28 48 8b 50 08 48 85 d2 74 1f 8b 4f 20 <4c> 8b 5a 20 48 f7 d9 65 49 01 0b 48 8b 52 20 65 48 ff 4a 08 48 c7 [ 4934.495933 < 0.000005>] RSP: 0018:ffffc916e86d7bc8 EFLAGS: 00010286 [ 4934.495935 < 0.000002>] RAX: ffff88a0e424c630 RBX: ffff88a1b659f368 RCX: 0000000000000178 [ 4934.495937 < 0.000002>] RDX: ffffff7fc0306230 RSI: ffffc916e86d7bf0 RDI: ffff889e5848bf00 [ 4934.495939 < 0.000002>] RBP: ffffc916e86d7c20 R08: fffff6fa51d96700 R09: ffff88a0e424c400 [ 4934.495941 < 0.000002>] R10: ffffc916e86d7bf8 R11: 8113070f005f00ff R12: ffff889e5848bf00 [ 4934.495943 < 0.000002>] R13: ffff88a04d210ec0 R14: ffff889fd7d6d300 R15: 0000000000000000 [ 4934.495945 < 0.000002>] FS: 0000000000000000(0000) GS:ffff88bdd023f000(0000) knlGS:0000000000000000 [ 4934.495947 < 0.000002>] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4934.495949 < 0.000002>] CR2: ffffff7fc0306250 CR3: 0000000003224000 CR4: 0000000000350ef0 [ 4934.495951 < 0.000002>] note: z_wr_int_2[3151] exited with irqs disabled