Gemini-ZFS-Fix.md
· 2.4 KiB · Markdown
Orginalformat
When you have 128 GB of RAM, ZFS defaults to a 64 GB ARC max ceiling. If your Windows VM pins 96 GB via hugepages, your system is mathematically trying to commit 160 GB of RAM on a 128 GB physical machine.
Since hugepages cannot be unpinned, the host kernel starves, leading directly to the brutal cascading page faults and segmentation faults you experienced.
Technical Case Notes:
- Root Cause 1 (Kernel Panic): An interface bug between Linux Kernel 6.18 code allocation profiling hooks (CONFIG_MEM_ALLOC_PROFILING) and the out-of-tree OpenZFS driver module. Background ZFS I/O workers drop pointers during slab memory cleanups, triggering __alloc_tagging_slab_free_hook page faults.
- Root Cause 2 (Segfaults / Exit Code 139): Core memory allocation collision. Pinned Windows VM static hugepages (96 GB) combined with default OpenZFS ARC ceilings (50% of host RAM = 64 GB) exceeded physical RAM capacity (128 GB). This starved the host kernel during intensive parallel nixos-rebuild compiler routines, triggering segmentation faults across Python, Sphinx, and VS Code extraction tools.
- Neutralize Kernel Allocation Profiling HooksForce the Linux kernel to bypass the unpatched memory tracking hooks that conflict with out-of-tree file system drivers.Action: Upgrade to a fixed minor kernel release (e.g., 6.18.48+) and append alloc_tag_boot=off directly to the boot loader command line parameters.
```
# Disable hugepages before recompiling kernel
sudo sysctl -w vm.nr_hugepages=0
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
```
ZFS Notes
- ZFS ARC Mem Use: `zarcstat`
- ZFS ARC Summary: `zarcsummary`
- ZFS ARC Size Floor: `cat /sys/module/zfs/parameters/zfs_arc_min`
- ZFS ARC Size Ceiling: `cat /sys/module/zfs/parameters/zfs_arc_max`
- ZFS ARC with alloc 50% of system mem by default, which could crash into a memory hugepage.
- Balance Hugepages and ZFS ARC mem directly via the kernel command line
- `cat /proc/spl/kstat/zfs/arcstats`
```
boot.kernelParams = [
# Balance Hugepage and ZFS ARC mem alloc limits directly via kernel command line
"hugepages=96"
"hugepagesz=1G"
"default_hugepagesz=1G"
"transparent_hugepage=never" # Blocks the host kernel from modifying or merging memory blocks behind the back of OpenZFS and your VM hypervisor.
"zfs.zfs_arc_min=4294967296"
"zfs.zfs_arc_max=12884901888"
]
```
When you have 128 GB of RAM, ZFS defaults to a 64 GB ARC max ceiling. If your Windows VM pins 96 GB via hugepages, your system is mathematically trying to commit 160 GB of RAM on a 128 GB physical machine. Since hugepages cannot be unpinned, the host kernel starves, leading directly to the brutal cascading page faults and segmentation faults you experienced.
Technical Case Notes:
- Root Cause 1 (Kernel Panic): An interface bug between Linux Kernel 6.18 code allocation profiling hooks (CONFIG_MEM_ALLOC_PROFILING) and the out-of-tree OpenZFS driver module. Background ZFS I/O workers drop pointers during slab memory cleanups, triggering __alloc_tagging_slab_free_hook page faults.
- Root Cause 2 (Segfaults / Exit Code 139): Core memory allocation collision. Pinned Windows VM static hugepages (96 GB) combined with default OpenZFS ARC ceilings (50% of host RAM = 64 GB) exceeded physical RAM capacity (128 GB). This starved the host kernel during intensive parallel nixos-rebuild compiler routines, triggering segmentation faults across Python, Sphinx, and VS Code extraction tools.
- Neutralize Kernel Allocation Profiling HooksForce the Linux kernel to bypass the unpatched memory tracking hooks that conflict with out-of-tree file system drivers.Action: Upgrade to a fixed minor kernel release (e.g., 6.18.48+) and append alloc_tag_boot=off directly to the boot loader command line parameters.
# Disable hugepages before recompiling kernel
sudo sysctl -w vm.nr_hugepages=0
sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
ZFS Notes
- ZFS ARC Mem Use:
zarcstat - ZFS ARC Summary:
zarcsummary - ZFS ARC Size Floor:
cat /sys/module/zfs/parameters/zfs_arc_min - ZFS ARC Size Ceiling:
cat /sys/module/zfs/parameters/zfs_arc_max - ZFS ARC with alloc 50% of system mem by default, which could crash into a memory hugepage.
- Balance Hugepages and ZFS ARC mem directly via the kernel command line
cat /proc/spl/kstat/zfs/arcstats
boot.kernelParams = [
# Balance Hugepage and ZFS ARC mem alloc limits directly via kernel command line
"hugepages=96"
"hugepagesz=1G"
"default_hugepagesz=1G"
"transparent_hugepage=never" # Blocks the host kernel from modifying or merging memory blocks behind the back of OpenZFS and your VM hypervisor.
"zfs.zfs_arc_min=4294967296"
"zfs.zfs_arc_max=12884901888"
]