ZZ / Qwen38-MTP-NInfer-Install.md

## Install NInfer Server for fast Qwen3.8-27B inference NInfer installs are tied to a specific GPU. - 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build - 4090 - UDPSendToFailed/ninfer-4090 - NInfer port for Qwen 3.8 27B on RTX 4090 - https://github.com/UDPSendToFailed/ninfer-4090/tree/feat/rtx-4090-sm89-native - 4090 Linux Docker - sergiuszm/ninfer-4090 - Qwen3.8-27B on one RTX 4090 - full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots - https://github.com/sergiuszm/ninfer-4090#quick-start-linux- 3090 - - 3090 - Don-Chad/ninfer-3090 - Fast Qwen3.8-27B inference on one RTX 3090 - ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds. - https://github.com/Don-Chad/ninfer-3090 ### NInfer install - Setup file structure - Compile and install vcpkg - Compile and install curl from vcpkg - Install FFMPEG from Chocolatey - Configure Ninfer - Build NInfer - Run `ninfer-serve.exe` - Query `ninfer-serve.exe` File Structure ```sh H:/AI/Code/ninfer-$gpu/ H:/AI/Code/Tool/vcpkg/ H:/AI/Code/Model/qwen3_8_27b.ninfer # 18GB H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically sets the CUDA_PATH in Windows system env variables ``` Install vcpkg.exe ```sh git clone https://github.com/Microsoft/vcpkg git fetch --unshallow # required for vcpkg to install pkgs git pull cd vcpkg ./bootstrap-vcpkg.sh ./vcpkg.exe integrate install ./vcpkg.exe install curl ``` Configure Ninfer with FFMPEG from Chocolatey and curl compiled by MSVC ```sh cd H:/AI/Code/ninfer-4090 # Clear your build folder build-sm89 then configure. time cmake -B build-sm89 -G "Visual Studio 17 2022" -A x64 \ -DCMAKE_CUDA_ARCHITECTURES=89 \ -DCMAKE_CUDA_FLAGS="--generate-code=arch=compute_89,code=sm_89" \ -DNINFER_BUILD_APPS=ON \ -DNINFER_BUILD_BENCHMARKS=ON \ -DCMAKE_PREFIX_PATH="C:/Users/IT/AppData/Local/UniGetUI/Chocolatey/lib/ffmpeg-shared/tools/ffmpeg-9.0.1-full_build-shared;H:/AI/Code/Tool/vcpkg/packages/curl_x64-windows" \ -DCMAKE_TOOLCHAIN_FILE="H:/AI/Code/Tool/vcpkg/scripts/buildsystems/vcpkg.cmake" \ -DCOMPILE_DEFINITIONS="CURL_STATICLIB" ``` Build NInfer ~20mins ```sh time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninfer_bench --parallel 32 ``` #### Run NInfer Server ```sh time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 --port 8080 ``` Open port 8080 for LAN access to ninfer-serve.exe ```ps1 New-NetFirewallRule -DisplayName "NInfer Server (8080)" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8080 ``` ### Query NInfer Server Curl ```sh curl http://$AI_HOST:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "qwen3.8-27b", "messages": [{"role": "user", "content": "Reply with one short sentence."}], "max_tokens": 64 }' ``` Aider Chat ```sh # aider.oai: alias aoai="aider.openai" aider.openai(){ aider_files="$@" local ai_model="qwen3.8-27b" local ollama_api_base_aider="http://$AI_HOST:8080/v1" local OPENAI_API_KEY="dummy" time aider $aider_files \ --no-show-model-warnings \ --model "openai/$ai_model" \ --openai-api-base "$ollama_api_base_aider" \ --openai-api-key "$OPENAI_API_KEY" \ --dark-mode --code-theme solarized-dark \ --notifications --notifications-command "ssh ai minecraft.sound" --watch-files } # --- ``` NInfer.exe ```sh ./build-sm89/apps/Release/ninfer.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" \ --prompt "Explain prefill and decode in three sentences." \ --max-context 16384 \ --max-new 256 \ --spec mtp --draft-tokens 3 \ --lm-head-draft ```