## Install NInfer Server for fast Qwen3.8-27B inference

NInfer installs are tied to a specific GPU.
- 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build
- 4090 - UDPSendToFailed/ninfer-4090 - NInfer port for Qwen 3.8 27B on RTX 4090 - https://github.com/UDPSendToFailed/ninfer-4090/tree/feat/rtx-4090-sm89-native
- 4090 Linux Docker - sergiuszm/ninfer-4090 - Qwen3.8-27B on one RTX 4090 - full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots - https://github.com/sergiuszm/ninfer-4090#quick-start-linux- 3090 - 
- 3090 - Don-Chad/ninfer-3090 - Fast Qwen3.8-27B inference on one RTX 3090 - ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds. - https://github.com/Don-Chad/ninfer-3090

### NInfer install
- Setup file structure
- Compile and install vcpkg
- Compile and install curl from vcpkg
- Install FFMPEG from Chocolatey
- Configure Ninfer
- Build NInfer
- Run `ninfer-serve.exe`
- Query `ninfer-serve.exe`

File Structure
```sh
H:/AI/Code/ninfer-$gpu/
H:/AI/Code/Tool/vcpkg/
H:/AI/Code/Model/qwen3_8_27b.ninfer # 18GB
H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically sets the CUDA_PATH in Windows system env variables
```

Install vcpkg.exe
```sh
git clone https://github.com/Microsoft/vcpkg
git fetch --unshallow # required for vcpkg to install pkgs
git pull
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg.exe integrate install
./vcpkg.exe install curl
```

Configure Ninfer with FFMPEG from Chocolatey and curl compiled by MSVC
```sh
cd H:/AI/Code/ninfer-4090
# Clear your build folder build-sm89 then configure.
time cmake -B build-sm89 -G "Visual Studio 17 2022" -A x64 \
  -DCMAKE_CUDA_ARCHITECTURES=89 \
  -DCMAKE_CUDA_FLAGS="--generate-code=arch=compute_89,code=sm_89" \
  -DNINFER_BUILD_APPS=ON \
  -DNINFER_BUILD_BENCHMARKS=ON \
  -DCMAKE_PREFIX_PATH="C:/Users/IT/AppData/Local/UniGetUI/Chocolatey/lib/ffmpeg-shared/tools/ffmpeg-9.0.1-full_build-shared;H:/AI/Code/Tool/vcpkg/packages/curl_x64-windows" \
  -DCMAKE_TOOLCHAIN_FILE="H:/AI/Code/Tool/vcpkg/scripts/buildsystems/vcpkg.cmake" \
  -DCOMPILE_DEFINITIONS="CURL_STATICLIB"
```

Build NInfer ~20mins
```sh
time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninfer_bench --parallel 32                                                                     
```

#### Run NInfer Server
```sh
time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
```

Open port 8080 for LAN access to ninfer-serve.exe
```ps1
New-NetFirewallRule -DisplayName "NInfer Server (8080)" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8080
```

### Query NInfer Server

Curl
```sh
curl http://$AI_HOST:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3.8-27b",
"messages": [{"role": "user", "content": "Reply with one short sentence."}],
"max_tokens": 64
}'
```

Aider Chat
```sh
# aider.oai:  
alias aoai="aider.openai"
aider.openai(){

aider_files="$@"

local ai_model="qwen3.8-27b"
local ollama_api_base_aider="http://$AI_HOST:8080/v1"
local OPENAI_API_KEY="dummy"

time aider $aider_files \
  --no-show-model-warnings \
  --model "openai/$ai_model" \
  --openai-api-base "$ollama_api_base_aider" \
  --openai-api-key "$OPENAI_API_KEY" \
  --dark-mode --code-theme solarized-dark \
  --notifications --notifications-command "ssh ai minecraft.sound" --watch-files

}
# ---
```

NInfer.exe
```sh
./build-sm89/apps/Release/ninfer.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" \
    --prompt "Explain prefill and decode in three sentences." \
    --max-context 16384 \
    --max-new 256 \
    --spec mtp --draft-tokens 3 \
    --lm-head-draft

```