Curtir 0
Bifurcar 0

Última atividade 1789265414

export AI_HOST="ip"; and update the target drive.

Install

ZZ's Avatar ZZ revisou este gist 1787091186. Ir para a revisão

1 file changed, 6 insertions, 6 deletions

Qwen38-MTP-Ninfer-Install.md

@@ -1,13 +1,13 @@
1 1
2 - ## Install Ninfer Server for Fast Qwen3.8-27B inference
2 + ## Install NInfer Server for Fast Qwen3.8-27B inference
3 3
4 - Ninfer installs are tied to a specific GPU.
4 + NInfer installs are tied to a specific GPU.
5 5 - 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build
6 6 - 4090 - UDPSendToFailed/ninfer-4090 - NInfer port for Qwen 3.8 27B on RTX 4090 - https://github.com/UDPSendToFailed/ninfer-4090/tree/feat/rtx-4090-sm89-native
7 7 - 4090 Linux Docker - sergiuszm/ninfer-4090 - Qwen3.8-27B on one RTX 4090 - full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots - https://github.com/sergiuszm/ninfer-4090#quick-start-linux- 3090 -
8 8 - 3090 - Don-Chad/ninfer-3090 - Fast Qwen3.8-27B inference on one RTX 3090 - ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds. - https://github.com/Don-Chad/ninfer-3090
9 9
10 - ### Ninfer install
10 + ### NInfer install
11 11 - Setup file structure
12 12 - Compile and install vcpkg
13 13 - Compile and install curl from vcpkg
@@ -50,12 +50,12 @@ time cmake -B build-sm89 -G "Visual Studio 17 2022" -A x64 \
50 50 -DCOMPILE_DEFINITIONS="CURL_STATICLIB"
51 51 ```
52 52
53 - Build Ninfer ~20mins
53 + Build NInfer ~20mins
54 54 ```sh
55 55 time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninfer_bench --parallel 32
56 56 ```
57 57
58 - #### Run Ninfer Server
58 + #### Run NInfer Server
59 59 ```sh
60 60 time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
61 61 ```
@@ -65,7 +65,7 @@ Open port 8080 for LAN access to ninfer-serve.exe
65 65 New-NetFirewallRule -DisplayName "NInfer Server (8080)" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8080
66 66 ```
67 67
68 - ### Query Ninfer Server
68 + ### Query NInfer Server
69 69
70 70 Curl
71 71 ```sh

ZZ's Avatar ZZ revisou este gist 1787080979. Ir para a revisão

1 file changed, 2 insertions, 2 deletions

Qwen38-MTP-Ninfer-Install.md

@@ -92,8 +92,8 @@ time aider $aider_files \
92 92 --no-show-model-warnings \
93 93 --model "openai/$ai_model" \
94 94 --openai-api-base "$ollama_api_base_aider" \
95 - -openai-api-key "ollama" \
96 - --dark-mode --code-theme solarized-dark \
95 + --openai-api-key "ollama" \
96 + --dark-mode --code-theme solarized-dark \
97 97 --notifications --notifications-command "ssh ai minecraft.sound" --watch-files
98 98
99 99 }

ZZ's Avatar ZZ revisou este gist 1787080954. Ir para a revisão

1 file changed, 2 insertions, 2 deletions

Qwen38-MTP-Ninfer-Install.md

@@ -28,7 +28,7 @@ H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically s
28 28 Install vcpkg.exe
29 29 ```sh
30 30 git clone https://github.com/Microsoft/vcpkg
31 - git fetch --unshallow # to make sure
31 + git fetch --unshallow # required for vcpkg to install pkgs
32 32 git pull
33 33 cd vcpkg
34 34 ./bootstrap-vcpkg.sh
@@ -92,7 +92,7 @@ time aider $aider_files \
92 92 --no-show-model-warnings \
93 93 --model "openai/$ai_model" \
94 94 --openai-api-base "$ollama_api_base_aider" \
95 - --openai-api-key "ollama" \
95 + -openai-api-key "ollama" \
96 96 --dark-mode --code-theme solarized-dark \
97 97 --notifications --notifications-command "ssh ai minecraft.sound" --watch-files
98 98

ZZ's Avatar ZZ revisou este gist 1787078879. Ir para a revisão

1 file changed, 5 insertions

Qwen38-MTP-Ninfer-Install.md

@@ -60,6 +60,11 @@ time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninf
60 60 time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
61 61 ```
62 62
63 + Open port 8080 for LAN access to ninfer-serve.exe
64 + ```ps1
65 + New-NetFirewallRule -DisplayName "NInfer Server (8080)" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8080
66 + ```
67 +
63 68 ### Query Ninfer Server
64 69
65 70 Curl

ZZ's Avatar ZZ revisou este gist 1787078582. Ir para a revisão

1 file changed, 97 insertions

Qwen38-MTP-Ninfer-Install.md(arquivo criado)

@@ -0,0 +1,97 @@
1 +
2 + ## Install Ninfer Server for Fast Qwen3.8-27B inference
3 +
4 + Ninfer installs are tied to a specific GPU.
5 + - 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build
6 + - 4090 - UDPSendToFailed/ninfer-4090 - NInfer port for Qwen 3.8 27B on RTX 4090 - https://github.com/UDPSendToFailed/ninfer-4090/tree/feat/rtx-4090-sm89-native
7 + - 4090 Linux Docker - sergiuszm/ninfer-4090 - Qwen3.8-27B on one RTX 4090 - full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots - https://github.com/sergiuszm/ninfer-4090#quick-start-linux- 3090 -
8 + - 3090 - Don-Chad/ninfer-3090 - Fast Qwen3.8-27B inference on one RTX 3090 - ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds. - https://github.com/Don-Chad/ninfer-3090
9 +
10 + ### Ninfer install
11 + - Setup file structure
12 + - Compile and install vcpkg
13 + - Compile and install curl from vcpkg
14 + - Install FFMPEG from Chocolatey
15 + - Configure Ninfer
16 + - Build Ninfer
17 + - Run `ninfer-serve.exe`
18 + - Query `ninfer-serve.exe`
19 +
20 + File Structure
21 + ```sh
22 + H:/AI/Code/ninfer-$gpu/
23 + H:/AI/Code/Tool/vcpkg/
24 + H:/AI/Code/Model/qwen3_8_27b.ninfer # 18GB
25 + H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically sets the CUDA_PATH in Windows system env variables
26 + ```
27 +
28 + Install vcpkg.exe
29 + ```sh
30 + git clone https://github.com/Microsoft/vcpkg
31 + git fetch --unshallow # to make sure
32 + git pull
33 + cd vcpkg
34 + ./bootstrap-vcpkg.sh
35 + ./vcpkg.exe integrate install
36 + ./vcpkg.exe install curl
37 + ```
38 +
39 + Configure Ninfer with FFMPEG from Chocolatey and curl compiled by MSVC
40 + ```sh
41 + cd H:/AI/Code/ninfer-4090
42 + # Clear your build folder build-sm89 then configure.
43 + time cmake -B build-sm89 -G "Visual Studio 17 2022" -A x64 \
44 + -DCMAKE_CUDA_ARCHITECTURES=89 \
45 + -DCMAKE_CUDA_FLAGS="--generate-code=arch=compute_89,code=sm_89" \
46 + -DNINFER_BUILD_APPS=ON \
47 + -DNINFER_BUILD_BENCHMARKS=ON \
48 + -DCMAKE_PREFIX_PATH="C:/Users/IT/AppData/Local/UniGetUI/Chocolatey/lib/ffmpeg-shared/tools/ffmpeg-9.0.1-full_build-shared;H:/AI/Code/Tool/vcpkg/packages/curl_x64-windows" \
49 + -DCMAKE_TOOLCHAIN_FILE="H:/AI/Code/Tool/vcpkg/scripts/buildsystems/vcpkg.cmake" \
50 + -DCOMPILE_DEFINITIONS="CURL_STATICLIB"
51 + ```
52 +
53 + Build Ninfer ~20mins
54 + ```sh
55 + time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninfer_bench --parallel 32
56 + ```
57 +
58 + #### Run Ninfer Server
59 + ```sh
60 + time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
61 + ```
62 +
63 + ### Query Ninfer Server
64 +
65 + Curl
66 + ```sh
67 + curl http://$AI_HOST:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
68 + "model": "qwen3.8-27b",
69 + "messages": [{"role": "user", "content": "Reply with one short sentence."}],
70 + "max_tokens": 64
71 + }'
72 + ```
73 +
74 + Aider Chat
75 + ```sh
76 + # aider.oai:
77 + alias aoai="aider.openai"
78 + aider.openai(){
79 +
80 + aider_files="$@"
81 +
82 + local ai_model="qwen3.8-27b"
83 + local ollama_api_base_aider="http://$AI_HOST:8080/v1"
84 + local OPENAI_API_KEY="dummy"
85 +
86 + time aider $aider_files \
87 + --no-show-model-warnings \
88 + --model "openai/$ai_model" \
89 + --openai-api-base "$ollama_api_base_aider" \
90 + --openai-api-key "ollama" \
91 + --dark-mode --code-theme solarized-dark \
92 + --notifications --notifications-command "ssh ai minecraft.sound" --watch-files
93 +
94 + }
95 + # ---
96 + ```
97 +
Próximo Anterior

Desenvolvido por Opengist ⋅ Load: 28ms⋅

Português