ZZ revised this gist . Go to revision
No changes
ZZ revised this gist . Go to revision
No changes
ZZ revised this gist . Go to revision
1 file changed, 1 insertion, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -57,7 +57,7 @@ time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninf | |||
| 57 | 57 | ||
| 58 | 58 | #### Run NInfer Server | |
| 59 | 59 | ```sh | |
| 60 | - | time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080 | |
| 60 | + | time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 --port 8080 | |
| 61 | 61 | ``` | |
| 62 | 62 | ||
| 63 | 63 | Open port 8080 for LAN access to ninfer-serve.exe | |
ZZ revised this gist . Go to revision
No changes
ZZ revised this gist . Go to revision
No changes
ZZ revised this gist . Go to revision
1 file changed, 2 insertions, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -99,8 +99,9 @@ time aider $aider_files \ | |||
| 99 | 99 | } | |
| 100 | 100 | # --- | |
| 101 | 101 | ``` | |
| 102 | - | ```sh | |
| 102 | + | ||
| 103 | 103 | NInfer.exe | |
| 104 | + | ```sh | |
| 104 | 105 | ./build-sm89/apps/Release/ninfer.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" \ | |
| 105 | 106 | --prompt "Explain prefill and decode in three sentences." \ | |
| 106 | 107 | --max-context 16384 \ | |
ZZ revised this gist . Go to revision
1 file changed, 10 insertions, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -99,4 +99,13 @@ time aider $aider_files \ | |||
| 99 | 99 | } | |
| 100 | 100 | # --- | |
| 101 | 101 | ``` | |
| 102 | - | ||
| 102 | + | ```sh | |
| 103 | + | NInfer.exe | |
| 104 | + | ./build-sm89/apps/Release/ninfer.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" \ | |
| 105 | + | --prompt "Explain prefill and decode in three sentences." \ | |
| 106 | + | --max-context 16384 \ | |
| 107 | + | --max-new 256 \ | |
| 108 | + | --spec mtp --draft-tokens 3 \ | |
| 109 | + | --lm-head-draft | |
| 110 | + | ||
| 111 | + | ``` | |
ZZ revised this gist . Go to revision
1 file changed, 1 insertion, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -92,7 +92,7 @@ time aider $aider_files \ | |||
| 92 | 92 | --no-show-model-warnings \ | |
| 93 | 93 | --model "openai/$ai_model" \ | |
| 94 | 94 | --openai-api-base "$ollama_api_base_aider" \ | |
| 95 | - | --openai-api-key "ollama" \ | |
| 95 | + | --openai-api-key "$OPENAI_API_KEY" \ | |
| 96 | 96 | --dark-mode --code-theme solarized-dark \ | |
| 97 | 97 | --notifications --notifications-command "ssh ai minecraft.sound" --watch-files | |
| 98 | 98 | ||
ZZ revised this gist . Go to revision
1 file changed, 1 insertion, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -1,5 +1,5 @@ | |||
| 1 | 1 | ||
| 2 | - | ## Install NInfer Server for Fast Qwen3.8-27B inference | |
| 2 | + | ## Install NInfer Server for fast Qwen3.8-27B inference | |
| 3 | 3 | ||
| 4 | 4 | NInfer installs are tied to a specific GPU. | |
| 5 | 5 | - 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build | |
ZZ revised this gist . Go to revision
1 file changed, 1 insertion, 1 deletion
Qwen38-MTP-Ninfer-Install.md
| @@ -13,7 +13,7 @@ NInfer installs are tied to a specific GPU. | |||
| 13 | 13 | - Compile and install curl from vcpkg | |
| 14 | 14 | - Install FFMPEG from Chocolatey | |
| 15 | 15 | - Configure Ninfer | |
| 16 | - | - Build Ninfer | |
| 16 | + | - Build NInfer | |
| 17 | 17 | - Run `ninfer-serve.exe` | |
| 18 | 18 | - Query `ninfer-serve.exe` | |
| 19 | 19 | ||