Revision of Qwen38-MTP-NInfer-Install.md

ZZ 已修改 1787080954. 還原成這個修訂版本

1 file changed, 2 insertions, 2 deletions

Qwen38-MTP-Ninfer-Install.md

			@@ -28,7 +28,7 @@ H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically s
28	28		Install vcpkg.exe
29	29		```sh
30	30		git clone https://github.com/Microsoft/vcpkg
31		-	git fetch --unshallow # to make sure
	31	+	git fetch --unshallow # required for vcpkg to install pkgs
32	32		git pull
33	33		cd vcpkg
34	34		./bootstrap-vcpkg.sh
			@@ -92,7 +92,7 @@ time aider $aider_files \
92	92		--no-show-model-warnings \
93	93		--model "openai/$ai_model" \
94	94		--openai-api-base "$ollama_api_base_aider" \
95		-	--openai-api-key "ollama" \
	95	+	-openai-api-key "ollama" \
96	96		--dark-mode --code-theme solarized-dark \
97	97		--notifications --notifications-command "ssh ai minecraft.sound" --watch-files
98	98

ZZ 已修改 1787078879. 還原成這個修訂版本

1 file changed, 5 insertions

Qwen38-MTP-Ninfer-Install.md

			@@ -60,6 +60,11 @@ time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninf
60	60		time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
61	61		```
62	62
	63	+	Open port 8080 for LAN access to ninfer-serve.exe
	64	+	```ps1
	65	+	New-NetFirewallRule -DisplayName "NInfer Server (8080)" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8080
	66	+	```
	67	+
63	68		### Query Ninfer Server
64	69
65	70		Curl

ZZ 已修改 1787078582. 還原成這個修訂版本

1 file changed, 97 insertions

Qwen38-MTP-Ninfer-Install.md(檔案已創建)

		@@ -0,0 +1,97 @@
1	+
2	+	## Install Ninfer Server for Fast Qwen3.8-27B inference
3	+
4	+	Ninfer installs are tied to a specific GPU.
5	+	- 5090 - Neroued/ninfer - High-performance single-GPU inference for selected model checkpoints and GPUs. - https://github.com/Neroued/ninfer#build
6	+	- 4090 - UDPSendToFailed/ninfer-4090 - NInfer port for Qwen 3.8 27B on RTX 4090 - https://github.com/UDPSendToFailed/ninfer-4090/tree/feat/rtx-4090-sm89-native
7	+	- 4090 Linux Docker - sergiuszm/ninfer-4090 - Qwen3.8-27B on one RTX 4090 - full native 262K context via E8 4-bit KV, up to 149 tok/s code decode with MTP3, sm_89-retuned attention prefill, vision, llama.cpp-compatible /metrics + /slots - https://github.com/sergiuszm/ninfer-4090#quick-start-linux- 3090 -
8	+	- 3090 - Don-Chad/ninfer-3090 - Fast Qwen3.8-27B inference on one RTX 3090 - ReplaySSM, MTP3, reasoning effort, C1-C8 batching, and native Windows and Linux builds. - https://github.com/Don-Chad/ninfer-3090
9	+
10	+	### Ninfer install
11	+	- Setup file structure
12	+	- Compile and install vcpkg
13	+	- Compile and install curl from vcpkg
14	+	- Install FFMPEG from Chocolatey
15	+	- Configure Ninfer
16	+	- Build Ninfer
17	+	- Run `ninfer-serve.exe`
18	+	- Query `ninfer-serve.exe`
19	+
20	+	File Structure
21	+	```sh
22	+	H:/AI/Code/ninfer-$gpu/
23	+	H:/AI/Code/Tool/vcpkg/
24	+	H:/AI/Code/Model/qwen3_8_27b.ninfer # 18GB
25	+	H:/AI/Code/Tool/cuda_13.3.1_windows # Remote drive CUDA install. Automatically sets the CUDA_PATH in Windows system env variables
26	+	```
27	+
28	+	Install vcpkg.exe
29	+	```sh
30	+	git clone https://github.com/Microsoft/vcpkg
31	+	git fetch --unshallow # to make sure
32	+	git pull
33	+	cd vcpkg
34	+	./bootstrap-vcpkg.sh
35	+	./vcpkg.exe integrate install
36	+	./vcpkg.exe install curl
37	+	```
38	+
39	+	Configure Ninfer with FFMPEG from Chocolatey and curl compiled by MSVC
40	+	```sh
41	+	cd H:/AI/Code/ninfer-4090
42	+	# Clear your build folder build-sm89 then configure.
43	+	time cmake -B build-sm89 -G "Visual Studio 17 2022" -A x64 \
44	+	-DCMAKE_CUDA_ARCHITECTURES=89 \
45	+	-DCMAKE_CUDA_FLAGS="--generate-code=arch=compute_89,code=sm_89" \
46	+	-DNINFER_BUILD_APPS=ON \
47	+	-DNINFER_BUILD_BENCHMARKS=ON \
48	+	-DCMAKE_PREFIX_PATH="C:/Users/IT/AppData/Local/UniGetUI/Chocolatey/lib/ffmpeg-shared/tools/ffmpeg-9.0.1-full_build-shared;H:/AI/Code/Tool/vcpkg/packages/curl_x64-windows" \
49	+	-DCMAKE_TOOLCHAIN_FILE="H:/AI/Code/Tool/vcpkg/scripts/buildsystems/vcpkg.cmake" \
50	+	-DCOMPILE_DEFINITIONS="CURL_STATICLIB"
51	+	```
52	+
53	+	Build Ninfer ~20mins
54	+	```sh
55	+	time cmake --build build-sm89 --config Release --target ninfer ninfer-serve ninfer_bench --parallel 32
56	+	```
57	+
58	+	#### Run Ninfer Server
59	+	```sh
60	+	time ./build-sm89/apps/Release/ninfer-serve.exe "H:/AI/Code/Model/qwen3_8_27b.ninfer" --kv-dtype rk4v4-e8 --spec mtp --draft-tokens 4 --lm-head-draft --max-context 220000 --preserve-thinking --host 0.0.0.0 -p 8080
61	+	```
62	+
63	+	### Query Ninfer Server
64	+
65	+	Curl
66	+	```sh
67	+	curl http://$AI_HOST:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
68	+	"model": "qwen3.8-27b",
69	+	"messages": [{"role": "user", "content": "Reply with one short sentence."}],
70	+	"max_tokens": 64
71	+	}'
72	+	```
73	+
74	+	Aider Chat
75	+	```sh
76	+	# aider.oai:
77	+	alias aoai="aider.openai"
78	+	aider.openai(){
79	+
80	+	aider_files="$@"
81	+
82	+	local ai_model="qwen3.8-27b"
83	+	local ollama_api_base_aider="http://$AI_HOST:8080/v1"
84	+	local OPENAI_API_KEY="dummy"
85	+
86	+	time aider $aider_files \
87	+	--no-show-model-warnings \
88	+	--model "openai/$ai_model" \
89	+	--openai-api-base "$ollama_api_base_aider" \
90	+	--openai-api-key "ollama" \
91	+	--dark-mode --code-theme solarized-dark \
92	+	--notifications --notifications-command "ssh ai minecraft.sound" --watch-files
93	+
94	+	}
95	+	# ---
96	+	```
97	+