Text Classification
GGUF
English
rknn
rknn3
rk1828
rk1820
rockchip
npu
edge-ai
kev
decision
w4a16
Eval Results (legacy)
conversational
Instructions to use jj90/Kev-0.8B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/Kev-0.8B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- Ollama
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/Kev-0.8B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Lemonade
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/Kev-0.8B-RKNN3-RK1828
Run and chat with the model
lemonade run user.Kev-0.8B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/Kev-0.8B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/Kev-0.8B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/Kev-0.8B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add model card, deployment guide and host-side code
Browse files- .gitattributes +10 -34
- LICENSE +202 -0
- README.md +271 -0
- docs/DEPLOY.md +206 -0
- host/e2e_check.py +176 -0
- host/kev_daemon.cpp +411 -0
- host/kev_serve.py +450 -0
- host/serve_assets/Ginv.npy +3 -0
- host/serve_assets/head_k.npy +3 -0
- host/serve_assets/head_q.npy +3 -0
- host/serve_assets/temperature.npy +3 -0
.gitattributes
CHANGED
|
@@ -1,35 +1,11 @@
|
|
| 1 |
-
|
| 2 |
-
*.
|
| 3 |
-
*.
|
| 4 |
-
*.
|
| 5 |
-
*.
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
-
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
-
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
-
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
-
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
-
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
*.
|
| 18 |
-
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
-
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
-
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
-
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
-
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
-
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
-
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
-
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
-
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
-
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
-
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
-
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
-
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
-
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
-
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
-
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
-
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
-
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
| 1 |
+
# RKNN3 artifacts (hundreds of MB, must go through LFS)
|
| 2 |
+
*.rknn filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.weight filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.embed.bin filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.gguf filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
|
| 7 |
+
# Host-side precomputed assets
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
|
| 10 |
+
# Keep Windows launchers as-is (CRLF, pure ASCII)
|
| 11 |
+
*.cmd text eol=crlf
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
LICENSE
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
| 178 |
+
|
| 179 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 180 |
+
|
| 181 |
+
To apply the Apache License to your work, attach the following
|
| 182 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 183 |
+
replaced with your own identifying information. (Don't include
|
| 184 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 185 |
+
comment syntax for the file format. We also recommend that a
|
| 186 |
+
file or class name and description of purpose be included on the
|
| 187 |
+
same "printed page" as the copyright notice for easier
|
| 188 |
+
identification within third-party archives.
|
| 189 |
+
|
| 190 |
+
Copyright 2026 Alibaba Cloud
|
| 191 |
+
|
| 192 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 193 |
+
you may not use this file except in compliance with the License.
|
| 194 |
+
You may obtain a copy of the License at
|
| 195 |
+
|
| 196 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 197 |
+
|
| 198 |
+
Unless required by applicable law or agreed to in writing, software
|
| 199 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 200 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 201 |
+
See the License for the specific language governing permissions and
|
| 202 |
+
limitations under the License.
|
README.md
ADDED
|
@@ -0,0 +1,271 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
+
inference: false
|
| 7 |
+
tags:
|
| 8 |
+
- rknn
|
| 9 |
+
- rknn3
|
| 10 |
+
- rk1828
|
| 11 |
+
- rk1820
|
| 12 |
+
- rockchip
|
| 13 |
+
- npu
|
| 14 |
+
- edge-ai
|
| 15 |
+
- kev
|
| 16 |
+
- decision
|
| 17 |
+
- text-classification
|
| 18 |
+
- w4a16
|
| 19 |
+
base_model: Qwen/Qwen3.5-0.8B-Base
|
| 20 |
+
base_model_relation: finetune
|
| 21 |
+
model-index:
|
| 22 |
+
- name: Kev-0.8B-RKNN3-RK1828
|
| 23 |
+
results:
|
| 24 |
+
- task:
|
| 25 |
+
type: text-classification
|
| 26 |
+
name: Structured decision (choice / noul / score)
|
| 27 |
+
dataset:
|
| 28 |
+
type: kev-decision-v2
|
| 29 |
+
name: Kev decision-v2 (60 held-out)
|
| 30 |
+
split: dev
|
| 31 |
+
metrics:
|
| 32 |
+
- type: accuracy
|
| 33 |
+
value: 0.817
|
| 34 |
+
name: Accuracy (device)
|
| 35 |
+
- task:
|
| 36 |
+
type: text-classification
|
| 37 |
+
name: Structured decision (choice / noul / score)
|
| 38 |
+
dataset:
|
| 39 |
+
type: kev-decision-v2
|
| 40 |
+
name: Kev decision-v2 (26 mixed-type, end-to-end HTTP)
|
| 41 |
+
split: dev
|
| 42 |
+
metrics:
|
| 43 |
+
- type: accuracy
|
| 44 |
+
value: 0.885
|
| 45 |
+
name: Accuracy (HTTP service)
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
# Kev-0.8B-RKNN3-RK1828
|
| 49 |
+
|
| 50 |
+
**Kev-0.8B (decision model) converted to RKNN3 and running on the Rockchip RK1828 NPU — with zero accuracy loss versus the original fp32 model.**
|
| 51 |
+
|
| 52 |
+
Kev is *not* a chat model. It runs a single prefill and scores options with a pointer head; it never generates tokens.
|
| 53 |
+
You can verify this yourself: ask it `1+1`, and you get nonsense. That is the model behaving correctly.
|
| 54 |
+
|
| 55 |
+
This repository contains the converted artifacts plus the minimal host-side code needed to actually
|
| 56 |
+
use them, because a plain `rkllm3-server` deployment **cannot** run Kev (see [Why this needs host code](#why-this-needs-host-code)).
|
| 57 |
+
|
| 58 |
+
| | |
|
| 59 |
+
| --- | --- |
|
| 60 |
+
| Base model | [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) @ `dc7cdfe` |
|
| 61 |
+
| Adapter | [`jaredpalmer/kev-0.8b`](https://huggingface.co/jaredpalmer/kev-0.8b) (LoRA r16, merged into base) |
|
| 62 |
+
| Target hardware | RK1828 NPU (RK1820/RK1828 AI SDK), host = Windows PC / RK3588 / RK3576 |
|
| 63 |
+
| Toolchain | RKNN3 **v1.1.0** (`runtime/rknn3_110`), `target_platform='rk1820'` |
|
| 64 |
+
| Quantization | `w4a16`, `quantized_algorithm='normal'`, `quantized_method='group32'` (weight-only, no calibration set) |
|
| 65 |
+
| NPU footprint | **802 MB** (as reported by `rknn-smi`) |
|
| 66 |
+
| Latency | ~0.9 s per decision end-to-end, ~75 ms per prefill (~100 tokens) |
|
| 67 |
+
|
| 68 |
+
---
|
| 69 |
+
|
| 70 |
+
## Files
|
| 71 |
+
|
| 72 |
+
| File | Size | What it is |
|
| 73 |
+
| --- | --- | --- |
|
| 74 |
+
| `rknn/Kev-0.8B.rknn` | 20.6 MiB | RKNN3 graph. Output is `[1, 1, 248320]` fp16 logits |
|
| 75 |
+
| `rknn/Kev-0.8B.weight` | 498 MiB | Quantized backbone weights |
|
| 76 |
+
| `rknn/Kev-0.8B.embed.bin` | 485 MiB | Token embedding table, `248320 × 1024` fp16 (also the dequantization matrix `W`) |
|
| 77 |
+
| `rknn/Kev-0.8B.tokenizer.gguf` | 10.4 MiB | Tokenizer. Identical to Qwen3.5-0.8B — LoRA does not touch the vocab |
|
| 78 |
+
| `rknn/start.cmd` | 1.4 KiB | `rkllm3-server` launch line (for the "text-only" sanity check) |
|
| 79 |
+
| `host/kev_daemon.cpp` | — | Device-side runner: loads the model once, then takes `TOK`/`R`/`A` commands and dumps logits per request |
|
| 80 |
+
| `host/kev_serve.py` | — | HTTP service exposing `POST /v1/systemone` |
|
| 81 |
+
| `host/serve_assets/*.npy` | 6.1 MiB | `Ginv.npy` (1024×1024), `head_q.npy`, `head_k.npy`, `temperature.npy` — precomputed, so you do not have to invert a 1024×1024 normal-equation matrix at startup |
|
| 82 |
+
| `host/e2e_check.py` | — | End-to-end accuracy check against the decision-v2 labels |
|
| 83 |
+
| `docs/DEPLOY.md` | — | Step-by-step deployment, including the traps |
|
| 84 |
+
|
| 85 |
+
Total: ~1.06 GB.
|
| 86 |
+
|
| 87 |
+
---
|
| 88 |
+
|
| 89 |
+
## Why this needs host code
|
| 90 |
+
|
| 91 |
+
Kev's scoring path is:
|
| 92 |
+
|
| 93 |
+
```
|
| 94 |
+
h_decide = backbone(state + question + "<decide>") # hidden state, 1024-d, *before* lm_head
|
| 95 |
+
h_opt_k = backbone(state + question + option_k + "</opt>")
|
| 96 |
+
z_k = (k(h_opt_k) · q(h_decide)) / sqrt(256)
|
| 97 |
+
answer = softmax(z / T) # T = 2.3510958125672174
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
The runtime exposes **vocab logits**, not hidden states. The usual conclusion is "blocked, needs retraining".
|
| 101 |
+
It is not blocked, because `logits = h @ Wᵀ` is an **overdetermined** system:
|
| 102 |
+
|
| 103 |
+
```
|
| 104 |
+
W ∈ R^(248320 × 1024), unknown h ∈ R^1024, 248320 equations
|
| 105 |
+
h = (WᵀW)⁻¹ Wᵀ · logits
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
`W` is the (shared) embedding matrix, so it is already on disk. The per-coordinate quantization noise gets
|
| 109 |
+
averaged over ~248k equations, i.e. suppressed by roughly √248320 ≈ 500×. Measured cosine similarity
|
| 110 |
+
between device-recovered `h` and the fp32 reference is **0.96–0.99**, with top-1 agreeing on every sampled position.
|
| 111 |
+
|
| 112 |
+
So the original pointer head runs unmodified on the host. **No retraining, no downgrade to an embedding model.**
|
| 113 |
+
|
| 114 |
+
`Ginv.npy` is the precomputed `(WᵀW)⁻¹Wᵀ`, so recovery is a single `1024×1024 @ 1024` matmul per position.
|
| 115 |
+
|
| 116 |
+
Two consequences worth knowing before you deploy:
|
| 117 |
+
|
| 118 |
+
- **The graph only keeps the last position.** There is no `num_logits_to_keep` in the public API and
|
| 119 |
+
`clear_kvcache` has no truncate. To read the `</opt>` positions you must re-run a truncated sequence.
|
| 120 |
+
A request with `K` options therefore costs **K+1 prefills** — that is a hard floor, not an implementation choice.
|
| 121 |
+
- **Per-call fixed overhead is ~50 ms**, independent of length. Latency scales with *number of calls*, not token count.
|
| 122 |
+
Batching long states into one prefix helps a lot; splitting into many short calls does not.
|
| 123 |
+
|
| 124 |
+
---
|
| 125 |
+
|
| 126 |
+
## Usage
|
| 127 |
+
|
| 128 |
+
### 1. Sanity check (text only)
|
| 129 |
+
|
| 130 |
+
```bat
|
| 131 |
+
cd rknn
|
| 132 |
+
start.cmd
|
| 133 |
+
```
|
| 134 |
+
|
| 135 |
+
Then `POST /v1/chat/completions`. Output will be garbage — this only proves the weights loaded and the NPU
|
| 136 |
+
is running. **Do not use this to judge whether the conversion worked.**
|
| 137 |
+
|
| 138 |
+
### 2. Real usage: decision API
|
| 139 |
+
|
| 140 |
+
Build `host/kev_daemon.cpp` against the RKNN3 SDK session demo, run it, then:
|
| 141 |
+
|
| 142 |
+
```bash
|
| 143 |
+
python host/kev_serve.py # listens on :17890
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
```bash
|
| 147 |
+
curl -X POST http://127.0.0.1:17890/v1/systemone \
|
| 148 |
+
-H "Content-Type: application/json" \
|
| 149 |
+
-d '{
|
| 150 |
+
"state": {"ticket": "Checkout returns 500 for EU customers", "age_hours": 30},
|
| 151 |
+
"questions": {
|
| 152 |
+
"team": {"type": "choice", "instructions": "Which team should own this?",
|
| 153 |
+
"criteria": {"billing": "payment, invoices, refunds",
|
| 154 |
+
"platform": "infra, deploys, runtime",
|
| 155 |
+
"growth": "onboarding, activation"}},
|
| 156 |
+
"urgent": {"type": "noul", "instructions": "Is this urgent?"},
|
| 157 |
+
"sev": {"type": "score", "instructions": "Rate the severity",
|
| 158 |
+
"criteria": ["cosmetic", "degraded", "broken", "outage"]}
|
| 159 |
+
}
|
| 160 |
+
}'
|
| 161 |
+
```
|
| 162 |
+
|
| 163 |
+
```json
|
| 164 |
+
{"answers": {
|
| 165 |
+
"team": {"type": "choice", "answer": "billing", "confidence": 0.7638, "probabilities": {...}},
|
| 166 |
+
"urgent": {"type": "noul", "p_true": 0.912},
|
| 167 |
+
"sev": {"type": "score", "score": 2.31, "legend": [...], "score_confidence": 0.54}
|
| 168 |
+
}}
|
| 169 |
+
```
|
| 170 |
+
|
| 171 |
+
Response schema follows the upstream `kev/api.py::to_answers` conventions:
|
| 172 |
+
|
| 173 |
+
- `noul` returns `p_true` only (the upstream API does not binarize it)
|
| 174 |
+
- `choice` returns `confidence = (p_max − 1/K) / (1 − 1/K)`, so `K=2` with `p=1.0` gives `confidence=1.0`
|
| 175 |
+
- `score` returns the probability-weighted expected level as `score`, plus per-level probabilities
|
| 176 |
+
|
| 177 |
+
Full commands and the build line: [`docs/DEPLOY.md`](docs/DEPLOY.md).
|
| 178 |
+
|
| 179 |
+
---
|
| 180 |
+
|
| 181 |
+
## Performance
|
| 182 |
+
|
| 183 |
+
Measured on RK1828 (5051.8 MB NPU), Windows host, single model loaded.
|
| 184 |
+
|
| 185 |
+
| Metric | Value |
|
| 186 |
+
| --- | --- |
|
| 187 |
+
| Accuracy, decision-v2 dev 60 (fp32 reference, host) | 49/60 = **0.817** |
|
| 188 |
+
| Accuracy, decision-v2 dev 60 (device, w4a16 + recovered `h`) | 49/60 = **0.817** |
|
| 189 |
+
| Per-question argmax agreement, device vs fp32 | **58/60 = 96.7%** |
|
| 190 |
+
| End-to-end HTTP, 26 mixed-type questions | 23/26 = **0.885** |
|
| 191 |
+
| Prefill throughput (long sequences) | 1200–1500 tok/s |
|
| 192 |
+
| Single prefill, ~100 tokens | ~75 ms |
|
| 193 |
+
| Per-call fixed overhead | ~50 ms |
|
| 194 |
+
| One decision (K=5) | ~0.5 s |
|
| 195 |
+
| End-to-end HTTP per question | ~0.9 s |
|
| 196 |
+
| Model load | ~30 s, once |
|
| 197 |
+
| NPU memory | 802 MB |
|
| 198 |
+
|
| 199 |
+
The two disagreeing questions (q9, q10) cancel out — one is right on device and wrong on fp32, the other the reverse.
|
| 200 |
+
That is the signature of the *same function* being computed, with numerical noise flipping borderline cases.
|
| 201 |
+
|
| 202 |
+
Coexistence, measured with `rknn-smi` on the same device:
|
| 203 |
+
|
| 204 |
+
| Configuration | NPU used | Delta |
|
| 205 |
+
| --- | --- | --- |
|
| 206 |
+
| baseline (proxy only) | 34 MB | — |
|
| 207 |
+
| Kev-0.8B alone | 802 MB | — |
|
| 208 |
+
| Kev-0.8B + Qwen3-ASR-0.6B | 2848 MB | +2047 MB |
|
| 209 |
+
| Kev-0.8B + Qwen3-ASR-1.7B | 2953 MB | +2152 MB |
|
| 210 |
+
|
| 211 |
+
Kev can share the NPU with an ASR model. Both fit; only ~105 MB separates the two ASR sizes.
|
| 212 |
+
Measure *coexisting* configurations, not standalone ones — a standalone run reports 779 MB for the 0.6B ASR,
|
| 213 |
+
which will make you over-budget your plan.
|
| 214 |
+
|
| 215 |
+
---
|
| 216 |
+
|
| 217 |
+
## Limitations
|
| 218 |
+
|
| 219 |
+
- **Not a chat model.** Text generation output is meaningless by design. Any benchmark that prompts it for answers is invalid.
|
| 220 |
+
- **Numeric noise flips borderline decisions.** 96.7% per-question agreement means ~3% of hard cases can differ from
|
| 221 |
+
the fp32 reference. If you need exact reproducibility, run the fp32 model.
|
| 222 |
+
- **K+1 prefill calls per request** cannot be reduced with this runtime (see above). Very large option lists get slow linearly.
|
| 223 |
+
- **Single-request serialization.** The service holds a global lock around one decision, because `K+1` prefills share
|
| 224 |
+
one KV cache; interleaving concurrent requests corrupts it. Throughput is therefore ~1 request at a time.
|
| 225 |
+
Concurrency is safe (tested), but it queues.
|
| 226 |
+
- **Host must be Windows / RK3588 / RK3576.** The RK1828 is a co-processor; an x86 Linux box can convert and simulate
|
| 227 |
+
but cannot drive the module.
|
| 228 |
+
- **Quantization format is fixed** to `w4a16 + normal + group32` by this toolchain. `w8a16` and `fp16` builds are rejected
|
| 229 |
+
(`request_type=float16` / `Failed to create initializer tensor`), so you cannot "build a higher-precision version"
|
| 230 |
+
to isolate quantization error.
|
| 231 |
+
|
| 232 |
+
---
|
| 233 |
+
|
| 234 |
+
## Provenance
|
| 235 |
+
|
| 236 |
+
- Base: `Qwen/Qwen3.5-0.8B-Base`, revision `dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68`
|
| 237 |
+
- Adapter: `jaredpalmer/kev-0.8b`, LoRA r16 / α32, task type `FEATURE_EXTRACTION`
|
| 238 |
+
- Merged via `merge_and_unload()`, then exported to ONNX with the official `rknn3-model-zoo/examples/Qwen3_5` pipeline
|
| 239 |
+
- `h`, the pointer head, and the option/decide token ids are all taken from the upstream Kev implementation unchanged
|
| 240 |
+
|
| 241 |
+
Special token ids used (from upstream `kev/model.py::SPECIAL`):
|
| 242 |
+
|
| 243 |
+
```
|
| 244 |
+
<state> 248060 <question> 248061 <opt> 248049 </opt> 248050 <decide> 248062
|
| 245 |
+
```
|
| 246 |
+
|
| 247 |
+
Artifact sizes match the officially published Qwen3.5-0.8B RKNN3 build file-for-file
|
| 248 |
+
(20.6 / 498 / 485 / 10.4 MiB), which is the practical proof that the conversion is structurally correct.
|
| 249 |
+
|
| 250 |
+
License: **Apache-2.0**, inherited from the base model.
|
| 251 |
+
|
| 252 |
+
---
|
| 253 |
+
|
| 254 |
+
## 中文摘要
|
| 255 |
+
|
| 256 |
+
Kev-0.8B 是决策模型,不是对话模型:它只跑一次 prefill,用指针头给候选项打分,不生成 token。
|
| 257 |
+
用"问它 1+1"来验收是错的——那样一定会得到无意义的回答。
|
| 258 |
+
|
| 259 |
+
RKNN3 运行时只暴露 vocab logits,而 Kev 需要 hidden state。但 `logits = h @ Wᵀ` 是超定方程
|
| 260 |
+
(248320 个方程解 1024 个未知数),`h = (WᵀW)⁻¹Wᵀ·logits` 可以精确解回来,
|
| 261 |
+
量化噪声被约 500 倍平均掉。因此**原版指针头可以原样跑在 host 侧,无需重训、无需改结构**。
|
| 262 |
+
|
| 263 |
+
| 项 | 结果 |
|
| 264 |
+
| --- | --- |
|
| 265 |
+
| 真机准确率(60 题) | 0.817,与 fp32 完全一致 |
|
| 266 |
+
| 逐题一致 | 58/60 |
|
| 267 |
+
| 端到端 HTTP(26 题三题型) | 0.885,约 0.9 s/题 |
|
| 268 |
+
| NPU 占用 | 802 MB |
|
| 269 |
+
|
| 270 |
+
本仓库包含转换产物 + 可复现的 host 侧代码(daemon、HTTP 服务、反解矩阵、头权重),
|
| 271 |
+
拿到即可跑通。部署步骤见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。
|
docs/DEPLOY.md
ADDED
|
@@ -0,0 +1,206 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 部署指南(RK1828)
|
| 2 |
+
|
| 3 |
+
> 这份文档只讲"怎么把它跑起来"。为什么这么做、踩过哪些坑,见仓库根 `README.md`
|
| 4 |
+
> 与上游项目里的 `PITFALLS.md`。
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## 0. 前置条件
|
| 9 |
+
|
| 10 |
+
| 项 | 要求 |
|
| 11 |
+
| --- | --- |
|
| 12 |
+
| Host | Windows PC(实测)/ RK3588 / RK3576 板。**x86 Linux 不行**——能转换、能跑模拟器,但不能直连模组 |
|
| 13 |
+
| 模组 | RK1828(RK1820 同 SDK) |
|
| 14 |
+
| 运行时 | RKNN3 **v1.1.0**,即 `runtime/rknn3_110`(`rkllm3-server.exe`、`librknn3_api.dll` 等) |
|
| 15 |
+
| 驱动/固件 | 与运行时同版本。版本不一致会 load 失败或静默退出 |
|
| 16 |
+
| 编译 daemon 额外需要 | Cygwin g++、RKNN3 SDK 的 `examples/rknn3_session_test_demo/{src,utils}`、`examples/3rdparty/tokenizer` |
|
| 17 |
+
| Python | 3.8+,`numpy` |
|
| 18 |
+
|
| 19 |
+
> 组件版本必须一致(toolkit / proxy / runtime / 固件)。混版本是最常见的"跑一半没反应"。
|
| 20 |
+
|
| 21 |
+
---
|
| 22 |
+
|
| 23 |
+
## 1. 布局
|
| 24 |
+
|
| 25 |
+
把仓库放到设备侧任意目录,建议:
|
| 26 |
+
|
| 27 |
+
```
|
| 28 |
+
<REPO>\
|
| 29 |
+
├── rknn\ # 本仓库的 rknn/ 目录
|
| 30 |
+
│ ├── Kev-0.8B.rknn
|
| 31 |
+
│ ├── Kev-0.8B.weight
|
| 32 |
+
│ ├── Kev-0.8B.embed.bin
|
| 33 |
+
│ ├── Kev-0.8B.tokenizer.gguf
|
| 34 |
+
│ └── start.cmd
|
| 35 |
+
├── host\
|
| 36 |
+
│ ├── kev_serve.py
|
| 37 |
+
│ ├── kev_daemon.cpp
|
| 38 |
+
│ ├── e2e_check.py
|
| 39 |
+
│ └── serve_assets\
|
| 40 |
+
└── runtime\rknn3_110\ # 从 SDK 拷来的运行时
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
`start.cmd` 会自动在**两个位置**找运行时:`<REPO>\runtime\rknn3_110`(本仓库下载后的布局)
|
| 44 |
+
和 `<包根>\runtime\rknn3_110`(放进 rk1828-console 的 `models\<模型>\` 时)。
|
| 45 |
+
两处都没有就报错并提示,不设 `RK_RUNTIME` 则不会静默用错版本:
|
| 46 |
+
|
| 47 |
+
```bat
|
| 48 |
+
set RK_RUNTIME=D:\somewhere\rknn3_110
|
| 49 |
+
start.cmd
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## 2. 第一步:确认模型能加载
|
| 55 |
+
|
| 56 |
+
```bat
|
| 57 |
+
cd <REPO>\rknn
|
| 58 |
+
start.cmd
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
看到 `server listening on 0.0.0.0:8888` 就说明权重进了 NPU。
|
| 62 |
+
|
| 63 |
+
**这一步只能证明"加载成功",不能证明"转换正确"。** Kev 不是对话模型,
|
| 64 |
+
用 `/v1/chat/completions` 问它问题一定得到胡言乱语——那是模型本身的行为,不是转换坏了。
|
| 65 |
+
真正的验收在 §4。
|
| 66 |
+
|
| 67 |
+
顺便看一眼占用:
|
| 68 |
+
|
| 69 |
+
```bat
|
| 70 |
+
rknn-smi
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
`used` 应该在 **800 MB** 量级(含 34 MB 基线)。明显偏小说明权重没真正进 NPU。
|
| 74 |
+
|
| 75 |
+
> 加载期间**不要**去轮询 `rknn-smi`。查询会卡在 `Weight sync`,把 30 秒的加载拖成几分钟甚至失败。
|
| 76 |
+
|
| 77 |
+
---
|
| 78 |
+
|
| 79 |
+
## 3. 第二步:编译 daemon
|
| 80 |
+
|
| 81 |
+
`rkllm3-server` 只能返回文本,拿不到 logits,所以真实使用需要自己的 host 程序。
|
| 82 |
+
|
| 83 |
+
准备 SDK 里的两样东西(都来自 RKNN3 SDK,不在本仓库内):
|
| 84 |
+
|
| 85 |
+
- `examples/rknn3_session_test_demo/` —— 里面 `src/`、`utils/`、`CMakeLists.txt`
|
| 86 |
+
- `examples/3rdparty/tokenizer/` —— 头文件 + `lib/Windows/x86_64/libtokenizer.a`
|
| 87 |
+
|
| 88 |
+
把 `host/kev_daemon.cpp` 放进 demo 的 `src/`(或任意位置,`-I` 指到就行),然后:
|
| 89 |
+
|
| 90 |
+
```bat
|
| 91 |
+
set PATH=C:\cygwin64\bin;<REPO>\runtime\rknn3_110\lib;<REPO>\runtime\rknn3_110\bin;%PATH%
|
| 92 |
+
cd /d <SDK>\examples\rknn3_session_test_demo
|
| 93 |
+
|
| 94 |
+
"C:\cygwin64\bin\g++.exe" -std=c++11 -I. -Iutils -I<REPO>\host -I<TOK>\include ^
|
| 95 |
+
src\kev_daemon.cpp utils\rknn3_session_test_common.cpp -o daemon.exe ^
|
| 96 |
+
<REPO>\runtime\rknn3_110\lib\librknn3_api.dll ^
|
| 97 |
+
<TOK>\lib\Windows\x86_64\libtokenizer.a -lpthread
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
三个必注意的点:
|
| 101 |
+
|
| 102 |
+
1. **`C:\cygwin64\bin` 必须在 PATH 最前面**,否则 `cc1plus` 找不到 `cygwin1.dll`,报一个和编译毫无关系的错。
|
| 103 |
+
2. **链接时直接给 `.dll` / `.a` 的完整路径**,Cygwin 的 ld 接受这种写法,不需要 `-L`/`-l`。
|
| 104 |
+
3. 编译产物 `daemon.exe` **必须能加载到 `librknn3_api.dll`**。计划任务不继承 PATH,
|
| 105 |
+
这种情况下会以 `rc=3221225781`(`0xC0000135`)**一个字的输出都没有**地退出。见 §6。
|
| 106 |
+
|
| 107 |
+
---
|
| 108 |
+
|
| 109 |
+
## 4. 第三步:起服务并验收
|
| 110 |
+
|
| 111 |
+
`kev_serve.py` 会自己拉起 daemon,不用手动启动:
|
| 112 |
+
|
| 113 |
+
```bat
|
| 114 |
+
set KEV_MODEL_DIR=<REPO>\rknn
|
| 115 |
+
set KEV_DAEMON=<SDK>\examples\rknn3_session_test_demo\daemon.exe
|
| 116 |
+
python <REPO>\host\kev_serve.py
|
| 117 |
+
```
|
| 118 |
+
|
| 119 |
+
不设这两个环境变量时,它按"`<包根>\models\Kev-0.8B`"和"脚本同目录 `rknn3_session_test_demo\daemon.exe`"找,
|
| 120 |
+
放进现成的 rk1828-console 包里就不用设。
|
| 121 |
+
|
| 122 |
+
默认端口 **17890**,可用 `KEV_PORT` 覆盖。等它打印 ready(模型加载约 30 s)后:
|
| 123 |
+
|
| 124 |
+
```bash
|
| 125 |
+
curl -X POST http://127.0.0.1:17890/v1/systemone ^
|
| 126 |
+
-H "Content-Type: application/json" ^
|
| 127 |
+
-d "{\"state\":{\"ticket\":\"Checkout returns 500 for EU customers\"},\"questions\":{\"team\":{\"type\":\"choice\",\"instructions\":\"Which team?\",\"criteria\":{\"billing\":\"payments\",\"platform\":\"infra\"}}}}"
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
返回:
|
| 131 |
+
|
| 132 |
+
```json
|
| 133 |
+
{"answers":{"team":{"type":"choice","answer":"billing","confidence":0.7638,"probabilities":{...}}}}
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
### 单项验收(不依赖数据集)
|
| 137 |
+
|
| 138 |
+
```bash
|
| 139 |
+
curl http://127.0.0.1:17890/healthz
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
### 整体验收(需要 decision-v2 数据��)
|
| 143 |
+
|
| 144 |
+
```bash
|
| 145 |
+
set KEV_URL=http://127.0.0.1:17890
|
| 146 |
+
python <REPO>\host\e2e_check.py
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
期望 **23/26 = 0.885**(三题型混合抽样)。数字明显偏低时先看是不是抽样把题型抽偏了,
|
| 150 |
+
再用 KEV_DEV 指到正确路径。
|
| 151 |
+
|
| 152 |
+
---
|
| 153 |
+
|
| 154 |
+
## 5. 接口
|
| 155 |
+
|
| 156 |
+
`POST /v1/systemone`
|
| 157 |
+
|
| 158 |
+
```json
|
| 159 |
+
{
|
| 160 |
+
"state": <任意 JSON:字符串 / 对象 / 数组,会按 YAML 风格渲染进 prompt>,
|
| 161 |
+
"questions": {
|
| 162 |
+
"<name>": {
|
| 163 |
+
"type": "choice" | "noul" | "score",
|
| 164 |
+
"instructions": "...",
|
| 165 |
+
"criteria": {"<选项>": "<描述>"} // choice:对象
|
| 166 |
+
| ["<档位描述>", ...] // score:数组
|
| 167 |
+
}
|
| 168 |
+
}
|
| 169 |
+
}
|
| 170 |
+
```
|
| 171 |
+
|
| 172 |
+
| type | 返回 |
|
| 173 |
+
| --- | --- |
|
| 174 |
+
| `choice` | `answer` + `confidence` + `probabilities` |
|
| 175 |
+
| `noul` | `p_true`(上游 API 只回概率,不转布尔) |
|
| 176 |
+
| `score` | `score`(概率加权期望档位)+ `legend` + 各档概率 + `score_confidence` |
|
| 177 |
+
|
| 178 |
+
错误码:`400` 参数不合法、`405` 方法不对、`413` body 超过 1 MiB、`500` 内部错误(堆栈只写日志,不回给客户端)。
|
| 179 |
+
|
| 180 |
+
---
|
| 181 |
+
|
| 182 |
+
## 6. 一定会踩的坑
|
| 183 |
+
|
| 184 |
+
| 现象 | 原因 | 处理 |
|
| 185 |
+
| --- | --- | --- |
|
| 186 |
+
| daemon 起来就退,**没有任何输出**,`rc=3221225781` | `librknn3_api.dll` 找不到(计划任务不继承 PATH) | 在 `Popen` 前把 `<runtime>\lib` 和 `\bin` 塞进 `os.environ["PATH"]` |
|
| 187 |
+
| `READY_DAEMON` 永远等不到,但进程活着 | C 运行时把管道 stdout 全缓冲了 | `setvbuf(stdout, NULL, _IOLBF, 0)`(`kev_daemon.cpp` 里已加) |
|
| 188 |
+
| `model_init ret=-2`,日志 `core mask 0x00000000 is not contiguous` | `cfg.run_core_mask` 默认 0 | 传 `0xff`(8 核) |
|
| 189 |
+
| 反复拆建 proxy,模组从总线掉线 | proxy 全机唯一 | 复用同一个 proxy 进程,只启停模型 |
|
| 190 |
+
| 改了代码但行为没变 | 残留进程占着端口 / 旧 exe | 先 `taskkill /F /IM daemon.exe`,用随机端口自测 |
|
| 191 |
+
| 断电后设备认不到 | 进了 MaskROM(`2207:180B`) | 跑一次 `uf` 工具,约 2 秒恢复 |
|
| 192 |
+
| `.cmd` 里的中文把命令截断 | cmd 按 OEM 代码页(936)读文件 | `.cmd` / `.bat` / `.ps1` 一律**纯 ASCII + CRLF** |
|
| 193 |
+
|
| 194 |
+
完整清单见上游项目的 `PITFALLS.md`(更细,含每一步的日志与判据)。
|
| 195 |
+
|
| 196 |
+
---
|
| 197 |
+
|
| 198 |
+
## 7. 已知限制
|
| 199 |
+
|
| 200 |
+
- **一次请求 = K+1 次 prefill**(`<decide>` 一次 + 每个 `</opt>` 一次)。图的输出固定 `[1,1,248320]`,
|
| 201 |
+
只保留最后一个位置;公开 API 没有 `num_logits_to_keep`,`clear_kvcache` 也没有 truncate。
|
| 202 |
+
这是硬下限,不是实现选择。
|
| 203 |
+
- 每次调用有约 **50 ms 固定开销**,与序列长度无关。延迟看**调用次数**,不是 token 数。
|
| 204 |
+
- 服务对整次决策加**全局锁**:并发是安全的,但会排队。`K+1` 次 prefill 共用一份 KV cache,
|
| 205 |
+
不加锁会串。
|
| 206 |
+
- 量化格式被工具链锁死为 `w4a16 + normal + group32`,做不了更高精度的对照版本。
|
host/e2e_check.py
ADDED
|
@@ -0,0 +1,176 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""端到端验证在线服务:从 decision-v2 取题 → 走 HTTP → 与标签比。
|
| 2 |
+
|
| 3 |
+
这一步验的是"服务路径"而不是"批量路径":
|
| 4 |
+
1. kev_serve 里的 render/option_text 与 plan_a 是否一致(不一致准确率会掉)
|
| 5 |
+
2. score 题型能不能跑通(此前只测过 choice/noul)
|
| 6 |
+
3. 三种题型混在一起时的逐题正确率
|
| 7 |
+
|
| 8 |
+
用法:在 Windows 上运行(服务与 daemon 都在本机)。
|
| 9 |
+
python e2e_check.py [每题最多选项数] [最多记录数]
|
| 10 |
+
"""
|
| 11 |
+
import json
|
| 12 |
+
import os
|
| 13 |
+
import random
|
| 14 |
+
import subprocess
|
| 15 |
+
import sys
|
| 16 |
+
import time
|
| 17 |
+
import urllib.request
|
| 18 |
+
|
| 19 |
+
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
| 20 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 21 |
+
PORT = random.randint(30000, 39999)
|
| 22 |
+
# 数据集默认 Windows 本机;在其他机器上跑时用 KEV_DEV 指过去
|
| 23 |
+
DEV = os.environ.get("KEV_DEV") or r"D:\kevsrc\evals\decision-v2\development.jsonl"
|
| 24 |
+
# 给了 KEV_URL 就直接连**已有服务**,不再自己起进程 ——
|
| 25 |
+
# 这样可以从另一台机器(走 SSH 端口转发)验证同一个实例,
|
| 26 |
+
# 而不是每次测都新起一个(新起的不代表线上那个的配置)。
|
| 27 |
+
EXTERNAL_URL = os.environ.get("KEV_URL")
|
| 28 |
+
if EXTERNAL_URL and "/v1/" not in EXTERNAL_URL:
|
| 29 |
+
# 允许只给 base(http://host:port),自动补全路径
|
| 30 |
+
EXTERNAL_URL = EXTERNAL_URL.rstrip("/") + "/v1/systemone"
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def render(v, indent=0):
|
| 34 |
+
pad = " " * indent
|
| 35 |
+
if v is None:
|
| 36 |
+
return ""
|
| 37 |
+
if isinstance(v, (str, int, float, bool)):
|
| 38 |
+
return str(v)
|
| 39 |
+
if isinstance(v, list):
|
| 40 |
+
return "\n".join(f"{pad}- {render(x, indent + 1).lstrip()}" for x in v)
|
| 41 |
+
return "\n".join(
|
| 42 |
+
f"{pad}{k}:\n{render(x, indent + 1)}" if isinstance(x, (dict, list))
|
| 43 |
+
else f"{pad}{k}: {render(x)}" for k, x in v.items())
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def option_text(name, desc):
|
| 47 |
+
return name if desc is None or desc == "" else f"{name}: {render(desc)}"
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def pick(path, max_recs, max_opts, seed=7):
|
| 51 |
+
"""抽记录,只留 2..max_opts 个选项且带 label 的题。"""
|
| 52 |
+
recs = []
|
| 53 |
+
with open(path, encoding="utf-8") as f:
|
| 54 |
+
for line in f:
|
| 55 |
+
r = json.loads(line)
|
| 56 |
+
qs, expect = {}, {}
|
| 57 |
+
for qid, q in r["questions"].items():
|
| 58 |
+
c = q.get("criteria") or {}
|
| 59 |
+
lab = q.get("label")
|
| 60 |
+
if lab is None:
|
| 61 |
+
continue
|
| 62 |
+
if q["type"] == "noul":
|
| 63 |
+
k = 2
|
| 64 |
+
exp = bool(lab) if isinstance(lab, bool) else str(lab).lower() in ("true", "yes")
|
| 65 |
+
elif q["type"] == "score":
|
| 66 |
+
k = len(c)
|
| 67 |
+
exp = lab if isinstance(lab, int) else None # 整数档位索引
|
| 68 |
+
else:
|
| 69 |
+
k = len(c)
|
| 70 |
+
exp = lab
|
| 71 |
+
if not (2 <= k <= max_opts):
|
| 72 |
+
continue
|
| 73 |
+
if q["type"] == "choice" and lab not in c:
|
| 74 |
+
continue
|
| 75 |
+
qs[qid] = {"type": q["type"],
|
| 76 |
+
"instructions": q["instructions"],
|
| 77 |
+
"criteria": c}
|
| 78 |
+
expect[qid] = {"kind": q["type"], "answer": exp,
|
| 79 |
+
"label": lab, "opts": k}
|
| 80 |
+
if qs:
|
| 81 |
+
recs.append({"state": r["state"], "questions": qs, "expect": expect})
|
| 82 |
+
# 先把**全部**候选读进来再打乱抽取。
|
| 83 |
+
# 早先是"读到 max_recs 就 break 然后打乱",结果前 12 条记录恰好全是 noul 题,
|
| 84 |
+
# 题型分布完全偏了(还以为是模型问题)。
|
| 85 |
+
random.Random(seed).shuffle(recs)
|
| 86 |
+
return recs[:max_recs]
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def main():
|
| 90 |
+
max_opts = int(sys.argv[1]) if len(sys.argv) > 1 else 4
|
| 91 |
+
max_recs = int(sys.argv[2]) if len(sys.argv) > 2 else 20
|
| 92 |
+
|
| 93 |
+
recs = pick(DEV, max_recs, max_opts)
|
| 94 |
+
nq = sum(len(r["questions"]) for r in recs)
|
| 95 |
+
print(f"取 {len(recs)} 条记录 / {nq} 道题(选项数 ≤{max_opts})", flush=True)
|
| 96 |
+
|
| 97 |
+
if EXTERNAL_URL:
|
| 98 |
+
print(f"使用已运行的服务:{EXTERNAL_URL}", flush=True)
|
| 99 |
+
p = None
|
| 100 |
+
else:
|
| 101 |
+
print("启动服务 …", flush=True)
|
| 102 |
+
p = subprocess.Popen([sys.executable, os.path.join(HERE, "kev_serve.py")],
|
| 103 |
+
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
| 104 |
+
env={**os.environ, "KEV_PORT": str(PORT)})
|
| 105 |
+
t0 = time.time()
|
| 106 |
+
if p is not None:
|
| 107 |
+
ready = False
|
| 108 |
+
while True:
|
| 109 |
+
line = p.stdout.readline()
|
| 110 |
+
if not line:
|
| 111 |
+
break
|
| 112 |
+
if b"listening on" in line:
|
| 113 |
+
ready = True
|
| 114 |
+
break
|
| 115 |
+
if time.time() - t0 > 240:
|
| 116 |
+
break
|
| 117 |
+
if not ready:
|
| 118 |
+
print("服务没起来", flush=True)
|
| 119 |
+
p.kill()
|
| 120 |
+
sys.exit(1)
|
| 121 |
+
print(f"服务就绪({time.time()-t0:.1f}s)", flush=True)
|
| 122 |
+
|
| 123 |
+
stat = {"choice": [0, 0], "noul": [0, 0], "score": [0, 0]}
|
| 124 |
+
fails = []
|
| 125 |
+
t_all = time.time()
|
| 126 |
+
for ri, r in enumerate(recs):
|
| 127 |
+
body = json.dumps({"state": r["state"], "questions": r["questions"]},
|
| 128 |
+
ensure_ascii=False).encode()
|
| 129 |
+
req = urllib.request.Request(EXTERNAL_URL or f"http://127.0.0.1:{PORT}/v1/systemone",
|
| 130 |
+
data=body, headers={"Content-Type": "application/json"})
|
| 131 |
+
with urllib.request.urlopen(req, timeout=600) as x:
|
| 132 |
+
resp = json.loads(x.read().decode())
|
| 133 |
+
ans = resp.get("answers", {})
|
| 134 |
+
for qid, ex in r["expect"].items():
|
| 135 |
+
got = ans.get(qid)
|
| 136 |
+
if not got or "error" in got:
|
| 137 |
+
fails.append((ri, qid, "ERR", str(got)[:60]))
|
| 138 |
+
continue
|
| 139 |
+
k = ex["kind"]
|
| 140 |
+
stat[k][1] += 1
|
| 141 |
+
if k == "noul":
|
| 142 |
+
ok = bool(got["answer"]) == bool(ex["answer"])
|
| 143 |
+
elif k == "score":
|
| 144 |
+
# label 是整数档位索引 → 与 argmax 档位比(不是与期望等级 score 比)
|
| 145 |
+
ok = int(got["argmax"]) == int(ex["label"])
|
| 146 |
+
else:
|
| 147 |
+
ok = got["answer"] == ex["answer"]
|
| 148 |
+
if ok:
|
| 149 |
+
stat[k][0] += 1
|
| 150 |
+
else:
|
| 151 |
+
got_v = got.get("argmax") if k == "score" else got.get("answer")
|
| 152 |
+
fails.append((ri, qid, k, f"{got_v} != {ex['label']}"))
|
| 153 |
+
if (ri + 1) % 5 == 0:
|
| 154 |
+
print(f" {ri+1}/{len(recs)} …", flush=True)
|
| 155 |
+
|
| 156 |
+
dt = time.time() - t_all
|
| 157 |
+
print("\n=== 分题型准确率 ===", flush=True)
|
| 158 |
+
tot_o = tot_n = 0
|
| 159 |
+
for k, (o, n) in stat.items():
|
| 160 |
+
if n:
|
| 161 |
+
print(f" {k:7s} {o}/{n} = {o/n:.3f}", flush=True)
|
| 162 |
+
tot_o += o
|
| 163 |
+
tot_n += n
|
| 164 |
+
print(f" 合计 {tot_o}/{tot_n} = {tot_o/max(tot_n,1):.3f}", flush=True)
|
| 165 |
+
print(f"\n{nq} 道题共 {dt:.1f}s,平均 {dt/max(nq,1):.2f}s/题", flush=True)
|
| 166 |
+
if fails:
|
| 167 |
+
print(f"\n失败 {len(fails)} 条,前 10:", flush=True)
|
| 168 |
+
for f in fails[:10]:
|
| 169 |
+
print(" ", f, flush=True)
|
| 170 |
+
# KEV_URL 模式下没有子进程(p is None),不能无条件 kill
|
| 171 |
+
if p:
|
| 172 |
+
p.kill()
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
if __name__ == "__main__":
|
| 176 |
+
main()
|
host/kev_daemon.cpp
ADDED
|
@@ -0,0 +1,411 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Copyright (c) 2026 by Rockchip Electronics Co., Ltd. All Rights Reserved.
|
| 2 |
+
//
|
| 3 |
+
// Licensed under the Apache License, Version 2.0 (the "License");
|
| 4 |
+
// you may not use this file except in compliance with the License.
|
| 5 |
+
// You may obtain a copy of the License at
|
| 6 |
+
//
|
| 7 |
+
// http://www.apache.org/licenses/LICENSE-2.0
|
| 8 |
+
//
|
| 9 |
+
// Unless required by applicable law or agreed to in writing, software
|
| 10 |
+
// distributed under the License is distributed on an "AS IS" BASIS,
|
| 11 |
+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 12 |
+
// See the License for the specific language governing permissions and
|
| 13 |
+
// limitations under the License.
|
| 14 |
+
|
| 15 |
+
#include "rknn3_session_test_common.h"
|
| 16 |
+
|
| 17 |
+
// ---- 我们加的:output_callback 里直接拿到 logits 张量并落盘 ----
|
| 18 |
+
// sampling_callback 不会被调用(运行时内部采样),但 output_tensors[0] 就是 vocabulary logits
|
| 19 |
+
static int g_kev_dumped = 0;
|
| 20 |
+
static int g_kev_seq = 0; // 当前第几条序列,用来给 logits 文件编号
|
| 21 |
+
static char g_out_path[512] = "D:\\rk1828-console\\kevrunner\\out\\cur.bin";
|
| 22 |
+
|
| 23 |
+
static int kev_output_callback(void* userdata, rknn3_tensor* outs, uint32_t n,
|
| 24 |
+
LLMOutputCallbackState state) {
|
| 25 |
+
(void)userdata;
|
| 26 |
+
if (!g_kev_dumped && n > 0 && state == RKLLM_OUTPUT_CALLBACK_PREFILL_FINISHED) {
|
| 27 |
+
const char* p = g_out_path;
|
| 28 |
+
FILE* f = fopen(p, "wb");
|
| 29 |
+
if (f) {
|
| 30 |
+
size_t sz = outs[0].mem->size;
|
| 31 |
+
fwrite(outs[0].mem->virt_addr, 1, sz, f);
|
| 32 |
+
fclose(f);
|
| 33 |
+
printf("[kev] dumped %zu bytes, state=%d, name=%s\n", sz, (int)state,
|
| 34 |
+
outs[0].attr->name);
|
| 35 |
+
g_kev_dumped = 1;
|
| 36 |
+
} else {
|
| 37 |
+
printf("[kev] fopen failed\n");
|
| 38 |
+
}
|
| 39 |
+
fflush(stdout);
|
| 40 |
+
}
|
| 41 |
+
return 0;
|
| 42 |
+
}
|
| 43 |
+
|
| 44 |
+
// ---- 极简 base64 解码,给 TOK 命令用(文本里有空格换行,用 base64 最省事)----
|
| 45 |
+
static int b64val(char c) {
|
| 46 |
+
if (c >= 'A' && c <= 'Z') return c - 'A';
|
| 47 |
+
if (c >= 'a' && c <= 'z') return c - 'a' + 26;
|
| 48 |
+
if (c >= '0' && c <= '9') return c - '0' + 52;
|
| 49 |
+
if (c == '+') return 62;
|
| 50 |
+
if (c == '/') return 63;
|
| 51 |
+
return -1;
|
| 52 |
+
}
|
| 53 |
+
static int b64decode(const char* in, int inlen, char* out, int outmax) {
|
| 54 |
+
int v = 0, bits = 0, n = 0;
|
| 55 |
+
for (int i = 0; i < inlen; i++) {
|
| 56 |
+
if (in[i] == '=' || in[i] == '\n' || in[i] == '\r') continue;
|
| 57 |
+
int d = b64val(in[i]);
|
| 58 |
+
if (d < 0) continue;
|
| 59 |
+
v = (v << 6) | d;
|
| 60 |
+
bits += 6;
|
| 61 |
+
if (bits >= 8) {
|
| 62 |
+
bits -= 8;
|
| 63 |
+
if (n < outmax) out[n++] = (char)((v >> bits) & 0xFF);
|
| 64 |
+
}
|
| 65 |
+
}
|
| 66 |
+
return n;
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
int main(int argc, char** argv)
|
| 70 |
+
{
|
| 71 |
+
// keep_history / ignore_eos_token / key_path are optional and parsed by
|
| 72 |
+
// argc position: passing fewer falls back to the defaults in [] below.
|
| 73 |
+
// Usage: %s <rknn> <weight> <tokenizer> <embedding> <max_context_len> <max_new_tokens> <core_mask> [keep_history] [ignore_eos_token]
|
| 74 |
+
// [key_path]
|
| 75 |
+
if (argc < 8 || argc > 12) { // 12 = 多一个 ids 文件
|
| 76 |
+
LOGW(
|
| 77 |
+
"Usage: %s <rknn_path> <weight_path> <tokenizer.gguf> <embedding.bin> <max_context_len> <max_new_tokens> <core_mask> [keep_history] [ignore_eos_token] [key_path]\n ",
|
| 78 |
+
argv[0]);
|
| 79 |
+
LOGW(
|
| 80 |
+
"Such as: %s ./model/Qwen2.5-0.5B.rknn ./model/Qwen2.5-0.5B.weight ./model/Qwen2.5-0.5B.tokenizer.gguf ./model/Qwen2.5-0.5B-embedding.bin 1024 256 0xff 0 0 ./model/key.env\n",
|
| 81 |
+
argv[0]);
|
| 82 |
+
LOGW(
|
| 83 |
+
"For encrypted model: %s ./model/Qwen2.5-0.5B.rknn ./model/Qwen2.5-0.5B.weight ./model/Qwen2.5-0.5B.tokenizer.gguf ./model/Qwen2.5-0.5B-embedding.bin 1024 256 0xff 0 0 ./model/key.env\n",
|
| 84 |
+
argv[0]);
|
| 85 |
+
|
| 86 |
+
return -1;
|
| 87 |
+
}
|
| 88 |
+
|
| 89 |
+
int ret = 0;
|
| 90 |
+
int i = 0; // 已处理的序列数(放在这里避免 goto 跨初始化)
|
| 91 |
+
char* model_path = argv[1];
|
| 92 |
+
char* weight_path = argv[2];
|
| 93 |
+
const char* tokenizer_path = argv[3];
|
| 94 |
+
const char* embedding_path = argv[4];
|
| 95 |
+
int32_t max_context_len = atoi(argv[5]);
|
| 96 |
+
int32_t max_new_tokens = atoi(argv[6]);
|
| 97 |
+
uint32_t core_mask = strtoul(argv[7], nullptr, 16);
|
| 98 |
+
// Optional positional args; defaults keep the old behaviour.
|
| 99 |
+
int keep_history = (argc > 8) ? atoi(argv[8]) : 0;
|
| 100 |
+
bool ignore_eos_token = (argc > 9) ? (atoi(argv[9]) != 0) : false;
|
| 101 |
+
const char* key_path = (argc > 10) ? argv[10] : nullptr;
|
| 102 |
+
const char* ids_path = (argc > 11) ? argv[11] : nullptr; // 每行一串 token id
|
| 103 |
+
|
| 104 |
+
// key 文件是占位参数;不存在就当没给,否则会被当成加密模型去解密
|
| 105 |
+
if (key_path) {
|
| 106 |
+
FILE* kf = fopen(key_path, "r");
|
| 107 |
+
if (kf) fclose(kf);
|
| 108 |
+
else {
|
| 109 |
+
printf("[kev] key file '%s' not found, ignore it\n", key_path);
|
| 110 |
+
key_path = nullptr;
|
| 111 |
+
}
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
rknn3_devices devs;
|
| 115 |
+
rknn3_context ctx = 0;
|
| 116 |
+
rknn3_input_output_num io_num;
|
| 117 |
+
VocabInfo vocab_info;
|
| 118 |
+
Tokenizer* tokenizer = nullptr;
|
| 119 |
+
struct stat emb_st;
|
| 120 |
+
struct embedding_info embedding_info;
|
| 121 |
+
RKLLMCallback callback;
|
| 122 |
+
rknn3_test_session_userdata_t session_userdata;
|
| 123 |
+
rknn3_llm_param params;
|
| 124 |
+
int n_params = 1;
|
| 125 |
+
RKLLMRunState state;
|
| 126 |
+
rknn3_lora loras_enabled[RKNN3_MAX_LORA_NUM];
|
| 127 |
+
rknn3_llm_config llm_config;
|
| 128 |
+
rknn3_kvcache_policy kvcache_policy = RKNN3_KVCACHE_POLICY_NORMAL;
|
| 129 |
+
rknn3_kvcache_policy_param kvcache_policy_param;
|
| 130 |
+
int llm_n_inputs = 1;
|
| 131 |
+
rknn3_llm_input* llm_inputs[llm_n_inputs];
|
| 132 |
+
rknn3_llm_infer_param llm_infer_param;
|
| 133 |
+
|
| 134 |
+
memset(&devs, 0, sizeof(devs));
|
| 135 |
+
memset(&io_num, 0, sizeof(io_num));
|
| 136 |
+
memset(&vocab_info, 0, sizeof(vocab_info));
|
| 137 |
+
memset(&emb_st, 0, sizeof(emb_st));
|
| 138 |
+
memset(&embedding_info, 0, sizeof(embedding_info));
|
| 139 |
+
memset(&callback, 0, sizeof(callback));
|
| 140 |
+
memset(¶ms, 0, sizeof(params));
|
| 141 |
+
memset(&state, 0, sizeof(state));
|
| 142 |
+
memset(&llm_config, 0, sizeof(llm_config));
|
| 143 |
+
memset(loras_enabled, 0, sizeof(loras_enabled));
|
| 144 |
+
memset(&kvcache_policy_param, 0, sizeof(kvcache_policy_param));
|
| 145 |
+
state.loras_enabled = loras_enabled;
|
| 146 |
+
memset(llm_inputs, 0, sizeof(llm_inputs));
|
| 147 |
+
memset(&llm_infer_param, 0, sizeof(llm_infer_param));
|
| 148 |
+
memset(&session_userdata, 0, sizeof(session_userdata));
|
| 149 |
+
|
| 150 |
+
// set output tensors index for output callback
|
| 151 |
+
rknn3_tensor output_tensors[1];
|
| 152 |
+
int n_output_tensors = 1;
|
| 153 |
+
int output_tensors_index[1] = {0};
|
| 154 |
+
memset(output_tensors, 0, sizeof(output_tensors));
|
| 155 |
+
|
| 156 |
+
std::vector<std::string> random_prompts = {
|
| 157 |
+
"请解释一下相对论的基本概念。",
|
| 158 |
+
"Please explain the basic concept of relativity",
|
| 159 |
+
};
|
| 160 |
+
int test_num = random_prompts.size();
|
| 161 |
+
|
| 162 |
+
// 若给了 ids 文件,就走批量模式:每行一串 token id,逐条 prefill 并 dump logits
|
| 163 |
+
std::vector<std::vector<int32_t>> seqs;
|
| 164 |
+
if (ids_path) {
|
| 165 |
+
FILE* fp = fopen(ids_path, "r");
|
| 166 |
+
if (!fp) { printf("cannot open ids file: %s\n", ids_path); return -1; }
|
| 167 |
+
char line[65536];
|
| 168 |
+
while (fgets(line, sizeof(line), fp)) {
|
| 169 |
+
std::vector<int32_t> one;
|
| 170 |
+
char* p = line;
|
| 171 |
+
while (*p) {
|
| 172 |
+
char* end = nullptr;
|
| 173 |
+
long v = strtol(p, &end, 10);
|
| 174 |
+
if (end == p) break;
|
| 175 |
+
one.push_back((int32_t)v);
|
| 176 |
+
p = end;
|
| 177 |
+
}
|
| 178 |
+
if (!one.empty()) seqs.push_back(one);
|
| 179 |
+
}
|
| 180 |
+
fclose(fp);
|
| 181 |
+
test_num = (int)seqs.size();
|
| 182 |
+
printf("[kev] loaded %d sequences from %s\n", test_num, ids_path);
|
| 183 |
+
}
|
| 184 |
+
|
| 185 |
+
printf("*******************************NEW TEST**********************************\n");
|
| 186 |
+
|
| 187 |
+
// find devices (devs already zero-initialised above)
|
| 188 |
+
const char* device_id = nullptr;
|
| 189 |
+
ret = rknn3_find_devices(&devs);
|
| 190 |
+
if (ret != RKNN3_SUCCESS || devs.n_devices == 0) {
|
| 191 |
+
printf("rknn3_find_devices failed, ret = %d\n", ret);
|
| 192 |
+
goto exit;
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
// init context and model
|
| 196 |
+
device_id = devs.devices[0].id;
|
| 197 |
+
ret = init_context_and_model(&ctx, model_path, weight_path, core_mask, key_path, device_id);
|
| 198 |
+
if (ret != RKNN3_SUCCESS) {
|
| 199 |
+
printf("rknn3_session_init_model fail! ret = %d\n", ret);
|
| 200 |
+
goto exit;
|
| 201 |
+
}
|
| 202 |
+
|
| 203 |
+
// Get Model Output Info for output callback (if needed)
|
| 204 |
+
if (n_output_tensors > 0) {
|
| 205 |
+
ret = setup_output_tensors(ctx, output_tensors, n_output_tensors, output_tensors_index, &io_num);
|
| 206 |
+
if (ret < 0) {
|
| 207 |
+
goto exit;
|
| 208 |
+
}
|
| 209 |
+
}
|
| 210 |
+
|
| 211 |
+
// get tokenizer and embedding
|
| 212 |
+
ret = get_tokenizer_and_embedding(tokenizer_path, &vocab_info, &tokenizer, &embedding_info, embedding_path, &emb_st);
|
| 213 |
+
if (ret < 0) {
|
| 214 |
+
printf("get_tokenizer_and_embedding failed, ret = %d\n", ret);
|
| 215 |
+
goto exit;
|
| 216 |
+
}
|
| 217 |
+
|
| 218 |
+
ret = rknn3_query(ctx, RKNN3_QUERY_LLM_CONFIG, &llm_config, sizeof(rknn3_llm_config));
|
| 219 |
+
if (ret != RKNN3_SUCCESS) {
|
| 220 |
+
printf("rknn3_query llm config failed! ret=%d", ret);
|
| 221 |
+
goto exit;
|
| 222 |
+
}
|
| 223 |
+
|
| 224 |
+
// Set basic session parameters
|
| 225 |
+
build_default_llm_params(¶ms, &vocab_info, max_context_len, ignore_eos_token);
|
| 226 |
+
|
| 227 |
+
// init session
|
| 228 |
+
session_userdata.session = rknn3_session_init(ctx, ¶ms, n_params);
|
| 229 |
+
if (!session_userdata.session) {
|
| 230 |
+
printf("rknn3_session_init failed, ret = %d\n", ret);
|
| 231 |
+
goto exit;
|
| 232 |
+
}
|
| 233 |
+
|
| 234 |
+
// set session callback
|
| 235 |
+
build_default_callback(&callback, tokenizer, &embedding_info, &session_userdata);
|
| 236 |
+
// sampling_callback 不会被运行时调用(内部采样),logits 从 output_callback 拿
|
| 237 |
+
callback.sampling_callback = NULL;
|
| 238 |
+
callback.sampling_userdata = NULL;
|
| 239 |
+
// 同时打开 output_callback:若它能触发而 sampling 不能,说明 sampling 被门控
|
| 240 |
+
callback.output_callback = kev_output_callback;
|
| 241 |
+
callback.output_userdata = &embedding_info;
|
| 242 |
+
callback.output_tensors = output_tensors;
|
| 243 |
+
callback.n_output_tensors = n_output_tensors;
|
| 244 |
+
|
| 245 |
+
ret = rknn3_session_set_callback(session_userdata.session, &callback);
|
| 246 |
+
if (ret != RKNN3_SUCCESS) {
|
| 247 |
+
printf("rknn3_session_set_callback failed, ret = %d\n", ret);
|
| 248 |
+
goto exit;
|
| 249 |
+
}
|
| 250 |
+
|
| 251 |
+
// set/update chat template
|
| 252 |
+
if (0) {
|
| 253 |
+
std::string system_prompt = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n";
|
| 254 |
+
std::string prompt_prefix = "<|im_start|>user\n";
|
| 255 |
+
std::string prompt_postfix = "<|im_end|>\n<|im_start|>assistant\n";
|
| 256 |
+
ret = rknn3_session_set_chat_template(session_userdata.session, system_prompt.c_str(), prompt_prefix.c_str(), prompt_postfix.c_str());
|
| 257 |
+
if (ret != RKNN3_SUCCESS) {
|
| 258 |
+
printf("rknn3_session_set_chat_template failed, ret = %d\n", ret);
|
| 259 |
+
goto exit;
|
| 260 |
+
}
|
| 261 |
+
}
|
| 262 |
+
|
| 263 |
+
// set kvcache policy
|
| 264 |
+
ret = rknn3_session_set_kvcache_policy(session_userdata.session, kvcache_policy, nullptr);
|
| 265 |
+
if (ret != RKNN3_SUCCESS) {
|
| 266 |
+
printf("rknn3_session_set_kvcache_policy failed, ret=%d\n", ret);
|
| 267 |
+
goto exit;
|
| 268 |
+
}
|
| 269 |
+
|
| 270 |
+
print_llm_config(&llm_config);
|
| 271 |
+
|
| 272 |
+
// Test the generic session inference: each iteration builds a typical prompt request,
|
| 273 |
+
// runs one round of generation to show the standard prompt input workflow.
|
| 274 |
+
char in_line[65536];
|
| 275 |
+
// 关键:stdout 接管道时 C 运行库默认全缓冲,READY_DAEMON 会卡在缓冲区里,
|
| 276 |
+
// 父进程永远等不到。必须显式改成行缓冲(只 fflush 一次不够,后面每行都要)。
|
| 277 |
+
setvbuf(stdout, NULL, _IOLBF, 0);
|
| 278 |
+
printf("READY_DAEMON\n");
|
| 279 |
+
fflush(stdout);
|
| 280 |
+
while (fgets(in_line, sizeof(in_line), stdin)) {
|
| 281 |
+
if (strncmp(in_line, "quit", 4) == 0) break;
|
| 282 |
+
|
| 283 |
+
// OUT <path> -> 指定下一次 dump 的落盘路径
|
| 284 |
+
if (strncmp(in_line, "OUT ", 4) == 0) {
|
| 285 |
+
char* q = in_line + 4;
|
| 286 |
+
while (*q == ' ') q++;
|
| 287 |
+
int n = (int)strlen(q);
|
| 288 |
+
while (n > 0 && (q[n-1] == '\n' || q[n-1] == '\r')) n--;
|
| 289 |
+
if (n > 0 && n < (int)sizeof(g_out_path)) {
|
| 290 |
+
memcpy(g_out_path, q, n);
|
| 291 |
+
g_out_path[n] = 0;
|
| 292 |
+
printf("OUTSET %s\n", g_out_path);
|
| 293 |
+
fflush(stdout);
|
| 294 |
+
}
|
| 295 |
+
continue;
|
| 296 |
+
}
|
| 297 |
+
|
| 298 |
+
// TOK <base64 text> -> 打印 "IDS n id1 id2 ..."
|
| 299 |
+
if (strncmp(in_line, "TOK ", 4) == 0) {
|
| 300 |
+
static char tbuf[65536];
|
| 301 |
+
static int32_t tids[8192];
|
| 302 |
+
int raw = b64decode(in_line + 4, (int)strlen(in_line + 4), tbuf, sizeof(tbuf) - 1);
|
| 303 |
+
tbuf[raw] = 0;
|
| 304 |
+
printf("[kev] TOK raw=%d head='%.40s' tok=%p\n", raw, tbuf, (void*)tokenizer);
|
| 305 |
+
fflush(stdout);
|
| 306 |
+
int n = tokenizer->Tokenize(tbuf, raw, tids, 8192);
|
| 307 |
+
printf("IDS %d", n);
|
| 308 |
+
for (int k = 0; k < n; k++) printf(" %d", tids[k]);
|
| 309 |
+
printf("\n");
|
| 310 |
+
fflush(stdout);
|
| 311 |
+
continue;
|
| 312 |
+
}
|
| 313 |
+
// R <ids> = 清空 kvcache 后从头 prefill(keep_history=0)
|
| 314 |
+
// A <ids> = 接在上一次之后追加 prefill(keep_history=1)
|
| 315 |
+
// 不带前缀 = 老行为(等价 R)
|
| 316 |
+
bool append = false;
|
| 317 |
+
char* pp = in_line;
|
| 318 |
+
if (in_line[0] == 'A' && in_line[1] == ' ') { append = true; pp = in_line + 2; }
|
| 319 |
+
else if (in_line[0] == 'R' && in_line[1] == ' ') { pp = in_line + 2; }
|
| 320 |
+
{
|
| 321 |
+
std::vector<int32_t> one;
|
| 322 |
+
char* p = pp;
|
| 323 |
+
while (*p) {
|
| 324 |
+
char* e = nullptr;
|
| 325 |
+
long v = strtol(p, &e, 10);
|
| 326 |
+
if (e == p) break;
|
| 327 |
+
one.push_back((int32_t)v);
|
| 328 |
+
p = e;
|
| 329 |
+
}
|
| 330 |
+
if (one.empty()) continue;
|
| 331 |
+
seqs.push_back(one);
|
| 332 |
+
}
|
| 333 |
+
printf("\n--------------------Input[%d]-------------------- \n", i);
|
| 334 |
+
|
| 335 |
+
llm_infer_param = build_default_infer_param(max_new_tokens, append ? 1 : 0);
|
| 336 |
+
|
| 337 |
+
g_kev_seq = i;
|
| 338 |
+
g_kev_dumped = 0;
|
| 339 |
+
// A 模式接着上次的 kvcache;R 模式(默认)从头开始
|
| 340 |
+
if (!append) {
|
| 341 |
+
rknn3_session_clear_kvcache(session_userdata.session, RKNN3_KVCACHE_CLEAR_ALL);
|
| 342 |
+
}
|
| 343 |
+
llm_inputs[0] = build_llm_input_tokens(seqs.back().data(), seqs.back().size(), false, false);
|
| 344 |
+
|
| 345 |
+
printf("\n--------------------Output---------------------- \n");
|
| 346 |
+
|
| 347 |
+
rknn3_test_timing_mark_start(&session_userdata.timing);
|
| 348 |
+
ret = rknn3_session_run(session_userdata.session, llm_inputs[0], llm_n_inputs, &llm_infer_param);
|
| 349 |
+
if (ret != RKNN3_SUCCESS) {
|
| 350 |
+
printf("rknn3_session_run failed, ret = %d\n", ret);
|
| 351 |
+
goto exit;
|
| 352 |
+
}
|
| 353 |
+
rknn3_test_timing_mark_end(&session_userdata.timing);
|
| 354 |
+
printf("DONE %d\n", i); // 每处理完一条就给客户端一个信号
|
| 355 |
+
fflush(stdout);
|
| 356 |
+
i++;
|
| 357 |
+
|
| 358 |
+
ret = rknn3_session_query_state(session_userdata.session, &state);
|
| 359 |
+
if (ret != RKNN3_SUCCESS) {
|
| 360 |
+
printf("rknn3_session_query_state failed, ret=%d\n", ret);
|
| 361 |
+
goto exit;
|
| 362 |
+
}
|
| 363 |
+
|
| 364 |
+
print_run_state(&state, true);
|
| 365 |
+
|
| 366 |
+
// clear kvcache
|
| 367 |
+
if (state.n_total_tokens >= (state.n_max_tokens - max_new_tokens)) {
|
| 368 |
+
ret = rknn3_session_clear_kvcache(session_userdata.session, RKNN3_KVCACHE_CLEAR_ALL);
|
| 369 |
+
if (ret != RKNN3_SUCCESS) {
|
| 370 |
+
printf("rknn3_session_clear_kvcache failed, ret=%d\n", ret);
|
| 371 |
+
goto exit;
|
| 372 |
+
}
|
| 373 |
+
}
|
| 374 |
+
|
| 375 |
+
print_perf_stats(&state, &session_userdata.timing);
|
| 376 |
+
|
| 377 |
+
// Free any heap memory owned by this iteration's input(s).
|
| 378 |
+
for (int j = 0; j < llm_n_inputs; j++) {
|
| 379 |
+
release_llm_input(ctx, llm_inputs[j]);
|
| 380 |
+
}
|
| 381 |
+
}
|
| 382 |
+
|
| 383 |
+
exit:
|
| 384 |
+
for (int j = 0; j < llm_n_inputs; j++) {
|
| 385 |
+
release_llm_input(ctx, llm_inputs[j]);
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
if (n_output_tensors > 0) {
|
| 389 |
+
cleanup_output_tensors(ctx, output_tensors, n_output_tensors);
|
| 390 |
+
}
|
| 391 |
+
|
| 392 |
+
if (session_userdata.session) {
|
| 393 |
+
rknn3_session_destroy(session_userdata.session);
|
| 394 |
+
}
|
| 395 |
+
|
| 396 |
+
if (ctx) {
|
| 397 |
+
rknn3_destroy(ctx);
|
| 398 |
+
}
|
| 399 |
+
|
| 400 |
+
embedding_info_release(&embedding_info, &emb_st);
|
| 401 |
+
|
| 402 |
+
if (tokenizer) {
|
| 403 |
+
delete tokenizer;
|
| 404 |
+
}
|
| 405 |
+
|
| 406 |
+
printf("*******************************END TEST**********************************\n");
|
| 407 |
+
|
| 408 |
+
fflush(stdout);
|
| 409 |
+
|
| 410 |
+
return 0;
|
| 411 |
+
}
|
host/kev_serve.py
ADDED
|
@@ -0,0 +1,450 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Kev 在 RK1828 上的在线决策服务(跑在 Windows)。
|
| 2 |
+
|
| 3 |
+
架构:
|
| 4 |
+
HTTP (本文件) --stdin/stdout--> daemon.exe --NPU--> logits
|
| 5 |
+
host 侧 numpy 做 反解 + 指针打分
|
| 6 |
+
|
| 7 |
+
一次请求 = K+1 次 prefill(<decide> 一次 + 每个 </opt> 一次),
|
| 8 |
+
因为图的输出只保留最后一个位置,取别的位置只能把序列截断重跑。
|
| 9 |
+
|
| 10 |
+
接口:POST /v1/systemone
|
| 11 |
+
{"state": <任意 JSON>, "questions": {"q1": {"type":"choice|noul|score",
|
| 12 |
+
"instructions": "...", "criteria": {...} | [...]}}}
|
| 13 |
+
"""
|
| 14 |
+
import base64
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import subprocess
|
| 18 |
+
import sys
|
| 19 |
+
import threading
|
| 20 |
+
import time
|
| 21 |
+
import traceback
|
| 22 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 23 |
+
|
| 24 |
+
import numpy as np
|
| 25 |
+
|
| 26 |
+
# 所有路径都从本文件位置推导,不写死盘符 —— 交付后整个目录搬到任何地方都能用。
|
| 27 |
+
# 布局:<包根>/kevrunner/kev_serve.py,<包根>/models/<模型>,<包根>/runtime/rknn3_110
|
| 28 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 29 |
+
ROOT = os.path.dirname(HERE)
|
| 30 |
+
# 两处可用环境变量覆盖,方便放进别的目录布局(不设时行为与原先完全一致)
|
| 31 |
+
MODEL = os.environ.get("KEV_MODEL_DIR") or os.path.join(ROOT, "models", "Kev-0.8B")
|
| 32 |
+
SEP = os.sep
|
| 33 |
+
WORK = HERE
|
| 34 |
+
DAEMON = os.environ.get("KEV_DAEMON") or os.path.join(
|
| 35 |
+
WORK, r"rknn3_session_test_demo\daemon.exe")
|
| 36 |
+
CUR = os.path.join(WORK, r"out\cur.bin")
|
| 37 |
+
# plan task has no console; pipe stdout EOFs immediately there, so use a file channel
|
| 38 |
+
CHAN = os.path.join(WORK, "chan.log")
|
| 39 |
+
|
| 40 |
+
# 官方 kev/model.py::SPECIAL 的 id
|
| 41 |
+
T_STATE, T_Q, T_OPT, T_OPT_END, T_DECIDE = 248060, 248061, 248049, 248050, 248062
|
| 42 |
+
VOCAB, HID = 248320, 1024
|
| 43 |
+
MAX_OPTS = 255
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
# ---------- 渲染(对齐官方 kev/api.py) ----------
|
| 47 |
+
def render(v, indent=0):
|
| 48 |
+
pad = " " * indent
|
| 49 |
+
if v is None:
|
| 50 |
+
return ""
|
| 51 |
+
if isinstance(v, (str, int, float, bool)):
|
| 52 |
+
return str(v)
|
| 53 |
+
if isinstance(v, list):
|
| 54 |
+
return "\n".join(f"{pad}- {render(x, indent + 1).lstrip()}" for x in v)
|
| 55 |
+
return "\n".join(
|
| 56 |
+
f"{pad}{k}:\n{render(x, indent + 1)}" if isinstance(x, (dict, list))
|
| 57 |
+
else f"{pad}{k}: {render(x)}" for k, x in v.items())
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def option_text(name, desc):
|
| 61 |
+
return name if desc is None or desc == "" else f"{name}: {render(desc)}"
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def question_options(q):
|
| 65 |
+
"""返回 (选项文本列表, 选项名列表, 题型)。"""
|
| 66 |
+
t = q["type"]
|
| 67 |
+
if t == "noul":
|
| 68 |
+
return ["No", "Yes"], ["false", "true"], "noul"
|
| 69 |
+
if t == "score":
|
| 70 |
+
crit = q["criteria"]
|
| 71 |
+
return [render(c) for c in crit], [str(i) for i in range(len(crit))], "score"
|
| 72 |
+
crit = q["criteria"]
|
| 73 |
+
names = list(crit)
|
| 74 |
+
return [option_text(k, crit[k]) for k in names], names, "choice"
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def round_prob(x):
|
| 78 |
+
return round(float(x), 4)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def choice_confidence(p):
|
| 82 |
+
"""(p_max - 1/K) / (1 - 1/K):均匀分布为 0,确定时为 1。"""
|
| 83 |
+
K = len(p)
|
| 84 |
+
return 1.0 if K == 1 else (float(max(p)) - 1 / K) / (1 - 1 / K)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def score_confidence(p):
|
| 88 |
+
"""1 - E|level - mode| / D,D = 均匀分布的平均绝对偏差。"""
|
| 89 |
+
L = len(p)
|
| 90 |
+
if L == 1:
|
| 91 |
+
return 1.0
|
| 92 |
+
p = [float(x) for x in p]
|
| 93 |
+
mode = max(range(L), key=p.__getitem__)
|
| 94 |
+
D = sum(abs(i - (L - 1) / 2) for i in range(L)) / L
|
| 95 |
+
return max(0.0, 1.0 - sum(pi * abs(i - mode) for i, pi in enumerate(p)) / D)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
# ---------- daemon 通信 ----------
|
| 99 |
+
class Daemon:
|
| 100 |
+
def __init__(self):
|
| 101 |
+
self.p = subprocess.Popen(
|
| 102 |
+
# 用 ntpath 拼:本文件在 Windows 上跑,os.path.join 会按 Windows 规则走,
|
| 103 |
+
# 但显式用 "\\" 更稳(曾被混合分隔符坑过)
|
| 104 |
+
[DAEMON]
|
| 105 |
+
+ [MODEL + SEP + f"Kev-0.8B{ext}"
|
| 106 |
+
for ext in (".rknn", ".weight", ".tokenizer.gguf", ".embed.bin")]
|
| 107 |
+
+ ["1024", "1", "0xff", "0", "0", "key.env"],
|
| 108 |
+
cwd=os.path.dirname(DAEMON),
|
| 109 |
+
stdin=subprocess.PIPE,
|
| 110 |
+
# 计划任务没有控制台,daemon 接管道时 stdout 会立刻 EOF
|
| 111 |
+
# (rc=None 但进程还活着)。落文件,有无控制台都成立。
|
| 112 |
+
stdout=open(CHAN, "w+b", buffering=0),
|
| 113 |
+
stderr=subprocess.STDOUT)
|
| 114 |
+
self.lock = threading.Lock()
|
| 115 |
+
self._buf = b""
|
| 116 |
+
self.offset = 0
|
| 117 |
+
self._buf = b""
|
| 118 |
+
# 等 daemon 在通道文件里写出 READY_DAEMON(模型加载约 30 s,给 180 s)
|
| 119 |
+
if not self._wait_for("READY_DAEMON", 180):
|
| 120 |
+
raise RuntimeError(
|
| 121 |
+
f"daemon 180s 内没有 READY_DAEMON(rc={self.p.poll()}),"
|
| 122 |
+
f"最后输出:{self._tail() or '(空)'}")
|
| 123 |
+
|
| 124 |
+
def _tail(self, n=6):
|
| 125 |
+
try:
|
| 126 |
+
with open(CHAN, "r", encoding="utf-8", errors="replace") as f:
|
| 127 |
+
return " | ".join(l.rstrip()[:110] for l in f.readlines()[-n:])
|
| 128 |
+
except OSError:
|
| 129 |
+
return ""
|
| 130 |
+
|
| 131 |
+
def _wait_for(self, prefix, timeout):
|
| 132 |
+
"""轮询通道文件,从 self.offset 往后找以 prefix 开头的行。
|
| 133 |
+
|
| 134 |
+
用文件而不是管道:计划任务没有控制台,管道方案在那里会立刻 EOF。
|
| 135 |
+
"""
|
| 136 |
+
t0 = time.time()
|
| 137 |
+
while time.time() - t0 < timeout:
|
| 138 |
+
chunk = b""
|
| 139 |
+
try:
|
| 140 |
+
with open(CHAN, "rb") as f:
|
| 141 |
+
f.seek(self.offset)
|
| 142 |
+
chunk = f.read()
|
| 143 |
+
except OSError:
|
| 144 |
+
pass
|
| 145 |
+
if chunk:
|
| 146 |
+
self.offset += len(chunk)
|
| 147 |
+
buf = self._buf + chunk
|
| 148 |
+
parts = buf.split(b"\n")
|
| 149 |
+
self._buf = parts.pop() # 不完整的尾巴留到下次
|
| 150 |
+
for raw in parts:
|
| 151 |
+
ln = raw.decode("utf-8", "replace").rstrip()
|
| 152 |
+
if ln.startswith(prefix):
|
| 153 |
+
return ln
|
| 154 |
+
if self.p.poll() is not None:
|
| 155 |
+
return None
|
| 156 |
+
time.sleep(0.05)
|
| 157 |
+
return None
|
| 158 |
+
|
| 159 |
+
def _cmd(self, line, stop_prefix):
|
| 160 |
+
self.p.stdin.write((line + "\n").encode("utf-8"))
|
| 161 |
+
self.p.stdin.flush()
|
| 162 |
+
out = self._wait_for(stop_prefix, 180)
|
| 163 |
+
if out is None:
|
| 164 |
+
raise RuntimeError(
|
| 165 |
+
f"等 {stop_prefix} 超时/退出(rc={self.p.poll()}):{self._tail()}")
|
| 166 |
+
return out
|
| 167 |
+
|
| 168 |
+
def tok(self, text):
|
| 169 |
+
b64 = base64.b64encode(text.encode("utf-8")).decode("ascii")
|
| 170 |
+
with self.lock:
|
| 171 |
+
out = self._cmd("TOK " + b64, "IDS ")
|
| 172 |
+
parts = out.split()
|
| 173 |
+
n = int(parts[1])
|
| 174 |
+
return [int(x) for x in parts[2:2 + n]]
|
| 175 |
+
|
| 176 |
+
def prefill(self, ids, append=False):
|
| 177 |
+
"""跑一次 prefill,返回 248320 维 logits。
|
| 178 |
+
|
| 179 |
+
append=False → R(先清 kvcache,从头算)
|
| 180 |
+
append=True → A(接在上一次之后,只算新增这段)
|
| 181 |
+
实测两者余弦 ≥ 0.998,top-1 一致,所以同一题内可以安全用增量。
|
| 182 |
+
"""
|
| 183 |
+
with self.lock:
|
| 184 |
+
self._cmd(("A " if append else "R ") + " ".join(str(x) for x in ids), "DONE")
|
| 185 |
+
return np.fromfile(CUR, dtype=np.float16).astype(np.float32)
|
| 186 |
+
|
| 187 |
+
def close(self):
|
| 188 |
+
try:
|
| 189 |
+
self.p.stdin.write(b"quit\n")
|
| 190 |
+
self.p.stdin.flush()
|
| 191 |
+
except Exception:
|
| 192 |
+
pass
|
| 193 |
+
try:
|
| 194 |
+
self.p.wait(timeout=10)
|
| 195 |
+
except Exception:
|
| 196 |
+
self.p.kill()
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
# ---------- 打分层 ----------
|
| 200 |
+
class Scorer:
|
| 201 |
+
def __init__(self, assets):
|
| 202 |
+
self.Ginv = np.load(os.path.join(assets, "Ginv.npy")) # 1024x1024
|
| 203 |
+
self.q = np.load(os.path.join(assets, "head_q.npy")) # 256x1024
|
| 204 |
+
self.k = np.load(os.path.join(assets, "head_k.npy")) # 256x1024
|
| 205 |
+
self.T = float(np.load(os.path.join(assets, "temperature.npy"))[0])
|
| 206 |
+
# 一次性读进内存(500 MB fp16)。用 memmap 的话每次 hidden() 都要从磁盘
|
| 207 |
+
# 重新读一遍 500 MB —— 实测把单请求从 7.6 s 拖到很慢。
|
| 208 |
+
print("loading embed weights to RAM …", flush=True)
|
| 209 |
+
W16 = np.fromfile(os.path.join(MODEL, "Kev-0.8B.embed.bin"),
|
| 210 |
+
dtype=np.float16).reshape(VOCAB, HID)
|
| 211 |
+
# 预转 float32 常驻(1 GB)。否则每次 hidden() 都要转 2.68 亿个元素,
|
| 212 |
+
# 实测单请求 7.6 s 里大部分耗在这上面。
|
| 213 |
+
print("casting weights to fp32 …", flush=True)
|
| 214 |
+
self.W = np.ascontiguousarray(W16, dtype=np.float32)
|
| 215 |
+
del W16
|
| 216 |
+
|
| 217 |
+
def hidden(self, logits):
|
| 218 |
+
"""h = (WᵀW)⁻¹ Wᵀ logits"""
|
| 219 |
+
# Wᵀ @ logits:memmap 分块算,避免一次性 500 MB 上浮
|
| 220 |
+
rhs = np.zeros(HID, dtype=np.float32)
|
| 221 |
+
step = 16384
|
| 222 |
+
for s in range(0, VOCAB, step):
|
| 223 |
+
e = min(s + step, VOCAB)
|
| 224 |
+
rhs += self.W[s:e].T @ logits[s:e]
|
| 225 |
+
return self.Ginv @ rhs
|
| 226 |
+
|
| 227 |
+
def scores(self, h_decide, h_opts):
|
| 228 |
+
z = (h_opts @ self.k.T) @ (self.q @ h_decide) / np.sqrt(self.q.shape[0])
|
| 229 |
+
z = z / self.T
|
| 230 |
+
z = z - z.max()
|
| 231 |
+
p = np.exp(z)
|
| 232 |
+
return p / p.sum()
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
# ---------- 决策 ----------
|
| 236 |
+
class BadRequest(Exception):
|
| 237 |
+
"""客户端参数错误 → 400。和「服务出故障」必须分开:
|
| 238 |
+
参数错回 500 会让人以为服务挂了,日志里也全是假异常。"""
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
def validate_request(req):
|
| 242 |
+
"""把能判的错都在**进模型之前**判掉。
|
| 243 |
+
|
| 244 |
+
这些情况以前会一路走到 question_options 里抛 KeyError,
|
| 245 |
+
被外层 catch 成 500 并回传 traceback —— 既误导调用方,又泄漏实现细节。
|
| 246 |
+
还有两处更糟的:缺 state 会白跑一次 prefill(几百毫秒),
|
| 247 |
+
未知 type 会被静默当成 choice 处理(拼错 "choices" 也能拿到答案)。
|
| 248 |
+
"""
|
| 249 |
+
if not isinstance(req, dict):
|
| 250 |
+
raise BadRequest("body must be a JSON object")
|
| 251 |
+
|
| 252 |
+
state = req.get("state")
|
| 253 |
+
# state 允许三种形态,这是 decision-v2 的真实分布,不是想当然:
|
| 254 |
+
# str 664 条 / dict 440 条 / list 72 条
|
| 255 |
+
# 一开始只允许 str,结果把 512 条合法请求挡成 400 —— 校验过���比不校验更糟。
|
| 256 |
+
if state is None:
|
| 257 |
+
raise BadRequest("state is required")
|
| 258 |
+
if isinstance(state, str):
|
| 259 |
+
if not state.strip():
|
| 260 |
+
raise BadRequest("state must not be empty")
|
| 261 |
+
elif isinstance(state, (dict, list)):
|
| 262 |
+
if not state:
|
| 263 |
+
raise BadRequest("state must not be empty")
|
| 264 |
+
else:
|
| 265 |
+
raise BadRequest(
|
| 266 |
+
f"state must be a string, object or array, got {type(state).__name__}")
|
| 267 |
+
|
| 268 |
+
qs = req.get("questions")
|
| 269 |
+
if not isinstance(qs, dict):
|
| 270 |
+
raise BadRequest("questions is required: an object {id: question}")
|
| 271 |
+
if not qs:
|
| 272 |
+
# 空 questions 以前返回 {"answers": {}} —— 拼错字段名(比如写成 question)
|
| 273 |
+
# 会静默"成功",调用方以为模型没意见。宁可报错。
|
| 274 |
+
raise BadRequest("questions must not be empty")
|
| 275 |
+
|
| 276 |
+
for qid, q in qs.items():
|
| 277 |
+
if not isinstance(q, dict):
|
| 278 |
+
raise BadRequest(f"questions[{qid}] must be an object")
|
| 279 |
+
kind = q.get("type")
|
| 280 |
+
if kind not in ("noul", "choice", "score"):
|
| 281 |
+
raise BadRequest(
|
| 282 |
+
f"questions[{qid}].type must be one of noul/choice/score, got {kind!r}")
|
| 283 |
+
if kind == "choice":
|
| 284 |
+
crit = q.get("criteria")
|
| 285 |
+
if not isinstance(crit, dict) or not crit:
|
| 286 |
+
raise BadRequest(f"questions[{qid}].criteria must be a non-empty object")
|
| 287 |
+
elif kind == "score":
|
| 288 |
+
crit = q.get("criteria")
|
| 289 |
+
if not isinstance(crit, list) or not crit:
|
| 290 |
+
raise BadRequest(f"questions[{qid}].criteria must be a non-empty array")
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
# 一次 decide 会连续调用多次 prefill(K+1 次),中间必须独占 daemon。
|
| 294 |
+
# 只在单次 _cmd 里加锁是不够的 —— HTTP 层是多线程的,两个请求会交替
|
| 295 |
+
# 使用同一份 kvcache,后一个的 append 会接在前一个的序列后面,结果全乱。
|
| 296 |
+
DECIDE_LOCK = threading.Lock()
|
| 297 |
+
|
| 298 |
+
MAX_BODY = 1 << 20 # 1 MB,够放很长的 state,也挡住恶意大包
|
| 299 |
+
|
| 300 |
+
|
| 301 |
+
def decide(dmn, sc, state, questions):
|
| 302 |
+
st_ids = dmn.tok(render(state))
|
| 303 |
+
S = [T_STATE] + st_ids
|
| 304 |
+
out = {}
|
| 305 |
+
for qid, q in questions.items():
|
| 306 |
+
texts, names, kind = question_options(q)
|
| 307 |
+
K = len(texts)
|
| 308 |
+
if not 1 <= K <= MAX_OPTS:
|
| 309 |
+
out[qid] = {"error": f"options must be 1..{MAX_OPTS}"}
|
| 310 |
+
continue
|
| 311 |
+
ins = [T_Q] + dmn.tok(render(q.get("instructions") or ""))
|
| 312 |
+
spans = [[T_OPT] + dmn.tok(t) + [T_OPT_END] for t in texts]
|
| 313 |
+
ids = S + ins + [x for sp in spans for x in sp] + [T_DECIDE]
|
| 314 |
+
|
| 315 |
+
dec = len(ids) - 1
|
| 316 |
+
ends, cur = [], len(S) + len(ins)
|
| 317 |
+
for sp in spans:
|
| 318 |
+
ends.append(cur + len(sp) - 1)
|
| 319 |
+
cur += len(sp)
|
| 320 |
+
|
| 321 |
+
# 增量 prefill:只算一次全序列,其余位置靠逐段追加。
|
| 322 |
+
# 全量要算 sum(len)=约 K+1 倍的 token;增量只算整条序列一遍。
|
| 323 |
+
prev, h_opts = -1, []
|
| 324 |
+
for i, e in enumerate(ends):
|
| 325 |
+
h_opts.append(sc.hidden(dmn.prefill(ids[prev + 1:e + 1], append=(i > 0))))
|
| 326 |
+
prev = e
|
| 327 |
+
h_dec = sc.hidden(dmn.prefill(ids[prev + 1:dec + 1], append=True))
|
| 328 |
+
h_opts = np.stack(h_opts)
|
| 329 |
+
prob = sc.scores(h_dec, h_opts)
|
| 330 |
+
|
| 331 |
+
# 输出结构对齐官方 kev/api.py::to_answers
|
| 332 |
+
if kind == "noul":
|
| 333 |
+
out[qid] = {"type": "noul", "noul": round_prob(prob[1]),
|
| 334 |
+
"answer": bool(prob[1] >= 0.5)}
|
| 335 |
+
elif kind == "score":
|
| 336 |
+
lv = sum(i * pi for i, pi in enumerate(prob))
|
| 337 |
+
out[qid] = {"type": "score", "score": round_prob(lv),
|
| 338 |
+
"legend": texts,
|
| 339 |
+
"probabilities": {str(i): round_prob(v) for i, v in enumerate(prob)},
|
| 340 |
+
"confidence": round_prob(score_confidence(prob)),
|
| 341 |
+
"argmax": int(prob.argmax())}
|
| 342 |
+
else:
|
| 343 |
+
i = int(prob.argmax())
|
| 344 |
+
out[qid] = {"type": "choice", "answer": names[i],
|
| 345 |
+
"confidence": round_prob(choice_confidence(prob)),
|
| 346 |
+
"probabilities": {n: round_prob(p) for n, p in zip(names, prob)}}
|
| 347 |
+
return {"answers": out}
|
| 348 |
+
|
| 349 |
+
|
| 350 |
+
def main():
|
| 351 |
+
assets = os.path.join(HERE, "serve_assets")
|
| 352 |
+
print("loading assets …", flush=True)
|
| 353 |
+
sc = Scorer(assets)
|
| 354 |
+
print("starting daemon …", flush=True)
|
| 355 |
+
# 计划任务不继承 PATH。daemon.exe 依赖 runtime\lib 里的 librknn3_api.dll 等,
|
| 356 |
+
# 找不到会以 0xC0000135 (rc=3221225781) 直接退出、一个字都不输出。
|
| 357 |
+
rt = os.path.join(ROOT, "runtime", "rknn3_110")
|
| 358 |
+
for d in (os.path.join(rt, "lib"), os.path.join(rt, "bin")):
|
| 359 |
+
if os.path.isdir(d) and d not in os.environ.get("PATH", ""):
|
| 360 |
+
os.environ["PATH"] = d + os.pathsep + os.environ.get("PATH", "")
|
| 361 |
+
print(f"[env] PATH head = {os.environ['PATH'].split(os.pathsep)[:2]}", flush=True)
|
| 362 |
+
dmn = Daemon()
|
| 363 |
+
print("daemon ready", flush=True)
|
| 364 |
+
|
| 365 |
+
class H(BaseHTTPRequestHandler):
|
| 366 |
+
def _json(self, code, obj):
|
| 367 |
+
raw = json.dumps(obj, ensure_ascii=False).encode()
|
| 368 |
+
self.send_response(code)
|
| 369 |
+
self.send_header("Content-Type", "application/json; charset=utf-8")
|
| 370 |
+
self.send_header("Content-Length", str(len(raw)))
|
| 371 |
+
self.end_headers()
|
| 372 |
+
self.wfile.write(raw)
|
| 373 |
+
|
| 374 |
+
def do_POST(self):
|
| 375 |
+
if self.path != "/v1/systemone":
|
| 376 |
+
self.send_error(404)
|
| 377 |
+
return
|
| 378 |
+
|
| 379 |
+
n = int(self.headers.get("Content-Length", 0) or 0)
|
| 380 |
+
if n > MAX_BODY:
|
| 381 |
+
self._json(413, {"error": f"body too large: {n} bytes (limit {MAX_BODY})"})
|
| 382 |
+
return
|
| 383 |
+
raw_in = self.rfile.read(n) if n > 0 else b""
|
| 384 |
+
if not raw_in:
|
| 385 |
+
self._json(400, {"error": "empty body"})
|
| 386 |
+
return
|
| 387 |
+
|
| 388 |
+
try:
|
| 389 |
+
req = json.loads(raw_in)
|
| 390 |
+
except json.JSONDecodeError as e:
|
| 391 |
+
# 4xx 而不是 5xx:这是调用方的问题,回 500 会让人以为服务挂了
|
| 392 |
+
self._json(400, {"error": f"invalid JSON at line {e.lineno} col {e.colno}: {e.msg}"})
|
| 393 |
+
return
|
| 394 |
+
|
| 395 |
+
try:
|
| 396 |
+
validate_request(req)
|
| 397 |
+
except BadRequest as e:
|
| 398 |
+
self._json(400, {"error": str(e)})
|
| 399 |
+
return
|
| 400 |
+
|
| 401 |
+
try:
|
| 402 |
+
# 整个决策独占 daemon:中间的 K+1 次 prefill 不能被别的请求插进来
|
| 403 |
+
with DECIDE_LOCK:
|
| 404 |
+
body = decide(dmn, sc, req["state"], req["questions"])
|
| 405 |
+
self._json(200, body)
|
| 406 |
+
except Exception as e:
|
| 407 |
+
# traceback 只进日志。响应里给调用方也没用,还泄漏实现细节。
|
| 408 |
+
with open(os.path.join(HERE, "serve_err.log"), "a", encoding="utf-8") as f:
|
| 409 |
+
f.write(traceback.format_exc() + "\n")
|
| 410 |
+
self._json(500, {"error": f"internal error: {type(e).__name__}: {e}"})
|
| 411 |
+
|
| 412 |
+
def do_GET(self):
|
| 413 |
+
# 探活:计划任务起来后用它确认"服务真的能答"
|
| 414 |
+
if self.path == "/healthz":
|
| 415 |
+
raw = json.dumps({"ok": True, "model": "Kev-0.8B",
|
| 416 |
+
"pid": os.getpid()}).encode()
|
| 417 |
+
self.send_response(200)
|
| 418 |
+
self.send_header("Content-Type", "application/json")
|
| 419 |
+
self.send_header("Content-Length", str(len(raw)))
|
| 420 |
+
self.end_headers()
|
| 421 |
+
self.wfile.write(raw)
|
| 422 |
+
elif self.path == "/v1/systemone":
|
| 423 |
+
# 存在但不接受 GET:405 比 404 准确,调用方一眼看出是方法错了
|
| 424 |
+
self.send_response(405)
|
| 425 |
+
self.send_header("Allow", "POST")
|
| 426 |
+
self.send_header("Content-Type", "application/json; charset=utf-8")
|
| 427 |
+
raw = json.dumps({"error": "use POST"}).encode()
|
| 428 |
+
self.send_header("Content-Length", str(len(raw)))
|
| 429 |
+
self.end_headers()
|
| 430 |
+
self.wfile.write(raw)
|
| 431 |
+
else:
|
| 432 |
+
self.send_error(404)
|
| 433 |
+
|
| 434 |
+
def log_message(self, *a):
|
| 435 |
+
pass
|
| 436 |
+
|
| 437 |
+
port = int(os.environ.get("KEV_PORT", "17890"))
|
| 438 |
+
host = os.environ.get("KEV_HOST", "127.0.0.1")
|
| 439 |
+
try:
|
| 440 |
+
srv = ThreadingHTTPServer((host, port), H)
|
| 441 |
+
except OSError as e:
|
| 442 |
+
# 端口被占是最常见的启动失败原因(旧进程没退干净),必须说清楚
|
| 443 |
+
print(f"BIND_FAILED {host}:{port} -> {e}", flush=True)
|
| 444 |
+
raise SystemExit(2)
|
| 445 |
+
print(f"listening on {host}:{port}", flush=True)
|
| 446 |
+
srv.serve_forever()
|
| 447 |
+
|
| 448 |
+
|
| 449 |
+
if __name__ == "__main__":
|
| 450 |
+
main()
|
host/serve_assets/Ginv.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9b556691b6b25ed0067e3bbf07a43a6505ff82bfc79e55acfbf9c052dc3f50cc
|
| 3 |
+
size 4194432
|
host/serve_assets/head_k.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7a305f5f8b4bdcc44be36f964b416892c3e835c6c2e6823d8e5d963927d703bd
|
| 3 |
+
size 1048704
|
host/serve_assets/head_q.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5c44393c6e8bc5ccd555f7bf4e73386b84930cdc6392e92673c528757238caa5
|
| 3 |
+
size 1048704
|
host/serve_assets/temperature.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:372426d801de013d10f2cb70cb10f6bb19732fece12e9567133cfc9808c9b64d
|
| 3 |
+
size 132
|