Image-Text-to-Text
Transformers
Safetensors
English
Chinese
groundinganything_vlm
text-generation
visual-grounding
object-detection
referring-expression-comprehension
pointing
ocr
document-layout
custom-code
conversational
custom_code
Instructions to use GroundingPI/GroundAnything-VLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use GroundingPI/GroundAnything-VLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="GroundingPI/GroundAnything-VLM", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("GroundingPI/GroundAnything-VLM", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use GroundingPI/GroundAnything-VLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "GroundingPI/GroundAnything-VLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "GroundingPI/GroundAnything-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/GroundingPI/GroundAnything-VLM
- SGLang
How to use GroundingPI/GroundAnything-VLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "GroundingPI/GroundAnything-VLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "GroundingPI/GroundAnything-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "GroundingPI/GroundAnything-VLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "GroundingPI/GroundAnything-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use GroundingPI/GroundAnything-VLM with Docker Model Runner:
docker model run hf.co/GroundingPI/GroundAnything-VLM
Simplify model card layout and evaluation; clarify license scope
#2
by Skywalker0410 - opened
- LICENSE +16 -47
- LICENSE-Apache-2.0 +202 -0
- LICENSE-Kimi-K3 +52 -0
- README.md +27 -96
- checksums.sha256 +4 -2
LICENSE
CHANGED
|
@@ -1,52 +1,21 @@
|
|
| 1 |
-
|
|
|
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
configuration files, inference and training code, and associated documentation
|
| 8 |
-
(collectively, the "Software") — to deal in the Software without restriction.
|
| 9 |
-
This includes, without limitation, the rights to use, copy, modify, merge,
|
| 10 |
-
publish, distribute, sublicense, and/or sell copies of the Software; to run,
|
| 11 |
-
deploy, fine-tune, or otherwise modify the Software and create derivative works
|
| 12 |
-
from it; and to permit persons to whom the Software is furnished to do so, in
|
| 13 |
-
each case subject to the following conditions:
|
| 14 |
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
|
|
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
party to exercise meaningful control over the inputs, parameters, or training
|
| 22 |
-
data. This does not include (a) end-user products with model capabilities solely
|
| 23 |
-
embedded within specific features or harnesses, or (b) mere relaying of requests
|
| 24 |
-
to models hosted by others.
|
| 25 |
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
US dollars (or the equivalent in other currencies) in total over any consecutive
|
| 29 |
-
12 months, the Licensee must enter into a separate agreement with Moonshot AI
|
| 30 |
-
before using the Software or its derivative works for any commercial purpose.
|
| 31 |
-
|
| 32 |
-
3. If the Software (or any derivative works thereof) is used for any of the
|
| 33 |
-
Licensee's commercial products or services that have more than 100 million
|
| 34 |
-
monthly active users, or more than 20 million US dollars (or equivalent in other
|
| 35 |
-
currencies) in monthly revenue, "Kimi K3" must be prominently displayed on the
|
| 36 |
-
user interface of such product or service.
|
| 37 |
-
|
| 38 |
-
4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal
|
| 39 |
-
use of the Software, defined as any use that does not make the Software, its
|
| 40 |
-
outputs, or its underlying capabilities available to third parties; or (b) any
|
| 41 |
-
use of the Software accessed through Moonshot AI's official products or
|
| 42 |
-
certified inference partners.
|
| 43 |
-
|
| 44 |
-
5. THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS”
|
| 45 |
-
BASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
|
| 46 |
-
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
|
| 47 |
-
AND NONINFRINGEMENT. IN NO EVENT SHALL MOONSHOT AI OR ITS AFFILIATES OR
|
| 48 |
-
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
| 49 |
-
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
| 50 |
-
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
| 51 |
-
|
| 52 |
-
For any questions regarding this license, please contact <license@moonshot.ai>.
|
|
|
|
| 1 |
+
License scope
|
| 2 |
+
=============
|
| 3 |
|
| 4 |
+
Original contributions by the GroundingPI and GroundAnything authors are
|
| 5 |
+
licensed under the Apache License, Version 2.0, provided in LICENSE-Apache-2.0.
|
| 6 |
+
The project imposes no additional use or commercial restrictions on those
|
| 7 |
+
contributions beyond that license.
|
| 8 |
|
| 9 |
+
This grant covers only rights held by the contributing authors. It does not
|
| 10 |
+
relicense third-party material or waive any upstream requirements.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
+
Kimi-derived material remains subject to the Kimi K3 License, reproduced
|
| 13 |
+
unchanged in LICENSE-Kimi-K3. Its conditions, including conditions that apply
|
| 14 |
+
to derivative works, continue to apply to the combined model package where
|
| 15 |
+
applicable. Apache 2.0 is not an alternative license for that material.
|
| 16 |
|
| 17 |
+
Other third-party components retain their own licenses and copyright notices.
|
| 18 |
+
Preserve the applicable license texts and notices when redistributing.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
+
Upstream Kimi K3 license source:
|
| 21 |
+
https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/LICENSE
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
LICENSE-Apache-2.0
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
| 178 |
+
|
| 179 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 180 |
+
|
| 181 |
+
To apply the Apache License to your work, attach the following
|
| 182 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 183 |
+
replaced with your own identifying information. (Don't include
|
| 184 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 185 |
+
comment syntax for the file format. We also recommend that a
|
| 186 |
+
file or class name and description of purpose be included on the
|
| 187 |
+
same "printed page" as the copyright notice for easier
|
| 188 |
+
identification within third-party archives.
|
| 189 |
+
|
| 190 |
+
Copyright [yyyy] [name of copyright owner]
|
| 191 |
+
|
| 192 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 193 |
+
you may not use this file except in compliance with the License.
|
| 194 |
+
You may obtain a copy of the License at
|
| 195 |
+
|
| 196 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 197 |
+
|
| 198 |
+
Unless required by applicable law or agreed to in writing, software
|
| 199 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 200 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 201 |
+
See the License for the specific language governing permissions and
|
| 202 |
+
limitations under the License.
|
LICENSE-Kimi-K3
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Kimi K3 License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Moonshot AI
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person (the "Licensee")
|
| 6 |
+
obtaining a copy of this software — including the model weights, parameters,
|
| 7 |
+
configuration files, inference and training code, and associated documentation
|
| 8 |
+
(collectively, the "Software") — to deal in the Software without restriction.
|
| 9 |
+
This includes, without limitation, the rights to use, copy, modify, merge,
|
| 10 |
+
publish, distribute, sublicense, and/or sell copies of the Software; to run,
|
| 11 |
+
deploy, fine-tune, or otherwise modify the Software and create derivative works
|
| 12 |
+
from it; and to permit persons to whom the Software is furnished to do so, in
|
| 13 |
+
each case subject to the following conditions:
|
| 14 |
+
|
| 15 |
+
1. The above copyright notice and this permission notice shall be included in
|
| 16 |
+
all copies or substantial portions of the Software. Licensee's use of the
|
| 17 |
+
Software must comply with applicable laws and regulations.
|
| 18 |
+
|
| 19 |
+
2. "Model as a Service" means giving a third party access to language model
|
| 20 |
+
inference or fine-tuning (e.g., via API) in a manner that allows such third
|
| 21 |
+
party to exercise meaningful control over the inputs, parameters, or training
|
| 22 |
+
data. This does not include (a) end-user products with model capabilities solely
|
| 23 |
+
embedded within specific features or harnesses, or (b) mere relaying of requests
|
| 24 |
+
to models hosted by others.
|
| 25 |
+
|
| 26 |
+
If the Licensee or any of its affiliates operates a Model as a Service business,
|
| 27 |
+
and the aggregate revenue of the Licensee and its affiliates exceeds 20 million
|
| 28 |
+
US dollars (or the equivalent in other currencies) in total over any consecutive
|
| 29 |
+
12 months, the Licensee must enter into a separate agreement with Moonshot AI
|
| 30 |
+
before using the Software or its derivative works for any commercial purpose.
|
| 31 |
+
|
| 32 |
+
3. If the Software (or any derivative works thereof) is used for any of the
|
| 33 |
+
Licensee's commercial products or services that have more than 100 million
|
| 34 |
+
monthly active users, or more than 20 million US dollars (or equivalent in other
|
| 35 |
+
currencies) in monthly revenue, "Kimi K3" must be prominently displayed on the
|
| 36 |
+
user interface of such product or service.
|
| 37 |
+
|
| 38 |
+
4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal
|
| 39 |
+
use of the Software, defined as any use that does not make the Software, its
|
| 40 |
+
outputs, or its underlying capabilities available to third parties; or (b) any
|
| 41 |
+
use of the Software accessed through Moonshot AI's official products or
|
| 42 |
+
certified inference partners.
|
| 43 |
+
|
| 44 |
+
5. THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS”
|
| 45 |
+
BASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
|
| 46 |
+
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
|
| 47 |
+
AND NONINFRINGEMENT. IN NO EVENT SHALL MOONSHOT AI OR ITS AFFILIATES OR
|
| 48 |
+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
| 49 |
+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
| 50 |
+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
| 51 |
+
|
| 52 |
+
For any questions regarding this license, please contact <license@moonshot.ai>.
|
README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
-
license_name: kimi-k3
|
| 4 |
-
license_link:
|
| 5 |
language:
|
| 6 |
- en
|
| 7 |
- zh
|
|
@@ -18,20 +18,18 @@ tags:
|
|
| 18 |
inference: false
|
| 19 |
---
|
| 20 |
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
# GroundAnything-VLM: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed
|
| 24 |
|
| 25 |
**Model family:** [GroundAnything — DLM / parallel decoding](https://huggingface.co/GroundingPI/GroundAnything) · [GroundAnything-VLM — autoregressive](https://huggingface.co/GroundingPI/GroundAnything-VLM).
|
| 26 |
|
| 27 |
**This repository contains the autoregressive GroundAnything-VLM checkpoint.** The family overview and figures below are shared with the DLM page; use the VLM serving recipe for these weights.
|
| 28 |
|
| 29 |
-
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig1-teaser.png" width="100%" alt="GroundAnything
|
| 30 |
|
| 31 |
## 🔗 Quick Links
|
| 32 |
|
| 33 |
- 🚀 **Online Demo:** Coming soon — XXX.
|
| 34 |
-
- 💻 **GitHub Code:**
|
| 35 |
- 📄 **Paper:** [arXiv:2609.39600](https://arxiv.org/abs/2609.39600).
|
| 36 |
- 🧪 **Evaluation data:** Coming soon — XXX.
|
| 37 |
|
|
@@ -49,13 +47,13 @@ An optional **self-speculative mode** achieves a **4.51× speedup** over the AR
|
|
| 49 |
|
| 50 |
<video controls playsinline preload="none" width="100%" poster="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/demo-poster.jpg" src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/demo.mp4"></video>
|
| 51 |
|
| 52 |
-
**Parallel
|
| 53 |
|
| 54 |
<video controls playsinline preload="none" width="100%" src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/decoding.mp4"></video>
|
| 55 |
|
| 56 |
### License/Terms of Use:
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
### Deployment Geography:
|
| 61 |
|
|
@@ -105,13 +103,7 @@ Global.
|
|
| 105 |
- **Spatial vocabulary:** 1,000 coordinate tokens shared with semantic labels and protocol markers.
|
| 106 |
- **DLM conversion:** the shared decoder and vocabulary head support both causal prediction and bidirectional response-block denoising. A mask token is added for diffusion generation.
|
| 107 |
|
| 108 |
-
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig2-architecture.png" width="100%" alt="GroundAnything
|
| 109 |
-
|
| 110 |
-
*Figure 2. Shared model architecture and the conversion to parallel grounding.*
|
| 111 |
-
|
| 112 |
-
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig4-attention-mask.png" width="640" alt="GroundAnything Figure 4: clean-stream causal and noisy-response block attention" /></p>
|
| 113 |
-
|
| 114 |
-
*Figure 4. The attention mask used during diffusion conversion. The clean stream uses causal attention. A noisy response block attends bidirectionally within its block and reads the clean conditioning context and strictly preceding clean response blocks. Conversion uses B=32; the illustration uses two-token blocks.*
|
| 115 |
|
| 116 |
## Input(s):
|
| 117 |
|
|
@@ -158,49 +150,35 @@ The default DLM service uses **BF16, Triton attention, eager execution, one GPU,
|
|
| 158 |
| [GroundAnything](https://huggingface.co/GroundingPI/GroundAnything) | Entropy-guided blockwise diffusion; optional self-speculation | **GAM** |
|
| 159 |
| [GroundAnything-VLM](https://huggingface.co/GroundingPI/GroundAnything-VLM) | Autoregressive generation | **GAM** |
|
| 160 |
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
## Testing and Evaluation Datasets:
|
| 164 |
-
|
| 165 |
-
### Data Modality:
|
| 166 |
-
|
| 167 |
-
Image and text, with task-specific box, point, text-region, or interaction annotations.
|
| 168 |
-
|
| 169 |
-
## Evaluation Dataset:
|
| 170 |
|
| 171 |
-
|
| 172 |
|
| 173 |
-
The
|
| 174 |
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
|
| 180 |
-
|
|
| 181 |
-
|
|
| 182 |
-
| `DLM` | Legacy
|
| 183 |
-
| `RLV2` |
|
| 184 |
-
| `VLM` | Generic VLM baselines | Explicit pixel, 0–1000, or 0–1 coordinate mode |
|
| 185 |
-
| `REXOMNI` | Rex-Omni adapter | Model-specific spatial-token parser |
|
| 186 |
-
| `LOCATEANYTHING` | LocateAnything adapter | Its own output parser and generation-mode setting |
|
| 187 |
-
| `GROUNDINGDINO` | External compatible GroundingDINO bridge | JSON coordinate responses |
|
| 188 |
|
| 189 |
-
**
|
| 190 |
|
| 191 |
-
|
| 192 |
|
| 193 |
## Quantitative Evaluation Benchmarks
|
| 194 |
|
| 195 |
-
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig7-grounding-performance.png" width="100%" alt="GroundAnything
|
| 196 |
-
|
| 197 |
-
*Figure 7. Paper-reported capability overview. Detailed task-level scores, baseline settings, and evaluation protocols are provided in the paper and supplementary material.*
|
| 198 |
|
| 199 |
## Inference:
|
| 200 |
|
| 201 |
### Installation
|
| 202 |
|
| 203 |
-
Run the commands from the **GroundAnything source repository root** after obtaining the code package, using **Linux x86_64 and Python 3.12**.
|
| 204 |
|
| 205 |
```bash
|
| 206 |
python3 -m pip install -r requirements.txt huggingface_hub
|
|
@@ -264,7 +242,7 @@ point_result = client.predict(
|
|
| 264 |
)
|
| 265 |
```
|
| 266 |
|
| 267 |
-
The HTTP client retains no model weights.
|
| 268 |
|
| 269 |
### Supported Tasks & Prompt Templates
|
| 270 |
|
|
@@ -299,64 +277,17 @@ After a block is complete, a causal forward reconstructs its authoritative KV ca
|
|
| 299 |
|
| 300 |
The model uses its own shared weights to draft tokens with bidirectional attention and verify them with causal attention. Verification accepts the **longest consecutive matching prefix**, stops at the first mismatch, applies the causal correction, and discards the rejected suffix cache states. The shipped speculative route uses greedy verification; it is not a general stochastic speculative sampler.
|
| 301 |
|
| 302 |
-
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig6-self-speculative-decoding.png" width="100%" alt="GroundAnything
|
| 303 |
-
|
| 304 |
-
*Figure 6. The paper studies two schedules: linear drafting and verification use two model forwards and 2B query tokens per round; quadratic fusion uses one forward after initialization with B(B + 1) query tokens. These counts describe queries and model calls, not total Transformer FLOPs.*
|
| 305 |
|
| 306 |
The documented `--decoder speculative` service is the linear shared-weight route. Exact greedy verification is relative to the converted model's causal branch; it does not imply identical outputs to the separately trained GroundAnything-VLM checkpoint.
|
| 307 |
|
| 308 |
-
### Benchmark Sampling Policy
|
| 309 |
-
|
| 310 |
-
Entropy-guided evaluation applies the five task profiles from `infer/decode/configs/task_profiles.json`:
|
| 311 |
-
|
| 312 |
-
| Task profile | Recipes | Temperature | Top-p | Output-token budget |
|
| 313 |
-
|:---|---:|---:|---:|---:|
|
| 314 |
-
| Strict single target | 14 | 0 | 1 | 512 |
|
| 315 |
-
| Medium, non-OCR | 12 | 0.1 | 0.95 | 4,096 |
|
| 316 |
-
| Dense, non-OCR | 8 | 0.1 | 0.95 | 8,192 |
|
| 317 |
-
| Medium OCR | 4 | 0.3 | 0.95 | 4,096 |
|
| 318 |
-
| Dense OCR | 4 | 0 | 1 | 4,096 |
|
| 319 |
-
|
| 320 |
-
The evaluator sends outer sampling parameters and the nested `custom_params.gam_dlm_decode` contract, adds the strict-single-target policy when required, and checks the server's actual decoder before sending requests. Unknown task profiles and mismatched decoders fail early. The entropy-guided benchmark policy rejects a global `max_tokens` override because it would replace the task budgets.
|
| 321 |
-
|
| 322 |
-
Self-speculative evaluation uses greedy sampling (`temperature=0`, `top_p=1`, repetition penalty 1). GroundAnything-VLM retains its own autoregressive task recipes. All of these routes use **GAM** prompts and spatial-token parsing.
|
| 323 |
-
|
| 324 |
-
### Run Evaluation
|
| 325 |
-
|
| 326 |
-
```bash
|
| 327 |
-
python3 run.py setup eval
|
| 328 |
-
|
| 329 |
-
# Match the command to the service that is already running.
|
| 330 |
-
python3 run.py eval --decoder denoise
|
| 331 |
-
python3 run.py eval --decoder speculative
|
| 332 |
-
python3 run.py eval --config configs/eval/vlm.yaml
|
| 333 |
-
```
|
| 334 |
-
|
| 335 |
-
Choose one evaluation command for the intended checkpoint/decoder:
|
| 336 |
-
|
| 337 |
-
| Checkpoint / decoder | Recipe | Mode | Endpoint model ID |
|
| 338 |
-
|:---|:---|:---|:---|
|
| 339 |
-
| GroundAnything / entropy-guided | `configs/eval/dlm.yaml` | **GAM** | `groundinganything` |
|
| 340 |
-
| GroundAnything / self-speculative | `configs/eval/dlm_speculative.yaml` | **GAM** | `groundinganything` |
|
| 341 |
-
| GroundAnything-VLM / autoregressive | `configs/eval/vlm.yaml` | **GAM** | `groundinganything-vlm` |
|
| 342 |
-
|
| 343 |
-
Use `service_contract: openai` and the corresponding port (8101 for DLM, 8102 for VLM). For a preflight that checks configured inputs without sending inference requests:
|
| 344 |
-
|
| 345 |
-
```bash
|
| 346 |
-
.venv-eval/bin/python scripts/evaluate.py configs/eval/dlm.yaml --dry-run
|
| 347 |
-
```
|
| 348 |
-
|
| 349 |
-
Configure the running service, `data_root`, dataset registry, task list, and a fresh `run_id` before launching. The default `limit: 8` is a smoke test; set **`limit: null`** for full evaluation. Evaluation connects to an existing service and does not start or change the decoder.
|
| 350 |
-
|
| 351 |
-
Each run saves `run.json` (configuration and provenance), `responses.jsonl` (raw responses, finish reasons, and token usage), task logs, and `summary.json` (metrics and completion state). Record the checkpoint revision and task configuration when comparing results.
|
| 352 |
-
|
| 353 |
## Inference Infrastructure
|
| 354 |
|
| 355 |
### SGLang Execution
|
| 356 |
|
| 357 |
The custom SGLang integration coordinates model loading, request scheduling, attention kernels, and KV-cache ownership for blockwise generation. Denoising uses bidirectional attention inside the active block; completed history is retained as causal KV. Self-speculation additionally verifies proposals and removes rejected suffix states. These cache semantics must be preserved when optimizing execution.
|
| 358 |
|
| 359 |
-
The supplied recipe selects **BF16 + Triton attention + eager execution**. DLM recipes record the engine source, decoder, effective settings,
|
| 360 |
|
| 361 |
### CUDA Graph and Selective FP8
|
| 362 |
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
+
license_name: apache-2.0-with-upstream-kimi-k3
|
| 4 |
+
license_link: LICENSE
|
| 5 |
language:
|
| 6 |
- en
|
| 7 |
- zh
|
|
|
|
| 18 |
inference: false
|
| 19 |
---
|
| 20 |
|
| 21 |
+
# <img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/logo.png" width="40" style="display: inline-block; vertical-align: middle;" alt="GroundAnything logo" /> GroundAnything-VLM: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed
|
|
|
|
|
|
|
| 22 |
|
| 23 |
**Model family:** [GroundAnything — DLM / parallel decoding](https://huggingface.co/GroundingPI/GroundAnything) · [GroundAnything-VLM — autoregressive](https://huggingface.co/GroundingPI/GroundAnything-VLM).
|
| 24 |
|
| 25 |
**This repository contains the autoregressive GroundAnything-VLM checkpoint.** The family overview and figures below are shared with the DLM page; use the VLM serving recipe for these weights.
|
| 26 |
|
| 27 |
+
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig1-teaser.png" width="100%" alt="GroundAnything: broad visual grounding and parallel visual evidence extraction" /></p>
|
| 28 |
|
| 29 |
## 🔗 Quick Links
|
| 30 |
|
| 31 |
- 🚀 **Online Demo:** Coming soon — XXX.
|
| 32 |
+
- 💻 **GitHub Code:** [groundingpi/GroundAnything](https://github.com/groundingpi/GroundAnything).
|
| 33 |
- 📄 **Paper:** [arXiv:2609.39600](https://arxiv.org/abs/2609.39600).
|
| 34 |
- 🧪 **Evaluation data:** Coming soon — XXX.
|
| 35 |
|
|
|
|
| 47 |
|
| 48 |
<video controls playsinline preload="none" width="100%" poster="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/demo-poster.jpg" src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/demo.mp4"></video>
|
| 49 |
|
| 50 |
+
**Parallel Decoding**
|
| 51 |
|
| 52 |
<video controls playsinline preload="none" width="100%" src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/decoding.mp4"></video>
|
| 53 |
|
| 54 |
### License/Terms of Use:
|
| 55 |
|
| 56 |
+
Our original contributions are available under **Apache 2.0**, with no additional restrictions imposed by this project. Third-party material retains its applicable licenses, including the **Kimi K3 License** for Kimi-derived material and applicable derivative works. Its conditions continue to apply when using or redistributing the combined model package. See [LICENSE](LICENSE) for the scope and full license texts.
|
| 57 |
|
| 58 |
### Deployment Geography:
|
| 59 |
|
|
|
|
| 103 |
- **Spatial vocabulary:** 1,000 coordinate tokens shared with semantic labels and protocol markers.
|
| 104 |
- **DLM conversion:** the shared decoder and vocabulary head support both causal prediction and bidirectional response-block denoising. A mask token is added for diffusion generation.
|
| 105 |
|
| 106 |
+
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig2-architecture.png" width="100%" alt="GroundAnything: vision-language architecture and autoregressive-to-diffusion conversion" /></p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
## Input(s):
|
| 109 |
|
|
|
|
| 150 |
| [GroundAnything](https://huggingface.co/GroundingPI/GroundAnything) | Entropy-guided blockwise diffusion; optional self-speculation | **GAM** |
|
| 151 |
| [GroundAnything-VLM](https://huggingface.co/GroundingPI/GroundAnything-VLM) | Autoregressive generation | **GAM** |
|
| 152 |
|
| 153 |
+
GroundAnything's main results use **entropy-guided decoding**; GroundAnything-VLM uses **autoregressive decoding**. Self-speculative decoding is an optional acceleration mode.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
|
| 155 |
+
## Evaluation
|
| 156 |
|
| 157 |
+
The evaluation toolkit supports **7 modes**:
|
| 158 |
|
| 159 |
+
| Mode | Supported models |
|
| 160 |
+
|:---|:---|
|
| 161 |
+
| **`GAM`** | **GroundingPI, GroundAnything, GroundAnything-VLM** |
|
| 162 |
+
| `VLM` | Generic vision-language baselines |
|
| 163 |
+
| `REXOMNI` | Rex-Omni |
|
| 164 |
+
| `LOCATEANYTHING` | LocateAnything |
|
| 165 |
+
| `GROUNDINGDINO` | GroundingDINO through a compatible service |
|
| 166 |
+
| `DLM` | Legacy diffusion checkpoints using the GAM protocol |
|
| 167 |
+
| `RLV2` | Legacy RL checkpoints using the GAM protocol |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
|
| 169 |
+
**All three released checkpoints use GAM mode.**
|
| 170 |
|
| 171 |
+
Evaluation code and instructions: [GitHub](https://github.com/groundingpi/GroundAnything).
|
| 172 |
|
| 173 |
## Quantitative Evaluation Benchmarks
|
| 174 |
|
| 175 |
+
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig7-grounding-performance.png" width="100%" alt="GroundAnything: GroundAnything and GroundAnything-VLM benchmark overview" /></p>
|
|
|
|
|
|
|
| 176 |
|
| 177 |
## Inference:
|
| 178 |
|
| 179 |
### Installation
|
| 180 |
|
| 181 |
+
Run the commands from the **GroundAnything source repository root** after obtaining the code package, using **Linux x86_64 and Python 3.12**. Use the serving profile in the supplied source bundle so that the custom model adapter, decoding implementation, and dependencies remain aligned.
|
| 182 |
|
| 183 |
```bash
|
| 184 |
python3 -m pip install -r requirements.txt huggingface_hub
|
|
|
|
| 242 |
)
|
| 243 |
```
|
| 244 |
|
| 245 |
+
The HTTP client retains no model weights.
|
| 246 |
|
| 247 |
### Supported Tasks & Prompt Templates
|
| 248 |
|
|
|
|
| 277 |
|
| 278 |
The model uses its own shared weights to draft tokens with bidirectional attention and verify them with causal attention. Verification accepts the **longest consecutive matching prefix**, stops at the first mismatch, applies the causal correction, and discards the rejected suffix cache states. The shipped speculative route uses greedy verification; it is not a general stochastic speculative sampler.
|
| 279 |
|
| 280 |
+
<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything-VLM/resolve/c367acf62cc8255782e3d19bd8c312f9b7ab35db/assets/fig6-self-speculative-decoding.png" width="100%" alt="GroundAnything: linear and quadratic self-speculative schedules with shared model weights" /></p>
|
|
|
|
|
|
|
| 281 |
|
| 282 |
The documented `--decoder speculative` service is the linear shared-weight route. Exact greedy verification is relative to the converted model's causal branch; it does not imply identical outputs to the separately trained GroundAnything-VLM checkpoint.
|
| 283 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
## Inference Infrastructure
|
| 285 |
|
| 286 |
### SGLang Execution
|
| 287 |
|
| 288 |
The custom SGLang integration coordinates model loading, request scheduling, attention kernels, and KV-cache ownership for blockwise generation. Denoising uses bidirectional attention inside the active block; completed history is retained as causal KV. Self-speculation additionally verifies proposals and removes rejected suffix states. These cache semantics must be preserved when optimizing execution.
|
| 289 |
|
| 290 |
+
The supplied recipe selects **BF16 + Triton attention + eager execution**. DLM recipes record the engine source, decoder, effective settings, and package versions in `outputs/sglang/<decoder>/engine_runtime.json`. The VLM service records its causal model/runtime configuration separately in `outputs/sglang/vlm/engine_runtime.json`. For higher service concurrency, use independent replicas with distinct devices, ports, and output directories; the default queue is not continuous multi-request batching.
|
| 291 |
|
| 292 |
### CUDA Graph and Selective FP8
|
| 293 |
|
checksums.sha256
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
7316e325dde0dd407bcc80ed5d3d080223e54572a4685b2aa1ee52041e41d466 added_tokens.json
|
| 4 |
a0bc6f6fc7a29a80017a433e8f03a1cc1236e838a944a2d034295a60c4f2fddb chat_template.jinja
|
| 5 |
2a23a7cf76128488c8866582bab06ffb1db387f7f7de82e06fafea65514ce10d config.json
|
|
@@ -22,3 +22,5 @@ a839e10631620ae5fe70ec3aa9d009af521f7ea03faa66b32adce287105003be modeling_groun
|
|
| 22 |
1a45e3949a652ba4fa8e65017b134404adee1146e1c61fa05e55cdcfa1279e01 tokenizer.json
|
| 23 |
d9089e4621f850e727a86c64110730d10feb459a9532e8bd8b77c8303d417bd3 tokenizer_config.json
|
| 24 |
ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910 vocab.json
|
|
|
|
|
|
|
|
|
| 1 |
+
c419c191a33774f4e1f834b3fb36033dc84e8ba316a3d1935eed1fa529884a74 LICENSE
|
| 2 |
+
1fc64576af2dec8f1eb4fe61170843b2fec0c0713157c7493920761bc945c266 README.md
|
| 3 |
7316e325dde0dd407bcc80ed5d3d080223e54572a4685b2aa1ee52041e41d466 added_tokens.json
|
| 4 |
a0bc6f6fc7a29a80017a433e8f03a1cc1236e838a944a2d034295a60c4f2fddb chat_template.jinja
|
| 5 |
2a23a7cf76128488c8866582bab06ffb1db387f7f7de82e06fafea65514ce10d config.json
|
|
|
|
| 22 |
1a45e3949a652ba4fa8e65017b134404adee1146e1c61fa05e55cdcfa1279e01 tokenizer.json
|
| 23 |
d9089e4621f850e727a86c64110730d10feb459a9532e8bd8b77c8303d417bd3 tokenizer_config.json
|
| 24 |
ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910 vocab.json
|
| 25 |
+
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE-Apache-2.0
|
| 26 |
+
20c797ce19af0c17de52c6afb144644768a591c521655f5ebf5712c9850f2887 LICENSE-Kimi-K3
|