FundusReasoner
FundusReasoner: An Interpretable Vision-Language Reasoning Model for Fundus Diagnosis
ไธญๆไป็ป
ๆจกๅ็ฎไป
FundusReasoner ๆฏไธไธช้ขๅๅฝฉ่ฒ็ผๅบ็ ง็ๅๆไธ่ฏๆญ็ๅฏ่งฃ้่ง่ง่ฏญ่จๆจกๅ๏ผVision-Language Model, VLM๏ผใ
่ฏฅๆจกๅๆจๅจไธไป ็ปๅบๆ็ป่ฏๆญ็ปๆ๏ผ่ฟ่ฝๅคๅฐ็ผๅบๅพๅไธญ็ๅฑ้จ่ง่ง่ฏๆฎใ็ ็ถๆ่ฟฐใ่ฏๆญๅๆไธๆ็ป่ฏๆญ่็ณป่ตทๆฅ๏ผไป่ๆไพๆดๅ ทๅฏๆฃๆฅๆง็็ผๅบๅพๅๅๆ่ฟ็จใ
FundusReasoner ๅบไบ Qwen3.5 ๅคๆจกๆๆจกๅๆๅปบ๏ผๅนถ้่ฟ้ๆญฅๅขๅ ไปปๅก้พๅบฆๅ่ฏๆฎ็ฒๅบฆ็่ฎญ็ปๆนๅผ๏ผไฝฟๆจกๅๅญฆไน ๏ผ
็ผ็ง่ฏญไน็่งฃ๏ผOphthalmic Semantic Adaptation๏ผ
ๅญฆไน ็ผๅบ็พ็ ใๆญฃๅธธ/ๅผๅธธ็ถๆใ็ผ็งๆฏ่ฏญๅ็ธๅ ณ่ง่ง้ฎ็ญไปปๅกใๅบๅ็บง่ง่งๅฎไฝ๏ผRegion-level Visual Grounding๏ผ
ไฝฟ็จ่พน็ๆกๆๅๆ ๅฝขๅผ๏ผๅฐ็ผ็ง่งฃๅ็ปๆๅ็ ็ถๆฆๅฟตไธ็ผๅบๅพๅไธญ็ๅ ทไฝๅบๅ่ฟ่กๅฏน้ฝใ็ปๆๅ่ฏๆฎ้พๆจ็๏ผStructured Evidence-chain Reasoning๏ผ
ๅจ่ฏๆญ่ฟ็จไธญไพๆฌก็ๆ็ ็ถๅบๅใๅฝฑๅ่กจ็ฐๆ่ฟฐใ่ฏๆฎๅๆไปฅๅๆ็ป่ฏๆญ๏ผไฝฟ่ฏๆญ็ป่ฎบ่ฝๅคไธๅพๅ่ฏๆฎๅปบ็ซๆพๅผ่็ณปใ
ๆจกๅ็็ปๆๅ่ฏๆญ่พๅบๅฝขๅผๅฆไธ๏ผ
<thinking>
<region>(x1, y1, x2, y2)</region>
<description>็ผๅบๅฝฑๅ่กจ็ฐๆ่ฟฐ</description>
...
<analyse>ๅบไบไธ่ฟฐๅฝฑๅ่ฏๆฎ่ฟ่ก่ฏๆญๅๆ</analyse>
</thinking>
<answer>ๆ็ป่ฏๆญ</answer>
ๅๆ ็ปไธ้็จ๏ผ
(x1, y1, x2, y2)
ๅ
ถไธญ (x1, y1) ไธบๅทฆไธ่ง๏ผ(x2, y2) ไธบๅณไธ่ง๏ผๅๆ ่ๅดไธบ 0~512ใ
้กน็ฎ่ตๆบ
ไปฃ็ ไปๅบ / Code Repository
https://gitee.com/guoyitongxue/fundus-reasoner-qwen35Hugging Face ็จๆทไธป้กต / Hugging Face Profile
https://huggingface.co/zym1105ๅฝๅๆจกๅไปๅบ / Model Repository
https://huggingface.co/zym1105/Qwen3.5FundusReasoner_base
ไปฃ็ ไปๅบๅ ๅซ FundusReasoner ็้กน็ฎไป็ปใ่ฎญ็ปไธ่ฏๆตไปฃ็ ๏ผไปฅๅ็ธๅ ณๅฎ้ช้ ็ฝฎใๅ็ปญๆจกๅไธ้กน็ฎๆดๆฐไนๅฐไผๅ ๅจไธ่ฟฐไปๅบไธญ็ปดๆคใ
้ขๅฎไนๆ็คบ่ฏ / Prompt Templates
ไธ้ข็ปๅบ FundusReasoner ่ฎญ็ปๅ่ฏๆต่ฟ็จไธญไฝฟ็จ็ๅ ็ฑปๅ ธๅๆ็คบ่ฏใ
<image>่กจ็คบ่พๅ ฅ็็ผๅบๅพๅใๅจๅฎ้ ๆจ็ๆถ๏ผๅบๆ็ งๆไฝฟ็จ็ๅคๆจกๆๆจ็ๆกๆถไผ ๅ ฅๅพๅ๏ผ่ไธๆฏๅฐ<image>ๅฝไฝๆฎ้ๆๆฌๅ็ฌ่พๅ ฅใ
1. ่งฃๅ็ปๆๅฎไฝ / Anatomical Structure Localization
<image>่ฏทๆ ๅบ่ง็็ไฝ็ฝฎ๏ผๅนถๆไพๅฏนๅบ็่พน็ๆกๅๆ ๏ผ[x1,y1,x2,y2]๏ผๅทฆไธ-ๅณไธ๏ผใ
่ฏฅไปปๅก่ฆๆฑๆจกๅๆ นๆฎ่พๅ ฅ็ผๅบ็ ง็ๅฎไฝๆๅฎ่งฃๅ็ปๆ๏ผๅนถ่พๅบๅฏนๅบ็ๅบๅๅๆ ใ
2. ๅ้ๅบๅ็ปๆ่ฏๅซ / Region Recognition
<image>่ฏทๅคๆญๅ้ๆก (0,261,53,316) ๅ
ๅฏนๅบ็ไธป่ฆ็็็ปๆๆฏไปไน๏ผๅช่พๅบๅ็งฐใ
่ฏฅไปปๅก็ปๅฎไธไธชๅ้ๅบๅ๏ผ่ฆๆฑๆจกๅ่ฏๅซๅ ถไธญไธป่ฆ็็ผๅบ่งฃๅ็ปๆใ
3. ๅบๅ-ๆฆๅฟต้ช่ฏ / Region-Concept Verification
<image>ไธๅๅ้ๅบๅๆฏๅฆไธบ(0,261,53,316)๏ผๅ้ๆก๏ผ่ง็ใ่ฏทๅ็ญๆฏๆๅฆใ
่ฏฅไปปๅก็จไบๅคๆญ็ปๅฎๅบๅไธๆๅฎ็ผ็งๆฆๅฟตๆฏๅฆๅน้ ใ
4. ็ปๆๅ่ฏๆฎ้พ่ฏๆญ / Structured Evidence-chain Diagnosis
<image>
ไฝ ๆฏไธๅ็ป้ชไธฐๅฏ็็ผ็งๅป็๏ผ้่ฆๆๆๅฎๆ ผๅผๅฏน็ผๅบๅพๅ่ฟ่กๅๆไธ่ฏๆญใ
่ฏทไธฅๆ ผๆ็
งไปฅไธ็ปๆ่พๅบ๏ผ
<thinking>
<region>(x1,y1,x2,y2)</region>
<description>ๆ่ฟฐ่ฏฅๅบๅ็ๅฏ็็
ๅ๏ผ่ฅ่ฏฅๅบๅๆ ๅผๅธธ๏ผๅไฝฟ็จๅๆ (0,0,512,512)๏ผๅนถๅฐๆ่ฟฐๅบๅฎๅไธบโๆช่งๅผๅธธโใ</description>
<!-- ๅฏๅค็ป๏ผๆฏไธช็
็ถไธ็ปใ่ฅ็
็ถๅๅ
จๅฑๆงๅๅ๏ผๆถๅๆดไธช็ผๅบ๏ผ๏ผไน้ๅ็ฌไฝไธบไธ็ป๏ผๅนถ็ปไธไฝฟ็จๅบๅ(0,0,512,512)ใ -->
<analyse>ๅบไบไธ่ฟฐ็
็ถ๏ผ็ปๅบโ็
็ถ็นๅพ โ ๆจ็ โ ่ฏๆญโ็ๅๆ่ฟ็จใ</analyse>
</thinking>
<answer>็ปๅบๆ็ป่ฏๆญ๏ผ่ฅๆดไฝๆ ๅผๅธธ๏ผ่ฏทๅฐๅ
ๅฎนๅบๅฎๅไธบโๆญฃๅธธ็ผๅบโใ</answer>
ใๅๆ ๆ ผๅผไธบ (x1,y1,x2,y2)๏ผๅๅซ่กจ็คบๅทฆไธ่งไธๅณไธ่ง๏ผๆฐๅผ่ๅด 0~512ใ
ใๆดไฝๆ ๅผๅธธๆถ๏ผไป
่พๅบไธ็ป๏ผregion=(0,0,512,512)ใdescription=โๆช่งๅผๅธธโ๏ผanswer=โๆญฃๅธธ็ผๅบโใ
ใๅ
จๅฑๆง็
็ถ้ไปฅ region=(0,0,512,512) ๅ็ฌๆ ๆณจ๏ผๅนถๅจ description ไธญๆ่ฟฐใ
ใๆฏไธช region ๅฟ
้กป็ดง่ทๅฏนๅบ็ descriptionใ
่ฟๆฏ FundusReasoner ็ๆ ธๅฟ่ฏๆญๅฝขๅผใๆจกๅ้ฆๅ
ๅฎไฝๅฏ็่ง่ง่ฏๆฎ๏ผ็ถๅๆ่ฟฐๅฏนๅบๅฝฑๅ่กจ็ฐ๏ผๅนถๅจ <analyse> ไธญๆดๅ่ฏๆฎ๏ผๆ็ปๅจ <answer> ไธญ็ปๅบ่ฏๆญใ
5. ็ดๆฅ่ฏๆญ / Direct Diagnosis
<image>่ฏฅๆฃ่
็็ผๅบๆ
ๅตๅฆไฝ๏ผ่ฏท็ดๆฅ่ฏๆญใ
่ฏฅไปปๅก็จไบ็ดๆฅ็ๆ็ผๅบ่ฏๆญ็ปๆ๏ผไธ่ฆๆฑ่พๅบๅฎๆด็็ปๆๅ่ฏๆฎ้พใ
ไฝฟ็จ่ฏดๆ
FundusReasoner ้ขๅ็ผๅบๅพๅ็ธๅ ณ็่ง่ง่ฏญ่จไปปๅก๏ผๅ ๆฌไฝไธ้ไบ๏ผ
- ็ผๅบ็พ็ ่ฏๆญ๏ผ
- ่งฃๅ็ปๆๅฎไฝ๏ผ
- ็ ็ถไธๅบๅ่ฏๅซ๏ผ
- ๅ้ๅบๅ้ช่ฏ๏ผ
- ๅพๅ็บง็ผ็ง่ง่ง้ฎ็ญ๏ผ
- ๅบไบๅฑ้จๅฝฑๅ่ฏๆฎ็็ปๆๅ่ฏๆญๆจ็ใ
ๅฏนไบ้่ฆๅฏ่งฃ้ๅๆ็ๅบๆฏ๏ผๆจ่ไฝฟ็จ็ปๆๅ่ฏๆฎ้พ่ฏๆญๆ็คบ่ฏ๏ผไปฅ่ทๅพ๏ผ
ๅบๅๅฎไฝ
โ
ๅฝฑๅ่กจ็ฐๆ่ฟฐ
โ
่ฏๆฎๅๆ
โ
ๆ็ป่ฏๆญ
็ธๆฏไป ่พๅบ็พ็ ๅ็งฐ๏ผ่ฏฅๅฝขๅผๅฏไปฅ็ดๆฅๆฃๆฅๆจกๅๆ็ป่ฏๆญๆไพๆฎ็ๅพๅๅบๅๅ่ง่ง่ฏๆฎใ
็ๆฌ่ฏดๆไธๅฝๅ้็จ่ๅด
ๅฝๅๅๅธ็ Qwen3.5FundusReasoner_base ไธบ FundusReasoner ็ๅบ็ก็ๆฌ๏ผไธป่ฆ้ขๅ็ฒ็ฒๅบฆ็ผๅบๅพๅ้ขๆ ๆณจใๅ้ๆ ็ญพ็ๆไธๅ็ปญไบบๅทฅ็ญ้๏ผๅนถไธๅปบ่ฎฎ็ดๆฅ็จไบ็ๅฎไธดๅบ่ฏๆญใ
้่ฆ็นๅซ่ฏดๆ็ๆฏ๏ผๅฝๅ base ็ๆฌๅจ่ฎญ็ป่ฟ็จไธญไฟ็ไบไธๅ่ฏญไน็ฒๅบฆๅไธๅ่ฏๆญๅฑ็บง็ๆ ็ญพ๏ผๅ ๆญค้จๅๆ ็ญพไน้ดๅฏ่ฝๅญๅจไธๅฎ็จๅบฆ็้ๅ ใไพๅฆ๏ผๅไธๅค้ป็ฝ่ฒ็ ็ถๅจ่พ็ฒ็ฒๅบฆไธๅฏไปฅ่ขซๆ่ฟฐไธบโ้ป็ฝ่ฒๆๅโ๏ผ่ๅจ็ปๅๅ ถๅฝขๆใๅๅธๅๅฑ้จ็นๅพ่ฟไธๆญฅๅคๆญๅ๏ผๅๅฏ่ฝ่ขซ่ฏๅซไธบโ็ป็่็ฃโใ่ฟ็ฑปๆ ็ญพๅนถ้ๅฎๅ จไบๆฅ๏ผ่ๆฏๅๆ ไบไปไธ่ฌๅฝฑๅ่กจ็ฐๅฐๆดๅ ทไฝ็ ็ๆฆๅฟต็้็บง็ปๅ่ฟ็จใ
ไธบไบๆ้ซๆจกๅๅจๆฐๆฎ้ขๆ ๆณจ้ถๆฎต็่ฆ็็๏ผๅนถไธบๅ็ปญไบบๅทฅๅฎกๆ ธใๆ ็ญพ็ญ้ๅๆฐๆฎๆด็ๆไพๆดๅ ๅ็ๅ้ไฟกๆฏ๏ผๅฝๅ base ็ๆฌๆๆไฟ็ไบไธๅๅฐบๅบฆๅไธๅๅฑ็บง็ๆ ็ญพไฝ็ณปใๅ ๆญค๏ผๅจ็ดๆฅไฝฟ็จๆจกๅ่ฟ่กๅพๅๅๆๆถ๏ผๆจกๅๅฏ่ฝ่พๅบ้จๅ่ฏญไน้ๅ ใๅฑ็บงไธๅๆ็ธไบๅ ๅซ็ๆ ็ญพใ
ๅบไบไธ่ฟฐๅๅ ๏ผๆฌ็ๆฌๆด้ๅ็จไบ๏ผ
- ๅคง่งๆจก็ผๅบๆฐๆฎ็็ฒ็ฒๅบฆ้ขๆ ๆณจ๏ผ
- ๅ้็ ็ถๅๅ้่ฏๆญๆ ็ญพ็ๆ๏ผ
- ไบบๅทฅๅฎกๆ ธๅ็ๆฐๆฎ็ญ้ไธ่พ ๅฉๆด็๏ผ
- ๅ็ปญๆไปคๆฐๆฎๅ็ปๆๅ่ฏๆฎ้พๆฐๆฎ็ๆๅปบใ
ๆฌ็ๆฌไธๅบ่ขซ่งไธบๆ็ป็ไธดๅบ่ฏๆญๆจกๅใ
ๆไปฌๆญฃๅจ่ฟไธๆญฅๆด็ๅไฟฎ่ฎขๆดๅ ไธฅๆ ผ็็ผ็ง่ฏๆญๆ ๅ๏ผๅนถๅบไบ่ฟไบๆ ๅๅผๅๅ็ปญ็ๆไปคๅพฎ่ฐ็ๆฌ๏ผinstruction-tuned version๏ผใ่ฏฅ็ๆฌๅฐ่ฟไธๆญฅ่ง่ๆ ็ญพไฝ็ณปใ่ฏๆญ็ฒๅบฆใ่พๅบๆ ผๅผๅไธดๅบๅคๅฎๆ ๅ๏ผไปฅๅๅฐๆ ็ญพๅไฝๅ่ฏญไนๅฑ็บงๆททๆ้ฎ้ขใ
็ธๅ ณ่ฏๆญๆ ๅ็ฎๅไปๅจ็ฑ็ผ็งๅป็่ฟ่กไฟฎ่ฎขๅๅฎกๆ ธใๅพ ๆ ๅๅฎๆๅ๏ผๆไปฌๅฐๅฐฝๅฟซๅๅธๅฏนๅบ็ๆไปคๅพฎ่ฐๆจกๅๅๆดๅฎๆด็ไฝฟ็จ่ฏดๆใ
็ ็ฉถ็จ้ๅฃฐๆ
ๆฌๆจกๅไธป่ฆ็จไบ็ง็ ไธๅญฆๆฏไบคๆตใ
FundusReasoner ๅฐๆชไฝไธบๅป็ๅจๆขฐๅฎๆไธดๅบ้ช่ฏ๏ผๅ ถ่พๅบๅฏ่ฝๅญๅจ้่ฏฏใ้ๆผๆไธๅ็กฎ็ๅปๅญฆๅคๆญใๅ ๆญค๏ผ
- ไธๅบๅฐๆจกๅ่พๅบไฝไธบ็ฌ็ซ็ไธดๅบ่ฏๆญไพๆฎ๏ผ
- ไธๅบๆฟไปฃๅ ทๆ่ต่ดจ็็ผ็งๅป็่ฟ่ก่ฏๆญๆๆฒป็ๅณ็ญ๏ผ
- ๅจๅปๅญฆ็ ็ฉถๅไธดๅบ็ธๅ ณๅฎ้ชไธญ๏ผๅบ็ฑไธไธไบบๅๅฏนๆจกๅ่พๅบ่ฟ่ก็ฌ็ซๅฎกๆ ธใ
English
Model Overview
FundusReasoner is an interpretable vision-language model designed for color fundus image analysis and diagnosis.
Rather than producing only a final disease label, FundusReasoner is designed to explicitly connect localized visual evidence, descriptions of ophthalmic findings, diagnostic analysis, and the final diagnosis, making the evidence supporting a prediction more directly inspectable.
FundusReasoner is built on a Qwen3.5 multimodal backbone and is trained with progressively increasing task complexity and evidence granularity. The framework consists of three main stages:
Ophthalmic Semantic Adaptation
The model learns fundus disease semantics, ophthalmic terminology, normal/abnormal concepts, and domain-specific visual question answering.Region-level Visual Grounding
Anatomical structures and pathological findings are aligned with localized image regions using bounding boxes or coordinate-based representations.Structured Evidence-chain Reasoning
The model generates localized regions, descriptions of visible findings, evidence analysis, and the final diagnosis in sequence, explicitly linking diagnostic conclusions to visual evidence.
The structured diagnostic output follows the general format:
<thinking>
<region>(x1, y1, x2, y2)</region>
<description>Visible ophthalmic finding</description>
...
<analyse>Evidence-based diagnostic analysis</analyse>
</thinking>
<answer>Final diagnosis</answer>
Coordinates follow the format:
(x1, y1, x2, y2)
where (x1, y1) and (x2, y2) represent the top-left and bottom-right corners, respectively. Coordinates are normalized to the 0โ512 image coordinate space used by FundusReasoner.
Resources
Code Repository
https://gitee.com/guoyitongxue/fundus-reasoner-qwen35Hugging Face Profile
https://huggingface.co/zym1105Model Repository
https://huggingface.co/zym1105/Qwen3.5FundusReasoner_base
The code repository contains the FundusReasoner project description, training and evaluation code, experiment configurations, and related resources.
Supported Task Types
FundusReasoner is designed for several fundus-oriented vision-language tasks, including:
- fundus disease diagnosis;
- anatomical structure localization;
- lesion and region recognition;
- region-concept verification;
- ophthalmic visual question answering;
- structured evidence-based diagnostic reasoning.
For interpretable diagnostic analysis, we recommend using the Structured Evidence-chain Diagnosis prompt shown above.
Its output can be interpreted as:
Region localization
โ
Finding description
โ
Evidence analysis
โ
Final diagnosis
This formulation makes it possible to inspect the visual regions and ophthalmic findings used by the model to support its diagnostic conclusion.
Version Notice and Current Scope
The currently released Qwen3.5FundusReasoner_base is the base version of FundusReasoner. It is primarily intended for coarse-grained fundus image pre-annotation, candidate label generation, and subsequent human screening, rather than direct use for real-world clinical diagnosis.
An important characteristic of the current base model is that it preserves labels at different semantic granularities and diagnostic levels. As a result, some labels may partially overlap. For example, the same yellow-white retinal lesion may first be described at a coarse level as a yellow-white plaque, and may subsequently be identified more specifically as drusen after considering its morphology, distribution, and local characteristics.
Such labels are not necessarily mutually exclusive. Instead, they may reflect a progressive refinement process from a general visual appearance to a more specific pathological concept.
To improve recall during large-scale data pre-annotation and to provide richer candidate information for later manual review, label filtering, and dataset curation, the current base version intentionally retains labels across multiple semantic scales and diagnostic levels. Consequently, direct model outputs may contain partially redundant, hierarchically related, or semantically overlapping labels.
For this reason, the current version is more suitable for:
- coarse-grained pre-annotation of large-scale fundus datasets;
- generation of candidate lesion and diagnosis labels;
- data screening and organization before expert review;
- construction of instruction-tuning data and structured evidence-chain annotations.
This release should not be regarded as a finalized clinical diagnostic model.
We are currently developing a subsequent instruction-tuned version based on a more strictly curated ophthalmic diagnostic standard. The upcoming version will further standardize label definitions, diagnostic granularity, output formats, and clinical decision criteria, with the goal of reducing redundant labels and ambiguity across semantic levels.
The corresponding diagnostic standards are still being revised and reviewed by ophthalmologists. Once this process is completed, we plan to release the instruction-tuned model together with more detailed usage guidelines as soon as possible.
Research-use Notice
FundusReasoner is released primarily for academic and research purposes.
The model has not been clinically validated as a medical device and may produce incorrect, incomplete, or misleading medical outputs. Therefore:
- model outputs should not be used as an independent basis for clinical diagnosis;
- the model must not replace qualified ophthalmologists or other healthcare professionals;
- outputs used in medical or clinical research should be independently reviewed by qualified experts.
Citation
Citation information for the FundusReasoner paper will be added after publication.
If you use this model before the formal publication of the paper, please cite or link to the FundusReasoner code repository and this Hugging Face model repository.
- Downloads last month
- 16