Harisundar commited on
Commit
7d2da0d
Β·
verified Β·
1 Parent(s): 9599cc6

Add dataset citations and training data attribution for open-source release

Browse files
Files changed (1) hide show
  1. README.md +152 -7
README.md CHANGED
@@ -33,6 +33,7 @@ the base weights β€” no PEFT adapter required at inference).
33
  - **Base model:** [`unsloth/Meta-Llama-3.1-8B`](https://huggingface.co/unsloth/Meta-Llama-3.1-8B)
34
  - **Code:** [PALL on GitHub](https://github.com/HARISUNDARRAJENDRAN/PALL)
35
  - **Companion VLM:** [`Harisundar/PALL-VLM`](https://huggingface.co/Harisundar/PALL-VLM)
 
36
  - **Language:** English
37
  - **License:** Llama 3.1 Community License
38
 
@@ -110,14 +111,77 @@ out = model.generate(inputs, max_new_tokens=400, do_sample=False)
110
  print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
111
  ```
112
 
113
- ## Training data
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
114
 
115
- Pulled from the private dataset repo [`Harisundar/pall`](https://huggingface.co/datasets/Harisundar/pall)
116
- (`cpt` / `sft` / `dpo` subsets). Sources include PubMed/PMC dental abstracts, ~200 dental
117
- textbooks, WHO/ADA-style clinical guidelines, open-source dental QA datasets, and synthesized
118
- dental QA derived from patient-style narratives. DPO preference pairs were constructed by
119
- scoring candidate responses on a 5-axis rubric (correctness, conciseness, hedging, clarity,
120
- safety) and pairing top- with bottom-quartile responses.
121
 
122
  ## Limitations & risks
123
 
@@ -179,7 +243,88 @@ If you use PALL, please cite:
179
  }
180
  ```
181
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
182
  ## Acknowledgements
183
 
184
  Built on Llama-3.1 (Meta), Unsloth, Hugging Face TRL/PEFT/Transformers, and bitsandbytes.
 
185
  For research and clinical-decision-**support** only.
 
33
  - **Base model:** [`unsloth/Meta-Llama-3.1-8B`](https://huggingface.co/unsloth/Meta-Llama-3.1-8B)
34
  - **Code:** [PALL on GitHub](https://github.com/HARISUNDARRAJENDRAN/PALL)
35
  - **Companion VLM:** [`Harisundar/PALL-VLM`](https://huggingface.co/Harisundar/PALL-VLM)
36
+ - **Training data:** [`Harisundar/pall`](https://huggingface.co/datasets/Harisundar/pall)
37
  - **Language:** English
38
  - **License:** Llama 3.1 Community License
39
 
 
111
  print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
112
  ```
113
 
114
+ ---
115
+
116
+ ## Training Data Sources & Acknowledgements
117
+
118
+ PALL is trained on data assembled from many publicly available sources. We gratefully
119
+ acknowledge the creators of these datasets. Row counts are from our dental-filtered
120
+ subsets; the original datasets may be larger.
121
+
122
+ ### CPT β€” Continued Pre-Training corpus (~406K documents, ~175M tokens)
123
+
124
+ | Source | Rows | Attribution |
125
+ |--------|-----:|-------------|
126
+ | [OpenAlex](https://openalex.org/) dental works | 199,518 | Priem, J., Piwowar, H., & Orr, R. (2022). *OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts.* arXiv:2205.01833 |
127
+ | [PubMed](https://pubmed.ncbi.nlm.nih.gov/) dental abstracts | 95,436 | U.S. National Library of Medicine / NCBI |
128
+ | [PMC](https://www.ncbi.nlm.nih.gov/pmc/) open-access dental full text | 49,895 | U.S. National Library of Medicine / NCBI |
129
+ | Dental textbooks (cleaned) | 32,923 | Various authors (see DPO sources below) |
130
+ | [ClinicalTrials.gov](https://clinicaltrials.gov/) dental studies | 15,379 | U.S. National Library of Medicine |
131
+ | [Wikipedia](https://www.wikipedia.org/) dental articles | 11,634 | Wikimedia Foundation (CC BY-SA) |
132
+ | HuggingFace dental extras | 1,098 | Community datasets |
133
+
134
+ ### SFT β€” Supervised Fine-Tuning (~412K instruction pairs)
135
+
136
+ | Source | Rows | Attribution |
137
+ |--------|-----:|-------------|
138
+ | [`OpenMed/Medical-Reasoning-SFT-Mega`](https://huggingface.co/datasets/OpenMed/Medical-Reasoning-SFT-Mega) | 24,550 | OpenMed team |
139
+ | PubMedQA dental (artificial subset) | 20,662 | Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., & Lu, X. (2019). *PubMedQA: A Dataset for Biomedical Research Question Answering.* EMNLP 2019 |
140
+ | [`ibivibiv/medical_instruct_en`](https://huggingface.co/datasets/ibivibiv/medical_instruct_en) | 19,336 | ibivibiv (HuggingFace) |
141
+ | [`FreedomIntelligence/ApolloCorpus`](https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus) | 18,016 | Wang, X. et al. (2024). *Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People.* arXiv:2403.03640 |
142
+ | ChatDoctor HealthcareMagic dental | 12,423 | Li, Y. et al. (2023). *ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge.* Cureus 15(6) |
143
+ | [`exafluence/Open-MedQA-Nexus`](https://huggingface.co/datasets/exafluence/Open-MedQA-Nexus) | 11,608 | Exafluence team |
144
+ | [`accolade2025/dental-llama2-35k`](https://huggingface.co/datasets/accolade2025/dental-llama2-35k) | 10,197 | accolade2025 (HuggingFace) |
145
+ | [`electricsheepafrica/oral-health-dental-disease`](https://huggingface.co/datasets/electricsheepafrica/oral-health-dental-disease) | 10,000 | electricsheepafrica (HuggingFace) |
146
+ | [`Intelligent-Internet/II-Medical-Reasoning-SFT`](https://huggingface.co/datasets/Intelligent-Internet/II-Medical-Reasoning-SFT) | 9,738 | Intelligent Internet team |
147
+ | [`ruslanmv/ai-medical-chatbot`](https://huggingface.co/datasets/ruslanmv/ai-medical-chatbot) | 9,681 | ruslanmv (HuggingFace) |
148
+ | [`AGBonnet/augmented-clinical-notes`](https://huggingface.co/datasets/AGBonnet/augmented-clinical-notes) | 16,265 | AGBonnet (HuggingFace) |
149
+ | [`FremyCompany/Asclepius-Synthetic-Clinical-Notes-QA-EN`](https://huggingface.co/datasets/FremyCompany/Asclepius-Synthetic-Clinical-Notes-QA-EN) | 8,138 | Kweon, S. et al. (2023). *Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes.* arXiv:2309.00237 |
150
+ | MedMCQA dental subset | 6,315 | Pal, A., Umapathi, L.K., & Sankarasubbu, M. (2022). *MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering.* CHIL 2022 |
151
+ | [`Mxode/Chinese-Medical-Instruct-1M`](https://huggingface.co/datasets/Mxode/Chinese-Medical-Instruct-1M) | 6,528 | Mxode (HuggingFace) |
152
+ | [`miriad/miriad-4.4M`](https://huggingface.co/datasets/miriad/miriad-4.4M) | 5,995 | Zheng, Q. et al. (2025). *MIRIAD: Augmenting LLMs with millions of medical query-response pairs.* arXiv:2506.06091 |
153
+ | HuatuoGPT2-SFT (GPT-4 generated) | 4,977 | Chen, J. et al. (2023). *HuatuoGPT-II: One-stage Training for Medical Adaption of LLMs.* arXiv:2311.09774 |
154
+ | [`BrainHealthAI/MedQA_mutilangual`](https://huggingface.co/datasets/BrainHealthAI/MedQA_mutilangual) | 3,970 | BrainHealthAI (HuggingFace); originally Jin, D. et al. (2021). *What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams.* Applied Sciences |
155
+ | Dental forum / Jonathan Kang dental | 2,419 | Community dental forums |
156
+ | [`Tuminha/dental-evidence-dataset`](https://huggingface.co/datasets/Tuminha/dental-evidence-dataset) | 1,931 | Tuminha (HuggingFace) |
157
+ | [`FreedomIntelligence/medical-o1-reasoning-SFT`](https://huggingface.co/datasets/FreedomIntelligence/medical-o1-reasoning-SFT) | 2,910 | FreedomIntelligence team |
158
+ | [`naazimsnh02/dentalgemma-instruct`](https://huggingface.co/datasets/naazimsnh02/dentalgemma-instruct) | 1,743 | naazimsnh02 (HuggingFace) |
159
+ | [`AnonymousSub/MedQuAD_47441_Question_Answer_Pairs`](https://huggingface.co/datasets/AnonymousSub/MedQuAD_47441_Question_Answer_Pairs) | 1,734 | Ben Abacha, A. & Demner-Fushman, D. (2019). *A Question-Entailment Approach to Question Answering.* BMC Bioinformatics |
160
+ | [`HPAI-BSC/Aloe-Beta-Medical-Collection`](https://huggingface.co/datasets/HPAI-BSC/Aloe-Beta-Medical-Collection) | 1,633 | Gururajan, A.K. et al. (2024). *Aloe: A Family of Fine-tuned Open Healthcare LLMs.* arXiv:2405.01886 |
161
+ | [`saidonepudi8/dental_tc`](https://huggingface.co/datasets/saidonepudi8/dental_tc) | 1,149 | saidonepudi8 (HuggingFace) |
162
+ | [`lavita/AlpaCare-MedInstruct-52k`](https://huggingface.co/datasets/lavita/AlpaCare-MedInstruct-52k) | 1,063 | Zhang, X. et al. (2023). *AlpaCare: Instruction-tuned Large Language Models for Medical Application.* arXiv:2310.14558 |
163
+ | Textbook-derived SFT | 2,039 | Various dental textbook authors |
164
+
165
+ ### DPO β€” Direct Preference Optimization (~10.7K preference triplets)
166
+
167
+ DPO preference pairs were constructed from dental textbook content by scoring candidate
168
+ responses on a 5-axis rubric (correctness, conciseness, hedging, clarity, safety) and
169
+ pairing top- with bottom-quartile responses. Source textbooks include works by:
170
+
171
+ - Hargreaves, K.M. β€” *Cohen's Pathways of the Pulp* (9th Ed.)
172
+ - Humes, H.D. β€” *Kelley's Textbook of Internal Medicine* (4th Ed.)
173
+ - Harvey, A. β€” *Lippincott's Illustrated Biochemistry* (5th Ed.)
174
+ - Myers, J. β€” *Oral Cancer Metastasis*
175
+ - Koolman, J. β€” *Color Atlas of Biochemistry* (2nd Ed.)
176
+ - Fonseca, R.J. β€” *Oral and Maxillofacial Surgery* (Vol I)
177
+ - Mehta, N.R. β€” *Head, Face, and Neck Pain*
178
+ - Fejerskov, O. & Kidd, E. β€” *Dental Caries: The Disease and its Clinical Management* (2nd Ed.)
179
+ - Mitchell, D.A. β€” *Oxford Handbook of Clinical Dentistry* (6th Ed.)
180
+ - Regezi, J.A. β€” *Oral Pathology* (7th Ed.)
181
+ - Malamed, S.F. β€” *Sedation: A Guide to Patient Management* (5th Ed.)
182
+ - And 100+ additional dental and medical textbooks
183
 
184
+ ---
 
 
 
 
 
185
 
186
  ## Limitations & risks
187
 
 
243
  }
244
  ```
245
 
246
+ ### Key dataset citations
247
+
248
+ ```bibtex
249
+ @inproceedings{jin2019pubmedqa,
250
+ title={PubMedQA: A Dataset for Biomedical Research Question Answering},
251
+ author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua},
252
+ booktitle={EMNLP-IJCNLP}, pages={2567--2577}, year={2019}
253
+ }
254
+ @InProceedings{pal2022medmcqa,
255
+ title={MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering},
256
+ author={Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan},
257
+ booktitle={CHIL}, series={PMLR}, volume={174}, pages={248--260}, year={2022}
258
+ }
259
+ @misc{wang2024apollo,
260
+ title={Apollo: Lightweight Multilingual Medical LLMs towards Democratizing Medical AI to 6B People},
261
+ author={Xidong Wang and Nuo Chen and Junyin Chen and Yan Hu and Yidong Wang and Xiangbo Wu
262
+ and Anningzhe Gao and Xiang Wan and Haizhou Li and Benyou Wang},
263
+ eprint={2403.03640}, archivePrefix={arXiv}, year={2024}
264
+ }
265
+ @article{li2023chatdoctor,
266
+ title={ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge},
267
+ author={Li, Yunxiang and Li, Zihan and Zhang, Kai and Dan, Ruilong and Jiang, Steve and Zhang, You},
268
+ journal={Cureus}, volume={15}, number={6}, year={2023}, doi={10.7759/cureus.40895}
269
+ }
270
+ @article{chen2023huatuogpt2,
271
+ title={HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs},
272
+ author={Chen, Junying and Wang, Xidong and Gao, Anningzhe and Jiang, Feng and Chen, Shunian
273
+ and Zhang, Hongbo and Song, Dingjie and Xie, Wenya and Kong, Chuyi and Li, Jianquan
274
+ and Wan, Xiang and Li, Haizhou and Wang, Benyou},
275
+ journal={arXiv preprint arXiv:2311.09774}, year={2023}
276
+ }
277
+ @misc{kweon2023asclepius,
278
+ title={Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes},
279
+ author={Sunjun Kweon and Junu Kim and Jiyoun Kim and Sujeong Im and Eunbyeol Cho and Seongsu Bae
280
+ and Jungwoo Oh and Gyubok Lee and Jong Hak Moon and Seng Chan You and Seungjin Baek
281
+ and Chang Hoon Han and Yoon Bin Jung and Yohan Jo and Edward Choi},
282
+ eprint={2309.00237}, archivePrefix={arXiv}, year={2023}
283
+ }
284
+ @misc{zhang2023alpacare,
285
+ title={AlpaCare: Instruction-tuned Large Language Models for Medical Application},
286
+ author={Xinlu Zhang and Chenxin Tian and Xianjun Yang and Lichang Chen and Zekun Li and Linda Ruth Petzold},
287
+ eprint={2310.14558}, archivePrefix={arXiv}, year={2023}
288
+ }
289
+ @article{benabacha2019medquad,
290
+ title={A Question-Entailment Approach to Question Answering},
291
+ author={Ben Abacha, Asma and Demner-Fushman, Dina},
292
+ journal={BMC Bioinformatics}, volume={20}, number={1}, pages={511:1--511:23}, year={2019}
293
+ }
294
+ @article{jin2021medqa,
295
+ title={What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams},
296
+ author={Jin, Di and Pan, Eileen and Oufattole, Nassim and Weng, Wei-Hung and Fang, Hanyi and Szolovits, Peter},
297
+ journal={Applied Sciences}, volume={11}, number={14}, pages={6421}, year={2021}, doi={10.3390/app11146421}
298
+ }
299
+ @article{priem2022openalex,
300
+ title={OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts},
301
+ author={Priem, Jason and Piwowar, Heather and Orr, Richard},
302
+ journal={arXiv preprint arXiv:2205.01833}, year={2022}
303
+ }
304
+ @misc{zheng2025miriad,
305
+ title={MIRIAD: Augmenting LLMs with millions of medical query-response pairs},
306
+ author={Qinyue Zheng and Salman Abdullah and Sam Rawal and Cyril Zakka and Sophie Ostmeier
307
+ and Maximilian Purk and Eduardo Reis and Eric J. Topol and Jure Leskovec and Michael Moor},
308
+ eprint={2506.06091}, archivePrefix={arXiv}, year={2025}
309
+ }
310
+ @misc{gururajan2024aloe,
311
+ title={Aloe: A Family of Fine-tuned Open Healthcare LLMs},
312
+ author={Ashwin Kumar Gururajan and Enrique Lopez-Cuena and Jordi Bayarri-Planas and Adrian Tormos
313
+ and Daniel Hinjos and Pablo Bernabeu-Perez and Anna Arias-Duart and Pablo Agustin Martin-Torres
314
+ and Lucia Urcelay-Ganzabal and Marta Gonzalez-Mallo and Sergio Alvarez-Napagao
315
+ and Eduard Ayguade-Parra and Ulises Cortes and Dario Garcia-Gasulla},
316
+ eprint={2405.01886}, archivePrefix={arXiv}, year={2024}
317
+ }
318
+ @misc{chen2024huatuogpto1,
319
+ title={HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs},
320
+ author={Junying Chen and Zhenyang Cai and Ke Ji and Xidong Wang and Wanlong Liu and Rongsheng Wang
321
+ and Jianye Hou and Benyou Wang},
322
+ eprint={2412.18925}, archivePrefix={arXiv}, year={2024}
323
+ }
324
+ ```
325
+
326
  ## Acknowledgements
327
 
328
  Built on Llama-3.1 (Meta), Unsloth, Hugging Face TRL/PEFT/Transformers, and bitsandbytes.
329
+ We thank all dataset creators listed above for making their data publicly available.
330
  For research and clinical-decision-**support** only.