Release BioJev-Nano full checkpoint

02030c8 27 days ago

8.26 kB

---
library_name: peft
base_model: Qwen/Qwen3.5-0.8B-Base
pipeline_tag: text-classification
tags:
- biojev
- biomedical
- qwen3.5
- peft
- qlora
- natural-language-inference
- text-classification
- biomedical-nlp
- system-one
language:
- en
---
# BioJev-Nano

**BioJev-Nano** is the 0.8B member of the BioJev biomedical decision-model family.

It is built from **Qwen/Qwen3.5-0.8B-Base** and follows the full BioJev adaptation pipeline:

```text
Qwen3.5-0.8B-Base
        │
        ▼
Biomedical DAPT
PubMed + PMC
        │
        ▼
General NLI
SNLI + MNLI + ANLI
        │
        ▼
Biomedical NLI
BioNLI + NLI4CT
        │
        ▼
BioJev-Nano
```

The released checkpoint is a **PEFT/QLoRA sequence-classification adapter** with three NLI labels:

```text
0 → contradiction
1 → entailment
2 → neutral
```

BioJev-Nano is primarily used for lightweight inference, controlled ablation experiments, and analysis of how biomedical domain adaptation and NLI specialization affect transfer.

## Links

- **BioJev-Nano weights:** https://huggingface.co/Gabriel382/BioJev-Nano
- **BioJev-4B weights:** https://huggingface.co/Gabriel382/BioJev
- **Source code:** https://github.com/Gabriel382/BioJev

## Authors

**Creator:** Gabriel Henrique Alencar Medeiros  
**Supervisor:** Lina F. Soualmia

BioJev is developed at **LITIS / Université de Rouen Normandie**.

---

## Training recipe

### Biomedical DAPT

The Nano model uses a lightweight biomedical DAPT stage initialized from Qwen3.5-0.8B-Base.

```text
Biomedical token budget: 50,000
Sequence length:          512
Method:                   QLoRA
```

The biomedical corpus combines PubMed and PMC sources.

### General NLI

```text
SNLI      35,000
MNLI      45,000
ANLI R1   20,000
----------------
Total    100,000
```

### Biomedical NLI

```text
BioNLI    30,000
NLI4CT    10,000
----------------
Total     40,000
```

The released checkpoint corresponds to the **full** Nano configuration:

```text
DAPT → General NLI → Biomedical NLI
```

---

## Evaluation

BioJev-Nano was evaluated on five biomedical benchmarks.

| Dataset | Macro-F1 |
|---|---:|
| BioNLI | 89.41 |
| NLI4CT | 46.07 |
| ChemProt | 14.33 |
| DDI2013 | 32.55 |
| BioRED | 16.16 |
| Relation-transfer mean (ChemProt/DDI2013/BioRED) | 21.01 |

These scores correspond to the full Nano configuration used in the BioJev Sprint 6 ablation study.

### Interpretation

The Nano ablations indicate that:

- general NLI training provides a strong contribution to transfer toward unseen biomedical tasks;
- biomedical NLI training strongly specializes the model toward biomedical inference;
- this specialization can reduce transfer to some unseen relation-classification tasks;
- biomedical DAPT has smaller and more task-dependent effects at the Nano scale.

BioNLI and NLI4CT can be seen during biomedical decision training depending on the experiment, so they should not be interpreted as clean zero-shot transfer tasks for the full model.

---

# Loading BioJev-Nano

Because BioJev is a **three-class PEFT sequence classifier**, the base model must be reconstructed with `num_labels=3` before loading the adapter.

Do **not** rely on a default two-label sequence-classification configuration.

Install:

```bash
pip install -U torch transformers peft accelerate
```

For optional 4-bit loading:

```bash
pip install -U bitsandbytes
```

## Python example

```python
import torch
from peft import PeftConfig, PeftModelForSequenceClassification
from transformers import AutoModelForSequenceClassification, AutoTokenizer
MODEL = "Gabriel382/BioJev-Nano"
LABEL2ID = {
    "contradiction": 0,
    "entailment": 1,
    "neutral": 2,
}
ID2LABEL = {v: k for k, v in LABEL2ID.items()}
peft_config = PeftConfig.from_pretrained(MODEL)
base = AutoModelForSequenceClassification.from_pretrained(
    peft_config.base_model_name_or_path,
    num_labels=3,
    label2id=LABEL2ID,
    id2label=ID2LABEL,
    dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto" if torch.cuda.is_available() else None,
)
model = PeftModelForSequenceClassification.from_pretrained(
    base,
    MODEL,
    is_trainable=False,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = tokenizer.pad_token_id
model.eval()
premise = "The clinical report describes a bacterial pneumonia."
hypothesis = "The patient has an infectious pulmonary disease."
inputs = tokenizer(
    premise,
    hypothesis,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)
device = next(model.parameters()).device
inputs = {k: v.to(device) for k, v in inputs.items()}
with torch.inference_mode():
    logits = model(**inputs).logits[0].float()
    probabilities = torch.softmax(logits, dim=-1)
for label, idx in LABEL2ID.items():
    print(f"{label:14s}: {probabilities[idx].item():.4f}")
```

A standalone version of this example is included in `example_inference.py`.

---

# Jev / System One compatibility API

The BioJev GitHub repository includes a compatibility layer that exposes existing BioJev sequence-classification checkpoints through a Jev/System-One-shaped API:

```text
POST /v1/systemone
```

Supported question types are:

```text
choice
noul
score
```

This is an **engineering compatibility bridge** over the BioJev NLI classifier. BioJev-Nano is not a native Ollama/System-One checkpoint.

## Install the BioJev API

Clone the main project:

```bash
git clone https://github.com/Gabriel382/BioJev.git
cd BioJev
python -m venv .venv
source .venv/bin/activate
pip install -e .
pip install fastapi uvicorn
```

Download or clone this model repository and point the server to its local directory.

Example:

```bash
python scripts/serve_systemone.py \
  --checkpoint /path/to/BioJev-Nano \
  --model-name biojev-nano \
  --load-in-4bit \
  --host 0.0.0.0 \
  --port 8000
```

Check the service:

```bash
curl http://127.0.0.1:8000/health
```

## System One choice example

```bash
curl http://127.0.0.1:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "biojev-nano",
    "state": "The patient has fever, productive cough, and a new lobar infiltrate.",
    "questions": {
      "diagnosis": {
        "type": "choice",
        "instructions": "Which diagnosis is best supported?",
        "criteria": {
          "pneumonia": "Community-acquired pneumonia",
          "asthma": "Acute asthma exacerbation",
          "migraine": "Migraine"
        }
      }
    }
  }'
```

For each candidate, the bridge constructs a premise/hypothesis NLI pair and computes:

```text
support(candidate) =
entailment_logit - contradiction_logit
```

Candidate supports are normalized with a softmax.

For robust Qwen3.5/PEFT compatibility, candidates are evaluated sequentially with `batch_size=1`.

---

## Repository files

A typical BioJev-Nano model repository contains:

```text
BioJev-Nano/
├── adapter_config.json
├── adapter_model.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── chat_template.jinja          # if present in the training checkpoint
├── training_manifest.json       # if present
├── README.md
├── example_inference.py
└── requirements.txt
```

The PEFT adapter does not duplicate the Qwen3.5-0.8B base weights. The base model is downloaded separately when BioJev-Nano is loaded.

---

## Scientific status

BioJev is a research project.

The model, its confidence estimates, and the System One compatibility layer are **not validated clinical decision systems**. They must not be used as the sole basis for diagnosis, treatment, or other high-stakes medical decisions.

---

## Citation

A formal paper citation will be added when the BioJev publication is available.

Until then, please cite the BioJev repository and the specific Hugging Face checkpoint used.

---

## Acknowledgements

BioJev-Nano was created by **Gabriel Henrique Alencar Medeiros** under the supervision of **Lina F. Soualmia** at **LITIS / Université de Rouen Normandie**.

BioJev builds on the Qwen3.5 model family and publicly available biomedical and natural-language-inference datasets.

	---
	library_name: peft
	base_model: Qwen/Qwen3.5-0.8B-Base
	pipeline_tag: text-classification
	tags:
	- biojev
	- biomedical
	- qwen3.5
	- peft
	- qlora
	- natural-language-inference
	- text-classification
	- biomedical-nlp
	- system-one
	language:
	- en
	---

	# BioJev-Nano

	BioJev-Nano is the 0.8B member of the BioJev biomedical decision-model family.

	It is built from Qwen/Qwen3.5-0.8B-Base and follows the full BioJev adaptation pipeline:

	```text
	Qwen3.5-0.8B-Base
	│
	▼
	Biomedical DAPT
	PubMed + PMC
	│
	▼
	General NLI
	SNLI + MNLI + ANLI
	│
	▼
	Biomedical NLI
	BioNLI + NLI4CT
	│
	▼
	BioJev-Nano
	```

	The released checkpoint is a PEFT/QLoRA sequence-classification adapter with three NLI labels:

	```text
	0 → contradiction
	1 → entailment
	2 → neutral
	```

	BioJev-Nano is primarily used for lightweight inference, controlled ablation experiments, and analysis of how biomedical domain adaptation and NLI specialization affect transfer.

	## Links

	- BioJev-Nano weights: https://huggingface.co/Gabriel382/BioJev-Nano
	- BioJev-4B weights: https://huggingface.co/Gabriel382/BioJev
	- Source code: https://github.com/Gabriel382/BioJev

	## Authors

	Creator: Gabriel Henrique Alencar Medeiros
	Supervisor: Lina F. Soualmia

	BioJev is developed at LITIS / Université de Rouen Normandie.

	---

	## Training recipe

	### Biomedical DAPT

	The Nano model uses a lightweight biomedical DAPT stage initialized from Qwen3.5-0.8B-Base.

	```text
	Biomedical token budget: 50,000
	Sequence length: 512
	Method: QLoRA
	```

	The biomedical corpus combines PubMed and PMC sources.

	### General NLI

	```text
	SNLI 35,000
	MNLI 45,000
	ANLI R1 20,000
	----------------
	Total 100,000
	```

	### Biomedical NLI

	```text
	BioNLI 30,000
	NLI4CT 10,000
	----------------
	Total 40,000
	```

	The released checkpoint corresponds to the full Nano configuration:

	```text
	DAPT → General NLI → Biomedical NLI
	```

	---

	## Evaluation

	BioJev-Nano was evaluated on five biomedical benchmarks.

	\| Dataset \| Macro-F1 \|
	\|---\|---:\|
	\| BioNLI \| 89.41 \|
	\| NLI4CT \| 46.07 \|
	\| ChemProt \| 14.33 \|
	\| DDI2013 \| 32.55 \|
	\| BioRED \| 16.16 \|
	\| Relation-transfer mean (ChemProt/DDI2013/BioRED) \| 21.01 \|

	These scores correspond to the full Nano configuration used in the BioJev Sprint 6 ablation study.

	### Interpretation

	The Nano ablations indicate that:

	- general NLI training provides a strong contribution to transfer toward unseen biomedical tasks;
	- biomedical NLI training strongly specializes the model toward biomedical inference;
	- this specialization can reduce transfer to some unseen relation-classification tasks;
	- biomedical DAPT has smaller and more task-dependent effects at the Nano scale.

	BioNLI and NLI4CT can be seen during biomedical decision training depending on the experiment, so they should not be interpreted as clean zero-shot transfer tasks for the full model.

	---

	# Loading BioJev-Nano

	Because BioJev is a three-class PEFT sequence classifier, the base model must be reconstructed with `num_labels=3` before loading the adapter.

	Do not rely on a default two-label sequence-classification configuration.

	Install:

	```bash
	pip install -U torch transformers peft accelerate
	```

	For optional 4-bit loading:

	```bash
	pip install -U bitsandbytes
	```

	## Python example

	```python
	import torch

	from peft import PeftConfig, PeftModelForSequenceClassification
	from transformers import AutoModelForSequenceClassification, AutoTokenizer

	MODEL = "Gabriel382/BioJev-Nano"

	LABEL2ID = {
	"contradiction": 0,
	"entailment": 1,
	"neutral": 2,
	}

	ID2LABEL = {v: k for k, v in LABEL2ID.items()}

	peft_config = PeftConfig.from_pretrained(MODEL)

	base = AutoModelForSequenceClassification.from_pretrained(
	peft_config.base_model_name_or_path,
	num_labels=3,
	label2id=LABEL2ID,
	id2label=ID2LABEL,
	dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
	device_map="auto" if torch.cuda.is_available() else None,
	)

	model = PeftModelForSequenceClassification.from_pretrained(
	base,
	MODEL,
	is_trainable=False,
	)

	tokenizer = AutoTokenizer.from_pretrained(MODEL)

	if tokenizer.pad_token_id is None:
	tokenizer.pad_token = tokenizer.eos_token

	model.config.pad_token_id = tokenizer.pad_token_id
	model.eval()

	premise = "The clinical report describes a bacterial pneumonia."
	hypothesis = "The patient has an infectious pulmonary disease."

	inputs = tokenizer(
	premise,
	hypothesis,
	return_tensors="pt",
	truncation=True,
	max_length=512,
	)

	device = next(model.parameters()).device
	inputs = {k: v.to(device) for k, v in inputs.items()}

	with torch.inference_mode():
	logits = model(**inputs).logits[0].float()
	probabilities = torch.softmax(logits, dim=-1)

	for label, idx in LABEL2ID.items():
	print(f"{label:14s}: {probabilities[idx].item():.4f}")
	```

	A standalone version of this example is included in `example_inference.py`.

	---

	# Jev / System One compatibility API

	The BioJev GitHub repository includes a compatibility layer that exposes existing BioJev sequence-classification checkpoints through a Jev/System-One-shaped API:

	```text
	POST /v1/systemone
	```

	Supported question types are:

	```text
	choice
	noul
	score
	```

	This is an engineering compatibility bridge over the BioJev NLI classifier. BioJev-Nano is not a native Ollama/System-One checkpoint.

	## Install the BioJev API

	Clone the main project:

	```bash
	git clone https://github.com/Gabriel382/BioJev.git
	cd BioJev

	python -m venv .venv
	source .venv/bin/activate

	pip install -e .
	pip install fastapi uvicorn
	```

	Download or clone this model repository and point the server to its local directory.

	Example:

	```bash
	python scripts/serve_systemone.py \
	--checkpoint /path/to/BioJev-Nano \
	--model-name biojev-nano \
	--load-in-4bit \
	--host 0.0.0.0 \
	--port 8000
	```

	Check the service:

	```bash
	curl http://127.0.0.1:8000/health
	```

	## System One choice example

	```bash
	curl http://127.0.0.1:8000/v1/systemone \
	-H "Content-Type: application/json" \
	-d '{
	"model": "biojev-nano",
	"state": "The patient has fever, productive cough, and a new lobar infiltrate.",
	"questions": {
	"diagnosis": {
	"type": "choice",
	"instructions": "Which diagnosis is best supported?",
	"criteria": {
	"pneumonia": "Community-acquired pneumonia",
	"asthma": "Acute asthma exacerbation",
	"migraine": "Migraine"
	}
	}
	}
	}'
	```

	For each candidate, the bridge constructs a premise/hypothesis NLI pair and computes:

	```text
	support(candidate) =
	entailment_logit - contradiction_logit
	```

	Candidate supports are normalized with a softmax.

	For robust Qwen3.5/PEFT compatibility, candidates are evaluated sequentially with `batch_size=1`.

	---

	## Repository files

	A typical BioJev-Nano model repository contains:

	```text
	BioJev-Nano/
	├── adapter_config.json
	├── adapter_model.safetensors
	├── tokenizer.json
	├── tokenizer_config.json
	├── chat_template.jinja # if present in the training checkpoint
	├── training_manifest.json # if present
	├── README.md
	├── example_inference.py
	└── requirements.txt
	```

	The PEFT adapter does not duplicate the Qwen3.5-0.8B base weights. The base model is downloaded separately when BioJev-Nano is loaded.

	---

	## Scientific status

	BioJev is a research project.

	The model, its confidence estimates, and the System One compatibility layer are not validated clinical decision systems. They must not be used as the sole basis for diagnosis, treatment, or other high-stakes medical decisions.

	---

	## Citation

	A formal paper citation will be added when the BioJev publication is available.

	Until then, please cite the BioJev repository and the specific Hugging Face checkpoint used.

	---

	## Acknowledgements

	BioJev-Nano was created by Gabriel Henrique Alencar Medeiros under the supervision of Lina F. Soualmia at LITIS / Université de Rouen Normandie.

	BioJev builds on the Qwen3.5 model family and publicly available biomedical and natural-language-inference datasets.