| # ReCon: Contrastive Explanation Model for OWL Reasoning (LoRA) |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| ReCon is a LoRA fine-tuned language model based on **meta-llama/Llama-3.1-8B-Instruct**, designed to generate **contrastive natural language explanations** for OWL reasoning tasks. |
|
|
| The model translates structured outputs from a Description Logic (DL) reasoner into human-readable explanations that answer: |
|
|
| > *“Why does fact A satisfy a class expression while foil B does not?”* |
|
|
| Unlike standard explanations, ReCon focuses on **contrastive reasoning**, highlighting: |
|
|
| * **Common evidence** (shared properties) |
| * **Differences** (missing or distinguishing properties) |
| * **Conflicts** (inconsistencies) |
|
|
| This work is part of a system combining: |
|
|
| * Symbolic reasoning (OWL reasoner) |
| * Natural language explanation (LLM) |
| * Graph-based visualization |
|
|
|
|
|
|
| --- |
|
|
| ### Key Contributions |
|
|
| * Converts **formal OWL reasoning outputs (JSON)** into natural language |
| * Reduces hallucination via **strict evidence-grounded prompting** |
| * Supports **contrastive explanations (fact vs foil)** |
| * Integrated into: |
|
|
| * Flask-based UI |
|
|
| --- |
|
|
| ### Model Information |
|
|
| * **Developed by:** Akash, Ashik and Sam (DICE Group, Paderborn University) |
| * **Model type:** Causal Language Model (LoRA adapted) |
| * **Base model:** meta-llama/Llama-3.1-8B-Instruct |
| * **Language(s):** English (primary), German (supported) |
| * **License:** Same as base model (Llama 3.1 license) |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| ### Direct Use |
|
|
| * OWL reasoning explanation generation |
| * Contrastive reasoning (fact vs foil) |
| * Explainable AI (XAI) workflows |
| * Knowledge graph interpretation |
|
|
| ### Downstream Use |
|
|
| * Integration with reasoning pipelines |
| * Explainability layer for symbolic + neural systems |
| * Educational tools for ontology reasoning |
|
|
| ### Out-of-Scope Use |
|
|
| * General chatbot usage |
| * Open-ended reasoning without structured input |
| * Tasks requiring external world knowledge |
|
|
| --- |
|
|
| ## How to Use |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| base_model = "meta-llama/Llama-3.1-8B-Instruct" |
| |
| tokenizer = AutoTokenizer.from_pretrained(base_model) |
| base = AutoModelForCausalLM.from_pretrained(base_model) |
| |
| model = PeftModel.from_pretrained(base, "vanishingMonk/Contrastive_Explainer") |
| ``` |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Training Objective |
|
|
| The model is fine-tuned to generate **faithful, structured explanations grounded strictly in reasoning evidence**, rather than free-form generation. |
|
|
| --- |
|
|
| ### Training Data |
|
|
| Training data was constructed from OWL reasoning tasks using datasets such as: |
|
|
| * Family ontology |
| * Carcinogenesis |
| * Lymphography |
|
|
| Each sample includes: |
|
|
| * Class expression |
| * Fact–foil pair |
| * Structured reasoning output: |
|
|
| * `common` |
| * `different` |
| * `conflicts` |
|
|
| Both formats were used: |
|
|
| * Instruction-based samples |
| * Multi-turn chat-based samples |
|
|
|
|
|
|
| --- |
|
|
| ### Prompt Engineering |
|
|
| A strict **three-layer prompt structure** was used: |
|
|
| 1. **System Prompt** |
|
|
| * Enforces evidence-only reasoning |
| * Controls output format |
| * Language constraints |
|
|
| 2. **Role Prompt** |
|
|
| * Defines model as: |
|
|
| > "Formal logician + OWL reasoner debugger" |
| |
| 3. **Context Prompt** |
|
|
| * Defines semantics of: |
|
|
| * common / different / conflicts |
| * Enforces exact reasoning patterns |
| * Fixes verdict: |
|
|
| * FACT = satisfies |
| * FOIL = fails |
|
|
| This significantly reduces hallucinations and improves logical consistency. |
|
|
|
|
|
|
| --- |
|
|
| ### Training Setup |
|
|
| * **Method:** LoRA (PEFT) |
| * **Rank (r):** 8 |
| * **Alpha:** 16 |
| * **Target modules:** |
|
|
| * q_proj, v_proj, o_proj |
| * gate_proj, up_proj, down_proj |
| * **Precision:** bf16 |
| * **Framework:** Torchtune |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| ### General Reasoning Benchmarks |
|
|
| | Benchmark | Metric | Fine-tuned | Base | |
| | -------------- | -------- | ---------- | ----- | |
| | ARC-Challenge | Accuracy | 0.544 | 0.553 | |
| | HellaSwag (EN) | Accuracy | 0.790 | 0.796 | |
| | HellaSwag (DE) | Accuracy | 0.615 | 0.615 | |
| | TruthfulQA | Accuracy | 0.533 | 0.545 | |
|
|
| → Minimal degradation in general reasoning performance |
|
|
| --- |
|
|
| ### Explanation Quality |
|
|
| | Metric | Fine-tuned | Base | |
| | --------- | ---------- | ------ | |
| | ROUGE-1 | 0.4066 | 0.5845 | |
| | ROUGE-2 | 0.1424 | 0.3443 | |
| | ROUGE-L | 0.2981 | 0.4013 | |
| | BERTScore | 0.8475 | 0.8705 | |
|
|
| **Important:** |
| Lower ROUGE/BERTScore reflects **stylistic differences**, not worse reasoning. |
| Human evaluation shows **higher logical fidelity and stability**. |
|
|
|
|
|
|
| --- |
|
|
| ## System Integration |
|
|
| ReCon is part of a full pipeline: |
|
|
| 1. OWL reasoner generates contrastive explanation (JSON) |
| 2. ReCon converts it into natural language |
| 3. Graphviz generates a visual explanation |
| 4. UI presents: |
|
|
| * Explanation |
| * Graph |
| * Chat interface |
|
|
| Also integrated into: |
|
|
| * Flask web interface |
| * Protégé plugin |
|
|
|
|
|
|
| --- |
|
|
| ## Limitations |
|
|
| * Requires structured reasoning input |
| * Cannot perform standalone logical inference |
| * Sensitive to prompt format |
| * May hallucinate if constraints are relaxed |
|
|
| --- |
|
|
| ## Future Work |
|
|
| * Scaling to larger ontologies |
| * Support for more DL constructs |
|
|
| --- |
|
|
| ## Citation |
|
|
| This model is part of an academic project at Paderborn University. |
| If you use this model, please reference this repository. |
|
|
|
|
| --- |
|
|
| ## Contact |
|
|
| For questions, collaborations, or research discussions, feel free to reach out to Akash. Email: akbaum@mail.uni-paderborn.de |
|
|
