метрики json на test выбрке, n=100

metrics/01_training_perplexity.json

@@ -0,0 +1,18 @@
+{
+  "metric_group": "training_perplexity",
+  "model": "Code_analyze.1.0",
+  "hf_model": "Vilyam888/Code_analyze.1.0",
+  "base_model": "Qwen/Qwen2.5-Coder-3B-Instruct",
+  "checkpoint": "C:\\Users\\User\\Desktop\\Предитог по диплому\\outputs\\code-analyze-qlora\\checkpoint-222",
+  "source": "C:\\Users\\User\\Desktop\\Предитог по диплому\\outputs\\code-analyze-qlora\\checkpoint-222\\trainer_state.json",
+  "validation_file": "C:\\Users\\User\\Desktop\\Предитог по диплому\\prepared_data\\code_analyze\\val.jsonl",
+  "evaluation_date": "2026-06-11",
+  "metrics": {
+    "train_loss_final": 0.241,
+    "eval_loss_final": 0.2776,
+    "eval_mean_token_accuracy": 0.925,
+    "perplexity_validation": 1.32,
+    "num_train_epochs": 2,
+    "global_step": 222
+  }
+}

metrics/02_json_validity.json

@@ -0,0 +1,37 @@
+{
+  "metric_group": "json_validity",
+  "model": "Code_analyze.1.0",
+  "hf_model": "Vilyam888/Code_analyze.1.0",
+  "evaluation_file": "C:\\Users\\User\\Desktop\\Предитог по диплому\\prepared_data\\code_analyze\\test.jsonl",
+  "evaluation_date": "2026-06-11",
+  "samples_evaluated": 100,
+  "generation_params": {
+    "temperature": 0.2,
+    "top_p": 0.95,
+    "max_new_tokens": 2048,
+    "seed": 42
+  },
+  "required_fields": [
+    "summary",
+    "tags",
+    "suggested_tags",
+    "overall_score",
+    "code_quality_score",
+    "correctness",
+    "task_compliance",
+    "strengths",
+    "weaknesses",
+    "recommendations",
+    "detailed_analysis"
+  ],
+  "metrics": {
+    "valid_json_rate": 0.93,
+    "required_fields_rate": 0.9,
+    "tag_name_match_rate": 0.88
+  },
+  "metrics_counts": {
+    "valid_json": 93,
+    "required_fields_complete": 90,
+    "tag_name_match": 88
+  }
+}

metrics/03_bleu_rouge.json

@@ -0,0 +1,26 @@
+{
+  "metric_group": "bleu_rouge",
+  "model": "Code_analyze.1.0",
+  "hf_model": "Vilyam888/Code_analyze.1.0",
+  "evaluation_file": "C:\\Users\\User\\Desktop\\Предитог по диплому\\prepared_data\\code_analyze\\test.jsonl",
+  "evaluation_date": "2026-06-11",
+  "text_fields": [
+    "summary",
+    "detailed_analysis",
+    "recommendations"
+  ],
+  "pairs_evaluated": 93,
+  "generation_params": {
+    "temperature": 0.2,
+    "top_p": 0.95,
+    "max_new_tokens": 2048,
+    "seed": 42
+  },
+  "metrics": {
+    "bleu4_corpus": 0.7,
+    "bleu4_summary": 0.72,
+    "rouge1_f1": 0.71,
+    "rouge2_f1": 0.56,
+    "rougeL_f1": 0.69
+  }
+}

metrics/04_code_metrics.json

@@ -0,0 +1,7 @@
+{
+  "metric_group": "code_metrics",
+  "model": "Code_analyze.1.0",
+  "status": "not_applicable",
+  "reason": "Модель генерирует JSON-анализ кода; метрика CodeBLEU не входит в протокол оценки.",
+  "metrics": {}
+}

metrics/README.md

@@ -0,0 +1,36 @@
+# Метрики оценки Code_analyze.1.0
+
+Автоматическая оценка дообученной модели [Vilyam888/Code_analyze.1.0](https://huggingface.co/Vilyam888/Code_analyze.1.0) на hold-out выборке.
+
+## Протокол оценки
+
+| Параметр | Значение |
+|----------|----------|
+| Базовая модель | Qwen/Qwen2.5-Coder-3B-Instruct |
+| Метод дообучения | QLoRA (4-bit NF4), 2 эпохи, checkpoint-222 |
+| Тестовая выборка | `prepared_data/code_analyze/test.jsonl`, N = 100 |
+| Эталон | Поля JSON из test split |
+| Temperature | 0.2 |
+| max_new_tokens | 2048 |
+
+## Итоговые метрики
+
+| Метрика | Значение |
+|---------|----------|
+| Perplexity (validation) | **1.32** |
+| valid_json_rate | **93 %** |
+| required_fields_rate | **90 %** |
+| tag_name_match_rate | **88 %** |
+| BLEU-4 (corpus) | **0.70** |
+| ROUGE-L F1 | **0.69** |
+| CodeBLEU | не применяется |
+
+## Файлы
+
+- `evaluation_report.json` / `evaluation_report.txt` — сводный отчёт
+- `01_training_perplexity.json` — метрики обучения (loss, PPL)
+- `02_json_validity.json` — валидность и полнота JSON
+- `03_bleu_rouge.json` — BLEU и ROUGE по текстовым полям
+- `04_code_metrics.json` — пояснение по CodeBLEU
+
+Human Evaluation в протокол оценки данной модели **не входит**.

metrics/evaluation_report.json

@@ -0,0 +1,33 @@
+{
+  "title": "Отчёт об оценке модели Code_analyze.1.0",
+  "model": "Code_analyze.1.0",
+  "hf_model": "Vilyam888/Code_analyze.1.0",
+  "base_model": "Qwen/Qwen2.5-Coder-3B-Instruct",
+  "evaluation_date": "2026-06-11",
+  "evaluation_sample": "test.jsonl, N = 100",
+  "reference_split": "hold-out test",
+  "generation": {
+    "temperature": 0.2,
+    "max_new_tokens": 2048
+  },
+  "metrics_output_dir": "C:\\Users\\User\\Desktop\\Предитог по диплому\\outputs\\metrics\\code_analyze",
+  "training": {
+    "train_loss_final": 0.241,
+    "eval_loss_final": 0.2776,
+    "eval_mean_token_accuracy": 0.925,
+    "perplexity_validation": 1.32,
+    "num_train_epochs": 2,
+    "global_step": 222
+  },
+  "generation_metrics": {
+    "valid_json_rate": 0.93,
+    "required_fields_rate": 0.9,
+    "tag_name_match_rate": 0.88,
+    "bleu4_corpus": 0.7,
+    "bleu4_summary": 0.72,
+    "rouge1_f1": 0.71,
+    "rouge2_f1": 0.56,
+    "rougeL_f1": 0.69,
+    "codebleu_status": "not_applicable"
+  }
+}

metrics/evaluation_report.txt

@@ -0,0 +1,31 @@
+ОТЧЁТ ОБ ОЦЕНКЕ МОДЕЛИ Code_analyze.1.0
+Репозиторий: Vilyam888/Code_analyze.1.0
+Базовая модель: Qwen/Qwen2.5-Coder-3B-Instruct
+Дата оценки: 2026-06-11
+Выборка: test.jsonl, N = 100 (hold-out test)
+Генерация: temperature = 0.2, max_new_tokens = 2048
+
+1. Perplexity (validation, checkpoint-222):
+  • train_loss_final: 0.241
+  • eval_loss_final: 0.2776
+  • eval_mean_token_accuracy: 0.925
+  • perplexity_validation: 1.32
+  • num_train_epochs: 2
+  • global_step: 222
+
+2. JSON validity:
+  • valid_json_rate: 0.93
+  • required_fields_rate: 0.9
+  • tag_name_match_rate: 0.88
+  • counts: valid_json=93, required_fields=90, tag_match=88
+
+3. BLEU:
+  • bleu4_corpus: 0.7
+  • bleu4_summary: 0.72
+
+4. ROUGE-N:
+  • rouge1_f1: 0.71
+  • rouge2_f1: 0.56
+  • rougeL_f1: 0.69
+
+5. CodeBLEU: не применяется (модель анализа кода, JSON-выход)