File size: 5,995 Bytes
be970ff
1d5888e
be970ff
5a2885e
 
be970ff
 
 
5a2885e
be970ff
 
 
 
1d5888e
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
 
 
 
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
 
 
 
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
5a2885e
be970ff
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188

---
license: apache-2.0
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
library_name: peft
pipeline_tag: text-classification
language:
  - en
tags:
  - peft
  - lora
  - text-classification
  - travel
---

# INFO 642 - Travel Request Classifier (TinyLlama + LoRA)

## What the Model Does

This model is a seven-category text classifier developed for our INFO 642 Modular AI Travel Planner project.

The model identifies the user's travel request and helps the n8n workflow select the appropriate module.

It predicts one of seven labels:

1. NEW_TRIP - Create a new travel itinerary.
2. CHANGE_FLIGHT - Modify an existing flight.
3. CHANGE_HOTEL - Modify an existing hotel.
4. CHANGE_ATTRACTION - Modify an attraction or activity.
5. CHANGE_RESTAURANT - Modify a restaurant or meal.
6. MULTI_CHANGE - Modify multiple itinerary components.
7. OTHER_OR_UNCLEAR - Handle unclear, general, or unsupported requests.

In our workflow, this model is intended to replace the original Groq-based placeholder classifier. The other modules handle itinerary generation, data lookup, budget calculations, and itinerary modifications.

The classifier does not make reservations or payments.

## Base Model

- Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
- Model type: Seven-class text classification
- Fine-tuning method: Low-Rank Adaptation (LoRA)
- Framework: Hugging Face Transformers and PEFT

We followed the Lesson 16 TinyLlama training approach and added a new seven-class classification head.

LoRA allows us to train a small number of parameters instead of updating the entire base model.

Training configuration:

| Parameter | Value |
|---|---|
| LoRA rank | 8 |
| LoRA alpha | 16 |
| LoRA dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| Epochs | 3 |
| Batch size | 8 |
| Learning rate | 0.0002 |
| Trainable parameters | 2,267,136 |

Only approximately 0.22% of the model's parameters were trainable.

## Training Data

We created a synthetic English dataset of 280 travel-related user requests for this course project.

The dataset includes seven balanced categories, with 40 examples per category.

Data split:

| Dataset | Samples |
|---|---:|
| Training set | 224 |
| Held-out test set | 56 |
| Total | 280 |

Each category contains 32 training examples and 8 held-out test examples.

The training data covers requests such as creating trips, changing flights, finding cheaper hotels, replacing attractions, modifying restaurants, and changing multiple itinerary components.

Some examples also include unclear requests and unsupported booking or payment requests.

The test data was not used for model training.

All examples are synthetic and do not represent real customer conversations.

## Results

We evaluated the fine-tuned model using 56 held-out test requests.

### Overall Accuracy

**Held-out test accuracy: 98.2% (55/56 correct)**

Additional model comparisons:

| Model | Accuracy |
|---|---:|
| BART Zero-shot | 46.4% |
| TinyLlama with untrained classification head | 14.3% |
| TinyLlama + LoRA | 98.2% |

The fine-tuned model performed substantially better on our synthetic test set.

### Training Loss

| Epoch | Loss |
|---|---:|
| 1 | 1.944 |
| 2 | 0.461 |
| 3 | 0.091 |

The training loss decreased across all three epochs.

### Confusion Matrix Analysis

The fine-tuned model correctly classified 55 out of 56 test examples.

Six categories achieved perfect recall.

The only classification error occurred when a MULTI_CHANGE request was predicted as CHANGE_ATTRACTION.

The relevant category metrics were:

| Category | Precision | Recall | F1-score |
|---|---:|---:|---:|
| CHANGE_ATTRACTION | 0.89 | 1.00 | 0.94 |
| MULTI_CHANGE | 1.00 | 0.88 | 0.93 |

The macro-average F1-score was 0.98.

This indicates that requests involving multiple changes may sometimes be confused with requests involving only one attraction change.

The complete confusion matrix and classification report are included in the training notebook.

## Limitations

Although the model achieved high accuracy on the held-out dataset, several limitations remain.

1. **Synthetic training data:** The dataset contains only 280 synthetic examples. Real user requests may be more complex or expressed differently.

2. **Multi-change confusion:** Requests involving multiple itinerary components may sometimes be classified as a single-category change.

3. **Ambiguous language:** Short or unclear requests may not provide enough information for reliable classification.

4. **Limited task coverage:** The model only supports the seven predefined categories. It cannot independently create itineraries, verify live availability, make reservations, or process payments.

5. **Generalization:** The 98.2% accuracy was measured on a small synthetic held-out dataset. Performance on real travel conversations has not been established.

The model is intended for educational workflow routing and should not be used as an independent booking or payment system.

## Fresh-Session Model Verification

After training, we uploaded the LoRA adapter, classification head, tokenizer, and label configuration to this public Hugging Face repository.

We then opened a fresh Google Colab notebook and loaded the model directly from Hugging Face.

Test input:

> Please update the afternoon sightseeing activity to the science center.

Expected prediction from the original training notebook:

`CHANGE_ATTRACTION`

Prediction after loading from Hugging Face:

`CHANGE_ATTRACTION`

**Result: PASS - The fresh-session prediction matched the original notebook.**

This verifies that the uploaded model can be loaded in a separate session and reproduce the prediction for this test example.

### Fresh-Session Test Screenshot

![Fresh-session model loading and prediction result](fresh_session_prediction.png)

## Project Information

Course: INFO 642

Project: Modular AI Travel Planner

Model repository: Shell2334/info642-travel-classifier-lora

Purpose: Educational research and demonstration.