Seth Claude Sonnet 5 commited on
Commit
0bbe01f
·
1 Parent(s): f928cd6

Chat Agent: full Agent Skills library (SKILL.md anatomy)

Browse files

Rebuilds the flat guardrail-string Skills feature into Anthropic's real
Agent Skills format: name + description + instructions (always injected
into the prompt), plus references/assets (indexed into the same
knowledge base the agentic search tool already searches), scripts
(stored/exported for documentation only — never executed, see
SKILLS-SECURITY.md), and evals (test cases graded by substring match or
an LLM judge). Skills export/import as real, portable SKILL.md files.

Also fixes a resilience gap found while testing live in the browser: an
indexing failure (e.g. the embeddings API being down) was crashing the
whole skill save with a 500 and losing the admin's text edits. The
skill's own fields now commit first, indexing runs in its own
try/except, and a failed index attempt rolls back cleanly so the next
save retries instead of getting stuck thinking it's already indexed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

SKILLS-SECURITY.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Chat Agent Skills — security notes
2
+
3
+ The Chat Agent's Skills feature follows Anthropic's Agent Skills anatomy: each skill is a
4
+ SKILL.md-format file with `name` + `description` + `instructions` (the "brain," always injected
5
+ into the agent's system prompt), plus optional `references/` (domain knowledge), `assets/`
6
+ (output templates), `scripts/` (mechanical helpers), and `evals/` (test cases). This document
7
+ covers the two things an admin authoring skills should understand before relying on them:
8
+ **scripts are never executed**, and **references/assets are visitor-searchable**.
9
+
10
+ ## Scripts are documentation only — never executed
11
+
12
+ You can author, store, export, and import script content in a skill's Scripts section. **The
13
+ agent never runs this code, under any circumstances.** This is a deliberate design decision, not
14
+ a missing feature, for the following reasons:
15
+
16
+ - **Shared blast radius.** The Chat Agent runs in the same process and container as the rest of
17
+ this CRM — the same SQLite database with real customer/lead data, the same `OPENAI_API_KEY`,
18
+ `GOOGLE_CLIENT_SECRET`, and session-signing secret. There is no separation between "the chat
19
+ agent's sandbox" and "everything else this app can touch."
20
+ - **No execution isolation on this hosting.** A safe code-execution feature needs an isolated,
21
+ ephemeral, no-network/no-filesystem/no-secret-access runtime per invocation. Standard Hugging
22
+ Face Spaces hosting (or a single Docker container generally) doesn't provide that — there's no
23
+ Docker-in-Docker or on-demand sandboxed compute available to spin up per script run.
24
+ - **Prompt injection changes the threat model.** The chat agent is driven by messages from
25
+ anonymous website visitors. Even if every script *body* is admin-authored and reviewed, a
26
+ visitor's message can influence what the agent decides to do with a tool at runtime. A
27
+ developer-supervised local coding agent choosing to run a script it just wrote is a different
28
+ threat model from a public-facing chat endpoint where an untrusted third party is one of the
29
+ inputs steering the conversation. Live script execution reachable from that surface is not a
30
+ risk this app takes on.
31
+ - **A README can't retroactively secure an unsafe architecture.** If real script execution is
32
+ ever wanted, it needs to be built as its own isolated execution service (ephemeral sandbox,
33
+ no ambient secrets, strict output size/time limits, admin-only trigger) — not bolted onto the
34
+ existing shared process.
35
+
36
+ **If you need the agent to actually *do* something** (call an API, run a calculation, look
37
+ something up), the safe path already exists: it's the same pattern as the built-in
38
+ `search_knowledge_base` tool. Ask for a new, narrow, purpose-built tool with its own explicit
39
+ inputs/outputs — reviewed and wired in like any other backend change — rather than arbitrary
40
+ script execution.
41
+
42
+ ## References and assets are visitor-searchable
43
+
44
+ Unlike scripts, a skill's **references** and **assets** *are* indexed into the same knowledge
45
+ base used for your Google Drive and website content, so the agentic `search_knowledge_base` tool
46
+ can retrieve and quote them for a visitor when relevant. Treat anything you put in a
47
+ reference/asset with the same review standard you'd apply to a page on your public website or a
48
+ document you've shared to the Drive folder the agent indexes — don't put anything there you
49
+ wouldn't want a visitor to potentially see quoted back to them.
50
+
51
+ ## Summary
52
+
53
+ | Skill section | Injected into every reply? | Visitor-searchable via the KB? | Executed? |
54
+ |-----------------|:---------------------------:|:-------------------------------:|:--------------:|
55
+ | Instructions | Yes, always | No | N/A (a prompt) |
56
+ | References | No | Yes | N/A |
57
+ | Assets | No | Yes | N/A |
58
+ | Scripts | No | No | **Never** |
backend/app/chat_agent_chat.py CHANGED
@@ -75,6 +75,16 @@ SEARCH_TOOL_SCHEMA = {
75
  _client: Optional[OpenAI] = None
76
 
77
 
 
 
 
 
 
 
 
 
 
 
78
  def _openai_client() -> OpenAI:
79
  global _client
80
  if _client is None:
@@ -173,13 +183,14 @@ def build_system_prompt(
173
  website_urls_seen: List[str] = []
174
  for i, c in enumerate(retrieved_chunks, start=1):
175
  title = c.get("title") or c.get("source_ref")
176
- if c.get("source_type") == "website":
 
177
  url = c.get("source_ref") or ""
178
- parts.append(f"[{i}] Website page \"{title}\" ({url})\n{c.get('text', '')[:1400]}")
179
  if url and url not in website_urls_seen:
180
  website_urls_seen.append(url)
181
  else:
182
- parts.append(f"[{i}] Internal document \"{title}\" (no public URL)\n{c.get('text', '')[:1400]}")
183
 
184
  if website_urls_seen:
185
  parts += [
@@ -282,14 +293,71 @@ def _format_tool_results(chunks: List[Dict[str, Any]]) -> str:
282
  parts: List[str] = []
283
  for i, c in enumerate(chunks, start=1):
284
  title = c.get("title") or c.get("source_ref")
285
- if c.get("source_type") == "website":
286
- label = f'Website page "{title}" ({c.get("source_ref")})'
 
287
  else:
288
- label = f'Internal document "{title}" (no public URL)'
289
  parts.append(f"[{i}] {label}\n{(c.get('text') or '')[:1400]}")
290
  return "\n\n".join(parts)
291
 
292
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
293
  def stream_agentic_reply_openai(
294
  system_prompt: str,
295
  history: List[Dict[str, str]],
 
75
  _client: Optional[OpenAI] = None
76
 
77
 
78
+ def _source_label(source_type: Optional[str], title: str) -> str:
79
+ if source_type == "website":
80
+ return f'Website page "{title}"'
81
+ if source_type == "skill_reference":
82
+ return f'Skill reference "{title}"'
83
+ if source_type == "skill_asset":
84
+ return f'Skill asset/template "{title}"'
85
+ return f'Internal document "{title}"'
86
+
87
+
88
  def _openai_client() -> OpenAI:
89
  global _client
90
  if _client is None:
 
183
  website_urls_seen: List[str] = []
184
  for i, c in enumerate(retrieved_chunks, start=1):
185
  title = c.get("title") or c.get("source_ref")
186
+ source_type = c.get("source_type")
187
+ if source_type == "website":
188
  url = c.get("source_ref") or ""
189
+ parts.append(f"[{i}] {_source_label(source_type, title)} ({url})\n{c.get('text', '')[:1400]}")
190
  if url and url not in website_urls_seen:
191
  website_urls_seen.append(url)
192
  else:
193
+ parts.append(f"[{i}] {_source_label(source_type, title)} (no public URL)\n{c.get('text', '')[:1400]}")
194
 
195
  if website_urls_seen:
196
  parts += [
 
293
  parts: List[str] = []
294
  for i, c in enumerate(chunks, start=1):
295
  title = c.get("title") or c.get("source_ref")
296
+ source_type = c.get("source_type")
297
+ if source_type == "website":
298
+ label = f'{_source_label(source_type, title)} ({c.get("source_ref")})'
299
  else:
300
+ label = f'{_source_label(source_type, title)} (no public URL)'
301
  parts.append(f"[{i}] {label}\n{(c.get('text') or '')[:1400]}")
302
  return "\n\n".join(parts)
303
 
304
 
305
+ def llm_judge(prompt: str, response: str, assertion_value: str) -> Dict[str, str]:
306
+ """Cheap LLM-graded eval assertion: does `response` satisfy the natural-language rubric in
307
+ `assertion_value`, given the original `prompt`? Same lightweight-call pattern as
308
+ extract_visitor_name above — not part of the live chat path, only used by the admin 'Run
309
+ evals' action on a skill.
310
+ """
311
+ try:
312
+ resp = _openai_client().chat.completions.create(
313
+ model=CHAT_MODEL_OPENAI,
314
+ messages=[
315
+ {
316
+ "role": "system",
317
+ "content": (
318
+ "You are grading whether a chat assistant's response satisfies a rubric. "
319
+ "Reply with exactly PASS or FAIL on the first line, then one short "
320
+ "sentence explaining why."
321
+ ),
322
+ },
323
+ {
324
+ "role": "user",
325
+ "content": (
326
+ f"Visitor asked: {prompt}\n\nAssistant replied: {response}\n\n"
327
+ f"Rubric: {assertion_value}"
328
+ ),
329
+ },
330
+ ],
331
+ temperature=0,
332
+ max_tokens=100,
333
+ )
334
+ text = (resp.choices[0].message.content or "").strip()
335
+ first_line, _, rest = text.partition("\n")
336
+ passed = first_line.strip().upper().startswith("PASS")
337
+ detail = rest.strip() or first_line.strip()
338
+ return {"status": "pass" if passed else "fail", "detail": detail[:500]}
339
+ except Exception as e:
340
+ logger.warning("llm_judge grading failed: %s", e)
341
+ return {"status": "error", "detail": str(e)[:500]}
342
+
343
+
344
+ def run_eval_assertion(prompt: str, response: str, assertion_type: str, assertion_value: str) -> Dict[str, str]:
345
+ """Grade one eval case's response against its assertion. must_contain/must_not_contain are
346
+ plain case-insensitive substring checks; llm_judge delegates to the grading call above."""
347
+ at = (assertion_type or "").strip()
348
+ av = assertion_value or ""
349
+ resp_lower = (response or "").lower()
350
+ if at == "must_contain":
351
+ passed = av.lower() in resp_lower
352
+ return {"status": "pass" if passed else "fail", "detail": f'Looked for "{av}" in the response.'}
353
+ if at == "must_not_contain":
354
+ passed = av.lower() not in resp_lower
355
+ return {"status": "pass" if passed else "fail", "detail": f'Checked the response did not contain "{av}".'}
356
+ if at == "llm_judge":
357
+ return llm_judge(prompt, response, av)
358
+ return {"status": "error", "detail": f"Unknown assertion_type: {at}"}
359
+
360
+
361
  def stream_agentic_reply_openai(
362
  system_prompt: str,
363
  history: List[Dict[str, str]],
backend/app/chat_agent_common.py CHANGED
@@ -4,6 +4,7 @@ ingestion modules, the chat engine, and the routes module — kept separate to a
4
 
5
  from __future__ import annotations
6
 
 
7
  import uuid
8
  from typing import Any, List, Optional
9
 
@@ -12,10 +13,22 @@ from sqlalchemy.orm import Session
12
 
13
  from .database import ChatAgentConfig, ChatAgentSkill
14
 
15
- DEFAULT_SKILLS: List[str] = [
16
- "Don't share internal pricing details unless they are already published on the website.",
17
- "Don't share existing client/customer names or details unless the website already has a public case study naming that client.",
18
- "Never reveal sensitive or private information (personal data, credentials, internal-only documents) to website visitors — always err on the side of caution.",
 
 
 
 
 
 
 
 
 
 
 
 
19
  ]
20
 
21
 
@@ -51,11 +64,14 @@ def ensure_default_skills(db: Session, tenant_id: int) -> None:
51
  has_any = db.query(ChatAgentSkill).filter(ChatAgentSkill.tenant_id == tenant_id).first()
52
  if has_any:
53
  return
54
- for i, text in enumerate(DEFAULT_SKILLS):
55
  db.add(
56
  ChatAgentSkill(
57
  tenant_id=tenant_id,
58
- text=text,
 
 
 
59
  is_default=1,
60
  is_active=1,
61
  sort_order=i,
@@ -65,7 +81,9 @@ def ensure_default_skills(db: Session, tenant_id: int) -> None:
65
 
66
 
67
  def compile_skills_md(db: Session, tenant_id: int) -> str:
68
- """Compile active skills into a 'Skills.md'-style block for the chat system prompt."""
 
 
69
  rows = (
70
  db.query(ChatAgentSkill)
71
  .filter(ChatAgentSkill.tenant_id == tenant_id, ChatAgentSkill.is_active == 1)
@@ -76,10 +94,221 @@ def compile_skills_md(db: Session, tenant_id: int) -> str:
76
  return ""
77
  lines = ["# Skills.md — Agent behavior rules (non-negotiable)", ""]
78
  for r in rows:
79
- t = safe_str(r.text)
80
- if t:
81
- lines.append(f"- {t}")
82
- return "\n".join(lines)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
 
84
 
85
  def config_to_dict(row: Optional[ChatAgentConfig]) -> dict:
 
4
 
5
  from __future__ import annotations
6
 
7
+ import re
8
  import uuid
9
  from typing import Any, List, Optional
10
 
 
13
 
14
  from .database import ChatAgentConfig, ChatAgentSkill
15
 
16
+ DEFAULT_SKILLS: List[dict] = [
17
+ {
18
+ "name": "no-internal-pricing",
19
+ "description": "Keep pricing conversations limited to what's already public.",
20
+ "instructions": "Don't share internal pricing details unless they are already published on the website.",
21
+ },
22
+ {
23
+ "name": "no-client-names",
24
+ "description": "Protect customer confidentiality unless a public case study already names them.",
25
+ "instructions": "Don't share existing client/customer names or details unless the website already has a public case study naming that client.",
26
+ },
27
+ {
28
+ "name": "no-sensitive-info",
29
+ "description": "Refuse to reveal sensitive, private, or internal-only information to visitors.",
30
+ "instructions": "Never reveal sensitive or private information (personal data, credentials, internal-only documents) to website visitors — always err on the side of caution.",
31
+ },
32
  ]
33
 
34
 
 
64
  has_any = db.query(ChatAgentSkill).filter(ChatAgentSkill.tenant_id == tenant_id).first()
65
  if has_any:
66
  return
67
+ for i, skill in enumerate(DEFAULT_SKILLS):
68
  db.add(
69
  ChatAgentSkill(
70
  tenant_id=tenant_id,
71
+ text=skill["instructions"],
72
+ name=skill["name"],
73
+ description=skill["description"],
74
+ instructions=skill["instructions"],
75
  is_default=1,
76
  is_active=1,
77
  sort_order=i,
 
81
 
82
 
83
  def compile_skills_md(db: Session, tenant_id: int) -> str:
84
+ """Compile all active skills' instructions into a merged 'Skills.md'-style block that's
85
+ always injected into the live chat system prompt. Each skill also has its own standalone,
86
+ exportable SKILL.md rendered by compile_skill_md() below."""
87
  rows = (
88
  db.query(ChatAgentSkill)
89
  .filter(ChatAgentSkill.tenant_id == tenant_id, ChatAgentSkill.is_active == 1)
 
94
  return ""
95
  lines = ["# Skills.md — Agent behavior rules (non-negotiable)", ""]
96
  for r in rows:
97
+ instructions = safe_str(r.instructions) or safe_str(r.text)
98
+ if not instructions:
99
+ continue
100
+ name = safe_str(r.name)
101
+ if name:
102
+ lines.append(f"## {name}")
103
+ description = safe_str(r.description)
104
+ if description:
105
+ lines.append(f"_{description}_")
106
+ lines.append("")
107
+ lines.append(instructions)
108
+ lines.append("")
109
+ return "\n".join(lines).rstrip()
110
+
111
+
112
+ # ---- SKILL.md compile / parse (Anthropic Agent Skills anatomy) ----
113
+ #
114
+ # Each skill is stored as structured columns (name, description, instructions, references_json,
115
+ # assets_json, scripts_json, evals_json) but is always rendered to / parsed from a single portable
116
+ # SKILL.md text file — YAML frontmatter (name + description) followed by the markdown instructions
117
+ # ("the brain"), then optional ## References / ## Assets / ## Scripts / ## Evals sections. This
118
+ # lets an admin export a skill, hand-edit or share it, and import it back losslessly.
119
+
120
+ _FRONTMATTER_RE = re.compile(r"\A---\s*\n(.*?)\n---\s*\n?", re.DOTALL)
121
+ _H3_RE = re.compile(r"^### (.+?)\s*$", re.MULTILINE)
122
+ _SECTION_RE = re.compile(r"^## (References|Assets|Scripts|Evals)\s*$", re.MULTILINE)
123
+ _EVAL_ITEM_RE = re.compile(
124
+ r"^- \*\*Prompt:\*\* (.*?)\n\s*\*\*Assertion \(([^)]*)\):\*\* (.*?)(?=\n- \*\*Prompt|\Z)",
125
+ re.DOTALL,
126
+ )
127
+
128
+
129
+ def _frontmatter_value(block: str, key: str) -> str:
130
+ m = re.search(rf"^{key}:\s*(.*)$", block, re.MULTILINE)
131
+ if not m:
132
+ return ""
133
+ val = m.group(1).strip()
134
+ if len(val) >= 2 and val[0] == val[-1] and val[0] in ("'", '"'):
135
+ val = val[1:-1]
136
+ return val
137
+
138
+
139
+ def _strip_code_fence(text: str) -> str:
140
+ text = text.strip()
141
+ m = re.match(r"^```[a-zA-Z0-9_+-]*\n(.*?)\n```$", text, re.DOTALL)
142
+ if m:
143
+ return m.group(1)
144
+ return text
145
+
146
+
147
+ def compile_skill_md(skill: ChatAgentSkill) -> str:
148
+ """Render one skill as a standalone, portable SKILL.md file."""
149
+ name = safe_str(skill.name) or f"skill-{skill.id}"
150
+ description = safe_str(skill.description)
151
+ instructions = safe_str(skill.instructions) or safe_str(skill.text)
152
+
153
+ lines = ["---", f"name: {name}", f"description: {description}", "---", "", instructions, ""]
154
+
155
+ references = skill.references_json or []
156
+ if references:
157
+ lines.append("## References")
158
+ lines.append("")
159
+ for ref in references:
160
+ title = safe_str(ref.get("title")) if isinstance(ref, dict) else ""
161
+ content = safe_str(ref.get("content")) if isinstance(ref, dict) else ""
162
+ lines.append(f"### {title or 'Untitled reference'}")
163
+ lines.append("")
164
+ lines.append(content)
165
+ lines.append("")
166
+
167
+ assets = skill.assets_json or []
168
+ if assets:
169
+ lines.append("## Assets")
170
+ lines.append("")
171
+ for asset in assets:
172
+ aname = safe_str(asset.get("name")) if isinstance(asset, dict) else ""
173
+ content = safe_str(asset.get("content")) if isinstance(asset, dict) else ""
174
+ lines.append(f"### {aname or 'Untitled asset'}")
175
+ lines.append("")
176
+ lines.append("```")
177
+ lines.append(content)
178
+ lines.append("```")
179
+ lines.append("")
180
+
181
+ scripts = skill.scripts_json or []
182
+ if scripts:
183
+ lines.append("## Scripts")
184
+ lines.append("")
185
+ lines.append("_Stored for documentation and portability only — never executed by the agent._")
186
+ lines.append("")
187
+ for script in scripts:
188
+ sname = safe_str(script.get("name")) if isinstance(script, dict) else ""
189
+ slang = safe_str(script.get("language")) if isinstance(script, dict) else ""
190
+ sdesc = safe_str(script.get("description")) if isinstance(script, dict) else ""
191
+ scontent = safe_str(script.get("content")) if isinstance(script, dict) else ""
192
+ header = f"### {sname or 'Untitled script'}"
193
+ if slang:
194
+ header += f" ({slang})"
195
+ lines.append(header)
196
+ lines.append("")
197
+ if sdesc:
198
+ lines.append(sdesc)
199
+ lines.append("")
200
+ lines.append(f"```{slang}")
201
+ lines.append(scontent)
202
+ lines.append("```")
203
+ lines.append("")
204
+
205
+ evals = skill.evals_json or []
206
+ if evals:
207
+ lines.append("## Evals")
208
+ lines.append("")
209
+ for ev in evals:
210
+ prompt = safe_str(ev.get("prompt")) if isinstance(ev, dict) else ""
211
+ atype = safe_str(ev.get("assertion_type")) if isinstance(ev, dict) else ""
212
+ aval = safe_str(ev.get("assertion_value")) if isinstance(ev, dict) else ""
213
+ lines.append(f"- **Prompt:** {prompt}")
214
+ lines.append(f" **Assertion ({atype}):** {aval}")
215
+ lines.append("")
216
+
217
+ return "\n".join(lines).rstrip() + "\n"
218
+
219
+
220
+ def _parse_h3_items(section_body: str, key: str) -> list:
221
+ items = []
222
+ parts = _H3_RE.split(section_body)
223
+ for i in range(1, len(parts), 2):
224
+ title = parts[i].strip()
225
+ body = (parts[i + 1] if i + 1 < len(parts) else "").strip()
226
+ if key == "name":
227
+ body = _strip_code_fence(body)
228
+ items.append({key: title, "content": body})
229
+ return items
230
+
231
+
232
+ def _parse_scripts_section(section_body: str) -> list:
233
+ scripts = []
234
+ parts = _H3_RE.split(section_body)
235
+ for i in range(1, len(parts), 2):
236
+ header = parts[i].strip()
237
+ body = (parts[i + 1] if i + 1 < len(parts) else "").strip()
238
+ m = re.match(r"^(.*?)\s*\(([^)]+)\)\s*$", header)
239
+ sname, slang = (m.group(1).strip(), m.group(2).strip()) if m else (header, "")
240
+ fence_m = re.search(r"```([a-zA-Z0-9_+-]*)\n(.*?)\n```", body, re.DOTALL)
241
+ if fence_m:
242
+ slang = slang or fence_m.group(1)
243
+ scontent = fence_m.group(2)
244
+ sdesc = body[: fence_m.start()].strip()
245
+ else:
246
+ scontent = body
247
+ sdesc = ""
248
+ scripts.append({"name": sname, "description": sdesc, "language": slang, "content": scontent})
249
+ return scripts
250
+
251
+
252
+ def _parse_evals_section(section_body: str) -> list:
253
+ evals = []
254
+ for m in _EVAL_ITEM_RE.finditer(section_body.strip() + "\n"):
255
+ evals.append(
256
+ {
257
+ "prompt": m.group(1).strip(),
258
+ "assertion_type": m.group(2).strip(),
259
+ "assertion_value": m.group(3).strip(),
260
+ }
261
+ )
262
+ return evals
263
+
264
+
265
+ def parse_skill_md(content: str) -> dict:
266
+ """Parse a SKILL.md file back into a dict of fields, matching compile_skill_md()'s format.
267
+ Name/description/instructions parse reliably from any valid frontmatter; References/Assets/
268
+ Scripts/Evals sections parse best-effort when they match the exported layout — a hand-edited
269
+ file that omits or reformats a section simply yields an empty list for it, not an error.
270
+ """
271
+ content = (content or "").replace("\r\n", "\n").strip("\n")
272
+ name = ""
273
+ description = ""
274
+ body = content
275
+
276
+ fm_match = _FRONTMATTER_RE.match(content + "\n")
277
+ if fm_match:
278
+ block = fm_match.group(1)
279
+ name = _frontmatter_value(block, "name")
280
+ description = _frontmatter_value(block, "description")
281
+ body = content[fm_match.end():]
282
+
283
+ parts = _SECTION_RE.split(body)
284
+ instructions = parts[0].strip()
285
+
286
+ references: list = []
287
+ assets: list = []
288
+ scripts: list = []
289
+ evals: list = []
290
+
291
+ for i in range(1, len(parts), 2):
292
+ section_name = parts[i]
293
+ section_body = parts[i + 1] if i + 1 < len(parts) else ""
294
+ if section_name == "References":
295
+ references = _parse_h3_items(section_body, "title")
296
+ elif section_name == "Assets":
297
+ assets = _parse_h3_items(section_body, "name")
298
+ elif section_name == "Scripts":
299
+ scripts = _parse_scripts_section(section_body)
300
+ elif section_name == "Evals":
301
+ evals = _parse_evals_section(section_body)
302
+
303
+ return {
304
+ "name": name,
305
+ "description": description,
306
+ "instructions": instructions,
307
+ "references": references,
308
+ "assets": assets,
309
+ "scripts": scripts,
310
+ "evals": evals,
311
+ }
312
 
313
 
314
  def config_to_dict(row: Optional[ChatAgentConfig]) -> dict:
backend/app/chat_agent_ingest.py CHANGED
@@ -1,8 +1,10 @@
1
  """
2
  Shared ingestion pipeline for the Chat Agent knowledge base: markdown chunking, OpenAI embeddings,
3
  content-hash diffing (skip unchanged Drive files/pages on re-index), and the reindex_drive /
4
- reindex_website orchestration functions that write ChatAgentDocument/ChatAgentChunk rows and a
5
- ChatAgentIndexLog entry for admin visibility.
 
 
6
  """
7
 
8
  from __future__ import annotations
@@ -316,3 +318,88 @@ def reindex_website(tenant_id: int) -> Dict[str, Any]:
316
  return {"ok": False, "error": str(e)}
317
  finally:
318
  db.close()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  """
2
  Shared ingestion pipeline for the Chat Agent knowledge base: markdown chunking, OpenAI embeddings,
3
  content-hash diffing (skip unchanged Drive files/pages on re-index), and the reindex_drive /
4
+ reindex_website / reindex_skill_content orchestration functions that write ChatAgentDocument/
5
+ ChatAgentChunk rows. Drive/website reindexing runs as background jobs and logs to
6
+ ChatAgentIndexLog for admin visibility; skill reference/asset indexing runs inline on skill save
7
+ (small enough not to need a background job) reusing the same upsert/embed/prune machinery.
8
  """
9
 
10
  from __future__ import annotations
 
318
  return {"ok": False, "error": str(e)}
319
  finally:
320
  db.close()
321
+
322
+
323
+ def reindex_skill_content(db, tenant_id: int, skill) -> Dict[str, Any]:
324
+ """Index a skill's references/assets into the same searchable knowledge base used for
325
+ Drive/website content, so the agentic search_knowledge_base tool can surface them on demand.
326
+ Reuses the caller's db session (called inline from the skill save route, not a background
327
+ job) and the existing upsert/embed/prune machinery above.
328
+ """
329
+ processed = 0
330
+ seen_refs: set = set()
331
+
332
+ for idx, ref in enumerate(skill.references_json or []):
333
+ if not isinstance(ref, dict):
334
+ continue
335
+ content = (ref.get("content") or "").strip()
336
+ if not content:
337
+ continue
338
+ source_ref = f"skill:{skill.id}:reference:{idx}"
339
+ seen_refs.add(source_ref)
340
+ title = (ref.get("title") or f"Reference {idx + 1}").strip()
341
+ if _upsert_document_and_chunks(
342
+ db,
343
+ tenant_id=tenant_id,
344
+ source_type="skill_reference",
345
+ source_ref=source_ref,
346
+ title=f"{skill.name or skill.id}: {title}",
347
+ markdown=content,
348
+ ):
349
+ processed += 1
350
+
351
+ for idx, asset in enumerate(skill.assets_json or []):
352
+ if not isinstance(asset, dict):
353
+ continue
354
+ content = (asset.get("content") or "").strip()
355
+ if not content:
356
+ continue
357
+ source_ref = f"skill:{skill.id}:asset:{idx}"
358
+ seen_refs.add(source_ref)
359
+ title = (asset.get("name") or f"Asset {idx + 1}").strip()
360
+ if _upsert_document_and_chunks(
361
+ db,
362
+ tenant_id=tenant_id,
363
+ source_type="skill_asset",
364
+ source_ref=source_ref,
365
+ title=f"{skill.name or skill.id}: {title}",
366
+ markdown=content,
367
+ ):
368
+ processed += 1
369
+
370
+ # Scope the prune to THIS skill's source_ref prefix — a plain source_type filter would wipe
371
+ # every other skill's reference/asset content too, since they all share "skill_reference"/
372
+ # "skill_asset" as source_type.
373
+ stale = (
374
+ db.query(ChatAgentDocument)
375
+ .filter(
376
+ ChatAgentDocument.tenant_id == tenant_id,
377
+ ChatAgentDocument.source_type.in_(["skill_reference", "skill_asset"]),
378
+ ChatAgentDocument.source_ref.like(f"skill:{skill.id}:%"),
379
+ ~ChatAgentDocument.source_ref.in_(seen_refs),
380
+ )
381
+ .all()
382
+ )
383
+ for d in stale:
384
+ db.query(ChatAgentChunk).filter(ChatAgentChunk.document_id == d.id).delete()
385
+ db.delete(d)
386
+
387
+ return {"ok": True, "processed": processed, "pruned": len(stale)}
388
+
389
+
390
+ def delete_skill_content(db, tenant_id: int, skill_id: int) -> int:
391
+ """Remove all indexed reference/asset content for a skill. Called when the skill itself is
392
+ deleted, since ChatAgentDocument has no FK cascade to ChatAgentSkill."""
393
+ docs = (
394
+ db.query(ChatAgentDocument)
395
+ .filter(
396
+ ChatAgentDocument.tenant_id == tenant_id,
397
+ ChatAgentDocument.source_type.in_(["skill_reference", "skill_asset"]),
398
+ ChatAgentDocument.source_ref.like(f"skill:{skill_id}:%"),
399
+ )
400
+ .all()
401
+ )
402
+ for d in docs:
403
+ db.query(ChatAgentChunk).filter(ChatAgentChunk.document_id == d.id).delete()
404
+ db.delete(d)
405
+ return len(docs)
backend/app/chat_agent_routes.py CHANGED
@@ -18,17 +18,26 @@ from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request
18
  from fastapi.responses import JSONResponse, PlainTextResponse, Response, StreamingResponse
19
  from sqlalchemy import func
20
  from sqlalchemy.orm import Session
 
21
 
22
  from .chat_agent_chat import (
23
  build_agentic_system_prompt,
24
  build_system_prompt,
25
  extract_email,
26
  extract_visitor_name,
 
27
  stream_agentic_reply_openai,
28
  stream_reply_gemini,
29
  )
30
- from .chat_agent_common import compile_skills_md, config_to_dict, get_or_create_config, safe_str
31
- from .chat_agent_ingest import reindex_drive, reindex_website
 
 
 
 
 
 
 
32
  from .chat_agent_retrieval import retrieve_relevant_chunks
33
  from .crm_lead_contacts import link_lead_to_contact
34
  from .database import (
@@ -48,6 +57,7 @@ from .models import (
48
  ChatAgentReindexRequest,
49
  ChatAgentSessionRequest,
50
  ChatAgentSkillCreateRequest,
 
51
  ChatAgentSkillPatchRequest,
52
  )
53
  from .oauth_utils import public_request_base
@@ -85,7 +95,13 @@ def _parse_drive_folder_id(url_or_id: str) -> str:
85
  def _skill_to_dict(r: ChatAgentSkill) -> dict:
86
  return {
87
  "id": r.id,
88
- "text": r.text or "",
 
 
 
 
 
 
89
  "is_default": bool(r.is_default),
90
  "is_active": bool(r.is_active),
91
  "sort_order": r.sort_order,
@@ -105,6 +121,20 @@ def _conversation_to_dict(r: ChatAgentConversation) -> dict:
105
  }
106
 
107
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
  def _embed_snippet(origin: str, public_id: str) -> str:
109
  if not origin:
110
  return ""
@@ -175,9 +205,17 @@ def create_skill(body: ChatAgentSkillCreateRequest, tc: TenantContext = Depends(
175
  .scalar()
176
  or 0
177
  )
 
178
  row = ChatAgentSkill(
179
  tenant_id=tc.tenant_id,
180
- text=body.text.strip(),
 
 
 
 
 
 
 
181
  is_default=0,
182
  is_active=1,
183
  sort_order=max_order + 1,
@@ -185,6 +223,7 @@ def create_skill(body: ChatAgentSkillCreateRequest, tc: TenantContext = Depends(
185
  tc.db.add(row)
186
  db_commit_with_retry(tc.db)
187
  tc.db.refresh(row)
 
188
  return _skill_to_dict(row)
189
 
190
 
@@ -197,14 +236,32 @@ def patch_skill(skill_id: int, body: ChatAgentSkillPatchRequest, tc: TenantConte
197
  )
198
  if not row:
199
  raise HTTPException(status_code=404, detail="Skill not found")
200
- if body.text is not None:
201
- row.text = body.text.strip()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
  if body.is_active is not None:
203
  row.is_active = 1 if body.is_active else 0
204
  if body.sort_order is not None:
205
  row.sort_order = body.sort_order
206
  row.updated_at = datetime.utcnow()
207
  db_commit_with_retry(tc.db)
 
 
208
  return _skill_to_dict(row)
209
 
210
 
@@ -217,11 +274,115 @@ def delete_skill(skill_id: int, tc: TenantContext = Depends(require_tenant_admin
217
  )
218
  if not row:
219
  raise HTTPException(status_code=404, detail="Skill not found")
 
220
  tc.db.delete(row)
221
  db_commit_with_retry(tc.db)
222
  return {"ok": True}
223
 
224
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
225
  # ---- Admin: reindex + status ----
226
 
227
 
 
18
  from fastapi.responses import JSONResponse, PlainTextResponse, Response, StreamingResponse
19
  from sqlalchemy import func
20
  from sqlalchemy.orm import Session
21
+ from sqlalchemy.orm.attributes import flag_modified
22
 
23
  from .chat_agent_chat import (
24
  build_agentic_system_prompt,
25
  build_system_prompt,
26
  extract_email,
27
  extract_visitor_name,
28
+ run_eval_assertion,
29
  stream_agentic_reply_openai,
30
  stream_reply_gemini,
31
  )
32
+ from .chat_agent_common import (
33
+ compile_skill_md,
34
+ compile_skills_md,
35
+ config_to_dict,
36
+ get_or_create_config,
37
+ parse_skill_md,
38
+ safe_str,
39
+ )
40
+ from .chat_agent_ingest import delete_skill_content, reindex_drive, reindex_skill_content, reindex_website
41
  from .chat_agent_retrieval import retrieve_relevant_chunks
42
  from .crm_lead_contacts import link_lead_to_contact
43
  from .database import (
 
57
  ChatAgentReindexRequest,
58
  ChatAgentSessionRequest,
59
  ChatAgentSkillCreateRequest,
60
+ ChatAgentSkillImportRequest,
61
  ChatAgentSkillPatchRequest,
62
  )
63
  from .oauth_utils import public_request_base
 
95
  def _skill_to_dict(r: ChatAgentSkill) -> dict:
96
  return {
97
  "id": r.id,
98
+ "name": r.name or "",
99
+ "description": r.description or "",
100
+ "instructions": r.instructions or r.text or "",
101
+ "references": r.references_json or [],
102
+ "assets": r.assets_json or [],
103
+ "scripts": r.scripts_json or [],
104
+ "evals": r.evals_json or [],
105
  "is_default": bool(r.is_default),
106
  "is_active": bool(r.is_active),
107
  "sort_order": r.sort_order,
 
121
  }
122
 
123
 
124
+ def _safe_reindex_skill_content(db: Session, tenant_id: int, skill: ChatAgentSkill) -> None:
125
+ """Index a skill's references/assets, but never let an indexing failure (e.g. the embeddings
126
+ API being down) fail the save of the skill's own name/description/instructions — those are
127
+ committed by the caller before this runs. On failure, roll back just the partial
128
+ document/chunk writes so a later save retries cleanly instead of leaving a document marked
129
+ up-to-date with zero chunks."""
130
+ try:
131
+ reindex_skill_content(db, tenant_id, skill)
132
+ db_commit_with_retry(db)
133
+ except Exception as e:
134
+ logger.warning("Skill content reindex failed for skill %s: %s", skill.id, e)
135
+ db.rollback()
136
+
137
+
138
  def _embed_snippet(origin: str, public_id: str) -> str:
139
  if not origin:
140
  return ""
 
205
  .scalar()
206
  or 0
207
  )
208
+ instructions = body.instructions.strip()
209
  row = ChatAgentSkill(
210
  tenant_id=tc.tenant_id,
211
+ name=body.name.strip(),
212
+ description=body.description.strip(),
213
+ instructions=instructions,
214
+ text=instructions,
215
+ references_json=[r.model_dump() for r in body.references],
216
+ assets_json=[a.model_dump() for a in body.assets],
217
+ scripts_json=[s.model_dump() for s in body.scripts],
218
+ evals_json=[e.model_dump() for e in body.evals],
219
  is_default=0,
220
  is_active=1,
221
  sort_order=max_order + 1,
 
223
  tc.db.add(row)
224
  db_commit_with_retry(tc.db)
225
  tc.db.refresh(row)
226
+ _safe_reindex_skill_content(tc.db, tc.tenant_id, row)
227
  return _skill_to_dict(row)
228
 
229
 
 
236
  )
237
  if not row:
238
  raise HTTPException(status_code=404, detail="Skill not found")
239
+ if body.name is not None:
240
+ row.name = body.name.strip()
241
+ if body.description is not None:
242
+ row.description = body.description.strip()
243
+ if body.instructions is not None:
244
+ row.instructions = body.instructions.strip()
245
+ row.text = row.instructions
246
+ content_changed = False
247
+ if body.references is not None:
248
+ row.references_json = [r.model_dump() for r in body.references]
249
+ content_changed = True
250
+ if body.assets is not None:
251
+ row.assets_json = [a.model_dump() for a in body.assets]
252
+ content_changed = True
253
+ if body.scripts is not None:
254
+ row.scripts_json = [s.model_dump() for s in body.scripts]
255
+ if body.evals is not None:
256
+ row.evals_json = [e.model_dump() for e in body.evals]
257
  if body.is_active is not None:
258
  row.is_active = 1 if body.is_active else 0
259
  if body.sort_order is not None:
260
  row.sort_order = body.sort_order
261
  row.updated_at = datetime.utcnow()
262
  db_commit_with_retry(tc.db)
263
+ if content_changed:
264
+ _safe_reindex_skill_content(tc.db, tc.tenant_id, row)
265
  return _skill_to_dict(row)
266
 
267
 
 
274
  )
275
  if not row:
276
  raise HTTPException(status_code=404, detail="Skill not found")
277
+ delete_skill_content(tc.db, tc.tenant_id, row.id)
278
  tc.db.delete(row)
279
  db_commit_with_retry(tc.db)
280
  return {"ok": True}
281
 
282
 
283
+ @router.get("/api/chat-agent/skills/{skill_id}/export")
284
+ def export_skill(skill_id: int, tc: TenantContext = Depends(require_tenant_admin)):
285
+ row = (
286
+ tc.db.query(ChatAgentSkill)
287
+ .filter(ChatAgentSkill.tenant_id == tc.tenant_id, ChatAgentSkill.id == skill_id)
288
+ .first()
289
+ )
290
+ if not row:
291
+ raise HTTPException(status_code=404, detail="Skill not found")
292
+ md = compile_skill_md(row)
293
+ filename = re.sub(r"[^a-zA-Z0-9_-]+", "-", (row.name or f"skill-{row.id}")).strip("-") or f"skill-{row.id}"
294
+ return PlainTextResponse(
295
+ md,
296
+ media_type="text/markdown",
297
+ headers={"Content-Disposition": f'attachment; filename="{filename}.SKILL.md"'},
298
+ )
299
+
300
+
301
+ @router.post("/api/chat-agent/skills/import")
302
+ def import_skill(body: ChatAgentSkillImportRequest, tc: TenantContext = Depends(require_tenant_admin)):
303
+ parsed = parse_skill_md(body.content)
304
+ if not parsed["instructions"]:
305
+ raise HTTPException(status_code=400, detail="Couldn't find any instructions in that SKILL.md content")
306
+ max_order = (
307
+ tc.db.query(func.max(ChatAgentSkill.sort_order))
308
+ .filter(ChatAgentSkill.tenant_id == tc.tenant_id)
309
+ .scalar()
310
+ or 0
311
+ )
312
+ row = ChatAgentSkill(
313
+ tenant_id=tc.tenant_id,
314
+ name=(parsed["name"] or "imported-skill").strip(),
315
+ description=parsed["description"].strip(),
316
+ instructions=parsed["instructions"].strip(),
317
+ text=parsed["instructions"].strip(),
318
+ references_json=parsed["references"],
319
+ assets_json=parsed["assets"],
320
+ scripts_json=parsed["scripts"],
321
+ evals_json=parsed["evals"],
322
+ is_default=0,
323
+ is_active=1,
324
+ sort_order=max_order + 1,
325
+ )
326
+ tc.db.add(row)
327
+ db_commit_with_retry(tc.db)
328
+ tc.db.refresh(row)
329
+ _safe_reindex_skill_content(tc.db, tc.tenant_id, row)
330
+ return _skill_to_dict(row)
331
+
332
+
333
+ @router.post("/api/chat-agent/skills/{skill_id}/evals/run")
334
+ def run_skill_evals(skill_id: int, tc: TenantContext = Depends(require_tenant_admin)):
335
+ row = (
336
+ tc.db.query(ChatAgentSkill)
337
+ .filter(ChatAgentSkill.tenant_id == tc.tenant_id, ChatAgentSkill.id == skill_id)
338
+ .first()
339
+ )
340
+ if not row:
341
+ raise HTTPException(status_code=404, detail="Skill not found")
342
+ evals = list(row.evals_json or [])
343
+ if not evals:
344
+ return {"skill": _skill_to_dict(row), "results": []}
345
+
346
+ skills_md = compile_skills_md(tc.db, tc.tenant_id)
347
+ results = []
348
+ for ev in evals[:10]:
349
+ prompt = (ev.get("prompt") or "").strip()
350
+ if not prompt:
351
+ continue
352
+ atype = ev.get("assertion_type") or "must_contain"
353
+ aval = ev.get("assertion_value") or ""
354
+ # has_visitor_info=True + already_asked_for_contact=True skip the lead-capture ask, so the
355
+ # eval run tests the actual answer to the prompt rather than a contact-info request.
356
+ system_prompt = build_agentic_system_prompt(
357
+ skills_md=skills_md,
358
+ greeting_message="",
359
+ has_visitor_info=True,
360
+ already_asked_for_contact=True,
361
+ )
362
+ try:
363
+ response_text = "".join(
364
+ stream_agentic_reply_openai(
365
+ system_prompt, [{"role": "user", "content": prompt}], tc.tenant_id, tc.db
366
+ )
367
+ ).strip()
368
+ except Exception as e:
369
+ logger.warning("Eval run failed to generate a response for skill %s: %s", skill_id, e)
370
+ response_text = ""
371
+ graded = run_eval_assertion(prompt, response_text, atype, aval)
372
+ ev["last_run_status"] = graded["status"]
373
+ ev["last_run_detail"] = (
374
+ f"{graded['detail']} — Response: {response_text[:300]}" if response_text else graded["detail"]
375
+ )
376
+ ev["last_run_at"] = datetime.utcnow().isoformat()
377
+ results.append({"prompt": prompt, "response": response_text, **graded})
378
+
379
+ row.evals_json = evals
380
+ flag_modified(row, "evals_json")
381
+ row.updated_at = datetime.utcnow()
382
+ db_commit_with_retry(tc.db)
383
+ return {"skill": _skill_to_dict(row), "results": results}
384
+
385
+
386
  # ---- Admin: reindex + status ----
387
 
388
 
backend/app/database.py CHANGED
@@ -439,13 +439,28 @@ class ChatAgentConfig(Base):
439
 
440
 
441
  class ChatAgentSkill(Base):
442
- """Admin-editable guardrail/behavior instruction, compiled into the chat system prompt."""
 
 
 
 
 
 
 
 
443
 
444
  __tablename__ = "chat_agent_skills"
445
 
446
  id = Column(Integer, primary_key=True, index=True)
447
  tenant_id = Column(Integer, ForeignKey("tenants.id"), nullable=False, index=True)
448
- text = Column(Text, nullable=False, default="")
 
 
 
 
 
 
 
449
  is_default = Column(Integer, default=0)
450
  is_active = Column(Integer, default=1)
451
  sort_order = Column(Integer, default=0)
@@ -756,6 +771,46 @@ def run_migrations(connection_engine):
756
  if "embed_height" not in ccols:
757
  conn.execute(text("ALTER TABLE cta_forms ADD COLUMN embed_height TEXT DEFAULT '480'"))
758
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
759
 
760
  # Create tables then migrate legacy SQLite schemas
761
  Base.metadata.create_all(bind=engine)
 
439
 
440
 
441
  class ChatAgentSkill(Base):
442
+ """
443
+ Admin-editable skill, following Anthropic's Agent Skills anatomy (SKILL.md: YAML
444
+ frontmatter name+description + markdown instructions, plus references/assets/scripts/evals).
445
+ `instructions` (the "brain") is always compiled into the chat system prompt; references and
446
+ assets are indexed into the same searchable knowledge base as Drive/website content so the
447
+ agentic search_knowledge_base tool can surface them on demand. `scripts_json` is stored/
448
+ exportable for documentation and portability ONLY — never executed by the agent (see
449
+ SKILLS-SECURITY.md for why).
450
+ """
451
 
452
  __tablename__ = "chat_agent_skills"
453
 
454
  id = Column(Integer, primary_key=True, index=True)
455
  tenant_id = Column(Integer, ForeignKey("tenants.id"), nullable=False, index=True)
456
+ text = Column(Text, nullable=False, default="") # legacy; superseded by `instructions` below
457
+ name = Column(String, default="")
458
+ description = Column(Text, default="")
459
+ instructions = Column(Text, default="")
460
+ references_json = Column(JSON, nullable=True) # [{title, content}]
461
+ assets_json = Column(JSON, nullable=True) # [{name, content}]
462
+ scripts_json = Column(JSON, nullable=True) # [{name, description, language, content}] — doc only
463
+ evals_json = Column(JSON, nullable=True) # [{prompt, assertion_type, assertion_value, last_run_*}]
464
  is_default = Column(Integer, default=0)
465
  is_active = Column(Integer, default=1)
466
  sort_order = Column(Integer, default=0)
 
771
  if "embed_height" not in ccols:
772
  conn.execute(text("ALTER TABLE cta_forms ADD COLUMN embed_height TEXT DEFAULT '480'"))
773
 
774
+ insp = inspect(connection_engine)
775
+ if insp.has_table("chat_agent_skills"):
776
+ skcols = [c["name"] for c in insp.get_columns("chat_agent_skills")]
777
+ if "name" not in skcols:
778
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN name TEXT DEFAULT ''"))
779
+ if "description" not in skcols:
780
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN description TEXT"))
781
+ if "instructions" not in skcols:
782
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN instructions TEXT"))
783
+ if "references_json" not in skcols:
784
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN references_json TEXT"))
785
+ if "assets_json" not in skcols:
786
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN assets_json TEXT"))
787
+ if "scripts_json" not in skcols:
788
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN scripts_json TEXT"))
789
+ if "evals_json" not in skcols:
790
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN evals_json TEXT"))
791
+ if "sort_order" not in skcols:
792
+ conn.execute(text("ALTER TABLE chat_agent_skills ADD COLUMN sort_order INTEGER DEFAULT 0"))
793
+ # Backfill legacy rows (created before `instructions` existed) so every skill has a
794
+ # real name/description/instructions triple, matching the SKILL.md anatomy.
795
+ conn.execute(
796
+ text(
797
+ "UPDATE chat_agent_skills SET instructions = text "
798
+ "WHERE (instructions IS NULL OR instructions = '') AND text IS NOT NULL"
799
+ )
800
+ )
801
+ conn.execute(
802
+ text(
803
+ "UPDATE chat_agent_skills SET name = 'skill-' || id "
804
+ "WHERE name IS NULL OR name = ''"
805
+ )
806
+ )
807
+ conn.execute(
808
+ text(
809
+ "UPDATE chat_agent_skills SET description = 'Legacy skill — edit to add a name and description' "
810
+ "WHERE description IS NULL OR description = ''"
811
+ )
812
+ )
813
+
814
 
815
  # Create tables then migrate legacy SQLite schemas
816
  Base.metadata.create_all(bind=engine)
backend/app/models.py CHANGED
@@ -354,16 +354,60 @@ class ChatAgentConfigPatch(BaseModel):
354
  reindex_interval_hours: Optional[int] = Field(None, ge=1, le=168)
355
 
356
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
357
  class ChatAgentSkillCreateRequest(BaseModel):
358
- text: str = Field(..., min_length=1, max_length=2000)
 
 
 
 
 
 
359
 
360
 
361
  class ChatAgentSkillPatchRequest(BaseModel):
362
- text: Optional[str] = Field(None, min_length=1, max_length=2000)
 
 
 
 
 
 
363
  is_active: Optional[bool] = None
364
  sort_order: Optional[int] = None
365
 
366
 
 
 
 
 
367
  class ChatAgentReindexRequest(BaseModel):
368
  source: Literal["drive", "website", "all"] = "all"
369
 
 
354
  reindex_interval_hours: Optional[int] = Field(None, ge=1, le=168)
355
 
356
 
357
+ class ChatAgentSkillReferenceItem(BaseModel):
358
+ title: str = Field("", max_length=300)
359
+ content: str = Field("", max_length=20000)
360
+
361
+
362
+ class ChatAgentSkillAssetItem(BaseModel):
363
+ name: str = Field("", max_length=300)
364
+ content: str = Field("", max_length=20000)
365
+
366
+
367
+ class ChatAgentSkillScriptItem(BaseModel):
368
+ # Stored/exported for documentation only — the agent never executes this content. See
369
+ # SKILLS-SECURITY.md.
370
+ name: str = Field("", max_length=300)
371
+ description: str = Field("", max_length=2000)
372
+ language: str = Field("", max_length=40)
373
+ content: str = Field("", max_length=20000)
374
+
375
+
376
+ class ChatAgentSkillEvalItem(BaseModel):
377
+ prompt: str = Field("", max_length=2000)
378
+ assertion_type: Literal["must_contain", "must_not_contain", "llm_judge"] = "must_contain"
379
+ assertion_value: str = Field("", max_length=2000)
380
+ last_run_status: Optional[str] = None
381
+ last_run_detail: Optional[str] = None
382
+ last_run_at: Optional[str] = None
383
+
384
+
385
  class ChatAgentSkillCreateRequest(BaseModel):
386
+ name: str = Field(..., min_length=1, max_length=200)
387
+ description: str = Field("", max_length=1000)
388
+ instructions: str = Field(..., min_length=1, max_length=8000)
389
+ references: List[ChatAgentSkillReferenceItem] = Field(default_factory=list, max_length=20)
390
+ assets: List[ChatAgentSkillAssetItem] = Field(default_factory=list, max_length=20)
391
+ scripts: List[ChatAgentSkillScriptItem] = Field(default_factory=list, max_length=20)
392
+ evals: List[ChatAgentSkillEvalItem] = Field(default_factory=list, max_length=10)
393
 
394
 
395
  class ChatAgentSkillPatchRequest(BaseModel):
396
+ name: Optional[str] = Field(None, min_length=1, max_length=200)
397
+ description: Optional[str] = Field(None, max_length=1000)
398
+ instructions: Optional[str] = Field(None, min_length=1, max_length=8000)
399
+ references: Optional[List[ChatAgentSkillReferenceItem]] = Field(None, max_length=20)
400
+ assets: Optional[List[ChatAgentSkillAssetItem]] = Field(None, max_length=20)
401
+ scripts: Optional[List[ChatAgentSkillScriptItem]] = Field(None, max_length=20)
402
+ evals: Optional[List[ChatAgentSkillEvalItem]] = Field(None, max_length=10)
403
  is_active: Optional[bool] = None
404
  sort_order: Optional[int] = None
405
 
406
 
407
+ class ChatAgentSkillImportRequest(BaseModel):
408
+ content: str = Field(..., min_length=1, max_length=100000)
409
+
410
+
411
  class ChatAgentReindexRequest(BaseModel):
412
  source: Literal["drive", "website", "all"] = "all"
413
 
frontend/src/components/settings/ChatAgentPanel.jsx CHANGED
@@ -7,10 +7,8 @@ import {
7
  Loader2,
8
  Mail,
9
  MessageSquare,
10
- Plus,
11
  RefreshCw,
12
  Send,
13
- Trash2,
14
  } from 'lucide-react';
15
  import AppShell from '@/components/layout/AppShell';
16
  import { Button } from '@/components/ui/button';
@@ -24,6 +22,7 @@ import {
24
  SelectTrigger,
25
  SelectValue,
26
  } from '@/components/ui/select';
 
27
  import { apiFetch } from '@/lib/api';
28
  import { cn } from '@/lib/utils';
29
 
@@ -325,11 +324,6 @@ export default function ChatAgentPanel({ onBack }) {
325
  const [error, setError] = useState('');
326
  const [savingConfig, setSavingConfig] = useState(false);
327
 
328
- const [skills, setSkills] = useState([]);
329
- const [skillsLoading, setSkillsLoading] = useState(true);
330
- const [newSkillText, setNewSkillText] = useState('');
331
- const [addingSkill, setAddingSkill] = useState(false);
332
-
333
  const [status, setStatus] = useState(null);
334
  const [statusLoading, setStatusLoading] = useState(true);
335
  const [reindexBusy, setReindexBusy] = useState('');
@@ -361,19 +355,6 @@ export default function ChatAgentPanel({ onBack }) {
361
  }
362
  }, []);
363
 
364
- const loadSkills = useCallback(async () => {
365
- setSkillsLoading(true);
366
- try {
367
- const r = await apiFetch('/api/chat-agent/skills');
368
- const data = r.ok ? await r.json() : { skills: [] };
369
- setSkills(data.skills || []);
370
- } catch {
371
- setSkills([]);
372
- } finally {
373
- setSkillsLoading(false);
374
- }
375
- }, []);
376
-
377
  const loadStatus = useCallback(async () => {
378
  setStatusLoading(true);
379
  try {
@@ -402,10 +383,9 @@ export default function ChatAgentPanel({ onBack }) {
402
 
403
  useEffect(() => {
404
  loadConfig();
405
- loadSkills();
406
  loadStatus();
407
  loadConversations();
408
- }, [loadConfig, loadSkills, loadStatus, loadConversations]);
409
 
410
  // Reflect Drive OAuth callback redirect (?chat_agent_drive=connected / ?chat_agent_drive_error=...)
411
  useEffect(() => {
@@ -485,56 +465,6 @@ export default function ChatAgentPanel({ onBack }) {
485
  }
486
  };
487
 
488
- const addSkill = async () => {
489
- const text = newSkillText.trim();
490
- if (!text) return;
491
- setAddingSkill(true);
492
- try {
493
- const res = await apiFetch('/api/chat-agent/skills', {
494
- method: 'POST',
495
- headers: { 'Content-Type': 'application/json' },
496
- body: JSON.stringify({ text }),
497
- });
498
- const data = await res.json();
499
- if (!res.ok) throw new Error(data.detail || 'Could not add skill');
500
- setSkills((prev) => [...prev, data]);
501
- setNewSkillText('');
502
- } catch (e) {
503
- setError(e.message || 'Could not add skill');
504
- } finally {
505
- setAddingSkill(false);
506
- }
507
- };
508
-
509
- const toggleSkillActive = async (skill) => {
510
- try {
511
- const res = await apiFetch(`/api/chat-agent/skills/${skill.id}`, {
512
- method: 'PATCH',
513
- headers: { 'Content-Type': 'application/json' },
514
- body: JSON.stringify({ is_active: !skill.is_active }),
515
- });
516
- const data = await res.json();
517
- if (!res.ok) throw new Error(data.detail || 'Update failed');
518
- setSkills((prev) => prev.map((s) => (s.id === skill.id ? data : s)));
519
- } catch (e) {
520
- setError(e.message || 'Update failed');
521
- }
522
- };
523
-
524
- const deleteSkill = async (skill) => {
525
- if (!window.confirm('Remove this skill?')) return;
526
- try {
527
- const res = await apiFetch(`/api/chat-agent/skills/${skill.id}`, { method: 'DELETE' });
528
- if (!res.ok) {
529
- const data = await res.json().catch(() => ({}));
530
- throw new Error(data.detail || 'Delete failed');
531
- }
532
- setSkills((prev) => prev.filter((s) => s.id !== skill.id));
533
- } catch (e) {
534
- setError(e.message || 'Delete failed');
535
- }
536
- };
537
-
538
  const openConversation = async (id) => {
539
  try {
540
  const r = await apiFetch(`/api/chat-agent/conversations/${id}`);
@@ -776,71 +706,7 @@ export default function ChatAgentPanel({ onBack }) {
776
 
777
  {/* Skills */}
778
  <TabsContent value="skills" className="space-y-4">
779
- <section className="rounded-2xl border border-slate-200 bg-white p-6 shadow-sm space-y-4">
780
- <div className="font-semibold text-slate-800">Skills &amp; guardrails</div>
781
- <p className="text-sm text-slate-600">
782
- Instructions the agent must always follow. Non-negotiable — the agent is told to
783
- keep these even under pressure from a visitor.
784
- </p>
785
- <div className="flex flex-col sm:flex-row gap-2">
786
- <Textarea
787
- placeholder="e.g. Always offer to schedule a demo call when asked about implementation."
788
- value={newSkillText}
789
- onChange={(e) => setNewSkillText(e.target.value)}
790
- rows={2}
791
- className="text-sm"
792
- />
793
- <Button type="button" onClick={addSkill} disabled={addingSkill} className="shrink-0 gap-1">
794
- {addingSkill ? <Loader2 className="h-4 w-4 animate-spin" /> : <Plus className="h-4 w-4" />}
795
- Add skill
796
- </Button>
797
- </div>
798
-
799
- {skillsLoading ? (
800
- <div className="flex justify-center py-8 text-slate-400">
801
- <Loader2 className="h-8 w-8 animate-spin" />
802
- </div>
803
- ) : skills.length === 0 ? (
804
- <p className="text-sm text-slate-500">No skills yet.</p>
805
- ) : (
806
- <ul className="space-y-2">
807
- {skills.map((s) => (
808
- <li
809
- key={s.id}
810
- className={cn(
811
- 'flex items-start gap-3 rounded-xl border px-3 py-2.5',
812
- s.is_active ? 'border-slate-200' : 'border-slate-100 bg-slate-50/60 opacity-60'
813
- )}
814
- >
815
- <input
816
- type="checkbox"
817
- className="mt-1 h-4 w-4 accent-violet-600"
818
- checked={!!s.is_active}
819
- onChange={() => toggleSkillActive(s)}
820
- title="Active"
821
- />
822
- <p className="flex-1 text-sm text-slate-700">
823
- {s.text}
824
- {s.is_default ? (
825
- <span className="ml-2 inline-flex rounded-full bg-violet-100 px-1.5 py-0.5 text-[10px] font-medium text-violet-800">
826
- default
827
- </span>
828
- ) : null}
829
- </p>
830
- <Button
831
- type="button"
832
- variant="ghost"
833
- size="icon"
834
- className="h-8 w-8 text-red-500 shrink-0"
835
- onClick={() => deleteSkill(s)}
836
- >
837
- <Trash2 className="h-4 w-4" />
838
- </Button>
839
- </li>
840
- ))}
841
- </ul>
842
- )}
843
- </section>
844
  </TabsContent>
845
 
846
  {/* Test Chat */}
 
7
  Loader2,
8
  Mail,
9
  MessageSquare,
 
10
  RefreshCw,
11
  Send,
 
12
  } from 'lucide-react';
13
  import AppShell from '@/components/layout/AppShell';
14
  import { Button } from '@/components/ui/button';
 
22
  SelectTrigger,
23
  SelectValue,
24
  } from '@/components/ui/select';
25
+ import ChatAgentSkillsTab from '@/components/settings/ChatAgentSkillsTab';
26
  import { apiFetch } from '@/lib/api';
27
  import { cn } from '@/lib/utils';
28
 
 
324
  const [error, setError] = useState('');
325
  const [savingConfig, setSavingConfig] = useState(false);
326
 
 
 
 
 
 
327
  const [status, setStatus] = useState(null);
328
  const [statusLoading, setStatusLoading] = useState(true);
329
  const [reindexBusy, setReindexBusy] = useState('');
 
355
  }
356
  }, []);
357
 
 
 
 
 
 
 
 
 
 
 
 
 
 
358
  const loadStatus = useCallback(async () => {
359
  setStatusLoading(true);
360
  try {
 
383
 
384
  useEffect(() => {
385
  loadConfig();
 
386
  loadStatus();
387
  loadConversations();
388
+ }, [loadConfig, loadStatus, loadConversations]);
389
 
390
  // Reflect Drive OAuth callback redirect (?chat_agent_drive=connected / ?chat_agent_drive_error=...)
391
  useEffect(() => {
 
465
  }
466
  };
467
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
468
  const openConversation = async (id) => {
469
  try {
470
  const r = await apiFetch(`/api/chat-agent/conversations/${id}`);
 
706
 
707
  {/* Skills */}
708
  <TabsContent value="skills" className="space-y-4">
709
+ <ChatAgentSkillsTab />
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
710
  </TabsContent>
711
 
712
  {/* Test Chat */}
frontend/src/components/settings/ChatAgentSkillsTab.jsx ADDED
@@ -0,0 +1,724 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import React, { useCallback, useEffect, useRef, useState } from 'react';
2
+ import {
3
+ AlertTriangle,
4
+ CheckCircle2,
5
+ ChevronDown,
6
+ ChevronRight,
7
+ Download,
8
+ FlaskConical,
9
+ Loader2,
10
+ Plus,
11
+ Trash2,
12
+ Upload,
13
+ XCircle,
14
+ } from 'lucide-react';
15
+ import { Button } from '@/components/ui/button';
16
+ import { Input } from '@/components/ui/input';
17
+ import { Textarea } from '@/components/ui/textarea';
18
+ import { apiFetch } from '@/lib/api';
19
+ import { cn } from '@/lib/utils';
20
+
21
+ const AUTOSAVE_MS = 600;
22
+
23
+ const ASSERTION_LABELS = {
24
+ must_contain: 'Response must contain',
25
+ must_not_contain: 'Response must NOT contain',
26
+ llm_judge: 'LLM judge rubric',
27
+ };
28
+
29
+ function draftFromSkill(s) {
30
+ return {
31
+ name: s.name || '',
32
+ description: s.description || '',
33
+ instructions: s.instructions || '',
34
+ references: s.references || [],
35
+ assets: s.assets || [],
36
+ scripts: s.scripts || [],
37
+ evals: s.evals || [],
38
+ };
39
+ }
40
+
41
+ function draftSnapshot(d) {
42
+ return JSON.stringify(d);
43
+ }
44
+
45
+ function emptyDraft() {
46
+ return { name: '', description: '', instructions: '', references: [], assets: [], scripts: [], evals: [] };
47
+ }
48
+
49
+ /** One collapsible References/Assets/Scripts/Evals section shared shell. */
50
+ function Section({ title, icon, count, open, onToggle, notice, children }) {
51
+ return (
52
+ <div className="rounded-xl border border-slate-100">
53
+ <button
54
+ type="button"
55
+ onClick={onToggle}
56
+ className="flex w-full items-center gap-2 px-3 py-2 text-left text-sm font-medium text-slate-700 hover:bg-slate-50 rounded-xl"
57
+ >
58
+ {open ? <ChevronDown className="h-3.5 w-3.5 shrink-0" /> : <ChevronRight className="h-3.5 w-3.5 shrink-0" />}
59
+ {icon}
60
+ {title}
61
+ <span className="text-xs font-normal text-slate-400">{count}</span>
62
+ </button>
63
+ {notice ? (
64
+ <div className="flex items-start gap-1.5 px-3 pb-1 text-xs text-amber-700">
65
+ <AlertTriangle className="h-3.5 w-3.5 shrink-0 mt-0.5" />
66
+ <span>{notice}</span>
67
+ </div>
68
+ ) : null}
69
+ {open ? <div className="space-y-2 px-3 pb-3">{children}</div> : null}
70
+ </div>
71
+ );
72
+ }
73
+
74
+ function SkillCard({ skill, onSaved, onDeleted, onError, defaultExpanded }) {
75
+ const [draft, setDraft] = useState(() => draftFromSkill(skill));
76
+ const [saveState, setSaveState] = useState('idle');
77
+ const [sectionsOpen, setSectionsOpen] = useState({
78
+ references: !!defaultExpanded,
79
+ assets: false,
80
+ scripts: false,
81
+ evals: false,
82
+ });
83
+ const [deleteBusy, setDeleteBusy] = useState(false);
84
+ const [evalRunBusy, setEvalRunBusy] = useState(false);
85
+ const [evalResults, setEvalResults] = useState(null);
86
+
87
+ const skipAutosaveRef = useRef(true);
88
+ const saveTimerRef = useRef(null);
89
+ const lastSyncedSnapshotRef = useRef(draftSnapshot(draftFromSkill(skill)));
90
+ const savedFadeTimerRef = useRef(null);
91
+
92
+ // Resync when the skill changes underneath us (e.g. an evals/run response overwrote
93
+ // last_run_status), but not when it's just an echo of what we ourselves just saved.
94
+ useEffect(() => {
95
+ const next = draftFromSkill(skill);
96
+ const snap = draftSnapshot(next);
97
+ if (snap === lastSyncedSnapshotRef.current) return;
98
+ skipAutosaveRef.current = true;
99
+ lastSyncedSnapshotRef.current = snap;
100
+ setDraft(next);
101
+ }, [skill]);
102
+
103
+ useEffect(() => {
104
+ if (skipAutosaveRef.current) {
105
+ skipAutosaveRef.current = false;
106
+ return;
107
+ }
108
+ const snapshot = draftSnapshot(draft);
109
+ if (snapshot === lastSyncedSnapshotRef.current) return;
110
+ if (!draft.name.trim() || !draft.instructions.trim()) return;
111
+
112
+ if (saveTimerRef.current) clearTimeout(saveTimerRef.current);
113
+ saveTimerRef.current = setTimeout(async () => {
114
+ if (draftSnapshot(draft) !== snapshot) return;
115
+ setSaveState('saving');
116
+ try {
117
+ const res = await apiFetch(`/api/chat-agent/skills/${skill.id}`, {
118
+ method: 'PATCH',
119
+ headers: { 'Content-Type': 'application/json' },
120
+ body: JSON.stringify({
121
+ name: draft.name.trim(),
122
+ description: draft.description,
123
+ instructions: draft.instructions.trim(),
124
+ references: draft.references,
125
+ assets: draft.assets,
126
+ scripts: draft.scripts,
127
+ evals: draft.evals,
128
+ }),
129
+ });
130
+ const data = await res.json();
131
+ if (!res.ok) throw new Error(data.detail || 'Save failed');
132
+ lastSyncedSnapshotRef.current = draftSnapshot(draftFromSkill(data));
133
+ onSaved(data);
134
+ if (savedFadeTimerRef.current) clearTimeout(savedFadeTimerRef.current);
135
+ setSaveState('saved');
136
+ savedFadeTimerRef.current = window.setTimeout(() => setSaveState('idle'), 2000);
137
+ } catch (e) {
138
+ onError(e.message || 'Save failed');
139
+ setSaveState('error');
140
+ }
141
+ }, AUTOSAVE_MS);
142
+ return () => {
143
+ if (saveTimerRef.current) clearTimeout(saveTimerRef.current);
144
+ };
145
+ // eslint-disable-next-line react-hooks/exhaustive-deps
146
+ }, [draft]);
147
+
148
+ const toggleSection = (key) => setSectionsOpen((s) => ({ ...s, [key]: !s[key] }));
149
+
150
+ const updateListItem = (key, idx, patch) => {
151
+ setDraft((d) => ({ ...d, [key]: d[key].map((item, i) => (i === idx ? { ...item, ...patch } : item)) }));
152
+ };
153
+ const addListItem = (key, item) => setDraft((d) => ({ ...d, [key]: [...d[key], item] }));
154
+ const removeListItem = (key, idx) => setDraft((d) => ({ ...d, [key]: d[key].filter((_, i) => i !== idx) }));
155
+
156
+ const toggleActive = async () => {
157
+ try {
158
+ const res = await apiFetch(`/api/chat-agent/skills/${skill.id}`, {
159
+ method: 'PATCH',
160
+ headers: { 'Content-Type': 'application/json' },
161
+ body: JSON.stringify({ is_active: !skill.is_active }),
162
+ });
163
+ const data = await res.json();
164
+ if (!res.ok) throw new Error(data.detail || 'Update failed');
165
+ onSaved(data);
166
+ } catch (e) {
167
+ onError(e.message || 'Update failed');
168
+ }
169
+ };
170
+
171
+ const handleDelete = async () => {
172
+ if (
173
+ !window.confirm(
174
+ `Delete skill "${skill.name || 'untitled'}"? This also removes its indexed references/assets from the knowledge base.`
175
+ )
176
+ )
177
+ return;
178
+ setDeleteBusy(true);
179
+ try {
180
+ const res = await apiFetch(`/api/chat-agent/skills/${skill.id}`, { method: 'DELETE' });
181
+ if (!res.ok) {
182
+ const data = await res.json().catch(() => ({}));
183
+ throw new Error(data.detail || 'Delete failed');
184
+ }
185
+ onDeleted(skill.id);
186
+ } catch (e) {
187
+ onError(e.message || 'Delete failed');
188
+ setDeleteBusy(false);
189
+ }
190
+ };
191
+
192
+ const handleExport = () => {
193
+ window.open(`/api/chat-agent/skills/${skill.id}/export`, '_blank');
194
+ };
195
+
196
+ const runEvals = async () => {
197
+ setEvalRunBusy(true);
198
+ setEvalResults(null);
199
+ try {
200
+ const res = await apiFetch(`/api/chat-agent/skills/${skill.id}/evals/run`, { method: 'POST' });
201
+ const data = await res.json();
202
+ if (!res.ok) throw new Error(data.detail || 'Eval run failed');
203
+ onSaved(data.skill);
204
+ setEvalResults(data.results);
205
+ } catch (e) {
206
+ onError(e.message || 'Eval run failed');
207
+ } finally {
208
+ setEvalRunBusy(false);
209
+ }
210
+ };
211
+
212
+ return (
213
+ <div
214
+ className={cn(
215
+ 'rounded-2xl border bg-white p-4 shadow-sm space-y-3',
216
+ skill.is_active ? 'border-slate-200' : 'border-slate-100 bg-slate-50/60 opacity-70'
217
+ )}
218
+ >
219
+ <div className="flex items-start gap-3">
220
+ <input
221
+ type="checkbox"
222
+ className="mt-2.5 h-4 w-4 accent-violet-600 shrink-0"
223
+ checked={!!skill.is_active}
224
+ onChange={toggleActive}
225
+ title="Active"
226
+ />
227
+ <div className="flex-1 space-y-2 min-w-0">
228
+ <div className="flex flex-wrap items-center gap-2">
229
+ <Input
230
+ value={draft.name}
231
+ onChange={(e) => setDraft((d) => ({ ...d, name: e.target.value }))}
232
+ placeholder="skill-name"
233
+ className="h-8 max-w-[220px] font-medium"
234
+ />
235
+ {skill.is_default ? (
236
+ <span className="inline-flex rounded-full bg-violet-100 px-1.5 py-0.5 text-[10px] font-medium text-violet-800">
237
+ default
238
+ </span>
239
+ ) : null}
240
+ <span className="text-xs text-slate-400">
241
+ {saveState === 'saving' ? 'Saving…' : saveState === 'saved' ? 'Saved' : saveState === 'error' ? 'Save failed' : ''}
242
+ </span>
243
+ <div className="ml-auto flex items-center gap-1">
244
+ <Button type="button" variant="ghost" size="sm" className="h-7 gap-1 text-xs" onClick={handleExport}>
245
+ <Download className="h-3.5 w-3.5" />
246
+ Export
247
+ </Button>
248
+ <Button
249
+ type="button"
250
+ variant="ghost"
251
+ size="icon"
252
+ className="h-7 w-7 text-red-500"
253
+ disabled={deleteBusy}
254
+ onClick={handleDelete}
255
+ >
256
+ {deleteBusy ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <Trash2 className="h-3.5 w-3.5" />}
257
+ </Button>
258
+ </div>
259
+ </div>
260
+ <Input
261
+ value={draft.description}
262
+ onChange={(e) => setDraft((d) => ({ ...d, description: e.target.value }))}
263
+ placeholder="One-line description (when the agent should use this skill)"
264
+ className="h-8 text-sm"
265
+ />
266
+ <Textarea
267
+ value={draft.instructions}
268
+ onChange={(e) => setDraft((d) => ({ ...d, instructions: e.target.value }))}
269
+ placeholder="Instructions — the brain of this skill. Always injected into the agent's system prompt."
270
+ rows={3}
271
+ className="text-sm"
272
+ />
273
+ </div>
274
+ </div>
275
+
276
+ <div className="space-y-2 pl-7">
277
+ <Section
278
+ title="References"
279
+ icon={null}
280
+ count={draft.references.length}
281
+ open={sectionsOpen.references}
282
+ onToggle={() => toggleSection('references')}
283
+ >
284
+ <p className="text-xs text-slate-500">
285
+ Domain knowledge the agent can search for on demand (indexed into the knowledge base).
286
+ </p>
287
+ {draft.references.map((r, idx) => (
288
+ <div key={idx} className="space-y-1 rounded-lg border border-slate-100 p-2">
289
+ <div className="flex items-center gap-2">
290
+ <Input
291
+ value={r.title}
292
+ onChange={(e) => updateListItem('references', idx, { title: e.target.value })}
293
+ placeholder="Title"
294
+ className="h-7 text-xs"
295
+ />
296
+ <Button
297
+ type="button"
298
+ variant="ghost"
299
+ size="icon"
300
+ className="h-7 w-7 text-red-500 shrink-0"
301
+ onClick={() => removeListItem('references', idx)}
302
+ >
303
+ <Trash2 className="h-3.5 w-3.5" />
304
+ </Button>
305
+ </div>
306
+ <Textarea
307
+ value={r.content}
308
+ onChange={(e) => updateListItem('references', idx, { content: e.target.value })}
309
+ placeholder="Reference content..."
310
+ rows={2}
311
+ className="text-xs"
312
+ />
313
+ </div>
314
+ ))}
315
+ <Button
316
+ type="button"
317
+ variant="outline"
318
+ size="sm"
319
+ className="h-7 gap-1 text-xs"
320
+ onClick={() => addListItem('references', { title: '', content: '' })}
321
+ >
322
+ <Plus className="h-3.5 w-3.5" />
323
+ Add reference
324
+ </Button>
325
+ </Section>
326
+
327
+ <Section
328
+ title="Assets"
329
+ icon={null}
330
+ count={draft.assets.length}
331
+ open={sectionsOpen.assets}
332
+ onToggle={() => toggleSection('assets')}
333
+ >
334
+ <p className="text-xs text-slate-500">
335
+ Output templates/snippets the agent can search for and reuse when answering.
336
+ </p>
337
+ {draft.assets.map((a, idx) => (
338
+ <div key={idx} className="space-y-1 rounded-lg border border-slate-100 p-2">
339
+ <div className="flex items-center gap-2">
340
+ <Input
341
+ value={a.name}
342
+ onChange={(e) => updateListItem('assets', idx, { name: e.target.value })}
343
+ placeholder="Name"
344
+ className="h-7 text-xs"
345
+ />
346
+ <Button
347
+ type="button"
348
+ variant="ghost"
349
+ size="icon"
350
+ className="h-7 w-7 text-red-500 shrink-0"
351
+ onClick={() => removeListItem('assets', idx)}
352
+ >
353
+ <Trash2 className="h-3.5 w-3.5" />
354
+ </Button>
355
+ </div>
356
+ <Textarea
357
+ value={a.content}
358
+ onChange={(e) => updateListItem('assets', idx, { content: e.target.value })}
359
+ placeholder="Asset content..."
360
+ rows={2}
361
+ className="text-xs"
362
+ />
363
+ </div>
364
+ ))}
365
+ <Button
366
+ type="button"
367
+ variant="outline"
368
+ size="sm"
369
+ className="h-7 gap-1 text-xs"
370
+ onClick={() => addListItem('assets', { name: '', content: '' })}
371
+ >
372
+ <Plus className="h-3.5 w-3.5" />
373
+ Add asset
374
+ </Button>
375
+ </Section>
376
+
377
+ <Section
378
+ title="Scripts"
379
+ icon={null}
380
+ count={draft.scripts.length}
381
+ open={sectionsOpen.scripts}
382
+ onToggle={() => toggleSection('scripts')}
383
+ notice="Stored for documentation/portability only — never executed by the agent. See SKILLS-SECURITY.md."
384
+ >
385
+ {draft.scripts.map((s, idx) => (
386
+ <div key={idx} className="space-y-1 rounded-lg border border-slate-100 p-2">
387
+ <div className="flex items-center gap-2">
388
+ <Input
389
+ value={s.name}
390
+ onChange={(e) => updateListItem('scripts', idx, { name: e.target.value })}
391
+ placeholder="Name"
392
+ className="h-7 text-xs"
393
+ />
394
+ <Input
395
+ value={s.language}
396
+ onChange={(e) => updateListItem('scripts', idx, { language: e.target.value })}
397
+ placeholder="language"
398
+ className="h-7 w-28 text-xs"
399
+ />
400
+ <Button
401
+ type="button"
402
+ variant="ghost"
403
+ size="icon"
404
+ className="h-7 w-7 text-red-500 shrink-0"
405
+ onClick={() => removeListItem('scripts', idx)}
406
+ >
407
+ <Trash2 className="h-3.5 w-3.5" />
408
+ </Button>
409
+ </div>
410
+ <Input
411
+ value={s.description}
412
+ onChange={(e) => updateListItem('scripts', idx, { description: e.target.value })}
413
+ placeholder="What this script is for"
414
+ className="h-7 text-xs"
415
+ />
416
+ <Textarea
417
+ value={s.content}
418
+ onChange={(e) => updateListItem('scripts', idx, { content: e.target.value })}
419
+ placeholder="Script content (not executed)..."
420
+ rows={3}
421
+ className="text-xs font-mono"
422
+ />
423
+ </div>
424
+ ))}
425
+ <Button
426
+ type="button"
427
+ variant="outline"
428
+ size="sm"
429
+ className="h-7 gap-1 text-xs"
430
+ onClick={() => addListItem('scripts', { name: '', description: '', language: '', content: '' })}
431
+ >
432
+ <Plus className="h-3.5 w-3.5" />
433
+ Add script
434
+ </Button>
435
+ </Section>
436
+
437
+ <Section
438
+ title="Evals"
439
+ icon={<FlaskConical className="h-3.5 w-3.5 shrink-0" />}
440
+ count={draft.evals.length}
441
+ open={sectionsOpen.evals}
442
+ onToggle={() => toggleSection('evals')}
443
+ >
444
+ <p className="text-xs text-slate-500">Test cases — a prompt plus an assertion the agent's reply must satisfy.</p>
445
+ {draft.evals.map((ev, idx) => {
446
+ const result = evalResults?.[idx];
447
+ return (
448
+ <div key={idx} className="space-y-1 rounded-lg border border-slate-100 p-2">
449
+ <div className="flex items-center gap-2">
450
+ <Input
451
+ value={ev.prompt}
452
+ onChange={(e) => updateListItem('evals', idx, { prompt: e.target.value })}
453
+ placeholder="Prompt to send the agent"
454
+ className="h-7 text-xs"
455
+ />
456
+ <Button
457
+ type="button"
458
+ variant="ghost"
459
+ size="icon"
460
+ className="h-7 w-7 text-red-500 shrink-0"
461
+ onClick={() => removeListItem('evals', idx)}
462
+ >
463
+ <Trash2 className="h-3.5 w-3.5" />
464
+ </Button>
465
+ </div>
466
+ <div className="flex items-center gap-2">
467
+ <select
468
+ value={ev.assertion_type}
469
+ onChange={(e) => updateListItem('evals', idx, { assertion_type: e.target.value })}
470
+ className="h-7 rounded-md border border-slate-200 bg-white text-xs px-1.5"
471
+ >
472
+ {Object.entries(ASSERTION_LABELS).map(([v, label]) => (
473
+ <option key={v} value={v}>
474
+ {label}
475
+ </option>
476
+ ))}
477
+ </select>
478
+ <Input
479
+ value={ev.assertion_value}
480
+ onChange={(e) => updateListItem('evals', idx, { assertion_value: e.target.value })}
481
+ placeholder={
482
+ ev.assertion_type === 'llm_judge' ? 'Rubric describing a passing answer' : 'Text to check for'
483
+ }
484
+ className="h-7 text-xs flex-1"
485
+ />
486
+ </div>
487
+ {ev.last_run_status ? (
488
+ <div
489
+ className={cn(
490
+ 'flex items-start gap-1.5 rounded-md px-2 py-1 text-xs',
491
+ ev.last_run_status === 'pass'
492
+ ? 'bg-emerald-50 text-emerald-800'
493
+ : ev.last_run_status === 'fail'
494
+ ? 'bg-red-50 text-red-700'
495
+ : 'bg-slate-100 text-slate-600'
496
+ )}
497
+ >
498
+ {ev.last_run_status === 'pass' ? (
499
+ <CheckCircle2 className="h-3.5 w-3.5 shrink-0 mt-px" />
500
+ ) : (
501
+ <XCircle className="h-3.5 w-3.5 shrink-0 mt-px" />
502
+ )}
503
+ <span>
504
+ {ev.last_run_status.toUpperCase()} · {ev.last_run_detail || ''}
505
+ {ev.last_run_at ? ` · ${new Date(ev.last_run_at).toLocaleString()}` : ''}
506
+ </span>
507
+ </div>
508
+ ) : null}
509
+ {result ? (
510
+ <p className="text-[11px] text-slate-400 truncate" title={result.response}>
511
+ Response: {result.response || '(empty)'}
512
+ </p>
513
+ ) : null}
514
+ </div>
515
+ );
516
+ })}
517
+ <div className="flex flex-wrap items-center gap-2">
518
+ <Button
519
+ type="button"
520
+ variant="outline"
521
+ size="sm"
522
+ className="h-7 gap-1 text-xs"
523
+ onClick={() =>
524
+ addListItem('evals', { prompt: '', assertion_type: 'must_contain', assertion_value: '' })
525
+ }
526
+ >
527
+ <Plus className="h-3.5 w-3.5" />
528
+ Add eval
529
+ </Button>
530
+ {draft.evals.length > 0 ? (
531
+ <Button
532
+ type="button"
533
+ variant="outline"
534
+ size="sm"
535
+ className="h-7 gap-1 text-xs"
536
+ disabled={evalRunBusy}
537
+ onClick={runEvals}
538
+ >
539
+ {evalRunBusy ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <FlaskConical className="h-3.5 w-3.5" />}
540
+ Run evals
541
+ </Button>
542
+ ) : null}
543
+ </div>
544
+ </Section>
545
+ </div>
546
+ </div>
547
+ );
548
+ }
549
+
550
+ export default function ChatAgentSkillsTab() {
551
+ const [skills, setSkills] = useState([]);
552
+ const [loading, setLoading] = useState(true);
553
+ const [error, setError] = useState('');
554
+ const [creating, setCreating] = useState(false);
555
+ const [lastCreatedId, setLastCreatedId] = useState(null);
556
+
557
+ const [importOpen, setImportOpen] = useState(false);
558
+ const [importText, setImportText] = useState('');
559
+ const [importBusy, setImportBusy] = useState(false);
560
+ const fileInputRef = useRef(null);
561
+
562
+ const loadSkills = useCallback(async () => {
563
+ setLoading(true);
564
+ try {
565
+ const r = await apiFetch('/api/chat-agent/skills');
566
+ const data = r.ok ? await r.json() : { skills: [] };
567
+ setSkills(data.skills || []);
568
+ } catch {
569
+ setSkills([]);
570
+ } finally {
571
+ setLoading(false);
572
+ }
573
+ }, []);
574
+
575
+ useEffect(() => {
576
+ loadSkills();
577
+ }, [loadSkills]);
578
+
579
+ const handleSaved = (updated) => {
580
+ setSkills((prev) => prev.map((s) => (s.id === updated.id ? updated : s)));
581
+ };
582
+ const handleDeleted = (id) => {
583
+ setSkills((prev) => prev.filter((s) => s.id !== id));
584
+ };
585
+
586
+ const addSkill = async () => {
587
+ setCreating(true);
588
+ setError('');
589
+ try {
590
+ const res = await apiFetch('/api/chat-agent/skills', {
591
+ method: 'POST',
592
+ headers: { 'Content-Type': 'application/json' },
593
+ body: JSON.stringify({
594
+ name: `new-skill-${skills.length + 1}`,
595
+ description: '',
596
+ instructions: 'Describe what the agent should do or avoid doing.',
597
+ }),
598
+ });
599
+ const data = await res.json();
600
+ if (!res.ok) throw new Error(data.detail || 'Could not add skill');
601
+ setSkills((prev) => [...prev, data]);
602
+ setLastCreatedId(data.id);
603
+ } catch (e) {
604
+ setError(e.message || 'Could not add skill');
605
+ } finally {
606
+ setCreating(false);
607
+ }
608
+ };
609
+
610
+ const runImport = async () => {
611
+ const content = importText.trim();
612
+ if (!content) return;
613
+ setImportBusy(true);
614
+ setError('');
615
+ try {
616
+ const res = await apiFetch('/api/chat-agent/skills/import', {
617
+ method: 'POST',
618
+ headers: { 'Content-Type': 'application/json' },
619
+ body: JSON.stringify({ content }),
620
+ });
621
+ const data = await res.json();
622
+ if (!res.ok) throw new Error(data.detail || 'Import failed');
623
+ setSkills((prev) => [...prev, data]);
624
+ setLastCreatedId(data.id);
625
+ setImportText('');
626
+ setImportOpen(false);
627
+ } catch (e) {
628
+ setError(e.message || 'Import failed');
629
+ } finally {
630
+ setImportBusy(false);
631
+ }
632
+ };
633
+
634
+ const handleFilePicked = (e) => {
635
+ const file = e.target.files?.[0];
636
+ if (!file) return;
637
+ const reader = new FileReader();
638
+ reader.onload = () => setImportText(String(reader.result || ''));
639
+ reader.readAsText(file);
640
+ e.target.value = '';
641
+ };
642
+
643
+ return (
644
+ <section className="rounded-2xl border border-slate-200 bg-white p-6 shadow-sm space-y-4">
645
+ <div className="flex flex-wrap items-center justify-between gap-2">
646
+ <div>
647
+ <div className="font-semibold text-slate-800">Skills</div>
648
+ <p className="text-sm text-slate-600">
649
+ Each skill is a real, exportable SKILL.md: name + description + instructions (always
650
+ applied), plus optional references/assets the agent can search for, scripts (documentation
651
+ only), and evals (test cases).
652
+ </p>
653
+ </div>
654
+ <div className="flex items-center gap-2">
655
+ <Button type="button" variant="outline" size="sm" className="gap-1.5" onClick={() => setImportOpen((v) => !v)}>
656
+ <Upload className="h-3.5 w-3.5" />
657
+ Import SKILL.md
658
+ </Button>
659
+ <Button type="button" size="sm" className="gap-1.5" onClick={addSkill} disabled={creating}>
660
+ {creating ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : <Plus className="h-3.5 w-3.5" />}
661
+ Add skill
662
+ </Button>
663
+ </div>
664
+ </div>
665
+
666
+ {importOpen ? (
667
+ <div className="space-y-2 rounded-xl border border-slate-100 bg-slate-50/60 p-3">
668
+ <div className="flex items-center gap-2">
669
+ <input
670
+ ref={fileInputRef}
671
+ type="file"
672
+ accept=".md,text/markdown,text/plain"
673
+ className="hidden"
674
+ onChange={handleFilePicked}
675
+ />
676
+ <Button type="button" variant="outline" size="sm" onClick={() => fileInputRef.current?.click()}>
677
+ Choose file...
678
+ </Button>
679
+ <span className="text-xs text-slate-500">or paste SKILL.md content below</span>
680
+ </div>
681
+ <Textarea
682
+ value={importText}
683
+ onChange={(e) => setImportText(e.target.value)}
684
+ placeholder={'---\nname: my-skill\ndescription: ...\n---\n\nInstructions...'}
685
+ rows={6}
686
+ className="text-xs font-mono"
687
+ />
688
+ <div className="flex justify-end gap-2">
689
+ <Button type="button" variant="ghost" size="sm" onClick={() => setImportOpen(false)}>
690
+ Cancel
691
+ </Button>
692
+ <Button type="button" size="sm" disabled={importBusy || !importText.trim()} onClick={runImport}>
693
+ {importBusy ? <Loader2 className="h-3.5 w-3.5 animate-spin" /> : null}
694
+ Import
695
+ </Button>
696
+ </div>
697
+ </div>
698
+ ) : null}
699
+
700
+ {error ? <p className="text-sm text-red-600 rounded-lg border border-red-200 bg-red-50 px-3 py-2">{error}</p> : null}
701
+
702
+ {loading ? (
703
+ <div className="flex justify-center py-8 text-slate-400">
704
+ <Loader2 className="h-8 w-8 animate-spin" />
705
+ </div>
706
+ ) : skills.length === 0 ? (
707
+ <p className="text-sm text-slate-500">No skills yet.</p>
708
+ ) : (
709
+ <div className="space-y-3">
710
+ {skills.map((s) => (
711
+ <SkillCard
712
+ key={s.id}
713
+ skill={s}
714
+ onSaved={handleSaved}
715
+ onDeleted={handleDeleted}
716
+ onError={setError}
717
+ defaultExpanded={s.id === lastCreatedId}
718
+ />
719
+ ))}
720
+ </div>
721
+ )}
722
+ </section>
723
+ );
724
+ }