Clean learned topics and persist deduped knowledge

app.py

@@ -55,7 +55,10 @@ def load_topics():
 
 
 def persist_topics():
+    global learned_topics
     with IO_LOCK:
+        normalized = ["dictionary words" if str(x).startswith("dictionary words") else x for x in learned_topics]
+        learned_topics = list(dict.fromkeys(normalized))
         with open(TOPICS_FILE, "w") as f:
             json.dump(learned_topics, f)
 
@@ -603,7 +606,7 @@ def learn_dictionary():
         if status:
             subprocess.run(["git", "-C", ROOT, "commit", "-m", f"Agron dictionary learning ({len(lines)} words)"], check=True)
             subprocess.run(["git", "-C", ROOT, "push", "origin", "main"], check=True)
-        learned_topics.append(f"dictionary words ({len(lines)})")
+        learned_topics.append("dictionary words")
         persist_topics()
         training_state["last"] = f"dictionary learned & pushed ({len(lines)} words)"
     except Exception as e:

memory/interactions.jsonl

@@ -31,3 +31,4 @@
 {"ts": 1792062087.812921, "user": "Can you search the web for PDMS PDT file import?", "assistant": "Here are live web results:\n1. To Create Imports for PDMS/S3D - Bentley Systems \u2014 Thewizard advances to the Mapping step. AsPDMS/S3D doesn't require any mappingfile,youcanskip the Mapping step. Tip: For non-OpenPlant schemaimports,youcanuse the mappingfileto create functional data by using the EC-SQL Query configuration. See Mapping.jason Configuration. Click Next. In theImportName field, enter a name. (https://docs.bentley.com/LiveContent/web/PlantSight%20Enterprise%20Help-v1/en/GUID-53FA37DF-CF72-4F1E-B04F-87740C9D3578.html)\n2. To Create Imports for PDMS/S3D - docs.bentley.com \u2014 Thewizard advances to the Mapping step. AsPDMS/S3D doesn't require any mappingfile,youcanskip the Mapping step. Tip: For non-OpenPlant schemaimports,youcanuse the mappingfileto create functional data by using the EC-SQL Query configuration. See Mapping.jason Configuration. Click Next. In theImportName field, enter a name. (https://docs.bentley.com/LiveContent/web/PlantSight%20PS3-vLatest/Help/en/topics/2821533/GUID-53FA37DF-CF72-4F1E-B04F-87740C9D3578.html)\n3. Import using PML - help.aveva.com \u2014 It is possible toimportP&IDs in non-graphics (tty) mode using PML. This enables batch scripts to executeimportswithout user interaction. For example: 1. Create a folder called TestImport in yourPDMS(or Marine) installation folder. 2. (https://help.aveva.com/AVEVA_Diagrams/14.1.4/SCHMM/SCHMM11.12.17.html)\n4. PDMS V11.6 Data Import Instructions | PDF | Text File - Scribd \u2014 1. The document provides instructions for importing different types of data intoPDMSV11.6 using various modules: 2. Data is imported using modules LEXICON, PROPCON, PARAGON, and SPECON throughimportfilesthat create and populate worlds with attributes, properties, templates, and specifications. 3. It describes starting each module, navigating to the correct database, running theimport... (https://www.scribd.com/document/332512363/WorkingInstruction-Complete-Data-Import-PDMS-V116-20130724)\n5. Import PDMS data - Intergraph Smart 3D - Help \u2014 What do you want to do?ImportPDMSdata Validate mapping details for importing Add thePDMSinterfaces to the catalog (https://docs.hexagonali.com/r/en-US/Intergraph-Smart-3D-Common/14/1004511)"}
 {"ts": 1792062125.5431235, "user": "How to import a PDT file into PDMS?", "assistant": "This article provides a detailed, step-by-step guide to importing 3D models intoPDMS, discusses common challenges and solutions, and offers tips to optimize your workflow for better accuracy and productivity. (source: PDFimport 3d models to pdms - staging.friends-library.org)"}
 {"ts": 1792062154.4077501, "user": "Tell me a short joke.", "assistant": "Here\u2019s a fun one-liner I found on the web:\n\nWhy don\u2019t scientists trust atoms?\n\nBecause they make up everything! \n\nThis is a classic short, easy-to-memorize joke suitable for all audiences. Enjoy sharing it!"}
+{"ts": 1792065110.0476015, "user": "what have you learned", "assistant": "I've learned from you and my training sources: 33 conversation exchanges (latest: 'Tell me a short joke.'); learned topics: base text model (WikiText-2), conversation memory, dictionary words (145), dictionary words (147), dictionary words (146), dictionary words (145), dictionary words (145), web: transformer architecture."}

memory/learned_topics.json

@@ -1 +1,6 @@
-["base text model (WikiText-2)", "conversation memory", "dictionary words (145)", "dictionary words (147)", "dictionary words (146)", "dictionary words (145)"]
+[
+  "base text model (WikiText-2)",
+  "conversation memory",
+  "dictionary words",
+  "web: transformer architecture"
+]