| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - agent |
| - chemistry |
| - environment |
| --- |
| ## Model Overview |
| This model is an **Contaminants of Emerging Concern Annotation Intelligent Agent** built on the Dify platform, integrated with the Norman knowledge base, Pubchemlite_exposomics database, and Invitrodb_v4.3 database. It enables high-throughput, large-scale annotation of emerging contaminants, including **usage classification** and **toxicity endpoints** by inputting the IUPAC name of the target contaminant. |
|
|
| ## Model Purpose |
| To construct a specialized knowledge database for emerging contaminants usage classification, which combines multi-source chemical/toxicological databases and AI agents. The core goals are: |
| 1. Realize fast and large-scale annotation of emerging contaminants' usage categories. |
| 2. Provide efficient query services for toxicity endpoints. |
| 3. Support high-throughput data analysis scenarios for emerging contaminants in environmental chemistry and toxicology research. |
|
|
| ## Key Definitions |
| | Term | Definition | |
| |------|------------| |
| | **System** | Refers to the *Contaminants of Emerging Concern Annotation Intelligent Agent* | |
| | **User** | Anyone authorized to use the functions of this system | |
| | **IUPAC Name** | A systematic naming convention formulated by IUPAC for accurately describing the composition and structure of chemical substances | |
| | **AI Agent** | A system based on large language models (LLMs) that understands user intentions and invokes multiple tools to solve complex tasks; in this system, it accepts IUPAC names of emerging contaminants and outputs usage classification and toxicity endpoints | |
| | **Norman Database** | A network for monitoring and evaluating environmental pollutants, facilitating European and international cooperation and data sharing in environmental pollution monitoring; classifies over 100,000 chemicals | |
| | **Pubchemlite_exposomics Database** | An open-source organic molecule information database derived from PubChem, applicable for mass spectrometry analysis and non-targeted identification of unknown pollutants | |
| | **Invitrodb_v4.3 Database** | The core database of US EPA ToxCast, storing a large amount of biological activity data, analysis workflows, and metadata of compounds generated by high-throughput screening (HTS) | |
|
|
| ## Quick Start |
| 1. Install Navicat on your computer and upload the **invitrodb_v4_3.sql** and **pubchemlite_exposomics_20251226.sql** databases. |
| 2. Deploy Docker and Dify. Set the Docker engine according to **dockerengine_setting.txt**. |
| 3. Create a custom tool named **sql_executor_pubchemlite_invitrodb_en** on the Dify platform. The detailed information for the schema tool is stored in **schema_tool.txt**. Note that you need to modify the URL address to your local address. |
| 4. Train the knowledge base on the FastGPT platform. The knowledge base file is **knowledge_database_input_iupac.csv**. |
| 5. Deploy the FDA **docker-compose.yml** and import the external knowledge base into Dify. |
| 6. On the Dify platform, select Import DSL File and upload the **dify_CECs_annotating.yml**. |
| 7. Before using this agent, you need to run Docker and the **step1_pubchemlite_invitrodb_to_dify_en.py** code. Note that in **step1_pubchemlite_invitrodb_to_dify_en.py**, the DB_CONFIGS needs to be modified to the username and password of your local SQL database. |
| 8. Once the setup is complete, you can start running. |
| 9. For batch queries, you can run the mini-program **step2_CECs_annotating_agent_v1.0.py**. |
| |
| ## System Architecture & Components |
| ### Core Databases Deployment |
| The system integrates three core databases with differentiated deployment strategies: |
| 1. **Norman Chemical Classification Database** |
| - Serves as a relational knowledge base, uploaded and parsed on the FastGPT platform, then embedded into the Dify platform. |
| - Optimized classification: Integrated or removed redundant categories, finally categorized chemicals into **9 classes**. |
| <img src="figure/Norman_category.png" alt="Norman_category" width="400"> |
| 2. **Pubchemlite_exposomics & Invitrodb_v4.3 Databases** |
| - Deployed in local SQL databases to support efficient local query and invocation. |
| - Query workflow: GPT-4o generates SQL statements → Extract valid SQL queries → Backend executes database queries and returns results. |
| |
| ### Agent Workflow Design (Dify Chatflow) |
| 1. **Base Model**: GPT-4o is used to generate SQL query statements and organize output data in JSON format for subsequent data extraction. |
| 2. **Custom Schema Tool** |
| - Created on the Dify platform to standardize SQL statement generation and API invocation logic. |
| - Implementation steps: Create custom tool → Configure tool name and Schema rules (see schema_tool.txt for details). |
| 3. **Knowledge Base Integration (FastGPT + Dify)** |
| - **FastGPT Knowledge Base Construction** |
| 1. Log in to FastGPT (https://fastgpt.aiown.top/) and enter the main interface. |
| 2. Import dataset (50,000+ chemicals with IUPAC names and categories from the Norman database). |
| 3. Connect the dataset to a FastGPT application and configure prompts (consistent with Few-shot prompts). |
| 4. Publish the application and export the API key for subsequent calls. |
| - **FastGPT-Dify Adaptor (FDA) Plugin** |
| - Resolves API incompatibility between FastGPT and Dify. |
| - Deployment steps: Create `docker-compose.yml` for FDA → Run `docker-compose up -d` in the configuration file directory to deploy the plugin. |
| - **Dify External Knowledge Base Connection**: Link the trained FastGPT knowledge base to Dify by importing the FastGPT API key and knowledge base ID. |
| <div align="center"> |
| <img src="figure/pipeline.jpg" alt="Norman_category" width="800"> |
| </div> |
| |
| |
| ## CECs BatchAnnotator (Desktop Version) |
| For chemists/researchers to automate batch compound info retrieval from CSV files (default column: `IUPAC_name`). Outputs standardized CSVs, failure logs, and raw API records for analysis/debugging. Requires valid Dify API key and accessible backend. |
| <div align="center"> |
| <img src="figure/CECs BatchAnnotator v1.0.png" alt="Norman_category" width="400"> |
| </div> |
|
|
| ## System Requirements |
| | Category | Specification | |
| |----------|---------------| |
| | Operating System | Windows 10 | |
| | Python Version | Python 3.8 or higher | |
| | Dependencies | Docker, FastGPT access rights, Dify platform account | |
|
|
| ## Usage Instructions |
| ### Preparations |
| 1. Complete Norman database classification optimization (merge into 9 categories). |
| 2. Deploy FDA plugin to connect FastGPT and Dify. |
| 3. Import the Norman dataset(knowledge_database_input_iupac.csv) into FastGPT, configure the application, and export the API key. |
| 4. Create a custom Schema tool on Dify and configure SQL invocation rules. |
| 5. Deploy Pubchemlite_exposomics (PubChemLite_exposomics_20251226.csv) and Invitrodb_v4.3 databases to local SQL and test query connectivity. |
| 6. Create a backend program (step1_pubchemlite_invitrodb_to_dify_en.py) to connect Dify and the SQL databases. |
|
|
| ### Inference Workflow |
| 1. Input the **IUPAC name** of the emerging contaminant into the Dify chat interface. |
| 2. The AI agent invokes: |
| - FastGPT knowledge base for **usage classification** via FDA plugin. |
| - Local SQL databases for **toxicity endpoints** via GPT-4o-generated SQL queries. |
| 3. Receive the structured output (JSON format) containing usage category and toxicity endpoints. |
|
|
| ## Limitations |
| 1. The accuracy of annotations depends on the completeness of the Norman, Pubchemlite_exposomics, and Invitrodb_v4.3 databases; unrecorded emerging contaminants may return empty results. |
| 2. Requires local deployment of SQL databases and FDA plugins, which has a certain threshold for environment configuration. |
| 3. Currently only supports input of **IUPAC names**; other naming formats (e.g., common names) are not supported. |
|
|
| ## Contact |
| laquh1086@163.com |
