---
license: cc-by-nc-4.0
tags:
- agent
- chemistry
- environment
---
## Model Overview
This model is an **Contaminants of Emerging Concern Annotation Intelligent Agent** built on the Dify platform, integrated with the Norman knowledge base, Pubchemlite_exposomics database, and Invitrodb_v4.3 database. It enables high-throughput, large-scale annotation of emerging contaminants, including **usage classification** and **toxicity endpoints** by inputting the IUPAC name of the target contaminant.

## Model Purpose
To construct a specialized knowledge database for emerging contaminants usage classification, which combines multi-source chemical/toxicological databases and AI agents. The core goals are:
1. Realize fast and large-scale annotation of emerging contaminants' usage categories.
2. Provide efficient query services for toxicity endpoints.
3. Support high-throughput data analysis scenarios for emerging contaminants in environmental chemistry and toxicology research.

## Key Definitions
| Term | Definition |
|------|------------|
| **System** | Refers to the *Contaminants of Emerging Concern Annotation Intelligent Agent* |
| **User** | Anyone authorized to use the functions of this system |
| **IUPAC Name** | A systematic naming convention formulated by IUPAC for accurately describing the composition and structure of chemical substances |
| **AI Agent** | A system based on large language models (LLMs) that understands user intentions and invokes multiple tools to solve complex tasks; in this system, it accepts IUPAC names of emerging contaminants and outputs usage classification and toxicity endpoints |
| **Norman Database** | A network for monitoring and evaluating environmental pollutants, facilitating European and international cooperation and data sharing in environmental pollution monitoring; classifies over 100,000 chemicals |
| **Pubchemlite_exposomics Database** | An open-source organic molecule information database derived from PubChem, applicable for mass spectrometry analysis and non-targeted identification of unknown pollutants |
| **Invitrodb_v4.3 Database** | The core database of US EPA ToxCast, storing a large amount of biological activity data, analysis workflows, and metadata of compounds generated by high-throughput screening (HTS) |

## Quick Start
1. Install Navicat on your computer and upload the **invitrodb_v4_3.sql** and **pubchemlite_exposomics_20251226.sql** databases.
2. Deploy Docker and Dify. Set the Docker engine according to **dockerengine_setting.txt**.
3. Create a custom tool named **sql_executor_pubchemlite_invitrodb_en** on the Dify platform. The detailed information for the schema tool is stored in **schema_tool.txt**. Note that you need to modify the URL address to your local address.
4. Train the knowledge base on the FastGPT platform. The knowledge base file is **knowledge_database_input_iupac.csv**.  
5. Deploy the FDA **docker-compose.yml** and import the external knowledge base into Dify.
6. On the Dify platform, select Import DSL File and upload the **dify_CECs_annotating.yml**.
7. Before using this agent, you need to run Docker and the **step1_pubchemlite_invitrodb_to_dify_en.py** code. Note that in **step1_pubchemlite_invitrodb_to_dify_en.py**, the DB_CONFIGS needs to be modified to the username and password of your local SQL database.
8. Once the setup is complete, you can start running.
9. For batch queries, you can run the mini-program **step2_CECs_annotating_agent_v1.0.py**.

## System Architecture & Components
### Core Databases Deployment
The system integrates three core databases with differentiated deployment strategies:
1. **Norman Chemical Classification Database**
   - Serves as a relational knowledge base, uploaded and parsed on the FastGPT platform, then embedded into the Dify platform.
   - Optimized classification: Integrated or removed redundant categories, finally categorized chemicals into **9 classes**.
    <img src="figure/Norman_category.png" alt="Norman_category" width="400">
2. **Pubchemlite_exposomics & Invitrodb_v4.3 Databases**
   - Deployed in local SQL databases to support efficient local query and invocation.
   - Query workflow: GPT-4o generates SQL statements → Extract valid SQL queries → Backend executes database queries and returns results.

### Agent Workflow Design (Dify Chatflow)
1. **Base Model**: GPT-4o is used to generate SQL query statements and organize output data in JSON format for subsequent data extraction.
2. **Custom Schema Tool**
   - Created on the Dify platform to standardize SQL statement generation and API invocation logic.
   - Implementation steps: Create custom tool → Configure tool name and Schema rules (see schema_tool.txt for details).
3. **Knowledge Base Integration (FastGPT + Dify)**
   - **FastGPT Knowledge Base Construction**
     1. Log in to FastGPT (https://fastgpt.aiown.top/) and enter the main interface.
     2. Import dataset (50,000+ chemicals with IUPAC names and categories from the Norman database).
     3. Connect the dataset to a FastGPT application and configure prompts (consistent with Few-shot prompts).
     4. Publish the application and export the API key for subsequent calls.
   - **FastGPT-Dify Adaptor (FDA) Plugin**
     - Resolves API incompatibility between FastGPT and Dify.
     - Deployment steps: Create `docker-compose.yml` for FDA → Run `docker-compose up -d` in the configuration file directory to deploy the plugin.
   - **Dify External Knowledge Base Connection**: Link the trained FastGPT knowledge base to Dify by importing the FastGPT API key and knowledge base ID.
<div align="center">
<img src="figure/pipeline.jpg" alt="Norman_category" width="800">
</div>


## CECs BatchAnnotator (Desktop Version)
For chemists/researchers to automate batch compound info retrieval from CSV files (default column: `IUPAC_name`). Outputs standardized CSVs, failure logs, and raw API records for analysis/debugging. Requires valid Dify API key and accessible backend.
<div align="center">
<img src="figure/CECs BatchAnnotator v1.0.png" alt="Norman_category" width="400">
</div>

## System Requirements
| Category | Specification |
|----------|---------------|
| Operating System | Windows 10 |
| Python Version | Python 3.8 or higher |
| Dependencies | Docker, FastGPT access rights, Dify platform account |

## Usage Instructions
### Preparations
1. Complete Norman database classification optimization (merge into 9 categories).
2. Deploy FDA plugin to connect FastGPT and Dify.
3. Import the Norman dataset(knowledge_database_input_iupac.csv) into FastGPT, configure the application, and export the API key.
4. Create a custom Schema tool on Dify and configure SQL invocation rules.
5. Deploy Pubchemlite_exposomics (PubChemLite_exposomics_20251226.csv) and Invitrodb_v4.3 databases to local SQL and test query connectivity.
6. Create a backend program (step1_pubchemlite_invitrodb_to_dify_en.py) to connect Dify and the SQL databases.

### Inference Workflow
1. Input the **IUPAC name** of the emerging contaminant into the Dify chat interface.
2. The AI agent invokes:
   - FastGPT knowledge base for **usage classification** via FDA plugin.
   - Local SQL databases for **toxicity endpoints** via GPT-4o-generated SQL queries.
3. Receive the structured output (JSON format) containing usage category and toxicity endpoints.

## Limitations
1. The accuracy of annotations depends on the completeness of the Norman, Pubchemlite_exposomics, and Invitrodb_v4.3 databases; unrecorded emerging contaminants may return empty results.
2. Requires local deployment of SQL databases and FDA plugins, which has a certain threshold for environment configuration.
3. Currently only supports input of **IUPAC names**; other naming formats (e.g., common names) are not supported.

## Contact
laquh1086@163.com