ncardian commited on
Commit
6a7d00b
ยท
verified ยท
1 Parent(s): 3df43dd

Upload 3 files

Browse files
Files changed (3) hide show
  1. README.md +231 -14
  2. app.py +1185 -0
  3. requirements.txt +11 -0
README.md CHANGED
@@ -1,14 +1,231 @@
1
- ---
2
- title: Chatbot Data Anaysis Coder
3
- emoji: ๐Ÿ‘
4
- colorFrom: indigo
5
- colorTo: gray
6
- sdk: streamlit
7
- sdk_version: 1.44.1
8
- app_file: app.py
9
- pinned: false
10
- license: mit
11
- short_description: ๐Ÿš€ A web-based interactive data analysis tool using LLM mode
12
- ---
13
-
14
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # **Documentation for DataCoder AI**
2
+ **Version**: 1.0
3
+ **Last Updated**: 2024-06-15
4
+
5
+ ---
6
+
7
+ ## **1. Introduction**
8
+ **DataCoder AI** is a web-based interactive data analysis tool that combines:
9
+ - **Jupyter-like Python execution**
10
+ - **AI-powered code generation** (via Groq API)
11
+ - **Data visualization**
12
+ - **Chat-based coding assistance**
13
+
14
+ Built with `Streamlit`, `Pandas`, and `Plotly`, designed for **data scientists** and **analysts**.
15
+ Here are the **required libraries** for your DataCoder AI application, extracted from your code:
16
+
17
+ ---
18
+
19
+ ### **Core Dependencies**
20
+ **Install via `pip install -r requirements.txt`:**
21
+ ```text
22
+ streamlit==1.32.0 # Web UI framework
23
+ pandas==2.0.0 # Data manipulation
24
+ matplotlib==3.7.0 # Basic visualization
25
+ seaborn==0.12.2 # Advanced plots
26
+ plotly==5.18.0 # Interactive charts
27
+ requests==2.31.0 # API calls (Groq)
28
+ langchain==0.1.0 # LLM interface (if used)
29
+ python-dotenv==1.0.0 # Environment variables
30
+ ```
31
+
32
+ ### **Optional/Implicit Dependencies**
33
+ ```text
34
+ numpy==1.24.0 # (Auto-installed with pandas)
35
+ scipy==1.10.0 # (For stats operations)
36
+ fuzzywuzzy==0.18.0 # (For column name matching - if used)
37
+ ```
38
+
39
+ ---
40
+
41
+ ### **Notes**:
42
+ 1. **Groq API**: No Python SDK required (uses raw `requests`).
43
+ 2. **Virtual Environment**: Recommended to avoid conflicts:
44
+ ```bash
45
+ python -m venv venv
46
+ source venv/bin/activate # Linux/Mac
47
+ venv\Scripts\activate # Windows
48
+ pip install -r requirements.txt
49
+ ```
50
+
51
+ 3. **Production Considerations**:
52
+ - Add `gunicorn` for deployment:
53
+ ```text
54
+ gunicorn==20.1.0
55
+ ```
56
+ - For Windows, use `waitress`:
57
+ ```text
58
+ waitress==2.1.2
59
+ ```
60
+
61
+ Let me know if you need version adjustments or encounter dependency conflicts!
62
+ ---
63
+
64
+ ## **2. System Architecture**
65
+
66
+ ### **2.1 Component Diagram**
67
+ ```mermaid
68
+ flowchart TB
69
+ subgraph Frontend
70
+ A[Streamlit UI] --> B[File Uploader]
71
+ A --> C[Code Editor]
72
+ A --> D[Output Display]
73
+ end
74
+
75
+ subgraph Backend
76
+ B --> E[Data Processor]
77
+ C --> F[Code Executor]
78
+ C --> G[Groq LLM API]
79
+ F --> H[Session State]
80
+ end
81
+
82
+ subgraph External
83
+ G -->|API Calls| I[Groq Cloud]
84
+ end
85
+ ```
86
+
87
+ ### **2.2 Data Flow**
88
+ 1. **Input**: CSV files โ†’ Pandas DataFrame
89
+ 2. **Processing**: Merge/clean data โ†’ Modified DataFrame
90
+ 3. **Execution**: Python code โ†’ Results (text/plots/variables)
91
+ 4. **Output**: Visualizations + Export files
92
+
93
+ ---
94
+
95
+ ## **3. Installation**
96
+
97
+ ### **3.1 Prerequisites**
98
+ - Python 3.8+
99
+ - `pip` package manager
100
+
101
+ ### **3.2 Setup**
102
+ ```bash
103
+ # Clone repository
104
+ git clone https://github.com/yourrepo/datacoder-ai.git
105
+ cd datacoder-ai
106
+
107
+ # Install dependencies
108
+ pip install -r requirements.txt
109
+
110
+ # Run the app
111
+ streamlit run app.py
112
+ ```
113
+
114
+ ### **3.3 Configuration**
115
+ Rename `.env.example` to `.env` and add your Groq API key:
116
+ ```env
117
+ GROQ_API_KEY=your_api_key_here
118
+ ```
119
+
120
+ ---
121
+
122
+ ## **4. User Guide**
123
+
124
+ ### **4.1 Data Upload**
125
+ 1. Click **"Upload CSV file(s)"**
126
+ 2. Select 1+ CSV files
127
+ 3. Configure merge options:
128
+ - **Horizontal**: Combine columns (align by index)
129
+ - **Vertical**: Stack rows
130
+
131
+ ### **4.2 Data Cleaning**
132
+ | Feature | How to Use |
133
+ |------------------|----------------------------------------|
134
+ | Handle NaN | Select: Keep/Remove/Fill with mean |
135
+ | Remove Duplicates| Choose: Keep first/last/all |
136
+ | Rename Columns | Enter: `old_name:new_name` |
137
+ | Drop Columns | Enter comma-separated column names |
138
+
139
+ ### **4.3 Code Execution**
140
+ #### **Auto-Run Mode**
141
+ 1. Type request (e.g., "Plot sales distribution")
142
+ 2. Click **โ–ถ๏ธ Execute** โ†’ AI generates and runs code
143
+
144
+ #### **Manual Mode**
145
+ 1. Write/edit Python code
146
+ 2. Click **โ–ถ๏ธ Execute**
147
+
148
+ ### **4.4 Chat Assistant**
149
+ Ask questions like:
150
+ - _"How to normalize this data?"_
151
+ - _"Explain the output of this code"_
152
+
153
+ ---
154
+
155
+ ## **5. API Reference**
156
+
157
+ ### **5.1 Core Functions**
158
+ #### `execute_code(code: str, df: pd.DataFrame) -> dict`
159
+ **Parameters**:
160
+ - `code`: Python code to execute
161
+ - `df`: Target DataFrame
162
+
163
+ **Returns**:
164
+ ```python
165
+ {
166
+ "success": bool,
167
+ "stdout": str, # Console output
168
+ "figure": Figure, # Matplotlib plot
169
+ "variables": dict # New variables created
170
+ }
171
+ ```
172
+
173
+ #### `analyze_code_execution(code: str, execution_result: dict, df: pd.DataFrame) -> str`
174
+ Generates natural language analysis of results in **Indonesian**.
175
+
176
+ ### **5.2 LLM Classes**
177
+ #### `GroqLLM()`
178
+ - **Model**: `qwen-2.5-coder-32b`
179
+ - **Temperature**: 0.3
180
+ - **Purpose**: Code generation
181
+
182
+ #### `GroqTextLLM()`
183
+ - **Model**: `deepseek-r1-distill-llama-70b`
184
+ - **Temperature**: 0.7
185
+ - **Purpose**: Chat explanations
186
+
187
+ ---
188
+
189
+ ## **6. Examples**
190
+
191
+ ### **6.1 Sample Workflow**
192
+ ```python
193
+ # AI-Generated Code Example (Auto-Run)
194
+ import matplotlib.pyplot as plt
195
+ plt.figure(figsize=(10, 6))
196
+ df['sales'].plot(kind='hist', bins=20)
197
+ plt.title('Sales Distribution')
198
+ plt.show()
199
+ ```
200
+
201
+ ### **6.2 Expected Output**
202
+ 1. Histogram plot rendered
203
+ 2. Console output:
204
+ ```text
205
+ Plot generated for column: sales
206
+ ```
207
+
208
+ ---
209
+
210
+ ## **7. Troubleshooting**
211
+
212
+ | Issue | Solution |
213
+ |--------------------------------|----------------------------------------|
214
+ | Groq API errors | Check quota/network connection |
215
+ | CSV parsing fails | Verify delimiter (use `delimiter=';'`)|
216
+ | Plot not showing | Ensure `plt.show()` is called |
217
+
218
+ ---
219
+
220
+ ## **8. License & Contribution**
221
+ - **License**: MIT
222
+ - **Contribute**: Fork + PRs welcome
223
+
224
+ ---
225
+
226
+ **Notes for Production**:
227
+ 1. Replace hardcoded API keys with environment variables
228
+ 2. Add user authentication for multi-user support
229
+ 3. Implement rate limiting for Groq API calls
230
+
231
+ For full source code, visit: [GitHub Repository Link]
app.py ADDED
@@ -0,0 +1,1185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import io
2
+ import contextlib
3
+ import streamlit as st
4
+ import pandas as pd
5
+ import matplotlib.pyplot as plt
6
+ import seaborn as sns
7
+ import requests
8
+ from datetime import datetime
9
+ from langchain.llms.base import LLM
10
+ import csv
11
+ import re
12
+ import pandas as pd
13
+ from io import StringIO
14
+ from typing import Optional, List
15
+ import plotly.express as px
16
+ import re
17
+
18
+ st.set_page_config(layout="wide")
19
+ # Custom CSS for Jupyter-like appearance
20
+ st.markdown("""
21
+ <style>
22
+ .block-container {
23
+ padding-top: 1rem;
24
+ padding-bottom: 1rem;
25
+ padding-left: 0rem;
26
+ padding-right: 0rem;
27
+ }
28
+ .element-container:has(.stColumn) {
29
+ gap: 0rem !important;
30
+ }
31
+ div[data-testid="column"] {
32
+ padding-left: 0rem !important;
33
+ padding-right: 0rem !important;
34
+ }
35
+ .stApp {
36
+ max-width: 1200px;
37
+ margin: 0 auto;
38
+ padding: 1rem;
39
+ }
40
+ .cell {
41
+ border: 1px solid #e0e0e0;
42
+ border-radius: 4px;
43
+ padding: 1rem;
44
+ margin-bottom: 1.5rem;
45
+ background-color: #f9f9f9;
46
+ }
47
+ .output {
48
+ border-left: 3px solid #4285f4;
49
+ padding-left: 1rem;
50
+ margin-top: 0.5rem;
51
+ }
52
+ .input-area {
53
+ background-color: #f5f5f5;
54
+ padding: 1rem;
55
+ border-radius: 4px;
56
+ margin-bottom: 1rem;
57
+ }
58
+ .stButton > button {
59
+ background-color: #111016; /* Warna background */
60
+ color: white; /* Warna teks */
61
+ border: 1.5px solid #ffffff; /* Border dengan warna yang sama seperti background */
62
+ padding: 0.1rem 0.5rem; /* Padding untuk tombol */
63
+ border-radius: 5px; /* Membuat border bulat (round) */
64
+ font-size: 16px; /* Ukuran font */
65
+ font-weight: bold; /* Membuat teks menjadi tebal */
66
+ box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); /* Tambahkan shadow agar tombol terlihat lebih modern */
67
+ transition: all 0.3s ease; /* Transisi halus saat hover */
68
+ }
69
+
70
+ .stButton > button:hover {
71
+ background-color: #357ae8; /* Ganti background saat hover */
72
+ border-color: #357ae8; /* Ganti warna border saat hover */
73
+ box-shadow: 0 6px 12px rgba(0, 0, 0, 0.15); /* Tambahkan efek bayangan saat hover */
74
+ }
75
+ .history-item {
76
+ margin-bottom: 2rem;
77
+ box-shadow: 0 1px 3px rgba(0,0,0,0.1);
78
+ }
79
+ .chat-message {
80
+ padding: 1rem;
81
+ border-radius: 0.5rem;
82
+ margin-bottom: 1rem;
83
+ display: flex;
84
+ }
85
+ .chat-message.user {
86
+ background-color: #262626;
87
+ align-self: flex-start; /* Pesan pengguna berada di sebelah kiri */
88
+ }
89
+ .chat-message.assistant {
90
+ background-color: #262626;
91
+ align-self: flex-end; /* Pesan asisten berada di sebelah kanan */
92
+ }
93
+ .chat-message {
94
+ font-size: 14px; /* Ukuran huruf standar */
95
+ }
96
+ .chat-message.user {
97
+ font-size: 16px; /* Ukuran huruf untuk pesan pengguna */
98
+ }
99
+ .chat-message.assistant {
100
+ font-size: 14px; /* Ukuran huruf untuk pesan asisten */
101
+ }
102
+ .sidebar {
103
+ width: 300px;
104
+ padding-right: 2rem;
105
+ }
106
+ .delete-btn {
107
+ background-color: #ff4b4b !important;
108
+ color: white !important;
109
+ border: none !important;
110
+ padding: 0.25rem 0.5rem !important;
111
+ font-size: 0.8rem !important;
112
+ margin-top: 0.5rem;
113
+ }
114
+ .delete-btn:hover {
115
+ background-color: #ff0000 !important;
116
+ }
117
+ </style>
118
+ """, unsafe_allow_html=True)
119
+
120
+ # Initialize session state
121
+ if 'execution_history' not in st.session_state:
122
+ st.session_state.execution_history = []
123
+ if 'df' not in st.session_state:
124
+ st.session_state.df = None
125
+
126
+ # =======================
127
+ # Custom LLMs
128
+ # =======================
129
+ class GroqLLM(LLM):
130
+ model: str = "qwen-2.5-coder-32b"
131
+ temperature: float = 0.3
132
+ api_key: str = "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"
133
+
134
+ @property
135
+ def _llm_type(self) -> str:
136
+ return "groq-llm"
137
+
138
+ def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
139
+ headers = {
140
+ "Authorization": f"Bearer {self.api_key}",
141
+ "Content-Type": "application/json",
142
+ }
143
+
144
+ payload = {
145
+ "messages": [{"role": "user", "content": prompt}],
146
+ "model": self.model,
147
+ "temperature": self.temperature,
148
+ }
149
+
150
+ try:
151
+ response = requests.post(
152
+ "https://api.groq.com/openai/v1/chat/completions",
153
+ headers=headers,
154
+ json=payload
155
+ )
156
+ response.raise_for_status()
157
+ return response.json()["choices"][0]["message"]["content"]
158
+ except Exception as e:
159
+ st.error(f"Error calling Groq API: {str(e)}")
160
+ return "Sorry, there was an error processing your request."
161
+
162
+ class GroqTextLLM(LLM):
163
+ model: str = "deepseek-r1-distill-llama-70b"
164
+ temperature: float = 0.7
165
+ api_key: str = "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"
166
+
167
+ @property
168
+ def _llm_type(self) -> str:
169
+ return "groq-text-llm"
170
+
171
+ def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
172
+ headers = {
173
+ "Authorization": f"Bearer {self.api_key}",
174
+ "Content-Type": "application/json",
175
+ }
176
+
177
+ payload = {
178
+ "messages": [{"role": "user", "content": prompt}],
179
+ "model": self.model,
180
+ "temperature": self.temperature,
181
+ }
182
+
183
+ try:
184
+ response = requests.post(
185
+ "https://api.groq.com/openai/v1/chat/completions",
186
+ headers=headers,
187
+ json=payload
188
+ )
189
+ response.raise_for_status()
190
+ return response.json()["choices"][0]["message"]["content"]
191
+ except Exception as e:
192
+ st.error(f"Error calling Groq API: {str(e)}")
193
+ return "Sorry, there was an error processing your request."
194
+
195
+ # Initialize LLMs
196
+ code_llm = GroqLLM()
197
+ text_llm = GroqTextLLM()
198
+
199
+
200
+ def analyze_code_execution(code: str, execution_result: dict, df: pd.DataFrame) -> str:
201
+ """
202
+ Analyze only the execution results (output and variables) to provide focused insights.
203
+
204
+ Args:
205
+ code: The Python code that was executed
206
+ execution_result: The dictionary containing execution results
207
+ df: The DataFrame being analyzed (for context only)
208
+
209
+ Returns:
210
+ str: Natural language analysis focused on the execution results
211
+ """
212
+ # Extract relevant information from execution_result
213
+ output = execution_result.get('stdout', 'Tidak ada output teks')
214
+ variables = execution_result.get('variables', {})
215
+ error = execution_result.get('error', None)
216
+
217
+ # Prepare variable summaries
218
+ var_summaries = []
219
+ for var_name, var_value in variables.items():
220
+ if isinstance(var_value, (pd.DataFrame, pd.Series)):
221
+ var_summaries.append(f"- {var_name}: {type(var_value).__name__} dengan bentuk {var_value.shape}")
222
+ elif isinstance(var_value, (plt.Figure, sns.axisgrid.Grid)):
223
+ var_summaries.append(f"- {var_name}: Visualisasi plot")
224
+ else:
225
+ var_summaries.append(f"- {var_name}: {type(var_value).__name__}")
226
+
227
+ # Prepare the prompt for the text model
228
+ prompt = f"""
229
+ Kamu adalah analis data yang menjelaskan hasil eksekusi kode Python kepada pemangku kepentingan non-teknis.
230
+
231
+ Hasil eksekusi yang perlu dianalisis:
232
+ 1. Output teks:
233
+ {output}
234
+
235
+ 2. Variabel yang dihasilkan:
236
+ {chr(10).join(var_summaries) if var_summaries else 'Tidak ada variabel baru dibuat'}
237
+
238
+ 3. Error (jika ada):
239
+ {error if error else 'Tidak ada error'}
240
+
241
+ analisis juga data berikut:
242
+ - Success: {execution_result['success']}
243
+ - Output: {execution_result.get('stdout', 'No output')}
244
+ - Figure: {execution_result.get('figure', 'None')}
245
+ - Error: {execution_result.get('error', 'None')}
246
+ - Variables created: {list(execution_result.get('variables', {}).keys())}
247
+ - {code}
248
+
249
+ Dataframe shape: {df.shape}
250
+
251
+ Berikan insight lebih lanjut berdasarkan seluruh kolom data yang ada pada DataFrame. Fokus pada:
252
+ - Fokuslah untuk membahas hasil eksekusi
253
+ - Jangan menganalisis dari data yg tidak ada
254
+ - Temuan atau pola menarik yang muncul dari data
255
+ - Hubungan atau korelasi antara kolom-kolom dalam data
256
+ - Faktor yang dapat mempengaruhi hasil analisis lebih lanjut
257
+ - Saran untuk pengolahan atau analisis lebih lanjut yang dapat dilakukan pada data ini
258
+ - Jangan membahas kode Python-nya
259
+ - Berikan insight secara singkat dan padat algoritma yang perlu digunakan seperti pada data science dan AI beserta kegunaannya terhadap analisis
260
+ - Gunakan bahasa Indonesia yang mudah dipahami
261
+ - Hanya gunakan kata ganti orang ketiga
262
+ """
263
+
264
+ # Get analysis from the text model
265
+ llm = GroqTextLLM()
266
+ response = llm(prompt)
267
+
268
+ # Bersihkan output untuk menghilangkan bagian yang tidak diinginkan
269
+ clean_response = re.sub(r'<\s*think\s*>.*?<\s*/\s*think\s*>', '', response, flags=re.DOTALL | re.IGNORECASE)
270
+
271
+ return clean_response
272
+
273
+
274
+
275
+ # =======================
276
+ # Code Execution Functions
277
+ # =======================
278
+
279
+ # =======================
280
+ def execute_code(code: str, df: pd.DataFrame):
281
+ """Execute Python code safely and return outputs"""
282
+ # Extract Python code from markdown blocks
283
+ if "```python" in code:
284
+ clean_code = code.split("```python")[1].split("```")[0].strip()
285
+ else:
286
+ clean_code = code.strip()
287
+
288
+ # Add required imports if missing
289
+ required_imports = [
290
+ "import pandas as pd",
291
+ "import matplotlib.pyplot as plt",
292
+ "import seaborn as sns"
293
+ ]
294
+
295
+ for imp in required_imports:
296
+ if imp not in clean_code:
297
+ clean_code = imp + "\n" + clean_code
298
+
299
+ # Prepare execution environment
300
+ local_vars = {
301
+ 'df': df,
302
+ 'plt': plt,
303
+ 'sns': sns,
304
+ 'pd': pd
305
+ }
306
+
307
+ # Capture outputs
308
+ stdout = io.StringIO()
309
+ stderr = io.StringIO()
310
+ fig = None
311
+
312
+ try:
313
+ with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
314
+ exec(clean_code, local_vars)
315
+
316
+ # Check for plots
317
+ if plt.gcf().get_axes():
318
+ fig = plt.gcf()
319
+
320
+ return {
321
+ 'code': clean_code,
322
+ 'stdout': stdout.getvalue(),
323
+ 'stderr': stderr.getvalue(),
324
+ 'figure': fig,
325
+ 'success': True,
326
+ 'variables': {k: v for k, v in local_vars.items()
327
+ if not k.startswith('_') and k not in ['df', 'plt', 'sns', 'pd']}
328
+ }
329
+
330
+ except Exception as e:
331
+ return {
332
+ 'code': clean_code,
333
+ 'error': str(e),
334
+ 'stderr': stderr.getvalue(),
335
+ 'success': False
336
+ }
337
+
338
+ def add_to_history(execution_result):
339
+ """Add execution result to history with timestamp"""
340
+ history_item = {
341
+ **execution_result,
342
+ 'timestamp': datetime.now().strftime("%Y-%m-%d %H:%M:%S")
343
+ }
344
+ st.session_state.execution_history.append(history_item)
345
+
346
+ # Generate explanation for successful executions
347
+ if execution_result['success']:
348
+ with st.spinner("Menganalisis hasil..."):
349
+ df_to_analyze = st.session_state.df_cleaned if 'df_cleaned' in st.session_state else st.session_state.df
350
+
351
+ analysis = analyze_code_execution(
352
+ code=execution_result['code'],
353
+ execution_result=execution_result,
354
+ df=df_to_analyze # Only used for context, not analyzed
355
+ )
356
+
357
+ history_item['explanation'] = f"""
358
+ **Analisis Hasil Eksekusi:**
359
+ {analysis}
360
+ """
361
+
362
+
363
+
364
+ # =======================
365
+ # Display Functions
366
+ # =======================
367
+
368
+ def display_code_with_highlight(code: str):
369
+ """Display formatted Python code using Streamlit's built-in code block"""
370
+ st.code(code, language='python')
371
+
372
+
373
+
374
+ def display_history():
375
+ """Display all execution history items"""
376
+ st.markdown("## Execution History")
377
+
378
+ if not st.session_state.execution_history:
379
+ st.info("No executions yet. Run some code to see results here.")
380
+ return
381
+
382
+ for i, item in enumerate(reversed(st.session_state.execution_history)):
383
+ with st.container():
384
+ st.markdown(f"### Execution #{len(st.session_state.execution_history)-i}")
385
+ st.caption(f"Executed at {item['timestamp']}")
386
+
387
+ # Display code
388
+ with st.expander("View Code", expanded=False):
389
+ display_code_with_highlight(item['code'])
390
+
391
+ # Display outputs
392
+ if item['success']:
393
+ if 'explanation' in item:
394
+ with st.expander("See Explanation"):
395
+ st.markdown("**Explanation:**")
396
+ st.info(item['explanation'])
397
+
398
+ if item['stdout']:
399
+ with st.expander("View Output", expanded=False):
400
+ st.markdown("**Output:**")
401
+ lines = item['stdout'].split('\n')
402
+ text_content = []
403
+ table_data = []
404
+ header_detected = False
405
+
406
+ # Iterasi melalui setiap baris untuk memisahkan teks dan tabel
407
+ for idx, line in enumerate(lines):
408
+ cleaned_line = line.strip()
409
+ if not cleaned_line:
410
+ continue
411
+
412
+ # Deteksi header tabel
413
+ if not header_detected:
414
+ if re.match(r"^[\w\s_]+$", cleaned_line) and len(cleaned_line.split()) > 1:
415
+ # Periksa baris berikutnya untuk memastikan ini adalah tabel
416
+ if (idx + 1 < len(lines)) and re.match(r"^\d+\s+[\d\.e+-]+", lines[idx+1].strip()):
417
+ header_detected = True
418
+ table_data = lines[idx:]
419
+ break
420
+ else:
421
+ text_content.append(cleaned_line)
422
+ else:
423
+ text_content.append(cleaned_line)
424
+
425
+ # Tampilkan konten teks
426
+ if text_content:
427
+ st.write("\n".join(text_content))
428
+
429
+ # Coba parsing dan tampilkan tabel jika ada
430
+ if table_data:
431
+ try:
432
+ # Bersihkan data untuk DataFrame
433
+ clean_table = [line.strip() for line in table_data if line.strip()]
434
+ df = pd.read_csv(StringIO("\n".join(clean_table)), sep=r"\s+", engine="python")
435
+ st.dataframe(df)
436
+ except Exception as e:
437
+ st.write("\n".join(table_data))
438
+
439
+
440
+ if item['figure']:
441
+ st.markdown("**Visualization:**")
442
+ fig = item['figure']
443
+ st.pyplot(fig)
444
+ plt.close()
445
+
446
+
447
+ if item['variables']:
448
+ with st.expander("Created Variables"):
449
+ st.json({k: str(type(v)) for k, v in item['variables'].items()})
450
+ else:
451
+ st.error("Execution failed")
452
+ st.error(item['error'])
453
+ if item['stderr']:
454
+ st.text("Error details:")
455
+ st.text(item['stderr'])
456
+
457
+ st.markdown("---")
458
+
459
+ # =======================
460
+ # Main App
461
+ # =======================
462
+ # Main app layout
463
+ col1a, col2a = st.columns([4, 7], gap="small")
464
+ st.markdown("<div style='margin-top: 5px'></div>", unsafe_allow_html=True)
465
+ show_col1 = st.toggle("Ask chat bot", value=True)
466
+ # Atur rasio kolom berdasarkan kondisi
467
+ if show_col1:
468
+ col1a, col2a = st.columns([4,7]) # Keduanya muncul
469
+ else:
470
+ col2a, _ = st.columns([1,0.0001,]) # Kolom 2 disembunyikan, col1 melebar penuh
471
+
472
+ with col1a:
473
+ if show_col1:
474
+ with st.container(height=570):
475
+ st.header("๐Ÿ’ฌ Coding Assistant")
476
+
477
+ # Inisialisasi state khusus untuk chatbot jika belum ada
478
+ if 'chatbot' not in st.session_state:
479
+ st.session_state.chatbot = {
480
+ 'chat_history': [],
481
+ 'prev_df': None,
482
+ 'prev_raw_dataframes': None
483
+ }
484
+
485
+ # Tombol hapus semua chat
486
+ if st.button("๐Ÿ—‘๏ธ Clear all chat", key="clear_all_chat"):
487
+ st.session_state.chatbot['chat_history'] = []
488
+
489
+ # Tampilkan chat history
490
+ # Tampilkan chat history per pasangan user-assistant
491
+ for i in range(0, len(st.session_state.chatbot['chat_history']), 2):
492
+ user_msg = st.session_state.chatbot['chat_history'][i]
493
+ assistant_msg = st.session_state.chatbot['chat_history'][i+1] if i+1 < len(st.session_state.chatbot['chat_history']) else None
494
+
495
+ with st.expander(f"Chat {i//2 + 1}", expanded=False):
496
+ with st.chat_message("user"):
497
+ st.markdown(user_msg["content"])
498
+ if assistant_msg:
499
+ with st.chat_message("assistant"):
500
+ st.markdown(assistant_msg["content"])
501
+
502
+
503
+ # Input chat
504
+ user_query = st.chat_input("Ask coding questions...", key="chatbot_input")
505
+
506
+ if user_query:
507
+ # Simpan state sebelumnya khusus untuk chatbot
508
+ st.session_state.chatbot['prev_df'] = st.session_state.get('df')
509
+ st.session_state.chatbot['prev_raw_dataframes'] = st.session_state.get('raw_dataframes')
510
+
511
+ # Tampilkan pesan user langsung
512
+ with st.chat_message("user"):
513
+ st.markdown(user_query)
514
+
515
+ # Tambahkan ke history
516
+ st.session_state.chatbot['chat_history'].append({"role": "user", "content": user_query})
517
+
518
+ with st.spinner("Thinking..."):
519
+ prompt = f"""
520
+ Anda adalah seorang analyst. Jawab pertanyaan berikut dengan jelas
521
+
522
+ Pertanyaan: {user_query}
523
+
524
+ Ketentuan jawaban:
525
+
526
+ 1. Gunakan bahasa Indonesia yang formal
527
+ 2. Jangan sertakan contoh kode
528
+ 3. Format kode dalam blok code
529
+ 4. Jelaskan istilah teknis dengan analogi sederhana
530
+ 5. Selalu berikan tag untuk proses berpikir anda
531
+ 6. Langsung jawab pada intinya
532
+ 7. Hanya tuliskan poin-poinnya saja
533
+ 8 Jawablah sesingkat mungkin
534
+ """
535
+ response = text_llm(prompt)
536
+ clean_response = re.sub(r'<\s*think\s*>.*?<\s*/\s*think\s*>', '', response, flags=re.DOTALL | re.IGNORECASE)
537
+
538
+ # Tampilkan dan simpan respon
539
+ with st.chat_message("assistant"):
540
+ st.markdown(clean_response)
541
+ st.session_state.chatbot['chat_history'].append({"role": "assistant", "content": clean_response})
542
+
543
+ with col2a: # Sidebar untuk chatbot
544
+ with st.container(height=570):
545
+ st.title("๐Ÿ“Š Data Analysis Powered by AI Assistance")
546
+ st.markdown("Analyze your CSV data with LLM Models")
547
+
548
+ def reset_data_state():
549
+ st.session_state.df = None
550
+ st.session_state.raw_dataframes = None
551
+ st.session_state.preview_df = None
552
+ st.session_state.file_names = [f.name for f in uploaded_files] if uploaded_files else []
553
+
554
+ # Initialize session state if not already present
555
+ if 'df' not in st.session_state:
556
+ st.session_state.df = None
557
+
558
+ st.header("CSV File Uploader")
559
+
560
+ uploaded_files = st.file_uploader(
561
+ "Upload CSV file(s)",
562
+ type=["csv"],
563
+ accept_multiple_files=True,
564
+ help="Upload one or more CSV files to combine"
565
+ )
566
+
567
+ if uploaded_files:
568
+ # Reset processing if files change
569
+ if 'prev_uploaded_files' not in st.session_state or st.session_state.prev_uploaded_files != [f.name for f in uploaded_files]:
570
+ st.session_state.df = None
571
+ st.session_state.raw_dataframes = None
572
+ st.session_state.preview_df = None
573
+ st.session_state.prev_uploaded_files = [f.name for f in uploaded_files]
574
+
575
+ # Process files but don't merge yet
576
+ if st.session_state.raw_dataframes is None:
577
+ dataframes = []
578
+ processed_files = 0
579
+
580
+ for uploaded_file in uploaded_files:
581
+ with st.expander(f"๐Ÿ“„ File: {uploaded_file.name}", expanded=False):
582
+ try:
583
+ # Try to detect delimiter
584
+ raw_data = uploaded_file.getvalue().decode("utf-8")
585
+ sniffer = csv.Sniffer()
586
+
587
+ try:
588
+ dialect = sniffer.sniff(raw_data.splitlines()[0])
589
+ delimiter = dialect.delimiter
590
+ except (csv.Error, IndexError):
591
+ # Try common delimiters if sniffing fails
592
+ for test_delim in [',', ';', '\t', '|']:
593
+ if test_delim in raw_data:
594
+ delimiter = test_delim
595
+ break
596
+ else:
597
+ delimiter = ',' # default
598
+
599
+ uploaded_file.seek(0) # Reset file pointer
600
+
601
+ # Read CSV with error handling
602
+ try:
603
+ df = pd.read_csv(uploaded_file, delimiter=delimiter, on_bad_lines='warn')
604
+ except Exception as e:
605
+ st.warning(f"Using Python engine for {uploaded_file.name} due to parsing issues")
606
+ df = pd.read_csv(uploaded_file, delimiter=delimiter, engine='python')
607
+
608
+ # Display file info
609
+ st.write(f"Shape: {df.shape}")
610
+ st.write(f"Detected delimiter: '{delimiter}'")
611
+ st.dataframe(df.head(5), use_container_width=True)
612
+
613
+ dataframes.append(df)
614
+ processed_files += 1
615
+
616
+ except Exception as e:
617
+ st.error(f"Error processing {uploaded_file.name}: {str(e)}")
618
+ continue
619
+
620
+ if processed_files == 0:
621
+ st.error("No files were successfully processed")
622
+ st.stop()
623
+
624
+ # Store raw dataframes in session state
625
+ st.session_state.raw_dataframes = dataframes
626
+
627
+ # Only proceed if we have raw dataframes
628
+ if st.session_state.raw_dataframes:
629
+ dataframes = st.session_state.raw_dataframes
630
+
631
+ st.subheader("Merge Configuration")
632
+ col1, col2 = st.columns([3, 1])
633
+
634
+ with col1:
635
+ merge_option = st.radio(
636
+ "Merge method:",
637
+ ("single data","Horizontal (concat columns)", "Vertical (concat rows)", "Horizontal (merge on common columns)"),
638
+ index=0,
639
+ horizontal=True,
640
+ key="merge_option"
641
+ )
642
+
643
+ # Additional merge parameters
644
+ if merge_option == "Horizontal (merge on common columns)":
645
+ if len(dataframes) > 0:
646
+ common_cols = set(dataframes[0].columns)
647
+ for df in dataframes[1:]:
648
+ common_cols.intersection_update(df.columns)
649
+
650
+ if common_cols:
651
+ selected_cols = st.multiselect(
652
+ "Select columns to merge on:",
653
+ options=list(common_cols),
654
+ default=list(common_cols),
655
+ key="merge_columns"
656
+ )
657
+ how_merge = st.selectbox(
658
+ "Merge type:",
659
+ ["inner", "outer", "left", "right"],
660
+ index=1,
661
+ key="how_merge"
662
+ )
663
+ else:
664
+ None
665
+
666
+ with col2:
667
+ st.write("") # Spacer
668
+ st.write("") # Spacer
669
+ preview_merge = st.button("๐Ÿ”„ Preview")
670
+ confirm_merge = st.button("โœ… Confirm")
671
+ reset_btn = st.button("๐Ÿ”„ Reset")
672
+
673
+ # Reset logic
674
+ if reset_btn:
675
+ reset_data_state()
676
+ st.rerun()
677
+
678
+ # Preview logic
679
+ if preview_merge:
680
+ with st.spinner("Generating merge preview..."):
681
+ try:
682
+ if merge_option == "single data":
683
+ preview_df = dataframes
684
+ elif merge_option == "Horizontal (concat columns)":
685
+ # Find common indices across ALL dataframes
686
+ common_indices = dataframes[0].index
687
+ for df in dataframes[1:]:
688
+ common_indices = common_indices.intersection(df.index)
689
+
690
+ if len(common_indices) == 0:
691
+ st.error("No common indices found across all dataframes")
692
+ st.stop()
693
+
694
+ # Filter all dataframes to only common indices
695
+ filtered_dfs = [df.loc[common_indices] for df in dataframes]
696
+
697
+ # Handle duplicate column names
698
+ all_columns = []
699
+ duplicate_counter = {}
700
+
701
+ for i, df in enumerate(filtered_dfs):
702
+ new_columns = []
703
+ for col in df.columns:
704
+ if col in all_columns:
705
+ if col not in duplicate_counter:
706
+ duplicate_counter[col] = 1
707
+ duplicate_counter[col] += 1
708
+ new_col = f"{col}_df{duplicate_counter[col]}"
709
+ new_columns.append(new_col)
710
+ else:
711
+ new_columns.append(col)
712
+ all_columns.extend(new_columns)
713
+ filtered_dfs[i].columns = new_columns
714
+
715
+ # Concatenate horizontally
716
+ preview_df = pd.concat(filtered_dfs, axis=1)
717
+
718
+ elif merge_option == "Vertical (concat rows)":
719
+ preview_df = pd.concat(dataframes, axis=0, ignore_index=True)
720
+
721
+ elif merge_option == "Horizontal (merge on common columns)":
722
+ def auto_merge_many(dfs):
723
+ if not dfs:
724
+ return None
725
+
726
+ merged_df = dfs[0]
727
+
728
+ for next_df in dfs[1:]:
729
+ common_keys = list(set(merged_df.columns) & set(next_df.columns))
730
+ if common_keys:
731
+ merged_df = pd.merge(merged_df, next_df, on=common_keys)
732
+ else:
733
+ print(f"[WARNING] Tidak ada key yang cocok antara:\n{merged_df.columns}\n&\n{next_df.columns}")
734
+
735
+ return merged_df
736
+
737
+ preview_df = auto_merge_many(dataframes)
738
+
739
+ st.session_state.preview_df = preview_df
740
+ st.success("Merge preview generated!")
741
+
742
+ # Show preview stats
743
+ st.subheader("Preview Results")
744
+ st.write(f"Shape: {preview_df.shape}")
745
+
746
+ # Show sample data with tabs
747
+ tab1, tab2 = st.tabs(["First Rows", "Last Rows"])
748
+ with tab1:
749
+ st.dataframe(preview_df.head(20), use_container_width=True)
750
+ with tab2:
751
+ st.dataframe(preview_df.tail(20), use_container_width=True)
752
+
753
+ # Show quick stats
754
+ with st.expander("Preview Statistics"):
755
+ st.write("Column types:")
756
+ st.dataframe(preview_df.dtypes.astype(str).reset_index().rename(
757
+ columns={'index': 'Column', 0: 'DataType'}
758
+ ))
759
+
760
+ st.write("Missing values:")
761
+ missing = preview_df.isnull().sum()
762
+ missing = missing[missing > 0].reset_index().rename(
763
+ columns={'index': 'Column', 0: 'Missing Count'}
764
+ )
765
+ if len(missing) > 0:
766
+ st.dataframe(missing)
767
+ else:
768
+ st.success("No missing values found!")
769
+
770
+ except Exception as e:
771
+ st.error(f"Merge preview failed: {str(e)}")
772
+ st.stop()
773
+
774
+ # Confirm merge logic
775
+ if confirm_merge and st.session_state.preview_df is not None:
776
+ st.session_state.df = st.session_state.preview_df
777
+ st.session_state.preview_df = None # Clear preview after confirmation
778
+ st.success("Merge confirmed! Data is now available for analysis.")
779
+ st.balloons()
780
+ st.rerun()
781
+
782
+ # Show raw data info
783
+ with st.expander("๐Ÿ“ฆ Raw Data Summary", expanded=False):
784
+ st.write(f"Total files loaded: {len(dataframes)}")
785
+ for i, df in enumerate(dataframes, 1):
786
+ st.write(f"### Dataframe {i}")
787
+ st.write(f"- Shape: {df.shape}")
788
+ st.write("- Columns:")
789
+ st.dataframe(pd.DataFrame({
790
+ 'Column': df.columns,
791
+ 'Type': df.dtypes.values,
792
+ 'Missing %': (df.isnull().mean() * 100).round(2)
793
+ }), hide_index=True)
794
+
795
+ if st.checkbox(f"Show sample data for Dataframe {i}", key=f"show_raw_{i}"):
796
+ st.dataframe(df.head(5), use_container_width=True)
797
+
798
+ # If merge is already confirmed, show the final data
799
+ if st.session_state.df is not None:
800
+ st.subheader("๐Ÿ” Merged Data Analysis")
801
+
802
+ # Data cleaning options (only after merge is confirmed)
803
+ with st.expander("๐Ÿงน Data Cleaning", expanded=True):
804
+ # Inisialisasi df_cleaned di session state jika belum ada
805
+
806
+ st.session_state.df_cleaned = st.session_state.df.copy()
807
+
808
+ clean_options = st.columns(4)
809
+
810
+ with clean_options[0]:
811
+ clean_nan = st.selectbox(
812
+ "Handle NaN values:",
813
+ ["Keep", "Remove rows", "Fill with zero", "Fill with mean"],
814
+ index=0,
815
+ key="clean_nan"
816
+ )
817
+
818
+ if st.button("Apply NaN Handling"):
819
+ if clean_nan == "Fill with zero":
820
+ st.session_state.df_cleaned = st.session_state.df_cleaned.fillna(0)
821
+ st.success("Filled NaN with zeros!")
822
+ elif clean_nan == "Fill with mean":
823
+ st.session_state.df_cleaned = st.session_state.df_cleaned.fillna(st.session_state.df_cleaned.mean())
824
+ st.success("Filled NaN with mean values!")
825
+ elif clean_nan == "Remove rows":
826
+ initial_rows = len(st.session_state.df_cleaned)
827
+ st.session_state.df_cleaned = st.session_state.df_cleaned.dropna()
828
+ removed_nan = initial_rows - len(st.session_state.df_cleaned)
829
+ st.success(f"Removed {removed_nan} rows with NaN values!")
830
+
831
+ with clean_options[1]:
832
+ clean_duplicates = st.selectbox(
833
+ "Handle duplicates:",
834
+ ["Keep", "Remove all", "Keep first", "Keep last"],
835
+ index=0,
836
+ key="clean_duplicates"
837
+ )
838
+
839
+ if st.button("Apply Deduplication"):
840
+ if clean_duplicates != "Keep":
841
+ initial_rows = len(st.session_state.df_cleaned)
842
+ if clean_duplicates == "Remove all":
843
+ st.session_state.df_cleaned = st.session_state.df_cleaned.drop_duplicates(keep=False)
844
+ elif clean_duplicates == "Keep first":
845
+ st.session_state.df_cleaned = st.session_state.df_cleaned.drop_duplicates(keep='first')
846
+ elif clean_duplicates == "Keep last":
847
+ st.session_state.df_cleaned = st.session_state.df_cleaned.drop_duplicates(keep='last')
848
+
849
+ removed_dups = initial_rows - len(st.session_state.df_cleaned)
850
+ st.success(f"Removed {removed_dups} duplicate rows!")
851
+
852
+ with clean_options[2]:
853
+ col_rename = st.text_input("Rename (e.g., 'old:new')")
854
+ if st.button("Rename Column") and col_rename and ":" in col_rename:
855
+ old, new = col_rename.split(":", 1)
856
+ if old in st.session_state.df_cleaned.columns:
857
+ st.session_state.df_cleaned = st.session_state.df_cleaned.rename(columns={old: new})
858
+ st.success(f"Renamed column '{old}' to '{new}'")
859
+ else:
860
+ st.warning(f"Column '{old}' not found")
861
+
862
+ with clean_options[3]:
863
+ # Fitur Drop Columns
864
+ cols_to_drop = st.text_input(
865
+ "Drop columns",
866
+ placeholder="column1, column2, ...",
867
+ key="drop_cols_input"
868
+ )
869
+
870
+ if st.button("Drop Columns"):
871
+ if cols_to_drop:
872
+ # Proses input: hilangkan spasi, split by comma, dan bersihkan
873
+ columns_list = [col.strip() for col in cols_to_drop.split(",") if col.strip()]
874
+
875
+ # Fungsi untuk mencocokkan nama kolom dengan fleksibilitas
876
+ def find_matching_column(col_name, df_columns):
877
+ col_name = col_name.lower().strip()
878
+ for df_col in df_columns:
879
+ if df_col.lower().strip() == col_name:
880
+ return df_col
881
+ return None
882
+
883
+ # Cari kolom yang cocok
884
+ matched_cols = []
885
+ not_found = []
886
+
887
+ for col in columns_list:
888
+ matched = find_matching_column(col, st.session_state.df_cleaned.columns)
889
+ if matched:
890
+ matched_cols.append(matched)
891
+ else:
892
+ not_found.append(col)
893
+
894
+ # Drop kolom yang ditemukan
895
+ if matched_cols:
896
+ initial_cols = st.session_state.df_cleaned.columns.tolist()
897
+ st.session_state.df_cleaned = st.session_state.df_cleaned.drop(columns=matched_cols)
898
+ st.success(f"Successfully dropped columns: {', '.join(matched_cols)}")
899
+
900
+ # Tampilkan warning untuk kolom yang tidak ditemukan
901
+ if not_found:
902
+ st.warning(f"Columns not found: {', '.join(not_found)}")
903
+
904
+ # Tampilkan perubahan
905
+ st.write(f"Remaining columns: {len(st.session_state.df_cleaned.columns)}")
906
+ else:
907
+ st.error("No matching columns found to drop")
908
+ else:
909
+ st.warning("Please enter column names to drop")
910
+
911
+ # Tampilkan info missing values setelah cleaning
912
+ missing_values = st.session_state.df_cleaned.isnull().sum().sum()
913
+ if missing_values > 0:
914
+ st.warning(f"โš ๏ธ Warning: There are still {missing_values} missing values in the data")
915
+ if st.button("Show Missing Values Details"):
916
+ missing_df = st.session_state.df_cleaned.isnull().sum()
917
+ missing_df = missing_df[missing_df > 0].reset_index()
918
+ missing_df.columns = ['Column', 'Missing Count']
919
+ st.dataframe(missing_df)
920
+ else:
921
+ st.success("โœ… No missing values remaining!")
922
+
923
+ # Final data display
924
+ st.session_state.df = st.session_state.df_cleaned
925
+ st.subheader("๐Ÿ“Š Final Data")
926
+ st.write(f"Shape: {st.session_state.df.shape}")
927
+
928
+ # Interactive data explorer
929
+ tab1, tab2, tab3, tab4 = st.tabs(["Data View", "Statistics", "Column Analysis", 'Type Data'])
930
+
931
+ with tab1:
932
+ num_rows = st.slider(
933
+ "Number of rows to display:",
934
+ min_value=5,
935
+ max_value=100,
936
+ value=20,
937
+ key="num_rows_view"
938
+ )
939
+ st.dataframe(st.session_state.df.head(num_rows), use_container_width=True)
940
+
941
+ with tab2:
942
+ st.write("### Descriptive Statistics")
943
+ st.dataframe(st.session_state.df.describe(include='all'), use_container_width=True)
944
+
945
+
946
+ with tab3:
947
+ selected_col = st.selectbox(
948
+ "Select column to analyze:",
949
+ st.session_state.df.columns,
950
+ key="col_analyze"
951
+ )
952
+
953
+ col_data = st.session_state.df[selected_col]
954
+ st.write(f"**Type:** {col_data.dtype}")
955
+ st.write(f"**Unique values:** {len(col_data.unique())}")
956
+ st.write(f"**Missing values:** {col_data.isnull().sum()} ({col_data.isnull().mean()*100:.2f}%)")
957
+
958
+ if col_data.nunique() < 10:
959
+ st.write("**Distribution**")
960
+ column_name = col_data.name # Nama kolom
961
+
962
+ # Masukkan col_data ke DataFrame agar px.histogram bisa membaca kolomnya
963
+ df_temp = pd.DataFrame({column_name: col_data})
964
+
965
+ fig = px.histogram(
966
+ df_temp,
967
+ x=column_name,
968
+ opacity=0.7,
969
+ color_discrete_sequence=["#636EFA"], # Opsional: warna
970
+ )
971
+
972
+ fig.update_traces(marker_line_color='black', marker_line_width=1)
973
+
974
+ fig.update_layout(
975
+ template='seaborn',
976
+ title=f"Histogram of {column_name}",
977
+ xaxis_title=column_name,
978
+ yaxis_title="Count", # Ganti dari "Density"
979
+ bargap=0.1,
980
+ font=dict(family="Verdana", size=13),
981
+ )
982
+
983
+ st.plotly_chart(fig, use_container_width=True)
984
+
985
+ else:
986
+ st.write("**Value counts (top 20):**")
987
+ st.dataframe(col_data.value_counts().head(20))
988
+
989
+ with tab4:
990
+ buffer = io.StringIO()
991
+ st.session_state.df.info(buf=buffer)
992
+ st.text(buffer.getvalue())
993
+
994
+ st.markdown("**Data Types:**")
995
+ st.json(st.session_state.df.dtypes.astype(str).to_dict())
996
+
997
+ st.markdown("**Missing Values:**")
998
+ st.json(st.session_state.df.isnull().sum().to_dict())
999
+
1000
+ st.markdown("**Duplicate:**")
1001
+ duplicate_count = st.session_state.df.duplicated().sum()
1002
+ st.write(f"Jumlah duplikat: {duplicate_count}")
1003
+
1004
+ # Menampilkan jumlah duplikat dalam format JSON yang benar
1005
+ st.json({"duplicate_count": duplicate_count})
1006
+
1007
+
1008
+ # Export options
1009
+ with st.expander("๐Ÿ’พ Export Data", expanded=False):
1010
+ export_format = st.radio(
1011
+ "Export format:",
1012
+ ["CSV", "Excel", "JSON"],
1013
+ horizontal=True
1014
+ )
1015
+
1016
+ export_filename = st.text_input("Filename (without extension)", "clean_data")
1017
+
1018
+ if st.button("Export Data"):
1019
+ try:
1020
+ if export_format == "CSV":
1021
+ csv = st.session_state.df.to_csv(index=False).encode('utf-8')
1022
+ st.download_button(
1023
+ label="Download CSV",
1024
+ data=csv,
1025
+ file_name=f"{export_filename}.csv",
1026
+ mime='text/csv'
1027
+ )
1028
+ elif export_format == "Excel":
1029
+ excel_buffer = io.BytesIO()
1030
+ st.session_state.df.to_excel(excel_buffer, index=False)
1031
+ st.download_button(
1032
+ label="Download Excel",
1033
+ data=excel_buffer,
1034
+ file_name=f"{export_filename}.xlsx",
1035
+ mime='application/vnd.ms-excel'
1036
+ )
1037
+ elif export_format == "JSON":
1038
+ json_str = st.session_state.df.to_json(orient='records', indent=2)
1039
+ st.download_button(
1040
+ label="Download JSON",
1041
+ data=json_str,
1042
+ file_name=f"{export_filename}.json",
1043
+ mime='application/json'
1044
+ )
1045
+ st.success("Export ready!")
1046
+ except Exception as e:
1047
+ st.error(f"Export failed: {str(e)}")
1048
+
1049
+ # Reset all button
1050
+ if st.button("๐Ÿ”„ Reset All Data", type="secondary"):
1051
+ st.session_state.df = None
1052
+ st.session_state.df_cleaned = None
1053
+ st.session_state.raw_dataframes = None
1054
+ st.session_state.preview_df = None
1055
+ st.rerun()
1056
+
1057
+ # Input area
1058
+ st.subheader("Code Input")
1059
+ input_method = st.radio("Input method:", ["Auto run code","Manual run code"])
1060
+
1061
+ user_input = st.text_area("Enter your Python code or question:", height=150,
1062
+ placeholder="Write code here or ask a question about your data...",
1063
+ key="user_input")
1064
+
1065
+ def generate_prompt(user_input, df_columns):
1066
+ """Generate optimized prompt for code generation"""
1067
+ return f"""
1068
+ Generate Python code to analyze pandas DataFrame 'df' according to request:
1069
+ {user_input}
1070
+
1071
+ STRICT RULES (NUMBERED PRIORITY):
1072
+ 1. CODE STRUCTURE:
1073
+ - Return complete executable code in ```python block
1074
+ - All new variables MUST be added as new columns to existing 'df' using df['new_column'] syntax
1075
+ - Never add a new variable (such as a list, Series, or array) to df if its length is less than the number of rows in df.
1076
+ - Never create new DataFrames or use merge/join operations
1077
+ - Print calculation result (such as a list, Series, or array) in console
1078
+ - Define all temporary variables explicitly
1079
+
1080
+ 2. DATA HANDLING:
1081
+ - Use only columns from: {list(df_columns)}
1082
+ - For new data storage:
1083
+ if isinstance(result, (pd.Series, list, np.ndarray)):
1084
+ df[new_col_name] = result
1085
+ elif numerical/string value:
1086
+ df[new_col_name] = [value]*len(df)
1087
+ - For column matching:
1088
+ from fuzzywuzzy import process
1089
+ def find_best_match(query, cols, threshold=70):
1090
+ # [existing fuzzy match implementation]
1091
+
1092
+ 3. COLUMN CREATION RULES:
1093
+ a. Untuk hasil operasi matematika:
1094
+ df['hasil_kalkulasi'] = df['col1'] + df['col2']
1095
+
1096
+ b. Untuk transformasi data:
1097
+ df['kategori_baru'] = df['col_existing'].apply(lambda x: x*2)
1098
+
1099
+ c. Untuk agregasi grup:
1100
+ df['rata_grup'] = df.groupby('kolom_grup')['target'].transform('mean')
1101
+
1102
+ d. Untuk hasil statistik:
1103
+ df['zscore'] = (df['nilai'] - df['nilai'].mean())/df['nilai'].std()
1104
+
1105
+ 4. VISUALIZATION:
1106
+ - Visualisasi harus menggunakan kolom dari df yang telah dimodifikasi
1107
+ - Contoh: plt.plot(df['kolom_baru'], df['kolom_lama'])
1108
+
1109
+ 5. OUTPUT HANDLING:
1110
+ - Untuk menampilkan kolom baru:
1111
+ print(f"Kolom baru yang dibuat:", list(df.columns[-N:])) # N = jumlah kolom baru
1112
+ print(df[['kolom_baru_1', 'kolom_baru_2']].head())
1113
+
1114
+ 6. ERROR PREVENTION:
1115
+ - Cek panjang data sebelum membuat kolom baru:
1116
+ if len(variable) != len(df):
1117
+ raise ValueError("Panjang variabel tidak sesuai dengan DataFrame")
1118
+ - Gunakan nama kolom unik:
1119
+ if 'nama_kolom' in df.columns:
1120
+ df['nama_kolom_rev'] = ... # tambahkan suffix jika sudah ada
1121
+
1122
+ FINAL CHECK:
1123
+ 1. Pastikan tidak ada operasi merge/join/concat
1124
+ 2. Verifikasi semua hasil kalkulasi ada di df sebagai kolom
1125
+ 3. Cek konsistensi indeks untuk kolom baru
1126
+ 4. Pastikan print terakhir menampilkan df dengan kolom baru
1127
+ 5. Jangan menampilkan penjelasan apapun dari kode
1128
+ 6. Jika user menyebut 'pivot' gunakan fungsi groupby
1129
+ """
1130
+
1131
+ # Mode handling
1132
+ if input_method == "Auto run code":
1133
+ col1, col2, col3 = st.columns(3)
1134
+ with col1: execute_btn = st.button("โ–ถ๏ธ Execute")
1135
+ with col2: clear_btn = st.button("๐Ÿ—‘๏ธ Clear History")
1136
+ with col3: show_vars = st.button("๐Ÿ“Š Show Variables")
1137
+ # Common button handling
1138
+ if clear_btn:
1139
+ st.session_state.execution_history = []
1140
+
1141
+ if show_vars and st.session_state.execution_history:
1142
+ latest_vars = st.session_state.execution_history[-1].get('variables', {})
1143
+ st.json({k: str(type(v)) for k, v in latest_vars.items()})
1144
+ if execute_btn and user_input:
1145
+ with st.spinner("Generating code with AI..."):
1146
+ generated_code = code_llm(generate_prompt(user_input, st.session_state.df.columns.tolist()))
1147
+ st.markdown("**Generated Code:**")
1148
+ st.code(generated_code, language='python')
1149
+
1150
+ result = execute_code(generated_code, st.session_state.df)
1151
+ add_to_history(result)
1152
+ st.rerun()
1153
+
1154
+ elif input_method == "Manual run code":
1155
+ col1, col2, col3, col4 = st.columns(4)
1156
+ with col1: code_btn = st.button("โ–ถ๏ธ Code")
1157
+ with col2: execute_btn = st.button("โ–ถ๏ธ Execute")
1158
+ with col3: clear_btn = st.button("๐Ÿ—‘๏ธ Clear History")
1159
+ with col4: show_vars = st.button("๐Ÿ“Š Show Variables")
1160
+
1161
+ if clear_btn:
1162
+ st.session_state.execution_history = []
1163
+
1164
+ if show_vars and st.session_state.execution_history:
1165
+ latest_vars = st.session_state.execution_history[-1].get('variables', {})
1166
+ st.json({k: str(type(v)) for k, v in latest_vars.items()})
1167
+
1168
+ if code_btn and user_input:
1169
+ with st.spinner("Membuat kode dengan AI..."):
1170
+ generated_code = code_llm(generate_prompt(user_input, st.session_state.df.columns.tolist()))
1171
+ st.session_state.generated_code = generated_code
1172
+ st.markdown("**Generated Code:**")
1173
+ st.code(generated_code, language='python')
1174
+
1175
+ if execute_btn:
1176
+ if 'generated_code' in st.session_state:
1177
+ result = execute_code(st.session_state.generated_code, st.session_state.df)
1178
+ add_to_history(result)
1179
+ elif user_input:
1180
+ result = execute_code(user_input, st.session_state.df)
1181
+ add_to_history(result)
1182
+ st.rerun()
1183
+
1184
+ # Display execution history
1185
+ display_history()
requirements.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ streamlit==1.32.0 # Web UI framework
2
+ pandas==2.0.0 # Data manipulation
3
+ matplotlib==3.7.0 # Basic visualization
4
+ seaborn==0.12.2 # Advanced plots
5
+ plotly==5.18.0 # Interactive charts
6
+ requests==2.31.0 # API calls (Groq)
7
+ langchain==0.1.0 # LLM interface (if used)
8
+ python-dotenv==1.0.0 # Environment variables
9
+ numpy==1.24.0 # (Auto-installed with pandas)
10
+ scipy==1.10.0 # (For stats operations)
11
+ fuzzywuzzy==0.18.0 # (For column name matching - if used)