Spaces:
Running
Running
File size: 12,946 Bytes
3ff49ac f7936e0 3ff49ac | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 | <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>A Project to quality check the submitted articles</title>
<style>
body {
font-family: Arial, sans-serif;
line-height: 1.6;
margin: 20px;
}
h1, h2 {
color: #333;
}
h2 {
margin-top: 30px;
}
ul {
list-style-type: disc;
margin-left: 20px;
}
p {
margin-bottom: 15px;
}
table{
border-collapse: collapse;
width: 95%;
border: 2px solid #2c3e50;
}
tr{
border-bottom: 2px solid #b60e0e;
}
td{
width: 15%;
vertical-align: top;
border: 2px solid #3498db;
}
</style>
</head>
<body>
<h2>Overview of the project</h2>
<p>
In this project we will evaluate the quality of all the articles submitted by the content writers.
All the articles are fetched based on a time window (last 3 days, last one week etc). Llamaindex workflow will be utilized
to check each article for its structure, any missing elements are there in the article and give an overall feedback.
We use a pydantic schema to predefine the structure and llm will be using the sturcture predict to evaluate.
A final report is created.
</p>
<h2>Here is the design, we will examine two approaches</h2>
<p>
First approach, each is a seperate task
<pre>
+----------------------------------+
| User Time Window |
| (e.g., "Last 24 Hours") |
+----------------+-----------------+
|
v
+----------------+-----------------+
| CMS System |
| (Fetch Articles in Time Window) |
+----------------+-----------------+
|
v
+-------------------------------------------------------------------+
| LLAMAINDEX WORKFLOW |
| |
| +-----------------------+ +--------------------+ |
| | Start Event | | Router / Dispatch | |
| | (Incoming CMS Data) +------------->| (Parallel Execution) | |
| +-----------------------+ +---------+----------+ |
| | |
| +-------------------+----------------------+ |
| | | | |
| v v v |
| +--------------+ +--------------+ +-------------------+ |
| | Task 1: | | Task 2: | | Task 3: | |
| | Structure | | Wellformed | | Content Quality | |
| | Analysis | | Validation | | Assessor | |
| +------+-------+ +------+-------+ +---------+---------+ |
| | | | |
| +-------------------+----------------------+ |
| | (Emit Partial Results) |
| v |
| +--------------------+ |
| | Gather / Step | |
| | (Collects 3 Tasks) | |
| +----------+---------+ |
| | |
| v |
| +--------------------+ |
| | Stop / Report Step| |
| | (Synthesize Data) | |
| +----------+---------+ |
+-----------------------------|-------------------------------------+
|
v
+--------------+-------------------+
| Final Quality Report |
| (Structure, Format & Insights) |
+----------------------------------+
</pre>
Second approach is all the tasks are clubbed to one single workflow step task.
<pre>
+----------------------------------+
| User Time Window |
| (e.g., "Last 24 Hours") |
+----------------+-----------------+
|
v
+----------------+-----------------+
| CMS System |
| (Fetch Articles in Time Window) |
+----------------+-----------------+
|
v
+-------------------------------------------------------------------+
| SINGLE-STEP STRUCTURED PREDICT |
| |
| +-----------------------+ +--------------------+ |
| | Raw Article Data +------------->| LLM Context | |
| +-----------------------+ | (Single Payload) | |
| +---------+----------+ |
| | |
| v |
| +--------------------+ |
| |llm.structured_predict |
| +---------+----------+ |
| | |
| v |
| +-----------------+------------------+
| | Unified Pydantic Schema Validation|
| | [Structure + Format + Quality] |
| +-----------------+------------------+
+----------------------------------------------------|--------------+
|
v
+-------------------------------------+------------+
| Final Article Report |
| (Instantly validated JSON / Typed Python Object) |
+--------------------------------------------------+
</pre>
Let's begin building the project. Code is organized as below.<br>
<pre>
├── article_schema.py --> Pydantic Schema for the structure
├── audit_workflow.py
├── create_report.py
├── fetch_articles.py --> Fetches the articles from a CMS
</pre>
<pre>
from pydantic import BaseModel, Field
from typing import List, Optional
#consider an article with an id, title, description fields
class ArticleStructure(BaseModel):
collectionId: str = Field(description="The unique identifier of the article.")
title: str = Field(short_summary="Check if title is engaging and properly cased.")
well_formed: bool = Field(description="Overall structural integrity of the article.")
missing_elements: Optional[List[str]] = Field(description="List of structural elements that are missing, if any.")
feedback: str = Field(description="Actionable feedback for the author.")
</pre>
Test the schema, by creating some dummy data point.<br>
<pre>
dummy_article = ArticleStructure(collectionId="1",title="some_thing", well_formed="yes", missing_elements=[], feedback="something")
print(dummy_article)
</pre>
Now, let's get the ball rolling and fetch some articles from a CMS system. If you don't have one, you may use some fastapi to implement a dummy article repository.
<br> or create a static set of text articles for testing the flow.
<pre>
import requests
from datetime import datetime, timedelta, timezone
def fetch_recent_articles(pocketbase_url: str):
# Calculate time threshold (e.g., 3 hours ago)
time_threshold = datetime.now(timezone.utc) - timedelta(days=50)
time_string = time_threshold.isoformat().replace("+00:00", "Z")
# Query PocketBase API, giving the full path where articles reside
filter_query = f"updated >= '{time_string}'"
url = f"{pocketbase_url}/api/collections/nature_posts/records"
params = {"filter": filter_query}
response = requests.get(url, params=params)
response.raise_for_status()
return response.json().get("items", [])
</pre>
Now, let's write a test audit program, we will use a locally hosted llm using openAI like. <br>
<pre>
import json
from llama_index.llms.openai_like import OpenAILike
import article_schema as asc
from llama_index.core.prompts import PromptTemplate
#creating a dummy article to test it
custom_llm = OpenAILike(model="gemma", api_base="http://localhost:8080/v1", api_key="not_needed", is_chat_model=True)
prompt = f"""
Analyze the following article for well-formed structure, tone, and readability.
CollectionId: {'something'}
Title: {'Rose, a beautiful plant'}
Content: {'Roses are very beautiful plants and grow in different colors. They are great addition to any garden! Roses are popular woody shrubs that belong to the genus Rosa.They grow colorful flowers in shades of red, pink, yellow, and white.Their stems have sharp prickles, commonly called thorns, to protect against animals.Most rose varieties produce a sweet and pleasant fragrance.'}
"""
# Use structured output to force the Pydantic schema evaluation
response = custom_llm.structured_predict(
output_cls=asc.ArticleStructure,
prompt=PromptTemplate(prompt)
)
print(response)
</pre>
If everything goes well, you should be able to see something like this. <br>
<pre>
collectionId='CollectionId' title='Rose, a beautiful plant' well_formed=True missing_elements=[] feedback='The article is well-formed with a clear structure. It follows a simple, direct narrative. The title is engaging and appropriately formatted. The content is concise and informative, providing a basic overview of roses. There are no missing elements.'
</pre>
Expand on the concept and implement a simple workflow using llamaindex to complete the quality check of the contents of the articles.<br>
<b>Full code will be presented at a later point of time!!</b>
</p>
</body>
</html> |