Articles_Quality_Check.html

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>A Project to quality check the submitted articles</title>
        <style>
            body {
                font-family: Arial, sans-serif;
                line-height: 1.6;
                margin: 20px;
            }
            h1, h2 {
                color: #333;
            }
            h2 {
                margin-top: 30px;
            }
            ul {
                list-style-type: disc;
                margin-left: 20px;
            }
            p {
                margin-bottom: 15px;
            }

            table{
                border-collapse: collapse;
                width: 95%;
                border: 2px solid #2c3e50;
            }
            
            tr{
                border-bottom: 2px solid #b60e0e;
            }

            td{
                width: 15%;
                vertical-align: top;
                border: 2px solid #3498db;
            }
        </style>
    </head>
    <body>
        <h2>Overview of the project</h2>
        <p>
            In this project we will evaluate the quality of all the articles submitted by the content writers.
            All the articles are fetched based on a time window (last 3 days, last one week etc). Llamaindex workflow will be utilized
            to check each article for its structure, any missing elements are there in the article and give an overall feedback.
            We use a pydantic schema to predefine the structure and llm will be using the sturcture predict to evaluate.
            A final report is created.
        </p>
        <h2>Here is the design, we will examine two approaches</h2>
        <p>
            First approach, each is a seperate task
            <pre>
                +----------------------------------+

               |        User Time Window          |
               |     (e.g., "Last 24 Hours")      |
               +----------------+-----------------+
                                |
                                v
               +----------------+-----------------+

               |          CMS System              |
               | (Fetch Articles in Time Window)  |
               +----------------+-----------------+
                                |
                                v
                +-------------------------------------------------------------------+

                |                     LLAMAINDEX WORKFLOW                           |
                |                                                                   |
                |   +-----------------------+              +--------------------+   |
                |   |    Start Event        |              |  Router / Dispatch |   |
                |   | (Incoming CMS Data)   +------------->|   (Parallel Execution) |   |
                |   +-----------------------+              +---------+----------+   |
                |                                                    |              |
                |         +-------------------+----------------------+              |
                |         |                   |                      |              |
                |         v                   v                      v              |
                |  +--------------+    +--------------+    +-------------------+    |
                |  |    Task 1:   |    |    Task 2:   |    |      Task 3:      |    |
                |  |  Structure   |    | Wellformed   |    | Content Quality   |    |
                |  |   Analysis   |    |  Validation  |    |     Assessor      |    |
                |  +------+-------+    +------+-------+    +---------+---------+    |
                |         |                   |                      |              |
                |         +-------------------+----------------------+              |
                |                             | (Emit Partial Results)              |
                |                             v                                     |
                |                  +--------------------+                           |
                |                  |    Gather / Step   |                           |
                |                  | (Collects 3 Tasks) |                           |
                |                  +----------+---------+                           |
                |                             |                                     |
                |                             v                                     |
                |                  +--------------------+                           |
                |                  |  Stop / Report Step|                           |
                |                  |  (Synthesize Data) |                           |
                |                  +----------+---------+                           |
                +-----------------------------|-------------------------------------+
                              |
                              v
               +--------------+-------------------+

               |       Final Quality Report       |
               |  (Structure, Format & Insights)  |
               +----------------------------------+
   
            </pre>
            Second approach is all the tasks are clubbed to one single workflow step task.
            <pre>
                +----------------------------------+

               |        User Time Window          |
               |     (e.g., "Last 24 Hours")      |
               +----------------+-----------------+
                                |
                                v
               +----------------+-----------------+

               |          CMS System              |
               | (Fetch Articles in Time Window)  |
               +----------------+-----------------+
                                |
                                v
                +-------------------------------------------------------------------+

                |                  SINGLE-STEP STRUCTURED PREDICT                   |
                |                                                                   |
                |   +-----------------------+              +--------------------+   |
                |   |   Raw Article Data    +------------->|    LLM Context     |   |
                |   +-----------------------+              |   (Single Payload) |   |
                |                                          +---------+----------+   |
                |                                                    |              |
                |                                                    v              |
                |                                          +--------------------+   |
                |                                          |llm.structured_predict  |
                |                                          +---------+----------+   |
                |                                                    |              |
                |                                                    v              |
                |                                  +-----------------+------------------+

                |                                  |   Unified Pydantic Schema Validation|
                |                                  |  [Structure + Format + Quality]    |
                |                                  +-----------------+------------------+
                +----------------------------------------------------|--------------+
                                    |
                                    v
               +-------------------------------------+------------+

               |               Final Article Report               |
               | (Instantly validated JSON / Typed Python Object) |
               +--------------------------------------------------+   
            </pre>
            Let's begin building the project. Code is organized as below.<br>
            <pre>
                ├── article_schema.py --> Pydantic Schema for the structure
                ├── audit_workflow.py
                ├── create_report.py
                ├── fetch_articles.py --> Fetches the articles from a CMS
            </pre>
            <pre>
                from pydantic import BaseModel, Field
                from typing import List, Optional
                #consider an article with an id, title, description fields
                class ArticleStructure(BaseModel):
                    collectionId: str = Field(description="The unique identifier of the article.")                   
                    title: str = Field(short_summary="Check if title is engaging and properly cased.")                   
                    well_formed: bool = Field(description="Overall structural integrity of the article.")
                    missing_elements: Optional[List[str]] = Field(description="List of structural elements that are missing, if any.")
                    feedback: str = Field(description="Actionable feedback for the author.")
            </pre>
            Test the schema, by creating some dummy data point.<br>
            <pre>
                dummy_article = ArticleStructure(collectionId="1",title="some_thing", well_formed="yes", missing_elements=[], feedback="something")
                print(dummy_article)
            </pre>
            Now, let's get the ball rolling and fetch some articles from a CMS system. If you don't have one, you may use some fastapi to implement a dummy article repository.
            <br> or create a static set of text articles for testing the flow.
            <pre>
                import requests
                from datetime import datetime, timedelta, timezone

                def fetch_recent_articles(pocketbase_url: str):
                    # Calculate time threshold (e.g., 3 hours ago)
                    time_threshold = datetime.now(timezone.utc) - timedelta(days=50)
                    time_string = time_threshold.isoformat().replace("+00:00", "Z")   
                    # Query PocketBase API, giving the full path where articles reside
                    filter_query = f"updated >= '{time_string}'"
                    url = f"{pocketbase_url}/api/collections/nature_posts/records"
                    params = {"filter": filter_query}
                    response = requests.get(url, params=params)
                    response.raise_for_status()
                    return response.json().get("items", [])
            </pre>
            Now, let's write a test audit program, we will use a locally hosted llm using openAI like. <br>
            <pre>
            import json
            from llama_index.llms.openai_like import OpenAILike
            import article_schema as asc
            from llama_index.core.prompts import PromptTemplate

            #creating a dummy article to test it
            custom_llm = OpenAILike(model="gemma", api_base="http://localhost:8080/v1", api_key="not_needed", is_chat_model=True)

            prompt = f"""
                    Analyze the following article for well-formed structure, tone, and readability.
                    CollectionId: {'something'}
                    Title: {'Rose, a beautiful plant'}
                    Content: {'Roses are very beautiful plants and grow in different colors. They are great addition to any garden! Roses are popular woody shrubs that belong to the genus Rosa.They grow colorful flowers in shades of red, pink, yellow, and white.Their stems have sharp prickles, commonly called thorns, to protect against animals.Most rose varieties produce a sweet and pleasant fragrance.'}
                    """

            # Use structured output to force the Pydantic schema evaluation
            response = custom_llm.structured_predict(
                    output_cls=asc.ArticleStructure,
                    prompt=PromptTemplate(prompt)
                    )

            print(response)   
            </pre>
            If everything goes well, you should be able to see something like this. <br>
            <pre>
                collectionId='CollectionId' title='Rose, a beautiful plant' well_formed=True missing_elements=[] feedback='The article is well-formed with a clear structure. It follows a simple, direct narrative. The title is engaging and appropriately formatted.  The content is concise and informative, providing a basic overview of roses.  There are no missing elements.'
            </pre>
            Expand on the concept and implement a simple workflow using llamaindex to complete the quality check of the contents of the articles.<br>
            <b>Full code will be presented at a later point of time!!</b>
        </p>
    </body>
</html>