File size: 2,895 Bytes
644f326
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
from langchain_core.document_loaders.base import BaseLoader
from langchain_core.documents import Document
from typing import Iterator, List
from src.core.exception import MyException
from src.core.logger import logger
import sys
from bs4 import BeautifulSoup


class HTMLLoader(BaseLoader):
    def __init__(self, file_path: str):
        self.file_path = file_path

    def lazy_load(self) -> Iterator[Document]:
        with logger.span("Lazy loading HTML: {file_path}", file_path=self.file_path):
            f = None
            try:
                logger.info("Opening HTML file", file_path=self.file_path)
                f = open(file=self.file_path, encoding="utf-8", errors="ignore")
                content = f.read()
                logger.debug("HTML file read", file_path=self.file_path, raw_size=len(content))

                soup = BeautifulSoup(content, "html.parser")
                for script in soup(["script", "style", "meta", "noscript"]):
                    script.decompose()

                text = soup.get_text(separator="\n")
                lines = (line.strip() for line in text.splitlines())
                chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
                text_clean = "\n".join(chunk for chunk in chunks if chunk)

                logger.info("HTML parsed and cleaned", file_path=self.file_path, clean_size=len(text_clean))
                yield Document(page_content=text_clean, metadata={"source": self.file_path, "page": 0})
                logger.info("HTML lazy load complete", file_path=self.file_path)

            except Exception as e:
                logger.error("Failed to lazy load HTML", file_path=self.file_path, error=str(e))
                raise MyException(e, sys)

            finally:
                if f:
                    f.close()

    def load(self) -> List[Document]:
        with logger.span("Loading HTML into memory: {file_path}", file_path=self.file_path):
            try:
                docs = list(self.lazy_load())
                logger.info("HTML load complete", file_path=self.file_path, doc_count=len(docs))
                return docs

            except Exception as e:
                logger.error("Failed to load HTML", file_path=self.file_path, error=str(e))
                raise MyException(e, sys)


async def html_loader(file_path: str):
    with logger.span("Async HTML loader: {file_path}", file_path=file_path):
        try:
            logger.info("Initiating HTML load", file_path=file_path)
            loader = HTMLLoader(file_path=file_path)
            docs = await loader.aload()
            logger.info("Async HTML load finished", file_path=file_path, doc_count=len(docs))
            return docs

        except Exception as e:
            logger.error("Async HTML loader failed", file_path=file_path, error=str(e))
            raise MyException(e, sys)