Spaces:
Runtime error
Runtime error
Commit History
-
Update embedding model to Google Generative AI and enhance vector store functionality
- Changed the embedding model from OpenAI to Google Generative AI in the EmbeddingManager class. - Updated the configuration to reflect the new embedding model path. - Modified validation checks to ensure the presence of the Google API key for RAG embeddings. - Added a new method to reset the vector store, allowing for a complete clear and recreation of the collection. - Enhanced logging to provide clearer feedback on embedding model initialization and vector store operations.4dfec96
-
Refactor LimitedEnsembleRetriever for improved compatibility and functionality
- Updated the `LimitedEnsembleRetriever` class to remove inheritance from `BaseRetriever`, simplifying its structure. - Changed the method names to align with the new invoke interface, replacing deprecated methods with `invoke` and `ainvoke`. - Added a compatibility method to handle both string input and other data types for the `invoke` method, enhancing usability. - Improved documentation within the class to clarify the purpose and functionality of methods.5da24ca -
Enhance vector store retrieval with limited results
- Introduced a new `LimitedEnsembleRetriever` class to limit the number of results returned by the ensemble retriever to a specified count (k). - Updated the `get_hybrid_retriever` method to return a `LimitedEnsembleRetriever` instance, ensuring that exactly k results are provided for improved performance and usability. - Enhanced logging to reflect the creation of the limited retriever with specified weights and result limits.9e9e9ff -
Refactor OCR configuration in DoclingParser to use EasyOCR exclusively
- Simplified the OCR method configuration by removing Tesseract support and defaulting to EasyOCR. - Updated the supported OCR methods list to reflect the removal of Tesseract, enhancing clarity and maintainability. - Improved logging to indicate the use of EasyOCR for CPU-only processing.18e6067 -
Enhance DoclingParser for CPU-only processing and improved error handling
- Implemented environment variables to enforce CPU-only mode for EasyOCR and CUDA libraries. - Enhanced error handling during ZeroGPU processing to provide more specific logging for pickle and CUDA errors. - Updated the CPU-only DocumentConverter creation method to ensure proper OCR fallback and logging. - Improved handling of kwargs to avoid unpicklable arguments during processing.5e0609f -
Implement ZeroGPU support in DoclingParser for enhanced document processing
- Added support for GPU processing using the ZeroGPU framework, allowing for accelerated document conversion. - Introduced methods for CPU-only processing and fallback mechanisms to ensure robust performance. - Updated the initialization process to defer converter creation until needed, preventing CUDA issues. - Enhanced error handling and logging for better debugging and user feedback during document conversion.d66e90c -
Enhance README and parser functionality for improved document processing
- Updated README to provide a comprehensive overview of the system, including a detailed table of contents and enhanced descriptions of key features. - Improved GeminiFlashParser to include detailed error messages for unsupported file types and added support for additional MIME types. - Enhanced MistralOcrParser to support DOCX and PPTX file types, improving document processing capabilities. - Introduced a new test suite for Gemini wrapper functionality to validate integration with MarkItDown and ensure robust performance.4a97b0c
-
Integrate Gemini API for enhanced image processing in MarkItDown
- Updated `app.py` to reflect the use of Gemini for image processing in MarkItDown. - Modified `requirements.txt` to include `ffmpeg-python` for audio processing and removed OpenAI dependency. - Enhanced `setup.sh` to install Gemini dependencies and updated installation instructions. - Introduced `gemini_client_wrapper.py` to create a wrapper for Gemini API, mimicking OpenAI's interface for compatibility with MarkItDown. - Added tests in `test_gemini_wrapper.py` to validate Gemini integration and MarkItDown functionality. - Refactored `markitdown_parser.py` to utilize Gemini for image files while maintaining standard processing for other formats. - Updated parser names and descriptions for clarity across various parsers.033e4ba -
Refactor UI components for modular architecture and enhance functionality
- Introduced a modular UI structure with dedicated components for Document Converter, Chat Interface, and Query Ranker. - Updated README to reflect the new modular UI architecture and its components. - Implemented content formatting utilities for Markdown and LaTeX rendering. - Enhanced file validation and threading utilities for improved user experience. - Added comprehensive styles for a cohesive UI design across components. - Established a test suite for the new UI components to ensure functionality and reliability.6ea41ec
-
Refactor document ingestion and chunking to support LaTeX content
- Updated `DocumentIngestionService` to generalize content ingestion, allowing both Markdown and LaTeX formats. - Introduced `LaTeXAwareChunker` for chunking LaTeX documents while preserving structures. - Enhanced `UnifiedDocumentChunker` to handle both Markdown and LaTeX content types seamlessly. - Modified `_process_latex_content` to return raw LaTeX for GOT-OCR without LLM conversion. - Improved UI rendering for LaTeX content, ensuring proper display using MathJax. - Added backward compatibility for existing Markdown ingestion methods.63279a9
-
Enhance UI with new Query Ranker feature and improve document search capabilities
- Introduced a new "Query Ranker" tab for searching and ranking document chunks with various retrieval methods: Similarity, MMR, BM25, and Hybrid. - Implemented real-time document chunk ranking with intelligent confidence scoring and actual similarity scores. - Updated the UI to a three-tab interface, integrating the Query Ranker alongside Document Converter and Chat. - Enhanced README to document the new Query Ranker feature and its functionalities. - Added comprehensive test suite for the Query Ranker implementation.623ad58 -
Add advanced retrieval strategies and update dependencies for RAG implementation
- Introduced BM25Retriever and EnsembleRetriever for enhanced document retrieval methods. - Updated `app.py`, `requirements.txt`, and `setup.sh` to include new dependencies for BM25 and community retrievers. - Enhanced `RAGChatService` to support multiple retrieval methods: similarity, MMR, BM25, and hybrid. - Updated README to document new retrieval strategies and configuration options. - Added comprehensive tests for retrieval methods and implementation structure.21c909d -
Enhance Docling and Mistral OCR parsers with improved response handling and logging
- Updated the `_format_batch_output` method in `DoclingParser` for better output formatting. - Enhanced response text extraction logic in `DoclingParser` to handle various response structures from the Gemini API, including detailed logging for debugging. - Removed unnecessary whitespace in `MistralOcrParser` for cleaner code. - Improved error handling and logging for API errors in `DoclingParser` to facilitate troubleshooting.c61b4e2 -
Enhance multi-document processing capabilities in parsers
- Implemented validation for batch file processing in both Docling and Mistral OCR parsers, ensuring size and type constraints are met. - Added support for multi-document processing in Docling, allowing up to 5 files with a combined size limit of 20MB. - Enhanced the `_create_batch_prompt` and `_format_batch_output` methods in both parsers to handle multiple documents effectively. - Updated README to reflect new multi-document processing features and parser capabilities.d437733 -
Implement multi-document processing capabilities and enhance UI
- Introduced support for processing up to 5 files simultaneously with a combined size limit of 20MB. - Added new processing types: Combined, Individual, Summary, and Comparison for enhanced document analysis. - Updated the Gemini Flash parser to handle multiple documents and format outputs based on processing type. - Enhanced the UI to dynamically display processing options and real-time validation for file uploads. - Unified document conversion method to streamline single and multi-file processing. - Improved error handling and logging for batch processing operations.111954a
-
Add data clearing service and vector store management
- Introduced `DataClearingService` for managing the clearing of vector store and chat history data. - Implemented `clear_all_documents` method in `VectorStoreManager` to remove all documents from the vector store. - Enhanced UI with a button to clear all data, integrating the new service for user interaction. - Updated logging to provide detailed feedback during data clearing operations.f46dfbd -
Refactor document ingestion and output file handling
- Updated `create_document_hash` to `create_file_hash` for full SHA-256 hashing. - Introduced duplicate checking and deletion methods in `DocumentIngestionService`. - Enhanced `ingest_markdown_content` to utilize original file content for deduplication. - Modified `_create_output_file` to preserve original filename and handle conflicts. - Improved logging for document processing and ingestion status.3f1b4af -
Import configuration from core module in gemini_flash_parser.py to enhance parser functionality.
63f3b68 -
Update .gitignore and enhance README with data management instructions
- Added log files to .gitignore to prevent unnecessary tracking. - Expanded README to include new commands for clearing data and running the application with fresh data. - Clarified what data gets cleared during testing, improving usability for developers.a4f1c9e
-
Enhance RAG (Retrieval-Augmented Generation) functionality and dependencies
- Added RAG configuration management and integrated support for LangChain, ChromaDB, and sentence-transformers. - Implemented document ingestion, chunking, and embedding management for RAG capabilities. - Updated application to check and install necessary RAG dependencies automatically. - Enhanced README to include RAG features and usage instructions. - Refactored environment checks to ensure RAG dependencies are available. - Improved UI to support document conversion and chat functionalities with RAG integration. - Updated requirements.txt to include new RAG-related packages.575f1c7 -
Add Docling support for advanced document processing
- Integrated Docling library for enhanced PDF understanding and table structure recognition. - Updated application to check for Docling installation and install it if missing. - Enhanced README with usage instructions for Docling alongside MarkItDown. - Added Docling configuration options and parser implementation. - Updated requirements.txt to include Docling as a dependency. - Refactored environment checks and logging for better error handling related to Docling. - Introduced a new Docling parser for document conversion to Markdown.c0c51c2
-
Implement environment-based UI launch configuration in main.py
- Added detection for running in Hugging Face Spaces using the SPACE_ID environment variable. - Configured the UI launch to use '0.0.0.0' for server_name when in Hugging Face Spaces, and 'localhost' for local development. - Maintained the server_port and share settings for both environments.57f6aa0 -
Refactor and enhance application structure for Markit_v2
- Introduced centralized configuration management for API keys and application settings. - Implemented a new environment manager for dependency checks and setup. - Refactored document processing into a dedicated service layer for improved organization. - Enhanced error handling with custom exceptions for better clarity. - Updated README with new configuration options and usage instructions. - Added lightweight launcher for local development. - Improved logging setup for better debugging and information tracking. - Updated .gitignore to include new files and directories. This commit lays the groundwork for a more modular and maintainable codebase, facilitating future feature additions and improvements.a773878
-
Minor UI edit
55627c9 -
New feature: Mistral OCR
- changed the requirement txt and added new mistral ocr parser script98482ce -
Modifying the UI
49c5606 -
Change in UI : the bottom margin was white which was ugly so it's being change
7022f7f -
Initial Implementation of Markitdown. Implemented:
- Basic python usage - Image integration using OpenAI - Plugin disabled by defaultdbdd7c8
-
Approach #2 -- converting latex output from GOT OCR to markdown
5b7f920 -
restore to version 1
23ad33e -
Tabular is not defined
34d180e -
Latex2Markdown display changes --attemtp1
33f1b65 -
Error: Error processing document with GOT-OCR: cannot pickle '_thread.lock' object
4cac30a -
ERROR - Failed to load GOT-OCR model: CUDA must not be initialized in the main process on Spaces with Stateless GPU environment.
3415bc4 -
runtime error fix
36e49b4 -
Please work
4fab3b3 -
handling zerogpu usage
610b772 -
enable zerogpu
62f9c09
-
CUDA dfloat 16 issue again
dcdb7ac -
fixing the missing path of run_ocr_2.0.py by editing got_ocr_parser.py
7d77a56 -
changes on logging for better debugging
5bb2b30 -
missing run_ocr_2.0.py file
c9c21c7 -
change in strategy --implementing github got ocr instead of hugging face model
0f5865d -
script to convert markdown to latex, changing UI output to fit right with got ocr
ad248f7 -
restore to check point
2184c47 -
failed to load got ocr model
f89451e -
Error: Too many output
fa54d05 -
Error: Error processing document with GOT-OCR: GOTQwenForCausalLM.chat() got an unexpected keyword argument 'format'
c4c3253 -
adding format=true
1312a63 -
complete reimplementation of got ocr
3332d94