Spaces:
Runtime error
Runtime error
Commit History
-
Update embedding model to Google Generative AI and enhance vector store functionality
- Changed the embedding model from OpenAI to Google Generative AI in the EmbeddingManager class. - Updated the configuration to reflect the new embedding model path. - Modified validation checks to ensure the presence of the Google API key for RAG embeddings. - Added a new method to reset the vector store, allowing for a complete clear and recreation of the collection. - Enhanced logging to provide clearer feedback on embedding model initialization and vector store operations.4dfec96
-
Update README to include demo video section
- Added a new section for a demo video showcasing Markit's features, including multi-document processing and RAG chat. - Included a link to the YouTube video and an embedded thumbnail for better visibility. - Updated the table of contents to reflect the addition of the demo video section.bf4414c -
Refactor LimitedEnsembleRetriever for improved compatibility and functionality
- Updated the `LimitedEnsembleRetriever` class to remove inheritance from `BaseRetriever`, simplifying its structure. - Changed the method names to align with the new invoke interface, replacing deprecated methods with `invoke` and `ainvoke`. - Added a compatibility method to handle both string input and other data types for the `invoke` method, enhancing usability. - Improved documentation within the class to clarify the purpose and functionality of methods.5da24ca -
Add GIF files using Git LFS
53ff214 -
Enhance vector store retrieval with limited results
- Introduced a new `LimitedEnsembleRetriever` class to limit the number of results returned by the ensemble retriever to a specified count (k). - Updated the `get_hybrid_retriever` method to return a `LimitedEnsembleRetriever` instance, ensuring that exactly k results are provided for improved performance and usability. - Enhanced logging to reflect the creation of the limited retriever with specified weights and result limits.9e9e9ff -
Refactor OCR configuration in DoclingParser to use EasyOCR exclusively
- Simplified the OCR method configuration by removing Tesseract support and defaulting to EasyOCR. - Updated the supported OCR methods list to reflect the removal of Tesseract, enhancing clarity and maintainability. - Improved logging to indicate the use of EasyOCR for CPU-only processing.18e6067 -
Enhance DoclingParser for CPU-only processing and improved error handling
- Implemented environment variables to enforce CPU-only mode for EasyOCR and CUDA libraries. - Enhanced error handling during ZeroGPU processing to provide more specific logging for pickle and CUDA errors. - Updated the CPU-only DocumentConverter creation method to ensure proper OCR fallback and logging. - Improved handling of kwargs to avoid unpicklable arguments during processing.5e0609f -
Implement ZeroGPU support in DoclingParser for enhanced document processing
- Added support for GPU processing using the ZeroGPU framework, allowing for accelerated document conversion. - Introduced methods for CPU-only processing and fallback mechanisms to ensure robust performance. - Updated the initialization process to defer converter creation until needed, preventing CUDA issues. - Enhanced error handling and logging for better debugging and user feedback during document conversion.d66e90c -
Add images with Git LFS tracking
9123559 -
Enhance README and parser functionality for improved document processing
- Updated README to provide a comprehensive overview of the system, including a detailed table of contents and enhanced descriptions of key features. - Improved GeminiFlashParser to include detailed error messages for unsupported file types and added support for additional MIME types. - Enhanced MistralOcrParser to support DOCX and PPTX file types, improving document processing capabilities. - Introduced a new test suite for Gemini wrapper functionality to validate integration with MarkItDown and ensure robust performance.4a97b0c
-
Merge pull request #15 from ansemin/development
Unverified signatureIntegrate Gemini API for enhanced image processing in MarkItDownd85616d -
Integrate Gemini API for enhanced image processing in MarkItDown
- Updated `app.py` to reflect the use of Gemini for image processing in MarkItDown. - Modified `requirements.txt` to include `ffmpeg-python` for audio processing and removed OpenAI dependency. - Enhanced `setup.sh` to install Gemini dependencies and updated installation instructions. - Introduced `gemini_client_wrapper.py` to create a wrapper for Gemini API, mimicking OpenAI's interface for compatibility with MarkItDown. - Added tests in `test_gemini_wrapper.py` to validate Gemini integration and MarkItDown functionality. - Refactored `markitdown_parser.py` to utilize Gemini for image files while maintaining standard processing for other formats. - Updated parser names and descriptions for clarity across various parsers.033e4ba -
Merge pull request #14 from ansemin/development
Unverified signatureRefactor UI components for modular architecture and enhance functiona…71af303 -
Refactor UI components for modular architecture and enhance functionality
- Introduced a modular UI structure with dedicated components for Document Converter, Chat Interface, and Query Ranker. - Updated README to reflect the new modular UI architecture and its components. - Implemented content formatting utilities for Markdown and LaTeX rendering. - Enhanced file validation and threading utilities for improved user experience. - Added comprehensive styles for a cohesive UI design across components. - Established a test suite for the new UI components to ensure functionality and reliability.6ea41ec
-
Enhance README to reflect new features and improvements in document processing
- Updated GOT-OCR section to highlight native LaTeX output and Mathpix rendering capabilities. - Introduced Smart Content-Aware Chunking for both Markdown and LaTeX, detailing preservation of structures and automatic format detection. - Added new configuration options for LaTeX chunking in the application settings. - Expanded usage instructions for GOT-OCR and other parsers, clarifying output formats. - Included a new section on advanced LaTeX processing features and use cases for better user guidance.2a9686e -
Merge pull request #12 from ansemin/development
Unverified signatureRefactor document ingestion and chunking to support LaTeX contentab0e18b -
Refactor document ingestion and chunking to support LaTeX content
- Updated `DocumentIngestionService` to generalize content ingestion, allowing both Markdown and LaTeX formats. - Introduced `LaTeXAwareChunker` for chunking LaTeX documents while preserving structures. - Enhanced `UnifiedDocumentChunker` to handle both Markdown and LaTeX content types seamlessly. - Modified `_process_latex_content` to return raw LaTeX for GOT-OCR without LLM conversion. - Improved UI rendering for LaTeX content, ensuring proper display using MathJax. - Added backward compatibility for existing Markdown ingestion methods.63279a9
-
Merge pull request #10 from ansemin/development
Unverified signatureEnhance UI with new Query Ranker feature and improve document search …e8b3cf9 -
Enhance UI with new Query Ranker feature and improve document search capabilities
- Introduced a new "Query Ranker" tab for searching and ranking document chunks with various retrieval methods: Similarity, MMR, BM25, and Hybrid. - Implemented real-time document chunk ranking with intelligent confidence scoring and actual similarity scores. - Updated the UI to a three-tab interface, integrating the Query Ranker alongside Document Converter and Chat. - Enhanced README to document the new Query Ranker feature and its functionalities. - Added comprehensive test suite for the Query Ranker implementation.623ad58 -
Merge pull request #9 from ansemin/development
Unverified signatureAdd advanced retrieval strategies and update dependencies for RAG imp…b2cd0ff -
Add advanced retrieval strategies and update dependencies for RAG implementation
- Introduced BM25Retriever and EnsembleRetriever for enhanced document retrieval methods. - Updated `app.py`, `requirements.txt`, and `setup.sh` to include new dependencies for BM25 and community retrievers. - Enhanced `RAGChatService` to support multiple retrieval methods: similarity, MMR, BM25, and hybrid. - Updated README to document new retrieval strategies and configuration options. - Added comprehensive tests for retrieval methods and implementation structure.21c909d -
Merge pull request #8 from ansemin/development
Unverified signatureDevelopment4b774be -
Enhance Docling and Mistral OCR parsers with improved response handling and logging
- Updated the `_format_batch_output` method in `DoclingParser` for better output formatting. - Enhanced response text extraction logic in `DoclingParser` to handle various response structures from the Gemini API, including detailed logging for debugging. - Removed unnecessary whitespace in `MistralOcrParser` for cleaner code. - Improved error handling and logging for API errors in `DoclingParser` to facilitate troubleshooting.c61b4e2 -
Enhance multi-document processing capabilities in parsers
- Implemented validation for batch file processing in both Docling and Mistral OCR parsers, ensuring size and type constraints are met. - Added support for multi-document processing in Docling, allowing up to 5 files with a combined size limit of 20MB. - Enhanced the `_create_batch_prompt` and `_format_batch_output` methods in both parsers to handle multiple documents effectively. - Updated README to reflect new multi-document processing features and parser capabilities.d437733 -
Merge pull request #6 from ansemin/development
Unverified signatureImplement multi-document processing capabilities and enhance UI615c16c -
Implement multi-document processing capabilities and enhance UI
- Introduced support for processing up to 5 files simultaneously with a combined size limit of 20MB. - Added new processing types: Combined, Individual, Summary, and Comparison for enhanced document analysis. - Updated the Gemini Flash parser to handle multiple documents and format outputs based on processing type. - Enhanced the UI to dynamically display processing options and real-time validation for file uploads. - Unified document conversion method to streamline single and multi-file processing. - Improved error handling and logging for batch processing operations.111954a
-
Enhance README and add data management features
- Updated README to include new "Clear All Data" functionality for user-friendly data management. - Added detailed instructions for in-app data clearing, including environment detection and user feedback. - Introduced `DataClearingService` for comprehensive data management, ensuring safe operations and automatic session resets. - Enhanced UI elements to reflect new data management capabilities and improved status reporting.35912f6 -
Add data clearing service and vector store management
- Introduced `DataClearingService` for managing the clearing of vector store and chat history data. - Implemented `clear_all_documents` method in `VectorStoreManager` to remove all documents from the vector store. - Enhanced UI with a button to clear all data, integrating the new service for user interaction. - Updated logging to provide detailed feedback during data clearing operations.f46dfbd -
Merge pull request #4 from ansemin/development
Unverified signatureRefactor document ingestion and output file handlingf901c17 -
Refactor document ingestion and output file handling
- Updated `create_document_hash` to `create_file_hash` for full SHA-256 hashing. - Introduced duplicate checking and deletion methods in `DocumentIngestionService`. - Enhanced `ingest_markdown_content` to utilize original file content for deduplication. - Modified `_create_output_file` to preserve original filename and handle conflicts. - Improved logging for document processing and ingestion status.3f1b4af -
Import configuration from core module in gemini_flash_parser.py to enhance parser functionality.
63f3b68 -
Update .gitignore to ignore all log files by changing the pattern from '==' to '='. This ensures that log files are properly excluded from version control.
6b709cc -
Remove =* files from repo and apply .gitignore
6099d74 -
Merge pull request #3 from ansemin/development
Unverified signatureUpdate .gitignore and enhance README with data management instructions and minor UI fixfbd1da2 -
Update .gitignore and enhance README with data management instructions
- Added log files to .gitignore to prevent unnecessary tracking. - Expanded README to include new commands for clearing data and running the application with fresh data. - Clarified what data gets cleared during testing, improving usability for developers.a4f1c9e -
Merge pull request #2 from ansemin/development
Unverified signature- Added RAG configuration management and integrated support for LangChain, ChromaDB, and sentence-transformers. - Implemented document ingestion, chunking, and embedding management for RAG capabilities. - Updated application to check and install necessary RAG dependencies automatically. - Enhanced README to include RAG features and usage instructions. - Refactored environment checks to ensure RAG dependencies are available. - Improved UI to support document conversion and chat functionalities with RAG integration. - Updated requirements.txt to include new RAG-related packages.a799aba
-
Enhance RAG (Retrieval-Augmented Generation) functionality and dependencies
- Added RAG configuration management and integrated support for LangChain, ChromaDB, and sentence-transformers. - Implemented document ingestion, chunking, and embedding management for RAG capabilities. - Updated application to check and install necessary RAG dependencies automatically. - Enhanced README to include RAG features and usage instructions. - Refactored environment checks to ensure RAG dependencies are available. - Improved UI to support document conversion and chat functionalities with RAG integration. - Updated requirements.txt to include new RAG-related packages.575f1c7 -
Add Docling support for advanced document processing
- Integrated Docling library for enhanced PDF understanding and table structure recognition. - Updated application to check for Docling installation and install it if missing. - Enhanced README with usage instructions for Docling alongside MarkItDown. - Added Docling configuration options and parser implementation. - Updated requirements.txt to include Docling as a dependency. - Refactored environment checks and logging for better error handling related to Docling. - Introduced a new Docling parser for document conversion to Markdown.c0c51c2
-
Implement environment-based UI launch configuration in main.py
- Added detection for running in Hugging Face Spaces using the SPACE_ID environment variable. - Configured the UI launch to use '0.0.0.0' for server_name when in Hugging Face Spaces, and 'localhost' for local development. - Maintained the server_port and share settings for both environments.57f6aa0 -
Clean up .gitignore by removing merge conflict markers and redundant entries, ensuring proper file exclusions for the project.
0a84dd0 -
Merge branch 'main' of https://github.com/ansemin/Markit_v2
48300e6 -
Update .gitignore
f6ac414 -
Merge pull request #1 from ansemin/development
Unverified signatureRefactor and enhance application structure for Markit_v2ebe2164 -
Refactor and enhance application structure for Markit_v2
- Introduced centralized configuration management for API keys and application settings. - Implemented a new environment manager for dependency checks and setup. - Refactored document processing into a dedicated service layer for improved organization. - Enhanced error handling with custom exceptions for better clarity. - Updated README with new configuration options and usage instructions. - Added lightweight launcher for local development. - Improved logging setup for better debugging and information tracking. - Updated .gitignore to include new files and directories. This commit lays the groundwork for a more modular and maintainable codebase, facilitating future feature additions and improvements.a773878
-
Minor UI edit
55627c9 -
Incompatibility with Google-genai and pydantic
Update pydantic to 2.10.6fae09c1 -
Error: The user requested pydantic==2.7.1
gradio 5.14.0 depends on pydantic>=2.0 google-genai 0.1.0 depends on pydantic<3.0.0dev and >=2.0.0 mistralai 1.6.0 depends on pydantic>=2.10.3 Fix: set pydantic>=2.10.30531b76 -
New feature: Mistral OCR
- changed the requirement txt and added new mistral ocr parser script98482ce -
Error: 'GPU task aborted', or 'You have exceeded your GPU quota (60s requested vs. 0s left). Create a free account to get more daily usage quota.'
Description: This is an error due to ZeroGPU free limitation, to resolve this you need to add 'hf_oauth: true' and subcribe to pro hugging face spaceb3c2847 -
Modifying the UI
49c5606