Spaces:
Runtime error
Runtime error
Commit History
-
Integrate Gemini API for enhanced image processing in MarkItDown
- Updated `app.py` to reflect the use of Gemini for image processing in MarkItDown. - Modified `requirements.txt` to include `ffmpeg-python` for audio processing and removed OpenAI dependency. - Enhanced `setup.sh` to install Gemini dependencies and updated installation instructions. - Introduced `gemini_client_wrapper.py` to create a wrapper for Gemini API, mimicking OpenAI's interface for compatibility with MarkItDown. - Added tests in `test_gemini_wrapper.py` to validate Gemini integration and MarkItDown functionality. - Refactored `markitdown_parser.py` to utilize Gemini for image files while maintaining standard processing for other formats. - Updated parser names and descriptions for clarity across various parsers.033e4ba -
Refactor UI components for modular architecture and enhance functionality
- Introduced a modular UI structure with dedicated components for Document Converter, Chat Interface, and Query Ranker. - Updated README to reflect the new modular UI architecture and its components. - Implemented content formatting utilities for Markdown and LaTeX rendering. - Enhanced file validation and threading utilities for improved user experience. - Added comprehensive styles for a cohesive UI design across components. - Established a test suite for the new UI components to ensure functionality and reliability.6ea41ec
-
Refactor document ingestion and chunking to support LaTeX content
- Updated `DocumentIngestionService` to generalize content ingestion, allowing both Markdown and LaTeX formats. - Introduced `LaTeXAwareChunker` for chunking LaTeX documents while preserving structures. - Enhanced `UnifiedDocumentChunker` to handle both Markdown and LaTeX content types seamlessly. - Modified `_process_latex_content` to return raw LaTeX for GOT-OCR without LLM conversion. - Improved UI rendering for LaTeX content, ensuring proper display using MathJax. - Added backward compatibility for existing Markdown ingestion methods.63279a9
-
Enhance UI with new Query Ranker feature and improve document search capabilities
- Introduced a new "Query Ranker" tab for searching and ranking document chunks with various retrieval methods: Similarity, MMR, BM25, and Hybrid. - Implemented real-time document chunk ranking with intelligent confidence scoring and actual similarity scores. - Updated the UI to a three-tab interface, integrating the Query Ranker alongside Document Converter and Chat. - Enhanced README to document the new Query Ranker feature and its functionalities. - Added comprehensive test suite for the Query Ranker implementation.623ad58 -
Implement multi-document processing capabilities and enhance UI
- Introduced support for processing up to 5 files simultaneously with a combined size limit of 20MB. - Added new processing types: Combined, Individual, Summary, and Comparison for enhanced document analysis. - Updated the Gemini Flash parser to handle multiple documents and format outputs based on processing type. - Enhanced the UI to dynamically display processing options and real-time validation for file uploads. - Unified document conversion method to streamline single and multi-file processing. - Improved error handling and logging for batch processing operations.111954a
-
Add data clearing service and vector store management
- Introduced `DataClearingService` for managing the clearing of vector store and chat history data. - Implemented `clear_all_documents` method in `VectorStoreManager` to remove all documents from the vector store. - Enhanced UI with a button to clear all data, integrating the new service for user interaction. - Updated logging to provide detailed feedback during data clearing operations.f46dfbd -
Refactor document ingestion and output file handling
- Updated `create_document_hash` to `create_file_hash` for full SHA-256 hashing. - Introduced duplicate checking and deletion methods in `DocumentIngestionService`. - Enhanced `ingest_markdown_content` to utilize original file content for deduplication. - Modified `_create_output_file` to preserve original filename and handle conflicts. - Improved logging for document processing and ingestion status.3f1b4af -
Update .gitignore and enhance README with data management instructions
- Added log files to .gitignore to prevent unnecessary tracking. - Expanded README to include new commands for clearing data and running the application with fresh data. - Clarified what data gets cleared during testing, improving usability for developers.a4f1c9e
-
Enhance RAG (Retrieval-Augmented Generation) functionality and dependencies
- Added RAG configuration management and integrated support for LangChain, ChromaDB, and sentence-transformers. - Implemented document ingestion, chunking, and embedding management for RAG capabilities. - Updated application to check and install necessary RAG dependencies automatically. - Enhanced README to include RAG features and usage instructions. - Refactored environment checks to ensure RAG dependencies are available. - Improved UI to support document conversion and chat functionalities with RAG integration. - Updated requirements.txt to include new RAG-related packages.575f1c7
-
Refactor and enhance application structure for Markit_v2
- Introduced centralized configuration management for API keys and application settings. - Implemented a new environment manager for dependency checks and setup. - Refactored document processing into a dedicated service layer for improved organization. - Enhanced error handling with custom exceptions for better clarity. - Updated README with new configuration options and usage instructions. - Added lightweight launcher for local development. - Improved logging setup for better debugging and information tracking. - Updated .gitignore to include new files and directories. This commit lays the groundwork for a more modular and maintainable codebase, facilitating future feature additions and improvements.a773878
-
Modifying the UI
49c5606 -
Change in UI : the bottom margin was white which was ugly so it's being change
7022f7f -
Initial Implementation of Markitdown. Implemented:
- Basic python usage - Image integration using OpenAI - Plugin disabled by defaultdbdd7c8
-
script to convert markdown to latex, changing UI output to fit right with got ocr
ad248f7
-
minor UI change
c337a68 -
Removing chat with document feature
bdc060b -
restore
66f3c4d -
Remove chat with document feature, updated ui, remove ocrmac, pinned transformer version to 4.37 for GOT OCR
e66626c -
Implement Markit_GOT files into Markit_v2
dda982a