|
Download README.md from neelnsoni13/LargeMultiExcelsCleanDeepSeek: direct link, hf CLI and curl.
- Browser
- Download file 2.26 kB
-
https://huggingface.co/neelnsoni13/LargeMultiExcelsCleanDeepSeek/resolve/main/README.md
- Command line
-
hf download hf://neelnsoni13/LargeMultiExcelsCleanDeepSeek/README.md
-
curl -L -o README.md https://huggingface.co/neelnsoni13/LargeMultiExcelsCleanDeepSeek/resolve/main/README.md
2.26 kB
| license: mit | |
| # CSV Data Processing & Merging Pipeline | |
| ## Overview | |
| This project processes and merges multiple CSV datasets by intelligently mapping column headers, cleaning data, and ensuring consistency. It leverages AI-based semantic similarity to standardize headers and applies filtering rules for valid data entries. | |
| ## Features | |
| - **Recursive CSV File Search:** Collects all CSV files from a specified directory. | |
| - **AI-Powered Header Mapping:** Uses Sentence-BERT (`paraphrase-MiniLM-L6-v2`) and DeepSeek-R1:14B to map headers to predefined target columns. | |
| - **Data Cleaning & Validation:** | |
| - Filters valid phone numbers (7 to 15 digits). | |
| - Normalizes column names for consistency. | |
| - **Merging & Structuring:** Combines multiple datasets into a structured format. | |
| ## Installation | |
| ### Step 1: Create a Virtual Environment | |
| ```sh | |
| python -m venv venv | |
| source venv/bin/activate # On macOS/Linux | |
| venv\Scripts\activate # On Windows | |
| ``` | |
| ### Step 2: Install Dependencies | |
| ```sh | |
| pip install -r requirements.txt | |
| ``` | |
| ### Step 3: Ensure Required Models are Installed | |
| ```sh | |
| python -m sentence_transformers.sentence_transformer | |
| ``` | |
| ## Usage | |
| ### Running the Script | |
| ```sh | |
| python main.py --root_directory path/to/csv/files | |
| ``` | |
| ### Arguments | |
| - `--root_directory`: Path to the folder containing CSV files. | |
| ## Project Structure | |
| ``` | |
| project/ | |
| │── 1_CleanLargeMultipleExcels.py # Main script | |
| ``` | |
| ## Functions | |
| ### `read_csv_files(root_directory)` | |
| Recursively scans a directory to find all CSV files. | |
| ### `calculate_similarity(source_list, target_list)` | |
| Computes cosine similarity between dataset headers and target headers. | |
| ### `get_header_similarity(headers)` | |
| Uses DeepSeek AI to match dataset headers to predefined target columns. | |
| ### `is_valid_mobile(value)` | |
| Checks if a given value is a valid phone number. | |
| ### `merge_dataframes(file_paths)` | |
| Reads, maps, cleans, and merges CSV data. | |
| ## Future Enhancements | |
| - **Parallel Processing:** Implement multiprocessing for handling large datasets. | |
| - **Improved Header Mapping:** Train a custom model for better accuracy. | |
| - **Batch Processing:** Process CSV files in chunks to optimize memory usage. | |
| ## License | |
| This project is licensed under the MIT License. | |
| --- | |
| **NEEL N SONI:** | |