Spaces:
Paused
Paused
feat: Populate Qdrant Cloud Cluster with multilingual Indic corpora across passage and longdoc strategies
3c23784 Download knowledge_base/processed/en_corpus.jsonl from rishik1111/vector-backend: direct link, hf CLI and curl.
- Browser
- Download file 15.4 kB
-
https://huggingface.co/spaces/rishik1111/vector-backend/resolve/main/knowledge_base/processed/en_corpus.jsonl
- Command line
-
hf download hf://spaces/rishik1111/vector-backend/knowledge_base/processed/en_corpus.jsonl
-
curl -L -o en_corpus.jsonl https://huggingface.co/spaces/rishik1111/vector-backend/resolve/main/knowledge_base/processed/en_corpus.jsonl
15.4 kB
| {"passage_id": "en_p_000001", "text": "J. Robert Oppenheimer was an American theoretical physicist and director of the Manhattan Project's Los Alamos Laboratory during World War II.", "source_lang": "en", "source_query_ids": [101], "is_selected": 1} | |
| {"passage_id": "en_p_000002", "text": "The Manhattan Project was a research and development undertaking during World War II that produced the first nuclear weapons.", "source_lang": "en", "source_query_ids": [101], "is_selected": 1} | |
| {"passage_id": "en_p_000003", "text": "Hacker House Goa 2026 is a developer hackathon focused on voice-enabled multilingual Indic AI applications and low-latency RAG systems.", "source_lang": "en", "source_query_ids": [102], "is_selected": 1} | |
| {"passage_id": "en_p_000004", "text": "Retrieval-Augmented Generation (RAG) optimizes LLM outputs by referencing an authoritative knowledge base outside of its training data sources.", "source_lang": "en", "source_query_ids": [103], "is_selected": 1} | |
| {"passage_id": "en_p_000005", "text": "FAISS is a library for efficient similarity search and clustering of dense vectors developed by Meta AI.", "source_lang": "en", "source_query_ids": [104], "is_selected": 1} | |
| {"passage_id": "en_p_000006", "text": "Goa is a state located on the southwestern coast of India within the Konkan region, famous for its beaches, architecture, and tropical climate.", "source_lang": "en", "source_query_ids": [105], "is_selected": 1} | |
| {"passage_id": "en_p_000007", "text": "Sarvam AI develops foundational artificial intelligence models and speech recognition tools specifically tailored for Indian languages.", "source_lang": "en", "source_query_ids": [106], "is_selected": 1} | |
| {"passage_id": "en_p_000008", "text": "Devanagari is an Indic script used for writing Hindi, Marathi, Nepali, and Sanskrit languages.", "source_lang": "en", "source_query_ids": [107], "is_selected": 1} | |
| {"passage_id": "en_p_000009", "text": "TextRank is a graph-based ranking algorithm used for keyword extraction and automatic text summarization.", "source_lang": "en", "source_query_ids": [108], "is_selected": 1} | |
| {"passage_id": "en_p_000010", "text": "Singular Value Decomposition (SVD) is a matrix factorization method that identifies principal components and factual energy in context synthesis.", "source_lang": "en", "source_query_ids": [109], "is_selected": 1} | |
| {"passage_id": "en_p_000011", "text": "The human heart has four chambers: the right atrium, left atrium, right ventricle, and left ventricle.", "source_lang": "en", "source_query_ids": [110], "is_selected": 1} | |
| {"passage_id": "en_p_000012", "text": "Cricket is the most popular sport in India, with the Indian Premier League (IPL) being one of the largest professional T20 cricket leagues globally.", "source_lang": "en", "source_query_ids": [111], "is_selected": 1} | |
| {"passage_id": "en_p_000013", "text": "The Olympic Games are a major international multi-sport event held every four years, featuring summer and winter sports competitions.", "source_lang": "en", "source_query_ids": [112], "is_selected": 1} | |
| {"passage_id": "en_p_000014", "text": "Indian cinema is the largest film industry in the world by annual film output, comprising Bollywood, Tollywood, and vibrant regional film industries.", "source_lang": "en", "source_query_ids": [113], "is_selected": 1} | |
| {"passage_id": "en_p_000015", "text": "The Academy Awards, commonly known as the Oscars, honor artistic and technical merit in the global film industry.", "source_lang": "en", "source_query_ids": [114], "is_selected": 1} | |
| {"passage_id": "en_p_000016", "text": "Social media platforms facilitate online community building, real-time news distribution, and global digital communication.", "source_lang": "en", "source_query_ids": [115], "is_selected": 1} | |
| {"passage_id": "en_p_000017", "text": "Digital India is a government initiative launched to transform India into a digitally empowered society and knowledge economy.", "source_lang": "en", "source_query_ids": [116], "is_selected": 1} | |
| {"passage_id": "en_p_000018", "text": "The Himalayas form the highest mountain range in the world, hosting peaks such as Mount Everest and acting as the source for major rivers like the Ganges.", "source_lang": "en", "source_query_ids": [117], "is_selected": 1} | |
| {"passage_id": "en_p_000019", "text": "The Western Ghats are a mountain range running parallel to India's western coast, recognized as a UNESCO World Heritage site and biodiversity hotspot.", "source_lang": "en", "source_query_ids": [118], "is_selected": 1} | |
| {"passage_id": "en_p_000020", "text": "Artificial Intelligence and Machine Learning enable computers to analyze complex datasets, perform natural language processing, and automate tasks.", "source_lang": "en", "source_query_ids": [119], "is_selected": 1} | |
| {"passage_id": "en_p_000021", "text": "Cloud computing provides on-demand access to computing resources including servers, storage, databases, and software over the internet.", "source_lang": "en", "source_query_ids": [120], "is_selected": 1} | |
| {"passage_id": "en_p_000022", "text": "Stock markets enable public companies to raise capital by issuing equity shares, allowing investors to trade ownership stakes.", "source_lang": "en", "source_query_ids": [121], "is_selected": 1} | |
| {"passage_id": "en_p_000023", "text": "The Reserve Bank of India (RBI) is India's central banking institution that formulates monetary policy and regulates the financial system.", "source_lang": "en", "source_query_ids": [122], "is_selected": 1} | |
| {"passage_id": "en_p_tech_001", "text": "Python is a high-level programming language widely used in data science, with NumPy providing fast multi-dimensional arrays and Pandas offering DataFrame structures for data manipulation.", "source_lang": "en", "source_query_ids": [401], "is_selected": 1} | |
| {"passage_id": "en_p_tech_002", "text": "Data preprocessing cleans raw datasets by handling missing values and outliers, while SQL queries extract data from databases, and Matplotlib creates data visualizations.", "source_lang": "en", "source_query_ids": [402], "is_selected": 1} | |
| {"passage_id": "en_p_tech_003", "text": "Git and GitHub enable version control for code collaboration, Linux command line manages server environments, APIs exchange JSON data, and Jupyter Notebooks facilitate interactive data analysis.", "source_lang": "en", "source_query_ids": [403], "is_selected": 1} | |
| {"passage_id": "en_p_tech_004", "text": "Linear algebra forms the foundation of machine learning, using vectors and matrices to represent datasets, spatial transformations, and neural network weights.", "source_lang": "en", "source_query_ids": [404], "is_selected": 1} | |
| {"passage_id": "en_p_tech_005", "text": "Statistics analyzes mean, median, variance, and standard deviation, while probability theory, correlation, covariance, and Bayes theorem calculate conditional likelihoods in ML models.", "source_lang": "en", "source_query_ids": [405], "is_selected": 1} | |
| {"passage_id": "en_p_tech_006", "text": "Calculus basics including derivatives and gradients guide optimization algorithms like gradient descent to minimize loss functions during machine learning model training.", "source_lang": "en", "source_query_ids": [406], "is_selected": 1} | |
| {"passage_id": "en_p_tech_007", "text": "Machine learning paradigms include supervised learning with labeled data, unsupervised learning for clustering unlabelled data, and reinforcement learning using reward feedback.", "source_lang": "en", "source_query_ids": [407], "is_selected": 1} | |
| {"passage_id": "en_p_tech_008", "text": "Linear regression predicts continuous numeric values, logistic regression performs binary classification, while decision trees and random forests ensemble multiple decision rules.", "source_lang": "en", "source_query_ids": [408], "is_selected": 1} | |
| {"passage_id": "en_p_tech_009", "text": "XGBoost provides gradient boosted decision trees, Support Vector Machines (SVM) create maximum-margin classification boundaries, K-Nearest Neighbors (KNN) classifies by proximity, K-Means clusters data, and PCA reduces dimensionality.", "source_lang": "en", "source_query_ids": [409], "is_selected": 1} | |
| {"passage_id": "en_p_tech_010", "text": "Feature engineering transforms raw variables, train-validation-test splitting prevents data leakage, cross-validation evaluates generalization, hyperparameter tuning optimizes parameters, and metrics like precision and recall assess ML performance.", "source_lang": "en", "source_query_ids": [410], "is_selected": 1} | |
| {"passage_id": "en_p_tech_011", "text": "Neural networks process inputs through multi-layer perceptrons, using activation functions like ReLU and backpropagation with loss gradients to update synaptic weights.", "source_lang": "en", "source_query_ids": [411], "is_selected": 1} | |
| {"passage_id": "en_p_tech_012", "text": "Convolutional Neural Networks (CNNs) process spatial images, Recurrent Neural Networks (RNNs) and LSTMs handle sequential text data, and transfer learning reuses pretrained feature extractors.", "source_lang": "en", "source_query_ids": [412], "is_selected": 1} | |
| {"passage_id": "en_p_tech_013", "text": "Transformer architectures utilize self-attention mechanisms to weigh relationships between all tokens in a sequence, mapping text into continuous vector embeddings.", "source_lang": "en", "source_query_ids": [413], "is_selected": 1} | |
| {"passage_id": "en_p_tech_014", "text": "Large Language Models (LLMs) break text into sub-word tokens, process prompts via prompt engineering, and undergo fine-tuning or parameter-efficient LoRA and QLoRA adaptation.", "source_lang": "en", "source_query_ids": [414], "is_selected": 1} | |
| {"passage_id": "en_p_tech_015", "text": "Retrieval-Augmented Generation (RAG) splits text into semantic chunks, stores vectors in vector databases for sub-second semantic search, and reranks candidates before generation.", "source_lang": "en", "source_query_ids": [415], "is_selected": 1} | |
| {"passage_id": "en_p_tech_016", "text": "AI agents execute multi-step agentic workflows using tool calling, grounding outputs to prevent hallucinations, with orchestration frameworks like LangChain and LangGraph.", "source_lang": "en", "source_query_ids": [416], "is_selected": 1} | |
| {"passage_id": "en_p_tech_017", "text": "Multimodal AI integrates text, vision, speech-to-text (STT), and text-to-speech (TTS), evaluating performance through automated LLM benchmark frameworks.", "source_lang": "en", "source_query_ids": [417], "is_selected": 1} | |
| {"passage_id": "en_p_tech_018", "text": "Computer vision utilizes OpenCV for image processing, YOLO for real-time object detection, image segmentation for pixel-level masks, and OCR for text extraction.", "source_lang": "en", "source_query_ids": [418], "is_selected": 1} | |
| {"passage_id": "en_p_tech_019", "text": "Vision Transformers (ViTs) adapt self-attention mechanisms to image patches, enabling multimodal vision-language models to perform visual question answering and image description.", "source_lang": "en", "source_query_ids": [419], "is_selected": 1} | |
| {"passage_id": "en_p_tech_020", "text": "MLOps standardizes machine learning lifecycle management using FastAPI REST endpoints, Docker containerization, cloud model serving, MLflow monitoring, and INT8 quantization for edge optimization.", "source_lang": "en", "source_query_ids": [420], "is_selected": 1} | |
| {"passage_id": "en_p_mov_001", "text": "Inception (2010, IMDb: 8.8) is a sci-fi thriller directed by Christopher Nolan, starring Leonardo DiCaprio as Dom Cobb, Joseph Gordon-Levitt, and Elliot Page. Dom Cobb is a thief who steals corporate secrets by entering dreams, and is tasked with planting an idea into a CEO's subconscious mind (inception).", "source_lang": "en", "source_query_ids": [701], "is_selected": 1} | |
| {"passage_id": "en_p_mov_002", "text": "The Dark Knight (2008, IMDb: 9.0) is a superhero film directed by Christopher Nolan, starring Christian Bale as Bruce Wayne / Batman and Heath Ledger as the Joker. Batman must accept one of the greatest psychological and physical tests of his ability to fight injustice when the Joker unleashes chaos on Gotham City.", "source_lang": "en", "source_query_ids": [702], "is_selected": 1} | |
| {"passage_id": "en_p_mov_003", "text": "Oppenheimer (2023, IMDb: 8.9) is a biographical drama directed by Christopher Nolan, starring Cillian Murphy as J. Robert Oppenheimer, Emily Blunt, Matt Damon, and Robert Downey Jr. The film chronicles Oppenheimer's leadership of the Manhattan Project at Los Alamos and the development of the atomic bomb.", "source_lang": "en", "source_query_ids": [703], "is_selected": 1} | |
| {"passage_id": "en_p_mov_004", "text": "Interstellar (2014, IMDb: 8.7) is a space epic directed by Christopher Nolan, starring Matthew McConaughey, Anne Hathaway, and Jessica Chastain. When Earth becomes uninhabitable, a team of ex-NASA astronauts travels through a wormhole near Saturn in search of a new habitable planet for humanity.", "source_lang": "en", "source_query_ids": [704], "is_selected": 1} | |
| {"passage_id": "en_p_mov_005", "text": "Titanic (1997, IMDb: 7.9) is an epic romance and disaster film directed by James Cameron, starring Leonardo DiCaprio as Jack Dawson and Kate Winslet as Rose DeWitt Bukater. A seventeen-year-old aristocrat falls in love with a poor artist aboard the luxurious, ill-fated R.M.S. Titanic.", "source_lang": "en", "source_query_ids": [705], "is_selected": 1} | |
| {"passage_id": "en_p_mov_006", "text": "The Godfather (1972, IMDb: 9.2) is a crime masterpiece directed by Francis Ford Coppola, starring Marlon Brando as Vito Corleone and Al Pacino as Michael Corleone. The story follows the aging patriarch of an Italian-American organized crime dynasty transferring control of his empire to his reluctant son.", "source_lang": "en", "source_query_ids": [706], "is_selected": 1} | |
| {"passage_id": "en_p_mov_007", "text": "Hollywood top actors list includes Leonardo DiCaprio, Robert De Niro, Tom Cruise, Meryl Streep, Scarlett Johansson, Cillian Murphy, Christian Bale, Brad Pitt, and Morgan Freeman, working with renowned directors like Steven Spielberg, Christopher Nolan, and Quentin Tarantino.", "source_lang": "en", "source_query_ids": [707], "is_selected": 1} | |
| {"passage_id": "en_p_mov_008", "text": "Baahubali: The Beginning (2015, IMDb: 8.0) and Baahubali 2: The Conclusion (2017, IMDb: 8.2) are epic Telugu fantasy action films directed by S.S. Rajamouli, starring Prabhas as Amarendra & Mahendra Baahubali, Rana Daggubati, Anushka Shetty, and Ramya Krishnan. An adventurous prince learns about his royal lineage and seeks revenge against a tyrant king.", "source_lang": "en", "source_query_ids": [708], "is_selected": 1} | |
| {"passage_id": "en_p_mov_009", "text": "RRR (2022, IMDb: 7.8) is a global blockbuster directed by S.S. Rajamouli, starring Ram Charan as Alluri Sitarama Raju and Jr NTR as Komaram Bheem, with Oscar-winning music by M.M. Keeravani for Naatu Naatu. Two 1920s Indian revolutionaries fight against British colonial oppressors.", "source_lang": "en", "source_query_ids": [709], "is_selected": 1} | |
| {"passage_id": "en_p_mov_010", "text": "Pushpa: The Rise (2021, IMDb: 7.6) and Pushpa 2: The Rule are Telugu action thrillers directed by Sukumar, starring Allu Arjun as Pushpa Raj, Rashmika Mandanna, and Fahadh Faasil. A coolie rises through the ranks of a red sandalwood smuggling syndicate in Andhra Pradesh.", "source_lang": "en", "source_query_ids": [710], "is_selected": 1} | |