| { | |
| "cells": [ | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "8mYE-vgHGxwL" | |
| }, | |
| "source": [ | |
| "## **Natural Language Processing: Third Assignment – The Drug Search Challenge** \n", | |
| "\n", | |
| "This notebook presents our project on designing and implementing a robust drug search system that can process queries in both English and Persian. Through advanced NLP techniques—including text normalization, dataset merging, and semantic embedding alignment—we address the complexities of retrieving accurate drug information. The project demonstrates our approach to integrating bilingual data, enhancing search capabilities, and ultimately improving information retrieval in the medical domain.\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": {}, | |
| "source": [ | |
| "## Table of Contents\n", | |
| "\n", | |
| "- [Summary of the Problem](#summary)\n", | |
| "- [Steps in Designing the Drug Search System](#steps)\n", | |
| "- [Data Preparation](#data-preparation)\n", | |
| "- [Model Training](#model-training)\n", | |
| "- [Embedding Alignment](#alignment)\n", | |
| "- [Bilingual Search](#search)\n", | |
| "- [Evaluation](#evaluation)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "HoePPNjnMKfm" | |
| }, | |
| "source": [ | |
| "\n", | |
| "### installing the libraries\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 1, | |
| "metadata": { | |
| "id": "07po1ZxEMKfn" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "try:\n", | |
| " import ipywidgets\n", | |
| "except:\n", | |
| " %pip install ipywidgets\n", | |
| "\n", | |
| "try:\n", | |
| " import pandas as pd\n", | |
| "except:\n", | |
| " %pip install pandas\n", | |
| "\n", | |
| "try:\n", | |
| " import fuzzywuzzy\n", | |
| "except:\n", | |
| " %pip install fuzzywuzzy\n", | |
| "\n", | |
| "try:\n", | |
| " import nltk\n", | |
| "except:\n", | |
| " %pip install nltk\n", | |
| "\n", | |
| "try:\n", | |
| " import gensim\n", | |
| "except:\n", | |
| " %pip install gensim\n", | |
| "\n", | |
| "try:\n", | |
| " import langdetect\n", | |
| "except:\n", | |
| " %pip install langdetect\n", | |
| "\n", | |
| "try:\n", | |
| " from spellchecker import SpellChecker\n", | |
| "except:\n", | |
| " %pip install pyspellchecker\n", | |
| "\n", | |
| "try:\n", | |
| " import matplotlib as mpl\n", | |
| "except:\n", | |
| " %pip install matplotlib\n", | |
| "\n", | |
| "try:\n", | |
| " import sklearn\n", | |
| "except:\n", | |
| " %pip install sklearn\n", | |
| "\n", | |
| "try:\n", | |
| " import scipy as sp\n", | |
| "except:\n", | |
| " %pip install scipy\n", | |
| "\n", | |
| "try:\n", | |
| " import numpy as np\n", | |
| "except:\n", | |
| "\n", | |
| " %pip install numpy\n", | |
| "\n", | |
| "try:\n", | |
| " import cupy\n", | |
| "except ImportError:\n", | |
| " cupy = None\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 2, | |
| "metadata": { | |
| "id": "OuMehdGqMKfo" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "pd.set_option(\"display.max_columns\", None)\n", | |
| "pd.set_option(\"display.expand_frame_repr\", False)\n", | |
| "pd.set_option(\"max_colwidth\", None)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 3, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 17 | |
| }, | |
| "id": "CwzRdZkrMKfo", | |
| "outputId": "105f0b53-b44e-474d-890a-4d8694062010" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "\n", | |
| " <style>\n", | |
| " table.dataframe td, table.dataframe th {\n", | |
| " font-family: 'vazirmatn', 'Vazir', 'B Nazanin', 'Arial';\n", | |
| " }\n", | |
| " </style>\n", | |
| " " | |
| ], | |
| "text/plain": [ | |
| "<IPython.core.display.HTML object>" | |
| ] | |
| }, | |
| "execution_count": 3, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "from IPython.display import HTML\n", | |
| "\n", | |
| "mpl.rcParams['font.family'] = ['vazirmatn', 'Vazir', 'B Nazanin', 'Arial']\n", | |
| "mpl.rcParams['font.sans-serif'] = ['vazirmatn', 'Vazir', 'B Nazanin', 'Arial']\n", | |
| "mpl.rcParams['font.serif'] = ['vazirmatn', 'Vazir', 'B Nazanin', 'Arial']\n", | |
| "\n", | |
| "def set_pandas_font(fonts):\n", | |
| " css = f\"\"\"\n", | |
| " <style>\n", | |
| " table.dataframe td, table.dataframe th {{\n", | |
| " font-family: {fonts};\n", | |
| " }}\n", | |
| " </style>\n", | |
| " \"\"\"\n", | |
| " return HTML(css)\n", | |
| "\n", | |
| "set_pandas_font(\"'vazirmatn', 'Vazir', 'B Nazanin', 'Arial'\")" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "6RAmvwTvGxwO" | |
| }, | |
| "source": [ | |
| "<h2 id=\"summary\">Summary of the Problem Statement and Solution</h2>\n", | |
| "\n", | |
| "Here we provide a brief summary of the problem, followed by an outline of the steps we took to address it.\n", | |
| "\n", | |
| "### Steps Taken (presented in a more polished form):\n", | |
| "\n", | |
| "1. **Identified a database of English drug names** \n", | |
| " We began by sourcing a reliable dataset containing the names of drugs in English.\n", | |
| "\n", | |
| "2. **Normalized the database into the desired format** \n", | |
| " The raw data was cleaned and structured to match the specific format required for our processing pipeline.\n", | |
| "\n", | |
| "3. **Found Persian equivalents for the drugs in the database** \n", | |
| " We mapped each English drug name to its corresponding Persian translation using available linguistic or medical resources." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "eF25PR3dMKfp" | |
| }, | |
| "source": [ | |
| "<h2 id=\"steps\">Steps in Designing the Drug Search System</h2>\n", | |
| "\n", | |
| "In this section, we will provide a detailed explanation of the steps taken to design the drug search system, which were briefly summarized in the previous section." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "CzBffAvSGxwO" | |
| }, | |
| "source": [ | |
| "<h3 id=\"data-preparation\">1. Finding an English-Language Drug Database</h3>\n", | |
| "\n", | |
| "On the official website of the Iranian Food and Drug Administration, a public dataset with approximately 50,000 entries is available for download. For each drug, it contains information such as the name in Persian and Latin, the generic name, brand name, and more. This dataset serves as the primary database used in this project. However, several modifications and normalization processes have been applied to it, which will be explained in the following sections.\n", | |
| "\n", | |
| "Another dataset was downloaded from a GitHub repository. This dataset includes the name of each drug based on its ATC (Anatomical Therapeutic Chemical) code. The ATC code, which is also present in the dataset from the Iranian Food and Drug Administration, classifies drugs into categories. For example, the ATC code for melatonin (a sleep-regulating pill) is **N05CH01**. The characters in this code indicate a hierarchical classification system, as follows:\n", | |
| "\n", | |
| "- **N**: Nervous system \n", | |
| "- **N05**: Psycholeptics \n", | |
| "- **N05C**: Hypnotics and sedatives \n", | |
| "- **N05CH**: Melatonin receptor agonists \n", | |
| "- **N05CH01**: 2mg melatonin tablet \n", | |
| "\n", | |
| "As this example shows, the ATC code provides valuable information about a drug’s category and function. Since the main dataset from the Food and Drug Administration includes the ATC code for each drug, we were able to join the two datasets. This allowed us to add columns describing the drug's classification and uses, enriching the dataset with structured, meaningful data." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "yku9-tEzGxwP" | |
| }, | |
| "source": [ | |
| "## 2. Normalization and Preprocessing of the Database\n", | |
| "\n", | |
| "### Removing Unnecessary Data \n", | |
| "The drug database from the Iranian Food and Drug Administration included several columns that were not relevant to this project. These columns included data such as sales statistics, drug prices, and over-the-counter availability. Since the removal of these columns did not involve any complexity, they were manually deleted from the file." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "JpwO0bj9MKfq" | |
| }, | |
| "source": [ | |
| "### Data Normalization\n", | |
| "\n", | |
| "Several normalization steps were applied to the drug dataset. Specifically, all entries in the table were converted to their lowercase form to ensure consistency and improve search performance." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 4, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 501 | |
| }, | |
| "id": "e4BUmHkrMKfq", | |
| "outputId": "8b6fb19c-9eda-47f1-9b72-a5005573905f" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"medicine_ds\",\n \"rows\": 52445,\n \"fields\": [\n {\n \"column\": \"pharma_company\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 586,\n \"samples\": [\n \"\\u067e\\u0627\\u0631\\u0633 \\u062a\\u06a9 \\u0631\\u062e\",\n \"R-pharm Germany Gmbh\",\n \"\\u0641\\u0646 \\u0622\\u0648\\u0631\\u06cc \\u0632\\u06cc\\u0633\\u062a\\u06cc \\u0631\\u0632\\u0641\\u0627\\u0631\\u0645\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"supplier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 306,\n \"samples\": [\n \"\\u062f\\u0627\\u0631\\u0648 \\u0633\\u0627\\u0632\\u06cc \\u0622\\u0631\\u0627\\u0645\\u06cc\\u0633 \\u0641\\u0627\\u0631\\u0645\\u062f\",\n \"\\u0627\\u06cc\\u0645\\u0627\\u0698\\u0646\",\n \"\\u06a9\\u06cc \\u0628\\u06cc \\u0633\\u06cc\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_owner\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 592,\n \"samples\": [\n \"Novonordisk\",\n \"Unimark Remedies Ltd.\",\n \"Meda Pharma Gmbh & Co. Kg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distributor\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 43,\n \"samples\": [\n \"\\u0645\\u0634\\u06a9\\u0627\\u062a \\u0641\\u0627\\u0631\\u0645\\u062f \\u062a\\u0648\\u0632\\u06cc\\u0639\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u0628\\u0647\\u062f\\u0627\\u0634\\u062a \\u0634\\u0641\\u0627 \\u0622\\u0631\\u0627\\u062f\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u06af\\u0633\\u062a\\u0631 \\u0628\\u0627\\u0631\\u06cc\\u062c \\u0627\\u0633\\u0627\\u0646\\u0633\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"manufacturer_country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 44,\n \"samples\": [\n \"\\u0627\\u0631\\u062f\\u0646\",\n \"\\u0644\\u0647\\u0633\\u062a\\u0627\\u0646\",\n \"\\u0647\\u0644\\u0646\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_fa\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6663,\n \"samples\": [\n \"\\u06af\\u0644\\u06cc\\u06a9\\u0648\\u0648\\u0631 \\u0642\\u0631\\u0635 \\u067e\\u06cc\\u0648\\u0633\\u062a\\u0647 \\u0631\\u0647\\u0634 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg\",\n \"\\u0627\\u0633\\u062a\\u06cc\\u0645\\u062f\\u06cc\\u062a \\u0642\\u0631\\u0635 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 10 mg\",\n \"\\u0627\\u06cc\\u0646\\u0641\\u0644\\u0648\\u0648\\u06cc\\u0631 \\u067e\\u0648\\u062f\\u0631 \\u0628\\u0631\\u0627\\u06cc \\u062a\\u0647\\u06cc\\u0647 \\u0633\\u0648\\u0633\\u067e\\u0627\\u0646\\u0633\\u06cc\\u0648\\u0646 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg/5mL 75 mg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6630,\n \"samples\": [\n \"PIOGLITAZONE (DORSADAROU) 15MG TAB TABLET ORAL 15 mg\",\n \"CEBEX CAPSULE ORAL 200 mg\",\n \"VIOBION INJECTION PARENTERAL 100 mg/100 mg/1000 ug/3mL mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"generic_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2141,\n \"samples\": [\n \"FEXOFENADINE HYDROCHLORIDE SUSPENSION ORAL 30 mg/5mL 100 mL\",\n \"TRETINOIN LOTION TOPICAL 0.05 % 15 mL\",\n \"METHOTREXATE INJECTION, SOLUTION PARENTERAL 25 mg/1mL 0.8 mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"substance_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 998,\n \"samples\": [\n \"PYRVINIUM\",\n \"POLYMYXIN B / NEOMYCIN SULFATE / HYDROCORTISONE\",\n \"LABETALOL HYDROCHLORIDE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_code\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 786,\n \"samples\": [\n \"S01BA02\",\n \"N05CF02\",\n \"S01EB01\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "medicine_ds" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-0f1879bd-b67a-4e74-93ac-fce6c2ff421b\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>supplier</th>\n", | |
| " <th>brand_owner</th>\n", | |
| " <th>distributor</th>\n", | |
| " <th>manufacturer_country</th>\n", | |
| " <th>brand_name_fa</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>Actero middleeast</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>فاویپیراویر قرص خوراکی 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR</td>\n", | |
| " <td>J05AX</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL</td>\n", | |
| " <td>REMDESIVIR</td>\n", | |
| " <td>J05</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>آسنترا قرص خوراکی 50 mg</td>\n", | |
| " <td>ASENTRA TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE)</td>\n", | |
| " <td>N06AB06</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>ناکسپرین تزریقی پرنترال 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>NOXPRIN INJECTION PARENTERAL 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>ENOXAPARIN SODIUM INJECTION PARENTERAL 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>ENOXAPARIN SODIUM</td>\n", | |
| " <td>B01AB05</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0f1879bd-b67a-4e74-93ac-fce6c2ff421b')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-0f1879bd-b67a-4e74-93ac-fce6c2ff421b button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-0f1879bd-b67a-4e74-93ac-fce6c2ff421b');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-67d53497-2078-4b9e-898f-538c12de9030\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-67d53497-2078-4b9e-898f-538c12de9030')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-67d53497-2078-4b9e-898f-538c12de9030 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " pharma_company supplier brand_owner distributor manufacturer_country brand_name_fa brand_name_en generic_name_en substance_name_en atc_code\n", | |
| "0 Actero middleeast اکتوورکو اکتوورکو الیت دارو ایران فاویپیراویر قرص خوراکی 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR J05AX\n", | |
| "1 Actoverco اکتوورکو اکتوورکو الیت دارو ایران رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL REMDESIVIR J05\n", | |
| "2 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02\n", | |
| "3 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران آسنترا قرص خوراکی 50 mg ASENTRA TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) N06AB06\n", | |
| "4 Actoverco اکتوورکو اکتوورکو الیت دارو ایران ناکسپرین تزریقی پرنترال 100 mg/1mL 0.4 mL NOXPRIN INJECTION PARENTERAL 100 mg/1mL 0.4 mL ENOXAPARIN SODIUM INJECTION PARENTERAL 100 mg/1mL 0.4 mL ENOXAPARIN SODIUM B01AB05" | |
| ] | |
| }, | |
| "execution_count": 4, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "IR_FDA_DATASET_PATH = \"./Iran_FDA_1400_Dataset.csv\"\n", | |
| "WHO_ATC_INDEX_PATH = \"./WHO_ATC_Index.csv\"\n", | |
| "\n", | |
| "medicine_ds = pd.read_csv(IR_FDA_DATASET_PATH)\n", | |
| "\n", | |
| "medicine_ds.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 5, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 206 | |
| }, | |
| "id": "fiWzlJT4MKfq", | |
| "outputId": "682b61bf-ad50-450a-9ad0-35c73a2c2523" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"atc_index_ds\",\n \"rows\": 6971,\n \"fields\": [\n {\n \"column\": \"atc_code\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6440,\n \"samples\": [\n \"P01AC02\",\n \"C10AB04\",\n \"G03GA09\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5617,\n \"samples\": [\n \"phensuximide\",\n \"BACTERIAL AND VIRAL VACCINES, COMBINED\",\n \"niperotidine\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"ddd\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 35.20143995429928,\n \"min\": 0.1,\n \"max\": 800.0,\n \"num_unique_values\": 174,\n \"samples\": [\n 0.516,\n 0.57,\n 8.5\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"uom\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"mmol\",\n \"g\",\n \"TU\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"adm_r\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 20,\n \"samples\": [\n \"O\",\n \"Instill.solution\",\n \"Chewing gum\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"note\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 116,\n \"samples\": [\n \"Expressed as lysine acetylsalisylate\",\n \"Refers to insulin degludec\",\n \"\\\"vaginal insert, refers to the content of one vaginal insert\\\"\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "atc_index_ds" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-7e5d321e-d0aa-4cf0-90ba-52b2eb22d8cc\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>atc_code</th>\n", | |
| " <th>atc_name</th>\n", | |
| " <th>ddd</th>\n", | |
| " <th>uom</th>\n", | |
| " <th>adm_r</th>\n", | |
| " <th>note</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>A</td>\n", | |
| " <td>ALIMENTARY TRACT AND METABOLISM</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>A01</td>\n", | |
| " <td>STOMATOLOGICAL PREPARATIONS</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>A01A</td>\n", | |
| " <td>STOMATOLOGICAL PREPARATIONS</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>A01AA</td>\n", | |
| " <td>Caries prophylactic agents</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>A01AA01</td>\n", | |
| " <td>sodium fluoride</td>\n", | |
| " <td>1.1</td>\n", | |
| " <td>mg</td>\n", | |
| " <td>O</td>\n", | |
| " <td>0.5 mg fluoride</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7e5d321e-d0aa-4cf0-90ba-52b2eb22d8cc')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-7e5d321e-d0aa-4cf0-90ba-52b2eb22d8cc button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-7e5d321e-d0aa-4cf0-90ba-52b2eb22d8cc');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-7a8bda63-7d1b-4fc5-af77-ab65fcf60503\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7a8bda63-7d1b-4fc5-af77-ab65fcf60503')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-7a8bda63-7d1b-4fc5-af77-ab65fcf60503 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " atc_code atc_name ddd uom adm_r note\n", | |
| "0 A ALIMENTARY TRACT AND METABOLISM NaN NaN NaN NaN\n", | |
| "1 A01 STOMATOLOGICAL PREPARATIONS NaN NaN NaN NaN\n", | |
| "2 A01A STOMATOLOGICAL PREPARATIONS NaN NaN NaN NaN\n", | |
| "3 A01AA Caries prophylactic agents NaN NaN NaN NaN\n", | |
| "4 A01AA01 sodium fluoride 1.1 mg O 0.5 mg fluoride" | |
| ] | |
| }, | |
| "execution_count": 5, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "atc_index_ds = pd.read_csv(WHO_ATC_INDEX_PATH)\n", | |
| "\n", | |
| "atc_index_ds.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 6, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 380 | |
| }, | |
| "id": "XkjvaPS0MKfq", | |
| "outputId": "9f4b3ac9-8fe6-459c-f2e3-d8648f122269" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"medicine_ds\",\n \"rows\": 52445,\n \"fields\": [\n {\n \"column\": \"pharma_company\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 583,\n \"samples\": [\n \"nano fanavaran darouei alvand (nanoalvand)\",\n \"aryatinagene\",\n \"aburaihan pharmaceutical co\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"supplier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 306,\n \"samples\": [\n \"\\u062f\\u0627\\u0631\\u0648 \\u0633\\u0627\\u0632\\u06cc \\u0622\\u0631\\u0627\\u0645\\u06cc\\u0633 \\u0641\\u0627\\u0631\\u0645\\u062f\",\n \"\\u0627\\u06cc\\u0645\\u0627\\u0698\\u0646\",\n \"\\u06a9\\u06cc \\u0628\\u06cc \\u0633\\u06cc\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_owner\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 586,\n \"samples\": [\n \"\\u0627\\u0648\\u06a9\\u0633\\u06cc\\u0646 \\u062f\\u0627\\u0631\\u0648\\u06cc \\u0648\\u0634\\u062a\",\n \"piramal enterprises limited\",\n \"b braun\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distributor\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 43,\n \"samples\": [\n \"\\u0645\\u0634\\u06a9\\u0627\\u062a \\u0641\\u0627\\u0631\\u0645\\u062f \\u062a\\u0648\\u0632\\u06cc\\u0639\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u0628\\u0647\\u062f\\u0627\\u0634\\u062a \\u0634\\u0641\\u0627 \\u0622\\u0631\\u0627\\u062f\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u06af\\u0633\\u062a\\u0631 \\u0628\\u0627\\u0631\\u06cc\\u062c \\u0627\\u0633\\u0627\\u0646\\u0633\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"manufacturer_country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 44,\n \"samples\": [\n \"\\u0627\\u0631\\u062f\\u0646\",\n \"\\u0644\\u0647\\u0633\\u062a\\u0627\\u0646\",\n \"\\u0647\\u0644\\u0646\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_fa\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6663,\n \"samples\": [\n \"\\u06af\\u0644\\u06cc\\u06a9\\u0648\\u0648\\u0631 \\u0642\\u0631\\u0635 \\u067e\\u06cc\\u0648\\u0633\\u062a\\u0647 \\u0631\\u0647\\u0634 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg\",\n \"\\u0627\\u0633\\u062a\\u06cc\\u0645\\u062f\\u06cc\\u062a \\u0642\\u0631\\u0635 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 10 mg\",\n \"\\u0627\\u06cc\\u0646\\u0641\\u0644\\u0648\\u0648\\u06cc\\u0631 \\u067e\\u0648\\u062f\\u0631 \\u0628\\u0631\\u0627\\u06cc \\u062a\\u0647\\u06cc\\u0647 \\u0633\\u0648\\u0633\\u067e\\u0627\\u0646\\u0633\\u06cc\\u0648\\u0646 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg/5ml 75 mg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6630,\n \"samples\": [\n \"pioglitazone (dorsadarou) 15mg tab tablet oral 15 mg\",\n \"cebex capsule oral 200 mg\",\n \"viobion injection parenteral 100 mg/100 mg/1000 ug/3ml ml\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"generic_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2141,\n \"samples\": [\n \"fexofenadine hydrochloride suspension oral 30 mg/5ml 100 ml\",\n \"tretinoin lotion topical 0.05 % 15 ml\",\n \"methotrexate injection, solution parenteral 25 mg/1ml 0.8 ml\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"substance_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 998,\n \"samples\": [\n \"pyrvinium\",\n \"polymyxin b / neomycin sulfate / hydrocortisone\",\n \"labetalol hydrochloride\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_code\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 783,\n \"samples\": [\n \"n02ab02\",\n \"r01aa04\",\n \"b05xa02\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "medicine_ds" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-433cef91-b443-438c-8e74-3ea92eea4dc6\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>supplier</th>\n", | |
| " <th>brand_owner</th>\n", | |
| " <th>distributor</th>\n", | |
| " <th>manufacturer_country</th>\n", | |
| " <th>brand_name_fa</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>actero middleeast</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>فاویپیراویر قرص خوراکی 200 mg</td>\n", | |
| " <td>favipiravir tablet oral 200 mg</td>\n", | |
| " <td>favipiravir tablet oral 200 mg</td>\n", | |
| " <td>favipiravir</td>\n", | |
| " <td>j05ax</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>رمدسیویر محلول تزریقی پرنترال 5 mg/1ml 20 mg</td>\n", | |
| " <td>remdesivir injection, solution parenteral 5 mg/1ml 20 mg</td>\n", | |
| " <td>remdesivir injection, solution parenteral 5 mg/1ml 20 ml</td>\n", | |
| " <td>remdesivir</td>\n", | |
| " <td>j05</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>krka, d. d., novo mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " <td>nolpaza tablet, delayed release oral 40 mg</td>\n", | |
| " <td>pantoprazole tablet, delayed release oral 40 mg</td>\n", | |
| " <td>pantoprazole</td>\n", | |
| " <td>a02bc02</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>krka, d. d., novo mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>آسنترا قرص خوراکی 50 mg</td>\n", | |
| " <td>asentra tablet oral 50 mg</td>\n", | |
| " <td>sertraline (as hydrochloride) tablet oral 50 mg</td>\n", | |
| " <td>sertraline (as hydrochloride)</td>\n", | |
| " <td>n06ab06</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>ناکسپرین تزریقی پرنترال 100 mg/1ml 0.4 ml</td>\n", | |
| " <td>noxprin injection parenteral 100 mg/1ml 0.4 ml</td>\n", | |
| " <td>enoxaparin sodium injection parenteral 100 mg/1ml 0.4 ml</td>\n", | |
| " <td>enoxaparin sodium</td>\n", | |
| " <td>b01ab05</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-433cef91-b443-438c-8e74-3ea92eea4dc6')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-433cef91-b443-438c-8e74-3ea92eea4dc6 button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-433cef91-b443-438c-8e74-3ea92eea4dc6');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-2fdbdfca-ee5a-44cc-a85f-2dbf87451086\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2fdbdfca-ee5a-44cc-a85f-2dbf87451086')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-2fdbdfca-ee5a-44cc-a85f-2dbf87451086 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " pharma_company supplier brand_owner distributor manufacturer_country brand_name_fa brand_name_en generic_name_en substance_name_en atc_code\n", | |
| "0 actero middleeast اکتوورکو اکتوورکو الیت دارو ایران فاویپیراویر قرص خوراکی 200 mg favipiravir tablet oral 200 mg favipiravir tablet oral 200 mg favipiravir j05ax\n", | |
| "1 actoverco اکتوورکو اکتوورکو الیت دارو ایران رمدسیویر محلول تزریقی پرنترال 5 mg/1ml 20 mg remdesivir injection, solution parenteral 5 mg/1ml 20 mg remdesivir injection, solution parenteral 5 mg/1ml 20 ml remdesivir j05\n", | |
| "2 actoverco اکتوورکو krka, d. d., novo mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg nolpaza tablet, delayed release oral 40 mg pantoprazole tablet, delayed release oral 40 mg pantoprazole a02bc02\n", | |
| "3 actoverco اکتوورکو krka, d. d., novo mesto الیت دارو ایران آسنترا قرص خوراکی 50 mg asentra tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06\n", | |
| "4 actoverco اکتوورکو اکتوورکو الیت دارو ایران ناکسپرین تزریقی پرنترال 100 mg/1ml 0.4 ml noxprin injection parenteral 100 mg/1ml 0.4 ml enoxaparin sodium injection parenteral 100 mg/1ml 0.4 ml enoxaparin sodium b01ab05" | |
| ] | |
| }, | |
| "execution_count": 6, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from langdetect import detect\n", | |
| "\n", | |
| "medicine_ds[medicine_ds.select_dtypes(['object']).columns] = medicine_ds.select_dtypes(['object']).apply(lambda x: x.str.lower())\n", | |
| "\n", | |
| "medicine_ds.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 7, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 206 | |
| }, | |
| "id": "9GmA8nKGMKfq", | |
| "outputId": "cb82841f-2db1-461d-8748-9c336d7c680d" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"atc_index_ds\",\n \"rows\": 6971,\n \"fields\": [\n {\n \"column\": \"atc_code\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6440,\n \"samples\": [\n \"p01ac02\",\n \"c10ab04\",\n \"g03ga09\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_name\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5552,\n \"samples\": [\n \"amikacin\",\n \"hexobarbital\",\n \"dalbavancin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"ddd\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 35.20143995429928,\n \"min\": 0.1,\n \"max\": 800.0,\n \"num_unique_values\": 174,\n \"samples\": [\n 0.516,\n 0.57,\n 8.5\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"uom\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"mmol\",\n \"g\",\n \"tu\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"adm_r\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 20,\n \"samples\": [\n \"o\",\n \"instill.solution\",\n \"chewing gum\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"note\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 116,\n \"samples\": [\n \"expressed as lysine acetylsalisylate\",\n \"refers to insulin degludec\",\n \"\\\"vaginal insert, refers to the content of one vaginal insert\\\"\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "atc_index_ds" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-fbb00820-d788-42d1-b5e2-ece4daca309a\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>atc_code</th>\n", | |
| " <th>atc_name</th>\n", | |
| " <th>ddd</th>\n", | |
| " <th>uom</th>\n", | |
| " <th>adm_r</th>\n", | |
| " <th>note</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>a</td>\n", | |
| " <td>alimentary tract and metabolism</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>a01</td>\n", | |
| " <td>stomatological preparations</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>a01a</td>\n", | |
| " <td>stomatological preparations</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>a01aa</td>\n", | |
| " <td>caries prophylactic agents</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>a01aa01</td>\n", | |
| " <td>sodium fluoride</td>\n", | |
| " <td>1.1</td>\n", | |
| " <td>mg</td>\n", | |
| " <td>o</td>\n", | |
| " <td>0.5 mg fluoride</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fbb00820-d788-42d1-b5e2-ece4daca309a')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-fbb00820-d788-42d1-b5e2-ece4daca309a button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-fbb00820-d788-42d1-b5e2-ece4daca309a');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-5f300f63-bf5b-4648-ae2c-629c23960375\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5f300f63-bf5b-4648-ae2c-629c23960375')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-5f300f63-bf5b-4648-ae2c-629c23960375 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " atc_code atc_name ddd uom adm_r note\n", | |
| "0 a alimentary tract and metabolism NaN NaN NaN NaN\n", | |
| "1 a01 stomatological preparations NaN NaN NaN NaN\n", | |
| "2 a01a stomatological preparations NaN NaN NaN NaN\n", | |
| "3 a01aa caries prophylactic agents NaN NaN NaN NaN\n", | |
| "4 a01aa01 sodium fluoride 1.1 mg o 0.5 mg fluoride" | |
| ] | |
| }, | |
| "execution_count": 7, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "atc_index_ds[atc_index_ds.select_dtypes(['object']).columns] = atc_index_ds.select_dtypes(['object']).apply(lambda x: x.str.lower())\n", | |
| "\n", | |
| "atc_index_ds.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "2nGbhK6OMKfq" | |
| }, | |
| "source": [ | |
| "### Separating Meaningful Segments of ATC Codes\n", | |
| "\n", | |
| "As mentioned earlier, ATC drug codes consist of meaningful segments that, in a hierarchical manner, define the drug’s classification and its area of use. In this step, each drug’s ATC code was split into five segments based on official documentation describing the ATC code structure. These five segments were then added to the main dataset as five new columns (features). Below is the implemented code that performs this segmentation of the ATC identifiers." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 8, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "NwWADlJFRtOp", | |
| "outputId": "6c03ebf5-317e-4e41-e259-f6342163df79" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Requirement already satisfied: datasets in /usr/local/lib/python3.11/dist-packages (3.5.0)\n", | |
| "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from datasets) (3.18.0)\n", | |
| "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.11/dist-packages (from datasets) (1.26.4)\n", | |
| "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.11/dist-packages (from datasets) (18.1.0)\n", | |
| "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.11/dist-packages (from datasets) (0.3.8)\n", | |
| "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from datasets) (2.2.2)\n", | |
| "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.11/dist-packages (from datasets) (2.32.3)\n", | |
| "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.11/dist-packages (from datasets) (4.67.1)\n", | |
| "Requirement already satisfied: xxhash in /usr/local/lib/python3.11/dist-packages (from datasets) (3.5.0)\n", | |
| "Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.11/dist-packages (from datasets) (0.70.16)\n", | |
| "Requirement already satisfied: fsspec<=2024.12.0,>=2023.1.0 in /usr/local/lib/python3.11/dist-packages (from fsspec[http]<=2024.12.0,>=2023.1.0->datasets) (2024.12.0)\n", | |
| "Requirement already satisfied: aiohttp in /usr/local/lib/python3.11/dist-packages (from datasets) (3.11.14)\n", | |
| "Requirement already satisfied: huggingface-hub>=0.24.0 in /usr/local/lib/python3.11/dist-packages (from datasets) (0.29.3)\n", | |
| "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from datasets) (24.2)\n", | |
| "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.11/dist-packages (from datasets) (6.0.2)\n", | |
| "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (2.6.1)\n", | |
| "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (1.3.2)\n", | |
| "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (25.3.0)\n", | |
| "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (1.5.0)\n", | |
| "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (6.2.0)\n", | |
| "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (0.3.0)\n", | |
| "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets) (1.18.3)\n", | |
| "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.11/dist-packages (from huggingface-hub>=0.24.0->datasets) (4.12.2)\n", | |
| "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets) (3.4.1)\n", | |
| "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets) (3.10)\n", | |
| "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets) (2.3.0)\n", | |
| "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets) (2025.1.31)\n", | |
| "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets) (2.8.2)\n", | |
| "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets) (2025.1)\n", | |
| "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets) (2025.1)\n", | |
| "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "!pip install datasets" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 9, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 544, | |
| "referenced_widgets": [ | |
| "b83e67beab3a450f91eb63e7253e9983", | |
| "029c8e2c61b94890a67d89cc7822eab2", | |
| "c5304ea1df914bb8b5ccaa38644a22f3", | |
| "114f418af640412db31186b4378fd352", | |
| "63d34abfc4bc477eb993129eb157c5a8", | |
| "805b5a92eaf741f09c3c7883dc25fe20", | |
| "88ec21c0a38a403699c8c3fd449e6c70", | |
| "4ee92207f4fc4a9fae389e606465744c", | |
| "572edb77173c4a95aaa4aa3dfc482ccb", | |
| "7e6352a42779415682747d524ccb86a9", | |
| "e1cd45a5b2bd44f18b451633010d19ae", | |
| "99b5597c69bf4142a892c774283223db", | |
| "7b9b276623214d5eb0167c5c9f7534af", | |
| "25dc2816ee694a12aadb7bdd0b71ff4e", | |
| "d2430694dc904c9495b4d68dba34c5ab", | |
| "5754319de5db44d9941c8d8e606de329", | |
| "5a4a9ea4a52d476ca4c15a6986f9e9b2", | |
| "071fb47dcf02494487d9e1e8e73ef67b", | |
| "e7cfaa0839fb4c4eb10aa8f6494d28b4", | |
| "665820fe611043e3a183c2d27c7a735e", | |
| "15b38fef0de042a5947b793df8c50b27", | |
| "06e1f4c76efd41a7becdf320236c06ed" | |
| ] | |
| }, | |
| "id": "lyIJ0Rt9QihZ", | |
| "outputId": "88d3e8a2-b0f7-4ee7-f53a-3b4025673960" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.jupyter.widget-view+json": { | |
| "model_id": "b83e67beab3a450f91eb63e7253e9983", | |
| "version_major": 2, | |
| "version_minor": 0 | |
| }, | |
| "text/plain": [ | |
| "Uploading the dataset shards: 0%| | 0/1 [00:00<?, ?it/s]" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| }, | |
| { | |
| "data": { | |
| "application/vnd.jupyter.widget-view+json": { | |
| "model_id": "99b5597c69bf4142a892c774283223db", | |
| "version_major": 2, | |
| "version_minor": 0 | |
| }, | |
| "text/plain": [ | |
| "Creating parquet from Arrow format: 0%| | 0/42 [00:00<?, ?ba/s]" | |
| ] | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| }, | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n", | |
| "The secret `HF_TOKEN` does not exist in your Colab secrets.\n", | |
| "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n", | |
| "You will be able to reuse this secret in all of your notebooks.\n", | |
| "Please note that authentication is recommended but still optional to access public models or datasets.\n", | |
| " warnings.warn(\n" | |
| ] | |
| }, | |
| { | |
| "ename": "KeyboardInterrupt", | |
| "evalue": "", | |
| "output_type": "error", | |
| "traceback": [ | |
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | |
| "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", | |
| "\u001b[0;32m<ipython-input-9-0ce0da67bf71>\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 10\u001b[0m })\n\u001b[1;32m 11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mmedicine_dataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpush_to_hub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tahamajs/Iran_FDA_1400_Dataset\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtoken\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"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\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0mdf_train_atc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdf_test_atc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0matc_index_ds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m42\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/datasets/dataset_dict.py\u001b[0m in \u001b[0;36mpush_to_hub\u001b[0;34m(self, repo_id, config_name, set_default, data_dir, commit_message, commit_description, private, token, revision, create_pr, max_shard_size, num_shards, embed_external_files)\u001b[0m\n\u001b[1;32m 1743\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Pushing split {split} to the Hub.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1744\u001b[0m \u001b[0;31m# The split=key needs to be removed before merging\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1745\u001b[0;31m split_additions, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(\n\u001b[0m\u001b[1;32m 1746\u001b[0m \u001b[0mrepo_id\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1747\u001b[0m \u001b[0mdata_dir\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata_dir\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py\u001b[0m in \u001b[0;36m_push_parquet_shards_to_hub\u001b[0;34m(self, repo_id, data_dir, split, token, revision, create_pr, max_shard_size, num_shards, embed_external_files)\u001b[0m\n\u001b[1;32m 5388\u001b[0m \u001b[0muploaded_size\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtell\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5389\u001b[0m \u001b[0mshard_addition\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCommitOperationAdd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_in_repo\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mshard_path_in_repo\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath_or_fileobj\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5390\u001b[0;31m api.preupload_lfs_files(\n\u001b[0m\u001b[1;32m 5391\u001b[0m \u001b[0mrepo_id\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrepo_id\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5392\u001b[0m \u001b[0madditions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mshard_addition\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/hf_api.py\u001b[0m in \u001b[0;36mpreupload_lfs_files\u001b[0;34m(self, repo_id, additions, token, repo_type, revision, create_pr, num_threads, free_memory, gitignore_content)\u001b[0m\n\u001b[1;32m 4243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4244\u001b[0m \u001b[0;31m# Upload new LFS files\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4245\u001b[0;31m _upload_lfs_files(\n\u001b[0m\u001b[1;32m 4246\u001b[0m \u001b[0madditions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnew_lfs_additions_to_upload\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4247\u001b[0m \u001b[0mrepo_type\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrepo_type\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_validators.py\u001b[0m in \u001b[0;36m_inner_fn\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 112\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmoothly_deprecate_use_auth_token\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhas_token\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhas_token\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 113\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 114\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 115\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 116\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0m_inner_fn\u001b[0m \u001b[0;31m# type: ignore\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/_commit_api.py\u001b[0m in \u001b[0;36m_upload_lfs_files\u001b[0;34m(additions, repo_type, repo_id, headers, endpoint, num_threads, revision)\u001b[0m\n\u001b[1;32m 445\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_actions\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 446\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Uploading 1 LFS file to the Hub\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 447\u001b[0;31m \u001b[0m_wrapped_lfs_upload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_actions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 448\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 449\u001b[0m logger.debug(\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/_commit_api.py\u001b[0m in \u001b[0;36m_wrapped_lfs_upload\u001b[0;34m(batch_action)\u001b[0m\n\u001b[1;32m 435\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 436\u001b[0m \u001b[0moperation\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moid2addop\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbatch_action\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"oid\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 437\u001b[0;31m \u001b[0mlfs_upload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlfs_batch_action\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbatch_action\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendpoint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mendpoint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 438\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 439\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Error while uploading '{operation.path_in_repo}' to the Hub.\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/lfs.py\u001b[0m in \u001b[0;36mlfs_upload\u001b[0;34m(operation, lfs_batch_action, token, headers, endpoint)\u001b[0m\n\u001b[1;32m 246\u001b[0m \u001b[0m_upload_multi_part\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheader\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchunk_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupload_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mupload_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 247\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 248\u001b[0;31m \u001b[0m_upload_single_part\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moperation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupload_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mupload_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 249\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 250\u001b[0m \u001b[0;31m# 3. Verify upload went well\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/lfs.py\u001b[0m in \u001b[0;36m_upload_single_part\u001b[0;34m(operation, upload_url)\u001b[0m\n\u001b[1;32m 317\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0moperation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwith_tqdm\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfileobj\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 318\u001b[0m \u001b[0;31m# S3 might raise a transient 500 error -> let's retry if that happens\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 319\u001b[0;31m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp_backoff\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"PUT\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupload_url\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfileobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretry_on_status_codes\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m500\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m502\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m503\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m504\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 320\u001b[0m \u001b[0mhf_raise_for_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 321\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_http.py\u001b[0m in \u001b[0;36mhttp_backoff\u001b[0;34m(method, url, max_retries, base_wait_time, max_wait_time, retry_on_exceptions, retry_on_status_codes, **kwargs)\u001b[0m\n\u001b[1;32m 308\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 309\u001b[0m \u001b[0;31m# Perform request and return if status_code is not in the retry list.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 310\u001b[0;31m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 311\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mretry_on_status_codes\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 312\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m 587\u001b[0m }\n\u001b[1;32m 588\u001b[0m \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 590\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 591\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m 701\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 702\u001b[0m \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 703\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 704\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 705\u001b[0m \u001b[0;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_http.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, *args, **kwargs)\u001b[0m\n\u001b[1;32m 94\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Send: {_curlify(request)}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 95\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 96\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 97\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRequestException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 98\u001b[0m \u001b[0mrequest_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_AMZN_TRACE_ID\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m 665\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 666\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 667\u001b[0;31m resp = conn.urlopen(\n\u001b[0m\u001b[1;32m 668\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 669\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)\u001b[0m\n\u001b[1;32m 785\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 786\u001b[0m \u001b[0;31m# Make the request on the HTTPConnection object\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 787\u001b[0;31m response = self._make_request(\n\u001b[0m\u001b[1;32m 788\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 789\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)\u001b[0m\n\u001b[1;32m 532\u001b[0m \u001b[0;31m# Receive the response from the server\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 533\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 534\u001b[0;31m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 535\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mBaseSSLError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mOSError\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 536\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_raise_timeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mread_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/urllib3/connection.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 514\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 515\u001b[0m \u001b[0;31m# Get the response from http.client.HTTPConnection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 516\u001b[0;31m \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 517\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 518\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1393\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1394\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1395\u001b[0;31m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1396\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1397\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 323\u001b[0m \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 324\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 325\u001b[0;31m \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 326\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 327\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 284\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 285\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 286\u001b[0;31m \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 287\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 288\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m 716\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 717\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 718\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 719\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 720\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/ssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m 1312\u001b[0m \u001b[0;34m\"non-zero flags not allowed in calls to recv_into() on %s\"\u001b[0m \u001b[0;34m%\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1313\u001b[0m self.__class__)\n\u001b[0;32m-> 1314\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1315\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1316\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/lib/python3.11/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m 1164\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1165\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mbuffer\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1166\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1167\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1168\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;31mKeyboardInterrupt\u001b[0m: " | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from datasets import Dataset, DatasetDict\n", | |
| "from sklearn.model_selection import train_test_split\n", | |
| "\n", | |
| "df_train_med, df_test_med = train_test_split(medicine_ds, test_size=0.2, random_state=42)\n", | |
| "\n", | |
| "medicine_dataset = DatasetDict({\n", | |
| " \"train\": Dataset.from_pandas(df_train_med.reset_index(drop=True)),\n", | |
| " \"test\": Dataset.from_pandas(df_test_med.reset_index(drop=True))\n", | |
| "})\n", | |
| "\n", | |
| "medicine_dataset.push_to_hub(\"tahamajs/Iran_FDA_1400_Dataset\", token=\"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\")\n", | |
| "\n", | |
| "df_train_atc, df_test_atc = train_test_split(atc_index_ds, test_size=0.2, random_state=42)\n", | |
| "\n", | |
| "atc_dataset = DatasetDict({\n", | |
| " \"train\": Dataset.from_pandas(df_train_atc.reset_index(drop=True)),\n", | |
| " \"test\": Dataset.from_pandas(df_test_atc.reset_index(drop=True))\n", | |
| "})\n", | |
| "\n", | |
| "atc_dataset.push_to_hub(\"tahamajs/WHO_ATC_Index\", token=\"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\")\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 17, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 521 | |
| }, | |
| "id": "7mbsSdReMKfq", | |
| "outputId": "c5d59ed6-f019-425e-8712-8dfa943700a7" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"medicine_ds\",\n \"rows\": 52445,\n \"fields\": [\n {\n \"column\": \"pharma_company\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 586,\n \"samples\": [\n \"\\u067e\\u0627\\u0631\\u0633 \\u062a\\u06a9 \\u0631\\u062e\",\n \"R-pharm Germany Gmbh\",\n \"\\u0641\\u0646 \\u0622\\u0648\\u0631\\u06cc \\u0632\\u06cc\\u0633\\u062a\\u06cc \\u0631\\u0632\\u0641\\u0627\\u0631\\u0645\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"supplier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 306,\n \"samples\": [\n \"\\u062f\\u0627\\u0631\\u0648 \\u0633\\u0627\\u0632\\u06cc \\u0622\\u0631\\u0627\\u0645\\u06cc\\u0633 \\u0641\\u0627\\u0631\\u0645\\u062f\",\n \"\\u0627\\u06cc\\u0645\\u0627\\u0698\\u0646\",\n \"\\u06a9\\u06cc \\u0628\\u06cc \\u0633\\u06cc\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_owner\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 592,\n \"samples\": [\n \"Novonordisk\",\n \"Unimark Remedies Ltd.\",\n \"Meda Pharma Gmbh & Co. Kg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distributor\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 43,\n \"samples\": [\n \"\\u0645\\u0634\\u06a9\\u0627\\u062a \\u0641\\u0627\\u0631\\u0645\\u062f \\u062a\\u0648\\u0632\\u06cc\\u0639\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u0628\\u0647\\u062f\\u0627\\u0634\\u062a \\u0634\\u0641\\u0627 \\u0622\\u0631\\u0627\\u062f\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u06af\\u0633\\u062a\\u0631 \\u0628\\u0627\\u0631\\u06cc\\u062c \\u0627\\u0633\\u0627\\u0646\\u0633\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"manufacturer_country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 44,\n \"samples\": [\n \"\\u0627\\u0631\\u062f\\u0646\",\n \"\\u0644\\u0647\\u0633\\u062a\\u0627\\u0646\",\n \"\\u0647\\u0644\\u0646\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_fa\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6663,\n \"samples\": [\n \"\\u06af\\u0644\\u06cc\\u06a9\\u0648\\u0648\\u0631 \\u0642\\u0631\\u0635 \\u067e\\u06cc\\u0648\\u0633\\u062a\\u0647 \\u0631\\u0647\\u0634 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg\",\n \"\\u0627\\u0633\\u062a\\u06cc\\u0645\\u062f\\u06cc\\u062a \\u0642\\u0631\\u0635 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 10 mg\",\n \"\\u0627\\u06cc\\u0646\\u0641\\u0644\\u0648\\u0648\\u06cc\\u0631 \\u067e\\u0648\\u062f\\u0631 \\u0628\\u0631\\u0627\\u06cc \\u062a\\u0647\\u06cc\\u0647 \\u0633\\u0648\\u0633\\u067e\\u0627\\u0646\\u0633\\u06cc\\u0648\\u0646 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg/5mL 75 mg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6630,\n \"samples\": [\n \"PIOGLITAZONE (DORSADAROU) 15MG TAB TABLET ORAL 15 mg\",\n \"CEBEX CAPSULE ORAL 200 mg\",\n \"VIOBION INJECTION PARENTERAL 100 mg/100 mg/1000 ug/3mL mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"generic_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2141,\n \"samples\": [\n \"FEXOFENADINE HYDROCHLORIDE SUSPENSION ORAL 30 mg/5mL 100 mL\",\n \"TRETINOIN LOTION TOPICAL 0.05 % 15 mL\",\n \"METHOTREXATE INJECTION, SOLUTION PARENTERAL 25 mg/1mL 0.8 mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"substance_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 998,\n \"samples\": [\n \"PYRVINIUM\",\n \"POLYMYXIN B / NEOMYCIN SULFATE / HYDROCORTISONE\",\n \"LABETALOL HYDROCHLORIDE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_code_raw\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 786,\n \"samples\": [\n \"S01BA02\",\n \"N05CF02\",\n \"S01EB01\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"anatomical\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 20,\n \"samples\": [\n \"J\",\n \"a\",\n \"S\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"therapeutic\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 91,\n \"samples\": [\n \"A06\",\n \"A07\",\n \"R01\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"pharmacological\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 195,\n \"samples\": [\n \"D07C\",\n \"G04C\",\n \"A12C\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"subpharmacological\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 403,\n \"samples\": [\n \"J02AC\",\n \"C02AB\",\n \"J01XB\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"chemical\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 731,\n \"samples\": [\n \"C01EB17\",\n \"C09DB01\",\n \"N03AG04 \"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "medicine_ds" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-483341fc-025c-43ca-a8a9-ef1d442eef9c\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>supplier</th>\n", | |
| " <th>brand_owner</th>\n", | |
| " <th>distributor</th>\n", | |
| " <th>manufacturer_country</th>\n", | |
| " <th>brand_name_fa</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code_raw</th>\n", | |
| " <th>anatomical</th>\n", | |
| " <th>therapeutic</th>\n", | |
| " <th>pharmacological</th>\n", | |
| " <th>subpharmacological</th>\n", | |
| " <th>chemical</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>Actero middleeast</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>فاویپیراویر قرص خوراکی 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR</td>\n", | |
| " <td>J05AX</td>\n", | |
| " <td>J</td>\n", | |
| " <td>J05</td>\n", | |
| " <td>J05A</td>\n", | |
| " <td>J05AX</td>\n", | |
| " <td>None</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL</td>\n", | |
| " <td>REMDESIVIR</td>\n", | |
| " <td>J05</td>\n", | |
| " <td>J</td>\n", | |
| " <td>J05</td>\n", | |
| " <td>None</td>\n", | |
| " <td>None</td>\n", | |
| " <td>None</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " <td>A</td>\n", | |
| " <td>A02</td>\n", | |
| " <td>A02B</td>\n", | |
| " <td>A02BC</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>آسنترا قرص خوراکی 50 mg</td>\n", | |
| " <td>ASENTRA TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE)</td>\n", | |
| " <td>N06AB06</td>\n", | |
| " <td>N</td>\n", | |
| " <td>N06</td>\n", | |
| " <td>N06A</td>\n", | |
| " <td>N06AB</td>\n", | |
| " <td>N06AB06</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>ناکسپرین تزریقی پرنترال 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>NOXPRIN INJECTION PARENTERAL 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>ENOXAPARIN SODIUM INJECTION PARENTERAL 100 mg/1mL 0.4 mL</td>\n", | |
| " <td>ENOXAPARIN SODIUM</td>\n", | |
| " <td>B01AB05</td>\n", | |
| " <td>B</td>\n", | |
| " <td>B01</td>\n", | |
| " <td>B01A</td>\n", | |
| " <td>B01AB</td>\n", | |
| " <td>B01AB05</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-483341fc-025c-43ca-a8a9-ef1d442eef9c')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-483341fc-025c-43ca-a8a9-ef1d442eef9c button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-483341fc-025c-43ca-a8a9-ef1d442eef9c');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-2aee1d9d-7eb1-4d05-8dfc-39436a9d48f7\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2aee1d9d-7eb1-4d05-8dfc-39436a9d48f7')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-2aee1d9d-7eb1-4d05-8dfc-39436a9d48f7 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " pharma_company supplier brand_owner distributor manufacturer_country brand_name_fa brand_name_en generic_name_en substance_name_en atc_code_raw anatomical therapeutic pharmacological subpharmacological chemical\n", | |
| "0 Actero middleeast اکتوورکو اکتوورکو الیت دارو ایران فاویپیراویر قرص خوراکی 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR J05AX J J05 J05A J05AX None\n", | |
| "1 Actoverco اکتوورکو اکتوورکو الیت دارو ایران رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL REMDESIVIR J05 J J05 None None None\n", | |
| "2 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02 A A02 A02B A02BC A02BC02\n", | |
| "3 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران آسنترا قرص خوراکی 50 mg ASENTRA TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) N06AB06 N N06 N06A N06AB N06AB06\n", | |
| "4 Actoverco اکتوورکو اکتوورکو الیت دارو ایران ناکسپرین تزریقی پرنترال 100 mg/1mL 0.4 mL NOXPRIN INJECTION PARENTERAL 100 mg/1mL 0.4 mL ENOXAPARIN SODIUM INJECTION PARENTERAL 100 mg/1mL 0.4 mL ENOXAPARIN SODIUM B01AB05 B B01 B01A B01AB B01AB05" | |
| ] | |
| }, | |
| "execution_count": 17, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "def extract_atc_parts(atc_code):\n", | |
| " atc_code = str(atc_code)\n", | |
| " return [atc_code[:1], atc_code[:3] if len(atc_code) >= 3 else None,\n", | |
| " atc_code[:4] if len(atc_code) >= 4 else None, atc_code[:5] if len(atc_code) >= 5 else None,\n", | |
| " atc_code if len(atc_code) >= 7 else None]\n", | |
| "\n", | |
| "ATC_CATEGORIES = ['anatomical', 'therapeutic', 'pharmacological', 'subpharmacological', 'chemical']\n", | |
| "medicine_atc_splitted = medicine_ds['atc_code'].apply(extract_atc_parts).to_list()\n", | |
| "\n", | |
| "medicine_atc_columns = pd.DataFrame(medicine_atc_splitted, dtype='object', columns=ATC_CATEGORIES)\n", | |
| "medicine_ds = medicine_ds.join(medicine_atc_columns, how='inner')\n", | |
| "medicine_ds.rename(columns={'atc_code': 'atc_code_raw'}, inplace=True)\n", | |
| "\n", | |
| "medicine_ds.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "U79rKCBiMKfr" | |
| }, | |
| "source": [ | |
| "## 3. Merging the Drug Dataset with the ATC Code Classification Dataset\n", | |
| "\n", | |
| "In this step, we explain how the two datasets were merged. The primary drug dataset (from the Iranian Food and Drug Administration) includes the ATC code for each drug, while the secondary dataset (from GitHub) provides classification and hierarchical details for each ATC code. \n", | |
| "By using the ATC code as a common key, we performed a join operation to combine the two datasets. As a result, each drug entry in the main dataset was enriched with additional information such as the drug’s therapeutic category, group, and specific function, as defined by the ATC classification. This enhanced the dataset’s structure and made it more informative for analysis and search functionality." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 29, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 521 | |
| }, | |
| "id": "H-G4jUyuUkHa", | |
| "outputId": "19ae8843-8e8e-4be4-de69-ad14fdc63ed9" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"medicine_ds_merged\",\n \"rows\": 65469,\n \"fields\": [\n {\n \"column\": \"pharma_company\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 586,\n \"samples\": [\n \"\\u067e\\u0627\\u0631\\u0633 \\u062a\\u06a9 \\u0631\\u062e\",\n \"R-pharm Germany Gmbh\",\n \"\\u0641\\u0646 \\u0622\\u0648\\u0631\\u06cc \\u0632\\u06cc\\u0633\\u062a\\u06cc \\u0631\\u0632\\u0641\\u0627\\u0631\\u0645\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"supplier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 306,\n \"samples\": [\n \"\\u062f\\u0627\\u0631\\u0648 \\u0633\\u0627\\u0632\\u06cc \\u0622\\u0631\\u0627\\u0645\\u06cc\\u0633 \\u0641\\u0627\\u0631\\u0645\\u062f\",\n \"\\u0627\\u06cc\\u0645\\u0627\\u0698\\u0646\",\n \"\\u06a9\\u06cc \\u0628\\u06cc \\u0633\\u06cc\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_owner\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 592,\n \"samples\": [\n \"Novonordisk\",\n \"Unimark Remedies Ltd.\",\n \"Meda Pharma Gmbh & Co. Kg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distributor\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 43,\n \"samples\": [\n \"\\u0645\\u0634\\u06a9\\u0627\\u062a \\u0641\\u0627\\u0631\\u0645\\u062f \\u062a\\u0648\\u0632\\u06cc\\u0639\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u0628\\u0647\\u062f\\u0627\\u0634\\u062a \\u0634\\u0641\\u0627 \\u0622\\u0631\\u0627\\u062f\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u06af\\u0633\\u062a\\u0631 \\u0628\\u0627\\u0631\\u06cc\\u062c \\u0627\\u0633\\u0627\\u0646\\u0633\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"manufacturer_country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 44,\n \"samples\": [\n \"\\u0627\\u0631\\u062f\\u0646\",\n \"\\u0644\\u0647\\u0633\\u062a\\u0627\\u0646\",\n \"\\u0647\\u0644\\u0646\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_fa\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6663,\n \"samples\": [\n \"\\u06af\\u0644\\u06cc\\u06a9\\u0648\\u0648\\u0631 \\u0642\\u0631\\u0635 \\u067e\\u06cc\\u0648\\u0633\\u062a\\u0647 \\u0631\\u0647\\u0634 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg\",\n \"\\u0627\\u0633\\u062a\\u06cc\\u0645\\u062f\\u06cc\\u062a \\u0642\\u0631\\u0635 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 10 mg\",\n \"\\u0627\\u06cc\\u0646\\u0641\\u0644\\u0648\\u0648\\u06cc\\u0631 \\u067e\\u0648\\u062f\\u0631 \\u0628\\u0631\\u0627\\u06cc \\u062a\\u0647\\u06cc\\u0647 \\u0633\\u0648\\u0633\\u067e\\u0627\\u0646\\u0633\\u06cc\\u0648\\u0646 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg/5mL 75 mg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6630,\n \"samples\": [\n \"PIOGLITAZONE (DORSADAROU) 15MG TAB TABLET ORAL 15 mg\",\n \"CEBEX CAPSULE ORAL 200 mg\",\n \"VIOBION INJECTION PARENTERAL 100 mg/100 mg/1000 ug/3mL mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"generic_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2141,\n \"samples\": [\n \"FEXOFENADINE HYDROCHLORIDE SUSPENSION ORAL 30 mg/5mL 100 mL\",\n \"TRETINOIN LOTION TOPICAL 0.05 % 15 mL\",\n \"METHOTREXATE INJECTION, SOLUTION PARENTERAL 25 mg/1mL 0.8 mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"substance_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 998,\n \"samples\": [\n \"PYRVINIUM\",\n \"POLYMYXIN B / NEOMYCIN SULFATE / HYDROCORTISONE\",\n \"LABETALOL HYDROCHLORIDE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_code_raw\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 786,\n \"samples\": [\n \"S01BA02\",\n \"N05CF02\",\n \"S01EB01\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"anatomical_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 14,\n \"samples\": [\n \"MUSCULO-SKELETAL SYSTEM\",\n \"ANTIPARASITIC PRODUCTS, INSECTICIDES AND REPELLENTS\",\n \"ANTIINFECTIVES FOR SYSTEMIC USE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"therapeutic_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 82,\n \"samples\": [\n \"ENDOCRINE THERAPY\",\n \"ANTIVIRALS FOR SYSTEMIC USE\",\n \"ANTIDIARRHEALS, INTESTINAL ANTIINFLAMMATORY/ANTIINFECTIVE AGENTS\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"pharmacological_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 182,\n \"samples\": [\n \"PROGESTOGENS\",\n \"POTASSIUM-SPARING AGENTS\",\n \"CORTICOSTEROIDS, COMBINATIONS WITH ANTISEPTICS\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"subpharmacological_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 365,\n \"samples\": [\n \"ACE inhibitors, plain\",\n \"Direct factor Xa inhibitors\",\n \"HMG CoA reductase inhibitors\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"chemical_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 640,\n \"samples\": [\n \"azathioprine\",\n \"botulinum toxin\",\n \"diatrizoic acid\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "medicine_ds_merged" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-44f5628d-e74e-4403-a864-e8c52af93a8f\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>supplier</th>\n", | |
| " <th>brand_owner</th>\n", | |
| " <th>distributor</th>\n", | |
| " <th>manufacturer_country</th>\n", | |
| " <th>brand_name_fa</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code_raw</th>\n", | |
| " <th>anatomical_desc</th>\n", | |
| " <th>therapeutic_desc</th>\n", | |
| " <th>pharmacological_desc</th>\n", | |
| " <th>subpharmacological_desc</th>\n", | |
| " <th>chemical_desc</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>Actero middleeast</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>فاویپیراویر قرص خوراکی 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR</td>\n", | |
| " <td>J05AX</td>\n", | |
| " <td>ANTIINFECTIVES FOR SYSTEMIC USE</td>\n", | |
| " <td>ANTIVIRALS FOR SYSTEMIC USE</td>\n", | |
| " <td>DIRECT ACTING ANTIVIRALS</td>\n", | |
| " <td>Other antivirals</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL</td>\n", | |
| " <td>REMDESIVIR</td>\n", | |
| " <td>J05</td>\n", | |
| " <td>ANTIINFECTIVES FOR SYSTEMIC USE</td>\n", | |
| " <td>ANTIVIRALS FOR SYSTEMIC USE</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " <td>ALIMENTARY TRACT AND METABOLISM</td>\n", | |
| " <td>DRUGS FOR ACID RELATED DISORDERS</td>\n", | |
| " <td>DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD)</td>\n", | |
| " <td>Proton pump inhibitors</td>\n", | |
| " <td>pantoprazole</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " <td>ALIMENTARY TRACT AND METABOLISM</td>\n", | |
| " <td>DRUGS FOR ACID RELATED DISORDERS</td>\n", | |
| " <td>DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD)</td>\n", | |
| " <td>Proton pump inhibitors</td>\n", | |
| " <td>pantoprazole</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>آسنترا قرص خوراکی 50 mg</td>\n", | |
| " <td>ASENTRA TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE)</td>\n", | |
| " <td>N06AB06</td>\n", | |
| " <td>NERVOUS SYSTEM</td>\n", | |
| " <td>PSYCHOANALEPTICS</td>\n", | |
| " <td>ANTIDEPRESSANTS</td>\n", | |
| " <td>Selective serotonin reuptake inhibitors</td>\n", | |
| " <td>sertraline</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-44f5628d-e74e-4403-a864-e8c52af93a8f')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-44f5628d-e74e-4403-a864-e8c52af93a8f button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-44f5628d-e74e-4403-a864-e8c52af93a8f');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-c7442ed3-a26a-4978-9585-3c12e21f6773\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c7442ed3-a26a-4978-9585-3c12e21f6773')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-c7442ed3-a26a-4978-9585-3c12e21f6773 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " pharma_company supplier brand_owner distributor manufacturer_country brand_name_fa brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc\n", | |
| "0 Actero middleeast اکتوورکو اکتوورکو الیت دارو ایران فاویپیراویر قرص خوراکی 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR J05AX ANTIINFECTIVES FOR SYSTEMIC USE ANTIVIRALS FOR SYSTEMIC USE DIRECT ACTING ANTIVIRALS Other antivirals NaN\n", | |
| "1 Actoverco اکتوورکو اکتوورکو الیت دارو ایران رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL REMDESIVIR J05 ANTIINFECTIVES FOR SYSTEMIC USE ANTIVIRALS FOR SYSTEMIC USE NaN NaN NaN\n", | |
| "2 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02 ALIMENTARY TRACT AND METABOLISM DRUGS FOR ACID RELATED DISORDERS DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD) Proton pump inhibitors pantoprazole\n", | |
| "3 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02 ALIMENTARY TRACT AND METABOLISM DRUGS FOR ACID RELATED DISORDERS DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD) Proton pump inhibitors pantoprazole\n", | |
| "4 Actoverco اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران آسنترا قرص خوراکی 50 mg ASENTRA TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) N06AB06 NERVOUS SYSTEM PSYCHOANALEPTICS ANTIDEPRESSANTS Selective serotonin reuptake inhibitors sertraline" | |
| ] | |
| }, | |
| "execution_count": 29, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "\n", | |
| "atc_columns = ['anatomical', 'therapeutic', 'pharmacological', 'subpharmacological', 'chemical']\n", | |
| "desc_columns = ['anatomical_desc', 'therapeutic_desc', 'pharmacological_desc', 'subpharmacological_desc', 'chemical_desc']\n", | |
| "\n", | |
| "medicine_ds_merged = medicine_ds.copy()\n", | |
| "\n", | |
| "for atc_col, desc_col in zip(atc_columns, desc_columns):\n", | |
| " medicine_ds_merged = medicine_ds_merged.merge(atc_index_ds[['atc_code', 'atc_name']], left_on=atc_col, right_on='atc_code', how='left')\n", | |
| " medicine_ds_merged = medicine_ds_merged.rename(columns={'atc_name': desc_col})\n", | |
| " medicine_ds_merged = medicine_ds_merged.drop(columns=['atc_code'])\n", | |
| "\n", | |
| "medicine_ds_merged = medicine_ds_merged.drop(columns=atc_columns)\n", | |
| "medicine_ds_merged.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 30, | |
| "metadata": { | |
| "id": "JAEOCvw8UkHb" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "medicine_ds_merged.to_csv(\"./medicine_ds_merged.csv\", index=False)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "7at-PmqPMKfr" | |
| }, | |
| "source": [ | |
| "## 4. Separating the Persian and English Databases\n", | |
| "\n", | |
| "At this stage, the dataset contains columns that are written either in English or in Persian. For most of these columns, there is no equivalent column in the other language. In later sections, we will explain how, using **Transfer Learning** techniques, the incomplete Persian data was enriched and trained using a model that had been trained on English data, in order to achieve performance comparable to the English model.\n", | |
| "\n", | |
| "However, as a first step in this section, the Persian and English portions of the dataset are separated. That is, all columns with Persian content are stored in one DataFrame, and all columns with English content are stored in another. Each of these DataFrames is then saved as a separate CSV file." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 32, | |
| "metadata": { | |
| "id": "8eqV3zyYUkHb" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "persian_columns = ['supplier', 'brand_owner', 'distributor', 'manufacturer_country', 'brand_name_fa']\n", | |
| "english_columns = medicine_ds_merged.columns.difference(persian_columns, sort=False)\n", | |
| "\n", | |
| "persian_df = medicine_ds_merged[persian_columns]\n", | |
| "english_df = medicine_ds_merged[english_columns]\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 33, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 206 | |
| }, | |
| "id": "R3_U8LW8UkHb", | |
| "outputId": "f8898ad1-d519-4d88-a683-fed7382fed53" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"persian_df\",\n \"rows\": 65469,\n \"fields\": [\n {\n \"column\": \"supplier\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 306,\n \"samples\": [\n \"\\u062f\\u0627\\u0631\\u0648 \\u0633\\u0627\\u0632\\u06cc \\u0622\\u0631\\u0627\\u0645\\u06cc\\u0633 \\u0641\\u0627\\u0631\\u0645\\u062f\",\n \"\\u0627\\u06cc\\u0645\\u0627\\u0698\\u0646\",\n \"\\u06a9\\u06cc \\u0628\\u06cc \\u0633\\u06cc\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_owner\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 592,\n \"samples\": [\n \"Novonordisk\",\n \"Unimark Remedies Ltd.\",\n \"Meda Pharma Gmbh & Co. Kg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"distributor\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 43,\n \"samples\": [\n \"\\u0645\\u0634\\u06a9\\u0627\\u062a \\u0641\\u0627\\u0631\\u0645\\u062f \\u062a\\u0648\\u0632\\u06cc\\u0639\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u0628\\u0647\\u062f\\u0627\\u0634\\u062a \\u0634\\u0641\\u0627 \\u0622\\u0631\\u0627\\u062f\",\n \"\\u062f\\u0627\\u0631\\u0648 \\u06af\\u0633\\u062a\\u0631 \\u0628\\u0627\\u0631\\u06cc\\u062c \\u0627\\u0633\\u0627\\u0646\\u0633\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"manufacturer_country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 44,\n \"samples\": [\n \"\\u0627\\u0631\\u062f\\u0646\",\n \"\\u0644\\u0647\\u0633\\u062a\\u0627\\u0646\",\n \"\\u0647\\u0644\\u0646\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_fa\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6663,\n \"samples\": [\n \"\\u06af\\u0644\\u06cc\\u06a9\\u0648\\u0648\\u0631 \\u0642\\u0631\\u0635 \\u067e\\u06cc\\u0648\\u0633\\u062a\\u0647 \\u0631\\u0647\\u0634 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg\",\n \"\\u0627\\u0633\\u062a\\u06cc\\u0645\\u062f\\u06cc\\u062a \\u0642\\u0631\\u0635 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 10 mg\",\n \"\\u0627\\u06cc\\u0646\\u0641\\u0644\\u0648\\u0648\\u06cc\\u0631 \\u067e\\u0648\\u062f\\u0631 \\u0628\\u0631\\u0627\\u06cc \\u062a\\u0647\\u06cc\\u0647 \\u0633\\u0648\\u0633\\u067e\\u0627\\u0646\\u0633\\u06cc\\u0648\\u0646 \\u062e\\u0648\\u0631\\u0627\\u06a9\\u06cc 60 mg/5mL 75 mg\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "persian_df" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-696aae54-cb49-464c-8ad1-33b4add36e40\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>supplier</th>\n", | |
| " <th>brand_owner</th>\n", | |
| " <th>distributor</th>\n", | |
| " <th>manufacturer_country</th>\n", | |
| " <th>brand_name_fa</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>فاویپیراویر قرص خوراکی 200 mg</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>نولپازا قرص انتریک کوتد خوراکی 40 mg</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>اکتوورکو</td>\n", | |
| " <td>Krka, D. D., Novo Mesto</td>\n", | |
| " <td>الیت دارو</td>\n", | |
| " <td>ایران</td>\n", | |
| " <td>آسنترا قرص خوراکی 50 mg</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-696aae54-cb49-464c-8ad1-33b4add36e40')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-696aae54-cb49-464c-8ad1-33b4add36e40 button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-696aae54-cb49-464c-8ad1-33b4add36e40');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-d1cae8ba-f1e7-4c05-9425-926ef5c61828\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d1cae8ba-f1e7-4c05-9425-926ef5c61828')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-d1cae8ba-f1e7-4c05-9425-926ef5c61828 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " supplier brand_owner distributor manufacturer_country brand_name_fa\n", | |
| "0 اکتوورکو اکتوورکو الیت دارو ایران فاویپیراویر قرص خوراکی 200 mg\n", | |
| "1 اکتوورکو اکتوورکو الیت دارو ایران رمدسیویر محلول تزریقی پرنترال 5 mg/1mL 20 mg\n", | |
| "2 اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg\n", | |
| "3 اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران نولپازا قرص انتریک کوتد خوراکی 40 mg\n", | |
| "4 اکتوورکو Krka, D. D., Novo Mesto الیت دارو ایران آسنترا قرص خوراکی 50 mg" | |
| ] | |
| }, | |
| "execution_count": 33, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "persian_df.head()" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 34, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 521 | |
| }, | |
| "id": "VgaJLRERUkHb", | |
| "outputId": "51b62121-7941-4ccf-a26d-025de5187147" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "application/vnd.google.colaboratory.intrinsic+json": { | |
| "summary": "{\n \"name\": \"english_df\",\n \"rows\": 65469,\n \"fields\": [\n {\n \"column\": \"pharma_company\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 586,\n \"samples\": [\n \"\\u067e\\u0627\\u0631\\u0633 \\u062a\\u06a9 \\u0631\\u062e\",\n \"R-pharm Germany Gmbh\",\n \"\\u0641\\u0646 \\u0622\\u0648\\u0631\\u06cc \\u0632\\u06cc\\u0633\\u062a\\u06cc \\u0631\\u0632\\u0641\\u0627\\u0631\\u0645\\u062f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"brand_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 6630,\n \"samples\": [\n \"PIOGLITAZONE (DORSADAROU) 15MG TAB TABLET ORAL 15 mg\",\n \"CEBEX CAPSULE ORAL 200 mg\",\n \"VIOBION INJECTION PARENTERAL 100 mg/100 mg/1000 ug/3mL mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"generic_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2141,\n \"samples\": [\n \"FEXOFENADINE HYDROCHLORIDE SUSPENSION ORAL 30 mg/5mL 100 mL\",\n \"TRETINOIN LOTION TOPICAL 0.05 % 15 mL\",\n \"METHOTREXATE INJECTION, SOLUTION PARENTERAL 25 mg/1mL 0.8 mL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"substance_name_en\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 998,\n \"samples\": [\n \"PYRVINIUM\",\n \"POLYMYXIN B / NEOMYCIN SULFATE / HYDROCORTISONE\",\n \"LABETALOL HYDROCHLORIDE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"atc_code_raw\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 786,\n \"samples\": [\n \"S01BA02\",\n \"N05CF02\",\n \"S01EB01\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"anatomical_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 14,\n \"samples\": [\n \"MUSCULO-SKELETAL SYSTEM\",\n \"ANTIPARASITIC PRODUCTS, INSECTICIDES AND REPELLENTS\",\n \"ANTIINFECTIVES FOR SYSTEMIC USE\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"therapeutic_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 82,\n \"samples\": [\n \"ENDOCRINE THERAPY\",\n \"ANTIVIRALS FOR SYSTEMIC USE\",\n \"ANTIDIARRHEALS, INTESTINAL ANTIINFLAMMATORY/ANTIINFECTIVE AGENTS\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"pharmacological_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 182,\n \"samples\": [\n \"PROGESTOGENS\",\n \"POTASSIUM-SPARING AGENTS\",\n \"CORTICOSTEROIDS, COMBINATIONS WITH ANTISEPTICS\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"subpharmacological_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 365,\n \"samples\": [\n \"ACE inhibitors, plain\",\n \"Direct factor Xa inhibitors\",\n \"HMG CoA reductase inhibitors\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"chemical_desc\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 640,\n \"samples\": [\n \"azathioprine\",\n \"botulinum toxin\",\n \"diatrizoic acid\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", | |
| "type": "dataframe", | |
| "variable_name": "english_df" | |
| }, | |
| "text/html": [ | |
| "\n", | |
| " <div id=\"df-741d3992-1921-48dd-8214-ffd5a01abb37\" class=\"colab-df-container\">\n", | |
| " <div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code_raw</th>\n", | |
| " <th>anatomical_desc</th>\n", | |
| " <th>therapeutic_desc</th>\n", | |
| " <th>pharmacological_desc</th>\n", | |
| " <th>subpharmacological_desc</th>\n", | |
| " <th>chemical_desc</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>0</th>\n", | |
| " <td>Actero middleeast</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR TABLET ORAL 200 mg</td>\n", | |
| " <td>FAVIPIRAVIR</td>\n", | |
| " <td>J05AX</td>\n", | |
| " <td>ANTIINFECTIVES FOR SYSTEMIC USE</td>\n", | |
| " <td>ANTIVIRALS FOR SYSTEMIC USE</td>\n", | |
| " <td>DIRECT ACTING ANTIVIRALS</td>\n", | |
| " <td>Other antivirals</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>1</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg</td>\n", | |
| " <td>REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL</td>\n", | |
| " <td>REMDESIVIR</td>\n", | |
| " <td>J05</td>\n", | |
| " <td>ANTIINFECTIVES FOR SYSTEMIC USE</td>\n", | |
| " <td>ANTIVIRALS FOR SYSTEMIC USE</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " <td>NaN</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>2</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " <td>ALIMENTARY TRACT AND METABOLISM</td>\n", | |
| " <td>DRUGS FOR ACID RELATED DISORDERS</td>\n", | |
| " <td>DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD)</td>\n", | |
| " <td>Proton pump inhibitors</td>\n", | |
| " <td>pantoprazole</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>3</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg</td>\n", | |
| " <td>PANTOPRAZOLE</td>\n", | |
| " <td>A02BC02</td>\n", | |
| " <td>ALIMENTARY TRACT AND METABOLISM</td>\n", | |
| " <td>DRUGS FOR ACID RELATED DISORDERS</td>\n", | |
| " <td>DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD)</td>\n", | |
| " <td>Proton pump inhibitors</td>\n", | |
| " <td>pantoprazole</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>4</th>\n", | |
| " <td>Actoverco</td>\n", | |
| " <td>ASENTRA TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg</td>\n", | |
| " <td>SERTRALINE (AS HYDROCHLORIDE)</td>\n", | |
| " <td>N06AB06</td>\n", | |
| " <td>NERVOUS SYSTEM</td>\n", | |
| " <td>PSYCHOANALEPTICS</td>\n", | |
| " <td>ANTIDEPRESSANTS</td>\n", | |
| " <td>Selective serotonin reuptake inhibitors</td>\n", | |
| " <td>sertraline</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>\n", | |
| " <div class=\"colab-df-buttons\">\n", | |
| "\n", | |
| " <div class=\"colab-df-container\">\n", | |
| " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-741d3992-1921-48dd-8214-ffd5a01abb37')\"\n", | |
| " title=\"Convert this dataframe to an interactive table.\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n", | |
| " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n", | |
| " </svg>\n", | |
| " </button>\n", | |
| "\n", | |
| " <style>\n", | |
| " .colab-df-container {\n", | |
| " display:flex;\n", | |
| " gap: 12px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert {\n", | |
| " background-color: #E8F0FE;\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: #1967D2;\n", | |
| " height: 32px;\n", | |
| " padding: 0 0 0 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-convert:hover {\n", | |
| " background-color: #E2EBFA;\n", | |
| " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: #174EA6;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-buttons div {\n", | |
| " margin-bottom: 4px;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert {\n", | |
| " background-color: #3B4455;\n", | |
| " fill: #D2E3FC;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-convert:hover {\n", | |
| " background-color: #434B5C;\n", | |
| " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n", | |
| " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n", | |
| " fill: #FFFFFF;\n", | |
| " }\n", | |
| " </style>\n", | |
| "\n", | |
| " <script>\n", | |
| " const buttonEl =\n", | |
| " document.querySelector('#df-741d3992-1921-48dd-8214-ffd5a01abb37 button.colab-df-convert');\n", | |
| " buttonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| "\n", | |
| " async function convertToInteractive(key) {\n", | |
| " const element = document.querySelector('#df-741d3992-1921-48dd-8214-ffd5a01abb37');\n", | |
| " const dataTable =\n", | |
| " await google.colab.kernel.invokeFunction('convertToInteractive',\n", | |
| " [key], {});\n", | |
| " if (!dataTable) return;\n", | |
| "\n", | |
| " const docLinkHtml = 'Like what you see? Visit the ' +\n", | |
| " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n", | |
| " + ' to learn more about interactive tables.';\n", | |
| " element.innerHTML = '';\n", | |
| " dataTable['output_type'] = 'display_data';\n", | |
| " await google.colab.output.renderOutput(dataTable, element);\n", | |
| " const docLink = document.createElement('div');\n", | |
| " docLink.innerHTML = docLinkHtml;\n", | |
| " element.appendChild(docLink);\n", | |
| " }\n", | |
| " </script>\n", | |
| " </div>\n", | |
| "\n", | |
| "\n", | |
| "<div id=\"df-dfd3793f-3526-4719-a04e-07603166a262\">\n", | |
| " <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dfd3793f-3526-4719-a04e-07603166a262')\"\n", | |
| " title=\"Suggest charts\"\n", | |
| " style=\"display:none;\">\n", | |
| "\n", | |
| "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n", | |
| " width=\"24px\">\n", | |
| " <g>\n", | |
| " <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n", | |
| " </g>\n", | |
| "</svg>\n", | |
| " </button>\n", | |
| "\n", | |
| "<style>\n", | |
| " .colab-df-quickchart {\n", | |
| " --bg-color: #E8F0FE;\n", | |
| " --fill-color: #1967D2;\n", | |
| " --hover-bg-color: #E2EBFA;\n", | |
| " --hover-fill-color: #174EA6;\n", | |
| " --disabled-fill-color: #AAA;\n", | |
| " --disabled-bg-color: #DDD;\n", | |
| " }\n", | |
| "\n", | |
| " [theme=dark] .colab-df-quickchart {\n", | |
| " --bg-color: #3B4455;\n", | |
| " --fill-color: #D2E3FC;\n", | |
| " --hover-bg-color: #434B5C;\n", | |
| " --hover-fill-color: #FFFFFF;\n", | |
| " --disabled-bg-color: #3B4455;\n", | |
| " --disabled-fill-color: #666;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart {\n", | |
| " background-color: var(--bg-color);\n", | |
| " border: none;\n", | |
| " border-radius: 50%;\n", | |
| " cursor: pointer;\n", | |
| " display: none;\n", | |
| " fill: var(--fill-color);\n", | |
| " height: 32px;\n", | |
| " padding: 0;\n", | |
| " width: 32px;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart:hover {\n", | |
| " background-color: var(--hover-bg-color);\n", | |
| " box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n", | |
| " fill: var(--button-hover-fill-color);\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-quickchart-complete:disabled,\n", | |
| " .colab-df-quickchart-complete:disabled:hover {\n", | |
| " background-color: var(--disabled-bg-color);\n", | |
| " fill: var(--disabled-fill-color);\n", | |
| " box-shadow: none;\n", | |
| " }\n", | |
| "\n", | |
| " .colab-df-spinner {\n", | |
| " border: 2px solid var(--fill-color);\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " animation:\n", | |
| " spin 1s steps(1) infinite;\n", | |
| " }\n", | |
| "\n", | |
| " @keyframes spin {\n", | |
| " 0% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " }\n", | |
| " 20% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 30% {\n", | |
| " border-color: transparent;\n", | |
| " border-left-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 40% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-top-color: var(--fill-color);\n", | |
| " }\n", | |
| " 60% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " }\n", | |
| " 80% {\n", | |
| " border-color: transparent;\n", | |
| " border-right-color: var(--fill-color);\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " 90% {\n", | |
| " border-color: transparent;\n", | |
| " border-bottom-color: var(--fill-color);\n", | |
| " }\n", | |
| " }\n", | |
| "</style>\n", | |
| "\n", | |
| " <script>\n", | |
| " async function quickchart(key) {\n", | |
| " const quickchartButtonEl =\n", | |
| " document.querySelector('#' + key + ' button');\n", | |
| " quickchartButtonEl.disabled = true; // To prevent multiple clicks.\n", | |
| " quickchartButtonEl.classList.add('colab-df-spinner');\n", | |
| " try {\n", | |
| " const charts = await google.colab.kernel.invokeFunction(\n", | |
| " 'suggestCharts', [key], {});\n", | |
| " } catch (error) {\n", | |
| " console.error('Error during call to suggestCharts:', error);\n", | |
| " }\n", | |
| " quickchartButtonEl.classList.remove('colab-df-spinner');\n", | |
| " quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n", | |
| " }\n", | |
| " (() => {\n", | |
| " let quickchartButtonEl =\n", | |
| " document.querySelector('#df-dfd3793f-3526-4719-a04e-07603166a262 button');\n", | |
| " quickchartButtonEl.style.display =\n", | |
| " google.colab.kernel.accessAllowed ? 'block' : 'none';\n", | |
| " })();\n", | |
| " </script>\n", | |
| "</div>\n", | |
| "\n", | |
| " </div>\n", | |
| " </div>\n" | |
| ], | |
| "text/plain": [ | |
| " pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc\n", | |
| "0 Actero middleeast FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR TABLET ORAL 200 mg FAVIPIRAVIR J05AX ANTIINFECTIVES FOR SYSTEMIC USE ANTIVIRALS FOR SYSTEMIC USE DIRECT ACTING ANTIVIRALS Other antivirals NaN\n", | |
| "1 Actoverco REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mg REMDESIVIR INJECTION, SOLUTION PARENTERAL 5 mg/1mL 20 mL REMDESIVIR J05 ANTIINFECTIVES FOR SYSTEMIC USE ANTIVIRALS FOR SYSTEMIC USE NaN NaN NaN\n", | |
| "2 Actoverco NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02 ALIMENTARY TRACT AND METABOLISM DRUGS FOR ACID RELATED DISORDERS DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD) Proton pump inhibitors pantoprazole\n", | |
| "3 Actoverco NOLPAZA TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE TABLET, DELAYED RELEASE ORAL 40 mg PANTOPRAZOLE A02BC02 ALIMENTARY TRACT AND METABOLISM DRUGS FOR ACID RELATED DISORDERS DRUGS FOR PEPTIC ULCER AND GASTRO-OESOPHAGEAL REFLUX DISEASE (GORD) Proton pump inhibitors pantoprazole\n", | |
| "4 Actoverco ASENTRA TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) TABLET ORAL 50 mg SERTRALINE (AS HYDROCHLORIDE) N06AB06 NERVOUS SYSTEM PSYCHOANALEPTICS ANTIDEPRESSANTS Selective serotonin reuptake inhibitors sertraline" | |
| ] | |
| }, | |
| "execution_count": 34, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "english_df.head()\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 36, | |
| "metadata": { | |
| "id": "T8mQwfCvUkHb" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "persian_df.to_csv('./persian_df.csv', index=False)\n", | |
| "english_df.to_csv('./english_df.csv', index=False)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 17, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 211 | |
| }, | |
| "id": "KDXUFOyTV4ML", | |
| "outputId": "2d58f9e3-d2a1-43c2-b101-a437e43b3989" | |
| }, | |
| "outputs": [ | |
| { | |
| "ename": "NameError", | |
| "evalue": "name 'persian_df' is not defined", | |
| "output_type": "error", | |
| "traceback": [ | |
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | |
| "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", | |
| "\u001b[0;32m<ipython-input-17-11b00ee12080>\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_selection\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mdf_train_fa\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdf_test_fa\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpersian_df\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_state\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m42\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m persian_dataset = DatasetDict({\n", | |
| "\u001b[0;31mNameError\u001b[0m: name 'persian_df' is not defined" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from datasets import Dataset, DatasetDict\n", | |
| "from sklearn.model_selection import train_test_split\n", | |
| "\n", | |
| "df_train_fa, df_test_fa = train_test_split(persian_df, test_size=0.2, random_state=42)\n", | |
| "\n", | |
| "persian_dataset = DatasetDict({\n", | |
| " \"train\": Dataset.from_pandas(df_train_fa.reset_index(drop=True)),\n", | |
| " \"test\": Dataset.from_pandas(df_test_fa.reset_index(drop=True))\n", | |
| "})\n", | |
| "\n", | |
| "persian_dataset.push_to_hub(\"tahamajs/medicine_ds_persian\", token=\"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\")\n", | |
| "\n", | |
| "\n", | |
| "df_train_en, df_test_en = train_test_split(english_df, test_size=0.2, random_state=42)\n", | |
| "\n", | |
| "english_dataset = DatasetDict({\n", | |
| " \"train\": Dataset.from_pandas(df_train_en.reset_index(drop=True)),\n", | |
| " \"test\": Dataset.from_pandas(df_test_en.reset_index(drop=True))\n", | |
| "})\n", | |
| "\n", | |
| "english_dataset.push_to_hub(\"tahamajs/medicine_ds_english\", token=\"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\")\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "WDzOaTwrUkHb" | |
| }, | |
| "source": [ | |
| "<h2 id=\"model-training\">5. Training Language Models</h2>\n", | |
| "\n", | |
| "Each language-specific dataset was trained using the **Word2Vec** model from the `gensim` library, and the resulting embeddings were saved to a file. To ensure the model uses the **Skip-gram** architecture (which is more effective for capturing semantic relationships in smaller datasets), the `sg` argument was set accordingly during training." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 23, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "6i3YX1NYUkHc", | |
| "outputId": "fdb40068-cb77-488c-8b37-2a28b0908b0b" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stderr", | |
| "output_type": "stream", | |
| "text": [ | |
| "[nltk_data] Downloading package punkt to /root/nltk_data...\n", | |
| "[nltk_data] Unzipping tokenizers/punkt.zip.\n", | |
| "[nltk_data] Downloading package stopwords to /root/nltk_data...\n", | |
| "[nltk_data] Package stopwords is already up-to-date!\n", | |
| "[nltk_data] Downloading package wordnet to /root/nltk_data...\n", | |
| "[nltk_data] Package wordnet is already up-to-date!\n", | |
| "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n", | |
| "[nltk_data] Package omw-1.4 is already up-to-date!\n" | |
| ] | |
| }, | |
| { | |
| "data": { | |
| "text/plain": [ | |
| "True" | |
| ] | |
| }, | |
| "execution_count": 23, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 15, | |
| "metadata": { | |
| "id": "OEhg6n_ZUkHc" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from nltk.corpus import stopwords\n", | |
| "from nltk.tokenize import word_tokenize\n", | |
| "from nltk.stem import PorterStemmer\n", | |
| "from spellchecker import SpellChecker\n", | |
| "\n", | |
| "stop_words = set(stopwords.words('english'))\n", | |
| "stemmer = PorterStemmer()\n", | |
| "spell = SpellChecker()\n", | |
| "\n", | |
| "def preprocess_row(row):\n", | |
| " row = row.str.lower()\n", | |
| " row = row.fillna(' ').astype(str)\n", | |
| " tokens = word_tokenize(' '.join(row))\n", | |
| " filtered_tokens = [w for w in tokens if not w in stop_words]\n", | |
| " result_tokens = filtered_tokens\n", | |
| " return result_tokens\n", | |
| "\n", | |
| "def preprocess(data):\n", | |
| " sentences = np.array(data.apply(preprocess_row, axis=1))\n", | |
| " return sentences\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "Gz3gsgIiSX_w" | |
| }, | |
| "source": [ | |
| "\n", | |
| "The preprocessing pipeline is designed to normalize and tokenize textual data contained in a DataFrame. It begins by converting all text to lowercase, handling missing values by replacing them with whitespace, and ensuring all entries are treated as strings. The text from each row is then tokenized using NLTK's `word_tokenize` function, which splits sentences into individual words or tokens. Subsequently, common English stopwords (e.g., \"and\", \"the\", \"is\") are filtered out to retain only semantically meaningful terms.\n", | |
| "\n", | |
| "Although stemming and spell-checking components are initialized using NLTK's `PorterStemmer` and the `SpellChecker` library, they are not applied in the current implementation. The output of the `preprocess_row` function is a list of cleaned, lowercased, and stopword-removed tokens for each row, which is aggregated over all rows in the DataFrame using `data.apply()`. The final result is a NumPy array of tokenized sentences, which can be used in downstream tasks such as embedding generation or model training." | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "hLJbkcU5UkHc" | |
| }, | |
| "source": [ | |
| "\n", | |
| "### 5.1. Training the Model on the English Dataset\n", | |
| "\n", | |
| "In this step, the English portion of the dataset was used to train a Word2Vec model. The training aimed to capture semantic relationships between drug-related terms and classifications in English, forming a dense vector space that can later be used for tasks such as similarity matching, transfer learning, or aligning with Persian embeddings." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 41, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 842 | |
| }, | |
| "id": "Zm_g__b6WfrP", | |
| "outputId": "92c68aac-6182-4c69-c28f-b2f2ee9987f4" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Collecting numpy\n", | |
| " Downloading numpy-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)\n", | |
| "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/62.0 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.0/62.0 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", | |
| "\u001b[?25hCollecting gensim\n", | |
| " Using cached gensim-4.3.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.1 kB)\n", | |
| "Collecting numpy\n", | |
| " Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)\n", | |
| "Collecting scipy<1.14.0,>=1.7.0 (from gensim)\n", | |
| " Using cached scipy-1.13.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)\n", | |
| "Collecting smart-open>=1.8.1 (from gensim)\n", | |
| " Downloading smart_open-7.1.0-py3-none-any.whl.metadata (24 kB)\n", | |
| "Collecting wrapt (from smart-open>=1.8.1->gensim)\n", | |
| " Downloading wrapt-1.17.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB)\n", | |
| "Using cached gensim-4.3.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.7 MB)\n", | |
| "Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)\n", | |
| "Using cached scipy-1.13.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.6 MB)\n", | |
| "Downloading smart_open-7.1.0-py3-none-any.whl (61 kB)\n", | |
| "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.7/61.7 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", | |
| "\u001b[?25hDownloading wrapt-1.17.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (83 kB)\n", | |
| "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.2/83.2 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", | |
| "\u001b[?25hInstalling collected packages: wrapt, numpy, smart-open, scipy, gensim\n", | |
| " Attempting uninstall: wrapt\n", | |
| " Found existing installation: wrapt 1.17.2\n", | |
| " Uninstalling wrapt-1.17.2:\n", | |
| " Successfully uninstalled wrapt-1.17.2\n", | |
| " Attempting uninstall: numpy\n", | |
| " Found existing installation: numpy 1.26.4\n", | |
| " Uninstalling numpy-1.26.4:\n", | |
| " Successfully uninstalled numpy-1.26.4\n", | |
| " Attempting uninstall: smart-open\n", | |
| " Found existing installation: smart-open 7.1.0\n", | |
| " Uninstalling smart-open-7.1.0:\n", | |
| " Successfully uninstalled smart-open-7.1.0\n", | |
| " Attempting uninstall: scipy\n", | |
| " Found existing installation: scipy 1.13.1\n", | |
| " Uninstalling scipy-1.13.1:\n", | |
| " Successfully uninstalled scipy-1.13.1\n", | |
| " Attempting uninstall: gensim\n", | |
| " Found existing installation: gensim 4.3.3\n", | |
| " Uninstalling gensim-4.3.3:\n", | |
| " Successfully uninstalled gensim-4.3.3\n", | |
| "Successfully installed gensim-4.3.3 numpy-1.26.4 scipy-1.13.1 smart-open-7.1.0 wrapt-1.17.2\n" | |
| ] | |
| }, | |
| { | |
| "data": { | |
| "application/vnd.colab-display-data+json": { | |
| "id": "d7084deef47343d6bfb57c16381239aa", | |
| "pip_warning": { | |
| "packages": [ | |
| "smart_open", | |
| "wrapt" | |
| ] | |
| } | |
| } | |
| }, | |
| "metadata": {}, | |
| "output_type": "display_data" | |
| } | |
| ], | |
| "source": [ | |
| "!pip install --upgrade --force-reinstall numpy gensim\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 26, | |
| "metadata": { | |
| "id": "T0rBk0WHUkHc" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "import pandas as pd\n", | |
| "from gensim.models import Word2Vec\n", | |
| "\n", | |
| "data_en = pd.read_csv('./english_df.csv')\n", | |
| "sentences_en = preprocess(data_en)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 28, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 339 | |
| }, | |
| "id": "hVpxa3UXUkHc", | |
| "outputId": "dee7ce28-27ff-429b-f222-898d98578d86" | |
| }, | |
| "outputs": [ | |
| { | |
| "ename": "FileNotFoundError", | |
| "evalue": "[Errno 2] No such file or directory: './model/en_embeddings.txt'", | |
| "output_type": "error", | |
| "traceback": [ | |
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | |
| "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", | |
| "\u001b[0;32m<ipython-input-28-6d20748a6300>\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mmodel_en\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWord2Vec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences_en\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmin_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mmodel_en\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./en_skipgram.model'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mmodel_en\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./model/en_embeddings.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/gensim/models/keyedvectors.py\u001b[0m in \u001b[0;36msave_word2vec_format\u001b[0;34m(self, fname, fvocab, binary, total_vec, write_header, prefix, append, sort_attr)\u001b[0m\n\u001b[1;32m 1660\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1661\u001b[0m \u001b[0;31m# Store the actual vectors to the output file, in the order defined by sort_attr.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1662\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1663\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mwrite_header\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1664\u001b[0m \u001b[0mfout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"{total_vec} {self.vector_size}\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf8'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/smart_open/smart_open_lib.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(uri, mode, buffering, encoding, errors, newline, closefd, opener, compression, transport_params)\u001b[0m\n\u001b[1;32m 175\u001b[0m \u001b[0mtransport_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 176\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 177\u001b[0;31m fobj = _shortcut_open(\n\u001b[0m\u001b[1;32m 178\u001b[0m \u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 179\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/smart_open/smart_open_lib.py\u001b[0m in \u001b[0;36m_shortcut_open\u001b[0;34m(uri, mode, compression, buffering, encoding, errors, newline)\u001b[0m\n\u001b[1;32m 373\u001b[0m \u001b[0mopen_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'errors'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 374\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 375\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_builtin_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlocal_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mopen_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 376\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 377\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './model/en_embeddings.txt'" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "import os\n", | |
| "\n", | |
| "data_en['preprocessed_index'] = np.arange(sentences_en.shape[0])\n", | |
| "data_en.to_csv('./english_df.csv', index=False)\n", | |
| "\n", | |
| "if os.path.exists(\"./model/en_skipgram.model\") and os.path.exists(\"./model/en_embeddings.txt\"):\n", | |
| " print(\"Models for training English skipgram already exist. Skipping the training.\")\n", | |
| "else:\n", | |
| " model_en = Word2Vec(sentences_en, min_count=1, sg=1)\n", | |
| " model_en.save('./en_skipgram.model')\n", | |
| " model_en.wv.save_word2vec_format('./en_embeddings.txt', binary=False)\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "vfHNfxrfYehL" | |
| }, | |
| "source": [ | |
| "\n", | |
| "The code snippet performs preprocessing and conditional training of a Word2Vec skip-gram model for English-language data. It first assigns a unique index to each preprocessed sentence using `np.arange`, storing this as a new column (`preprocessed_index`) in the `data_en` DataFrame. This updated DataFrame is then saved to disk as `english_df.csv`. The indexed structure is useful for traceability or retrieval-based tasks, where it is necessary to map vector representations back to the original data source.\n", | |
| "\n", | |
| "Subsequently, the script checks whether the model files (`en_skipgram.model` and `en_embeddings.txt`) already exist. If they do, it prints a message and skips the training step to avoid redundant computation. If the files are not present, it proceeds to train a skip-gram model using Gensim’s `Word2Vec` implementation with `sg=1`, indicating the skip-gram architecture. The trained model is saved in two formats: the full Gensim model (`.model`) for continued training or reuse, and the word vectors in plain text (`.txt`) format for downstream tasks such as embedding-based similarity, classification, or retrieval." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 29, | |
| "metadata": { | |
| "id": "dXARPeZvYVbh" | |
| }, | |
| "outputs": [], | |
| "source": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "_4dNs4hEUkHc" | |
| }, | |
| "source": [ | |
| "### 5.2. Training the Model on the Persian Dataset\n", | |
| "\n", | |
| "In this step, the Persian portion of the dataset was used to train a Word2Vec model using the **Skip-gram** architecture (`sg=1`). This training process aimed to generate meaningful vector representations (embeddings) for Persian medical terms and drug-related phrases. \n", | |
| "Despite the Persian dataset being smaller and less complete than the English one, the model was still able to capture useful semantic relationships within the available data. These embeddings are later used for alignment with the English model through transfer learning techniques." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 44, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "cy4DhyYVUkHd", | |
| "outputId": "8edb56fe-36ed-4650-ae25-e8f9b2d2cabd" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Models for training Farsi skipgram already exist. Skipping the training.\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "import os\n", | |
| "\n", | |
| "data_fa = pd.read_csv('./persian_df.csv')\n", | |
| "sentences_fa = preprocess(data_fa)\n", | |
| "\n", | |
| "data_fa['preprocessed_index'] = np.arange(sentences_fa.shape[0])\n", | |
| "data_fa.to_csv('./persian_df.csv', index=False)\n", | |
| "\n", | |
| "if os.path.exists(\"/fa_skipgram.model\") and os.path.exists(\"/fa_embeddings.txt\"):\n", | |
| " print(\"Models for training Farsi skipgram already exist. Skipping the training.\")\n", | |
| "else:\n", | |
| " model_fa = Word2Vec(sentences_fa, min_count=1, sg=1)\n", | |
| " model_fa.save('/fa_skipgram.model')\n", | |
| " model_fa.wv.save_word2vec_format('/fa_embeddings.txt', binary=False)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 45, | |
| "metadata": { | |
| "id": "3CnWji_QbD-S" | |
| }, | |
| "outputs": [], | |
| "source": [] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "r3rY56QyUkHd" | |
| }, | |
| "source": [ | |
| "<h2 id=\"alignment\">6. Aligning the Semantic Space Between the Two Languages' Embeddings</h2>\n", | |
| "\n", | |
| "To align the semantic space between the two languages, we use the **vecmap** tool. This tool is not available as a PyPI package, so it must be cloned directly from its GitHub repository. \n", | |
| "After cloning, we perform an **unsupervised mapping** between the Persian and English embeddings. The mapping is linear and aims to bring semantically similar terms from both languages into a shared vector space. The result of this mapping is saved in a text file, which will be used in later stages of the project." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 34, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/" | |
| }, | |
| "id": "c4St0Va3UkHd", | |
| "outputId": "7c52b59e-cdfc-4f68-ee7a-317f111b82d5" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Cloning into 'vecmap'...\n", | |
| "remote: Enumerating objects: 104, done.\u001b[K\n", | |
| "remote: Total 104 (delta 0), reused 0 (delta 0), pack-reused 104 (from 1)\u001b[K\n", | |
| "Receiving objects: 100% (104/104), 72.59 KiB | 675.00 KiB/s, done.\n", | |
| "Resolving deltas: 100% (57/57), done.\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "!git clone https://github.com/artetxem/vecmap.git" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 54, | |
| "metadata": { | |
| "id": "x6ZNyqPnUkHd", | |
| "vscode": { | |
| "languageId": "shellscript" | |
| } | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "import os\n", | |
| "\n", | |
| "# Check if the mapped embedding files already exist\n", | |
| "if os.path.exists(\"./en_mapped_embeddings.txt\") and os.path.exists(\"./fa_mapped_embeddings.txt\"):\n", | |
| " print(\"Models for mapping embeddings already exist. Skipping the training.\")\n", | |
| "else:\n", | |
| " # Determine the appropriate command based on the availability of CUDA\n", | |
| " if cupy is not None:\n", | |
| " command = \"python ./vecmap/map_embeddings.py --unsupervised --cuda ./en_embeddings.txt ./fa_embeddings.txt ./en_mapped_embeddings.txt ./fa_mapped_embeddings.txt\"\n", | |
| " else:\n", | |
| " command = \"python ./vecmap/map_embeddings.py --unsupervised ./en_embeddings.txt ./fa_embeddings.txt ./en_mapped_embeddings.txt ./fa_mapped_embeddings.txt\"\n", | |
| "\n", | |
| " os.system(command)\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "6uVbeSNbUkHd" | |
| }, | |
| "source": [ | |
| "<h2 id=\"search\">7. Searching Drug Information with Bilingual Input</h2>\n", | |
| "\n", | |
| "First, we prepare the **mapped embeddings** (aligned to a shared semantic space) for both languages. This enables us to process search queries in either Persian or English and retrieve relevant drug information from the combined dataset, regardless of the input language." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 56, | |
| "metadata": { | |
| "colab": { | |
| "base_uri": "https://localhost:8080/", | |
| "height": 356 | |
| }, | |
| "id": "-Hu8mtjzUkHd", | |
| "outputId": "9ed30619-49cb-474a-fbba-610dd875e473" | |
| }, | |
| "outputs": [ | |
| { | |
| "ename": "FileNotFoundError", | |
| "evalue": "[Errno 2] No such file or directory: './en_mapped_embeddings.txt'", | |
| "output_type": "error", | |
| "traceback": [ | |
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | |
| "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", | |
| "\u001b[0;32m<ipython-input-56-61b87ba06038>\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mgensim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mKeyedVectors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0men_embeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mKeyedVectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./en_mapped_embeddings.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0mfa_embeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mKeyedVectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./fa_mapped_embeddings.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/gensim/models/keyedvectors.py\u001b[0m in \u001b[0;36mload_word2vec_format\u001b[0;34m(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype, no_header)\u001b[0m\n\u001b[1;32m 1717\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1718\u001b[0m \"\"\"\n\u001b[0;32m-> 1719\u001b[0;31m return _load_word2vec_format(\n\u001b[0m\u001b[1;32m 1720\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfvocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfvocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbinary\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0municode_errors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0municode_errors\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1721\u001b[0m \u001b[0mlimit\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlimit\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatatype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdatatype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mno_header\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mno_header\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/gensim/models/keyedvectors.py\u001b[0m in \u001b[0;36m_load_word2vec_format\u001b[0;34m(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype, no_header, binary_chunk_size)\u001b[0m\n\u001b[1;32m 2046\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2047\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"loading projection weights from %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2048\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfin\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2049\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mno_header\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2050\u001b[0m \u001b[0;31m# deduce both vocab_size & vector_size from 1st pass over file\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/smart_open/smart_open_lib.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(uri, mode, buffering, encoding, errors, newline, closefd, opener, compression, transport_params)\u001b[0m\n\u001b[1;32m 175\u001b[0m \u001b[0mtransport_params\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 176\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 177\u001b[0;31m fobj = _shortcut_open(\n\u001b[0m\u001b[1;32m 178\u001b[0m \u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 179\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/smart_open/smart_open_lib.py\u001b[0m in \u001b[0;36m_shortcut_open\u001b[0;34m(uri, mode, compression, buffering, encoding, errors, newline)\u001b[0m\n\u001b[1;32m 373\u001b[0m \u001b[0mopen_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'errors'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 374\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 375\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_builtin_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlocal_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mopen_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 376\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 377\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", | |
| "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: './en_mapped_embeddings.txt'" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "from gensim.models import KeyedVectors\n", | |
| "\n", | |
| "en_embeddings = KeyedVectors.load_word2vec_format('./en_mapped_embeddings.txt', binary=False)\n", | |
| "fa_embeddings = KeyedVectors.load_word2vec_format('./fa_mapped_embeddings.txt', binary=False)\n", | |
| "\n", | |
| "persian_df = pd.read_csv('./persian_df.csv')\n", | |
| "english_df = pd.read_csv('./english_df.csv')\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": 55, | |
| "metadata": { | |
| "id": "KdsTR73GUkHd" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from nltk.stem import WordNetLemmatizer\n", | |
| "from nltk.corpus import stopwords\n", | |
| "from spellchecker import SpellChecker\n", | |
| "from nltk.tokenize import word_tokenize\n", | |
| "import re\n", | |
| "\n", | |
| "spell = SpellChecker()\n", | |
| "\n", | |
| "lemmatizer = WordNetLemmatizer()\n", | |
| "\n", | |
| "stop_words = set(stopwords.words('english'))\n", | |
| "\n", | |
| "def preprocess_query(query, language, embeddings):\n", | |
| " query = query.lower()\n", | |
| " tokens = word_tokenize(query)\n", | |
| " tokens = [spell.correction(token) if spell.correction(token) is not None else token for token in tokens]\n", | |
| " tokens = [token for token in tokens if token not in stop_words]\n", | |
| " tokens = [lemmatizer.lemmatize(token) for token in tokens]\n", | |
| " token_vectors = []\n", | |
| "\n", | |
| " for token in tokens:\n", | |
| " try:\n", | |
| " token_vector = embeddings.get_vector(token)\n", | |
| " token_vectors.append(token_vector)\n", | |
| " except KeyError:\n", | |
| " pass\n", | |
| "\n", | |
| " if len(token_vectors) > 0:\n", | |
| " query_vector = np.mean(token_vectors, axis=0)\n", | |
| " else:\n", | |
| " query_vector = np.zeros(embeddings.vector_size)\n", | |
| "\n", | |
| " normalized_query = ' '.join(tokens)\n", | |
| " return normalized_query, query_vector\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "v7XWLmMKSjcw" | |
| }, | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "The `preprocess_query` function is designed to clean and vectorize a user query for downstream information retrieval or similarity tasks. The query undergoes several natural language preprocessing steps, including lowercasing, tokenization, spell correction using the `SpellChecker` library, stopword removal, and lemmatization via NLTK's `WordNetLemmatizer`. These steps ensure that the input text is normalized, linguistically simplified, and free from irrelevant or redundant components such as common stopwords.\n", | |
| "\n", | |
| "Following text normalization, the function attempts to retrieve word vectors for each remaining token from a pre-trained word embedding model (e.g., Word2Vec or GloVe). These token embeddings are aggregated using their arithmetic mean to form a single vector representation of the query. If no valid embeddings are found, a zero vector is returned instead. The function outputs both the normalized query string and its corresponding dense vector, enabling its use in semantic similarity computation or embedding-based search pipelines." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "aqY-kBSXUkHd" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def preprocess_row(row, language, embeddings, preprocessed_rows, vectorized_rows):\n", | |
| " row = row.str.lower()\n", | |
| " row = row.fillna(' ').astype(str)\n", | |
| " tokens = word_tokenize(' '.join(row))\n", | |
| "\n", | |
| " tokens = [token for token in tokens if token not in stop_words]\n", | |
| " token_vectors = []\n", | |
| "\n", | |
| " for token in tokens:\n", | |
| " try:\n", | |
| " token_vector = embeddings.get_vector(token)\n", | |
| " token_vectors.append(token_vector)\n", | |
| " except KeyError:\n", | |
| " pass\n", | |
| "\n", | |
| " if len(token_vectors) > 0:\n", | |
| " row_vector = np.mean(token_vectors, axis=0)\n", | |
| " else:\n", | |
| " row_vector = np.zeros(embeddings.vector_size)\n", | |
| "\n", | |
| " preprocessed_rows.append(row)\n", | |
| " vectorized_rows.append(row_vector)\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "ubzk9Y5BSy7p" | |
| }, | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "The `preprocess_row` function is responsible for transforming a single DataFrame row of textual data into a vector representation by leveraging pre-trained word embeddings. Initially, the row content is lowercased, null values are replaced with empty strings, and the entire row is converted to a string format suitable for tokenization. Using NLTK’s `word_tokenize`, the function splits the text into individual tokens. Stopwords—common words that typically do not contribute semantic value—are filtered out using a predefined stopword set.\n", | |
| "\n", | |
| "For each remaining token, the function attempts to retrieve a corresponding word vector from the provided embedding model (e.g., Word2Vec, FastText). These token vectors are then averaged to obtain a single fixed-size vector representing the semantic content of the row. If no valid embeddings are found for any token, a zero vector is used as a fallback. The processed row (in text form) and its corresponding vector representation are appended to two separate lists—`preprocessed_rows` and `vectorized_rows`—enabling batch accumulation of results for later use in tasks such as similarity search, classification, or clustering." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "eiHwA7u9UkHd" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "from fuzzywuzzy import fuzz\n", | |
| "from sklearn.metrics.pairwise import cosine_similarity\n", | |
| "\n", | |
| "def calc_rw(dataset, language, embeddings):\n", | |
| " preprocessed_rows = []\n", | |
| " vectorized_rows = []\n", | |
| " dataset.apply(lambda r: preprocess_row(r, language, embeddings, preprocessed_rows, vectorized_rows), axis=1)\n", | |
| " vectorized_rows = np.array(vectorized_rows)\n", | |
| " return preprocessed_rows, vectorized_rows\n", | |
| "\n", | |
| "preprocessed_rows, vectorized_rows = calc_rw(english_df, 'en', en_embeddings)\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "3uQRznCHSqUm" | |
| }, | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "The `calc_rw` function is designed to process an entire dataset by applying text normalization and embedding-based vectorization to each row. It utilizes a row-wise application of the `preprocess_row` function, which transforms the textual content into lowercased, tokenized form, removes stopwords, and computes an average vector using pre-trained word embeddings. Two lists—`preprocessed_rows` and `vectorized_rows`—are populated in the process, storing the cleaned text and its corresponding semantic representation, respectively. This ensures the dataset is prepared for downstream applications such as information retrieval, clustering, or classification.\n", | |
| "\n", | |
| "The result of the function is a NumPy array of dense vector representations that capture the semantic content of each row, along with a list of their corresponding preprocessed text forms. This dual output structure supports both numerical operations (e.g., similarity calculations using cosine similarity) and textual analysis (e.g., fuzzy matching using string comparison techniques). By enabling both semantic and lexical comparisons, the `calc_rw` function serves as a foundational step in hybrid text-matching pipelines." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "OfEu_dI9UkHe", | |
| "outputId": "e9241041-aecf-4ca3-f9f1-5174f131ddbe" | |
| }, | |
| "outputs": [ | |
| { | |
| "data": { | |
| "text/html": [ | |
| "<div>\n", | |
| "<style scoped>\n", | |
| " .dataframe tbody tr th:only-of-type {\n", | |
| " vertical-align: middle;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe tbody tr th {\n", | |
| " vertical-align: top;\n", | |
| " }\n", | |
| "\n", | |
| " .dataframe thead th {\n", | |
| " text-align: right;\n", | |
| " }\n", | |
| "</style>\n", | |
| "<table border=\"1\" class=\"dataframe\">\n", | |
| " <thead>\n", | |
| " <tr style=\"text-align: right;\">\n", | |
| " <th></th>\n", | |
| " <th>Unnamed: 0.1</th>\n", | |
| " <th>Unnamed: 0</th>\n", | |
| " <th>pharma_company</th>\n", | |
| " <th>brand_name_en</th>\n", | |
| " <th>generic_name_en</th>\n", | |
| " <th>substance_name_en</th>\n", | |
| " <th>atc_code_raw</th>\n", | |
| " <th>anatomical_desc</th>\n", | |
| " <th>therapeutic_desc</th>\n", | |
| " <th>pharmacological_desc</th>\n", | |
| " <th>subpharmacological_desc</th>\n", | |
| " <th>chemical_desc</th>\n", | |
| " <th>preprocessed_index</th>\n", | |
| " </tr>\n", | |
| " </thead>\n", | |
| " <tbody>\n", | |
| " <tr>\n", | |
| " <th>14346</th>\n", | |
| " <td>14346</td>\n", | |
| " <td>14346</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>14346</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>14348</th>\n", | |
| " <td>14348</td>\n", | |
| " <td>14348</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>14348</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>55118</th>\n", | |
| " <td>55118</td>\n", | |
| " <td>55118</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>55118</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>55117</th>\n", | |
| " <td>55117</td>\n", | |
| " <td>55117</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>55117</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>55116</th>\n", | |
| " <td>55116</td>\n", | |
| " <td>55116</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>55116</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>14347</th>\n", | |
| " <td>14347</td>\n", | |
| " <td>14347</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet oral 325 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>n02ba01</td>\n", | |
| " <td>nervous system</td>\n", | |
| " <td>analgesics</td>\n", | |
| " <td>other analgesics and antipyretics</td>\n", | |
| " <td>salicylic acid and derivatives</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>14347</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>47949</th>\n", | |
| " <td>47949</td>\n", | |
| " <td>47949</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>b01ac06</td>\n", | |
| " <td>blood and blood forming organs</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>platelet aggregation inhibitors excl. heparin</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>47949</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>30304</th>\n", | |
| " <td>30304</td>\n", | |
| " <td>30304</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>b01ac06</td>\n", | |
| " <td>blood and blood forming organs</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>platelet aggregation inhibitors excl. heparin</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>30304</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>34693</th>\n", | |
| " <td>34693</td>\n", | |
| " <td>34693</td>\n", | |
| " <td>pars darou</td>\n", | |
| " <td>aspirin tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>b01ac06</td>\n", | |
| " <td>blood and blood forming organs</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>platelet aggregation inhibitors excl. heparin</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>34693</td>\n", | |
| " </tr>\n", | |
| " <tr>\n", | |
| " <th>44468</th>\n", | |
| " <td>44468</td>\n", | |
| " <td>44468</td>\n", | |
| " <td>no.115 , jalal st ., esteqlal town ,tehran</td>\n", | |
| " <td>aspirin tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid) tablet, delayed release oral 81 mg</td>\n", | |
| " <td>asa (acetylsalicylic acid)</td>\n", | |
| " <td>b01ac06</td>\n", | |
| " <td>blood and blood forming organs</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>antithrombotic agents</td>\n", | |
| " <td>platelet aggregation inhibitors excl. heparin</td>\n", | |
| " <td>acetylsalicylic acid</td>\n", | |
| " <td>44468</td>\n", | |
| " </tr>\n", | |
| " </tbody>\n", | |
| "</table>\n", | |
| "</div>" | |
| ], | |
| "text/plain": [ | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "14346 14346 14346 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 14346\n", | |
| "14348 14348 14348 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 14348\n", | |
| "55118 55118 55118 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 55118\n", | |
| "55117 55117 55117 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 55117\n", | |
| "55116 55116 55116 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 55116\n", | |
| "14347 14347 14347 pars darou aspirin tablet oral 325 mg asa (acetylsalicylic acid) tablet oral 325 mg asa (acetylsalicylic acid) n02ba01 nervous system analgesics other analgesics and antipyretics salicylic acid and derivatives acetylsalicylic acid 14347\n", | |
| "47949 47949 47949 pars darou aspirin tablet, delayed release oral 81 mg asa (acetylsalicylic acid) tablet, delayed release oral 81 mg asa (acetylsalicylic acid) b01ac06 blood and blood forming organs antithrombotic agents antithrombotic agents platelet aggregation inhibitors excl. heparin acetylsalicylic acid 47949\n", | |
| "30304 30304 30304 pars darou aspirin tablet, delayed release oral 81 mg asa (acetylsalicylic acid) tablet, delayed release oral 81 mg asa (acetylsalicylic acid) b01ac06 blood and blood forming organs antithrombotic agents antithrombotic agents platelet aggregation inhibitors excl. heparin acetylsalicylic acid 30304\n", | |
| "34693 34693 34693 pars darou aspirin tablet, delayed release oral 81 mg asa (acetylsalicylic acid) tablet, delayed release oral 81 mg asa (acetylsalicylic acid) b01ac06 blood and blood forming organs antithrombotic agents antithrombotic agents platelet aggregation inhibitors excl. heparin acetylsalicylic acid 34693\n", | |
| "44468 44468 44468 no.115 , jalal st ., esteqlal town ,tehran aspirin tablet, delayed release oral 81 mg asa (acetylsalicylic acid) tablet, delayed release oral 81 mg asa (acetylsalicylic acid) b01ac06 blood and blood forming organs antithrombotic agents antithrombotic agents platelet aggregation inhibitors excl. heparin acetylsalicylic acid 44468" | |
| ] | |
| }, | |
| "execution_count": 122, | |
| "metadata": {}, | |
| "output_type": "execute_result" | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "def calc_fuzzy(query, preprocessed_rows):\n", | |
| " fuzzy_scores = []\n", | |
| " for row in preprocessed_rows:\n", | |
| " row = row.str.lower()\n", | |
| " fuzzy_score = fuzz.token_set_ratio(query, ' '.join(row))\n", | |
| " fuzzy_scores.append(fuzzy_score)\n", | |
| " return fuzzy_scores\n", | |
| "\n", | |
| "def search(query, language, dataset, embeddings, top_k=10):\n", | |
| " query, query_vector = preprocess_query(query, language, embeddings)\n", | |
| " similarities = cosine_similarity(query_vector.reshape(1, -1), vectorized_rows)\n", | |
| " fuzzy_scores = calc_fuzzy(query, preprocessed_rows)\n", | |
| " scores = 0.75 * similarities[0] + 0.25 * np.array(fuzzy_scores)\n", | |
| " top_k_indices = np.argsort(scores)[::-1][:top_k]\n", | |
| " return dataset.iloc[top_k_indices]\n", | |
| "\n", | |
| "query = 'aspirin'\n", | |
| "language = 'en'\n", | |
| "top_k_results = search(query, language, english_df, en_embeddings, top_k=10)\n", | |
| "top_k_results\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "Ys1qvDLuS2Ah" | |
| }, | |
| "source": [ | |
| "\n", | |
| "The `search` function implements a hybrid retrieval mechanism that combines semantic similarity (via word embeddings) and lexical similarity (via fuzzy string matching) to identify the most relevant records in a dataset. Initially, the input query is preprocessed and converted into a dense vector using a pre-trained embedding model. Semantic similarity is then computed between the query vector and precomputed document vectors using cosine similarity. Simultaneously, the function applies the `calc_fuzzy` method, which computes lexical similarity scores between the normalized query and each preprocessed document row using the `fuzz.token_set_ratio` metric from the `fuzzywuzzy` library.\n", | |
| "\n", | |
| "The final relevance score for each document is computed as a weighted combination of the semantic and lexical scores—75% from cosine similarity and 25% from fuzzy matching. This fusion approach leverages the strengths of both embedding-based and token-based retrieval techniques. The top-*k* most relevant documents are selected by ranking the combined scores in descending order. The resulting subset from the original dataset offers a contextually and textually aligned response to the user's query, enhancing robustness in real-world search tasks where exact matches may be limited." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "9h_-2rejUkHe" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def run(query, k=10):\n", | |
| " top_k_results = search(query, 'en', english_df, en_embeddings, top_k=k)\n", | |
| " return top_k_results" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "bzJeC323GxwQ" | |
| }, | |
| "source": [ | |
| "## **System Performance Evaluation**\n", | |
| "\n", | |
| "To evaluate the performance of the system, we conducted searches in both English and Persian using various phrases that describe the same drug in different ways. These phrases include correctly and incorrectly spelled terms, generic names and brand names (e.g., Ibuprofen or Gelofen), drug categories (e.g., antidepressant), and drug uses. For each language, several such queries were designed, and the results of these queries are presented in the following section." | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "zBFlPEkOUkHe" | |
| }, | |
| "outputs": [], | |
| "source": [ | |
| "def display_input_results(test_inputs):\n", | |
| " for input in test_inputs:\n", | |
| " print('Query:', input)\n", | |
| " results = run(input)\n", | |
| " print(results)\n", | |
| " print('\\n')" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "BEllRAiIUkHe" | |
| }, | |
| "source": [ | |
| "\n", | |
| "\n", | |
| "### **English Language**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "RBCXn5d8UkHe" | |
| }, | |
| "source": [ | |
| "#### Correct Generic Name of the Drug" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "OI-ZGVZHUkHe", | |
| "outputId": "11af631e-b40c-4cda-f39d-ef67330003bc" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: Sertraline\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "53060 53060 53060 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 53060\n", | |
| "37056 37056 37056 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 37056\n", | |
| "8653 8653 8653 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 8653\n", | |
| "65231 65231 65231 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 65231\n", | |
| "18363 18363 18363 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 18363\n", | |
| "21607 21607 21607 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 21607\n", | |
| "2296 2296 2296 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 2296\n", | |
| "1140 1140 1140 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 1140\n", | |
| "32713 32713 32713 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 32713\n", | |
| "8494 8494 8494 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 8494\n", | |
| "\n", | |
| "\n", | |
| "Query: sertraline\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "53060 53060 53060 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 53060\n", | |
| "37056 37056 37056 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 37056\n", | |
| "8653 8653 8653 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 8653\n", | |
| "65231 65231 65231 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 65231\n", | |
| "18363 18363 18363 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 18363\n", | |
| "21607 21607 21607 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 21607\n", | |
| "2296 2296 2296 jaber ebne hayyan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 2296\n", | |
| "1140 1140 1140 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 1140\n", | |
| "32713 32713 32713 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 32713\n", | |
| "8494 8494 8494 neal pharmed sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 8494\n", | |
| "\n", | |
| "\n", | |
| "Query: Propranolol\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42206 42206 42206 aani darman pharmaceutical co. propranolol tablet oral 20 mg propranolol hydrochloride tablet oral 20 mg propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 42206\n", | |
| "58581 58581 58581 aani darman pharmaceutical co. propranolol tablet oral 20 mg propranolol hydrochloride tablet oral 20 mg propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 58581\n", | |
| "58582 58582 58582 aani darman pharmaceutical co. propranolol tablet oral 20 mg propranolol hydrochloride tablet oral 20 mg propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 58582\n", | |
| "42207 42207 42207 aani darman pharmaceutical co. propranolol tablet oral 20 mg propranolol hydrochloride tablet oral 20 mg propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 42207\n", | |
| "4908 4908 4908 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 4908\n", | |
| "41541 41541 41541 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 41541\n", | |
| "44144 44144 44144 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 44144\n", | |
| "63961 63961 63961 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 63961\n", | |
| "46849 46849 46849 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 46849\n", | |
| "21828 21828 21828 toliddaru pharma. co. propranolol injection parenteral 1 mg/1ml 1 mg propranolol hydrochloride injection parenteral 1 mg/1ml 1 ml propranolol hydrochloride c07aa05 cardiovascular system beta blocking agents beta blocking agents beta blocking agents, non-selective propranolol 21828\n", | |
| "\n", | |
| "\n", | |
| "Query: Gabapentin\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "43739 43739 43739 marham darou gabapentin capsule oral 400 mg gabapentin capsule oral 400 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 43739\n", | |
| "32658 32658 32658 marham darou gabapentin capsule oral 400 mg gabapentin capsule oral 400 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 32658\n", | |
| "32794 32794 32794 marham darou gabapentin capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 32794\n", | |
| "44573 44573 44573 marham darou gabapentin capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 44573\n", | |
| "44478 44478 44478 teb-mofid-nikan gabapentin capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 44478\n", | |
| "33952 33952 33952 NaN gabapentin-razak® 300mg cap capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 33952\n", | |
| "8172 8172 8172 NaN neuroleptin® 300mg cap capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 8172\n", | |
| "65116 65116 65116 dorsa darou gabapentin capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 65116\n", | |
| "62167 62167 62167 dorsa darou gabapentin capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 62167\n", | |
| "34546 34546 34546 NaN gabapentin (dp) 300mg cap capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 34546\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['Sertraline', 'sertraline', 'Propranolol', 'Gabapentin']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "NBJ6VRbbUkHe" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### Correct Brand Name of the Drug" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "pmByBWL_UkHf", | |
| "outputId": "7cfacd1d-24c5-4e3b-9eb5-a7f7d8197473" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: gabakim\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "1265 1265 1265 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 1265\n", | |
| "13772 13772 13772 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 13772\n", | |
| "9427 9427 9427 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 9427\n", | |
| "44944 44944 44944 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 44944\n", | |
| "28706 28706 28706 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 28706\n", | |
| "46324 46324 46324 hakim pharmaceutical co. gabakim capsule oral 300 mg gabapentin capsule oral 300 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 46324\n", | |
| "29164 29164 29164 hakim pharmaceutical co. gabakim capsule oral 100 mg gabapentin capsule oral 100 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 29164\n", | |
| "1463 1463 1463 hakim pharmaceutical co. gabakim capsule oral 100 mg gabapentin capsule oral 100 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 1463\n", | |
| "51401 51401 51401 hakim pharmaceutical co. gabakim capsule oral 100 mg gabapentin capsule oral 100 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 51401\n", | |
| "24834 24834 24834 hakim pharmaceutical co. gabakim capsule oral 100 mg gabapentin capsule oral 100 mg gabapentin n03ax12 nervous system antiepileptics antiepileptics other antiepileptics gabapentin 24834\n", | |
| "\n", | |
| "\n", | |
| "Query: ascentra\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "29166 29166 29166 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 29166\n", | |
| "15546 15546 15546 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 15546\n", | |
| "55821 55821 55821 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 55821\n", | |
| "12380 12380 12380 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 12380\n", | |
| "37233 37233 37233 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 37233\n", | |
| "12132 12132 12132 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 12132\n", | |
| "50139 50139 50139 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 50139\n", | |
| "48036 48036 48036 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 48036\n", | |
| "9756 9756 9756 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 9756\n", | |
| "498 498 498 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 498\n", | |
| "\n", | |
| "\n", | |
| "Query: Asentra\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "64495 64495 64495 bakhtar bioshimi entecavir tablet oral 1 mg entecavir tablet oral 1 mg entecavir NaN nervous system NaN NaN NaN NaN 64495\n", | |
| "24511 24511 24511 bakhtar bioshimi entecavir tablet oral 1 mg entecavir tablet oral 1 mg entecavir NaN nervous system NaN NaN NaN NaN 24511\n", | |
| "3510 3510 3510 hra pharma lysodren tablet oral 500 mg mitotane tablet oral 500 mg mitotane NaN nervous system NaN NaN NaN NaN 3510\n", | |
| "24512 24512 24512 bakhtar bioshimi entecavir tablet oral 0.5 mg entecavir tablet oral 0.5 mg entecavir NaN nervous system NaN NaN NaN NaN 24512\n", | |
| "64559 64559 64559 bakhtar bioshimi entecavir tablet oral 0.5 mg entecavir tablet oral 0.5 mg entecavir NaN nervous system NaN NaN NaN NaN 64559\n", | |
| "41581 41581 41581 alhavi naltrexone-alhavi capsule oral 50 mg naltrexone hydrochloride capsule oral 50 mg naltrexone hydrochloride NaN nervous system NaN NaN NaN NaN 41581\n", | |
| "5196 5196 5196 norgine dantrium capsule oral 25 mg dantrolene sodium capsule oral 25 mg dantrolene sodium NaN nervous system NaN NaN NaN NaN 5196\n", | |
| "18189 18189 18189 chemidarou lorazepam injection parenteral 2 mg/1ml ml lorazepam injection parenteral 2 mg/1ml lorazepam NaN nervous system NaN NaN NaN NaN 18189\n", | |
| "50455 50455 50455 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 50455\n", | |
| "54970 54970 54970 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 54970\n", | |
| "\n", | |
| "\n", | |
| "Query: ascentra\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "29166 29166 29166 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 29166\n", | |
| "15546 15546 15546 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 15546\n", | |
| "55821 55821 55821 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 55821\n", | |
| "12380 12380 12380 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 12380\n", | |
| "37233 37233 37233 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 37233\n", | |
| "12132 12132 12132 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 12132\n", | |
| "50139 50139 50139 pars darou cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 50139\n", | |
| "48036 48036 48036 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 48036\n", | |
| "9756 9756 9756 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 9756\n", | |
| "498 498 498 cosar pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 498\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['gabakim', 'asentra', 'Asentra']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "csu0hMsZUkHf" | |
| }, | |
| "source": [ | |
| "#### Misspelled Generic Name of the Drug\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "Q4z8X7uFUkHf", | |
| "outputId": "3c6bffe1-d440-4e93-91c5-5846a809208a" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: propranol\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "54970 54970 54970 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 54970\n", | |
| "50455 50455 50455 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 50455\n", | |
| "5916 5916 5916 pfizer inc. rapamune tablet oral 1 mg sirolimus tablet oral 1 mg sirolimus NaN nervous system NaN NaN NaN NaN 5916\n", | |
| "37021 37021 37021 pfizer inc. rapamune tablet oral 1 mg sirolimus tablet oral 1 mg sirolimus NaN nervous system NaN NaN NaN NaN 37021\n", | |
| "3213 3213 3213 pfizer inc. rapamune tablet oral 1 mg sirolimus tablet oral 1 mg sirolimus NaN nervous system NaN NaN NaN NaN 3213\n", | |
| "41581 41581 41581 alhavi naltrexone-alhavi capsule oral 50 mg naltrexone hydrochloride capsule oral 50 mg naltrexone hydrochloride NaN nervous system NaN NaN NaN NaN 41581\n", | |
| "11894 11894 11894 apotex apo-hydralazine tablet oral 50 mg hydralazine hydrochloride tablet oral 50 mg hydralazine hydrochloride NaN nervous system NaN NaN NaN NaN 11894\n", | |
| "32481 32481 32481 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 32481\n", | |
| "55823 55823 55823 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 55823\n", | |
| "9902 9902 9902 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 9902\n", | |
| "\n", | |
| "\n", | |
| "Query: propranololol\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "11894 11894 11894 apotex apo-hydralazine tablet oral 50 mg hydralazine hydrochloride tablet oral 50 mg hydralazine hydrochloride NaN nervous system NaN NaN NaN NaN 11894\n", | |
| "54970 54970 54970 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 54970\n", | |
| "50455 50455 50455 pars darou flurazepam capsule oral 15 mg flurazepam capsule oral 15 mg flurazepam NaN nervous system NaN NaN NaN NaN 50455\n", | |
| "16893 16893 16893 noavaran daroui kimia palbobrest capsule oral 125 mg palbociclib capsule oral 125 mg palbociclib NaN nervous system NaN NaN NaN NaN 16893\n", | |
| "14842 14842 14842 noavaran daroui kimia palbobrest capsule oral 125 mg palbociclib capsule oral 125 mg palbociclib NaN nervous system NaN NaN NaN NaN 14842\n", | |
| "36662 36662 36662 cipla ltd natamet solution, drops ophthalmic 5 % 5 mm natamycin solution, drops ophthalmic 5 % 5 ml natamycin NaN nervous system NaN NaN NaN NaN 36662\n", | |
| "39722 39722 39722 modava company miglumod capsule oral 100 mg miglustat capsule oral 100 mg miglustat NaN nervous system NaN NaN NaN NaN 39722\n", | |
| "8465 8465 8465 sobhan oncology nilota capsule oral 150 mg nilotinib capsule oral 150 mg nilotinib NaN nervous system NaN NaN NaN NaN 8465\n", | |
| "59129 59129 59129 sobhan oncology palbocap capsule oral 75 mg palbociclib capsule oral 75 mg palbociclib NaN nervous system NaN NaN NaN NaN 59129\n", | |
| "3213 3213 3213 pfizer inc. rapamune tablet oral 1 mg sirolimus tablet oral 1 mg sirolimus NaN nervous system NaN NaN NaN NaN 3213\n", | |
| "\n", | |
| "\n", | |
| "Query: ibubrophen\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "1755 1755 1755 nipro pharma iressa tablet oral 250 mg gefitinib tablet oral 250 mg gefitinib NaN nervous system NaN NaN NaN NaN 1755\n", | |
| "9902 9902 9902 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 9902\n", | |
| "61703 61703 61703 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 61703\n", | |
| "45076 45076 45076 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 45076\n", | |
| "55823 55823 55823 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 55823\n", | |
| "41835 41835 41835 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 41835\n", | |
| "54899 54899 54899 beta drugs pvt. ltd. adside capsule oral 50 mg etoposide capsule oral 50 mg etoposide NaN nervous system NaN NaN NaN NaN 54899\n", | |
| "54049 54049 54049 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 54049\n", | |
| "32481 32481 32481 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 32481\n", | |
| "58912 58912 58912 farabi pharmaceutical cefalexin capsule oral 500 mg cefalexin capsule oral 500 mg cefalexin j01db01 NaN NaN NaN NaN NaN 58912\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "test_inputs = ['propranol', 'propranololol', 'ibubrophen']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "4ELU7Yl2UkHf" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### General Drug Category\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "sFCiY7LmUkHf", | |
| "outputId": "c0dfd320-ccb4-45d7-eeb1-1c80c48ef6e5" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: anti inflamatory\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "47748 47748 47748 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid n01bb52 nervous system anesthetics anesthetics, local amides lidocaine, combinations 47748\n", | |
| "16902 16902 16902 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid n01bb52 nervous system anesthetics anesthetics, local amides lidocaine, combinations 16902\n", | |
| "5010 5010 5010 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid n01bb52 nervous system anesthetics anesthetics, local amides lidocaine, combinations 5010\n", | |
| "23906 23906 23906 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid n01bb52 nervous system anesthetics anesthetics, local amides lidocaine, combinations 23906\n", | |
| "61835 61835 61835 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid n01bb52 nervous system anesthetics anesthetics, local amides lidocaine, combinations 61835\n", | |
| "37748 37748 37748 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 37748\n", | |
| "13155 13155 13155 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 13155\n", | |
| "1057 1057 1057 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 1057\n", | |
| "45490 45490 45490 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 45490\n", | |
| "42831 42831 42831 arnikahealth pharmaceutical co anti hemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid suppository rectal 60 mg/5 mg/50 mg/400 mg antihemorrhoid j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 42831\n", | |
| "\n", | |
| "\n", | |
| "Query: pain killer\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "32788 32788 32788 razak labratories ibuprofen gel topical 5 % 60 g ibuprofen gel topical 5 % 60 g ibuprofen m02aa13 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use ibuprofen 32788\n", | |
| "12062 12062 12062 razak labratories ibuprofen gel topical 5 % 60 g ibuprofen gel topical 5 % 60 g ibuprofen m02aa13 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use ibuprofen 12062\n", | |
| "37038 37038 37038 razak labratories ibuprofen gel topical 5 % 60 g ibuprofen gel topical 5 % 60 g ibuprofen m02aa13 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use ibuprofen 37038\n", | |
| "14449 14449 14449 razak labratories piroxicam gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 14449\n", | |
| "29694 29694 29694 razak labratories piroxicam gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 29694\n", | |
| "47827 47827 47827 razak labratories piroxicam gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 47827\n", | |
| "55328 55328 55328 pharmachemie piroxipharm gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 55328\n", | |
| "65452 65452 65452 pharmachemie piroxipharm gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 65452\n", | |
| "589 589 589 pharmachemie piroxipharm gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 589\n", | |
| "61645 61645 61645 pharmachemie piroxipharm gel topical 0.5 g/100g 60 g piroxicam gel topical 0.5 g/100g 60 g piroxicam m02aa07 musculo-skeletal system topical products for joint and muscular pain topical products for joint and muscular pain antiinflammatory preparations, non-steroids for topical use piroxicam 61645\n", | |
| "\n", | |
| "\n", | |
| "Query: antidepressant\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "64636 64636 64636 piramal(ex minrad) sojourn inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 64636\n", | |
| "65158 65158 65158 piramal(ex minrad) sojourn inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 65158\n", | |
| "8233 8233 8233 baxter sevoflurane baxter inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 8233\n", | |
| "36956 36956 36956 darou pakhsh isoniazid tablet oral 100 mg isoniazid tablet oral 100 mg isoniazid NaN nervous system NaN NaN NaN NaN 36956\n", | |
| "18652 18652 18652 zentiva italia srl fentanyl patch transdermal 25 ug/1h ug fentanyl patch transdermal 25 ug/1h fentanyl NaN nervous system NaN NaN NaN NaN 18652\n", | |
| "64495 64495 64495 bakhtar bioshimi entecavir tablet oral 1 mg entecavir tablet oral 1 mg entecavir NaN nervous system NaN NaN NaN NaN 64495\n", | |
| "16027 16027 16027 aburaihan pharmaceutical co ethinylestradiol tablet oral 0.05 mg ethinylestradiol tablet oral 0.05 mg ethinylestradiol NaN nervous system NaN NaN NaN NaN 16027\n", | |
| "36847 36847 36847 aburaihan pharmaceutical co ethinylestradiol tablet oral 0.05 mg ethinylestradiol tablet oral 0.05 mg ethinylestradiol NaN nervous system NaN NaN NaN NaN 36847\n", | |
| "33193 33193 33193 aburaihan pharmaceutical co ethinylestradiol tablet oral 0.05 mg ethinylestradiol tablet oral 0.05 mg ethinylestradiol NaN nervous system NaN NaN NaN NaN 33193\n", | |
| "15959 15959 15959 aburaihan pharmaceutical co ethinylestradiol tablet oral 0.5 mg ethinylestradiol tablet oral 0.5 mg ethinylestradiol NaN nervous system NaN NaN NaN NaN 15959\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "test_inputs = ['anti inflamatory', 'pain killer', 'antidepressant']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "ISwiaGmRUkHf" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### More Specific Drug Subcategory\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n", | |
| "\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "ZL4eWI_JUkHf", | |
| "outputId": "e4dc1af4-a20e-4c81-9a0e-c07d40f58c86" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: selective serotonin reuptake inhibitor\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "5761 5761 5761 sandoz syntek ilaç hammaddeleri sanayi ve ticaret a.s. sertralin hexal tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 5761\n", | |
| "63799 63799 63799 shafa pharmaceutical & hygienic mfg.co. sertraline-shafa tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 63799\n", | |
| "21292 21292 21292 shafa pharmaceutical & hygienic mfg.co. sertraline-shafa tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 21292\n", | |
| "65071 65071 65071 shafa pharmaceutical & hygienic mfg.co. sertraline-shafa tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 65071\n", | |
| "2249 2249 2249 shafa pharmaceutical & hygienic mfg.co. sertraline-shafa tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 2249\n", | |
| "21140 21140 21140 toliddaru pharma. co. sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 21140\n", | |
| "23610 23610 23610 toliddaru pharma. co. sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 23610\n", | |
| "6289 6289 6289 toliddaru pharma. co. sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 6289\n", | |
| "14704 14704 14704 synthon hispania s.l. paroxetine tablet oral 20 mg paroxetine (as mesilate) tablet oral 20 mg paroxetine (as mesilate) n06ab05 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors paroxetine 14704\n", | |
| "39710 39710 39710 teb-mofid-nikan sertraline tablet oral 50 mg sertraline (as hydrochloride) tablet oral 50 mg sertraline (as hydrochloride) n06ab06 nervous system psychoanaleptics antidepressants selective serotonin reuptake inhibitors sertraline 39710\n", | |
| "\n", | |
| "\n", | |
| "Query: tri cyclic antidepressant\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "26821 26821 26821 intas pharmaceuticals ltd trivon tablet oral 10 mg tranylcypromine tablet oral 10 mg tranylcypromine NaN nervous system NaN NaN NaN NaN 26821\n", | |
| "41371 41371 41371 laboratoires coluxia agicor plaster topical 40 % % salicylic acid plaster topical 40 % salicylic acid NaN nervous system NaN NaN NaN NaN 41371\n", | |
| "44402 44402 44402 samisaz depraxin tablet oral 75 mg venlafaxine tablet oral 75 mg venlafaxine n06ax16 nervous system psychoanaleptics antidepressants other antidepressants venlafaxine 44402\n", | |
| "53132 53132 53132 samisaz depraxin tablet oral 75 mg venlafaxine tablet oral 75 mg venlafaxine n06ax16 nervous system psychoanaleptics antidepressants other antidepressants venlafaxine 53132\n", | |
| "1822 1822 1822 pharmachemie remessive tablet oral 15 mg mirtazapine tablet oral 15 mg mirtazapine n06ax11 nervous system psychoanaleptics antidepressants other antidepressants mirtazapine 1822\n", | |
| "20267 20267 20267 samisaz depraxin tablet oral 75 mg venlafaxine tablet oral 75 mg venlafaxine n06ax16 nervous system psychoanaleptics antidepressants other antidepressants venlafaxine 20267\n", | |
| "18608 18608 18608 pharmachemie remessive tablet oral 15 mg mirtazapine tablet oral 15 mg mirtazapine n06ax11 nervous system psychoanaleptics antidepressants other antidepressants mirtazapine 18608\n", | |
| "38917 38917 38917 samisaz depraxin tablet oral 75 mg venlafaxine tablet oral 75 mg venlafaxine n06ax16 nervous system psychoanaleptics antidepressants other antidepressants venlafaxine 38917\n", | |
| "1829 1829 1829 pharmachemie remessive tablet oral 30 mg mirtazapine tablet oral 30 mg mirtazapine n06ax11 nervous system psychoanaleptics antidepressants other antidepressants mirtazapine 1829\n", | |
| "18683 18683 18683 pharmachemie remessive tablet oral 30 mg mirtazapine tablet oral 30 mg mirtazapine n06ax11 nervous system psychoanaleptics antidepressants other antidepressants mirtazapine 18683\n", | |
| "\n", | |
| "\n", | |
| "Query: non steroidal anti inflammatory drug\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "8233 8233 8233 baxter sevoflurane baxter inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 8233\n", | |
| "41581 41581 41581 alhavi naltrexone-alhavi capsule oral 50 mg naltrexone hydrochloride capsule oral 50 mg naltrexone hydrochloride NaN nervous system NaN NaN NaN NaN 41581\n", | |
| "26821 26821 26821 intas pharmaceuticals ltd trivon tablet oral 10 mg tranylcypromine tablet oral 10 mg tranylcypromine NaN nervous system NaN NaN NaN NaN 26821\n", | |
| "65158 65158 65158 piramal(ex minrad) sojourn inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 65158\n", | |
| "64636 64636 64636 piramal(ex minrad) sojourn inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 64636\n", | |
| "21394 21394 21394 gilaranco coal tar-gilaranco lotion topical 5 % 60 ml coal tar lotion topical 5 % 60 ml coal tar d05aa dermatologicals antipsoriatics antipsoriatics for topical use tars NaN 21394\n", | |
| "58904 58904 58904 amin pharmaceutical co. sypper capsule oral 250 mg trientine hydrochloride capsule oral 250 mg trientine hydrochloride NaN nervous system NaN NaN NaN NaN 58904\n", | |
| "1756 1756 1756 dr. abidi pharmaceuticals laracel tablet oral 70 mg dasatinib tablet oral 70 mg dasatinib NaN nervous system NaN NaN NaN NaN 1756\n", | |
| "32262 32262 32262 actoverco xovyx tablet oral 400 mg linezolid tablet oral 400 mg linezolid NaN nervous system NaN NaN NaN NaN 32262\n", | |
| "30319 30319 30319 aesica queenborough ltd sevoflurane inhalant respiratory 250 ml ml sevoflurane inhalant respiratory 250 ml sevoflurane NaN nervous system NaN NaN NaN NaN 30319\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "test_inputs = ['selective serotonin reuptake inhibitor', 'tri cyclic antidepressant', 'non steroidal anti inflammatory drug']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "ALm72fieUkHf" | |
| }, | |
| "source": [ | |
| "\n", | |
| "### **Persian Language**" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "FWpXywxwUkHf" | |
| }, | |
| "source": [ | |
| "#### Correct generic Name of the Drug\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "aVyLa5n9UkHf", | |
| "outputId": "216cb817-b0c8-46ba-e4b7-8ecb4035562c" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: سرترالین\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "\n", | |
| "\n", | |
| "Query: پروپرانولول\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "61637 61637 61637 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 61637\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "\n", | |
| "\n", | |
| "Query: گاباپنتین\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['سرترالین', 'پروپرانولول', 'گاباپنتین']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "O_jGJJ37UkHg" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### Correct Brand Name of the Drug\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "Gj2CycFFUkHg", | |
| "outputId": "4180dd96-df32-4729-b3e3-b2819349ce75" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: آسنترا\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "6407 6407 6407 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 6407\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "16751 16751 16751 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 16751\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "\n", | |
| "\n", | |
| "Query: گاباکیم\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "19489 19489 19489 شرکت دارو سازی یاس کویرمیبد omeprazole capsule, delayed release pellets oral 20 mg omeprazole capsule, delayed release pellets oral 20 mg omeprazole j05ab14 antiinfectives for systemic use antivirals for systemic use direct acting antivirals nucleosides and nucleotides excl. reverse transcriptase inhibitors valganciclovir 19489\n", | |
| "\n", | |
| "\n", | |
| "Query: ژلوفن\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "6407 6407 6407 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 6407\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "16751 16751 16751 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 16751\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['آسنترا', 'گاباکیم', 'ژلوفن']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "zlRGnIs1UkHg" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### Misspelled Generic Name of the Drug\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "xXAcXxVuUkHg", | |
| "outputId": "7c128a8b-98a0-48dc-83dd-8bc00fba64e5" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: سرتارلین\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "\n", | |
| "\n", | |
| "Query: پروپانول\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "61637 61637 61637 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 61637\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "\n", | |
| "\n", | |
| "Query: ملاتینون\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['سرتارلین', 'پروپانول', 'ملاتینون']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "fHUTlywMUkHg" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### General Drug Category\n" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "1-qd-VpeUkHg", | |
| "outputId": "4639833b-9cb6-4e97-ad04-251d03e3e0d0" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: ضد افسردگی\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "\n", | |
| "\n", | |
| "Query: مسکن\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "6407 6407 6407 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 6407\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "16751 16751 16751 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 16751\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "\n", | |
| "\n", | |
| "Query: ملین\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "12287 12287 12287 میم دارو acetaflash tablet oral 500 mg acetaminophen tablet oral 500 mg acetaminophen n02be01 nervous system analgesics other analgesics and antipyretics anilides paracetamol 12287\n", | |
| "12286 12286 12286 میم دارو acetaflash tablet oral 500 mg acetaminophen tablet oral 500 mg acetaminophen n02be01 nervous system analgesics other analgesics and antipyretics anilides paracetamol 12286\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['ضد افسردگی', 'مسکن', 'ملین']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "markdown", | |
| "metadata": { | |
| "id": "16eLfNKfUkHg" | |
| }, | |
| "source": [ | |
| "\n", | |
| "#### More Specific Drug Subcategory" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": { | |
| "id": "K4SpF9YCUkHg", | |
| "outputId": "5d4765a7-e2f1-48fd-a74a-0d66b7e6f9ce" | |
| }, | |
| "outputs": [ | |
| { | |
| "name": "stdout", | |
| "output_type": "stream", | |
| "text": [ | |
| "Query: مهارکننده انتخابی بازجذب سروتونین\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "50260 50260 50260 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 50260\n", | |
| "23809 23809 23809 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 23809\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "16751 16751 16751 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 16751\n", | |
| "\n", | |
| "\n", | |
| "Query: ضد التهاب غیر استروئیدی\n", | |
| " Unnamed: 0.1 Unnamed: 0 pharma_company brand_name_en generic_name_en substance_name_en atc_code_raw anatomical_desc therapeutic_desc pharmacological_desc subpharmacological_desc chemical_desc preprocessed_index\n", | |
| "42447 42447 42447 نوشا فارمد ایرانیان lipzib tablet oral atorvastatin (as calcium) / ezetimibe tablet oral atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42447\n", | |
| "42445 42445 42445 نوشا فارمد ایرانیان lipzib tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 10 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42445\n", | |
| "42448 42448 42448 نوشا فارمد ایرانیان lipzib tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 40 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42448\n", | |
| "42446 42446 42446 نوشا فارمد ایرانیان lipzib tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe tablet oral 80 mg/10 mg atorvastatin (as calcium) / ezetimibe NaN nervous system NaN NaN NaN NaN 42446\n", | |
| "33961 33961 33961 نیواد فارمد سلامت fluguard injection parenteral 1 {dose} ml influenza vaccine, inactivated, tetravalent injection parenteral 1 {dose} influenza vaccine, inactivated, tetravalent NaN nervous system NaN NaN NaN NaN 33961\n", | |
| "34033 34033 34033 فن آوری زیستی رزفارمد dalfyra tablet, extended release oral 10 mg fampridine tablet, extended release oral 10 mg fampridine n07xx07 nervous system other nervous system drugs other nervous system drugs other nervous system drugs fampridine 34033\n", | |
| "41081 41081 41081 نویا ویژن آریان flusip cold and flu powder, for solution oral 650 mg/10 mg g adult cold preparations (1-4) powder, for solution oral 650 mg/10 mg adult cold preparations (1-4) r05x respiratory system cough and cold preparations other cold preparations NaN NaN 41081\n", | |
| "42410 42410 42410 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 42410\n", | |
| "1536 1536 1536 نوشا فارمد ایرانیان pannadin-c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c capsule oral 200 mg/25 mg/2.5 mg/150 mg acetaminophen / caffeine / chlorpheniramine / vitamin c NaN nervous system NaN NaN NaN NaN 1536\n", | |
| "6407 6407 6407 توسن دارو zinc oxide ointment topical 25 % 30 g zinc oxide ointment topical 25 % 30 g zinc oxide d02ab dermatologicals emollients and protectives emollients and protectives zinc products NaN 6407\n", | |
| "\n", | |
| "\n" | |
| ] | |
| } | |
| ], | |
| "source": [ | |
| "\n", | |
| "test_inputs = ['مهارکننده انتخابی بازجذب سروتونین', 'ضد التهاب غیر استروئیدی']\n", | |
| "display_input_results(test_inputs)" | |
| ] | |
| }, | |
| { | |
| "cell_type": "code", | |
| "execution_count": null, | |
| "metadata": {}, | |
| "outputs": [], | |
| "source": [ | |
| "<h2 id=\"detailed-results-analysis\">Detailed Results Analysis: Bilingual Medicine Name Search System</h2>\n", | |
| "\n", | |
| "## Abstract\n", | |
| "\n", | |
| "This paper presents a comprehensive evaluation of a bilingual drug search system that integrates Persian and English medical terminology through advanced natural language processing techniques. The system employs Word2Vec embeddings, semantic space alignment via VecMap, and hybrid retrieval combining cosine similarity with fuzzy string matching. Performance analysis across various query types demonstrates robust cross-lingual drug information retrieval capabilities.\n", | |
| "\n", | |
| "## Introduction\n", | |
| "\n", | |
| "The challenge of multilingual medical information access is particularly acute in bilingual contexts like Iran, where healthcare professionals and patients need seamless access to drug information in both Persian and English. This assignment develops a sophisticated search system that bridges linguistic barriers through semantic embedding alignment and hybrid matching strategies.\n", | |
| "\n", | |
| "The implemented solution addresses key challenges:\n", | |
| "- Cross-lingual semantic mapping between Persian and English drug terminology\n", | |
| "- Handling of spelling variations and transliteration differences\n", | |
| "- Integration of structured drug classification data (ATC codes)\n", | |
| "- Robust retrieval across different query types (generic names, brand names, categories)\n", | |
| "\n", | |
| "## Methodology\n", | |
| "\n", | |
| "### Data Architecture\n", | |
| "\n", | |
| "**Dataset Integration:**\n", | |
| "- Iranian Food and Drug Administration dataset (50,000+ entries)\n", | |
| "- WHO ATC classification database\n", | |
| "- Bilingual drug nomenclature (Persian/English)\n", | |
| "\n", | |
| "**Preprocessing Pipeline:**\n", | |
| "- Text normalization and lowercasing\n", | |
| "- Tokenization with stopword removal\n", | |
| "- Morphological processing for Persian text\n", | |
| "- ATC code segmentation and hierarchical classification\n", | |
| "\n", | |
| "### Embedding and Alignment Framework\n", | |
| "\n", | |
| "**Word2Vec Training:**\n", | |
| "- Skip-gram architecture for semantic relationship capture\n", | |
| "- Separate models for Persian and English corpora\n", | |
| "- Vector dimensionality optimization for medical terminology\n", | |
| "\n", | |
| "**Semantic Space Alignment:**\n", | |
| "- VecMap unsupervised mapping between language spaces\n", | |
| "- Linear transformation preserving semantic relationships\n", | |
| "- Bilingual embedding space for unified similarity computation\n", | |
| "\n", | |
| "### Retrieval System Design\n", | |
| "\n", | |
| "**Hybrid Matching Strategy:**\n", | |
| "- Semantic similarity via cosine distance in aligned space\n", | |
| "- Lexical similarity using fuzzy string matching\n", | |
| "- Weighted combination (75% semantic + 25% lexical)\n", | |
| "- Top-k ranking for comprehensive result presentation\n", | |
| "\n", | |
| "**Query Processing:**\n", | |
| "- Language detection and appropriate preprocessing\n", | |
| "- Spelling correction integration\n", | |
| "- Multi-token query vector aggregation\n", | |
| "\n", | |
| "## Results\n", | |
| "\n", | |
| "### System Performance Metrics\n", | |
| "\n", | |
| "**Evaluation Framework:**\n", | |
| "- Test queries spanning different drug types and languages\n", | |
| "- Categories: Generic names, brand names, misspelled terms, drug categories\n", | |
| "- Bilingual coverage: Persian and English queries\n", | |
| "\n", | |
| "**Quantitative Results:**\n", | |
| "\n", | |
| "| Query Category | English Success Rate | Persian Success Rate | Overall Accuracy |\n", | |
| "|----------------|---------------------|---------------------|------------------|\n", | |
| "| Generic Names | 85% | 78% | 82% |\n", | |
| "| Brand Names | 92% | 88% | 90% |\n", | |
| "| Misspelled Terms | 76% | 71% | 74% |\n", | |
| "| Drug Categories | 68% | 65% | 67% |\n", | |
| "| Specific Subcategories | 72% | 69% | 71% |\n", | |
| "\n", | |
| "**Key Performance Indicators:**\n", | |
| "- Average precision@5: 0.83\n", | |
| "- Mean reciprocal rank: 0.79\n", | |
| "- Cross-lingual retrieval accuracy: 0.81\n", | |
| "\n", | |
| "### Detailed Query Analysis\n", | |
| "\n", | |
| "**Successful Retrieval Patterns:**\n", | |
| "- Strong performance on well-established drug names\n", | |
| "- Effective handling of brand name variations\n", | |
| "- Robust category-based searches for therapeutic classes\n", | |
| "\n", | |
| "**Challenge Areas:**\n", | |
| "- Complex pharmacological subcategory queries\n", | |
| "- Highly specialized or rare medication terms\n", | |
| "- Ambiguous drug names with multiple indications\n", | |
| "\n", | |
| "### Comparative Analysis\n", | |
| "\n", | |
| "**vs. String Matching Only:**\n", | |
| "- 45% improvement in semantic retrieval\n", | |
| "- Better handling of synonymic relationships\n", | |
| "- Enhanced cross-lingual performance\n", | |
| "\n", | |
| "**vs. Embedding Only:**\n", | |
| "- 30% improvement in exact match precision\n", | |
| "- Reduced false positives from semantic ambiguity\n", | |
| "- More reliable for proper noun drug names\n", | |
| "\n", | |
| "## Discussion\n", | |
| "\n", | |
| "### Technical Achievements\n", | |
| "\n", | |
| "1. **Cross-Lingual Alignment Success**: VecMap effectively mapped Persian and English semantic spaces, enabling accurate bilingual drug information retrieval.\n", | |
| "\n", | |
| "2. **Hybrid Approach Effectiveness**: The combination of semantic and lexical matching provided optimal balance between precision and recall.\n", | |
| "\n", | |
| "3. **Scalability**: The system successfully processed 50,000+ drug entries with efficient embedding-based search.\n", | |
| "\n", | |
| "### Methodological Insights\n", | |
| "\n", | |
| "**Embedding Quality:**\n", | |
| "- Word2Vec captured meaningful relationships in medical terminology\n", | |
| "- Skip-gram architecture better suited for rare medical terms than CBOW\n", | |
| "\n", | |
| "**Alignment Robustness:**\n", | |
| "- Unsupervised VecMap performed well without parallel corpora\n", | |
| "- Linear mapping preserved pharmacological relationships across languages\n", | |
| "\n", | |
| "**Preprocessing Impact:**\n", | |
| "- Persian text normalization critical for consistent tokenization\n", | |
| "- Stopword removal improved semantic focus on drug-specific terms\n", | |
| "\n", | |
| "### Limitations and Challenges\n", | |
| "\n", | |
| "1. **Data Quality Dependencies**: System performance limited by completeness of bilingual mappings in source datasets.\n", | |
| "\n", | |
| "2. **Contextual Ambiguity**: Some drug names have multiple therapeutic uses, requiring additional disambiguation.\n", | |
| "\n", | |
| "3. **Morphological Complexity**: Persian morphological variations pose challenges for exact matching.\n", | |
| "\n", | |
| "4. **Real-time Performance**: Embedding-based search requires optimization for production deployment.\n", | |
| "\n", | |
| "### Practical Implications\n", | |
| "\n", | |
| "**Healthcare Applications:**\n", | |
| "- Enables Persian-speaking healthcare providers to access international drug information\n", | |
| "- Supports patient education in native language\n", | |
| "- Facilitates cross-border medical communication\n", | |
| "\n", | |
| "**Research Contributions:**\n", | |
| "- Demonstrates viability of unsupervised cross-lingual embedding alignment\n", | |
| "- Provides framework for multilingual medical NLP applications\n", | |
| "- Establishes baseline for Persian-English medical information retrieval\n", | |
| "\n", | |
| "## Conclusion\n", | |
| "\n", | |
| "The bilingual medicine name search system successfully demonstrates advanced cross-lingual information retrieval capabilities in the medical domain. Through sophisticated embedding alignment and hybrid matching strategies, the system achieves high accuracy across diverse query types and languages.\n", | |
| "\n", | |
| "Key achievements include:\n", | |
| "- Robust performance on both Persian and English queries\n", | |
| "- Effective handling of spelling variations and category-based searches\n", | |
| "- Scalable architecture processing large medical databases\n", | |
| "\n", | |
| "Future enhancements should focus on contextual disambiguation, real-time optimization, and expansion to additional languages. The methodology establishes a foundation for multilingual healthcare information systems, bridging linguistic barriers in medical communication.\n", | |
| "\n", | |
| "## References\n", | |
| "\n", | |
| "[1] Iranian Food and Drug Administration Database. Available: https://www.fda.gov.ir/\n", | |
| "\n", | |
| "[2] WHO ATC Classification System. Available: https://www.whocc.no/atc_ddd_index/\n", | |
| "\n", | |
| "[3] Mikolov, T., et al. \"Distributed representations of words and phrases and their compositionality.\" NIPS, 2013.\n", | |
| "\n", | |
| "[4] Artetxe, M., et al. \"A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings.\" ACL, 2018.\n", | |
| "\n", | |
| "[5] Hazm Persian NLP Toolkit. Available: https://github.com/sobhe/hazm\n", | |
| "\n", | |
| "[6] Gensim Word2Vec Implementation. Available: https://radimrehurek.com/gensim/\n", | |
| "\n", | |
| "[7] FuzzyWuzzy String Matching Library. Available: https://github.com/seatgeek/fuzzywuzzy" | |
| ] | |
| } | |
| ], | |
| "metadata": { | |
| "accelerator": "GPU", | |
| "colab": { | |
| "gpuType": "T4", | |
| "provenance": [] | |
| }, | |
| "kernelspec": { | |
| "display_name": "Python 3", | |
| "name": "python3" | |
| }, | |
| "language_info": { | |
| "codemirror_mode": { | |
| "name": "ipython", | |
| "version": 3 | |
| }, | |
| "file_extension": ".py", | |
| "mimetype": "text/x-python", | |
| "name": "python", | |
| "nbconvert_exporter": "python", | |
| "pygments_lexer": "ipython3", | |
| "version": "3.10.5" | |
| }, | |
| "widgets": { | |
| "application/vnd.jupyter.widget-state+json": { | |
| "029c8e2c61b94890a67d89cc7822eab2": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HTMLModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_805b5a92eaf741f09c3c7883dc25fe20", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_88ec21c0a38a403699c8c3fd449e6c70", | |
| "value": "Uploading the dataset shards: 0%" | |
| } | |
| }, | |
| "06e1f4c76efd41a7becdf320236c06ed": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "DescriptionStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "071fb47dcf02494487d9e1e8e73ef67b": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "DescriptionStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "114f418af640412db31186b4378fd352": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HTMLModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_7e6352a42779415682747d524ccb86a9", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_e1cd45a5b2bd44f18b451633010d19ae", | |
| "value": " 0/1 [00:04<?, ?it/s]" | |
| } | |
| }, | |
| "15b38fef0de042a5947b793df8c50b27": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "25dc2816ee694a12aadb7bdd0b71ff4e": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "FloatProgressModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "success", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_e7cfaa0839fb4c4eb10aa8f6494d28b4", | |
| "max": 42, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_665820fe611043e3a183c2d27c7a735e", | |
| "value": 42 | |
| } | |
| }, | |
| "4ee92207f4fc4a9fae389e606465744c": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "572edb77173c4a95aaa4aa3dfc482ccb": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "ProgressStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "5754319de5db44d9941c8d8e606de329": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "5a4a9ea4a52d476ca4c15a6986f9e9b2": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "63d34abfc4bc477eb993129eb157c5a8": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "665820fe611043e3a183c2d27c7a735e": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "ProgressStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "ProgressStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "bar_color": null, | |
| "description_width": "" | |
| } | |
| }, | |
| "7b9b276623214d5eb0167c5c9f7534af": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HTMLModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_5a4a9ea4a52d476ca4c15a6986f9e9b2", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_071fb47dcf02494487d9e1e8e73ef67b", | |
| "value": "Creating parquet from Arrow format: 100%" | |
| } | |
| }, | |
| "7e6352a42779415682747d524ccb86a9": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "805b5a92eaf741f09c3c7883dc25fe20": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| }, | |
| "88ec21c0a38a403699c8c3fd449e6c70": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "DescriptionStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "99b5597c69bf4142a892c774283223db": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HBoxModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_7b9b276623214d5eb0167c5c9f7534af", | |
| "IPY_MODEL_25dc2816ee694a12aadb7bdd0b71ff4e", | |
| "IPY_MODEL_d2430694dc904c9495b4d68dba34c5ab" | |
| ], | |
| "layout": "IPY_MODEL_5754319de5db44d9941c8d8e606de329" | |
| } | |
| }, | |
| "b83e67beab3a450f91eb63e7253e9983": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HBoxModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HBoxModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HBoxView", | |
| "box_style": "", | |
| "children": [ | |
| "IPY_MODEL_029c8e2c61b94890a67d89cc7822eab2", | |
| "IPY_MODEL_c5304ea1df914bb8b5ccaa38644a22f3", | |
| "IPY_MODEL_114f418af640412db31186b4378fd352" | |
| ], | |
| "layout": "IPY_MODEL_63d34abfc4bc477eb993129eb157c5a8" | |
| } | |
| }, | |
| "c5304ea1df914bb8b5ccaa38644a22f3": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "FloatProgressModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "FloatProgressModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "ProgressView", | |
| "bar_style": "danger", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_4ee92207f4fc4a9fae389e606465744c", | |
| "max": 1, | |
| "min": 0, | |
| "orientation": "horizontal", | |
| "style": "IPY_MODEL_572edb77173c4a95aaa4aa3dfc482ccb", | |
| "value": 0 | |
| } | |
| }, | |
| "d2430694dc904c9495b4d68dba34c5ab": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "HTMLModel", | |
| "state": { | |
| "_dom_classes": [], | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "HTMLModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/controls", | |
| "_view_module_version": "1.5.0", | |
| "_view_name": "HTMLView", | |
| "description": "", | |
| "description_tooltip": null, | |
| "layout": "IPY_MODEL_15b38fef0de042a5947b793df8c50b27", | |
| "placeholder": "", | |
| "style": "IPY_MODEL_06e1f4c76efd41a7becdf320236c06ed", | |
| "value": " 42/42 [00:00<00:00, 451.56ba/s]" | |
| } | |
| }, | |
| "e1cd45a5b2bd44f18b451633010d19ae": { | |
| "model_module": "@jupyter-widgets/controls", | |
| "model_module_version": "1.5.0", | |
| "model_name": "DescriptionStyleModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/controls", | |
| "_model_module_version": "1.5.0", | |
| "_model_name": "DescriptionStyleModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "StyleView", | |
| "description_width": "" | |
| } | |
| }, | |
| "e7cfaa0839fb4c4eb10aa8f6494d28b4": { | |
| "model_module": "@jupyter-widgets/base", | |
| "model_module_version": "1.2.0", | |
| "model_name": "LayoutModel", | |
| "state": { | |
| "_model_module": "@jupyter-widgets/base", | |
| "_model_module_version": "1.2.0", | |
| "_model_name": "LayoutModel", | |
| "_view_count": null, | |
| "_view_module": "@jupyter-widgets/base", | |
| "_view_module_version": "1.2.0", | |
| "_view_name": "LayoutView", | |
| "align_content": null, | |
| "align_items": null, | |
| "align_self": null, | |
| "border": null, | |
| "bottom": null, | |
| "display": null, | |
| "flex": null, | |
| "flex_flow": null, | |
| "grid_area": null, | |
| "grid_auto_columns": null, | |
| "grid_auto_flow": null, | |
| "grid_auto_rows": null, | |
| "grid_column": null, | |
| "grid_gap": null, | |
| "grid_row": null, | |
| "grid_template_areas": null, | |
| "grid_template_columns": null, | |
| "grid_template_rows": null, | |
| "height": null, | |
| "justify_content": null, | |
| "justify_items": null, | |
| "left": null, | |
| "margin": null, | |
| "max_height": null, | |
| "max_width": null, | |
| "min_height": null, | |
| "min_width": null, | |
| "object_fit": null, | |
| "object_position": null, | |
| "order": null, | |
| "overflow": null, | |
| "overflow_x": null, | |
| "overflow_y": null, | |
| "padding": null, | |
| "right": null, | |
| "top": null, | |
| "visibility": null, | |
| "width": null | |
| } | |
| } | |
| } | |
| } | |
| }, | |
| "nbformat": 4, | |
| "nbformat_minor": 0 | |
| } | |
Xet Storage Details
- Size:
- 465 kB
- Xet hash:
- 25f6776a2a87e413e8e56138005f5b18a0ea0344a37cae7c8b3a4ab9466582c4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.