Upload SeedTranslate (2).ipynb
Browse files- SeedTranslate (2).ipynb +148 -0
SeedTranslate (2).ipynb
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"nbformat": 4,
|
| 3 |
+
"nbformat_minor": 0,
|
| 4 |
+
"metadata": {
|
| 5 |
+
"colab": {
|
| 6 |
+
"provenance": []
|
| 7 |
+
},
|
| 8 |
+
"kernelspec": {
|
| 9 |
+
"name": "python3",
|
| 10 |
+
"display_name": "Python 3"
|
| 11 |
+
},
|
| 12 |
+
"language_info": {
|
| 13 |
+
"name": "python"
|
| 14 |
+
}
|
| 15 |
+
},
|
| 16 |
+
"cells": [
|
| 17 |
+
{
|
| 18 |
+
"cell_type": "markdown",
|
| 19 |
+
"source": [
|
| 20 |
+
"**Welcome to your first Colab! First off, what is this? Where are we? What is going on?!?!**\n",
|
| 21 |
+
"\n",
|
| 22 |
+
"*In short:*\n",
|
| 23 |
+
"This is a **Google Colab**, think of this as a Google Doc but for coding. We use this platform as a big equalizer since it doesn't matter if you own a Mac, PC or Chromebook because every Colab runs in the cloud thanks to their servers!\n",
|
| 24 |
+
"\n",
|
| 25 |
+
"Lets get this party started! In just 4 lines we will be running our first translations model! To begin, mouse over the command under me and you should see a Play button appear. Go ahead and press it and wait approximately 30 seconds for it to download all the dependencies. Make sure at the bottom of the page you see a green checkmark to ensure a successful run!\n",
|
| 26 |
+
"\n",
|
| 27 |
+
"\n",
|
| 28 |
+
"*Reminder! It will NOT download anything on your computer, so if you come back to this program again later you will always need to begin from the top!*"
|
| 29 |
+
],
|
| 30 |
+
"metadata": {
|
| 31 |
+
"id": "xBrUa2CSXOXr"
|
| 32 |
+
}
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
"cell_type": "code",
|
| 36 |
+
"source": [
|
| 37 |
+
"!pip install transformers sentencepiece"
|
| 38 |
+
],
|
| 39 |
+
"metadata": {
|
| 40 |
+
"id": "FnAf9krqUwBz"
|
| 41 |
+
},
|
| 42 |
+
"execution_count": null,
|
| 43 |
+
"outputs": []
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"cell_type": "markdown",
|
| 47 |
+
"source": [
|
| 48 |
+
"Ok so let me give you a basic translations of what we just ran!\n",
|
| 49 |
+
"\n",
|
| 50 |
+
"pip = Package Installer Python - with this command we are telling the machine what program we are using to download our dependencies!\n",
|
| 51 |
+
"\n",
|
| 52 |
+
"install = this is the partner to Pip! We are specifing we want to install things, as opposed to 'pip unistall' or 'pip list' commands\n",
|
| 53 |
+
"\n",
|
| 54 |
+
"transformers = No this isn't a cool crime fighting robot (Yet! Maybe we can make one for a future project?) This is a tool offered by Hugging face to download and train pre trained models. Think of this as a smartphone, a super cool and complex tool to allow you to do all kinds of cool things (Phonecalls, pictures, angry birds, etc) [Read more about transformers here!](https://huggingface.co/docs/transformers/index)\n",
|
| 55 |
+
"\n",
|
| 56 |
+
"sentencepiece = This is a language-independent subword tokenizer and detokenizer for language models (I know, a mouth full!) But just think of this as a special iPhone app thats great at understanding languages [Read more about sentencepieces here!](https://medium.com/codex/sentencepiece-a-simple-and-language-independent-subword-tokenizer-and-detokenizer-for-neural-text-ffda431e704e)\n",
|
| 57 |
+
"\n",
|
| 58 |
+
"**Essentially translate**\n",
|
| 59 |
+
"\"*pip install transformers sentencepiece*\" *to* \"Buy a iPhone, download Google Translate\"\n",
|
| 60 |
+
"\n",
|
| 61 |
+
"**Next Step:**\n",
|
| 62 |
+
"Ok once you are ready, Mouse over the text below me and press play!\n"
|
| 63 |
+
],
|
| 64 |
+
"metadata": {
|
| 65 |
+
"id": "pjAy2TD5XQQu"
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"cell_type": "code",
|
| 70 |
+
"source": [
|
| 71 |
+
"from transformers import pipeline"
|
| 72 |
+
],
|
| 73 |
+
"metadata": {
|
| 74 |
+
"id": "EcU9xP_5L0rQ"
|
| 75 |
+
},
|
| 76 |
+
"execution_count": null,
|
| 77 |
+
"outputs": []
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"cell_type": "markdown",
|
| 81 |
+
"source": [
|
| 82 |
+
"**Translation**\n",
|
| 83 |
+
"\n",
|
| 84 |
+
"Go into \"tranformers\" (the thing we just downloaded), we are going to setup the pipeline feature. In short Pipeline just makes sure we keeping things short and simple, it exists simply to offer a simple API dedicated to several tasks. [If you want to read more!](https://huggingface.co/docs/transformers/main_classes/pipelines)\n",
|
| 85 |
+
"\n",
|
| 86 |
+
"**If there are no failures, keep going!**"
|
| 87 |
+
],
|
| 88 |
+
"metadata": {
|
| 89 |
+
"id": "i4KjeiLiXQ5s"
|
| 90 |
+
}
|
| 91 |
+
},
|
| 92 |
+
{
|
| 93 |
+
"cell_type": "code",
|
| 94 |
+
"source": [
|
| 95 |
+
"pipe = pipeline(task='text2text-generation', model='facebook/m2m100_418M')"
|
| 96 |
+
],
|
| 97 |
+
"metadata": {
|
| 98 |
+
"id": "yXUBigNfU02q"
|
| 99 |
+
},
|
| 100 |
+
"execution_count": null,
|
| 101 |
+
"outputs": []
|
| 102 |
+
},
|
| 103 |
+
{
|
| 104 |
+
"cell_type": "markdown",
|
| 105 |
+
"source": [
|
| 106 |
+
"**Ok we are almost there!**\n",
|
| 107 |
+
"\n",
|
| 108 |
+
"We just activated a Pipe. Think of a Pipe from the game \"Super Mario\" taking Mario from one specific area to another. With the \"transformers\" package we have thosands of things we can do, but in this case we just need to translate text. So lets just use a Pipe and get to what we need, in this case 'text2text-generation' for this project!\n",
|
| 109 |
+
"\n",
|
| 110 |
+
"Now the second part of our command have we have a model. This one is given to Facebook and is called \"m2m100\" or Many-to-Many multilingual translations. It is basically a super powerful translation dataset of 100 languages (hence the number 100) [Click me to read more about it!](https://huggingface.co/facebook/m2m100_418M)\n",
|
| 111 |
+
"\n",
|
| 112 |
+
"**You know what to do! Lets give it a spin**"
|
| 113 |
+
],
|
| 114 |
+
"metadata": {
|
| 115 |
+
"id": "nnD2qTT4XRg7"
|
| 116 |
+
}
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"cell_type": "code",
|
| 120 |
+
"source": [
|
| 121 |
+
"pipe(\"I love rock and roll\", forced_bos_token_id=pipe.tokenizer.get_lang_id(lang='zh'))"
|
| 122 |
+
],
|
| 123 |
+
"metadata": {
|
| 124 |
+
"id": "8e6HmITZU-YN"
|
| 125 |
+
},
|
| 126 |
+
"execution_count": null,
|
| 127 |
+
"outputs": []
|
| 128 |
+
},
|
| 129 |
+
{
|
| 130 |
+
"cell_type": "markdown",
|
| 131 |
+
"source": [
|
| 132 |
+
"Thats it! Congratulations, you just setup and ran your first language translation model and learned the basics of Google Colab and Hugging face!\n",
|
| 133 |
+
"\n",
|
| 134 |
+
"\n",
|
| 135 |
+
"\n",
|
| 136 |
+
"\n",
|
| 137 |
+
"\n",
|
| 138 |
+
"```\n",
|
| 139 |
+
"# This project was made with <3 by Igor!\n",
|
| 140 |
+
"```\n",
|
| 141 |
+
"\n"
|
| 142 |
+
],
|
| 143 |
+
"metadata": {
|
| 144 |
+
"id": "lVsc_TgeoexY"
|
| 145 |
+
}
|
| 146 |
+
}
|
| 147 |
+
]
|
| 148 |
+
}
|