diff --git "a/hands_on_machine_learning.json" "b/hands_on_machine_learning.json" new file mode 100644--- /dev/null +++ "b/hands_on_machine_learning.json" @@ -0,0 +1,88 @@ +[ + { + "part": "Part I", + "chapters": [ + { + "chapter": 1, + "text": "Chapter 1. The Machine\nLearning Landscape\nWhen most people hear “Machine Learning,” they picture a robot: a\ndependable butler or a deadly Terminator, depending on who you ask. But\nMachine Learning is not just a futuristic fantasy; it’s already here. In fact,\nit has been around for decades in some specialized applications, such as\nOptical Character Recognition (OCR). But the first ML application that\nreally became mainstream, improving the lives of hundreds of millions of\npeople, took over the world back in the 1990s: the spam filter. It’s not\nexactly a self-aware Skynet, but it does technically qualify as Machine\nLearning (it has actually learned so well that you seldom need to flag an\nemail as spam anymore). It was followed by hundreds of ML applications\nthat now quietly power hundreds of products and features that you use\nregularly, from better recommendations to voice search.\nWhere does Machine Learning start and where does it end? What exactly\ndoes it mean for a machine to learn something? If I download a copy of\nWikipedia, has my computer really learned something? Is it suddenly\nsmarter? In this chapter we will start by clarifying what Machine Learning\nis and why you may want to use it.\nThen, before we set out to explore the Machine Learning continent, we\nwill take a look at the map and learn about the main regions and the most\nnotable landmarks: supervised versus unsupervised learning, online versus\nbatch learning, instance-based versus model-based learning. Then we will\nlook at the workflow of a typical ML project, discuss the main challenges\nyou may face, and cover how to evaluate and fine-tune a Machine\nLearning system.\nThis chapter introduces a lot of fundamental concepts (and jargon) that\nevery data scientist should know by heart. It will be a high-level overview\n(it’s the only chapter without much code), all rather simple, but you should\nmake sure everything is crystal clear to you before continuing on to the\nrest of the book. So grab a coffee and let’s get started!\nTIP\nIf you already know all the Machine Learning basics, you may want to skip directly\nto Chapter 2. If you are not sure, try to answer all the questions listed at the end of\nthe chapter before moving on.\nWhat Is Machine Learning?\nMachine Learning is the science (and art) of programming computers so\nthey can learn from data.\nHere is a slightly more general definition:\n[Machine Learning is the] field of study that gives computers the ability\nto learn without being explicitly programmed.\n—Arthur Samuel, 1959\nAnd a more engineering-oriented one:\nA computer program is said to learn from experience E with respect to\nsome task T and some performance measure P, if its performance on T,\nas measured by P, improves with experience E.\n—Tom Mitchell, 1997\nYour spam filter is a Machine Learning program that, given examples of\nspam emails (e.g., flagged by users) and examples of regular (nonspam,\nalso called “ham”) emails, can learn to flag spam. The examples that the\nsystem uses to learn are called the training set. Each training example is\ncalled a training instance (or sample). In this case, the task T is to flag\nspam for new emails, the experience E is the training data, and the\nperformance measure P needs to be defined; for example, you can use the\nratio of correctly classified emails. This particular performance measure is\ncalled accuracy, and it is often used in classification tasks.\nIf you just download a copy of Wikipedia, your computer has a lot more\ndata, but it is not suddenly better at any task. Thus, downloading a copy of\nWikipedia is not Machine Learning.\nWhy Use Machine Learning?\nConsider how you would write a spam filter using traditional\nprogramming techniques (Figure 1-1):\n1. First you would consider what spam typically looks like. You\nmight notice that some words or phrases (such as “4U,” “credit\ncard,” “free,” and “amazing”) tend to come up a lot in the subject\nline. Perhaps you would also notice a few other patterns in the\nsender’s name, the email’s body, and other parts of the email.\n2. You would write a detection algorithm for each of the patterns\nthat you noticed, and your program would flag emails as spam if a\nnumber of these patterns were detected.\n3. You would test your program and repeat steps 1 and 2 until it was\ngood enough to launch.\nFigure 1-1. The traditional approach\nSince the problem is difficult, your program will likely become a long list\nof complex rules—pretty hard to maintain.\nIn contrast, a spam filter based on Machine Learning techniques\nautomatically learns which words and phrases are good predictors of spam\nby detecting unusually frequent patterns of words in the spam examples\ncompared to the ham examples (Figure 1-2). The program is much shorter,\neasier to maintain, and most likely more accurate.\nWhat if spammers notice that all their emails containing “4U” are\nblocked? They might start writing “For U” instead. A spam filter using\ntraditional programming techniques would need to be updated to flag “For\nU” emails. If spammers keep working around your spam filter, you will\nneed to keep writing new rules forever.\nIn contrast, a spam filter based on Machine Learning techniques\nautomatically notices that “For U” has become unusually frequent in spam\nflagged by users, and it starts flagging them without your intervention\n(Figure 1-3).\nFigure 1-2. The Machine Learning approach\nFigure 1-3. Automatically adapting to change\nAnother area where Machine Learning shines is for problems that either\nare too complex for traditional approaches or have no known algorithm.\nFor example, consider speech recognition. Say you want to start simple\nand write a program capable of distinguishing the words “one” and “two.”\nYou might notice that the word “two” starts with a high-pitch sound (“T”),\nso you could hardcode an algorithm that measures high-pitch sound\nintensity and use that to distinguish ones and twos —but obviously this\ntechnique will not scale to thousands of words spoken by millions of very\ndifferent people in noisy environments and in dozens of languages. The\nbest solution (at least today) is to write an algorithm that learns by itself,\ngiven many example recordings for each word.\nFinally, Machine Learning can help humans learn (Figure 1-4). ML\nalgorithms can be inspected to see what they have learned (although for\nsome algorithms this can be tricky). For instance, once a spam filter has\nbeen trained on enough spam, it can easily be inspected to reveal the list of\nwords and combinations of words that it believes are the best predictors of\nspam. Sometimes this will reveal unsuspected correlations or new trends,\nand thereby lead to a better understanding of the problem. Applying ML\ntechniques to dig into large amounts of data can help discover patterns that\nwere not immediately apparent. This is called data mining.\nFigure 1-4. Machine Learning can help humans learn\nTo summarize, Machine Learning is great for:\nProblems for which existing solutions require a lot of fine-tuning\nor long lists of rules: one Machine Learning algorithm can often\nsimplify code and perform better than the traditional approach.\nComplex problems for which using a traditional approach yields\nno good solution: the best Machine Learning techniques can\nperhaps find a solution.\nFluctuating environments: a Machine Learning system can adapt\nto new data.\nGetting insights about complex problems and large amounts of\ndata.\nExamples of Applications\nLet’s look at some concrete examples of Machine Learning tasks, along\nwith the techniques that can tackle them:\nAnalyzing images of products on a production line to automatically\nclassify them\nThis is image classification, typically performed using convolutional\nneural networks (CNNs; see Chapter 14).\nDetecting tumors in brain scans\nThis is semantic segmentation, where each pixel in the image is\nclassified (as we want to determine the exact location and shape of\ntumors), typically using CNNs as well.\nAutomatically classifying news articles\nThis is natural language processing (NLP), and more specifically text\nclassification, which can be tackled using recurrent neural networks\n(RNNs), CNNs, or Transformers (see Chapter 16).\nAutomatically flagging offensive comments on discussion forums\nThis is also text classification, using the same NLP tools.\nSummarizing long documents automatically\nThis is a branch of NLP called text summarization, again using the\nsame tools.\nCreating a chatbot or a personal assistant\nThis involves many NLP components, including natural language\nunderstanding (NLU) and question-answering modules.\nForecasting your company’s revenue next year, based on many\nperformance metrics\nThis is a regression task (i.e., predicting values) that may be tackled\nusing any regression model, such as a Linear Regression or Polynomial\nRegression model (see Chapter 4), a regression SVM (see Chapter 5), a\nregression Random Forest (see Chapter 7), or an artificial neural\nnetwork (see Chapter 10). If you want to take into account sequences\nof past performance metrics, you may want to use RNNs, CNNs, or\nTransformers (see Chapters 15 and 16).\nMaking your app react to voice commands\nThis is speech recognition, which requires processing audio samples:\nsince they are long and complex sequences, they are typically\nprocessed using RNNs, CNNs, or Transformers (see Chapters 15 and\n16).\nDetecting credit card fraud\nThis is anomaly detection (see Chapter 9).\nSegmenting clients based on their purchases so that you can design a\ndifferent marketing strategy for each segment\nThis is clustering (see Chapter 9).\nRepresenting a complex, high-dimensional dataset in a clear and insightful\ndiagram\nThis is data visualization, often involving dimensionality reduction\ntechniques (see Chapter 8).\nRecommending a product that a client may be interested in, based on past\npurchases\nThis is a recommender system. One approach is to feed past purchases\n(and other information about the client) to an artificial neural network\n(see Chapter 10), and get it to output the most likely next purchase.\nThis neural net would typically be trained on past sequences of\npurchases across all clients.\nBuilding an intelligent bot for a game\nThis is often tackled using Reinforcement Learning (RL; see\nChapter 18), which is a branch of Machine Learning that trains agents\n(such as bots) to pick the actions that will maximize their rewards over\ntime (e.g., a bot may get a reward every time the player loses some life\npoints), within a given environment (such as the game). The famous\nAlphaGo program that beat the world champion at the game of Go was\nbuilt using RL.\nThis list could go on and on, but hopefully it gives you a sense of the\nincredible breadth and complexity of the tasks that Machine Learning can\ntackle, and the types of techniques that you would use for each task.\nTypes of Machine Learning Systems\nThere are so many different types of Machine Learning systems that it is\nuseful to classify them in broad categories, based on the following criteria:\nWhether or not they are trained with human supervision\n(supervised, unsupervised, semisupervised, and Reinforcement\nLearning)\nWhether or not they can learn incrementally on the fly (online\nversus batch learning)\nWhether they work by simply comparing new data points to\nknown data points, or instead by detecting patterns in the training\ndata and building a predictive model, much like scientists do\n(instance-based versus model-based learning)\nThese criteria are not exclusive; you can combine them in any way you\nlike. For example, a state-of-the-art spam filter may learn on the fly using\na deep neural network model trained using examples of spam and ham;\nthis makes it an online, model-based, supervised learning system.\nLet’s look at each of these criteria a bit more closely.\nSupervised/Unsupervised Learning\nMachine Learning systems can be classified according to the amount and\ntype of supervision they get during training. There are four major\ncategories: supervised learning, unsupervised learning, semisupervised\nlearning, and Reinforcement Learning.\nSupervised learning\nIn supervised learning, the training set you feed to the algorithm includes\nthe desired solutions, called labels (Figure 1-5).\nFigure 1-5. A labeled training set for spam classification (an example of supervised learning)\nA typical supervised learning task is classification. The spam filter is a\ngood example of this: it is trained with many example emails along with\ntheir class (spam or ham), and it must learn how to classify new emails.\nAnother typical task is to predict a target numeric value, such as the price\nof a car, given a set of features (mileage, age, brand, etc.) called\npredictors. This sort of task is called regression (Figure 1-6). To train the\nsystem, you need to give it many examples of cars, including both their\npredictors and their labels (i.e., their prices).\nNOTE\nIn Machine Learning an attribute is a data type (e.g., “mileage”), while a feature has\nseveral meanings, depending on the context, but generally means an attribute plus its\nvalue (e.g., “mileage = 15,000”). Many people use the words attribute and feature\ninterchangeably.\nNote that some regression algorithms can be used for classification as\nwell, and vice versa. For example, Logistic Regression is commonly used\nfor classification, as it can output a value that corresponds to the\nprobability of belonging to a given class (e.g., 20% chance of being spam).\nFigure 1-6. A regression problem: predict a value, given an input feature (there are usually\nmultiple input features, and sometimes multiple output values)\nHere are some of the most important supervised learning algorithms\n(covered in this book):\nk-Nearest Neighbors\n1\nLinear Regression\nLogistic Regression\nSupport Vector Machines (SVMs)\nDecision Trees and Random Forests\nNeural networks\nUnsupervised learning\nIn unsupervised learning, as you might guess, the training data is\nunlabeled (Figure 1-7). The system tries to learn without a teacher.\nFigure 1-7. An unlabeled training set for unsupervised learning\nHere are some of the most important unsupervised learning algorithms\n(most of these are covered in Chapters 8 and 9):\nClustering\nK-Means\nDBSCAN\nHierarchical Cluster Analysis (HCA)\nAnomaly detection and novelty detection\n2\nOne-class SVM\nIsolation Forest\nVisualization and dimensionality reduction\nPrincipal Component Analysis (PCA)\nKernel PCA\nLocally Linear Embedding (LLE)\nt-Distributed Stochastic Neighbor Embedding (t-SNE)\nAssociation rule learning\nApriori\nEclat\nFor example, say you have a lot of data about your blog’s visitors. You\nmay want to run a clustering algorithm to try to detect groups of similar\nvisitors (Figure 1-8). At no point do you tell the algorithm which group a\nvisitor belongs to: it finds those connections without your help. For\nexample, it might notice that 40% of your visitors are males who love\ncomic books and generally read your blog in the evening, while 20% are\nyoung sci-fi lovers who visit during the weekends. If you use a\nhierarchical clustering algorithm, it may also subdivide each group into\nsmaller groups. This may help you target your posts for each group.\nFigure 1-8. Clustering\nVisualization algorithms are also good examples of unsupervised learning\nalgorithms: you feed them a lot of complex and unlabeled data, and they\noutput a 2D or 3D representation of your data that can easily be plotted\n(Figure 1-9). These algorithms try to preserve as much structure as they\ncan (e.g., trying to keep separate clusters in the input space from\noverlapping in the visualization) so that you can understand how the data\nis organized and perhaps identify unsuspected patterns.\nFigure 1-9. Example of a t-SNE visualization highlighting semantic clusters\nA related task is dimensionality reduction, in which the goal is to simplify\nthe data without losing too much information. One way to do this is to\nmerge several correlated features into one. For example, a car’s mileage\nmay be strongly correlated with its age, so the dimensionality reduction\nalgorithm will merge them into one feature that represents the car’s wear\nand tear. This is called feature extraction.\nTIP\nIt is often a good idea to try to reduce the dimension of your training data using a\ndimensionality reduction algorithm before you feed it to another Machine Learning\nalgorithm (such as a supervised learning algorithm). It will run much faster, the data\nwill take up less disk and memory space, and in some cases it may also perform\nbetter.\nYet another important unsupervised task is anomaly detection—for\nexample, detecting unusual credit card transactions to prevent fraud,\n3\ncatching manufacturing defects, or automatically removing outliers from a\ndataset before feeding it to another learning algorithm. The system is\nshown mostly normal instances during training, so it learns to recognize\nthem; then, when it sees a new instance, it can tell whether it looks like a\nnormal one or whether it is likely an anomaly (see Figure 1-10). A very\nsimilar task is novelty detection: it aims to detect new instances that look\ndifferent from all instances in the training set. This requires having a very\n“clean” training set, devoid of any instance that you would like the\nalgorithm to detect. For example, if you have thousands of pictures of\ndogs, and 1% of these pictures represent Chihuahuas, then a novelty\ndetection algorithm should not treat new pictures of Chihuahuas as\nnovelties. On the other hand, anomaly detection algorithms may consider\nthese dogs as so rare and so different from other dogs that they would\nlikely classify them as anomalies (no offense to Chihuahuas).\nFigure 1-10. Anomaly detection\nFinally, another common unsupervised task is association rule learning, in\nwhich the goal is to dig into large amounts of data and discover interesting\nrelations between attributes. For example, suppose you own a\nsupermarket. Running an association rule on your sales logs may reveal\nthat people who purchase barbecue sauce and potato chips also tend to buy\nsteak. Thus, you may want to place these items close to one another.\nSemisupervised learning\nSince labeling data is usually time-consuming and costly, you will often\nhave plenty of unlabeled instances, and few labeled instances. Some\nalgorithms can deal with data that’s partially labeled. This is called\nsemisupervised learning (Figure 1-11).\nFigure 1-11. Semisupervised learning with two classes (triangles and squares): the unlabeled\nexamples (circles) help classify a new instance (the cross) into the triangle class rather than the\nsquare class, even though it is closer to the labeled squares\nSome photo-hosting services, such as Google Photos, are good examples\nof this. Once you upload all your family photos to the service, it\nautomatically recognizes that the same person A shows up in photos 1, 5,\nand 11, while another person B shows up in photos 2, 5, and 7. This is the\nunsupervised part of the algorithm (clustering). Now all the system needs\nis for you to tell it who these people are. Just add one label per person and\nit is able to name everyone in every photo, which is useful for searching\nphotos.\nMost semisupervised learning algorithms are combinations of\nunsupervised and supervised algorithms. For example, deep belief\nnetworks (DBNs) are based on unsupervised components called restricted\nBoltzmann machines (RBMs) stacked on top of one another. RBMs are\n4\ntrained sequentially in an unsupervised manner, and then the whole system\nis fine-tuned using supervised learning techniques.\nReinforcement Learning\nReinforcement Learning isag a very different beast. The learning system,\ncalled an agent in this context, can observe the environment, select and\nperform actions, and get rewards in return (or penalties in the form of\nnegative rewards, as shown in Figure 1-12). It must then learn by itself\nwhat is the best strategy, called a policy, to get the most reward over time.\nA policy defines what action the agent should choose when it is in a given\nsituation.\nFigure 1-12. Reinforcement Learning\nFor example, many robots implement Reinforcement Learning algorithms\nto learn how to walk. DeepMind’s AlphaGo program is also a good\nexample of Reinforcement Learning: it made the headlines in May 2017\nwhen it beat the world champion Ke Jie at the game of Go. It learned its\nwinning policy by analyzing millions of games, and then playing many\ngames against itself. Note that learning was turned off during the games\nagainst the champion; AlphaGo was just applying the policy it had\nlearned.\nBatch and Online Learning\nAnother criterion used to classify Machine Learning systems is whether or\nnot the system can learn incrementally from a stream of incoming data.\nBatch learning\nIn batch learning, the system is incapable of learning incrementally: it\nmust be trained using all the available data. This will generally take a lot\nof time and computing resources, so it is typically done offline. First the\nsystem is trained, and then it is launched into production and runs without\nlearning anymore; it just applies what it has learned. This is called offline\nlearning.\nIf you want a batch learning system to know about new data (such as a new\ntype of spam), you need to train a new version of the system from scratch\non the full dataset (not just the new data, but also the old data), then stop\nthe old system and replace it with the new one.\nFortunately, the whole process of training, evaluating, and launching a\nMachine Learning system can be automated fairly easily (as shown in\nFigure 1-3), so even a batch learning system can adapt to change. Simply\nupdate the data and train a new version of the system from scratch as often\nas needed.\nThis solution is simple and often works fine, but training using the full set\nof data can take many hours, so you would typically train a new system\nonly every 24 hours or even just weekly. If your system needs to adapt to\nrapidly changing data (e.g., to predict stock prices), then you need a more\nreactive solution.\nAlso, training on the full set of data requires a lot of computing resources\n(CPU, memory space, disk space, disk I/O, network I/O, etc.). If you have\na lot of data and you automate your system to train from scratch every day,\nit will end up costing you a lot of money. If the amount of data is huge, it\nmay even be impossible to use a batch learning algorithm.\nFinally, if your system needs to be able to learn autonomously and it has\nlimited resources (e.g., a smartphone application or a rover on Mars), then\ncarrying around large amounts of training data and taking up a lot of\nresources to train for hours every day is a showstopper.\nFortunately, a better option in all these cases is to use algorithms that are\ncapable of learning incrementally.\nOnline learning\nIn online learning, you train the system incrementally by feeding it data\ninstances sequentially, either individually or in small groups called mini-\nbatches. Each learning step is fast and cheap, so the system can learn\nabout new data on the fly, as it arrives (see Figure 1-13).\nFigure 1-13. In online learning, a model is trained and launched into production, and then it\nkeeps learning as new data comes in\nOnline learning is great for systems that receive data as a continuous flow\n(e.g., stock prices) and need to adapt to change rapidly or autonomously. It\nis also a good option if you have limited computing resources: once an\nonline learning system has learned about new data instances, it does not\nneed them anymore, so you can discard them (unless you want to be able\nto roll back to a previous state and “replay” the data). This can save a huge\namount of space.\nOnline learning algorithms can also be used to train systems on huge\ndatasets that cannot fit in one machine’s main memory (this is called out-\nof-core learning). The algorithm loads part of the data, runs a training step\non that data, and repeats the process until it has run on all of the data (see\nFigure 1-14).\nWARNING\nOut-of-core learning is usually done offline (i.e., not on the live system), so online\nlearning can be a confusing name. Think of it as incremental learning.\nOne important parameter of online learning systems is how fast they\nshould adapt to changing data: this is called the learning rate. If you set a\nhigh learning rate, then your system will rapidly adapt to new data, but it\nwill also tend to quickly forget the old data (you don’t want a spam filter\nto flag only the latest kinds of spam it was shown). Conversely, if you set a\nlow learning rate, the system will have more inertia; that is, it will learn\nmore slowly, but it will also be less sensitive to noise in the new data or to\nsequences of nonrepresentative data points (outliers).\nFigure 1-14. Using online learning to handle huge datasets\nA big challenge with online learning is that if bad data is fed to the\nsystem, the system’s performance will gradually decline. If it’s a live\nsystem, your clients will notice. For example, bad data could come from a\nmalfunctioning sensor on a robot, or from someone spamming a search\nengine to try to rank high in search results. To reduce this risk, you need to\nmonitor your system closely and promptly switch learning off (and\npossibly revert to a previously working state) if you detect a drop in\nperformance. You may also want to monitor the input data and react to\nabnormal data (e.g., using an anomaly detection algorithm).\nInstance-Based Versus Model-Based Learning\nOne more way to categorize Machine Learning systems is by how they\ngeneralize. Most Machine Learning tasks are about making predictions.\nThis means that given a number of training examples, the system needs to\nbe able to make good predictions for (generalize to) examples it has never\nseen before. Having a good performance measure on the training data is\ngood, but insufficient; the true goal is to perform well on new instances.\nThere are two main approaches to generalization: instance-based learning\nand model-based learning.\nInstance-based learning\nPossibly the most trivial form of learning is simply to learn by heart. If\nyou were to create a spam filter this way, it would just flag all emails that\nare identical to emails that have already been flagged by users—not the\nworst solution, but certainly not the best.\nInstead of just flagging emails that are identical to known spam emails,\nyour spam filter could be programmed to also flag emails that are very\nsimilar to known spam emails. This requires a measure of similarity\nbetween two emails. A (very basic) similarity measure between two\nemails could be to count the number of words they have in common. The\nsystem would flag an email as spam if it has many words in common with\na known spam email.\nThis is called instance-based learning: the system learns the examples by\nheart, then generalizes to new cases by using a similarity measure to\ncompare them to the learned examples (or a subset of them). For example,\nin Figure 1-15 the new instance would be classified as a triangle because\nthe majority of the most similar instances belong to that class.\nFigure 1-15. Instance-based learning\nModel-based learning\nAnother way to generalize from a set of examples is to build a model of\nthese examples and then use that model to make predictions. This is called\nmodel-based learning (Figure 1-16).\nFigure 1-16. Model-based learning\nFor example, suppose you want to know if money makes people happy, so\nyou download the Better Life Index data from the OECD’s website and\nstats about gross domestic product (GDP) per capita from the IMF’s\nwebsite. Then you join the tables and sort by GDP per capita. Table 1-1\nshows an excerpt of what you get.\nTable 1-1. Does money make people happier?\nCountry\nGDP per capita (USD) Life satisfaction\nHungary\n12,240\n4.9\nKorea\n27,195\n5.8\nFrance\n37,675\n6.5\nAustralia\n50,962\n7.3\nUnited States 55,805\n7.2\nLet’s plot the data for these countries (Figure 1-17).\nFigure 1-17. Do you see a trend here?\nThere does seem to be a trend here! Although the data is noisy (i.e., partly\nrandom), it looks like life satisfaction goes up more or less linearly as the\ncountry’s GDP per capita increases. So you decide to model life\nsatisfaction as a linear function of GDP per capita. This step is called\nmodel selection: you selected a linear model of life satisfaction with just\none attribute, GDP per capita (Equation 1-1).\nEquation 1-1. A simple linear model\nlife_satisfaction = θ0 + θ1 × GDP_per_capita\nThis model has two model parameters, θ and θ . By tweaking these\nparameters, you can make your model represent any linear function, as\nshown in Figure 1-18.\n0\n1 5\nFigure 1-18. A few possible linear models\nBefore you can use your model, you need to define the parameter values θ\nand θ . How can you know which values will make your model perform\nbest? To answer this question, you need to specify a performance measure.\nYou can either define a utility function (or fitness function) that measures\nhow good your model is, or you can define a cost function that measures\nhow bad it is. For Linear Regression problems, people typically use a cost\nfunction that measures the distance between the linear model’s predictions\nand the training examples; the objective is to minimize this distance.\nThis is where the Linear Regression algorithm comes in: you feed it your\ntraining examples, and it finds the parameters that make the linear model\nfit best to your data. This is called training the model. In our case, the\nalgorithm finds that the optimal parameter values are θ = 4.85 and θ =\n4.91 × 10\n.\n0\n1\n0\n1\n–5\nWARNING\nConfusingly, the same word “model” can refer to a type of model (e.g., Linear\nRegression), to a fully specified model architecture (e.g., Linear Regression with one\ninput and one output), or to the final trained model ready to be used for predictions\n(e.g., Linear Regression with one input and one output, using θ = 4.85 and θ = 4.91\n× 10\n). Model selection consists in choosing the type of model and fully specifying\nits architecture. Training a model means running an algorithm to find the model\nparameters that will make it best fit the training data (and hopefully make good\npredictions on new data).\nNow the model fits the training data as closely as possible (for a linear\nmodel), as you can see in Figure 1-19.\nFigure 1-19. The linear model that fits the training data best\nYou are finally ready to run the model to make predictions. For example,\nsay you want to know how happy Cypriots are, and the OECD data does\nnot have the answer. Fortunately, you can use your model to make a good\nprediction: you look up Cyprus’s GDP per capita, find $22,587, and then\napply your model and find that life satisfaction is likely to be somewhere\naround 4.85 + 22,587 × 4.91 × 10 = 5.96.\n0\n1\n–5\n-5\nTo whet your appetite, Example 1-1 shows the Python code that loads the\ndata, prepares it, creates a scatterplot for visualization, and then trains a\nlinear model and makes a prediction.\nExample 1-1. Training and running a linear model using Scikit-Learn\nimport matplotlib.pyplot as plt \nimport numpy as np \nimport pandas as pd \nimport sklearn.linear_model \n \n# Load the data \noecd_bli = pd.read_csv(\"oecd_bli_2015.csv\", thousands=',') \ngdp_per_capita = \npd.read_csv(\"gdp_per_capita.csv\",thousands=',',delimiter='\\t', \n encoding='latin1', na_values=\"n/a\") \n \n \n# Prepare the data \ncountry_stats = prepare_country_stats(oecd_bli, gdp_per_capita) \nX = np.c_[country_stats[\"GDP per capita\"]] \ny = np.c_[country_stats[\"Life satisfaction\"]] \n \n# Visualize the data \ncountry_stats.plot(kind='scatter', x=\"GDP per capita\", y='Life \nsatisfaction') \nplt.show() \n \n# Select a linear model \nmodel = sklearn.linear_model.LinearRegression() \n \n# Train the model \nmodel.fit(X, y) \n \n# Make a prediction for Cyprus \nX_new = [[22587]] # Cyprus's GDP per capita \nprint(model.predict(X_new)) # outputs [[ 5.96242338]]\n6\n7\nNOTE\nIf you had used an instance-based learning algorithm instead, you would have found\nthat Slovenia has the closest GDP per capita to that of Cyprus ($20,732), and since\nthe OECD data tells us that Slovenians’ life satisfaction is 5.7, you would have\npredicted a life satisfaction of 5.7 for Cyprus. If you zoom out a bit and look at the\ntwo next-closest countries, you will find Portugal and Spain with life satisfactions of\n5.1 and 6.5, respectively. Averaging these three values, you get 5.77, which is pretty\nclose to your model-based prediction. This simple algorithm is called k-Nearest\nNeighbors regression (in this example, k = 3).\nReplacing the Linear Regression model with k-Nearest Neighbors regression in the\nprevious code is as simple as replacing these two lines:\nimport sklearn.linear_model \nmodel = sklearn.linear_model.LinearRegression()\nwith these two:\nimport sklearn.neighbors \nmodel = sklearn.neighbors.KNeighborsRegressor( \n n_neighbors=3)\nIf all went well, your model will make good predictions. If not, you may\nneed to use more attributes (employment rate, health, air pollution, etc.),\nget more or better-quality training data, or perhaps select a more powerful\nmodel (e.g., a Polynomial Regression model).\nIn summary:\nYou studied the data.\nYou selected a model.\nYou trained it on the training data (i.e., the learning algorithm\nsearched for the model parameter values that minimize a cost\nfunction).\nFinally, you applied the model to make predictions on new cases\n(this is called inference), hoping that this model will generalize\nwell.\nThis is what a typical Machine Learning project looks like. In Chapter 2\nyou will experience this firsthand by going through a project end to end.\nWe have covered a lot of ground so far: you now know what Machine\nLearning is really about, why it is useful, what some of the most common\ncategories of ML systems are, and what a typical project workflow looks\nlike. Now let’s look at what can go wrong in learning and prevent you from\nmaking accurate predictions.\nMain Challenges of Machine Learning\nIn short, since your main task is to select a learning algorithm and train it\non some data, the two things that can go wrong are “bad algorithm” and\n“bad data.” Let’s start with examples of bad data.\nInsufficient Quantity of Training Data\nFor a toddler to learn what an apple is, all it takes is for you to point to an\napple and say “apple” (possibly repeating this procedure a few times).\nNow the child is able to recognize apples in all sorts of colors and shapes.\nGenius.\nMachine Learning is not quite there yet; it takes a lot of data for most\nMachine Learning algorithms to work properly. Even for very simple\nproblems you typically need thousands of examples, and for complex\nproblems such as image or speech recognition you may need millions of\nexamples (unless you can reuse parts of an existing model).\nTHE UNREASONABLE EFFECTIVENESS OF DATA\nIn a famous paper published in 2001, Microsoft researchers Michele\nBanko and Eric Brill showed that very different Machine Learning\nalgorithms, including fairly simple ones, performed almost identically\nwell on a complex problem of natural language disambiguation once\nthey were given enough data (as you can see in Figure 1-20).\nFigure 1-20. The importance of data versus algorithms\nAs the authors put it, “these results suggest that we may want to\nreconsider the trade-off between spending time and money on\nalgorithm development versus spending it on corpus development.”\n8\n9\nThe idea that data matters more than algorithms for complex problems\nwas further popularized by Peter Norvig et al. in a paper titled “The\nUnreasonable Effectiveness of Data”, published in 2009.\n It should be\nnoted, however, that small- and medium-sized datasets are still very\ncommon, and it is not always easy or cheap to get extra training data —\nso don’t abandon algorithms just yet.\nNonrepresentative Training Data\nIn order to generalize well, it is crucial that your training data be\nrepresentative of the new cases you want to generalize to. This is true\nwhether you use instance-based learning or model-based learning.\nFor example, the set of countries we used earlier for training the linear\nmodel was not perfectly representative; a few countries were missing.\nFigure 1-21 shows what the data looks like when you add the missing\ncountries.\nFigure 1-21. A more representative training sample\nIf you train a linear model on this data, you get the solid line, while the old\nmodel is represented by the dotted line. As you can see, not only does\nadding a few missing countries significantly alter the model, but it makes\nit clear that such a simple linear model is probably never going to work\nwell. It seems that very rich countries are not happier than moderately rich\ncountries (in fact, they seem unhappier), and conversely some poor\ncountries seem happier than many rich countries.\n10\nBy using a nonrepresentative training set, we trained a model that is\nunlikely to make accurate predictions, especially for very poor and very\nrich countries.\nIt is crucial to use a training set that is representative of the cases you\nwant to generalize to. This is often harder than it sounds: if the sample is\ntoo small, you will have sampling noise (i.e., nonrepresentative data as a\nresult of chance), but even very large samples can be nonrepresentative if\nthe sampling method is flawed. This is called sampling bias.\nEXAMPLES OF SAMPLING BIAS\nPerhaps the most famous example of sampling bias happened during\nthe US presidential election in 1936, which pitted Landon against\nRoosevelt: the Literary Digest conducted a very large poll, sending\nmail to about 10 million people. It got 2.4 million answers, and\npredicted with high confidence that Landon would get 57% of the\nvotes. Instead, Roosevelt won with 62% of the votes. The flaw was in\nthe Literary Digest’s sampling method:\nFirst, to obtain the addresses to send the polls to, the Literary\nDigest used telephone directories, lists of magazine\nsubscribers, club membership lists, and the like. All of these\nlists tended to favor wealthier people, who were more likely\nto vote Republican (hence Landon).\nSecond, less than 25% of the people who were polled\nanswered. Again this introduced a sampling bias, by\npotentially ruling out people who didn’t care much about\npolitics, people who didn’t like the Literary Digest, and other\nkey groups. This is a special type of sampling bias called\nnonresponse bias.\nHere is another example: say you want to build a system to recognize\nfunk music videos. One way to build your training set is to search for\n“funk music” on YouTube and use the resulting videos. But this\nassumes that YouTube’s search engine returns a set of videos that are\nrepresentative of all the funk music videos on YouTube. In reality, the\nsearch results are likely to be biased toward popular artists (and if you\nlive in Brazil you will get a lot of “funk carioca” videos, which sound\nnothing like James Brown). On the other hand, how else can you get a\nlarge training set?\nPoor-Quality Data\nObviously, if your training data is full of errors, outliers, and noise (e.g.,\ndue to poor-quality measurements), it will make it harder for the system to\ndetect the underlying patterns, so your system is less likely to perform\nwell. It is often well worth the effort to spend time cleaning up your\ntraining data. The truth is, most data scientists spend a significant part of\ntheir time doing just that. The following are a couple of examples of when\nyou’d want to clean up training data:\nIf some instances are clearly outliers, it may help to simply\ndiscard them or try to fix the errors manually.\nIf some instances are missing a few features (e.g., 5% of your\ncustomers did not specify their age), you must decide whether you\nwant to ignore this attribute altogether, ignore these instances, fill\nin the missing values (e.g., with the median age), or train one\nmodel with the feature and one model without it.\nIrrelevant Features\nAs the saying goes: garbage in, garbage out. Your system will only be\ncapable of learning if the training data contains enough relevant features\nand not too many irrelevant ones. A critical part of the success of a\nMachine Learning project is coming up with a good set of features to train\non. This process, called feature engineering, involves the following steps:\nFeature selection (selecting the most useful features to train on\namong existing features)\nFeature extraction (combining existing features to produce a\nmore useful one —as we saw earlier, dimensionality reduction\nalgorithms can help)\nCreating new features by gathering new data\nNow that we have looked at many examples of bad data, let’s look at a\ncouple of examples of bad algorithms.\nOverfitting the Training Data\nSay you are visiting a foreign country and the taxi driver rips you off. You\nmight be tempted to say that all taxi drivers in that country are thieves.\nOvergeneralizing is something that we humans do all too often, and\nunfortunately machines can fall into the same trap if we are not careful. In\nMachine Learning this is called overfitting: it means that the model\nperforms well on the training data, but it does not generalize well.\nFigure 1-22 shows an example of a high-degree polynomial life\nsatisfaction model that strongly overfits the training data. Even though it\nperforms much better on the training data than the simple linear model,\nwould you really trust its predictions?\nFigure 1-22. Overfitting the training data\nComplex models such as deep neural networks can detect subtle patterns\nin the data, but if the training set is noisy, or if it is too small (which\nintroduces sampling noise), then the model is likely to detect patterns in\nthe noise itself. Obviously these patterns will not generalize to new\ninstances. For example, say you feed your life satisfaction model many\nmore attributes, including uninformative ones such as the country’s name.\nIn that case, a complex model may detect patterns like the fact that all\ncountries in the training data with a w in their name have a life satisfaction\ngreater than 7: New Zealand (7.3), Norway (7.4), Sweden (7.2), and\nSwitzerland (7.5). How confident are you that the w-satisfaction rule\ngeneralizes to Rwanda or Zimbabwe? Obviously this pattern occurred in\nthe training data by pure chance, but the model has no way to tell whether\na pattern is real or simply the result of noise in the data.\nWARNING\nOverfitting happens when the model is too complex relative to the amount and\nnoisiness of the training data. Here are possible solutions:\nSimplify the model by selecting one with fewer parameters (e.g., a linear\nmodel rather than a high-degree polynomial model), by reducing the\nnumber of attributes in the training data, or by constraining the model.\nGather more training data.\nReduce the noise in the training data (e.g., fix data errors and remove\noutliers).\nConstraining a model to make it simpler and reduce the risk of overfitting\nis called regularization. For example, the linear model we defined earlier\nhas two parameters, θ and θ . This gives the learning algorithm two\ndegrees of freedom to adapt the model to the training data: it can tweak\nboth the height (θ ) and the slope (θ ) of the line. If we forced θ = 0, the\nalgorithm would have only one degree of freedom and would have a much\nharder time fitting the data properly: all it could do is move the line up or\ndown to get as close as possible to the training instances, so it would end\nup around the mean. A very simple model indeed! If we allow the\nalgorithm to modify θ but we force it to keep it small, then the learning\nalgorithm will effectively have somewhere in between one and two\ndegrees of freedom. It will produce a model that’s simpler than one with\ntwo degrees of freedom, but more complex than one with just one. You\nwant to find the right balance between fitting the training data perfectly\nand keeping the model simple enough to ensure that it will generalize\nwell.\nFigure 1-23 shows three models. The dotted line represents the original\nmodel that was trained on the countries represented as circles (without the\ncountries represented as squares), the dashed line is our second model\n0\n1\n0\n1\n1\n1\ntrained with all countries (circles and squares), and the solid line is a\nmodel trained with the same data as the first model but with a\nregularization constraint. You can see that regularization forced the model\nto have a smaller slope: this model does not fit the training data (circles)\nas well as the first model, but it actually generalizes better to new\nexamples that it did not see during training (squares).\nFigure 1-23. Regularization reduces the risk of overfitting\nThe amount of regularization to apply during learning can be controlled by\na hyperparameter. A hyperparameter is a parameter of a learning\nalgorithm (not of the model). As such, it is not affected by the learning\nalgorithm itself; it must be set prior to training and remains constant\nduring training. If you set the regularization hyperparameter to a very\nlarge value, you will get an almost flat model (a slope close to zero); the\nlearning algorithm will almost certainly not overfit the training data, but it\nwill be less likely to find a good solution. Tuning hyperparameters is an\nimportant part of building a Machine Learning system (you will see a\ndetailed example in the next chapter).\nUnderfitting the Training Data\nAs you might guess, underfitting is the opposite of overfitting: it occurs\nwhen your model is too simple to learn the underlying structure of the\ndata. For example, a linear model of life satisfaction is prone to underfit;\nreality is just more complex than the model, so its predictions are bound to\nbe inaccurate, even on the training examples.\nHere are the main options for fixing this problem:\nSelect a more powerful model, with more parameters.\nFeed better features to the learning algorithm (feature\nengineering).\nReduce the constraints on the model (e.g., reduce the\nregularization hyperparameter).\nStepping Back\nBy now you know a lot about Machine Learning. However, we went\nthrough so many concepts that you may be feeling a little lost, so let’s step\nback and look at the big picture:\nMachine Learning is about making machines get better at some\ntask by learning from data, instead of having to explicitly code\nrules.\nThere are many different types of ML systems: supervised or not,\nbatch or online, instance-based or model-based.\nIn an ML project you gather data in a training set, and you feed\nthe training set to a learning algorithm. If the algorithm is model-\nbased, it tunes some parameters to fit the model to the training set\n(i.e., to make good predictions on the training set itself), and then\nhopefully it will be able to make good predictions on new cases as\nwell. If the algorithm is instance-based, it just learns the\nexamples by heart and generalizes to new instances by using a\nsimilarity measure to compare them to the learned instances.\nThe system will not perform well if your training set is too small,\nor if the data is not representative, is noisy, or is polluted with\nirrelevant features (garbage in, garbage out). Lastly, your model\nneeds to be neither too simple (in which case it will underfit) nor\ntoo complex (in which case it will overfit).\nThere’s just one last important topic to cover: once you have trained a\nmodel, you don’t want to just “hope” it generalizes to new cases. You want\nto evaluate it and fine-tune it if necessary. Let’s see how to do that.\nTesting and Validating\nThe only way to know how well a model will generalize to new cases is to\nactually try it out on new cases. One way to do that is to put your model in\nproduction and monitor how well it performs. This works well, but if your\nmodel is horribly bad, your users will complain—not the best idea.\nA better option is to split your data into two sets: the training set and the\ntest set. As these names imply, you train your model using the training set,\nand you test it using the test set. The error rate on new cases is called the\ngeneralization error (or out-of-sample error), and by evaluating your\nmodel on the test set, you get an estimate of this error. This value tells you\nhow well your model will perform on instances it has never seen before.\nIf the training error is low (i.e., your model makes few mistakes on the\ntraining set) but the generalization error is high, it means that your model\nis overfitting the training data.\nTIP\nIt is common to use 80% of the data for training and hold out 20% for testing.\nHowever, this depends on the size of the dataset: if it contains 10 million instances,\nthen holding out 1% means your test set will contain 100,000 instances, probably\nmore than enough to get a good estimate of the generalization error.\nHyperparameter Tuning and Model Selection\nEvaluating a model is simple enough: just use a test set. But suppose you\nare hesitating between two types of models (say, a linear model and a\npolynomial model): how can you decide between them? One option is to\ntrain both and compare how well they generalize using the test set.\nNow suppose that the linear model generalizes better, but you want to\napply some regularization to avoid overfitting. The question is, how do\nyou choose the value of the regularization hyperparameter? One option is\nto train 100 different models using 100 different values for this\nhyperparameter. Suppose you find the best hyperparameter value that\nproduces a model with the lowest generalization error —say, just 5% error.\nYou launch this model into production, but unfortunately it does not\nperform as well as expected and produces 15% errors. What just\nhappened?\nThe problem is that you measured the generalization error multiple times\non the test set, and you adapted the model and hyperparameters to produce\nthe best model for that particular set. This means that the model is\nunlikely to perform as well on new data.\nA common solution to this problem is called holdout validation: you\nsimply hold out part of the training set to evaluate several candidate\nmodels and select the best one. The new held-out set is called the\nvalidation set (or sometimes the development set, or dev set). More\nspecifically, you train multiple models with various hyperparameters on\nthe reduced training set (i.e., the full training set minus the validation set),\nand you select the model that performs best on the validation set. After\nthis holdout validation process, you train the best model on the full\ntraining set (including the validation set), and this gives you the final\nmodel. Lastly, you evaluate this final model on the test set to get an\nestimate of the generalization error.\nThis solution usually works quite well. However, if the validation set is\ntoo small, then model evaluations will be imprecise: you may end up\nselecting a suboptimal model by mistake. Conversely, if the validation set\nis too large, then the remaining training set will be much smaller than the\nfull training set. Why is this bad? Well, since the final model will be\ntrained on the full training set, it is not ideal to compare candidate models\ntrained on a much smaller training set. It would be like selecting the\nfastest sprinter to participate in a marathon. One way to solve this problem\nis to perform repeated cross-validation, using many small validation sets.\nEach model is evaluated once per validation set after it is trained on the\nrest of the data. By averaging out all the evaluations of a model, you get a\nmuch more accurate measure of its performance. There is a drawback,\nhowever: the training time is multiplied by the number of validation sets.\nData Mismatch\nIn some cases, it’s easy to get a large amount of data for training, but this\ndata probably won’t be perfectly representative of the data that will be\nused in production. For example, suppose you want to create a mobile app\nto take pictures of flowers and automatically determine their species. You\ncan easily download millions of pictures of flowers on the web, but they\nwon’t be perfectly representative of the pictures that will actually be taken\nusing the app on a mobile device. Perhaps you only have 10,000\nrepresentative pictures (i.e., actually taken with the app). In this case, the\nmost important rule to remember is that the validation set and the test set\nmust be as representative as possible of the data you expect to use in\nproduction, so they should be composed exclusively of representative\npictures: you can shuffle them and put half in the validation set and half in\nthe test set (making sure that no duplicates or near-duplicates end up in\nboth sets). But after training your model on the web pictures, if you\nobserve that the performance of the model on the validation set is\ndisappointing, you will not know whether this is because your model has\noverfit the training set, or whether this is just due to the mismatch between\nthe web pictures and the mobile app pictures. One solution is to hold out\nsome of the training pictures (from the web) in yet another set that\nAndrew Ng calls the train-dev set. After the model is trained (on the\ntraining set, not on the train-dev set), you can evaluate it on the train-dev\nset. If it performs well, then the model is not overfitting the training set. If\nit performs poorly on the validation set, the problem must be coming from\nthe data mismatch. You can try to tackle this problem by preprocessing the\nweb images to make them look more like the pictures that will be taken by\nthe mobile app, and then retraining the model. Conversely, if the model\nperforms poorly on the train-dev set, then it must have overfit the training\nset, so you should try to simplify or regularize the model, get more\ntraining data, and clean up the training data.\nNO FREE LUNCH THEOREM\nA model is a simplified version of the observations. The\nsimplifications are meant to discard the superfluous details that are\nunlikely to generalize to new instances. To decide what data to discard\nand what data to keep, you must make assumptions. For example, a\nlinear model makes the assumption that the data is fundamentally\nlinear and that the distance between the instances and the straight line\nis just noise, which can safely be ignored.\nIn a famous 1996 paper, David Wolpert demonstrated that if you\nmake absolutely no assumption about the data, then there is no reason\nto prefer one model over any other. This is called the No Free Lunch\n(NFL) theorem. For some datasets the best model is a linear model,\nwhile for other datasets it is a neural network. There is no model that\nis a priori guaranteed to work better (hence the name of the theorem).\nThe only way to know for sure which model is best is to evaluate them\nall. Since this is not possible, in practice you make some reasonable\nassumptions about the data and evaluate only a few reasonable models.\nFor example, for simple tasks you may evaluate linear models with\nvarious levels of regularization, and for a complex problem you may\nevaluate various neural networks.\nExercises\nIn this chapter we have covered some of the most important concepts in\nMachine Learning. In the next chapters we will dive deeper and write\nmore code, but before we do, make sure you know how to answer the\nfollowing questions:\n1. How would you define Machine Learning?\n11\n2. Can you name four types of problems where it shines?\n3. What is a labeled training set?\n4. What are the two most common supervised tasks?\n5. Can you name four common unsupervised tasks?\n6. What type of Machine Learning algorithm would you use to allow\na robot to walk in various unknown terrains?\n7. What type of algorithm would you use to segment your customers\ninto multiple groups?\n8. Would you frame the problem of spam detection as a supervised\nlearning problem or an unsupervised learning problem?\n9. What is an online learning system?\n10. What is out-of-core learning?\n11. What type of learning algorithm relies on a similarity measure to\nmake predictions?\n12. What is the difference between a model parameter and a learning\nalgorithm’s hyperparameter?\n13. What do model-based learning algorithms search for? What is the\nmost common strategy they use to succeed? How do they make\npredictions?\n14. Can you name four of the main challenges in Machine Learning?\n15. If your model performs great on the training data but generalizes\npoorly to new instances, what is happening? Can you name three\npossible solutions?\n16. What is a test set, and why would you want to use it?\n17. What is the purpose of a validation set?\n18. What is the train-dev set, when do you need it, and how do you\nuse it?\n19. What can go wrong if you tune hyperparameters using the test\nset?\nSolutions to these exercises are available in Appendix A.\n1 Fun fact: this odd-sounding name is a statistics term introduced by Francis Galton while he\nwas studying the fact that the children of tall people tend to be shorter than their parents.\nSince the children were shorter, he called this regression to the mean. This name was then\napplied to the methods he used to analyze correlations between variables.\n2 Some neural network architectures can be unsupervised, such as autoencoders and\nrestricted Boltzmann machines. They can also be semisupervised, such as in deep belief\nnetworks and unsupervised pretraining.\n3 Notice how animals are rather well separated from vehicles and how horses are close to\ndeer but far from birds. Figure reproduced with permission from Richard Socher et al.,\n“Zero-Shot Learning Through Cross-Modal Transfer,” Proceedings of the 26th\nInternational Conference on Neural Information Processing Systems 1 (2013): 935–943.\n4 That’s when the system works perfectly. In practice it often creates a few clusters per\nperson, and sometimes mixes up two people who look alike, so you may need to provide a\nfew labels per person and manually clean up some clusters.\n5 By convention, the Greek letter θ (theta) is frequently used to represent model parameters.\n6 The prepare_country_stats() function’s definition is not shown here (see this\nchapter’s Jupyter notebook if you want all the gory details). It’s just boring pandas code\nthat joins the life satisfaction data from the OECD with the GDP per capita data from the\nIMF.\n7 It’s OK if you don’t understand all the code yet; we will present Scikit-Learn in the\nfollowing chapters.\n8 For example, knowing whether to write “to,” “two,” or “too,” depending on the context.\n9 Figure reproduced with permission from Michele Banko and Eric Brill, “Scaling to Very\nVery Large Corpora for Natural Language Disambiguation,” Proceedings of the 39th\nAnnual Meeting of the Association for Computational Linguistics (2001): 26–33.\n10 Peter Norvig et al., “The Unreasonable Effectiveness of Data,” IEEE Intelligent Systems\n24, no. 2 (2009): 8–12.\n11 David Wolpert, “The Lack of A Priori Distinctions Between Learning Algorithms,” Neural\nComputation 8, no. 7 (1996): 1341–1390." + }, + { + "chapter": 2, + "text": "Chapter 2. End-to-End Machine\nLearning Project\nIn this chapter you will work through an example project end to end,\npretending to be a recently hired data scientist at a real estate company. Here\nare the main steps you will go through:\n1. Look at the big picture.\n2. Get the data.\n3. Discover and visualize the data to gain insights.\n4. Prepare the data for Machine Learning algorithms.\n5. Select a model and train it.\n6. Fine-tune your model.\n7. Present your solution.\n8. Launch, monitor, and maintain your system.\nWorking with Real Data\nWhen you are learning about Machine Learning, it is best to experiment with\nreal-world data, not artificial datasets. Fortunately, there are thousands of\nopen datasets to choose from, ranging across all sorts of domains. Here are a\nfew places you can look to get data:\nPopular open data repositories\nUC Irvine Machine Learning Repository\nKaggle datasets\nAmazon’s AWS datasets\n1\nMeta portals (they list open data repositories)\nData Portals\nOpenDataMonitor\nQuandl\nOther pages listing many popular open data repositories\nWikipedia’s list of Machine Learning datasets\nQuora.com\nThe datasets subreddit\nIn this chapter we’ll use the California Housing Prices dataset from the\nStatLib repository (see Figure 2-1). This dataset is based on data from the\n1990 California census. It is not exactly recent (a nice house in the Bay Area\nwas still affordable at the time), but it has many qualities for learning, so we\nwill pretend it is recent data. For teaching purposes I’ve added a categorical\nattribute and removed a few features.\n2\nFigure 2-1. California housing prices\nLook at the Big Picture\nWelcome to the Machine Learning Housing Corporation! Your first task is to\nuse California census data to build a model of housing prices in the state. This\ndata includes metrics such as the population, median income, and median\nhousing price for each block group in California. Block groups are the\nsmallest geographical unit for which the US Census Bureau publishes sample\ndata (a block group typically has a population of 600 to 3,000 people). We\nwill call them “districts” for short.\nYour model should learn from this data and be able to predict the median\nhousing price in any district, given all the other metrics.\nTIP\nSince you are a well-organized data scientist, the first thing you should do is pull out\nyour Machine Learning project checklist. You can start with the one in Appendix B; it\nshould work reasonably well for most Machine Learning projects, but make sure to adapt\nit to your needs. In this chapter we will go through many checklist items, but we will also\nskip a few, either because they are self-explanatory or because they will be discussed in\nlater chapters.\nFrame the Problem\nThe first question to ask your boss is what exactly the business objective is.\nBuilding a model is probably not the end goal. How does the company expect\nto use and benefit from this model? Knowing the objective is important\nbecause it will determine how you frame the problem, which algorithms you\nwill select, which performance measure you will use to evaluate your model,\nand how much effort you will spend tweaking it.\nYour boss answers that your model’s output (a prediction of a district’s\nmedian housing price) will be fed to another Machine Learning system (see\nFigure 2-2), along with many other signals. This downstream system will\ndetermine whether it is worth investing in a given area or not. Getting this\nright is critical, as it directly affects revenue.\nFigure 2-2. A Machine Learning pipeline for real estate investments\n3\nPIPELINES\nA sequence of data processing components is called a data pipeline.\nPipelines are very common in Machine Learning systems, since there is a\nlot of data to manipulate and many data transformations to apply.\nComponents typically run asynchronously. Each component pulls in a\nlarge amount of data, processes it, and spits out the result in another data\nstore. Then, some time later, the next component in the pipeline pulls this\ndata and spits out its own output. Each component is fairly self-contained:\nthe interface between components is simply the data store. This makes the\nsystem simple to grasp (with the help of a data flow graph), and different\nteams can focus on different components. Moreover, if a component\nbreaks down, the downstream components can often continue to run\nnormally (at least for a while) by just using the last output from the\nbroken component. This makes the architecture quite robust.\nOn the other hand, a broken component can go unnoticed for some time if\nproper monitoring is not implemented. The data gets stale and the overall\nsystem’s performance drops.\nThe next question to ask your boss is what the current solution looks like (if\nany). The current situation will often give you a reference for performance, as\nwell as insights on how to solve the problem. Your boss answers that the\ndistrict housing prices are currently estimated manually by experts: a team\ngathers up-to-date information about a district, and when they cannot get the\nmedian housing price, they estimate it using complex rules.\nThis is costly and time-consuming, and their estimates are not great; in cases\nwhere they manage to find out the actual median housing price, they often\nrealize that their estimates were off by more than 20%. This is why the\ncompany thinks that it would be useful to train a model to predict a district’s\nmedian housing price, given other data about that district. The census data\nlooks like a great dataset to exploit for this purpose, since it includes the\nmedian housing prices of thousands of districts, as well as other data.\nWith all this information, you are now ready to start designing your system.\nFirst, you need to frame the problem: is it supervised, unsupervised, or\nReinforcement Learning? Is it a classification task, a regression task, or\nsomething else? Should you use batch learning or online learning techniques?\nBefore you read on, pause and try to answer these questions for yourself.\nHave you found the answers? Let’s see: it is clearly a typical supervised\nlearning task, since you are given labeled training examples (each instance\ncomes with the expected output, i.e., the district’s median housing price). It is\nalso a typical regression task, since you are asked to predict a value. More\nspecifically, this is a multiple regression problem, since the system will use\nmultiple features to make a prediction (it will use the district’s population, the\nmedian income, etc.). It is also a univariate regression problem, since we are\nonly trying to predict a single value for each district. If we were trying to\npredict multiple values per district, it would be a multivariate regression\nproblem. Finally, there is no continuous flow of data coming into the system,\nthere is no particular need to adjust to changing data rapidly, and the data is\nsmall enough to fit in memory, so plain batch learning should do just fine.\nTIP\nIf the data were huge, you could either split your batch learning work across multiple\nservers (using the MapReduce technique) or use an online learning technique.\nSelect a Performance Measure\nYour next step is to select a performance measure. A typical performance\nmeasure for regression problems is the Root Mean Square Error (RMSE). It\ngives an idea of how much error the system typically makes in its predictions,\nwith a higher weight for large errors. Equation 2-1 shows the mathematical\nformula to compute the RMSE.\nEquation 2-1. Root Mean Square Error (RMSE)\nRMSE (X, h) =\n\n \n\n⎷\nm\n∑\ni=1\n(h (x(i)) −y(i))\n2\n1\nm\nNOTATIONS\nThis equation introduces several very common Machine Learning\nnotations that we will use throughout this book:\nm is the number of instances in the dataset you are measuring the\nRMSE on.\nFor example, if you are evaluating the RMSE on a\nvalidation set of 2,000 districts, then m = 2,000.\nx is a vector of all the feature values (excluding the label) of the\ni instance in the dataset, and y is its label (the desired output\nvalue for that instance).\nFor example, if the first district in the dataset is located\nat longitude –118.29°, latitude 33.91°, and it has 1,416\ninhabitants with a median income of $38,372, and the\nmedian house value is $156,400 (ignoring the other\nfeatures for now), then:\nx(1) =\n⎛\n⎜ \n⎜ \n⎜ \n⎜\n⎝\n−118.29\n33.91\n1,416\n38,372\n⎞\n⎟ \n⎟ \n⎟ \n⎟\n⎠\nand:\ny(1) = 156,400\nX is a matrix containing all the feature values (excluding labels)\nof all instances in the dataset. There is one row per instance, and\nthe i row is equal to the transpose of x , noted (x ) .\nFor example, if the first district is as just described, then\nthe matrix X looks like this:\n(i)\nth\n(i)\nth\n(i)\n(i) ⊺4\nX =\n⎛\n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜\n⎝\n(x(1))\n⊺\n(x(2))\n⊺\n⋮\n(x(1999))\n⊺\n(x(2000))\n⊺\n⎞\n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟\n⎠\n=(\n−118.29\n33.91\n1,416\n38,372\n⋮\n⋮\n⋮\n⋮\n)\nh is your system’s prediction function, also called a hypothesis.\nWhen your system is given an instance’s feature vector x , it\noutputs a predicted value ŷ = h(x ) for that instance (ŷ is\npronounced “y-hat”).\nFor example, if your system predicts that the median\nhousing price in the first district is $158,400, then ŷ\n =\nh(x\n) = 158,400. The prediction error for this district is\nŷ\n – y\n = 2,000.\nRMSE(X,h) is the cost function measured on the set of examples\nusing your hypothesis h.\nWe use lowercase italic font for scalar values (such as m or y ) and\nfunction names (such as h), lowercase bold font for vectors (such as x ),\nand uppercase bold font for matrices (such as X).\nEven though the RMSE is generally the preferred performance measure for\nregression tasks, in some contexts you may prefer to use another function. For\nexample, suppose that there are many outlier districts. In that case, you may\nconsider using the mean absolute error (MAE, also called the average\nabsolute deviation; see Equation 2-2):\nEquation 2-2. Mean absolute error (MAE)\nMAE (X, h) =\nm\n∑\ni=1\n∣∣h (x(i)) −y(i)∣∣\n(i)\n(i)\n(i)\n(1)\n(1)\n(1)\n(1)\n(i)\n(i)\n1\nm\nBoth the RMSE and the MAE are ways to measure the distance between two\nvectors: the vector of predictions and the vector of target values. Various\ndistance measures, or norms, are possible:\nComputing the root of a sum of squares (RMSE) corresponds to the\nEuclidean norm: this is the notion of distance you are familiar with.\nIt is also called the ℓ norm, noted ∥ · ∥ (or just ∥ · ∥).\nComputing the sum of absolutes (MAE) corresponds to the ℓ norm,\nnoted ∥ · ∥. This is sometimes called the Manhattan norm because it\nmeasures the distance between two points in a city if you can only\ntravel along orthogonal city blocks.\nMore generally, the ℓ norm of a vector v containing n elements is\ndefined as ∥v∥k = (|v0|k + |v1|k + ⋯+ |vn|k) . ℓ gives the\nnumber of nonzero elements in the vector, and ℓ gives the\nmaximum absolute value in the vector.\nThe higher the norm index, the more it focuses on large values and\nneglects small ones. This is why the RMSE is more sensitive to\noutliers than the MAE. But when outliers are exponentially rare (like\nin a bell-shaped curve), the RMSE performs very well and is\ngenerally preferred.\nCheck the Assumptions\nLastly, it is good practice to list and verify the assumptions that have been\nmade so far (by you or others); this can help you catch serious issues early on.\nFor example, the district prices that your system outputs are going to be fed\ninto a downstream Machine Learning system, and you assume that these\nprices are going to be used as such. But what if the downstream system\nconverts the prices into categories (e.g., “cheap,” “medium,” or “expensive”)\nand then uses those categories instead of the prices themselves? In this case,\ngetting the price perfectly right is not important at all; your system just needs\nto get the category right. If that’s so, then the problem should have been\nframed as a classification task, not a regression task. You don’t want to find\nthis out after working on a regression system for months.\n2\n2\n1\n1\nk\n1\nk\n0\n∞\nFortunately, after talking with the team in charge of the downstream system,\nyou are confident that they do indeed need the actual prices, not just\ncategories. Great! You’re all set, the lights are green, and you can start coding\nnow!\nGet the Data\nIt’s time to get your hands dirty. Don’t hesitate to pick up your laptop and\nwalk through the following code examples in a Jupyter notebook. The full\nJupyter notebook is available at https://github.com/ageron/handson-ml2.\nCreate the Workspace\nFirst you will need to have Python installed. It is probably already installed on\nyour system. If not, you can get it at https://www.python.org/.\nNext you need to create a workspace directory for your Machine Learning\ncode and datasets. Open a terminal and type the following commands (after\nthe $ prompts):\n$ export ML_PATH=\"$HOME/ml\" # You can change the path if you prefer \n$ mkdir -p $ML_PATH \nYou will need a number of Python modules: Jupyter, NumPy, pandas,\nMatplotlib, and Scikit-Learn. If you already have Jupyter running with all\nthese modules installed, you can safely skip to “Download the Data”. If you\ndon’t have them yet, there are many ways to install them (and their\ndependencies). You can use your system’s packaging system (e.g., apt-get on\nUbuntu, or MacPorts or Homebrew on macOS), install a Scientific Python\ndistribution such as Anaconda and use its packaging system, or just use\nPython’s own packaging system, pip, which is included by default with the\nPython binary installers (since Python 2.7.9). You can check to see if pip is\ninstalled by typing the following command:\n$ python3 -m pip --version \npip 19.0.2 from [...]/lib/python3.6/site-packages (python 3.6) \n5\n6\nYou should make sure you have a recent version of pip installed. To upgrade\nthe pip module, type the following (the exact version may differ):\n$ python3 -m pip install --user -U pip \nCollecting pip \n[...] \nSuccessfully installed pip-19.0.2 \n7\nCREATING AN ISOLATED ENVIRONMENT\nIf you would like to work in an isolated environment (which is strongly\nrecommended so that you can work on different projects without having\nconflicting library versions), install virtualenv by running the following\npip command (again, if you want virtualenv to be installed for all users on\nyour machine, remove --user and run this command with administrator\nrights):\n$ python3 -m pip install --user -U virtualenv \nCollecting virtualenv \n[...] \nSuccessfully installed virtualenv \nNow you can create an isolated Python environment by typing this:\n$ cd $ML_PATH \n$ virtualenv my_env \nUsing base prefix '[...]' \nNew python executable in [...]/ml/my_env/bin/python3.6 \nAlso creating executable in [...]/ml/my_env/bin/python \nInstalling setuptools, pip, wheel...done. \nNow every time you want to activate this environment, just open a\nterminal and type the following:\n$ cd $ML_PATH \n$ source my_env/bin/activate # on Linux or macOS \n$ .\\my_env\\Scripts\\activate # on Windows \nTo deactivate this environment, type deactivate. While the environment\nis active, any package you install using pip will be installed in this\nisolated environment, and Python will only have access to these packages\n(if you also want access to the system’s packages, you should create the\nenvironment using virtualenv’s --system-site-packages option). Check\nout virtualenv’s documentation for more information.\nNow you can install all the required modules and their dependencies using\nthis simple pip command (if you are not using a virtualenv, you will need the\n8\n--user option or administrator rights):\n$ python3 -m pip install -U jupyter matplotlib numpy pandas scipy scikit-\nlearn \nCollecting jupyter \n Downloading jupyter-1.0.0-py2.py3-none-any.whl \nCollecting matplotlib \n [...] \nTo check your installation, try to import every module like this:\n$ python3 -c \"import jupyter, matplotlib, numpy, pandas, scipy, sklearn\" \nThere should be no output and no error. Now you can fire up Jupyter by typing\nthe following:\n$ jupyter notebook \n[I 15:24 NotebookApp] Serving notebooks from local directory: [...]/ml \n[I 15:24 NotebookApp] 0 active kernels \n[I 15:24 NotebookApp] The Jupyter Notebook is running at: \nhttp://localhost:8888/ \n[I 15:24 NotebookApp] Use Control-C to stop this server and shut down all \nkernels (twice to skip confirmation). \nA Jupyter server is now running in your terminal, listening to port 8888. You\ncan visit this server by opening your web browser to http://localhost:8888/\n(this usually happens automatically when the server starts). You should see\nyour empty workspace directory (containing only the env directory if you\nfollowed the preceding virtualenv instructions).\nNow create a new Python notebook by clicking the New button and selecting\nthe appropriate Python version (see Figure 2-3). Doing that will create a new\nnotebook file called Untitled.ipynb in your workspace, start a Jupyter Python\nkernel to run the notebook, and open this notebook in a new tab. You should\nstart by renaming this notebook to “Housing” (this will automatically rename\nthe file to Housing.ipynb) by clicking Untitled and typing the new name.\n9\nFigure 2-3. Your workspace in Jupyter\nA notebook contains a list of cells. Each cell can contain executable code or\nformatted text. Right now the notebook contains only one empty code cell,\nlabeled “In [1]:”. Try typing print(\"Hello world!\") in the cell and clicking\nthe play button (see Figure 2-4) or pressing Shift-Enter. This sends the current\ncell to this notebook’s Python kernel, which runs it and returns the output. The\nresult is displayed below the cell, and since you’ve reached the end of the\nnotebook, a new cell is automatically created. Go through the User Interface\nTour from Jupyter’s Help menu to learn the basics.\nFigure 2-4. Hello world Python notebook\nDownload the Data\nIn typical environments your data would be available in a relational database\n(or some other common data store) and spread across multiple\ntables/documents/files. To access it, you would first need to get your\ncredentials and access authorizations\n and familiarize yourself with the data\nschema. In this project, however, things are much simpler: you will just\ndownload a single compressed file, housing.tgz, which contains a comma-\nseparated values (CSV) file called housing.csv with all the data.\nYou could use your web browser to download the file and run tar xzf\nhousing.tgz to decompress it and extract the CSV file, but it is preferable to\ncreate a small function to do that. Having a function that downloads the data\nis useful in particular if the data changes regularly: you can write a small\nscript that uses the function to fetch the latest data (or you can set up a\nscheduled job to do that automatically at regular intervals). Automating the\nprocess of fetching the data is also useful if you need to install the dataset on\nmultiple machines.\nHere is the function to fetch the data:\nimport os \nimport tarfile \nimport urllib \n \nDOWNLOAD_ROOT = \"https://raw.githubusercontent.com/ageron/handson-\nml2/master/\" \nHOUSING_PATH = os.path.join(\"datasets\", \"housing\") \nHOUSING_URL = DOWNLOAD_ROOT + \"datasets/housing/housing.tgz\" \n \ndef fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH): \n os.makedirs(housing_path, exist_ok=True) \n tgz_path = os.path.join(housing_path, \"housing.tgz\") \n urllib.request.urlretrieve(housing_url, tgz_path) \n housing_tgz = tarfile.open(tgz_path) \n housing_tgz.extractall(path=housing_path) \n housing_tgz.close()\nNow when you call fetch_housing_data(), it creates a datasets/housing\ndirectory in your workspace, downloads the housing.tgz file, and extracts the\nhousing.csv file from it in this directory.\nNow let’s load the data using pandas. Once again, you should write a small\nfunction to load the data:\n10\n11\nimport pandas as pd \n \ndef load_housing_data(housing_path=HOUSING_PATH): \n csv_path = os.path.join(housing_path, \"housing.csv\") \n return pd.read_csv(csv_path)\nThis function returns a pandas DataFrame object containing all the data.\nTake a Quick Look at the Data Structure\nLet’s take a look at the top five rows using the DataFrame’s head() method\n(see Figure 2-5).\nFigure 2-5. Top five rows in the dataset\nEach row represents one district. There are 10 attributes (you can see the first\n6 in the screenshot): longitude, latitude, housing_median_age,\ntotal_rooms, total_bedrooms, population, households, median_income,\nmedian_house_value, and ocean_proximity.\nThe info() method is useful to get a quick description of the data, in\nparticular the total number of rows, each attribute’s type, and the number of\nnonnull values (see Figure 2-6).\nFigure 2-6. Housing info\nThere are 20,640 instances in the dataset, which means that it is fairly small\nby Machine Learning standards, but it’s perfect to get started. Notice that the\ntotal_bedrooms attribute has only 20,433 nonnull values, meaning that 207\ndistricts are missing this feature. We will need to take care of this later.\nAll attributes are numerical, except the ocean_proximity field. Its type is\nobject, so it could hold any kind of Python object. But since you loaded this\ndata from a CSV file, you know that it must be a text attribute. When you\nlooked at the top five rows, you probably noticed that the values in the\nocean_proximity column were repetitive, which means that it is probably a\ncategorical attribute. You can find out what categories exist and how many\ndistricts belong to each category by using the value_counts() method:\n>>> housing[\"ocean_proximity\"].value_counts() \n<1H OCEAN 9136 \nINLAND 6551 \nNEAR OCEAN 2658 \nNEAR BAY 2290 \nISLAND 5 \nName: ocean_proximity, dtype: int64\nLet’s look at the other fields. The describe() method shows a summary of\nthe numerical attributes (Figure 2-7).\nFigure 2-7. Summary of each numerical attribute\nThe count, mean, min, and max rows are self-explanatory. Note that the null\nvalues are ignored (so, for example, the count of total_bedrooms is 20,433,\nnot 20,640). The std row shows the standard deviation, which measures how\ndispersed the values are.\n The 25%, 50%, and 75% rows show the\ncorresponding percentiles: a percentile indicates the value below which a\ngiven percentage of observations in a group of observations fall. For example,\n25% of the districts have a housing_median_age lower than 18, while 50%\nare lower than 29 and 75% are lower than 37. These are often called the 25th\npercentile (or first quartile), the median, and the 75th percentile (or third\nquartile).\nAnother quick way to get a feel of the type of data you are dealing with is to\nplot a histogram for each numerical attribute. A histogram shows the number\nof instances (on the vertical axis) that have a given value range (on the\nhorizontal axis). You can either plot this one attribute at a time, or you can\ncall the hist() method on the whole dataset (as shown in the following code\nexample), and it will plot a histogram for each numerical attribute (see\nFigure 2-8):\n%matplotlib inline # only in a Jupyter notebook \nimport matplotlib.pyplot as plt \nhousing.hist(bins=50, figsize=(20,15)) \nplt.show()\n12\nNOTE\nThe hist() method relies on Matplotlib, which in turn relies on a user-specified\ngraphical backend to draw on your screen. So before you can plot anything, you need to\nspecify which backend Matplotlib should use. The simplest option is to use Jupyter’s\nmagic command %matplotlib inline. This tells Jupyter to set up Matplotlib so it uses\nJupyter’s own backend. Plots are then rendered within the notebook itself. Note that\ncalling show() is optional in a Jupyter notebook, as Jupyter will automatically display\nplots when a cell is executed.\nFigure 2-8. A histogram for each numerical attribute\nThere are a few things you might notice in these histograms:\n1. First, the median income attribute does not look like it is expressed\nin US dollars (USD). After checking with the team that collected the\ndata, you are told that the data has been scaled and capped at 15\n(actually, 15.0001) for higher median incomes, and at 0.5 (actually,\n0.4999) for lower median incomes. The numbers represent roughly\ntens of thousands of dollars (e.g., 3 actually means about $30,000).\nWorking with preprocessed attributes is common in Machine\nLearning, and it is not necessarily a problem, but you should try to\nunderstand how the data was computed.\n2. The housing median age and the median house value were also\ncapped. The latter may be a serious problem since it is your target\nattribute (your labels). Your Machine Learning algorithms may learn\nthat prices never go beyond that limit. You need to check with your\nclient team (the team that will use your system’s output) to see if this\nis a problem or not. If they tell you that they need precise predictions\neven beyond $500,000, then you have two options:\na. Collect proper labels for the districts whose labels were\ncapped.\nb. Remove those districts from the training set (and also from\nthe test set, since your system should not be evaluated poorly\nif it predicts values beyond $500,000).\n3. These attributes have very different scales. We will discuss this later\nin this chapter, when we explore feature scaling.\n4. Finally, many histograms are tail-heavy: they extend much farther to\nthe right of the median than to the left. This may make it a bit harder\nfor some Machine Learning algorithms to detect patterns. We will try\ntransforming these attributes later on to have more bell-shaped\ndistributions.\nHopefully you now have a better understanding of the kind of data you are\ndealing with.\nWARNING\nWait! Before you look at the data any further, you need to create a test set, put it aside,\nand never look at it.\nCreate a Test Set\nIt may sound strange to voluntarily set aside part of the data at this stage.\nAfter all, you have only taken a quick glance at the data, and surely you\nshould learn a whole lot more about it before you decide what algorithms to\nuse, right? This is true, but your brain is an amazing pattern detection system,\nwhich means that it is highly prone to overfitting: if you look at the test set,\nyou may stumble upon some seemingly interesting pattern in the test data that\nleads you to select a particular kind of Machine Learning model. When you\nestimate the generalization error using the test set, your estimate will be too\noptimistic, and you will launch a system that will not perform as well as\nexpected. This is called data snooping bias.\nCreating a test set is theoretically simple: pick some instances randomly,\ntypically 20% of the dataset (or less if your dataset is very large), and set\nthem aside:\nimport numpy as np \n \ndef split_train_test(data, test_ratio): \n shuffled_indices = np.random.permutation(len(data)) \n test_set_size = int(len(data) * test_ratio) \n test_indices = shuffled_indices[:test_set_size] \n train_indices = shuffled_indices[test_set_size:] \n return data.iloc[train_indices], data.iloc[test_indices]\nYou can then use this function like this:\n>>> train_set, test_set = split_train_test(housing, 0.2) \n>>> len(train_set) \n16512 \n>>> len(test_set) \n4128\nWell, this works, but it is not perfect: if you run the program again, it will\ngenerate a different test set! Over time, you (or your Machine Learning\nalgorithms) will get to see the whole dataset, which is what you want to avoid.\nOne solution is to save the test set on the first run and then load it in\nsubsequent runs. Another option is to set the random number generator’s seed\n(e.g., with np.random.seed(42))\n before calling\n13\n14\nnp.random.permutation() so that it always generates the same shuffled\nindices.\nBut both these solutions will break the next time you fetch an updated dataset.\nTo have a stable train/test split even after updating the dataset, a common\nsolution is to use each instance’s identifier to decide whether or not it should\ngo in the test set (assuming instances have a unique and immutable\nidentifier). For example, you could compute a hash of each instance’s\nidentifier and put that instance in the test set if the hash is lower than or equal\nto 20% of the maximum hash value. This ensures that the test set will remain\nconsistent across multiple runs, even if you refresh the dataset. The new test\nset will contain 20% of the new instances, but it will not contain any instance\nthat was previously in the training set.\nHere is a possible implementation:\nfrom zlib import crc32 \n \ndef test_set_check(identifier, test_ratio): \n return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32 \n \ndef split_train_test_by_id(data, test_ratio, id_column): \n ids = data[id_column] \n in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio)) \n return data.loc[~in_test_set], data.loc[in_test_set]\nUnfortunately, the housing dataset does not have an identifier column. The\nsimplest solution is to use the row index as the ID:\nhousing_with_id = housing.reset_index() # adds an `index` column \ntrain_set, test_set = split_train_test_by_id(housing_with_id, 0.2, \"index\")\nIf you use the row index as a unique identifier, you need to make sure that\nnew data gets appended to the end of the dataset and that no row ever gets\ndeleted. If this is not possible, then you can try to use the most stable features\nto build a unique identifier. For example, a district’s latitude and longitude are\nguaranteed to be stable for a few million years, so you could combine them\ninto an ID like so:15\nhousing_with_id[\"id\"] = housing[\"longitude\"] * 1000 + housing[\"latitude\"] \ntrain_set, test_set = split_train_test_by_id(housing_with_id, 0.2, \"id\")\nScikit-Learn provides a few functions to split datasets into multiple subsets in\nvarious ways. The simplest function is train_test_split(), which does\npretty much the same thing as the function split_train_test(), with a\ncouple of additional features. First, there is a random_state parameter that\nallows you to set the random generator seed. Second, you can pass it multiple\ndatasets with an identical number of rows, and it will split them on the same\nindices (this is very useful, for example, if you have a separate DataFrame for\nlabels):\nfrom sklearn.model_selection import train_test_split \n \ntrain_set, test_set = train_test_split(housing, test_size=0.2, \nrandom_state=42)\nSo far we have considered purely random sampling methods. This is generally\nfine if your dataset is large enough (especially relative to the number of\nattributes), but if it is not, you run the risk of introducing a significant\nsampling bias. When a survey company decides to call 1,000 people to ask\nthem a few questions, they don’t just pick 1,000 people randomly in a phone\nbook. They try to ensure that these 1,000 people are representative of the\nwhole population. For example, the US population is 51.3% females and\n48.7% males, so a well-conducted survey in the US would try to maintain this\nratio in the sample: 513 female and 487 male. This is called stratified\nsampling: the population is divided into homogeneous subgroups called\nstrata, and the right number of instances are sampled from each stratum to\nguarantee that the test set is representative of the overall population. If the\npeople running the survey used purely random sampling, there would be about\na 12% chance of sampling a skewed test set that was either less than 49%\nfemale or more than 54% female. Either way, the survey results would be\nsignificantly biased.\nSuppose you chatted with experts who told you that the median income is a\nvery important attribute to predict median housing prices. You may want to\nensure that the test set is representative of the various categories of incomes\nin the whole dataset. Since the median income is a continuous numerical\nattribute, you first need to create an income category attribute. Let’s look at\nthe median income histogram more closely (back in Figure 2-8): most median\nincome values are clustered around 1.5 to 6 (i.e., $15,000–$60,000), but some\nmedian incomes go far beyond 6. It is important to have a sufficient number\nof instances in your dataset for each stratum, or else the estimate of a\nstratum’s importance may be biased. This means that you should not have too\nmany strata, and each stratum should be large enough. The following code\nuses the pd.cut() function to create an income category attribute with five\ncategories (labeled from 1 to 5): category 1 ranges from 0 to 1.5 (i.e., less\nthan $15,000), category 2 from 1.5 to 3, and so on:\nhousing[\"income_cat\"] = pd.cut(housing[\"median_income\"], \n bins=[0., 1.5, 3.0, 4.5, 6., np.inf], \n labels=[1, 2, 3, 4, 5])\nThese income categories are represented in Figure 2-9:\nhousing[\"income_cat\"].hist()\nFigure 2-9. Histogram of income categories\nNow you are ready to do stratified sampling based on the income category.\nFor this you can use Scikit-Learn’s StratifiedShuffleSplit class:\nfrom sklearn.model_selection import StratifiedShuffleSplit \n \nsplit = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42) \nfor train_index, test_index in split.split(housing, housing[\"income_cat\"]): \n strat_train_set = housing.loc[train_index] \n strat_test_set = housing.loc[test_index]\nLet’s see if this worked as expected. You can start by looking at the income\ncategory proportions in the test set:\n>>> strat_test_set[\"income_cat\"].value_counts() / len(strat_test_set) \n3 0.350533 \n2 0.318798 \n4 0.176357 \n5 0.114583 \n1 0.039729 \nName: income_cat, dtype: float64\nWith similar code you can measure the income category proportions in the\nfull dataset. Figure 2-10 compares the income category proportions in the\noverall dataset, in the test set generated with stratified sampling, and in a test\nset generated using purely random sampling. As you can see, the test set\ngenerated using stratified sampling has income category proportions almost\nidentical to those in the full dataset, whereas the test set generated using\npurely random sampling is skewed.\nFigure 2-10. Sampling bias comparison of stratified versus purely random sampling\nNow you should remove the income_cat attribute so the data is back to its\noriginal state:\nfor set_ in (strat_train_set, strat_test_set): \n set_.drop(\"income_cat\", axis=1, inplace=True)\nWe spent quite a bit of time on test set generation for a good reason: this is an\noften neglected but critical part of a Machine Learning project. Moreover,\nmany of these ideas will be useful later when we discuss cross-validation.\nNow it’s time to move on to the next stage: exploring the data.\nDiscover and Visualize the Data to Gain\nInsights\nSo far you have only taken a quick glance at the data to get a general\nunderstanding of the kind of data you are manipulating. Now the goal is to go\ninto a little more depth.\nFirst, make sure you have put the test set aside and you are only exploring the\ntraining set. Also, if the training set is very large, you may want to sample an\nexploration set, to make manipulations easy and fast. In our case, the set is\nquite small, so you can just work directly on the full set. Let’s create a copy so\nthat you can play with it without harming the training set:\nhousing = strat_train_set.copy()\nVisualizing Geographical Data\nSince there is geographical information (latitude and longitude), it is a good\nidea to create a scatterplot of all districts to visualize the data (Figure 2-11):\nhousing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\")\nFigure 2-11. A geographical scatterplot of the data\nThis looks like California all right, but other than that it is hard to see any\nparticular pattern. Setting the alpha option to 0.1 makes it much easier to\nvisualize the places where there is a high density of data points (Figure 2-12):\nhousing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.1)\nFigure 2-12. A better visualization that highlights high-density areas\nNow that’s much better: you can clearly see the high-density areas, namely\nthe Bay Area and around Los Angeles and San Diego, plus a long line of fairly\nhigh density in the Central Valley, in particular around Sacramento and\nFresno.\nOur brains are very good at spotting patterns in pictures, but you may need to\nplay around with visualization parameters to make the patterns stand out.\nNow let’s look at the housing prices (Figure 2-13). The radius of each circle\nrepresents the district’s population (option s), and the color represents the\nprice (option c). We will use a predefined color map (option cmap) called jet,\nwhich ranges from blue (low values) to red (high prices):\nhousing.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.4, \n s=housing[\"population\"]/100, label=\"population\", figsize=(10,7), \n c=\"median_house_value\", cmap=plt.get_cmap(\"jet\"), colorbar=True, \n) \nplt.legend()\n16\nFigure 2-13. California housing prices: red is expensive, blue is cheap, larger circles indicate areas\nwith a larger population\nThis image tells you that the housing prices are very much related to the\nlocation (e.g., close to the ocean) and to the population density, as you\nprobably knew already. A clustering algorithm should be useful for detecting\nthe main cluster and for adding new features that measure the proximity to the\ncluster centers. The ocean proximity attribute may be useful as well, although\nin Northern California the housing prices in coastal districts are not too high,\nso it is not a simple rule.\nLooking for Correlations\nSince the dataset is not too large, you can easily compute the standard\ncorrelation coefficient (also called Pearson’s r) between every pair of\nattributes using the corr() method:\ncorr_matrix = housing.corr()\nNow let’s look at how much each attribute correlates with the median house\nvalue:\n>>> corr_matrix[\"median_house_value\"].sort_values(ascending=False) \nmedian_house_value 1.000000 \nmedian_income 0.687170 \ntotal_rooms 0.135231 \nhousing_median_age 0.114220 \nhouseholds 0.064702 \ntotal_bedrooms 0.047865 \npopulation -0.026699 \nlongitude -0.047279 \nlatitude -0.142826 \nName: median_house_value, dtype: float64\nThe correlation coefficient ranges from –1 to 1. When it is close to 1, it means\nthat there is a strong positive correlation; for example, the median house\nvalue tends to go up when the median income goes up. When the coefficient is\nclose to –1, it means that there is a strong negative correlation; you can see a\nsmall negative correlation between the latitude and the median house value\n(i.e., prices have a slight tendency to go down when you go north). Finally,\ncoefficients close to 0 mean that there is no linear correlation. Figure 2-14\nshows various plots along with the correlation coefficient between their\nhorizontal and vertical axes.\nFigure 2-14. Standard correlation coefficient of various datasets (source: Wikipedia; public domain\nimage)\nWARNING\nThe correlation coefficient only measures linear correlations (“if x goes up, then y\ngenerally goes up/down”). It may completely miss out on nonlinear relationships (e.g.,\n“if x is close to 0, then y generally goes up”). Note how all the plots of the bottom row\nhave a correlation coefficient equal to 0, despite the fact that their axes are clearly not\nindependent: these are examples of nonlinear relationships. Also, the second row shows\nexamples where the correlation coefficient is equal to 1 or –1; notice that this has nothing\nto do with the slope. For example, your height in inches has a correlation coefficient of 1\nwith your height in feet or in nanometers.\nAnother way to check for correlation between attributes is to use the pandas\nscatter_matrix() function, which plots every numerical attribute against\nevery other numerical attribute. Since there are now 11 numerical attributes,\nyou would get 11 = 121 plots, which would not fit on a page—so let’s just\nfocus on a few promising attributes that seem most correlated with the\nmedian housing value (Figure 2-15):\nfrom pandas.plotting import scatter_matrix \n \nattributes = [\"median_house_value\", \"median_income\", \"total_rooms\", \n \"housing_median_age\"] \nscatter_matrix(housing[attributes], figsize=(12, 8))\n2\nFigure 2-15. This scatter matrix plots every numerical attribute against every other numerical\nattribute, plus a histogram of each numerical attribute\nThe main diagonal (top left to bottom right) would be full of straight lines if\npandas plotted each variable against itself, which would not be very useful. So\ninstead pandas displays a histogram of each attribute (other options are\navailable; see the pandas documentation for more details).\nThe most promising attribute to predict the median house value is the median\nincome, so let’s zoom in on their correlation scatterplot (Figure 2-16):\nhousing.plot(kind=\"scatter\", x=\"median_income\", y=\"median_house_value\", \n alpha=0.1)\nFigure 2-16. Median income versus median house value\nThis plot reveals a few things. First, the correlation is indeed very strong; you\ncan clearly see the upward trend, and the points are not too dispersed. Second,\nthe price cap that we noticed earlier is clearly visible as a horizontal line at\n$500,000. But this plot reveals other less obvious straight lines: a horizontal\nline around $450,000, another around $350,000, perhaps one around $280,000,\nand a few more below that. You may want to try removing the corresponding\ndistricts to prevent your algorithms from learning to reproduce these data\nquirks.\nExperimenting with Attribute Combinations\nHopefully the previous sections gave you an idea of a few ways you can\nexplore the data and gain insights. You identified a few data quirks that you\nmay want to clean up before feeding the data to a Machine Learning\nalgorithm, and you found interesting correlations between attributes, in\nparticular with the target attribute. You also noticed that some attributes have\na tail-heavy distribution, so you may want to transform them (e.g., by\ncomputing their logarithm). Of course, your mileage will vary considerably\nwith each project, but the general ideas are similar.\nOne last thing you may want to do before preparing the data for Machine\nLearning algorithms is to try out various attribute combinations. For example,\nthe total number of rooms in a district is not very useful if you don’t know\nhow many households there are. What you really want is the number of rooms\nper household. Similarly, the total number of bedrooms by itself is not very\nuseful: you probably want to compare it to the number of rooms. And the\npopulation per household also seems like an interesting attribute combination\nto look at. Let’s create these new attributes:\nhousing[\"rooms_per_household\"] = housing[\"total_rooms\"]/housing[\"households\"] \nhousing[\"bedrooms_per_room\"] = \nhousing[\"total_bedrooms\"]/housing[\"total_rooms\"] \nhousing[\"population_per_household\"]=housing[\"population\"]/housing[\"households\n\"]\nAnd now let’s look at the correlation matrix again:\n>>> corr_matrix = housing.corr() \n>>> corr_matrix[\"median_house_value\"].sort_values(ascending=False) \nmedian_house_value 1.000000 \nmedian_income 0.687160 \nrooms_per_household 0.146285 \ntotal_rooms 0.135097 \nhousing_median_age 0.114110 \nhouseholds 0.064506 \ntotal_bedrooms 0.047689 \npopulation_per_household -0.021985 \npopulation -0.026920 \nlongitude -0.047432 \nlatitude -0.142724 \nbedrooms_per_room -0.259984 \nName: median_house_value, dtype: float64\nHey, not bad! The new bedrooms_per_room attribute is much more correlated\nwith the median house value than the total number of rooms or bedrooms.\nApparently houses with a lower bedroom/room ratio tend to be more\nexpensive. The number of rooms per household is also more informative than\nthe total number of rooms in a district—obviously the larger the houses, the\nmore expensive they are.\nThis round of exploration does not have to be absolutely thorough; the point is\nto start off on the right foot and quickly gain insights that will help you get a\nfirst reasonably good prototype. But this is an iterative process: once you get\na prototype up and running, you can analyze its output to gain more insights\nand come back to this exploration step.\nPrepare the Data for Machine Learning\nAlgorithms\nIt’s time to prepare the data for your Machine Learning algorithms. Instead of\ndoing this manually, you should write functions for this purpose, for several\ngood reasons:\nThis will allow you to reproduce these transformations easily on any\ndataset (e.g., the next time you get a fresh dataset).\nYou will gradually build a library of transformation functions that\nyou can reuse in future projects.\nYou can use these functions in your live system to transform the new\ndata before feeding it to your algorithms.\nThis will make it possible for you to easily try various\ntransformations and see which combination of transformations works\nbest.\nBut first let’s revert to a clean training set (by copying strat_train_set\nonce again). Let’s also separate the predictors and the labels, since we don’t\nnecessarily want to apply the same transformations to the predictors and the\ntarget values (note that drop() creates a copy of the data and does not affect\nstrat_train_set):\nhousing = strat_train_set.drop(\"median_house_value\", axis=1) \nhousing_labels = strat_train_set[\"median_house_value\"].copy()\nData Cleaning\nMost Machine Learning algorithms cannot work with missing features, so\nlet’s create a few functions to take care of them. We saw earlier that the\ntotal_bedrooms attribute has some missing values, so let’s fix this. You have\nthree options:\n1. Get rid of the corresponding districts.\n2. Get rid of the whole attribute.\n3. Set the values to some value (zero, the mean, the median, etc.).\nYou can accomplish these easily using DataFrame’s dropna(), drop(), and\nfillna() methods:\nhousing.dropna(subset=[\"total_bedrooms\"]) # option 1 \nhousing.drop(\"total_bedrooms\", axis=1) # option 2 \nmedian = housing[\"total_bedrooms\"].median() # option 3 \nhousing[\"total_bedrooms\"].fillna(median, inplace=True)\nIf you choose option 3, you should compute the median value on the training\nset and use it to fill the missing values in the training set. Don’t forget to save\nthe median value that you have computed. You will need it later to replace\nmissing values in the test set when you want to evaluate your system, and also\nonce the system goes live to replace missing values in new data.\nScikit-Learn provides a handy class to take care of missing values:\nSimpleImputer. Here is how to use it. First, you need to create a\nSimpleImputer instance, specifying that you want to replace each attribute’s\nmissing values with the median of that attribute:\nfrom sklearn.impute import SimpleImputer \n \nimputer = SimpleImputer(strategy=\"median\")\nSince the median can only be computed on numerical attributes, you need to\ncreate a copy of the data without the text attribute ocean_proximity:\nhousing_num = housing.drop(\"ocean_proximity\", axis=1)\nNow you can fit the imputer instance to the training data using the fit()\nmethod:\nimputer.fit(housing_num)\nThe imputer has simply computed the median of each attribute and stored the\nresult in its statistics_ instance variable. Only the total_bedrooms\nattribute had missing values, but we cannot be sure that there won’t be any\nmissing values in new data after the system goes live, so it is safer to apply\nthe imputer to all the numerical attributes:\n>>> imputer.statistics_ \narray([ -118.51 , 34.26 , 29. , 2119.5 , 433. , 1164. , 408. , 3.5409]) \n>>> housing_num.median().values \narray([ -118.51 , 34.26 , 29. , 2119.5 , 433. , 1164. , 408. , 3.5409])\nNow you can use this “trained” imputer to transform the training set by\nreplacing missing values with the learned medians:\nX = imputer.transform(housing_num)\nThe result is a plain NumPy array containing the transformed features. If you\nwant to put it back into a pandas DataFrame, it’s simple:\nhousing_tr = pd.DataFrame(X, columns=housing_num.columns, \n index=housing_num.index)\nSCIKIT-LEARN DESIGN\nScikit-Learn’s API is remarkably well designed. These are the main\ndesign principles:\nConsistency\nAll objects share a consistent and simple interface:\nEstimators\nAny object that can estimate some parameters based on a dataset is\ncalled an estimator (e.g., an imputer is an estimator). The\nestimation itself is performed by the fit() method, and it takes\nonly a dataset as a parameter (or two for supervised learning\nalgorithms; the second dataset contains the labels). Any other\nparameter needed to guide the estimation process is considered a\nhyperparameter (such as an imputer’s strategy), and it must be\nset as an instance variable (generally via a constructor parameter).\nTransformers\nSome estimators (such as an imputer) can also transform a\ndataset; these are called transformers. Once again, the API is\nsimple: the transformation is performed by the transform()\nmethod with the dataset to transform as a parameter. It returns the\ntransformed dataset. This transformation generally relies on the\nlearned parameters, as is the case for an imputer. All transformers\nalso have a convenience method called fit_transform() that is\nequivalent to calling fit() and then transform() (but sometimes\nfit_transform() is optimized and runs much faster).\nPredictors\nFinally, some estimators, given a dataset, are capable of making\npredictions; they are called predictors. For example, the\nLinearRegression model in the previous chapter was a predictor:\ngiven a country’s GDP per capita, it predicted life satisfaction. A\npredictor has a predict() method that takes a dataset of new\n17\ninstances and returns a dataset of corresponding predictions. It\nalso has a score() method that measures the quality of the\npredictions, given a test set (and the corresponding labels, in the\ncase of supervised learning algorithms).\nInspection\nAll the estimator’s hyperparameters are accessible directly via public\ninstance variables (e.g., imputer.strategy), and all the estimator’s\nlearned parameters are accessible via public instance variables with an\nunderscore suffix (e.g., imputer.statistics_).\nNonproliferation of classes\nDatasets are represented as NumPy arrays or SciPy sparse matrices,\ninstead of homemade classes. Hyperparameters are just regular Python\nstrings or numbers.\nComposition\nExisting building blocks are reused as much as possible. For example,\nit is easy to create a Pipeline estimator from an arbitrary sequence of\ntransformers followed by a final estimator, as we will see.\nSensible defaults\nScikit-Learn provides reasonable default values for most parameters,\nmaking it easy to quickly create a baseline working system.\nHandling Text and Categorical Attributes\nSo far we have only dealt with numerical attributes, but now let’s look at text\nattributes. In this dataset, there is just one: the ocean_proximity attribute.\nLet’s look at its value for the first 10 instances:\n>>> housing_cat = housing[[\"ocean_proximity\"]] \n>>> housing_cat.head(10) \n ocean_proximity \n17606 <1H OCEAN \n18632 <1H OCEAN \n18\n14650 NEAR OCEAN \n3230 INLAND \n3555 <1H OCEAN \n19480 INLAND \n8879 <1H OCEAN \n13685 INLAND \n4937 <1H OCEAN \n4861 <1H OCEAN\nIt’s not arbitrary text: there are a limited number of possible values, each of\nwhich represents a category. So this attribute is a categorical attribute. Most\nMachine Learning algorithms prefer to work with numbers, so let’s convert\nthese categories from text to numbers. For this, we can use Scikit-Learn’s\nOrdinalEncoder class:\n>>> from sklearn.preprocessing import OrdinalEncoder \n>>> ordinal_encoder = OrdinalEncoder() \n>>> housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat) \n>>> housing_cat_encoded[:10] \narray([[0.], \n [0.], \n [4.], \n [1.], \n [0.], \n [1.], \n [0.], \n [1.], \n [0.], \n [0.]])\nYou can get the list of categories using the categories_ instance variable. It\nis a list containing a 1D array of categories for each categorical attribute (in\nthis case, a list containing a single array since there is just one categorical\nattribute):\n>>> ordinal_encoder.categories_ \n[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'], \n dtype=object)]\nOne issue with this representation is that ML algorithms will assume that two\nnearby values are more similar than two distant values. This may be fine in\nsome cases (e.g., for ordered categories such as “bad,” “average,” “good,” and\n19\n“excellent”), but it is obviously not the case for the ocean_proximity column\n(for example, categories 0 and 4 are clearly more similar than categories 0\nand 1). To fix this issue, a common solution is to create one binary attribute\nper category: one attribute equal to 1 when the category is “<1H OCEAN”\n(and 0 otherwise), another attribute equal to 1 when the category is\n“INLAND” (and 0 otherwise), and so on. This is called one-hot encoding,\nbecause only one attribute will be equal to 1 (hot), while the others will be 0\n(cold). The new attributes are sometimes called dummy attributes. Scikit-\nLearn provides a OneHotEncoder class to convert categorical values into one-\nhot vectors:\n>>> from sklearn.preprocessing import OneHotEncoder \n>>> cat_encoder = OneHotEncoder() \n>>> housing_cat_1hot = cat_encoder.fit_transform(housing_cat) \n>>> housing_cat_1hot \n<16512x5 sparse matrix of type '' \n with 16512 stored elements in Compressed Sparse Row format>\nNotice that the output is a SciPy sparse matrix, instead of a NumPy array.\nThis is very useful when you have categorical attributes with thousands of\ncategories. After one-hot encoding, we get a matrix with thousands of\ncolumns, and the matrix is full of 0s except for a single 1 per row. Using up\ntons of memory mostly to store zeros would be very wasteful, so instead a\nsparse matrix only stores the location of the nonzero elements. You can use it\nmostly like a normal 2D array, but if you really want to convert it to a\n(dense) NumPy array, just call the toarray() method:\n>>> housing_cat_1hot.toarray() \narray([[1., 0., 0., 0., 0.], \n [1., 0., 0., 0., 0.], \n [0., 0., 0., 0., 1.], \n ..., \n [0., 1., 0., 0., 0.], \n [1., 0., 0., 0., 0.], \n [0., 0., 0., 1., 0.]])\nOnce again, you can get the list of categories using the encoder’s\ncategories_ instance variable:\n20\n21\n>>> cat_encoder.categories_ \n[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'], \n dtype=object)]\nTIP\nIf a categorical attribute has a large number of possible categories (e.g., country code,\nprofession, species), then one-hot encoding will result in a large number of input\nfeatures. This may slow down training and degrade performance. If this happens, you\nmay want to replace the categorical input with useful numerical features related to the\ncategories: for example, you could replace the ocean_proximity feature with the\ndistance to the ocean (similarly, a country code could be replaced with the country’s\npopulation and GDP per capita). Alternatively, you could replace each category with a\nlearnable, low-dimensional vector called an embedding. Each category’s representation\nwould be learned during training. This is an example of representation learning (see\nChapters 13 and 17 for more details).\nCustom Transformers\nAlthough Scikit-Learn provides many useful transformers, you will need to\nwrite your own for tasks such as custom cleanup operations or combining\nspecific attributes. You will want your transformer to work seamlessly with\nScikit-Learn functionalities (such as pipelines), and since Scikit-Learn relies\non duck typing (not inheritance), all you need to do is create a class and\nimplement three methods: fit() (returning self), transform(), and\nfit_transform().\nYou can get the last one for free by simply adding TransformerMixin as a\nbase class. If you add BaseEstimator as a base class (and avoid *args and\n**kargs in your constructor), you will also get two extra methods\n(get_params() and set_params()) that will be useful for automatic\nhyperparameter tuning.\nFor example, here is a small transformer class that adds the combined\nattributes we discussed earlier:\nfrom sklearn.base import BaseEstimator, TransformerMixin \n \nrooms_ix, bedrooms_ix, population_ix, households_ix = 3, 4, 5, 6 \n \nclass CombinedAttributesAdder(BaseEstimator, TransformerMixin): \n def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs \n self.add_bedrooms_per_room = add_bedrooms_per_room \n def fit(self, X, y=None): \n return self # nothing else to do \n def transform(self, X, y=None): \n rooms_per_household = X[:, rooms_ix] / X[:, households_ix] \n population_per_household = X[:, population_ix] / X[:, households_ix] \n if self.add_bedrooms_per_room: \n bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix] \n return np.c_[X, rooms_per_household, population_per_household, \n bedrooms_per_room] \n \n else: \n return np.c_[X, rooms_per_household, population_per_household] \n \nattr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False) \nhousing_extra_attribs = attr_adder.transform(housing.values)\nIn this example the transformer has one hyperparameter,\nadd_bedrooms_per_room, set to True by default (it is often helpful to provide\nsensible defaults). This hyperparameter will allow you to easily find out\nwhether adding this attribute helps the Machine Learning algorithms or not.\nMore generally, you can add a hyperparameter to gate any data preparation\nstep that you are not 100% sure about. The more you automate these data\npreparation steps, the more combinations you can automatically try out,\nmaking it much more likely that you will find a great combination (and\nsaving you a lot of time).\nFeature Scaling\nOne of the most important transformations you need to apply to your data is\nfeature scaling. With few exceptions, Machine Learning algorithms don’t\nperform well when the input numerical attributes have very different scales.\nThis is the case for the housing data: the total number of rooms ranges from\nabout 6 to 39,320, while the median incomes only range from 0 to 15. Note\nthat scaling the target values is generally not required.\nThere are two common ways to get all attributes to have the same scale: min-\nmax scaling and standardization.\nMin-max scaling (many people call this normalization) is the simplest: values\nare shifted and rescaled so that they end up ranging from 0 to 1. We do this by\nsubtracting the min value and dividing by the max minus the min. Scikit-\nLearn provides a transformer called MinMaxScaler for this. It has a\nfeature_range hyperparameter that lets you change the range if, for some\nreason, you don’t want 0–1.\nStandardization is different: first it subtracts the mean value (so standardized\nvalues always have a zero mean), and then it divides by the standard deviation\nso that the resulting distribution has unit variance. Unlike min-max scaling,\nstandardization does not bound values to a specific range, which may be a\nproblem for some algorithms (e.g., neural networks often expect an input\nvalue ranging from 0 to 1). However, standardization is much less affected by\noutliers. For example, suppose a district had a median income equal to 100\n(by mistake). Min-max scaling would then crush all the other values from 0–\n15 down to 0–0.15, whereas standardization would not be much affected.\nScikit-Learn provides a transformer called StandardScaler for\nstandardization.\nWARNING\nAs with all the transformations, it is important to fit the scalers to the training data only,\nnot to the full dataset (including the test set). Only then can you use them to transform the\ntraining set and the test set (and new data).\nTransformation Pipelines\nAs you can see, there are many data transformation steps that need to be\nexecuted in the right order. Fortunately, Scikit-Learn provides the Pipeline\nclass to help with such sequences of transformations. Here is a small pipeline\nfor the numerical attributes:\nfrom sklearn.pipeline import Pipeline \nfrom sklearn.preprocessing import StandardScaler \n \nnum_pipeline = Pipeline([ \n ('imputer', SimpleImputer(strategy=\"median\")), \n ('attribs_adder', CombinedAttributesAdder()), \n ('std_scaler', StandardScaler()), \n ]) \n \nhousing_num_tr = num_pipeline.fit_transform(housing_num)\nThe Pipeline constructor takes a list of name/estimator pairs defining a\nsequence of steps. All but the last estimator must be transformers (i.e., they\nmust have a fit_transform() method). The names can be anything you like\n(as long as they are unique and don’t contain double underscores, __); they\nwill come in handy later for hyperparameter tuning.\nWhen you call the pipeline’s fit() method, it calls fit_transform()\nsequentially on all transformers, passing the output of each call as the\nparameter to the next call until it reaches the final estimator, for which it calls\nthe fit() method.\nThe pipeline exposes the same methods as the final estimator. In this example,\nthe last estimator is a StandardScaler, which is a transformer, so the\npipeline has a transform() method that applies all the transforms to the data\nin sequence (and of course also a fit_transform() method, which is the one\nwe used).\nSo far, we have handled the categorical columns and the numerical columns\nseparately. It would be more convenient to have a single transformer able to\nhandle all columns, applying the appropriate transformations to each column.\nIn version 0.20, Scikit-Learn introduced the ColumnTransformer for this\npurpose, and the good news is that it works great with pandas DataFrames.\nLet’s use it to apply all the transformations to the housing data:\nfrom sklearn.compose import ColumnTransformer \n \nnum_attribs = list(housing_num) \ncat_attribs = [\"ocean_proximity\"] \n \nfull_pipeline = ColumnTransformer([ \n (\"num\", num_pipeline, num_attribs), \n (\"cat\", OneHotEncoder(), cat_attribs), \n ]) \n \nhousing_prepared = full_pipeline.fit_transform(housing)\nFirst we import the ColumnTransformer class, next we get the list of\nnumerical column names and the list of categorical column names, and then\nwe construct a ColumnTransformer. The constructor requires a list of tuples,\nwhere each tuple contains a name, a transformer, and a list of names (or\nindices) of columns that the transformer should be applied to. In this example,\nwe specify that the numerical columns should be transformed using the\nnum_pipeline that we defined earlier, and the categorical columns should be\ntransformed using a OneHotEncoder. Finally, we apply this\nColumnTransformer to the housing data: it applies each transformer to the\nappropriate columns and concatenates the outputs along the second axis (the\ntransformers must return the same number of rows).\nNote that the OneHotEncoder returns a sparse matrix, while the\nnum_pipeline returns a dense matrix. When there is such a mix of sparse and\ndense matrices, the ColumnTransformer estimates the density of the final\nmatrix (i.e., the ratio of nonzero cells), and it returns a sparse matrix if the\ndensity is lower than a given threshold (by default, sparse_threshold=0.3).\nIn this example, it returns a dense matrix. And that’s it! We have a\npreprocessing pipeline that takes the full housing data and applies the\nappropriate transformations to each column.\nTIP\nInstead of using a transformer, you can specify the string \"drop\" if you want the\ncolumns to be dropped, or you can specify \"passthrough\" if you want the columns to\nbe left untouched. By default, the remaining columns (i.e., the ones that were not listed)\nwill be dropped, but you can set the remainder hyperparameter to any transformer (or to\n\"passthrough\") if you want these columns to be handled differently.\nIf you are using Scikit-Learn 0.19 or earlier, you can use a third-party library\nsuch as sklearn-pandas, or you can roll out your own custom transformer to\nget the same functionality as the ColumnTransformer. Alternatively, you can\nuse the FeatureUnion class, which can apply different transformers and\nconcatenate their outputs. But you cannot specify different columns for each\ntransformer; they all apply to the whole data. It is possible to work around\n22\nthis limitation using a custom transformer for column selection (see the\nJupyter notebook for an example).\nSelect and Train a Model\nAt last! You framed the problem, you got the data and explored it, you\nsampled a training set and a test set, and you wrote transformation pipelines\nto clean up and prepare your data for Machine Learning algorithms\nautomatically. You are now ready to select and train a Machine Learning\nmodel.\nTraining and Evaluating on the Training Set\nThe good news is that thanks to all these previous steps, things are now going\nto be much simpler than you might think. Let’s first train a Linear Regression\nmodel, like we did in the previous chapter:\nfrom sklearn.linear_model import LinearRegression \n \nlin_reg = LinearRegression() \nlin_reg.fit(housing_prepared, housing_labels)\nDone! You now have a working Linear Regression model. Let’s try it out on a\nfew instances from the training set:\n>>> some_data = housing.iloc[:5] \n>>> some_labels = housing_labels.iloc[:5] \n>>> some_data_prepared = full_pipeline.transform(some_data) \n>>> print(\"Predictions:\", lin_reg.predict(some_data_prepared)) \nPredictions: [ 210644.6045 317768.8069 210956.4333 59218.9888 \n189747.5584] \n>>> print(\"Labels:\", list(some_labels)) \nLabels: [286600.0, 340600.0, 196900.0, 46300.0, 254500.0]\nIt works, although the predictions are not exactly accurate (e.g., the first\nprediction is off by close to 40%!). Let’s measure this regression model’s\nRMSE on the whole training set using Scikit-Learn’s mean_squared_error()\nfunction:\n>>> from sklearn.metrics import mean_squared_error \n>>> housing_predictions = lin_reg.predict(housing_prepared) \n>>> lin_mse = mean_squared_error(housing_labels, housing_predictions) \n>>> lin_rmse = np.sqrt(lin_mse) \n>>> lin_rmse \n68628.19819848922\nThis is better than nothing, but clearly not a great score: most districts’\nmedian_housing_values range between $120,000 and $265,000, so a typical\nprediction error of $68,628 is not very satisfying. This is an example of a\nmodel underfitting the training data. When this happens it can mean that the\nfeatures do not provide enough information to make good predictions, or that\nthe model is not powerful enough. As we saw in the previous chapter, the\nmain ways to fix underfitting are to select a more powerful model, to feed the\ntraining algorithm with better features, or to reduce the constraints on the\nmodel. This model is not regularized, which rules out the last option. You\ncould try to add more features (e.g., the log of the population), but first let’s\ntry a more complex model to see how it does.\nLet’s train a DecisionTreeRegressor. This is a powerful model, capable of\nfinding complex nonlinear relationships in the data (Decision Trees are\npresented in more detail in Chapter 6). The code should look familiar by now:\nfrom sklearn.tree import DecisionTreeRegressor \n \ntree_reg = DecisionTreeRegressor() \ntree_reg.fit(housing_prepared, housing_labels)\nNow that the model is trained, let’s evaluate it on the training set:\n>>> housing_predictions = tree_reg.predict(housing_prepared) \n>>> tree_mse = mean_squared_error(housing_labels, housing_predictions) \n>>> tree_rmse = np.sqrt(tree_mse) \n>>> tree_rmse \n0.0\nWait, what!? No error at all? Could this model really be absolutely perfect?\nOf course, it is much more likely that the model has badly overfit the data.\nHow can you be sure? As we saw earlier, you don’t want to touch the test set\nuntil you are ready to launch a model you are confident about, so you need to\nuse part of the training set for training and part of it for model validation.\nBetter Evaluation Using Cross-Validation\nOne way to evaluate the Decision Tree model would be to use the\ntrain_test_split() function to split the training set into a smaller training\nset and a validation set, then train your models against the smaller training set\nand evaluate them against the validation set. It’s a bit of work, but nothing too\ndifficult, and it would work fairly well.\nA great alternative is to use Scikit-Learn’s K-fold cross-validation feature.\nThe following code randomly splits the training set into 10 distinct subsets\ncalled folds, then it trains and evaluates the Decision Tree model 10 times,\npicking a different fold for evaluation every time and training on the other 9\nfolds. The result is an array containing the 10 evaluation scores:\nfrom sklearn.model_selection import cross_val_score \nscores = cross_val_score(tree_reg, housing_prepared, housing_labels, \n scoring=\"neg_mean_squared_error\", cv=10) \ntree_rmse_scores = np.sqrt(-scores)\nWARNING\nScikit-Learn’s cross-validation features expect a utility function (greater is better) rather\nthan a cost function (lower is better), so the scoring function is actually the opposite of\nthe MSE (i.e., a negative value), which is why the preceding code computes -scores\nbefore calculating the square root.\nLet’s look at the results:\n>>> def display_scores(scores): \n... print(\"Scores:\", scores) \n... print(\"Mean:\", scores.mean()) \n... print(\"Standard deviation:\", scores.std()) \n... \n>>> display_scores(tree_rmse_scores) \nScores: [70194.33680785 66855.16363941 72432.58244769 70758.73896782 \n 71115.88230639 75585.14172901 70262.86139133 70273.6325285 \n 75366.87952553 71231.65726027] \nMean: 71407.68766037929 \nStandard deviation: 2439.4345041191004\nNow the Decision Tree doesn’t look as good as it did earlier. In fact, it seems\nto perform worse than the Linear Regression model! Notice that cross-\nvalidation allows you to get not only an estimate of the performance of your\nmodel, but also a measure of how precise this estimate is (i.e., its standard\ndeviation). The Decision Tree has a score of approximately 71,407, generally\n±2,439. You would not have this information if you just used one validation\nset. But cross-validation comes at the cost of training the model several times,\nso it is not always possible.\nLet’s compute the same scores for the Linear Regression model just to be\nsure:\n>>> lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels, \n... scoring=\"neg_mean_squared_error\", cv=10) \n... \n>>> lin_rmse_scores = np.sqrt(-lin_scores) \n>>> display_scores(lin_rmse_scores) \nScores: [66782.73843989 66960.118071 70347.95244419 74739.57052552 \n 68031.13388938 71193.84183426 64969.63056405 68281.61137997 \n 71552.91566558 67665.10082067] \nMean: 69052.46136345083 \nStandard deviation: 2731.674001798348\nThat’s right: the Decision Tree model is overfitting so badly that it performs\nworse than the Linear Regression model.\nLet’s try one last model now: the RandomForestRegressor. As we will see in\nChapter 7, Random Forests work by training many Decision Trees on random\nsubsets of the features, then averaging out their predictions. Building a model\non top of many other models is called Ensemble Learning, and it is often a\ngreat way to push ML algorithms even further. We will skip most of the code\nsince it is essentially the same as for the other models:\n>>> from sklearn.ensemble import RandomForestRegressor \n>>> forest_reg = RandomForestRegressor() \n>>> forest_reg.fit(housing_prepared, housing_labels) \n>>> [...] \n>>> forest_rmse \n18603.515021376355 \n>>> display_scores(forest_rmse_scores) \nScores: [49519.80364233 47461.9115823 50029.02762854 52325.28068953 \n 49308.39426421 53446.37892622 48634.8036574 47585.73832311 \n 53490.10699751 50021.5852922 ] \nMean: 50182.303100336096 \nStandard deviation: 2097.0810550985693\nWow, this is much better: Random Forests look very promising. However,\nnote that the score on the training set is still much lower than on the\nvalidation sets, meaning that the model is still overfitting the training set.\nPossible solutions for overfitting are to simplify the model, constrain it (i.e.,\nregularize it), or get a lot more training data. Before you dive much deeper\ninto Random Forests, however, you should try out many other models from\nvarious categories of Machine Learning algorithms (e.g., several Support\nVector Machines with different kernels, and possibly a neural network),\nwithout spending too much time tweaking the hyperparameters. The goal is to\nshortlist a few (two to five) promising models.\nTIP\nYou should save every model you experiment with so that you can come back easily to\nany model you want. Make sure you save both the hyperparameters and the trained\nparameters, as well as the cross-validation scores and perhaps the actual predictions as\nwell. This will allow you to easily compare scores across model types, and compare the\ntypes of errors they make. You can easily save Scikit-Learn models by using Python’s\npickle module or by using the joblib library, which is more efficient at serializing\nlarge NumPy arrays (you can install this library using pip):\nimport joblib \n \njoblib.dump(my_model, \"my_model.pkl\") \n# and later... \nmy_model_loaded = joblib.load(\"my_model.pkl\")\nFine-Tune Your Model\nLet’s assume that you now have a shortlist of promising models. You now\nneed to fine-tune them. Let’s look at a few ways you can do that.\nGrid Search\nOne option would be to fiddle with the hyperparameters manually, until you\nfind a great combination of hyperparameter values. This would be very\ntedious work, and you may not have time to explore many combinations.\nInstead, you should get Scikit-Learn’s GridSearchCV to search for you. All\nyou need to do is tell it which hyperparameters you want it to experiment with\nand what values to try out, and it will use cross-validation to evaluate all the\npossible combinations of hyperparameter values. For example, the following\ncode searches for the best combination of hyperparameter values for the\nRandomForestRegressor:\nfrom sklearn.model_selection import GridSearchCV \n \nparam_grid = [ \n {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]}, \n {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, \n4]}, \n ] \n \nforest_reg = RandomForestRegressor() \n \ngrid_search = GridSearchCV(forest_reg, param_grid, cv=5, \n scoring='neg_mean_squared_error', \n return_train_score=True) \n \ngrid_search.fit(housing_prepared, housing_labels)\nTIP\nWhen you have no idea what value a hyperparameter should have, a simple approach is\nto try out consecutive powers of 10 (or a smaller number if you want a more fine-grained\nsearch, as shown in this example with the n_estimators hyperparameter).\nThis param_grid tells Scikit-Learn to first evaluate all 3 × 4 = 12\ncombinations of n_estimators and max_features hyperparameter values\nspecified in the first dict (don’t worry about what these hyperparameters\nmean for now; they will be explained in Chapter 7), then try all 2 × 3 = 6\ncombinations of hyperparameter values in the second dict, but this time with\nthe bootstrap hyperparameter set to False instead of True (which is the\ndefault value for this hyperparameter).\nThe grid search will explore 12 + 6 = 18 combinations of\nRandomForestRegressor hyperparameter values, and it will train each model\n5 times (since we are using five-fold cross validation). In other words, all in\nall, there will be 18 × 5 = 90 rounds of training! It may take quite a long time,\nbut when it is done you can get the best combination of parameters like this:\n>>> grid_search.best_params_ \n{'max_features': 8, 'n_estimators': 30}\nTIP\nSince 8 and 30 are the maximum values that were evaluated, you should probably try\nsearching again with higher values; the score may continue to improve.\nYou can also get the best estimator directly:\n>>> grid_search.best_estimator_ \nRandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None, \n max_features=8, max_leaf_nodes=None, min_impurity_decrease=0.0, \n min_impurity_split=None, min_samples_leaf=1, \n min_samples_split=2, min_weight_fraction_leaf=0.0, \n n_estimators=30, n_jobs=None, oob_score=False, random_state=None, \n verbose=0, warm_start=False)\nNOTE\nIf GridSearchCV is initialized with refit=True (which is the default), then once it finds\nthe best estimator using cross-validation, it retrains it on the whole training set. This is\nusually a good idea, since feeding it more data will likely improve its performance.\nAnd of course the evaluation scores are also available:\n>>> cvres = grid_search.cv_results_ \n>>> for mean_score, params in zip(cvres[\"mean_test_score\"], cvres[\"params\"]): \n... print(np.sqrt(-mean_score), params) \n... \n63669.05791727153 {'max_features': 2, 'n_estimators': 3} \n55627.16171305252 {'max_features': 2, 'n_estimators': 10} \n53384.57867637289 {'max_features': 2, 'n_estimators': 30} \n60965.99185930139 {'max_features': 4, 'n_estimators': 3} \n52740.98248528835 {'max_features': 4, 'n_estimators': 10} \n50377.344409590376 {'max_features': 4, 'n_estimators': 30} \n58663.84733372485 {'max_features': 6, 'n_estimators': 3} \n52006.15355973719 {'max_features': 6, 'n_estimators': 10} \n50146.465964159885 {'max_features': 6, 'n_estimators': 30} \n57869.25504027614 {'max_features': 8, 'n_estimators': 3} \n51711.09443660957 {'max_features': 8, 'n_estimators': 10} \n49682.25345942335 {'max_features': 8, 'n_estimators': 30} \n62895.088889905004 {'bootstrap': False, 'max_features': 2, 'n_estimators': 3} \n54658.14484390074 {'bootstrap': False, 'max_features': 2, 'n_estimators': 10} \n59470.399594730654 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3} \n52725.01091081235 {'bootstrap': False, 'max_features': 3, 'n_estimators': 10} \n57490.612956065226 {'bootstrap': False, 'max_features': 4, 'n_estimators': 3} \n51009.51445842374 {'bootstrap': False, 'max_features': 4, 'n_estimators': 10}\nIn this example, we obtain the best solution by setting the max_features\nhyperparameter to 8 and the n_estimators hyperparameter to 30. The RMSE\nscore for this combination is 49,682, which is slightly better than the score\nyou got earlier using the default hyperparameter values (which was 50,182).\nCongratulations, you have successfully fine-tuned your best model!\nTIP\nDon’t forget that you can treat some of the data preparation steps as hyperparameters. For\nexample, the grid search will automatically find out whether or not to add a feature you\nwere not sure about (e.g., using the add_bedrooms_per_room hyperparameter of your\nCombinedAttributesAdder transformer). It may similarly be used to automatically find\nthe best way to handle outliers, missing features, feature selection, and more.\nRandomized Search\nThe grid search approach is fine when you are exploring relatively few\ncombinations, like in the previous example, but when the hyperparameter\nsearch space is large, it is often preferable to use RandomizedSearchCV\ninstead. This class can be used in much the same way as the GridSearchCV\nclass, but instead of trying out all possible combinations, it evaluates a given\nnumber of random combinations by selecting a random value for each\nhyperparameter at every iteration. This approach has two main benefits:\nIf you let the randomized search run for, say, 1,000 iterations, this\napproach will explore 1,000 different values for each hyperparameter\n(instead of just a few values per hyperparameter with the grid search\napproach).\nSimply by setting the number of iterations, you have more control\nover the computing budget you want to allocate to hyperparameter\nsearch.\nEnsemble Methods\nAnother way to fine-tune your system is to try to combine the models that\nperform best. The group (or “ensemble”) will often perform better than the\nbest individual model (just like Random Forests perform better than the\nindividual Decision Trees they rely on), especially if the individual models\nmake very different types of errors. We will cover this topic in more detail in\nChapter 7.\nAnalyze the Best Models and Their Errors\nYou will often gain good insights on the problem by inspecting the best\nmodels. For example, the RandomForestRegressor can indicate the relative\nimportance of each attribute for making accurate predictions:\n>>> feature_importances = grid_search.best_estimator_.feature_importances_ \n>>> feature_importances \narray([7.33442355e-02, 6.29090705e-02, 4.11437985e-02, 1.46726854e-02, \n 1.41064835e-02, 1.48742809e-02, 1.42575993e-02, 3.66158981e-01, \n 5.64191792e-02, 1.08792957e-01, 5.33510773e-02, 1.03114883e-02, \n 1.64780994e-01, 6.02803867e-05, 1.96041560e-03, 2.85647464e-03])\nLet’s display these importance scores next to their corresponding attribute\nnames:\n>>> extra_attribs = [\"rooms_per_hhold\", \"pop_per_hhold\", \"bedrooms_per_room\"] \n>>> cat_encoder = full_pipeline.named_transformers_[\"cat\"] \n>>> cat_one_hot_attribs = list(cat_encoder.categories_[0]) \n_\n_\n_\n(\n_\ng\n_[ ])\n>>> attributes = num_attribs + extra_attribs + cat_one_hot_attribs \n>>> sorted(zip(feature_importances, attributes), reverse=True) \n[(0.3661589806181342, 'median_income'), \n (0.1647809935615905, 'INLAND'), \n (0.10879295677551573, 'pop_per_hhold'), \n (0.07334423551601242, 'longitude'), \n (0.0629090704826203, 'latitude'), \n (0.05641917918195401, 'rooms_per_hhold'), \n (0.05335107734767581, 'bedrooms_per_room'), \n (0.041143798478729635, 'housing_median_age'), \n (0.014874280890402767, 'population'), \n (0.014672685420543237, 'total_rooms'), \n (0.014257599323407807, 'households'), \n (0.014106483453584102, 'total_bedrooms'), \n (0.010311488326303787, '<1H OCEAN'), \n (0.002856474637320158, 'NEAR OCEAN'), \n (0.00196041559947807, 'NEAR BAY'), \n (6.028038672736599e-05, 'ISLAND')]\nWith this information, you may want to try dropping some of the less useful\nfeatures (e.g., apparently only one ocean_proximity category is really useful,\nso you could try dropping the others).\nYou should also look at the specific errors that your system makes, then try to\nunderstand why it makes them and what could fix the problem (adding extra\nfeatures or getting rid of uninformative ones, cleaning up outliers, etc.).\nEvaluate Your System on the Test Set\nAfter tweaking your models for a while, you eventually have a system that\nperforms sufficiently well. Now is the time to evaluate the final model on the\ntest set. There is nothing special about this process; just get the predictors and\nthe labels from your test set, run your full_pipeline to transform the data\n(call transform(), not fit_transform()—you do not want to fit the test\nset!), and evaluate the final model on the test set:\nfinal_model = grid_search.best_estimator_ \n \nX_test = strat_test_set.drop(\"median_house_value\", axis=1) \ny_test = strat_test_set[\"median_house_value\"].copy() \n \nX_test_prepared = full_pipeline.transform(X_test) \n \nfinal_predictions = final_model.predict(X_test_prepared) \n \nfinal_mse = mean_squared_error(y_test, final_predictions) \nfinal_rmse = np.sqrt(final_mse) # => evaluates to 47,730.2\nIn some cases, such a point estimate of the generalization error will not be\nquite enough to convince you to launch: what if it is just 0.1% better than the\nmodel currently in production? You might want to have an idea of how precise\nthis estimate is. For this, you can compute a 95% confidence interval for the\ngeneralization error using scipy.stats.t.interval():\n>>> from scipy import stats \n>>> confidence = 0.95 \n>>> squared_errors = (final_predictions - y_test) ** 2 \n>>> np.sqrt(stats.t.interval(confidence, len(squared_errors) - 1, \n... loc=squared_errors.mean(), \n... scale=stats.sem(squared_errors))) \n... \narray([45685.10470776, 49691.25001878])\nIf you did a lot of hyperparameter tuning, the performance will usually be\nslightly worse than what you measured using cross-validation (because your\nsystem ends up fine-tuned to perform well on the validation data and will\nlikely not perform as well on unknown datasets). It is not the case in this\nexample, but when this happens you must resist the temptation to tweak the\nhyperparameters to make the numbers look good on the test set; the\nimprovements would be unlikely to generalize to new data.\nNow comes the project prelaunch phase: you need to present your solution\n(highlighting what you have learned, what worked and what did not, what\nassumptions were made, and what your system’s limitations are), document\neverything, and create nice presentations with clear visualizations and easy-\nto-remember statements (e.g., “the median income is the number one\npredictor of housing prices”). In this California housing example, the final\nperformance of the system is not better than the experts’ price estimates,\nwhich were often off by about 20%, but it may still be a good idea to launch\nit, especially if this frees up some time for the experts so they can work on\nmore interesting and productive tasks.\nLaunch, Monitor, and Maintain Your System\nPerfect, you got approval to launch! You now need to get your solution ready\nfor production (e.g., polish the code, write documentation and tests, and so\non). Then you can deploy your model to your production environment. One\nway to do this is to save the trained Scikit-Learn model (e.g., using joblib),\nincluding the full preprocessing and prediction pipeline, then load this trained\nmodel within your production environment and use it to make predictions by\ncalling its predict() method. For example, perhaps the model will be used\nwithin a website: the user will type in some data about a new district and click\nthe Estimate Price button. This will send a query containing the data to the\nweb server, which will forward it to your web application, and finally your\ncode will simply call the model’s predict() method (you want to load the\nmodel upon server startup, rather than every time the model is used).\nAlternatively, you can wrap the model within a dedicated web service that\nyour web application can query through a REST API\n (see Figure 2-17). This\nmakes it easier to upgrade your model to new versions without interrupting\nthe main application. It also simplifies scaling, since you can start as many\nweb services as needed and load-balance the requests coming from your web\napplication across these web services. Moreover, it allows your web\napplication to use any language, not just Python.\nFigure 2-17. A model deployed as a web service and used by a web application\nAnother popular strategy is to deploy your model on the cloud, for example\non Google Cloud AI Platform (formerly known as Google Cloud ML Engine):\njust save your model using joblib and upload it to Google Cloud Storage\n(GCS), then head over to Google Cloud AI Platform and create a new model\nversion, pointing it to the GCS file. That’s it! This gives you a simple web\nservice that takes care of load balancing and scaling for you. It take JSON\nrequests containing the input data (e.g., of a district) and returns JSON\nresponses containing the predictions. You can then use this web service in\n23\nyour website (or whatever production environment you are using). As we will\nsee in Chapter 19, deploying TensorFlow models on AI Platform is not much\ndifferent from deploying Scikit-Learn models.\nBut deployment is not the end of the story. You also need to write monitoring\ncode to check your system’s live performance at regular intervals and trigger\nalerts when it drops. This could be a steep drop, likely due to a broken\ncomponent in your infrastructure, but be aware that it could also be a gentle\ndecay that could easily go unnoticed for a long time. This is quite common\nbecause models tend to “rot” over time: indeed, the world changes, so if the\nmodel was trained with last year’s data, it may not be adapted to today’s data.\nWARNING\nEven a model trained to classify pictures of cats and dogs may need to be retrained\nregularly, not because cats and dogs will mutate overnight, but because cameras keep\nchanging, along with image formats, sharpness, brightness, and size ratios. Moreover,\npeople may love different breeds next year, or they may decide to dress their pets with\ntiny hats—who knows?\nSo you need to monitor your model’s live performance. But how do you that?\nWell, it depends. In some cases, the model’s performance can be inferred from\ndownstream metrics. For example, if your model is part of a recommender\nsystem and it suggests products that the users may be interested in, then it’s\neasy to monitor the number of recommended products sold each day. If this\nnumber drops (compared to non-recommended products), then the prime\nsuspect is the model. This may be because the data pipeline is broken, or\nperhaps the model needs to be retrained on fresh data (as we will discuss\nshortly).\nHowever, it’s not always possible to determine the model’s performance\nwithout any human analysis. For example, suppose you trained an image\nclassification model (see Chapter 3) to detect several product defects on a\nproduction line. How can you get an alert if the model’s performance drops,\nbefore thousands of defective products get shipped to your clients? One\nsolution is to send to human raters a sample of all the pictures that the model\nclassified (especially pictures that the model wasn’t so sure about).\nDepending on the task, the raters may need to be experts, or they could be\nnonspecialists, such as workers on a crowdsourcing platform (e.g., Amazon\nMechanical Turk). In some applications they could even be the users\nthemselves, responding for example via surveys or repurposed captchas.\nEither way, you need to put in place a monitoring system (with or without\nhuman raters to evaluate the live model), as well as all the relevant processes\nto define what to do in case of failures and how to prepare for them.\nUnfortunately, this can be a lot of work. In fact, it is often much more work\nthan building and training a model.\nIf the data keeps evolving, you will need to update your datasets and retrain\nyour model regularly. You should probably automate the whole process as\nmuch as possible. Here are a few things you can automate:\nCollect fresh data regularly and label it (e.g., using human raters).\nWrite a script to train the model and fine-tune the hyperparameters\nautomatically. This script could run automatically, for example every\nday or every week, depending on your needs.\nWrite another script that will evaluate both the new model and the\nprevious model on the updated test set, and deploy the model to\nproduction if the performance has not decreased (if it did, make sure\nyou investigate why).\nYou should also make sure you evaluate the model’s input data quality.\nSometimes performance will degrade slightly because of a poor-quality signal\n(e.g., a malfunctioning sensor sending random values, or another team’s\noutput becoming stale), but it may take a while before your system’s\nperformance degrades enough to trigger an alert. If you monitor your model’s\ninputs, you may catch this earlier. For example, you could trigger an alert if\nmore and more inputs are missing a feature, or if its mean or standard\ndeviation drifts too far from the training set, or a categorical feature starts\ncontaining new categories.\nFinally, make sure you keep backups of every model you create and have the\nprocess and tools in place to roll back to a previous model quickly, in case the\nnew model starts failing badly for some reason. Having backups also makes it\n24\npossible to easily compare new models with previous ones. Similarly, you\nshould keep backups of every version of your datasets so that you can roll\nback to a previous dataset if the new one ever gets corrupted (e.g., if the fresh\ndata that gets added to it turns out to be full of outliers). Having backups of\nyour datasets also allows you to evaluate any model against any previous\ndataset.\nTIP\nYou may want to create several subsets of the test set in order to evaluate how well your\nmodel performs on specific parts of the data. For example, you may want to have a\nsubset containing only the most recent data, or a test set for specific kinds of inputs (e.g.,\ndistricts located inland versus districts located near the ocean). This will give you a\ndeeper understanding of your model’s strengths and weaknesses.\nAs you can see, Machine Learning involves quite a lot of infrastructure, so\ndon’t be surprised if your first ML project takes a lot of effort and time to\nbuild and deploy to production. Fortunately, once all the infrastructure is in\nplace, going from idea to production will be much faster.\nTry It Out!\nHopefully this chapter gave you a good idea of what a Machine Learning\nproject looks like as well as showing you some of the tools you can use to\ntrain a great system. As you can see, much of the work is in the data\npreparation step: building monitoring tools, setting up human evaluation\npipelines, and automating regular model training. The Machine Learning\nalgorithms are important, of course, but it is probably preferable to be\ncomfortable with the overall process and know three or four algorithms well\nrather than to spend all your time exploring advanced algorithms.\nSo, if you have not already done so, now is a good time to pick up a laptop,\nselect a dataset that you are interested in, and try to go through the whole\nprocess from A to Z. A good place to start is on a competition website such as\nhttp://kaggle.com/: you will have a dataset to play with, a clear goal, and\npeople to share the experience with. Have fun!\nExercises\nThe following exercises are all based on this chapter’s housing dataset:\n1. Try a Support Vector Machine regressor (sklearn.svm.SVR) with\nvarious hyperparameters, such as kernel=\"linear\" (with various\nvalues for the C hyperparameter) or kernel=\"rbf\" (with various\nvalues for the C and gamma hyperparameters). Don’t worry about what\nthese hyperparameters mean for now. How does the best SVR\npredictor perform?\n2. Try replacing GridSearchCV with RandomizedSearchCV.\n3. Try adding a transformer in the preparation pipeline to select only the\nmost important attributes.\n4. Try creating a single pipeline that does the full data preparation plus\nthe final prediction.\n5. Automatically explore some preparation options using\nGridSearchCV.\nSolutions to these exercises can be found in the Jupyter notebooks available at\nhttps://github.com/ageron/handson-ml2.\n1 The example project is fictitious; the goal is to illustrate the main steps of a Machine Learning\nproject, not to learn anything about the real estate business.\n2 The original dataset appeared in R. Kelley Pace and Ronald Barry, “Sparse Spatial\nAutoregressions,” Statistics & Probability Letters 33, no. 3 (1997): 291–297.\n3 A piece of information fed to a Machine Learning system is often called a signal, in reference\nto Claude Shannon’s information theory, which he developed at Bell Labs to improve\ntelecommunications. His theory: you want a high signal-to-noise ratio.\n4 Recall that the transpose operator flips a column vector into a row vector (and vice versa).\n5 The latest version of Python 3 is recommended. Python 2.7+ may work too, but now that it’s\ndeprecated, all major scientific libraries are dropping support for it, so you should migrate to\nPython 3 as soon as possible.\n6 I’ll show the installation steps using pip in a bash shell on a Linux or macOS system. You may\nneed to adapt these commands to your own system. On Windows, I recommend installing\nAnaconda instead.\n7 If you want to upgrade pip for all users on your machine rather than just your own user, you\nshould remove the --user option and make sure you have administrator rights (e.g., by adding\nsudo before the whole command on Linux or macOS).\n8 Alternative tools include venv (very similar to virtualenv and included in the standard library),\nvirtualenvwrapper (provides extra functionalities on top of virtualenv), pyenv (allows easy\nswitching between Python versions), and pipenv (a great packaging tool by the same author as\nthe popular requests library, built on top of pip and virtualenv).\n9 Note that Jupyter can handle multiple versions of Python, and even many other languages\nsuch as R or Octave.\n10 You might also need to check legal constraints, such as private fields that should never be\ncopied to unsafe data stores.\n11 In a real project you would save this code in a Python file, but for now you can just write it in\nyour Jupyter notebook.\n12 The standard deviation is generally denoted σ (the Greek letter sigma), and it is the square root\nof the variance, which is the average of the squared deviation from the mean. When a feature\nhas a bell-shaped normal distribution (also called a Gaussian distribution), which is very\ncommon, the “68-95-99.7” rule applies: about 68% of the values fall within 1σ of the mean,\n95% within 2σ, and 99.7% within 3σ.\n13 In this book, when a code example contains a mix of code and outputs, as is the case here, it is\nformatted like in the Python interpreter, for better readability: the code lines are prefixed with\n>>> (or ... for indented blocks), and the outputs have no prefix.\n14 You will often see people set the random seed to 42. This number has no special property,\nother than to be the Answer to the Ultimate Question of Life, the Universe, and Everything.\n15 The location information is actually quite coarse, and as a result many districts will have the\nexact same ID, so they will end up in the same set (test or train). This introduces some\nunfortunate sampling bias.\n16 If you are reading this in grayscale, grab a red pen and scribble over most of the coastline from\nthe Bay Area down to San Diego (as you might expect). You can add a patch of yellow around\nSacramento as well.\n17 For more details on the design principles, see Lars Buitinck et al., “API Design for Machine\nLearning Software: Experiences from the Scikit-Learn Project” ,” arXiv preprint\narXiv:1309.0238 (2013).\n18 Some predictors also provide methods to measure the confidence of their predictions.\n19 This class is available in Scikit-Learn 0.20 and later. If you use an earlier version, please\nconsider upgrading, or use the pandas Series.factorize() method.\n20 Before Scikit-Learn 0.20, the method could only encode integer categorical values, but since\n0.20 it can also handle other types of inputs, including text categorical inputs.\n21 See SciPy’s documentation for more details.\n22 Just like for pipelines, the name can be anything as long as it does not contain double\nunderscores.\n23 In a nutshell, a REST (or RESTful) API is an HTTP-based API that follows some conventions,\nsuch as using standard HTTP verbs to read, update, create, or delete resources (GET, POST,\nPUT, and DELETE) and using JSON for the inputs and outputs.\n24 A captcha is a test to ensure a user is not a robot. These tests have often been used as a cheap\nway to label training data." + }, + { + "chapter": 3, + "text": "Chapter 3. Classification\nIn Chapter 1 I mentioned that the most common supervised learning tasks\nare regression (predicting values) and classification (predicting classes).\nIn Chapter 2 we explored a regression task, predicting housing values,\nusing various algorithms such as Linear Regression, Decision Trees, and\nRandom Forests (which will be explained in further detail in later\nchapters). Now we will turn our attention to classification systems.\nMNIST\nIn this chapter we will be using the MNIST dataset, which is a set of\n70,000 small images of digits handwritten by high school students and\nemployees of the US Census Bureau. Each image is labeled with the digit\nit represents. This set has been studied so much that it is often called the\n“hello world” of Machine Learning: whenever people come up with a new\nclassification algorithm they are curious to see how it will perform on\nMNIST, and anyone who learns Machine Learning tackles this dataset\nsooner or later.\nScikit-Learn provides many helper functions to download popular\ndatasets. MNIST is one of them. The following code fetches the MNIST\ndataset:\n>>> from sklearn.datasets import fetch_openml \n>>> mnist = fetch_openml('mnist_784', version=1) \n>>> mnist.keys() \ndict_keys(['data', 'target', 'feature_names', 'DESCR', 'details', \n 'categories', 'url'])\nDatasets loaded by Scikit-Learn generally have a similar dictionary\nstructure, including the following:\nA DESCR key describing the dataset\n1\nA data key containing an array with one row per instance and one\ncolumn per feature\nA target key containing an array with the labels\nLet’s look at these arrays:\n>>> X, y = mnist[\"data\"], mnist[\"target\"] \n>>> X.shape \n(70000, 784) \n>>> y.shape \n(70000,)\nThere are 70,000 images, and each image has 784 features. This is because\neach image is 28 × 28 pixels, and each feature simply represents one\npixel’s intensity, from 0 (white) to 255 (black). Let’s take a peek at one\ndigit from the dataset. All you need to do is grab an instance’s feature\nvector, reshape it to a 28 × 28 array, and display it using Matplotlib’s\nimshow() function:\nimport matplotlib as mpl \nimport matplotlib.pyplot as plt \n \nsome_digit = X[0] \nsome_digit_image = some_digit.reshape(28, 28) \n \nplt.imshow(some_digit_image, cmap=\"binary\") \nplt.axis(\"off\") \nplt.show()\nThis looks like a 5, and indeed that’s what the label tells us:\n>>> y[0] \n'5'\nNote that the label is a string. Most ML algorithms expect numbers, so\nlet’s cast y to integer:\n>>> y = y.astype(np.uint8)\nTo give you a feel for the complexity of the classification task, Figure 3-1\nshows a few more images from the MNIST dataset.\nFigure 3-1. Digits from the MNIST dataset\nBut wait! You should always create a test set and set it aside before\ninspecting the data closely. The MNIST dataset is actually already split\ninto a training set (the first 60,000 images) and a test set (the last 10,000\nimages):\nX_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], \ny[60000:]\nThe training set is already shuffled for us, which is good because this\nguarantees that all cross-validation folds will be similar (you don’t want\none fold to be missing some digits). Moreover, some learning algorithms\nare sensitive to the order of the training instances, and they perform poorly\nif they get many similar instances in a row. Shuffling the dataset ensures\nthat this won’t happen.\nTraining a Binary Classifier\nLet’s simplify the problem for now and only try to identify one digit—for\nexample, the number 5. This “5-detector” will be an example of a binary\nclassifier, capable of distinguishing between just two classes, 5 and not-5.\nLet’s create the target vectors for this classification task:\ny_train_5 = (y_train == 5) # True for all 5s, False for all other digits \ny_test_5 = (y_test == 5)\nNow let’s pick a classifier and train it. A good place to start is with a\nStochastic Gradient Descent (SGD) classifier, using Scikit-Learn’s\nSGDClassifier class. This classifier has the advantage of being capable\nof handling very large datasets efficiently. This is in part because SGD\ndeals with training instances independently, one at a time (which also\nmakes SGD well suited for online learning), as we will see later. Let’s\ncreate an SGDClassifier and train it on the whole training set:\nfrom sklearn.linear_model import SGDClassifier \n \nsgd_clf = SGDClassifier(random_state=42) \nsgd_clf.fit(X_train, y_train_5)\nTIP\nThe SGDClassifier relies on randomness during training (hence the name\n“stochastic”). If you want reproducible results, you should set the random_state\nparameter.\n2\nNow we can use it to detect images of the number 5:\n>>> sgd_clf.predict([some_digit]) \narray([ True])\nThe classifier guesses that this image represents a 5 (True). Looks like it\nguessed right in this particular case! Now, let’s evaluate this model’s\nperformance.\nPerformance Measures\nEvaluating a classifier is often significantly trickier than evaluating a\nregressor, so we will spend a large part of this chapter on this topic. There\nare many performance measures available, so grab another coffee and get\nready to learn many new concepts and acronyms!\nMeasuring Accuracy Using Cross-Validation\nA good way to evaluate a model is to use cross-validation, just as you did\nin Chapter 2.\nIMPLEMENTING CROSS-VALIDATION\nOccasionally you will need more control over the cross-validation\nprocess than what Scikit-Learn provides off the shelf. In these cases,\nyou can implement cross-validation yourself. The following code does\nroughly the same thing as Scikit-Learn’s cross_val_score()\nfunction, and it prints the same result:\nfrom sklearn.model_selection import StratifiedKFold \nfrom sklearn.base import clone \n \nskfolds = StratifiedKFold(n_splits=3, random_state=42) \n \nfor train_index, test_index in skfolds.split(X_train, y_train_5): \n clone_clf = clone(sgd_clf) \n X_train_folds = X_train[train_index] \n y_train_folds = y_train_5[train_index] \n X_test_fold = X_train[test_index] \n y_test_fold = y_train_5[test_index] \n \n clone_clf.fit(X_train_folds, y_train_folds) \n y_pred = clone_clf.predict(X_test_fold) \n n_correct = sum(y_pred == y_test_fold) \n print(n_correct / len(y_pred)) # prints 0.9502, 0.96565, and \n0.96495\nThe StratifiedKFold class performs stratified sampling (as\nexplained in Chapter 2) to produce folds that contain a representative\nratio of each class. At each iteration the code creates a clone of the\nclassifier, trains that clone on the training folds, and makes predictions\non the test fold. Then it counts the number of correct predictions and\noutputs the ratio of correct predictions.\nLet’s use the cross_val_score() function to evaluate our\nSGDClassifier model, using K-fold cross-validation with three folds.\nRemember that K-fold cross-validation means splitting the training set\ninto K folds (in this case, three), then making predictions and evaluating\nthem on each fold using a model trained on the remaining folds (see\nChapter 2):\n>>> from sklearn.model_selection import cross_val_score \n>>> cross_val_score(sgd_clf, X_train, y_train_5, cv=3, \nscoring=\"accuracy\") \narray([0.96355, 0.93795, 0.95615])\nWow! Above 93% accuracy (ratio of correct predictions) on all cross-\nvalidation folds? This looks amazing, doesn’t it? Well, before you get too\nexcited, let’s look at a very dumb classifier that just classifies every single\nimage in the “not-5” class:\nfrom sklearn.base import BaseEstimator \n \nclass Never5Classifier(BaseEstimator): \n def fit(self, X, y=None): \n pass \n def predict(self, X): \n return np.zeros((len(X), 1), dtype=bool)\nCan you guess this model’s accuracy? Let’s find out:\n>>> never_5_clf = Never5Classifier() \n>>> cross_val_score(never_5_clf, X_train, y_train_5, cv=3, \nscoring=\"accuracy\") \narray([0.91125, 0.90855, 0.90915])\nThat’s right, it has over 90% accuracy! This is simply because only about\n10% of the images are 5s, so if you always guess that an image is not a 5,\nyou will be right about 90% of the time. Beats Nostradamus.\nThis demonstrates why accuracy is generally not the preferred\nperformance measure for classifiers, especially when you are dealing with\nskewed datasets (i.e., when some classes are much more frequent than\nothers).\nConfusion Matrix\nA much better way to evaluate the performance of a classifier is to look at\nthe confusion matrix. The general idea is to count the number of times\ninstances of class A are classified as class B. For example, to know the\nnumber of times the classifier confused images of 5s with 3s, you would\nlook in the fifth row and third column of the confusion matrix.\nTo compute the confusion matrix, you first need to have a set of\npredictions so that they can be compared to the actual targets. You could\nmake predictions on the test set, but let’s keep it untouched for now\n(remember that you want to use the test set only at the very end of your\nproject, once you have a classifier that you are ready to launch). Instead,\nyou can use the cross_val_predict() function:\nfrom sklearn.model_selection import cross_val_predict \n \ny_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)\nJust like the cross_val_score() function, cross_val_predict()\nperforms K-fold cross-validation, but instead of returning the evaluation\nscores, it returns the predictions made on each test fold. This means that\nyou get a clean prediction for each instance in the training set (“clean”\nmeaning that the prediction is made by a model that never saw the data\nduring training).\nNow you are ready to get the confusion matrix using the\nconfusion_matrix() function. Just pass it the target classes (y_train_5)\nand the predicted classes (y_train_pred):\n>>> from sklearn.metrics import confusion_matrix \n>>> confusion_matrix(y_train_5, y_train_pred) \narray([[53057, 1522], \n [ 1325, 4096]])\nEach row in a confusion matrix represents an actual class, while each\ncolumn represents a predicted class. The first row of this matrix considers\nnon-5 images (the negative class): 53,057 of them were correctly\nclassified as non-5s (they are called true negatives), while the remaining\n1,522 were wrongly classified as 5s (false positives). The second row\nconsiders the images of 5s (the positive class): 1,325 were wrongly\nclassified as non-5s (false negatives), while the remaining 4,096 were\ncorrectly classified as 5s (true positives). A perfect classifier would have\nonly true positives and true negatives, so its confusion matrix would have\nnonzero values only on its main diagonal (top left to bottom right):\n>>> y_train_perfect_predictions = y_train_5 # pretend we reached \nperfection \n>>> confusion_matrix(y_train_5, y_train_perfect_predictions) \narray([[54579, 0], \n [ 0, 5421]])\nThe confusion matrix gives you a lot of information, but sometimes you\nmay prefer a more concise metric. An interesting one to look at is the\naccuracy of the positive predictions; this is called the precision of the\nclassifier (Equation 3-1).\nEquation 3-1. Precision\nprecision =\nTP is the number of true positives, and FP is the number of false positives.\nA trivial way to have perfect precision is to make one single positive\nprediction and ensure it is correct (precision = 1/1 = 100%). But this\nwould not be very useful, since the classifier would ignore all but one\npositive instance. So precision is typically used along with another metric\nnamed recall, also called sensitivity or the true positive rate (TPR): this is\nthe ratio of positive instances that are correctly detected by the classifier\n(Equation 3-2).\nEquation 3-2. Recall\nrecall =\nTP\nTP + FP\nTP\nTP + FN\nFN is, of course, the number of false negatives.\nIf you are confused about the confusion matrix, Figure 3-2 may help.\nFigure 3-2. An illustrated confusion matrix shows examples of true negatives (top left), false\npositives (top right), false negatives (lower left), and true positives (lower right)\nPrecision and Recall\nScikit-Learn provides several functions to compute classifier metrics,\nincluding precision and recall:\n>>> from sklearn.metrics import precision_score, recall_score \n>>> precision_score(y_train_5, y_train_pred) # == 4096 / (4096 + 1522) \n0.7290850836596654 \n>>> recall_score(y_train_5, y_train_pred) # == 4096 / (4096 + 1325) \n0.7555801512636044\nNow your 5-detector does not look as shiny as it did when you looked at\nits accuracy. When it claims an image represents a 5, it is correct only\n72.9% of the time. Moreover, it only detects 75.6% of the 5s.\nIt is often convenient to combine precision and recall into a single metric\ncalled the F score, in particular if you need a simple way to compare two\nclassifiers. The F score is the harmonic mean of precision and recall\n(Equation 3-3). Whereas the regular mean treats all values equally, the\n1\n1\nharmonic mean gives much more weight to low values. As a result, the\nclassifier will only get a high F score if both recall and precision are high.\nEquation 3-3. F\nF1 =\n= 2 ×\n=\nTo compute the F score, simply call the f1_score() function:\n>>> from sklearn.metrics import f1_score \n>>> f1_score(y_train_5, y_train_pred) \n0.7420962043663375\nThe F score favors classifiers that have similar precision and recall. This\nis not always what you want: in some contexts you mostly care about\nprecision, and in other contexts you really care about recall. For example,\nif you trained a classifier to detect videos that are safe for kids, you would\nprobably prefer a classifier that rejects many good videos (low recall) but\nkeeps only safe ones (high precision), rather than a classifier that has a\nmuch higher recall but lets a few really bad videos show up in your\nproduct (in such cases, you may even want to add a human pipeline to\ncheck the classifier’s video selection). On the other hand, suppose you\ntrain a classifier to detect shoplifters in surveillance images: it is probably\nfine if your classifier has only 30% precision as long as it has 99% recall\n(sure, the security guards will get a few false alerts, but almost all\nshoplifters will get caught).\nUnfortunately, you can’t have it both ways: increasing precision reduces\nrecall, and vice versa. This is called the precision/recall trade-off.\nPrecision/Recall Trade-off\nTo understand this trade-off, let’s look at how the SGDClassifier makes\nits classification decisions. For each instance, it computes a score based on\na decision function. If that score is greater than a threshold, it assigns the\n1\n1\n2\n+\n1\nprecision\n1\nrecall\nprecision × recall\nprecision + recall\nTP\nTP + FN+FP\n2\n1\n1\ninstance to the positive class; otherwise it assigns it to the negative class.\nFigure 3-3 shows a few digits positioned from the lowest score on the left\nto the highest score on the right. Suppose the decision threshold is\npositioned at the central arrow (between the two 5s): you will find 4 true\npositives (actual 5s) on the right of that threshold, and 1 false positive\n(actually a 6). Therefore, with that threshold, the precision is 80% (4 out\nof 5). But out of 6 actual 5s, the classifier only detects 4, so the recall is\n67% (4 out of 6). If you raise the threshold (move it to the arrow on the\nright), the false positive (the 6) becomes a true negative, thereby\nincreasing the precision (up to 100% in this case), but one true positive\nbecomes a false negative, decreasing recall down to 50%. Conversely,\nlowering the threshold increases recall and reduces precision.\nFigure 3-3. In this precision/recall trade-off, images are ranked by their classifier score, and\nthose above the chosen decision threshold are considered positive; the higher the threshold, the\nlower the recall, but (in general) the higher the precision\nScikit-Learn does not let you set the threshold directly, but it does give\nyou access to the decision scores that it uses to make predictions. Instead\nof calling the classifier’s predict() method, you can call its\ndecision_function() method, which returns a score for each instance,\nand then use any threshold you want to make predictions based on those\nscores:\n>>> y_scores = sgd_clf.decision_function([some_digit]) \n>>> y_scores \narray([2412.53175101]) \n>>> threshold = 0 \n>>> y_some_digit_pred = (y_scores > threshold) \narray([ True])\nThe SGDClassifier uses a threshold equal to 0, so the previous code\nreturns the same result as the predict() method (i.e., True). Let’s raise\nthe threshold:\n>>> threshold = 8000 \n>>> y_some_digit_pred = (y_scores > threshold) \n>>> y_some_digit_pred \narray([False])\nThis confirms that raising the threshold decreases recall. The image\nactually represents a 5, and the classifier detects it when the threshold is 0,\nbut it misses it when the threshold is increased to 8,000.\nHow do you decide which threshold to use? First, use the\ncross_val_predict() function to get the scores of all instances in the\ntraining set, but this time specify that you want to return decision scores\ninstead of predictions:\ny_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, \n method=\"decision_function\")\nWith these scores, use the precision_recall_curve() function to\ncompute precision and recall for all possible thresholds:\nfrom sklearn.metrics import precision_recall_curve \n \nprecisions, recalls, thresholds = precision_recall_curve(y_train_5, \ny_scores)\nFinally, use Matplotlib to plot precision and recall as functions of the\nthreshold value (Figure 3-4):\ndef plot_precision_recall_vs_threshold(precisions, recalls, thresholds): \n plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\") \n plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\") \n [...] # highlight the threshold and add the legend, axis label, and \ngrid \n \nplot_precision_recall_vs_threshold(precisions, recalls, thresholds) \nplt.show()\nFigure 3-4. Precision and recall versus the decision threshold\nNOTE\nYou may wonder why the precision curve is bumpier than the recall curve in\nFigure 3-4. The reason is that precision may sometimes go down when you raise the\nthreshold (although in general it will go up). To understand why, look back at\nFigure 3-3 and notice what happens when you start from the central threshold and\nmove it just one digit to the right: precision goes from 4/5 (80%) down to 3/4 (75%).\nOn the other hand, recall can only go down when the threshold is increased, which\nexplains why its curve looks smooth.\nAnother way to select a good precision/recall trade-off is to plot precision\ndirectly against recall, as shown in Figure 3-5 (the same threshold as\nearlier is highlighted).\nFigure 3-5. Precision versus recall\nYou can see that precision really starts to fall sharply around 80% recall.\nYou will probably want to select a precision/recall trade-off just before\nthat drop—for example, at around 60% recall. But of course, the choice\ndepends on your project.\nSuppose you decide to aim for 90% precision. You look up the first plot\nand find that you need to use a threshold of about 8,000. To be more\nprecise you can search for the lowest threshold that gives you at least 90%\nprecision (np.argmax() will give you the first index of the maximum\nvalue, which in this case means the first True value):\nthreshold_90_precision = thresholds[np.argmax(precisions >= 0.90)] # \n~7816\nTo make predictions (on the training set for now), instead of calling the\nclassifier’s predict() method, you can run this code:\ny_train_pred_90 = (y_scores >= threshold_90_precision)\nLet’s check these predictions’ precision and recall:\n>>> precision_score(y_train_5, y_train_pred_90) \n0.9000380083618396 \n>>> recall_score(y_train_5, y_train_pred_90) \n0.4368197749492714\nGreat, you have a 90% precision classifier! As you can see, it is fairly easy\nto create a classifier with virtually any precision you want: just set a high\nenough threshold, and you’re done. But wait, not so fast. A high-precision\nclassifier is not very useful if its recall is too low!\nTIP\nIf someone says, “Let’s reach 99% precision,” you should ask, “At what recall?”\nThe ROC Curve\nThe receiver operating characteristic (ROC) curve is another common\ntool used with binary classifiers. It is very similar to the precision/recall\ncurve, but instead of plotting precision versus recall, the ROC curve plots\nthe true positive rate (another name for recall) against the false positive\nrate (FPR). The FPR is the ratio of negative instances that are incorrectly\nclassified as positive. It is equal to 1 – the true negative rate (TNR), which\nis the ratio of negative instances that are correctly classified as negative.\nThe TNR is also called specificity. Hence, the ROC curve plots sensitivity\n(recall) versus 1 – specificity.\nTo plot the ROC curve, you first use the roc_curve() function to compute\nthe TPR and FPR for various threshold values:\nfrom sklearn.metrics import roc_curve \n \nfpr, tpr, thresholds = roc_curve(y_train_5, y_scores)\nThen you can plot the FPR against the TPR using Matplotlib. This code\nproduces the plot in Figure 3-6:\ndef plot_roc_curve(fpr, tpr, label=None): \n plt.plot(fpr, tpr, linewidth=2, label=label) \n plt.plot([0, 1], [0, 1], 'k--') # Dashed diagonal \n [...] # Add axis labels and grid \n \nplot_roc_curve(fpr, tpr) \nplt.show()\nOnce again there is a trade-off: the higher the recall (TPR), the more false\npositives (FPR) the classifier produces. The dotted line represents the\nROC curve of a purely random classifier; a good classifier stays as far\naway from that line as possible (toward the top-left corner).\nFigure 3-6. This ROC curve plots the false positive rate against the true positive rate for all\npossible thresholds; the red circle highlights the chosen ratio (at 43.68% recall)\nOne way to compare classifiers is to measure the area under the curve\n(AUC). A perfect classifier will have a ROC AUC equal to 1, whereas a\npurely random classifier will have a ROC AUC equal to 0.5. Scikit-Learn\nprovides a function to compute the ROC AUC:\n>>> from sklearn.metrics import roc_auc_score \n>>> roc_auc_score(y_train_5, y_scores) \n0.9611778893101814\nTIP\nSince the ROC curve is so similar to the precision/recall (PR) curve, you may wonder\nhow to decide which one to use. As a rule of thumb, you should prefer the PR curve\nwhenever the positive class is rare or when you care more about the false positives\nthan the false negatives. Otherwise, use the ROC curve. For example, looking at the\nprevious ROC curve (and the ROC AUC score), you may think that the classifier is\nreally good. But this is mostly because there are few positives (5s) compared to the\nnegatives (non-5s). In contrast, the PR curve makes it clear that the classifier has\nroom for improvement (the curve could be closer to the top-left corner).\nLet’s now train a RandomForestClassifier and compare its ROC curve\nand ROC AUC score to those of the SGDClassifier. First, you need to get\nscores for each instance in the training set. But due to the way it works\n(see Chapter 7), the RandomForestClassifier class does not have a\ndecision_function() method. Instead, it has a predict_proba()\nmethod. Scikit-Learn classifiers generally have one or the other, or both.\nThe predict_proba() method returns an array containing a row per\ninstance and a column per class, each containing the probability that the\ngiven instance belongs to the given class (e.g., 70% chance that the image\nrepresents a 5):\nfrom sklearn.ensemble import RandomForestClassifier \n \nforest_clf = RandomForestClassifier(random_state=42) \ny_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, \n method=\"predict_proba\")\nThe roc_curve() function expects labels and scores, but instead of scores\nyou can give it class probabilities. Let’s use the positive class’s probability\nas the score:\ny_scores_forest = y_probas_forest[:, 1] # score = proba of positive \nclass \nfpr_forest, tpr_forest, thresholds_forest = \nroc_curve(y_train_5,y_scores_forest)\nNow you are ready to plot the ROC curve. It is useful to plot the first ROC\ncurve as well to see how they compare (Figure 3-7):\nplt.plot(fpr, tpr, \"b:\", label=\"SGD\") \nplot_roc_curve(fpr_forest, tpr_forest, \"Random Forest\") \nplt.legend(loc=\"lower right\") \nplt.show()\nFigure 3-7. Comparing ROC curves: the Random Forest classifier is superior to the SGD\nclassifier because its ROC curve is much closer to the top-left corner, and it has a greater AUC\nAs you can see in Figure 3-7, the RandomForestClassifier’s ROC curve\nlooks much better than the SGDClassifier’s: it comes much closer to the\ntop-left corner. As a result, its ROC AUC score is also significantly better:\n>>> roc_auc_score(y_train_5, y_scores_forest) \n0.9983436731328145\nTry measuring the precision and recall scores: you should find 99.0%\nprecision and 86.6% recall. Not too bad!\nYou now know how to train binary classifiers, choose the appropriate\nmetric for your task, evaluate your classifiers using cross-validation,\nselect the precision/recall trade-off that fits your needs, and use ROC\ncurves and ROC AUC scores to compare various models. Now let’s try to\ndetect more than just the 5s.\nMulticlass Classification\nWhereas binary classifiers distinguish between two classes, multiclass\nclassifiers (also called multinomial classifiers) can distinguish between\nmore than two classes.\nSome algorithms (such as SGD classifiers, Random Forest classifiers, and\nnaive Bayes classifiers) are capable of handling multiple classes natively.\nOthers (such as Logistic Regression or Support Vector Machine\nclassifiers) are strictly binary classifiers. However, there are various\nstrategies that you can use to perform multiclass classification with\nmultiple binary classifiers.\nOne way to create a system that can classify the digit images into 10\nclasses (from 0 to 9) is to train 10 binary classifiers, one for each digit (a\n0-detector, a 1-detector, a 2-detector, and so on). Then when you want to\nclassify an image, you get the decision score from each classifier for that\nimage and you select the class whose classifier outputs the highest score.\nThis is called the one-versus-the-rest (OvR) strategy (also called one-\nversus-all).\nAnother strategy is to train a binary classifier for every pair of digits: one\nto distinguish 0s and 1s, another to distinguish 0s and 2s, another for 1s\nand 2s, and so on. This is called the one-versus-one (OvO) strategy. If\nthere are N classes, you need to train N × (N – 1) / 2 classifiers. For the\nMNIST problem, this means training 45 binary classifiers! When you want\nto classify an image, you have to run the image through all 45 classifiers\nand see which class wins the most duels. The main advantage of OvO is\nthat each classifier only needs to be trained on the part of the training set\nfor the two classes that it must distinguish.\nSome algorithms (such as Support Vector Machine classifiers) scale\npoorly with the size of the training set. For these algorithms OvO is\npreferred because it is faster to train many classifiers on small training\nsets than to train few classifiers on large training sets. For most binary\nclassification algorithms, however, OvR is preferred.\nScikit-Learn detects when you try to use a binary classification algorithm\nfor a multiclass classification task, and it automatically runs OvR or OvO,\ndepending on the algorithm. Let’s try this with a Support Vector Machine\nclassifier (see Chapter 5), using the sklearn.svm.SVC class:\n>>> from sklearn.svm import SVC \n>>> svm_clf = SVC() \n>>> svm_clf.fit(X_train, y_train) # y_train, not y_train_5 \n>>> svm_clf.predict([some_digit]) \narray([5], dtype=uint8)\nThat was easy! This code trains the SVC on the training set using the\noriginal target classes from 0 to 9 (y_train), instead of the 5-versus-the-\nrest target classes (y_train_5). Then it makes a prediction (a correct one\nin this case). Under the hood, Scikit-Learn actually used the OvO strategy:\nit trained 45 binary classifiers, got their decision scores for the image, and\nselected the class that won the most duels.\nIf you call the decision_function() method, you will see that it returns\n10 scores per instance (instead of just 1). That’s one score per class:\n>>> some_digit_scores = svm_clf.decision_function([some_digit]) \n>>> some_digit_scores \narray([[ 2.92492871, 7.02307409, 3.93648529, 0.90117363, 5.96945908, \n 9.5 , 1.90718593, 8.02755089, -0.13202708, \n4.94216947]])\nThe highest score is indeed the one corresponding to class 5:\n>>> np.argmax(some_digit_scores) \n5 \n>>> svm_clf.classes_ \narray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8) \n>>> svm_clf.classes_[5] \n5\nWARNING\nWhen a classifier is trained, it stores the list of target classes in its classes_\nattribute, ordered by value. In this case, the index of each class in the classes_\narray conveniently matches the class itself (e.g., the class at index 5 happens to be\nclass 5), but in general you won’t be so lucky.\nIf you want to force Scikit-Learn to use one-versus-one or one-versus-the-\nrest, you can use the OneVsOneClassifier or OneVsRestClassifier\nclasses. Simply create an instance and pass a classifier to its constructor\n(it does not even have to be a binary classifier). For example, this code\ncreates a multiclass classifier using the OvR strategy, based on an SVC:\n>>> from sklearn.multiclass import OneVsRestClassifier \n>>> ovr_clf = OneVsRestClassifier(SVC()) \n>>> ovr_clf.fit(X_train, y_train) \n>>> ovr_clf.predict([some_digit]) \narray([5], dtype=uint8) \n>>> len(ovr_clf.estimators_) \n10\nTraining an SGDClassifier (or a RandomForestClassifier) is just as\neasy:\n>>> sgd_clf.fit(X_train, y_train) \n>>> sgd_clf.predict([some_digit]) \narray([5], dtype=uint8)\nThis time Scikit-Learn did not have to run OvR or OvO because SGD\nclassifiers can directly classify instances into multiple classes. The\ndecision_function() method now returns one value per class. Let’s look\nat the score that the SGD classifier assigned to each class:\n>>> sgd_clf.decision_function([some_digit]) \narray([[-15955.22628, -38080.96296, -13326.66695, 573.52692, \n-17680.68466, \n 2412.53175, -25526.86498, -12290.15705, -7946.05205, \n-10631.35889]])\nYou can see that the classifier is fairly confident about its prediction:\nalmost all scores are largely negative, while class 5 has a score of 2412.5.\nThe model has a slight doubt regarding class 3, which gets a score of\n573.5. Now of course you want to evaluate this classifier. As usual, you\ncan use cross-validation. Use the cross_val_score() function to evaluate\nthe SGDClassifier’s accuracy:\n>>> cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring=\"accuracy\") \narray([0.8489802 , 0.87129356, 0.86988048])\nIt gets over 84% on all test folds. If you used a random classifier, you\nwould get 10% accuracy, so this is not such a bad score, but you can still\ndo much better. Simply scaling the inputs (as discussed in Chapter 2)\nincreases accuracy above 89%:\n>>> from sklearn.preprocessing import StandardScaler \n>>> scaler = StandardScaler() \n>>> X_train_scaled = scaler.fit_transform(X_train.astype(np.float64)) \n>>> cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, \nscoring=\"accuracy\") \narray([0.89707059, 0.8960948 , 0.90693604])\nError Analysis\nIf this were a real project, you would now follow the steps in your\nMachine Learning project checklist (see Appendix B). You’d explore data\npreparation options, try out multiple models (shortlisting the best ones and\nfine-tuning their hyperparameters using GridSearchCV), and automate as\nmuch as possible. Here, we will assume that you have found a promising\nmodel and you want to find ways to improve it. One way to do this is to\nanalyze the types of errors it makes.\nFirst, look at the confusion matrix. You need to make predictions using the\ncross_val_predict() function, then call the confusion_matrix()\nfunction, just like you did earlier:\n>>> y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, \ncv=3) \n>>> conf_mx = confusion_matrix(y_train, y_train_pred) \n>>> conf_mx \narray([[5578, 0, 22, 7, 8, 45, 35, 5, 222, 1], \n [ 0, 6410, 35, 26, 4, 44, 4, 8, 198, 13], \n [ 28, 27, 5232, 100, 74, 27, 68, 37, 354, 11], \n [ 23, 18, 115, 5254, 2, 209, 26, 38, 373, 73], \n [ 11, 14, 45, 12, 5219, 11, 33, 26, 299, 172], \n [ 26, 16, 31, 173, 54, 4484, 76, 14, 482, 65], \n [ 31, 17, 45, 2, 42, 98, 5556, 3, 123, 1], \n [ 20, 10, 53, 27, 50, 13, 3, 5696, 173, 220], \n [ 17, 64, 47, 91, 3, 125, 24, 11, 5421, 48], \n [ 24, 18, 29, 67, 116, 39, 1, 174, 329, 5152]])\nThat’s a lot of numbers. It’s often more convenient to look at an image\nrepresentation of the confusion matrix, using Matplotlib’s matshow()\nfunction:\nplt.matshow(conf_mx, cmap=plt.cm.gray) \nplt.show()\nThis confusion matrix looks pretty good, since most images are on the\nmain diagonal, which means that they were classified correctly. The 5s\nlook slightly darker than the other digits, which could mean that there are\nfewer images of 5s in the dataset or that the classifier does not perform as\nwell on 5s as on other digits. In fact, you can verify that both are the case.\nLet’s focus the plot on the errors. First, you need to divide each value in\nthe confusion matrix by the number of images in the corresponding class\nso that you can compare error rates instead of absolute numbers of errors\n(which would make abundant classes look unfairly bad):\nrow_sums = conf_mx.sum(axis=1, keepdims=True) \nnorm_conf_mx = conf_mx / row_sums\nFill the diagonal with zeros to keep only the errors, and plot the result:\nnp.fill_diagonal(norm_conf_mx, 0) \nplt.matshow(norm_conf_mx, cmap=plt.cm.gray) \nplt.show()\nYou can clearly see the kinds of errors the classifier makes. Remember\nthat rows represent actual classes, while columns represent predicted\nclasses. The column for class 8 is quite bright, which tells you that many\nimages get misclassified as 8s. However, the row for class 8 is not that\nbad, telling you that actual 8s in general get properly classified as 8s. As\nyou can see, the confusion matrix is not necessarily symmetrical. You can\nalso see that 3s and 5s often get confused (in both directions).\nAnalyzing the confusion matrix often gives you insights into ways to\nimprove your classifier. Looking at this plot, it seems that your efforts\nshould be spent on reducing the false 8s. For example, you could try to\ngather more training data for digits that look like 8s (but are not) so that\nthe classifier can learn to distinguish them from real 8s. Or you could\nengineer new features that would help the classifier—for example, writing\nan algorithm to count the number of closed loops (e.g., 8 has two, 6 has\none, 5 has none). Or you could preprocess the images (e.g., using Scikit-\nImage, Pillow, or OpenCV) to make some patterns, such as closed loops,\nstand out more.\nAnalyzing individual errors can also be a good way to gain insights on\nwhat your classifier is doing and why it is failing, but it is more difficult\nand time-consuming. For example, let’s plot examples of 3s and 5s (the\nplot_digits() function just uses Matplotlib’s imshow() function; see\nthis chapter’s Jupyter notebook for details):\ncl_a, cl_b = 3, 5 \nX_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)] \nX_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)] \nX_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)] \nX_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)] \n \nplt.figure(figsize=(8,8)) \nplt.subplot(221); plot_digits(X_aa[:25], images_per_row=5) \nplt.subplot(222); plot_digits(X_ab[:25], images_per_row=5) \nplt.subplot(223); plot_digits(X_ba[:25], images_per_row=5) \nplt.subplot(224); plot_digits(X_bb[:25], images_per_row=5) \nplt.show()\nThe two 5 × 5 blocks on the left show digits classified as 3s, and the two 5\n× 5 blocks on the right show images classified as 5s. Some of the digits\nthat the classifier gets wrong (i.e., in the bottom-left and top-right blocks)\nare so badly written that even a human would have trouble classifying\nthem (e.g., the 5 in the first row and second column truly looks like a\nbadly written 3). However, most misclassified images seem like obvious\nerrors to us, and it’s hard to understand why the classifier made the\nmistakes it did. The reason is that we used a simple SGDClassifier,\nwhich is a linear model. All it does is assign a weight per class to each\npixel, and when it sees a new image it just sums up the weighted pixel\n3\nintensities to get a score for each class. So since 3s and 5s differ only by a\nfew pixels, this model will easily confuse them.\nThe main difference between 3s and 5s is the position of the small line\nthat joins the top line to the bottom arc. If you draw a 3 with the junction\nslightly shifted to the left, the classifier might classify it as a 5, and vice\nversa. In other words, this classifier is quite sensitive to image shifting\nand rotation. So one way to reduce the 3/5 confusion would be to\npreprocess the images to ensure that they are well centered and not too\nrotated. This will probably help reduce other errors as well.\nMultilabel Classification\nUntil now each instance has always been assigned to just one class. In\nsome cases you may want your classifier to output multiple classes for\neach instance. Consider a face-recognition classifier: what should it do if\nit recognizes several people in the same picture? It should attach one tag\nper person it recognizes. Say the classifier has been trained to recognize\nthree faces, Alice, Bob, and Charlie. Then when the classifier is shown a\npicture of Alice and Charlie, it should output [1, 0, 1] (meaning “Alice\nyes, Bob no, Charlie yes”). Such a classification system that outputs\nmultiple binary tags is called a multilabel classification system.\nWe won’t go into face recognition just yet, but let’s look at a simpler\nexample, just for illustration purposes:\nfrom sklearn.neighbors import KNeighborsClassifier \n \ny_train_large = (y_train >= 7) \ny_train_odd = (y_train % 2 == 1) \ny_multilabel = np.c_[y_train_large, y_train_odd] \n \nknn_clf = KNeighborsClassifier() \nknn_clf.fit(X_train, y_multilabel)\nThis code creates a y_multilabel array containing two target labels for\neach digit image: the first indicates whether or not the digit is large (7, 8,\nor 9), and the second indicates whether or not it is odd. The next lines\ncreate a KNeighborsClassifier instance (which supports multilabel\nclassification, though not all classifiers do), and we train it using the\nmultiple targets array. Now you can make a prediction, and notice that it\noutputs two labels:\n>>> knn_clf.predict([some_digit]) \narray([[False, True]])\nAnd it gets it right! The digit 5 is indeed not large (False) and odd (True).\nThere are many ways to evaluate a multilabel classifier, and selecting the\nright metric really depends on your project. One approach is to measure\nthe F score for each individual label (or any other binary classifier metric\ndiscussed earlier), then simply compute the average score. This code\ncomputes the average F score across all labels:\n>>> y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_multilabel, \ncv=3) \n>>> f1_score(y_multilabel, y_train_knn_pred, average=\"macro\") \n0.976410265560605\nThis assumes that all labels are equally important, however, which may\nnot be the case. In particular, if you have many more pictures of Alice than\nof Bob or Charlie, you may want to give more weight to the classifier’s\nscore on pictures of Alice. One simple option is to give each label a weight\nequal to its support (i.e., the number of instances with that target label). To\ndo this, simply set average=\"weighted\" in the preceding code.\nMultioutput Classification\nThe last type of classification task we are going to discuss here is called\nmultioutput–multiclass classification (or simply multioutput\nclassification). It is simply a generalization of multilabel classification\nwhere each label can be multiclass (i.e., it can have more than two\npossible values).\n1\n1\n4\nTo illustrate this, let’s build a system that removes noise from images. It\nwill take as input a noisy digit image, and it will (hopefully) output a clean\ndigit image, represented as an array of pixel intensities, just like the\nMNIST images. Notice that the classifier’s output is multilabel (one label\nper pixel) and each label can have multiple values (pixel intensity ranges\nfrom 0 to 255). It is thus an example of a multioutput classification\nsystem.\nNOTE\nThe line between classification and regression is sometimes blurry, such as in this\nexample. Arguably, predicting pixel intensity is more akin to regression than to\nclassification. Moreover, multioutput systems are not limited to classification tasks;\nyou could even have a system that outputs multiple labels per instance, including\nboth class labels and value labels.\nLet’s start by creating the training and test sets by taking the MNIST\nimages and adding noise to their pixel intensities with NumPy’s\nrandint() function. The target images will be the original images:\nnoise = np.random.randint(0, 100, (len(X_train), 784)) \nX_train_mod = X_train + noise \nnoise = np.random.randint(0, 100, (len(X_test), 784)) \nX_test_mod = X_test + noise \ny_train_mod = X_train \ny_test_mod = X_test\nLet’s take a peek at an image from the test set (yes, we’re snooping on the\ntest data, so you should be frowning right now):\nOn the left is the noisy input image, and on the right is the clean target\nimage. Now let’s train the classifier and make it clean this image:\nknn_clf.fit(X_train_mod, y_train_mod) \nclean_digit = knn_clf.predict([X_test_mod[some_index]]) \nplot_digit(clean_digit)\nLooks close enough to the target! This concludes our tour of classification.\nYou should now know how to select good metrics for classification tasks,\npick the appropriate precision/recall trade-off, compare classifiers, and\nmore generally build good classification systems for a variety of tasks.\nExercises\n1. Try to build a classifier for the MNIST dataset that achieves over\n97% accuracy on the test set. Hint: the KNeighborsClassifier\nworks quite well for this task; you just need to find good\nhyperparameter values (try a grid search on the weights and\nn_neighbors hyperparameters).\n2. Write a function that can shift an MNIST image in any direction\n(left, right, up, or down) by one pixel. Then, for each image in\nthe training set, create four shifted copies (one per direction) and\nadd them to the training set. Finally, train your best model on this\nexpanded training set and measure its accuracy on the test set.\nYou should observe that your model performs even better now!\nThis technique of artificially growing the training set is called\ndata augmentation or training set expansion.\n3. Tackle the Titanic dataset. A great place to start is on Kaggle.\n4. Build a spam classifier (a more challenging exercise):\nDownload examples of spam and ham from Apache\nSpamAssassin’s public datasets.\nUnzip the datasets and familiarize yourself with the data\nformat.\nSplit the datasets into a training set and a test set.\nWrite a data preparation pipeline to convert each email\ninto a feature vector. Your preparation pipeline should\ntransform an email into a (sparse) vector that indicates\nthe presence or absence of each possible word. For\nexample, if all emails only ever contain four words,\n“Hello,” “how,” “are,” “you,” then the email “Hello you\nHello Hello you” would be converted into a vector [1, 0,\n0, 1] (meaning [“Hello” is present, “how” is absent, “are”\nis absent, “you” is present]), or [3, 0, 0, 2] if you prefer to\ncount the number of occurrences of each word.\n5\nYou may want to add hyperparameters to your\npreparation pipeline to control whether or not to strip off\nemail headers, convert each email to lowercase, remove\npunctuation, replace all URLs with “URL,” replace all\nnumbers with “NUMBER,” or even perform stemming\n(i.e., trim off word endings; there are Python libraries\navailable to do this).\nFinally, try out several classifiers and see if you can build\na great spam classifier, with both high recall and high\nprecision.\nSolutions to these exercises can be found in the Jupyter notebooks\navailable at https://github.com/ageron/handson-ml2.\n1 By default Scikit-Learn caches downloaded datasets in a directory called\n$HOME/scikit_learn_data.\n2 Shuffling may be a bad idea in some contexts—for example, if you are working on time\nseries data (such as stock market prices or weather conditions). We will explore this in the\nnext chapters.\n3 But remember that our brain is a fantastic pattern recognition system, and our visual\nsystem does a lot of complex preprocessing before any information reaches our\nconsciousness, so the fact that it feels simple does not mean that it is.\n4 Scikit-Learn offers a few other averaging options and multilabel classifier metrics; see the\ndocumentation for more details.\n5 You can use the shift() function from the scipy.ndimage.interpolation module.\nFor example, shift(image, [2, 1], cval=0) shifts the image two pixels down and one\npixel to the right." + }, + { + "chapter": 4, + "text": "Chapter 4. Training Models\nSo far we have treated Machine Learning models and their training\nalgorithms mostly like black boxes. If you went through some of the\nexercises in the previous chapters, you may have been surprised by how\nmuch you can get done without knowing anything about what’s under the\nhood: you optimized a regression system, you improved a digit image\nclassifier, and you even built a spam classifier from scratch, all this without\nknowing how they actually work. Indeed, in many situations you don’t really\nneed to know the implementation details.\nHowever, having a good understanding of how things work can help you\nquickly home in on the appropriate model, the right training algorithm to\nuse, and a good set of hyperparameters for your task. Understanding what’s\nunder the hood will also help you debug issues and perform error analysis\nmore efficiently. Lastly, most of the topics discussed in this chapter will be\nessential in understanding, building, and training neural networks (discussed\nin Part II of this book).\nIn this chapter we will start by looking at the Linear Regression model, one\nof the simplest models there is. We will discuss two very different ways to\ntrain it:\nUsing a direct “closed-form” equation that directly computes the\nmodel parameters that best fit the model to the training set (i.e., the\nmodel parameters that minimize the cost function over the training\nset).\nUsing an iterative optimization approach called Gradient Descent\n(GD) that gradually tweaks the model parameters to minimize the\ncost function over the training set, eventually converging to the\nsame set of parameters as the first method. We will look at a few\nvariants of Gradient Descent that we will use again and again when\nwe study neural networks in Part II: Batch GD, Mini-batch GD, and\nStochastic GD.\nNext we will look at Polynomial Regression, a more complex model that can\nfit nonlinear datasets. Since this model has more parameters than Linear\nRegression, it is more prone to overfitting the training data, so we will look\nat how to detect whether or not this is the case using learning curves, and\nthen we will look at several regularization techniques that can reduce the\nrisk of overfitting the training set.\nFinally, we will look at two more models that are commonly used for\nclassification tasks: Logistic Regression and Softmax Regression.\nWARNING\nThere will be quite a few math equations in this chapter, using basic notions of linear\nalgebra and calculus. To understand these equations, you will need to know what\nvectors and matrices are; how to transpose them, multiply them, and inverse them; and\nwhat partial derivatives are. If you are unfamiliar with these concepts, please go\nthrough the linear algebra and calculus introductory tutorials available as Jupyter\nnotebooks in the online supplemental material. For those who are truly allergic to\nmathematics, you should still go through this chapter and simply skip the equations;\nhopefully, the text will be sufficient to help you understand most of the concepts.\nLinear Regression\nIn Chapter 1 we looked at a simple regression model of life satisfaction:\nlife_satisfaction = θ + θ × GDP_per_capita.\nThis model is just a linear function of the input feature GDP_per_capita. θ\nand θ are the model’s parameters.\nMore generally, a linear model makes a prediction by simply computing a\nweighted sum of the input features, plus a constant called the bias term (also\ncalled the intercept term), as shown in Equation 4-1.\nEquation 4-1. Linear Regression model prediction\nˆy = θ0 + θ1x1 + θ2x2 + ⋯+ θnxn\nIn this equation:\n0\n1\n0\n1\nŷ is the predicted value.\nn is the number of features.\nx is the i feature value.\nθ is the j model parameter (including the bias term θ and the\nfeature weights θ , θ , ⋯, θ ).\nThis can be written much more concisely using a vectorized form, as shown\nin Equation 4-2.\nEquation 4-2. Linear Regression model prediction (vectorized form)\nˆy = hθ(x) = θ ⋅x\nIn this equation:\nθ is the model’s parameter vector, containing the bias term θ and\nthe feature weights θ to θ .\nx is the instance’s feature vector, containing x to x , with x always\nequal to 1.\nθ · x is the dot product of the vectors θ and x, which is of course\nequal to θ0x0 + θ1x1 + θ2x2 + ⋯+ θnxn.\nh is the hypothesis function, using the model parameters θ.\nNOTE\nIn Machine Learning, vectors are often represented as column vectors, which are 2D\narrays with a single column. If θ and x are column vectors, then the prediction is \nˆy = θ⊺x, where θ⊺ is the transpose of θ (a row vector instead of a column vector) and \nθ⊺x is the matrix multiplication of θ⊺ and x. It is of course the same prediction, except\nthat it is now represented as a single-cell matrix rather than a scalar value. In this book I\nwill use this notation to avoid switching between dot products and matrix\nmultiplications.\nOK, that’s the Linear Regression model—but how do we train it? Well, recall\nthat training a model means setting its parameters so that the model best fits\ni\nth\nj\nth\n0\n1\n2\nn\n0\n1\nn\n0\nn\n0\nθ\nthe training set. For this purpose, we first need a measure of how well (or\npoorly) the model fits the training data. In Chapter 2 we saw that the most\ncommon performance measure of a regression model is the Root Mean\nSquare Error (RMSE) (Equation 2-1). Therefore, to train a Linear Regression\nmodel, we need to find the value of θ that minimizes the RMSE. In practice,\nit is simpler to minimize the mean squared error (MSE) than the RMSE, and\nit leads to the same result (because the value that minimizes a function also\nminimizes its square root).\nThe MSE of a Linear Regression hypothesis h on a training set X is\ncalculated using Equation 4-3.\nEquation 4-3. MSE cost function for a Linear Regression model\nMSE (X, hθ) =\nm\n∑\ni=1\n(θ⊺x(i) −y(i))\n2\nMost of these notations were presented in Chapter 2 (see “Notations”). The\nonly difference is that we write h instead of just h to make it clear that the\nmodel is parametrized by the vector θ. To simplify notations, we will just\nwrite MSE(θ) instead of MSE(X, h ).\nThe Normal Equation\nTo find the value of θ that minimizes the cost function, there is a closed-form\nsolution—in other words, a mathematical equation that gives the result\ndirectly. This is called the Normal Equation (Equation 4-4).\nEquation 4-4. Normal Equation\nˆθ = (X⊺X)−1 X⊺ y\nIn this equation:\nˆθ is the value of θ that minimizes the cost function.\ny is the vector of target values containing y\n to y\n.\n1\nθ\n1\nm\nθ\nθ\n(1)\n(m)\nLet’s generate some linear-looking data to test this equation on (Figure 4-1):\nimport numpy as np \n \nX = 2 * np.random.rand(100, 1) \ny = 4 + 3 * X + np.random.randn(100, 1)\nFigure 4-1. Randomly generated linear dataset\nNow let’s compute ˆθ using the Normal Equation. We will use the inv()\nfunction from NumPy’s linear algebra module (np.linalg) to compute the\ninverse of a matrix, and the dot() method for matrix multiplication:\nX_b = np.c_[np.ones((100, 1)), X] # add x0 = 1 to each instance \ntheta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)\nThe function that we used to generate the data is y = 4 + 3x + Gaussian\nnoise. Let’s see what the equation found:\n>>> theta_best \narray([[4.21509616], \n [2.77011339]])\n1\nWe would have hoped for θ = 4 and θ = 3 instead of θ = 4.215 and θ =\n2.770. Close enough, but the noise made it impossible to recover the exact\nparameters of the original function.\nNow we can make predictions using ˆθ:\n>>> X_new = np.array([[0], [2]]) \n>>> X_new_b = np.c_[np.ones((2, 1)), X_new] # add x0 = 1 to each instance \n>>> y_predict = X_new_b.dot(theta_best) \n>>> y_predict \narray([[4.21509616], \n [9.75532293]])\nLet’s plot this model’s predictions (Figure 4-2):\nplt.plot(X_new, y_predict, \"r-\") \nplt.plot(X, y, \"b.\") \nplt.axis([0, 2, 0, 15]) \nplt.show()\nFigure 4-2. Linear Regression model predictions\nPerforming Linear Regression using Scikit-Learn is simple:\n0\n1\n0\n1\n2\n>>> from sklearn.linear_model import LinearRegression \n>>> lin_reg = LinearRegression() \n>>> lin_reg.fit(X, y) \n>>> lin_reg.intercept_, lin_reg.coef_ \n(array([4.21509616]), array([[2.77011339]])) \n>>> lin_reg.predict(X_new) \narray([[4.21509616], \n [9.75532293]])\nThe LinearRegression class is based on the scipy.linalg.lstsq()\nfunction (the name stands for “least squares”), which you could call directly:\n>>> theta_best_svd, residuals, rank, s = np.linalg.lstsq(X_b, y, rcond=1e-6) \n>>> theta_best_svd \narray([[4.21509616], \n [2.77011339]])\nThis function computes ˆθ = X+y, where X+ is the pseudoinverse of X\n(specifically, the Moore-Penrose inverse). You can use np.linalg.pinv()\nto compute the pseudoinverse directly:\n>>> np.linalg.pinv(X_b).dot(y) \narray([[4.21509616], \n [2.77011339]])\nThe pseudoinverse itself is computed using a standard matrix factorization\ntechnique called Singular Value Decomposition (SVD) that can decompose\nthe training set matrix X into the matrix multiplication of three matrices U Σ\nV (see numpy.linalg.svd()). The pseudoinverse is computed as \nX+ = VΣ+U⊺. To compute the matrix Σ+, the algorithm takes Σ and sets\nto zero all values smaller than a tiny threshold value, then it replaces all the\nnonzero values with their inverse, and finally it transposes the resulting\nmatrix. This approach is more efficient than computing the Normal\nEquation, plus it handles edge cases nicely: indeed, the Normal Equation\nmay not work if the matrix X X is not invertible (i.e., singular), such as if m\n< n or if some features are redundant, but the pseudoinverse is always\ndefined.\n⊺\n⊺\nComputational Complexity\nThe Normal Equation computes the inverse of X X, which is an (n + 1) × (n\n+ 1) matrix (where n is the number of features). The computational\ncomplexity of inverting such a matrix is typically about O(n\n) to O(n ),\ndepending on the implementation. In other words, if you double the number\nof features, you multiply the computation time by roughly 2\n = 5.3 to 2 =\n8.\nThe SVD approach used by Scikit-Learn’s LinearRegression class is about\nO(n ). If you double the number of features, you multiply the computation\ntime by roughly 4.\nWARNING\nBoth the Normal Equation and the SVD approach get very slow when the number of\nfeatures grows large (e.g., 100,000). On the positive side, both are linear with regard to\nthe number of instances in the training set (they are O(m)), so they handle large training\nsets efficiently, provided they can fit in memory.\nAlso, once you have trained your Linear Regression model (using the\nNormal Equation or any other algorithm), predictions are very fast: the\ncomputational complexity is linear with regard to both the number of\ninstances you want to make predictions on and the number of features. In\nother words, making predictions on twice as many instances (or twice as\nmany features) will take roughly twice as much time.\nNow we will look at a very different way to train a Linear Regression model,\nwhich is better suited for cases where there are a large number of features or\ntoo many training instances to fit in memory.\nGradient Descent\nGradient Descent is a generic optimization algorithm capable of finding\noptimal solutions to a wide range of problems. The general idea of Gradient\n⊺\n2.4\n3\n2.4\n3\n2\nDescent is to tweak parameters iteratively in order to minimize a cost\nfunction.\nSuppose you are lost in the mountains in a dense fog, and you can only feel\nthe slope of the ground below your feet. A good strategy to get to the bottom\nof the valley quickly is to go downhill in the direction of the steepest slope.\nThis is exactly what Gradient Descent does: it measures the local gradient of\nthe error function with regard to the parameter vector θ, and it goes in the\ndirection of descending gradient. Once the gradient is zero, you have reached\na minimum!\nConcretely, you start by filling θ with random values (this is called random\ninitialization). Then you improve it gradually, taking one baby step at a time,\neach step attempting to decrease the cost function (e.g., the MSE), until the\nalgorithm converges to a minimum (see Figure 4-3).\nFigure 4-3. In this depiction of Gradient Descent, the model parameters are initialized randomly\nand get tweaked repeatedly to minimize the cost function; the learning step size is proportional to\nthe slope of the cost function, so the steps gradually get smaller as the parameters approach the\nminimum\nAn important parameter in Gradient Descent is the size of the steps,\ndetermined by the learning rate hyperparameter. If the learning rate is too\nsmall, then the algorithm will have to go through many iterations to\nconverge, which will take a long time (see Figure 4-4).\nFigure 4-4. The learning rate is too small\nOn the other hand, if the learning rate is too high, you might jump across the\nvalley and end up on the other side, possibly even higher up than you were\nbefore. This might make the algorithm diverge, with larger and larger\nvalues, failing to find a good solution (see Figure 4-5).\nFigure 4-5. The learning rate is too large\nFinally, not all cost functions look like nice, regular bowls. There may be\nholes, ridges, plateaus, and all sorts of irregular terrains, making\nconvergence to the minimum difficult. Figure 4-6 shows the two main\nchallenges with Gradient Descent. If the random initialization starts the\nalgorithm on the left, then it will converge to a local minimum, which is not\nas good as the global minimum. If it starts on the right, then it will take a\nvery long time to cross the plateau. And if you stop too early, you will never\nreach the global minimum.\nFigure 4-6. Gradient Descent pitfalls\nFortunately, the MSE cost function for a Linear Regression model happens to\nbe a convex function, which means that if you pick any two points on the\ncurve, the line segment joining them never crosses the curve. This implies\nthat there are no local minima, just one global minimum. It is also a\ncontinuous function with a slope that never changes abruptly. These two\nfacts have a great consequence: Gradient Descent is guaranteed to approach\narbitrarily close the global minimum (if you wait long enough and if the\nlearning rate is not too high).\nIn fact, the cost function has the shape of a bowl, but it can be an elongated\nbowl if the features have very different scales. Figure 4-7 shows Gradient\nDescent on a training set where features 1 and 2 have the same scale (on the\nleft), and on a training set where feature 1 has much smaller values than\nfeature 2 (on the right).\n3\n4\nFigure 4-7. Gradient Descent with (left) and without (right) feature scaling\nAs you can see, on the left the Gradient Descent algorithm goes straight\ntoward the minimum, thereby reaching it quickly, whereas on the right it\nfirst goes in a direction almost orthogonal to the direction of the global\nminimum, and it ends with a long march down an almost flat valley. It will\neventually reach the minimum, but it will take a long time.\nWARNING\nWhen using Gradient Descent, you should ensure that all features have a similar scale\n(e.g., using Scikit-Learn’s StandardScaler class), or else it will take much longer to\nconverge.\nThis diagram also illustrates the fact that training a model means searching\nfor a combination of model parameters that minimizes a cost function (over\nthe training set). It is a search in the model’s parameter space: the more\nparameters a model has, the more dimensions this space has, and the harder\nthe search is: searching for a needle in a 300-dimensional haystack is much\ntrickier than in 3 dimensions. Fortunately, since the cost function is convex\nin the case of Linear Regression, the needle is simply at the bottom of the\nbowl.\nBatch Gradient Descent\nTo implement Gradient Descent, you need to compute the gradient of the\ncost function with regard to each model parameter θ . In other words, you\nneed to calculate how much the cost function will change if you change θ\njust a little bit. This is called a partial derivative. It is like asking “What is\nthe slope of the mountain under my feet if I face east?” and then asking the\nsame question facing north (and so on for all other dimensions, if you can\nimagine a universe with more than three dimensions). Equation 4-5\ncomputes the partial derivative of the cost function with regard to parameter\nθ , noted \n MSE(θ).\nEquation 4-5. Partial derivatives of the cost function\nMSE (θ) =\nm\n∑\ni=1\n(θ⊺x(i) −y(i)) x(i)\nj\nInstead of computing these partial derivatives individually, you can use\nEquation 4-6 to compute them all in one go. The gradient vector, noted\n∇MSE(θ), contains all the partial derivatives of the cost function (one for\neach model parameter).\nEquation 4-6. Gradient vector of the cost function\n∇θ MSE (θ) =\n⎛\n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜ \n⎜\n⎝\nMSE (θ)\nMSE (θ)\n⋮\nMSE (θ)\n⎞\n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟ \n⎟\n⎠\n=\nX⊺(Xθ −y)\nj\nj\nj\n∂\n∂θj\n∂\n∂θj\n2\nm\nθ\n∂\n∂θ0\n∂\n∂θ1\n∂\n∂θn\n2\nm\nWARNING\nNotice that this formula involves calculations over the full training set X, at each\nGradient Descent step! This is why the algorithm is called Batch Gradient Descent: it\nuses the whole batch of training data at every step (actually, Full Gradient Descent\nwould probably be a better name). As a result it is terribly slow on very large training\nsets (but we will see much faster Gradient Descent algorithms shortly). However,\nGradient Descent scales well with the number of features; training a Linear Regression\nmodel when there are hundreds of thousands of features is much faster using Gradient\nDescent than using the Normal Equation or SVD decomposition.\nOnce you have the gradient vector, which points uphill, just go in the\nopposite direction to go downhill. This means subtracting ∇MSE(θ) from θ.\nThis is where the learning rate η comes into play: multiply the gradient\nvector by η to determine the size of the downhill step (Equation 4-7).\nEquation 4-7. Gradient Descent step\nθ(next step) = θ −η∇θ MSE(θ)\nLet’s look at a quick implementation of this algorithm:\neta = 0.1 # learning rate \nn_iterations = 1000 \nm = 100 \n \ntheta = np.random.randn(2,1) # random initialization \n \nfor iteration in range(n_iterations): \n gradients = 2/m * X_b.T.dot(X_b.dot(theta) - y) \n theta = theta - eta * gradients\nThat wasn’t too hard! Let’s look at the resulting theta:\n>>> theta \narray([[4.21509616], \n [2.77011339]])\nHey, that’s exactly what the Normal Equation found! Gradient Descent\nworked perfectly. But what if you had used a different learning rate eta?\nθ\n5\nFigure 4-8 shows the first 10 steps of Gradient Descent using three different\nlearning rates (the dashed line represents the starting point).\nFigure 4-8. Gradient Descent with various learning rates\nOn the left, the learning rate is too low: the algorithm will eventually reach\nthe solution, but it will take a long time. In the middle, the learning rate\nlooks pretty good: in just a few iterations, it has already converged to the\nsolution. On the right, the learning rate is too high: the algorithm diverges,\njumping all over the place and actually getting further and further away from\nthe solution at every step.\nTo find a good learning rate, you can use grid search (see Chapter 2).\nHowever, you may want to limit the number of iterations so that grid search\ncan eliminate models that take too long to converge.\nYou may wonder how to set the number of iterations. If it is too low, you will\nstill be far away from the optimal solution when the algorithm stops; but if it\nis too high, you will waste time while the model parameters do not change\nanymore. A simple solution is to set a very large number of iterations but to\ninterrupt the algorithm when the gradient vector becomes tiny—that is, when\nits norm becomes smaller than a tiny number ϵ (called the tolerance)—\nbecause this happens when Gradient Descent has (almost) reached the\nminimum.\nCONVERGENCE RATE\nWhen the cost function is convex and its slope does not change abruptly\n(as is the case for the MSE cost function), Batch Gradient Descent with a\nfixed learning rate will eventually converge to the optimal solution, but\nyou may have to wait a while: it can take O(1/ϵ) iterations to reach the\noptimum within a range of ϵ, depending on the shape of the cost\nfunction. If you divide the tolerance by 10 to have a more precise\nsolution, then the algorithm may have to run about 10 times longer.\nStochastic Gradient Descent\nThe main problem with Batch Gradient Descent is the fact that it uses the\nwhole training set to compute the gradients at every step, which makes it\nvery slow when the training set is large. At the opposite extreme, Stochastic\nGradient Descent picks a random instance in the training set at every step\nand computes the gradients based only on that single instance. Obviously,\nworking on a single instance at a time makes the algorithm much faster\nbecause it has very little data to manipulate at every iteration. It also makes\nit possible to train on huge training sets, since only one instance needs to be\nin memory at each iteration (Stochastic GD can be implemented as an out-\nof-core algorithm; see Chapter 1).\nOn the other hand, due to its stochastic (i.e., random) nature, this algorithm\nis much less regular than Batch Gradient Descent: instead of gently\ndecreasing until it reaches the minimum, the cost function will bounce up\nand down, decreasing only on average. Over time it will end up very close to\nthe minimum, but once it gets there it will continue to bounce around, never\nsettling down (see Figure 4-9). So once the algorithm stops, the final\nparameter values are good, but not optimal.\nFigure 4-9. With Stochastic Gradient Descent, each training step is much faster but also much more\nstochastic than when using Batch Gradient Descent\nWhen the cost function is very irregular (as in Figure 4-6), this can actually\nhelp the algorithm jump out of local minima, so Stochastic Gradient Descent\nhas a better chance of finding the global minimum than Batch Gradient\nDescent does.\nTherefore, randomness is good to escape from local optima, but bad because\nit means that the algorithm can never settle at the minimum. One solution to\nthis dilemma is to gradually reduce the learning rate. The steps start out\nlarge (which helps make quick progress and escape local minima), then get\nsmaller and smaller, allowing the algorithm to settle at the global minimum.\nThis process is akin to simulated annealing, an algorithm inspired from the\nprocess in metallurgy of annealing, where molten metal is slowly cooled\ndown. The function that determines the learning rate at each iteration is\ncalled the learning schedule. If the learning rate is reduced too quickly, you\nmay get stuck in a local minimum, or even end up frozen halfway to the\nminimum. If the learning rate is reduced too slowly, you may jump around\nthe minimum for a long time and end up with a suboptimal solution if you\nhalt training too early.\nThis code implements Stochastic Gradient Descent using a simple learning\nschedule:\nn_epochs = 50 \nt0, t1 = 5, 50 # learning schedule hyperparameters \n \ndef learning_schedule(t): \n return t0 / (t + t1) \n \ntheta = np.random.randn(2,1) # random initialization \n \nfor epoch in range(n_epochs): \n for i in range(m): \n random_index = np.random.randint(m) \n xi = X_b[random_index:random_index+1] \n yi = y[random_index:random_index+1] \n gradients = 2 * xi.T.dot(xi.dot(theta) - yi) \n eta = learning_schedule(epoch * m + i) \n theta = theta - eta * gradients\nBy convention we iterate by rounds of m iterations; each round is called an\nepoch. While the Batch Gradient Descent code iterated 1,000 times through\nthe whole training set, this code goes through the training set only 50 times\nand reaches a pretty good solution:\n>>> theta \narray([[4.21076011], \n [2.74856079]])\nFigure 4-10 shows the first 20 steps of training (notice how irregular the\nsteps are).\nFigure 4-10. The first 20 steps of Stochastic Gradient Descent\nNote that since instances are picked randomly, some instances may be picked\nseveral times per epoch, while others may not be picked at all. If you want to\nbe sure that the algorithm goes through every instance at each epoch, another\napproach is to shuffle the training set (making sure to shuffle the input\nfeatures and the labels jointly), then go through it instance by instance, then\nshuffle it again, and so on. However, this approach generally converges more\nslowly.\nWARNING\nWhen using Stochastic Gradient Descent, the training instances must be independent\nand identically distributed (IID) to ensure that the parameters get pulled toward the\nglobal optimum, on average. A simple way to ensure this is to shuffle the instances\nduring training (e.g., pick each instance randomly, or shuffle the training set at the\nbeginning of each epoch). If you do not shuffle the instances—for example, if the\ninstances are sorted by label—then SGD will start by optimizing for one label, then the\nnext, and so on, and it will not settle close to the global minimum.\nTo perform Linear Regression using Stochastic GD with Scikit-Learn, you\ncan use the SGDRegressor class, which defaults to optimizing the squared\nerror cost function. The following code runs for maximum 1,000 epochs or\nuntil the loss drops by less than 0.001 during one epoch (max_iter=1000,\ntol=1e-3). It starts with a learning rate of 0.1 (eta0=0.1), using the default\nlearning schedule (different from the preceding one). Lastly, it does not use\nany regularization (penalty=None; more details on this shortly):\nfrom sklearn.linear_model import SGDRegressor \nsgd_reg = SGDRegressor(max_iter=1000, tol=1e-3, penalty=None, eta0=0.1) \nsgd_reg.fit(X, y.ravel())\nOnce again, you find a solution quite close to the one returned by the Normal\nEquation:\n>>> sgd_reg.intercept_, sgd_reg.coef_ \n(array([4.24365286]), array([2.8250878]))\nMini-batch Gradient Descent\nThe last Gradient Descent algorithm we will look at is called Mini-batch\nGradient Descent. It is simple to understand once you know Batch and\nStochastic Gradient Descent: at each step, instead of computing the gradients\nbased on the full training set (as in Batch GD) or based on just one instance\n(as in Stochastic GD), Mini-batch GD computes the gradients on small\nrandom sets of instances called mini-batches. The main advantage of Mini-\nbatch GD over Stochastic GD is that you can get a performance boost from\nhardware optimization of matrix operations, especially when using GPUs.\nThe algorithm’s progress in parameter space is less erratic than with\nStochastic GD, especially with fairly large mini-batches. As a result, Mini-\nbatch GD will end up walking around a bit closer to the minimum than\nStochastic GD—but it may be harder for it to escape from local minima (in\nthe case of problems that suffer from local minima, unlike Linear\nRegression). Figure 4-11 shows the paths taken by the three Gradient\nDescent algorithms in parameter space during training. They all end up near\nthe minimum, but Batch GD’s path actually stops at the minimum, while\nboth Stochastic GD and Mini-batch GD continue to walk around. However,\ndon’t forget that Batch GD takes a lot of time to take each step, and\nStochastic GD and Mini-batch GD would also reach the minimum if you\nused a good learning schedule.\nFigure 4-11. Gradient Descent paths in parameter space\nLet’s compare the algorithms we’ve discussed so far for Linear Regression\n(recall that m is the number of training instances and n is the number of\nfeatures); see Table 4-1.\n6\nTable 4-1. Comparison of algorithms for Linear Regression\nAlgorithm\nLarge\nm\nOut-of-core\nsupport\nLarge\nn\nHyperparams Scaling\nrequired\nScikit-Learn\nNormal\nEquation\nFast\nNo\nSlow\n0\nNo\nN/A\nSVD\nFast\nNo\nSlow\n0\nNo\nLinearRegression\nBatch GD\nSlow\nNo\nFast\n2\nYes\nSGDRegressor\nStochastic\nGD\nFast\nYes\nFast\n≥2\nYes\nSGDRegressor\nMini-batch\nGD\nFast\nYes\nFast\n≥2\nYes\nSGDRegressor\nNOTE\nThere is almost no difference after training: all these algorithms end up with very\nsimilar models and make predictions in exactly the same way.\nPolynomial Regression\nWhat if your data is more complex than a straight line? Surprisingly, you can\nuse a linear model to fit nonlinear data. A simple way to do this is to add\npowers of each feature as new features, then train a linear model on this\nextended set of features. This technique is called Polynomial Regression.\nLet’s look at an example. First, let’s generate some nonlinear data, based on\na simple quadratic equation (plus some noise; see Figure 4-12):\nm = 100 \nX = 6 * np.random.rand(m, 1) - 3 \ny = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)\n7\nFigure 4-12. Generated nonlinear and noisy dataset\nClearly, a straight line will never fit this data properly. So let’s use Scikit-\nLearn’s PolynomialFeatures class to transform our training data, adding\nthe square (second-degree polynomial) of each feature in the training set as a\nnew feature (in this case there is just one feature):\n>>> from sklearn.preprocessing import PolynomialFeatures \n>>> poly_features = PolynomialFeatures(degree=2, include_bias=False) \n>>> X_poly = poly_features.fit_transform(X) \n>>> X[0] \narray([-0.75275929]) \n>>> X_poly[0] \narray([-0.75275929, 0.56664654])\nX_poly now contains the original feature of X plus the square of this feature.\nNow you can fit a LinearRegression model to this extended training data\n(Figure 4-13):\n>>> lin_reg = LinearRegression() \n>>> lin_reg.fit(X_poly, y) \n>>> lin_reg.intercept_, lin_reg.coef_ \n(array([1.78134581]), array([[0.93366893, 0.56456263]]))\nFigure 4-13. Polynomial Regression model predictions\nNot bad: the model estimates ˆy = 0.56x12 + 0.93x1 + 1.78 when in fact the\noriginal function was y = 0.5x12 + 1.0x1 + 2.0 + Gaussian noise.\nNote that when there are multiple features, Polynomial Regression is capable\nof finding relationships between features (which is something a plain Linear\nRegression model cannot do). This is made possible by the fact that\nPolynomialFeatures also adds all combinations of features up to the given\ndegree. For example, if there were two features a and b,\nPolynomialFeatures with degree=3 would not only add the features a , a ,\nb , and b , but also the combinations ab, a b, and ab .\nWARNING\nPolynomialFeatures(degree=d) transforms an array containing n features into an\narray containing \n features, where n! is the factorial of n, equal to 1 × 2 × 3 ×\n⋯ × n. Beware of the combinatorial explosion of the number of features!\n2\n3\n2\n3\n2\n2\n(n + d)!\nd!n!\nLearning Curves\nIf you perform high-degree Polynomial Regression, you will likely fit the\ntraining data much better than with plain Linear Regression. For example,\nFigure 4-14 applies a 300-degree polynomial model to the preceding training\ndata, and compares the result with a pure linear model and a quadratic model\n(second-degree polynomial). Notice how the 300-degree polynomial model\nwiggles around to get as close as possible to the training instances.\nFigure 4-14. High-degree Polynomial Regression\nThis high-degree Polynomial Regression model is severely overfitting the\ntraining data, while the linear model is underfitting it. The model that will\ngeneralize best in this case is the quadratic model, which makes sense\nbecause the data was generated using a quadratic model. But in general you\nwon’t know what function generated the data, so how can you decide how\ncomplex your model should be? How can you tell that your model is\noverfitting or underfitting the data?\nIn Chapter 2 you used cross-validation to get an estimate of a model’s\ngeneralization performance. If a model performs well on the training data\nbut generalizes poorly according to the cross-validation metrics, then your\nmodel is overfitting. If it performs poorly on both, then it is underfitting.\nThis is one way to tell when a model is too simple or too complex.\nAnother way to tell is to look at the learning curves: these are plots of the\nmodel’s performance on the training set and the validation set as a function\nof the training set size (or the training iteration). To generate the plots, train\nthe model several times on different sized subsets of the training set. The\nfollowing code defines a function that, given some training data, plots the\nlearning curves of a model:\nfrom sklearn.metrics import mean_squared_error \nfrom sklearn.model_selection import train_test_split \n \ndef plot_learning_curves(model, X, y): \n X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2) \n train_errors, val_errors = [], [] \n for m in range(1, len(X_train)): \n model.fit(X_train[:m], y_train[:m]) \n y_train_predict = model.predict(X_train[:m]) \n y_val_predict = model.predict(X_val) \n train_errors.append(mean_squared_error(y_train[:m], \ny_train_predict)) \n val_errors.append(mean_squared_error(y_val, y_val_predict)) \n plt.plot(np.sqrt(train_errors), \"r-+\", linewidth=2, label=\"train\") \n plt.plot(np.sqrt(val_errors), \"b-\", linewidth=3, label=\"val\")\nLet’s look at the learning curves of the plain Linear Regression model (a\nstraight line; see Figure 4-15):\nlin_reg = LinearRegression() \nplot_learning_curves(lin_reg, X, y)\nFigure 4-15. Learning curves\nThis model that’s underfitting deserves a bit of explanation. First, let’s look\nat the performance on the training data: when there are just one or two\ninstances in the training set, the model can fit them perfectly, which is why\nthe curve starts at zero. But as new instances are added to the training set, it\nbecomes impossible for the model to fit the training data perfectly, both\nbecause the data is noisy and because it is not linear at all. So the error on\nthe training data goes up until it reaches a plateau, at which point adding new\ninstances to the training set doesn’t make the average error much better or\nworse. Now let’s look at the performance of the model on the validation data.\nWhen the model is trained on very few training instances, it is incapable of\ngeneralizing properly, which is why the validation error is initially quite big.\nThen, as the model is shown more training examples, it learns, and thus the\nvalidation error slowly goes down. However, once again a straight line\ncannot do a good job modeling the data, so the error ends up at a plateau,\nvery close to the other curve.\nThese learning curves are typical of a model that’s underfitting. Both curves\nhave reached a plateau; they are close and fairly high.\nTIP\nIf your model is underfitting the training data, adding more training examples will not\nhelp. You need to use a more complex model or come up with better features.\nNow let’s look at the learning curves of a 10th-degree polynomial model on\nthe same data (Figure 4-16):\nfrom sklearn.pipeline import Pipeline \n \npolynomial_regression = Pipeline([ \n (\"poly_features\", PolynomialFeatures(degree=10, \ninclude_bias=False)), \n (\"lin_reg\", LinearRegression()), \n ]) \n \nplot_learning_curves(polynomial_regression, X, y)\nFigure 4-16. Learning curves for the 10th-degree polynomial model\nThese learning curves look a bit like the previous ones, but there are two\nvery important differences:\nThe error on the training data is much lower than with the Linear\nRegression model.\nThere is a gap between the curves. This means that the model\nperforms significantly better on the training data than on the\nvalidation data, which is the hallmark of an overfitting model. If\nyou used a much larger training set, however, the two curves would\ncontinue to get closer.\nTIP\nOne way to improve an overfitting model is to feed it more training data until the\nvalidation error reaches the training error.\nTHE BIAS/VARIANCE TRADE-OFF\nAn important theoretical result of statistics and Machine Learning is the\nfact that a model’s generalization error can be expressed as the sum of\nthree very different errors:\nBias\nThis part of the generalization error is due to wrong assumptions,\nsuch as assuming that the data is linear when it is actually quadratic.\nA high-bias model is most likely to underfit the training data.\nVariance\nThis part is due to the model’s excessive sensitivity to small\nvariations in the training data. A model with many degrees of\nfreedom (such as a high-degree polynomial model) is likely to have\nhigh variance and thus overfit the training data.\nIrreducible error\nThis part is due to the noisiness of the data itself. The only way to\nreduce this part of the error is to clean up the data (e.g., fix the data\nsources, such as broken sensors, or detect and remove outliers).\nIncreasing a model’s complexity will typically increase its variance and\nreduce its bias. Conversely, reducing a model’s complexity increases its\nbias and reduces its variance. This is why it is called a trade-off.\nRegularized Linear Models\nAs we saw in Chapters 1 and 2, a good way to reduce overfitting is to\nregularize the model (i.e., to constrain it): the fewer degrees of freedom it\nhas, the harder it will be for it to overfit the data. A simple way to regularize\na polynomial model is to reduce the number of polynomial degrees.\nFor a linear model, regularization is typically achieved by constraining the\nweights of the model. We will now look at Ridge Regression, Lasso\n8\nRegression, and Elastic Net, which implement three different ways to\nconstrain the weights.\nRidge Regression\nRidge Regression (also called Tikhonov regularization) is a regularized\nversion of Linear Regression: a regularization term equal to α ∑n\ni=1 θi\n2 is\nadded to the cost function. This forces the learning algorithm to not only fit\nthe data but also keep the model weights as small as possible. Note that the\nregularization term should only be added to the cost function during training.\nOnce the model is trained, you want to use the unregularized performance\nmeasure to evaluate the model’s performance.\nNOTE\nIt is quite common for the cost function used during training to be different from the\nperformance measure used for testing. Apart from regularization, another reason they\nmight be different is that a good training cost function should have optimization-\nfriendly derivatives, while the performance measure used for testing should be as close\nas possible to the final objective. For example, classifiers are often trained using a cost\nfunction such as the log loss (discussed in a moment) but evaluated using\nprecision/recall.\nThe hyperparameter α controls how much you want to regularize the model.\nIf α = 0, then Ridge Regression is just Linear Regression. If α is very large,\nthen all weights end up very close to zero and the result is a flat line going\nthrough the data’s mean. Equation 4-8 presents the Ridge Regression cost\nfunction.\nEquation 4-8. Ridge Regression cost function\nJ(θ) = MSE(θ) + α\n∑n\ni=1 θi\n2\nNote that the bias term θ is not regularized (the sum starts at i = 1, not 0). If\nwe define w as the vector of feature weights (θ to θ ), then the\nregularization term is equal to ½(∥ w ∥) , where ∥ w ∥ represents the ℓ\n9\n1\n2\n0\n1\nn\n2 2\n2\n2\nnorm of the weight vector. For Gradient Descent, just add αw to the MSE\ngradient vector (Equation 4-6).\nWARNING\nIt is important to scale the data (e.g., using a StandardScaler) before performing\nRidge Regression, as it is sensitive to the scale of the input features. This is true of most\nregularized models.\nFigure 4-17 shows several Ridge models trained on some linear data using\ndifferent α values. On the left, plain Ridge models are used, leading to linear\npredictions. On the right, the data is first expanded using\nPolynomialFeatures(degree=10), then it is scaled using a\nStandardScaler, and finally the Ridge models are applied to the resulting\nfeatures: this is Polynomial Regression with Ridge regularization. Note how\nincreasing α leads to flatter (i.e., less extreme, more reasonable) predictions,\nthus reducing the model’s variance but increasing its bias.\nFigure 4-17. A linear model (left) and a polynomial model (right), both with various levels of Ridge\nregularization\nAs with Linear Regression, we can perform Ridge Regression either by\ncomputing a closed-form equation or by performing Gradient Descent. The\npros and cons are the same. Equation 4-9 shows the closed-form solution,\n10\nwhere A is the (n + 1) × (n + 1) identity matrix, except with a 0 in the top-\nleft cell, corresponding to the bias term.\nEquation 4-9. Ridge Regression closed-form solution\nˆθ = (X⊺X + αA)−1 X⊺ y\nHere is how to perform Ridge Regression with Scikit-Learn using a closed-\nform solution (a variant of Equation 4-9 that uses a matrix factorization\ntechnique by André-Louis Cholesky):\n>>> from sklearn.linear_model import Ridge \n>>> ridge_reg = Ridge(alpha=1, solver=\"cholesky\") \n>>> ridge_reg.fit(X, y) \n>>> ridge_reg.predict([[1.5]]) \narray([[1.55071465]])\nAnd using Stochastic Gradient Descent:\n>>> sgd_reg = SGDRegressor(penalty=\"l2\") \n>>> sgd_reg.fit(X, y.ravel()) \n>>> sgd_reg.predict([[1.5]]) \narray([1.47012588])\nThe penalty hyperparameter sets the type of regularization term to use.\nSpecifying \"l2\" indicates that you want SGD to add a regularization term to\nthe cost function equal to half the square of the ℓ norm of the weight vector:\nthis is simply Ridge Regression.\nLasso Regression\nLeast Absolute Shrinkage and Selection Operator Regression (usually\nsimply called Lasso Regression) is another regularized version of Linear\nRegression: just like Ridge Regression, it adds a regularization term to the\ncost function, but it uses the ℓ norm of the weight vector instead of half the\nsquare of the ℓ norm (see Equation 4-10).\nEquation 4-10. Lasso Regression cost function\n11\n12\n2\n1\n2\nJ(θ) = MSE(θ) + α ∑n\ni=1|θi|\nFigure 4-18 shows the same thing as Figure 4-17 but replaces Ridge models\nwith Lasso models and uses smaller α values.\nFigure 4-18. A linear model (left) and a polynomial model (right), both using various levels of Lasso\nregularization\nAn important characteristic of Lasso Regression is that it tends to eliminate\nthe weights of the least important features (i.e., set them to zero). For\nexample, the dashed line in the righthand plot in Figure 4-18 (with α = 10 )\nlooks quadratic, almost linear: all the weights for the high-degree\npolynomial features are equal to zero. In other words, Lasso Regression\nautomatically performs feature selection and outputs a sparse model (i.e.,\nwith few nonzero feature weights).\nYou can get a sense of why this is the case by looking at Figure 4-19: the\naxes represent two model parameters, and the background contours represent\ndifferent loss functions. In the top-left plot, the contours represent the ℓ\nloss (|θ | + |θ |), which drops linearly as you get closer to any axis. For\nexample, if you initialize the model parameters to θ = 2 and θ = 0.5,\nrunning Gradient Descent will decrement both parameters equally (as\nrepresented by the dashed yellow line); therefore θ will reach 0 first (since\nit was closer to 0 to begin with). After that, Gradient Descent will roll down\nthe gutter until it reaches θ = 0 (with a bit of bouncing around, since the\ngradients of ℓ never get close to 0: they are either –1 or 1 for each\n-7\n1\n1\n2\n1\n2\n2\n1\n1\nparameter). In the top-right plot, the contours represent Lasso’s cost function\n(i.e., an MSE cost function plus an ℓ loss). The small white circles show the\npath that Gradient Descent takes to optimize some model parameters that\nwere initialized around θ = 0.25 and θ = –1: notice once again how the path\nquickly reaches θ = 0, then rolls down the gutter and ends up bouncing\naround the global optimum (represented by the red square). If we increased\nα, the global optimum would move left along the dashed yellow line, while\nif we decreased α, the global optimum would move right (in this example,\nthe optimal parameters for the unregularized MSE are θ = 2 and θ = 0.5).\nFigure 4-19. Lasso versus Ridge regularization\nThe two bottom plots show the same thing but with an ℓ penalty instead. In\nthe bottom-left plot, you can see that the ℓ loss decreases with the distance\nto the origin, so Gradient Descent just takes a straight path toward that point.\nIn the bottom-right plot, the contours represent Ridge Regression’s cost\nfunction (i.e., an MSE cost function plus an ℓ loss). There are two main\n1\n1\n2\n2\n1\n2\n2\n2\n2\ndifferences with Lasso. First, the gradients get smaller as the parameters\napproach the global optimum, so Gradient Descent naturally slows down,\nwhich helps convergence (as there is no bouncing around). Second, the\noptimal parameters (represented by the red square) get closer and closer to\nthe origin when you increase α, but they never get eliminated entirely.\nTIP\nTo avoid Gradient Descent from bouncing around the optimum at the end when using\nLasso, you need to gradually reduce the learning rate during training (it will still bounce\naround the optimum, but the steps will get smaller and smaller, so it will converge).\nThe Lasso cost function is not differentiable at θ = 0 (for i = 1, 2, ⋯, n), but\nGradient Descent still works fine if you use a subgradient vector g instead\nwhen any θ = 0. Equation 4-11 shows a subgradient vector equation you can\nuse for Gradient Descent with the Lasso cost function.\nEquation 4-11. Lasso Regression subgradient vector\ng(θ, J) = ∇θ MSE(θ) + α\n⎛\n⎜ \n⎜ \n⎜ \n⎜ \n⎜\n⎝\nsign(θ1)\nsign(θ2)\n⋮\nsign(θn)\n⎞\n⎟ \n⎟ \n⎟ \n⎟ \n⎟\n⎠\n  where  sign(θi) =\n⎧\n⎪\n⎨\n⎪\n⎩\n−1\nif θi < 0\n0\nif θi = 0\n+1\nif θi > 0\nHere is a small Scikit-Learn example using the Lasso class:\n>>> from sklearn.linear_model import Lasso \n>>> lasso_reg = Lasso(alpha=0.1) \n>>> lasso_reg.fit(X, y) \n>>> lasso_reg.predict([[1.5]]) \narray([1.53788174])\nNote that you could instead use SGDRegressor(penalty=\"l1\").\nElastic Net\ni\n13\ni\nElastic Net is a middle ground between Ridge Regression and Lasso\nRegression. The regularization term is a simple mix of both Ridge and\nLasso’s regularization terms, and you can control the mix ratio r. When r =\n0, Elastic Net is equivalent to Ridge Regression, and when r = 1, it is\nequivalent to Lasso Regression (see Equation 4-12).\nEquation 4-12. Elastic Net cost function\nJ(θ) = MSE(θ) + rα ∑n\ni=1|θi|+\nα ∑n\ni=1 θi\n2\nSo when should you use plain Linear Regression (i.e., without any\nregularization), Ridge, Lasso, or Elastic Net? It is almost always preferable\nto have at least a little bit of regularization, so generally you should avoid\nplain Linear Regression. Ridge is a good default, but if you suspect that only\na few features are useful, you should prefer Lasso or Elastic Net because\nthey tend to reduce the useless features’ weights down to zero, as we have\ndiscussed. In general, Elastic Net is preferred over Lasso because Lasso may\nbehave erratically when the number of features is greater than the number of\ntraining instances or when several features are strongly correlated.\nHere is a short example that uses Scikit-Learn’s ElasticNet (l1_ratio\ncorresponds to the mix ratio r):\n>>> from sklearn.linear_model import ElasticNet \n>>> elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5) \n>>> elastic_net.fit(X, y) \n>>> elastic_net.predict([[1.5]]) \narray([1.54333232])\nEarly Stopping\nA very different way to regularize iterative learning algorithms such as\nGradient Descent is to stop training as soon as the validation error reaches a\nminimum. This is called early stopping. Figure 4-20 shows a complex model\n(in this case, a high-degree Polynomial Regression model) being trained\nwith Batch Gradient Descent. As the epochs go by the algorithm learns, and\nits prediction error (RMSE) on the training set goes down, along with its\nprediction error on the validation set. After a while though, the validation\nerror stops decreasing and starts to go back up. This indicates that the model\n1−r\n2\nhas started to overfit the training data. With early stopping you just stop\ntraining as soon as the validation error reaches the minimum. It is such a\nsimple and efficient regularization technique that Geoffrey Hinton called it a\n“beautiful free lunch.”\nFigure 4-20. Early stopping regularization\nTIP\nWith Stochastic and Mini-batch Gradient Descent, the curves are not so smooth, and it\nmay be hard to know whether you have reached the minimum or not. One solution is to\nstop only after the validation error has been above the minimum for some time (when\nyou are confident that the model will not do any better), then roll back the model\nparameters to the point where the validation error was at a minimum.\nHere is a basic implementation of early stopping:\nfrom sklearn.base import clone \n \n# prepare the data \npoly_scaler = Pipeline([ \n (\"poly_features\", PolynomialFeatures(degree=90, \ninclude_bias=False)), \n (\"std_scaler\", StandardScaler()) \n ]) \nX_train_poly_scaled = poly_scaler.fit_transform(X_train) \nX_val_poly_scaled = poly_scaler.transform(X_val) \n \nsgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True, \n penalty=None, learning_rate=\"constant\", eta0=0.0005) \n \nminimum_val_error = float(\"inf\") \nbest_epoch = None \nbest_model = None \nfor epoch in range(1000): \n sgd_reg.fit(X_train_poly_scaled, y_train) # continues where it left off \n y_val_predict = sgd_reg.predict(X_val_poly_scaled) \n val_error = mean_squared_error(y_val, y_val_predict) \n if val_error < minimum_val_error: \n minimum_val_error = val_error \n best_epoch = epoch \n best_model = clone(sgd_reg)\nNote that with warm_start=True, when the fit() method is called it\ncontinues training where it left off, instead of restarting from scratch.\nLogistic Regression\nAs we discussed in Chapter 1, some regression algorithms can be used for\nclassification (and vice versa). Logistic Regression (also called Logit\nRegression) is commonly used to estimate the probability that an instance\nbelongs to a particular class (e.g., what is the probability that this email is\nspam?). If the estimated probability is greater than 50%, then the model\npredicts that the instance belongs to that class (called the positive class,\nlabeled “1”), and otherwise it predicts that it does not (i.e., it belongs to the\nnegative class, labeled “0”). This makes it a binary classifier.\nEstimating Probabilities\nSo how does Logistic Regression work? Just like a Linear Regression model,\na Logistic Regression model computes a weighted sum of the input features\n(plus a bias term), but instead of outputting the result directly like the Linear\nRegression model does, it outputs the logistic of this result (see Equation 4-\n13).\nEquation 4-13. Logistic Regression model estimated probability (vectorized form)\nˆp = hθ (x) = σ (x⊺θ)\nThe logistic—noted σ(·)—is a sigmoid function (i.e., S-shaped) that outputs a\nnumber between 0 and 1. It is defined as shown in Equation 4-14 and\nFigure 4-21.\nEquation 4-14. Logistic function\nσ (t) =\nFigure 4-21. Logistic function\nOnce the Logistic Regression model has estimated the probability ˆp = h (x)\nthat an instance x belongs to the positive class, it can make its prediction ŷ\neasily (see Equation 4-15).\nEquation 4-15. Logistic Regression model prediction\nˆy ={ 0\nif ˆp < 0.5\n1\nif ˆp ≥0.5\nNotice that σ(t) < 0.5 when t < 0, and σ(t) ≥ 0.5 when t ≥ 0, so a Logistic\nRegression model predicts 1 if x θ is positive and 0 if it is negative.\n1\n1 + exp(−t)\nθ\n⊺\nNOTE\nThe score t is often called the logit. The name comes from the fact that the logit\nfunction, defined as logit(p) = log(p / (1 – p)), is the inverse of the logistic function.\nIndeed, if you compute the logit of the estimated probability p, you will find that the\nresult is t. The logit is also called the log-odds, since it is the log of the ratio between the\nestimated probability for the positive class and the estimated probability for the negative\nclass.\nTraining and Cost Function\nNow you know how a Logistic Regression model estimates probabilities and\nmakes predictions. But how is it trained? The objective of training is to set\nthe parameter vector θ so that the model estimates high probabilities for\npositive instances (y = 1) and low probabilities for negative instances (y = 0).\nThis idea is captured by the cost function shown in Equation 4-16 for a\nsingle training instance x.\nEquation 4-16. Cost function of a single training instance\nc(θ) ={\n−log(ˆp)\nif y = 1\n−log(1 −ˆp)\nif y = 0\nThis cost function makes sense because –log(t) grows very large when t\napproaches 0, so the cost will be large if the model estimates a probability\nclose to 0 for a positive instance, and it will also be very large if the model\nestimates a probability close to 1 for a negative instance. On the other hand,\n–log(t) is close to 0 when t is close to 1, so the cost will be close to 0 if the\nestimated probability is close to 0 for a negative instance or close to 1 for a\npositive instance, which is precisely what we want.\nThe cost function over the whole training set is the average cost over all\ntraining instances. It can be written in a single expression called the log loss,\nshown in Equation 4-17.\nEquation 4-17. Logistic Regression cost function (log loss)\nJ(θ) = −\n∑m\ni=1[y(i)log(ˆp(i))+(1 −y(i))log(1 −ˆp(i))]\n1\nm\nThe bad news is that there is no known closed-form equation to compute the\nvalue of θ that minimizes this cost function (there is no equivalent of the\nNormal Equation). The good news is that this cost function is convex, so\nGradient Descent (or any other optimization algorithm) is guaranteed to find\nthe global minimum (if the learning rate is not too large and you wait long\nenough). The partial derivatives of the cost function with regard to the j\nmodel parameter θ are given by Equation 4-18.\nEquation 4-18. Logistic cost function partial derivatives\nJ(θ) =\nm\n∑\ni=1\n(σ (θ⊺x(i)) −y(i)) x\n(i)\nj\nThis equation looks very much like Equation 4-5: for each instance it\ncomputes the prediction error and multiplies it by the j feature value, and\nthen it computes the average over all training instances. Once you have the\ngradient vector containing all the partial derivatives, you can use it in the\nBatch Gradient Descent algorithm. That’s it: you now know how to train a\nLogistic Regression model. For Stochastic GD you would take one instance\nat a time, and for Mini-batch GD you would use a mini-batch at a time.\nDecision Boundaries\nLet’s use the iris dataset to illustrate Logistic Regression. This is a famous\ndataset that contains the sepal and petal length and width of 150 iris flowers\nof three different species: Iris setosa, Iris versicolor, and Iris virginica (see\nFigure 4-22).\nth\nj\n∂\n∂θj\n1\nm\nth\nFigure 4-22. Flowers of three iris plant species\nLet’s try to build a classifier to detect the Iris virginica type based only on\nthe petal width feature. First let’s load the data:\n>>> from sklearn import datasets \n>>> iris = datasets.load_iris() \n>>> list(iris.keys()) \n['data', 'target', 'target_names', 'DESCR', 'feature_names', 'filename'] \n>>> X = iris[\"data\"][:, 3:] # petal width \n>>> y = (iris[\"target\"] == 2).astype(np.int) # 1 if Iris virginica, else 0\nNow let’s train a Logistic Regression model:\nfrom sklearn.linear_model import LogisticRegression \n \nlog_reg = LogisticRegression() \nlog_reg.fit(X, y)\nLet’s look at the model’s estimated probabilities for flowers with petal\nwidths varying from 0 cm to 3 cm (Figure 4-23):\nX_new = np.linspace(0, 3, 1000).reshape(-1, 1) \ny_proba = log_reg.predict_proba(X_new) \nplt.plot(X_new, y_proba[:, 1], \"g-\", label=\"Iris virginica\") \n14\n15\nplt.plot(X_new, y_proba[:, 0], \"b--\", label=\"Not Iris virginica\") \n# + more Matplotlib code to make the image look pretty\nFigure 4-23. Estimated probabilities and decision boundary\nThe petal width of Iris virginica flowers (represented by triangles) ranges\nfrom 1.4 cm to 2.5 cm, while the other iris flowers (represented by squares)\ngenerally have a smaller petal width, ranging from 0.1 cm to 1.8 cm. Notice\nthat there is a bit of overlap. Above about 2 cm the classifier is highly\nconfident that the flower is an Iris virginica (it outputs a high probability for\nthat class), while below 1 cm it is highly confident that it is not an Iris\nvirginica (high probability for the “Not Iris virginica” class). In between\nthese extremes, the classifier is unsure. However, if you ask it to predict the\nclass (using the predict() method rather than the predict_proba()\nmethod), it will return whichever class is the most likely. Therefore, there is\na decision boundary at around 1.6 cm where both probabilities are equal to\n50%: if the petal width is higher than 1.6 cm, the classifier will predict that\nthe flower is an Iris virginica, and otherwise it will predict that it is not\n(even if it is not very confident):\n>>> log_reg.predict([[1.7], [1.5]]) \narray([1, 0])\nFigure 4-24 shows the same dataset, but this time displaying two features:\npetal width and length. Once trained, the Logistic Regression classifier can,\nbased on these two features, estimate the probability that a new flower is an\nIris virginica. The dashed line represents the points where the model\nestimates a 50% probability: this is the model’s decision boundary. Note that\nit is a linear boundary. Each parallel line represents the points where the\nmodel outputs a specific probability, from 15% (bottom left) to 90% (top\nright). All the flowers beyond the top-right line have an over 90% chance of\nbeing Iris virginica, according to the model.\nFigure 4-24. Linear decision boundary\nJust like the other linear models, Logistic Regression models can be\nregularized using ℓ or ℓ penalties. Scikit-Learn actually adds an ℓ penalty\nby default.\nNOTE\nThe hyperparameter controlling the regularization strength of a Scikit-Learn\nLogisticRegression model is not alpha (as in other linear models), but its inverse:\nC. The higher the value of C, the less the model is regularized.\nSoftmax Regression\nThe Logistic Regression model can be generalized to support multiple\nclasses directly, without having to train and combine multiple binary\nclassifiers (as discussed in Chapter 3). This is called Softmax Regression, or\nMultinomial Logistic Regression.\nThe idea is simple: when given an instance x, the Softmax Regression model\nfirst computes a score s (x) for each class k, then estimates the probability of\neach class by applying the softmax function (also called the normalized\n16\n1\n2\n2\nk\nexponential) to the scores. The equation to compute s (x) should look\nfamiliar, as it is just like the equation for Linear Regression prediction (see\nEquation 4-19).\nEquation 4-19. Softmax score for class k\nsk (x) = x⊺θ(k)\nNote that each class has its own dedicated parameter vector θ\n. All these\nvectors are typically stored as rows in a parameter matrix Θ.\nOnce you have computed the score of every class for the instance x, you can\nestimate the probability ˆp that the instance belongs to class k by running the\nscores through the softmax function (Equation 4-20). The function computes\nthe exponential of every score, then normalizes them (dividing by the sum of\nall the exponentials). The scores are generally called logits or log-odds\n(although they are actually unnormalized log-odds).\nEquation 4-20. Softmax function\nˆpk = σ(s(x))k =\nIn this equation:\nK is the number of classes.\ns(x) is a vector containing the scores of each class for the instance x.\nσ(s(x)) is the estimated probability that the instance x belongs to\nclass k, given the scores of each class for that instance.\nJust like the Logistic Regression classifier, the Softmax Regression\nclassifier predicts the class with the highest estimated probability (which is\nsimply the class with the highest score), as shown in Equation 4-21.\nEquation 4-21. Softmax Regression classifier prediction\nk\n(k)\nk\nexp(sk (x))\n∑K\nj=1 exp(sj (x))\nk\nˆy = argmax\nk\nσ(s(x))k = argmax\nk\nsk (x) = argmax\nk\n((θ(k))\n⊺\nx)\nThe argmax operator returns the value of a variable that maximizes a\nfunction. In this equation, it returns the value of k that maximizes the\nestimated probability σ(s(x)) .\nTIP\nThe Softmax Regression classifier predicts only one class at a time (i.e., it is multiclass,\nnot multioutput), so it should be used only with mutually exclusive classes, such as\ndifferent types of plants. You cannot use it to recognize multiple people in one picture.\nNow that you know how the model estimates probabilities and makes\npredictions, let’s take a look at training. The objective is to have a model that\nestimates a high probability for the target class (and consequently a low\nprobability for the other classes). Minimizing the cost function shown in\nEquation 4-22, called the cross entropy, should lead to this objective because\nit penalizes the model when it estimates a low probability for a target class.\nCross entropy is frequently used to measure how well a set of estimated\nclass probabilities matches the target classes.\nEquation 4-22. Cross entropy cost function\nJ(Θ) = −\n∑m\ni=1 ∑K\nk=1 y(i)\nk log(ˆp(i)\nk )\nIn this equation:\ny(i)\nk is the target probability that the i instance belongs to class k.\nIn general, it is either equal to 1 or 0, depending on whether the\ninstance belongs to the class or not.\nNotice that when there are just two classes (K = 2), this cost function is\nequivalent to the Logistic Regression’s cost function (log loss; see Equation\n4-17).\nk\n1\nm\nth\nCROSS ENTROPY\nCross entropy originated from information theory. Suppose you want to\nefficiently transmit information about the weather every day. If there are\neight options (sunny, rainy, etc.), you could encode each option using\nthree bits because 2 = 8. However, if you think it will be sunny almost\nevery day, it would be much more efficient to code “sunny” on just one\nbit (0) and the other seven options on four bits (starting with a 1). Cross\nentropy measures the average number of bits you actually send per\noption. If your assumption about the weather is perfect, cross entropy\nwill be equal to the entropy of the weather itself (i.e., its intrinsic\nunpredictability). But if your assumptions are wrong (e.g., if it rains\noften), cross entropy will be greater by an amount called the Kullback–\nLeibler (KL) divergence.\nThe cross entropy between two probability distributions p and q is\ndefined as H (p, q) = −∑x p (x) log q (x) (at least when the\ndistributions are discrete). For more details, check out my video on the\nsubject.\nThe gradient vector of this cost function with regard to θ\n is given by\nEquation 4-23.\nEquation 4-23. Cross entropy gradient vector for class k\n∇θ(k) J (Θ) =\nm\n∑\ni=1\n(ˆp(i)\nk −y(i)\nk )x(i)\nNow you can compute the gradient vector for every class, then use Gradient\nDescent (or any other optimization algorithm) to find the parameter matrix\nΘ that minimizes the cost function.\nLet’s use Softmax Regression to classify the iris flowers into all three\nclasses. Scikit-Learn’s LogisticRegression uses one-versus-the-rest by\ndefault when you train it on more than two classes, but you can set the\nmulti_class hyperparameter to \"multinomial\" to switch it to Softmax\n3\n(k)\n1\nm\nRegression. You must also specify a solver that supports Softmax\nRegression, such as the \"lbfgs\" solver (see Scikit-Learn’s documentation\nfor more details). It also applies ℓ regularization by default, which you can\ncontrol using the hyperparameter C:\nX = iris[\"data\"][:, (2, 3)] # petal length, petal width \ny = iris[\"target\"] \n \nsoftmax_reg = LogisticRegression(multi_class=\"multinomial\",solver=\"lbfgs\", \nC=10) \nsoftmax_reg.fit(X, y)\nSo the next time you find an iris with petals that are 5 cm long and 2 cm\nwide, you can ask your model to tell you what type of iris it is, and it will\nanswer Iris virginica (class 2) with 94.2% probability (or Iris versicolor\nwith 5.8% probability):\n>>> softmax_reg.predict([[5, 2]]) \narray([2]) \n>>> softmax_reg.predict_proba([[5, 2]]) \narray([[6.38014896e-07, 5.74929995e-02, 9.42506362e-01]])\nFigure 4-25 shows the resulting decision boundaries, represented by the\nbackground colors. Notice that the decision boundaries between any two\nclasses are linear. The figure also shows the probabilities for the Iris\nversicolor class, represented by the curved lines (e.g., the line labeled with\n0.450 represents the 45% probability boundary). Notice that the model can\npredict a class that has an estimated probability below 50%. For example, at\nthe point where all decision boundaries meet, all classes have an equal\nestimated probability of 33%.\n2\nFigure 4-25. Softmax Regression decision boundaries\nExercises\n1. Which Linear Regression training algorithm can you use if you have\na training set with millions of features?\n2. Suppose the features in your training set have very different scales.\nWhich algorithms might suffer from this, and how? What can you\ndo about it?\n3. Can Gradient Descent get stuck in a local minimum when training a\nLogistic Regression model?\n4. Do all Gradient Descent algorithms lead to the same model,\nprovided you let them run long enough?\n5. Suppose you use Batch Gradient Descent and you plot the validation\nerror at every epoch. If you notice that the validation error\nconsistently goes up, what is likely going on? How can you fix this?\n6. Is it a good idea to stop Mini-batch Gradient Descent immediately\nwhen the validation error goes up?\n7. Which Gradient Descent algorithm (among those we discussed) will\nreach the vicinity of the optimal solution the fastest? Which will\nactually converge? How can you make the others converge as well?\n8. Suppose you are using Polynomial Regression. You plot the learning\ncurves and you notice that there is a large gap between the training\nerror and the validation error. What is happening? What are three\nways to solve this?\n9. Suppose you are using Ridge Regression and you notice that the\ntraining error and the validation error are almost equal and fairly\nhigh. Would you say that the model suffers from high bias or high\nvariance? Should you increase the regularization hyperparameter α\nor reduce it?\n10. Why would you want to use:\na. Ridge Regression instead of plain Linear Regression (i.e.,\nwithout any regularization)?\nb. Lasso instead of Ridge Regression?\nc. Elastic Net instead of Lasso?\n11. Suppose you want to classify pictures as outdoor/indoor and\ndaytime/nighttime. Should you implement two Logistic Regression\nclassifiers or one Softmax Regression classifier?\n12. Implement Batch Gradient Descent with early stopping for Softmax\nRegression (without using Scikit-Learn).\nSolutions to these exercises are available in Appendix A.\n1 It is often the case that a learning algorithm will try to optimize a different function than the\nperformance measure used to evaluate the final model. This is generally because that function\nis easier to compute, because it has useful differentiation properties that the performance\nmeasure lacks, or because we want to constrain the model during training, as you will see\nwhen we discuss regularization.\n2 Note that Scikit-Learn separates the bias term (intercept_) from the feature weights\n(coef_).\n3 Technically speaking, its derivative is Lipschitz continuous.\n4 Since feature 1 is smaller, it takes a larger change in θ to affect the cost function, which is\nwhy the bowl is elongated along the θ axis.\n1\n1\n5 Eta (η) is the seventh letter of the Greek alphabet.\n6 While the Normal Equation can only perform Linear Regression, the Gradient Descent\nalgorithms can be used to train many other models, as we will see.\n7 A quadratic equation is of the form y = ax\n2 + bx + c.\n8 This notion of bias is not to be confused with the bias term of linear models.\n9 It is common to use the notation J(θ) for cost functions that don’t have a short name; we will\noften use this notation throughout the rest of this book. The context will make it clear which\ncost function is being discussed.\n10 Norms are discussed in Chapter 2.\n11 A square matrix full of 0s except for 1s on the main diagonal (top left to bottom right).\n12 Alternatively you can use the Ridge class with the \"sag\" solver. Stochastic Average GD is a\nvariant of Stochastic GD. For more details, see the presentation “Minimizing Finite Sums with\nthe Stochastic Average Gradient Algorithm” by Mark Schmidt et al. from the University of\nBritish Columbia.\n13 You can think of a subgradient vector at a nondifferentiable point as an intermediate vector\nbetween the gradient vectors around that point.\n14 Photos reproduced from the corresponding Wikipedia pages. Iris virginica photo by Frank\nMayfield (Creative Commons BY-SA 2.0), Iris versicolor photo by D. Gordon E. Robertson\n(Creative Commons BY-SA 3.0), Iris setosa photo public domain.\n15 NumPy’s reshape() function allows one dimension to be –1, which means “unspecified”:\nthe value is inferred from the length of the array and the remaining dimensions.\n16 It is the the set of points x such that θ + θ x + θ x = 0, which defines a straight line.\n0\n1 1\n2 2" + }, + { + "chapter": 5, + "text": "Chapter 5. Support Vector Machines\nA Support Vector Machine (SVM) is a powerful and versatile Machine Learning model,\ncapable of performing linear or nonlinear classification, regression, and even outlier\ndetection. It is one of the most popular models in Machine Learning, and anyone interested\nin Machine Learning should have it in their toolbox. SVMs are particularly well suited for\nclassification of complex small- or medium-sized datasets.\nThis chapter will explain the core concepts of SVMs, how to use them, and how they work.\nLinear SVM Classification\nThe fundamental idea behind SVMs is best explained with some pictures. Figure 5-1 shows\npart of the iris dataset that was introduced at the end of Chapter 4. The two classes can\nclearly be separated easily with a straight line (they are linearly separable). The left plot\nshows the decision boundaries of three possible linear classifiers. The model whose\ndecision boundary is represented by the dashed line is so bad that it does not even separate\nthe classes properly. The other two models work perfectly on this training set, but their\ndecision boundaries come so close to the instances that these models will probably not\nperform as well on new instances. In contrast, the solid line in the plot on the right\nrepresents the decision boundary of an SVM classifier; this line not only separates the two\nclasses but also stays as far away from the closest training instances as possible. You can\nthink of an SVM classifier as fitting the widest possible street (represented by the parallel\ndashed lines) between the classes. This is called large margin classification.\nFigure 5-1. Large margin classification\nNotice that adding more training instances “off the street” will not affect the decision\nboundary at all: it is fully determined (or “supported”) by the instances located on the edge\nof the street. These instances are called the support vectors (they are circled in Figure 5-1).\nFigure 5-2. Sensitivity to feature scales\nWARNING\nSVMs are sensitive to the feature scales, as you can see in Figure 5-2: in the left plot, the vertical scale is\nmuch larger than the horizontal scale, so the widest possible street is close to horizontal. After feature\nscaling (e.g., using Scikit-Learn’s StandardScaler), the decision boundary in the right plot looks much\nbetter.\nSoft Margin Classification\nIf we strictly impose that all instances must be off the street and on the right side, this is\ncalled hard margin classification. There are two main issues with hard margin\nclassification. First, it only works if the data is linearly separable. Second, it is sensitive to\noutliers. Figure 5-3 shows the iris dataset with just one additional outlier: on the left, it is\nimpossible to find a hard margin; on the right, the decision boundary ends up very different\nfrom the one we saw in Figure 5-1 without the outlier, and it will probably not generalize\nas well.\nFigure 5-3. Hard margin sensitivity to outliers\nTo avoid these issues, use a more flexible model. The objective is to find a good balance\nbetween keeping the street as large as possible and limiting the margin violations (i.e.,\ninstances that end up in the middle of the street or even on the wrong side). This is called\nsoft margin classification.\nWhen creating an SVM model using Scikit-Learn, we can specify a number of\nhyperparameters. C is one of those hyperparameters. If we set it to a low value, then we end\nup with the model on the left of Figure 5-4. With a high value, we get the model on the\nright. Margin violations are bad. It’s usually better to have few of them. However, in this\ncase the model on the left has a lot of margin violations but will probably generalize better.\nFigure 5-4. Large margin (left) versus fewer margin violations (right)\nTIP\nIf your SVM model is overfitting, you can try regularizing it by reducing C.\nThe following Scikit-Learn code loads the iris dataset, scales the features, and then trains a\nlinear SVM model (using the LinearSVC class with C=1 and the hinge loss function,\ndescribed shortly) to detect Iris virginica flowers:\nimport numpy as np \nfrom sklearn import datasets \nfrom sklearn.pipeline import Pipeline \nfrom sklearn.preprocessing import StandardScaler \nfrom sklearn.svm import LinearSVC \n \niris = datasets.load_iris() \nX = iris[\"data\"][:, (2, 3)] # petal length, petal width \ny = (iris[\"target\"] == 2).astype(np.float64) # Iris virginica \n \nsvm_clf = Pipeline([ \n (\"scaler\", StandardScaler()), \n (\"linear_svc\", LinearSVC(C=1, loss=\"hinge\")), \n ]) \n \nsvm_clf.fit(X, y)\nThe resulting model is represented on the left in Figure 5-4.\nThen, as usual, you can use the model to make predictions:\n>>> svm_clf.predict([[5.5, 1.7]]) \narray([1.])\nNOTE\nUnlike Logistic Regression classifiers, SVM classifiers do not output probabilities for each class.\nInstead of using the LinearSVC class, we could use the SVC class with a linear kernel.\nWhen creating the SVC model, we would write SVC(kernel=\"linear\", C=1). Or we\ncould use the SGDClassifier class, with SGDClassifier(loss=\"hinge\",\nalpha=1/(m*C)). This applies regular Stochastic Gradient Descent (see Chapter 4) to train\na linear SVM classifier. It does not converge as fast as the LinearSVC class, but it can be\nuseful to handle online classification tasks or huge datasets that do not fit in memory (out-\nof-core training).\nTIP\nThe LinearSVC class regularizes the bias term, so you should center the training set first by subtracting its\nmean. This is automatic if you scale the data using the StandardScaler. Also make sure you set the loss\nhyperparameter to \"hinge\", as it is not the default value. Finally, for better performance, you should set\nthe dual hyperparameter to False, unless there are more features than training instances (we will discuss\nduality later in the chapter).\nNonlinear SVM Classification\nAlthough linear SVM classifiers are efficient and work surprisingly well in many cases,\nmany datasets are not even close to being linearly separable. One approach to handling\nnonlinear datasets is to add more features, such as polynomial features (as you did in\nChapter 4); in some cases this can result in a linearly separable dataset. Consider the left\nplot in Figure 5-5: it represents a simple dataset with just one feature, x . This dataset is\nnot linearly separable, as you can see. But if you add a second feature x = (x ) , the\nresulting 2D dataset is perfectly linearly separable.\n1\n2\n1 2\nFigure 5-5. Adding features to make a dataset linearly separable\nTo implement this idea using Scikit-Learn, create a Pipeline containing a\nPolynomialFeatures transformer (discussed in “Polynomial Regression”), followed by a\nStandardScaler and a LinearSVC. Let’s test this on the moons dataset: this is a toy dataset\nfor binary classification in which the data points are shaped as two interleaving half circles\n(see Figure 5-6). You can generate this dataset using the make_moons() function:\nfrom sklearn.datasets import make_moons \nfrom sklearn.pipeline import Pipeline \nfrom sklearn.preprocessing import PolynomialFeatures \n \nX, y = make_moons(n_samples=100, noise=0.15) \npolynomial_svm_clf = Pipeline([ \n (\"poly_features\", PolynomialFeatures(degree=3)), \n (\"scaler\", StandardScaler()), \n (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\")) \n ]) \n \npolynomial_svm_clf.fit(X, y)\nFigure 5-6. Linear SVM classifier using polynomial features\nPolynomial Kernel\nAdding polynomial features is simple to implement and can work great with all sorts of\nMachine Learning algorithms (not just SVMs). That said, at a low polynomial degree, this\nmethod cannot deal with very complex datasets, and with a high polynomial degree it\ncreates a huge number of features, making the model too slow.\nFortunately, when using SVMs you can apply an almost miraculous mathematical\ntechnique called the kernel trick (explained in a moment). The kernel trick makes it\npossible to get the same result as if you had added many polynomial features, even with\nvery high-degree polynomials, without actually having to add them. So there is no\ncombinatorial explosion of the number of features because you don’t actually add any\nfeatures. This trick is implemented by the SVC class. Let’s test it on the moons dataset:\nfrom sklearn.svm import SVC \npoly_kernel_svm_clf = Pipeline([ \n (\"scaler\", StandardScaler()), \n (\"svm_clf\", SVC(kernel=\"poly\", degree=3, coef0=1, C=5)) \n ]) \npoly_kernel_svm_clf.fit(X, y)\nThis code trains an SVM classifier using a third-degree polynomial kernel. It is represented\non the left in Figure 5-7. On the right is another SVM classifier using a 10th-degree\npolynomial kernel. Obviously, if your model is overfitting, you might want to reduce the\npolynomial degree. Conversely, if it is underfitting, you can try increasing it. The\nhyperparameter coef0 controls how much the model is influenced by high-degree\npolynomials versus low-degree polynomials.\nFigure 5-7. SVM classifiers with a polynomial kernel\nTIP\nA common approach to finding the right hyperparameter values is to use grid search (see Chapter 2). It is\noften faster to first do a very coarse grid search, then a finer grid search around the best values found.\nHaving a good sense of what each hyperparameter actually does can also help you search in the right part\nof the hyperparameter space.\nSimilarity Features\nAnother technique to tackle nonlinear problems is to add features computed using a\nsimilarity function, which measures how much each instance resembles a particular\nlandmark. For example, let’s take the 1D dataset discussed earlier and add two landmarks\nto it at x = –2 and x = 1 (see the left plot in Figure 5-8). Next, let’s define the similarity\nfunction to be the Gaussian Radial Basis Function (RBF) with γ = 0.3 (see Equation 5-1).\nEquation 5-1. Gaussian RBF\nϕγ (x, ℓ) = exp(−γ∥x −ℓ∥2)\nThis is a bell-shaped function varying from 0 (very far away from the landmark) to 1 (at\nthe landmark). Now we are ready to compute the new features. For example, let’s look at\nthe instance x = –1: it is located at a distance of 1 from the first landmark and 2 from the\nsecond landmark. Therefore its new features are x = exp(–0.3 × 1 ) ≈ 0.74 and x = exp(–\n0.3 × 2 ) ≈ 0.30. The plot on the right in Figure 5-8 shows the transformed dataset\n(dropping the original features). As you can see, it is now linearly separable.\n1\n1\n1\n2\n2\n3\n2\nFigure 5-8. Similarity features using the Gaussian RBF\nYou may wonder how to select the landmarks. The simplest approach is to create a\nlandmark at the location of each and every instance in the dataset. Doing that creates many\ndimensions and thus increases the chances that the transformed training set will be linearly\nseparable. The downside is that a training set with m instances and n features gets\ntransformed into a training set with m instances and m features (assuming you drop the\noriginal features). If your training set is very large, you end up with an equally large\nnumber of features.\nGaussian RBF Kernel\nJust like the polynomial features method, the similarity features method can be useful with\nany Machine Learning algorithm, but it may be computationally expensive to compute all\nthe additional features, especially on large training sets. Once again the kernel trick does\nits SVM magic, making it possible to obtain a similar result as if you had added many\nsimilarity features. Let’s try the SVC class with the Gaussian RBF kernel:\nrbf_kernel_svm_clf = Pipeline([ \n (\"scaler\", StandardScaler()), \n (\"svm_clf\", SVC(kernel=\"rbf\", gamma=5, C=0.001)) \n ]) \nrbf_kernel_svm_clf.fit(X, y)\nThis model is represented at the bottom left in Figure 5-9. The other plots show models\ntrained with different values of hyperparameters gamma (γ) and C. Increasing gamma makes\nthe bell-shaped curve narrower (see the righthand plots in Figure 5-8). As a result, each\ninstance’s range of influence is smaller: the decision boundary ends up being more\nirregular, wiggling around individual instances. Conversely, a small gamma value makes the\nbell-shaped curve wider: instances have a larger range of influence, and the decision\nboundary ends up smoother. So γ acts like a regularization hyperparameter: if your model\nis overfitting, you should reduce it; if it is underfitting, you should increase it (similar to\nthe C hyperparameter).\nFigure 5-9. SVM classifiers using an RBF kernel\nOther kernels exist but are used much more rarely. Some kernels are specialized for\nspecific data structures. String kernels are sometimes used when classifying text\ndocuments or DNA sequences (e.g., using the string subsequence kernel or kernels based\non the Levenshtein distance).\nTIP\nWith so many kernels to choose from, how can you decide which one to use? As a rule of thumb, you\nshould always try the linear kernel first (remember that LinearSVC is much faster than\nSVC(kernel=\"linear\")), especially if the training set is very large or if it has plenty of features. If the\ntraining set is not too large, you should also try the Gaussian RBF kernel; it works well in most cases.\nThen if you have spare time and computing power, you can experiment with a few other kernels, using\ncross-validation and grid search. You’d want to experiment like that especially if there are kernels\nspecialized for your training set’s data structure.\nComputational Complexity\nThe LinearSVC class is based on the liblinear library, which implements an optimized\nalgorithm for linear SVMs. It does not support the kernel trick, but it scales almost\nlinearly with the number of training instances and the number of features. Its training time\ncomplexity is roughly O(m × n).\n1\nThe algorithm takes longer if you require very high precision. This is controlled by the\ntolerance hyperparameter ϵ (called tol in Scikit-Learn). In most classification tasks, the\ndefault tolerance is fine.\nThe SVC class is based on the libsvm library, which implements an algorithm that supports\nthe kernel trick. The training time complexity is usually between O(m × n) and O(m ×\nn). Unfortunately, this means that it gets dreadfully slow when the number of training\ninstances gets large (e.g., hundreds of thousands of instances). This algorithm is perfect for\ncomplex small or medium-sized training sets. It scales well with the number of features,\nespecially with sparse features (i.e., when each instance has few nonzero features). In this\ncase, the algorithm scales roughly with the average number of nonzero features per\ninstance. Table 5-1 compares Scikit-Learn’s SVM classification classes.\nTable 5-1. Comparison of Scikit-Learn classes for SVM classification\nClass\nTime complexity\nOut-of-core support Scaling required Kernel trick\nLinearSVC\nO(m × n)\nNo\nYes\nNo\nSGDClassifier O(m × n)\nYes\nYes\nNo\nSVC\nO(m² × n) to O(m³ × n) No\nYes\nYes\nSVM Regression\nAs mentioned earlier, the SVM algorithm is versatile: not only does it support linear and\nnonlinear classification, but it also supports linear and nonlinear regression. To use SVMs\nfor regression instead of classification, the trick is to reverse the objective: instead of\ntrying to fit the largest possible street between two classes while limiting margin\nviolations, SVM Regression tries to fit as many instances as possible on the street while\nlimiting margin violations (i.e., instances off the street). The width of the street is\ncontrolled by a hyperparameter, ϵ. Figure 5-10 shows two linear SVM Regression models\ntrained on some random linear data, one with a large margin (ϵ = 1.5) and the other with a\nsmall margin (ϵ = 0.5).\n2\n2\n3\nFigure 5-10. SVM Regression\nAdding more training instances within the margin does not affect the model’s predictions;\nthus, the model is said to be ϵ-insensitive.\nYou can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The\nfollowing code produces the model represented on the left in Figure 5-10 (the training data\nshould be scaled and centered first):\nfrom sklearn.svm import LinearSVR \n \nsvm_reg = LinearSVR(epsilon=1.5) \nsvm_reg.fit(X, y)\nTo tackle nonlinear regression tasks, you can use a kernelized SVM model. Figure 5-11\nshows SVM Regression on a random quadratic training set, using a second-degree\npolynomial kernel. There is little regularization in the left plot (i.e., a large C value), and\nmuch more regularization in the right plot (i.e., a small C value).\nFigure 5-11. SVM Regression using a second-degree polynomial kernel\nThe following code uses Scikit-Learn’s SVR class (which supports the kernel trick) to\nproduce the model represented on the left in Figure 5-11:\nfrom sklearn.svm import SVR \n \nsvm_poly_reg = SVR(kernel=\"poly\", degree=2, C=100, epsilon=0.1) \nsvm_poly_reg.fit(X, y)\nThe SVR class is the regression equivalent of the SVC class, and the LinearSVR class is the\nregression equivalent of the LinearSVC class. The LinearSVR class scales linearly with the\nsize of the training set (just like the LinearSVC class), while the SVR class gets much too\nslow when the training set grows large (just like the SVC class).\nNOTE\nSVMs can also be used for outlier detection; see Scikit-Learn’s documentation for more details.\nUnder the Hood\nThis section explains how SVMs make predictions and how their training algorithms work,\nstarting with linear SVM classifiers. If you are just getting started with Machine Learning,\nyou can safely skip it and go straight to the exercises at the end of this chapter, and come\nback later when you want to get a deeper understanding of SVMs.\nFirst, a word about notations. In Chapter 4 we used the convention of putting all the model\nparameters in one vector θ, including the bias term θ and the input feature weights θ to\nθ , and adding a bias input x = 1 to all instances. In this chapter we will use a convention\nthat is more convenient (and more common) when dealing with SVMs: the bias term will\nbe called b, and the feature weights vector will be called w. No bias feature will be added\nto the input feature vectors.\nDecision Function and Predictions\nThe linear SVM classifier model predicts the class of a new instance x by simply\ncomputing the decision function w x + b = w x + ⋯ + w x + b. If the result is positive,\nthe predicted class ŷ is the positive class (1), and otherwise it is the negative class (0); see\nEquation 5-2.\nEquation 5-2. Linear SVM classifier prediction\nˆy ={ 0\nif w⊺x + b < 0,\n1\nif w⊺x + b ≥0\n0\n1\nn\n0\n⊺\n1\n1\nn\nn\nFigure 5-12 shows the decision function that corresponds to the model in the left in\nFigure 5-4: it is a 2D plane because this dataset has two features (petal width and petal\nlength). The decision boundary is the set of points where the decision function is equal to\n0: it is the intersection of two planes, which is a straight line (represented by the thick solid\nline).\nFigure 5-12. Decision function for the iris dataset\nThe dashed lines represent the points where the decision function is equal to 1 or –1: they\nare parallel and at equal distance to the decision boundary, and they form a margin around\nit. Training a linear SVM classifier means finding the values of w and b that make this\nmargin as wide as possible while avoiding margin violations (hard margin) or limiting\nthem (soft margin).\nTraining Objective\nConsider the slope of the decision function: it is equal to the norm of the weight vector, ∥ w\n∥. If we divide this slope by 2, the points where the decision function is equal to ±1 are\ngoing to be twice as far away from the decision boundary. In other words, dividing the\nslope by 2 will multiply the margin by 2. This may be easier to visualize in 2D, as shown in\nFigure 5-13. The smaller the weight vector w, the larger the margin.\n3\nFigure 5-13. A smaller weight vector results in a larger margin\nSo we want to minimize ∥ w ∥ to get a large margin. If we also want to avoid any margin\nviolations (hard margin), then we need the decision function to be greater than 1 for all\npositive training instances and lower than –1 for negative training instances. If we define\nt = –1 for negative instances (if y = 0) and t = 1 for positive instances (if y = 1), then\nwe can express this constraint as t (w x + b) ≥ 1 for all instances.\nWe can therefore express the hard margin linear SVM classifier objective as the\nconstrained optimization problem in Equation 5-3.\nEquation 5-3. Hard margin linear SVM classifier objective\nminimize\nw,b\nw⊺w\nsubject to\nt(i) (w⊺x(i) + b) ≥1\nfor i = 1, 2, ⋯, m\nNOTE\nWe are minimizing w w, which is equal to ∥ w ∥, rather than minimizing ∥ w ∥. Indeed, ∥ w ∥ has a\nnice, simple derivative (it is just w), while ∥ w ∥ is not differentiable at w = 0. Optimization algorithms work\nmuch better on differentiable functions.\nTo get the soft margin objective, we need to introduce a slack variable ζ ≥ 0 for each\ninstance: ζ measures how much the i instance is allowed to violate the margin. We now\nhave two conflicting objectives: make the slack variables as small as possible to reduce the\nmargin violations, and make w w as small as possible to increase the margin. This is\nwhere the C hyperparameter comes in: it allows us to define the tradeoff between these two\nobjectives. This gives us the constrained optimization problem in Equation 5-4.\nEquation 5-4. Soft margin linear SVM classifier objective\n(i)\n(i)\n(i)\n(i)\n(i)\n⊺\n(i)\n1\n2\n1\n2\n⊺\n1\n2\n2\n1\n2\n2\n(i)\n4\n(i)\nth\n1\n2\n⊺\nminimize\nw,b,ζ\nw⊺w + C\nm\n∑\ni=1\nζ(i)\nsubject to\nt(i) (w⊺x(i) + b) ≥1 −ζ(i)\nand\nζ(i) ≥0\nfor i = 1, 2, ⋯, m\nQuadratic Programming\nThe hard margin and soft margin problems are both convex quadratic optimization\nproblems with linear constraints. Such problems are known as Quadratic Programming\n(QP) problems. Many off-the-shelf solvers are available to solve QP problems by using a\nvariety of techniques that are outside the scope of this book.\nThe general problem formulation is given by Equation 5-5.\nEquation 5-5. Quadratic Programming problem\nMinimize\np\np⊺Hp\n+\nf ⊺p\nsubject to\nAp ≤b\nwhere\n⎧\n⎪ \n⎪ \n⎪ \n⎪ \n⎪ \n⎪ \n⎪\n⎨\n⎪ \n⎪ \n⎪ \n⎪ \n⎪ \n⎪ \n⎪\n⎩\np\nis an np-dimensional vector (np = number of parameters),\nH\nis an np × np matrix,\nf\nis an np-dimensional vector,\nA\nis an nc × np matrix (nc = number of constraints),\nb\nis an nc-dimensional vector.\nNote that the expression A p ≤ b defines n constraints: p a ≤ b for i = 1, 2, ⋯, n ,\nwhere a is the vector containing the elements of the i row of A and b is the i element\nof b.\nYou can easily verify that if you set the QP parameters in the following way, you get the\nhard margin linear SVM classifier objective:\nn = n + 1, where n is the number of features (the +1 is for the bias term).\nn = m, where m is the number of training instances.\nH is the n × n identity matrix, except with a zero in the top-left cell (to ignore\nthe bias term).\nf = 0, an n -dimensional vector full of 0s.\nb = –1, an n -dimensional vector full of –1s.\na = –t ˙x , where ˙x is equal to x with an extra bias feature ˙x = 1.\n1\n2\n5\n1\n2\nc\n⊺\n(i)\n(i)\nc\n(i)\nth\n(i)\nth\np\nc\np\np\np\nc\n(i)\n(i)\n(i)\n(i)\n(i)\n0\nOne way to train a hard margin linear SVM classifier is to use an off-the-shelf QP solver\nand pass it the preceding parameters. The resulting vector p will contain the bias term b =\np and the feature weights w = p for i = 1, 2, ⋯, n. Similarly, you can use a QP solver to\nsolve the soft margin problem (see the exercises at the end of the chapter).\nTo use the kernel trick, we are going to look at a different constrained optimization\nproblem.\nThe Dual Problem\nGiven a constrained optimization problem, known as the primal problem, it is possible to\nexpress a different but closely related problem, called its dual problem. The solution to the\ndual problem typically gives a lower bound to the solution of the primal problem, but\nunder some conditions it can have the same solution as the primal problem. Luckily, the\nSVM problem happens to meet these conditions, so you can choose to solve the primal\nproblem or the dual problem; both will have the same solution. Equation 5-6 shows the\ndual form of the linear SVM objective (if you are interested in knowing how to derive the\ndual problem from the primal problem, see Appendix C).\nEquation 5-6. Dual form of the linear SVM objective\nminimize\nα\nm\n∑\ni=1\nm\n∑\nj=1\nα(i)α(j)t(i)t(j)x(i)⊺x(j)\n−\nm\n∑\ni=1\nα(i)\nsubject to\nα(i) ≥0\nfor i = 1, 2, ⋯, m\nOnce you find the vector ˆα that minimizes this equation (using a QP solver), use Equation\n5-7 to compute ˆ\nw and ˆb that minimize the primal problem.\nEquation 5-7. From the dual solution to the primal solution\nˆ\nw =\nm\n∑\ni=1\nˆα(i)t(i)x(i)\nˆb =\nm\n∑(t(i) −ˆ\nw⊺x(i))\nThe dual problem is faster to solve than the primal one when the number of training\ninstances is smaller than the number of features. More importantly, the dual problem\nmakes the kernel trick possible, while the primal does not. So what is this kernel trick,\nanyway?\nKernelized SVMs\n0\ni\ni\n6\n1\n2\n1\nns\ni=1\nˆα(i)>0\nSuppose you want to apply a second-degree polynomial transformation to a two-\ndimensional training set (such as the moons training set), then train a linear SVM classifier\non the transformed training set. Equation 5-8 shows the second-degree polynomial\nmapping function ϕ that you want to apply.\nEquation 5-8. Second-degree polynomial mapping\nϕ(x)= ϕ(( x1\nx2\n))=\n⎛\n⎜\n⎝\nx12\n√2 x1x2\nx22\n⎞\n⎟\n⎠\nNotice that the transformed vector is 3D instead of 2D. Now let’s look at what happens to a\ncouple of 2D vectors, a and b, if we apply this second-degree polynomial mapping and then\ncompute the dot product of the transformed vectors (See Equation 5-9).\nEquation 5-9. Kernel trick for a second-degree polynomial mapping\nϕ(a)⊺ϕ (b)\n=\n⎛\n⎜\n⎝\na12\n√2 a1a2\na22\n⎞\n⎟\n⎠\n⊺\n⎛\n⎜ \n⎜\n⎝\nb1\n2\n√2 b1b2\nb2\n2\n⎞\n⎟ \n⎟\n⎠\n= a12b1\n2 + 2a1b1a2b2 + a22b2\n2\n= (a1b1 + a2b2)2 = (( a1\na2\n)\n⊺\n( b1\nb2\n))\n2\n= (a⊺b)2\nHow about that? The dot product of the transformed vectors is equal to the square of the\ndot product of the original vectors: ϕ(a) ϕ(b) = (a b) .\nHere is the key insight: if you apply the transformation ϕ to all training instances, then the\ndual problem (see Equation 5-6) will contain the dot product ϕ(x ) ϕ(x ). But if ϕ is the\nsecond-degree polynomial transformation defined in Equation 5-8, then you can replace\nthis dot product of transformed vectors simply by (x(i)⊺x(j))\n2\n. So, you don’t need to\ntransform the training instances at all; just replace the dot product by its square in Equation\n5-6. The result will be strictly the same as if you had gone through the trouble of\ntransforming the training set then fitting a linear SVM algorithm, but this trick makes the\nwhole process much more computationally efficient.\nThe function K(a, b) = (a b) is a second-degree polynomial kernel. In Machine Learning,\na kernel is a function capable of computing the dot product ϕ(a) ϕ(b), based only on the\noriginal vectors a and b, without having to compute (or even to know about) the\ntransformation ϕ. Equation 5-10 lists some of the most commonly used kernels.\nEquation 5-10. Common kernels\n7\n⊺\n⊺\n2\n(i) ⊺\n(j)\n⊺\n2\n⊺\nLinear:\nK (a, b) = a⊺b\nPolynomial:\nK (a, b) = (γa⊺b + r)d\nGaussian RBF:\nK (a, b) = exp (−γ∥a −b∥2)\nSigmoid:\nK (a, b) = tanh(γa⊺b + r)\nMERCER’S THEOREM\nAccording to Mercer’s theorem, if a function K(a, b) respects a few mathematical\nconditions called Mercer’s conditions (e.g., K must be continuous and symmetric in its\narguments so that K(a, b) = K(b, a), etc.), then there exists a function ϕ that maps a\nand b into another space (possibly with much higher dimensions) such that K(a, b) =\nϕ(a) ϕ(b). You can use K as a kernel because you know ϕ exists, even if you don’t\nknow what ϕ is. In the case of the Gaussian RBF kernel, it can be shown that ϕ maps\neach training instance to an infinite-dimensional space, so it’s a good thing you don’t\nneed to actually perform the mapping!\nNote that some frequently used kernels (such as the sigmoid kernel) don’t respect all of\nMercer’s conditions, yet they generally work well in practice.\nThere is still one loose end we must tie up. Equation 5-7 shows how to go from the dual\nsolution to the primal solution in the case of a linear SVM classifier. But if you apply the\nkernel trick, you end up with equations that include ϕ(x ). In fact, ˆ\nw must have the same\nnumber of dimensions as ϕ(x ), which may be huge or even infinite, so you can’t compute\nit. But how can you make predictions without knowing ˆ\nw? Well, the good news is that you\ncan plug the formula for ˆ\nw from Equation 5-7 into the decision function for a new instance\nx\n, and you get an equation with only dot products between input vectors. This makes it\npossible to use the kernel trick (Equation 5-11).\nEquation 5-11. Making predictions with a kernelized SVM\nhˆ\nw,ˆb(ϕ(x(n))) = ˆ\nw⊺ϕ (x(n)) + ˆb = (\nm\n∑\ni=1\nˆα(i)t(i)ϕ (x(i)))\n⊺\nϕ (x(n)) + ˆb\n=\nm\n∑\ni=1\nˆα(i)t(i)(ϕ(x(i))\n⊺\nϕ (x(n)))+ˆb\n=\nm\n∑ˆα(i)t(i)K (x(i), x(n)) + ˆb\nNote that since α ≠ 0 only for support vectors, making predictions involves computing\nthe dot product of the new input vector x\n with only the support vectors, not all the\n⊺\n(i)\n(i)\n(n)\ni=1\nˆα(i)>0\n(i)\n(n)\ntraining instances. Of course, you need to use the same trick to compute the bias term ˆb\n(Equation 5-12).\nEquation 5-12. Using the kernel trick to compute the bias term\nˆb\n=\nm\n∑(t(i) −ˆ\nw⊺ϕ (x(i)))=\nm\n∑(t(i) −(\nm\n∑\nj=1\nˆα(j)t(j)ϕ (x(j)))\n⊺\nϕ (x(i)))\n=\nm\n∑\n⎛\n⎜ \n⎜ \n⎜\n⎝\nt(i) −\nm\n∑ˆα(j)t(j)K (x(i), x(j))\n⎞\n⎟ \n⎟ \n⎟\n⎠\nIf you are starting to get a headache, it’s perfectly normal: it’s an unfortunate side effect of\nthe kernel trick.\nOnline SVMs\nBefore concluding this chapter, let’s take a quick look at online SVM classifiers (recall that\nonline learning means learning incrementally, typically as new instances arrive).\nFor linear SVM classifiers, one method for implementing an online SVM classifier is to\nuse Gradient Descent (e.g., using SGDClassifier) to minimize the cost function in\nEquation 5-13, which is derived from the primal problem. Unfortunately, Gradient Descent\nconverges much more slowly than the methods based on QP.\nEquation 5-13. Linear SVM classifier cost function\nJ (w, b) =\nw⊺w\n+\nC\nm\n∑\ni=1\nmax(0, 1 −t(i) (w⊺x(i) + b))\nThe first sum in the cost function will push the model to have a small weight vector w,\nleading to a larger margin. The second sum computes the total of all margin violations. An\ninstance’s margin violation is equal to 0 if it is located off the street and on the correct\nside, or else it is proportional to the distance to the correct side of the street. Minimizing\nthis term ensures that the model makes the margin violations as small and as few as\npossible.\n1\nns\ni=1\nˆα(i)>0\n1\nns\ni=1\nˆα(i)>0\n1\nns\ni=1\nˆα(i)>0\nj=1\nˆα(j)>0\n1\n2\nHINGE LOSS\nThe function max(0, 1 – t) is called the hinge loss function (see the following image). It\nis equal to 0 when t ≥ 1. Its derivative (slope) is equal to –1 if t < 1 and 0 if t > 1. It is\nnot differentiable at t = 1, but just like for Lasso Regression (see “Lasso Regression”),\nyou can still use Gradient Descent using any subderivative at t = 1 (i.e., any value\nbetween –1 and 0).\nIt is also possible to implement online kernelized SVMs, as described in the papers\n“Incremental and Decremental Support Vector Machine Learning” and “Fast Kernel\nClassifiers with Online and Active Learning”. These kernelized SVMs are implemented in\nMatlab and C++. For large-scale nonlinear problems, you may want to consider using\nneural networks instead (see Part II).\nExercises\n1. What is the fundamental idea behind Support Vector Machines?\n2. What is a support vector?\n3. Why is it important to scale the inputs when using SVMs?\n4. Can an SVM classifier output a confidence score when it classifies an instance?\nWhat about a probability?\n5. Should you use the primal or the dual form of the SVM problem to train a model\non a training set with millions of instances and hundreds of features?\n8\n9\n6. Say you’ve trained an SVM classifier with an RBF kernel, but it seems to underfit\nthe training set. Should you increase or decrease γ (gamma)? What about C?\n7. How should you set the QP parameters (H, f, A, and b) to solve the soft margin\nlinear SVM classifier problem using an off-the-shelf QP solver?\n8. Train a LinearSVC on a linearly separable dataset. Then train an SVC and a\nSGDClassifier on the same dataset. See if you can get them to produce roughly\nthe same model.\n9. Train an SVM classifier on the MNIST dataset. Since SVM classifiers are binary\nclassifiers, you will need to use one-versus-the-rest to classify all 10 digits. You\nmay want to tune the hyperparameters using small validation sets to speed up the\nprocess. What accuracy can you reach?\n10. Train an SVM regressor on the California housing dataset.\nSolutions to these exercises are available in Appendix A.\n1 Chih-Jen Lin et al., “A Dual Coordinate Descent Method for Large-Scale Linear SVM,” Proceedings of the 25th\nInternational Conference on Machine Learning (2008): 408–415.\n2 John Platt, “Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machines”\n(Microsoft Research technical report, April 21, 1998), https://www.microsoft.com/en-us/research/wp-\ncontent/uploads/2016/02/tr-98-14.pdf.\n3 More generally, when there are n features, the decision function is an n-dimensional hyperplane, and the\ndecision boundary is an (n – 1)-dimensional hyperplane.\n4 Zeta (ζ) is the sixth letter of the Greek alphabet.\n5 To learn more about Quadratic Programming, you can start by reading Stephen Boyd and Lieven\nVandenberghe’s book Convex Optimization (Cambridge University Press, 2004) or watch Richard Brown’s series\nof video lectures.\n6 The objective function is convex, and the inequality constraints are continuously differentiable and convex\nfunctions.\n7 As explained in Chapter 4, the dot product of two vectors a and b is normally noted a · b. However, in Machine\nLearning, vectors are frequently represented as column vectors (i.e., single-column matrices), so the dot product\nis achieved by computing a\n⊺b. To remain consistent with the rest of the book, we will use this notation here,\nignoring the fact that this technically results in a single-cell matrix rather than a scalar value.\n8 Gert Cauwenberghs and Tomaso Poggio, “Incremental and Decremental Support Vector Machine Learning,”\nProceedings of the 13th International Conference on Neural Information Processing Systems (2000): 388–394.\n9 Antoine Bordes et al., “Fast Kernel Classifiers with Online and Active Learning,” Journal of Machine Learning\nResearch 6 (2005): 1579–1619." + }, + { + "chapter": 6, + "text": "Chapter 6. Decision Trees\nLike SVMs, Decision Trees are versatile Machine Learning algorithms that can perform both\nclassification and regression tasks, and even multioutput tasks. They are powerful\nalgorithms, capable of fitting complex datasets. For example, in Chapter 2 you trained a\nDecisionTreeRegressor model on the California housing dataset, fitting it perfectly\n(actually, overfitting it).\nDecision Trees are also the fundamental components of Random Forests (see Chapter 7),\nwhich are among the most powerful Machine Learning algorithms available today.\nIn this chapter we will start by discussing how to train, visualize, and make predictions with\nDecision Trees. Then we will go through the CART training algorithm used by Scikit-Learn,\nand we will discuss how to regularize trees and use them for regression tasks. Finally, we\nwill discuss some of the limitations of Decision Trees.\nTraining and Visualizing a Decision Tree\nTo understand Decision Trees, let’s build one and take a look at how it makes predictions.\nThe following code trains a DecisionTreeClassifier on the iris dataset (see Chapter 4):\nfrom sklearn.datasets import load_iris \nfrom sklearn.tree import DecisionTreeClassifier \n \niris = load_iris() \nX = iris.data[:, 2:] # petal length and width \ny = iris.target \n \ntree_clf = DecisionTreeClassifier(max_depth=2) \ntree_clf.fit(X, y)\nYou can visualize the trained Decision Tree by first using the export_graphviz() method\nto output a graph definition file called iris_tree.dot:\nfrom sklearn.tree import export_graphviz \n \nexport_graphviz( \n tree_clf, \n out_file=image_path(\"iris_tree.dot\"), \n feature_names=iris.feature_names[2:], \n class_names=iris.target_names, \n rounded=True, \n filled=True \n )\nThen you can use the dot command-line tool from the Graphviz package to convert this .dot\nfile to a variety of formats, such as PDF or PNG. This command line converts the .dot file\nto a .png image file:\n$ dot -Tpng iris_tree.dot -o iris_tree.png \nYour first Decision Tree looks like Figure 6-1.\nFigure 6-1. Iris Decision Tree\nMaking Predictions\nLet’s see how the tree represented in Figure 6-1 makes predictions. Suppose you find an iris\nflower and you want to classify it. You start at the root node (depth 0, at the top): this node\nasks whether the flower’s petal length is smaller than 2.45 cm. If it is, then you move down\nto the root’s left child node (depth 1, left). In this case, it is a leaf node (i.e., it does not have\nany child nodes), so it does not ask any questions: simply look at the predicted class for that\nnode, and the Decision Tree predicts that your flower is an Iris setosa (class=setosa).\nNow suppose you find another flower, and this time the petal length is greater than 2.45 cm.\nYou must move down to the root’s right child node (depth 1, right), which is not a leaf node,\nso the node asks another question: is the petal width smaller than 1.75 cm? If it is, then your\nflower is most likely an Iris versicolor (depth 2, left). If not, it is likely an Iris virginica\n(depth 2, right). It’s really that simple.\n1\nNOTE\nOne of the many qualities of Decision Trees is that they require very little data preparation. In fact, they\ndon’t require feature scaling or centering at all.\nA node’s samples attribute counts how many training instances it applies to. For example,\n100 training instances have a petal length greater than 2.45 cm (depth 1, right), and of those\n100, 54 have a petal width smaller than 1.75 cm (depth 2, left). A node’s value attribute tells\nyou how many training instances of each class this node applies to: for example, the bottom-\nright node applies to 0 Iris setosa, 1 Iris versicolor, and 45 Iris virginica. Finally, a node’s\ngini attribute measures its impurity: a node is “pure” (gini=0) if all training instances it\napplies to belong to the same class. For example, since the depth-1 left node applies only to\nIris setosa training instances, it is pure and its gini score is 0. Equation 6-1 shows how the\ntraining algorithm computes the gini score G of the i node. The depth-2 left node has a\ngini score equal to 1 – (0/54) – (49/54) – (5/54) ≈ 0.168.\nEquation 6-1. Gini impurity\nGi = 1 −\nn\n∑\nk=1\npi,k\n2\nIn this equation:\np is the ratio of class k instances among the training instances in the i node.\nNOTE\nScikit-Learn uses the CART algorithm, which produces only binary trees: nonleaf nodes always have two\nchildren (i.e., questions only have yes/no answers). However, other algorithms such as ID3 can produce\nDecision Trees with nodes that have more than two children.\nFigure 6-2 shows this Decision Tree’s decision boundaries. The thick vertical line represents\nthe decision boundary of the root node (depth 0): petal length = 2.45 cm. Since the lefthand\narea is pure (only Iris setosa), it cannot be split any further. However, the righthand area is\nimpure, so the depth-1 right node splits it at petal width = 1.75 cm (represented by the\ndashed line). Since max_depth was set to 2, the Decision Tree stops right there. If you set\nmax_depth to 3, then the two depth-2 nodes would each add another decision boundary\n(represented by the dotted lines).\ni\nth\n2\n2\n2\ni,k\nth\nFigure 6-2. Decision Tree decision boundaries\nMODEL INTERPRETATION: WHITE BOX VERSUS BLACK BOX\nDecision Trees are intuitive, and their decisions are easy to interpret. Such models are\noften called white box models. In contrast, as we will see, Random Forests or neural\nnetworks are generally considered black box models. They make great predictions, and\nyou can easily check the calculations that they performed to make these predictions;\nnevertheless, it is usually hard to explain in simple terms why the predictions were\nmade. For example, if a neural network says that a particular person appears on a\npicture, it is hard to know what contributed to this prediction: did the model recognize\nthat person’s eyes? Their mouth? Their nose? Their shoes? Or even the couch that they\nwere sitting on? Conversely, Decision Trees provide nice, simple classification rules that\ncan even be applied manually if need be (e.g., for flower classification).\nEstimating Class Probabilities\nA Decision Tree can also estimate the probability that an instance belongs to a particular\nclass k. First it traverses the tree to find the leaf node for this instance, and then it returns the\nratio of training instances of class k in this node. For example, suppose you have found a\nflower whose petals are 5 cm long and 1.5 cm wide. The corresponding leaf node is the\ndepth-2 left node, so the Decision Tree should output the following probabilities: 0% for Iris\nsetosa (0/54), 90.7% for Iris versicolor (49/54), and 9.3% for Iris virginica (5/54). And if\nyou ask it to predict the class, it should output Iris versicolor (class 1) because it has the\nhighest probability. Let’s check this:\n>>> tree_clf.predict_proba([[5, 1.5]]) \narray([[0. , 0.90740741, 0.09259259]]) \n>>> tree_clf.predict([[5, 1.5]]) \narray([1])\nPerfect! Notice that the estimated probabilities would be identical anywhere else in the\nbottom-right rectangle of Figure 6-2—for example, if the petals were 6 cm long and 1.5 cm\nwide (even though it seems obvious that it would most likely be an Iris virginica in this\ncase).\nThe CART Training Algorithm\nScikit-Learn uses the Classification and Regression Tree (CART) algorithm to train Decision\nTrees (also called “growing” trees). The algorithm works by first splitting the training set\ninto two subsets using a single feature k and a threshold t (e.g., “petal length ≤ 2.45 cm”).\nHow does it choose k and t ? It searches for the pair (k, t ) that produces the purest subsets\n(weighted by their size). Equation 6-2 gives the cost function that the algorithm tries to\nminimize.\nEquation 6-2. CART cost function for classification\nJ (k, tk) =\nGleft +\nGright\nwhere {\nGleft/right measures the impurity of the left/right subset,\nmleft/right is the number of instances in the left/right subset.\nOnce the CART algorithm has successfully split the training set in two, it splits the subsets\nusing the same logic, then the sub-subsets, and so on, recursively. It stops recursing once it\nreaches the maximum depth (defined by the max_depth hyperparameter), or if it cannot find\na split that will reduce impurity. A few other hyperparameters (described in a moment)\ncontrol additional stopping conditions (min_samples_split, min_samples_leaf,\nmin_weight_fraction_leaf, and max_leaf_nodes).\nWARNING\nAs you can see, the CART algorithm is a greedy algorithm: it greedily searches for an optimum split at the\ntop level, then repeats the process at each subsequent level. It does not check whether or not the split will\nlead to the lowest possible impurity several levels down. A greedy algorithm often produces a solution that’s\nreasonably good but not guaranteed to be optimal.\nUnfortunately, finding the optimal tree is known to be an NP-Complete problem: it requires O(exp(m))\ntime, making the problem intractable even for small training sets. This is why we must settle for a\n“reasonably good” solution.\nComputational Complexity\nk\nk\nk\nmleft\nm\nmright\nm\n2\nMaking predictions requires traversing the Decision Tree from the root to a leaf. Decision\nTrees generally are approximately balanced, so traversing the Decision Tree requires going\nthrough roughly O(log (m)) nodes. Since each node only requires checking the value of one\nfeature, the overall prediction complexity is O(log (m)), independent of the number of\nfeatures. So predictions are very fast, even when dealing with large training sets.\nThe training algorithm compares all features (or less if max_features is set) on all samples\nat each node. Comparing all features on all samples at each node results in a training\ncomplexity of O(n × m log (m)). For small training sets (less than a few thousand instances),\nScikit-Learn can speed up training by presorting the data (set presort=True), but doing that\nslows down training considerably for larger training sets.\nGini Impurity or Entropy?\nBy default, the Gini impurity measure is used, but you can select the entropy impurity\nmeasure instead by setting the criterion hyperparameter to \"entropy\". The concept of\nentropy originated in thermodynamics as a measure of molecular disorder: entropy\napproaches zero when molecules are still and well ordered. Entropy later spread to a wide\nvariety of domains, including Shannon’s information theory, where it measures the average\ninformation content of a message: entropy is zero when all messages are identical. In\nMachine Learning, entropy is frequently used as an impurity measure: a set’s entropy is zero\nwhen it contains instances of only one class. Equation 6-3 shows the definition of the\nentropy of the i node. For example, the depth-2 left node in Figure 6-1 has an entropy equal\nto −\nlog2 (\n) −\nlog2 (\n) ≈ 0.445.\nEquation 6-3. Entropy\nHi = −\nn\n∑pi,k log2 (pi,k)\nSo, should you use Gini impurity or entropy? The truth is, most of the time it does not make\na big difference: they lead to similar trees. Gini impurity is slightly faster to compute, so it\nis a good default. However, when they differ, Gini impurity tends to isolate the most\nfrequent class in its own branch of the tree, while entropy tends to produce slightly more\nbalanced trees.\nRegularization Hyperparameters\nDecision Trees make very few assumptions about the training data (as opposed to linear\nmodels, which assume that the data is linear, for example). If left unconstrained, the tree\nstructure will adapt itself to the training data, fitting it very closely—indeed, most likely\n2\n3\n2\n2\n4\nth\n49\n54\n49\n54\n5\n54\n5\n54\nk=1\npi,k≠0\n5\noverfitting it. Such a model is often called a nonparametric model, not because it does not\nhave any parameters (it often has a lot) but because the number of parameters is not\ndetermined prior to training, so the model structure is free to stick closely to the data. In\ncontrast, a parametric model, such as a linear model, has a predetermined number of\nparameters, so its degree of freedom is limited, reducing the risk of overfitting (but\nincreasing the risk of underfitting).\nTo avoid overfitting the training data, you need to restrict the Decision Tree’s freedom\nduring training. As you know by now, this is called regularization. The regularization\nhyperparameters depend on the algorithm used, but generally you can at least restrict the\nmaximum depth of the Decision Tree. In Scikit-Learn, this is controlled by the max_depth\nhyperparameter (the default value is None, which means unlimited). Reducing max_depth\nwill regularize the model and thus reduce the risk of overfitting.\nThe DecisionTreeClassifier class has a few other parameters that similarly restrict the\nshape of the Decision Tree: min_samples_split (the minimum number of samples a node\nmust have before it can be split), min_samples_leaf (the minimum number of samples a\nleaf node must have), min_weight_fraction_leaf (same as min_samples_leaf but\nexpressed as a fraction of the total number of weighted instances), max_leaf_nodes (the\nmaximum number of leaf nodes), and max_features (the maximum number of features that\nare evaluated for splitting at each node). Increasing min_* hyperparameters or reducing\nmax_* hyperparameters will regularize the model.\nNOTE\nOther algorithms work by first training the Decision Tree without restrictions, then pruning (deleting)\nunnecessary nodes. A node whose children are all leaf nodes is considered unnecessary if the purity\nimprovement it provides is not statistically significant. Standard statistical tests, such as the χ test (chi-\nsquared test), are used to estimate the probability that the improvement is purely the result of chance (which\nis called the null hypothesis). If this probability, called the p-value, is higher than a given threshold\n(typically 5%, controlled by a hyperparameter), then the node is considered unnecessary and its children are\ndeleted. The pruning continues until all unnecessary nodes have been pruned.\nFigure 6-3 shows two Decision Trees trained on the moons dataset (introduced in Chapter 5).\nOn the left the Decision Tree is trained with the default hyperparameters (i.e., no\nrestrictions), and on the right it’s trained with min_samples_leaf=4. It is quite obvious that\nthe model on the left is overfitting, and the model on the right will probably generalize\nbetter.\n2\nFigure 6-3. Regularization using min_samples_leaf\nRegression\nDecision Trees are also capable of performing regression tasks. Let’s build a regression tree\nusing Scikit-Learn’s DecisionTreeRegressor class, training it on a noisy quadratic dataset\nwith max_depth=2:\nfrom sklearn.tree import DecisionTreeRegressor \n \ntree_reg = DecisionTreeRegressor(max_depth=2) \ntree_reg.fit(X, y)\nThe resulting tree is represented in Figure 6-4.\nFigure 6-4. A Decision Tree for regression\nThis tree looks very similar to the classification tree you built earlier. The main difference is\nthat instead of predicting a class in each node, it predicts a value. For example, suppose you\nwant to make a prediction for a new instance with x = 0.6. You traverse the tree starting at\nthe root, and you eventually reach the leaf node that predicts value=0.111. This prediction\nis the average target value of the 110 training instances associated with this leaf node, and it\nresults in a mean squared error equal to 0.015 over these 110 instances.\nThis model’s predictions are represented on the left in Figure 6-5. If you set max_depth=3,\nyou get the predictions represented on the right. Notice how the predicted value for each\nregion is always the average target value of the instances in that region. The algorithm splits\neach region in a way that makes most training instances as close as possible to that predicted\nvalue.\nFigure 6-5. Predictions of two Decision Tree regression models\nThe CART algorithm works mostly the same way as earlier, except that instead of trying to\nsplit the training set in a way that minimizes impurity, it now tries to split the training set in\na way that minimizes the MSE. Equation 6-4 shows the cost function that the algorithm tries\nto minimize.\nEquation 6-4. CART cost function for regression\nJ (k, tk) =\nMSEleft +\nMSEright\nwhere\n⎧\n⎪\n⎨\n⎪\n⎩\nMSEnode = ∑i∈node (ˆynode −y(i))\n2\nˆynode =\n∑i∈node y(i)\nJust like for classification tasks, Decision Trees are prone to overfitting when dealing with\nregression tasks. Without any regularization (i.e., using the default hyperparameters), you\nget the predictions on the left in Figure 6-6. These predictions are obviously overfitting the\ntraining set very badly. Just setting min_samples_leaf=10 results in a much more\nreasonable model, represented on the right in Figure 6-6.\n1\nmleft\nm\nmright\nm\n1\nmnode\nFigure 6-6. Regularizing a Decision Tree regressor\nInstability\nHopefully by now you are convinced that Decision Trees have a lot going for them: they are\nsimple to understand and interpret, easy to use, versatile, and powerful. However, they do\nhave a few limitations. First, as you may have noticed, Decision Trees love orthogonal\ndecision boundaries (all splits are perpendicular to an axis), which makes them sensitive to\ntraining set rotation. For example, Figure 6-7 shows a simple linearly separable dataset: on\nthe left, a Decision Tree can split it easily, while on the right, after the dataset is rotated by\n45°, the decision boundary looks unnecessarily convoluted. Although both Decision Trees fit\nthe training set perfectly, it is very likely that the model on the right will not generalize well.\nOne way to limit this problem is to use Principal Component Analysis (see Chapter 8),\nwhich often results in a better orientation of the training data.\nFigure 6-7. Sensitivity to training set rotation\nMore generally, the main issue with Decision Trees is that they are very sensitive to small\nvariations in the training data. For example, if you just remove the widest Iris versicolor\nfrom the iris training set (the one with petals 4.8 cm long and 1.8 cm wide) and train a new\nDecision Tree, you may get the model represented in Figure 6-8. As you can see, it looks\nvery different from the previous Decision Tree (Figure 6-2). Actually, since the training\nalgorithm used by Scikit-Learn is stochastic, you may get very different models even on the\nsame training data (unless you set the random_state hyperparameter).\nFigure 6-8. Sensitivity to training set details\nRandom Forests can limit this instability by averaging predictions over many trees, as we\nwill see in the next chapter.\nExercises\n1. What is the approximate depth of a Decision Tree trained (without restrictions) on a\ntraining set with one million instances?\n2. Is a node’s Gini impurity generally lower or greater than its parent’s? Is it generally\nlower/greater, or always lower/greater?\n3. If a Decision Tree is overfitting the training set, is it a good idea to try decreasing\nmax_depth?\n4. If a Decision Tree is underfitting the training set, is it a good idea to try scaling the\ninput features?\n5. If it takes one hour to train a Decision Tree on a training set containing 1 million\ninstances, roughly how much time will it take to train another Decision Tree on a\ntraining set containing 10 million instances?\n6. If your training set contains 100,000 instances, will setting presort=True speed up\ntraining?\n7. Train and fine-tune a Decision Tree for the moons dataset by following these steps:\n6\na. Use make_moons(n_samples=10000, noise=0.4) to generate a moons\ndataset.\nb. Use train_test_split() to split the dataset into a training set and a test\nset.\nc. Use grid search with cross-validation (with the help of the GridSearchCV\nclass) to find good hyperparameter values for a DecisionTreeClassifier.\nHint: try various values for max_leaf_nodes.\nd. Train it on the full training set using these hyperparameters, and measure\nyour model’s performance on the test set. You should get roughly 85% to\n87% accuracy.\n8. Grow a forest by following these steps:\na. Continuing the previous exercise, generate 1,000 subsets of the training set,\neach containing 100 instances selected randomly. Hint: you can use Scikit-\nLearn’s ShuffleSplit class for this.\nb. Train one Decision Tree on each subset, using the best hyperparameter\nvalues found in the previous exercise. Evaluate these 1,000 Decision Trees\non the test set. Since they were trained on smaller sets, these Decision\nTrees will likely perform worse than the first Decision Tree, achieving only\nabout 80% accuracy.\nc. Now comes the magic. For each test set instance, generate the predictions\nof the 1,000 Decision Trees, and keep only the most frequent prediction\n(you can use SciPy’s mode() function for this). This approach gives you\nmajority-vote predictions over the test set.\nd. Evaluate these predictions on the test set: you should obtain a slightly\nhigher accuracy than your first model (about 0.5 to 1.5% higher).\nCongratulations, you have trained a Random Forest classifier!\nSolutions to these exercises are available in Appendix A.\n1 Graphviz is an open source graph visualization software package, available at http://www.graphviz.org/.\n2 P is the set of problems that can be solved in polynomial time. NP is the set of problems whose solutions can be\nverified in polynomial time. An NP-Hard problem is a problem to which any NP problem can be reduced in\npolynomial time. An NP-Complete problem is both NP and NP-Hard. A major open mathematical question is\nwhether or not P = NP. If P ≠ NP (which seems likely), then no polynomial algorithm will ever be found for any\nNP-Complete problem (except perhaps on a quantum computer).\n3 log is the binary logarithm. It is equal to log (m) = log(m) / log(2).\n4 A reduction of entropy is often called an information gain.\n2\n2\n5 See Sebastian Raschka’s interesting analysis for more details.\n6 It randomly selects the set of features to evaluate at each node." + }, + { + "chapter": 7, + "text": "Chapter 7. Ensemble Learning and\nRandom Forests\nSuppose you pose a complex question to thousands of random people, then aggregate\ntheir answers. In many cases you will find that this aggregated answer is better than an\nexpert’s answer. This is called the wisdom of the crowd. Similarly, if you aggregate the\npredictions of a group of predictors (such as classifiers or regressors), you will often get\nbetter predictions than with the best individual predictor. A group of predictors is called\nan ensemble; thus, this technique is called Ensemble Learning, and an Ensemble\nLearning algorithm is called an Ensemble method.\nAs an example of an Ensemble method, you can train a group of Decision Tree\nclassifiers, each on a different random subset of the training set. To make predictions,\nyou obtain the predictions of all the individual trees, then predict the class that gets the\nmost votes (see the last exercise in Chapter 6). Such an ensemble of Decision Trees is\ncalled a Random Forest, and despite its simplicity, this is one of the most powerful\nMachine Learning algorithms available today.\nAs discussed in Chapter 2, you will often use Ensemble methods near the end of a\nproject, once you have already built a few good predictors, to combine them into an\neven better predictor. In fact, the winning solutions in Machine Learning competitions\noften involve several Ensemble methods (most famously in the Netflix Prize\ncompetition).\nIn this chapter we will discuss the most popular Ensemble methods, including bagging,\nboosting, and stacking. We will also explore Random Forests.\nVoting Classifiers\nSuppose you have trained a few classifiers, each one achieving about 80% accuracy.\nYou may have a Logistic Regression classifier, an SVM classifier, a Random Forest\nclassifier, a K-Nearest Neighbors classifier, and perhaps a few more (see Figure 7-1).\nFigure 7-1. Training diverse classifiers\nA very simple way to create an even better classifier is to aggregate the predictions of\neach classifier and predict the class that gets the most votes. This majority-vote\nclassifier is called a hard voting classifier (see Figure 7-2).\nFigure 7-2. Hard voting classifier predictions\nSomewhat surprisingly, this voting classifier often achieves a higher accuracy than the\nbest classifier in the ensemble. In fact, even if each classifier is a weak learner\n(meaning it does only slightly better than random guessing), the ensemble can still be a\nstrong learner (achieving high accuracy), provided there are a sufficient number of\nweak learners and they are sufficiently diverse.\nHow is this possible? The following analogy can help shed some light on this mystery.\nSuppose you have a slightly biased coin that has a 51% chance of coming up heads and\n49% chance of coming up tails. If you toss it 1,000 times, you will generally get more\nor less 510 heads and 490 tails, and hence a majority of heads. If you do the math, you\nwill find that the probability of obtaining a majority of heads after 1,000 tosses is close\nto 75%. The more you toss the coin, the higher the probability (e.g., with 10,000 tosses,\nthe probability climbs over 97%). This is due to the law of large numbers: as you keep\ntossing the coin, the ratio of heads gets closer and closer to the probability of heads\n(51%). Figure 7-3 shows 10 series of biased coin tosses. You can see that as the number\nof tosses increases, the ratio of heads approaches 51%. Eventually all 10 series end up\nso close to 51% that they are consistently above 50%.\nFigure 7-3. The law of large numbers\nSimilarly, suppose you build an ensemble containing 1,000 classifiers that are\nindividually correct only 51% of the time (barely better than random guessing). If you\npredict the majority voted class, you can hope for up to 75% accuracy! However, this is\nonly true if all classifiers are perfectly independent, making uncorrelated errors, which\nis clearly not the case because they are trained on the same data. They are likely to\nmake the same types of errors, so there will be many majority votes for the wrong class,\nreducing the ensemble’s accuracy.\nTIP\nEnsemble methods work best when the predictors are as independent from one another as possible.\nOne way to get diverse classifiers is to train them using very different algorithms. This increases the\nchance that they will make very different types of errors, improving the ensemble’s accuracy.\nThe following code creates and trains a voting classifier in Scikit-Learn, composed of\nthree diverse classifiers (the training set is the moons dataset, introduced in Chapter 5):\nfrom sklearn.ensemble import RandomForestClassifier \nfrom sklearn.ensemble import VotingClassifier \nfrom sklearn.linear_model import LogisticRegression \nfrom sklearn.svm import SVC \n \nlog_clf = LogisticRegression() \nrnd_clf = RandomForestClassifier() \nsvm_clf = SVC() \n \nvoting_clf = VotingClassifier( \n estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)], \n voting='hard') \nvoting_clf.fit(X_train, y_train)\nLet’s look at each classifier’s accuracy on the test set:\n>>> from sklearn.metrics import accuracy_score \n>>> for clf in (log_clf, rnd_clf, svm_clf, voting_clf): \n... clf.fit(X_train, y_train) \n... y_pred = clf.predict(X_test) \n... print(clf.__class__.__name__, accuracy_score(y_test, y_pred)) \n... \nLogisticRegression 0.864 \nRandomForestClassifier 0.896 \nSVC 0.888 \nVotingClassifier 0.904\nThere you have it! The voting classifier slightly outperforms all the individual\nclassifiers.\nIf all classifiers are able to estimate class probabilities (i.e., they all have a\npredict_proba() method), then you can tell Scikit-Learn to predict the class with the\nhighest class probability, averaged over all the individual classifiers. This is called soft\nvoting. It often achieves higher performance than hard voting because it gives more\nweight to highly confident votes. All you need to do is replace voting=\"hard\" with\nvoting=\"soft\" and ensure that all classifiers can estimate class probabilities. This is\nnot the case for the SVC class by default, so you need to set its probability\nhyperparameter to True (this will make the SVC class use cross-validation to estimate\nclass probabilities, slowing down training, and it will add a predict_proba() method).\nIf you modify the preceding code to use soft voting, you will find that the voting\nclassifier achieves over 91.2% accuracy!\nBagging and Pasting\nOne way to get a diverse set of classifiers is to use very different training algorithms, as\njust discussed. Another approach is to use the same training algorithm for every\npredictor and train them on different random subsets of the training set. When sampling\nis performed with replacement, this method is called bagging (short for bootstrap\naggregating ). When sampling is performed without replacement, it is called pasting.\nIn other words, both bagging and pasting allow training instances to be sampled several\ntimes across multiple predictors, but only bagging allows training instances to be\nsampled several times for the same predictor. This sampling and training process is\nrepresented in Figure 7-4.\nFigure 7-4. Bagging and pasting involves training several predictors on different random samples of the training\nset\nOnce all predictors are trained, the ensemble can make a prediction for a new instance\nby simply aggregating the predictions of all predictors. The aggregation function is\ntypically the statistical mode (i.e., the most frequent prediction, just like a hard voting\nclassifier) for classification, or the average for regression. Each individual predictor has\na higher bias than if it were trained on the original training set, but aggregation reduces\nboth bias and variance. Generally, the net result is that the ensemble has a similar bias\nbut a lower variance than a single predictor trained on the original training set.\nAs you can see in Figure 7-4, predictors can all be trained in parallel, via different CPU\ncores or even different servers. Similarly, predictions can be made in parallel. This is\none of the reasons bagging and pasting are such popular methods: they scale very well.\n1\n2\n3\n4\nBagging and Pasting in Scikit-Learn\nScikit-Learn offers a simple API for both bagging and pasting with the\nBaggingClassifier class (or BaggingRegressor for regression). The following code\ntrains an ensemble of 500 Decision Tree classifiers: each is trained on 100 training\ninstances randomly sampled from the training set with replacement (this is an example\nof bagging, but if you want to use pasting instead, just set bootstrap=False). The\nn_jobs parameter tells Scikit-Learn the number of CPU cores to use for training and\npredictions (–1 tells Scikit-Learn to use all available cores):\nfrom sklearn.ensemble import BaggingClassifier \nfrom sklearn.tree import DecisionTreeClassifier \n \nbag_clf = BaggingClassifier( \n DecisionTreeClassifier(), n_estimators=500, \n max_samples=100, bootstrap=True, n_jobs=-1) \nbag_clf.fit(X_train, y_train) \ny_pred = bag_clf.predict(X_test)\nNOTE\nThe BaggingClassifier automatically performs soft voting instead of hard voting if the base\nclassifier can estimate class probabilities (i.e., if it has a predict_proba() method), which is the case\nwith Decision Tree classifiers.\nFigure 7-5 compares the decision boundary of a single Decision Tree with the decision\nboundary of a bagging ensemble of 500 trees (from the preceding code), both trained on\nthe moons dataset. As you can see, the ensemble’s predictions will likely generalize\nmuch better than the single Decision Tree’s predictions: the ensemble has a comparable\nbias but a smaller variance (it makes roughly the same number of errors on the training\nset, but the decision boundary is less irregular).\n5\nFigure 7-5. A single Decision Tree (left) versus a bagging ensemble of 500 trees (right)\nBootstrapping introduces a bit more diversity in the subsets that each predictor is\ntrained on, so bagging ends up with a slightly higher bias than pasting; but the extra\ndiversity also means that the predictors end up being less correlated, so the ensemble’s\nvariance is reduced. Overall, bagging often results in better models, which explains why\nit is generally preferred. However, if you have spare time and CPU power, you can use\ncross-validation to evaluate both bagging and pasting and select the one that works best.\nOut-of-Bag Evaluation\nWith bagging, some instances may be sampled several times for any given predictor,\nwhile others may not be sampled at all. By default a BaggingClassifier samples m\ntraining instances with replacement (bootstrap=True), where m is the size of the\ntraining set. This means that only about 63% of the training instances are sampled on\naverage for each predictor. The remaining 37% of the training instances that are not\nsampled are called out-of-bag (oob) instances. Note that they are not the same 37% for\nall predictors.\nSince a predictor never sees the oob instances during training, it can be evaluated on\nthese instances, without the need for a separate validation set. You can evaluate the\nensemble itself by averaging out the oob evaluations of each predictor.\nIn Scikit-Learn, you can set oob_score=True when creating a BaggingClassifier to\nrequest an automatic oob evaluation after training. The following code demonstrates\nthis. The resulting evaluation score is available through the oob_score_ variable:\n>>> bag_clf = BaggingClassifier( \n... DecisionTreeClassifier(), n_estimators=500, \n... bootstrap=True, n_jobs=-1, oob_score=True) \n... \n>>> bag_clf.fit(X_train, y_train) \n6\n>>> bag_clf.oob_score_ \n0.90133333333333332\nAccording to this oob evaluation, this BaggingClassifier is likely to achieve about\n90.1% accuracy on the test set. Let’s verify this:\n>>> from sklearn.metrics import accuracy_score \n>>> y_pred = bag_clf.predict(X_test) \n>>> accuracy_score(y_test, y_pred) \n0.91200000000000003\nWe get 91.2% accuracy on the test set—close enough!\nThe oob decision function for each training instance is also available through the\noob_decision_function_ variable. In this case (since the base estimator has a\npredict_proba() method), the decision function returns the class probabilities for\neach training instance. For example, the oob evaluation estimates that the first training\ninstance has a 68.25% probability of belonging to the positive class (and 31.75% of\nbelonging to the negative class):\n>>> bag_clf.oob_decision_function_ \narray([[0.31746032, 0.68253968], \n [0.34117647, 0.65882353], \n [1. , 0. ], \n ... \n [1. , 0. ], \n [0.03108808, 0.96891192], \n [0.57291667, 0.42708333]])\nRandom Patches and Random Subspaces\nThe BaggingClassifier class supports sampling the features as well. Sampling is\ncontrolled by two hyperparameters: max_features and bootstrap_features. They\nwork the same way as max_samples and bootstrap, but for feature sampling instead of\ninstance sampling. Thus, each predictor will be trained on a random subset of the input\nfeatures.\nThis technique is particularly useful when you are dealing with high-dimensional inputs\n(such as images). Sampling both training instances and features is called the Random\nPatches method. Keeping all training instances (by setting bootstrap=False and\nmax_samples=1.0) but sampling features (by setting bootstrap_features to True\nand/or max_features to a value smaller than 1.0) is called the Random Subspaces\nmethod.\n7\n8\nSampling features results in even more predictor diversity, trading a bit more bias for a\nlower variance.\nRandom Forests\nAs we have discussed, a Random Forest is an ensemble of Decision Trees, generally\ntrained via the bagging method (or sometimes pasting), typically with max_samples set\nto the size of the training set. Instead of building a BaggingClassifier and passing it a\nDecisionTreeClassifier, you can instead use the RandomForestClassifier class,\nwhich is more convenient and optimized for Decision Trees\n (similarly, there is a\nRandomForestRegressor class for regression tasks). The following code uses all\navailable CPU cores to train a Random Forest classifier with 500 trees (each limited to\nmaximum 16 nodes):\nfrom sklearn.ensemble import RandomForestClassifier \n \nrnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1) \nrnd_clf.fit(X_train, y_train) \n \ny_pred_rf = rnd_clf.predict(X_test)\nWith a few exceptions, a RandomForestClassifier has all the hyperparameters of a\nDecisionTreeClassifier (to control how trees are grown), plus all the\nhyperparameters of a BaggingClassifier to control the ensemble itself.\nThe Random Forest algorithm introduces extra randomness when growing trees; instead\nof searching for the very best feature when splitting a node (see Chapter 6), it searches\nfor the best feature among a random subset of features. The algorithm results in greater\ntree diversity, which (again) trades a higher bias for a lower variance, generally yielding\nan overall better model. The following BaggingClassifier is roughly equivalent to the\nprevious RandomForestClassifier:\nbag_clf = BaggingClassifier( \n DecisionTreeClassifier(splitter=\"random\", max_leaf_nodes=16), \n n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1)\nExtra-Trees\nWhen you are growing a tree in a Random Forest, at each node only a random subset of\nthe features is considered for splitting (as discussed earlier). It is possible to make trees\neven more random by also using random thresholds for each feature rather than\nsearching for the best possible thresholds (like regular Decision Trees do).\n9\n10\n11\nA forest of such extremely random trees is called an Extremely Randomized Trees\nensemble\n (or Extra-Trees for short). Once again, this technique trades more bias for a\nlower variance. It also makes Extra-Trees much faster to train than regular Random\nForests, because finding the best possible threshold for each feature at every node is one\nof the most time-consuming tasks of growing a tree.\nYou can create an Extra-Trees classifier using Scikit-Learn’s ExtraTreesClassifier\nclass. Its API is identical to the RandomForestClassifier class. Similarly, the\nExtraTreesRegressor class has the same API as the RandomForestRegressor class.\nTIP\nIt is hard to tell in advance whether a RandomForestClassifier will perform better or worse than an\nExtraTreesClassifier. Generally, the only way to know is to try both and compare them using\ncross-validation (tuning the hyperparameters using grid search).\nFeature Importance\nYet another great quality of Random Forests is that they make it easy to measure the\nrelative importance of each feature. Scikit-Learn measures a feature’s importance by\nlooking at how much the tree nodes that use that feature reduce impurity on average\n(across all trees in the forest). More precisely, it is a weighted average, where each\nnode’s weight is equal to the number of training samples that are associated with it (see\nChapter 6).\nScikit-Learn computes this score automatically for each feature after training, then it\nscales the results so that the sum of all importances is equal to 1. You can access the\nresult using the feature_importances_ variable. For example, the following code\ntrains a RandomForestClassifier on the iris dataset (introduced in Chapter 4) and\noutputs each feature’s importance. It seems that the most important features are the\npetal length (44%) and width (42%), while sepal length and width are rather\nunimportant in comparison (11% and 2%, respectively):\n>>> from sklearn.datasets import load_iris \n>>> iris = load_iris() \n>>> rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1) \n>>> rnd_clf.fit(iris[\"data\"], iris[\"target\"]) \n>>> for name, score in zip(iris[\"feature_names\"], rnd_clf.feature_importances_): \n... print(name, score) \n... \nsepal length (cm) 0.112492250999 \nsepal width (cm) 0.0231192882825 \npetal length (cm) 0.441030464364 \npetal width (cm) 0.423357996355\n12\nSimilarly, if you train a Random Forest classifier on the MNIST dataset (introduced in\nChapter 3) and plot each pixel’s importance, you get the image represented in Figure 7-\n6.\nFigure 7-6. MNIST pixel importance (according to a Random Forest classifier)\nRandom Forests are very handy to get a quick understanding of what features actually\nmatter, in particular if you need to perform feature selection.\nBoosting\nBoosting (originally called hypothesis boosting) refers to any Ensemble method that can\ncombine several weak learners into a strong learner. The general idea of most boosting\nmethods is to train predictors sequentially, each trying to correct its predecessor. There\nare many boosting methods available, but by far the most popular are AdaBoost\n (short\nfor Adaptive Boosting) and Gradient Boosting. Let’s start with AdaBoost.\nAdaBoost\nOne way for a new predictor to correct its predecessor is to pay a bit more attention to\nthe training instances that the predecessor underfitted. This results in new predictors\nfocusing more and more on the hard cases. This is the technique used by AdaBoost.\n13\nFor example, when training an AdaBoost classifier, the algorithm first trains a base\nclassifier (such as a Decision Tree) and uses it to make predictions on the training set.\nThe algorithm then increases the relative weight of misclassified training instances.\nThen it trains a second classifier, using the updated weights, and again makes\npredictions on the training set, updates the instance weights, and so on (see Figure 7-7).\nFigure 7-7. AdaBoost sequential training with instance weight updates\nFigure 7-8 shows the decision boundaries of five consecutive predictors on the moons\ndataset (in this example, each predictor is a highly regularized SVM classifier with an\nRBF kernel\n). The first classifier gets many instances wrong, so their weights get\nboosted. The second classifier therefore does a better job on these instances, and so on.\nThe plot on the right represents the same sequence of predictors, except that the\nlearning rate is halved (i.e., the misclassified instance weights are boosted half as much\nat every iteration). As you can see, this sequential learning technique has some\nsimilarities with Gradient Descent, except that instead of tweaking a single predictor’s\nparameters to minimize a cost function, AdaBoost adds predictors to the ensemble,\ngradually making it better.\n14\nFigure 7-8. Decision boundaries of consecutive predictors\nOnce all predictors are trained, the ensemble makes predictions very much like bagging\nor pasting, except that predictors have different weights depending on their overall\naccuracy on the weighted training set.\nWARNING\nThere is one important drawback to this sequential learning technique: it cannot be parallelized (or\nonly partially), since each predictor can only be trained after the previous predictor has been trained\nand evaluated. As a result, it does not scale as well as bagging or pasting.\nLet’s take a closer look at the AdaBoost algorithm. Each instance weight w is initially\nset to \n. A first predictor is trained, and its weighted error rate r is computed on the\ntraining set; see Equation 7-1.\nEquation 7-1. Weighted error rate of the j predictor\nrj =\nwhere ˆy\n(i)\nj\nis the jth predictor’s prediction for the ith instance.\nThe predictor’s weight α is then computed using Equation 7-2, where η is the learning\nrate hyperparameter (defaults to 1).\n The more accurate the predictor is, the higher its\nweight will be. If it is just guessing randomly, then its weight will be close to zero.\nHowever, if it is most often wrong (i.e., less accurate than random guessing), then its\nweight will be negative.\n(i)\n1\nm\n1\nth\nm\n∑\nw(i)\ni=1\nˆy(i)\nj ≠y(i)\nm\n∑\ni=1\nw(i)\nj\n15\nEquation 7-2. Predictor weight\nαj = η log\nNext, the AdaBoost algorithm updates the instance weights, using Equation 7-3, which\nboosts the weights of the misclassified instances.\nEquation 7-3. Weight update rule\nfor i = 1, 2, ⋯, m\nw(i) ←{\nw(i)\nif ˆyj\n(i) = y(i)\nw(i) exp (αj) if ˆyj\n(i) ≠y(i)\nThen all the instance weights are normalized (i.e., divided by ∑m\ni=1 w(i)).\nFinally, a new predictor is trained using the updated weights, and the whole process is\nrepeated (the new predictor’s weight is computed, the instance weights are updated,\nthen another predictor is trained, and so on). The algorithm stops when the desired\nnumber of predictors is reached, or when a perfect predictor is found.\nTo make predictions, AdaBoost simply computes the predictions of all the predictors\nand weighs them using the predictor weights α . The predicted class is the one that\nreceives the majority of weighted votes (see Equation 7-4).\nEquation 7-4. AdaBoost predictions\nˆy (x) = argmax\nk\nN\n∑\nαj\nwhere N is the number of predictors.\nScikit-Learn uses a multiclass version of AdaBoost called SAMME (which stands for\nStagewise Additive Modeling using a Multiclass Exponential loss function). When there\nare just two classes, SAMME is equivalent to AdaBoost. If the predictors can estimate\nclass probabilities (i.e., if they have a predict_proba() method), Scikit-Learn can use\na variant of SAMME called SAMME.R (the R stands for “Real”), which relies on class\nprobabilities rather than predictions and generally performs better.\nThe following code trains an AdaBoost classifier based on 200 Decision Stumps using\nScikit-Learn’s AdaBoostClassifier class (as you might expect, there is also an\nAdaBoostRegressor class). A Decision Stump is a Decision Tree with max_depth=1—\n1 −rj\nrj\nj\nj=1\nˆyj(x)=k\n16\nin other words, a tree composed of a single decision node plus two leaf nodes. This is\nthe default base estimator for the AdaBoostClassifier class:\nfrom sklearn.ensemble import AdaBoostClassifier \n \nada_clf = AdaBoostClassifier( \n DecisionTreeClassifier(max_depth=1), n_estimators=200, \n algorithm=\"SAMME.R\", learning_rate=0.5) \nada_clf.fit(X_train, y_train)\nTIP\nIf your AdaBoost ensemble is overfitting the training set, you can try reducing the number of\nestimators or more strongly regularizing the base estimator.\nGradient Boosting\nAnother very popular boosting algorithm is Gradient Boosting. Just like AdaBoost,\nGradient Boosting works by sequentially adding predictors to an ensemble, each one\ncorrecting its predecessor. However, instead of tweaking the instance weights at every\niteration like AdaBoost does, this method tries to fit the new predictor to the residual\nerrors made by the previous predictor.\nLet’s go through a simple regression example, using Decision Trees as the base\npredictors (of course, Gradient Boosting also works great with regression tasks). This is\ncalled Gradient Tree Boosting, or Gradient Boosted Regression Trees (GBRT). First,\nlet’s fit a DecisionTreeRegressor to the training set (for example, a noisy quadratic\ntraining set):\nfrom sklearn.tree import DecisionTreeRegressor \n \ntree_reg1 = DecisionTreeRegressor(max_depth=2) \ntree_reg1.fit(X, y)\nNext, we’ll train a second DecisionTreeRegressor on the residual errors made by the\nfirst predictor:\ny2 = y - tree_reg1.predict(X) \ntree_reg2 = DecisionTreeRegressor(max_depth=2) \ntree_reg2.fit(X, y2)\nThen we train a third regressor on the residual errors made by the second predictor:\n17\ny3 = y2 - tree_reg2.predict(X) \ntree_reg3 = DecisionTreeRegressor(max_depth=2) \ntree_reg3.fit(X, y3)\nNow we have an ensemble containing three trees. It can make predictions on a new\ninstance simply by adding up the predictions of all the trees:\ny_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))\nFigure 7-9 represents the predictions of these three trees in the left column, and the\nensemble’s predictions in the right column. In the first row, the ensemble has just one\ntree, so its predictions are exactly the same as the first tree’s predictions. In the second\nrow, a new tree is trained on the residual errors of the first tree. On the right you can see\nthat the ensemble’s predictions are equal to the sum of the predictions of the first two\ntrees. Similarly, in the third row another tree is trained on the residual errors of the\nsecond tree. You can see that the ensemble’s predictions gradually get better as trees are\nadded to the ensemble.\nA simpler way to train GBRT ensembles is to use Scikit-Learn’s\nGradientBoostingRegressor class. Much like the RandomForestRegressor class, it\nhas hyperparameters to control the growth of Decision Trees (e.g., max_depth,\nmin_samples_leaf), as well as hyperparameters to control the ensemble training, such\nas the number of trees (n_estimators). The following code creates the same ensemble\nas the previous one:\nfrom sklearn.ensemble import GradientBoostingRegressor \n \ngbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0) \ngbrt.fit(X, y)\nFigure 7-9. In this depiction of Gradient Boosting, the first predictor (top left) is trained normally, then each\nconsecutive predictor (middle left and lower left) is trained on the previous predictor’s residuals; the right column\nshows the resulting ensemble’s predictions\nThe learning_rate hyperparameter scales the contribution of each tree. If you set it to\na low value, such as 0.1, you will need more trees in the ensemble to fit the training set,\nbut the predictions will usually generalize better. This is a regularization technique\ncalled shrinkage. Figure 7-10 shows two GBRT ensembles trained with a low learning\nrate: the one on the left does not have enough trees to fit the training set, while the one\non the right has too many trees and overfits the training set.\nFigure 7-10. GBRT ensembles with not enough predictors (left) and too many (right)\nIn order to find the optimal number of trees, you can use early stopping (see Chapter 4).\nA simple way to implement this is to use the staged_predict() method: it returns an\niterator over the predictions made by the ensemble at each stage of training (with one\ntree, two trees, etc.). The following code trains a GBRT ensemble with 120 trees, then\nmeasures the validation error at each stage of training to find the optimal number of\ntrees, and finally trains another GBRT ensemble using the optimal number of trees:\nimport numpy as np \nfrom sklearn.model_selection import train_test_split \nfrom sklearn.metrics import mean_squared_error \n \nX_train, X_val, y_train, y_val = train_test_split(X, y) \n \ngbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120) \ngbrt.fit(X_train, y_train) \n \nerrors = [mean_squared_error(y_val, y_pred) \n for y_pred in gbrt.staged_predict(X_val)] \nbst_n_estimators = np.argmin(errors) + 1 \n \ngbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators) \ngbrt_best.fit(X_train, y_train)\nThe validation errors are represented on the left of Figure 7-11, and the best model’s\npredictions are represented on the right.\nFigure 7-11. Tuning the number of trees using early stopping\nIt is also possible to implement early stopping by actually stopping training early\n(instead of training a large number of trees first and then looking back to find the\noptimal number). You can do so by setting warm_start=True, which makes Scikit-\nLearn keep existing trees when the fit() method is called, allowing incremental\ntraining. The following code stops training when the validation error does not improve\nfor five iterations in a row:\ngbrt = GradientBoostingRegressor(max_depth=2, warm_start=True) \n \nmin_val_error = float(\"inf\") \nerror_going_up = 0 \nfor n_estimators in range(1, 120): \n gbrt.n_estimators = n_estimators \n gbrt.fit(X_train, y_train) \n y_pred = gbrt.predict(X_val) \n val_error = mean_squared_error(y_val, y_pred) \n if val_error < min_val_error: \n min_val_error = val_error \n error_going_up = 0 \n else: \n error_going_up += 1 \n if error_going_up == 5: \n break # early stopping\nThe GradientBoostingRegressor class also supports a subsample hyperparameter,\nwhich specifies the fraction of training instances to be used for training each tree. For\nexample, if subsample=0.25, then each tree is trained on 25% of the training instances,\nselected randomly. As you can probably guess by now, this technique trades a higher\nbias for a lower variance. It also speeds up training considerably. This is called\nStochastic Gradient Boosting.\nNOTE\nIt is possible to use Gradient Boosting with other cost functions. This is controlled by the loss\nhyperparameter (see Scikit-Learn’s documentation for more details).\nIt is worth noting that an optimized implementation of Gradient Boosting is available in\nthe popular Python library XGBoost, which stands for Extreme Gradient Boosting. This\npackage was initially developed by Tianqi Chen as part of the Distributed (Deep)\nMachine Learning Community (DMLC), and it aims to be extremely fast, scalable, and\nportable. In fact, XGBoost is often an important component of the winning entries in\nML competitions. XGBoost’s API is quite similar to Scikit-Learn’s:\nimport xgboost \n \nxgb_reg = xgboost.XGBRegressor() \nxgb_reg.fit(X_train, y_train) \ny_pred = xgb_reg.predict(X_val)\nXGBoost also offers several nice features, such as automatically taking care of early\nstopping:\nxgb_reg.fit(X_train, y_train, \n eval_set=[(X_val, y_val)], early_stopping_rounds=2) \ny_pred = xgb_reg.predict(X_val)\nYou should definitely check it out!\nStacking\nThe last Ensemble method we will discuss in this chapter is called stacking (short for\nstacked generalization). It is based on a simple idea: instead of using trivial functions\n(such as hard voting) to aggregate the predictions of all predictors in an ensemble, why\ndon’t we train a model to perform this aggregation? Figure 7-12 shows such an\nensemble performing a regression task on a new instance. Each of the bottom three\npredictors predicts a different value (3.1, 2.7, and 2.9), and then the final predictor\n(called a blender, or a meta learner) takes these predictions as inputs and makes the\nfinal prediction (3.0).\n18\nFigure 7-12. Aggregating predictions using a blending predictor\nTo train the blender, a common approach is to use a hold-out set.\n Let’s see how it\nworks. First, the training set is split into two subsets. The first subset is used to train the\npredictors in the first layer (see Figure 7-13).\n19\nFigure 7-13. Training the first layer\nNext, the first layer’s predictors are used to make predictions on the second (held-out)\nset (see Figure 7-14). This ensures that the predictions are “clean,” since the predictors\nnever saw these instances during training. For each instance in the hold-out set, there\nare three predicted values. We can create a new training set using these predicted values\nas input features (which makes this new training set 3D), and keeping the target values.\nThe blender is trained on this new training set, so it learns to predict the target value,\ngiven the first layer’s predictions.\nFigure 7-14. Training the blender\nIt is actually possible to train several different blenders this way (e.g., one using Linear\nRegression, another using Random Forest Regression), to get a whole layer of blenders.\nThe trick is to split the training set into three subsets: the first one is used to train the\nfirst layer, the second one is used to create the training set used to train the second layer\n(using predictions made by the predictors of the first layer), and the third one is used to\ncreate the training set to train the third layer (using predictions made by the predictors\nof the second layer). Once this is done, we can make a prediction for a new instance by\ngoing through each layer sequentially, as shown in Figure 7-15.\nFigure 7-15. Predictions in a multilayer stacking ensemble\nUnfortunately, Scikit-Learn does not support stacking directly, but it is not too hard to\nroll out your own implementation (see the following exercises). Alternatively, you can\nuse an open source implementation such as brew.\nExercises\n1. If you have trained five different models on the exact same training data, and\nthey all achieve 95% precision, is there any chance that you can combine these\nmodels to get better results? If so, how? If not, why?\n2. What is the difference between hard and soft voting classifiers?\n3. Is it possible to speed up training of a bagging ensemble by distributing it\nacross multiple servers? What about pasting ensembles, boosting ensembles,\nRandom Forests, or stacking ensembles?\n4. What is the benefit of out-of-bag evaluation?\n5. What makes Extra-Trees more random than regular Random Forests? How can\nthis extra randomness help? Are Extra-Trees slower or faster than regular\nRandom Forests?\n6. If your AdaBoost ensemble underfits the training data, which hyperparameters\nshould you tweak and how?\n7. If your Gradient Boosting ensemble overfits the training set, should you\nincrease or decrease the learning rate?\n8. Load the MNIST data (introduced in Chapter 3), and split it into a training set,\na validation set, and a test set (e.g., use 50,000 instances for training, 10,000\nfor validation, and 10,000 for testing). Then train various classifiers, such as a\nRandom Forest classifier, an Extra-Trees classifier, and an SVM classifier.\nNext, try to combine them into an ensemble that outperforms each individual\nclassifier on the validation set, using soft or hard voting. Once you have found\none, try it on the test set. How much better does it perform compared to the\nindividual classifiers?\n9. Run the individual classifiers from the previous exercise to make predictions\non the validation set, and create a new training set with the resulting\npredictions: each training instance is a vector containing the set of predictions\nfrom all your classifiers for an image, and the target is the image’s class. Train\na classifier on this new training set. Congratulations, you have just trained a\nblender, and together with the classifiers it forms a stacking ensemble! Now\nevaluate the ensemble on the test set. For each image in the test set, make\npredictions with all your classifiers, then feed the predictions to the blender to\nget the ensemble’s predictions. How does it compare to the voting classifier\nyou trained earlier?\nSolutions to these exercises are available in Appendix A.\n1 Leo Breiman, “Bagging Predictors,” Machine Learning 24, no. 2 (1996): 123–140.\n2 In statistics, resampling with replacement is called bootstrapping.\n3 Leo Breiman, “Pasting Small Votes for Classification in Large Databases and On-Line,” Machine Learning\n36, no. 1–2 (1999): 85–103.\n4 Bias and variance were introduced in Chapter 4.\n5 max_samples can alternatively be set to a float between 0.0 and 1.0, in which case the max number of\ninstances to sample is equal to the size of the training set times max_samples.\n6 As m grows, this ratio approaches 1 – exp(–1) ≈ 63.212%.\n7 Gilles Louppe and Pierre Geurts, “Ensembles on Random Patches,” Lecture Notes in Computer Science\n7523 (2012): 346–361.\n8 Tin Kam Ho, “The Random Subspace Method for Constructing Decision Forests,” IEEE Transactions on\nPattern Analysis and Machine Intelligence 20, no. 8 (1998): 832–844.\n9 Tin Kam Ho, “Random Decision Forests,” Proceedings of the Third International Conference on Document\nAnalysis and Recognition 1 (1995): 278.\n10 The BaggingClassifier class remains useful if you want a bag of something other than Decision Trees.\n11 There are a few notable exceptions: splitter is absent (forced to \"random\"), presort is absent (forced to\nFalse), max_samples is absent (forced to 1.0), and base_estimator is absent (forced to\nDecisionTreeClassifier with the provided hyperparameters).\n12 Pierre Geurts et al., “Extremely Randomized Trees,” Machine Learning 63, no. 1 (2006): 3–42.\n13 Yoav Freund and Robert E. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an\nApplication to Boosting,” Journal of Computer and System Sciences 55, no. 1 (1997): 119–139.\n14 This is just for illustrative purposes. SVMs are generally not good base predictors for AdaBoost; they are\nslow and tend to be unstable with it.\n15 The original AdaBoost algorithm does not use a learning rate hyperparameter.\n16 For more details, see Ji Zhu et al., “Multi-Class AdaBoost,” Statistics and Its Interface 2, no. 3 (2009): 349–\n360.\n17 Gradient Boosting was first introduced in Leo Breiman’s 1997 paper “Arcing the Edge” and was further\ndeveloped in the 1999 paper “Greedy Function Approximation: A Gradient Boosting Machine” by Jerome\nH. Friedman.\n18 David H. Wolpert, “Stacked Generalization,” Neural Networks 5, no. 2 (1992): 241–259.\n19 Alternatively, it is possible to use out-of-fold predictions. In some contexts this is called stacking, while\nusing a hold-out set is called blending. For many people these terms are synonymous." + }, + { + "chapter": 8, + "text": "Chapter 8. Dimensionality\nReduction\nMany Machine Learning problems involve thousands or even millions of\nfeatures for each training instance. Not only do all these features make\ntraining extremely slow, but they can also make it much harder to find a\ngood solution, as we will see. This problem is often referred to as the\ncurse of dimensionality.\nFortunately, in real-world problems, it is often possible to reduce the\nnumber of features considerably, turning an intractable problem into a\ntractable one. For example, consider the MNIST images (introduced in\nChapter 3): the pixels on the image borders are almost always white, so\nyou could completely drop these pixels from the training set without\nlosing much information. Figure 7-6 confirms that these pixels are utterly\nunimportant for the classification task. Additionally, two neighboring\npixels are often highly correlated: if you merge them into a single pixel\n(e.g., by taking the mean of the two pixel intensities), you will not lose\nmuch information.\nWARNING\nReducing dimensionality does cause some information loss (just like compressing an\nimage to JPEG can degrade its quality), so even though it will speed up training, it\nmay make your system perform slightly worse. It also makes your pipelines a bit\nmore complex and thus harder to maintain. So, if training is too slow, you should\nfirst try to train your system with the original data before considering using\ndimensionality reduction. In some cases, reducing the dimensionality of the training\ndata may filter out some noise and unnecessary details and thus result in higher\nperformance, but in general it won’t; it will just speed up training.\nApart from speeding up training, dimensionality reduction is also\nextremely useful for data visualization (or DataViz). Reducing the number\nof dimensions down to two (or three) makes it possible to plot a condensed\nview of a high-dimensional training set on a graph and often gain some\nimportant insights by visually detecting patterns, such as clusters.\nMoreover, DataViz is essential to communicate your conclusions to people\nwho are not data scientists—in particular, decision makers who will use\nyour results.\nIn this chapter we will discuss the curse of dimensionality and get a sense\nof what goes on in high-dimensional space. Then, we will consider the two\nmain approaches to dimensionality reduction (projection and Manifold\nLearning), and we will go through three of the most popular\ndimensionality reduction techniques: PCA, Kernel PCA, and LLE.\nThe Curse of Dimensionality\nWe are so used to living in three dimensions that our intuition fails us\nwhen we try to imagine a high-dimensional space. Even a basic 4D\nhypercube is incredibly hard to picture in our minds (see Figure 8-1), let\nalone a 200-dimensional ellipsoid bent in a 1,000-dimensional space.\nFigure 8-1. Point, segment, square, cube, and tesseract (0D to 4D hypercubes)\nIt turns out that many things behave very differently in high-dimensional\nspace. For example, if you pick a random point in a unit square (a 1 × 1\nsquare), it will have only about a 0.4% chance of being located less than\n1\n2\n0.001 from a border (in other words, it is very unlikely that a random point\nwill be “extreme” along any dimension). But in a 10,000-dimensional unit\nhypercube, this probability is greater than 99.999999%. Most points in a\nhigh-dimensional hypercube are very close to the border.\nHere is a more troublesome difference: if you pick two points randomly in\na unit square, the distance between these two points will be, on average,\nroughly 0.52. If you pick two random points in a unit 3D cube, the average\ndistance will be roughly 0.66. But what about two points picked randomly\nin a 1,000,000-dimensional hypercube? The average distance, believe it or\nnot, will be about 408.25 (roughly √1, 000, 000/6)! This is\ncounterintuitive: how can two points be so far apart when they both lie\nwithin the same unit hypercube? Well, there’s just plenty of space in high\ndimensions. As a result, high-dimensional datasets are at risk of being\nvery sparse: most training instances are likely to be far away from each\nother. This also means that a new instance will likely be far away from any\ntraining instance, making predictions much less reliable than in lower\ndimensions, since they will be based on much larger extrapolations. In\nshort, the more dimensions the training set has, the greater the risk of\noverfitting it.\nIn theory, one solution to the curse of dimensionality could be to increase\nthe size of the training set to reach a sufficient density of training\ninstances. Unfortunately, in practice, the number of training instances\nrequired to reach a given density grows exponentially with the number of\ndimensions. With just 100 features (significantly fewer than in the MNIST\nproblem), you would need more training instances than atoms in the\nobservable universe in order for training instances to be within 0.1 of each\nother on average, assuming they were spread out uniformly across all\ndimensions.\nMain Approaches for Dimensionality\nReduction\n3\nBefore we dive into specific dimensionality reduction algorithms, let’s\ntake a look at the two main approaches to reducing dimensionality:\nprojection and Manifold Learning.\nProjection\nIn most real-world problems, training instances are not spread out\nuniformly across all dimensions. Many features are almost constant, while\nothers are highly correlated (as discussed earlier for MNIST). As a result,\nall training instances lie within (or close to) a much lower-dimensional\nsubspace of the high-dimensional space. This sounds very abstract, so let’s\nlook at an example. In Figure 8-2 you can see a 3D dataset represented by\ncircles.\nFigure 8-2. A 3D dataset lying close to a 2D subspace\nNotice that all training instances lie close to a plane: this is a lower-\ndimensional (2D) subspace of the high-dimensional (3D) space. If we\nproject every training instance perpendicularly onto this subspace (as\nrepresented by the short lines connecting the instances to the plane), we\nget the new 2D dataset shown in Figure 8-3. Ta-da! We have just reduced\nthe dataset’s dimensionality from 3D to 2D. Note that the axes correspond\nto new features z and z (the coordinates of the projections on the plane).\nFigure 8-3. The new 2D dataset after projection\nHowever, projection is not always the best approach to dimensionality\nreduction. In many cases the subspace may twist and turn, such as in the\nfamous Swiss roll toy dataset represented in Figure 8-4.\n1\n2\nFigure 8-4. Swiss roll dataset\nSimply projecting onto a plane (e.g., by dropping x ) would squash\ndifferent layers of the Swiss roll together, as shown on the left side of\nFigure 8-5. What you really want is to unroll the Swiss roll to obtain the\n2D dataset on the right side of Figure 8-5.\nFigure 8-5. Squashing by projecting onto a plane (left) versus unrolling the Swiss roll (right)\n3\nManifold Learning\nThe Swiss roll is an example of a 2D manifold. Put simply, a 2D manifold\nis a 2D shape that can be bent and twisted in a higher-dimensional space.\nMore generally, a d-dimensional manifold is a part of an n-dimensional\nspace (where d < n) that locally resembles a d-dimensional hyperplane. In\nthe case of the Swiss roll, d = 2 and n = 3: it locally resembles a 2D plane,\nbut it is rolled in the third dimension.\nMany dimensionality reduction algorithms work by modeling the\nmanifold on which the training instances lie; this is called Manifold\nLearning. It relies on the manifold assumption, also called the manifold\nhypothesis, which holds that most real-world high-dimensional datasets lie\nclose to a much lower-dimensional manifold. This assumption is very\noften empirically observed.\nOnce again, think about the MNIST dataset: all handwritten digit images\nhave some similarities. They are made of connected lines, the borders are\nwhite, and they are more or less centered. If you randomly generated\nimages, only a ridiculously tiny fraction of them would look like\nhandwritten digits. In other words, the degrees of freedom available to you\nif you try to create a digit image are dramatically lower than the degrees\nof freedom you would have if you were allowed to generate any image you\nwanted. These constraints tend to squeeze the dataset into a lower-\ndimensional manifold.\nThe manifold assumption is often accompanied by another implicit\nassumption: that the task at hand (e.g., classification or regression) will be\nsimpler if expressed in the lower-dimensional space of the manifold. For\nexample, in the top row of Figure 8-6 the Swiss roll is split into two\nclasses: in the 3D space (on the left), the decision boundary would be\nfairly complex, but in the 2D unrolled manifold space (on the right), the\ndecision boundary is a straight line.\nHowever, this implicit assumption does not always hold. For example, in\nthe bottom row of Figure 8-6, the decision boundary is located at x = 5.\nThis decision boundary looks very simple in the original 3D space (a\n1\nvertical plane), but it looks more complex in the unrolled manifold (a\ncollection of four independent line segments).\nIn short, reducing the dimensionality of your training set before training a\nmodel will usually speed up training, but it may not always lead to a better\nor simpler solution; it all depends on the dataset.\nHopefully you now have a good sense of what the curse of dimensionality\nis and how dimensionality reduction algorithms can fight it, especially\nwhen the manifold assumption holds. The rest of this chapter will go\nthrough some of the most popular algorithms.\nFigure 8-6. The decision boundary may not always be simpler with lower dimensions\nPCA\nPrincipal Component Analysis (PCA) is by far the most popular\ndimensionality reduction algorithm. First it identifies the hyperplane that\nlies closest to the data, and then it projects the data onto it, just like in\nFigure 8-2.\nPreserving the Variance\nBefore you can project the training set onto a lower-dimensional\nhyperplane, you first need to choose the right hyperplane. For example, a\nsimple 2D dataset is represented on the left in Figure 8-7, along with three\ndifferent axes (i.e., 1D hyperplanes). On the right is the result of the\nprojection of the dataset onto each of these axes. As you can see, the\nprojection onto the solid line preserves the maximum variance, while the\nprojection onto the dotted line preserves very little variance and the\nprojection onto the dashed line preserves an intermediate amount of\nvariance.\nFigure 8-7. Selecting the subspace to project on\nIt seems reasonable to select the axis that preserves the maximum amount\nof variance, as it will most likely lose less information than the other\nprojections. Another way to justify this choice is that it is the axis that\nminimizes the mean squared distance between the original dataset and its\nprojection onto that axis. This is the rather simple idea behind PCA.\nPrincipal Components\n4\nPCA identifies the axis that accounts for the largest amount of variance in\nthe training set. In Figure 8-7, it is the solid line. It also finds a second\naxis, orthogonal to the first one, that accounts for the largest amount of\nremaining variance. In this 2D example there is no choice: it is the dotted\nline. If it were a higher-dimensional dataset, PCA would also find a third\naxis, orthogonal to both previous axes, and a fourth, a fifth, and so on—as\nmany axes as the number of dimensions in the dataset.\nThe i axis is called the i principal component (PC) of the data. In\nFigure 8-7, the first PC is the axis on which vector c lies, and the second\nPC is the axis on which vector c lies. In Figure 8-2 the first two PCs are\nthe orthogonal axes on which the two arrows lie, on the plane, and the\nthird PC is the axis orthogonal to that plane.\nNOTE\nFor each principal component, PCA finds a zero-centered unit vector pointing in the\ndirection of the PC. Since two opposing unit vectors lie on the same axis, the\ndirection of the unit vectors returned by PCA is not stable: if you perturb the training\nset slightly and run PCA again, the unit vectors may point in the opposite direction\nas the original vectors. However, they will generally still lie on the same axes. In\nsome cases, a pair of unit vectors may even rotate or swap (if the variances along\nthese two axes are close), but the plane they define will generally remain the same.\nSo how can you find the principal components of a training set? Luckily,\nthere is a standard matrix factorization technique called Singular Value\nDecomposition (SVD) that can decompose the training set matrix X into\nthe matrix multiplication of three matrices U Σ V , where V contains the\nunit vectors that define all the principal components that we are looking\nfor, as shown in Equation 8-1.\nEquation 8-1. Principal components matrix\nV =\n⎛\n⎜\n⎝\n∣\n∣\n∣\nc1\nc2\n⋯\ncn\n∣\n∣\n∣\n⎞\n⎟\n⎠\nth\nth\n1\n2\n⊺\nThe following Python code uses NumPy’s svd() function to obtain all the\nprincipal components of the training set, then extracts the two unit vectors\nthat define the first two PCs:\nX_centered = X - X.mean(axis=0) \nU, s, Vt = np.linalg.svd(X_centered) \nc1 = Vt.T[:, 0] \nc2 = Vt.T[:, 1]\nWARNING\nPCA assumes that the dataset is centered around the origin. As we will see, Scikit-\nLearn’s PCA classes take care of centering the data for you. If you implement PCA\nyourself (as in the preceding example), or if you use other libraries, don’t forget to\ncenter the data first.\nProjecting Down to d Dimensions\nOnce you have identified all the principal components, you can reduce the\ndimensionality of the dataset down to d dimensions by projecting it onto\nthe hyperplane defined by the first d principal components. Selecting this\nhyperplane ensures that the projection will preserve as much variance as\npossible. For example, in Figure 8-2 the 3D dataset is projected down to\nthe 2D plane defined by the first two principal components, preserving a\nlarge part of the dataset’s variance. As a result, the 2D projection looks\nvery much like the original 3D dataset.\nTo project the training set onto the hyperplane and obtain a reduced dataset\nX\n of dimensionality d, compute the matrix multiplication of the\ntraining set matrix X by the matrix W , defined as the matrix containing\nthe first d columns of V, as shown in Equation 8-2.\nEquation 8-2. Projecting the training set down to d dimensions\nXd-proj = XWd\nd-proj\nd\nThe following Python code projects the training set onto the plane defined\nby the first two principal components:\nW2 = Vt.T[:, :2] \nX2D = X_centered.dot(W2)\nThere you have it! You now know how to reduce the dimensionality of any\ndataset down to any number of dimensions, while preserving as much\nvariance as possible.\nUsing Scikit-Learn\nScikit-Learn’s PCA class uses SVD decomposition to implement PCA, just\nlike we did earlier in this chapter. The following code applies PCA to\nreduce the dimensionality of the dataset down to two dimensions (note\nthat it automatically takes care of centering the data):\nfrom sklearn.decomposition import PCA \n \npca = PCA(n_components = 2) \nX2D = pca.fit_transform(X)\nAfter fitting the PCA transformer to the dataset, its components_ attribute\nholds the transpose of W (e.g., the unit vector that defines the first\nprincipal component is equal to pca.components_.T[:, 0]).\nExplained Variance Ratio\nAnother useful piece of information is the explained variance ratio of\neach principal component, available via the explained_variance_ratio_\nvariable. The ratio indicates the proportion of the dataset’s variance that\nlies along each principal component. For example, let’s look at the\nexplained variance ratios of the first two components of the 3D dataset\nrepresented in Figure 8-2:\nd\n>>> pca.explained_variance_ratio_ \narray([0.84248607, 0.14631839])\nThis output tells you that 84.2% of the dataset’s variance lies along the\nfirst PC, and 14.6% lies along the second PC. This leaves less than 1.2%\nfor the third PC, so it is reasonable to assume that the third PC probably\ncarries little information.\nChoosing the Right Number of Dimensions\nInstead of arbitrarily choosing the number of dimensions to reduce down\nto, it is simpler to choose the number of dimensions that add up to a\nsufficiently large portion of the variance (e.g., 95%). Unless, of course,\nyou are reducing dimensionality for data visualization—in that case you\nwill want to reduce the dimensionality down to 2 or 3.\nThe following code performs PCA without reducing dimensionality, then\ncomputes the minimum number of dimensions required to preserve 95%\nof the training set’s variance:\npca = PCA() \npca.fit(X_train) \ncumsum = np.cumsum(pca.explained_variance_ratio_) \nd = np.argmax(cumsum >= 0.95) + 1\nYou could then set n_components=d and run PCA again. But there is a\nmuch better option: instead of specifying the number of principal\ncomponents you want to preserve, you can set n_components to be a float\nbetween 0.0 and 1.0, indicating the ratio of variance you wish to preserve:\npca = PCA(n_components=0.95) \nX_reduced = pca.fit_transform(X_train)\nYet another option is to plot the explained variance as a function of the\nnumber of dimensions (simply plot cumsum; see Figure 8-8). There will\nusually be an elbow in the curve, where the explained variance stops\ngrowing fast. In this case, you can see that reducing the dimensionality\ndown to about 100 dimensions wouldn’t lose too much explained variance.\nFigure 8-8. Explained variance as a function of the number of dimensions\nPCA for Compression\nAfter dimensionality reduction, the training set takes up much less space.\nAs an example, try applying PCA to the MNIST dataset while preserving\n95% of its variance. You should find that each instance will have just over\n150 features, instead of the original 784 features. So, while most of the\nvariance is preserved, the dataset is now less than 20% of its original size!\nThis is a reasonable compression ratio, and you can see how this size\nreduction can speed up a classification algorithm (such as an SVM\nclassifier) tremendously.\nIt is also possible to decompress the reduced dataset back to 784\ndimensions by applying the inverse transformation of the PCA projection.\nThis won’t give you back the original data, since the projection lost a bit\nof information (within the 5% variance that was dropped), but it will\nlikely be close to the original data. The mean squared distance between the\noriginal data and the reconstructed data (compressed and then\ndecompressed) is called the reconstruction error.\nThe following code compresses the MNIST dataset down to 154\ndimensions, then uses the inverse_transform() method to decompress it\nback to 784 dimensions:\npca = PCA(n_components = 154) \nX_reduced = pca.fit_transform(X_train) \nX_recovered = pca.inverse_transform(X_reduced)\nFigure 8-9 shows a few digits from the original training set (on the left),\nand the corresponding digits after compression and decompression. You\ncan see that there is a slight image quality loss, but the digits are still\nmostly intact.\nFigure 8-9. MNIST compression that preserves 95% of the variance\nThe equation of the inverse transformation is shown in Equation 8-3.\nEquation 8-3. PCA inverse transformation, back to the original number of dimensions\nXrecovered = Xd-projWd\n⊺\nRandomized PCA\nIf you set the svd_solver hyperparameter to \"randomized\", Scikit-Learn\nuses a stochastic algorithm called Randomized PCA that quickly finds an\napproximation of the first d principal components. Its computational\ncomplexity is O(m × d ) + O(d ), instead of O(m × n ) + O(n ) for the full\nSVD approach, so it is dramatically faster than full SVD when d is much\nsmaller than n:\nrnd_pca = PCA(n_components=154, svd_solver=\"randomized\") \nX_reduced = rnd_pca.fit_transform(X_train)\nBy default, svd_solver is actually set to \"auto\": Scikit-Learn\nautomatically uses the randomized PCA algorithm if m or n is greater than\n500 and d is less than 80% of m or n, or else it uses the full SVD approach.\nIf you want to force Scikit-Learn to use full SVD, you can set the\nsvd_solver hyperparameter to \"full\".\nIncremental PCA\nOne problem with the preceding implementations of PCA is that they\nrequire the whole training set to fit in memory in order for the algorithm\nto run. Fortunately, Incremental PCA (IPCA) algorithms have been\ndeveloped. They allow you to split the training set into mini-batches and\nfeed an IPCA algorithm one mini-batch at a time. This is useful for large\ntraining sets and for applying PCA online (i.e., on the fly, as new instances\narrive).\nThe following code splits the MNIST dataset into 100 mini-batches (using\nNumPy’s array_split() function) and feeds them to Scikit-Learn’s\nIncrementalPCA class to reduce the dimensionality of the MNIST\ndataset down to 154 dimensions (just like before). Note that you must call\nthe partial_fit() method with each mini-batch, rather than the fit()\nmethod with the whole training set:\n2\n3\n2\n3\n5\nfrom sklearn.decomposition import IncrementalPCA \n \nn_batches = 100 \ninc_pca = IncrementalPCA(n_components=154) \nfor X_batch in np.array_split(X_train, n_batches): \n inc_pca.partial_fit(X_batch) \n \nX_reduced = inc_pca.transform(X_train)\nAlternatively, you can use NumPy’s memmap class, which allows you to\nmanipulate a large array stored in a binary file on disk as if it were\nentirely in memory; the class loads only the data it needs in memory, when\nit needs it. Since the IncrementalPCA class uses only a small part of the\narray at any given time, the memory usage remains under control. This\nmakes it possible to call the usual fit() method, as you can see in the\nfollowing code:\nX_mm = np.memmap(filename, dtype=\"float32\", mode=\"readonly\", shape=(m, \nn)) \n \nbatch_size = m // n_batches \ninc_pca = IncrementalPCA(n_components=154, batch_size=batch_size) \ninc_pca.fit(X_mm)\nKernel PCA\nIn Chapter 5 we discussed the kernel trick, a mathematical technique that\nimplicitly maps instances into a very high-dimensional space (called the\nfeature space), enabling nonlinear classification and regression with\nSupport Vector Machines. Recall that a linear decision boundary in the\nhigh-dimensional feature space corresponds to a complex nonlinear\ndecision boundary in the original space.\nIt turns out that the same trick can be applied to PCA, making it possible\nto perform complex nonlinear projections for dimensionality reduction.\nThis is called Kernel PCA (kPCA). It is often good at preserving clusters\nof instances after projection, or sometimes even unrolling datasets that lie\nclose to a twisted manifold.\n6\nThe following code uses Scikit-Learn’s KernelPCA class to perform kPCA\nwith an RBF kernel (see Chapter 5 for more details about the RBF kernel\nand other kernels):\nfrom sklearn.decomposition import KernelPCA \n \nrbf_pca = KernelPCA(n_components = 2, kernel=\"rbf\", gamma=0.04) \nX_reduced = rbf_pca.fit_transform(X)\nFigure 8-10 shows the Swiss roll, reduced to two dimensions using a linear\nkernel (equivalent to simply using the PCA class), an RBF kernel, and a\nsigmoid kernel.\nFigure 8-10. Swiss roll reduced to 2D using kPCA with various kernels\nSelecting a Kernel and Tuning Hyperparameters\nAs kPCA is an unsupervised learning algorithm, there is no obvious\nperformance measure to help you select the best kernel and\nhyperparameter values. That said, dimensionality reduction is often a\npreparation step for a supervised learning task (e.g., classification), so you\ncan use grid search to select the kernel and hyperparameters that lead to\nthe best performance on that task. The following code creates a two-step\npipeline, first reducing dimensionality to two dimensions using kPCA,\nthen applying Logistic Regression for classification. Then it uses\nGridSearchCV to find the best kernel and gamma value for kPCA in order\nto get the best classification accuracy at the end of the pipeline:\nfrom sklearn.model_selection import GridSearchCV \nfrom sklearn.linear_model import LogisticRegression \nfrom sklearn.pipeline import Pipeline \n \nclf = Pipeline([ \n (\"kpca\", KernelPCA(n_components=2)), \n (\"log_reg\", LogisticRegression()) \n ]) \n \nparam_grid = [{ \n \"kpca__gamma\": np.linspace(0.03, 0.05, 10), \n \"kpca__kernel\": [\"rbf\", \"sigmoid\"] \n }] \n \ngrid_search = GridSearchCV(clf, param_grid, cv=3) \ngrid_search.fit(X, y)\nThe best kernel and hyperparameters are then available through the\nbest_params_ variable:\n>>> print(grid_search.best_params_) \n{'kpca__gamma': 0.043333333333333335, 'kpca__kernel': 'rbf'}\nAnother approach, this time entirely unsupervised, is to select the kernel\nand hyperparameters that yield the lowest reconstruction error. Note that\nreconstruction is not as easy as with linear PCA. Here’s why. Figure 8-11\nshows the original Swiss roll 3D dataset (top left) and the resulting 2D\ndataset after kPCA is applied using an RBF kernel (top right). Thanks to\nthe kernel trick, this transformation is mathematically equivalent to using\nthe feature map φ to map the training set to an infinite-dimensional\nfeature space (bottom right), then projecting the transformed training set\ndown to 2D using linear PCA.\nNotice that if we could invert the linear PCA step for a given instance in\nthe reduced space, the reconstructed point would lie in feature space, not\nin the original space (e.g., like the one represented by an X in the\ndiagram). Since the feature space is infinite-dimensional, we cannot\ncompute the reconstructed point, and therefore we cannot compute the true\nreconstruction error. Fortunately, it is possible to find a point in the\noriginal space that would map close to the reconstructed point. This point\nis called the reconstruction pre-image. Once you have this pre-image, you\ncan measure its squared distance to the original instance. You can then\nselect the kernel and hyperparameters that minimize this reconstruction\npre-image error.\nFigure 8-11. Kernel PCA and the reconstruction pre-image error\nYou may be wondering how to perform this reconstruction. One solution is\nto train a supervised regression model, with the projected instances as the\ntraining set and the original instances as the targets. Scikit-Learn will do\nthis automatically if you set fit_inverse_transform=True, as shown in\nthe following code:\nrbf_pca = KernelPCA(n_components = 2, kernel=\"rbf\", gamma=0.0433, \n fit_inverse_transform=True) \nX_reduced = rbf_pca.fit_transform(X) \nX_preimage = rbf_pca.inverse_transform(X_reduced)\n7\nNOTE\nBy default, fit_inverse_transform=False and KernelPCA has no\ninverse_transform() method. This method only gets created when you set\nfit_inverse_transform=True.\nYou can then compute the reconstruction pre-image error:\n>>> from sklearn.metrics import mean_squared_error \n>>> mean_squared_error(X, X_preimage) \n32.786308795766132\nNow you can use grid search with cross-validation to find the kernel and\nhyperparameters that minimize this error.\nLLE\nLocally Linear Embedding (LLE) is another powerful nonlinear\ndimensionality reduction (NLDR) technique. It is a Manifold Learning\ntechnique that does not rely on projections, like the previous algorithms\ndo. In a nutshell, LLE works by first measuring how each training instance\nlinearly relates to its closest neighbors (c.n.), and then looking for a low-\ndimensional representation of the training set where these local\nrelationships are best preserved (more details shortly). This approach\nmakes it particularly good at unrolling twisted manifolds, especially when\nthere is not too much noise.\nThe following code uses Scikit-Learn’s LocallyLinearEmbedding class to\nunroll the Swiss roll:\nfrom sklearn.manifold import LocallyLinearEmbedding \n \nlle = LocallyLinearEmbedding(n_components=2, n_neighbors=10) \nX_reduced = lle.fit_transform(X)\n8\nThe resulting 2D dataset is shown in Figure 8-12. As you can see, the\nSwiss roll is completely unrolled, and the distances between instances are\nlocally well preserved. However, distances are not preserved on a larger\nscale: the left part of the unrolled Swiss roll is stretched, while the right\npart is squeezed. Nevertheless, LLE did a pretty good job at modeling the\nmanifold.\nFigure 8-12. Unrolled Swiss roll using LLE\nHere’s how LLE works: for each training instance x , the algorithm\nidentifies its k closest neighbors (in the preceding code k = 10), then tries\nto reconstruct x as a linear function of these neighbors. More\nspecifically, it finds the weights w such that the squared distance between\nx and ∑m\nj=1 wi,jx(j) is as small as possible, assuming w = 0 if x is not\none of the k closest neighbors of x . Thus the first step of LLE is the\nconstrained optimization problem described in Equation 8-4, where W is\nthe weight matrix containing all the weights w . The second constraint\nsimply normalizes the weights for each training instance x .\nEquation 8-4. LLE step one: linearly modeling local relationships\n(i)\n(i)\ni,j\n(i)\ni,j\n(j)\n(i)\ni,j\n(i)\nˆ\nW = argmin\nW\nm\n∑\ni=1\n(x(i) −\nm\n∑\nj=1\nwi,jx(j))\n2\nsubject to {\nwi,j = 0\nif x(j) is not one of the k c.n. of x(i)\n∑m\nj=1 wi,j = 1\nfor i = 1, 2, ⋯, m\nAfter this step, the weight matrix ˆ\nW (containing the weights ˆwi,j) encodes\nthe local linear relationships between the training instances. The second\nstep is to map the training instances into a d-dimensional space (where d <\nn) while preserving these local relationships as much as possible. If z is\nthe image of x in this d-dimensional space, then we want the squared\ndistance between z and ∑m\nj=1 ˆwi,jz(j) to be as small as possible. This idea\nleads to the unconstrained optimization problem described in Equation 8-\n5. It looks very similar to the first step, but instead of keeping the\ninstances fixed and finding the optimal weights, we are doing the reverse:\nkeeping the weights fixed and finding the optimal position of the\ninstances’ images in the low-dimensional space. Note that Z is the matrix\ncontaining all z .\nEquation 8-5. LLE step two: reducing dimensionality while preserving relationships\nˆZ = argmin\nZ\nm\n∑\ni=1\n(z(i) −\nm\n∑\nj=1\nˆwi,jz(j))\n2\nScikit-Learn’s LLE implementation has the following computational\ncomplexity: O(m log(m)n log(k)) for finding the k nearest neighbors,\nO(mnk ) for optimizing the weights, and O(dm ) for constructing the low-\ndimensional representations. Unfortunately, the m in the last term makes\nthis algorithm scale poorly to very large datasets.\nOther Dimensionality Reduction Techniques\n(i)\n(i)\n(i)\n(i)\n3\n2\n2\nThere are many other dimensionality reduction techniques, several of\nwhich are available in Scikit-Learn. Here are some of the most popular\nones:\nRandom Projections\nAs its name suggests, projects the data to a lower-dimensional space\nusing a random linear projection. This may sound crazy, but it turns\nout that such a random projection is actually very likely to preserve\ndistances well, as was demonstrated mathematically by William B.\nJohnson and Joram Lindenstrauss in a famous lemma. The quality of\nthe dimensionality reduction depends on the number of instances and\nthe target dimensionality, but surprisingly not on the initial\ndimensionality. Check out the documentation for the\nsklearn.random_projection package for more details.\nMultidimensional Scaling (MDS)\nReduces dimensionality while trying to preserve the distances between\nthe instances.\nIsomap\nCreates a graph by connecting each instance to its nearest neighbors,\nthen reduces dimensionality while trying to preserve the geodesic\ndistances between the instances.\nt-Distributed Stochastic Neighbor Embedding (t-SNE)\nReduces dimensionality while trying to keep similar instances close\nand dissimilar instances apart. It is mostly used for visualization, in\nparticular to visualize clusters of instances in high-dimensional space\n(e.g., to visualize the MNIST images in 2D).\nLinear Discriminant Analysis (LDA)\nIs a classification algorithm, but during training it learns the most\ndiscriminative axes between the classes, and these axes can then be\nused to define a hyperplane onto which to project the data. The benefit\n9\nof this approach is that the projection will keep classes as far apart as\npossible, so LDA is a good technique to reduce dimensionality before\nrunning another classification algorithm such as an SVM classifier.\nFigure 8-13 shows the results of a few of these techniques.\nFigure 8-13. Using various techniques to reduce the Swill roll to 2D\nExercises\n1. What are the main motivations for reducing a dataset’s\ndimensionality? What are the main drawbacks?\n2. What is the curse of dimensionality?\n3. Once a dataset’s dimensionality has been reduced, is it possible to\nreverse the operation? If so, how? If not, why?\n4. Can PCA be used to reduce the dimensionality of a highly\nnonlinear dataset?\n5. Suppose you perform PCA on a 1,000-dimensional dataset, setting\nthe explained variance ratio to 95%. How many dimensions will\nthe resulting dataset have?\n6. In what cases would you use vanilla PCA, Incremental PCA,\nRandomized PCA, or Kernel PCA?\n7. How can you evaluate the performance of a dimensionality\nreduction algorithm on your dataset?\n8. Does it make any sense to chain two different dimensionality\nreduction algorithms?\n9. Load the MNIST dataset (introduced in Chapter 3) and split it into\na training set and a test set (take the first 60,000 instances for\ntraining, and the remaining 10,000 for testing). Train a Random\nForest classifier on the dataset and time how long it takes, then\nevaluate the resulting model on the test set. Next, use PCA to\nreduce the dataset’s dimensionality, with an explained variance\nratio of 95%. Train a new Random Forest classifier on the\nreduced dataset and see how long it takes. Was training much\nfaster? Next, evaluate the classifier on the test set. How does it\ncompare to the previous classifier?\n10. Use t-SNE to reduce the MNIST dataset down to two dimensions\nand plot the result using Matplotlib. You can use a scatterplot\nusing 10 different colors to represent each image’s target class.\nAlternatively, you can replace each dot in the scatterplot with the\ncorresponding instance’s class (a digit from 0 to 9), or even plot\nscaled-down versions of the digit images themselves (if you plot\nall digits, the visualization will be too cluttered, so you should\neither draw a random sample or plot an instance only if no other\ninstance has already been plotted at a close distance). You should\nget a nice visualization with well-separated clusters of digits. Try\nusing other dimensionality reduction algorithms such as PCA,\nLLE, or MDS and compare the resulting visualizations.\nSolutions to these exercises are available in Appendix A.\n1 Well, four dimensions if you count time, and a few more if you are a string theorist.\n2 Watch a rotating tesseract projected into 3D space at https://homl.info/30. Image by\nWikipedia user NerdBoy1392 (Creative Commons BY-SA 3.0). Reproduced from\nhttps://en.wikipedia.org/wiki/Tesseract.\n3 Fun fact: anyone you know is probably an extremist in at least one dimension (e.g., how\nmuch sugar they put in their coffee), if you consider enough dimensions.\n4 Karl Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space,” The\nLondon, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2, no. 11\n(1901): 559-572, https://homl.info/pca.\n5 Scikit-Learn uses the algorithm described in David A. Ross et al., “Incremental Learning\nfor Robust Visual Tracking,” International Journal of Computer Vision 77, no. 1–3 (2008):\n125–141.\n6 Bernhard Schölkopf et al., “Kernel Principal Component Analysis,” in Lecture Notes in\nComputer Science 1327 (Berlin: Springer, 1997): 583–588.\n7 If you set fit_inverse_transform=True, Scikit-Learn will use the algorithm (based on\nKernel Ridge Regression) described in Gokhan H. Bakır et al., “Learning to Find Pre-\nImages”, Proceedings of the 16th International Conference on Neural Information\nProcessing Systems (2004): 449–456.\n8 Sam T. Roweis and Lawrence K. Saul, “Nonlinear Dimensionality Reduction by Locally\nLinear Embedding,” Science 290, no. 5500 (2000): 2323–2326.\n9 The geodesic distance between two nodes in a graph is the number of nodes on the\nshortest path between these nodes." + }, + { + "chapter": 9, + "text": "Chapter 9. Unsupervised\nLearning Techniques\nAlthough most of the applications of Machine Learning today are based on\nsupervised learning (and as a result, this is where most of the investments\ngo to), the vast majority of the available data is unlabeled: we have the\ninput features X, but we do not have the labels y. The computer scientist\nYann LeCun famously said that “if intelligence was a cake, unsupervised\nlearning would be the cake, supervised learning would be the icing on the\ncake, and reinforcement learning would be the cherry on the cake.” In\nother words, there is a huge potential in unsupervised learning that we\nhave only barely started to sink our teeth into.\nSay you want to create a system that will take a few pictures of each item\non a manufacturing production line and detect which items are defective.\nYou can fairly easily create a system that will take pictures automatically,\nand this might give you thousands of pictures every day. You can then\nbuild a reasonably large dataset in just a few weeks. But wait, there are no\nlabels! If you want to train a regular binary classifier that will predict\nwhether an item is defective or not, you will need to label every single\npicture as “defective” or “normal.” This will generally require human\nexperts to sit down and manually go through all the pictures. This is a\nlong, costly, and tedious task, so it will usually only be done on a small\nsubset of the available pictures. As a result, the labeled dataset will be\nquite small, and the classifier’s performance will be disappointing.\nMoreover, every time the company makes any change to its products, the\nwhole process will need to be started over from scratch. Wouldn’t it be\ngreat if the algorithm could just exploit the unlabeled data without needing\nhumans to label every picture? Enter unsupervised learning.\nIn Chapter 8 we looked at the most common unsupervised learning task:\ndimensionality reduction. In this chapter we will look at a few more\nunsupervised learning tasks and algorithms:\nClustering\nThe goal is to group similar instances together into clusters. Clustering\nis a great tool for data analysis, customer segmentation, recommender\nsystems, search engines, image segmentation, semi-supervised\nlearning, dimensionality reduction, and more.\nAnomaly detection\nThe objective is to learn what “normal” data looks like, and then use\nthat to detect abnormal instances, such as defective items on a\nproduction line or a new trend in a time series.\nDensity estimation\nThis is the task of estimating the probability density function (PDF) of\nthe random process that generated the dataset. Density estimation is\ncommonly used for anomaly detection: instances located in very low-\ndensity regions are likely to be anomalies. It is also useful for data\nanalysis and visualization.\nReady for some cake? We will start with clustering, using K-Means and\nDBSCAN, and then we will discuss Gaussian mixture models and see how\nthey can be used for density estimation, clustering, and anomaly detection.\nClustering\nAs you enjoy a hike in the mountains, you stumble upon a plant you have\nnever seen before. You look around and you notice a few more. They are\nnot identical, yet they are sufficiently similar for you to know that they\nmost likely belong to the same species (or at least the same genus). You\nmay need a botanist to tell you what species that is, but you certainly don’t\nneed an expert to identify groups of similar-looking objects. This is called\nclustering: it is the task of identifying similar instances and assigning\nthem to clusters, or groups of similar instances.\nJust like in classification, each instance gets assigned to a group. However,\nunlike classification, clustering is an unsupervised task. Consider\nFigure 9-1: on the left is the iris dataset (introduced in Chapter 4), where\neach instance’s species (i.e., its class) is represented with a different\nmarker. It is a labeled dataset, for which classification algorithms such as\nLogistic Regression, SVMs, or Random Forest classifiers are well suited.\nOn the right is the same dataset, but without the labels, so you cannot use a\nclassification algorithm anymore. This is where clustering algorithms step\nin: many of them can easily detect the lower-left cluster. It is also quite\neasy to see with our own eyes, but it is not so obvious that the upper-right\ncluster is composed of two distinct sub-clusters. That said, the dataset has\ntwo additional features (sepal length and width), not represented here, and\nclustering algorithms can make good use of all features, so in fact they\nidentify the three clusters fairly well (e.g., using a Gaussian mixture\nmodel, only 5 instances out of 150 are assigned to the wrong cluster).\nFigure 9-1. Classification (left) versus clustering (right)\nClustering is used in a wide variety of applications, including these:\nFor customer segmentation\nYou can cluster your customers based on their purchases and their\nactivity on your website. This is useful to understand who your\ncustomers are and what they need, so you can adapt your products and\nmarketing campaigns to each segment. For example, customer\nsegmentation can be useful in recommender systems to suggest content\nthat other users in the same cluster enjoyed.\nFor data analysis\nWhen you analyze a new dataset, it can be helpful to run a clustering\nalgorithm, and then analyze each cluster separately.\nAs a dimensionality reduction technique\nOnce a dataset has been clustered, it is usually possible to measure\neach instance’s affinity with each cluster (affinity is any measure of\nhow well an instance fits into a cluster). Each instance’s feature vector\nx can then be replaced with the vector of its cluster affinities. If there\nare k clusters, then this vector is k-dimensional. This vector is\ntypically much lower-dimensional than the original feature vector, but\nit can preserve enough information for further processing.\nFor anomaly detection (also called outlier detection)\nAny instance that has a low affinity to all the clusters is likely to be an\nanomaly. For example, if you have clustered the users of your website\nbased on their behavior, you can detect users with unusual behavior,\nsuch as an unusual number of requests per second. Anomaly detection\nis particularly useful in detecting defects in manufacturing, or for\nfraud detection.\nFor semi-supervised learning\nIf you only have a few labels, you could perform clustering and\npropagate the labels to all the instances in the same cluster. This\ntechnique can greatly increase the number of labels available for a\nsubsequent supervised learning algorithm, and thus improve its\nperformance.\nFor search engines\nSome search engines let you search for images that are similar to a\nreference image. To build such a system, you would first apply a\nclustering algorithm to all the images in your database; similar images\nwould end up in the same cluster. Then when a user provides a\nreference image, all you need to do is use the trained clustering model\nto find this image’s cluster, and you can then simply return all the\nimages from this cluster.\nTo segment an image\nBy clustering pixels according to their color, then replacing each\npixel’s color with the mean color of its cluster, it is possible to\nconsiderably reduce the number of different colors in the image. Image\nsegmentation is used in many object detection and tracking systems, as\nit makes it easier to detect the contour of each object.\nThere is no universal definition of what a cluster is: it really depends on\nthe context, and different algorithms will capture different kinds of\nclusters. Some algorithms look for instances centered around a particular\npoint, called a centroid. Others look for continuous regions of densely\npacked instances: these clusters can take on any shape. Some algorithms\nare hierarchical, looking for clusters of clusters. And the list goes on.\nIn this section, we will look at two popular clustering algorithms, K-\nMeans and DBSCAN, and explore some of their applications, such as\nnonlinear dimensionality reduction, semi-supervised learning, and\nanomaly detection.\nK-Means\nConsider the unlabeled dataset represented in Figure 9-2: you can clearly\nsee five blobs of instances. The K-Means algorithm is a simple algorithm\ncapable of clustering this kind of dataset very quickly and efficiently,\noften in just a few iterations. It was proposed by Stuart Lloyd at Bell Labs\nin 1957 as a technique for pulse-code modulation, but it was only\npublished outside of the company in 1982. In 1965, Edward W. Forgy had\npublished virtually the same algorithm, so K-Means is sometimes referred\nto as Lloyd–Forgy.\n1\nFigure 9-2. An unlabeled dataset composed of five blobs of instances\nLet’s train a K-Means clusterer on this dataset. It will try to find each\nblob’s center and assign each instance to the closest blob:\nfrom sklearn.cluster import KMeans \nk = 5 \nkmeans = KMeans(n_clusters=k) \ny_pred = kmeans.fit_predict(X)\nNote that you have to specify the number of clusters k that the algorithm\nmust find. In this example, it is pretty obvious from looking at the data\nthat k should be set to 5, but in general it is not that easy. We will discuss\nthis shortly.\nEach instance was assigned to one of the five clusters. In the context of\nclustering, an instance’s label is the index of the cluster that this instance\ngets assigned to by the algorithm: this is not to be confused with the class\nlabels in classification (remember that clustering is an unsupervised\nlearning task). The KMeans instance preserves a copy of the labels of the\ninstances it was trained on, available via the labels_ instance variable:\n>>> y_pred \narray([4, 0, 1, ..., 2, 1, 0], dtype=int32) \n>>> y_pred is kmeans.labels_ \nTrue\nWe can also take a look at the five centroids that the algorithm found:\n>>> kmeans.cluster_centers_ \narray([[-2.80389616, 1.80117999], \n [ 0.20876306, 2.25551336], \n [-2.79290307, 2.79641063], \n [-1.46679593, 2.28585348], \n [-2.80037642, 1.30082566]])\nYou can easily assign new instances to the cluster whose centroid is\nclosest:\n>>> X_new = np.array([[0, 2], [3, 2], [-3, 3], [-3, 2.5]]) \n>>> kmeans.predict(X_new) \narray([1, 1, 2, 2], dtype=int32)\nIf you plot the cluster’s decision boundaries, you get a Voronoi tessellation\n(see Figure 9-3, where each centroid is represented with an X).\nFigure 9-3. K-Means decision boundaries (Voronoi tessellation)\nThe vast majority of the instances were clearly assigned to the appropriate\ncluster, but a few instances were probably mislabeled (especially near the\nboundary between the top-left cluster and the central cluster). Indeed, the\nK-Means algorithm does not behave very well when the blobs have very\ndifferent diameters because all it cares about when assigning an instance\nto a cluster is the distance to the centroid.\nInstead of assigning each instance to a single cluster, which is called hard\nclustering, it can be useful to give each instance a score per cluster, which\nis called soft clustering. The score can be the distance between the\ninstance and the centroid; conversely, it can be a similarity score (or\naffinity), such as the Gaussian Radial Basis Function (introduced in\nChapter 5). In the KMeans class, the transform() method measures the\ndistance from each instance to every centroid:\n>>> kmeans.transform(X_new) \narray([[2.81093633, 0.32995317, 2.9042344 , 1.49439034, 2.88633901], \n [5.80730058, 2.80290755, 5.84739223, 4.4759332 , 5.84236351], \n [1.21475352, 3.29399768, 0.29040966, 1.69136631, 1.71086031], \n [0.72581411, 3.21806371, 0.36159148, 1.54808703, 1.21567622]])\nIn this example, the first instance in X_new is located at a distance of 2.81\nfrom the first centroid, 0.33 from the second centroid, 2.90 from the third\ncentroid, 1.49 from the fourth centroid, and 2.89 from the fifth centroid. If\nyou have a high-dimensional dataset and you transform it this way, you\nend up with a k-dimensional dataset: this transformation can be a very\nefficient nonlinear dimensionality reduction technique.\nThe K-Means algorithm\nSo, how does the algorithm work? Well, suppose you were given the\ncentroids. You could easily label all the instances in the dataset by\nassigning each of them to the cluster whose centroid is closest.\nConversely, if you were given all the instance labels, you could easily\nlocate all the centroids by computing the mean of the instances for each\ncluster. But you are given neither the labels nor the centroids, so how can\nyou proceed? Well, just start by placing the centroids randomly (e.g., by\npicking k instances at random and using their locations as centroids). Then\nlabel the instances, update the centroids, label the instances, update the\ncentroids, and so on until the centroids stop moving. The algorithm is\nguaranteed to converge in a finite number of steps (usually quite small); it\nwill not oscillate forever.\nYou can see the algorithm in action in Figure 9-4: the centroids are\ninitialized randomly (top left), then the instances are labeled (top right),\nthen the centroids are updated (center left), the instances are relabeled\n(center right), and so on. As you can see, in just three iterations, the\nalgorithm has reached a clustering that seems close to optimal.\nNOTE\nThe computational complexity of the algorithm is generally linear with regard to the\nnumber of instances m, the number of clusters k, and the number of dimensions n.\nHowever, this is only true when the data has a clustering structure. If it does not,\nthen in the worst-case scenario the complexity can increase exponentially with the\nnumber of instances. In practice, this rarely happens, and K-Means is generally one\nof the fastest clustering algorithms.\n2\nFigure 9-4. The K-Means algorithm\nAlthough the algorithm is guaranteed to converge, it may not converge to\nthe right solution (i.e., it may converge to a local optimum): whether it\ndoes or not depends on the centroid initialization. Figure 9-5 shows two\nsuboptimal solutions that the algorithm can converge to if you are not\nlucky with the random initialization step.\nFigure 9-5. Suboptimal solutions due to unlucky centroid initializations\nLet’s look at a few ways you can mitigate this risk by improving the\ncentroid initialization.\nCentroid initialization methods\nIf you happen to know approximately where the centroids should be (e.g.,\nif you ran another clustering algorithm earlier), then you can set the init\nhyperparameter to a NumPy array containing the list of centroids, and set\nn_init to 1:\ngood_init = np.array([[-3, 3], [-3, 2], [-3, 1], [-1, 2], [0, 2]]) \nkmeans = KMeans(n_clusters=5, init=good_init, n_init=1)\nAnother solution is to run the algorithm multiple times with different\nrandom initializations and keep the best solution. The number of random\ninitializations is controlled by the n_init hyperparameter: by default, it is\nequal to 10, which means that the whole algorithm described earlier runs\n10 times when you call fit(), and Scikit-Learn keeps the best solution.\nBut how exactly does it know which solution is the best? It uses a\nperformance metric! That metric is called the model’s inertia, which is the\nmean squared distance between each instance and its closest centroid. It is\nroughly equal to 223.3 for the model on the left in Figure 9-5, 237.5 for\nthe model on the right in Figure 9-5, and 211.6 for the model in Figure 9-\n3. The KMeans class runs the algorithm n_init times and keeps the model\nwith the lowest inertia. In this example, the model in Figure 9-3 will be\nselected (unless we are very unlucky with n_init consecutive random\ninitializations). If you are curious, a model’s inertia is accessible via the\ninertia_ instance variable:\n>>> kmeans.inertia_ \n211.59853725816856\nThe score() method returns the negative inertia. Why negative? Because\na predictor’s score() method must always respect Scikit-Learn’s “greater\nis better” rule: if a predictor is better than another, its score() method\nshould return a greater score.\n>>> kmeans.score(X) \n-211.59853725816856\nAn important improvement to the K-Means algorithm, K-Means++, was\nproposed in a 2006 paper by David Arthur and Sergei Vassilvitskii. They\nintroduced a smarter initialization step that tends to select centroids that\nare distant from one another, and this improvement makes the K-Means\nalgorithm much less likely to converge to a suboptimal solution. They\nshowed that the additional computation required for the smarter\ninitialization step is well worth it because it makes it possible to\ndrastically reduce the number of times the algorithm needs to be run to\nfind the optimal solution. Here is the K-Means++ initialization algorithm:\n1. Take one centroid c\n, chosen uniformly at random from the\ndataset.\n2. Take a new centroid c , choosing an instance x with probability \nD(x(i))\n2 / ∑m\nj=1 D(x(j))\n2, where D(x ) is the distance between\nthe instance x and the closest centroid that was already chosen.\nThis probability distribution ensures that instances farther away\nfrom already chosen centroids are much more likely be selected\nas centroids.\n3. Repeat the previous step until all k centroids have been chosen.\nThe KMeans class uses this initialization method by default. If you want to\nforce it to use the original method (i.e., picking k instances randomly to\ndefine the initial centroids), then you can set the init hyperparameter to\n\"random\". You will rarely need to do this.\nAccelerated K-Means and mini-batch K-Means\n3\n(1)\n(i)\n(i)\n(i)\n(i)\nAnother important improvement to the K-Means algorithm was proposed\nin a 2003 paper by Charles Elkan. It considerably accelerates the\nalgorithm by avoiding many unnecessary distance calculations. Elkan\nachieved this by exploiting the triangle inequality (i.e., that a straight line\nis always the shortest distance between two points ) and by keeping track\nof lower and upper bounds for distances between instances and centroids.\nThis is the algorithm the KMeans class uses by default (you can force it to\nuse the original algorithm by setting the algorithm hyperparameter to\n\"full\", although you probably will never need to).\nYet another important variant of the K-Means algorithm was proposed in a\n2010 paper by David Sculley. Instead of using the full dataset at each\niteration, the algorithm is capable of using mini-batches, moving the\ncentroids just slightly at each iteration. This speeds up the algorithm\ntypically by a factor of three or four and makes it possible to cluster huge\ndatasets that do not fit in memory. Scikit-Learn implements this algorithm\nin the MiniBatchKMeans class. You can just use this class like the KMeans\nclass:\nfrom sklearn.cluster import MiniBatchKMeans \n \nminibatch_kmeans = MiniBatchKMeans(n_clusters=5) \nminibatch_kmeans.fit(X)\nIf the dataset does not fit in memory, the simplest option is to use the\nmemmap class, as we did for incremental PCA in Chapter 8. Alternatively,\nyou can pass one mini-batch at a time to the partial_fit() method, but\nthis will require much more work, since you will need to perform multiple\ninitializations and select the best one yourself (see the mini-batch K-\nMeans section of the notebook for an example).\nAlthough the Mini-batch K-Means algorithm is much faster than the\nregular K-Means algorithm, its inertia is generally slightly worse,\nespecially as the number of clusters increases. You can see this in\nFigure 9-6: the plot on the left compares the inertias of Mini-batch K-\nMeans and regular K-Means models trained on the previous dataset using\n4\n5\n6\nvarious numbers of clusters k. The difference between the two curves\nremains fairly constant, but this difference becomes more and more\nsignificant as k increases, since the inertia becomes smaller and smaller. In\nthe plot on the right, you can see that Mini-batch K-Means is much faster\nthan regular K-Means, and this difference increases with k.\nFigure 9-6. Mini-batch K-Means has a higher inertia than K-Means (left) but it is much faster\n(right), especially as k increases\nFinding the optimal number of clusters\nSo far, we have set the number of clusters k to 5 because it was obvious by\nlooking at the data that this was the correct number of clusters. But in\ngeneral, it will not be so easy to know how to set k, and the result might be\nquite bad if you set it to the wrong value. As you can see in Figure 9-7,\nsetting k to 3 or 8 results in fairly bad models.\nFigure 9-7. Bad choices for the number of clusters: when k is too small, separate clusters get\nmerged (left), and when k is too large, some clusters get chopped into multiple pieces (right)\nYou might be thinking that we could just pick the model with the lowest\ninertia, right? Unfortunately, it is not that simple. The inertia for k=3 is\n653.2, which is much higher than for k=5 (which was 211.6). But with k=8,\nthe inertia is just 119.1. The inertia is not a good performance metric when\ntrying to choose k because it keeps getting lower as we increase k. Indeed,\nthe more clusters there are, the closer each instance will be to its closest\ncentroid, and therefore the lower the inertia will be. Let’s plot the inertia\nas a function of k (see Figure 9-8).\nFigure 9-8. When plotting the inertia as a function of the number of clusters k, the curve often\ncontains an inflexion point called the “elbow”\nAs you can see, the inertia drops very quickly as we increase k up to 4, but\nthen it decreases much more slowly as we keep increasing k. This curve\nhas roughly the shape of an arm, and there is an “elbow” at k = 4. So, if we\ndid not know better, 4 would be a good choice: any lower value would be\ndramatic, while any higher value would not help much, and we might just\nbe splitting perfectly good clusters in half for no good reason.\nThis technique for choosing the best value for the number of clusters is\nrather coarse. A more precise approach (but also more computationally\nexpensive) is to use the silhouette score, which is the mean silhouette\ncoefficient over all the instances. An instance’s silhouette coefficient is\nequal to (b – a) / max(a, b), where a is the mean distance to the other\ninstances in the same cluster (i.e., the mean intra-cluster distance) and b is\nthe mean nearest-cluster distance (i.e., the mean distance to the instances\nof the next closest cluster, defined as the one that minimizes b, excluding\nthe instance’s own cluster). The silhouette coefficient can vary between –1\nand +1. A coefficient close to +1 means that the instance is well inside its\nown cluster and far from other clusters, while a coefficient close to 0\nmeans that it is close to a cluster boundary, and finally a coefficient close\nto –1 means that the instance may have been assigned to the wrong cluster.\nTo compute the silhouette score, you can use Scikit-Learn’s\nsilhouette_score() function, giving it all the instances in the dataset\nand the labels they were assigned:\n>>> from sklearn.metrics import silhouette_score \n>>> silhouette_score(X, kmeans.labels_) \n0.655517642572828\nLet’s compare the silhouette scores for different numbers of clusters (see\nFigure 9-9).\nFigure 9-9. Selecting the number of clusters k using the silhouette score\nAs you can see, this visualization is much richer than the previous one:\nalthough it confirms that k = 4 is a very good choice, it also underlines the\nfact that k = 5 is quite good as well, and much better than k = 6 or 7. This\nwas not visible when comparing inertias.\nAn even more informative visualization is obtained when you plot every\ninstance’s silhouette coefficient, sorted by the cluster they are assigned to\nand by the value of the coefficient. This is called a silhouette diagram (see\nFigure 9-10). Each diagram contains one knife shape per cluster. The\nshape’s height indicates the number of instances the cluster contains, and\nits width represents the sorted silhouette coefficients of the instances in\nthe cluster (wider is better). The dashed line indicates the mean silhouette\ncoefficient.\nFigure 9-10. Analyzing the silhouette diagrams for various values of k\nThe vertical dashed lines represent the silhouette score for each number of\nclusters. When most of the instances in a cluster have a lower coefficient\nthan this score (i.e., if many of the instances stop short of the dashed line,\nending to the left of it), then the cluster is rather bad since this means its\ninstances are much too close to other clusters. We can see that when k = 3\nand when k = 6, we get bad clusters. But when k = 4 or k = 5, the clusters\nlook pretty good: most instances extend beyond the dashed line, to the\nright and closer to 1.0. When k = 4, the cluster at index 1 (the third from\nthe top) is rather big. When k = 5, all clusters have similar sizes. So, even\nthough the overall silhouette score from k = 4 is slightly greater than for k\n= 5, it seems like a good idea to use k = 5 to get clusters of similar sizes.\nLimits of K-Means\nDespite its many merits, most notably being fast and scalable, K-Means is\nnot perfect. As we saw, it is necessary to run the algorithm several times to\navoid suboptimal solutions, plus you need to specify the number of\nclusters, which can be quite a hassle. Moreover, K-Means does not behave\nvery well when the clusters have varying sizes, different densities, or\nnonspherical shapes. For example, Figure 9-11 shows how K-Means\nclusters a dataset containing three ellipsoidal clusters of different\ndimensions, densities, and orientations.\nFigure 9-11. K-Means fails to cluster these ellipsoidal blobs properly\nAs you can see, neither of these solutions is any good. The solution on the\nleft is better, but it still chops off 25% of the middle cluster and assigns it\nto the cluster on the right. The solution on the right is just terrible, even\nthough its inertia is lower. So, depending on the data, different clustering\nalgorithms may perform better. On these types of elliptical clusters,\nGaussian mixture models work great.\nTIP\nIt is important to scale the input features before you run K-Means, or the clusters\nmay be very stretched and K-Means will perform poorly. Scaling the features does\nnot guarantee that all the clusters will be nice and spherical, but it generally improves\nthings.\nNow let’s look at a few ways we can benefit from clustering. We will use\nK-Means, but feel free to experiment with other clustering algorithms.\nUsing Clustering for Image Segmentation\nImage segmentation is the task of partitioning an image into multiple\nsegments. In semantic segmentation, all pixels that are part of the same\nobject type get assigned to the same segment. For example, in a self-\ndriving car’s vision system, all pixels that are part of a pedestrian’s image\nmight be assigned to the “pedestrian” segment (there would be one\nsegment containing all the pedestrians). In instance segmentation, all\npixels that are part of the same individual object are assigned to the same\nsegment. In this case there would be a different segment for each\npedestrian. The state of the art in semantic or instance segmentation today\nis achieved using complex architectures based on convolutional neural\nnetworks (see Chapter 14). Here, we are going to do something much\nsimpler: color segmentation. We will simply assign pixels to the same\nsegment if they have a similar color. In some applications, this may be\nsufficient. For example, if you want to analyze satellite images to measure\nhow much total forest area there is in a region, color segmentation may be\njust fine.\nFirst, use Matplotlib’s imread() function to load the image (see the upper-\nleft image in Figure 9-12):\n>>> from matplotlib.image import imread # or `from imageio import \nimread` \n>>> image = \nimread(os.path.join(\"images\",\"unsupervised_learning\",\"ladybug.png\")) \n>>> image.shape \n(533, 800, 3)\nThe image is represented as a 3D array. The first dimension’s size is the\nheight; the second is the width; and the third is the number of color\nchannels, in this case red, green, and blue (RGB). In other words, for each\npixel there is a 3D vector containing the intensities of red, green, and blue,\neach between 0.0 and 1.0 (or between 0 and 255, if you use\nimageio.imread()). Some images may have fewer channels, such as\ngrayscale images (one channel). And some images may have more\nchannels, such as images with an additional alpha channel for\ntransparency or satellite images, which often contain channels for many\nlight frequencies (e.g., infrared). The following code reshapes the array to\nget a long list of RGB colors, then it clusters these colors using K-Means:\nX = image.reshape(-1, 3) \nkmeans = KMeans(n_clusters=8).fit(X) \nsegmented_img = kmeans.cluster_centers_[kmeans.labels_] \nsegmented_img = segmented_img.reshape(image.shape)\nFor example, it may identify a color cluster for all shades of green. Next,\nfor each color (e.g., dark green), it looks for the mean color of the pixel’s\ncolor cluster. For example, all shades of green may be replaced with the\nsame light green color (assuming the mean color of the green cluster is\nlight green). Finally, it reshapes this long list of colors to get the same\nshape as the original image. And we’re done!\nThis outputs the image shown in the upper right of Figure 9-12. You can\nexperiment with various numbers of clusters, as shown in the figure. When\nyou use fewer than eight clusters, notice that the ladybug’s flashy red color\nfails to get a cluster of its own: it gets merged with colors from the\nenvironment. This is because K-Means prefers clusters of similar sizes.\nThe ladybug is small—much smaller than the rest of the image—so even\nthough its color is flashy, K-Means fails to dedicate a cluster to it.\nFigure 9-12. Image segmentation using K-Means with various numbers of color clusters\nThat wasn’t too hard, was it? Now let’s look at another application of\nclustering: preprocessing.\nUsing Clustering for Preprocessing\nClustering can be an efficient approach to dimensionality reduction, in\nparticular as a preprocessing step before a supervised learning algorithm.\nAs an example of using clustering for dimensionality reduction, let’s\ntackle the digits dataset, which is a simple MNIST-like dataset containing\n1,797 grayscale 8 × 8 images representing the digits 0 to 9. First, load the\ndataset:\nfrom sklearn.datasets import load_digits \n \nX_digits, y_digits = load_digits(return_X_y=True)\nNow, split it into a training set and a test set:\nfrom sklearn.model_selection import train_test_split \n \nX_train, X_test, y_train, y_test = train_test_split(X_digits, y_digits)\nNext, fit a Logistic Regression model:\nfrom sklearn.linear_model import LogisticRegression \n \nlog_reg = LogisticRegression() \nlog_reg.fit(X_train, y_train)\nLet’s evaluate its accuracy on the test set:\n>>> log_reg.score(X_test, y_test) \n0.9688888888888889\nOK, that’s our baseline: 96.9% accuracy. Let’s see if we can do better by\nusing K-Means as a preprocessing step. We will create a pipeline that will\nfirst cluster the training set into 50 clusters and replace the images with\ntheir distances to these 50 clusters, then apply a Logistic Regression\nmodel:\nfrom sklearn.pipeline import Pipeline \n \npipeline = Pipeline([ \n (\"kmeans\", KMeans(n_clusters=50)), \n (\"log_reg\", LogisticRegression()), \n]) \npipeline.fit(X_train, y_train)\nWARNING\nSince there are 10 different digits, it is tempting to set the number of clusters to 10.\nHowever, each digit can be written several different ways, so it is preferable to use a\nlarger number of clusters, such as 50.\nNow let’s evaluate this classification pipeline:\n>>> pipeline.score(X_test, y_test) \n0.9777777777777777\nHow about that? We reduced the error rate by almost 30% (from about\n3.1% to about 2.2%)!\nBut we chose the number of clusters k arbitrarily; we can surely do better.\nSince K-Means is just a preprocessing step in a classification pipeline,\nfinding a good value for k is much simpler than earlier. There’s no need to\nperform silhouette analysis or minimize the inertia; the best value of k is\nsimply the one that results in the best classification performance during\ncross-validation. We can use GridSearchCV to find the optimal number of\nclusters:\nfrom sklearn.model_selection import GridSearchCV \n \nparam_grid = dict(kmeans__n_clusters=range(2, 100)) \ngrid_clf = GridSearchCV(pipeline, param_grid, cv=3, verbose=2) \ngrid_clf.fit(X_train, y_train)\nLet’s look at the best value for k and the performance of the resulting\npipeline:\n>>> grid_clf.best_params_ \n{'kmeans__n_clusters': 99} \n>>> grid_clf.score(X_test, y_test) \n0.9822222222222222\nWith k = 99 clusters, we get a significant accuracy boost, reaching 98.22%\naccuracy on the test set. Cool! You may want to keep exploring higher\nvalues for k, since 99 was the largest value in the range we explored.\nUsing Clustering for Semi-Supervised Learning\nAnother use case for clustering is in semi-supervised learning, when we\nhave plenty of unlabeled instances and very few labeled instances. Let’s\ntrain a Logistic Regression model on a sample of 50 labeled instances\nfrom the digits dataset:\nn_labeled = 50 \nlog_reg = LogisticRegression() \nlog_reg.fit(X_train[:n_labeled], y_train[:n_labeled])\nWhat is the performance of this model on the test set?\n>>> log_reg.score(X_test, y_test) \n0.8333333333333334\nThe accuracy is just 83.3%. It should come as no surprise that this is much\nlower than earlier, when we trained the model on the full training set. Let’s\nsee how we can do better. First, let’s cluster the training set into 50\nclusters. Then for each cluster, let’s find the image closest to the centroid.\nWe will call these images the representative images:\nk = 50 \nkmeans = KMeans(n_clusters=k) \nX_digits_dist = kmeans.fit_transform(X_train) \nrepresentative_digit_idx = np.argmin(X_digits_dist, axis=0) \nX_representative_digits = X_train[representative_digit_idx]\nFigure 9-13 shows these 50 representative images.\nFigure 9-13. Fifty representative digit images (one per cluster)\nLet’s look at each image and manually label it:\ny_representative_digits = np.array([4, 8, 0, 6, 8, 3, ..., 7, 6, 2, 3, 1, \n1])\nNow we have a dataset with just 50 labeled instances, but instead of being\nrandom instances, each of them is a representative image of its cluster.\nLet’s see if the performance is any better:\n>>> log_reg = LogisticRegression() \n>>> log_reg.fit(X_representative_digits, y_representative_digits) \n>>> log_reg.score(X_test, y_test) \n0.9222222222222223\nWow! We jumped from 83.3% accuracy to 92.2%, although we are still\nonly training the model on 50 instances. Since it is often costly and painful\nto label instances, especially when it has to be done manually by experts,\nit is a good idea to label representative instances rather than just random\ninstances.\nBut perhaps we can go one step further: what if we propagated the labels\nto all the other instances in the same cluster? This is called label\npropagation:\ny_train_propagated = np.empty(len(X_train), dtype=np.int32) \nfor i in range(k): \n y_train_propagated[kmeans.labels_==i] = y_representative_digits[i]\nNow let’s train the model again and look at its performance:\n>>> log_reg = LogisticRegression() \n>>> log_reg.fit(X_train, y_train_propagated) \n>>> log_reg.score(X_test, y_test) \n0.9333333333333333\nWe got a reasonable accuracy boost, but nothing absolutely astounding.\nThe problem is that we propagated each representative instance’s label to\nall the instances in the same cluster, including the instances located close\nto the cluster boundaries, which are more likely to be mislabeled. Let’s see\nwhat happens if we only propagate the labels to the 20% of the instances\nthat are closest to the centroids:\npercentile_closest = 20 \n \nX_cluster_dist = X_digits_dist[np.arange(len(X_train)), kmeans.labels_] \nfor i in range(k): \n in_cluster = (kmeans.labels_ == i) \n cluster_dist = X_cluster_dist[in_cluster] \n cutoff_distance = np.percentile(cluster_dist, percentile_closest) \n above_cutoff = (X_cluster_dist > cutoff_distance) \n X_cluster_dist[in_cluster & above_cutoff] = -1 \n \npartially_propagated = (X_cluster_dist != -1) \nX_train_partially_propagated = X_train[partially_propagated] \ny_train_partially_propagated = y_train_propagated[partially_propagated]\nNow let’s train the model again on this partially propagated dataset:\n>>> log_reg = LogisticRegression() \n>>> log_reg.fit(X_train_partially_propagated, \ny_train_partially_propagated) \n>>> log_reg.score(X_test, y_test) \n0.94\nNice! With just 50 labeled instances (only 5 examples per class on\naverage!), we got 94.0% accuracy, which is pretty close to the performance\nof Logistic Regression on the fully labeled digits dataset (which was\n96.9%). This good performance is due to the fact that the propagated\nlabels are actually pretty good—their accuracy is very close to 99%, as the\nfollowing code shows:\n>>> np.mean(y_train_partially_propagated == \ny_train[partially_propagated]) \n0.9896907216494846\nACTIVE LEARNING\nTo continue improving your model and your training set, the next step\ncould be to do a few rounds of active learning, which is when a human\nexpert interacts with the learning algorithm, providing labels for\nspecific instances when the algorithm requests them. There are many\ndifferent strategies for active learning, but one of the most common\nones is called uncertainty sampling. Here is how it works:\n1. The model is trained on the labeled instances gathered so far,\nand this model is used to make predictions on all the\nunlabeled instances.\n2. The instances for which the model is most uncertain (i.e.,\nwhen its estimated probability is lowest) are given to the\nexpert to be labeled.\n3. You iterate this process until the performance improvement\nstops being worth the labeling effort.\nOther strategies include labeling the instances that would result in the\nlargest model change, or the largest drop in the model’s validation\nerror, or the instances that different models disagree on (e.g., an SVM\nor a Random Forest).\nBefore we move on to Gaussian mixture models, let’s take a look at\nDBSCAN, another popular clustering algorithm that illustrates a very\ndifferent approach based on local density estimation. This approach allows\nthe algorithm to identify clusters of arbitrary shapes.\nDBSCAN\nThis algorithm defines clusters as continuous regions of high density. Here\nis how it works:\nFor each instance, the algorithm counts how many instances are\nlocated within a small distance ε (epsilon) from it. This region is\ncalled the instance’s ε-neighborhood.\nIf an instance has at least min_samples instances in its ε-\nneighborhood (including itself), then it is considered a core\ninstance. In other words, core instances are those that are located\nin dense regions.\nAll instances in the neighborhood of a core instance belong to the\nsame cluster. This neighborhood may include other core\ninstances; therefore, a long sequence of neighboring core\ninstances forms a single cluster.\nAny instance that is not a core instance and does not have one in\nits neighborhood is considered an anomaly.\nThis algorithm works well if all the clusters are dense enough and if they\nare well separated by low-density regions. The DBSCAN class in Scikit-\nLearn is as simple to use as you might expect. Let’s test it on the moons\ndataset, introduced in Chapter 5:\nfrom sklearn.cluster import DBSCAN \nfrom sklearn.datasets import make_moons \n \nX, y = make_moons(n_samples=1000, noise=0.05) \ndbscan = DBSCAN(eps=0.05, min_samples=5) \ndbscan.fit(X)\nThe labels of all the instances are now available in the labels_ instance\nvariable:\n>>> dbscan.labels_ \narray([ 0, 2, -1, -1, 1, 0, 0, 0, ..., 3, 2, 3, 3, 4, 2, 6, \n3])\nNotice that some instances have a cluster index equal to –1, which means\nthat they are considered as anomalies by the algorithm. The indices of the\ncore instances are available in the core_sample_indices_ instance\nvariable, and the core instances themselves are available in the\ncomponents_ instance variable:\n>>> len(dbscan.core_sample_indices_) \n808 \n>>> dbscan.core_sample_indices_ \narray([ 0, 4, 5, 6, 7, 8, 10, 11, ..., 992, 993, 995, 997, 998, \n999]) \n>>> dbscan.components_ \narray([[-0.02137124, 0.40618608], \n [-0.84192557, 0.53058695], \n ... \n [-0.94355873, 0.3278936 ], \n [ 0.79419406, 0.60777171]])\nThis clustering is represented in the lefthand plot of Figure 9-14. As you\ncan see, it identified quite a lot of anomalies, plus seven different clusters.\nHow disappointing! Fortunately, if we widen each instance’s neighborhood\nby increasing eps to 0.2, we get the clustering on the right, which looks\nperfect. Let’s continue with this model.\nFigure 9-14. DBSCAN clustering using two different neighborhood radiuses\nSomewhat surprisingly, the DBSCAN class does not have a predict()\nmethod, although it has a fit_predict() method. In other words, it\ncannot predict which cluster a new instance belongs to. This\nimplementation decision was made because different classification\nalgorithms can be better for different tasks, so the authors decided to let\nthe user choose which one to use. Moreover, it’s not hard to implement.\nFor example, let’s train a KNeighborsClassifier:\nfrom sklearn.neighbors import KNeighborsClassifier \n \nknn = KNeighborsClassifier(n_neighbors=50) \nknn.fit(dbscan.components_, dbscan.labels_[dbscan.core_sample_indices_])\nNow, given a few new instances, we can predict which cluster they most\nlikely belong to and even estimate a probability for each cluster:\n>>> X_new = np.array([[-0.5, 0], [0, 0.5], [1, -0.1], [2, 1]]) \n>>> knn.predict(X_new) \narray([1, 0, 1, 0]) \n>>> knn.predict_proba(X_new) \narray([[0.18, 0.82], \n [1. , 0. ], \n [0.12, 0.88], \n [1. , 0. ]])\nNote that we only trained the classifier on the core instances, but we could\nalso have chosen to train it on all the instances, or all but the anomalies:\nthis choice depends on the final task.\nThe decision boundary is represented in Figure 9-15 (the crosses represent\nthe four instances in X_new). Notice that since there is no anomaly in the\ntraining set, the classifier always chooses a cluster, even when that cluster\nis far away. It is fairly straightforward to introduce a maximum distance,\nin which case the two instances that are far away from both clusters are\nclassified as anomalies. To do this, use the kneighbors() method of the\nKNeighborsClassifier. Given a set of instances, it returns the distances\nand the indices of the k nearest neighbors in the training set (two matrices,\neach with k columns):\n>>> y_dist, y_pred_idx = knn.kneighbors(X_new, n_neighbors=1) \n>>> y_pred = dbscan.labels_[dbscan.core_sample_indices_][y_pred_idx] \n>>> y_pred[y_dist > 0.2] = -1 \n>>> y_pred.ravel() \narray([-1, 0, 1, -1])\nFigure 9-15. Decision boundary between two clusters\nIn short, DBSCAN is a very simple yet powerful algorithm capable of\nidentifying any number of clusters of any shape. It is robust to outliers,\nand it has just two hyperparameters (eps and min_samples). If the density\nvaries significantly across the clusters, however, it can be impossible for it\nto capture all the clusters properly. Its computational complexity is\nroughly O(m log m), making it pretty close to linear with regard to the\nnumber of instances, but Scikit-Learn’s implementation can require up to\nO(m ) memory if eps is large.\nTIP\nYou may also want to try Hierarchical DBSCAN (HDBSCAN), which is implemented\nin the scikit-learn-contrib project.\nOther Clustering Algorithms\nScikit-Learn implements several more clustering algorithms that you\nshould take a look at. We cannot cover them all in detail here, but here is a\nbrief overview:\nAgglomerative clustering\n2\nA hierarchy of clusters is built from the bottom up. Think of many tiny\nbubbles floating on water and gradually attaching to each other until\nthere’s one big group of bubbles. Similarly, at each iteration,\nagglomerative clustering connects the nearest pair of clusters (starting\nwith individual instances). If you drew a tree with a branch for every\npair of clusters that merged, you would get a binary tree of clusters,\nwhere the leaves are the individual instances. This approach scales\nvery well to large numbers of instances or clusters. It can capture\nclusters of various shapes, it produces a flexible and informative\ncluster tree instead of forcing you to choose a particular cluster scale,\nand it can be used with any pairwise distance. It can scale nicely to\nlarge numbers of instances if you provide a connectivity matrix, which\nis a sparse m × m matrix that indicates which pairs of instances are\nneighbors (e.g., returned by\nsklearn.neighbors.kneighbors_graph()). Without a connectivity\nmatrix, the algorithm does not scale well to large datasets.\nBIRCH\nThe BIRCH (Balanced Iterative Reducing and Clustering using\nHierarchies) algorithm was designed specifically for very large\ndatasets, and it can be faster than batch K-Means, with similar results,\nas long as the number of features is not too large (<20). During\ntraining, it builds a tree structure containing just enough information\nto quickly assign each new instance to a cluster, without having to\nstore all the instances in the tree: this approach allows it to use limited\nmemory, while handling huge datasets.\nMean-Shift\nThis algorithm starts by placing a circle centered on each instance;\nthen for each circle it computes the mean of all the instances located\nwithin it, and it shifts the circle so that it is centered on the mean.\nNext, it iterates this mean-shifting step until all the circles stop\nmoving (i.e., until each of them is centered on the mean of the\ninstances it contains). Mean-Shift shifts the circles in the direction of\nhigher density, until each of them has found a local density maximum.\nFinally, all the instances whose circles have settled in the same place\n(or close enough) are assigned to the same cluster. Mean-Shift has\nsome of the same features as DBSCAN, like how it can find any\nnumber of clusters of any shape, it has very few hyperparameters (just\none—the radius of the circles, called the bandwidth), and it relies on\nlocal density estimation. But unlike DBSCAN, Mean-Shift tends to\nchop clusters into pieces when they have internal density variations.\nUnfortunately, its computational complexity is O(m ), so it is not\nsuited for large datasets.\nAffinity propagation\nThis algorithm uses a voting system, where instances vote for similar\ninstances to be their representatives, and once the algorithm converges,\neach representative and its voters form a cluster. Affinity propagation\ncan detect any number of clusters of different sizes. Unfortunately, this\nalgorithm has a computational complexity of O(m ), so it too is not\nsuited for large datasets.\nSpectral clustering\nThis algorithm takes a similarity matrix between the instances and\ncreates a low-dimensional embedding from it (i.e., it reduces its\ndimensionality), then it uses another clustering algorithm in this low-\ndimensional space (Scikit-Learn’s implementation uses K-Means.)\nSpectral clustering can capture complex cluster structures, and it can\nalso be used to cut graphs (e.g., to identify clusters of friends on a\nsocial network). It does not scale well to large numbers of instances,\nand it does not behave well when the clusters have very different sizes.\nNow let’s dive into Gaussian mixture models, which can be used for\ndensity estimation, clustering, and anomaly detection.\nGaussian Mixtures\n2\n2\nA Gaussian mixture model (GMM) is a probabilistic model that assumes\nthat the instances were generated from a mixture of several Gaussian\ndistributions whose parameters are unknown. All the instances generated\nfrom a single Gaussian distribution form a cluster that typically looks like\nan ellipsoid. Each cluster can have a different ellipsoidal shape, size,\ndensity, and orientation, just like in Figure 9-11. When you observe an\ninstance, you know it was generated from one of the Gaussian\ndistributions, but you are not told which one, and you do not know what\nthe parameters of these distributions are.\nThere are several GMM variants. In the simplest variant, implemented in\nthe GaussianMixture class, you must know in advance the number k of\nGaussian distributions. The dataset X is assumed to have been generated\nthrough the following probabilistic process:\nFor each instance, a cluster is picked randomly from among k\nclusters. The probability of choosing the j cluster is defined by\nthe cluster’s weight, ϕ . The index of the cluster chosen for the\ni instance is noted z .\nIf z =j, meaning the i instance has been assigned to the j\ncluster, the location x of this instance is sampled randomly from\nthe Gaussian distribution with mean μ and covariance matrix\nΣ . This is noted x(i) ∼N (μ(j), Σ(j)).\nThis generative process can be represented as a graphical model. Figure 9-\n16 represents the structure of the conditional dependencies between\nrandom variables.\nth\n(j) 7\nth\n(i)\n(i)\nth\nth\n(i)\n(j)\n(j)\nFigure 9-16. A graphical representation of a Gaussian mixture model, including its parameters\n(squares), random variables (circles), and their conditional dependencies (solid arrows)\nHere is how to interpret the figure:\nThe circles represent random variables.\nThe squares represent fixed values (i.e., parameters of the model).\nThe large rectangles are called plates. They indicate that their\ncontent is repeated several times.\nThe number at the bottom right of each plate indicates how many\ntimes its content is repeated. So, there are m random variables z\n(from z\n to z\n) and m random variables x . There are also k\nmeans μ and k covariance matrices Σ . Lastly, there is just one\nweight vector ϕ (containing all the weights ϕ\n to ϕ\n).\nEach variable z is drawn from the categorical distribution with\nweights ϕ. Each variable x is drawn from the normal\ndistribution, with the mean and covariance matrix defined by its\ncluster z .\nThe solid arrows represent conditional dependencies. For\nexample, the probability distribution for each random variable z\n8\n(i)\n(1)\n(m)\n(i)\n(j)\n(j)\n(1)\n(k)\n(i)\n(i)\n(i)\n(i)\ndepends on the weight vector ϕ. Note that when an arrow crosses\na plate boundary, it means that it applies to all the repetitions of\nthat plate. For example, the weight vector ϕ conditions the\nprobability distributions of all the random variables x\n to x\n.\nThe squiggly arrow from z to x represents a switch: depending\non the value of z , the instance x will be sampled from a\ndifferent Gaussian distribution. For example, if z =j, then\nx(i) ∼N (μ(j), Σ(j)).\nShaded nodes indicate that the value is known. So, in this case,\nonly the random variables x have known values: they are called\nobserved variables. The unknown random variables z are called\nlatent variables.\nSo, what can you do with such a model? Well, given the dataset X, you\ntypically want to start by estimating the weights ϕ and all the distribution\nparameters μ\n to μ\n and Σ\n to Σ\n. Scikit-Learn’s GaussianMixture\nclass makes this super easy:\nfrom sklearn.mixture import GaussianMixture \n \ngm = GaussianMixture(n_components=3, n_init=10) \ngm.fit(X)\nLet’s look at the parameters that the algorithm estimated:\n>>> gm.weights_ \narray([0.20965228, 0.4000662 , 0.39028152]) \n>>> gm.means_ \narray([[ 3.39909717, 1.05933727], \n [-1.40763984, 1.42710194], \n [ 0.05135313, 0.07524095]]) \n>>> gm.covariances_ \narray([[[ 1.14807234, -0.03270354], \n [-0.03270354, 0.95496237]], \n \n [[ 0.63478101, 0.72969804], \n [ 0.72969804, 1.1609872 ]], \n \n(1)\n(m)\n(i)\n(i)\n(i)\n(i)\n(i)\n(i)\n(i)\n(1)\n(k)\n(1)\n(k)\n [[ 0.68809572, 0.79608475], \n [ 0.79608475, 1.21234145]]])\nGreat, it worked fine! Indeed, the weights that were used to generate the\ndata were 0.2, 0.4, and 0.4; and similarly, the means and covariance\nmatrices were very close to those found by the algorithm. But how? This\nclass relies on the Expectation-Maximization (EM) algorithm, which has\nmany similarities with the K-Means algorithm: it also initializes the\ncluster parameters randomly, then it repeats two steps until convergence,\nfirst assigning instances to clusters (this is called the expectation step) and\nthen updating the clusters (this is called the maximization step). Sounds\nfamiliar, right? In the context of clustering, you can think of EM as a\ngeneralization of K-Means that not only finds the cluster centers (μ\n to\nμ\n), but also their size, shape, and orientation (Σ\n to Σ\n), as well as\ntheir relative weights (ϕ\n to ϕ\n). Unlike K-Means, though, EM uses soft\ncluster assignments, not hard assignments. For each instance, during the\nexpectation step, the algorithm estimates the probability that it belongs to\neach cluster (based on the current cluster parameters). Then, during the\nmaximization step, each cluster is updated using all the instances in the\ndataset, with each instance weighted by the estimated probability that it\nbelongs to that cluster. These probabilities are called the responsibilities\nof the clusters for the instances. During the maximization step, each\ncluster’s update will mostly be impacted by the instances it is most\nresponsible for.\nWARNING\nUnfortunately, just like K-Means, EM can end up converging to poor solutions, so it\nneeds to be run several times, keeping only the best solution. This is why we set\nn_init to 10. Be careful: by default n_init is set to 1.\nYou can check whether or not the algorithm converged and how many\niterations it took:\n(1)\n(k)\n(1)\n(k)\n(1)\n(k)\n>>> gm.converged_ \nTrue \n>>> gm.n_iter_ \n3\nNow that you have an estimate of the location, size, shape, orientation, and\nrelative weight of each cluster, the model can easily assign each instance\nto the most likely cluster (hard clustering) or estimate the probability that\nit belongs to a particular cluster (soft clustering). Just use the predict()\nmethod for hard clustering, or the predict_proba() method for soft\nclustering:\n>>> gm.predict(X) \narray([2, 2, 1, ..., 0, 0, 0]) \n>>> gm.predict_proba(X) \narray([[2.32389467e-02, 6.77397850e-07, 9.76760376e-01], \n [1.64685609e-02, 6.75361303e-04, 9.82856078e-01], \n [2.01535333e-06, 9.99923053e-01, 7.49319577e-05], \n ..., \n [9.99999571e-01, 2.13946075e-26, 4.28788333e-07], \n [1.00000000e+00, 1.46454409e-41, 5.12459171e-16], \n [1.00000000e+00, 8.02006365e-41, 2.27626238e-15]])\nA Gaussian mixture model is a generative model, meaning you can sample\nnew instances from it (note that they are ordered by cluster index):\n>>> X_new, y_new = gm.sample(6) \n>>> X_new \narray([[ 2.95400315, 2.63680992], \n [-1.16654575, 1.62792705], \n [-1.39477712, -1.48511338], \n [ 0.27221525, 0.690366 ], \n [ 0.54095936, 0.48591934], \n [ 0.38064009, -0.56240465]]) \n \n>>> y_new \narray([0, 1, 2, 2, 2, 2])\nIt is also possible to estimate the density of the model at any given\nlocation. This is achieved using the score_samples() method: for each\ninstance it is given, this method estimates the log of the probability\ndensity function (PDF) at that location. The greater the score, the higher\nthe density:\n>>> gm.score_samples(X) \narray([-2.60782346, -3.57106041, -3.33003479, ..., -3.51352783, \n -4.39802535, -3.80743859])\nIf you compute the exponential of these scores, you get the value of the\nPDF at the location of the given instances. These are not probabilities, but\nprobability densities: they can take on any positive value, not just a value\nbetween 0 and 1. To estimate the probability that an instance will fall\nwithin a particular region, you would have to integrate the PDF over that\nregion (if you do so over the entire space of possible instance locations,\nthe result will be 1).\nFigure 9-17 shows the cluster means, the decision boundaries (dashed\nlines), and the density contours of this model.\nFigure 9-17. Cluster means, decision boundaries, and density contours of a trained Gaussian\nmixture model\nNice! The algorithm clearly found an excellent solution. Of course, we\nmade its task easy by generating the data using a set of 2D Gaussian\ndistributions (unfortunately, real-life data is not always so Gaussian and\nlow-dimensional). We also gave the algorithm the correct number of\nclusters. When there are many dimensions, or many clusters, or few\ninstances, EM can struggle to converge to the optimal solution. You might\nneed to reduce the difficulty of the task by limiting the number of\nparameters that the algorithm has to learn. One way to do this is to limit\nthe range of shapes and orientations that the clusters can have. This can be\nachieved by imposing constraints on the covariance matrices. To do this,\nset the covariance_type hyperparameter to one of the following values:\n\"spherical\"\nAll clusters must be spherical, but they can have different diameters\n(i.e., different variances).\n\"diag\"\nClusters can take on any ellipsoidal shape of any size, but the\nellipsoid’s axes must be parallel to the coordinate axes (i.e., the\ncovariance matrices must be diagonal).\n\"tied\"\nAll clusters must have the same ellipsoidal shape, size, and orientation\n(i.e., all clusters share the same covariance matrix).\nBy default, covariance_type is equal to \"full\", which means that each\ncluster can take on any shape, size, and orientation (it has its own\nunconstrained covariance matrix). Figure 9-18 plots the solutions found by\nthe EM algorithm when covariance_type is set to \"tied\" or\n\"spherical.”\nFigure 9-18. Gaussian mixtures for tied clusters (left) and spherical clusters (right)\nNOTE\nThe computational complexity of training a GaussianMixture model depends on\nthe number of instances m, the number of dimensions n, the number of clusters k,\nand the constraints on the covariance matrices. If covariance_type is \"spherical\nor \"diag\", it is O(kmn), assuming the data has a clustering structure. If\ncovariance_type is \"tied\" or \"full\", it is O(kmn + kn ), so it will not scale to\nlarge numbers of features.\nGaussian mixture models can also be used for anomaly detection. Let’s see\nhow.\nAnomaly Detection Using Gaussian Mixtures\nAnomaly detection (also called outlier detection) is the task of detecting\ninstances that deviate strongly from the norm. These instances are called\nanomalies, or outliers, while the normal instances are called inliers.\nAnomaly detection is useful in a wide variety of applications, such as\nfraud detection, detecting defective products in manufacturing, or\nremoving outliers from a dataset before training another model (which can\nsignificantly improve the performance of the resulting model).\n2\n3\nUsing a Gaussian mixture model for anomaly detection is quite simple:\nany instance located in a low-density region can be considered an anomaly.\nYou must define what density threshold you want to use. For example, in a\nmanufacturing company that tries to detect defective products, the ratio of\ndefective products is usually well known. Say it is equal to 4%. You then\nset the density threshold to be the value that results in having 4% of the\ninstances located in areas below that threshold density. If you notice that\nyou get too many false positives (i.e., perfectly good products that are\nflagged as defective), you can lower the threshold. Conversely, if you have\ntoo many false negatives (i.e., defective products that the system does not\nflag as defective), you can increase the threshold. This is the usual\nprecision/recall trade-off (see Chapter 3). Here is how you would identify\nthe outliers using the fourth percentile lowest density as the threshold (i.e.,\napproximately 4% of the instances will be flagged as anomalies):\ndensities = gm.score_samples(X) \ndensity_threshold = np.percentile(densities, 4) \nanomalies = X[densities < density_threshold]\nFigure 9-19 represents these anomalies as stars.\nFigure 9-19. Anomaly detection using a Gaussian mixture model\nA closely related task is novelty detection: it differs from anomaly\ndetection in that the algorithm is assumed to be trained on a “clean”\ndataset, uncontaminated by outliers, whereas anomaly detection does not\nmake this assumption. Indeed, outlier detection is often used to clean up a\ndataset.\nTIP\nGaussian mixture models try to fit all the data, including the outliers, so if you have\ntoo many of them, this will bias the model’s view of “normality,” and some outliers\nmay wrongly be considered as normal. If this happens, you can try to fit the model\nonce, use it to detect and remove the most extreme outliers, then fit the model again\non the cleaned-up dataset. Another approach is to use robust covariance estimation\nmethods (see the EllipticEnvelope class).\nJust like K-Means, the GaussianMixture algorithm requires you to\nspecify the number of clusters. So, how can you find it?\nSelecting the Number of Clusters\nWith K-Means, you could use the inertia or the silhouette score to select\nthe appropriate number of clusters. But with Gaussian mixtures, it is not\npossible to use these metrics because they are not reliable when the\nclusters are not spherical or have different sizes. Instead, you can try to\nfind the model that minimizes a theoretical information criterion, such as\nthe Bayesian information criterion (BIC) or the Akaike information\ncriterion (AIC), defined in Equation 9-1.\nEquation 9-1. Bayesian information criterion (BIC) and Akaike information criterion (AIC)\nBIC =\nlog (m) p −2 log (ˆL)\nAIC =\n2p −2 log(ˆL)\nIn these equations:\nm is the number of instances, as always.\np is the number of parameters learned by the model.\nˆL is the maximized value of the likelihood function of the model.\nBoth the BIC and the AIC penalize models that have more parameters to\nlearn (e.g., more clusters) and reward models that fit the data well. They\noften end up selecting the same model. When they differ, the model\nselected by the BIC tends to be simpler (fewer parameters) than the one\nselected by the AIC, but tends to not fit the data quite as well (this is\nespecially true for larger datasets).\nLIKELIHOOD FUNCTION\nThe terms “probability” and “likelihood” are often used\ninterchangeably in the English language, but they have very different\nmeanings in statistics. Given a statistical model with some parameters\nθ, the word “probability” is used to describe how plausible a future\noutcome x is (knowing the parameter values θ), while the word\n“likelihood” is used to describe how plausible a particular set of\nparameter values θ are, after the outcome x is known.\nConsider a 1D mixture model of two Gaussian distributions centered\nat –4 and +1. For simplicity, this toy model has a single parameter θ\nthat controls the standard deviations of both distributions. The top-left\ncontour plot in Figure 9-20 shows the entire model f(x; θ) as a function\nof both x and θ. To estimate the probability distribution of a future\noutcome x, you need to set the model parameter θ. For example, if you\nset θ to 1.3 (the horizontal line), you get the probability density\nfunction f(x; θ=1.3) shown in the lower-left plot. Say you want to\nestimate the probability that x will fall between –2 and +2. You must\ncalculate the integral of the PDF on this range (i.e., the surface of the\nshaded region). But what if you don’t know θ, and instead if you have\nobserved a single instance x=2.5 (the vertical line in the upper-left\nplot)? In this case, you get the likelihood function L\n(θ|x=2.5)=f(x=2.5; θ), represented in the upper-right plot.\nFigure 9-20. A model’s parametric function (top left), and some derived functions: a PDF\n(lower left), a likelihood function (top right), and a log likelihood function (lower right)\nIn short, the PDF is a function of x (with θ fixed), while the likelihood\nfunction is a function of θ (with x fixed). It is important to understand\nthat the likelihood function is not a probability distribution: if you\nintegrate a probability distribution over all possible values of x, you\nalways get 1; but if you integrate the likelihood function over all\npossible values of θ, the result can be any positive value.\nGiven a dataset X, a common task is to try to estimate the most likely\nvalues for the model parameters. To do this, you must find the values\nthat maximize the likelihood function, given X. In this example, if you\nhave observed a single instance x=2.5, the maximum likelihood\nestimate (MLE) of θ is ˆθ=1.5. If a prior probability distribution g over\nθ exists, it is possible to take it into account by maximizing L\n(θ|x)g(θ) rather than just maximizing L (θ|x). This is called maximum\na-posteriori (MAP) estimation. Since MAP constrains the parameter\nvalues, you can think of it as a regularized version of MLE.\nNotice that maximizing the likelihood function is equivalent to\nmaximizing its logarithm (represented in the lower-righthand plot in\nFigure 9-20). Indeed the logarithm is a strictly increasing function, so\nif θ maximizes the log likelihood, it also maximizes the likelihood. It\nturns out that it is generally easier to maximize the log likelihood. For\nexample, if you observed several independent instances x\n to x\n,\nyou would need to find the value of θ that maximizes the product of\nthe individual likelihood functions. But it is equivalent, and much\nsimpler, to maximize the sum (not the product) of the log likelihood\nfunctions, thanks to the magic of the logarithm which converts\nproducts into sums: log(ab)=log(a)+log(b).\nOnce you have estimated ˆθ, the value of θ that maximizes the\nlikelihood function, then you are ready to compute ˆL = L (ˆθ, X),\nwhich is the value used to compute the AIC and BIC; you can think of\nit as a measure of how well the model fits the data.\nTo compute the BIC and AIC, call the bic() and aic() methods:\n>>> gm.bic(X) \n8189.74345832983 \n>>> gm.aic(X) \n8102.518178214792\nFigure 9-21 shows the BIC for different numbers of clusters k. As you can\nsee, both the BIC and the AIC are lowest when k=3, so it is most likely the\nbest choice. Note that we could also search for the best value for the\ncovariance_type hyperparameter. For example, if it is \"spherical\"\nrather than \"full\", then the model has significantly fewer parameters to\nlearn, but it does not fit the data as well.\n(1)\n(m)\nFigure 9-21. AIC and BIC for different numbers of clusters k\nBayesian Gaussian Mixture Models\nRather than manually searching for the optimal number of clusters, you\ncan use the BayesianGaussianMixture class, which is capable of giving\nweights equal (or close) to zero to unnecessary clusters. Set the number of\nclusters n_components to a value that you have good reason to believe is\ngreater than the optimal number of clusters (this assumes some minimal\nknowledge about the problem at hand), and the algorithm will eliminate\nthe unnecessary clusters automatically. For example, let’s set the number\nof clusters to 10 and see what happens:\n>>> from sklearn.mixture import BayesianGaussianMixture \n>>> bgm = BayesianGaussianMixture(n_components=10, n_init=10) \n>>> bgm.fit(X) \n>>> np.round(bgm.weights_, 2) \narray([0.4 , 0.21, 0.4 , 0. , 0. , 0. , 0. , 0. , 0. , 0. ])\nPerfect: the algorithm automatically detected that only three clusters are\nneeded, and the resulting clusters are almost identical to the ones in\nFigure 9-17.\nIn this model, the cluster parameters (including the weights, means, and\ncovariance matrices) are not treated as fixed model parameters anymore,\nbut as latent random variables, like the cluster assignments (see Figure 9-\n22). So z now includes both the cluster parameters and the cluster\nassignments.\nThe Beta distribution is commonly used to model random variables whose\nvalues lie within a fixed range. In this case, the range is from 0 to 1. The\nStick-Breaking Process (SBP) is best explained through an example:\nsuppose Φ=[0.3, 0.6, 0.5,…], then 30% of the instances will be assigned to\ncluster 0, then 60% of the remaining instances will be assigned to cluster\n1, then 50% of the remaining instances will be assigned to cluster 2, and\nso on. This process is a good model for datasets where new instances are\nmore likely to join large clusters than small clusters (e.g., people are more\nlikely to move to larger cities). If the concentration α is high, then Φ\nvalues will likely be close to 0, and the SBP generate many clusters.\nConversely, if the concentration is low, then Φ values will likely be close\nto 1, and there will be few clusters. Finally, the Wishart distribution is\nused to sample covariance matrices: the parameters d and V control the\ndistribution of cluster shapes.\nFigure 9-22. Bayesian Gaussian mixture model\nPrior knowledge about the latent variables z can be encoded in a\nprobability distribution p(z) called the prior. For example, we may have a\nprior belief that the clusters are likely to be few (low concentration), or\nconversely, that they are likely to be plentiful (high concentration). This\nprior belief about the number of clusters can be adjusted using the\nweight_concentration_prior hyperparameter. Setting it to 0.01 or\n10,000 gives very different clusterings (see Figure 9-23). The more data\nwe have, however, the less the priors matter. In fact, to plot diagrams with\nsuch large differences, you must use very strong priors and little data.\nFigure 9-23. Using different concentration priors on the same data results in different numbers\nof clusters\nBayes’ theorem (Equation 9-2) tells us how to update the probability\ndistribution over the latent variables after we observe some data X. It\ncomputes the posterior distribution p(z|X), which is the conditional\nprobability of z given X.\nEquation 9-2. Bayes’ theorem\np (z|X) = posterior =\n=\nUnfortunately, in a Gaussian mixture model (and many other problems),\nthe denominator p(x) is intractable, as it requires integrating over all the\npossible values of z (Equation 9-3), which would require considering all\npossible combinations of cluster parameters and cluster assignments.\nEquation 9-3. The evidence p(X) is often intractable\nlikelihood × prior\nevidence\np(X|z) p(z)\np(X)\np(X) = ∫p(X|z)p(z)dz\nThis intractability is one of the central problems in Bayesian statistics, and\nthere are several approaches to solving it. One of them is variational\ninference, which picks a family of distributions q(z; λ) with its own\nvariational parameters λ (lambda), then optimizes these parameters to\nmake q(z) a good approximation of p(z|X). This is achieved by finding the\nvalue of λ that minimizes the KL divergence from q(z) to p(z|X), noted\nD\n(q‖p). The KL divergence equation is shown in Equation 9-4, and it\ncan be rewritten as the log of the evidence (log p(X)) minus the evidence\nlower bound (ELBO). Since the log of the evidence does not depend on q,\nit is a constant term, so minimizing the KL divergence just requires\nmaximizing the ELBO.\nEquation 9-4. KL divergence from q(z) to p(z|X)\nDKL (q ∥p) =\nEq[log\n]\n=\nEq[log q(z) −log p(z | X)]\n=\nEq[log q (z) −log\n]\n=\nEq[log q(z) −log p(z, X) + log p(X)]\n=\nEq[log q(z)]−Eq[log p(z, X)]+Eq[log p(X)]\n=\nEq[log p(X)]−(Eq[log p(z, X)]−Eq[log q(z)])\n=\nlog p(X) −ELBO\nwhere ELBO = Eq[log p(z, X)]−Eq[log q(z)]\nIn practice, there are different techniques to maximize the ELBO. In mean\nfield variational inference, it is necessary to pick the family of\ndistributions q(z; λ) and the prior p(z) very carefully to ensure that the\nequation for the ELBO simplifies to a form that can be computed.\nUnfortunately, there is no general way to do this. Picking the right family\nKL\nq(z)\np(z | X)\np(z, X)\np(X)\nof distributions and the right prior depends on the task and requires some\nmathematical skills. For example, the distributions and lower-bound\nequations used in Scikit-Learn’s BayesianGaussianMixture class are\npresented in the documentation. From these equations it is possible to\nderive update equations for the cluster parameters and assignment\nvariables: these are then used very much like in the Expectation-\nMaximization algorithm. In fact, the computational complexity of the\nBayesianGaussianMixture class is similar to that of the\nGaussianMixture class (but generally significantly slower). A simpler\napproach to maximizing the ELBO is called black box stochastic\nvariational inference (BBSVI): at each iteration, a few samples are drawn\nfrom q, and they are used to estimate the gradients of the ELBO with\nregard to the variational parameters λ, which are then used in a gradient\nascent step. This approach makes it possible to use Bayesian inference\nwith any kind of model (provided it is differentiable), even deep neural\nnetworks; using Bayesian inference with deep neural networks is called\nBayesian Deep Learning.\nTIP\nIf you want to dive deeper into Bayesian statistics, check out the book Bayesian\nData Analysis by Andrew Gelman et al. (Chapman & Hall).\nGaussian mixture models work great on clusters with ellipsoidal shapes,\nbut if you try to fit a dataset with different shapes, you may have bad\nsurprises. For example, let’s see what happens if we use a Bayesian\nGaussian mixture model to cluster the moons dataset (see Figure 9-24).\nFigure 9-24. Fitting a Gaussian mixture to nonellipsoidal clusters\nOops! The algorithm desperately searched for ellipsoids, so it found eight\ndifferent clusters instead of two. The density estimation is not too bad, so\nthis model could perhaps be used for anomaly detection, but it failed to\nidentify the two moons. Let’s now look at a few clustering algorithms\ncapable of dealing with arbitrarily shaped clusters.\nOther Algorithms for Anomaly and Novelty Detection\nScikit-Learn implements other algorithms dedicated to anomaly detection\nor novelty detection:\nPCA (and other dimensionality reduction techniques with an\ninverse_transform() method)\nIf you compare the reconstruction error of a normal instance with the\nreconstruction error of an anomaly, the latter will usually be much\nlarger. This is a simple and often quite efficient anomaly detection\napproach (see this chapter’s exercises for an application of this\napproach).\nFast-MCD (minimum covariance determinant)\nImplemented by the EllipticEnvelope class, this algorithm is useful\nfor outlier detection, in particular to clean up a dataset. It assumes that\nthe normal instances (inliers) are generated from a single Gaussian\ndistribution (not a mixture). It also assumes that the dataset is\ncontaminated with outliers that were not generated from this Gaussian\ndistribution. When the algorithm estimates the parameters of the\nGaussian distribution (i.e., the shape of the elliptic envelope around\nthe inliers), it is careful to ignore the instances that are most likely\noutliers. This technique gives a better estimation of the elliptic\nenvelope and thus makes the algorithm better at identifying the\noutliers.\nIsolation Forest\nThis is an efficient algorithm for outlier detection, especially in high-\ndimensional datasets. The algorithm builds a Random Forest in which\neach Decision Tree is grown randomly: at each node, it picks a feature\nrandomly, then it picks a random threshold value (between the min and\nmax values) to split the dataset in two. The dataset gradually gets\nchopped into pieces this way, until all instances end up isolated from\nthe other instances. Anomalies are usually far from other instances, so\non average (across all the Decision Trees) they tend to get isolated in\nfewer steps than normal instances.\nLocal Outlier Factor (LOF)\nThis algorithm is also good for outlier detection. It compares the\ndensity of instances around a given instance to the density around its\nneighbors. An anomaly is often more isolated than its k nearest\nneighbors.\nOne-class SVM\nThis algorithm is better suited for novelty detection. Recall that a\nkernelized SVM classifier separates two classes by first (implicitly)\nmapping all the instances to a high-dimensional space, then separating\nthe two classes using a linear SVM classifier within this high-\ndimensional space (see Chapter 5). Since we just have one class of\ninstances, the one-class SVM algorithm instead tries to separate the\ninstances in high-dimensional space from the origin. In the original\nspace, this will correspond to finding a small region that encompasses\nall the instances. If a new instance does not fall within this region, it is\nan anomaly. There are a few hyperparameters to tweak: the usual ones\nfor a kernelized SVM, plus a margin hyperparameter that corresponds\nto the probability of a new instance being mistakenly considered as\nnovel when it is in fact normal. It works great, especially with high-\ndimensional datasets, but like all SVMs it does not scale to large\ndatasets.\nExercises\n1. How would you define clustering? Can you name a few clustering\nalgorithms?\n2. What are some of the main applications of clustering algorithms?\n3. Describe two techniques to select the right number of clusters\nwhen using K-Means.\n4. What is label propagation? Why would you implement it, and\nhow?\n5. Can you name two clustering algorithms that can scale to large\ndatasets? And two that look for regions of high density?\n6. Can you think of a use case where active learning would be\nuseful? How would you implement it?\n7. What is the difference between anomaly detection and novelty\ndetection?\n8. What is a Gaussian mixture? What tasks can you use it for?\n9. Can you name two techniques to find the right number of clusters\nwhen using a Gaussian mixture model?\n10. The classic Olivetti faces dataset contains 400 grayscale 64 × 64–\npixel images of faces. Each image is flattened to a 1D vector of\nsize 4,096. 40 different people were photographed (10 times\neach), and the usual task is to train a model that can predict which\nperson is represented in each picture. Load the dataset using the\nsklearn.datasets.fetch_olivetti_faces() function, then\nsplit it into a training set, a validation set, and a test set (note that\nthe dataset is already scaled between 0 and 1). Since the dataset is\nquite small, you probably want to use stratified sampling to\nensure that there are the same number of images per person in\neach set. Next, cluster the images using K-Means, and ensure that\nyou have a good number of clusters (using one of the techniques\ndiscussed in this chapter). Visualize the clusters: do you see\nsimilar faces in each cluster?\n11. Continuing with the Olivetti faces dataset, train a classifier to\npredict which person is represented in each picture, and evaluate\nit on the validation set. Next, use K-Means as a dimensionality\nreduction tool, and train a classifier on the reduced set. Search for\nthe number of clusters that allows the classifier to get the best\nperformance: what performance can you reach? What if you\nappend the features from the reduced set to the original features\n(again, searching for the best number of clusters)?\n12. Train a Gaussian mixture model on the Olivetti faces dataset. To\nspeed up the algorithm, you should probably reduce the dataset’s\ndimensionality (e.g., use PCA, preserving 99% of the variance).\nUse the model to generate some new faces (using the sample()\nmethod), and visualize them (if you used PCA, you will need to\nuse its inverse_transform() method). Try to modify some\nimages (e.g., rotate, flip, darken) and see if the model can detect\nthe anomalies (i.e., compare the output of the score_samples()\nmethod for normal images and for anomalies).\n13. Some dimensionality reduction techniques can also be used for\nanomaly detection. For example, take the Olivetti faces dataset\nand reduce it with PCA, preserving 99% of the variance. Then\ncompute the reconstruction error for each image. Next, take some\nof the modified images you built in the previous exercise, and\nlook at their reconstruction error: notice how much larger the\nreconstruction error is. If you plot a reconstructed image, you will\nsee why: it tries to reconstruct a normal face.\nSolutions to these exercises are available in Appendix A.\n1 Stuart P. Lloyd, “Least Squares Quantization in PCM,” IEEE Transactions on Information\nTheory 28, no. 2 (1982): 129–137.\n2 That’s because the mean squared distance between the instances and their closest centroid\ncan only go down at each step.\n3 David Arthur and Sergei Vassilvitskii, “k-Means++: The Advantages of Careful Seeding,”\nProceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms (2007):\n1027–1035.\n4 Charles Elkan, “Using the Triangle Inequality to Accelerate k-Means,” Proceedings of the\n20th International Conference on Machine Learning (2003): 147–153.\n5 The triangle inequality is AC ≤ AB + BC where A, B and C are three points and AB, AC,\nand BC are the distances between these points.\n6 David Sculley, “Web-Scale K-Means Clustering,” Proceedings of the 19th International\nConference on World Wide Web (2010): 1177–1178.\n7 Phi (ϕ or φ) is the 21st letter of the Greek alphabet.\n8 Most of these notations are standard, but a few additional notations were taken from the\nWikipedia article on plate notation." + } + ] + }, + { + "part": "Part II", + "chapters": [ + { + "chapter": 10, + "text": "Chapter 10. Introduction to\nArtificial Neural Networks with\nKeras\nBirds inspired us to fly, burdock plants inspired Velcro, and nature has\ninspired countless more inventions. It seems only logical, then, to look at\nthe brain’s architecture for inspiration on how to build an intelligent\nmachine. This is the logic that sparked artificial neural networks (ANNs):\nan ANN is a Machine Learning model inspired by the networks of\nbiological neurons found in our brains. However, although planes were\ninspired by birds, they don’t have to flap their wings. Similarly, ANNs\nhave gradually become quite different from their biological cousins. Some\nresearchers even argue that we should drop the biological analogy\naltogether (e.g., by saying “units” rather than “neurons”), lest we restrict\nour creativity to biologically plausible systems.\nANNs are at the very core of Deep Learning. They are versatile, powerful,\nand scalable, making them ideal to tackle large and highly complex\nMachine Learning tasks such as classifying billions of images (e.g.,\nGoogle Images), powering speech recognition services (e.g., Apple’s Siri),\nrecommending the best videos to watch to hundreds of millions of users\nevery day (e.g., YouTube), or learning to beat the world champion at the\ngame of Go (DeepMind’s AlphaGo).\nThe first part of this chapter introduces artificial neural networks, starting\nwith a quick tour of the very first ANN architectures and leading up to\nMultilayer Perceptrons (MLPs), which are heavily used today (other\narchitectures will be explored in the next chapters). In the second part, we\nwill look at how to implement neural networks using the popular Keras\nAPI. This is a beautifully designed and simple high-level API for building,\ntraining, evaluating, and running neural networks. But don’t be fooled by\n1\nits simplicity: it is expressive and flexible enough to let you build a wide\nvariety of neural network architectures. In fact, it will probably be\nsufficient for most of your use cases. And should you ever need extra\nflexibility, you can always write custom Keras components using its\nlower-level API, as we will see in Chapter 12.\nBut first, let’s go back in time to see how artificial neural networks came\nto be!\nFrom Biological to Artificial Neurons\nSurprisingly, ANNs have been around for quite a while: they were first\nintroduced back in 1943 by the neurophysiologist Warren McCulloch and\nthe mathematician Walter Pitts. In their landmark paper “A Logical\nCalculus of Ideas Immanent in Nervous Activity,” McCulloch and Pitts\npresented a simplified computational model of how biological neurons\nmight work together in animal brains to perform complex computations\nusing propositional logic. This was the first artificial neural network\narchitecture. Since then many other architectures have been invented, as\nwe will see.\nThe early successes of ANNs led to the widespread belief that we would\nsoon be conversing with truly intelligent machines. When it became clear\nin the 1960s that this promise would go unfulfilled (at least for quite a\nwhile), funding flew elsewhere, and ANNs entered a long winter. In the\nearly 1980s, new architectures were invented and better training\ntechniques were developed, sparking a revival of interest in connectionism\n(the study of neural networks). But progress was slow, and by the 1990s\nother powerful Machine Learning techniques were invented, such as\nSupport Vector Machines (see Chapter 5). These techniques seemed to\noffer better results and stronger theoretical foundations than ANNs, so\nonce again the study of neural networks was put on hold.\nWe are now witnessing yet another wave of interest in ANNs. Will this\nwave die out like the previous ones did? Well, here are a few good reasons\n2\nto believe that this time is different and that the renewed interest in ANNs\nwill have a much more profound impact on our lives:\nThere is now a huge quantity of data available to train neural\nnetworks, and ANNs frequently outperform other ML techniques\non very large and complex problems.\nThe tremendous increase in computing power since the 1990s now\nmakes it possible to train large neural networks in a reasonable\namount of time. This is in part due to Moore’s law (the number of\ncomponents in integrated circuits has doubled about every 2 years\nover the last 50 years), but also thanks to the gaming industry,\nwhich has stimulated the production of powerful GPU cards by\nthe millions. Moreover, cloud platforms have made this power\naccessible to everyone.\nThe training algorithms have been improved. To be fair they are\nonly slightly different from the ones used in the 1990s, but these\nrelatively small tweaks have had a huge positive impact.\nSome theoretical limitations of ANNs have turned out to be\nbenign in practice. For example, many people thought that ANN\ntraining algorithms were doomed because they were likely to get\nstuck in local optima, but it turns out that this is rather rare in\npractice (and when it is the case, they are usually fairly close to\nthe global optimum).\nANNs seem to have entered a virtuous circle of funding and\nprogress. Amazing products based on ANNs regularly make the\nheadline news, which pulls more and more attention and funding\ntoward them, resulting in more and more progress and even more\namazing products.\nBiological Neurons\nBefore we discuss artificial neurons, let’s take a quick look at a biological\nneuron (represented in Figure 10-1). It is an unusual-looking cell mostly\nfound in animal brains. It’s composed of a cell body containing the nucleus\nand most of the cell’s complex components, many branching extensions\ncalled dendrites, plus one very long extension called the axon. The axon’s\nlength may be just a few times longer than the cell body, or up to tens of\nthousands of times longer. Near its extremity the axon splits off into many\nbranches called telodendria, and at the tip of these branches are minuscule\nstructures called synaptic terminals (or simply synapses), which are\nconnected to the dendrites or cell bodies of other neurons. Biological\nneurons produce short electrical impulses called action potentials (APs, or\njust signals) which travel along the axons and make the synapses release\nchemical signals called neurotransmitters. When a neuron receives a\nsufficient amount of these neurotransmitters within a few milliseconds, it\nfires its own electrical impulses (actually, it depends on the\nneurotransmitters, as some of them inhibit the neuron from firing).\nFigure 10-1. Biological neuron\nThus, individual biological neurons seem to behave in a rather simple way,\nbut they are organized in a vast network of billions, with each neuron\ntypically connected to thousands of other neurons. Highly complex\n3\n4\ncomputations can be performed by a network of fairly simple neurons,\nmuch like a complex anthill can emerge from the combined efforts of\nsimple ants. The architecture of biological neural networks (BNNs) is\nstill the subject of active research, but some parts of the brain have been\nmapped, and it seems that neurons are often organized in consecutive\nlayers, especially in the cerebral cortex (i.e., the outer layer of your brain),\nas shown in Figure 10-2.\nFigure 10-2. Multiple layers in a biological neural network (human cortex)\nLogical Computations with Neurons\nMcCulloch and Pitts proposed a very simple model of the biological\nneuron, which later became known as an artificial neuron: it has one or\nmore binary (on/off) inputs and one binary output. The artificial neuron\nactivates its output when more than a certain number of its inputs are\nactive. In their paper, they showed that even with such a simplified model\nit is possible to build a network of artificial neurons that computes any\nlogical proposition you want. To see how such a network works, let’s build\na few ANNs that perform various logical computations (see Figure 10-3),\nassuming that a neuron is activated when at least two of its inputs are\nactive.\n5\n6\nFigure 10-3. ANNs performing simple logical computations\nLet’s see what these networks do:\nThe first network on the left is the identity function: if neuron A\nis activated, then neuron C gets activated as well (since it receives\ntwo input signals from neuron A); but if neuron A is off, then\nneuron C is off as well.\nThe second network performs a logical AND: neuron C is\nactivated only when both neurons A and B are activated (a single\ninput signal is not enough to activate neuron C).\nThe third network performs a logical OR: neuron C gets activated\nif either neuron A or neuron B is activated (or both).\nFinally, if we suppose that an input connection can inhibit the\nneuron’s activity (which is the case with biological neurons), then\nthe fourth network computes a slightly more complex logical\nproposition: neuron C is activated only if neuron A is active and\nneuron B is off. If neuron A is active all the time, then you get a\nlogical NOT: neuron C is active when neuron B is off, and vice\nversa.\nYou can imagine how these networks can be combined to compute\ncomplex logical expressions (see the exercises at the end of the chapter for\nan example).\nThe Perceptron\nThe Perceptron is one of the simplest ANN architectures, invented in 1957\nby Frank Rosenblatt. It is based on a slightly different artificial neuron\n(see Figure 10-4) called a threshold logic unit (TLU), or sometimes a\nlinear threshold unit (LTU). The inputs and output are numbers (instead of\nbinary on/off values), and each input connection is associated with a\nweight. The TLU computes a weighted sum of its inputs (z = w x + w x\n+ ⋯ + w x = x w), then applies a step function to that sum and outputs\nthe result: h (x) = step(z), where z = x w.\nFigure 10-4. Threshold logic unit: an artificial neuron which computes a weighted sum of its\ninputs then applies a step function\nThe most common step function used in Perceptrons is the Heaviside step\nfunction (see Equation 10-1). Sometimes the sign function is used instead.\nEquation 10-1. Common step functions used in Perceptrons (assuming threshold = 0)\nheaviside (z) ={ 0\nif z < 0\n1\nif z ≥0\nsgn (z) =\n⎧\n⎪\n⎨\n⎪\n⎩\n−1\nif z < 0\n0\nif z = 0\n+1\nif z > 0\n1\n1\n2\n2\nn\nn\n⊺\nw\n⊺\nA single TLU can be used for simple linear binary classification. It\ncomputes a linear combination of the inputs, and if the result exceeds a\nthreshold, it outputs the positive class. Otherwise it outputs the negative\nclass (just like a Logistic Regression or linear SVM classifier). You could,\nfor example, use a single TLU to classify iris flowers based on petal length\nand width (also adding an extra bias feature x = 1, just like we did in\nprevious chapters). Training a TLU in this case means finding the right\nvalues for w , w , and w (the training algorithm is discussed shortly).\nA Perceptron is simply composed of a single layer of TLUs, with each\nTLU connected to all the inputs. When all the neurons in a layer are\nconnected to every neuron in the previous layer (i.e., its input neurons),\nthe layer is called a fully connected layer, or a dense layer. The inputs of\nthe Perceptron are fed to special passthrough neurons called input\nneurons: they output whatever input they are fed. All the input neurons\nform the input layer. Moreover, an extra bias feature is generally added\n(x = 1): it is typically represented using a special type of neuron called a\nbias neuron, which outputs 1 all the time. A Perceptron with two inputs\nand three outputs is represented in Figure 10-5. This Perceptron can\nclassify instances simultaneously into three different binary classes, which\nmakes it a multioutput classifier.\n0\n0\n1\n2\n7\n0\nFigure 10-5. Architecture of a Perceptron with two input neurons, one bias neuron, and three\noutput neurons\nThanks to the magic of linear algebra, Equation 10-2 makes it possible to\nefficiently compute the outputs of a layer of artificial neurons for several\ninstances at once.\nEquation 10-2. Computing the outputs of a fully connected layer\nhW,b (X) = ϕ (XW + b)\nIn this equation:\nAs always, X represents the matrix of input features. It has one\nrow per instance and one column per feature.\nThe weight matrix W contains all the connection weights except\nfor the ones from the bias neuron. It has one row per input neuron\nand one column per artificial neuron in the layer.\nThe bias vector b contains all the connection weights between the\nbias neuron and the artificial neurons. It has one bias term per\nartificial neuron.\nThe function ϕ is called the activation function: when the\nartificial neurons are TLUs, it is a step function (but we will\ndiscuss other activation functions shortly).\nSo, how is a Perceptron trained? The Perceptron training algorithm\nproposed by Rosenblatt was largely inspired by Hebb’s rule. In his 1949\nbook The Organization of Behavior (Wiley), Donald Hebb suggested that\nwhen a biological neuron triggers another neuron often, the connection\nbetween these two neurons grows stronger. Siegrid Löwel later\nsummarized Hebb’s idea in the catchy phrase, “Cells that fire together,\nwire together”; that is, the connection weight between two neurons tends\nto increase when they fire simultaneously. This rule later became known as\nHebb’s rule (or Hebbian learning). Perceptrons are trained using a variant\nof this rule that takes into account the error made by the network when it\nmakes a prediction; the Perceptron learning rule reinforces connections\nthat help reduce the error. More specifically, the Perceptron is fed one\ntraining instance at a time, and for each instance it makes its predictions.\nFor every output neuron that produced a wrong prediction, it reinforces the\nconnection weights from the inputs that would have contributed to the\ncorrect prediction. The rule is shown in Equation 10-3.\nEquation 10-3. Perceptron learning rule (weight update)\nwi,j(next step) = wi,j + η (yj −ˆyj) xi\nIn this equation:\nw is the connection weight between the i input neuron and the\nj output neuron.\nx is the i input value of the current training instance.\nˆyj is the output of the j output neuron for the current training\ninstance.\ni, j\nth\nth\ni\nth\nth\ny is the target output of the j output neuron for the current\ntraining instance.\nη is the learning rate.\nThe decision boundary of each output neuron is linear, so Perceptrons are\nincapable of learning complex patterns (just like Logistic Regression\nclassifiers). However, if the training instances are linearly separable,\nRosenblatt demonstrated that this algorithm would converge to a\nsolution. This is called the Perceptron convergence theorem.\nScikit-Learn provides a Perceptron class that implements a single-TLU\nnetwork. It can be used pretty much as you would expect—for example, on\nthe iris dataset (introduced in Chapter 4):\nimport numpy as np \nfrom sklearn.datasets import load_iris \nfrom sklearn.linear_model import Perceptron \n \niris = load_iris() \nX = iris.data[:, (2, 3)] # petal length, petal width \ny = (iris.target == 0).astype(np.int) # Iris setosa? \n \nper_clf = Perceptron() \nper_clf.fit(X, y) \n \ny_pred = per_clf.predict([[2, 0.5]])\nYou may have noticed that the Perceptron learning algorithm strongly\nresembles Stochastic Gradient Descent. In fact, Scikit-Learn’s Perceptron\nclass is equivalent to using an SGDClassifier with the following\nhyperparameters: loss=\"perceptron\", learning_rate=\"constant\",\neta0=1 (the learning rate), and penalty=None (no regularization).\nNote that contrary to Logistic Regression classifiers, Perceptrons do not\noutput a class probability; rather, they make predictions based on a hard\nthreshold. This is one reason to prefer Logistic Regression over\nPerceptrons.\nj\nth\n8\nIn their 1969 monograph Perceptrons, Marvin Minsky and Seymour\nPapert highlighted a number of serious weaknesses of Perceptrons—in\nparticular, the fact that they are incapable of solving some trivial problems\n(e.g., the Exclusive OR (XOR) classification problem; see the left side of\nFigure 10-6). This is true of any other linear classification model (such as\nLogistic Regression classifiers), but researchers had expected much more\nfrom Perceptrons, and some were so disappointed that they dropped neural\nnetworks altogether in favor of higher-level problems such as logic,\nproblem solving, and search.\nIt turns out that some of the limitations of Perceptrons can be eliminated\nby stacking multiple Perceptrons. The resulting ANN is called a\nMultilayer Perceptron (MLP). An MLP can solve the XOR problem, as\nyou can verify by computing the output of the MLP represented on the\nright side of Figure 10-6: with inputs (0, 0) or (1, 1), the network outputs\n0, and with inputs (0, 1) or (1, 0) it outputs 1. All connections have a\nweight equal to 1, except the four connections where the weight is shown.\nTry verifying that this network indeed solves the XOR problem!\nFigure 10-6. XOR classification problem and an MLP that solves it\nThe Multilayer Perceptron and Backpropagation\nAn MLP is composed of one (passthrough) input layer, one or more layers\nof TLUs, called hidden layers, and one final layer of TLUs called the\noutput layer (see Figure 10-7). The layers close to the input layer are\nusually called the lower layers, and the ones close to the outputs are\nusually called the upper layers. Every layer except the output layer\nincludes a bias neuron and is fully connected to the next layer.\nFigure 10-7. Architecture of a Multilayer Perceptron with two inputs, one hidden layer of four\nneurons, and three output neurons (the bias neurons are shown here, but usually they are\nimplicit)\nNOTE\nThe signal flows only in one direction (from the inputs to the outputs), so this\narchitecture is an example of a feedforward neural network (FNN).\nWhen an ANN contains a deep stack of hidden layers, it is called a deep\nneural network (DNN). The field of Deep Learning studies DNNs, and\nmore generally models containing deep stacks of computations. Even so,\nmany people talk about Deep Learning whenever neural networks are\ninvolved (even shallow ones).\nFor many years researchers struggled to find a way to train MLPs, without\nsuccess. But in 1986, David Rumelhart, Geoffrey Hinton, and Ronald\nWilliams published a groundbreaking paper\n that introduced the\nbackpropagation training algorithm, which is still used today. In short, it\nis Gradient Descent (introduced in Chapter 4) using an efficient technique\nfor computing the gradients automatically: in just two passes through the\nnetwork (one forward, one backward), the backpropagation algorithm is\nable to compute the gradient of the network’s error with regard to every\nsingle model parameter. In other words, it can find out how each\nconnection weight and each bias term should be tweaked in order to reduce\nthe error. Once it has these gradients, it just performs a regular Gradient\nDescent step, and the whole process is repeated until the network\nconverges to the solution.\nNOTE\nAutomatically computing gradients is called automatic differentiation, or autodiff.\nThere are various autodiff techniques, with different pros and cons. The one used by\nbackpropagation is called reverse-mode autodiff. It is fast and precise, and is well\nsuited when the function to differentiate has many variables (e.g., connection\nweights) and few outputs (e.g., one loss). If you want to learn more about autodiff,\ncheck out Appendix D.\nLet’s run through this algorithm in a bit more detail:\nIt handles one mini-batch at a time (for example, containing 32\ninstances each), and it goes through the full training set multiple\ntimes. Each pass is called an epoch.\n9\n10\n11\nEach mini-batch is passed to the network’s input layer, which\nsends it to the first hidden layer. The algorithm then computes the\noutput of all the neurons in this layer (for every instance in the\nmini-batch). The result is passed on to the next layer, its output is\ncomputed and passed to the next layer, and so on until we get the\noutput of the last layer, the output layer. This is the forward pass:\nit is exactly like making predictions, except all intermediate\nresults are preserved since they are needed for the backward pass.\nNext, the algorithm measures the network’s output error (i.e., it\nuses a loss function that compares the desired output and the\nactual output of the network, and returns some measure of the\nerror).\nThen it computes how much each output connection contributed\nto the error. This is done analytically by applying the chain rule\n(perhaps the most fundamental rule in calculus), which makes this\nstep fast and precise.\nThe algorithm then measures how much of these error\ncontributions came from each connection in the layer below, again\nusing the chain rule, working backward until the algorithm\nreaches the input layer. As explained earlier, this reverse pass\nefficiently measures the error gradient across all the connection\nweights in the network by propagating the error gradient\nbackward through the network (hence the name of the algorithm).\nFinally, the algorithm performs a Gradient Descent step to tweak\nall the connection weights in the network, using the error\ngradients it just computed.\nThis algorithm is so important that it’s worth summarizing it again: for\neach training instance, the backpropagation algorithm first makes a\nprediction (forward pass) and measures the error, then goes through each\nlayer in reverse to measure the error contribution from each connection\n(reverse pass), and finally tweaks the connection weights to reduce the\nerror (Gradient Descent step).\nWARNING\nIt is important to initialize all the hidden layers’ connection weights randomly, or\nelse training will fail. For example, if you initialize all weights and biases to zero,\nthen all neurons in a given layer will be perfectly identical, and thus\nbackpropagation will affect them in exactly the same way, so they will remain\nidentical. In other words, despite having hundreds of neurons per layer, your model\nwill act as if it had only one neuron per layer: it won’t be too smart. If instead you\nrandomly initialize the weights, you break the symmetry and allow backpropagation\nto train a diverse team of neurons.\nIn order for this algorithm to work properly, its authors made a key change\nto the MLP’s architecture: they replaced the step function with the logistic\n(sigmoid) function, σ(z) = 1 / (1 + exp(–z)). This was essential because the\nstep function contains only flat segments, so there is no gradient to work\nwith (Gradient Descent cannot move on a flat surface), while the logistic\nfunction has a well-defined nonzero derivative everywhere, allowing\nGradient Descent to make some progress at every step. In fact, the\nbackpropagation algorithm works well with many other activation\nfunctions, not just the logistic function. Here are two other popular\nchoices:\nThe hyperbolic tangent function: tanh(z) = 2σ(2z) – 1\nJust like the logistic function, this activation function is S-shaped,\ncontinuous, and differentiable, but its output value ranges from –1 to 1\n(instead of 0 to 1 in the case of the logistic function). That range tends\nto make each layer’s output more or less centered around 0 at the\nbeginning of training, which often helps speed up convergence.\nThe Rectified Linear Unit function: ReLU(z) = max(0, z)\nThe ReLU function is continuous but unfortunately not differentiable\nat z = 0 (the slope changes abruptly, which can make Gradient Descent\nbounce around), and its derivative is 0 for z < 0. In practice, however,\nit works very well and has the advantage of being fast to compute, so it\nhas become the default.\n Most importantly, the fact that it does not\nhave a maximum output value helps reduce some issues during\nGradient Descent (we will come back to this in Chapter 11).\nThese popular activation functions and their derivatives are represented in\nFigure 10-8. But wait! Why do we need activation functions in the first\nplace? Well, if you chain several linear transformations, all you get is a\nlinear transformation. For example, if f(x) = 2x + 3 and g(x) = 5x – 1, then\nchaining these two linear functions gives you another linear function:\nf(g(x)) = 2(5x – 1) + 3 = 10x + 1. So if you don’t have some nonlinearity\nbetween layers, then even a deep stack of layers is equivalent to a single\nlayer, and you can’t solve very complex problems with that. Conversely, a\nlarge enough DNN with nonlinear activations can theoretically\napproximate any continuous function.\nFigure 10-8. Activation functions and their derivatives\nOK! You know where neural nets came from, what their architecture is,\nand how to compute their outputs. You’ve also learned about the\nbackpropagation algorithm. But what exactly can you do with them?\nRegression MLPs\nFirst, MLPs can be used for regression tasks. If you want to predict a\nsingle value (e.g., the price of a house, given many of its features), then\nyou just need a single output neuron: its output is the predicted value. For\n12\nmultivariate regression (i.e., to predict multiple values at once), you need\none output neuron per output dimension. For example, to locate the center\nof an object in an image, you need to predict 2D coordinates, so you need\ntwo output neurons. If you also want to place a bounding box around the\nobject, then you need two more numbers: the width and the height of the\nobject. So, you end up with four output neurons.\nIn general, when building an MLP for regression, you do not want to use\nany activation function for the output neurons, so they are free to output\nany range of values. If you want to guarantee that the output will always\nbe positive, then you can use the ReLU activation function in the output\nlayer. Alternatively, you can use the softplus activation function, which is a\nsmooth variant of ReLU: softplus(z) = log(1 + exp(z)). It is close to 0\nwhen z is negative, and close to z when z is positive. Finally, if you want to\nguarantee that the predictions will fall within a given range of values, then\nyou can use the logistic function or the hyperbolic tangent, and then scale\nthe labels to the appropriate range: 0 to 1 for the logistic function and –1\nto 1 for the hyperbolic tangent.\nThe loss function to use during training is typically the mean squared\nerror, but if you have a lot of outliers in the training set, you may prefer to\nuse the mean absolute error instead. Alternatively, you can use the Huber\nloss, which is a combination of both.\nTIP\nThe Huber loss is quadratic when the error is smaller than a threshold δ (typically 1)\nbut linear when the error is larger than δ. The linear part makes it less sensitive to\noutliers than the mean squared error, and the quadratic part allows it to converge\nfaster and be more precise than the mean absolute error.\nTable 10-1 summarizes the typical architecture of a regression MLP.\nTable 10-1. Typical regression MLP architecture\nHyperparameter\nTypical value\n# input neurons\nOne per input feature (e.g., 28 x 28 = 784 for MNIST)\n# hidden layers\nDepends on the problem, but typically 1 to 5\n# neurons per hidden\nlayer\nDepends on the problem, but typically 10 to 100\n# output neurons\n1 per prediction dimension\nHidden activation\nReLU (or SELU, see Chapter 11)\nOutput activation\nNone, or ReLU/softplus (if positive outputs) or logistic/tanh (if\nbounded outputs)\nLoss function\nMSE or MAE/Huber (if outliers)\nClassification MLPs\nMLPs can also be used for classification tasks. For a binary classification\nproblem, you just need a single output neuron using the logistic activation\nfunction: the output will be a number between 0 and 1, which you can\ninterpret as the estimated probability of the positive class. The estimated\nprobability of the negative class is equal to one minus that number.\nMLPs can also easily handle multilabel binary classification tasks (see\nChapter 3). For example, you could have an email classification system\nthat predicts whether each incoming email is ham or spam, and\nsimultaneously predicts whether it is an urgent or nonurgent email. In this\ncase, you would need two output neurons, both using the logistic\nactivation function: the first would output the probability that the email is\nspam, and the second would output the probability that it is urgent. More\ngenerally, you would dedicate one output neuron for each positive class.\nNote that the output probabilities do not necessarily add up to 1. This lets\nthe model output any combination of labels: you can have nonurgent ham,\nurgent ham, nonurgent spam, and perhaps even urgent spam (although that\nwould probably be an error).\nIf each instance can belong only to a single class, out of three or more\npossible classes (e.g., classes 0 through 9 for digit image classification),\nthen you need to have one output neuron per class, and you should use the\nsoftmax activation function for the whole output layer (see Figure 10-9).\nThe softmax function (introduced in Chapter 4) will ensure that all the\nestimated probabilities are between 0 and 1 and that they add up to 1\n(which is required if the classes are exclusive). This is called multiclass\nclassification.\nFigure 10-9. A modern MLP (including ReLU and softmax) for classification\nRegarding the loss function, since we are predicting probability\ndistributions, the cross-entropy loss (also called the log loss, see\nChapter 4) is generally a good choice.\nTable 10-2 summarizes the typical architecture of a classification MLP.\nTable 10-2. Typical classification MLP architecture\nHyperparameter\nBinary\nclassification\nMultilabel binary\nclassification\nMulticlass\nclassification\nInput and hidden\nlayers\nSame as\nregression\nSame as regression\nSame as regression\n# output neurons\n1\n1 per label\n1 per class\nOutput layer\nactivation\nLogistic\nLogistic\nSoftmax\nLoss function\nCross entropy\nCross entropy\nCross entropy\nTIP\nBefore we go on, I recommend you go through exercise 1 at the end of this chapter.\nYou will play with various neural network architectures and visualize their outputs\nusing the TensorFlow Playground. This will be very useful to better understand\nMLPs, including the effects of all the hyperparameters (number of layers and\nneurons, activation functions, and more).\nNow you have all the concepts you need to start implementing MLPs with\nKeras!\nImplementing MLPs with Keras\nKeras is a high-level Deep Learning API that allows you to easily build,\ntrain, evaluate, and execute all sorts of neural networks. Its documentation\n(or specification) is available at https://keras.io/. The reference\nimplementation, also called Keras, was developed by François Chollet as\npart of a research project\n and was released as an open source project in\nMarch 2015. It quickly gained popularity, owing to its ease of use,\nflexibility, and beautiful design. To perform the heavy computations\nrequired by neural networks, this reference implementation relies on a\ncomputation backend. At present, you can choose from three popular open\n13\nsource Deep Learning libraries: TensorFlow, Microsoft Cognitive Toolkit\n(CNTK), and Theano. Therefore, to avoid any confusion, we will refer to\nthis reference implementation as multibackend Keras.\nSince late 2016, other implementations have been released. You can now\nrun Keras on Apache MXNet, Apple’s Core ML, JavaScript or TypeScript\n(to run Keras code in a web browser), and PlaidML (which can run on all\nsorts of GPU devices, not just Nvidia). Moreover, TensorFlow itself now\ncomes bundled with its own Keras implementation, tf.keras. It only\nsupports TensorFlow as the backend, but it has the advantage of offering\nsome very useful extra features (see Figure 10-10): for example, it\nsupports TensorFlow’s Data API, which makes it easy to load and\npreprocess data efficiently. For this reason, we will use tf.keras in this\nbook. However, in this chapter we will not use any of the TensorFlow-\nspecific features, so the code should run fine on other Keras\nimplementations as well (at least in Python), with only minor\nmodifications, such as changing the imports.\nFigure 10-10. Two implementations of the Keras API: multibackend Keras (left) and tf.keras\n(right)\nThe most popular Deep Learning library, after Keras and TensorFlow, is\nFacebook’s PyTorch library. The good news is that its API is quite similar\nto Keras’s (in part because both APIs were inspired by Scikit-Learn and\nChainer), so once you know Keras, it is not difficult to switch to PyTorch,\nif you ever want to. PyTorch’s popularity grew exponentially in 2018,\nlargely thanks to its simplicity and excellent documentation, which were\nnot TensorFlow 1.x’s main strengths. However, TensorFlow 2 is arguably\njust as simple as PyTorch, as it has adopted Keras as its official high-level\nAPI and its developers have greatly simplified and cleaned up the rest of\nthe API. The documentation has also been completely reorganized, and it\nis much easier to find what you need now. Similarly, PyTorch’s main\nweaknesses (e.g., limited portability and no computation graph analysis)\nhave been largely addressed in PyTorch 1.0. Healthy competition is\nbeneficial to everyone.\nAll right, it’s time to code! As tf.keras is bundled with TensorFlow, let’s\nstart by installing TensorFlow.\nInstalling TensorFlow 2\nAssuming you installed Jupyter and Scikit-Learn by following the\ninstallation instructions in Chapter 2, use pip to install TensorFlow. If you\ncreated an isolated environment using virtualenv, you first need to activate\nit:\n$ cd $ML_PATH # Your ML working directory (e.g., \n$HOME/ml) \n$ source my_env/bin/activate # on Linux or macOS \n$ .\\my_env\\Scripts\\activate # on Windows \nNext, install TensorFlow 2 (if you are not using a virtualenv, you will need\nadministrator rights, or to add the --user option):\n$ python3 -m pip install --upgrade tensorflow \nNOTE\nFor GPU support, at the time of this writing you need to install tensorflow-gpu\ninstead of tensorflow, but the TensorFlow team is working on having a single\nlibrary that will support both CPU-only and GPU-equipped systems. You will still\nneed to install extra libraries for GPU support (see https://tensorflow.org/install for\nmore details). We will look at GPUs in more depth in Chapter 19.\nTo test your installation, open a Python shell or a Jupyter notebook, then\nimport TensorFlow and tf.keras and print their versions:\n>>> import tensorflow as tf \n>>> from tensorflow import keras \n>>> tf.__version__ \n'2.0.0' \n>>> keras.__version__ \n'2.2.4-tf'\nThe second version is the version of the Keras API implemented by\ntf.keras. Note that it ends with -tf, highlighting the fact that tf.keras\nimplements the Keras API, plus some extra TensorFlow-specific features.\nNow let’s use tf.keras! We’ll start by building a simple image classifier.\nBuilding an Image Classifier Using the Sequential API\nFirst, we need to load a dataset. In this chapter we will tackle Fashion\nMNIST, which is a drop-in replacement of MNIST (introduced in\nChapter 3). It has the exact same format as MNIST (70,000 grayscale\nimages of 28 × 28 pixels each, with 10 classes), but the images represent\nfashion items rather than handwritten digits, so each class is more diverse,\nand the problem turns out to be significantly more challenging than\nMNIST. For example, a simple linear model reaches about 92% accuracy\non MNIST, but only about 83% on Fashion MNIST.\nUsing Keras to load the dataset\nKeras provides some utility functions to fetch and load common datasets,\nincluding MNIST, Fashion MNIST, and the California housing dataset we\nused in Chapter 2. Let’s load Fashion MNIST:\nfashion_mnist = keras.datasets.fashion_mnist \n(X_train_full, y_train_full), (X_test, y_test) = \nfashion_mnist.load_data()\nWhen loading MNIST or Fashion MNIST using Keras rather than Scikit-\nLearn, one important difference is that every image is represented as a 28\n× 28 array rather than a 1D array of size 784. Moreover, the pixel\nintensities are represented as integers (from 0 to 255) rather than floats\n(from 0.0 to 255.0). Let’s take a look at the shape and data type of the\ntraining set:\n>>> X_train_full.shape \n(60000, 28, 28) \n>>> X_train_full.dtype \ndtype('uint8')\nNote that the dataset is already split into a training set and a test set, but\nthere is no validation set, so we’ll create one now. Additionally, since we\nare going to train the neural network using Gradient Descent, we must\nscale the input features. For simplicity, we’ll scale the pixel intensities\ndown to the 0–1 range by dividing them by 255.0 (this also converts them\nto floats):\nX_valid, X_train = X_train_full[:5000] / 255.0, X_train_full[5000:] / \n255.0 \ny_valid, y_train = y_train_full[:5000], y_train_full[5000:]\nWith MNIST, when the label is equal to 5, it means that the image\nrepresents the handwritten digit 5. Easy. For Fashion MNIST, however, we\nneed the list of class names to know what we are dealing with:\nclass_names = [\"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\", \n \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]\nFor example, the first image in the training set represents a coat:\n>>> class_names[y_train[0]] \n'Coat'\nFigure 10-11 shows some samples from the Fashion MNIST dataset.\nFigure 10-11. Samples from Fashion MNIST\nCreating the model using the Sequential API\nNow let’s build the neural network! Here is a classification MLP with two\nhidden layers:\nmodel = keras.models.Sequential() \nmodel.add(keras.layers.Flatten(input_shape=[28, 28])) \nmodel.add(keras.layers.Dense(300, activation=\"relu\")) \nmodel.add(keras.layers.Dense(100, activation=\"relu\")) \nmodel.add(keras.layers.Dense(10, activation=\"softmax\"))\nLet’s go through this code line by line:\nThe first line creates a Sequential model. This is the simplest\nkind of Keras model for neural networks that are just composed of\na single stack of layers connected sequentially. This is called the\nSequential API.\nNext, we build the first layer and add it to the model. It is a\nFlatten layer whose role is to convert each input image into a 1D\narray: if it receives input data X, it computes X.reshape(-1, 1).\nThis layer does not have any parameters; it is just there to do\nsome simple preprocessing. Since it is the first layer in the model,\nyou should specify the input_shape, which doesn’t include the\nbatch size, only the shape of the instances. Alternatively, you\ncould add a keras.layers.InputLayer as the first layer, setting\ninput_shape=[28,28].\nNext we add a Dense hidden layer with 300 neurons. It will use\nthe ReLU activation function. Each Dense layer manages its own\nweight matrix, containing all the connection weights between the\nneurons and their inputs. It also manages a vector of bias terms\n(one per neuron). When it receives some input data, it computes\nEquation 10-2.\nThen we add a second Dense hidden layer with 100 neurons, also\nusing the ReLU activation function.\nFinally, we add a Dense output layer with 10 neurons (one per\nclass), using the softmax activation function (because the classes\nare exclusive).\nTIP\nSpecifying activation=\"relu\" is equivalent to specifying\nactivation=keras.activations.relu. Other activation functions are available\nin the keras.activations package, we will use many of them in this book. See\nhttps://keras.io/activations/ for the full list.\nInstead of adding the layers one by one as we just did, you can pass a list\nof layers when creating the Sequential model:\nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dense(300, activation=\"relu\"), \n keras.layers.Dense(100, activation=\"relu\"), \n keras.layers.Dense(10, activation=\"softmax\") \n])\nUSING CODE EXAMPLES FROM KERAS.IO\nCode examples documented on keras.io will work fine with tf.keras,\nbut you need to change the imports. For example, consider this\nkeras.io code:\nfrom keras.layers import Dense \noutput_layer = Dense(10)\nYou must change the imports like this:\nfrom tensorflow.keras.layers import Dense \noutput_layer = Dense(10)\nOr simply use full paths, if you prefer:\nfrom tensorflow import keras \noutput_layer = keras.layers.Dense(10)\nThis approach is more verbose, but I use it in this book so you can\neasily see which packages to use, and to avoid confusion between\nstandard classes and custom classes. In production code, I prefer the\nprevious approach. Many people also use from tensorflow.keras\nimport layers followed by layers.Dense(10).\nThe model’s summary() method displays all the model’s layers,\nincluding each layer’s name (which is automatically generated unless you\nset it when creating the layer), its output shape (None means the batch size\ncan be anything), and its number of parameters. The summary ends with\nthe total number of parameters, including trainable and non-trainable\nparameters. Here we only have trainable parameters (we will see examples\nof non-trainable parameters in Chapter 11):\n>>> model.summary() \nModel: \"sequential\" \n_________________________________________________________________ \n14\nLayer (type) Output Shape Param # \n================================================================= \nflatten (Flatten) (None, 784) 0 \n_________________________________________________________________ \ndense (Dense) (None, 300) 235500 \n_________________________________________________________________ \ndense_1 (Dense) (None, 100) 30100 \n_________________________________________________________________ \ndense_2 (Dense) (None, 10) 1010 \n================================================================= \nTotal params: 266,610 \nTrainable params: 266,610 \nNon-trainable params: 0 \n_________________________________________________________________\nNote that Dense layers often have a lot of parameters. For example, the\nfirst hidden layer has 784 × 300 connection weights, plus 300 bias terms,\nwhich adds up to 235,500 parameters! This gives the model quite a lot of\nflexibility to fit the training data, but it also means that the model runs the\nrisk of overfitting, especially when you do not have a lot of training data.\nWe will come back to this later.\nYou can easily get a model’s list of layers, to fetch a layer by its index, or\nyou can fetch it by name:\n>>> model.layers \n[, \n , \n , \n ] \n>>> hidden1 = model.layers[1] \n>>> hidden1.name \n'dense' \n>>> model.get_layer('dense') is hidden1 \nTrue\nAll the parameters of a layer can be accessed using its get_weights() and\nset_weights() methods. For a Dense layer, this includes both the\nconnection weights and the bias terms:\n>>> weights, biases = hidden1.get_weights() \n>>> weights \narray([[ 0.02448617, -0.00877795, -0.02189048, ..., -0.02766046, \n 0.03859074, -0.06889391], \n ..., \n [-0.06022581, 0.01577859, -0.02585464, ..., -0.00527829, \n 0.00272203, -0.06793761]], dtype=float32) \n>>> weights.shape \n(784, 300) \n>>> biases \narray([0., 0., 0., 0., 0., 0., 0., 0., 0., ..., 0., 0., 0.], \ndtype=float32) \n>>> biases.shape \n(300,)\nNotice that the Dense layer initialized the connection weights randomly\n(which is needed to break symmetry, as we discussed earlier), and the\nbiases were initialized to zeros, which is fine. If you ever want to use a\ndifferent initialization method, you can set kernel_initializer (kernel\nis another name for the matrix of connection weights) or\nbias_initializer when creating the layer. We will discuss initializers\nfurther in Chapter 11, but if you want the full list, see\nhttps://keras.io/initializers/.\nNOTE\nThe shape of the weight matrix depends on the number of inputs. This is why it is\nrecommended to specify the input_shape when creating the first layer in a\nSequential model. However, if you do not specify the input shape, it’s OK: Keras\nwill simply wait until it knows the input shape before it actually builds the model.\nThis will happen either when you feed it actual data (e.g., during training), or when\nyou call its build() method. Until the model is really built, the layers will not have\nany weights, and you will not be able to do certain things (such as print the model\nsummary or save the model). So, if you know the input shape when creating the\nmodel, it is best to specify it.\nCompiling the model\nAfter a model is created, you must call its compile() method to specify\nthe loss function and the optimizer to use. Optionally, you can specify a\nlist of extra metrics to compute during training and evaluation:\nmodel.compile(loss=\"sparse_categorical_crossentropy\", \n optimizer=\"sgd\", \n metrics=[\"accuracy\"])\nNOTE\nUsing loss=\"sparse_categorical_crossentropy\" is equivalent to using\nloss=keras.losses.sparse_categorical_crossentropy. Similarly, specifying\noptimizer=\"sgd\" is equivalent to specifying\noptimizer=keras.optimizers.SGD(), and metrics=[\"accuracy\"] is\nequivalent to metrics=[keras.metrics.sparse_categorical_accuracy]\n(when using this loss). We will use many other losses, optimizers, and metrics in this\nbook; for the full lists, see https://keras.io/losses, https://keras.io/optimizers, and\nhttps://keras.io/metrics.\nThis code requires some explanation. First, we use the\n\"sparse_categorical_crossentropy\" loss because we have sparse\nlabels (i.e., for each instance, there is just a target class index, from 0 to 9\nin this case), and the classes are exclusive. If instead we had one target\nprobability per class for each instance (such as one-hot vectors, e.g. [0.,\n0., 0., 1., 0., 0., 0., 0., 0., 0.] to represent class 3), then we\nwould need to use the \"categorical_crossentropy\" loss instead. If we\nwere doing binary classification (with one or more binary labels), then we\nwould use the \"sigmoid\" (i.e., logistic) activation function in the output\nlayer instead of the \"softmax\" activation function, and we would use the\n\"binary_crossentropy\" loss.\nTIP\nIf you want to convert sparse labels (i.e., class indices) to one-hot vector labels, use\nthe keras.utils.to_categorical() function. To go the other way round, use the\nnp.argmax() function with axis=1.\nRegarding the optimizer, \"sgd\" means that we will train the model using\nsimple Stochastic Gradient Descent. In other words, Keras will perform\nthe backpropagation algorithm described earlier (i.e., reverse-mode\nautodiff plus Gradient Descent). We will discuss more efficient optimizers\nin Chapter 11 (they improve the Gradient Descent part, not the autodiff).\nNOTE\nWhen using the SGD optimizer, it is important to tune the learning rate. So, you will\ngenerally want to use optimizer=keras.optimizers.SGD(lr=???) to set the\nlearning rate, rather than optimizer=\"sgd\", which defaults to lr=0.01.\nFinally, since this is a classifier, it’s useful to measure its \"accuracy\"\nduring training and evaluation.\nTraining and evaluating the model\nNow the model is ready to be trained. For this we simply need to call its\nfit() method:\n>>> history = model.fit(X_train, y_train, epochs=30, \n... validation_data=(X_valid, y_valid)) \n... \nTrain on 55000 samples, validate on 5000 samples \nEpoch 1/30 \n55000/55000 [======] - 3s 49us/sample - loss: 0.7218 - accuracy: \n0.7660 \n - val_loss: 0.4973 - val_accuracy: \n0.8366 \nEpoch 2/30 \n55000/55000 [======] - 2s 45us/sample - loss: 0.4840 - accuracy: \n0.8327 \n - val_loss: 0.4456 - val_accuracy: \n0.8480 \n[...] \nEpoch 30/30 \n55000/55000 [======] - 3s 53us/sample - loss: 0.2252 - accuracy: \n0.9192 \n - val_loss: 0.2999 - val_accuracy: \n0.8926\nWe pass it the input features (X_train) and the target classes (y_train),\nas well as the number of epochs to train (or else it would default to just 1,\nwhich would definitely not be enough to converge to a good solution). We\nalso pass a validation set (this is optional). Keras will measure the loss and\nthe extra metrics on this set at the end of each epoch, which is very useful\nto see how well the model really performs. If the performance on the\ntraining set is much better than on the validation set, your model is\nprobably overfitting the training set (or there is a bug, such as a data\nmismatch between the training set and the validation set).\nAnd that’s it! The neural network is trained.\n At each epoch during\ntraining, Keras displays the number of instances processed so far (along\nwith a progress bar), the mean training time per sample, and the loss and\naccuracy (or any other extra metrics you asked for) on both the training set\nand the validation set. You can see that the training loss went down, which\nis a good sign, and the validation accuracy reached 89.26% after 30\nepochs. That’s not too far from the training accuracy, so there does not\nseem to be much overfitting going on.\nTIP\nInstead of passing a validation set using the validation_data argument, you could\nset validation_split to the ratio of the training set that you want Keras to use for\nvalidation. For example, validation_split=0.1 tells Keras to use the last 10% of\nthe data (before shuffling) for validation.\nIf the training set was very skewed, with some classes being\noverrepresented and others underrepresented, it would be useful to set the\nclass_weight argument when calling the fit() method, which would\ngive a larger weight to underrepresented classes and a lower weight to\noverrepresented classes. These weights would be used by Keras when\ncomputing the loss. If you need per-instance weights, set the\n15\nsample_weight argument (it supersedes class_weight). Per-instance\nweights could be useful if some instances were labeled by experts while\nothers were labeled using a crowdsourcing platform: you might want to\ngive more weight to the former. You can also provide sample weights (but\nnot class weights) for the validation set by adding them as a third item in\nthe validation_data tuple.\nThe fit() method returns a History object containing the training\nparameters (history.params), the list of epochs it went through\n(history.epoch), and most importantly a dictionary (history.history)\ncontaining the loss and extra metrics it measured at the end of each epoch\non the training set and on the validation set (if any). If you use this\ndictionary to create a pandas DataFrame and call its plot() method, you\nget the learning curves shown in Figure 10-12:\nimport pandas as pd \nimport matplotlib.pyplot as plt \n \npd.DataFrame(history.history).plot(figsize=(8, 5)) \nplt.grid(True) \nplt.gca().set_ylim(0, 1) # set the vertical range to [0-1] \nplt.show()\nFigure 10-12. Learning curves: the mean training loss and accuracy measured over each epoch,\nand the mean validation loss and accuracy measured at the end of each epoch\nYou can see that both the training accuracy and the validation accuracy\nsteadily increase during training, while the training loss and the validation\nloss decrease. Good! Moreover, the validation curves are close to the\ntraining curves, which means that there is not too much overfitting. In this\nparticular case, the model looks like it performed better on the validation\nset than on the training set at the beginning of training. But that’s not the\ncase: indeed, the validation error is computed at the end of each epoch,\nwhile the training error is computed using a running mean during each\nepoch. So the training curve should be shifted by half an epoch to the left.\nIf you do that, you will see that the training and validation curves overlap\nalmost perfectly at the beginning of training.\nTIP\nWhen plotting the training curve, it should be shifted by half an epoch to the left.\nThe training set performance ends up beating the validation performance,\nas is generally the case when you train for long enough. You can tell that\nthe model has not quite converged yet, as the validation loss is still going\ndown, so you should probably continue training. It’s as simple as calling\nthe fit() method again, since Keras just continues training where it left\noff (you should be able to reach close to 89% validation accuracy).\nIf you are not satisfied with the performance of your model, you should go\nback and tune the hyperparameters. The first one to check is the learning\nrate. If that doesn’t help, try another optimizer (and always retune the\nlearning rate after changing any hyperparameter). If the performance is\nstill not great, then try tuning model hyperparameters such as the number\nof layers, the number of neurons per layer, and the types of activation\nfunctions to use for each hidden layer. You can also try tuning other\nhyperparameters, such as the batch size (it can be set in the fit() method\nusing the batch_size argument, which defaults to 32). We will get back to\nhyperparameter tuning at the end of this chapter. Once you are satisfied\nwith your model’s validation accuracy, you should evaluate it on the test\nset to estimate the generalization error before you deploy the model to\nproduction. You can easily do this using the evaluate() method (it also\nsupports several other arguments, such as batch_size and\nsample_weight; please check the documentation for more details):\n>>> model.evaluate(X_test, y_test) \n10000/10000 [==========] - 0s 29us/sample - loss: 0.3340 - accuracy: \n0.8851 \n[0.3339798209667206, 0.8851]\nAs we saw in Chapter 2, it is common to get slightly lower performance on\nthe test set than on the validation set, because the hyperparameters are\ntuned on the validation set, not the test set (however, in this example, we\ndid not do any hyperparameter tuning, so the lower accuracy is just bad\nluck). Remember to resist the temptation to tweak the hyperparameters on\nthe test set, or else your estimate of the generalization error will be too\noptimistic.\nUsing the model to make predictions\nNext, we can use the model’s predict() method to make predictions on\nnew instances. Since we don’t have actual new instances, we will just use\nthe first three instances of the test set:\n>>> X_new = X_test[:3] \n>>> y_proba = model.predict(X_new) \n>>> y_proba.round(2) \narray([[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.01, 0. , 0.96], \n [0. , 0. , 0.98, 0. , 0.02, 0. , 0. , 0. , 0. , 0. ], \n [0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]], \n dtype=float32)\nAs you can see, for each instance the model estimates one probability per\nclass, from class 0 to class 9. For example, for the first image it estimates\nthat the probability of class 9 (ankle boot) is 96%, the probability of class\n5 (sandal) is 3%, the probability of class 7 (sneaker) is 1%, and the\nprobabilities of the other classes are negligible. In other words, it\n“believes” the first image is footwear, most likely ankle boots but possibly\nsandals or sneakers. If you only care about the class with the highest\nestimated probability (even if that probability is quite low), then you can\nuse the predict_classes() method instead:\n>>> y_pred = model.predict_classes(X_new) \n>>> y_pred \narray([9, 2, 1]) \n>>> np.array(class_names)[y_pred] \narray(['Ankle boot', 'Pullover', 'Trouser'], dtype='>> y_new = y_test[:3] \n>>> y_new \narray([9, 2, 1])\nFigure 10-13. Correctly classified Fashion MNIST images\nNow you know how to use the Sequential API to build, train, evaluate, and\nuse a classification MLP. But what about regression?\nBuilding a Regression MLP Using the Sequential API\nLet’s switch to the California housing problem and tackle it using a\nregression neural network. For simplicity, we will use Scikit-Learn’s\nfetch_california_housing() function to load the data. This dataset is\nsimpler than the one we used in Chapter 2, since it contains only\nnumerical features (there is no ocean_proximity feature), and there is no\nmissing value. After loading the data, we split it into a training set, a\nvalidation set, and a test set, and we scale all the features:\nfrom sklearn.datasets import fetch_california_housing \nfrom sklearn.model_selection import train_test_split \nfrom sklearn.preprocessing import StandardScaler \n \nhousing = fetch_california_housing() \n \nX_train_full, X_test, y_train_full, y_test = train_test_split( \n housing.data, housing.target) \nX_train, X_valid, y_train, y_valid = train_test_split( \n X_train_full, y_train_full) \n \nscaler = StandardScaler() \nX_train = scaler.fit_transform(X_train) \nX_valid = scaler.transform(X_valid) \nX_test = scaler.transform(X_test)\nUsing the Sequential API to build, train, evaluate, and use a regression\nMLP to make predictions is quite similar to what we did for classification.\nThe main differences are the fact that the output layer has a single neuron\n(since we only want to predict a single value) and uses no activation\nfunction, and the loss function is the mean squared error. Since the dataset\nis quite noisy, we just use a single hidden layer with fewer neurons than\nbefore, to avoid overfitting:\nmodel = keras.models.Sequential([ \n keras.layers.Dense(30, activation=\"relu\", \ninput_shape=X_train.shape[1:]), \n keras.layers.Dense(1) \n]) \nmodel.compile(loss=\"mean_squared_error\", optimizer=\"sgd\") \nhistory = model.fit(X_train, y_train, epochs=20, \n validation_data=(X_valid, y_valid)) \nmse_test = model.evaluate(X_test, y_test) \nX_new = X_test[:3] # pretend these are new instances \ny_pred = model.predict(X_new)\nAs you can see, the Sequential API is quite easy to use. However, although\nSequential models are extremely common, it is sometimes useful to\nbuild neural networks with more complex topologies, or with multiple\ninputs or outputs. For this purpose, Keras offers the Functional API.\nBuilding Complex Models Using the Functional API\nOne example of a nonsequential neural network is a Wide & Deep neural\nnetwork. This neural network architecture was introduced in a 2016 paper\nby Heng-Tze Cheng et al.\n It connects all or part of the inputs directly to\nthe output layer, as shown in Figure 10-14. This architecture makes it\npossible for the neural network to learn both deep patterns (using the deep\npath) and simple rules (through the short path).\n In contrast, a regular\nMLP forces all the data to flow through the full stack of layers; thus,\nsimple patterns in the data may end up being distorted by this sequence of\ntransformations.\n16\n17\nFigure 10-14. Wide & Deep neural network\nLet’s build such a neural network to tackle the California housing\nproblem:\ninput_ = keras.layers.Input(shape=X_train.shape[1:]) \nhidden1 = keras.layers.Dense(30, activation=\"relu\")(input_) \nhidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1) \nconcat = keras.layers.Concatenate()([input_, hidden2]) \noutput = keras.layers.Dense(1)(concat) \nmodel = keras.Model(inputs=[input_], outputs=[output])\nLet’s go through each line of this code:\nFirst, we need to create an Input object.\n This is a specification\nof the kind of input the model will get, including its shape and\ndtype. A model may actually have multiple inputs, as we will see\nshortly.\nNext, we create a Dense layer with 30 neurons, using the ReLU\nactivation function. As soon as it is created, notice that we call it\nlike a function, passing it the input. This is why this is called the\nFunctional API. Note that we are just telling Keras how it should\nconnect the layers together; no actual data is being processed yet.\nWe then create a second hidden layer, and again we use it as a\nfunction. Note that we pass it the output of the first hidden layer.\nNext, we create a Concatenate layer, and once again we\nimmediately use it like a function, to concatenate the input and\nthe output of the second hidden layer. You may prefer the\nkeras.layers.concatenate() function, which creates a\nConcatenate layer and immediately calls it with the given inputs.\nThen we create the output layer, with a single neuron and no\nactivation function, and we call it like a function, passing it the\nresult of the concatenation.\nLastly, we create a Keras Model, specifying which inputs and\noutputs to use.\nOnce you have built the Keras model, everything is exactly like earlier, so\nthere’s no need to repeat it here: you must compile the model, train it,\nevaluate it, and use it to make predictions.\nBut what if you want to send a subset of the features through the wide path\nand a different subset (possibly overlapping) through the deep path (see\nFigure 10-15)? In this case, one solution is to use multiple inputs. For\nexample, suppose we want to send five features through the wide path\n(features 0 to 4), and six features through the deep path (features 2 to 7):\n18\ninput_A = keras.layers.Input(shape=[5], name=\"wide_input\") \ninput_B = keras.layers.Input(shape=[6], name=\"deep_input\") \nhidden1 = keras.layers.Dense(30, activation=\"relu\")(input_B) \nhidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1) \nconcat = keras.layers.concatenate([input_A, hidden2]) \noutput = keras.layers.Dense(1, name=\"output\")(concat) \nmodel = keras.Model(inputs=[input_A, input_B], outputs=[output])\nFigure 10-15. Handling multiple inputs\nThe code is self-explanatory. You should name at least the most important\nlayers, especially when the model gets a bit complex like this. Note that\nwe specified inputs=[input_A, input_B] when creating the model. Now\nwe can compile the model as usual, but when we call the fit() method,\ninstead of passing a single input matrix X_train, we must pass a pair of\nmatrices (X_train_A, X_train_B): one per input.\n The same is true for\nX_valid, and also for X_test and X_new when you call evaluate() or\npredict():\nmodel.compile(loss=\"mse\", optimizer=keras.optimizers.SGD(lr=1e-3)) \n \nX_train_A, X_train_B = X_train[:, :5], X_train[:, 2:] \nX_valid_A, X_valid_B = X_valid[:, :5], X_valid[:, 2:] \nX_test_A, X_test_B = X_test[:, :5], X_test[:, 2:] \nX_new_A, X_new_B = X_test_A[:3], X_test_B[:3] \n \nhistory = model.fit((X_train_A, X_train_B), y_train, epochs=20, \n validation_data=((X_valid_A, X_valid_B), y_valid)) \nmse_test = model.evaluate((X_test_A, X_test_B), y_test) \ny_pred = model.predict((X_new_A, X_new_B))\nThere are many use cases in which you may want to have multiple outputs:\nThe task may demand it. For instance, you may want to locate and\nclassify the main object in a picture. This is both a regression task\n(finding the coordinates of the object’s center, as well as its width\nand height) and a classification task.\nSimilarly, you may have multiple independent tasks based on the\nsame data. Sure, you could train one neural network per task, but\nin many cases you will get better results on all tasks by training a\nsingle neural network with one output per task. This is because\nthe neural network can learn features in the data that are useful\nacross tasks. For example, you could perform multitask\nclassification on pictures of faces, using one output to classify the\nperson’s facial expression (smiling, surprised, etc.) and another\noutput to identify whether they are wearing glasses or not.\nAnother use case is as a regularization technique (i.e., a training\nconstraint whose objective is to reduce overfitting and thus\n19\nimprove the model’s ability to generalize). For example, you may\nwant to add some auxiliary outputs in a neural network\narchitecture (see Figure 10-16) to ensure that the underlying part\nof the network learns something useful on its own, without\nrelying on the rest of the network.\nFigure 10-16. Handling multiple outputs, in this example to add an auxiliary output for\nregularization\nAdding extra outputs is quite easy: just connect them to the appropriate\nlayers and add them to your model’s list of outputs. For example, the\nfollowing code builds the network represented in Figure 10-16:\n[...] # Same as above, up to the main output layer \noutput = keras.layers.Dense(1, name=\"main_output\")(concat) \naux_output = keras.layers.Dense(1, name=\"aux_output\")(hidden2) \nmodel = keras.Model(inputs=[input_A, input_B], outputs=[output, \naux_output])\nEach output will need its own loss function. Therefore, when we compile\nthe model, we should pass a list of losses\n (if we pass a single loss, Keras\n20\nwill assume that the same loss must be used for all outputs). By default,\nKeras will compute all these losses and simply add them up to get the final\nloss used for training. We care much more about the main output than\nabout the auxiliary output (as it is just used for regularization), so we want\nto give the main output’s loss a much greater weight. Fortunately, it is\npossible to set all the loss weights when compiling the model:\nmodel.compile(loss=[\"mse\", \"mse\"], loss_weights=[0.9, 0.1], \noptimizer=\"sgd\")\nNow when we train the model, we need to provide labels for each output.\nIn this example, the main output and the auxiliary output should try to\npredict the same thing, so they should use the same labels. So instead of\npassing y_train, we need to pass (y_train, y_train) (and the same\ngoes for y_valid and y_test):\nhistory = model.fit( \n [X_train_A, X_train_B], [y_train, y_train], epochs=20, \n validation_data=([X_valid_A, X_valid_B], [y_valid, y_valid]))\nWhen we evaluate the model, Keras will return the total loss, as well as all\nthe individual losses:\ntotal_loss, main_loss, aux_loss = model.evaluate( \n [X_test_A, X_test_B], [y_test, y_test])\nSimilarly, the predict() method will return predictions for each output:\ny_pred_main, y_pred_aux = model.predict([X_new_A, X_new_B])\nAs you can see, you can build any sort of architecture you want quite\neasily with the Functional API. Let’s look at one last way you can build\nKeras models.\nUsing the Subclassing API to Build Dynamic Models\nBoth the Sequential API and the Functional API are declarative: you start\nby declaring which layers you want to use and how they should be\nconnected, and only then can you start feeding the model some data for\ntraining or inference. This has many advantages: the model can easily be\nsaved, cloned, and shared; its structure can be displayed and analyzed; the\nframework can infer shapes and check types, so errors can be caught early\n(i.e., before any data ever goes through the model). It’s also fairly easy to\ndebug, since the whole model is a static graph of layers. But the flip side is\njust that: it’s static. Some models involve loops, varying shapes,\nconditional branching, and other dynamic behaviors. For such cases, or\nsimply if you prefer a more imperative programming style, the\nSubclassing API is for you.\nSimply subclass the Model class, create the layers you need in the\nconstructor, and use them to perform the computations you want in the\ncall() method. For example, creating an instance of the following\nWideAndDeepModel class gives us an equivalent model to the one we just\nbuilt with the Functional API. You can then compile it, evaluate it, and use\nit to make predictions, exactly like we just did:\nclass WideAndDeepModel(keras.Model): \n def __init__(self, units=30, activation=\"relu\", **kwargs): \n super().__init__(**kwargs) # handles standard args (e.g., name) \n self.hidden1 = keras.layers.Dense(units, activation=activation) \n self.hidden2 = keras.layers.Dense(units, activation=activation) \n self.main_output = keras.layers.Dense(1) \n self.aux_output = keras.layers.Dense(1) \n \n def call(self, inputs): \n input_A, input_B = inputs \n hidden1 = self.hidden1(input_B) \n hidden2 = self.hidden2(hidden1) \n concat = keras.layers.concatenate([input_A, hidden2]) \n main_output = self.main_output(concat) \n aux_output = self.aux_output(hidden2) \n return main_output, aux_output \n \nmodel = WideAndDeepModel()\nThis example looks very much like the Functional API, except we do not\nneed to create the inputs; we just use the input argument to the call()\nmethod, and we separate the creation of the layers\n in the constructor\nfrom their usage in the call() method. The big difference is that you can\ndo pretty much anything you want in the call() method: for loops, if\nstatements, low-level TensorFlow operations—your imagination is the\nlimit (see Chapter 12)! This makes it a great API for researchers\nexperimenting with new ideas.\nThis extra flexibility does come at a cost: your model’s architecture is\nhidden within the call() method, so Keras cannot easily inspect it; it\ncannot save or clone it; and when you call the summary() method, you\nonly get a list of layers, without any information on how they are\nconnected to each other. Moreover, Keras cannot check types and shapes\nahead of time, and it is easier to make mistakes. So unless you really need\nthat extra flexibility, you should probably stick to the Sequential API or\nthe Functional API.\nTIP\nKeras models can be used just like regular layers, so you can easily combine them to\nbuild complex architectures.\nNow that you know how to build and train neural nets using Keras, you\nwill want to save them!\nSaving and Restoring a Model\nWhen using the Sequential API or the Functional API, saving a trained\nKeras model is as simple as it gets:\nmodel = keras.layers.Sequential([...]) # or keras.Model([...]) \nmodel.compile([...]) \nmodel.fit([...]) \nmodel.save(\"my_keras_model.h5\")\n21\nKeras will use the HDF5 format to save both the model’s architecture\n(including every layer’s hyperparameters) and the values of all the model\nparameters for every layer (e.g., connection weights and biases). It also\nsaves the optimizer (including its hyperparameters and any state it may\nhave).\nYou will typically have a script that trains a model and saves it, and one or\nmore scripts (or web services) that load the model and use it to make\npredictions. Loading the model is just as easy:\nmodel = keras.models.load_model(\"my_keras_model.h5\")\nWARNING\nThis will work when using the Sequential API or the Functional API, but\nunfortunately not when using model subclassing. You can use save_weights() and\nload_weights() to at least save and restore the model parameters, but you will\nneed to save and restore everything else yourself.\nBut what if training lasts several hours? This is quite common, especially\nwhen training on large datasets. In this case, you should not only save your\nmodel at the end of training, but also save checkpoints at regular intervals\nduring training, to avoid losing everything if your computer crashes. But\nhow can you tell the fit() method to save checkpoints? Use callbacks.\nUsing Callbacks\nThe fit() method accepts a callbacks argument that lets you specify a\nlist of objects that Keras will call at the start and end of training, at the\nstart and end of each epoch, and even before and after processing each\nbatch. For example, the ModelCheckpoint callback saves checkpoints of\nyour model at regular intervals during training, by default at the end of\neach epoch:\n[...] # build and compile the model \ncheckpoint_cb = keras.callbacks.ModelCheckpoint(\"my_keras_model.h5\") \nhistory = model.fit(X_train, y_train, epochs=10, callbacks=\n[checkpoint_cb])\nMoreover, if you use a validation set during training, you can set\nsave_best_only=True when creating the ModelCheckpoint. In this case,\nit will only save your model when its performance on the validation set is\nthe best so far. This way, you do not need to worry about training for too\nlong and overfitting the training set: simply restore the last model saved\nafter training, and this will be the best model on the validation set. The\nfollowing code is a simple way to implement early stopping (introduced in\nChapter 4):\ncheckpoint_cb = keras.callbacks.ModelCheckpoint(\"my_keras_model.h5\", \n save_best_only=True) \nhistory = model.fit(X_train, y_train, epochs=10, \n validation_data=(X_valid, y_valid), \n callbacks=[checkpoint_cb]) \nmodel = keras.models.load_model(\"my_keras_model.h5\") # roll back to best \nmodel\nAnother way to implement early stopping is to simply use the\nEarlyStopping callback. It will interrupt training when it measures no\nprogress on the validation set for a number of epochs (defined by the\npatience argument), and it will optionally roll back to the best model.\nYou can combine both callbacks to save checkpoints of your model (in\ncase your computer crashes) and interrupt training early when there is no\nmore progress (to avoid wasting time and resources):\nearly_stopping_cb = keras.callbacks.EarlyStopping(patience=10, \n \nrestore_best_weights=True) \nhistory = model.fit(X_train, y_train, epochs=100, \n validation_data=(X_valid, y_valid), \n callbacks=[checkpoint_cb, early_stopping_cb])\nThe number of epochs can be set to a large value since training will stop\nautomatically when there is no more progress. In this case, there is no\nneed to restore the best model saved because the EarlyStopping callback\nwill keep track of the best weights and restore them for you at the end of\ntraining.\nTIP\nThere are many other callbacks available in the keras.callbacks package.\nIf you need extra control, you can easily write your own custom callbacks.\nAs an example of how to do that, the following custom callback will\ndisplay the ratio between the validation loss and the training loss during\ntraining (e.g., to detect overfitting):\nclass PrintValTrainRatioCallback(keras.callbacks.Callback): \n def on_epoch_end(self, epoch, logs): \n print(\"\\nval/train: {:.2f}\".format(logs[\"val_loss\"] / \nlogs[\"loss\"]))\nAs you might expect, you can implement on_train_begin(),\non_train_end(), on_epoch_begin(), on_epoch_end(),\non_batch_begin(), and on_batch_end(). Callbacks can also be used\nduring evaluation and predictions, should you ever need them (e.g., for\ndebugging). For evaluation, you should implement on_test_begin(),\non_test_end(), on_test_batch_begin(), or on_test_batch_end()\n(called by evaluate()), and for prediction you should implement\non_predict_begin(), on_predict_end(), on_predict_batch_begin(),\nor on_predict_batch_end() (called by predict()).\nNow let’s take a look at one more tool you should definitely have in your\ntoolbox when using tf.keras: TensorBoard.\nUsing TensorBoard for Visualization\nTensorBoard is a great interactive visualization tool that you can use to\nview the learning curves during training, compare learning curves between\nmultiple runs, visualize the computation graph, analyze training statistics,\nview images generated by your model, visualize complex\nmultidimensional data projected down to 3D and automatically clustered\nfor you, and more! This tool is installed automatically when you install\nTensorFlow, so you already have it.\nTo use it, you must modify your program so that it outputs the data you\nwant to visualize to special binary log files called event files. Each binary\ndata record is called a summary. The TensorBoard server will monitor the\nlog directory, and it will automatically pick up the changes and update the\nvisualizations: this allows you to visualize live data (with a short delay),\nsuch as the learning curves during training. In general, you want to point\nthe TensorBoard server to a root log directory and configure your program\nso that it writes to a different subdirectory every time it runs. This way, the\nsame TensorBoard server instance will allow you to visualize and compare\ndata from multiple runs of your program, without getting everything\nmixed up.\nLet’s start by defining the root log directory we will use for our\nTensorBoard logs, plus a small function that will generate a subdirectory\npath based on the current date and time so that it’s different at every run.\nYou may want to include extra information in the log directory name, such\nas hyperparameter values that you are testing, to make it easier to know\nwhat you are looking at in TensorBoard:\nimport os \nroot_logdir = os.path.join(os.curdir, \"my_logs\") \n \ndef get_run_logdir(): \n import time \n run_id = time.strftime(\"run_%Y_%m_%d-%H_%M_%S\") \n return os.path.join(root_logdir, run_id) \n \nrun_logdir = get_run_logdir() # e.g., './my_logs/run_2019_06_07-15_15_22'\nThe good news is that Keras provides a nice TensorBoard() callback:\n[...] # Build and compile your model \ntensorboard_cb = keras.callbacks.TensorBoard(run_logdir) \nhistory = model.fit(X_train, y_train, epochs=30, \n validation_data=(X_valid, y_valid), \n callbacks=[tensorboard_cb])\nAnd that’s all there is to it! It could hardly be easier to use. If you run this\ncode, the TensorBoard() callback will take care of creating the log\ndirectory for you (along with its parent directories if needed), and during\ntraining it will create event files and write summaries to them. After\nrunning the program a second time (perhaps changing some\nhyperparameter value), you will end up with a directory structure similar\nto this one:\nmy_logs/ \n├── run_2019_06_07-15_15_22 \n│   ├── train \n│   │   ├── \nevents.out.tfevents.1559891732.mycomputer.local.38511.694049.v2 \n│   │   ├── events.out.tfevents.1559891732.mycomputer.local.profile-empty \n│   │   └── plugins/profile/2019-06-07_15-15-32 \n│   │   └── local.trace \n│   └── validation \n│   └── \nevents.out.tfevents.1559891733.mycomputer.local.38511.696430.v2 \n└── run_2019_06_07-15_15_49 \n └── [...]\nThere’s one directory per run, each containing one subdirectory for\ntraining logs and one for validation logs. Both contain event files, but the\ntraining logs also include profiling traces: this allows TensorBoard to\nshow you exactly how much time the model spent on each part of your\nmodel, across all your devices, which is great for locating performance\nbottlenecks.\nNext you need to start the TensorBoard server. One way to do this is by\nrunning a command in a terminal. If you installed TensorFlow within a\nvirtualenv, you should activate it. Next, run the following command at the\nroot of the project (or from anywhere else, as long as you point to the\nappropriate log directory):\n$ tensorboard --logdir=./my_logs --port=6006 \nTensorBoard 2.0.0 at http://mycomputer.local:6006/ (Press CTRL+C to quit) \nIf your shell cannot find the tensorboard script, then you must update your\nPATH environment variable so that it contains the directory in which the\nscript was installed (alternatively, you can just replace tensorboard in the\ncommand line with python3 -m tensorboard.main). Once the server is\nup, you can open a web browser and go to http://localhost:6006.\nAlternatively, you can use TensorBoard directly within Jupyter, by running\nthe following commands. The first line loads the TensorBoard extension,\nand the second line starts a TensorBoard server on port 6006 (unless it is\nalready started) and connects to it:\n%load_ext tensorboard \n%tensorboard --logdir=./my_logs --port=6006\nEither way, you should see TensorBoard’s web interface. Click the\nSCALARS tab to view the learning curves (see Figure 10-17). At the\nbottom left, select the logs you want to visualize (e.g., the training logs\nfrom the first and second run), and click the epoch_loss scalar. Notice\nthat the training loss went down nicely during both runs, but the second\nrun went down much faster. Indeed, we used a learning rate of 0.05\n(optimizer=keras.optimizers.SGD(lr=0.05)) instead of 0.001.\nFigure 10-17. Visualizing learning curves with TensorBoard\nYou can also visualize the whole graph, the learned weights (projected to\n3D), or the profiling traces. The TensorBoard() callback has options to\nlog extra data too, such as embeddings (see Chapter 13).\nAdditionally, TensorFlow offers a lower-level API in the tf.summary\npackage. The following code creates a SummaryWriter using the\ncreate_file_writer() function, and it uses this writer as a context to\nlog scalars, histograms, images, audio, and text, all of which can then be\nvisualized using TensorBoard (give it a try!):\ntest_logdir = get_run_logdir() \nwriter = tf.summary.create_file_writer(test_logdir) \nwith writer.as_default(): \n for step in range(1, 1000 + 1): \n tf.summary.scalar(\"my_scalar\", np.sin(step / 10), step=step) \n data = (np.random.randn(100) + 2) * step / 100 # some random data \n tf.summary.histogram(\"my_hist\", data, buckets=50, step=step) \n images = np.random.rand(2, 32, 32, 3) # random 32×32 RGB images \n tf.summary.image(\"my_images\", images * step / 1000, step=step) \n texts = [\"The step is \" + str(step), \"Its square is \" + \nstr(step**2)] \n tf.summary.text(\"my_text\", texts, step=step) \n sine_wave = tf.math.sin(tf.range(12000) / 48000 * 2 * np.pi * \nstep) \n audio = tf.reshape(tf.cast(sine_wave, tf.float32), [1, -1, 1]) \n tf.summary.audio(\"my_audio\", audio, sample_rate=48000, step=step)\nThis is actually a useful visualization tool to have, even beyond\nTensorFlow or Deep Learning.\nLet’s summarize what you’ve learned so far in this chapter: we saw where\nneural nets came from, what an MLP is and how you can use it for\nclassification and regression, how to use tf.keras’s Sequential API to build\nMLPs, and how to use the Functional API or the Subclassing API to build\nmore complex model architectures. You learned how to save and restore a\nmodel and how to use callbacks for checkpointing, early stopping, and\nmore. Finally, you learned how to use TensorBoard for visualization. You\ncan already go ahead and use neural networks to tackle many problems!\nHowever, you may wonder how to choose the number of hidden layers, the\nnumber of neurons in the network, and all the other hyperparameters. Let’s\nlook at this now.\nFine-Tuning Neural Network\nHyperparameters\nThe flexibility of neural networks is also one of their main drawbacks:\nthere are many hyperparameters to tweak. Not only can you use any\nimaginable network architecture, but even in a simple MLP you can\nchange the number of layers, the number of neurons per layer, the type of\nactivation function to use in each layer, the weight initialization logic, and\nmuch more. How do you know what combination of hyperparameters is\nthe best for your task?\nOne option is to simply try many combinations of hyperparameters and\nsee which one works best on the validation set (or use K-fold cross-\nvalidation). For example, we can use GridSearchCV or\nRandomizedSearchCV to explore the hyperparameter space, as we did in\nChapter 2. To do this, we need to wrap our Keras models in objects that\nmimic regular Scikit-Learn regressors. The first step is to create a function\nthat will build and compile a Keras model, given a set of hyperparameters:\ndef build_model(n_hidden=1, n_neurons=30, learning_rate=3e-3, \ninput_shape=[8]): \n model = keras.models.Sequential() \n model.add(keras.layers.InputLayer(input_shape=input_shape)) \n for layer in range(n_hidden): \n model.add(keras.layers.Dense(n_neurons, activation=\"relu\")) \n model.add(keras.layers.Dense(1)) \n optimizer = keras.optimizers.SGD(lr=learning_rate) \n model.compile(loss=\"mse\", optimizer=optimizer) \n return model\nThis function creates a simple Sequential model for univariate\nregression (only one output neuron), with the given input shape and the\ngiven number of hidden layers and neurons, and it compiles it using an\nSGD optimizer configured with the specified learning rate. It is good\npractice to provide reasonable defaults to as many hyperparameters as you\ncan, as Scikit-Learn does.\nNext, let’s create a KerasRegressor based on this build_model()\nfunction:\nkeras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)\nThe KerasRegressor object is a thin wrapper around the Keras model\nbuilt using build_model(). Since we did not specify any hyperparameters\nwhen creating it, it will use the default hyperparameters we defined in\nbuild_model(). Now we can use this object like a regular Scikit-Learn\nregressor: we can train it using its fit() method, then evaluate it using its\nscore() method, and use it to make predictions using its predict()\nmethod, as you can see in the following code:\nkeras_reg.fit(X_train, y_train, epochs=100, \n validation_data=(X_valid, y_valid), \n callbacks=[keras.callbacks.EarlyStopping(patience=10)]) \nmse_test = keras_reg.score(X_test, y_test) \ny_pred = keras_reg.predict(X_new)\nNote that any extra parameter you pass to the fit() method will get\npassed to the underlying Keras model. Also note that the score will be the\nopposite of the MSE because Scikit-Learn wants scores, not losses (i.e.,\nhigher should be better).\nWe don’t want to train and evaluate a single model like this, though we\nwant to train hundreds of variants and see which one performs best on the\nvalidation set. Since there are many hyperparameters, it is preferable to\nuse a randomized search rather than grid search (as we discussed in\nChapter 2). Let’s try to explore the number of hidden layers, the number of\nneurons, and the learning rate:\nfrom scipy.stats import reciprocal \nfrom sklearn.model_selection import RandomizedSearchCV \n \nparam_distribs = { \n \"n_hidden\": [0, 1, 2, 3], \n \"n_neurons\": np.arange(1, 100), \n \"learning_rate\": reciprocal(3e-4, 3e-2), \n} \n \nrnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=10, \ncv=3) \nrnd_search_cv.fit(X_train, y_train, epochs=100, \n validation_data=(X_valid, y_valid), \n callbacks=[keras.callbacks.EarlyStopping(patience=10)])\nThis is identical to what we did in Chapter 2, except here we pass extra\nparameters to the fit() method, and they get relayed to the underlying\nKeras models. Note that RandomizedSearchCV uses K-fold cross-\nvalidation, so it does not use X_valid and y_valid, which are only used\nfor early stopping.\nThe exploration may last many hours, depending on the hardware, the size\nof the dataset, the complexity of the model, and the values of n_iter and\ncv. When it’s over, you can access the best parameters found, the best\nscore, and the trained Keras model like this:\n>>> rnd_search_cv.best_params_ \n{'learning_rate': 0.0033625641252688094, 'n_hidden': 2, 'n_neurons': 42} \n>>> rnd_search_cv.best_score_ \n-0.3189529188278931 \n>>> model = rnd_search_cv.best_estimator_.model\nYou can now save this model, evaluate it on the test set, and, if you are\nsatisfied with its performance, deploy it to production. Using randomized\nsearch is not too hard, and it works well for many fairly simple problems.\nWhen training is slow, however (e.g., for more complex problems with\nlarger datasets), this approach will only explore a tiny portion of the\nhyperparameter space. You can partially alleviate this problem by\nassisting the search process manually: first run a quick random search\nusing wide ranges of hyperparameter values, then run another search using\nsmaller ranges of values centered on the best ones found during the first\nrun, and so on. This approach will hopefully zoom in on a good set of\nhyperparameters. However, it’s very time consuming, and probably not the\nbest use of your time.\nFortunately, there are many techniques to explore a search space much\nmore efficiently than randomly. Their core idea is simple: when a region\nof the space turns out to be good, it should be explored more. Such\ntechniques take care of the “zooming” process for you and lead to much\nbetter solutions in much less time. Here are some Python libraries you can\nuse to optimize hyperparameters:\nHyperopt\nA popular library for optimizing over all sorts of complex search\nspaces (including real values, such as the learning rate, and discrete\nvalues, such as the number of layers).\nHyperas, kopt, or Talos\nUseful libraries for optimizing hyperparameters for Keras models (the\nfirst two are based on Hyperopt).\nKeras Tuner\nAn easy-to-use hyperparameter optimization library by Google for\nKeras models, with a hosted service for visualization and analysis.\nScikit-Optimize (skopt)\nA general-purpose optimization library. The BayesSearchCV class\nperforms Bayesian optimization using an interface similar to\nGridSearchCV.\nSpearmint\nA Bayesian optimization library.\nHyperband\nA fast hyperparameter tuning library based on the recent Hyperband\npaper\n by Lisha Li et al.\nSklearn-Deap\nA hyperparameter optimization library based on evolutionary\nalgorithms, with a GridSearchCV-like interface.\nMoreover, many companies offer services for hyperparameter\noptimization. We’ll discuss Google Cloud AI Platform’s hyperparameter\ntuning service in Chapter 19. Other options include services by Arimo and\nSigOpt, and CallDesk’s Oscar.\nHyperparameter tuning is still an active area of research, and evolutionary\nalgorithms are making a comeback. For example, check out DeepMind’s\nexcellent 2017 paper,\n where the authors jointly optimize a population of\nmodels and their hyperparameters. Google has also used an evolutionary\napproach, not just to search for hyperparameters but also to look for the\nbest neural network architecture for the problem; their AutoML suite is\nalready available as a cloud service. Perhaps the days of building neural\n22\n23\nnetworks manually will soon be over? Check out Google’s post on this\ntopic. In fact, evolutionary algorithms have been used successfully to train\nindividual neural networks, replacing the ubiquitous Gradient Descent!\nFor an example, see the 2017 post by Uber where the authors introduce\ntheir Deep Neuroevolution technique.\nBut despite all this exciting progress and all these tools and services, it\nstill helps to have an idea of what values are reasonable for each\nhyperparameter so that you can build a quick prototype and restrict the\nsearch space. The following sections provide guidelines for choosing the\nnumber of hidden layers and neurons in an MLP and for selecting good\nvalues for some of the main hyperparameters.\nNumber of Hidden Layers\nFor many problems, you can begin with a single hidden layer and get\nreasonable results. An MLP with just one hidden layer can theoretically\nmodel even the most complex functions, provided it has enough neurons.\nBut for complex problems, deep networks have a much higher parameter\nefficiency than shallow ones: they can model complex functions using\nexponentially fewer neurons than shallow nets, allowing them to reach\nmuch better performance with the same amount of training data.\nTo understand why, suppose you are asked to draw a forest using some\ndrawing software, but you are forbidden to copy and paste anything. It\nwould take an enormous amount of time: you would have to draw each\ntree individually, branch by branch, leaf by leaf. If you could instead draw\none leaf, copy and paste it to draw a branch, then copy and paste that\nbranch to create a tree, and finally copy and paste this tree to make a\nforest, you would be finished in no time. Real-world data is often\nstructured in such a hierarchical way, and deep neural networks\nautomatically take advantage of this fact: lower hidden layers model low-\nlevel structures (e.g., line segments of various shapes and orientations),\nintermediate hidden layers combine these low-level structures to model\nintermediate-level structures (e.g., squares, circles), and the highest\nhidden layers and the output layer combine these intermediate structures\nto model high-level structures (e.g., faces).\nNot only does this hierarchical architecture help DNNs converge faster to\na good solution, but it also improves their ability to generalize to new\ndatasets. For example, if you have already trained a model to recognize\nfaces in pictures and you now want to train a new neural network to\nrecognize hairstyles, you can kickstart the training by reusing the lower\nlayers of the first network. Instead of randomly initializing the weights\nand biases of the first few layers of the new neural network, you can\ninitialize them to the values of the weights and biases of the lower layers\nof the first network. This way the network will not have to learn from\nscratch all the low-level structures that occur in most pictures; it will only\nhave to learn the higher-level structures (e.g., hairstyles). This is called\ntransfer learning.\nIn summary, for many problems you can start with just one or two hidden\nlayers and the neural network will work just fine. For instance, you can\neasily reach above 97% accuracy on the MNIST dataset using just one\nhidden layer with a few hundred neurons, and above 98% accuracy using\ntwo hidden layers with the same total number of neurons, in roughly the\nsame amount of training time. For more complex problems, you can ramp\nup the number of hidden layers until you start overfitting the training set.\nVery complex tasks, such as large image classification or speech\nrecognition, typically require networks with dozens of layers (or even\nhundreds, but not fully connected ones, as we will see in Chapter 14), and\nthey need a huge amount of training data. You will rarely have to train\nsuch networks from scratch: it is much more common to reuse parts of a\npretrained state-of-the-art network that performs a similar task. Training\nwill then be a lot faster and require much less data (we will discuss this in\nChapter 11).\nNumber of Neurons per Hidden Layer\nThe number of neurons in the input and output layers is determined by the\ntype of input and output your task requires. For example, the MNIST task\nrequires 28 × 28 = 784 input neurons and 10 output neurons.\nAs for the hidden layers, it used to be common to size them to form a\npyramid, with fewer and fewer neurons at each layer—the rationale being\nthat many low-level features can coalesce into far fewer high-level\nfeatures. A typical neural network for MNIST might have 3 hidden layers,\nthe first with 300 neurons, the second with 200, and the third with 100.\nHowever, this practice has been largely abandoned because it seems that\nusing the same number of neurons in all hidden layers performs just as\nwell in most cases, or even better; plus, there is only one hyperparameter\nto tune, instead of one per layer. That said, depending on the dataset, it can\nsometimes help to make the first hidden layer bigger than the others.\nJust like the number of layers, you can try increasing the number of\nneurons gradually until the network starts overfitting. But in practice, it’s\noften simpler and more efficient to pick a model with more layers and\nneurons than you actually need, then use early stopping and other\nregularization techniques to prevent it from overfitting. Vincent\nVanhoucke, a scientist at Google, has dubbed this the “stretch pants”\napproach: instead of wasting time looking for pants that perfectly match\nyour size, just use large stretch pants that will shrink down to the right\nsize. With this approach, you avoid bottleneck layers that could ruin your\nmodel. On the flip side, if a layer has too few neurons, it will not have\nenough representational power to preserve all the useful information from\nthe inputs (e.g., a layer with two neurons can only output 2D data, so if it\nprocesses 3D data, some information will be lost). No matter how big and\npowerful the rest of the network is, that information will never be\nrecovered.\nTIP\nIn general you will get more bang for your buck by increasing the number of layers\ninstead of the number of neurons per layer.\nLearning Rate, Batch Size, and Other Hyperparameters\nThe numbers of hidden layers and neurons are not the only\nhyperparameters you can tweak in an MLP. Here are some of the most\nimportant ones, as well as tips on how to set them:\nLearning rate\nThe learning rate is arguably the most important hyperparameter. In\ngeneral, the optimal learning rate is about half of the maximum\nlearning rate (i.e., the learning rate above which the training algorithm\ndiverges, as we saw in Chapter 4). One way to find a good learning rate\nis to train the model for a few hundred iterations, starting with a very\nlow learning rate (e.g., 10 ) and gradually increasing it up to a very\nlarge value (e.g., 10). This is done by multiplying the learning rate by a\nconstant factor at each iteration (e.g., by exp(log(10 )/500) to go from\n10 to 10 in 500 iterations). If you plot the loss as a function of the\nlearning rate (using a log scale for the learning rate), you should see it\ndropping at first. But after a while, the learning rate will be too large,\nso the loss will shoot back up: the optimal learning rate will be a bit\nlower than the point at which the loss starts to climb (typically about\n10 times lower than the turning point). You can then reinitialize your\nmodel and train it normally using this good learning rate. We will look\nat more learning rate techniques in Chapter 11.\nOptimizer\nChoosing a better optimizer than plain old Mini-batch Gradient\nDescent (and tuning its hyperparameters) is also quite important. We\nwill see several advanced optimizers in Chapter 11.\nBatch size\nThe batch size can have a significant impact on your model’s\nperformance and training time. The main benefit of using large batch\nsizes is that hardware accelerators like GPUs can process them\nefficiently (see Chapter 19), so the training algorithm will see more\ninstances per second. Therefore, many researchers and practitioners\nrecommend using the largest batch size that can fit in GPU RAM.\n-5\n6\n-5\nThere’s a catch, though: in practice, large batch sizes often lead to\ntraining instabilities, especially at the beginning of training, and the\nresulting model may not generalize as well as a model trained with a\nsmall batch size. In April 2018, Yann LeCun even tweeted “Friends\ndon’t let friends use mini-batches larger than 32,” citing a 2018\npaper\n by Dominic Masters and Carlo Luschi which concluded that\nusing small batches (from 2 to 32) was preferable because small\nbatches led to better models in less training time. Other papers point in\nthe opposite direction, however; in 2017, papers by Elad Hoffer et al.\nand Priya Goyal et al.\n showed that it was possible to use very large\nbatch sizes (up to 8,192) using various techniques such as warming up\nthe learning rate (i.e., starting training with a small learning rate, then\nramping it up, as we will see in Chapter 11). This led to a very short\ntraining time, without any generalization gap. So, one strategy is to try\nto use a large batch size, using learning rate warmup, and if training is\nunstable or the final performance is disappointing, then try using a\nsmall batch size instead.\nActivation function\nWe discussed how to choose the activation function earlier in this\nchapter: in general, the ReLU activation function will be a good\ndefault for all hidden layers. For the output layer, it really depends on\nyour task.\nNumber of iterations\nIn most cases, the number of training iterations does not actually need\nto be tweaked: just use early stopping instead.\nTIP\nThe optimal learning rate depends on the other hyperparameters—especially the\nbatch size—so if you modify any hyperparameter, make sure to update the learning\nrate as well.\n24\n25\n26\nFor more best practices regarding tuning neural network hyperparameters,\ncheck out the excellent 2018 paper\n by Leslie Smith.\nThis concludes our introduction to artificial neural networks and their\nimplementation with Keras. In the next few chapters, we will discuss\ntechniques to train very deep nets. We will also explore how to customize\nmodels using TensorFlow’s lower-level API and how to load and\npreprocess data efficiently using the Data API. And we will dive into other\npopular neural network architectures: convolutional neural networks for\nimage processing, recurrent neural networks for sequential data,\nautoencoders for representation learning, and generative adversarial\nnetworks to model and generate data.\nExercises\n1. The TensorFlow Playground is a handy neural network simulator\nbuilt by the TensorFlow team. In this exercise, you will train\nseveral binary classifiers in just a few clicks, and tweak the\nmodel’s architecture and its hyperparameters to gain some\nintuition on how neural networks work and what their\nhyperparameters do. Take some time to explore the following:\na. The patterns learned by a neural net. Try training the\ndefault neural network by clicking the Run button (top\nleft). Notice how it quickly finds a good solution for the\nclassification task. The neurons in the first hidden layer\nhave learned simple patterns, while the neurons in the\nsecond hidden layer have learned to combine the simple\npatterns of the first hidden layer into more complex\npatterns. In general, the more layers there are, the more\ncomplex the patterns can be.\nb. Activation functions. Try replacing the tanh activation\nfunction with a ReLU activation function, and train the\nnetwork again. Notice that it finds a solution even faster,\n27\n28\nbut this time the boundaries are linear. This is due to the\nshape of the ReLU function.\nc. The risk of local minima. Modify the network\narchitecture to have just one hidden layer with three\nneurons. Train it multiple times (to reset the network\nweights, click the Reset button next to the Play button).\nNotice that the training time varies a lot, and sometimes\nit even gets stuck in a local minimum.\nd. What happens when neural nets are too small. Remove\none neuron to keep just two. Notice that the neural\nnetwork is now incapable of finding a good solution,\neven if you try multiple times. The model has too few\nparameters and systematically underfits the training set.\ne. What happens when neural nets are large enough. Set the\nnumber of neurons to eight, and train the network several\ntimes. Notice that it is now consistently fast and never\ngets stuck. This highlights an important finding in neural\nnetwork theory: large neural networks almost never get\nstuck in local minima, and even when they do these local\noptima are almost as good as the global optimum.\nHowever, they can still get stuck on long plateaus for a\nlong time.\nf. The risk of vanishing gradients in deep networks. Select\nthe spiral dataset (the bottom-right dataset under\n“DATA”), and change the network architecture to have\nfour hidden layers with eight neurons each. Notice that\ntraining takes much longer and often gets stuck on\nplateaus for long periods of time. Also notice that the\nneurons in the highest layers (on the right) tend to evolve\nfaster than the neurons in the lowest layers (on the left).\nThis problem, called the “vanishing gradients” problem,\ncan be alleviated with better weight initialization and\nother techniques, better optimizers (such as AdaGrad or\nAdam), or Batch Normalization (discussed in\nChapter 11).\ng. Go further. Take an hour or so to play around with other\nparameters and get a feel for what they do, to build an\nintuitive understanding about neural networks.\n2. Draw an ANN using the original artificial neurons (like the ones\nin Figure 10-3) that computes A ⊕ B (where ⊕ represents the\nXOR operation). Hint: A ⊕ B = (A ∧ ¬ B ∨ (¬ A ∧ B).\n3. Why is it generally preferable to use a Logistic Regression\nclassifier rather than a classical Perceptron (i.e., a single layer of\nthreshold logic units trained using the Perceptron training\nalgorithm)? How can you tweak a Perceptron to make it\nequivalent to a Logistic Regression classifier?\n4. Why was the logistic activation function a key ingredient in\ntraining the first MLPs?\n5. Name three popular activation functions. Can you draw them?\n6. Suppose you have an MLP composed of one input layer with 10\npassthrough neurons, followed by one hidden layer with 50\nartificial neurons, and finally one output layer with 3 artificial\nneurons. All artificial neurons use the ReLU activation function.\nWhat is the shape of the input matrix X?\nWhat are the shapes of the hidden layer’s weight vector\nW and its bias vector b ?\nWhat are the shapes of the output layer’s weight vector\nW and its bias vector b ?\nWhat is the shape of the network’s output matrix Y?\nh\nh\no\no\nWrite the equation that computes the network’s output\nmatrix Y as a function of X, W , b , W , and b .\n7. How many neurons do you need in the output layer if you want to\nclassify email into spam or ham? What activation function should\nyou use in the output layer? If instead you want to tackle MNIST,\nhow many neurons do you need in the output layer, and which\nactivation function should you use? What about for getting your\nnetwork to predict housing prices, as in Chapter 2?\n8. What is backpropagation and how does it work? What is the\ndifference between backpropagation and reverse-mode autodiff?\n9. Can you list all the hyperparameters you can tweak in a basic\nMLP? If the MLP overfits the training data, how could you tweak\nthese hyperparameters to try to solve the problem?\n10. Train a deep MLP on the MNIST dataset (you can load it using\nkeras.datasets.mnist.load_data(). See if you can get over\n98% precision. Try searching for the optimal learning rate by\nusing the approach presented in this chapter (i.e., by growing the\nlearning rate exponentially, plotting the error, and finding the\npoint where the error shoots up). Try adding all the bells and\nwhistles—save checkpoints, use early stopping, and plot learning\ncurves using TensorBoard.\nSolutions to these exercises are available in Appendix A.\n1 You can get the best of both worlds by being open to biological inspirations without being\nafraid to create biologically unrealistic models, as long as they work well.\n2 Warren S. McCulloch and Walter Pitts, “A Logical Calculus of the Ideas Immanent in\nNervous Activity,” The Bulletin of Mathematical Biology 5, no. 4 (1943): 115–113.\n3 They are not actually attached, just so close that they can very quickly exchange chemical\nsignals.\n4 Image by Bruce Blaus (Creative Commons 3.0). Reproduced from\nhttps://en.wikipedia.org/wiki/Neuron.\nh\nh\no\no\n5 In the context of Machine Learning, the phrase “neural networks” generally refers to\nANNs, not BNNs.\n6 Drawing of a cortical lamination by S. Ramon y Cajal (public domain). Reproduced from\nhttps://en.wikipedia.org/wiki/Cerebral_cortex.\n7 The name Perceptron is sometimes used to mean a tiny network with a single TLU.\n8 Note that this solution is not unique: when data points are linearly separable, there is an\ninfinity of hyperplanes that can separate them.\n9 In the 1990s, an ANN with more than two hidden layers was considered deep. Nowadays,\nit is common to see ANNs with dozens of layers, or even hundreds, so the definition of\n“deep” is quite fuzzy.\n10 David Rumelhart et al. “Learning Internal Representations by Error Propagation,”\n(Defense Technical Information Center technical report, September 1985).\n11 This technique was actually independently invented several times by various researchers\nin different fields, starting with Paul Werbos in 1974.\n12 Biological neurons seem to implement a roughly sigmoid (S-shaped) activation function,\nso researchers stuck to sigmoid functions for a very long time. But it turns out that ReLU\ngenerally works better in ANNs. This is one of the cases where the biological analogy was\nmisleading.\n13 Project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).\n14 You can use keras.utils.plot_model() to generate an image of your model.\n15 If your training or validation data does not match the expected shape, you will get an\nexception. This is perhaps the most common error, so you should get familiar with the error\nmessage. The message is actually quite clear: for example, if you try to train this model\nwith an array containing flattened images (X_train.reshape(-1, 784)), then you will\nget the following exception: “ValueError: Error when checking input: expected\nflatten_input to have 3 dimensions, but got array with shape (60000, 784).”\n16 Heng-Tze Cheng et al., “Wide & Deep Learning for Recommender Systems,”\nProceedings of the First Workshop on Deep Learning for Recommender Systems (2016): 7–\n10.\n17 The short path can also be used to provide manually engineered features to the neural\nnetwork.\n18 The name input_ is used to avoid overshadowing Python’s built-in input() function.\n19 Alternatively, you can pass a dictionary mapping the input names to the input values, like\n{\"wide_input\": X_train_A, \"deep_input\": X_train_B}. This is especially useful\nwhen there are many inputs, to avoid getting the order wrong.\n20 Alternatively, you can pass a dictionary that maps each output name to the corresponding\nloss. Just like for the inputs, this is useful when there are multiple outputs, to avoid getting\nthe order wrong. The loss weights and metrics (discussed shortly) can also be set using\ndictionaries.\n21 Keras models have an output attribute, so we cannot use that name for the main output\nlayer, which is why we renamed it to main_output.\n22 Lisha Li et al., “Hyperband: A Novel Bandit-Based Approach to Hyperparameter\nOptimization,” Journal of Machine Learning Research 18 (April 2018): 1–52.\n23 Max Jaderberg et al., “Population Based Training of Neural Networks,” arXiv preprint\narXiv:1711.09846 (2017).\n24 Dominic Masters and Carlo Luschi, “Revisiting Small Batch Training for Deep Neural\nNetworks,” arXiv preprint arXiv:1804.07612 (2018).\n25 Elad Hoffer et al., “Train Longer, Generalize Better: Closing the Generalization Gap in\nLarge Batch Training of Neural Networks,” Proceedings of the 31st International\nConference on Neural Information Processing Systems (2017): 1729–1739.\n26 Priya Goyal et al., “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” arXiv\npreprint arXiv:1706.02677 (2017).\n27 Leslie N. Smith, “A Disciplined Approach to Neural Network Hyper-Parameters: Part 1—\nLearning Rate, Batch Size, Momentum, and Weight Decay,” arXiv preprint\narXiv:1803.09820 (2018).\n28 A few extra ANN architectures are presented in Appendix E." + }, + { + "chapter": 11, + "text": "Chapter 11. Training Deep\nNeural Networks\nIn Chapter 10 we introduced artificial neural networks and trained our first\ndeep neural networks. But they were shallow nets, with just a few hidden\nlayers. What if you need to tackle a complex problem, such as detecting\nhundreds of types of objects in high-resolution images? You may need to\ntrain a much deeper DNN, perhaps with 10 layers or many more, each\ncontaining hundreds of neurons, linked by hundreds of thousands of\nconnections. Training a deep DNN isn’t a walk in the park. Here are some\nof the problems you could run into:\nYou may be faced with the tricky vanishing gradients problem or\nthe related exploding gradients problem. This is when the\ngradients grow smaller and smaller, or larger and larger, when\nflowing backward through the DNN during training. Both of these\nproblems make lower layers very hard to train.\nYou might not have enough training data for such a large network,\nor it might be too costly to label.\nTraining may be extremely slow.\nA model with millions of parameters would severely risk\noverfitting the training set, especially if there are not enough\ntraining instances or if they are too noisy.\nIn this chapter we will go through each of these problems and present\ntechniques to solve them. We will start by exploring the vanishing and\nexploding gradients problems and some of their most popular solutions.\nNext, we will look at transfer learning and unsupervised pretraining, which\ncan help you tackle complex tasks even when you have little labeled data.\nThen we will discuss various optimizers that can speed up training large\nmodels tremendously. Finally, we will go through a few popular\nregularization techniques for large neural networks.\nWith these tools, you will be able to train very deep nets. Welcome to\nDeep Learning!\nThe Vanishing/Exploding Gradients\nProblems\nAs we discussed in Chapter 10, the backpropagation algorithm works by\ngoing from the output layer to the input layer, propagating the error\ngradient along the way. Once the algorithm has computed the gradient of\nthe cost function with regard to each parameter in the network, it uses\nthese gradients to update each parameter with a Gradient Descent step.\nUnfortunately, gradients often get smaller and smaller as the algorithm\nprogresses down to the lower layers. As a result, the Gradient Descent\nupdate leaves the lower layers’ connection weights virtually unchanged,\nand training never converges to a good solution. We call this the vanishing\ngradients problem. In some cases, the opposite can happen: the gradients\ncan grow bigger and bigger until layers get insanely large weight updates\nand the algorithm diverges. This is the exploding gradients problem,\nwhich surfaces in recurrent neural networks (see Chapter 15). More\ngenerally, deep neural networks suffer from unstable gradients; different\nlayers may learn at widely different speeds.\nThis unfortunate behavior was empirically observed long ago, and it was\none of the reasons deep neural networks were mostly abandoned in the\nearly 2000s. It wasn’t clear what caused the gradients to be so unstable\nwhen training a DNN, but some light was shed in a 2010 paper by Xavier\nGlorot and Yoshua Bengio. The authors found a few suspects, including\nthe combination of the popular logistic sigmoid activation function and\nthe weight initialization technique that was most popular at the time (i.e.,\na normal distribution with a mean of 0 and a standard deviation of 1). In\nshort, they showed that with this activation function and this initialization\n1\nscheme, the variance of the outputs of each layer is much greater than the\nvariance of its inputs. Going forward in the network, the variance keeps\nincreasing after each layer until the activation function saturates at the top\nlayers. This saturation is actually made worse by the fact that the logistic\nfunction has a mean of 0.5, not 0 (the hyperbolic tangent function has a\nmean of 0 and behaves slightly better than the logistic function in deep\nnetworks).\nLooking at the logistic activation function (see Figure 11-1), you can see\nthat when inputs become large (negative or positive), the function\nsaturates at 0 or 1, with a derivative extremely close to 0. Thus, when\nbackpropagation kicks in it has virtually no gradient to propagate back\nthrough the network; and what little gradient exists keeps getting diluted\nas backpropagation progresses down through the top layers, so there is\nreally nothing left for the lower layers.\nFigure 11-1. Logistic activation function saturation\nGlorot and He Initialization\nIn their paper, Glorot and Bengio propose a way to significantly alleviate\nthe unstable gradients problem. They point out that we need the signal to\nflow properly in both directions: in the forward direction when making\npredictions, and in the reverse direction when backpropagating gradients.\nWe don’t want the signal to die out, nor do we want it to explode and\nsaturate. For the signal to flow properly, the authors argue that we need the\nvariance of the outputs of each layer to be equal to the variance of its\ninputs, and we need the gradients to have equal variance before and after\nflowing through a layer in the reverse direction (please check out the paper\nif you are interested in the mathematical details). It is actually not\npossible to guarantee both unless the layer has an equal number of inputs\nand neurons (these numbers are called the fan-in and fan-out of the layer),\nbut Glorot and Bengio proposed a good compromise that has proven to\nwork very well in practice: the connection weights of each layer must be\ninitialized randomly as described in Equation 11-1, where \nfanavg = (fanin + fanout) /2. This initialization strategy is called Xavier\ninitialization or Glorot initialization, after the paper’s first author.\nEquation 11-1. Glorot initialization (when using the logistic activation function)\nNormal distribution with mean 0 and variance σ2 =\nOr a uniform distribution between  −r and  + r, with r = √\nIf you replace fan\n with fan in Equation 11-1, you get an initialization\nstrategy that Yann LeCun proposed in the 1990s. He called it LeCun\ninitialization. Genevieve Orr and Klaus-Robert Müller even recommended\nit in their 1998 book Neural Networks: Tricks of the Trade (Springer).\nLeCun initialization is equivalent to Glorot initialization when fan =\nfan\n. It took over a decade for researchers to realize how important this\ntrick is. Using Glorot initialization can speed up training considerably, and\nit is one of the tricks that led to the success of Deep Learning.\nSome papers have provided similar strategies for different activation\nfunctions. These strategies differ only by the scale of the variance and\nwhether they use fan\n or fan , as shown in Table 11-1 (for the uniform\n2\n1\nfanavg\n3\nfanavg\navg\nin\nin\nout\n3\navg\nin\ndistribution, just compute r = √3σ2). The initialization strategy for the\nReLU activation function (and its variants, including the ELU activation\ndescribed shortly) is sometimes called He initialization, after the paper’s\nfirst author. The SELU activation function will be explained later in this\nchapter. It should be used with LeCun initialization (preferably with a\nnormal distribution, as we will see).\nTable 11-1. Initialization parameters for each\ntype of activation function\nInitialization Activation functions\nσ² (Normal)\nGlorot\nNone, tanh, logistic, softmax 1 / fan\nHe\nReLU and variants\n2 / fan\nLeCun\nSELU\n1 / fan\nBy default, Keras uses Glorot initialization with a uniform distribution.\nWhen creating a layer, you can change this to He initialization by setting\nkernel_initializer=\"he_uniform\" or\nkernel_initializer=\"he_normal\" like this:\nkeras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")\nIf you want He initialization with a uniform distribution but based on\nfan\n rather than fan , you can use the VarianceScaling initializer like\nthis:\nhe_avg_init = keras.initializers.VarianceScaling(scale=2., \nmode='fan_avg', \n distribution='uniform') \nkeras.layers.Dense(10, activation=\"sigmoid\", \nkernel_initializer=he_avg_init)\nNonsaturating Activation Functions\navg\nin\nin\navg\nin\nOne of the insights in the 2010 paper by Glorot and Bengio was that the\nproblems with unstable gradients were in part due to a poor choice of\nactivation function. Until then most people had assumed that if Mother\nNature had chosen to use roughly sigmoid activation functions in\nbiological neurons, they must be an excellent choice. But it turns out that\nother activation functions behave much better in deep neural networks—in\nparticular, the ReLU activation function, mostly because it does not\nsaturate for positive values (and because it is fast to compute).\nUnfortunately, the ReLU activation function is not perfect. It suffers from\na problem known as the dying ReLUs: during training, some neurons\neffectively “die,” meaning they stop outputting anything other than 0. In\nsome cases, you may find that half of your network’s neurons are dead,\nespecially if you used a large learning rate. A neuron dies when its weights\nget tweaked in such a way that the weighted sum of its inputs are negative\nfor all instances in the training set. When this happens, it just keeps\noutputting zeros, and Gradient Descent does not affect it anymore because\nthe gradient of the ReLU function is zero when its input is negative.\nTo solve this problem, you may want to use a variant of the ReLU\nfunction, such as the leaky ReLU. This function is defined as\nLeakyReLU (z) = max(αz, z) (see Figure 11-2). The hyperparameter α\ndefines how much the function “leaks”: it is the slope of the function for z\n< 0 and is typically set to 0.01. This small slope ensures that leaky ReLUs\nnever die; they can go into a long coma, but they have a chance to\neventually wake up. A 2015 paper compared several variants of the ReLU\nactivation function, and one of its conclusions was that the leaky variants\nalways outperformed the strict ReLU activation function. In fact, setting α\n= 0.2 (a huge leak) seemed to result in better performance than α = 0.01 (a\nsmall leak). The paper also evaluated the randomized leaky ReLU\n(RReLU), where α is picked randomly in a given range during training and\nis fixed to an average value during testing. RReLU also performed fairly\nwell and seemed to act as a regularizer (reducing the risk of overfitting the\ntraining set). Finally, the paper evaluated the parametric leaky ReLU\n(PReLU), where α is authorized to be learned during training (instead of\n4\nα\n5\nbeing a hyperparameter, it becomes a parameter that can be modified by\nbackpropagation like any other parameter). PReLU was reported to\nstrongly outperform ReLU on large image datasets, but on smaller datasets\nit runs the risk of overfitting the training set.\nFigure 11-2. Leaky ReLU: like ReLU, but with a small slope for negative values\nLast but not least, a 2015 paper by Djork-Arné Clevert et al. proposed a\nnew activation function called the exponential linear unit (ELU) that\noutperformed all the ReLU variants in the authors’ experiments: training\ntime was reduced, and the neural network performed better on the test set.\nFigure 11-3 graphs the function, and Equation 11-2 shows its definition.\nEquation 11-2. ELU activation function\nELUα (z) ={ α(exp(z) −1)\nif z < 0\nz\nif z ≥0\n6\nFigure 11-3. ELU activation function\nThe ELU activation function looks a lot like the ReLU function, with a few\nmajor differences:\nIt takes on negative values when z < 0, which allows the unit to\nhave an average output closer to 0 and helps alleviate the\nvanishing gradients problem. The hyperparameter α defines the\nvalue that the ELU function approaches when z is a large negative\nnumber. It is usually set to 1, but you can tweak it like any other\nhyperparameter.\nIt has a nonzero gradient for z < 0, which avoids the dead neurons\nproblem.\nIf α is equal to 1 then the function is smooth everywhere,\nincluding around z = 0, which helps speed up Gradient Descent\nsince it does not bounce as much to the left and right of z = 0.\nThe main drawback of the ELU activation function is that it is slower to\ncompute than the ReLU function and its variants (due to the use of the\nexponential function). Its faster convergence rate during training\ncompensates for that slow computation, but still, at test time an ELU\nnetwork will be slower than a ReLU network.\nThen, a 2017 paper by Günter Klambauer et al. introduced the Scaled\nELU (SELU) activation function: as its name suggests, it is a scaled\nvariant of the ELU activation function. The authors showed that if you\nbuild a neural network composed exclusively of a stack of dense layers,\nand if all hidden layers use the SELU activation function, then the network\nwill self-normalize: the output of each layer will tend to preserve a mean\nof 0 and standard deviation of 1 during training, which solves the\nvanishing/exploding gradients problem. As a result, the SELU activation\nfunction often significantly outperforms other activation functions for\nsuch neural nets (especially deep ones). There are, however, a few\nconditions for self-normalization to happen (see the paper for the\nmathematical justification):\nThe input features must be standardized (mean 0 and standard\ndeviation 1).\nEvery hidden layer’s weights must be initialized with LeCun\nnormal initialization. In Keras, this means setting\nkernel_initializer=\"lecun_normal\".\nThe network’s architecture must be sequential. Unfortunately, if\nyou try to use SELU in nonsequential architectures, such as\nrecurrent networks (see Chapter 15) or networks with skip\nconnections (i.e., connections that skip layers, such as in Wide &\nDeep nets), self-normalization will not be guaranteed, so SELU\nwill not necessarily outperform other activation functions.\nThe paper only guarantees self-normalization if all layers are\ndense, but some researchers have noted that the SELU activation\nfunction can improve performance in convolutional neural nets as\nwell (see Chapter 14).\n7\nTIP\nSo, which activation function should you use for the hidden layers of your deep\nneural networks? Although your mileage will vary, in general SELU > ELU > leaky\nReLU (and its variants) > ReLU > tanh > logistic. If the network’s architecture\nprevents it from self-normalizing, then ELU may perform better than SELU (since\nSELU is not smooth at z = 0). If you care a lot about runtime latency, then you may\nprefer leaky ReLU. If you don’t want to tweak yet another hyperparameter, you may\nuse the default α values used by Keras (e.g., 0.3 for leaky ReLU). If you have spare\ntime and computing power, you can use cross-validation to evaluate other activation\nfunctions, such as RReLU if your network is overfitting or PReLU if you have a\nhuge training set. That said, because ReLU is the most used activation function (by\nfar), many libraries and hardware accelerators provide ReLU-specific optimizations;\ntherefore, if speed is your priority, ReLU might still be the best choice.\nTo use the leaky ReLU activation function, create a LeakyReLU layer and\nadd it to your model just after the layer you want to apply it to:\nmodel = keras.models.Sequential([ \n [...] \n keras.layers.Dense(10, kernel_initializer=\"he_normal\"), \n keras.layers.LeakyReLU(alpha=0.2), \n [...] \n])\nFor PReLU, replace LeakyRelu(alpha=0.2) with PReLU(). There is\ncurrently no official implementation of RReLU in Keras, but you can\nfairly easily implement your own (to learn how to do that, see the\nexercises at the end of Chapter 12).\nFor SELU activation, set activation=\"selu\" and\nkernel_initializer=\"lecun_normal\" when creating a layer:\nlayer = keras.layers.Dense(10, activation=\"selu\", \n kernel_initializer=\"lecun_normal\")\nBatch Normalization\nAlthough using He initialization along with ELU (or any variant of ReLU)\ncan significantly reduce the danger of the vanishing/exploding gradients\nproblems at the beginning of training, it doesn’t guarantee that they won’t\ncome back during training.\nIn a 2015 paper, Sergey Ioffe and Christian Szegedy proposed a technique\ncalled Batch Normalization (BN) that addresses these problems. The\ntechnique consists of adding an operation in the model just before or after\nthe activation function of each hidden layer. This operation simply zero-\ncenters and normalizes each input, then scales and shifts the result using\ntwo new parameter vectors per layer: one for scaling, the other for\nshifting. In other words, the operation lets the model learn the optimal\nscale and mean of each of the layer’s inputs. In many cases, if you add a\nBN layer as the very first layer of your neural network, you do not need to\nstandardize your training set (e.g., using a StandardScaler); the BN layer\nwill do it for you (well, approximately, since it only looks at one batch at a\ntime, and it can also rescale and shift each input feature).\nIn order to zero-center and normalize the inputs, the algorithm needs to\nestimate each input’s mean and standard deviation. It does so by\nevaluating the mean and standard deviation of the input over the current\nmini-batch (hence the name “Batch Normalization”). The whole operation\nis summarized step by step in Equation 11-3.\nEquation 11-3. Batch Normalization algorithm\n1.\nμB =\nmB\n∑\ni=1\nx(i)\n2.\nσB2 =\nmB\n∑\ni=1\n(x(i) −μB)\n2\n3.\nˆx(i) =\n4.\nz(i) = γ ⊗ˆx(i) + β\n8\n1\nmB\n1\nmB\nx(i) −μB\n√σB2 + ε\nIn this algorithm:\nμ is the vector of input means, evaluated over the whole mini-\nbatch B (it contains one mean per input).\nσ is the vector of input standard deviations, also evaluated over\nthe whole mini-batch (it contains one standard deviation per\ninput).\nm is the number of instances in the mini-batch.\nˆx is the vector of zero-centered and normalized inputs for\ninstance i.\nγ is the output scale parameter vector for the layer (it contains one\nscale parameter per input).\n⊗ represents element-wise multiplication (each input is\nmultiplied by its corresponding output scale parameter).\nβ is the output shift (offset) parameter vector for the layer (it\ncontains one offset parameter per input). Each input is offset by\nits corresponding shift parameter.\nε is a tiny number that avoids division by zero (typically 10\n).\nThis is called a smoothing term.\nz is the output of the BN operation. It is a rescaled and shifted\nversion of the inputs.\nSo during training, BN standardizes its inputs, then rescales and offsets\nthem. Good! What about at test time? Well, it’s not that simple. Indeed, we\nmay need to make predictions for individual instances rather than for\nbatches of instances: in this case, we will have no way to compute each\ninput’s mean and standard deviation. Moreover, even if we do have a batch\nof instances, it may be too small, or the instances may not be independent\nand identically distributed, so computing statistics over the batch\ninstances would be unreliable. One solution could be to wait until the end\nof training, then run the whole training set through the neural network and\nB\nB\nB\n(i)\n–5\n(i)\ncompute the mean and standard deviation of each input of the BN layer.\nThese “final” input means and standard deviations could then be used\ninstead of the batch input means and standard deviations when making\npredictions. However, most implementations of Batch Normalization\nestimate these final statistics during training by using a moving average of\nthe layer’s input means and standard deviations. This is what Keras does\nautomatically when you use the BatchNormalization layer. To sum up,\nfour parameter vectors are learned in each batch-normalized layer: γ (the\noutput scale vector) and β (the output offset vector) are learned through\nregular backpropagation, and μ (the final input mean vector) and σ (the\nfinal input standard deviation vector) are estimated using an exponential\nmoving average. Note that μ and σ are estimated during training, but they\nare used only after training (to replace the batch input means and standard\ndeviations in Equation 11-3).\nIoffe and Szegedy demonstrated that Batch Normalization considerably\nimproved all the deep neural networks they experimented with, leading to\na huge improvement in the ImageNet classification task (ImageNet is a\nlarge database of images classified into many classes, commonly used to\nevaluate computer vision systems). The vanishing gradients problem was\nstrongly reduced, to the point that they could use saturating activation\nfunctions such as the tanh and even the logistic activation function. The\nnetworks were also much less sensitive to the weight initialization. The\nauthors were able to use much larger learning rates, significantly speeding\nup the learning process. Specifically, they note that:\nApplied to a state-of-the-art image classification model, Batch\nNormalization achieves the same accuracy with 14 times fewer training\nsteps, and beats the original model by a significant margin. […] Using\nan ensemble of batch-normalized networks, we improve upon the best\npublished result on ImageNet classification: reaching 4.9% top-5\nvalidation error (and 4.8% test error), exceeding the accuracy of human\nraters.\nFinally, like a gift that keeps on giving, Batch Normalization acts like a\nregularizer, reducing the need for other regularization techniques (such as\ndropout, described later in this chapter).\nBatch Normalization does, however, add some complexity to the model\n(although it can remove the need for normalizing the input data, as we\ndiscussed earlier). Moreover, there is a runtime penalty: the neural\nnetwork makes slower predictions due to the extra computations required\nat each layer. Fortunately, it’s often possible to fuse the BN layer with the\nprevious layer, after training, thereby avoiding the runtime penalty. This is\ndone by updating the previous layer’s weights and biases so that it directly\nproduces outputs of the appropriate scale and offset. For example, if the\nprevious layer computes XW + b, then the BN layer will compute γ⊗(XW\n+ b – μ)/σ + β (ignoring the smoothing term ε in the denominator). If we\ndefine W′ = γ⊗W/σ and b′ = γ⊗(b – μ)/σ + β, the equation simplifies to\nXW′ + b′. So if we replace the previous layer’s weights and biases (W and\nb) with the updated weights and biases (W′ and b′), we can get rid of the\nBN layer (TFLite’s optimizer does this automatically; see Chapter 19).\nNOTE\nYou may find that training is rather slow, because each epoch takes much more time\nwhen you use Batch Normalization. This is usually counterbalanced by the fact that\nconvergence is much faster with BN, so it will take fewer epochs to reach the same\nperformance. All in all, wall time will usually be shorter (this is the time measured by\nthe clock on your wall).\nImplementing Batch Normalization with Keras\nAs with most things with Keras, implementing Batch Normalization is\nsimple and intuitive. Just add a BatchNormalization layer before or after\neach hidden layer’s activation function, and optionally add a BN layer as\nwell as the first layer in your model. For example, this model applies BN\nafter every hidden layer and as the first layer in the model (after flattening\nthe input images):\nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.BatchNormalization(), \n keras.layers.Dense(300, activation=\"elu\", \nkernel_initializer=\"he_normal\"), \n keras.layers.BatchNormalization(), \n keras.layers.Dense(100, activation=\"elu\", \nkernel_initializer=\"he_normal\"), \n keras.layers.BatchNormalization(), \n keras.layers.Dense(10, activation=\"softmax\") \n])\nThat’s all! In this tiny example with just two hidden layers, it’s unlikely\nthat Batch Normalization will have a very positive impact; but for deeper\nnetworks it can make a tremendous difference.\nLet’s display the model summary:\n>>> model.summary() \nModel: \"sequential_3\" \n_________________________________________________________________ \nLayer (type) Output Shape Param # \n================================================================= \nflatten_3 (Flatten) (None, 784) 0 \n_________________________________________________________________ \nbatch_normalization_v2 (Batc (None, 784) 3136 \n_________________________________________________________________ \ndense_50 (Dense) (None, 300) 235500 \n_________________________________________________________________ \nbatch_normalization_v2_1 (Ba (None, 300) 1200 \n_________________________________________________________________ \ndense_51 (Dense) (None, 100) 30100 \n_________________________________________________________________ \nbatch_normalization_v2_2 (Ba (None, 100) 400 \n_________________________________________________________________ \ndense_52 (Dense) (None, 10) 1010 \n================================================================= \nTotal params: 271,346 \nTrainable params: 268,978 \nNon-trainable params: 2,368\nAs you can see, each BN layer adds four parameters per input: γ, β, μ, and\nσ (for example, the first BN layer adds 3,136 parameters, which is 4 ×\n784). The last two parameters, μ and σ, are the moving averages; they are\nnot affected by backpropagation, so Keras calls them “non-trainable” (if\n9\nyou count the total number of BN parameters, 3,136 + 1,200 + 400, and\ndivide by 2, you get 2,368, which is the total number of non-trainable\nparameters in this model).\nLet’s look at the parameters of the first BN layer. Two are trainable (by\nbackpropagation), and two are not:\n>>> [(var.name, var.trainable) for var in model.layers[1].variables] \n[('batch_normalization_v2/gamma:0', True), \n ('batch_normalization_v2/beta:0', True), \n ('batch_normalization_v2/moving_mean:0', False), \n ('batch_normalization_v2/moving_variance:0', False)]\nNow when you create a BN layer in Keras, it also creates two operations\nthat will be called by Keras at each iteration during training. These\noperations will update the moving averages. Since we are using the\nTensorFlow backend, these operations are TensorFlow operations (we will\ndiscuss TF operations in Chapter 12):\n>>> model.layers[1].updates \n[, \n ]\nThe authors of the BN paper argued in favor of adding the BN layers\nbefore the activation functions, rather than after (as we just did). There is\nsome debate about this, as which is preferable seems to depend on the task\n—you can experiment with this too to see which option works best on your\ndataset. To add the BN layers before the activation functions, you must\nremove the activation function from the hidden layers and add them as\nseparate layers after the BN layers. Moreover, since a Batch\nNormalization layer includes one offset parameter per input, you can\nremove the bias term from the previous layer (just pass use_bias=False\nwhen creating it):\nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.BatchNormalization(), \n keras.layers.Dense(300, kernel_initializer=\"he_normal\", \nuse_bias=False), \n keras.layers.BatchNormalization(), \n keras.layers.Activation(\"elu\"), \n keras.layers.Dense(100, kernel_initializer=\"he_normal\", \nuse_bias=False), \n keras.layers.BatchNormalization(), \n keras.layers.Activation(\"elu\"), \n keras.layers.Dense(10, activation=\"softmax\") \n])\nThe BatchNormalization class has quite a few hyperparameters you can\ntweak. The defaults will usually be fine, but you may occasionally need to\ntweak the momentum. This hyperparameter is used by the\nBatchNormalization layer when it updates the exponential moving\naverages; given a new value v (i.e., a new vector of input means or\nstandard deviations computed over the current batch), the layer updates\nthe running average ˆv using the following equation:\nˆv ←ˆv × momentum + v × (1 −momentum)\nA good momentum value is typically close to 1; for example, 0.9, 0.99, or\n0.999 (you want more 9s for larger datasets and smaller mini-batches).\nAnother important hyperparameter is axis: it determines which axis\nshould be normalized. It defaults to –1, meaning that by default it will\nnormalize the last axis (using the means and standard deviations computed\nacross the other axes). When the input batch is 2D (i.e., the batch shape is\n[batch size, features]), this means that each input feature will be\nnormalized based on the mean and standard deviation computed across all\nthe instances in the batch. For example, the first BN layer in the previous\ncode example will independently normalize (and rescale and shift) each of\nthe 784 input features. If we move the first BN layer before the Flatten\nlayer, then the input batches will be 3D, with shape [batch size, height,\nwidth]; therefore, the BN layer will compute 28 means and 28 standard\ndeviations (1 per column of pixels, computed across all instances in the\nbatch and across all rows in the column), and it will normalize all pixels in\na given column using the same mean and standard deviation. There will\nalso be just 28 scale parameters and 28 shift parameters. If instead you\nstill want to treat each of the 784 pixels independently, then you should set\naxis=[1, 2].\nNotice that the BN layer does not perform the same computation during\ntraining and after training: it uses batch statistics during training and the\n“final” statistics after training (i.e., the final values of the moving\naverages). Let’s take a peek at the source code of this class to see how this\nis handled:\nclass BatchNormalization(keras.layers.Layer): \n [...] \n def call(self, inputs, training=None): \n [...]\nThe call() method is the one that performs the computations; as you can\nsee, it has an extra training argument, which is set to None by default,\nbut the fit() method sets to it to 1 during training. If you ever need to\nwrite a custom layer, and it must behave differently during training and\ntesting, add a training argument to the call() method and use this\nargument in the method to decide what to compute\n (we will discuss\ncustom layers in Chapter 12).\nBatchNormalization has become one of the most-used layers in deep\nneural networks, to the point that it is often omitted in the diagrams, as it\nis assumed that BN is added after every layer. But a recent paper by\nHongyi Zhang et al. may change this assumption: by using a novel fixed-\nupdate (fixup) weight initialization technique, the authors managed to\ntrain a very deep neural network (10,000 layers!) without BN, achieving\nstate-of-the-art performance on complex image classification tasks. As\nthis is bleeding-edge research, however, you may want to wait for\nadditional research to confirm this finding before you drop Batch\nNormalization.\nGradient Clipping\n10\n11\nAnother popular technique to mitigate the exploding gradients problem is\nto clip the gradients during backpropagation so that they never exceed\nsome threshold. This is called Gradient Clipping.\n This technique is most\noften used in recurrent neural networks, as Batch Normalization is tricky\nto use in RNNs, as we will see in Chapter 15. For other types of networks,\nBN is usually sufficient.\nIn Keras, implementing Gradient Clipping is just a matter of setting the\nclipvalue or clipnorm argument when creating an optimizer, like this:\noptimizer = keras.optimizers.SGD(clipvalue=1.0) \nmodel.compile(loss=\"mse\", optimizer=optimizer)\nThis optimizer will clip every component of the gradient vector to a value\nbetween –1.0 and 1.0. This means that all the partial derivatives of the loss\n(with regard to each and every trainable parameter) will be clipped\nbetween –1.0 and 1.0. The threshold is a hyperparameter you can tune.\nNote that it may change the orientation of the gradient vector. For\ninstance, if the original gradient vector is [0.9, 100.0], it points mostly in\nthe direction of the second axis; but once you clip it by value, you get [0.9,\n1.0], which points roughly in the diagonal between the two axes. In\npractice, this approach works well. If you want to ensure that Gradient\nClipping does not change the direction of the gradient vector, you should\nclip by norm by setting clipnorm instead of clipvalue. This will clip the\nwhole gradient if its ℓ norm is greater than the threshold you picked. For\nexample, if you set clipnorm=1.0, then the vector [0.9, 100.0] will be\nclipped to [0.00899964, 0.9999595], preserving its orientation but almost\neliminating the first component. If you observe that the gradients explode\nduring training (you can track the size of the gradients using\nTensorBoard), you may want to try both clipping by value and clipping by\nnorm, with different thresholds, and see which option performs best on the\nvalidation set.\nReusing Pretrained Layers\n12\n2\nIt is generally not a good idea to train a very large DNN from scratch:\ninstead, you should always try to find an existing neural network that\naccomplishes a similar task to the one you are trying to tackle (we will\ndiscuss how to find them in Chapter 14), then reuse the lower layers of this\nnetwork. This technique is called transfer learning. It will not only speed\nup training considerably, but also require significantly less training data.\nSuppose you have access to a DNN that was trained to classify pictures\ninto 100 different categories, including animals, plants, vehicles, and\neveryday objects. You now want to train a DNN to classify specific types\nof vehicles. These tasks are very similar, even partly overlapping, so you\nshould try to reuse parts of the first network (see Figure 11-4).\nFigure 11-4. Reusing pretrained layers\nNOTE\nIf the input pictures of your new task don’t have the same size as the ones used in\nthe original task, you will usually have to add a preprocessing step to resize them to\nthe size expected by the original model. More generally, transfer learning will work\nbest when the inputs have similar low-level features.\nThe output layer of the original model should usually be replaced because\nit is most likely not useful at all for the new task, and it may not even have\nthe right number of outputs for the new task.\nSimilarly, the upper hidden layers of the original model are less likely to\nbe as useful as the lower layers, since the high-level features that are most\nuseful for the new task may differ significantly from the ones that were\nmost useful for the original task. You want to find the right number of\nlayers to reuse.\nTIP\nThe more similar the tasks are, the more layers you want to reuse (starting with the\nlower layers). For very similar tasks, try keeping all the hidden layers and just\nreplacing the output layer.\nTry freezing all the reused layers first (i.e., make their weights non-\ntrainable so that Gradient Descent won’t modify them), then train your\nmodel and see how it performs. Then try unfreezing one or two of the top\nhidden layers to let backpropagation tweak them and see if performance\nimproves. The more training data you have, the more layers you can\nunfreeze. It is also useful to reduce the learning rate when you unfreeze\nreused layers: this will avoid wrecking their fine-tuned weights.\nIf you still cannot get good performance, and you have little training data,\ntry dropping the top hidden layer(s) and freezing all the remaining hidden\nlayers again. You can iterate until you find the right number of layers to\nreuse. If you have plenty of training data, you may try replacing the top\nhidden layers instead of dropping them, and even adding more hidden\nlayers.\nTransfer Learning with Keras\nLet’s look at an example. Suppose the Fashion MNIST dataset only\ncontained eight classes—for example, all the classes except for sandal and\nshirt. Someone built and trained a Keras model on that set and got\nreasonably good performance (>90% accuracy). Let’s call this model A.\nYou now want to tackle a different task: you have images of sandals and\nshirts, and you want to train a binary classifier (positive=shirt,\nnegative=sandal). Your dataset is quite small; you only have 200 labeled\nimages. When you train a new model for this task (let’s call it model B)\nwith the same architecture as model A, it performs reasonably well (97.2%\naccuracy). But since it’s a much easier task (there are just two classes),\nyou were hoping for more. While drinking your morning coffee, you\nrealize that your task is quite similar to task A, so perhaps transfer\nlearning can help? Let’s find out!\nFirst, you need to load model A and create a new model based on that\nmodel’s layers. Let’s reuse all the layers except for the output layer:\nmodel_A = keras.models.load_model(\"my_model_A.h5\") \nmodel_B_on_A = keras.models.Sequential(model_A.layers[:-1]) \nmodel_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))\nNote that model_A and model_B_on_A now share some layers. When you\ntrain model_B_on_A, it will also affect model_A. If you want to avoid that,\nyou need to clone model_A before you reuse its layers. To do this, you\nclone model A’s architecture with clone.model(), then copy its weights\n(since clone_model() does not clone the weights):\nmodel_A_clone = keras.models.clone_model(model_A) \nmodel_A_clone.set_weights(model_A.get_weights())\nNow you could train model_B_on_A for task B, but since the new output\nlayer was initialized randomly it will make large errors (at least during the\nfirst few epochs), so there will be large error gradients that may wreck the\nreused weights. To avoid this, one approach is to freeze the reused layers\nduring the first few epochs, giving the new layer some time to learn\nreasonable weights. To do this, set every layer’s trainable attribute to\nFalse and compile the model:\nfor layer in model_B_on_A.layers[:-1]: \n layer.trainable = False \n \nmodel_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=\"sgd\", \n metrics=[\"accuracy\"])\nNOTE\nYou must always compile your model after you freeze or unfreeze layers.\nNow you can train the model for a few epochs, then unfreeze the reused\nlayers (which requires compiling the model again) and continue training to\nfine-tune the reused layers for task B. After unfreezing the reused layers, it\nis usually a good idea to reduce the learning rate, once again to avoid\ndamaging the reused weights:\nhistory = model_B_on_A.fit(X_train_B, y_train_B, epochs=4, \n validation_data=(X_valid_B, y_valid_B)) \n \nfor layer in model_B_on_A.layers[:-1]: \n layer.trainable = True \n \noptimizer = keras.optimizers.SGD(lr=1e-4) # the default lr is 1e-2 \nmodel_B_on_A.compile(loss=\"binary_crossentropy\", optimizer=optimizer, \n metrics=[\"accuracy\"]) \nhistory = model_B_on_A.fit(X_train_B, y_train_B, epochs=16, \n validation_data=(X_valid_B, y_valid_B))\nSo, what’s the final verdict? Well, this model’s test accuracy is 99.25%,\nwhich means that transfer learning reduced the error rate from 2.8% down\nto almost 0.7%! That’s a factor of four!\n>>> model_B_on_A.evaluate(X_test_B, y_test_B) \n[0.06887910133600235, 0.9925]\nAre you convinced? You shouldn’t be: I cheated! I tried many\nconfigurations until I found one that demonstrated a strong improvement.\nIf you try to change the classes or the random seed, you will see that the\nimprovement generally drops, or even vanishes or reverses. What I did is\ncalled “torturing the data until it confesses.” When a paper just looks too\npositive, you should be suspicious: perhaps the flashy new technique does\nnot actually help much (in fact, it may even degrade performance), but the\nauthors tried many variants and reported only the best results (which may\nbe due to sheer luck), without mentioning how many failures they\nencountered on the way. Most of the time, this is not malicious at all, but it\nis part of the reason so many results in science can never be reproduced.\nWhy did I cheat? It turns out that transfer learning does not work very well\nwith small dense networks, presumably because small networks learn few\npatterns, and dense networks learn very specific patterns, which are\nunlikely to be useful in other tasks. Transfer learning works best with deep\nconvolutional neural networks, which tend to learn feature detectors that\nare much more general (especially in the lower layers). We will revisit\ntransfer learning in Chapter 14, using the techniques we just discussed\n(and this time there will be no cheating, I promise!).\nUnsupervised Pretraining\nSuppose you want to tackle a complex task for which you don’t have much\nlabeled training data, but unfortunately you cannot find a model trained on\na similar task. Don’t lose hope! First, you should try to gather more\nlabeled training data, but if you can’t, you may still be able to perform\nunsupervised pretraining (see Figure 11-5). Indeed, it is often cheap to\ngather unlabeled training examples, but expensive to label them. If you\ncan gather plenty of unlabeled training data, you can try to use it to train\nan unsupervised model, such as an autoencoder or a generative adversarial\nnetwork (see Chapter 17). Then you can reuse the lower layers of the\nautoencoder or the lower layers of the GAN’s discriminator, add the output\nlayer for your task on top, and fine-tune the final network using supervised\nlearning (i.e., with the labeled training examples).\nIt is this technique that Geoffrey Hinton and his team used in 2006 and\nwhich led to the revival of neural networks and the success of Deep\nLearning. Until 2010, unsupervised pretraining—typically with restricted\nBoltzmann machines (RBMs; see Appendix E)—was the norm for deep\nnets, and only after the vanishing gradients problem was alleviated did it\nbecome much more common to train DNNs purely using supervised\nlearning. Unsupervised pretraining (today typically using autoencoders or\nGANs rather than RBMs) is still a good option when you have a complex\ntask to solve, no similar model you can reuse, and little labeled training\ndata but plenty of unlabeled training data.\nNote that in the early days of Deep Learning it was difficult to train deep\nmodels, so people would use a technique called greedy layer-wise\npretraining (depicted in Figure 11-5). They would first train an\nunsupervised model with a single layer, typically an RBM, then they\nwould freeze that layer and add another one on top of it, then train the\nmodel again (effectively just training the new layer), then freeze the new\nlayer and add another layer on top of it, train the model again, and so on.\nNowadays, things are much simpler: people generally train the full\nunsupervised model in one shot (i.e., in Figure 11-5, just start directly at\nstep three) and use autoencoders or GANs rather than RBMs.\nFigure 11-5. In unsupervised training, a model is trained on the unlabeled data (or on all the\ndata) using an unsupervised learning technique, then it is fine-tuned for the final task on the\nlabeled data using a supervised learning technique; the unsupervised part may train one layer at\na time as shown here, or it may train the full model directly\nPretraining on an Auxiliary Task\nIf you do not have much labeled training data, one last option is to train a\nfirst neural network on an auxiliary task for which you can easily obtain or\ngenerate labeled training data, then reuse the lower layers of that network\nfor your actual task. The first neural network’s lower layers will learn\nfeature detectors that will likely be reusable by the second neural network.\nFor example, if you want to build a system to recognize faces, you may\nonly have a few pictures of each individual—clearly not enough to train a\ngood classifier. Gathering hundreds of pictures of each person would not\nbe practical. You could, however, gather a lot of pictures of random people\non the web and train a first neural network to detect whether or not two\ndifferent pictures feature the same person. Such a network would learn\ngood feature detectors for faces, so reusing its lower layers would allow\nyou to train a good face classifier that uses little training data.\nFor natural language processing (NLP) applications, you can download a\ncorpus of millions of text documents and automatically generate labeled\ndata from it. For example, you could randomly mask out some words and\ntrain a model to predict what the missing words are (e.g., it should predict\nthat the missing word in the sentence “What ___ you saying?” is probably\n“are” or “were”). If you can train a model to reach good performance on\nthis task, then it will already know quite a lot about language, and you can\ncertainly reuse it for your actual task and fine-tune it on your labeled data\n(we will discuss more pretraining tasks in Chapter 15).\nNOTE\nSelf-supervised learning is when you automatically generate the labels from the data\nitself, then you train a model on the resulting “labeled” dataset using supervised\nlearning techniques. Since this approach requires no human labeling whatsoever, it is\nbest classified as a form of unsupervised learning.\nFaster Optimizers\nTraining a very large deep neural network can be painfully slow. So far we\nhave seen four ways to speed up training (and reach a better solution):\napplying a good initialization strategy for the connection weights, using a\ngood activation function, using Batch Normalization, and reusing parts of\na pretrained network (possibly built on an auxiliary task or using\nunsupervised learning). Another huge speed boost comes from using a\nfaster optimizer than the regular Gradient Descent optimizer. In this\nsection we will present the most popular algorithms: momentum\noptimization, Nesterov Accelerated Gradient, AdaGrad, RMSProp, and\nfinally Adam and Nadam optimization.\nMomentum Optimization\nImagine a bowling ball rolling down a gentle slope on a smooth surface: it\nwill start out slowly, but it will quickly pick up momentum until it\neventually reaches terminal velocity (if there is some friction or air\nresistance). This is the very simple idea behind momentum optimization,\nproposed by Boris Polyak in 1964.\n In contrast, regular Gradient Descent\nwill simply take small, regular steps down the slope, so the algorithm will\ntake much more time to reach the bottom.\nRecall that Gradient Descent updates the weights θ by directly subtracting\nthe gradient of the cost function J(θ) with regard to the weights (∇J(θ))\nmultiplied by the learning rate η. The equation is: θ ← θ – η∇J(θ). It does\nnot care about what the earlier gradients were. If the local gradient is tiny,\nit goes very slowly.\nMomentum optimization cares a great deal about what previous gradients\nwere: at each iteration, it subtracts the local gradient from the momentum\nvector m (multiplied by the learning rate η), and it updates the weights by\nadding this momentum vector (see Equation 11-4). In other words, the\ngradient is used for acceleration, not for speed. To simulate some sort of\nfriction mechanism and prevent the momentum from growing too large,\nthe algorithm introduces a new hyperparameter β, called the momentum,\nwhich must be set between 0 (high friction) and 1 (no friction). A typical\nmomentum value is 0.9.\nEquation 11-4. Momentum algorithm\n1.\nm ←βm −η∇θJ (θ)\n2.\nθ ←θ + m\nYou can easily verify that if the gradient remains constant, the terminal\nvelocity (i.e., the maximum size of the weight updates) is equal to that\ngradient multiplied by the learning rate η multiplied by \n (ignoring the\nsign). For example, if β = 0.9, then the terminal velocity is equal to 10\ntimes the gradient times the learning rate, so momentum optimization\nends up going 10 times faster than Gradient Descent! This allows\nmomentum optimization to escape from plateaus much faster than\n13\nθ\nθ\n1\n1−β\nGradient Descent. We saw in Chapter 4 that when the inputs have very\ndifferent scales, the cost function will look like an elongated bowl (see\nFigure 4-7). Gradient Descent goes down the steep slope quite fast, but\nthen it takes a very long time to go down the valley. In contrast,\nmomentum optimization will roll down the valley faster and faster until it\nreaches the bottom (the optimum). In deep neural networks that don’t use\nBatch Normalization, the upper layers will often end up having inputs with\nvery different scales, so using momentum optimization helps a lot. It can\nalso help roll past local optima.\nNOTE\nDue to the momentum, the optimizer may overshoot a bit, then come back,\novershoot again, and oscillate like this many times before stabilizing at the\nminimum. This is one of the reasons it’s good to have a bit of friction in the system:\nit gets rid of these oscillations and thus speeds up convergence.\nImplementing momentum optimization in Keras is a no-brainer: just use\nthe SGD optimizer and set its momentum hyperparameter, then lie back and\nprofit!\noptimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)\nThe one drawback of momentum optimization is that it adds yet another\nhyperparameter to tune. However, the momentum value of 0.9 usually\nworks well in practice and almost always goes faster than regular Gradient\nDescent.\nNesterov Accelerated Gradient\nOne small variant to momentum optimization, proposed by Yurii Nesterov\nin 1983,\n is almost always faster than vanilla momentum optimization.\nThe Nesterov Accelerated Gradient (NAG) method, also known as\nNesterov momentum optimization, measures the gradient of the cost\n14\nfunction not at the local position θ but slightly ahead in the direction of\nthe momentum, at θ + βm (see Equation 11-5).\nEquation 11-5. Nesterov Accelerated Gradient algorithm\n1.\nm ←βm −η∇θJ (θ + βm)\n2.\nθ ←θ + m\nThis small tweak works because in general the momentum vector will be\npointing in the right direction (i.e., toward the optimum), so it will be\nslightly more accurate to use the gradient measured a bit farther in that\ndirection rather than the gradient at the original position, as you can see in\nFigure 11-6 (where ∇ represents the gradient of the cost function\nmeasured at the starting point θ, and ∇ represents the gradient at the point\nlocated at θ + βm).\nAs you can see, the Nesterov update ends up slightly closer to the\noptimum. After a while, these small improvements add up and NAG ends\nup being significantly faster than regular momentum optimization.\nMoreover, note that when the momentum pushes the weights across a\nvalley, ∇ continues to push farther across the valley, while ∇ pushes\nback toward the bottom of the valley. This helps reduce oscillations and\nthus NAG converges faster.\nNAG is generally faster than regular momentum optimization. To use it,\nsimply set nesterov=True when creating the SGD optimizer:\noptimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True)\n1\n2\n1\n2\nFigure 11-6. Regular versus Nesterov momentum optimization: the former applies the gradients\ncomputed before the momentum step, while the latter applies the gradients computed after\nAdaGrad\nConsider the elongated bowl problem again: Gradient Descent starts by\nquickly going down the steepest slope, which does not point straight\ntoward the global optimum, then it very slowly goes down to the bottom of\nthe valley. It would be nice if the algorithm could correct its direction\nearlier to point a bit more toward the global optimum. The AdaGrad\nalgorithm\n achieves this correction by scaling down the gradient vector\nalong the steepest dimensions (see Equation 11-6).\nEquation 11-6. AdaGrad algorithm\n1.\ns ←s + ∇θJ (θ) ⊗∇θJ (θ)\n2.\nθ ←θ −η ∇θJ (θ) ⊘√s + ε\nThe first step accumulates the square of the gradients into the vector s\n(recall that the ⊗ symbol represents the element-wise multiplication).\nThis vectorized form is equivalent to computing s ← s + (∂ J(θ) / ∂ θ )\nfor each element s of the vector s; in other words, each s accumulates the\n15\ni\ni\ni 2\ni\ni\nsquares of the partial derivative of the cost function with regard to\nparameter θ . If the cost function is steep along the i dimension, then s\nwill get larger and larger at each iteration.\nThe second step is almost identical to Gradient Descent, but with one big\ndifference: the gradient vector is scaled down by a factor of √s + ε (the ⊘\nsymbol represents the element-wise division, and ε is a smoothing term to\navoid division by zero, typically set to 10\n). This vectorized form is\nequivalent to simultaneously computing θi ←θi −η ∂J(θ)/∂θi/√si + ε\nfor all parameters θ .\nIn short, this algorithm decays the learning rate, but it does so faster for\nsteep dimensions than for dimensions with gentler slopes. This is called an\nadaptive learning rate. It helps point the resulting updates more directly\ntoward the global optimum (see Figure 11-7). One additional benefit is\nthat it requires much less tuning of the learning rate hyperparameter η.\nFigure 11-7. AdaGrad versus Gradient Descent: the former can correct its direction earlier to\npoint to the optimum\nAdaGrad frequently performs well for simple quadratic problems, but it\noften stops too early when training neural networks. The learning rate gets\nscaled down so much that the algorithm ends up stopping entirely before\nreaching the global optimum. So even though Keras has an Adagrad\ni\nth\ni\n–10\ni\noptimizer, you should not use it to train deep neural networks (it may be\nefficient for simpler tasks such as Linear Regression, though). Still,\nunderstanding AdaGrad is helpful to grasp the other adaptive learning rate\noptimizers.\nRMSProp\nAs we’ve seen, AdaGrad runs the risk of slowing down a bit too fast and\nnever converging to the global optimum. The RMSProp algorithm\n fixes\nthis by accumulating only the gradients from the most recent iterations (as\nopposed to all the gradients since the beginning of training). It does so by\nusing exponential decay in the first step (see Equation 11-7).\nEquation 11-7. RMSProp algorithm\n1.\ns ←βs + (1 −β) ∇θJ (θ) ⊗∇θJ (θ)\n2.\nθ ←θ −η ∇θJ (θ) ⊘√s + ε\nThe decay rate β is typically set to 0.9. Yes, it is once again a new\nhyperparameter, but this default value often works well, so you may not\nneed to tune it at all.\nAs you might expect, Keras has an RMSprop optimizer:\noptimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9)\nExcept on very simple problems, this optimizer almost always performs\nmuch better than AdaGrad. In fact, it was the preferred optimization\nalgorithm of many researchers until Adam optimization came around.\nAdam and Nadam Optimization\nAdam,\n which stands for adaptive moment estimation, combines the ideas\nof momentum optimization and RMSProp: just like momentum\noptimization, it keeps track of an exponentially decaying average of past\n16\n17\ngradients; and just like RMSProp, it keeps track of an exponentially\ndecaying average of past squared gradients (see Equation 11-8).\nEquation 11-8. Adam algorithm\n1.\nm ←β1m −(1 −β1) ∇θJ (θ)\n2.\ns ←β2s + (1 −β2) ∇θJ (θ) ⊗∇θJ (θ)\n3.\nˆ\nm ←\n4.\nˆs ←\n5.\nθ ←θ + η ˆ\nm ⊘√ˆs + ε\nIn this equation, t represents the iteration number (starting at 1).\nIf you just look at steps 1, 2, and 5, you will notice Adam’s close\nsimilarity to both momentum optimization and RMSProp. The only\ndifference is that step 1 computes an exponentially decaying average\nrather than an exponentially decaying sum, but these are actually\nequivalent except for a constant factor (the decaying average is just 1 – β\ntimes the decaying sum). Steps 3 and 4 are somewhat of a technical detail:\nsince m and s are initialized at 0, they will be biased toward 0 at the\nbeginning of training, so these two steps will help boost m and s at the\nbeginning of training.\nThe momentum decay hyperparameter β is typically initialized to 0.9,\nwhile the scaling decay hyperparameter β is often initialized to 0.999. As\nearlier, the smoothing term ε is usually initialized to a tiny number such as\n10\n. These are the default values for the Adam class (to be precise,\nepsilon defaults to None, which tells Keras to use\nkeras.backend.epsilon(), which defaults to 10\n; you can change it\nusing keras.backend.set_epsilon()). Here is how to create an Adam\noptimizer using Keras:\noptimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)\n18\nm\n1 −β1\n⊺\ns\n1 −β2\n⊺\n1\n1\n2\n–7\n–7\nSince Adam is an adaptive learning rate algorithm (like AdaGrad and\nRMSProp), it requires less tuning of the learning rate hyperparameter η.\nYou can often use the default value η = 0.001, making Adam even easier to\nuse than Gradient Descent.\nTIP\nIf you are starting to feel overwhelmed by all these different techniques and are\nwondering how to choose the right ones for your task, don’t worry: some practical\nguidelines are provided at the end of this chapter.\nFinally, two variants of Adam are worth mentioning:\nAdaMax\nNotice that in step 2 of Equation 11-8, Adam accumulates the squares\nof the gradients in s (with a greater weight for more recent weights). In\nstep 5, if we ignore ε and steps 3 and 4 (which are technical details\nanyway), Adam scales down the parameter updates by the square root\nof s. In short, Adam scales down the parameter updates by the ℓ norm\nof the time-decayed gradients (recall that the ℓ norm is the square\nroot of the sum of squares). AdaMax, introduced in the same paper as\nAdam, replaces the ℓ norm with the ℓ norm (a fancy way of saying\nthe max). Specifically, it replaces step 2 in Equation 11-8 with \ns ←max(β2s, ∇θJ (θ)), it drops step 4, and in step 5 it scales down\nthe gradient updates by a factor of s, which is just the max of the time-\ndecayed gradients. In practice, this can make AdaMax more stable than\nAdam, but it really depends on the dataset, and in general Adam\nperforms better. So, this is just one more optimizer you can try if you\nexperience problems with Adam on some task.\nNadam\nNadam optimization is Adam optimization plus the Nesterov trick, so\nit will often converge slightly faster than Adam. In his report\nintroducing this technique,\n the researcher Timothy Dozat compares\n2\n2\n2\n∞\n19\nmany different optimizers on various tasks and finds that Nadam\ngenerally outperforms Adam but is sometimes outperformed by\nRMSProp.\nWARNING\nAdaptive optimization methods (including RMSProp, Adam, and Nadam\noptimization) are often great, converging fast to a good solution. However, a 2017\npaper\n by Ashia C. Wilson et al. showed that they can lead to solutions that\ngeneralize poorly on some datasets. So when you are disappointed by your model’s\nperformance, try using plain Nesterov Accelerated Gradient instead: your dataset\nmay just be allergic to adaptive gradients. Also check out the latest research, because\nit’s moving fast.\nAll the optimization techniques discussed so far only rely on the first-\norder partial derivatives (Jacobians). The optimization literature also\ncontains amazing algorithms based on the second-order partial derivatives\n(the Hessians, which are the partial derivatives of the Jacobians).\nUnfortunately, these algorithms are very hard to apply to deep neural\nnetworks because there are n Hessians per output (where n is the number\nof parameters), as opposed to just n Jacobians per output. Since DNNs\ntypically have tens of thousands of parameters, the second-order\noptimization algorithms often don’t even fit in memory, and even when\nthey do, computing the Hessians is just too slow.\n20\n2\nTRAINING SPARSE MODELS\nAll the optimization algorithms just presented produce dense models,\nmeaning that most parameters will be nonzero. If you need a blazingly\nfast model at runtime, or if you need it to take up less memory, you\nmay prefer to end up with a sparse model instead.\nOne easy way to achieve this is to train the model as usual, then get rid\nof the tiny weights (set them to zero). Note that this will typically not\nlead to a very sparse model, and it may degrade the model’s\nperformance.\nA better option is to apply strong ℓ regularization during training (we\nwill see how later in this chapter), as it pushes the optimizer to zero\nout as many weights as it can (as discussed in “Lasso Regression” in\nChapter 4).\nIf these techniques remain insufficient, check out the TensorFlow\nModel Optimization Toolkit (TF-MOT), which provides a pruning API\ncapable of iteratively removing connections during training based on\ntheir magnitude.\nTable 11-2 compares all the optimizers we’ve discussed so far (* is bad, **\nis average, and *** is good).\n1\nTable 11-2. Optimizer comparison\nClass\nConvergence speed Convergence quality\nSGD\n*\n***\nSGD(momentum=...)\n**\n***\nSGD(momentum=..., nesterov=True) **\n***\nAdagrad\n***\n* (stops too early)\nRMSprop\n***\n** or ***\nAdam\n***\n** or ***\nNadam\n***\n** or ***\nAdaMax\n***\n** or ***\nLearning Rate Scheduling\nFinding a good learning rate is very important. If you set it much too high,\ntraining may diverge (as we discussed in “Gradient Descent”). If you set it\ntoo low, training will eventually converge to the optimum, but it will take\na very long time. If you set it slightly too high, it will make progress very\nquickly at first, but it will end up dancing around the optimum, never\nreally settling down. If you have a limited computing budget, you may\nhave to interrupt training before it has converged properly, yielding a\nsuboptimal solution (see Figure 11-8).\nFigure 11-8. Learning curves for various learning rates η\nAs we discussed in Chapter 10, you can find a good learning rate by\ntraining the model for a few hundred iterations, exponentially increasing\nthe learning rate from a very small value to a very large value, and then\nlooking at the learning curve and picking a learning rate slightly lower\nthan the one at which the learning curve starts shooting back up. You can\nthen reinitialize your model and train it with that learning rate.\nBut you can do better than a constant learning rate: if you start with a large\nlearning rate and then reduce it once training stops making fast progress,\nyou can reach a good solution faster than with the optimal constant\nlearning rate. There are many different strategies to reduce the learning\nrate during training. It can also be beneficial to start with a low learning\nrate, increase it, then drop it again. These strategies are called learning\nschedules (we briefly introduced this concept in Chapter 4). These are the\nmost commonly used learning schedules:\nPower scheduling\nSet the learning rate to a function of the iteration number t: η(t) = η /\n(1 + t/s) . The initial learning rate η , the power c (typically set to 1),\nand the steps s are hyperparameters. The learning rate drops at each\nstep. After s steps, it is down to η / 2. After s more steps, it is down to\nη / 3, then it goes down to η / 4, then η / 5, and so on. As you can\n0\nc\n0\n0\n0\n0\n0\nsee, this schedule first drops quickly, then more and more slowly. Of\ncourse, power scheduling requires tuning η and s (and possibly c).\nExponential scheduling\nSet the learning rate to η(t) = η 0.1 . The learning rate will gradually\ndrop by a factor of 10 every s steps. While power scheduling reduces\nthe learning rate more and more slowly, exponential scheduling keeps\nslashing it by a factor of 10 every s steps.\nPiecewise constant scheduling\nUse a constant learning rate for a number of epochs (e.g., η = 0.1 for 5\nepochs), then a smaller learning rate for another number of epochs\n(e.g., η = 0.001 for 50 epochs), and so on. Although this solution can\nwork very well, it requires fiddling around to figure out the right\nsequence of learning rates and how long to use each of them.\nPerformance scheduling\nMeasure the validation error every N steps (just like for early\nstopping), and reduce the learning rate by a factor of λ when the error\nstops dropping.\n1cycle scheduling\nContrary to the other approaches, 1cycle (introduced in a 2018 paper\nby Leslie Smith) starts by increasing the initial learning rate η ,\ngrowing linearly up to η halfway through training. Then it decreases\nthe learning rate linearly down to η again during the second half of\ntraining, finishing the last few epochs by dropping the rate down by\nseveral orders of magnitude (still linearly). The maximum learning\nrate η is chosen using the same approach we used to find the optimal\nlearning rate, and the initial learning rate η is chosen to be roughly 10\ntimes lower. When using a momentum, we start with a high\nmomentum first (e.g., 0.95), then drop it down to a lower momentum\nduring the first half of training (e.g., down to 0.85, linearly), and then\nbring it back up to the maximum value (e.g., 0.95) during the second\n0\n0\nt/s\n0\n1\n21\n0\n1\n0\n1\n0\nhalf of training, finishing the last few epochs with that maximum\nvalue. Smith did many experiments showing that this approach was\noften able to speed up training considerably and reach better\nperformance. For example, on the popular CIFAR10 image dataset, this\napproach reached 91.9% validation accuracy in just 100 epochs,\ninstead of 90.3% accuracy in 800 epochs through a standard approach\n(with the same neural network architecture).\nA 2013 paper\n by Andrew Senior et al. compared the performance of\nsome of the most popular learning schedules when using momentum\noptimization to train deep neural networks for speech recognition. The\nauthors concluded that, in this setting, both performance scheduling and\nexponential scheduling performed well. They favored exponential\nscheduling because it was easy to tune and it converged slightly faster to\nthe optimal solution (they also mentioned that it was easier to implement\nthan performance scheduling, but in Keras both options are easy). That\nsaid, the 1cycle approach seems to perform even better.\nImplementing power scheduling in Keras is the easiest option: just set the\ndecay hyperparameter when creating an optimizer:\noptimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)\nThe decay is the inverse of s (the number of steps it takes to divide the\nlearning rate by one more unit), and Keras assumes that c is equal to 1.\nExponential scheduling and piecewise scheduling are quite simple too.\nYou first need to define a function that takes the current epoch and returns\nthe learning rate. For example, let’s implement exponential scheduling:\ndef exponential_decay_fn(epoch): \n return 0.01 * 0.1**(epoch / 20)\nIf you do not want to hardcode η and s, you can create a function that\nreturns a configured function:\n22\n0\ndef exponential_decay(lr0, s): \n def exponential_decay_fn(epoch): \n return lr0 * 0.1**(epoch / s) \n return exponential_decay_fn \n \nexponential_decay_fn = exponential_decay(lr0=0.01, s=20)\nNext, create a LearningRateScheduler callback, giving it the schedule\nfunction, and pass this callback to the fit() method:\nlr_scheduler = \nkeras.callbacks.LearningRateScheduler(exponential_decay_fn) \nhistory = model.fit(X_train_scaled, y_train, [...], callbacks=\n[lr_scheduler])\nThe LearningRateScheduler will update the optimizer’s learning_rate\nattribute at the beginning of each epoch. Updating the learning rate once\nper epoch is usually enough, but if you want it to be updated more often,\nfor example at every step, you can always write your own callback (see the\n“Exponential Scheduling” section of the notebook for an example).\nUpdating the learning rate at every step makes sense if there are many\nsteps per epoch. Alternatively, you can use the\nkeras.optimizers.schedules approach, described shortly.\nThe schedule function can optionally take the current learning rate as a\nsecond argument. For example, the following schedule function multiplies\nthe previous learning rate by 0.1\n, which results in the same exponential\ndecay (except the decay now starts at the beginning of epoch 0 instead of\n1):\ndef exponential_decay_fn(epoch, lr): \n return lr * 0.1**(1 / 20)\nThis implementation relies on the optimizer’s initial learning rate\n(contrary to the previous implementation), so make sure to set it\nappropriately.\n1/20\nWhen you save a model, the optimizer and its learning rate get saved\nalong with it. This means that with this new schedule function, you could\njust load a trained model and continue training where it left off, no\nproblem. Things are not so simple if your schedule function uses the\nepoch argument, however: the epoch does not get saved, and it gets reset\nto 0 every time you call the fit() method. If you were to continue\ntraining a model where it left off, this could lead to a very large learning\nrate, which would likely damage your model’s weights. One solution is to\nmanually set the fit() method’s initial_epoch argument so the epoch\nstarts at the right value.\nFor piecewise constant scheduling, you can use a schedule function like\nthe following one (as earlier, you can define a more general function if\nyou want; see the “Piecewise Constant Scheduling” section of the\nnotebook for an example), then create a LearningRateScheduler callback\nwith this function and pass it to the fit() method, just like we did for\nexponential scheduling:\ndef piecewise_constant_fn(epoch): \n if epoch < 5: \n return 0.01 \n elif epoch < 15: \n return 0.005 \n else: \n return 0.001\nFor performance scheduling, use the ReduceLROnPlateau callback. For\nexample, if you pass the following callback to the fit() method, it will\nmultiply the learning rate by 0.5 whenever the best validation loss does\nnot improve for five consecutive epochs (other options are available;\nplease check the documentation for more details):\nlr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\nLastly, tf.keras offers an alternative way to implement learning rate\nscheduling: define the learning rate using one of the schedules available in\nkeras.optimizers.schedules, then pass this learning rate to any\noptimizer. This approach updates the learning rate at each step rather than\nat each epoch. For example, here is how to implement the same\nexponential schedule as the exponential_decay_fn() function we\ndefined earlier:\ns = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = \n32) \nlearning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1) \noptimizer = keras.optimizers.SGD(learning_rate)\nThis is nice and simple, plus when you save the model, the learning rate\nand its schedule (including its state) get saved as well. This approach,\nhowever, is not part of the Keras API; it is specific to tf.keras.\nAs for the 1cycle approach, the implementation poses no particular\ndifficulty: just create a custom callback that modifies the learning rate at\neach iteration (you can update the optimizer’s learning rate by changing\nself.model.optimizer.lr). See the “1Cycle scheduling” section of the\nnotebook for an example.\nTo sum up, exponential decay, performance scheduling, and 1cycle can\nconsiderably speed up convergence, so give them a try!\nAvoiding Overfitting Through Regularization\nWith four parameters I can fit an elephant and with five I can make him\nwiggle his trunk.\n—John von Neumann, cited by Enrico Fermi in Nature\n427\nWith thousands of parameters, you can fit the whole zoo. Deep neural\nnetworks typically have tens of thousands of parameters, sometimes even\nmillions. This gives them an incredible amount of freedom and means\nthey can fit a huge variety of complex datasets. But this great flexibility\nalso makes the network prone to overfitting the training set. We need\nregularization.\nWe already implemented one of the best regularization techniques in\nChapter 10: early stopping. Moreover, even though Batch Normalization\nwas designed to solve the unstable gradients problems, it also acts like a\npretty good regularizer. In this section we will examine other popular\nregularization techniques for neural networks: ℓ and ℓ regularization,\ndropout, and max-norm regularization.\nℓ and ℓ Regularization\nJust like you did in Chapter 4 for simple linear models, you can use ℓ\nregularization to constrain a neural network’s connection weights, and/or\nℓ regularization if you want a sparse model (with many weights equal to\n0). Here is how to apply ℓ regularization to a Keras layer’s connection\nweights, using a regularization factor of 0.01:\nlayer = keras.layers.Dense(100, activation=\"elu\", \n kernel_initializer=\"he_normal\", \n \nkernel_regularizer=keras.regularizers.l2(0.01))\nThe l2() function returns a regularizer that will be called at each step\nduring training to compute the regularization loss. This is then added to\nthe final loss. As you might expect, you can just use\nkeras.regularizers.l1() if you want ℓ regularization; if you want\nboth ℓ and ℓ regularization, use keras.regularizers.l1_l2()\n(specifying both regularization factors).\nSince you will typically want to apply the same regularizer to all layers in\nyour network, as well as using the same activation function and the same\ninitialization strategy in all hidden layers, you may find yourself repeating\nthe same arguments. This makes the code ugly and error-prone. To avoid\nthis, you can try refactoring your code to use loops. Another option is to\n1\n2\n1\n2\n2\n1\n2\n1\n1\n2\nuse Python’s functools.partial() function, which lets you create a thin\nwrapper for any callable, with some default argument values:\nfrom functools import partial \n \nRegularizedDense = partial(keras.layers.Dense, \n activation=\"elu\", \n kernel_initializer=\"he_normal\", \n \nkernel_regularizer=keras.regularizers.l2(0.01)) \n \nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n RegularizedDense(300), \n RegularizedDense(100), \n RegularizedDense(10, activation=\"softmax\", \n kernel_initializer=\"glorot_uniform\") \n])\nDropout\nDropout is one of the most popular regularization techniques for deep\nneural networks. It was proposed in a paper\n by Geoffrey Hinton in 2012\nand further detailed in a 2014 paper\n by Nitish Srivastava et al., and it has\nproven to be highly successful: even the state-of-the-art neural networks\nget a 1–2% accuracy boost simply by adding dropout. This may not sound\nlike a lot, but when a model already has 95% accuracy, getting a 2%\naccuracy boost means dropping the error rate by almost 40% (going from\n5% error to roughly 3%).\nIt is a fairly simple algorithm: at every training step, every neuron\n(including the input neurons, but always excluding the output neurons) has\na probability p of being temporarily “dropped out,” meaning it will be\nentirely ignored during this training step, but it may be active during the\nnext step (see Figure 11-9). The hyperparameter p is called the dropout\nrate, and it is typically set between 10% and 50%: closer to 20–30% in\nrecurrent neural nets (see Chapter 15), and closer to 40–50% in\nconvolutional neural networks (see Chapter 14). After training, neurons\n23\n24\ndon’t get dropped anymore. And that’s all (except for a technical detail we\nwill discuss momentarily).\nFigure 11-9. With dropout regularization, at each training iteration a random subset of all\nneurons in one or more layers—except the output layer—are “dropped out”; these neurons\noutput 0 at this iteration (represented by the dashed arrows)\nIt’s surprising at first that this destructive technique works at all. Would a\ncompany perform better if its employees were told to toss a coin every\nmorning to decide whether or not to go to work? Well, who knows;\nperhaps it would! The company would be forced to adapt its organization;\nit could not rely on any single person to work the coffee machine or\nperform any other critical tasks, so this expertise would have to be spread\nacross several people. Employees would have to learn to cooperate with\nmany of their coworkers, not just a handful of them. The company would\nbecome much more resilient. If one person quit, it wouldn’t make much of\na difference. It’s unclear whether this idea would actually work for\ncompanies, but it certainly does for neural networks. Neurons trained with\ndropout cannot co-adapt with their neighboring neurons; they have to be as\nuseful as possible on their own. They also cannot rely excessively on just a\nfew input neurons; they must pay attention to each of their input neurons.\nThey end up being less sensitive to slight changes in the inputs. In the end,\nyou get a more robust network that generalizes better.\nAnother way to understand the power of dropout is to realize that a unique\nneural network is generated at each training step. Since each neuron can be\neither present or absent, there are a total of 2 possible networks (where N\nis the total number of droppable neurons). This is such a huge number that\nit is virtually impossible for the same neural network to be sampled twice.\nOnce you have run 10,000 training steps, you have essentially trained\n10,000 different neural networks (each with just one training instance).\nThese neural networks are obviously not independent because they share\nmany of their weights, but they are nevertheless all different. The resulting\nneural network can be seen as an averaging ensemble of all these smaller\nneural networks.\nTIP\nIn practice, you can usually apply dropout only to the neurons in the top one to three\nlayers (excluding the output layer).\nThere is one small but important technical detail. Suppose p = 50%, in\nwhich case during testing a neuron would be connected to twice as many\ninput neurons as it would be (on average) during training. To compensate\nfor this fact, we need to multiply each neuron’s input connection weights\nby 0.5 after training. If we don’t, each neuron will get a total input signal\nroughly twice as large as what the network was trained on and will be\nunlikely to perform well. More generally, we need to multiply each input\nconnection weight by the keep probability (1 – p) after training.\nAlternatively, we can divide each neuron’s output by the keep probability\nduring training (these alternatives are not perfectly equivalent, but they\nwork equally well).\nN\nTo implement dropout using Keras, you can use the\nkeras.layers.Dropout layer. During training, it randomly drops some\ninputs (setting them to 0) and divides the remaining inputs by the keep\nprobability. After training, it does nothing at all; it just passes the inputs to\nthe next layer. The following code applies dropout regularization before\nevery Dense layer, using a dropout rate of 0.2:\nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dropout(rate=0.2), \n keras.layers.Dense(300, activation=\"elu\", \nkernel_initializer=\"he_normal\"), \n keras.layers.Dropout(rate=0.2), \n keras.layers.Dense(100, activation=\"elu\", \nkernel_initializer=\"he_normal\"), \n keras.layers.Dropout(rate=0.2), \n keras.layers.Dense(10, activation=\"softmax\") \n])\nWARNING\nSince dropout is only active during training, comparing the training loss and the\nvalidation loss can be misleading. In particular, a model may be overfitting the\ntraining set and yet have similar training and validation losses. So make sure to\nevaluate the training loss without dropout (e.g., after training).\nIf you observe that the model is overfitting, you can increase the dropout\nrate. Conversely, you should try decreasing the dropout rate if the model\nunderfits the training set. It can also help to increase the dropout rate for\nlarge layers, and reduce it for small ones. Moreover, many state-of-the-art\narchitectures only use dropout after the last hidden layer, so you may want\nto try this if full dropout is too strong.\nDropout does tend to significantly slow down convergence, but it usually\nresults in a much better model when tuned properly. So, it is generally well\nworth the extra time and effort.\nTIP\nIf you want to regularize a self-normalizing network based on the SELU activation\nfunction (as discussed earlier), you should use alpha dropout: this is a variant of\ndropout that preserves the mean and standard deviation of its inputs (it was\nintroduced in the same paper as SELU, as regular dropout would break self-\nnormalization).\nMonte Carlo (MC) Dropout\nIn 2016, a paper\n by Yarin Gal and Zoubin Ghahramani added a few more\ngood reasons to use dropout:\nFirst, the paper established a profound connection between\ndropout networks (i.e., neural networks containing a Dropout\nlayer before every weight layer) and approximate Bayesian\ninference,\n giving dropout a solid mathematical justification.\nSecond, the authors introduced a powerful technique called MC\nDropout, which can boost the performance of any trained dropout\nmodel without having to retrain it or even modify it at all,\nprovides a much better measure of the model’s uncertainty, and is\nalso amazingly simple to implement.\nIf this all sounds like a “one weird trick” advertisement, then take a look\nat the following code. It is the full implementation of MC Dropout,\nboosting the dropout model we trained earlier without retraining it:\ny_probas = np.stack([model(X_test_scaled, training=True) \n for sample in range(100)]) \ny_proba = y_probas.mean(axis=0)\nWe just make 100 predictions over the test set, setting training=True to\nensure that the Dropout layer is active, and stack the predictions. Since\ndropout is active, all the predictions will be different. Recall that\npredict() returns a matrix with one row per instance and one column per\nclass. Because there are 10,000 instances in the test set and 10 classes, this\n25\n26\nis a matrix of shape [10000, 10]. We stack 100 such matrices, so y_probas\nis an array of shape [100, 10000, 10]. Once we average over the first\ndimension (axis=0), we get y_proba, an array of shape [10000, 10], like\nwe would get with a single prediction. That’s all! Averaging over multiple\npredictions with dropout on gives us a Monte Carlo estimate that is\ngenerally more reliable than the result of a single prediction with dropout\noff. For example, let’s look at the model’s prediction for the first instance\nin the test set, with dropout off:\n>>> np.round(model.predict(X_test_scaled[:1]), 2) \narray([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]], \n dtype=float32)\nThe model seems almost certain that this image belongs to class 9 (ankle\nboot). Should you trust it? Is there really so little room for doubt?\nCompare this with the predictions made when dropout is activated:\n>>> np.round(y_probas[:, :1], 2) \narray([[[0. , 0. , 0. , 0. , 0. , 0.14, 0. , 0.17, 0. , 0.68]], \n [[0. , 0. , 0. , 0. , 0. , 0.16, 0. , 0.2 , 0. , 0.64]], \n [[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.01, 0. , 0.97]], \n [...]\nThis tells a very different story: apparently, when we activate dropout, the\nmodel is not sure anymore. It still seems to prefer class 9, but sometimes\nit hesitates with classes 5 (sandal) and 7 (sneaker), which makes sense\ngiven they’re all footwear. Once we average over the first dimension, we\nget the following MC Dropout predictions:\n>>> np.round(y_proba[:1], 2) \narray([[0. , 0. , 0. , 0. , 0. , 0.22, 0. , 0.16, 0. , 0.62]], \n dtype=float32)\nThe model still thinks this image belongs to class 9, but only with a 62%\nconfidence, which seems much more reasonable than 99%. Plus it’s useful\nto know exactly which other classes it thinks are likely. And you can also\ntake a look at the standard deviation of the probability estimates:\n>>> y_std = y_probas.std(axis=0) \n>>> np.round(y_std[:1], 2) \narray([[0. , 0. , 0. , 0. , 0. , 0.28, 0. , 0.21, 0.02, 0.32]], \n dtype=float32)\nApparently there’s quite a lot of variance in the probability estimates: if\nyou were building a risk-sensitive system (e.g., a medical or financial\nsystem), you should probably treat such an uncertain prediction with\nextreme caution. You definitely would not treat it like a 99% confident\nprediction. Moreover, the model’s accuracy got a small boost from 86.8 to\n86.9:\n>>> accuracy = np.sum(y_pred == y_test) / len(y_test) \n>>> accuracy \n0.8694\nNOTE\nThe number of Monte Carlo samples you use (100 in this example) is a\nhyperparameter you can tweak. The higher it is, the more accurate the predictions\nand their uncertainty estimates will be. However, if you double it, inference time will\nalso be doubled. Moreover, above a certain number of samples, you will notice little\nimprovement. So your job is to find the right trade-off between latency and\naccuracy, depending on your application.\nIf your model contains other layers that behave in a special way during\ntraining (such as BatchNormalization layers), then you should not force\ntraining mode like we just did. Instead, you should replace the Dropout\nlayers with the following MCDropout class:\nclass MCDropout(keras.layers.Dropout): \n def call(self, inputs): \n return super().call(inputs, training=True)\nHere, we just subclass the Dropout layer and override the call() method\nto force its training argument to True (see Chapter 12). Similarly, you\ncould define an MCAlphaDropout class by subclassing AlphaDropout\n27\ninstead. If you are creating a model from scratch, it’s just a matter of using\nMCDropout rather than Dropout. But if you have a model that was already\ntrained using Dropout, you need to create a new model that’s identical to\nthe existing model except that it replaces the Dropout layers with\nMCDropout, then copy the existing model’s weights to your new model.\nIn short, MC Dropout is a fantastic technique that boosts dropout models\nand provides better uncertainty estimates. And of course, since it is just\nregular dropout during training, it also acts like a regularizer.\nMax-Norm Regularization\nAnother regularization technique that is popular for neural networks is\ncalled max-norm regularization: for each neuron, it constrains the weights\nw of the incoming connections such that ∥ w ∥ ≤ r, where r is the max-\nnorm hyperparameter and ∥ · ∥ is the ℓ norm.\nMax-norm regularization does not add a regularization loss term to the\noverall loss function. Instead, it is typically implemented by computing\n∥w∥ after each training step and rescaling w if needed (w ←w\n).\nReducing r increases the amount of regularization and helps reduce\noverfitting. Max-norm regularization can also help alleviate the unstable\ngradients problems (if you are not using Batch Normalization).\nTo implement max-norm regularization in Keras, set the\nkernel_constraint argument of each hidden layer to a max_norm()\nconstraint with the appropriate max value, like this:\nkeras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\", \n kernel_constraint=keras.constraints.max_norm(1.))\nAfter each training iteration, the model’s fit() method will call the object\nreturned by max_norm(), passing it the layer’s weights and getting\nrescaled weights in return, which then replace the layer’s weights. As\nyou’ll see in Chapter 12, you can define your own custom constraint\n2\n2\n2\n2\nr\n∥w ∥2\nfunction if necessary and use it as the kernel_constraint. You can also\nconstrain the bias terms by setting the bias_constraint argument.\nThe max_norm() function has an axis argument that defaults to 0. A\nDense layer usually has weights of shape [number of inputs, number of\nneurons], so using axis=0 means that the max-norm constraint will apply\nindependently to each neuron’s weight vector. If you want to use max-\nnorm with convolutional layers (see Chapter 14), make sure to set the\nmax_norm() constraint’s axis argument appropriately (usually axis=[0,\n1, 2]).\nSummary and Practical Guidelines\nIn this chapter we have covered a wide range of techniques, and you may\nbe wondering which ones you should use. This depends on the task, and\nthere is no clear consensus yet, but I have found the configuration in\nTable 11-3 to work fine in most cases, without requiring much\nhyperparameter tuning. That said, please do not consider these defaults as\nhard rules!\nTable 11-3. Default DNN configuration\nHyperparameter\nDefault value\nKernel initializer\nHe initialization\nActivation function\nELU\nNormalization\nNone if shallow; Batch Norm if deep\nRegularization\nEarly stopping (+ℓ reg. if needed)\nOptimizer\nMomentum optimization (or RMSProp or Nadam)\nLearning rate schedule 1cycle\n2\nIf the network is a simple stack of dense layers, then it can self-normalize,\nand you should use the configuration in Table 11-4 instead.\nTable 11-4. DNN configuration for a self-normalizing net\nHyperparameter\nDefault value\nKernel initializer\nLeCun initialization\nActivation function\nSELU\nNormalization\nNone (self-normalization)\nRegularization\nAlpha dropout if needed\nOptimizer\nMomentum optimization (or RMSProp or Nadam)\nLearning rate schedule 1cycle\nDon’t forget to normalize the input features! You should also try to reuse\nparts of a pretrained neural network if you can find one that solves a\nsimilar problem, or use unsupervised pretraining if you have a lot of\nunlabeled data, or use pretraining on an auxiliary task if you have a lot of\nlabeled data for a similar task.\nWhile the previous guidelines should cover most cases, here are some\nexceptions:\nIf you need a sparse model, you can use ℓ regularization (and\noptionally zero out the tiny weights after training). If you need an\neven sparser model, you can use the TensorFlow Model\nOptimization Toolkit. This will break self-normalization, so you\nshould use the default configuration in this case.\nIf you need a low-latency model (one that performs lightning-fast\npredictions), you may need to use fewer layers, fold the Batch\nNormalization layers into the previous layers, and possibly use a\nfaster activation function such as leaky ReLU or just ReLU.\nHaving a sparse model will also help. Finally, you may want to\n1\nreduce the float precision from 32 bits to 16 or even 8 bits (see\n“Deploying a Model to a Mobile or Embedded Device”). Again,\ncheck out TF-MOT.\nIf you are building a risk-sensitive application, or inference\nlatency is not very important in your application, you can use MC\nDropout to boost performance and get more reliable probability\nestimates, along with uncertainty estimates.\nWith these guidelines, you are now ready to train very deep nets! I hope\nyou are now convinced that you can go quite a long way using just Keras.\nThere may come a time, however, when you need to have even more\ncontrol; for example, to write a custom loss function or to tweak the\ntraining algorithm. For such cases you will need to use TensorFlow’s\nlower-level API, as you will see in the next chapter.\nExercises\n1. Is it OK to initialize all the weights to the same value as long as\nthat value is selected randomly using He initialization?\n2. Is it OK to initialize the bias terms to 0?\n3. Name three advantages of the SELU activation function over\nReLU.\n4. In which cases would you want to use each of the following\nactivation functions: SELU, leaky ReLU (and its variants), ReLU,\ntanh, logistic, and softmax?\n5. What may happen if you set the momentum hyperparameter too\nclose to 1 (e.g., 0.99999) when using an SGD optimizer?\n6. Name three ways you can produce a sparse model.\n7. Does dropout slow down training? Does it slow down inference\n(i.e., making predictions on new instances)? What about MC\nDropout?\n8. Practice training a deep neural network on the CIFAR10 image\ndataset:\na. Build a DNN with 20 hidden layers of 100 neurons each\n(that’s too many, but it’s the point of this exercise). Use\nHe initialization and the ELU activation function.\nb. Using Nadam optimization and early stopping, train the\nnetwork on the CIFAR10 dataset. You can load it with\nkeras.datasets.cifar10.load_ data(). The dataset is\ncomposed of 60,000 32 × 32–pixel color images (50,000\nfor training, 10,000 for testing) with 10 classes, so you’ll\nneed a softmax output layer with 10 neurons. Remember\nto search for the right learning rate each time you change\nthe model’s architecture or hyperparameters.\nc. Now try adding Batch Normalization and compare the\nlearning curves: Is it converging faster than before? Does\nit produce a better model? How does it affect training\nspeed?\nd. Try replacing Batch Normalization with SELU, and make\nthe necessary adjustements to ensure the network self-\nnormalizes (i.e., standardize the input features, use\nLeCun normal initialization, make sure the DNN contains\nonly a sequence of dense layers, etc.).\ne. Try regularizing the model with alpha dropout. Then,\nwithout retraining your model, see if you can achieve\nbetter accuracy using MC Dropout.\nf. Retrain your model using 1cycle scheduling and see if it\nimproves training speed and model accuracy.\nSolutions to these exercises are available in Appendix A.\n1 Xavier Glorot and Yoshua Bengio, “Understanding the Difficulty of Training Deep\nFeedforward Neural Networks,” Proceedings of the 13th International Conference on\nArtificial Intelligence and Statistics (2010): 249–256.\n2 Here’s an analogy: if you set a microphone amplifier’s knob too close to zero, people\nwon’t hear your voice, but if you set it too close to the max, your voice will be saturated\nand people won’t understand what you are saying. Now imagine a chain of such amplifiers:\nthey all need to be set properly in order for your voice to come out loud and clear at the\nend of the chain. Your voice has to come out of each amplifier at the same amplitude as it\ncame in.\n3 E.g., Kaiming He et al., “Delving Deep into Rectifiers: Surpassing Human-Level\nPerformance on ImageNet Classification,” Proceedings of the 2015 IEEE International\nConference on Computer Vision (2015): 1026–1034.\n4 Unless it is part of the first hidden layer, a dead neuron may sometimes come back to life:\nGradient Descent may indeed tweak neurons in the layers below in such a way that the\nweighted sum of the dead neuron’s inputs is positive again.\n5 Bing Xu et al., “Empirical Evaluation of Rectified Activations in Convolutional Network,”\narXiv preprint arXiv:1505.00853 (2015).\n6 Djork-Arné Clevert et al., “Fast and Accurate Deep Network Learning by Exponential\nLinear Units (ELUs),” Proceedings of the International Conference on Learning\nRepresentations (2016).\n7 Günter Klambauer et al., “Self-Normalizing Neural Networks,” Proceedings of the 31st\nInternational Conference on Neural Information Processing Systems (2017): 972–981.\n8 Sergey Ioffe and Christian Szegedy, “Batch Normalization: Accelerating Deep Network\nTraining by Reducing Internal Covariate Shift,” Proceedings of the 32nd International\nConference on Machine Learning (2015): 448–456.\n9 However, they are estimated during training, based on the training data, so arguably they\nare trainable. In Keras, “non-trainable” really means “untouched by backpropagation.”\n10 The Keras API also specifies a keras.backend.learning_phase() function that should\nreturn 1 during training and 0 otherwise.\n11 Hongyi Zhang et al., “Fixup Initialization: Residual Learning Without Normalization,”\narXiv preprint arXiv:1901.09321 (2019).\n12 Razvan Pascanu et al., “On the Difficulty of Training Recurrent Neural Networks,”\nProceedings of the 30th International Conference on Machine Learning (2013): 1310–\n1318.\n13 Boris T. Polyak, “Some Methods of Speeding Up the Convergence of Iteration Methods,”\nUSSR Computational Mathematics and Mathematical Physics 4, no. 5 (1964): 1–17.\n14 Yurii Nesterov, “A Method for Unconstrained Convex Minimization Problem with the\nRate of Convergence O(1/k\n2 ),” Doklady AN USSR 269 (1983): 543–547.\n15 John Duchi et al., “Adaptive Subgradient Methods for Online Learning and Stochastic\nOptimization,” Journal of Machine Learning Research 12 (2011): 2121��2159.\n16 This algorithm was created by Geoffrey Hinton and Tijmen Tieleman in 2012 and\npresented by Geoffrey Hinton in his Coursera class on neural networks (slides:\nhttps://homl.info/57; video: https://homl.info/58). Amusingly, since the authors did not write\na paper to describe the algorithm, researchers often cite “slide 29 in lecture 6” in their\npapers.\n17 Diederik P. Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization,” arXiv\npreprint arXiv:1412.6980 (2014).\n18 These are estimations of the mean and (uncentered) variance of the gradients. The mean is\noften called the first moment while the variance is often called the second moment, hence\nthe name of the algorithm.\n19 Timothy Dozat, “Incorporating Nesterov Momentum into Adam” (2016).\n20 Ashia C. Wilson et al., “The Marginal Value of Adaptive Gradient Methods in Machine\nLearning,” Advances in Neural Information Processing Systems 30 (2017): 4148–4158.\n21 Leslie N. Smith, “A Disciplined Approach to Neural Network Hyper-Parameters: Part 1—\nLearning Rate, Batch Size, Momentum, and Weight Decay,” arXiv preprint\narXiv:1803.09820 (2018).\n22 Andrew Senior et al., “An Empirical Study of Learning Rates in Deep Neural Networks for\nSpeech Recognition,” Proceedings of the IEEE International Conference on Acoustics,\nSpeech, and Signal Processing (2013): 6724–6728.\n23 Geoffrey E. Hinton et al., “Improving Neural Networks by Preventing Co-Adaptation of\nFeature Detectors,” arXiv preprint arXiv:1207.0580 (2012).\n24 Nitish Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from\nOverfitting,” Journal of Machine Learning Research 15 (2014): 1929–1958.\n25 Yarin Gal and Zoubin Ghahramani, “Dropout as a Bayesian Approximation: Representing\nModel Uncertainty in Deep Learning,” Proceedings of the 33rd International Conference\non Machine Learning (2016): 1050–1059.\n26 Specifically, they show that training a dropout network is mathematically equivalent to\napproximate Bayesian inference in a specific type of probabilistic model called a Deep\nGaussian Process.\n27 This MCDropout class will work with all Keras APIs, including the Sequential API. If you\nonly care about the Functional API or the Subclassing API, you do not have to create an\nMCDropout class; you can create a regular Dropout layer and call it with training=True." + }, + { + "chapter": 12, + "text": "Chapter 12. Custom Models and\nTraining with TensorFlow\nUp until now, we’ve used only TensorFlow’s high-level API, tf.keras, but it\nalready got us pretty far: we built various neural network architectures,\nincluding regression and classification nets, Wide & Deep nets, and self-\nnormalizing nets, using all sorts of techniques, such as Batch\nNormalization, dropout, and learning rate schedules. In fact, 95% of the\nuse cases you will encounter will not require anything other than tf.keras\n(and tf.data; see Chapter 13). But now it’s time to dive deeper into\nTensorFlow and take a look at its lower-level Python API. This will be\nuseful when you need extra control to write custom loss functions, custom\nmetrics, layers, models, initializers, regularizers, weight constraints, and\nmore. You may even need to fully control the training loop itself, for\nexample to apply special transformations or constraints to the gradients\n(beyond just clipping them) or to use multiple optimizers for different\nparts of the network. We will cover all these cases in this chapter, and we\nwill also look at how you can boost your custom models and training\nalgorithms using TensorFlow’s automatic graph generation feature. But\nfirst, let’s take a quick tour of TensorFlow.\nNOTE\nTensorFlow 2.0 (beta) was released in June 2019, making TensorFlow much easier to\nuse. The first edition of this book used TF 1, while this edition uses TF 2.\nA Quick Tour of TensorFlow\nAs you know, TensorFlow is a powerful library for numerical computation,\nparticularly well suited and fine-tuned for large-scale Machine Learning\n(but you could use it for anything else that requires heavy computations).\nIt was developed by the Google Brain team and it powers many of\nGoogle’s large-scale services, such as Google Cloud Speech, Google\nPhotos, and Google Search. It was open sourced in November 2015, and it\nis now the most popular Deep Learning library (in terms of citations in\npapers, adoption in companies, stars on GitHub, etc.). Countless projects\nuse TensorFlow for all sorts of Machine Learning tasks, such as image\nclassification, natural language processing, recommender systems, and\ntime series forecasting.\nSo what does TensorFlow offer? Here’s a summary:\nIts core is very similar to NumPy, but with GPU support.\nIt supports distributed computing (across multiple devices and\nservers).\nIt includes a kind of just-in-time (JIT) compiler that allows it to\noptimize computations for speed and memory usage. It works by\nextracting the computation graph from a Python function, then\noptimizing it (e.g., by pruning unused nodes), and finally running\nit efficiently (e.g., by automatically running independent\noperations in parallel).\nComputation graphs can be exported to a portable format, so you\ncan train a TensorFlow model in one environment (e.g., using\nPython on Linux) and run it in another (e.g., using Java on an\nAndroid device).\nIt implements autodiff (see Chapter 10 and Appendix D) and\nprovides some excellent optimizers, such as RMSProp and Nadam\n(see Chapter 11), so you can easily minimize all sorts of loss\nfunctions.\nTensorFlow offers many more features built on top of these core features:\nthe most important is of course tf.keras, but it also has data loading and\npreprocessing ops (tf.data, tf.io, etc.), image processing ops\n1\n(tf.image), signal processing ops (tf.signal), and more (see Figure 12-\n1 for an overview of TensorFlow’s Python API).\nTIP\nWe will cover many of the packages and functions of the TensorFlow API, but it’s\nimpossible to cover them all, so you should really take some time to browse through\nthe API; you will find that it is quite rich and well documented.\nFigure 12-1. TensorFlow’s Python API\nAt the lowest level, each TensorFlow operation (op for short) is\nimplemented using highly efficient C++ code. Many operations have\nmultiple implementations called kernels: each kernel is dedicated to a\nspecific device type, such as CPUs, GPUs, or even TPUs (tensor\nprocessing units). As you may know, GPUs can dramatically speed up\ncomputations by splitting them into many smaller chunks and running\nthem in parallel across many GPU threads. TPUs are even faster: they are\n2\ncustom ASIC chips built specifically for Deep Learning operations (we\nwill discuss how to use TensorFlow with GPUs or TPUs in Chapter 19).\nTensorFlow’s architecture is shown in Figure 12-2. Most of the time your\ncode will use the high-level APIs (especially tf.keras and tf.data); but\nwhen you need more flexibility, you will use the lower-level Python API,\nhandling tensors directly. Note that APIs for other languages are also\navailable. In any case, TensorFlow’s execution engine will take care of\nrunning the operations efficiently, even across multiple devices and\nmachines if you tell it to.\nFigure 12-2. TensorFlow’s architecture\nTensorFlow runs not only on Windows, Linux, and macOS, but also on\nmobile devices (using TensorFlow Lite), including both iOS and Android\n(see Chapter 19). If you do not want to use the Python API, there are C++,\nJava, Go, and Swift APIs. There is even a JavaScript implementation\ncalled TensorFlow.js that makes it possible to run your models directly in\nyour browser.\nThere’s more to TensorFlow than the library. TensorFlow is at the center of\nan extensive ecosystem of libraries. First, there’s TensorBoard for\nvisualization (see Chapter 10). Next, there’s TensorFlow Extended (TFX),\n3\nwhich is a set of libraries built by Google to productionize TensorFlow\nprojects: it includes tools for data validation, preprocessing, model\nanalysis, and serving (with TF Serving; see Chapter 19). Google’s\nTensorFlow Hub provides a way to easily download and reuse pretrained\nneural networks. You can also get many neural network architectures,\nsome of them pretrained, in TensorFlow’s model garden. Check out the\nTensorFlow Resources and https://github.com/jtoy/awesome-tensorflow for\nmore TensorFlow-based projects. You will find hundreds of TensorFlow\nprojects on GitHub, so it is often easy to find existing code for whatever\nyou are trying to do.\nTIP\nMore and more ML papers are released along with their implementations, and\nsometimes even with pretrained models. Check out https://paperswithcode.com/ to\neasily find them.\nLast but not least, TensorFlow has a dedicated team of passionate and\nhelpful developers, as well as a large community contributing to\nimproving it. To ask technical questions, you should use\nhttp://stackoverflow.com/ and tag your question with tensorflow and\npython. You can file bugs and feature requests through GitHub. For general\ndiscussions, join the Google group.\nOK, it’s time to start coding!\nUsing TensorFlow like NumPy\nTensorFlow’s API revolves around tensors, which flow from operation to\noperation—hence the name TensorFlow. A tensor is usually a\nmultidimensional array (exactly like a NumPy ndarray), but it can also\nhold a scalar (a simple value, such as 42). These tensors will be important\nwhen we create custom cost functions, custom metrics, custom layers, and\nmore, so let’s see how to create and manipulate them.\nTensors and Operations\nYou can create a tensor with tf.constant(). For example, here is a tensor\nrepresenting a matrix with two rows and three columns of floats:\n>>> tf.constant([[1., 2., 3.], [4., 5., 6.]]) # matrix \n \n>>> tf.constant(42) # scalar \n\nJust like an ndarray, a tf.Tensor has a shape and a data type (dtype):\n>>> t = tf.constant([[1., 2., 3.], [4., 5., 6.]]) \n>>> t.shape \nTensorShape([2, 3]) \n>>> t.dtype \ntf.float32\nIndexing works much like in NumPy:\n>>> t[:, 1:] \n \n>>> t[..., 1, tf.newaxis] \n\nMost importantly, all sorts of tensor operations are available:\n>>> t + 10 \n \n>>> tf.square(t) \n \n>>> t @ tf.transpose(t) \n\nNote that writing t + 10 is equivalent to calling tf.add(t, 10) (indeed,\nPython calls the magic method t.__add__(10), which just calls\ntf.add(t, 10)). Other operators like - and * are also supported. The @\noperator was added in Python 3.5, for matrix multiplication: it is\nequivalent to calling the tf.matmul() function.\nYou will find all the basic math operations you need (tf.add(),\ntf.multiply(), tf.square(), tf.exp(), tf.sqrt(), etc.) and most\noperations that you can find in NumPy (e.g., tf.reshape(),\ntf.squeeze(), tf.tile()). Some functions have a different name than in\nNumPy; for instance, tf.reduce_mean(), tf.reduce_sum(),\ntf.reduce_max(), and tf.math.log() are the equivalent of np.mean(),\nnp.sum(), np.max() and np.log(). When the name differs, there is often\na good reason for it. For example, in TensorFlow you must write\ntf.transpose(t); you cannot just write t.T like in NumPy. The reason is\nthat the tf.transpose() function does not do exactly the same thing as\nNumPy’s T attribute: in TensorFlow, a new tensor is created with its own\ncopy of the transposed data, while in NumPy, t.T is just a transposed view\non the same data. Similarly, the tf.reduce_sum() operation is named this\nway because its GPU kernel (i.e., GPU implementation) uses a reduce\nalgorithm that does not guarantee the order in which the elements are\nadded: because 32-bit floats have limited precision, the result may change\never so slightly every time you call this operation. The same is true of\ntf.reduce_mean() (but of course tf.reduce_max() is deterministic).\nNOTE\nMany functions and classes have aliases. For example, tf.add() and\ntf.math.add() are the same function. This allows TensorFlow to have concise\nnames for the most common operations while preserving well-organized packages.\nKERAS’ LOW-LEVEL API\nThe Keras API has its own low-level API, located in keras.backend.\nIt includes functions like square(), exp(), and sqrt(). In tf.keras,\nthese functions generally just call the corresponding TensorFlow\noperations. If you want to write code that will be portable to other\nKeras implementations, you should use these Keras functions.\nHowever, they only cover a subset of all functions available in\nTensorFlow, so in this book we will use the TensorFlow operations\ndirectly. Here is as simple example using keras.backend, which is\ncommonly named K for short:\n>>> from tensorflow import keras \n>>> K = keras.backend \n>>> K.square(K.transpose(t)) + 10 \n\nTensors and NumPy\nTensors play nice with NumPy: you can create a tensor from a NumPy\narray, and vice versa. You can even apply TensorFlow operations to\nNumPy arrays and NumPy operations to tensors:\n>>> a = np.array([2., 4., 5.]) \n>>> tf.constant(a) \n \n4\n>>> t.numpy() # or np.array(t) \narray([[1., 2., 3.], \n [4., 5., 6.]], dtype=float32) \n>>> tf.square(a) \n \n>>> np.square(t) \narray([[ 1., 4., 9.], \n [16., 25., 36.]], dtype=float32)\nWARNING\nNotice that NumPy uses 64-bit precision by default, while TensorFlow uses 32-bit.\nThis is because 32-bit precision is generally more than enough for neural networks,\nplus it runs faster and uses less RAM. So when you create a tensor from a NumPy\narray, make sure to set dtype=tf.float32.\nType Conversions\nType conversions can significantly hurt performance, and they can easily\ngo unnoticed when they are done automatically. To avoid this, TensorFlow\ndoes not perform any type conversions automatically: it just raises an\nexception if you try to execute an operation on tensors with incompatible\ntypes. For example, you cannot add a float tensor and an integer tensor,\nand you cannot even add a 32-bit float and a 64-bit float:\n>>> tf.constant(2.) + tf.constant(40) \nTraceback[...]InvalidArgumentError[...]expected to be a float[...] \n>>> tf.constant(2.) + tf.constant(40., dtype=tf.float64) \nTraceback[...]InvalidArgumentError[...]expected to be a double[...]\nThis may be a bit annoying at first, but remember that it’s for a good\ncause! And of course you can use tf.cast() when you really need to\nconvert types:\n>>> t2 = tf.constant(40., dtype=tf.float64) \n>>> tf.constant(2.0) + tf.cast(t2, tf.float32) \n\nVariables\nThe tf.Tensor values we’ve seen so far are immutable: you cannot\nmodify them. This means that we cannot use regular tensors to implement\nweights in a neural network, since they need to be tweaked by\nbackpropagation. Plus, other parameters may also need to change over\ntime (e.g., a momentum optimizer keeps track of past gradients). What we\nneed is a tf.Variable:\n>>> v = tf.Variable([[1., 2., 3.], [4., 5., 6.]]) \n>>> v \n\nA tf.Variable acts much like a tf.Tensor: you can perform the same\noperations with it, it plays nicely with NumPy as well, and it is just as\npicky with types. But it can also be modified in place using the assign()\nmethod (or assign_add() or assign_sub(), which increment or\ndecrement the variable by the given value). You can also modify\nindividual cells (or slices), by using the cell’s (or slice’s) assign()\nmethod (direct item assignment will not work) or by using the\nscatter_update() or scatter_nd_update() methods:\nv.assign(2 * v) # => [[2., 4., 6.], [8., 10., 12.]] \nv[0, 1].assign(42) # => [[2., 42., 6.], [8., 10., 12.]] \nv[:, 2].assign([0., 1.]) # => [[2., 42., 0.], [8., 10., 1.]] \nv.scatter_nd_update(indices=[[0, 0], [1, 2]], updates=[100., 200.]) \n # => [[100., 42., 0.], [8., 10., 200.]]\nNOTE\nIn practice you will rarely have to create variables manually, since Keras provides an\nadd_weight() method that will take care of it for you, as we will see. Moreover,\nmodel parameters will generally be updated directly by the optimizers, so you will\nrarely need to update variables manually.\nOther Data Structures\nTensorFlow supports several other data structures, including the following\n(please see the “Tensors and Operations” section in the notebook or\nAppendix F for more details):\nSparse tensors (tf.SparseTensor)\nEfficiently represent tensors containing mostly zeros. The tf.sparse\npackage contains operations for sparse tensors.\nTensor arrays (tf.TensorArray)\nAre lists of tensors. They have a fixed size by default but can\noptionally be made dynamic. All tensors they contain must have the\nsame shape and data type.\nRagged tensors (tf.RaggedTensor)\nRepresent static lists of lists of tensors, where every tensor has the\nsame shape and data type. The tf.ragged package contains operations\nfor ragged tensors.\nString tensors\nAre regular tensors of type tf.string. These represent byte strings,\nnot Unicode strings, so if you create a string tensor using a Unicode\nstring (e.g., a regular Python 3 string like \"café\"), then it will get\nencoded to UTF-8 automatically (e.g., b\"caf\\xc3\\xa9\").\nAlternatively, you can represent Unicode strings using tensors of type\ntf.int32, where each item represents a Unicode code point (e.g., [99,\n97, 102, 233]). The tf.strings package (with an s) contains ops\nfor byte strings and Unicode strings (and to convert one into the other).\nIt’s important to note that a tf.string is atomic, meaning that its\nlength does not appear in the tensor’s shape. Once you convert it to a\nUnicode tensor (i.e., a tensor of type tf.int32 holding Unicode code\npoints), the length appears in the shape.\nSets\nAre represented as regular tensors (or sparse tensors). For example,\ntf.constant([[1, 2], [3, 4]]) represents the two sets {1, 2} and\n{3, 4}. More generally, each set is represented by a vector in the\ntensor’s last axis. You can manipulate sets using operations from the\ntf.sets package.\nQueues\nStore tensors across multiple steps. TensorFlow offers various kinds of\nqueues: simple First In, First Out (FIFO) queues (FIFOQueue), queues\nthat can prioritize some items (PriorityQueue), shuffle their items\n(RandomShuffleQueue), and batch items of different shapes by\npadding (PaddingFIFOQueue). These classes are all in the tf.queue\npackage.\nWith tensors, operations, variables, and various data structures at your\ndisposal, you are now ready to customize your models and training\nalgorithms!\nCustomizing Models and Training Algorithms\nLet’s start by creating a custom loss function, which is a simple and\ncommon use case.\nCustom Loss Functions\nSuppose you want to train a regression model, but your training set is a bit\nnoisy. Of course, you start by trying to clean up your dataset by removing\nor fixing the outliers, but that turns out to be insufficient; the dataset is\nstill noisy. Which loss function should you use? The mean squared error\nmight penalize large errors too much and cause your model to be\nimprecise. The mean absolute error would not penalize outliers as much,\nbut training might take a while to converge, and the trained model might\nnot be very precise. This is probably a good time to use the Huber loss\n(introduced in Chapter 10) instead of the good old MSE. The Huber loss is\nnot currently part of the official Keras API, but it is available in tf.keras\n(just use an instance of the keras.losses.Huber class). But let’s pretend\nit’s not there: implementing it is easy as pie! Just create a function that\ntakes the labels and predictions as arguments, and use TensorFlow\noperations to compute every instance’s loss:\ndef huber_fn(y_true, y_pred): \n error = y_true - y_pred \n is_small_error = tf.abs(error) < 1 \n squared_loss = tf.square(error) / 2 \n linear_loss = tf.abs(error) - 0.5 \n return tf.where(is_small_error, squared_loss, linear_loss)\nWARNING\nFor better performance, you should use a vectorized implementation, as in this\nexample. Moreover, if you want to benefit from TensorFlow’s graph features, you\nshould use only TensorFlow operations.\nIt is also preferable to return a tensor containing one loss per instance,\nrather than returning the mean loss. This way, Keras can apply class\nweights or sample weights when requested (see Chapter 10).\nNow you can use this loss when you compile the Keras model, then train\nyour model:\nmodel.compile(loss=huber_fn, optimizer=\"nadam\") \nmodel.fit(X_train, y_train, [...])\nAnd that’s it! For each batch during training, Keras will call the\nhuber_fn() function to compute the loss and use it to perform a Gradient\nDescent step. Moreover, it will keep track of the total loss since the\nbeginning of the epoch, and it will display the mean loss.\nBut what happens to this custom loss when you save the model?\nSaving and Loading Models That Contain Custom\nComponents\nSaving a model containing a custom loss function works fine, as Keras\nsaves the name of the function. Whenever you load it, you’ll need to\nprovide a dictionary that maps the function name to the actual function.\nMore generally, when you load a model containing custom objects, you\nneed to map the names to the objects:\nmodel = keras.models.load_model(\"my_model_with_a_custom_loss.h5\", \n custom_objects={\"huber_fn\": huber_fn})\nWith the current implementation, any error between –1 and 1 is considered\n“small.” But what if you want a different threshold? One solution is to\ncreate a function that creates a configured loss function:\ndef create_huber(threshold=1.0): \n def huber_fn(y_true, y_pred): \n error = y_true - y_pred \n is_small_error = tf.abs(error) < threshold \n squared_loss = tf.square(error) / 2 \n linear_loss = threshold * tf.abs(error) - threshold**2 / 2 \n return tf.where(is_small_error, squared_loss, linear_loss) \n return huber_fn\nmodel.compile(loss=create_huber(2.0), optimizer=\"nadam\")\nUnfortunately, when you save the model, the threshold will not be saved.\nThis means that you will have to specify the threshold value when\nloading the model (note that the name to use is \"huber_fn\", which is the\nname of the function you gave Keras, not the name of the function that\ncreated it):\nmodel = \nkeras.models.load_model(\"my_model_with_a_custom_loss_threshold_2.h5\", \n custom_objects={\"huber_fn\": \ncreate_huber(2.0)})\nYou can solve this by creating a subclass of the keras.losses.Loss class,\nand then implementing its get_config() method:\nclass HuberLoss(keras.losses.Loss): \n def __init__(self, threshold=1.0, **kwargs): \n self.threshold = threshold \n super().__init__(**kwargs) \n def call(self, y_true, y_pred): \n error = y_true - y_pred \n is_small_error = tf.abs(error) < self.threshold \n squared_loss = tf.square(error) / 2 \n linear_loss = self.threshold * tf.abs(error) - self.threshold**2 \n/ 2 \n return tf.where(is_small_error, squared_loss, linear_loss) \n def get_config(self): \n base_config = super().get_config() \n return {**base_config, \"threshold\": self.threshold}\nWARNING\nThe Keras API currently only specifies how to use subclassing to define layers,\nmodels, callbacks, and regularizers. If you build other components (such as losses,\nmetrics, initializers, or constraints) using subclassing, they may not be portable to\nother Keras implementations. It’s likely that the Keras API will be updated to specify\nsubclassing for all these components as well.\nLet’s walk through this code:\nThe constructor accepts **kwargs and passes them to the parent\nconstructor, which handles standard hyperparameters: the name of\nthe loss and the reduction algorithm to use to aggregate the\nindividual instance losses. By default, it is\n\"sum_over_batch_size\", which means that the loss will be the\nsum of the instance losses, weighted by the sample weights, if\nany, and divided by the batch size (not by the sum of weights, so\nthis is not the weighted mean). Other possible values are \"sum\"\nand None.\n5\nThe call() method takes the labels and predictions, computes all\nthe instance losses, and returns them.\nThe get_config() method returns a dictionary mapping each\nhyperparameter name to its value. It first calls the parent class’s\nget_config() method, then adds the new hyperparameters to this\ndictionary (note that the convenient {**x} syntax was added in\nPython 3.5).\nYou can then use any instance of this class when you compile the model:\nmodel.compile(loss=HuberLoss(2.), optimizer=\"nadam\")\nWhen you save the model, the threshold will be saved along with it; and\nwhen you load the model, you just need to map the class name to the class\nitself:\nmodel = keras.models.load_model(\"my_model_with_a_custom_loss_class.h5\", \n custom_objects={\"HuberLoss\": HuberLoss})\nWhen you save a model, Keras calls the loss instance’s get_config()\nmethod and saves the config as JSON in the HDF5 file. When you load the\nmodel, it calls the from_config() class method on the HuberLoss class:\nthis method is implemented by the base class (Loss) and creates an\ninstance of the class, passing **config to the constructor.\nThat’s it for losses! That wasn’t too hard, was it? Just as simple are custom\nactivation functions, initializers, regularizers, and constraints. Let’s look\nat these now.\nCustom Activation Functions, Initializers, Regularizers,\nand Constraints\nMost Keras functionalities, such as losses, regularizers, constraints,\ninitializers, metrics, activation functions, layers, and even full models, can\nbe customized in very much the same way. Most of the time, you will just\nneed to write a simple function with the appropriate inputs and outputs.\nHere are examples of a custom activation function (equivalent to\nkeras.activations.softplus() or tf.nn.softplus()), a custom\nGlorot initializer (equivalent to\nkeras.initializers.glorot_normal()), a custom ℓ regularizer\n(equivalent to keras.regularizers.l1(0.01)), and a custom constraint\nthat ensures weights are all positive (equivalent to\nkeras.constraints.nonneg() or tf.nn.relu()):\ndef my_softplus(z): # return value is just tf.nn.softplus(z) \n return tf.math.log(tf.exp(z) + 1.0) \n \ndef my_glorot_initializer(shape, dtype=tf.float32): \n stddev = tf.sqrt(2. / (shape[0] + shape[1])) \n return tf.random.normal(shape, stddev=stddev, dtype=dtype) \n \ndef my_l1_regularizer(weights): \n return tf.reduce_sum(tf.abs(0.01 * weights)) \n \ndef my_positive_weights(weights): # return value is just \ntf.nn.relu(weights) \n return tf.where(weights < 0., tf.zeros_like(weights), weights)\nAs you can see, the arguments depend on the type of custom function.\nThese custom functions can then be used normally; for example:\nlayer = keras.layers.Dense(30, activation=my_softplus, \n kernel_initializer=my_glorot_initializer, \n kernel_regularizer=my_l1_regularizer, \n kernel_constraint=my_positive_weights)\nThe activation function will be applied to the output of this Dense layer,\nand its result will be passed on to the next layer. The layer’s weights will\nbe initialized using the value returned by the initializer. At each training\nstep the weights will be passed to the regularization function to compute\nthe regularization loss, which will be added to the main loss to get the\nfinal loss used for training. Finally, the constraint function will be called\n1\nafter each training step, and the layer’s weights will be replaced by the\nconstrained weights.\nIf a function has hyperparameters that need to be saved along with the\nmodel, then you will want to subclass the appropriate class, such as\nkeras.regularizers.Regularizer, keras.constraints.Constraint,\nkeras.initializers.Initializer, or keras.layers.Layer (for any\nlayer, including activation functions). Much like we did for the custom\nloss, here is a simple class for ℓ regularization that saves its factor\nhyperparameter (this time we do not need to call the parent constructor or\nthe get_config() method, as they are not defined by the parent class):\nclass MyL1Regularizer(keras.regularizers.Regularizer): \n def __init__(self, factor): \n self.factor = factor \n def __call__(self, weights): \n return tf.reduce_sum(tf.abs(self.factor * weights)) \n def get_config(self): \n return {\"factor\": self.factor}\nNote that you must implement the call() method for losses, layers\n(including activation functions), and models, or the __call__() method\nfor regularizers, initializers, and constraints. For metrics, things are a bit\ndifferent, as we will see now.\nCustom Metrics\nLosses and metrics are conceptually not the same thing: losses (e.g., cross\nentropy) are used by Gradient Descent to train a model, so they must be\ndifferentiable (at least where they are evaluated), and their gradients\nshould not be 0 everywhere. Plus, it’s OK if they are not easily\ninterpretable by humans. In contrast, metrics (e.g., accuracy) are used to\nevaluate a model: they must be more easily interpretable, and they can be\nnon-differentiable or have 0 gradients everywhere.\nThat said, in most cases, defining a custom metric function is exactly the\nsame as defining a custom loss function. In fact, we could even use the\n1\nHuber loss function we created earlier as a metric; it would work just fine\n(and persistence would also work the same way, in this case only saving\nthe name of the function, \"huber_fn\"):\nmodel.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[create_huber(2.0)])\nFor each batch during training, Keras will compute this metric and keep\ntrack of its mean since the beginning of the epoch. Most of the time, this is\nexactly what you want. But not always! Consider a binary classifier’s\nprecision, for example. As we saw in Chapter 3, precision is the number of\ntrue positives divided by the number of positive predictions (including\nboth true positives and false positives). Suppose the model made five\npositive predictions in the first batch, four of which were correct: that’s\n80% precision. Then suppose the model made three positive predictions in\nthe second batch, but they were all incorrect: that’s 0% precision for the\nsecond batch. If you just compute the mean of these two precisions, you\nget 40%. But wait a second—that’s not the model’s precision over these\ntwo batches! Indeed, there were a total of four true positives (4 + 0) out of\neight positive predictions (5 + 3), so the overall precision is 50%, not\n40%. What we need is an object that can keep track of the number of true\npositives and the number of false positives and that can compute their\nratio when requested. This is precisely what the\nkeras.metrics.Precision class does:\n>>> precision = keras.metrics.Precision() \n>>> precision([0, 1, 1, 1, 0, 1, 0, 1], [1, 1, 0, 1, 0, 1, 0, 1]) \n \n>>> precision([0, 1, 0, 0, 1, 0, 1, 1], [1, 0, 1, 1, 0, 0, 0, 0]) \n\nIn this example, we created a Precision object, then we used it like a\nfunction, passing it the labels and predictions for the first batch, then for\nthe second batch (note that we could also have passed sample weights). We\nused the same number of true and false positives as in the example we just\ndiscussed. After the first batch, it returns a precision of 80%; then after the\n6\nsecond batch, it returns 50% (which is the overall precision so far, not the\nsecond batch’s precision). This is called a streaming metric (or stateful\nmetric), as it is gradually updated, batch after batch.\nAt any point, we can call the result() method to get the current value of\nthe metric. We can also look at its variables (tracking the number of true\nand false positives) by using the variables attribute, and we can reset\nthese variables using the reset_states() method:\n>>> p.result() \n \n>>> p.variables \n[, \n ] \n>>> p.reset_states() # both variables get reset to 0.0\nIf you need to create such a streaming metric, create a subclass of the\nkeras.metrics.Metric class. Here is a simple example that keeps track\nof the total Huber loss and the number of instances seen so far. When\nasked for the result, it returns the ratio, which is simply the mean Huber\nloss:\nclass HuberMetric(keras.metrics.Metric): \n def __init__(self, threshold=1.0, **kwargs): \n super().__init__(**kwargs) # handles base args (e.g., dtype) \n self.threshold = threshold \n self.huber_fn = create_huber(threshold) \n self.total = self.add_weight(\"total\", initializer=\"zeros\") \n self.count = self.add_weight(\"count\", initializer=\"zeros\") \n def update_state(self, y_true, y_pred, sample_weight=None): \n metric = self.huber_fn(y_true, y_pred) \n self.total.assign_add(tf.reduce_sum(metric)) \n self.count.assign_add(tf.cast(tf.size(y_true), tf.float32)) \n def result(self): \n return self.total / self.count \n def get_config(self): \n base_config = super().get_config() \n return {**base_config, \"threshold\": self.threshold}\nLet’s walk through this code:7\nThe constructor uses the add_weight() method to create the\nvariables needed to keep track of the metric’s state over multiple\nbatches—in this case, the sum of all Huber losses (total) and the\nnumber of instances seen so far (count). You could just create\nvariables manually if you preferred. Keras tracks any\ntf.Variable that is set as an attribute (and more generally, any\n“trackable” object, such as layers or models).\nThe update_state() method is called when you use an instance\nof this class as a function (as we did with the Precision object).\nIt updates the variables, given the labels and predictions for one\nbatch (and sample weights, but in this case we ignore them).\nThe result() method computes and returns the final result, in\nthis case the mean Huber metric over all instances. When you use\nthe metric as a function, the update_state() method gets called\nfirst, then the result() method is called, and its output is\nreturned.\nWe also implement the get_config() method to ensure the\nthreshold gets saved along with the model.\nThe default implementation of the reset_states() method\nresets all variables to 0.0 (but you can override it if needed).\nNOTE\nKeras will take care of variable persistence seamlessly; no action is required.\nWhen you define a metric using a simple function, Keras automatically\ncalls it for each batch, and it keeps track of the mean during each epoch,\njust like we did manually. So the only benefit of our HuberMetric class is\nthat the threshold will be saved. But of course, some metrics, like\nprecision, cannot simply be averaged over batches: in those cases, there’s\nno other option than to implement a streaming metric.\nNow that we have built a streaming metric, building a custom layer will\nseem like a walk in the park!\nCustom Layers\nYou may occasionally want to build an architecture that contains an exotic\nlayer for which TensorFlow does not provide a default implementation. In\nthis case, you will need to create a custom layer. Or you may simply want\nto build a very repetitive architecture, containing identical blocks of layers\nrepeated many times, and it would be convenient to treat each block of\nlayers as a single layer. For example, if the model is a sequence of layers\nA, B, C, A, B, C, A, B, C, then you might want to define a custom layer D\ncontaining layers A, B, C, so your model would then simply be D, D, D.\nLet’s see how to build custom layers.\nFirst, some layers have no weights, such as keras.layers.Flatten or\nkeras.layers.ReLU. If you want to create a custom layer without any\nweights, the simplest option is to write a function and wrap it in a\nkeras.layers.Lambda layer. For example, the following layer will apply\nthe exponential function to its inputs:\nexponential_layer = keras.layers.Lambda(lambda x: tf.exp(x))\nThis custom layer can then be used like any other layer, using the\nSequential API, the Functional API, or the Subclassing API. You can also\nuse it as an activation function (or you could use activation=tf.exp,\nactivation=keras.activations.exponential, or simply\nactivation=\"exponential\"). The exponential layer is sometimes used in\nthe output layer of a regression model when the values to predict have\nvery different scales (e.g., 0.001, 10., 1,000.).\nAs you’ve probably guessed by now, to build a custom stateful layer (i.e., a\nlayer with weights), you need to create a subclass of the\nkeras.layers.Layer class. For example, the following class implements\na simplified version of the Dense layer:\nclass MyDense(keras.layers.Layer): \n def __init__(self, units, activation=None, **kwargs): \n super().__init__(**kwargs) \n self.units = units \n self.activation = keras.activations.get(activation) \n \n def build(self, batch_input_shape): \n self.kernel = self.add_weight( \n name=\"kernel\", shape=[batch_input_shape[-1], self.units], \n initializer=\"glorot_normal\") \n self.bias = self.add_weight( \n name=\"bias\", shape=[self.units], initializer=\"zeros\") \n super().build(batch_input_shape) # must be at the end \n \n def call(self, X): \n return self.activation(X @ self.kernel + self.bias) \n \n def compute_output_shape(self, batch_input_shape): \n return tf.TensorShape(batch_input_shape.as_list()[:-1] + \n[self.units]) \n \n def get_config(self): \n base_config = super().get_config() \n return {**base_config, \"units\": self.units, \n \"activation\": \nkeras.activations.serialize(self.activation)}\nLet’s walk through this code:\nThe constructor takes all the hyperparameters as arguments (in\nthis example, units and activation), and importantly it also\ntakes a **kwargs argument. It calls the parent constructor,\npassing it the kwargs: this takes care of standard arguments such\nas input_shape, trainable, and name. Then it saves the\nhyperparameters as attributes, converting the activation\nargument to the appropriate activation function using the\nkeras.activations.get() function (it accepts functions,\nstandard strings like \"relu\" or \"selu\", or simply None).8\nThe build() method’s role is to create the layer’s variables by\ncalling the add_weight() method for each weight. The build()\nmethod is called the first time the layer is used. At that point,\nKeras will know the shape of this layer’s inputs, and it will pass it\nto the build() method, which is often necessary to create some\nof the weights. For example, we need to know the number of\nneurons in the previous layer in order to create the connection\nweights matrix (i.e., the \"kernel\"): this corresponds to the size of\nthe last dimension of the inputs. At the end of the build()\nmethod (and only at the end), you must call the parent’s build()\nmethod: this tells Keras that the layer is built (it just sets\nself.built=True).\nThe call() method performs the desired operations. In this case,\nwe compute the matrix multiplication of the inputs X and the\nlayer’s kernel, we add the bias vector, and we apply the activation\nfunction to the result, and this gives us the output of the layer.\nThe compute_output_shape() method simply returns the shape\nof this layer’s outputs. In this case, it is the same shape as the\ninputs, except the last dimension is replaced with the number of\nneurons in the layer. Note that in tf.keras, shapes are instances of\nthe tf.TensorShape class, which you can convert to Python lists\nusing as_list().\nThe get_config() method is just like in the previous custom\nclasses. Note that we save the activation function’s full\nconfiguration by calling keras.activations.serialize().\nYou can now use a MyDense layer just like any other layer!\n9\nNOTE\nYou can generally omit the compute_output_shape() method, as tf.keras\nautomatically infers the output shape, except when the layer is dynamic (as we will\nsee shortly). In other Keras implementations, this method is either required or its\ndefault implementation assumes the output shape is the same as the input shape.\nTo create a layer with multiple inputs (e.g., Concatenate), the argument to\nthe call() method should be a tuple containing all the inputs, and\nsimilarly the argument to the compute_output_shape() method should\nbe a tuple containing each input’s batch shape. To create a layer with\nmultiple outputs, the call() method should return the list of outputs, and\ncompute_output_shape() should return the list of batch output shapes\n(one per output). For example, the following toy layer takes two inputs and\nreturns three outputs:\nclass MyMultiLayer(keras.layers.Layer): \n def call(self, X): \n X1, X2 = X \n return [X1 + X2, X1 * X2, X1 / X2] \n \n def compute_output_shape(self, batch_input_shape): \n b1, b2 = batch_input_shape \n return [b1, b1, b1] # should probably handle broadcasting rules\nThis layer may now be used like any other layer, but of course only using\nthe Functional and Subclassing APIs, not the Sequential API (which only\naccepts layers with one input and one output).\nIf your layer needs to have a different behavior during training and during\ntesting (e.g., if it uses Dropout or BatchNormalization layers), then you\nmust add a training argument to the call() method and use this\nargument to decide what to do. For example, let’s create a layer that adds\nGaussian noise during training (for regularization) but does nothing during\ntesting (Keras has a layer that does the same thing,\nkeras.layers.GaussianNoise):\nclass MyGaussianNoise(keras.layers.Layer): \n def __init__(self, stddev, **kwargs): \n super().__init__(**kwargs) \n self.stddev = stddev \n \n def call(self, X, training=None): \n if training: \n noise = tf.random.normal(tf.shape(X), stddev=self.stddev) \n return X + noise \n else: \n return X \n \n def compute_output_shape(self, batch_input_shape): \n return batch_input_shape\nWith that, you can now build any custom layer you need! Now let’s create\ncustom models.\nCustom Models\nWe already looked at creating custom model classes in Chapter 10, when\nwe discussed the Subclassing API.\n It’s straightforward: subclass the\nkeras.Model class, create layers and variables in the constructor, and\nimplement the call() method to do whatever you want the model to do.\nSuppose you want to build the model represented in Figure 12-3.\n10\nFigure 12-3. Custom model example: an arbitrary model with a custom ResidualBlock layer\ncontaining a skip connection\nThe inputs go through a first dense layer, then through a residual block\ncomposed of two dense layers and an addition operation (as we will see in\nChapter 14, a residual block adds its inputs to its outputs), then through\nthis same residual block three more times, then through a second residual\nblock, and the final result goes through a dense output layer. Note that this\nmodel does not make much sense; it’s just an example to illustrate the fact\nthat you can easily build any kind of model you want, even one that\ncontains loops and skip connections. To implement this model, it is best to\nfirst create a ResidualBlock layer, since we are going to create a couple\nof identical blocks (and we might want to reuse it in another model):\nclass ResidualBlock(keras.layers.Layer): \n def __init__(self, n_layers, n_neurons, **kwargs): \n super().__init__(**kwargs) \n self.hidden = [keras.layers.Dense(n_neurons, activation=\"elu\", \n kernel_initializer=\"he_normal\") \n for _ in range(n_layers)] \n \n def call(self, inputs): \n Z = inputs \n for layer in self.hidden: \n Z = layer(Z) \n return inputs + Z\nThis layer is a bit special since it contains other layers. This is handled\ntransparently by Keras: it automatically detects that the hidden attribute\ncontains trackable objects (layers in this case), so their variables are\nautomatically added to this layer’s list of variables. The rest of this class is\nself-explanatory. Next, let’s use the Subclassing API to define the model\nitself:\nclass ResidualRegressor(keras.Model): \n def __init__(self, output_dim, **kwargs): \n super().__init__(**kwargs) \n self.hidden1 = keras.layers.Dense(30, activation=\"elu\", \n kernel_initializer=\"he_normal\") \n self.block1 = ResidualBlock(2, 30) \n self.block2 = ResidualBlock(2, 30) \n self.out = keras.layers.Dense(output_dim) \n \n def call(self, inputs): \n Z = self.hidden1(inputs) \n for _ in range(1 + 3): \n Z = self.block1(Z) \n Z = self.block2(Z) \n return self.out(Z)\nWe create the layers in the constructor and use them in the call()\nmethod. This model can then be used like any other model (compile it, fit\nit, evaluate it, and use it to make predictions). If you also want to be able\nto save the model using the save() method and load it using the\nkeras.models.load_model() function, you must implement the\nget_config() method (as we did earlier) in both the ResidualBlock\nclass and the ResidualRegressor class. Alternatively, you can save and\nload the weights using the save_weights() and load_weights()\nmethods.\nThe Model class is a subclass of the Layer class, so models can be defined\nand used exactly like layers. But a model has some extra functionalities,\nincluding of course its compile(), fit(), evaluate(), and predict()\nmethods (and a few variants), plus the get_layers() method (which can\nreturn any of the model’s layers by name or by index) and the save()\nmethod (and support for keras.models.load_model() and\nkeras.models.clone_model()).\nTIP\nIf models provide more functionality than layers, why not just define every layer as a\nmodel? Well, technically you could, but it is usually cleaner to distinguish the\ninternal components of your model (i.e., layers or reusable blocks of layers) from the\nmodel itself (i.e., the object you will train). The former should subclass the Layer\nclass, while the latter should subclass the Model class.\nWith that, you can naturally and concisely build almost any model that\nyou find in a paper, using the Sequential API, the Functional API, the\nSubclassing API, or even a mix of these. “Almost” any model? Yes, there\nare still a few things that we need to look at: first, how to define losses or\nmetrics based on model internals, and second, how to build a custom\ntraining loop.\nLosses and Metrics Based on Model Internals\nThe custom losses and metrics we defined earlier were all based on the\nlabels and the predictions (and optionally sample weights). There will be\ntimes when you want to define losses based on other parts of your model,\nsuch as the weights or activations of its hidden layers. This may be useful\nfor regularization purposes or to monitor some internal aspect of your\nmodel.\nTo define a custom loss based on model internals, compute it based on any\npart of the model you want, then pass the result to the add_loss()\nmethod.For example, let’s build a custom regression MLP model\ncomposed of a stack of five hidden layers plus an output layer. This\ncustom model will also have an auxiliary output on top of the upper\nhidden layer. The loss associated to this auxiliary output will be called the\nreconstruction loss (see Chapter 17): it is the mean squared difference\nbetween the reconstruction and the inputs. By adding this reconstruction\nloss to the main loss, we will encourage the model to preserve as much\ninformation as possible through the hidden layers—even information that\nis not directly useful for the regression task itself. In practice, this loss\nsometimes improves generalization (it is a regularization loss). Here is the\ncode for this custom model with a custom reconstruction loss:\nclass ReconstructingRegressor(keras.Model): \n def __init__(self, output_dim, **kwargs): \n super().__init__(**kwargs) \n self.hidden = [keras.layers.Dense(30, activation=\"selu\", \n \nkernel_initializer=\"lecun_normal\") \n for _ in range(5)] \n self.out = keras.layers.Dense(output_dim) \n \n def build(self, batch_input_shape): \n n_inputs = batch_input_shape[-1] \n self.reconstruct = keras.layers.Dense(n_inputs) \n super().build(batch_input_shape) \n \n def call(self, inputs): \n Z = inputs \n for layer in self.hidden: \n Z = layer(Z) \n reconstruction = self.reconstruct(Z) \n recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs)) \n self.add_loss(0.05 * recon_loss) \n return self.out(Z)\nLet’s go through this code:\nThe constructor creates the DNN with five dense hidden layers\nand one dense output layer.\nThe build() method creates an extra dense layer which will be\nused to reconstruct the inputs of the model. It must be created\nhere because its number of units must be equal to the number of\ninputs, and this number is unknown before the build() method is\ncalled.\nThe call() method processes the inputs through all five hidden\nlayers, then passes the result through the reconstruction layer,\nwhich produces the reconstruction.\nThen the call() method computes the reconstruction loss (the\nmean squared difference between the reconstruction and the\ninputs), and adds it to the model’s list of losses using the\nadd_loss() method. Notice that we scale down the\nreconstruction loss by multiplying it by 0.05 (this is a\nhyperparameter you can tune). This ensures that the\nreconstruction loss does not dominate the main loss.\nFinally, the call() method passes the output of the hidden layers\nto the output layer and returns its output.\nSimilarly, you can add a custom metric based on model internals by\ncomputing it in any way you want, as long as the result is the output of a\nmetric object. For example, you can create a keras.metrics.Mean object\nin the constructor, then call it in the call() method, passing it the\nrecon_loss, and finally add it to the model by calling the model’s\nadd_metric() method. This way, when you train the model, Keras will\ndisplay both the mean loss over each epoch (the loss is the sum of the\nmain loss plus 0.05 times the reconstruction loss) and the mean\nreconstruction error over each epoch. Both will go down during training:\nEpoch 1/5 \n11610/11610 [=============] [...] loss: 4.3092 - reconstruction_error: \n1.7360 \nEpoch 2/5 \n11610/11610 [=============] [...] loss: 1.1232 - reconstruction_error: \n11\n0.8964 \n[...]\nIn over 99% of cases, everything we have discussed so far will be\nsufficient to implement whatever model you want to build, even with\ncomplex architectures, losses, and metrics. However, in some rare cases\nyou may need to customize the training loop itself. Before we get there,\nwe need to look at how to compute gradients automatically in TensorFlow.\nComputing Gradients Using Autodiff\nTo understand how to use autodiff (see Chapter 10 and Appendix D) to\ncompute gradients automatically, let’s consider a simple toy function:\ndef f(w1, w2): \n return 3 * w1 ** 2 + 2 * w1 * w2\nIf you know calculus, you can analytically find that the partial derivative\nof this function with regard to w1 is 6 * w1 + 2 * w2. You can also find\nthat its partial derivative with regard to w2 is 2 * w1. For example, at the\npoint (w1, w2) = (5, 3), these partial derivatives are equal to 36 and 10,\nrespectively, so the gradient vector at this point is (36, 10). But if this were\na neural network, the function would be much more complex, typically\nwith tens of thousands of parameters, and finding the partial derivatives\nanalytically by hand would be an almost impossible task. One solution\ncould be to compute an approximation of each partial derivative by\nmeasuring how much the function’s output changes when you tweak the\ncorresponding parameter:\n>>> w1, w2 = 5, 3 \n>>> eps = 1e-6 \n>>> (f(w1 + eps, w2) - f(w1, w2)) / eps \n36.000003007075065 \n>>> (f(w1, w2 + eps) - f(w1, w2)) / eps \n10.000000003174137\nLooks about right! This works rather well and is easy to implement, but it\nis just an approximation, and importantly you need to call f() at least\nonce per parameter (not twice, since we could compute f(w1, w2) just\nonce). Needing to call f() at least once per parameter makes this approach\nintractable for large neural networks. So instead, we should use autodiff.\nTensorFlow makes this pretty simple:\nw1, w2 = tf.Variable(5.), tf.Variable(3.) \nwith tf.GradientTape() as tape: \n z = f(w1, w2) \n \ngradients = tape.gradient(z, [w1, w2])\nWe first define two variables w1 and w2, then we create a\ntf.GradientTape context that will automatically record every operation\nthat involves a variable, and finally we ask this tape to compute the\ngradients of the result z with regard to both variables [w1, w2]. Let’s take\na look at the gradients that TensorFlow computed:\n>>> gradients \n[, \n ]\nPerfect! Not only is the result accurate (the precision is only limited by the\nfloating-point errors), but the gradient() method only goes through the\nrecorded computations once (in reverse order), no matter how many\nvariables there are, so it is incredibly efficient. It’s like magic!\nTIP\nTo save memory, only put the strict minimum inside the tf.GradientTape() block.\nAlternatively, pause recording by creating a with tape.stop_recording() block\ninside the tf.GradientTape() block.\nThe tape is automatically erased immediately after you call its\ngradient() method, so you will get an exception if you try to call\ngradient() twice:\nwith tf.GradientTape() as tape: \n z = f(w1, w2) \n \ndz_dw1 = tape.gradient(z, w1) # => tensor 36.0 \ndz_dw2 = tape.gradient(z, w2) # RuntimeError!\nIf you need to call gradient() more than once, you must make the tape\npersistent and delete it each time you are done with it to free resources:\nwith tf.GradientTape(persistent=True) as tape: \n z = f(w1, w2) \n \ndz_dw1 = tape.gradient(z, w1) # => tensor 36.0 \ndz_dw2 = tape.gradient(z, w2) # => tensor 10.0, works fine now! \ndel tape\nBy default, the tape will only track operations involving variables, so if\nyou try to compute the gradient of z with regard to anything other than a\nvariable, the result will be None:\nc1, c2 = tf.constant(5.), tf.constant(3.) \nwith tf.GradientTape() as tape: \n z = f(c1, c2) \n \ngradients = tape.gradient(z, [c1, c2]) # returns [None, None]\nHowever, you can force the tape to watch any tensors you like, to record\nevery operation that involves them. You can then compute gradients with\nregard to these tensors, as if they were variables:\nwith tf.GradientTape() as tape: \n tape.watch(c1) \n tape.watch(c2) \n z = f(c1, c2) \n \ngradients = tape.gradient(z, [c1, c2]) # returns [tensor 36., tensor 10.]\n12\nThis can be useful in some cases, like if you want to implement a\nregularization loss that penalizes activations that vary a lot when the\ninputs vary little: the loss will be based on the gradient of the activations\nwith regard to the inputs. Since the inputs are not variables, you would\nneed to tell the tape to watch them.\nMost of the time a gradient tape is used to compute the gradients of a\nsingle value (usually the loss) with regard to a set of values (usually the\nmodel parameters). This is where reverse-mode autodiff shines, as it just\nneeds to do one forward pass and one reverse pass to get all the gradients\nat once. If you try to compute the gradients of a vector, for example a\nvector containing multiple losses, then TensorFlow will compute the\ngradients of the vector’s sum. So if you ever need to get the individual\ngradients (e.g., the gradients of each loss with regard to the model\nparameters), you must call the tape’s jabobian() method: it will perform\nreverse-mode autodiff once for each loss in the vector (all in parallel by\ndefault). It is even possible to compute second-order partial derivatives\n(the Hessians, i.e., the partial derivatives of the partial derivatives), but\nthis is rarely needed in practice (see the “Computing Gradients with\nAutodiff” section of the notebook for an example).\nIn some cases you may want to stop gradients from backpropagating\nthrough some part of your neural network. To do this, you must use the\ntf.stop_gradient() function. The function returns its inputs during the\nforward pass (like tf.identity()), but it does not let gradients through\nduring backpropagation (it acts like a constant):\ndef f(w1, w2): \n return 3 * w1 ** 2 + tf.stop_gradient(2 * w1 * w2) \n \nwith tf.GradientTape() as tape: \n z = f(w1, w2) # same result as without stop_gradient() \n \ngradients = tape.gradient(z, [w1, w2]) # => returns [tensor 30., None]\nFinally, you may occasionally run into some numerical issues when\ncomputing gradients. For example, if you compute the gradients of the\nmy_softplus() function for large inputs, the result will be NaN:\n>>> x = tf.Variable([100.]) \n>>> with tf.GradientTape() as tape: \n... z = my_softplus(x) \n... \n>>> tape.gradient(z, [x]) \n\nThis is because computing the gradients of this function using autodiff\nleads to some numerical difficulties: due to floating-point precision errors,\nautodiff ends up computing infinity divided by infinity (which returns\nNaN). Fortunately, we can analytically find that the derivative of the\nsoftplus function is just 1 / (1 + 1 / exp(x)), which is numerically stable.\nNext, we can tell TensorFlow to use this stable function when computing\nthe gradients of the my_softplus() function by decorating it with\n@tf.custom_gradient and making it return both its normal output and\nthe function that computes the derivatives (note that it will receive as\ninput the gradients that were backpropagated so far, down to the softplus\nfunction; and according to the chain rule, we should multiply them with\nthis function’s gradients):\n@tf.custom_gradient \ndef my_better_softplus(z): \n exp = tf.exp(z) \n def my_softplus_gradients(grad): \n return grad / (1 + 1 / exp) \n return tf.math.log(exp + 1), my_softplus_gradients\nNow when we compute the gradients of the my_better_softplus()\nfunction, we get the proper result, even for large input values (however,\nthe main output still explodes because of the exponential; one workaround\nis to use tf.where() to return the inputs when they are large).\nCongratulations! You can now compute the gradients of any function\n(provided it is differentiable at the point where you compute it), even\nblocking backpropagation when needed, and write your own gradient\nfunctions! This is probably more flexibility than you will ever need, even\nif you build your own custom training loops, as we will see now.\nCustom Training Loops\nIn some rare cases, the fit() method may not be flexible enough for what\nyou need to do. For example, the Wide & Deep paper we discussed in\nChapter 10 uses two different optimizers: one for the wide path and the\nother for the deep path. Since the fit() method only uses one optimizer\n(the one that we specify when compiling the model), implementing this\npaper requires writing your own custom loop.\nYou may also like to write custom training loops simply to feel more\nconfident that they do precisely what you intend them to do (perhaps you\nare unsure about some details of the fit() method). It can sometimes feel\nsafer to make everything explicit. However, remember that writing a\ncustom training loop will make your code longer, more error-prone, and\nharder to maintain.\nTIP\nUnless you really need the extra flexibility, you should prefer using the fit()\nmethod rather than implementing your own training loop, especially if you work in a\nteam.\nFirst, let’s build a simple model. No need to compile it, since we will\nhandle the training loop manually:\nl2_reg = keras.regularizers.l2(0.05) \nmodel = keras.models.Sequential([ \n keras.layers.Dense(30, activation=\"elu\", \nkernel_initializer=\"he_normal\", \n kernel_regularizer=l2_reg), \n keras.layers.Dense(1, kernel_regularizer=l2_reg) \n])\nNext, let’s create a tiny function that will randomly sample a batch of\ninstances from the training set (in Chapter 13 we will discuss the Data\nAPI, which offers a much better alternative):\ndef random_batch(X, y, batch_size=32): \n idx = np.random.randint(len(X), size=batch_size) \n return X[idx], y[idx]\nLet’s also define a function that will display the training status, including\nthe number of steps, the total number of steps, the mean loss since the\nstart of the epoch (i.e., we will use the Mean metric to compute it), and\nother metrics:\ndef print_status_bar(iteration, total, loss, metrics=None): \n metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result()) \n for m in [loss] + (metrics or [])]) \n end = \"\" if iteration < total else \"\\n\" \n print(\"\\r{}/{} - \".format(iteration, total) + metrics, \n end=end)\nThis code is self-explanatory, unless you are unfamiliar with Python string\nformatting: {:.4f} will format a float with four digits after the decimal\npoint, and using \\r (carriage return) along with end=\"\" ensures that the\nstatus bar always gets printed on the same line. In the notebook, the\nprint_status_bar() function includes a progress bar, but you could use\nthe handy tqdm library instead.\nWith that, let’s get down to business! First, we need to define some\nhyperparameters and choose the optimizer, the loss function, and the\nmetrics (just the MAE in this example):\nn_epochs = 5 \nbatch_size = 32 \nn_steps = len(X_train) // batch_size \noptimizer = keras.optimizers.Nadam(lr=0.01) \nloss_fn = keras.losses.mean_squared_error \nmean_loss = keras.metrics.Mean() \nmetrics = [keras.metrics.MeanAbsoluteError()]\nAnd now we are ready to build the custom loop!\nfor epoch in range(1, n_epochs + 1): \n print(\"Epoch {}/{}\".format(epoch, n_epochs)) \n for step in range(1, n_steps + 1): \n X_batch, y_batch = random_batch(X_train_scaled, y_train) \n with tf.GradientTape() as tape: \n y_pred = model(X_batch, training=True) \n main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred)) \n loss = tf.add_n([main_loss] + model.losses) \n gradients = tape.gradient(loss, model.trainable_variables) \n optimizer.apply_gradients(zip(gradients, \nmodel.trainable_variables)) \n mean_loss(loss) \n for metric in metrics: \n metric(y_batch, y_pred) \n print_status_bar(step * batch_size, len(y_train), mean_loss, \nmetrics) \n print_status_bar(len(y_train), len(y_train), mean_loss, metrics) \n for metric in [mean_loss] + metrics: \n metric.reset_states()\nThere’s a lot going on in this code, so let’s walk through it:\nWe create two nested loops: one for the epochs, the other for the\nbatches within an epoch.\nThen we sample a random batch from the training set.\nInside the tf.GradientTape() block, we make a prediction for\none batch (using the model as a function), and we compute the\nloss: it is equal to the main loss plus the other losses (in this\nmodel, there is one regularization loss per layer). Since the\nmean_squared_error() function returns one loss per instance,\nwe compute the mean over the batch using tf.reduce_mean() (if\nyou wanted to apply different weights to each instance, this is\nwhere you would do it). The regularization losses are already\nreduced to a single scalar each, so we just need to sum them\n(using tf.add_n(), which sums multiple tensors of the same\nshape and data type).\nNext, we ask the tape to compute the gradient of the loss with\nregard to each trainable variable (not all variables!), and we apply\nthem to the optimizer to perform a Gradient Descent step.\nThen we update the mean loss and the metrics (over the current\nepoch), and we display the status bar.\nAt the end of each epoch, we display the status bar again to make\nit look complete\n and to print a line feed, and we reset the states\nof the mean loss and the metrics.\nIf you set the optimizer’s clipnorm or clipvalue hyperparameter, it will\ntake care of this for you. If you want to apply any other transformation to\nthe gradients, simply do so before calling the apply_gradients()\nmethod.\nIf you add weight constraints to your model (e.g., by setting\nkernel_constraint or bias_constraint when creating a layer), you\nshould update the training loop to apply these constraints just after\napply_gradients():\nfor variable in model.variables: \n if variable.constraint is not None: \n variable.assign(variable.constraint(variable))\nMost importantly, this training loop does not handle layers that behave\ndifferently during training and testing (e.g., BatchNormalization or\nDropout). To handle these, you need to call the model with\ntraining=True and make sure it propagates this to every layer that needs\nit.\nAs you can see, there are quite a lot of things you need to get right, and it’s\neasy to make a mistake. But on the bright side, you get full control, so it’s\nyour call.\nNow that you know how to customize any part of your models\n and\ntraining algorithms, let’s see how you can use TensorFlow’s automatic\n13\n14\ngraph generation feature: it can speed up your custom code considerably,\nand it will also make it portable to any platform supported by TensorFlow\n(see Chapter 19).\nTensorFlow Functions and Graphs\nIn TensorFlow 1, graphs were unavoidable (as were the complexities that\ncame with them) because they were a central part of TensorFlow’s API. In\nTensorFlow 2, they are still there, but not as central, and they’re much\n(much!) simpler to use. To show just how simple, let’s start with a trivial\nfunction that computes the cube of its input:\ndef cube(x): \n return x ** 3\nWe can obviously call this function with a Python value, such as an int or a\nfloat, or we can call it with a tensor:\n>>> cube(2) \n8 \n>>> cube(tf.constant(2.0)) \n\nNow, let’s use tf.function() to convert this Python function to a\nTensorFlow Function:\n>>> tf_cube = tf.function(cube) \n>>> tf_cube \n\nThis TF Function can then be used exactly like the original Python\nfunction, and it will return the same result (but as tensors):\n>>> tf_cube(2) \n \n>>> tf_cube(tf.constant(2.0)) \n\nUnder the hood, tf.function() analyzed the computations performed by\nthe cube() function and generated an equivalent computation graph! As\nyou can see, it was rather painless (we will see how this works shortly).\nAlternatively, we could have used tf.function as a decorator; this is\nactually more common:\n@tf.function \ndef tf_cube(x): \n return x ** 3\nThe original Python function is still available via the TF Function’s\npython_function attribute, in case you ever need it:\n>>> tf_cube.python_function(2) \n8\nTensorFlow optimizes the computation graph, pruning unused nodes,\nsimplifying expressions (e.g., 1 + 2 would get replaced with 3), and more.\nOnce the optimized graph is ready, the TF Function efficiently executes\nthe operations in the graph, in the appropriate order (and in parallel when\nit can). As a result, a TF Function will usually run much faster than the\noriginal Python function, especially if it performs complex\ncomputations.\n Most of the time you will not really need to know more\nthan that: when you want to boost a Python function, just transform it into\na TF Function. That’s all!\nMoreover, when you write a custom loss function, a custom metric, a\ncustom layer, or any other custom function and you use it in a Keras\nmodel (as we did throughout this chapter), Keras automatically converts\nyour function into a TF Function—no need to use tf.function(). So\nmost of the time, all this magic is 100% transparent.\n15\nTIP\nYou can tell Keras not to convert your Python functions to TF Functions by setting\ndynamic=True when creating a custom layer or a custom model. Alternatively, you\ncan set run_eagerly=True when calling the model’s compile() method.\nBy default, a TF Function generates a new graph for every unique set of\ninput shapes and data types and caches it for subsequent calls. For\nexample, if you call tf_cube(tf.constant(10)), a graph will be\ngenerated for int32 tensors of shape []. Then if you call\ntf_cube(tf.constant(20)), the same graph will be reused. But if you\nthen call tf_cube(tf.constant([10, 20])), a new graph will be\ngenerated for int32 tensors of shape [2]. This is how TF Functions handle\npolymorphism (i.e., varying argument types and shapes). However, this is\nonly true for tensor arguments: if you pass numerical Python values to a\nTF Function, a new graph will be generated for every distinct value: for\nexample, calling tf_cube(10) and tf_cube(20) will generate two graphs.\nWARNING\nIf you call a TF Function many times with different numerical Python values, then\nmany graphs will be generated, slowing down your program and using up a lot of\nRAM (you must delete the TF Function to release it). Python values should be\nreserved for arguments that will have few unique values, such as hyperparameters\nlike the number of neurons per layer. This allows TensorFlow to better optimize each\nvariant of your model.\nAutoGraph and Tracing\nSo how does TensorFlow generate graphs? It starts by analyzing the\nPython function’s source code to capture all the control flow statements,\nsuch as for loops, while loops, and if statements, as well as break,\ncontinue, and return statements. This first step is called AutoGraph. The\nreason TensorFlow has to analyze the source code is that Python does not\nprovide any other way to capture control flow statements: it offers magic\nmethods like __add__() and __mul__() to capture operators like + and *,\nbut there are no __while__() or __if__() magic methods. After\nanalyzing the function’s code, AutoGraph outputs an upgraded version of\nthat function in which all the control flow statements are replaced by the\nappropriate TensorFlow operations, such as tf.while_loop() for loops\nand tf.cond() for if statements. For example, in Figure 12-4, AutoGraph\nanalyzes the source code of the sum_squares() Python function, and it\ngenerates the tf__sum_squares() function. In this function, the for loop\nis replaced by the definition of the loop_body() function (containing the\nbody of the original for loop), followed by a call to the for_stmt()\nfunction. This call will build the appropriate tf.while_loop() operation\nin the computation graph.\nFigure 12-4. How TensorFlow generates graphs using AutoGraph and tracing\nNext, TensorFlow calls this “upgraded” function, but instead of passing the\nargument, it passes a symbolic tensor—a tensor without any actual value,\nonly a name, a data type, and a shape. For example, if you call\nsum_squares(tf.constant(10)), then the tf__sum_squares() function\nwill be called with a symbolic tensor of type int32 and shape []. The\nfunction will run in graph mode, meaning that each TensorFlow operation\nwill add a node in the graph to represent itself and its output tensor(s) (as\nopposed to the regular mode, called eager execution, or eager mode). In\ngraph mode, TF operations do not perform any computations. This should\nfeel familiar if you know TensorFlow 1, as graph mode was the default\nmode. In Figure 12-4, you can see the tf__sum_squares() function being\ncalled with a symbolic tensor as its argument (in this case, an int32 tensor\nof shape []) and the final graph being generated during tracing. The nodes\nrepresent operations, and the arrows represent tensors (both the generated\nfunction and the graph are simplified).\nTIP\nTo view the generated function’s source code, you can call\ntf.autograph.to_code(sum_squares.python_function). The code is not\nmeant to be pretty, but it can sometimes help for debugging.\nTF Function Rules\nMost of the time, converting a Python function that performs TensorFlow\noperations into a TF Function is trivial: decorate it with @tf.function or\nlet Keras take care of it for you. However, there are a few rules to respect:\nIf you call any external library, including NumPy or even the\nstandard library, this call will run only during tracing; it will not\nbe part of the graph. Indeed, a TensorFlow graph can only include\nTensorFlow constructs (tensors, operations, variables, datasets,\nand so on). So, make sure you use tf.reduce_sum() instead of\nnp.sum(), tf.sort() instead of the built-in sorted() function,\nand so on (unless you really want the code to run only during\ntracing). This has a few additional implications:\nIf you define a TF Function f(x) that just returns\nnp.random.rand(), a random number will only be\ngenerated when the function is traced, so\nf(tf.constant(2.)) and f(tf.constant(3.)) will\nreturn the same random number, but\nf(tf.constant([2., 3.])) will return a different one.\nIf you replace np.random.rand() with\ntf.random.uniform([]), then a new random number\nwill be generated upon every call, since the operation\nwill be part of the graph.\nIf your non-TensorFlow code has side effects (such as\nlogging something or updating a Python counter), then\nyou should not expect those side effects to occur every\ntime you call the TF Function, as they will only occur\nwhen the function is traced.\nYou can wrap arbitrary Python code in a\ntf.py_function() operation, but doing so will hinder\nperformance, as TensorFlow will not be able to do any\ngraph optimization on this code. It will also reduce\nportability, as the graph will only run on platforms where\nPython is available (and where the right libraries are\ninstalled).\nYou can call other Python functions or TF Functions, but they\nshould follow the same rules, as TensorFlow will capture their\noperations in the computation graph. Note that these other\nfunctions do not need to be decorated with @tf.function.\nIf the function creates a TensorFlow variable (or any other stateful\nTensorFlow object, such as a dataset or a queue), it must do so\nupon the very first call, and only then, or else you will get an\nexception. It is usually preferable to create variables outside of\nthe TF Function (e.g., in the build() method of a custom layer).\nIf you want to assign a new value to the variable, make sure you\ncall its assign() method, instead of using the = operator.\nThe source code of your Python function should be available to\nTensorFlow. If the source code is unavailable (for example, if you\ndefine your function in the Python shell, which does not give\naccess to the source code, or if you deploy only the compiled\n*.pyc Python files to production), then the graph generation\nprocess will fail or have limited functionality.\nTensorFlow will only capture for loops that iterate over a tensor\nor a dataset. So make sure you use for i in tf.range(x) rather\nthan for i in range(x), or else the loop will not be captured in\nthe graph. Instead, it will run during tracing. (This may be what\nyou want if the for loop is meant to build the graph, for example\nto create each layer in a neural network.)\nAs always, for performance reasons, you should prefer a\nvectorized implementation whenever you can, rather than using\nloops.\nIt’s time to sum up! In this chapter we started with a brief overview of\nTensorFlow, then we looked at TensorFlow’s low-level API, including\ntensors, operations, variables, and special data structures. We then used\nthese tools to customize almost every component in tf.keras. Finally, we\nlooked at how TF Functions can boost performance, how graphs are\ngenerated using AutoGraph and tracing, and what rules to follow when you\nwrite TF Functions (if you would like to open the black box a bit further,\nfor example to explore the generated graphs, you will find technical\ndetails in Appendix G).\nIn the next chapter, we will look at how to efficiently load and preprocess\ndata with TensorFlow.\nExercises\n1. How would you describe TensorFlow in a short sentence? What\nare its main features? Can you name other popular Deep Learning\nlibraries?\n2. Is TensorFlow a drop-in replacement for NumPy? What are the\nmain differences between the two?\n3. Do you get the same result with tf.range(10) and\ntf.constant(np.arange(10))?\n4. Can you name six other data structures available in TensorFlow,\nbeyond regular tensors?\n5. A custom loss function can be defined by writing a function or by\nsubclassing the keras.losses.Loss class. When would you use\neach option?\n6. Similarly, a custom metric can be defined in a function or a\nsubclass of keras.metrics.Metric. When would you use each\noption?\n7. When should you create a custom layer versus a custom model?\n8. What are some use cases that require writing your own custom\ntraining loop?\n9. Can custom Keras components contain arbitrary Python code, or\nmust they be convertible to TF Functions?\n10. What are the main rules to respect if you want a function to be\nconvertible to a TF Function?\n11. When would you need to create a dynamic Keras model? How do\nyou do that? Why not make all your models dynamic?\n12. Implement a custom layer that performs Layer Normalization (we\nwill use this type of layer in Chapter 15):\na. The build() method should define two trainable weights\nα and β, both of shape input_shape[-1:] and data type\ntf.float32. α should be initialized with 1s, and β with\n0s.\nb. The call() method should compute the mean μ and\nstandard deviation σ of each instance’s features. For this,\nyou can use tf.nn.moments(inputs, axes=-1,\nkeepdims=True), which returns the mean μ and the\nvariance σ of all instances (compute the square root of\nthe variance to get the standard deviation). Then the\nfunction should compute and return α⊗(X - μ)/(σ + ε) +\nβ, where ⊗ represents itemwise multiplication (*) and ε\nis a smoothing term (small constant to avoid division by\nzero, e.g., 0.001).\nc. Ensure that your custom layer produces the same (or very\nnearly the same) output as the\nkeras.layers.LayerNormalization layer.\n13. Train a model using a custom training loop to tackle the Fashion\nMNIST dataset (see Chapter 10).\na. Display the epoch, iteration, mean training loss, and\nmean accuracy over each epoch (updated at each\niteration), as well as the validation loss and accuracy at\nthe end of each epoch.\nb. Try using a different optimizer with a different learning\nrate for the upper layers and the lower layers.\nSolutions to these exercises are available in Appendix A.\n1 TensorFlow includes another Deep Learning API called the Estimators API, but the\nTensorFlow team recommends using tf.keras instead.\n2 If you ever need to (but you probably won’t), you can write your own operations using\nthe C++ API.\n2\n3 To learn more about TPUs and how they work, check out https://homl.info/tpus.\n4 A notable exception is tf.math.log(), which is commonly used but doesn’t have a\ntf.log() alias (as it might be confused with logging).\n5 It would not be a good idea to use a weighted mean: if you did, then two instances with\nthe same weight but in different batches would have a different impact on training,\ndepending on the total weight of each batch.\n6 However, the Huber loss is seldom used as a metric (the MAE or MSE is preferred).\n7 This class is for illustration purposes only. A simpler and better implementation would just\nsubclass the keras.metrics.Mean class; see the “Streaming metrics” section of the\nnotebook for an example.\n8 This function is specific to tf.keras. You could use keras.activations.Activation\ninstead.\n9 The Keras API calls this argument input_shape, but since it also includes the batch\ndimension, I prefer to call it batch_input_shape. Same for compute_output_shape().\n10 The name “Subclassing API” usually refers only to the creation of custom models by\nsubclassing, although many other things can be created by subclassing, as we saw in this\nchapter.\n11 You can also call add_loss() on any layer inside the model, as the model recursively\ngathers losses from all of its layers.\n12 If the tape goes out of scope, for example when the function that used it returns, Python’s\ngarbage collector will delete it for you.\n13 The truth is we did not process every single instance in the training set, because we\nsampled instances randomly: some were processed more than once, while others were not\nprocessed at all. Likewise, if the training set size is not a multiple of the batch size, we will\nmiss a few instances. In practice that’s fine.\n14 With the exception of optimizers, as very few people ever customize these; see the\n“Custom Optimizers” section in the notebook for an example.\n15 However, in this trivial example, the computation graph is so small that there is nothing at\nall to optimize, so tf_cube() actually runs much slower than cube()." + }, + { + "chapter": 13, + "text": "Chapter 13. Loading and\nPreprocessing Data with\nTensorFlow\nSo far we have used only datasets that fit in memory, but Deep Learning\nsystems are often trained on very large datasets that will not fit in RAM.\nIngesting a large dataset and preprocessing it efficiently can be tricky to\nimplement with other Deep Learning libraries, but TensorFlow makes it\neasy thanks to the Data API: you just create a dataset object, and tell it\nwhere to get the data and how to transform it. TensorFlow takes care of all\nthe implementation details, such as multithreading, queuing, batching, and\nprefetching. Moreover, the Data API works seamlessly with tf.keras!\nOff the shelf, the Data API can read from text files (such as CSV files),\nbinary files with fixed-size records, and binary files that use TensorFlow’s\nTFRecord format, which supports records of varying sizes. TFRecord is a\nflexible and efficient binary format based on Protocol Buffers (an open\nsource binary format). The Data API also has support for reading from\nSQL databases. Moreover, many open source extensions are available to\nread from all sorts of data sources, such as Google’s BigQuery service.\nReading huge datasets efficiently is not the only difficulty: the data also\nneeds to be preprocessed, usually normalized. Moreover, it is not always\ncomposed strictly of convenient numerical fields: there may be text\nfeatures, categorical features, and so on. These need to be encoded, for\nexample using one-hot encoding, bag-of-words encoding, or embeddings\n(as we will see, an embedding is a trainable dense vector that represents a\ncategory or token). One option to handle all this preprocessing is to write\nyour own custom preprocessing layers. Another is to use the standard\npreprocessing layers provided by Keras.\nIn this chapter, we will cover the Data API, the TFRecord format, and how\nto create custom preprocessing layers and use the standard Keras ones. We\nwill also take a quick look at a few related projects from TensorFlow’s\necosystem:\nTF Transform (tf.Transform)\nMakes it possible to write a single preprocessing function that can be\nrun in batch mode on your full training set, before training (to speed it\nup), and then exported to a TF Function and incorporated into your\ntrained model so that once it is deployed in production it can take care\nof preprocessing new instances on the fly.\nTF Datasets (TFDS)\nProvides a convenient function to download many common datasets of\nall kinds, including large ones like ImageNet, as well as convenient\ndataset objects to manipulate them using the Data API.\nSo let’s get started!\nThe Data API\nThe whole Data API revolves around the concept of a dataset: as you\nmight suspect, this represents a sequence of data items. Usually you will\nuse datasets that gradually read data from disk, but for simplicity let’s\ncreate a dataset entirely in RAM using\ntf.data.Dataset.from_tensor_slices():\n>>> X = tf.range(10) # any data tensor \n>>> dataset = tf.data.Dataset.from_tensor_slices(X) \n>>> dataset \n\nThe from_tensor_slices() function takes a tensor and creates a\ntf.data.Dataset whose elements are all the slices of X (along the first\ndimension), so this dataset contains 10 items: tensors 0, 1, 2, …, 9. In this\ncase we would have obtained the same dataset if we had used\ntf.data.Dataset.range(10).\nYou can simply iterate over a dataset’s items like this:\n>>> for item in dataset: \n... print(item) \n... \ntf.Tensor(0, shape=(), dtype=int32) \ntf.Tensor(1, shape=(), dtype=int32) \ntf.Tensor(2, shape=(), dtype=int32) \n[...] \ntf.Tensor(9, shape=(), dtype=int32)\nChaining Transformations\nOnce you have a dataset, you can apply all sorts of transformations to it by\ncalling its transformation methods. Each method returns a new dataset, so\nyou can chain transformations like this (this chain is illustrated in\nFigure 13-1):\n>>> dataset = dataset.repeat(3).batch(7) \n>>> for item in dataset: \n... print(item) \n... \ntf.Tensor([0 1 2 3 4 5 6], shape=(7,), dtype=int32) \ntf.Tensor([7 8 9 0 1 2 3], shape=(7,), dtype=int32) \ntf.Tensor([4 5 6 7 8 9 0], shape=(7,), dtype=int32) \ntf.Tensor([1 2 3 4 5 6 7], shape=(7,), dtype=int32) \ntf.Tensor([8 9], shape=(2,), dtype=int32)\nFigure 13-1. Chaining dataset transformations\nIn this example, we first call the repeat() method on the original dataset,\nand it returns a new dataset that will repeat the items of the original\ndataset three times. Of course, this will not copy all the data in memory\nthree times! (If you call this method with no arguments, the new dataset\nwill repeat the source dataset forever, so the code that iterates over the\ndataset will have to decide when to stop.) Then we call the batch()\nmethod on this new dataset, and again this creates a new dataset. This one\nwill group the items of the previous dataset in batches of seven items.\nFinally, we iterate over the items of this final dataset. As you can see, the\nbatch() method had to output a final batch of size two instead of seven,\nbut you can call it with drop_remainder=True if you want it to drop this\nfinal batch so that all batches have the exact same size.\nWARNING\nThe dataset methods do not modify datasets, they create new ones, so make sure to\nkeep a reference to these new datasets (e.g., with dataset = ...), or else nothing\nwill happen.\nYou can also transform the items by calling the map() method. For\nexample, this creates a new dataset with all items doubled:\n>>> dataset = dataset.map(lambda x: x * 2) # Items: [0,2,4,6,8,10,12]\nThis function is the one you will call to apply any preprocessing you want\nto your data. Sometimes this will include computations that can be quite\nintensive, such as reshaping or rotating an image, so you will usually want\nto spawn multiple threads to speed things up: it’s as simple as setting the\nnum_parallel_calls argument. Note that the function you pass to the\nmap() method must be convertible to a TF Function (see Chapter 12).\nWhile the map() method applies a transformation to each item, the\napply() method applies a transformation to the dataset as a whole. For\nexample, the following code applies the unbatch() function to the dataset\n(this function is currently experimental, but it will most likely move to the\ncore API in a future release). Each item in the new dataset will be a single-\ninteger tensor instead of a batch of seven integers:\n>>> dataset = dataset.apply(tf.data.experimental.unbatch()) # Items: \n0,2,4,...\nIt is also possible to simply filter the dataset using the filter() method:\n>>> dataset = dataset.filter(lambda x: x < 10) # Items: 0 2 4 6 8 0 2 4 \n6...\nYou will often want to look at just a few items from a dataset. You can use\nthe take() method for that:\n>>> for item in dataset.take(3): \n... print(item) \n... \ntf.Tensor(0, shape=(), dtype=int64) \ntf.Tensor(2, shape=(), dtype=int64) \ntf.Tensor(4, shape=(), dtype=int64)\nShuffling the Data\nAs you know, Gradient Descent works best when the instances in the\ntraining set are independent and identically distributed (see Chapter 4). A\nsimple way to ensure this is to shuffle the instances, using the shuffle()\nmethod. It will create a new dataset that will start by filling up a buffer\nwith the first items of the source dataset. Then, whenever it is asked for an\nitem, it will pull one out randomly from the buffer and replace it with a\nfresh one from the source dataset, until it has iterated entirely through the\nsource dataset. At this point it continues to pull out items randomly from\nthe buffer until it is empty. You must specify the buffer size, and it is\nimportant to make it large enough, or else shuffling will not be very\neffective. Just don’t exceed the amount of RAM you have, and even if\nyou have plenty of it, there’s no need to go beyond the dataset’s size. You\ncan provide a random seed if you want the same random order every time\nyou run your program. For example, the following code creates and\ndisplays a dataset containing the integers 0 to 9, repeated 3 times, shuffled\nusing a buffer of size 5 and a random seed of 42, and batched with a batch\nsize of 7:\n>>> dataset = tf.data.Dataset.range(10).repeat(3) # 0 to 9, three times \n>>> dataset = dataset.shuffle(buffer_size=5, seed=42).batch(7) \n>>> for item in dataset: \n... print(item) \n... \ntf.Tensor([0 2 3 6 7 9 4], shape=(7,), dtype=int64) \ntf.Tensor([5 0 1 1 8 6 5], shape=(7,), dtype=int64) \ntf.Tensor([4 8 7 1 2 3 0], shape=(7,), dtype=int64) \ntf.Tensor([5 4 2 7 8 9 9], shape=(7,), dtype=int64) \ntf.Tensor([3 6], shape=(2,), dtype=int64)\nTIP\nIf you call repeat() on a shuffled dataset, by default it will generate a new order at\nevery iteration. This is generally a good idea, but if you prefer to reuse the same\norder at each iteration (e.g., for tests or debugging), you can set\nreshuffle_each_iteration=False.\nFor a large dataset that does not fit in memory, this simple shuffling-\nbuffer approach may not be sufficient, since the buffer will be small\ncompared to the dataset. One solution is to shuffle the source data itself\n1\n(for example, on Linux you can shuffle text files using the shuf\ncommand). This will definitely improve shuffling a lot! Even if the source\ndata is shuffled, you will usually want to shuffle it some more, or else the\nsame order will be repeated at each epoch, and the model may end up\nbeing biased (e.g., due to some spurious patterns present by chance in the\nsource data’s order). To shuffle the instances some more, a common\napproach is to split the source data into multiple files, then read them in a\nrandom order during training. However, instances located in the same file\nwill still end up close to each other. To avoid this you can pick multiple\nfiles randomly and read them simultaneously, interleaving their records.\nThen on top of that you can add a shuffling buffer using the shuffle()\nmethod. If all this sounds like a lot of work, don’t worry: the Data API\nmakes all this possible in just a few lines of code. Let’s see how to do this.\nInterleaving lines from multiple files\nFirst, let’s suppose that you’ve loaded the California housing dataset,\nshuffled it (unless it was already shuffled), and split it into a training set, a\nvalidation set, and a test set. Then you split each set into many CSV files\nthat each look like this (each row contains eight input features plus the\ntarget median house value):\nMedInc,HouseAge,AveRooms,AveBedrms,Popul,AveOccup,Lat,Long,MedianHouseVal\nue \n3.5214,15.0,3.0499,1.1065,1447.0,1.6059,37.63,-122.43,1.442 \n5.3275,5.0,6.4900,0.9910,3464.0,3.4433,33.69,-117.39,1.687 \n3.1,29.0,7.5423,1.5915,1328.0,2.2508,38.44,-122.98,1.621 \n[...]\nLet’s also suppose train_filepaths contains the list of training file paths\n(and you also have valid_filepaths and test_filepaths):\n>>> train_filepaths \n['datasets/housing/my_train_00.csv', \n'datasets/housing/my_train_01.csv',...]\nAlternatively, you could use file patterns; for example, train_filepaths\n= \"datasets/housing/my_train_*.csv\". Now let’s create a dataset\ncontaining only these file paths:\nfilepath_dataset = tf.data.Dataset.list_files(train_filepaths, seed=42)\nBy default, the list_files() function returns a dataset that shuffles the\nfile paths. In general this is a good thing, but you can set shuffle=False\nif you do not want that for some reason.\nNext, you can call the interleave() method to read from five files at a\ntime and interleave their lines (skipping the first line of each file, which is\nthe header row, using the skip() method):\nn_readers = 5 \ndataset = filepath_dataset.interleave( \n lambda filepath: tf.data.TextLineDataset(filepath).skip(1), \n cycle_length=n_readers)\nThe interleave() method will create a dataset that will pull five file\npaths from the filepath_dataset, and for each one it will call the\nfunction you gave it (a lambda in this example) to create a new dataset (in\nthis case a TextLineDataset). To be clear, at this stage there will be seven\ndatasets in all: the filepath dataset, the interleave dataset, and the five\nTextLineDatasets created internally by the interleave dataset. When we\niterate over the interleave dataset, it will cycle through these five\nTextLineDatasets, reading one line at a time from each until all datasets\nare out of items. Then it will get the next five file paths from the\nfilepath_dataset and interleave them the same way, and so on until it\nruns out of file paths.\nTIP\nFor interleaving to work best, it is preferable to have files of identical length;\notherwise the ends of the longest files will not be interleaved.\nBy default, interleave() does not use parallelism; it just reads one line\nat a time from each file, sequentially. If you want it to actually read files\nin parallel, you can set the num_parallel_calls argument to the number\nof threads you want (note that the map() method also has this argument).\nYou can even set it to tf.data.experimental.AUTOTUNE to make\nTensorFlow choose the right number of threads dynamically based on the\navailable CPU (however, this is an experimental feature for now). Let’s\nlook at what the dataset contains now:\n>>> for line in dataset.take(5): \n... print(line.numpy()) \n... \nb'4.2083,44.0,5.3232,0.9171,846.0,2.3370,37.47,-122.2,2.782' \nb'4.1812,52.0,5.7013,0.9965,692.0,2.4027,33.73,-118.31,3.215' \nb'3.6875,44.0,4.5244,0.9930,457.0,3.1958,34.04,-118.15,1.625' \nb'3.3456,37.0,4.5140,0.9084,458.0,3.2253,36.67,-121.7,2.526' \nb'3.5214,15.0,3.0499,1.1065,1447.0,1.6059,37.63,-122.43,1.442'\nThese are the first rows (ignoring the header row) of five CSV files,\nchosen randomly. Looks good! But as you can see, these are just byte\nstrings; we need to parse them and scale the data.\nPreprocessing the Data\nLet’s implement a small function that will perform this preprocessing:\nX_mean, X_std = [...] # mean and scale of each feature in the training \nset \nn_inputs = 8 \n \ndef preprocess(line): \n defs = [0.] * n_inputs + [tf.constant([], dtype=tf.float32)] \n fields = tf.io.decode_csv(line, record_defaults=defs) \n x = tf.stack(fields[:-1]) \n y = tf.stack(fields[-1:]) \n return (x - X_mean) / X_std, y\nLet’s walk through this code:\nFirst, the code assumes that we have precomputed the mean and\nstandard deviation of each feature in the training set. X_mean and\nX_std are just 1D tensors (or NumPy arrays) containing eight\nfloats, one per input feature.\nThe preprocess() function takes one CSV line and starts by\nparsing it. For this it uses the tf.io.decode_csv() function,\nwhich takes two arguments: the first is the line to parse, and the\nsecond is an array containing the default value for each column in\nthe CSV file. This array tells TensorFlow not only the default\nvalue for each column, but also the number of columns and their\ntypes. In this example, we tell it that all feature columns are floats\nand that missing values should default to 0, but we provide an\nempty array of type tf.float32 as the default value for the last\ncolumn (the target): the array tells TensorFlow that this column\ncontains floats, but that there is no default value, so it will raise\nan exception if it encounters a missing value.\nThe decode_csv() function returns a list of scalar tensors (one\nper column), but we need to return 1D tensor arrays. So we call\ntf.stack() on all tensors except for the last one (the target): this\nwill stack these tensors into a 1D array. We then do the same for\nthe target value (this makes it a 1D tensor array with a single\nvalue, rather than a scalar tensor).\nFinally, we scale the input features by subtracting the feature\nmeans and then dividing by the feature standard deviations, and\nwe return a tuple containing the scaled features and the target.\nLet’s test this preprocessing function:\n>>> \npreprocess(b'4.2083,44.0,5.3232,0.9171,846.0,2.3370,37.47,-122.2,2.782') \n(, \n )\nLooks good! We can now apply the function to the dataset.\nPutting Everything Together\nTo make the code reusable, let’s put together everything we have discussed\nso far into a small helper function: it will create and return a dataset that\nwill efficiently load California housing data from multiple CSV files,\npreprocess it, shuffle it, optionally repeat it, and batch it (see Figure 13-2):\ndef csv_reader_dataset(filepaths, repeat=1, n_readers=5, \n n_read_threads=None, shuffle_buffer_size=10000, \n n_parse_threads=5, batch_size=32): \n dataset = tf.data.Dataset.list_files(filepaths) \n dataset = dataset.interleave( \n lambda filepath: tf.data.TextLineDataset(filepath).skip(1), \n cycle_length=n_readers, num_parallel_calls=n_read_threads) \n dataset = dataset.map(preprocess, num_parallel_calls=n_parse_threads) \n dataset = dataset.shuffle(shuffle_buffer_size).repeat(repeat) \n return dataset.batch(batch_size).prefetch(1)\nEverything should make sense in this code, except the very last line\n(prefetch(1)), which is important for performance.\nFigure 13-2. Loading and preprocessing data from multiple CSV files\nPrefetching\nBy calling prefetch(1) at the end, we are creating a dataset that will do\nits best to always be one batch ahead. In other words, while our training\nalgorithm is working on one batch, the dataset will already be working in\nparallel on getting the next batch ready (e.g., reading the data from disk\nand preprocessing it). This can improve performance dramatically, as is\nillustrated in Figure 13-3. If we also ensure that loading and preprocessing\nare multithreaded (by setting num_parallel_calls when calling\ninterleave() and map()), we can exploit multiple cores on the CPU and\nhopefully make preparing one batch of data shorter than running a training\nstep on the GPU: this way the GPU will be almost 100% utilized (except\nfor the data transfer time from the CPU to the GPU ), and training will run\nmuch faster.\n2\n3\nFigure 13-3. With prefetching, the CPU and the GPU work in parallel: as the GPU works on one\nbatch, the CPU works on the next\nTIP\nIf you plan to purchase a GPU card, its processing power and its memory size are of\ncourse very important (in particular, a large amount of RAM is crucial for computer\nvision). Just as important to get good performance is its memory bandwidth; this is\nthe number of gigabytes of data it can get into or out of its RAM per second.\nIf the dataset is small enough to fit in memory, you can significantly speed\nup training by using the dataset’s cache() method to cache its content to\nRAM. You should generally do this after loading and preprocessing the\ndata, but before shuffling, repeating, batching, and prefetching. This way,\neach instance will only be read and preprocessed once (instead of once per\nepoch), but the data will still be shuffled differently at each epoch, and the\nnext batch will still be prepared in advance.\nYou now know how to build efficient input pipelines to load and\npreprocess data from multiple text files. We have discussed the most\ncommon dataset methods, but there are a few more you may want to look\nat: concatenate(), zip(), window(), reduce(), shard(), flat_map(),\nand padded_batch(). There are also a couple more class methods:\nfrom_generator() and from_tensors(), which create a new dataset\nfrom a Python generator or a list of tensors, respectively. Please check the\nAPI documentation for more details. Also note that there are experimental\nfeatures available in tf.data.experimental, many of which will likely\nmake it to the core API in future releases (e.g., check out the CsvDataset\nclass, as well as the make_csv_dataset() method, which takes care of\ninferring the type of each column).\nUsing the Dataset with tf.keras\nNow we can use the csv_reader_dataset() function to create a dataset\nfor the training set. Note that we do not need to repeat it, as this will be\ntaken care of by tf.keras. We also create datasets for the validation set and\nthe test set:\ntrain_set = csv_reader_dataset(train_filepaths) \nvalid_set = csv_reader_dataset(valid_filepaths) \ntest_set = csv_reader_dataset(test_filepaths)\nAnd now we can simply build and train a Keras model using these\ndatasets. All we need to do is pass the training and validation datasets to\nthe fit() method, instead of X_train, y_train, X_valid, and y_valid:\nmodel = keras.models.Sequential([...]) \nmodel.compile([...]) \nmodel.fit(train_set, epochs=10, validation_data=valid_set)\n4\n5\nSimilarly, we can pass a dataset to the evaluate() and predict()\nmethods:\nmodel.evaluate(test_set) \nnew_set = test_set.take(3).map(lambda X, y: X) # pretend we have 3 new \ninstances \nmodel.predict(new_set) # a dataset containing new instances\nUnlike the other sets, the new_set will usually not contain labels (if it\ndoes, Keras will ignore them). Note that in all these cases, you can still use\nNumPy arrays instead of datasets if you want (but of course they need to\nhave been loaded and preprocessed first).\nIf you want to build your own custom training loop (as in Chapter 12), you\ncan just iterate over the training set, very naturally:\nfor X_batch, y_batch in train_set: \n [...] # perform one Gradient Descent step\nIn fact, it is even possible to create a TF Function (see Chapter 12) that\nperforms the whole training loop:\n@tf.function \ndef train(model, optimizer, loss_fn, n_epochs, [...]): \n train_set = csv_reader_dataset(train_filepaths, repeat=n_epochs, \n[...]) \n for X_batch, y_batch in train_set: \n with tf.GradientTape() as tape: \n y_pred = model(X_batch) \n main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred)) \n loss = tf.add_n([main_loss] + model.losses) \n grads = tape.gradient(loss, model.trainable_variables) \n optimizer.apply_gradients(zip(grads, model.trainable_variables))\nCongratulations, you now know how to build powerful input pipelines\nusing the Data API! However, so far we have used CSV files, which are\ncommon, simple, and convenient but not really efficient, and do not\nsupport large or complex data structures (such as images or audio) very\nwell. So let’s see how to use TFRecords instead.\nTIP\nIf you are happy with CSV files (or whatever other format you are using), you do not\nhave to use TFRecords. As the saying goes, if it ain’t broke, don’t fix it! TFRecords\nare useful when the bottleneck during training is loading and parsing the data.\nThe TFRecord Format\nThe TFRecord format is TensorFlow’s preferred format for storing large\namounts of data and reading it efficiently. It is a very simple binary format\nthat just contains a sequence of binary records of varying sizes (each\nrecord is comprised of a length, a CRC checksum to check that the length\nwas not corrupted, then the actual data, and finally a CRC checksum for\nthe data). You can easily create a TFRecord file using the\ntf.io.TFRecordWriter class:\nwith tf.io.TFRecordWriter(\"my_data.tfrecord\") as f: \n f.write(b\"This is the first record\") \n f.write(b\"And this is the second record\")\nAnd you can then use a tf.data.TFRecordDataset to read one or more\nTFRecord files:\nfilepaths = [\"my_data.tfrecord\"] \ndataset = tf.data.TFRecordDataset(filepaths) \nfor item in dataset: \n print(item)\nThis will output:\ntf.Tensor(b'This is the first record', shape=(), dtype=string) \ntf.Tensor(b'And this is the second record', shape=(), dtype=string)\nTIP\nBy default, a TFRecordDataset will read files one by one, but you can make it read\nmultiple files in parallel and interleave their records by setting\nnum_parallel_reads. Alternatively, you could obtain the same result by using\nlist_files() and interleave() as we did earlier to read multiple CSV files.\nCompressed TFRecord Files\nIt can sometimes be useful to compress your TFRecord files, especially if\nthey need to be loaded via a network connection. You can create a\ncompressed TFRecord file by setting the options argument:\noptions = tf.io.TFRecordOptions(compression_type=\"GZIP\") \nwith tf.io.TFRecordWriter(\"my_compressed.tfrecord\", options) as f: \n [...]\nWhen reading a compressed TFRecord file, you need to specify the\ncompression type:\ndataset = tf.data.TFRecordDataset([\"my_compressed.tfrecord\"], \n compression_type=\"GZIP\")\nA Brief Introduction to Protocol Buffers\nEven though each record can use any binary format you want, TFRecord\nfiles usually contain serialized protocol buffers (also called protobufs).\nThis is a portable, extensible, and efficient binary format developed at\nGoogle back in 2001 and made open source in 2008; protobufs are now\nwidely used, in particular in gRPC, Google’s remote procedure call\nsystem. They are defined using a simple language that looks like this:\nsyntax = \"proto3\"; \nmessage Person { \n string name = 1; \n int32 id = 2; \n repeated string email = 3; \n}\nThis definition says we are using version 3 of the protobuf format, and it\nspecifies that each Person object may (optionally) have a name of type\nstring, an id of type int32, and zero or more email fields, each of type\nstring. The numbers 1, 2, and 3 are the field identifiers: they will be used\nin each record’s binary representation. Once you have a definition in a\n.proto file, you can compile it. This requires protoc, the protobuf\ncompiler, to generate access classes in Python (or some other language).\nNote that the protobuf definitions we will use have already been compiled\nfor you, and their Python classes are part of TensorFlow, so you will not\nneed to use protoc. All you need to know is how to use protobuf access\nclasses in Python. To illustrate the basics, let’s look at a simple example\nthat uses the access classes generated for the Person protobuf (the code is\nexplained in the comments):\n>>> from person_pb2 import Person # import the generated access class \n>>> person = Person(name=\"Al\", id=123, email=[\"a@b.com\"]) # create a \nPerson \n>>> print(person) # display the Person \nname: \"Al\" \nid: 123 \nemail: \"a@b.com\" \n>>> person.name # read a field \n\"Al\" \n>>> person.name = \"Alice\" # modify a field \n>>> person.email[0] # repeated fields can be accessed like arrays \n\"a@b.com\" \n>>> person.email.append(\"c@d.com\") # add an email address \n>>> s = person.SerializeToString() # serialize the object to a byte \nstring \n>>> s \nb'\\n\\x05Alice\\x10{\\x1a\\x07a@b.com\\x1a\\x07c@d.com' \n>>> person2 = Person() # create a new Person \n>>> person2.ParseFromString(s) # parse the byte string (27 bytes long) \n27 \n>>> person == person2 # now they are equal \nTrue\nIn short, we import the Person class generated by protoc, we create an\ninstance and play with it, visualizing it and reading and writing some\n6\nfields, then we serialize it using the SerializeToString() method. This\nis the binary data that is ready to be saved or transmitted over the network.\nWhen reading or receiving this binary data, we can parse it using the\nParseFromString() method, and we get a copy of the object that was\nserialized.\nWe could save the serialized Person object to a TFRecord file, then we\ncould load and parse it: everything would work fine. However,\nSerializeToString() and ParseFromString() are not TensorFlow\noperations (and neither are the other operations in this code), so they\ncannot be included in a TensorFlow Function (except by wrapping them in\na tf.py_function() operation, which would make the code slower and\nless portable, as we saw in Chapter 12). Fortunately, TensorFlow does\ninclude special protobuf definitions for which it provides parsing\noperations.\nTensorFlow Protobufs\nThe main protobuf typically used in a TFRecord file is the Example\nprotobuf, which represents one instance in a dataset. It contains a list of\nnamed features, where each feature can either be a list of byte strings, a\nlist of floats, or a list of integers. Here is the protobuf definition:\nsyntax = \"proto3\"; \nmessage BytesList { repeated bytes value = 1; } \nmessage FloatList { repeated float value = 1 [packed = true]; } \nmessage Int64List { repeated int64 value = 1 [packed = true]; } \nmessage Feature { \n oneof kind { \n BytesList bytes_list = 1; \n FloatList float_list = 2; \n Int64List int64_list = 3; \n } \n}; \nmessage Features { map feature = 1; }; \nmessage Example { Features features = 1; };\n7\nThe definitions of BytesList, FloatList, and Int64List are\nstraightforward enough. Note that [packed = true] is used for repeated\nnumerical fields, for a more efficient encoding. A Feature contains either\na BytesList, a FloatList, or an Int64List. A Features (with an s)\ncontains a dictionary that maps a feature name to the corresponding\nfeature value. And finally, an Example contains only a Features object.\nHere is how you could create a tf.train.Example representing the same\nperson as earlier and write it to a TFRecord file:\nfrom tensorflow.train import BytesList, FloatList, Int64List \nfrom tensorflow.train import Feature, Features, Example \n \nperson_example = Example( \n features=Features( \n feature={ \n \"name\": Feature(bytes_list=BytesList(value=[b\"Alice\"])), \n \"id\": Feature(int64_list=Int64List(value=[123])), \n \"emails\": Feature(bytes_list=BytesList(value=[b\"a@b.com\", \n b\"c@d.com\"])) \n }))\nThe code is a bit verbose and repetitive, but it’s rather straightforward (and\nyou could easily wrap it inside a small helper function). Now that we have\nan Example protobuf, we can serialize it by calling its\nSerializeToString() method, then write the resulting data to a\nTFRecord file:\nwith tf.io.TFRecordWriter(\"my_contacts.tfrecord\") as f: \n f.write(person_example.SerializeToString())\nNormally you would write much more than one Example! Typically, you\nwould create a conversion script that reads from your current format (say,\nCSV files), creates an Example protobuf for each instance, serializes them,\nand saves them to several TFRecord files, ideally shuffling them in the\nprocess. This requires a bit of work, so once again make sure it is really\nnecessary (perhaps your pipeline works fine with CSV files).\n8\nNow that we have a nice TFRecord file containing a serialized Example,\nlet’s try to load it.\nLoading and Parsing Examples\nTo load the serialized Example protobufs, we will use a\ntf.data.TFRecordDataset once again, and we will parse each Example\nusing tf.io.parse_single_example(). This is a TensorFlow operation,\nso it can be included in a TF Function. It requires at least two arguments: a\nstring scalar tensor containing the serialized data, and a description of\neach feature. The description is a dictionary that maps each feature name\nto either a tf.io.FixedLenFeature descriptor indicating the feature’s\nshape, type, and default value, or a tf.io.VarLenFeature descriptor\nindicating only the type (if the length of the feature’s list may vary, such as\nfor the \"emails\" feature).\nThe following code defines a description dictionary, then it iterates over\nthe TFRecordDataset and parses the serialized Example protobuf this\ndataset contains:\nfeature_description = { \n \"name\": tf.io.FixedLenFeature([], tf.string, default_value=\"\"), \n \"id\": tf.io.FixedLenFeature([], tf.int64, default_value=0), \n \"emails\": tf.io.VarLenFeature(tf.string), \n} \n \nfor serialized_example in \ntf.data.TFRecordDataset([\"my_contacts.tfrecord\"]): \n parsed_example = tf.io.parse_single_example(serialized_example, \n feature_description)\nThe fixed-length features are parsed as regular tensors, but the variable-\nlength features are parsed as sparse tensors. You can convert a sparse\ntensor to a dense tensor using tf.sparse.to_dense(), but in this case it\nis simpler to just access its values:\n>>> tf.sparse.to_dense(parsed_example[\"emails\"], default_value=b\"\") \n \n>>> parsed_example[\"emails\"].values \n\nA BytesList can contain any binary data you want, including any\nserialized object. For example, you can use tf.io.encode_jpeg() to\nencode an image using the JPEG format and put this binary data in a\nBytesList. Later, when your code reads the TFRecord, it will start by\nparsing the Example, then it will need to call tf.io.decode_jpeg() to\nparse the data and get the original image (or you can use\ntf.io.decode_image(), which can decode any BMP, GIF, JPEG, or PNG\nimage). You can also store any tensor you want in a BytesList by\nserializing the tensor using tf.io.serialize_tensor() then putting the\nresulting byte string in a BytesList feature. Later, when you parse the\nTFRecord, you can parse this data using tf.io.parse_tensor().\nInstead of parsing examples one by one using\ntf.io.parse_single_example(), you may want to parse them batch by\nbatch using tf.io.parse_example():\ndataset = tf.data.TFRecordDataset([\"my_contacts.tfrecord\"]).batch(10) \nfor serialized_examples in dataset: \n parsed_examples = tf.io.parse_example(serialized_examples, \n feature_description)\nAs you can see, the Example protobuf will probably be sufficient for most\nuse cases. However, it may be a bit cumbersome to use when you are\ndealing with lists of lists. For example, suppose you want to classify text\ndocuments. Each document may be represented as a list of sentences,\nwhere each sentence is represented as a list of words. And perhaps each\ndocument also has a list of comments, where each comment is represented\nas a list of words. There may be some contextual data too, such as the\ndocument’s author, title, and publication date. TensorFlow’s\nSequenceExample protobuf is designed for such use cases.\nHandling Lists of Lists Using the SequenceExample\nProtobuf\nHere is the definition of the SequenceExample protobuf:\nmessage FeatureList { repeated Feature feature = 1; }; \nmessage FeatureLists { map feature_list = 1; }; \nmessage SequenceExample { \n Features context = 1; \n FeatureLists feature_lists = 2; \n};\nA SequenceExample contains a Features object for the contextual data\nand a FeatureLists object that contains one or more named FeatureList\nobjects (e.g., a FeatureList named \"content\" and another named\n\"comments\"). Each FeatureList contains a list of Feature objects, each\nof which may be a list of byte strings, a list of 64-bit integers, or a list of\nfloats (in this example, each Feature would represent a sentence or a\ncomment, perhaps in the form of a list of word identifiers). Building a\nSequenceExample, serializing it, and parsing it is similar to building,\nserializing, and parsing an Example, but you must use\ntf.io.parse_single_sequence_example() to parse a single\nSequenceExample or tf.io.parse_sequence_example() to parse a\nbatch. Both functions return a tuple containing the context features (as a\ndictionary) and the feature lists (also as a dictionary). If the feature lists\ncontain sequences of varying sizes (as in the preceding example), you may\nwant to convert them to ragged tensors, using\ntf.RaggedTensor.from_sparse() (see the notebook for the full code):\nparsed_context, parsed_feature_lists = \ntf.io.parse_single_sequence_example( \n serialized_sequence_example, context_feature_descriptions, \n sequence_feature_descriptions) \nparsed_content = \ntf.RaggedTensor.from_sparse(parsed_feature_lists[\"content\"])\nNow that you know how to efficiently store, load, and parse data, the next\nstep is to prepare it so that it can be fed to a neural network.\nPreprocessing the Input Features\nPreparing your data for a neural network requires converting all features\ninto numerical features, generally normalizing them, and more. In\nparticular, if your data contains categorical features or text features, they\nneed to be converted to numbers. This can be done ahead of time when\npreparing your data files, using any tool you like (e.g., NumPy, pandas, or\nScikit-Learn). Alternatively, you can preprocess your data on the fly when\nloading it with the Data API (e.g., using the dataset’s map() method, as we\nsaw earlier), or you can include a preprocessing layer directly in your\nmodel. Let’s look at this last option now.\nFor example, here is how you can implement a standardization layer using\na Lambda layer. For each feature, it subtracts the mean and divides by its\nstandard deviation (plus a tiny smoothing term to avoid division by zero):\nmeans = np.mean(X_train, axis=0, keepdims=True) \nstds = np.std(X_train, axis=0, keepdims=True) \neps = keras.backend.epsilon() \nmodel = keras.models.Sequential([ \n keras.layers.Lambda(lambda inputs: (inputs - means) / (stds + eps)), \n [...] # other layers \n])\nThat’s not too hard! However, you may prefer to use a nice self-contained\ncustom layer (much like Scikit-Learn’s StandardScaler), rather than\nhaving global variables like means and stds dangling around:\nclass Standardization(keras.layers.Layer): \n def adapt(self, data_sample): \n self.means_ = np.mean(data_sample, axis=0, keepdims=True) \n self.stds_ = np.std(data_sample, axis=0, keepdims=True) \n_\np\n(\n_\np\n,\n,\np\n)\n def call(self, inputs): \n return (inputs - self.means_) / (self.stds_ + \nkeras.backend.epsilon())\nBefore you can use this standardization layer, you will need to adapt it to\nyour dataset by calling the adapt() method and passing it a data sample.\nThis will allow it to use the appropriate mean and standard deviation for\neach feature:\nstd_layer = Standardization() \nstd_layer.adapt(data_sample)\nThis sample must be large enough to be representative of your dataset, but\nit does not have to be the full training set: in general, a few hundred\nrandomly selected instances will suffice (however, this depends on your\ntask). Next, you can use this preprocessing layer like a normal layer:\nmodel = keras.Sequential() \nmodel.add(std_layer) \n[...] # create the rest of the model \nmodel.compile([...]) \nmodel.fit([...])\nIf you are thinking that Keras should contain a standardization layer like\nthis one, here’s some good news for you: by the time you read this, the\nkeras.layers.Normalization layer will probably be available. It will\nwork very much like our custom Standardization layer: first, create the\nlayer, then adapt it to your dataset by passing a data sample to the adapt()\nmethod, and finally use the layer normally.\nNow let’s look at categorical features. We will start by encoding them as\none-hot vectors.\nEncoding Categorical Features Using One-Hot Vectors\nConsider the ocean_proximity feature in the California housing dataset\nwe explored in Chapter 2: it is a categorical feature with five possible\nvalues: \"<1H OCEAN\", \"INLAND\", \"NEAR OCEAN\", \"NEAR BAY\", and\n\"ISLAND\". We need to encode this feature before we feed it to a neural\nnetwork. Since there are very few categories, we can use one-hot encoding.\nFor this, we first need to map each category to its index (0 to 4), which can\nbe done using a lookup table:\nvocab = [\"<1H OCEAN\", \"INLAND\", \"NEAR OCEAN\", \"NEAR BAY\", \"ISLAND\"] \nindices = tf.range(len(vocab), dtype=tf.int64) \ntable_init = tf.lookup.KeyValueTensorInitializer(vocab, indices) \nnum_oov_buckets = 2 \ntable = tf.lookup.StaticVocabularyTable(table_init, num_oov_buckets)\nLet’s go through this code:\nWe first define the vocabulary: this is the list of all possible\ncategories.\nThen we create a tensor with the corresponding indices (0 to 4).\nNext, we create an initializer for the lookup table, passing it the\nlist of categories and their corresponding indices. In this example,\nwe already have this data, so we use a\nKeyValueTensorInitializer; but if the categories were listed in\na text file (with one category per line), we would use a\nTextFileInitializer instead.\nIn the last two lines we create the lookup table, giving it the\ninitializer and specifying the number of out-of-vocabulary (oov)\nbuckets. If we look up a category that does not exist in the\nvocabulary, the lookup table will compute a hash of this category\nand use it to assign the unknown category to one of the oov\nbuckets. Their indices start after the known categories, so in this\nexample the indices of the two oov buckets are 5 and 6.\nWhy use oov buckets? Well, if the number of categories is large (e.g., zip\ncodes, cities, words, products, or users) and the dataset is large as well, or\nit keeps changing, then getting the full list of categories may not be\nconvenient. One solution is to define the vocabulary based on a data\nsample (rather than the whole training set) and add some oov buckets for\nthe other categories that were not in the data sample. The more unknown\ncategories you expect to find during training, the more oov buckets you\nshould use. Indeed, if there are not enough oov buckets, there will be\ncollisions: different categories will end up in the same bucket, so the\nneural network will not be able to distinguish them (at least not based on\nthis feature).\nNow let’s use the lookup table to encode a small batch of categorical\nfeatures to one-hot vectors:\n>>> categories = tf.constant([\"NEAR BAY\", \"DESERT\", \"INLAND\", \"INLAND\"]) \n>>> cat_indices = table.lookup(categories) \n>>> cat_indices \n \n>>> cat_one_hot = tf.one_hot(cat_indices, depth=len(vocab) + \nnum_oov_buckets) \n>>> cat_one_hot \n\nAs you can see, \"NEAR BAY\" was mapped to index 3, the unknown\ncategory \"DESERT\" was mapped to one of the two oov buckets (at index 5),\nand \"INLAND\" was mapped to index 1, twice. Then we used tf.one_hot()\nto one-hot encode these indices. Notice that we have to tell this function\nthe total number of indices, which is equal to the vocabulary size plus the\nnumber of oov buckets. Now you know how to encode categorical features\nto one-hot vectors using TensorFlow!\nJust like earlier, it wouldn’t be too difficult to bundle all of this logic into\na nice self-contained class. Its adapt() method would take a data sample\nand extract all the distinct categories it contains. It would create a lookup\ntable to map each category to its index (including unknown categories\nusing oov buckets). Then its call() method would use the lookup table to\nmap the input categories to their indices. Well, here’s more good news: by\nthe time you read this, Keras will probably include a layer called\nkeras.layers.TextVectorization, which will be capable of doing\nexactly that: its adapt() method will extract the vocabulary from a data\nsample, and its call() method will convert each category to its index in\nthe vocabulary. You could add this layer at the beginning of your model,\nfollowed by a Lambda layer that would apply the tf.one_hot() function,\nif you want to convert these indices to one-hot vectors.\nThis may not be the best solution, though. The size of each one-hot vector\nis the vocabulary length plus the number of oov buckets. This is fine when\nthere are just a few possible categories, but if the vocabulary is large, it is\nmuch more efficient to encode them using embeddings instead.\nTIP\nAs a rule of thumb, if the number of categories is lower than 10, then one-hot\nencoding is generally the way to go (but your mileage may vary!). If the number of\ncategories is greater than 50 (which is often the case when you use hash buckets),\nthen embeddings are usually preferable. In between 10 and 50 categories, you may\nwant to experiment with both options and see which one works best for your use\ncase.\nEncoding Categorical Features Using Embeddings\nAn embedding is a trainable dense vector that represents a category. By\ndefault, embeddings are initialized randomly, so for example the \"NEAR\nBAY\" category could be represented initially by a random vector such as\n[0.131, 0.890], while the \"NEAR OCEAN\" category might be represented\nby another random vector such as [0.631, 0.791]. In this example, we\nuse 2D embeddings, but the number of dimensions is a hyperparameter\nyou can tweak. Since these embeddings are trainable, they will gradually\nimprove during training; and as they represent fairly similar categories,\nGradient Descent will certainly end up pushing them closer together, while\nit will tend to move them away from the \"INLAND\" category’s embedding\n(see Figure 13-4). Indeed, the better the representation, the easier it will be\nfor the neural network to make accurate predictions, so training tends to\nmake embeddings useful representations of the categories. This is called\nrepresentation learning (we will see other types of representation learning\nin Chapter 17).\nFigure 13-4. Embeddings will gradually improve during training\nWORD EMBEDDINGS\nNot only will embeddings generally be useful representations for the\ntask at hand, but quite often these same embeddings can be reused\nsuccessfully for other tasks. The most common example of this is\nword embeddings (i.e., embeddings of individual words): when you are\nworking on a natural language processing task, you are often better off\nreusing pretrained word embeddings than training your own.\nThe idea of using vectors to represent words dates back to the 1960s,\nand many sophisticated techniques have been used to generate useful\nvectors, including using neural networks. But things really took off in\n2013, when Tomáš Mikolov and other Google researchers published a\npaper describing an efficient technique to learn word embeddings\nusing neural networks, significantly outperforming previous attempts.\nThis allowed them to learn embeddings on a very large corpus of text:\nthey trained a neural network to predict the words near any given\nword, and obtained astounding word embeddings. For example,\nsynonyms had very close embeddings, and semantically related words\nsuch as France, Spain, and Italy ended up clustered together.\nIt’s not just about proximity, though: word embeddings were also\norganized along meaningful axes in the embedding space. Here is a\nfamous example: if you compute King – Man + Woman (adding and\nsubtracting the embedding vectors of these words), then the result will\nbe very close to the embedding of the word Queen (see Figure 13-5).\nIn other words, the word embeddings encode the concept of gender!\nSimilarly, you can compute Madrid – Spain + France, and the result is\nclose to Paris, which seems to show that the notion of capital city was\nalso encoded in the embeddings.\n9\nFigure 13-5. Word embeddings of similar words tend to be close, and some axes seem to\nencode meaningful concepts\nUnfortunately, word embeddings sometimes capture our worst biases.\nFor example, although they correctly learn that Man is to King as\nWoman is to Queen, they also seem to learn that Man is to Doctor as\nWoman is to Nurse: quite a sexist bias! To be fair, this particular\nexample is probably exaggerated, as was pointed out in a 2019 paper\nby Malvina Nissim et al. Nevertheless, ensuring fairness in Deep\nLearning algorithms is an important and active research topic.\nLet’s look at how we could implement embeddings manually, to\nunderstand how they work (then we will use a simple Keras layer instead).\nFirst, we need to create an embedding matrix containing each category’s\nembedding, initialized randomly; it will have one row per category and per\noov bucket, and one column per embedding dimension:\nembedding_dim = 2 \nembed_init = tf.random.uniform([len(vocab) + num_oov_buckets, \nembedding_dim]) \nembedding_matrix = tf.Variable(embed_init)\n10\nIn this example we are using 2D embeddings, but as a rule of thumb\nembeddings typically have 10 to 300 dimensions, depending on the task\nand the vocabulary size (you will have to tune this hyperparameter).\nThis embedding matrix is a random 6 × 2 matrix, stored in a variable (so it\ncan be tweaked by Gradient Descent during training):\n>>> embedding_matrix \n\nNow let’s encode the same batch of categorical features as earlier, but this\ntime using these embeddings:\n>>> categories = tf.constant([\"NEAR BAY\", \"DESERT\", \"INLAND\", \"INLAND\"]) \n>>> cat_indices = table.lookup(categories) \n>>> cat_indices \n \n>>> tf.nn.embedding_lookup(embedding_matrix, cat_indices) \n\nThe tf.nn.embedding_lookup() function looks up the rows in the\nembedding matrix, at the given indices—that’s all it does. For example,\nthe lookup table says that the \"INLAND\" category is at index 1, so the\ntf.nn.embedding_lookup() function returns the embedding at row 1 in\nthe embedding matrix (twice): [0.3528825, 0.46448255].\nKeras provides a keras.layers.Embedding layer that handles the\nembedding matrix (trainable, by default); when the layer is created it\ninitializes the embedding matrix randomly, and then when it is called with\nsome category indices it returns the rows at those indices in the\nembedding matrix:\n>>> embedding = keras.layers.Embedding(input_dim=len(vocab) + \nnum_oov_buckets, \n... output_dim=embedding_dim) \n... \n>>> embedding(cat_indices) \n\nPutting everything together, we can now create a Keras model that can\nprocess categorical features (along with regular numerical features) and\nlearn an embedding for each category (as well as for each oov bucket):\nregular_inputs = keras.layers.Input(shape=[8]) \ncategories = keras.layers.Input(shape=[], dtype=tf.string) \ncat_indices = keras.layers.Lambda(lambda cats: table.lookup(cats))\n(categories) \ncat_embed = keras.layers.Embedding(input_dim=6, output_dim=2)\n(cat_indices) \nencoded_inputs = keras.layers.concatenate([regular_inputs, cat_embed]) \noutputs = keras.layers.Dense(1)(encoded_inputs) \nmodel = keras.models.Model(inputs=[regular_inputs, categories], \n outputs=[outputs])\nThis model takes two inputs: a regular input containing eight numerical\nfeatures per instance, plus a categorical input (containing one categorical\nfeature per instance). It uses a Lambda layer to look up each category’s\nindex, then it looks up the embeddings for these indices. Next, it\nconcatenates the embeddings and the regular inputs in order to give the\nencoded inputs, which are ready to be fed to a neural network. We could\nadd any kind of neural network at this point, but we just add a dense output\nlayer, and we create the Keras model.\nWhen the keras.layers.TextVectorization layer is available, you can\ncall its adapt() method to make it extract the vocabulary from a data\nsample (it will take care of creating the lookup table for you). Then you\ncan add it to your model, and it will perform the index lookup (replacing\nthe Lambda layer in the previous code example).\nNOTE\nOne-hot encoding followed by a Dense layer (with no activation function and no\nbiases) is equivalent to an Embedding layer. However, the Embedding layer uses\nway fewer computations (the performance difference becomes clear when the size of\nthe embedding matrix grows). The Dense layer’s weight matrix plays the role of the\nembedding matrix. For example, using one-hot vectors of size 20 and a Dense layer\nwith 10 units is equivalent to using an Embedding layer with input_dim=20 and\noutput_dim=10. As a result, it would be wasteful to use more embedding\ndimensions than the number of units in the layer that follows the Embedding layer.\nNow let’s look a bit more closely at the Keras preprocessing layers.\nKeras Preprocessing Layers\nThe TensorFlow team is working on providing a set of standard Keras\npreprocessing layers. They will probably be available by the time you read\nthis; however, the API may change slightly by then, so please refer to the\nnotebook for this chapter if anything behaves unexpectedly. This new API\nwill likely supersede the existing Feature Columns API, which is harder to\nuse and less intuitive (if you want to learn more about the Feature\nColumns API anyway, please check out the notebook for this chapter).\nWe already discussed two of these layers: the\nkeras.layers.Normalization layer that will perform feature\nstandardization (it will be equivalent to the Standardization layer we\ndefined earlier), and the TextVectorization layer that will be capable of\nencoding each word in the inputs into its index in the vocabulary. In both\ncases, you create the layer, you call its adapt() method with a data\nsample, and then you use the layer normally in your model. The other\npreprocessing layers will follow the same pattern.\nThe API will also include a keras.layers.Discretization layer that\nwill chop continuous data into different bins and encode each bin as a one-\nhot vector. For example, you could use it to discretize prices into three\ncategories, (low, medium, high), which would be encoded as [1, 0, 0], [0,\n1, 0], and [0, 0, 1], respectively. Of course this loses a lot of information,\nbut in some cases it can help the model detect patterns that would\notherwise not be obvious when just looking at the continuous values.\nWARNING\nThe Discretization layer will not be differentiable, and it should only be used at\nthe start of your model. Indeed, the model’s preprocessing layers will be frozen\nduring training, so their parameters will not be affected by Gradient Descent, and\nthus they do not need to be differentiable. This also means that you should not use\nan Embedding layer directly in a custom preprocessing layer, if you want it to be\ntrainable: instead, it should be added separately to your model, as in the previous\ncode example.\nIt will also be possible to chain multiple preprocessing layers using the\nPreprocessingStage class. For example, the following code will create a\npreprocessing pipeline that will first normalize the inputs, then discretize\nthem (this may remind you of Scikit-Learn pipelines). After you adapt this\npipeline to a data sample, you can use it like a regular layer in your\nmodels (but again, only at the start of the model, since it contains a\nnondifferentiable preprocessing layer):\nnormalization = keras.layers.Normalization() \ndiscretization = keras.layers.Discretization([...]) \npipeline = keras.layers.PreprocessingStage([normalization, \ndiscretization]) \npipeline.adapt(data_sample)\nThe TextVectorization layer will also have an option to output word-\ncount vectors instead of word indices. For example, if the vocabulary\ncontains three words, say [\"and\", \"basketball\", \"more\"], then the\ntext \"more and more\" will be mapped to the vector [1, 0, 2]: the word\n\"and\" appears once, the word \"basketball\" does not appear at all, and\nthe word \"more\" appears twice. This text representation is called a bag of\nwords, since it completely loses the order of the words. Common words\nlike \"and\" will have a large value in most texts, even though they are\nusually the least interesting (e.g., in the text \"more and more\nbasketball\" the word \"basketball\" is clearly the most important,\nprecisely because it is not a very frequent word). So, the word counts\nshould be normalized in a way that reduces the importance of frequent\nwords. A common way to do this is to divide each word count by the log of\nthe total number of training instances in which the word appears. This\ntechnique is called Term-Frequency × Inverse-Document-Frequency (TF-\nIDF). For example, let’s imagine that the words \"and\", \"basketball\", and\n\"more\" appear respectively in 200, 10, and 100 text instances in the\ntraining set: in this case, the final vector will be [1/log(200),\n0/log(10), 2/log(100)], which is approximately equal to [0.19, 0.,\n0.43]. The TextVectorization layer will (likely) have an option to\nperform TF-IDF.\nNOTE\nIf the standard preprocessing layers are insufficient for your task, you will still have\nthe option to create your own custom preprocessing layer, much like we did earlier\nwith the Standardization class. Create a subclass of the\nkeras.layers.PreprocessingLayer class with an adapt() method, which\nshould take a data_sample argument and optionally an extra reset_state\nargument: if True, then the adapt() method should reset any existing state before\ncomputing the new state; if False, it should try to update the existing state.\nAs you can see, these Keras preprocessing layers will make preprocessing\nmuch easier! Now, whether you choose to write your own preprocessing\nlayers or use Keras’s (or even use the Feature Columns API), all the\npreprocessing will be done on the fly. During training, however, it may be\npreferable to perform preprocessing ahead of time. Let’s see why we’d\nwant to do that and how we’d go about it.\nTF Transform\nIf preprocessing is computationally expensive, then handling it before\ntraining rather than on the fly may give you a significant speedup: the data\nwill be preprocessed just once per instance before training, rather than\nonce per instance and per epoch during training. As mentioned earlier, if\nthe dataset is small enough to fit in RAM, you can use its cache()\nmethod. But if it is too large, then tools like Apache Beam or Spark will\nhelp. They let you run efficient data processing pipelines over large\namounts of data, even distributed across multiple servers, so you can use\nthem to preprocess all the training data before training.\nThis works great and indeed can speed up training, but there is one\nproblem: once your model is trained, suppose you want to deploy it to a\nmobile app. In that case you will need to write some code in your app to\ntake care of preprocessing the data before it is fed to the model. And\nsuppose you also want to deploy the model to TensorFlow.js so that it runs\nin a web browser? Once again, you will need to write some preprocessing\ncode. This can become a maintenance nightmare: whenever you want to\nchange the preprocessing logic, you will need to update your Apache\nBeam code, your mobile app code, and your JavaScript code. This is not\nonly time-consuming, but also error-prone: you may end up with subtle\ndifferences between the preprocessing operations performed before\ntraining and the ones performed in your app or in the browser. This\ntraining/serving skew will lead to bugs or degraded performance.\nOne improvement would be to take the trained model (trained on data that\nwas preprocessed by your Apache Beam or Spark code) and, before\ndeploying it to your app or the browser, add extra preprocessing layers to\ntake care of preprocessing on the fly. That’s definitely better, since now\nyou just have two versions of your preprocessing code: the Apache Beam\nor Spark code, and the preprocessing layers’ code.\nBut what if you could define your preprocessing operations just once? This\nis what TF Transform was designed for. It is part of TensorFlow Extended\n(TFX), an end-to-end platform for productionizing TensorFlow models.\nFirst, to use a TFX component such as TF Transform, you must install it;\nit does not come bundled with TensorFlow. You then define your\npreprocessing function just once (in Python), by using TF Transform\nfunctions for scaling, bucketizing, and more. You can also use any\nTensorFlow operation you need. Here is what this preprocessing function\nmight look like if we just had two features:\nimport tensorflow_transform as tft \n \ndef preprocess(inputs): # inputs = a batch of input features \n median_age = inputs[\"housing_median_age\"] \n ocean_proximity = inputs[\"ocean_proximity\"] \n standardized_age = tft.scale_to_z_score(median_age) \n ocean_proximity_id = \ntft.compute_and_apply_vocabulary(ocean_proximity) \n return { \n \"standardized_median_age\": standardized_age, \n \"ocean_proximity_id\": ocean_proximity_id \n }\nNext, TF Transform lets you apply this preprocess() function to the\nwhole training set using Apache Beam (it provides an\nAnalyzeAndTransformDataset class that you can use for this purpose in\nyour Apache Beam pipeline). In the process, it will also compute all the\nnecessary statistics over the whole training set: in this example, the mean\nand standard deviation of the housing_median_age feature, and the\nvocabulary for the ocean_proximity feature. The components that\ncompute these statistics are called analyzers.\nImportantly, TF Transform will also generate an equivalent TensorFlow\nFunction that you can plug into the model you deploy. This TF Function\nincludes some constants that correspond to all the all the necessary\nstatistics computed by Apache Beam (the mean, standard deviation, and\nvocabulary).\nWith the Data API, TFRecords, the Keras preprocessing layers, and TF\nTransform, you can build highly scalable input pipelines for training and\nbenefit from fast and portable data preprocessing in production.\nBut what if you just wanted to use a standard dataset? Well in that case,\nthings are much simpler: just use TFDS!\nThe TensorFlow Datasets (TFDS) Project\nThe TensorFlow Datasets project makes it very easy to download common\ndatasets, from small ones like MNIST or Fashion MNIST to huge datasets\nlike ImageNet (you will need quite a bit of disk space!). The list includes\nimage datasets, text datasets (including translation datasets), and audio\nand video datasets. You can visit https://homl.info/tfds to view the full list,\nalong with a description of each dataset.\nTFDS is not bundled with TensorFlow, so you need to install the\ntensorflow-datasets library (e.g., using pip). Then call the\ntfds.load() function, and it will download the data you want (unless it\nwas already downloaded earlier) and return the data as a dictionary of\ndatasets (typically one for training and one for testing, but this depends on\nthe dataset you choose). For example, let’s download MNIST:\nimport tensorflow_datasets as tfds \n \ndataset = tfds.load(name=\"mnist\") \nmnist_train, mnist_test = dataset[\"train\"], dataset[\"test\"]\nYou can then apply any transformation you want (typically shuffling,\nbatching, and prefetching), and you’re ready to train your model. Here is a\nsimple example:\nmnist_train = mnist_train.shuffle(10000).batch(32).prefetch(1) \nfor item in mnist_train: \n images = item[\"image\"] \n labels = item[\"label\"] \n [...]\nTIP\nThe load() function shuffles each data shard it downloads (only for the training\nset). This may not be sufficient, so it’s best to shuffle the training data some more.\nNote that each item in the dataset is a dictionary containing both the\nfeatures and the labels. But Keras expects each item to be a tuple\ncontaining two elements (again, the features and the labels). You could\ntransform the dataset using the map() method, like this:\nmnist_train = mnist_train.shuffle(10000).batch(32) \nmnist_train = mnist_train.map(lambda items: (items[\"image\"], \nitems[\"label\"])) \nmnist_train = mnist_train.prefetch(1)\nBut it’s simpler to ask the load() function to do this for you by setting\nas_supervised=True (obviously this works only for labeled datasets).\nYou can also specify the batch size if you want. Then you can pass the\ndataset directly to your tf.keras model:\ndataset = tfds.load(name=\"mnist\", batch_size=32, as_supervised=True) \nmnist_train = dataset[\"train\"].prefetch(1) \nmodel = keras.models.Sequential([...]) \nmodel.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"sgd\") \nmodel.fit(mnist_train, epochs=5)\nThis was quite a technical chapter, and you may feel that it is a bit far\nfrom the abstract beauty of neural networks, but the fact is Deep Learning\noften involves large amounts of data, and knowing how to load, parse, and\npreprocess it efficiently is a crucial skill to have. In the next chapter, we\nwill look at convolutional neural networks, which are among the most\nsuccessful neural net architectures for image processing and many other\napplications.\nExercises\n1. Why would you want to use the Data API?\n2. What are the benefits of splitting a large dataset into multiple\nfiles?\n3. During training, how can you tell that your input pipeline is the\nbottleneck? What can you do to fix it?\n4. Can you save any binary data to a TFRecord file, or only\nserialized protocol buffers?\n5. Why would you go through the hassle of converting all your data\nto the Example protobuf format? Why not use your own protobuf\ndefinition?\n6. When using TFRecords, when would you want to activate\ncompression? Why not do it systematically?\n7. Data can be preprocessed directly when writing the data files, or\nwithin the tf.data pipeline, or in preprocessing layers within your\nmodel, or using TF Transform. Can you list a few pros and cons\nof each option?\n8. Name a few common techniques you can use to encode\ncategorical features. What about text?\n9. Load the Fashion MNIST dataset (introduced in Chapter 10); split\nit into a training set, a validation set, and a test set; shuffle the\ntraining set; and save each dataset to multiple TFRecord files.\nEach record should be a serialized Example protobuf with two\nfeatures: the serialized image (use tf.io.serialize_tensor()\nto serialize each image), and the label. Then use tf.data to create\nan efficient dataset for each set. Finally, use a Keras model to\ntrain these datasets, including a preprocessing layer to standardize\neach input feature. Try to make the input pipeline as efficient as\npossible, using TensorBoard to visualize profiling data.\n11\n10. In this exercise you will download a dataset, split it, create a\ntf.data.Dataset to load it and preprocess it efficiently, then\nbuild and train a binary classification model containing an\nEmbedding layer:\na. Download the Large Movie Review Dataset, which\ncontains 50,000 movies reviews from the Internet Movie\nDatabase. The data is organized in two directories, train\nand test, each containing a pos subdirectory with 12,500\npositive reviews and a neg subdirectory with 12,500\nnegative reviews. Each review is stored in a separate text\nfile. There are other files and folders (including\npreprocessed bag-of-words), but we will ignore them in\nthis exercise.\nb. Split the test set into a validation set (15,000) and a test\nset (10,000).\nc. Use tf.data to create an efficient dataset for each set.\nd. Create a binary classification model, using a\nTextVectorization layer to preprocess each review. If\nthe TextVectorization layer is not yet available (or if\nyou like a challenge), try to create your own custom\npreprocessing layer: you can use the functions in the\ntf.strings package, for example lower() to make\neverything lowercase, regex_replace() to replace\npunctuation with spaces, and split() to split words on\nspaces. You should use a lookup table to output word\nindices, which must be prepared in the adapt() method.\ne. Add an Embedding layer and compute the mean\nembedding for each review, multiplied by the square root\nof the number of words (see Chapter 16). This rescaled\nmean embedding can then be passed to the rest of your\nmodel.\nf. Train the model and see what accuracy you get. Try to\noptimize your pipelines to make training as fast as\npossible.\ng. Use TFDS to load the same dataset more easily:\ntfds.load(\"imdb_reviews\").\nSolutions to these exercises are available in Appendix A.\n1 Imagine a sorted deck of cards on your left: suppose you just take the top three cards and\nshuffle them, then pick one randomly and put it to your right, keeping the other two in your\nhands. Take another card on your left, shuffle the three cards in your hands and pick one of\nthem randomly, and put it on your right. When you are done going through all the cards\nlike this, you will have a deck of cards on your right: do you think it will be perfectly\nshuffled?\n2 In general, just prefetching one batch is fine, but in some cases you may need to prefetch\na few more. Alternatively, you can let TensorFlow decide automatically by passing\ntf.data.experimental.AUTOTUNE (this is an experimental feature for now).\n3 But check out the tf.data.experimental.prefetch_to_device() function, which\ncan prefetch data directly to the GPU.\n4 Support for datasets is specific to tf.keras; this will not work in other implementations of\nthe Keras API.\n5 The fit() method will take care of repeating the training dataset. Alternatively, you\ncould call repeat() on the training dataset so that it repeats forever and specify the\nsteps_per_epoch argument when calling the fit() method. This may be useful in some\nrare cases, for example if you want to use a shuffle buffer that crosses over epochs.\n6 Since protobuf objects are meant to be serialized and transmitted, they are called\nmessages.\n7 This chapter contains the bare minimum you need to know about protobufs to use\nTFRecords. To learn more about protobufs, please visit https://homl.info/protobuf.\n8 Why was Example even defined, since it contains no more than a Features object? Well,\nTensorFlow’s developers may one day decide to add more fields to it. As long as the new\nExample definition still contains the features field, with the same ID, it will be backward\ncompatible. This extensibility is one of the great features of protobufs.\n9 Tomas Mikolov et al., “Distributed Representations of Words and Phrases and Their\nCompositionality,” Proceedings of the 26th International Conference on Neural\nInformation Processing Systems 2 (2013): 3111–3119.\n10 Malvina Nissim et al., “Fair Is Better Than Sensational: Man Is to Doctor as Woman Is to\nDoctor,” arXiv preprint arXiv:1905.09866 (2019).\n11 For large images, you could use tf.io.encode_jpeg() instead. This would save a lot of\nspace, but it would lose a bit of image quality." + }, + { + "chapter": 14, + "text": "Chapter 14. Deep Computer\nVision Using Convolutional\nNeural Networks\nAlthough IBM’s Deep Blue supercomputer beat the chess world champion\nGarry Kasparov back in 1996, it wasn’t until fairly recently that computers\nwere able to reliably perform seemingly trivial tasks such as detecting a\npuppy in a picture or recognizing spoken words. Why are these tasks so\neffortless to us humans? The answer lies in the fact that perception largely\ntakes place outside the realm of our consciousness, within specialized\nvisual, auditory, and other sensory modules in our brains. By the time\nsensory information reaches our consciousness, it is already adorned with\nhigh-level features; for example, when you look at a picture of a cute\npuppy, you cannot choose not to see the puppy, not to notice its cuteness.\nNor can you explain how you recognize a cute puppy; it’s just obvious to\nyou. Thus, we cannot trust our subjective experience: perception is not\ntrivial at all, and to understand it we must look at how the sensory\nmodules work.\nConvolutional neural networks (CNNs) emerged from the study of the\nbrain’s visual cortex, and they have been used in image recognition since\nthe 1980s. In the last few years, thanks to the increase in computational\npower, the amount of available training data, and the tricks presented in\nChapter 11 for training deep nets, CNNs have managed to achieve\nsuperhuman performance on some complex visual tasks. They power\nimage search services, self-driving cars, automatic video classification\nsystems, and more. Moreover, CNNs are not restricted to visual\nperception: they are also successful at many other tasks, such as voice\nrecognition and natural language processing. However, we will focus on\nvisual applications for now.\nIn this chapter we will explore where CNNs came from, what their\nbuilding blocks look like, and how to implement them using TensorFlow\nand Keras. Then we will discuss some of the best CNN architectures, as\nwell as other visual tasks, including object detection (classifying multiple\nobjects in an image and placing bounding boxes around them) and\nsemantic segmentation (classifying each pixel according to the class of the\nobject it belongs to).\nThe Architecture of the Visual Cortex\nDavid H. Hubel and Torsten Wiesel performed a series of experiments on\ncats in 1958 and 1959 (and a few years later on monkeys ), giving\ncrucial insights into the structure of the visual cortex (the authors received\nthe Nobel Prize in Physiology or Medicine in 1981 for their work). In\nparticular, they showed that many neurons in the visual cortex have a\nsmall local receptive field, meaning they react only to visual stimuli\nlocated in a limited region of the visual field (see Figure 14-1, in which\nthe local receptive fields of five neurons are represented by dashed\ncircles). The receptive fields of different neurons may overlap, and\ntogether they tile the whole visual field.\nMoreover, the authors showed that some neurons react only to images of\nhorizontal lines, while others react only to lines with different orientations\n(two neurons may have the same receptive field but react to different line\norientations). They also noticed that some neurons have larger receptive\nfields, and they react to more complex patterns that are combinations of\nthe lower-level patterns. These observations led to the idea that the higher-\nlevel neurons are based on the outputs of neighboring lower-level neurons\n(in Figure 14-1, notice that each neuron is connected only to a few neurons\nfrom the previous layer). This powerful architecture is able to detect all\nsorts of complex patterns in any area of the visual field.\n1\n2\n3\nFigure 14-1. Biological neurons in the visual cortex respond to specific patterns in small regions\nof the visual field called receptive fields; as the visual signal makes its way through consecutive\nbrain modules, neurons respond to more complex patterns in larger receptive fields.\nThese studies of the visual cortex inspired the neocognitron, introduced\nin 1980, which gradually evolved into what we now call convolutional\nneural networks. An important milestone was a 1998 paper by Yann\nLeCun et al. that introduced the famous LeNet-5 architecture, widely used\nby banks to recognize handwritten check numbers. This architecture has\nsome building blocks that you already know, such as fully connected\nlayers and sigmoid activation functions, but it also introduces two new\nbuilding blocks: convolutional layers and pooling layers. Let’s look at\nthem now.\nNOTE\nWhy not simply use a deep neural network with fully connected layers for image\nrecognition tasks? Unfortunately, although this works fine for small images (e.g.,\nMNIST), it breaks down for larger images because of the huge number of parameters\nit requires. For example, a 100 × 100–pixel image has 10,000 pixels, and if the first\nlayer has just 1,000 neurons (which already severely restricts the amount of\ninformation transmitted to the next layer), this means a total of 10 million\nconnections. And that’s just the first layer. CNNs solve this problem using partially\nconnected layers and weight sharing.\nConvolutional Layers\n4\n5\nThe most important building block of a CNN is the convolutional layer:\nneurons in the first convolutional layer are not connected to every single\npixel in the input image (like they were in the layers discussed in previous\nchapters), but only to pixels in their receptive fields (see Figure 14-2). In\nturn, each neuron in the second convolutional layer is connected only to\nneurons located within a small rectangle in the first layer. This\narchitecture allows the network to concentrate on small low-level features\nin the first hidden layer, then assemble them into larger higher-level\nfeatures in the next hidden layer, and so on. This hierarchical structure is\ncommon in real-world images, which is one of the reasons why CNNs\nwork so well for image recognition.\nFigure 14-2. CNN layers with rectangular local receptive fields\nNOTE\nAll the multilayer neural networks we’ve looked at so far had layers composed of a\nlong line of neurons, and we had to flatten input images to 1D before feeding them\nto the neural network. In a CNN each layer is represented in 2D, which makes it\neasier to match neurons with their corresponding inputs.\n6\nA neuron located in row i, column j of a given layer is connected to the\noutputs of the neurons in the previous layer located in rows i to i + f – 1,\ncolumns j to j + f – 1, where f and f are the height and width of the\nreceptive field (see Figure 14-3). In order for a layer to have the same\nheight and width as the previous layer, it is common to add zeros around\nthe inputs, as shown in the diagram. This is called zero padding.\nFigure 14-3. Connections between layers and zero padding\nIt is also possible to connect a large input layer to a much smaller layer by\nspacing out the receptive fields, as shown in Figure 14-4. This\ndramatically reduces the model’s computational complexity. The shift\nfrom one receptive field to the next is called the stride. In the diagram, a 5\n× 7 input layer (plus zero padding) is connected to a 3 × 4 layer, using 3 ×\n3 receptive fields and a stride of 2 (in this example the stride is the same\nin both directions, but it does not have to be so). A neuron located in row i,\ncolumn j in the upper layer is connected to the outputs of the neurons in\nh\nw\nh\nw\nthe previous layer located in rows i × s to i × s + f – 1, columns j × s to\nj × s + f – 1, where s and s are the vertical and horizontal strides.\nFigure 14-4. Reducing dimensionality using a stride of 2\nFilters\nA neuron’s weights can be represented as a small image the size of the\nreceptive field. For example, Figure 14-5 shows two possible sets of\nweights, called filters (or convolution kernels). The first one is represented\nas a black square with a vertical white line in the middle (it is a 7 × 7\nmatrix full of 0s except for the central column, which is full of 1s);\nneurons using these weights will ignore everything in their receptive field\nexcept for the central vertical line (since all inputs will get multiplied by\n0, except for the ones located in the central vertical line). The second filter\nis a black square with a horizontal white line in the middle. Once again,\nneurons using these weights will ignore everything in their receptive field\nexcept for the central horizontal line.\nh\nh\nh\nw\nw\nw\nh\nw\nNow if all neurons in a layer use the same vertical line filter (and the same\nbias term), and you feed the network the input image shown in Figure 14-5\n(the bottom image), the layer will output the top-left image. Notice that\nthe vertical white lines get enhanced while the rest gets blurred. Similarly,\nthe upper-right image is what you get if all neurons use the same\nhorizontal line filter; notice that the horizontal white lines get enhanced\nwhile the rest is blurred out. Thus, a layer full of neurons using the same\nfilter outputs a feature map, which highlights the areas in an image that\nactivate the filter the most. Of course, you do not have to define the filters\nmanually: instead, during training the convolutional layer will\nautomatically learn the most useful filters for its task, and the layers above\nwill learn to combine them into more complex patterns.\nFigure 14-5. Applying two different filters to get two feature maps\nStacking Multiple Feature Maps\nUp to now, for simplicity, I have represented the output of each\nconvolutional layer as a 2D layer, but in reality a convolutional layer has\nmultiple filters (you decide how many) and outputs one feature map per\nfilter, so it is more accurately represented in 3D (see Figure 14-6). It has\none neuron per pixel in each feature map, and all neurons within a given\nfeature map share the same parameters (i.e., the same weights and bias\nterm). Neurons in different feature maps use different parameters. A\nneuron’s receptive field is the same as described earlier, but it extends\nacross all the previous layers’ feature maps. In short, a convolutional layer\nsimultaneously applies multiple trainable filters to its inputs, making it\ncapable of detecting multiple features anywhere in its inputs.\nNOTE\nThe fact that all neurons in a feature map share the same parameters dramatically\nreduces the number of parameters in the model. Once the CNN has learned to\nrecognize a pattern in one location, it can recognize it in any other location. In\ncontrast, once a regular DNN has learned to recognize a pattern in one location, it\ncan recognize it only in that particular location.\nInput images are also composed of multiple sublayers: one per color\nchannel. There are typically three: red, green, and blue (RGB). Grayscale\nimages have just one channel, but some images may have much more—for\nexample, satellite images that capture extra light frequencies (such as\ninfrared).\nFigure 14-6. Convolutional layers with multiple feature maps, and images with three color\nchannels\nSpecifically, a neuron located in row i, column j of the feature map k in a\ngiven convolutional layer l is connected to the outputs of the neurons in\nthe previous layer l – 1, located in rows i × s to i × s + f – 1 and\ncolumns j × s to j × s + f – 1, across all feature maps (in layer l – 1).\nNote that all neurons located in the same row i and column j but in\ndifferent feature maps are connected to the outputs of the exact same\nneurons in the previous layer.\nEquation 14-1 summarizes the preceding explanations in one big\nmathematical equation: it shows how to compute the output of a given\nneuron in a convolutional layer. It is a bit ugly due to all the different\nh\nh\nh\nw\nw\nw\nindices, but all it does is calculate the weighted sum of all the inputs, plus\nthe bias term.\nEquation 14-1. Computing the output of a neuron in a convolutional layer\nzi,j,k = bk +\nfh−1\n∑\nu=0\nfw−1\n∑\nv=0\nfn′−1\n∑\nk'=0\nxi′,j′,k′. wu,v,k′,k\nwith { i' = i × sh + u\nj' = j × sw + v\nIn this equation:\nz\n is the output of the neuron located in row i, column j in\nfeature map k of the convolutional layer (layer l).\nAs explained earlier, s and s are the vertical and horizontal\nstrides, f and f are the height and width of the receptive field,\nand f is the number of feature maps in the previous layer (layer l\n– 1).\nx\n is the output of the neuron located in layer l – 1, row i′,\ncolumn j′, feature map k′ (or channel k′ if the previous layer is the\ninput layer).\nb is the bias term for feature map k (in layer l). You can think of\nit as a knob that tweaks the overall brightness of the feature map\nk.\nw\n is the connection weight between any neuron in feature\nmap k of the layer l and its input located at row u, column v\n(relative to the neuron’s receptive field), and feature map k′.\nTensorFlow Implementation\nIn TensorFlow, each input image is typically represented as a 3D tensor of\nshape [height, width, channels]. A mini-batch is represented as a 4D tensor\nof shape [mini-batch size, height, width, channels]. The weights of a\nconvolutional layer are represented as a 4D tensor of shape [f , f , f , f ].\ni, j, k\nh\nw\nh\nw\nn′\ni′, j′, k′\nk\nu, v, k′ ,k\nh\nw\nn′\nn\nThe bias terms of a convolutional layer are simply represented as a 1D\ntensor of shape [f ].\nLet’s look at a simple example. The following code loads two sample\nimages, using Scikit-Learn’s load_sample_image() (which loads two\ncolor images, one of a Chinese temple, and the other of a flower), then it\ncreates two filters and applies them to both images, and finally it displays\none of the resulting feature maps:\nfrom sklearn.datasets import load_sample_image \n \n# Load sample images \nchina = load_sample_image(\"china.jpg\") / 255 \nflower = load_sample_image(\"flower.jpg\") / 255 \nimages = np.array([china, flower]) \nbatch_size, height, width, channels = images.shape \n \n# Create 2 filters \nfilters = np.zeros(shape=(7, 7, channels, 2), dtype=np.float32) \nfilters[:, 3, :, 0] = 1 # vertical line \nfilters[3, :, :, 1] = 1 # horizontal line \n \noutputs = tf.nn.conv2d(images, filters, strides=1, padding=\"same\") \n \nplt.imshow(outputs[0, :, :, 1], cmap=\"gray\") # plot 1st image's 2nd \nfeature map \nplt.show()\nLet’s go through this code:\nThe pixel intensity for each color channel is represented as a byte\nfrom 0 to 255, so we scale these features simply by dividing by\n255, to get floats ranging from 0 to 1.\nThen we create two 7 × 7 filters (one with a vertical white line in\nthe middle, and the other with a horizontal white line in the\nmiddle).\nWe apply them to both images using the tf.nn.conv2d()\nfunction, which is part of TensorFlow’s low-level Deep Learning\nn\nAPI. In this example, we use zero padding (padding=\"same\") and\na stride of 2.\nFinally, we plot one of the resulting feature maps (similar to the\ntop-right image in Figure 14-5).\nThe tf.nn.conv2d() line deserves a bit more explanation:\nimages is the input mini-batch (a 4D tensor, as explained earlier).\nfilters is the set of filters to apply (also a 4D tensor, as\nexplained earlier).\nstrides is equal to 1, but it could also be a 1D array with four\nelements, where the two central elements are the vertical and\nhorizontal strides (s and s ). The first and last elements must\ncurrently be equal to 1. They may one day be used to specify a\nbatch stride (to skip some instances) and a channel stride (to skip\nsome of the previous layer’s feature maps or channels).\npadding must be either \"same\" or \"valid\":\nIf set to \"same\", the convolutional layer uses zero\npadding if necessary. The output size is set to the number\nof input neurons divided by the stride, rounded up. For\nexample, if the input size is 13 and the stride is 5 (see\nFigure 14-7), then the output size is 3 (i.e., 13 / 5 = 2.6,\nrounded up to 3). Then zeros are added as evenly as\npossible around the inputs, as needed. When strides=1,\nthe layer’s outputs will have the same spatial dimensions\n(width and height) as its inputs, hence the name same.\nIf set to \"valid\", the convolutional layer does not use\nzero padding and may ignore some rows and columns at\nthe bottom and right of the input image, depending on the\nstride, as shown in Figure 14-7 (for simplicity, only the\nh\nw\nhorizontal dimension is shown here, but of course the\nsame logic applies to the vertical dimension). This means\nthat every neuron’s receptive field lies strictly within\nvalid positions inside the input (it does not go out of\nbounds), hence the name valid.\nFigure 14-7. Padding=\"same” or “valid” (with input width 13, filter width 6, stride 5)\nIn this example we manually defined the filters, but in a real CNN you\nwould normally define filters as trainable variables so the neural net can\nlearn which filters work best, as explained earlier. Instead of manually\ncreating the variables, use the keras.layers.Conv2D layer:\nconv = keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, \n padding=\"same\", activation=\"relu\")\nThis code creates a Conv2D layer with 32 filters, each 3 × 3, using a stride\nof 1 (both horizontally and vertically) and \"same\" padding, and applying\nthe ReLU activation function to its outputs. As you can see, convolutional\nlayers have quite a few hyperparameters: you must choose the number of\nfilters, their height and width, the strides, and the padding type. As always,\nyou can use cross-validation to find the right hyperparameter values, but\nthis is very time-consuming. We will discuss common CNN architectures\nlater, to give you some idea of which hyperparameter values work best in\npractice.\nMemory Requirements\nAnother problem with CNNs is that the convolutional layers require a\nhuge amount of RAM. This is especially true during training, because the\nreverse pass of backpropagation requires all the intermediate values\ncomputed during the forward pass.\nFor example, consider a convolutional layer with 5 × 5 filters, outputting\n200 feature maps of size 150 × 100, with stride 1 and \"same\" padding. If\nthe input is a 150 × 100 RGB image (three channels), then the number of\nparameters is (5 × 5 × 3 + 1) × 200 = 15,200 (the + 1 corresponds to the\nbias terms), which is fairly small compared to a fully connected layer.\nHowever, each of the 200 feature maps contains 150 × 100 neurons, and\neach of these neurons needs to compute a weighted sum of its 5 × 5 × 3 =\n75 inputs: that’s a total of 225 million float multiplications. Not as bad as\na fully connected layer, but still quite computationally intensive.\nMoreover, if the feature maps are represented using 32-bit floats, then the\nconvolutional layer’s output will occupy 200 × 150 × 100 × 32 = 96\nmillion bits (12 MB) of RAM. And that’s just for one instance—if a\ntraining batch contains 100 instances, then this layer will use up 1.2 GB of\nRAM!\nDuring inference (i.e., when making a prediction for a new instance) the\nRAM occupied by one layer can be released as soon as the next layer has\nbeen computed, so you only need as much RAM as required by two\nconsecutive layers. But during training everything computed during the\nforward pass needs to be preserved for the reverse pass, so the amount of\nRAM needed is (at least) the total amount of RAM required by all layers.\n7\n8\nTIP\nIf training crashes because of an out-of-memory error, you can try reducing the\nmini-batch size. Alternatively, you can try reducing dimensionality using a stride, or\nremoving a few layers. Or you can try using 16-bit floats instead of 32-bit floats. Or\nyou could distribute the CNN across multiple devices.\nNow let’s look at the second common building block of CNNs: the pooling\nlayer.\nPooling Layers\nOnce you understand how convolutional layers work, the pooling layers\nare quite easy to grasp. Their goal is to subsample (i.e., shrink) the input\nimage in order to reduce the computational load, the memory usage, and\nthe number of parameters (thereby limiting the risk of overfitting).\nJust like in convolutional layers, each neuron in a pooling layer is\nconnected to the outputs of a limited number of neurons in the previous\nlayer, located within a small rectangular receptive field. You must define\nits size, the stride, and the padding type, just like before. However, a\npooling neuron has no weights; all it does is aggregate the inputs using an\naggregation function such as the max or mean. Figure 14-8 shows a max\npooling layer, which is the most common type of pooling layer. In this\nexample, we use a 2 × 2 pooling kernel, with a stride of 2 and no padding.\nOnly the max input value in each receptive field makes it to the next layer,\nwhile the other inputs are dropped. For example, in the lower-left\nreceptive field in Figure 14-8, the input values are 1, 5, 3, 2, so only the\nmax value, 5, is propagated to the next layer. Because of the stride of 2, the\noutput image has half the height and half the width of the input image\n(rounded down since we use no padding).\n9\nFigure 14-8. Max pooling layer (2 × 2 pooling kernel, stride 2, no padding)\nNOTE\nA pooling layer typically works on every input channel independently, so the output\ndepth is the same as the input depth.\nOther than reducing computations, memory usage, and the number of\nparameters, a max pooling layer also introduces some level of invariance\nto small translations, as shown in Figure 14-9. Here we assume that the\nbright pixels have a lower value than dark pixels, and we consider three\nimages (A, B, C) going through a max pooling layer with a 2 × 2 kernel\nand stride 2. Images B and C are the same as image A, but shifted by one\nand two pixels to the right. As you can see, the outputs of the max pooling\nlayer for images A and B are identical. This is what translation invariance\nmeans. For image C, the output is different: it is shifted one pixel to the\nright (but there is still 75% invariance). By inserting a max pooling layer\nevery few layers in a CNN, it is possible to get some level of translation\ninvariance at a larger scale. Moreover, max pooling offers a small amount\nof rotational invariance and a slight scale invariance. Such invariance\n(even if it is limited) can be useful in cases where the prediction should\nnot depend on these details, such as in classification tasks.\nFigure 14-9. Invariance to small translations\nHowever, max pooling has some downsides too. Firstly, it is obviously\nvery destructive: even with a tiny 2 × 2 kernel and a stride of 2, the output\nwill be two times smaller in both directions (so its area will be four times\nsmaller), simply dropping 75% of the input values. And in some\napplications, invariance is not desirable. Take semantic segmentation (the\ntask of classifying each pixel in an image according to the object that pixel\nbelongs to, which we’ll explore later in this chapter): obviously, if the\ninput image is translated by one pixel to the right, the output should also\nbe translated by one pixel to the right. The goal in this case is\nequivariance, not invariance: a small change to the inputs should lead to a\ncorresponding small change in the output.\nTensorFlow Implementation\nImplementing a max pooling layer in TensorFlow is quite easy. The\nfollowing code creates a max pooling layer using a 2 × 2 kernel. The\nstrides default to the kernel size, so this layer will use a stride of 2 (both\nhorizontally and vertically). By default, it uses \"valid\" padding (i.e., no\npadding at all):\nmax_pool = keras.layers.MaxPool2D(pool_size=2)\nTo create an average pooling layer, just use AvgPool2D instead of\nMaxPool2D. As you might expect, it works exactly like a max pooling\nlayer, except it computes the mean rather than the max. Average pooling\nlayers used to be very popular, but people mostly use max pooling layers\nnow, as they generally perform better. This may seem surprising, since\ncomputing the mean generally loses less information than computing the\nmax. But on the other hand, max pooling preserves only the strongest\nfeatures, getting rid of all the meaningless ones, so the next layers get a\ncleaner signal to work with. Moreover, max pooling offers stronger\ntranslation invariance than average pooling, and it requires slightly less\ncompute.\nNote that max pooling and average pooling can be performed along the\ndepth dimension rather than the spatial dimensions, although this is not as\ncommon. This can allow the CNN to learn to be invariant to various\nfeatures. For example, it could learn multiple filters, each detecting a\ndifferent rotation of the same pattern (such as hand-written digits; see\nFigure 14-10), and the depthwise max pooling layer would ensure that the\noutput is the same regardless of the rotation. The CNN could similarly\nlearn to be invariant to anything else: thickness, brightness, skew, color,\nand so on.\nFigure 14-10. Depthwise max pooling can help the CNN learn any invariance\nKeras does not include a depthwise max pooling layer, but TensorFlow’s\nlow-level Deep Learning API does: just use the tf.nn.max_pool()\nfunction, and specify the kernel size and strides as 4-tuples (i.e., tuples of\nsize 4). The first three values of each should be 1: this indicates that the\nkernel size and stride along the batch, height, and width dimensions should\nbe 1. The last value should be whatever kernel size and stride you want\nalong the depth dimension—for example, 3 (this must be a divisor of the\ninput depth; it will not work if the previous layer outputs 20 feature maps,\nsince 20 is not a multiple of 3):\noutput = tf.nn.max_pool(images, \n ksize=(1, 1, 1, 3), \n strides=(1, 1, 1, 3), \n padding=\"valid\")\nIf you want to include this as a layer in your Keras models, wrap it in a\nLambda layer (or create a custom Keras layer):\ndepth_pool = keras.layers.Lambda( \n lambda X: tf.nn.max_pool(X, ksize=(1, 1, 1, 3), strides=(1, 1, 1, 3), \n padding=\"valid\"))\nOne last type of pooling layer that you will often see in modern\narchitectures is the global average pooling layer. It works very\ndifferently: all it does is compute the mean of each entire feature map (it’s\nlike an average pooling layer using a pooling kernel with the same spatial\ndimensions as the inputs). This means that it just outputs a single number\nper feature map and per instance. Although this is of course extremely\ndestructive (most of the information in the feature map is lost), it can be\nuseful as the output layer, as we will see later in this chapter. To create\nsuch a layer, simply use the keras.layers.GlobalAvgPool2D class:\nglobal_avg_pool = keras.layers.GlobalAvgPool2D()\nIt’s equivalent to this simple Lambda layer, which computes the mean over\nthe spatial dimensions (height and width):\nglobal_avg_pool = keras.layers.Lambda(lambda X: tf.reduce_mean(X, axis=\n[1, 2]))\nNow you know all the building blocks to create convolutional neural\nnetworks. Let’s see how to assemble them.\nCNN Architectures\nTypical CNN architectures stack a few convolutional layers (each one\ngenerally followed by a ReLU layer), then a pooling layer, then another\nfew convolutional layers (+ReLU), then another pooling layer, and so on.\nThe image gets smaller and smaller as it progresses through the network,\nbut it also typically gets deeper and deeper (i.e., with more feature maps),\nthanks to the convolutional layers (see Figure 14-11). At the top of the\nstack, a regular feedforward neural network is added, composed of a few\nfully connected layers (+ReLUs), and the final layer outputs the prediction\n(e.g., a softmax layer that outputs estimated class probabilities).\nFigure 14-11. Typical CNN architecture\nTIP\nA common mistake is to use convolution kernels that are too large. For example,\ninstead of using a convolutional layer with a 5 × 5 kernel, stack two layers with 3 × 3\nkernels: it will use fewer parameters and require fewer computations, and it will\nusually perform better. One exception is for the first convolutional layer: it can\ntypically have a large kernel (e.g., 5 × 5), usually with a stride of 2 or more: this will\nreduce the spatial dimension of the image without losing too much information, and\nsince the input image only has three channels in general, it will not be too costly.\nHere is how you can implement a simple CNN to tackle the Fashion\nMNIST dataset (introduced in Chapter 10):\nmodel = keras.models.Sequential([ \n keras.layers.Conv2D(64, 7, activation=\"relu\", padding=\"same\", \n input_shape=[28, 28, 1]), \n keras.layers.MaxPooling2D(2), \n keras.layers.Conv2D(128, 3, activation=\"relu\", padding=\"same\"), \n keras.layers.Conv2D(128, 3, activation=\"relu\", padding=\"same\"), \n keras.layers.MaxPooling2D(2), \n keras.layers.Conv2D(256, 3, activation=\"relu\", padding=\"same\"), \n keras.layers.Conv2D(256, 3, activation=\"relu\", padding=\"same\"), \n keras.layers.MaxPooling2D(2), \n keras.layers.Flatten(), \n keras.layers.Dense(128, activation=\"relu\"), \n keras.layers.Dropout(0.5), \n keras.layers.Dense(64, activation=\"relu\"), \n keras.layers.Dropout(0.5), \n keras.layers.Dense(10, activation=\"softmax\") \n])\nLet’s go through this model:\nThe first layer uses 64 fairly large filters (7 × 7) but no stride\nbecause the input images are not very large. It also sets\ninput_shape=[28, 28, 1], because the images are 28 × 28\npixels, with a single color channel (i.e., grayscale).\nNext we have a max pooling layer which uses a pool size of 2, so\nit divides each spatial dimension by a factor of 2.\nThen we repeat the same structure twice: two convolutional layers\nfollowed by a max pooling layer. For larger images, we could\nrepeat this structure several more times (the number of repetitions\nis a hyperparameter you can tune).\nNote that the number of filters grows as we climb up the CNN\ntoward the output layer (it is initially 64, then 128, then 256): it\nmakes sense for it to grow, since the number of low-level features\nis often fairly low (e.g., small circles, horizontal lines), but there\nare many different ways to combine them into higher-level\nfeatures. It is a common practice to double the number of filters\nafter each pooling layer: since a pooling layer divides each spatial\ndimension by a factor of 2, we can afford to double the number of\nfeature maps in the next layer without fear of exploding the\nnumber of parameters, memory usage, or computational load.\nNext is the fully connected network, composed of two hidden\ndense layers and a dense output layer. Note that we must flatten\nits inputs, since a dense network expects a 1D array of features for\neach instance. We also add two dropout layers, with a dropout rate\nof 50% each, to reduce overfitting.\nThis CNN reaches over 92% accuracy on the test set. It’s not state of the\nart, but it is pretty good, and clearly much better than what we achieved\nwith dense networks in Chapter 10.\nOver the years, variants of this fundamental architecture have been\ndeveloped, leading to amazing advances in the field. A good measure of\nthis progress is the error rate in competitions such as the ILSVRC\nImageNet challenge. In this competition the top-five error rate for image\nclassification fell from over 26% to less than 2.3% in just six years. The\ntop-five error rate is the number of test images for which the system’s top\nfive predictions did not include the correct answer. The images are large\n(256 pixels high) and there are 1,000 classes, some of which are really\nsubtle (try distinguishing 120 dog breeds). Looking at the evolution of the\nwinning entries is a good way to understand how CNNs work.\nWe will first look at the classical LeNet-5 architecture (1998), then three\nof the winners of the ILSVRC challenge: AlexNet (2012), GoogLeNet\n(2014), and ResNet (2015).\nLeNet-5\nThe LeNet-5 architecture\n is perhaps the most widely known CNN\narchitecture. As mentioned earlier, it was created by Yann LeCun in 1998\nand has been widely used for handwritten digit recognition (MNIST). It is\ncomposed of the layers shown in Table 14-1.\n10\nTable 14-1. LeNet-5 architecture\nLayer Type\nMaps Size\nKernel size Stride Activation\nOut\nFully connected –\n10\n–\n–\nRBF\nF6\nFully connected –\n84\n–\n–\ntanh\nC5\nConvolution\n120\n1 × 1\n5 × 5\n1\ntanh\nS4\nAvg pooling\n16\n5 × 5\n2 × 2\n2\ntanh\nC3\nConvolution\n16\n10 × 10 5 × 5\n1\ntanh\nS2\nAvg pooling\n6\n14 × 14 2 × 2\n2\ntanh\nC1\nConvolution\n6\n28 × 28 5 × 5\n1\ntanh\nIn\nInput\n1\n32 × 32 –\n–\n–\nThere are a few extra details to be noted:\nMNIST images are 28 × 28 pixels, but they are zero-padded to 32\n× 32 pixels and normalized before being fed to the network. The\nrest of the network does not use any padding, which is why the\nsize keeps shrinking as the image progresses through the network.\nThe average pooling layers are slightly more complex than usual:\neach neuron computes the mean of its inputs, then multiplies the\nresult by a learnable coefficient (one per map) and adds a\nlearnable bias term (again, one per map), then finally applies the\nactivation function.\nMost neurons in C3 maps are connected to neurons in only three\nor four S2 maps (instead of all six S2 maps). See table 1 (page 8)\nin the original paper\n for details.\nThe output layer is a bit special: instead of computing the matrix\nmultiplication of the inputs and the weight vector, each neuron\noutputs the square of the Euclidian distance between its input\n10\nvector and its weight vector. Each output measures how much the\nimage belongs to a particular digit class. The cross-entropy cost\nfunction is now preferred, as it penalizes bad predictions much\nmore, producing larger gradients and converging faster.\nYann LeCun’s website features great demos of LeNet-5 classifying digits.\nAlexNet\nThe AlexNet CNN architecture won the 2012 ImageNet ILSVRC\nchallenge by a large margin: it achieved a top-five error rate of 17%, while\nthe second best achieved only 26%! It was developed by Alex Krizhevsky\n(hence the name), Ilya Sutskever, and Geoffrey Hinton. It is similar to\nLeNet-5, only much larger and deeper, and it was the first to stack\nconvolutional layers directly on top of one another, instead of stacking a\npooling layer on top of each convolutional layer. Table 14-2 presents this\narchitecture.\n11\nTable 14-2. AlexNet architecture\nLayer Type\nMaps\nSize\nKernel size Stride Padding Activation\nOut\nFully connected –\n1,000\n–\n–\n–\nSoftmax\nF9\nFully connected –\n4,096\n–\n–\n–\nReLU\nF8\nFully connected –\n4,096\n–\n–\n–\nReLU\nC7\nConvolution\n256\n13 × 13\n3 × 3\n1\nsame\nReLU\nC6\nConvolution\n384\n13 × 13\n3 × 3\n1\nsame\nReLU\nC5\nConvolution\n384\n13 × 13\n3 × 3\n1\nsame\nReLU\nS4\nMax pooling\n256\n13 × 13\n3 × 3\n2\nvalid\n–\nC3\nConvolution\n256\n27 × 27\n5 × 5\n1\nsame\nReLU\nS2\nMax pooling\n96\n27 × 27\n3 × 3\n2\nvalid\n–\nC1\nConvolution\n96\n55 × 55\n11 × 11\n4\nvalid\nReLU\nIn\nInput\n3 (RGB) 227 × 227 –\n–\n–\n–\nTo reduce overfitting, the authors used two regularization techniques.\nFirst, they applied dropout (introduced in Chapter 11) with a 50% dropout\nrate during training to the outputs of layers F8 and F9. Second, they\nperformed data augmentation by randomly shifting the training images by\nvarious offsets, flipping them horizontally, and changing the lighting\nconditions.\nDATA AUGMENTATION\nData augmentation artificially increases the size of the training set by\ngenerating many realistic variants of each training instance. This\nreduces overfitting, making this a regularization technique. The\ngenerated instances should be as realistic as possible: ideally, given an\nimage from the augmented training set, a human should not be able to\ntell whether it was augmented or not. Simply adding white noise will\nnot help; the modifications should be learnable (white noise is not).\nFor example, you can slightly shift, rotate, and resize every picture in\nthe training set by various amounts and add the resulting pictures to\nthe training set (see Figure 14-12). This forces the model to be more\ntolerant to variations in the position, orientation, and size of the\nobjects in the pictures. For a model that’s more tolerant of different\nlighting conditions, you can similarly generate many images with\nvarious contrasts. In general, you can also flip the pictures\nhorizontally (except for text, and other asymmetrical objects). By\ncombining these transformations, you can greatly increase the size of\nyour training set.\nFigure 14-12. Generating new training instances from existing ones\nAlexNet also uses a competitive normalization step immediately after the\nReLU step of layers C1 and C3, called local response normalization\n(LRN): the most strongly activated neurons inhibit other neurons located\nat the same position in neighboring feature maps (such competitive\nactivation has been observed in biological neurons). This encourages\ndifferent feature maps to specialize, pushing them apart and forcing them\nto explore a wider range of features, ultimately improving generalization.\nEquation 14-2 shows how to apply LRN.\nEquation 14-2. Local response normalization (LRN)\nbi = ai(k + α\njhigh\n∑\nj=jlow\naj2)\n−β\nwith\n⎧\n⎪ \n⎪\n⎨\n⎪ \n⎪\n⎩\njhigh = min(i +\n, fn −1)\njlow = max(0, i −\n)\nIn this equation:\nr\n2\nr\n2\nb is the normalized output of the neuron located in feature map i,\nat some row u and column v (note that in this equation we\nconsider only neurons located at this row and column, so u and v\nare not shown).\na is the activation of that neuron after the ReLU step, but before\nnormalization.\nk, α, β, and r are hyperparameters. k is called the bias, and r is\ncalled the depth radius.\nf is the number of feature maps.\nFor example, if r = 2 and a neuron has a strong activation, it will inhibit\nthe activation of the neurons located in the feature maps immediately\nabove and below its own.\nIn AlexNet, the hyperparameters are set as follows: r = 2, α = 0.00002, β =\n0.75, and k = 1. This step can be implemented using the\ntf.nn.local_response_normalization() function (which you can wrap\nin a Lambda layer if you want to use it in a Keras model).\nA variant of AlexNet called ZF Net\n was developed by Matthew Zeiler\nand Rob Fergus and won the 2013 ILSVRC challenge. It is essentially\nAlexNet with a few tweaked hyperparameters (number of feature maps,\nkernel size, stride, etc.).\nGoogLeNet\nThe GoogLeNet architecture was developed by Christian Szegedy et al.\nfrom Google Research,\n and it won the ILSVRC 2014 challenge by\npushing the top-five error rate below 7%. This great performance came in\nlarge part from the fact that the network was much deeper than previous\nCNNs (as you’ll see in Figure 14-14). This was made possible by\nsubnetworks called inception modules,\n which allow GoogLeNet to use\nparameters much more efficiently than previous architectures: GoogLeNet\ni\ni\nn\n12\n13\n14\nactually has 10 times fewer parameters than AlexNet (roughly 6 million\ninstead of 60 million).\nFigure 14-13 shows the architecture of an inception module. The notation\n“3 × 3 + 1(S)” means that the layer uses a 3 × 3 kernel, stride 1, and\n\"same\" padding. The input signal is first copied and fed to four different\nlayers. All convolutional layers use the ReLU activation function. Note\nthat the second set of convolutional layers uses different kernel sizes (1 ×\n1, 3 × 3, and 5 × 5), allowing them to capture patterns at different scales.\nAlso note that every single layer uses a stride of 1 and \"same\" padding\n(even the max pooling layer), so their outputs all have the same height and\nwidth as their inputs. This makes it possible to concatenate all the outputs\nalong the depth dimension in the final depth concatenation layer (i.e.,\nstack the feature maps from all four top convolutional layers). This\nconcatenation layer can be implemented in TensorFlow using the\ntf.concat() operation, with axis=3 (the axis is the depth).\nFigure 14-13. Inception module\nYou may wonder why inception modules have convolutional layers with 1\n× 1 kernels. Surely these layers cannot capture any features because they\nlook at only one pixel at a time? In fact, the layers serve three purposes:\nAlthough they cannot capture spatial patterns, they can capture\npatterns along the depth dimension.\nThey are configured to output fewer feature maps than their\ninputs, so they serve as bottleneck layers, meaning they reduce\ndimensionality. This cuts the computational cost and the number\nof parameters, speeding up training and improving generalization.\nEach pair of convolutional layers ([1 × 1, 3 × 3] and [1 × 1, 5 × 5])\nacts like a single powerful convolutional layer, capable of\ncapturing more complex patterns. Indeed, instead of sweeping a\nsimple linear classifier across the image (as a single\nconvolutional layer does), this pair of convolutional layers sweeps\na two-layer neural network across the image.\nIn short, you can think of the whole inception module as a convolutional\nlayer on steroids, able to output feature maps that capture complex\npatterns at various scales.\nWARNING\nThe number of convolutional kernels for each convolutional layer is a\nhyperparameter. Unfortunately, this means that you have six more hyperparameters\nto tweak for every inception layer you add.\nNow let’s look at the architecture of the GoogLeNet CNN (see Figure 14-\n14). The number of feature maps output by each convolutional layer and\neach pooling layer is shown before the kernel size. The architecture is so\ndeep that it has to be represented in three columns, but GoogLeNet is\nactually one tall stack, including nine inception modules (the boxes with\nthe spinning tops). The six numbers in the inception modules represent the\nnumber of feature maps output by each convolutional layer in the module\n(in the same order as in Figure 14-13). Note that all the convolutional\nlayers use the ReLU activation function.\nFigure 14-14. GoogLeNet architecture\nLet’s go through this network:\nThe first two layers divide the image’s height and width by 4 (so\nits area is divided by 16), to reduce the computational load. The\nfirst layer uses a large kernel size so that much of the information\nis preserved.\nThen the local response normalization layer ensures that the\nprevious layers learn a wide variety of features (as discussed\nearlier).\nTwo convolutional layers follow, where the first acts like a\nbottleneck layer. As explained earlier, you can think of this pair as\na single smarter convolutional layer.\nAgain, a local response normalization layer ensures that the\nprevious layers capture a wide variety of patterns.\nNext, a max pooling layer reduces the image height and width by\n2, again to speed up computations.\nThen comes the tall stack of nine inception modules, interleaved\nwith a couple max pooling layers to reduce dimensionality and\nspeed up the net.\nNext, the global average pooling layer outputs the mean of each\nfeature map: this drops any remaining spatial information, which\nis fine because there was not much spatial information left at that\npoint. Indeed, GoogLeNet input images are typically expected to\nbe 224 × 224 pixels, so after 5 max pooling layers, each dividing\nthe height and width by 2, the feature maps are down to 7 × 7.\nMoreover, it is a classification task, not localization, so it does\nnot matter where the object is. Thanks to the dimensionality\nreduction brought by this layer, there is no need to have several\nfully connected layers at the top of the CNN (like in AlexNet),\nand this considerably reduces the number of parameters in the\nnetwork and limits the risk of overfitting.\nThe last layers are self-explanatory: dropout for regularization,\nthen a fully connected layer with 1,000 units (since there are\n1,000 classes) and a softmax activation function to output\nestimated class probabilities.\nThis diagram is slightly simplified: the original GoogLeNet architecture\nalso included two auxiliary classifiers plugged on top of the third and sixth\ninception modules. They were both composed of one average pooling\nlayer, one convolutional layer, two fully connected layers, and a softmax\nactivation layer. During training, their loss (scaled down by 70%) was\nadded to the overall loss. The goal was to fight the vanishing gradients\nproblem and regularize the network. However, it was later shown that their\neffect was relatively minor.\nSeveral variants of the GoogLeNet architecture were later proposed by\nGoogle researchers, including Inception-v3 and Inception-v4, using\nslightly different inception modules and reaching even better performance.\nVGGNet\nThe runner-up in the ILSVRC 2014 challenge was VGGNet,\n developed\nby Karen Simonyan and Andrew Zisserman from the Visual Geometry\nGroup (VGG) research lab at Oxford University. It had a very simple and\nclassical architecture, with 2 or 3 convolutional layers and a pooling layer,\nthen again 2 or 3 convolutional layers and a pooling layer, and so on\n(reaching a total of just 16 or 19 convolutional layers, depending on the\nVGG variant), plus a final dense network with 2 hidden layers and the\noutput layer. It used only 3 × 3 filters, but many filters.\nResNet\nKaiming He et al. won the ILSVRC 2015 challenge using a Residual\nNetwork (or ResNet),\n that delivered an astounding top-five error rate\nunder 3.6%. The winning variant used an extremely deep CNN composed\nof 152 layers (other variants had 34, 50, and 101 layers). It confirmed the\ngeneral trend: models are getting deeper and deeper, with fewer and fewer\nparameters. The key to being able to train such a deep network is to use\nskip connections (also called shortcut connections): the signal feeding into\na layer is also added to the output of a layer located a bit higher up the\nstack. Let’s see why this is useful.\nWhen training a neural network, the goal is to make it model a target\nfunction h(x). If you add the input x to the output of the network (i.e., you\nadd a skip connection), then the network will be forced to model f(x) =\nh(x) – x rather than h(x). This is called residual learning (see Figure 14-\n15).\n15\n16\nFigure 14-15. Residual learning\nWhen you initialize a regular neural network, its weights are close to zero,\nso the network just outputs values close to zero. If you add a skip\nconnection, the resulting network just outputs a copy of its inputs; in other\nwords, it initially models the identity function. If the target function is\nfairly close to the identity function (which is often the case), this will\nspeed up training considerably.\nMoreover, if you add many skip connections, the network can start making\nprogress even if several layers have not started learning yet (see\nFigure 14-16). Thanks to skip connections, the signal can easily make its\nway across the whole network. The deep residual network can be seen as a\nstack of residual units (RUs), where each residual unit is a small neural\nnetwork with a skip connection.\nFigure 14-16. Regular deep neural network (left) and deep residual network (right)\nNow let’s look at ResNet’s architecture (see Figure 14-17). It is\nsurprisingly simple. It starts and ends exactly like GoogLeNet (except\nwithout a dropout layer), and in between is just a very deep stack of simple\nresidual units. Each residual unit is composed of two convolutional layers\n(and no pooling layer!), with Batch Normalization (BN) and ReLU\nactivation, using 3 × 3 kernels and preserving spatial dimensions (stride 1,\n\"same\" padding).\nFigure 14-17. ResNet architecture\nNote that the number of feature maps is doubled every few residual units,\nat the same time as their height and width are halved (using a\nconvolutional layer with stride 2). When this happens, the inputs cannot be\nadded directly to the outputs of the residual unit because they don’t have\nthe same shape (for example, this problem affects the skip connection\nrepresented by the dashed arrow in Figure 14-17). To solve this problem,\nthe inputs are passed through a 1 × 1 convolutional layer with stride 2 and\nthe right number of output feature maps (see Figure 14-18).\nFigure 14-18. Skip connection when changing feature map size and depth\nResNet-34 is the ResNet with 34 layers (only counting the convolutional\nlayers and the fully connected layer)\n containing 3 residual units that\noutput 64 feature maps, 4 RUs with 128 maps, 6 RUs with 256 maps, and 3\nRUs with 512 maps. We will implement this architecture later in this\nchapter.\nResNets deeper than that, such as ResNet-152, use slightly different\nresidual units. Instead of two 3 × 3 convolutional layers with, say, 256\nfeature maps, they use three convolutional layers: first a 1 × 1\nconvolutional layer with just 64 feature maps (4 times less), which acts as\na bottleneck layer (as discussed already), then a 3 × 3 layer with 64 feature\nmaps, and finally another 1 × 1 convolutional layer with 256 feature maps\n(4 times 64) that restores the original depth. ResNet-152 contains 3 such\nRUs that output 256 maps, then 8 RUs with 512 maps, a whopping 36 RUs\nwith 1,024 maps, and finally 3 RUs with 2,048 maps.\nNOTE\nGoogle’s Inception-v4\n architecture merged the ideas of GoogLeNet and ResNet\nand achieved a top-five error rate of close to 3% on ImageNet classification.\n17\n18\nXception\nAnother variant of the GoogLeNet architecture is worth noting: Xception\n(which stands for Extreme Inception) was proposed in 2016 by François\nChollet (the author of Keras), and it significantly outperformed Inception-\nv3 on a huge vision task (350 million images and 17,000 classes). Just like\nInception-v4, it merges the ideas of GoogLeNet and ResNet, but it\nreplaces the inception modules with a special type of layer called a\ndepthwise separable convolution layer (or separable convolution layer for\nshort\n). These layers had been used before in some CNN architectures,\nbut they were not as central as in the Xception architecture. While a\nregular convolutional layer uses filters that try to simultaneously capture\nspatial patterns (e.g., an oval) and cross-channel patterns (e.g., mouth +\nnose + eyes = face), a separable convolutional layer makes the strong\nassumption that spatial patterns and cross-channel patterns can be\nmodeled separately (see Figure 14-19). Thus, it is composed of two parts:\nthe first part applies a single spatial filter for each input feature map, then\nthe second part looks exclusively for cross-channel patterns—it is just a\nregular convolutional layer with 1 × 1 filters.\n19\n20\nFigure 14-19. Depthwise separable convolutional layer\nSince separable convolutional layers only have one spatial filter per input\nchannel, you should avoid using them after layers that have too few\nchannels, such as the input layer (granted, that’s what Figure 14-19\nrepresents, but it is just for illustration purposes). For this reason, the\nXception architecture starts with 2 regular convolutional layers, but then\nthe rest of the architecture uses only separable convolutions (34 in all),\nplus a few max pooling layers and the usual final layers (a global average\npooling layer and a dense output layer).\nYou might wonder why Xception is considered a variant of GoogLeNet,\nsince it contains no inception module at all. Well, as we discussed earlier,\nan inception module contains convolutional layers with 1 × 1 filters: these\nlook exclusively for cross-channel patterns. However, the convolutional\nlayers that sit on top of them are regular convolutional layers that look\nboth for spatial and cross-channel patterns. So you can think of an\ninception module as an intermediate between a regular convolutional layer\n(which considers spatial patterns and cross-channel patterns jointly) and a\nseparable convolutional layer (which considers them separately). In\npractice, it seems that separable convolutional layers generally perform\nbetter.\nTIP\nSeparable convolutional layers use fewer parameters, less memory, and fewer\ncomputations than regular convolutional layers, and in general they even perform\nbetter, so you should consider using them by default (except after layers with few\nchannels).\nThe ILSVRC 2016 challenge was won by the CUImage team from the\nChinese University of Hong Kong. They used an ensemble of many\ndifferent techniques, including a sophisticated object-detection system\ncalled GBD-Net,\n to achieve a top-five error rate below 3%. Although\nthis result is unquestionably impressive, the complexity of the solution\ncontrasted with the simplicity of ResNets. Moreover, one year later\nanother fairly simple architecture performed even better, as we will see\nnow.\nSENet\nThe winning architecture in the ILSVRC 2017 challenge was the Squeeze-\nand-Excitation Network (SENet).\n This architecture extends existing\narchitectures such as inception networks and ResNets, and boosts their\nperformance. This allowed SENet to win the competition with an\nastonishing 2.25% top-five error rate! The extended versions of inception\nnetworks and ResNets are called SE-Inception and SE-ResNet,\nrespectively. The boost comes from the fact that a SENet adds a small\nneural network, called an SE block, to every unit in the original\narchitecture (i.e., every inception module or every residual unit), as shown\nin Figure 14-20.\n21\n22\nFigure 14-20. SE-Inception module (left) and SE-ResNet unit (right)\nAn SE block analyzes the output of the unit it is attached to, focusing\nexclusively on the depth dimension (it does not look for any spatial\npattern), and it learns which features are usually most active together. It\nthen uses this information to recalibrate the feature maps, as shown in\nFigure 14-21. For example, an SE block may learn that mouths, noses, and\neyes usually appear together in pictures: if you see a mouth and a nose,\nyou should expect to see eyes as well. So if the block sees a strong\nactivation in the mouth and nose feature maps, but only mild activation in\nthe eye feature map, it will boost the eye feature map (more accurately, it\nwill reduce irrelevant feature maps). If the eyes were somewhat confused\nwith something else, this feature map recalibration will help resolve the\nambiguity.\nFigure 14-21. An SE block performs feature map recalibration\nAn SE block is composed of just three layers: a global average pooling\nlayer, a hidden dense layer using the ReLU activation function, and a\ndense output layer using the sigmoid activation function (see Figure 14-\n22).\nFigure 14-22. SE block architecture\nAs earlier, the global average pooling layer computes the mean activation\nfor each feature map: for example, if its input contains 256 feature maps,\nit will output 256 numbers representing the overall level of response for\neach filter. The next layer is where the “squeeze” happens: this layer has\nsignificantly fewer than 256 neurons—typically 16 times fewer than the\nnumber of feature maps (e.g., 16 neurons)—so the 256 numbers get\ncompressed into a small vector (e.g., 16 dimensions). This is a low-\ndimensional vector representation (i.e., an embedding) of the distribution\nof feature responses. This bottleneck step forces the SE block to learn a\ngeneral representation of the feature combinations (we will see this\nprinciple in action again when we discuss autoencoders in Chapter 17).\nFinally, the output layer takes the embedding and outputs a recalibration\nvector containing one number per feature map (e.g., 256), each between 0\nand 1. The feature maps are then multiplied by this recalibration vector, so\nirrelevant features (with a low recalibration score) get scaled down while\nrelevant features (with a recalibration score close to 1) are left alone.\nImplementing a ResNet-34 CNN Using Keras\nMost CNN architectures described so far are fairly straightforward to\nimplement (although generally you would load a pretrained network\ninstead, as we will see). To illustrate the process, let’s implement a\nResNet-34 from scratch using Keras. First, let’s create a ResidualUnit\nlayer:\nclass ResidualUnit(keras.layers.Layer): \n def __init__(self, filters, strides=1, activation=\"relu\", **kwargs): \n super().__init__(**kwargs) \n self.activation = keras.activations.get(activation) \n self.main_layers = [ \n keras.layers.Conv2D(filters, 3, strides=strides, \n padding=\"same\", use_bias=False), \n keras.layers.BatchNormalization(), \n self.activation, \n keras.layers.Conv2D(filters, 3, strides=1, \n padding=\"same\", use_bias=False), \n keras.layers.BatchNormalization()] \n self.skip_layers = [] \n if strides > 1: \n self.skip_layers = [ \n keras.layers.Conv2D(filters, 1, strides=strides, \n padding=\"same\", use_bias=False), \n keras.layers.BatchNormalization()] \n \n def call(self, inputs): \n Z = inputs \n for layer in self.main_layers: \n Z = layer(Z) \n skip_Z = inputs \n for layer in self.skip_layers: \n skip_Z = layer(skip_Z) \n return self.activation(Z + skip_Z)\nAs you can see, this code matches Figure 14-18 pretty closely. In the\nconstructor, we create all the layers we will need: the main layers are the\nones on the right side of the diagram, and the skip layers are the ones on\nthe left (only needed if the stride is greater than 1). Then in the call()\nmethod, we make the inputs go through the main layers and the skip layers\n(if any), then we add both outputs and apply the activation function.\nNext, we can build the ResNet-34 using a Sequential model, since it’s\nreally just a long sequence of layers (we can treat each residual unit as a\nsingle layer now that we have the ResidualUnit class):\nmodel = keras.models.Sequential() \nmodel.add(keras.layers.Conv2D(64, 7, strides=2, input_shape=[224, 224, \n3], \n padding=\"same\", use_bias=False)) \nmodel.add(keras.layers.BatchNormalization()) \nmodel.add(keras.layers.Activation(\"relu\")) \nmodel.add(keras.layers.MaxPool2D(pool_size=3, strides=2, padding=\"same\")) \nprev_filters = 64 \nfor filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3: \n strides = 1 if filters == prev_filters else 2 \n model.add(ResidualUnit(filters, strides=strides)) \n prev_filters = filters \nmodel.add(keras.layers.GlobalAvgPool2D()) \nmodel.add(keras.layers.Flatten()) \nmodel.add(keras.layers.Dense(10, activation=\"softmax\"))\nThe only slightly tricky part in this code is the loop that adds the\nResidualUnit layers to the model: as explained earlier, the first 3 RUs\nhave 64 filters, then the next 4 RUs have 128 filters, and so on. We then set\nthe stride to 1 when the number of filters is the same as in the previous\nRU, or else we set it to 2. Then we add the ResidualUnit, and finally we\nupdate prev_filters.\nIt is amazing that in fewer than 40 lines of code, we can build the model\nthat won the ILSVRC 2015 challenge! This demonstrates both the\nelegance of the ResNet model and the expressiveness of the Keras API.\nImplementing the other CNN architectures is not much harder. However,\nKeras comes with several of these architectures built in, so why not use\nthem instead?\nUsing Pretrained Models from Keras\nIn general, you won’t have to implement standard models like GoogLeNet\nor ResNet manually, since pretrained networks are readily available with a\nsingle line of code in the keras.applications package. For example, you\ncan load the ResNet-50 model, pretrained on ImageNet, with the following\nline of code:\nmodel = keras.applications.resnet50.ResNet50(weights=\"imagenet\")\nThat’s all! This will create a ResNet-50 model and download weights\npretrained on the ImageNet dataset. To use it, you first need to ensure that\nthe images have the right size. A ResNet-50 model expects 224 × 224-\npixel images (other models may expect other sizes, such as 299 × 299), so\nlet’s use TensorFlow’s tf.image.resize() function to resize the images\nwe loaded earlier:\nimages_resized = tf.image.resize(images, [224, 224])\nTIP\nThe tf.image.resize() will not preserve the aspect ratio. If this is a problem, try\ncropping the images to the appropriate aspect ratio before resizing. Both operations\ncan be done in one shot with tf.image.crop_and_resize().\nThe pretrained models assume that the images are preprocessed in a\nspecific way. In some cases they may expect the inputs to be scaled from 0\nto 1, or –1 to 1, and so on. Each model provides a preprocess_input()\nfunction that you can use to preprocess your images. These functions\nassume that the pixel values range from 0 to 255, so we must multiply\nthem by 255 (since earlier we scaled them to the 0–1 range):\ninputs = keras.applications.resnet50.preprocess_input(images_resized * \n255)\nNow we can use the pretrained model to make predictions:\nY_proba = model.predict(inputs)\nAs usual, the output Y_proba is a matrix with one row per image and one\ncolumn per class (in this case, there are 1,000 classes). If you want to\ndisplay the top K predictions, including the class name and the estimated\nprobability of each predicted class, use the decode_predictions()\nfunction. For each image, it returns an array containing the top K\npredictions, where each prediction is represented as an array containing\nthe class identifier,\n its name, and the corresponding confidence score:\ntop_K = keras.applications.resnet50.decode_predictions(Y_proba, top=3) \nfor image_index in range(len(images)): \n print(\"Image #{}\".format(image_index)) \n for class_id, name, y_proba in top_K[image_index]: \n print(\" {} - {:12s} {:.2f}%\".format(class_id, name, y_proba * \n100)) \n print()\n23\nThe output looks like this:\nImage #0 \n n03877845 - palace 42.87% \n n02825657 - bell_cote 40.57% \n n03781244 - monastery 14.56% \n \nImage #1 \n n04522168 - vase 46.83% \n n07930864 - cup 7.78% \n n11939491 - daisy 4.87%\nThe correct classes (monastery and daisy) appear in the top three results\nfor both images. That’s pretty good, considering that the model had to\nchoose from among 1,000 classes.\nAs you can see, it is very easy to create a pretty good image classifier\nusing a pretrained model. Other vision models are available in\nkeras.applications, including several ResNet variants, GoogLeNet\nvariants like Inception-v3 and Xception, VGGNet variants, and MobileNet\nand MobileNetV2 (lightweight models for use in mobile applications).\nBut what if you want to use an image classifier for classes of images that\nare not part of ImageNet? In that case, you may still benefit from the\npretrained models to perform transfer learning.\nPretrained Models for Transfer Learning\nIf you want to build an image classifier but you do not have enough\ntraining data, then it is often a good idea to reuse the lower layers of a\npretrained model, as we discussed in Chapter 11. For example, let’s train a\nmodel to classify pictures of flowers, reusing a pretrained Xception model.\nFirst, let’s load the dataset using TensorFlow Datasets (see Chapter 13):\nimport tensorflow_datasets as tfds \n \ndataset, info = tfds.load(\"tf_flowers\", as_supervised=True, \nwith_info=True) \ndataset_size = info.splits[\"train\"].num_examples # 3670 \nclass_names = info.features[\"label\"].names # [\"dandelion\", \"daisy\", ...] \nn_classes = info.features[\"label\"].num_classes # 5\nNote that you can get information about the dataset by setting\nwith_info=True. Here, we get the dataset size and the names of the\nclasses. Unfortunately, there is only a \"train\" dataset, no test set or\nvalidation set, so we need to split the training set. The TF Datasets project\nprovides an API for this. For example, let’s take the first 10% of the\ndataset for testing, the next 15% for validation, and the remaining 75% for\ntraining:\ntest_split, valid_split, train_split = tfds.Split.TRAIN.subsplit([10, 15, \n75]) \n \ntest_set = tfds.load(\"tf_flowers\", split=test_split, as_supervised=True) \nvalid_set = tfds.load(\"tf_flowers\", split=valid_split, \nas_supervised=True) \ntrain_set = tfds.load(\"tf_flowers\", split=train_split, \nas_supervised=True)\nNext we must preprocess the images. The CNN expects 224 × 224 images,\nso we need to resize them. We also need to run the images through\nXception’s preprocess_input() function:\ndef preprocess(image, label): \n resized_image = tf.image.resize(image, [224, 224]) \n final_image = \nkeras.applications.xception.preprocess_input(resized_image) \n return final_image, label\nLet’s apply this preprocessing function to all three datasets, shuffle the\ntraining set, and add batching and prefetching to all the datasets:\nbatch_size = 32 \ntrain_set = train_set.shuffle(1000) \ntrain_set = train_set.map(preprocess).batch(batch_size).prefetch(1) \nvalid_set = valid_set.map(preprocess).batch(batch_size).prefetch(1) \ntest_set = test_set.map(preprocess).batch(batch_size).prefetch(1)\nIf you want to perform some data augmentation, change the preprocessing\nfunction for the training set, adding some random transformations to the\ntraining images. For example, use tf.image.random_crop() to randomly\ncrop the images, use tf.image.random_flip_left_right() to randomly\nflip the images horizontally, and so on (see the “Pretrained Models for\nTransfer Learning” section of the notebook for an example).\nTIP\nThe keras.preprocessing.image.ImageDataGenerator class makes it easy to\nload images from disk and augment them in various ways: you can shift each image,\nrotate it, rescale it, flip it horizontally or vertically, shear it, or apply any\ntransformation function you want to it. This is very convenient for simple projects.\nHowever, building a tf.data pipeline has many advantages: it can read the images\nefficiently (e.g., in parallel) from any source, not just the local disk; you can\nmanipulate the Dataset as you wish; and if you write a preprocessing function\nbased on tf.image operations, this function can be used both in the tf.data pipeline\nand in the model you will deploy to production (see Chapter 19).\nNext let’s load an Xception model, pretrained on ImageNet. We exclude\nthe top of the network by setting include_top=False: this excludes the\nglobal average pooling layer and the dense output layer. We then add our\nown global average pooling layer, based on the output of the base model,\nfollowed by a dense output layer with one unit per class, using the softmax\nactivation function. Finally, we create the Keras Model:\nbase_model = keras.applications.xception.Xception(weights=\"imagenet\", \n include_top=False) \navg = keras.layers.GlobalAveragePooling2D()(base_model.output) \noutput = keras.layers.Dense(n_classes, activation=\"softmax\")(avg) \nmodel = keras.Model(inputs=base_model.input, outputs=output)\nAs explained in Chapter 11, it’s usually a good idea to freeze the weights\nof the pretrained layers, at least at the beginning of training:\nfor layer in base_model.layers: \n layer.trainable = False\nNOTE\nSince our model uses the base model’s layers directly, rather than the base_model\nobject itself, setting base_model.trainable=False would have no effect.\nFinally, we can compile the model and start training:\noptimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01) \nmodel.compile(loss=\"sparse_categorical_crossentropy\", \noptimizer=optimizer, \n metrics=[\"accuracy\"]) \nhistory = model.fit(train_set, epochs=5, validation_data=valid_set)\nWARNING\nThis will be very slow, unless you have a GPU. If you do not, then you should run\nthis chapter’s notebook in Colab, using a GPU runtime (it’s free!). See the\ninstructions at https://github.com/ageron/handson-ml2.\nAfter training the model for a few epochs, its validation accuracy should\nreach about 75–80% and stop making much progress. This means that the\ntop layers are now pretty well trained, so we are ready to unfreeze all the\nlayers (or you could try unfreezing just the top ones) and continue training\n(don’t forget to compile the model when you freeze or unfreeze layers).\nThis time we use a much lower learning rate to avoid damaging the\npretrained weights:\nfor layer in base_model.layers: \n layer.trainable = True \n \noptimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001) \nmodel.compile(...) \nhistory = model.fit(...)\nIt will take a while, but this model should reach around 95% accuracy on\nthe test set. With that, you can start training amazing image classifiers!\nBut there’s more to computer vision than just classification. For example,\nwhat if you also want to know where the flower is in the picture? Let’s\nlook at this now.\nClassification and Localization\nLocalizing an object in a picture can be expressed as a regression task, as\ndiscussed in Chapter 10: to predict a bounding box around the object, a\ncommon approach is to predict the horizontal and vertical coordinates of\nthe object’s center, as well as its height and width. This means we have\nfour numbers to predict. It does not require much change to the model; we\njust need to add a second dense output layer with four units (typically on\ntop of the global average pooling layer), and it can be trained using the\nMSE loss:\nbase_model = keras.applications.xception.Xception(weights=\"imagenet\", \n include_top=False) \navg = keras.layers.GlobalAveragePooling2D()(base_model.output) \nclass_output = keras.layers.Dense(n_classes, activation=\"softmax\")(avg) \nloc_output = keras.layers.Dense(4)(avg) \nmodel = keras.Model(inputs=base_model.input, \n outputs=[class_output, loc_output]) \nmodel.compile(loss=[\"sparse_categorical_crossentropy\", \"mse\"], \n loss_weights=[0.8, 0.2], # depends on what you care most \nabout \n optimizer=optimizer, metrics=[\"accuracy\"])\nBut now we have a problem: the flowers dataset does not have bounding\nboxes around the flowers. So, we need to add them ourselves. This is often\none of the hardest and most costly parts of a Machine Learning project:\ngetting the labels. It’s a good idea to spend time looking for the right tools.\nTo annotate images with bounding boxes, you may want to use an open\nsource image labeling tool like VGG Image Annotator, LabelImg,\nOpenLabeler, or ImgLab, or perhaps a commercial tool like LabelBox or\nSupervisely. You may also want to consider crowdsourcing platforms such\nas Amazon Mechanical Turk if you have a very large number of images to\nannotate. However, it is quite a lot of work to set up a crowdsourcing\nplatform, prepare the form to be sent to the workers, supervise them, and\nensure that the quality of the bounding boxes they produce is good, so\nmake sure it is worth the effort. If there are just a few thousand images to\nlabel, and you don’t plan to do this frequently, it may be preferable to do it\nyourself. Adriana Kovashka et al. wrote a very practical paper\n about\ncrowdsourcing in computer vision. I recommend you check it out, even if\nyou do not plan to use crowdsourcing.\nLet’s suppose you’ve obtained the bounding boxes for every image in the\nflowers dataset (for now we will assume there is a single bounding box per\nimage). You then need to create a dataset whose items will be batches of\npreprocessed images along with their class labels and their bounding\nboxes. Each item should be a tuple of the form (images,\n(class_labels, bounding_boxes)). Then you are ready to train your\nmodel!\nTIP\nThe bounding boxes should be normalized so that the horizontal and vertical\ncoordinates, as well as the height and width, all range from 0 to 1. Also, it is\ncommon to predict the square root of the height and width rather than the height and\nwidth directly: this way, a 10-pixel error for a large bounding box will not be\npenalized as much as a 10-pixel error for a small bounding box.\nThe MSE often works fairly well as a cost function to train the model, but\nit is not a great metric to evaluate how well the model can predict\nbounding boxes. The most common metric for this is the Intersection over\nUnion (IoU): the area of overlap between the predicted bounding box and\nthe target bounding box, divided by the area of their union (see Figure 14-\n23). In tf.keras, it is implemented by the tf.keras.metrics.MeanIoU\nclass.\n24\nFigure 14-23. Intersection over Union (IoU) metric for bounding boxes\nClassifying and localizing a single object is nice, but what if the images\ncontain multiple objects (as is often the case in the flowers dataset)?\nObject Detection\nThe task of classifying and localizing multiple objects in an image is\ncalled object detection. Until a few years ago, a common approach was to\ntake a CNN that was trained to classify and locate a single object, then\nslide it across the image, as shown in Figure 14-24. In this example, the\nimage was chopped into a 6 × 8 grid, and we show a CNN (the thick black\nrectangle) sliding across all 3 × 3 regions. When the CNN was looking at\nthe top left of the image, it detected part of the leftmost rose, and then it\ndetected that same rose again when it was first shifted one step to the\nright. At the next step, it started detecting part of the topmost rose, and\nthen it detected it again once it was shifted one more step to the right. You\nwould then continue to slide the CNN through the whole image, looking at\nall 3 × 3 regions. Moreover, since objects can have varying sizes, you\nwould also slide the CNN across regions of different sizes. For example,\nonce you are done with the 3 × 3 regions, you might want to slide the CNN\nacross all 4 × 4 regions as well.\nFigure 14-24. Detecting multiple objects by sliding a CNN across the image\nThis technique is fairly straightforward, but as you can see it will detect\nthe same object multiple times, at slightly different positions. Some post-\nprocessing will then be needed to get rid of all the unnecessary bounding\nboxes. A common approach for this is called non-max suppression. Here’s\nhow you do it:\n1. First, you need to add an extra objectness output to your CNN, to\nestimate the probability that a flower is indeed present in the\nimage (alternatively, you could add a “no-flower” class, but this\nusually does not work as well). It must use the sigmoid activation\nfunction, and you can train it using binary cross-entropy loss.\nThen get rid of all the bounding boxes for which the objectness\nscore is below some threshold: this will drop all the bounding\nboxes that don’t actually contain a flower.\n2. Find the bounding box with the highest objectness score, and get\nrid of all the other bounding boxes that overlap a lot with it (e.g.,\nwith an IoU greater than 60%). For example, in Figure 14-24, the\nbounding box with the max objectness score is the thick bounding\nbox over the topmost rose (the objectness score is represented by\nthe thickness of the bounding boxes). The other bounding box\nover that same rose overlaps a lot with the max bounding box, so\nwe will get rid of it.\n3. Repeat step two until there are no more bounding boxes to get rid\nof.\nThis simple approach to object detection works pretty well, but it requires\nrunning the CNN many times, so it is quite slow. Fortunately, there is a\nmuch faster way to slide a CNN across an image: using a fully\nconvolutional network (FCN).\nFully Convolutional Networks\nThe idea of FCNs was first introduced in a 2015 paper\n by Jonathan Long\net al., for semantic segmentation (the task of classifying every pixel in an\nimage according to the class of the object it belongs to). The authors\npointed out that you could replace the dense layers at the top of a CNN by\nconvolutional layers. To understand this, let’s look at an example: suppose\na dense layer with 200 neurons sits on top of a convolutional layer that\noutputs 100 feature maps, each of size 7 × 7 (this is the feature map size,\nnot the kernel size). Each neuron will compute a weighted sum of all 100 ×\n7 × 7 activations from the convolutional layer (plus a bias term). Now let’s\nsee what happens if we replace the dense layer with a convolutional layer\nusing 200 filters, each of size 7 × 7, and with \"valid\" padding. This layer\nwill output 200 feature maps, each 1 × 1 (since the kernel is exactly the\nsize of the input feature maps and we are using \"valid\" padding). In other\nwords, it will output 200 numbers, just like the dense layer did; and if you\nlook closely at the computations performed by a convolutional layer, you\nwill notice that these numbers will be precisely the same as those the\ndense layer produced. The only difference is that the dense layer’s output\n25\nwas a tensor of shape [batch size, 200], while the convolutional layer will\noutput a tensor of shape [batch size, 1, 1, 200].\nTIP\nTo convert a dense layer to a convolutional layer, the number of filters in the\nconvolutional layer must be equal to the number of units in the dense layer, the filter\nsize must be equal to the size of the input feature maps, and you must use \"valid\"\npadding. The stride may be set to 1 or more, as we will see shortly.\nWhy is this important? Well, while a dense layer expects a specific input\nsize (since it has one weight per input feature), a convolutional layer will\nhappily process images of any size\n (however, it does expect its inputs to\nhave a specific number of channels, since each kernel contains a different\nset of weights for each input channel). Since an FCN contains only\nconvolutional layers (and pooling layers, which have the same property), it\ncan be trained and executed on images of any size!\nFor example, suppose we’d already trained a CNN for flower classification\nand localization. It was trained on 224 × 224 images, and it outputs 10\nnumbers: outputs 0 to 4 are sent through the softmax activation function,\nand this gives the class probabilities (one per class); output 5 is sent\nthrough the logistic activation function, and this gives the objectness\nscore; outputs 6 to 9 do not use any activation function, and they represent\nthe bounding box’s center coordinates, as well as its height and width. We\ncan now convert its dense layers to convolutional layers. In fact, we don’t\neven need to retrain it; we can just copy the weights from the dense layers\nto the convolutional layers! Alternatively, we could have converted the\nCNN into an FCN before training.\nNow suppose the last convolutional layer before the output layer (also\ncalled the bottleneck layer) outputs 7 × 7 feature maps when the network\nis fed a 224 × 224 image (see the left side of Figure 14-25). If we feed the\nFCN a 448 × 448 image (see the right side of Figure 14-25), the bottleneck\nlayer will now output 14 × 14 feature maps.\n Since the dense output layer\n26\n27\nwas replaced by a convolutional layer using 10 filters of size 7 × 7, with\n\"valid\" padding and stride 1, the output will be composed of 10 features\nmaps, each of size 8 × 8 (since 14 – 7 + 1 = 8). In other words, the FCN\nwill process the whole image only once, and it will output an 8 × 8 grid\nwhere each cell contains 10 numbers (5 class probabilities, 1 objectness\nscore, and 4 bounding box coordinates). It’s exactly like taking the original\nCNN and sliding it across the image using 8 steps per row and 8 steps per\ncolumn. To visualize this, imagine chopping the original image into a 14 ×\n14 grid, then sliding a 7 × 7 window across this grid; there will be 8 × 8 =\n64 possible locations for the window, hence 8 × 8 predictions. However,\nthe FCN approach is much more efficient, since the network only looks at\nthe image once. In fact, You Only Look Once (YOLO) is the name of a very\npopular object detection architecture, which we’ll look at next.\nFigure 14-25. The same fully convolutional network processing a small image (left) and a large\none (right)\nYou Only Look Once (YOLO)\nYOLO is an extremely fast and accurate object detection architecture\nproposed by Joseph Redmon et al. in a 2015 paper,\n and subsequently\nimproved in 2016\n (YOLOv2) and in 2018\n (YOLOv3). It is so fast that\nit can run in real time on a video, as seen in Redmon’s demo.\nYOLOv3’s architecture is quite similar to the one we just discussed, but\nwith a few important differences:\nIt outputs five bounding boxes for each grid cell (instead of just\none), and each bounding box comes with an objectness score. It\nalso outputs 20 class probabilities per grid cell, as it was trained\non the PASCAL VOC dataset, which contains 20 classes. That’s a\ntotal of 45 numbers per grid cell: 5 bounding boxes, each with 4\ncoordinates, plus 5 objectness scores, plus 20 class probabilities.\nInstead of predicting the absolute coordinates of the bounding box\ncenters, YOLOv3 predicts an offset relative to the coordinates of\nthe grid cell, where (0, 0) means the top left of that cell and (1, 1)\nmeans the bottom right. For each grid cell, YOLOv3 is trained to\npredict only bounding boxes whose center lies in that cell (but the\nbounding box itself generally extends well beyond the grid cell).\nYOLOv3 applies the logistic activation function to the bounding\nbox coordinates to ensure they remain in the 0 to 1 range.\nBefore training the neural net, YOLOv3 finds five representative\nbounding box dimensions, called anchor boxes (or bounding box\npriors). It does this by applying the K-Means algorithm (see\nChapter 9) to the height and width of the training set bounding\nboxes. For example, if the training images contain many\npedestrians, then one of the anchor boxes will likely have the\ndimensions of a typical pedestrian. Then when the neural net\npredicts five bounding boxes per grid cell, it actually predicts how\nmuch to rescale each of the anchor boxes. For example, suppose\none anchor box is 100 pixels tall and 50 pixels wide, and the\nnetwork predicts, say, a vertical rescaling factor of 1.5 and a\n28\n29\n30\nhorizontal rescaling of 0.9 (for one of the grid cells). This will\nresult in a predicted bounding box of size 150 × 45 pixels. To be\nmore precise, for each grid cell and each anchor box, the network\npredicts the log of the vertical and horizontal rescaling factors.\nHaving these priors makes the network more likely to predict\nbounding boxes of the appropriate dimensions, and it also speeds\nup training because it will more quickly learn what reasonable\nbounding boxes look like.\nThe network is trained using images of different scales: every few\nbatches during training, the network randomly chooses a new\nimage dimension (from 330 × 330 to 608 × 608 pixels). This\nallows the network to learn to detect objects at different scales.\nMoreover, it makes it possible to use YOLOv3 at different scales:\nthe smaller scale will be less accurate but faster than the larger\nscale, so you can choose the right trade-off for your use case.\nThere are a few more innovations you might be interested in, such as the\nuse of skip connections to recover some of the spatial resolution that is\nlost in the CNN (we will discuss this shortly, when we look at semantic\nsegmentation). In the 2016 paper, the authors introduce the YOLO9000\nmodel that uses hierarchical classification: the model predicts a\nprobability for each node in a visual hierarchy called WordTree. This\nmakes it possible for the network to predict with high confidence that an\nimage represents, say, a dog, even though it is unsure what specific type of\ndog. I encourage you to go ahead and read all three papers: they are quite\npleasant to read, and they provide excellent examples of how Deep\nLearning systems can be incrementally improved.\nMEAN AVERAGE PRECISION (MAP)\nA very common metric used in object detection tasks is the mean\nAverage Precision (mAP). “Mean Average” sounds a bit redundant,\ndoesn’t it? To understand this metric, let’s go back to two\nclassification metrics we discussed in Chapter 3: precision and recall.\nRemember the trade-off: the higher the recall, the lower the precision.\nYou can visualize this in a precision/recall curve (see Figure 3-5). To\nsummarize this curve into a single number, we could compute its area\nunder the curve (AUC). But note that the precision/recall curve may\ncontain a few sections where precision actually goes up when recall\nincreases, especially at low recall values (you can see this at the top\nleft of Figure 3-5). This is one of the motivations for the mAP metric.\nSuppose the classifier has 90% precision at 10% recall, but 96%\nprecision at 20% recall. There’s really no trade-off here: it simply\nmakes more sense to use the classifier at 20% recall rather than at\n10% recall, as you will get both higher recall and higher precision. So\ninstead of looking at the precision at 10% recall, we should really be\nlooking at the maximum precision that the classifier can offer with at\nleast 10% recall. It would be 96%, not 90%. Therefore, one way to get\na fair idea of the model’s performance is to compute the maximum\nprecision you can get with at least 0% recall, then 10% recall, 20%,\nand so on up to 100%, and then calculate the mean of these maximum\nprecisions. This is called the Average Precision (AP) metric. Now\nwhen there are more than two classes, we can compute the AP for each\nclass, and then compute the mean AP (mAP). That’s it!\nIn an object detection system, there is an additional level of\ncomplexity: what if the system detected the correct class, but at the\nwrong location (i.e., the bounding box is completely off)? Surely we\nshould not count this as a positive prediction. One approach is to\ndefine an IOU threshold: for example, we may consider that a\nprediction is correct only if the IOU is greater than, say, 0.5, and the\npredicted class is correct. The corresponding mAP is generally noted\nmAP@0.5 (or mAP@50%, or sometimes just AP ). In some\ncompetitions (such as the PASCAL VOC challenge), this is what is\ndone. In others (such as the COCO competition), the mAP is computed\nfor different IOU thresholds (0.50, 0.55, 0.60, …, 0.95), and the final\nmetric is the mean of all these mAPs (noted AP@[.50:.95] or\nAP@[.50:0.05:.95]). Yes, that’s a mean mean average.\nSeveral YOLO implementations built using TensorFlow are available on\nGitHub. In particular, check out Zihao Zang’s TensorFlow 2\nimplementation. Other object detection models are available in the\nTensorFlow Models project, many with pretrained weights; and some have\neven been ported to TF Hub, such as SSD and Faster-RCNN,\n which are\nboth quite popular. SSD is also a “single shot” detection model, similar to\nYOLO. Faster R-CNN is more complex: the image first goes through a\nCNN, then the output is passed to a Region Proposal Network (RPN) that\nproposes bounding boxes that are most likely to contain an object, and a\nclassifier is run for each bounding box, based on the cropped output of the\nCNN.\nThe choice of detection system depends on many factors: speed, accuracy,\navailable pretrained models, training time, complexity, etc. The papers\ncontain tables of metrics, but there is quite a lot of variability in the\ntesting environments, and the technologies evolve so fast that it is difficult\nto make a fair comparison that will be useful for most people and remain\nvalid for more than a few months.\nSo, we can locate objects by drawing bounding boxes around them. Great!\nBut perhaps you want to be a bit more precise. Let’s see how to go down to\nthe pixel level.\nSemantic Segmentation\nIn semantic segmentation, each pixel is classified according to the class of\nthe object it belongs to (e.g., road, car, pedestrian, building, etc.), as shown\nin Figure 14-26. Note that different objects of the same class are not\n50\n31\n32\ndistinguished. For example, all the bicycles on the right side of the\nsegmented image end up as one big lump of pixels. The main difficulty in\nthis task is that when images go through a regular CNN, they gradually\nlose their spatial resolution (due to the layers with strides greater than 1);\nso, a regular CNN may end up knowing that there’s a person somewhere in\nthe bottom left of the image, but it will not be much more precise than\nthat.\nJust like for object detection, there are many different approaches to tackle\nthis problem, some quite complex. However, a fairly simple solution was\nproposed in the 2015 paper by Jonathan Long et al. we discussed earlier.\nThe authors start by taking a pretrained CNN and turning it into an FCN.\nThe CNN applies an overall stride of 32 to the input image (i.e., if you add\nup all the strides greater than 1), meaning the last layer outputs feature\nmaps that are 32 times smaller than the input image. This is clearly too\ncoarse, so they add a single upsampling layer that multiplies the resolution\nby 32.\nFigure 14-26. Semantic segmentation\nThere are several solutions available for upsampling (increasing the size\nof an image), such as bilinear interpolation, but that only works reasonably\nwell up to ×4 or ×8. Instead, they use a transposed convolutional layer:\nit is equivalent to first stretching the image by inserting empty rows and\ncolumns (full of zeros), then performing a regular convolution (see\nFigure 14-27). Alternatively, some people prefer to think of it as a regular\n33\nconvolutional layer that uses fractional strides (e.g., 1/2 in Figure 14-27).\nThe transposed convolutional layer can be initialized to perform\nsomething close to linear interpolation, but since it is a trainable layer, it\nwill learn to do better during training. In tf.keras, you can use the\nConv2DTranspose layer.\nFigure 14-27. Upsampling using a transposed convolutional layer\nNOTE\nIn a transposed convolutional layer, the stride defines how much the input will be\nstretched, not the size of the filter steps, so the larger the stride, the larger the output\n(unlike for convolutional layers or pooling layers).\nTENSORFLOW CONVOLUTION OPERATIONS\nTensorFlow also offers a few other kinds of convolutional layers:\nkeras.layers.Conv1D\nCreates a convolutional layer for 1D inputs, such as time series or\ntext (sequences of letters or words), as we will see in Chapter 15.\nkeras.layers.Conv3D\nCreates a convolutional layer for 3D inputs, such as 3D PET scans.\ndilation_rate\nSetting the dilation_rate hyperparameter of any convolutional\nlayer to a value of 2 or more creates an à-trous convolutional layer\n(“à trous” is French for “with holes”). This is equivalent to using a\nregular convolutional layer with a filter dilated by inserting rows\nand columns of zeros (i.e., holes). For example, a 1 × 3 filter equal\nto [[1,2,3]] may be dilated with a dilation rate of 4, resulting in\na dilated filter of [[1, 0, 0, 0, 2, 0, 0, 0, 3]]. This lets the\nconvolutional layer have a larger receptive field at no\ncomputational price and using no extra parameters.\ntf.nn.depthwise_conv2d()\nCan be used to create a depthwise convolutional layer (but you\nneed to create the variables yourself). It applies every filter to\nevery individual input channel independently. Thus, if there are f\nfilters and f input channels, then this will output f × f feature\nmaps.\nThis solution is OK, but still too imprecise. To do better, the authors added\nskip connections from lower layers: for example, they upsampled the\noutput image by a factor of 2 (instead of 32), and they added the output of\na lower layer that had this double resolution. Then they upsampled the\nresult by a factor of 16, leading to a total upsampling factor of 32 (see\nn\nn′\nn\nn′\nFigure 14-28). This recovered some of the spatial resolution that was lost\nin earlier pooling layers. In their best architecture, they used a second\nsimilar skip connection to recover even finer details from an even lower\nlayer. In short, the output of the original CNN goes through the following\nextra steps: upscale ×2, add the output of a lower layer (of the appropriate\nscale), upscale ×2, add the output of an even lower layer, and finally\nupscale ×8. It is even possible to scale up beyond the size of the original\nimage: this can be used to increase the resolution of an image, which is a\ntechnique called super-resolution.\nFigure 14-28. Skip layers recover some spatial resolution from lower layers\nOnce again, many GitHub repositories provide TensorFlow\nimplementations of semantic segmentation (TensorFlow 1 for now), and\nyou will even find pretrained instance segmentation models in the\nTensorFlow Models project. Instance segmentation is similar to semantic\nsegmentation, but instead of merging all objects of the same class into one\nbig lump, each object is distinguished from the others (e.g., it identifies\neach individual bicycle). At present, the instance segmentation models\navailable in the TensorFlow Models project are based on the Mask R-CNN\narchitecture, which was proposed in a 2017 paper:\n it extends the Faster\nR-CNN model by additionally producing a pixel mask for each bounding\nbox. So not only do you get a bounding box around each object, with a set\nof estimated class probabilities, but you also get a pixel mask that locates\npixels in the bounding box that belong to the object.\nAs you can see, the field of Deep Computer Vision is vast and moving fast,\nwith all sorts of architectures popping out every year, all based on\nconvolutional neural networks. The progress made in just a few years has\n34\nbeen astounding, and researchers are now focusing on harder and harder\nproblems, such as adversarial learning (which attempts to make the\nnetwork more resistant to images designed to fool it), explainability\n(understanding why the network makes a specific classification), realistic\nimage generation (which we will come back to in Chapter 17), and single-\nshot learning (a system that can recognize an object after it has seen it just\nonce). Some even explore completely novel architectures, such as\nGeoffrey Hinton’s capsule networks\n (I presented them in a couple of\nvideos, with the corresponding code in a notebook). Now on to the next\nchapter, where we will look at how to process sequential data such as time\nseries using recurrent neural networks and convolutional neural networks.\nExercises\n1. What are the advantages of a CNN over a fully connected DNN\nfor image classification?\n2. Consider a CNN composed of three convolutional layers, each\nwith 3 × 3 kernels, a stride of 2, and \"same\" padding. The lowest\nlayer outputs 100 feature maps, the middle one outputs 200, and\nthe top one outputs 400. The input images are RGB images of 200\n× 300 pixels.\nWhat is the total number of parameters in the CNN? If we are\nusing 32-bit floats, at least how much RAM will this network\nrequire when making a prediction for a single instance? What\nabout when training on a mini-batch of 50 images?\n3. If your GPU runs out of memory while training a CNN, what are\nfive things you could try to solve the problem?\n4. Why would you want to add a max pooling layer rather than a\nconvolutional layer with the same stride?\n5. When would you want to add a local response normalization\nlayer?\n35\n6. Can you name the main innovations in AlexNet, compared to\nLeNet-5? What about the main innovations in GoogLeNet,\nResNet, SENet, and Xception?\n7. What is a fully convolutional network? How can you convert a\ndense layer into a convolutional layer?\n8. What is the main technical difficulty of semantic segmentation?\n9. Build your own CNN from scratch and try to achieve the highest\npossible accuracy on MNIST.\n10. Use transfer learning for large image classification, going through\nthese steps:\na. Create a training set containing at least 100 images per\nclass. For example, you could classify your own pictures\nbased on the location (beach, mountain, city, etc.), or\nalternatively you can use an existing dataset (e.g., from\nTensorFlow Datasets).\nb. Split it into a training set, a validation set, and a test set.\nc. Build the input pipeline, including the appropriate\npreprocessing operations, and optionally add data\naugmentation.\nd. Fine-tune a pretrained model on this dataset.\n11. Go through TensorFlow’s Style Transfer tutorial. It is a fun way to\ngenerate art using Deep Learning.\nSolutions to these exercises are available in Appendix A.\n1 David H. Hubel, “Single Unit Activity in Striate Cortex of Unrestrained Cats,” The Journal\nof Physiology 147 (1959): 226–238.\n2 David H. Hubel and Torsten N. Wiesel, “Receptive Fields of Single Neurons in the Cat’s\nStriate Cortex,” The Journal of Physiology 148 (1959): 574–591.\n3 David H. Hubel and Torsten N. Wiesel, “Receptive Fields and Functional Architecture of\nMonkey Striate Cortex,” The Journal of Physiology 195 (1968): 215–243.\n4 Kunihiko Fukushima, “Neocognitron: A Self-Organizing Neural Network Model for a\nMechanism of Pattern Recognition Unaffected by Shift in Position,” Biological Cybernetics\n36 (1980): 193–202.\n5 Yann LeCun et al., “Gradient-Based Learning Applied to Document Recognition,”\nProceedings of the IEEE 86, no. 11 (1998): 2278–2324.\n6 A convolution is a mathematical operation that slides one function over another and\nmeasures the integral of their pointwise multiplication. It has deep connections with the\nFourier transform and the Laplace transform and is heavily used in signal processing.\nConvolutional layers actually use cross-correlations, which are very similar to convolutions\n(see https://homl.info/76 for more details).\n7 A fully connected layer with 150 × 100 neurons, each connected to all 150 × 100 × 3\ninputs, would have 150\n2 × 100\n2 × 3 = 675 million parameters!\n8 In the international system of units (SI), 1 MB = 1,000 KB = 1,000 × 1,000 bytes = 1,000\n× 1,000 × 8 bits.\n9 Other kernels we’ve discussed so far had weights, but pooling kernels do not: they are just\nstateless sliding windows.\n10 Yann LeCun et al., “Gradient-Based Learning Applied to Document Recognition,”\nProceedings of the IEEE 86, no. 11 (1998): 2278–2324.\n11 Alex Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural\nNetworks,” _Proceedings of the 25th International Conference on Neural Information\nProcessing Systems 1 (2012): 1097–1105.\n12 Matthew D. Zeiler and Rob Fergus, “Visualizing and Understanding Convolutional\nNetworks,” Proceedings of the European Conference on Computer Vision (2014): 818-833.\n13 Christian Szegedy et al., “Going Deeper with Convolutions,” Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition (2015): 1–9.\n14 In the 2010 movie Inception, the characters keep going deeper and deeper into multiple\nlayers of dreams; hence the name of these modules.\n15 Karen Simonyan and Andrew Zisserman, “Very Deep Convolutional Networks for Large-\nScale Image Recognition,” arXiv preprint arXiv:1409.1556 (2014).\n16 Kaiming He et al., “Deep Residual Learning for Image Recognition,” arXiv preprint\narXiv:1512:03385 (2015).\n17 It is a common practice when describing a neural network to count only layers with\nparameters.\n18 Christian Szegedy et al., “Inception–v4, Inception-ResNet and the Impact of Residual\nConnections on Learning,” arXiv preprint arXiv:1602.07261 (2016).\n19 François Chollet, “Xception: Deep Learning with Depthwise Separable Convolutions,”\narXiv preprint arXiv:1610.02357 (2016).\n20 This name can sometimes be ambiguous, since spatially separable convolutions are often\ncalled “separable convolutions” as well.\n21 Xingyu Zeng et al., “Crafting GBD-Net for Object Detection,” IEEE Transactions on\nPattern Analysis and Machine Intelligence 40, no. 9 (2018): 2109–2123.\n22 Jie Hu et al., “Squeeze-and-Excitation Networks,” Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition (2018): 7132–7141.\n23 In the ImageNet dataset, each image is associated to a word in the WordNet dataset: the\nclass ID is just a WordNet ID.\n24 Adriana Kovashka et al., “Crowdsourcing in Computer Vision,” Foundations and Trends\nin Computer Graphics and Vision 10, no. 3 (2014): 177–243.\n25 Jonathan Long et al., “Fully Convolutional Networks for Semantic Segmentation,”\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015):\n3431–3440.\n26 There is one small exception: a convolutional layer using \"valid\" padding will complain\nif the input size is smaller than the kernel size.\n27 This assumes we used only \"same\" padding in the network: indeed, \"valid\" padding\nwould reduce the size of the feature maps. Moreover, 448 can be neatly divided by 2\nseveral times until we reach 7, without any rounding error. If any layer uses a different\nstride than 1 or 2, then there may be some rounding error, so again the feature maps may\nend up being smaller.\n28 Joseph Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection,”\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016):\n779–788.\n29 Joseph Redmon and Ali Farhadi, “YOLO9000: Better, Faster, Stronger,” Proceedings of\nthe IEEE Conference on Computer Vision and Pattern Recognition (2017): 6517–6525.\n30 Joseph Redmon and Ali Farhadi, “YOLOv3: An Incremental Improvement,” arXiv\npreprint arXiv:1804.02767 (2018).\n31 Wei Liu et al., “SSD: Single Shot Multibox Detector,” Proceedings of the 14th European\nConference on Computer Vision 1 (2016): 21–37.\n32 Shaoqing Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region\nProposal Networks,” Proceedings of the 28th International Conference on Neural\nInformation Processing Systems 1 (2015): 91–99.\n33 This type of layer is sometimes referred to as a deconvolution layer, but it does not\nperform what mathematicians call a deconvolution, so this name should be avoided.\n34 Kaiming He et al., “Mask R-CNN,” arXiv preprint arXiv:1703.06870 (2017).\n35 Geoffrey Hinton et al., “Matrix Capsules with EM Routing,” Proceedings of the\nInternational Conference on Learning Representations (2018)." + }, + { + "chapter": 15, + "text": "Chapter 15. Processing\nSequences Using RNNs and\nCNNs\nThe batter hits the ball. The outfielder immediately starts running,\nanticipating the ball’s trajectory. He tracks it, adapts his movements, and\nfinally catches it (under a thunder of applause). Predicting the future is\nsomething you do all the time, whether you are finishing a friend’s\nsentence or anticipating the smell of coffee at breakfast. In this chapter we\nwill discuss recurrent neural networks (RNNs), a class of nets that can\npredict the future (well, up to a point, of course). They can analyze time\nseries data such as stock prices, and tell you when to buy or sell. In\nautonomous driving systems, they can anticipate car trajectories and help\navoid accidents. More generally, they can work on sequences of arbitrary\nlengths, rather than on fixed-sized inputs like all the nets we have\nconsidered so far. For example, they can take sentences, documents, or\naudio samples as input, making them extremely useful for natural\nlanguage processing applications such as automatic translation or speech-\nto-text.\nIn this chapter we will first look at the fundamental concepts underlying\nRNNs and how to train them using backpropagation through time, then we\nwill use them to forecast a time series. After that we’ll explore the two\nmain difficulties that RNNs face:\nUnstable gradients (discussed in Chapter 11), which can be\nalleviated using various techniques, including recurrent dropout\nand recurrent layer normalization\nA (very) limited short-term memory, which can be extended using\nLSTM and GRU cells\nRNNs are not the only types of neural networks capable of handling\nsequential data: for small sequences, a regular dense network can do the\ntrick; and for very long sequences, such as audio samples or text,\nconvolutional neural networks can actually work quite well too. We will\ndiscuss both of these possibilities, and we will finish this chapter by\nimplementing a WaveNet: this is a CNN architecture capable of handling\nsequences of tens of thousands of time steps. In Chapter 16, we will\ncontinue to explore RNNs and see how to use them for natural language\nprocessing, along with more recent architectures based on attention\nmechanisms. Let’s get started!\nRecurrent Neurons and Layers\nUp to now we have focused on feedforward neural networks, where the\nactivations flow only in one direction, from the input layer to the output\nlayer (a few exceptions are discussed in Appendix E). A recurrent neural\nnetwork looks very much like a feedforward neural network, except it also\nhas connections pointing backward. Let’s look at the simplest possible\nRNN, composed of one neuron receiving inputs, producing an output, and\nsending that output back to itself, as shown in Figure 15-1 (left). At each\ntime step t (also called a frame), this recurrent neuron receives the inputs\nx as well as its own output from the previous time step, y\n. Since there\nis no previous output at the first time step, it is generally set to 0. We can\nrepresent this tiny network against the time axis, as shown in Figure 15-1\n(right). This is called unrolling the network through time (it’s the same\nrecurrent neuron represented once per time step).\n(t)\n(t–1)\nFigure 15-1. A recurrent neuron (left) unrolled through time (right)\nYou can easily create a layer of recurrent neurons. At each time step t,\nevery neuron receives both the input vector x and the output vector from\nthe previous time step y\n, as shown in Figure 15-2. Note that both the\ninputs and outputs are vectors now (when there was just a single neuron,\nthe output was a scalar).\nFigure 15-2. A layer of recurrent neurons (left) unrolled through time (right)\nEach recurrent neuron has two sets of weights: one for the inputs x and\nthe other for the outputs of the previous time step, y\n. Let’s call these\nweight vectors w and w . If we consider the whole recurrent layer instead\nof just one recurrent neuron, we can place all the weight vectors in two\nweight matrices, W and W . The output vector of the whole recurrent\nlayer can then be computed pretty much as you might expect, as shown in\n(t)\n(t–1)\n(t)\n(t–1)\nx\ny\nx\ny\nEquation 15-1 (b is the bias vector and ϕ(·) is the activation function (e.g.,\nReLU ).\nEquation 15-1. Output of a recurrent layer for a single instance\ny(t) = ϕ(Wx\n⊺x(t) + Wy\n⊺y(t−1) + b)\nJust as with feedforward neural networks, we can compute a recurrent\nlayer’s output in one shot for a whole mini-batch by placing all the inputs\nat time step t in an input matrix X (see Equation 15-2).\nEquation 15-2. Outputs of a layer of recurrent neurons for all instances in a mini-batch\nY(t)\n= ϕ(X(t)Wx + Y(t−1)Wy + b)\n= ϕ([X(t)\nY(t−1)]W + b) with W =[ Wx\nWy\n]\nIn this equation:\nY is an m × n\n matrix containing the layer’s outputs at time\nstep t for each instance in the mini-batch (m is the number of\ninstances in the mini-batch and n\n is the number of neurons).\nX is an m × n\n matrix containing the inputs for all instances\n(n\n is the number of input features).\nW is an n\n × n\n matrix containing the connection\nweights for the inputs of the current time step.\nW is an n\n × n\n matrix containing the connection\nweights for the outputs of the previous time step.\nb is a vector of size n\n containing each neuron’s bias term.\nThe weight matrices W and W are often concatenated vertically\ninto a single weight matrix W of shape (n\n + n\n) ×\nn\n (see the second line of Equation 15-2).\n1\n(t)\n(t)\nneurons\nneurons\n(t)\ninputs\ninputs\nx\ninputs\nneurons\ny\nneurons\nneurons\nneurons\nx\ny\ninputs\nneurons\nneurons\nThe notation [X Y\n] represents the horizontal concatenation\nof the matrices X and Y\n.\nNotice that Y is a function of X and Y\n, which is a function of X\nand Y\n, which is a function of X\n and Y\n, and so on. This makes\nY a function of all the inputs since time t = 0 (that is, X\n, X\n, …, X ).\nAt the first time step, t = 0, there are no previous outputs, so they are\ntypically assumed to be all zeros.\nMemory Cells\nSince the output of a recurrent neuron at time step t is a function of all the\ninputs from previous time steps, you could say it has a form of memory. A\npart of a neural network that preserves some state across time steps is\ncalled a memory cell (or simply a cell). A single recurrent neuron, or a\nlayer of recurrent neurons, is a very basic cell, capable of learning only\nshort patterns (typically about 10 steps long, but this varies depending on\nthe task). Later in this chapter, we will look at some more complex and\npowerful types of cells capable of learning longer patterns (roughly 10\ntimes longer, but again, this depends on the task).\nIn general a cell’s state at time step t, denoted h (the “h” stands for\n“hidden”), is a function of some inputs at that time step and its state at the\nprevious time step: h = f(h\n, x ). Its output at time step t, denoted y ,\nis also a function of the previous state and the current inputs. In the case of\nthe basic cells we have discussed so far, the output is simply equal to the\nstate, but in more complex cells this is not always the case, as shown in\nFigure 15-3.\n(t)\n(t–1)\n(t)\n(t–1)\n(t)\n(t)\n(t–1)\n(t–1)\n(t–2)\n(t–2)\n(t–3)\n(t)\n(0)\n(1)\n(t)\n(t)\n(t)\n(t–1)\n(t)\n(t)\nFigure 15-3. A cell’s hidden state and its output may be different\nInput and Output Sequences\nAn RNN can simultaneously take a sequence of inputs and produce a\nsequence of outputs (see the top-left network in Figure 15-4). This type of\nsequence-to-sequence network is useful for predicting time series such as\nstock prices: you feed it the prices over the last N days, and it must output\nthe prices shifted by one day into the future (i.e., from N – 1 days ago to\ntomorrow).\nAlternatively, you could feed the network a sequence of inputs and ignore\nall outputs except for the last one (see the top-right network in Figure 15-\n4). In other words, this is a sequence-to-vector network. For example, you\ncould feed the network a sequence of words corresponding to a movie\nreview, and the network would output a sentiment score (e.g., from –1\n[hate] to +1 [love]).\nConversely, you could feed the network the same input vector over and\nover again at each time step and let it output a sequence (see the bottom-\nleft network of Figure 15-4). This is a vector-to-sequence network. For\nexample, the input could be an image (or the output of a CNN), and the\noutput could be a caption for that image.\nLastly, you could have a sequence-to-vector network, called an encoder,\nfollowed by a vector-to-sequence network, called a decoder (see the\nbottom-right network of Figure 15-4). For example, this could be used for\ntranslating a sentence from one language to another. You would feed the\nnetwork a sentence in one language, the encoder would convert this\nsentence into a single vector representation, and then the decoder would\ndecode this vector into a sentence in another language. This two-step\nmodel, called an Encoder–Decoder, works much better than trying to\ntranslate on the fly with a single sequence-to-sequence RNN (like the one\nrepresented at the top left): the last words of a sentence can affect the first\nwords of the translation, so you need to wait until you have seen the whole\nsentence before translating it. We will see how to implement an Encoder–\nDecoder in Chapter 16 (as we will see, it is a bit more complex than in\nFigure 15-4 suggests).\nFigure 15-4. Seq-to-seq (top left), seq-to-vector (top right), vector-to-seq (bottom left), and\nEncoder–Decoder (bottom right) networks\nSounds promising, but how do you train a recurrent neural network?\nTraining RNNs\nTo train an RNN, the trick is to unroll it through time (like we just did) and\nthen simply use regular backpropagation (see Figure 15-5). This strategy\nis called backpropagation through time (BPTT).\nJust like in regular backpropagation, there is a first forward pass through\nthe unrolled network (represented by the dashed arrows). Then the output\nsequence is evaluated using a cost function C(Y\n, Y\n, …Y\n) (where T\nis the max time step). Note that this cost function may ignore some\noutputs, as shown in Figure 15-5 (for example, in a sequence-to-vector\nRNN, all outputs are ignored except for the very last one). The gradients of\nthat cost function are then propagated backward through the unrolled\nnetwork (represented by the solid arrows). Finally the model parameters\nare updated using the gradients computed during BPTT. Note that the\ngradients flow backward through all the outputs used by the cost function,\nnot just through the final output (for example, in Figure 15-5 the cost\nfunction is computed using the last three outputs of the network, Y\n, Y\n,\nand Y\n, so gradients flow through these three outputs, but not through\nY\n and Y\n). Moreover, since the same parameters W and b are used at\neach time step, backpropagation will do the right thing and sum over all\ntime steps.\n(0)\n(1)\n(T)\n(2)\n(3)\n(4)\n(0)\n(1)\nFigure 15-5. Backpropagation through time\nFortunately, tf.keras takes care of all of this complexity for you—so let’s\nstart coding!\nForecasting a Time Series\nSuppose you are studying the number of active users per hour on your\nwebsite, or the daily temperature in your city, or your company’s financial\nhealth, measured quarterly using multiple metrics. In all these cases, the\ndata will be a sequence of one or more values per time step. This is called\na time series. In the first two examples there is a single value per time\nstep, so these are univariate time series, while in the financial example\nthere are multiple values per time step (e.g., the company’s revenue, debt,\nand so on), so it is a multivariate time series. A typical task is to predict\nfuture values, which is called forecasting. Another common task is to fill\nin the blanks: to predict (or rather “postdict”) missing values from the\npast. This is called imputation. For example, Figure 15-6 shows 3\nunivariate time series, each of them 50 time steps long, and the goal here\nis to forecast the value at the next time step (represented by the X) for\neach of them.\nFigure 15-6. Time series forecasting\nFor simplicity, we are using a time series generated by the\ngenerate_time_series() function, shown here:\ndef generate_time_series(batch_size, n_steps): \n freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1) \n time = np.linspace(0, 1, n_steps) \n series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10)) # \nwave 1 \n series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20)) # + \nwave 2 \n series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5) # + \nnoise \n return series[..., np.newaxis].astype(np.float32)\nThis function creates as many time series as requested (via the\nbatch_size argument), each of length n_steps, and there is just one\nvalue per time step in each series (i.e., all series are univariate). The\nfunction returns a NumPy array of shape [batch size, time steps, 1], where\neach series is the sum of two sine waves of fixed amplitudes but random\nfrequencies and phases, plus a bit of noise.\nNOTE\nWhen dealing with time series (and other types of sequences such as sentences), the\ninput features are generally represented as 3D arrays of shape [batch size, time steps,\ndimensionality], where dimensionality is 1 for univariate time series and more for\nmultivariate time series.\nNow let’s create a training set, a validation set, and a test set using this\nfunction:\nn_steps = 50 \nseries = generate_time_series(10000, n_steps + 1) \nX_train, y_train = series[:7000, :n_steps], series[:7000, -1] \nX_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1] \nX_test, y_test = series[9000:, :n_steps], series[9000:, -1]\nX_train contains 7,000 time series (i.e., its shape is [7000, 50, 1]), while\nX_valid contains 2,000 (from the 7,000th time series to the 8,999th) and\nX_test contains 1,000 (from the 9,000 to the 9,999 ). Since we want to\nforecast a single value for each series, the targets are column vectors (e.g.,\ny_train has a shape of [7000, 1]).\nBaseline Metrics\nBefore we start using RNNs, it is often a good idea to have a few baseline\nmetrics, or else we may end up thinking our model works great when in\nfact it is doing worse than basic models. For example, the simplest\napproach is to predict the last value in each series. This is called naive\nforecasting, and it is sometimes surprisingly difficult to outperform. In\nthis case, it gives us a mean squared error of about 0.020:\n>>> y_pred = X_valid[:, -1] \n>>> np.mean(keras.losses.mean_squared_error(y_valid, y_pred)) \n0.020211367\nth\nth\nAnother simple approach is to use a fully connected network. Since it\nexpects a flat list of features for each input, we need to add a Flatten\nlayer. Let’s just use a simple Linear Regression model so that each\nprediction will be a linear combination of the values in the time series:\nmodel = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[50, 1]), \n keras.layers.Dense(1) \n])\nIf we compile this model using the MSE loss and the default Adam\noptimizer, then fit it on the training set for 20 epochs and evaluate it on\nthe validation set, we get an MSE of about 0.004. That’s much better than\nthe naive approach!\nImplementing a Simple RNN\nLet’s see if we can beat that with a simple RNN:\nmodel = keras.models.Sequential([ \n keras.layers.SimpleRNN(1, input_shape=[None, 1]) \n])\nThat’s really the simplest RNN you can build. It just contains a single\nlayer, with a single neuron, as we saw in Figure 15-1. We do not need to\nspecify the length of the input sequences (unlike in the previous model),\nsince a recurrent neural network can process any number of time steps\n(this is why we set the first input dimension to None). By default, the\nSimpleRNN layer uses the hyperbolic tangent activation function. It works\nexactly as we saw earlier: the initial state h\n is set to 0, and it is passed\nto a single recurrent neuron, along with the value of the first time step,\nx\n. The neuron computes a weighted sum of these values and applies the\nhyperbolic tangent activation function to the result, and this gives the first\noutput, y . In a simple RNN, this output is also the new state h . This new\nstate is passed to the same recurrent neuron along with the next input\nvalue, x\n, and the process is repeated until the last time step. Then the\n(init)\n(0)\n0\n0\n(1)\nlayer just outputs the last value, y\n. All of this is performed\nsimultaneously for every time series.\nNOTE\nBy default, recurrent layers in Keras only return the final output. To make them\nreturn one output per time step, you must set return_sequences=True, as we will\nsee.\nIf you compile, fit, and evaluate this model (just like earlier, we train for\n20 epochs using Adam), you will find that its MSE reaches only 0.014, so\nit is better than the naive approach but it does not beat a simple linear\nmodel. Note that for each neuron, a linear model has one parameter per\ninput and per time step, plus a bias term (in the simple linear model we\nused, that’s a total of 51 parameters). In contrast, for each recurrent neuron\nin a simple RNN, there is just one parameter per input and per hidden state\ndimension (in a simple RNN, that’s just the number of recurrent neurons in\nthe layer), plus a bias term. In this simple RNN, that’s a total of just three\nparameters.\n49\nTREND AND SEASONALITY\nThere are many other models to forecast time series, such as weighted\nmoving average models or autoregressive integrated moving average\n(ARIMA) models. Some of them require you to first remove the trend\nand seasonality. For example, if you are studying the number of active\nusers on your website, and it is growing by 10% every month, you\nwould have to remove this trend from the time series. Once the model\nis trained and starts making predictions, you would have to add the\ntrend back to get the final predictions. Similarly, if you are trying to\npredict the amount of sunscreen lotion sold every month, you will\nprobably observe strong seasonality: since it sells well every summer,\na similar pattern will be repeated every year. You would have to\nremove this seasonality from the time series, for example by\ncomputing the difference between the value at each time step and the\nvalue one year earlier (this technique is called differencing). Again,\nafter the model is trained and makes predictions, you would have to\nadd the seasonal pattern back to get the final predictions.\nWhen using RNNs, it is generally not necessary to do all this, but it\nmay improve performance in some cases, since the model will not\nhave to learn the trend or the seasonality.\nApparently our simple RNN was too simple to get good performance. So\nlet’s try to add more recurrent layers!\nDeep RNNs\nIt is quite common to stack multiple layers of cells, as shown in Figure 15-\n7. This gives you a deep RNN.\nFigure 15-7. Deep RNN (left) unrolled through time (right)\nImplementing a deep RNN with tf.keras is quite simple: just stack\nrecurrent layers. In this example, we use three SimpleRNN layers (but we\ncould add any other type of recurrent layer, such as an LSTM layer or a GRU\nlayer, which we will discuss shortly):\nmodel = keras.models.Sequential([ \n keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, \n1]), \n keras.layers.SimpleRNN(20, return_sequences=True), \n keras.layers.SimpleRNN(1) \n])\nWARNING\nMake sure to set return_sequences=True for all recurrent layers (except the last\none, if you only care about the last output). If you don’t, they will output a 2D array\n(containing only the output of the last time step) instead of a 3D array (containing\noutputs for all time steps), and the next recurrent layer will complain that you are not\nfeeding it sequences in the expected 3D format.\nIf you compile, fit, and evaluate this model, you will find that it reaches an\nMSE of 0.003. We finally managed to beat the linear model!\nNote that the last layer is not ideal: it must have a single unit because we\nwant to forecast a univariate time series, and this means we must have a\nsingle output value per time step. However, having a single unit means\nthat the hidden state is just a single number. That’s really not much, and\nit’s probably not that useful; presumably, the RNN will mostly use the\nhidden states of the other recurrent layers to carry over all the information\nit needs from time step to time step, and it will not use the final layer’s\nhidden state very much. Moreover, since a SimpleRNN layer uses the tanh\nactivation function by default, the predicted values must lie within the\nrange –1 to 1. But what if you want to use another activation function? For\nboth these reasons, it might be preferable to replace the output layer with a\nDense layer: it would run slightly faster, the accuracy would be roughly\nthe same, and it would allow us to choose any output activation function\nwe want. If you make this change, also make sure to remove\nreturn_sequences=True from the second (now last) recurrent layer:\nmodel = keras.models.Sequential([ \n keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, \n1]), \n keras.layers.SimpleRNN(20), \n keras.layers.Dense(1) \n])\nIf you train this model, you will see that it converges faster and performs\njust as well. Plus, you could change the output activation function if you\nwanted.\nForecasting Several Time Steps Ahead\nSo far we have only predicted the value at the next time step, but we could\njust as easily have predicted the value several steps ahead by changing the\ntargets appropriately (e.g., to predict 10 steps ahead, just change the\ntargets to be the value 10 steps ahead instead of 1 step ahead). But what if\nwe want to predict the next 10 values?\nThe first option is to use the model we already trained, make it predict the\nnext value, then add that value to the inputs (acting as if this predicted\nvalue had actually occurred), and use the model again to predict the\nfollowing value, and so on, as in the following code:\nseries = generate_time_series(1, n_steps + 10) \nX_new, Y_new = series[:, :n_steps], series[:, n_steps:] \nX = X_new \nfor step_ahead in range(10): \n y_pred_one = model.predict(X[:, step_ahead:])[:, np.newaxis, :] \n X = np.concatenate([X, y_pred_one], axis=1) \n \nY_pred = X[:, n_steps:]\nAs you might expect, the prediction for the next step will usually be more\naccurate than the predictions for later time steps, since the errors might\naccumulate (as you can see in Figure 15-8). If you evaluate this approach\non the validation set, you will find an MSE of about 0.029. This is much\nhigher than the previous models, but it’s also a much harder task, so the\ncomparison doesn’t mean much. It’s much more meaningful to compare\nthis performance with naive predictions (just forecasting that the time\nseries will remain constant for 10 time steps) or with a simple linear\nmodel. The naive approach is terrible (it gives an MSE of about 0.223),\nbut the linear model gives an MSE of about 0.0188: it’s much better than\nusing our RNN to forecast the future one step at a time, and also much\nfaster to train and run. Still, if you only want to forecast a few time steps\nahead, on more complex tasks, this approach may work well.\nFigure 15-8. Forecasting 10 steps ahead, 1 step at a time\nThe second option is to train an RNN to predict all 10 next values at once.\nWe can still use a sequence-to-vector model, but it will output 10 values\ninstead of 1. However, we first need to change the targets to be vectors\ncontaining the next 10 values:\nseries = generate_time_series(10000, n_steps + 10) \nX_train, Y_train = series[:7000, :n_steps], series[:7000, -10:, 0] \nX_valid, Y_valid = series[7000:9000, :n_steps], series[7000:9000, -10:, \n0] \nX_test, Y_test = series[9000:, :n_steps], series[9000:, -10:, 0]\nNow we just need the output layer to have 10 units instead of 1:\nmodel = keras.models.Sequential([ \n keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, \n1]), \n keras.layers.SimpleRNN(20), \n keras.layers.Dense(10) \n])\nAfter training this model, you can predict the next 10 values at once very\neasily:\nY_pred = model.predict(X_new)\nThis model works nicely: the MSE for the next 10 time steps is about\n0.008. That’s much better than the linear model. But we can still do better:\nindeed, instead of training the model to forecast the next 10 values only at\nthe very last time step, we can train it to forecast the next 10 values at\neach and every time step. In other words, we can turn this sequence-to-\nvector RNN into a sequence-to-sequence RNN. The advantage of this\ntechnique is that the loss will contain a term for the output of the RNN at\neach and every time step, not just the output at the last time step. This\nmeans there will be many more error gradients flowing through the model,\nand they won’t have to flow only through time; they will also flow from\nthe output of each time step. This will both stabilize and speed up training.\nTo be clear, at time step 0 the model will output a vector containing the\nforecasts for time steps 1 to 10, then at time step 1 the model will forecast\ntime steps 2 to 11, and so on. So each target must be a sequence of the\nsame length as the input sequence, containing a 10-dimensional vector at\neach step. Let’s prepare these target sequences:\nY = np.empty((10000, n_steps, 10)) # each target is a sequence of 10D \nvectors \nfor step_ahead in range(1, 10 + 1): \n Y[:, :, step_ahead - 1] = series[:, step_ahead:step_ahead + n_steps, \n0] \nY_train = Y[:7000] \nY_valid = Y[7000:9000] \nY_test = Y[9000:]\nNOTE\nIt may be surprising that the targets will contain values that appear in the inputs\n(there is a lot of overlap between X_train and Y_train). Isn’t that cheating?\nFortunately, not at all: at each time step, the model only knows about past time steps,\nso it cannot look ahead. It is said to be a causal model.\nTo turn the model into a sequence-to-sequence model, we must set\nreturn_sequences=True in all recurrent layers (even the last one), and\nwe must apply the output Dense layer at every time step. Keras offers a\nTimeDistributed layer for this very purpose: it wraps any layer (e.g., a\nDense layer) and applies it at every time step of its input sequence. It does\nthis efficiently, by reshaping the inputs so that each time step is treated as\na separate instance (i.e., it reshapes the inputs from [batch size, time steps,\ninput dimensions] to [batch size × time steps, input dimensions]; in this\nexample, the number of input dimensions is 20 because the previous\nSimpleRNN layer has 20 units), then it runs the Dense layer, and finally it\nreshapes the outputs back to sequences (i.e., it reshapes the outputs from\n[batch size × time steps, output dimensions] to [batch size, time steps,\noutput dimensions]; in this example the number of output dimensions is\n10, since the Dense layer has 10 units). Here is the updated model:\nmodel = keras.models.Sequential([ \n keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, \n1]), \n keras.layers.SimpleRNN(20, return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(10)) \n])\nThe Dense layer actually supports sequences as inputs (and even higher-\ndimensional inputs): it handles them just like\nTimeDistributed(Dense(…)), meaning it is applied to the last input\ndimension only (independently across all time steps). Thus, we could\nreplace the last layer with just Dense(10). For the sake of clarity,\nhowever, we will keep using TimeDistributed(Dense(10)) because it\n2\nmakes it clear that the Dense layer is applied independently at each time\nstep and that the model will output a sequence, not just a single vector.\nAll outputs are needed during training, but only the output at the last time\nstep is useful for predictions and for evaluation. So although we will rely\non the MSE over all the outputs for training, we will use a custom metric\nfor evaluation, to only compute the MSE over the output at the last time\nstep:\ndef last_time_step_mse(Y_true, Y_pred): \n return keras.metrics.mean_squared_error(Y_true[:, -1], Y_pred[:, -1]) \n \noptimizer = keras.optimizers.Adam(lr=0.01) \nmodel.compile(loss=\"mse\", optimizer=optimizer, metrics=\n[last_time_step_mse])\nWe get a validation MSE of about 0.006, which is 25% better than the\nprevious model. You can combine this approach with the first one: just\npredict the next 10 values using this RNN, then concatenate these values to\nthe input time series and use the model again to predict the next 10 values,\nand repeat the process as many times as needed. With this approach, you\ncan generate arbitrarily long sequences. It may not be very accurate for\nlong-term predictions, but it may be just fine if your goal is to generate\noriginal music or text, as we will see in Chapter 16.\nTIP\nWhen forecasting time series, it is often useful to have some error bars along with\nyour predictions. For this, an efficient technique is MC Dropout, introduced in\nChapter 11: add an MC Dropout layer within each memory cell, dropping part of the\ninputs and hidden states. After training, to forecast a new time series, use the model\nmany times and compute the mean and standard deviation of the predictions at each\ntime step.\nSimple RNNs can be quite good at forecasting time series or handling\nother kinds of sequences, but they do not perform as well on long time\nseries or sequences. Let’s discuss why and see what we can do about it.\nHandling Long Sequences\nTo train an RNN on long sequences, we must run it over many time steps,\nmaking the unrolled RNN a very deep network. Just like any deep neural\nnetwork it may suffer from the unstable gradients problem, discussed in\nChapter 11: it may take forever to train, or training may be unstable.\nMoreover, when an RNN processes a long sequence, it will gradually\nforget the first inputs in the sequence. Let’s look at both these problems,\nstarting with the unstable gradients problem.\nFighting the Unstable Gradients Problem\nMany of the tricks we used in deep nets to alleviate the unstable gradients\nproblem can also be used for RNNs: good parameter initialization, faster\noptimizers, dropout, and so on. However, nonsaturating activation\nfunctions (e.g., ReLU) may not help as much here; in fact, they may\nactually lead the RNN to be even more unstable during training. Why?\nWell, suppose Gradient Descent updates the weights in a way that\nincreases the outputs slightly at the first time step. Because the same\nweights are used at every time step, the outputs at the second time step\nmay also be slightly increased, and those at the third, and so on until the\noutputs explode—and a nonsaturating activation function does not prevent\nthat. You can reduce this risk by using a smaller learning rate, but you can\nalso simply use a saturating activation function like the hyperbolic tangent\n(this explains why it is the default). In much the same way, the gradients\nthemselves can explode. If you notice that training is unstable, you may\nwant to monitor the size of the gradients (e.g., using TensorBoard) and\nperhaps use Gradient Clipping.\nMoreover, Batch Normalization cannot be used as efficiently with RNNs\nas with deep feedforward nets. In fact, you cannot use it between time\nsteps, only between recurrent layers. To be more precise, it is technically\npossible to add a BN layer to a memory cell (as we will see shortly) so that\nit will be applied at each time step (both on the inputs for that time step\nand on the hidden state from the previous step). However, the same BN\nlayer will be used at each time step, with the same parameters, regardless\nof the actual scale and offset of the inputs and hidden state. In practice,\nthis does not yield good results, as was demonstrated by César Laurent et\nal. in a 2015 paper: the authors found that BN was slightly beneficial only\nwhen it was applied to the inputs, not to the hidden states. In other words,\nit was slightly better than nothing when applied between recurrent layers\n(i.e., vertically in Figure 15-7), but not within recurrent layers (i.e.,\nhorizontally). In Keras this can be done simply by adding a\nBatchNormalization layer before each recurrent layer, but don’t expect\ntoo much from it.\nAnother form of normalization often works better with RNNs: Layer\nNormalization. This idea was introduced by Jimmy Lei Ba et al. in a 2016\npaper: it is very similar to Batch Normalization, but instead of\nnormalizing across the batch dimension, it normalizes across the features\ndimension. One advantage is that it can compute the required statistics on\nthe fly, at each time step, independently for each instance. This also means\nthat it behaves the same way during training and testing (as opposed to\nBN), and it does not need to use exponential moving averages to estimate\nthe feature statistics across all instances in the training set. Like BN, Layer\nNormalization learns a scale and an offset parameter for each input. In an\nRNN, it is typically used right after the linear combination of the inputs\nand the hidden states.\nLet’s use tf.keras to implement Layer Normalization within a simple\nmemory cell. For this, we need to define a custom memory cell. It is just\nlike a regular layer, except its call() method takes two arguments: the\ninputs at the current time step and the hidden states from the previous\ntime step. Note that the states argument is a list containing one or more\ntensors. In the case of a simple RNN cell it contains a single tensor equal\nto the outputs of the previous time step, but other cells may have multiple\nstate tensors (e.g., an LSTMCell has a long-term state and a short-term\nstate, as we will see shortly). A cell must also have a state_size attribute\nand an output_size attribute. In a simple RNN, both are simply equal to\n3\n4\nthe number of units. The following code implements a custom memory\ncell which will behave like a SimpleRNNCell, except it will also apply\nLayer Normalization at each time step:\nclass LNSimpleRNNCell(keras.layers.Layer): \n def __init__(self, units, activation=\"tanh\", **kwargs): \n super().__init__(**kwargs) \n self.state_size = units \n self.output_size = units \n self.simple_rnn_cell = keras.layers.SimpleRNNCell(units, \n \nactivation=None) \n self.layer_norm = keras.layers.LayerNormalization() \n self.activation = keras.activations.get(activation) \n def call(self, inputs, states): \n outputs, new_states = self.simple_rnn_cell(inputs, states) \n norm_outputs = self.activation(self.layer_norm(outputs)) \n return norm_outputs, [norm_outputs]\nThe code is quite straightforward. Our LNSimpleRNNCell class inherits\nfrom the keras.layers.Layer class, just like any custom layer. The\nconstructor takes the number of units and the desired activation function,\nand it sets the state_size and output_size attributes, then creates a\nSimpleRNNCell with no activation function (because we want to perform\nLayer Normalization after the linear operation but before the activation\nfunction). Then the constructor creates the LayerNormalization layer,\nand finally it fetches the desired activation function. The call() method\nstarts by applying the simple RNN cell, which computes a linear\ncombination of the current inputs and the previous hidden states, and it\nreturns the result twice (indeed, in a SimpleRNNCell, the outputs are just\nequal to the hidden states: in other words, new_states[0] is equal to\noutputs, so we can safely ignore new_states in the rest of the call()\nmethod). Next, the call() method applies Layer Normalization, followed\nby the activation function. Finally, it returns the outputs twice (once as the\noutputs, and once as the new hidden states). To use this custom cell, all we\nneed to do is create a keras.layers.RNN layer, passing it a cell instance:\n5\nmodel = keras.models.Sequential([ \n keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True, \n input_shape=[None, 1]), \n keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(10)) \n])\nSimilarly, you could create a custom cell to apply dropout between each\ntime step. But there’s a simpler way: all recurrent layers (except for\nkeras.layers.RNN) and all cells provided by Keras have a dropout\nhyperparameter and a recurrent_dropout hyperparameter: the former\ndefines the dropout rate to apply to the inputs (at each time step), and the\nlatter defines the dropout rate for the hidden states (also at each time step).\nNo need to create a custom cell to apply dropout at each time step in an\nRNN.\nWith these techniques, you can alleviate the unstable gradients problem\nand train an RNN much more efficiently. Now let’s look at how to deal\nwith the short-term memory problem.\nTackling the Short-Term Memory Problem\nDue to the transformations that the data goes through when traversing an\nRNN, some information is lost at each time step. After a while, the RNN’s\nstate contains virtually no trace of the first inputs. This can be a\nshowstopper. Imagine Dory the fish trying to translate a long sentence; by\nthe time she’s finished reading it, she has no clue how it started. To tackle\nthis problem, various types of cells with long-term memory have been\nintroduced. They have proven so successful that the basic cells are not\nused much anymore. Let’s first look at the most popular of these long-term\nmemory cells: the LSTM cell.\nLSTM cells\nThe Long Short-Term Memory (LSTM) cell was proposed in 1997 by\nSepp Hochreiter and Jürgen Schmidhuber and gradually improved over the\nyears by several researchers, such as Alex Graves, Haşim Sak, and\n6\n7\n8\nWojciech Zaremba. If you consider the LSTM cell as a black box, it can\nbe used very much like a basic cell, except it will perform much better;\ntraining will converge faster, and it will detect long-term dependencies in\nthe data. In Keras, you can simply use the LSTM layer instead of the\nSimpleRNN layer:\nmodel = keras.models.Sequential([ \n keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]), \n keras.layers.LSTM(20, return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(10)) \n])\nAlternatively, you could use the general-purpose keras.layers.RNN layer,\ngiving it an LSTMCell as an argument:\nmodel = keras.models.Sequential([ \n keras.layers.RNN(keras.layers.LSTMCell(20), return_sequences=True, \n input_shape=[None, 1]), \n keras.layers.RNN(keras.layers.LSTMCell(20), return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(10)) \n])\nHowever, the LSTM layer uses an optimized implementation when running\non a GPU (see Chapter 19), so in general it is preferable to use it (the RNN\nlayer is mostly useful when you define custom cells, as we did earlier).\nSo how does an LSTM cell work? Its architecture is shown in Figure 15-9.\nIf you don’t look at what’s inside the box, the LSTM cell looks exactly\nlike a regular cell, except that its state is split into two vectors: h and c\n(“c” stands for “cell”). You can think of h as the short-term state and c\nas the long-term state.\n9\n(t)\n(t)\n(t)\n(t)\nFigure 15-9. LSTM cell\nNow let’s open the box! The key idea is that the network can learn what to\nstore in the long-term state, what to throw away, and what to read from it.\nAs the long-term state c\n traverses the network from left to right, you\ncan see that it first goes through a forget gate, dropping some memories,\nand then it adds some new memories via the addition operation (which\nadds the memories that were selected by an input gate). The result c is\nsent straight out, without any further transformation. So, at each time step,\nsome memories are dropped and some memories are added. Moreover,\nafter the addition operation, the long-term state is copied and passed\nthrough the tanh function, and then the result is filtered by the output gate.\nThis produces the short-term state h (which is equal to the cell’s output\nfor this time step, y ). Now let’s look at where new memories come from\nand how the gates work.\nFirst, the current input vector x and the previous short-term state h\nare fed to four different fully connected layers. They all serve a different\npurpose:\n(t–1)\n(t)\n(t)\n(t)\n(t)\n(t–1)\nThe main layer is the one that outputs g . It has the usual role of\nanalyzing the current inputs x and the previous (short-term)\nstate h\n. In a basic cell, there is nothing other than this layer,\nand its output goes straight out to y and h . In contrast, in an\nLSTM cell this layer’s output does not go straight out, but instead\nits most important parts are stored in the long-term state (and the\nrest is dropped).\nThe three other layers are gate controllers. Since they use the\nlogistic activation function, their outputs range from 0 to 1. As\nyou can see, their outputs are fed to element-wise multiplication\noperations, so if they output 0s they close the gate, and if they\noutput 1s they open it. Specifically:\nThe forget gate (controlled by f ) controls which parts of\nthe long-term state should be erased.\nThe input gate (controlled by i ) controls which parts of\ng should be added to the long-term state.\nFinally, the output gate (controlled by o ) controls which\nparts of the long-term state should be read and output at\nthis time step, both to h and to y .\nIn short, an LSTM cell can learn to recognize an important input (that’s\nthe role of the input gate), store it in the long-term state, preserve it for as\nlong as it is needed (that’s the role of the forget gate), and extract it\nwhenever it is needed. This explains why these cells have been amazingly\nsuccessful at capturing long-term patterns in time series, long texts, audio\nrecordings, and more.\nEquation 15-3 summarizes how to compute the cell’s long-term state, its\nshort-term state, and its output at each time step for a single instance (the\nequations for a whole mini-batch are very similar).\nEquation 15-3. LSTM computations\n(t)\n(t)\n(t–1)\n(t)\n(t)\n(t)\n(t)\n(t)\n(t)\n(t)\n(t)\ni(t)\n= σ(Wxi\n⊺x(t) + Whi\n⊺h(t−1) + bi)\nf(t)\n= σ(Wxf\n⊺x(t) + Whf\n⊺h(t−1) + bf)\no(t)\n= σ(Wxo\n⊺x(t) + Who\n⊺h(t−1) + bo)\ng(t)\n= tanh(Wxg\n⊺x(t) + Whg\n⊺h(t−1) + bg)\nc(t)\n= f(t) ⊗c(t−1) + i(t) ⊗g(t)\ny(t)\n= h(t) = o(t) ⊗tanh (c(t))\nIn this equation:\nW , W , W , W are the weight matrices of each of the four\nlayers for their connection to the input vector x .\nW , W , W\n, and W\n are the weight matrices of each of the\nfour layers for their connection to the previous short-term state\nh\n.\nb , b , b , and b are the bias terms for each of the four layers.\nNote that TensorFlow initializes b to a vector full of 1s instead of\n0s. This prevents forgetting everything at the beginning of\ntraining.\nPeephole connections\nIn a regular LSTM cell, the gate controllers can look only at the input x\nand the previous short-term state h\n. It may be a good idea to give them\na bit more context by letting them peek at the long-term state as well. This\nidea was proposed by Felix Gers and Jürgen Schmidhuber in 2000.\n They\nproposed an LSTM variant with extra connections called peephole\nconnections: the previous long-term state c\n is added as an input to the\ncontrollers of the forget gate and the input gate, and the current long-term\nstate c is added as input to the controller of the output gate. This often\nimproves performance, but not always, and there is no clear pattern for\nwhich tasks are better off with or without them: you will have to try it on\nyour task and see if it helps.\nxi\nxf\nxo\nxg\n(t)\nhi\nhf\nho\nhg\n(t–1)\ni\nf\no\ng\nf\n(t)\n(t–1)\n10\n(t–1)\n(t)\nIn Keras, the LSTM layer is based on the keras.layers.LSTMCell cell,\nwhich does not support peepholes. The experimental\ntf.keras.experimental.PeepholeLSTMCell does, however, so you can\ncreate a keras.layers.RNN layer and pass a PeepholeLSTMCell to its\nconstructor.\nThere are many other variants of the LSTM cell. One particularly popular\nvariant is the GRU cell, which we will look at now.\nGRU cells\nThe Gated Recurrent Unit (GRU) cell (see Figure 15-10) was proposed by\nKyunghyun Cho et al. in a 2014 paper that also introduced the Encoder–\nDecoder network we discussed earlier.\nFigure 15-10. GRU cell\nThe GRU cell is a simplified version of the LSTM cell, and it seems to\nperform just as well\n (which explains its growing popularity). These are\nthe main simplifications:\n11\n12\nBoth state vectors are merged into a single vector h .\nA single gate controller z controls both the forget gate and the\ninput gate. If the gate controller outputs a 1, the forget gate is\nopen (= 1) and the input gate is closed (1 – 1 = 0). If it outputs a 0,\nthe opposite happens. In other words, whenever a memory must\nbe stored, the location where it will be stored is erased first. This\nis actually a frequent variant to the LSTM cell in and of itself.\nThere is no output gate; the full state vector is output at every\ntime step. However, there is a new gate controller r that controls\nwhich part of the previous state will be shown to the main layer\n(g ).\nEquation 15-4 summarizes how to compute the cell’s state at each time\nstep for a single instance.\nEquation 15-4. GRU computations\nz(t)\n= σ(Wxz\n⊺x(t) + Whz\n⊺h(t−1) + bz)\nr(t)\n= σ(Wxr\n⊺x(t) + Whr\n⊺h(t−1) + br)\ng(t)\n= tanh(Wxg\n⊺x(t) + Whg\n⊺(r(t) ⊗h(t−1)) + bg)\nh(t)\n= z(t) ⊗h(t−1) + (1 −z(t)) ⊗g(t)\nKeras provides a keras.layers.GRU layer (based on the\nkeras.layers.GRUCell memory cell); using it is just a matter of\nreplacing SimpleRNN or LSTM with GRU.\nLSTM and GRU cells are one of the main reasons behind the success of\nRNNs. Yet while they can tackle much longer sequences than simple\nRNNs, they still have a fairly limited short-term memory, and they have a\nhard time learning long-term patterns in sequences of 100 time steps or\nmore, such as audio samples, long time series, or long sentences. One way\nto solve this is to shorten the input sequences, for example using 1D\nconvolutional layers.\n(t)\n(t)\n(t)\n(t)\nUsing 1D convolutional layers to process sequences\nIn Chapter 14, we saw that a 2D convolutional layer works by sliding\nseveral fairly small kernels (or filters) across an image, producing\nmultiple 2D feature maps (one per kernel). Similarly, a 1D convolutional\nlayer slides several kernels across a sequence, producing a 1D feature map\nper kernel. Each kernel will learn to detect a single very short sequential\npattern (no longer than the kernel size). If you use 10 kernels, then the\nlayer’s output will be composed of 10 1-dimensional sequences (all of the\nsame length), or equivalently you can view this output as a single 10-\ndimensional sequence. This means that you can build a neural network\ncomposed of a mix of recurrent layers and 1D convolutional layers (or\neven 1D pooling layers). If you use a 1D convolutional layer with a stride\nof 1 and \"same\" padding, then the output sequence will have the same\nlength as the input sequence. But if you use \"valid\" padding or a stride\ngreater than 1, then the output sequence will be shorter than the input\nsequence, so make sure you adjust the targets accordingly. For example,\nthe following model is the same as earlier, except it starts with a 1D\nconvolutional layer that downsamples the input sequence by a factor of 2,\nusing a stride of 2. The kernel size is larger than the stride, so all inputs\nwill be used to compute the layer’s output, and therefore the model can\nlearn to preserve the useful information, dropping only the unimportant\ndetails. By shortening the sequences, the convolutional layer may help the\nGRU layers detect longer patterns. Note that we must also crop off the first\nthree time steps in the targets (since the kernel’s size is 4, the first output\nof the convolutional layer will be based on the input time steps 0 to 3), and\ndownsample the targets by a factor of 2:\nmodel = keras.models.Sequential([ \n keras.layers.Conv1D(filters=20, kernel_size=4, strides=2, \npadding=\"valid\", \n input_shape=[None, 1]), \n keras.layers.GRU(20, return_sequences=True), \n keras.layers.GRU(20, return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(10)) \n]) \n \nmodel.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse]) \nhistory = model.fit(X_train, Y_train[:, 3::2], epochs=20, \n validation_data=(X_valid, Y_valid[:, 3::2]))\nIf you train and evaluate this model, you will find that it is the best model\nso far. The convolutional layer really helps. In fact, it is actually possible\nto use only 1D convolutional layers and drop the recurrent layers entirely!\nWaveNet\nIn a 2016 paper,\n Aaron van den Oord and other DeepMind researchers\nintroduced an architecture called WaveNet. They stacked 1D convolutional\nlayers, doubling the dilation rate (how spread apart each neuron’s inputs\nare) at every layer: the first convolutional layer gets a glimpse of just two\ntime steps at a time, while the next one sees four time steps (its receptive\nfield is four time steps long), the next one sees eight time steps, and so on\n(see Figure 15-11). This way, the lower layers learn short-term patterns,\nwhile the higher layers learn long-term patterns. Thanks to the doubling\ndilation rate, the network can process extremely large sequences very\nefficiently.\nFigure 15-11. WaveNet architecture\nIn the WaveNet paper, the authors actually stacked 10 convolutional layers\nwith dilation rates of 1, 2, 4, 8, …, 256, 512, then they stacked another\ngroup of 10 identical layers (also with dilation rates 1, 2, 4, 8, …, 256,\n512), then again another identical group of 10 layers. They justified this\n13\narchitecture by pointing out that a single stack of 10 convolutional layers\nwith these dilation rates will act like a super-efficient convolutional layer\nwith a kernel of size 1,024 (except way faster, more powerful, and using\nsignificantly fewer parameters), which is why they stacked 3 such blocks.\nThey also left-padded the input sequences with a number of zeros equal to\nthe dilation rate before every layer, to preserve the same sequence length\nthroughout the network. Here is how to implement a simplified WaveNet\nto tackle the same sequences as earlier:\nmodel = keras.models.Sequential() \nmodel.add(keras.layers.InputLayer(input_shape=[None, 1])) \nfor rate in (1, 2, 4, 8) * 2: \n model.add(keras.layers.Conv1D(filters=20, kernel_size=2, \npadding=\"causal\", \n activation=\"relu\", dilation_rate=rate)) \nmodel.add(keras.layers.Conv1D(filters=10, kernel_size=1)) \nmodel.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse]) \nhistory = model.fit(X_train, Y_train, epochs=20, \n validation_data=(X_valid, Y_valid))\nThis Sequential model starts with an explicit input layer (this is simpler\nthan trying to set input_shape only on the first layer), then continues with\na 1D convolutional layer using \"causal\" padding: this ensures that the\nconvolutional layer does not peek into the future when making predictions\n(it is equivalent to padding the inputs with the right amount of zeros on the\nleft and using \"valid\" padding). We then add similar pairs of layers using\ngrowing dilation rates: 1, 2, 4, 8, and again 1, 2, 4, 8. Finally, we add the\noutput layer: a convolutional layer with 10 filters of size 1 and without any\nactivation function. Thanks to the padding layers, every convolutional\nlayer outputs a sequence of the same length as the input sequences, so the\ntargets we use during training can be the full sequences: no need to crop\nthem or downsample them.\nThe last two models offer the best performance so far in forecasting our\ntime series! In the WaveNet paper, the authors achieved state-of-the-art\nperformance on various audio tasks (hence the name of the architecture),\nincluding text-to-speech tasks, producing incredibly realistic voices across\n14\nseveral languages. They also used the model to generate music, one audio\nsample at a time. This feat is all the more impressive when you realize\nthat a single second of audio can contain tens of thousands of time steps—\neven LSTMs and GRUs cannot handle such long sequences.\nIn Chapter 16, we will continue to explore RNNs, and we will see how\nthey can tackle various NLP tasks.\nExercises\n1. Can you think of a few applications for a sequence-to-sequence\nRNN? What about a sequence-to-vector RNN, and a vector-to-\nsequence RNN?\n2. How many dimensions must the inputs of an RNN layer have?\nWhat does each dimension represent? What about its outputs?\n3. If you want to build a deep sequence-to-sequence RNN, which\nRNN layers should have return_sequences=True? What about a\nsequence-to-vector RNN?\n4. Suppose you have a daily univariate time series, and you want to\nforecast the next seven days. Which RNN architecture should you\nuse?\n5. What are the main difficulties when training RNNs? How can you\nhandle them?\n6. Can you sketch the LSTM cell’s architecture?\n7. Why would you want to use 1D convolutional layers in an RNN?\n8. Which neural network architecture could you use to classify\nvideos?\n9. Train a classification model for the SketchRNN dataset, available\nin TensorFlow Datasets.\n10. Download the Bach chorales dataset and unzip it. It is composed\nof 382 chorales composed by Johann Sebastian Bach. Each\nchorale is 100 to 640 time steps long, and each time step contains\n4 integers, where each integer corresponds to a note’s index on a\npiano (except for the value 0, which means that no note is played).\nTrain a model—recurrent, convolutional, or both—that can\npredict the next time step (four notes), given a sequence of time\nsteps from a chorale. Then use this model to generate Bach-like\nmusic, one note at a time: you can do this by giving the model the\nstart of a chorale and asking it to predict the next time step, then\nappending these time steps to the input sequence and asking the\nmodel for the next note, and so on. Also make sure to check out\nGoogle’s Coconet model, which was used for a nice Google\ndoodle about Bach.\nSolutions to these exercises are available in Appendix A.\n1 Note that many researchers prefer to use the hyperbolic tangent (tanh) activation function\nin RNNs rather than the ReLU activation function. For example, take a look at Vu Pham et\nal.’s 2013 paper “Dropout Improves Recurrent Neural Networks for Handwriting\nRecognition”. ReLU-based RNNs are also possible, as shown in Quoc V. Le et al.’s 2015\npaper “A Simple Way to Initialize Recurrent Networks of Rectified Linear Units”.\n2 Note that a TimeDistributed(Dense(n)) layer is equivalent to a Conv1D(n,\nfilter_size=1) layer.\n3 César Laurent et al., “Batch Normalized Recurrent Neural Networks,” Proceedings of the\nIEEE International Conference on Acoustics, Speech, and Signal Processing (2016): 2657–\n2661.\n4 Jimmy Lei Ba et al., “Layer Normalization,” arXiv preprint arXiv:1607.06450 (2016).\n5 It would have been simpler to inherit from SimpleRNNCell instead so that we wouldn’t\nhave to create an internal SimpleRNNCell or handle the state_size and output_size\nattributes, but the goal here was to show how to create a custom cell from scratch.\n6 A character from the animated movies Finding Nemo and Finding Dory who has short-\nterm memory loss.\n7 Sepp Hochreiter and Jürgen Schmidhuber, “Long Short-Term Memory,” Neural\nComputation 9, no. 8 (1997): 1735–1780.\n8 Haşim Sak et al., “Long Short-Term Memory Based Recurrent Neural Network\nArchitectures for Large Vocabulary Speech Recognition,” arXiv preprint arXiv:1402.1128\n(2014).\n9 Wojciech Zaremba et al., “Recurrent Neural Network Regularization,” arXiv preprint\narXiv:1409.2329 (2014).\n10 F. A. Gers and J. Schmidhuber, “Recurrent Nets That Time and Count,” Proceedings of\nthe IEEE-INNS-ENNS International Joint Conference on Neural Networks (2000): 189–\n194.\n11 Kyunghyun Cho et al., “Learning Phrase Representations Using RNN Encoder-Decoder\nfor Statistical Machine Translation,” Proceedings of the 2014 Conference on Empirical\nMethods in Natural Language Processing (2014): 1724–1734.\n12 A 2015 paper by Klaus Greff et al., “LSTM: A Search Space Odyssey”, seems to show\nthat all LSTM variants perform roughly the same.\n13 Aaron van den Oord et al., “WaveNet: A Generative Model for Raw Audio,” arXiv\npreprint arXiv:1609.03499 (2016).\n14 The complete WaveNet uses a few more tricks, such as skip connections like in a ResNet,\nand Gated Activation Units similar to those found in a GRU cell. Please see the notebook\nfor more details." + }, + { + "chapter": 16, + "text": "Chapter 16. Natural Language\nProcessing with RNNs and\nAttention\nWhen Alan Turing imagined his famous Turing test in 1950, his objective\nwas to evaluate a machine’s ability to match human intelligence. He could\nhave tested for many things, such as the ability to recognize cats in\npictures, play chess, compose music, or escape a maze, but, interestingly,\nhe chose a linguistic task. More specifically, he devised a chatbot capable\nof fooling its interlocutor into thinking it was human. This test does have\nits weaknesses: a set of hardcoded rules can fool unsuspecting or naive\nhumans (e.g., the machine could give vague predefined answers in\nresponse to some keywords; it could pretend that it is joking or drunk, to\nget a pass on its weirdest answers; or it could escape difficult questions by\nanswering them with its own questions), and many aspects of human\nintelligence are utterly ignored (e.g., the ability to interpret nonverbal\ncommunication such as facial expressions, or to learn a manual task). But\nthe test does highlight the fact that mastering language is arguably Homo\nsapiens’s greatest cognitive ability. Can we build a machine that can read\nand write natural language?\nA common approach for natural language tasks is to use recurrent neural\nnetworks. We will therefore continue to explore RNNs (introduced in\nChapter 15), starting with a character RNN, trained to predict the next\ncharacter in a sentence. This will allow us to generate some original text,\nand in the process we will see how to build a TensorFlow Dataset on a very\nlong sequence. We will first use a stateless RNN (which learns on random\nportions of text at each iteration, without any information on the rest of\nthe text), then we will build a stateful RNN (which preserves the hidden\nstate between training iterations and continues reading where it left off,\n1\n2\nallowing it to learn longer patterns). Next, we will build an RNN to\nperform sentiment analysis (e.g., reading movie reviews and extracting the\nrater’s feeling about the movie), this time treating sentences as sequences\nof words, rather than characters. Then we will show how RNNs can be\nused to build an Encoder–Decoder architecture capable of performing\nneural machine translation (NMT). For this, we will use the seq2seq API\nprovided by the TensorFlow Addons project.\nIn the second part of this chapter, we will look at attention mechanisms.\nAs their name suggests, these are neural network components that learn to\nselect the part of the inputs that the rest of the model should focus on at\neach time step. First we will see how to boost the performance of an RNN-\nbased Encoder–Decoder architecture using attention, then we will drop\nRNNs altogether and look at a very successful attention-only architecture\ncalled the Transformer. Finally, we will take a look at some of the most\nimportant advances in NLP in 2018 and 2019, including incredibly\npowerful language models such as GPT-2 and BERT, both based on\nTransformers.\nLet’s start with a simple and fun model that can write like Shakespeare\n(well, sort of).\nGenerating Shakespearean Text Using a\nCharacter RNN\nIn a famous 2015 blog post titled “The Unreasonable Effectiveness of\nRecurrent Neural Networks,” Andrej Karpathy showed how to train an\nRNN to predict the next character in a sentence. This Char-RNN can then\nbe used to generate novel text, one character at a time. Here is a small\nsample of the text generated by a Char-RNN model after it was trained on\nall of Shakespeare’s work:\nPANDARUS:\nAlas, I think he shall be come approached and the day\nWhen little srain would be attain’d into being never fed,\nAnd who is but a chain and subjects of his death,\nI should not sleep.\nNot exactly a masterpiece, but it is still impressive that the model was\nable to learn words, grammar, proper punctuation, and more, just by\nlearning to predict the next character in a sentence. Let’s look at how to\nbuild a Char-RNN, step by step, starting with the creation of the dataset.\nCreating the Training Dataset\nFirst, let’s download all of Shakespeare’s work, using Keras’s handy\nget_file() function and downloading the data from Andrej Karpathy’s\nChar-RNN project:\nshakespeare_url = \"https://homl.info/shakespeare\" # shortcut URL \nfilepath = keras.utils.get_file(\"shakespeare.txt\", shakespeare_url) \nwith open(filepath) as f: \n shakespeare_text = f.read()\nNext, we must encode every character as an integer. One option is to create\na custom preprocessing layer, as we did in Chapter 13. But in this case, it\nwill be simpler to use Keras’s Tokenizer class. First we need to fit a\ntokenizer to the text: it will find all the characters used in the text and map\neach of them to a different character ID, from 1 to the number of distinct\ncharacters (it does not start at 0, so we can use that value for masking, as\nwe will see later in this chapter):\ntokenizer = keras.preprocessing.text.Tokenizer(char_level=True) \ntokenizer.fit_on_texts([shakespeare_text])\nWe set char_level=True to get character-level encoding rather than the\ndefault word-level encoding. Note that this tokenizer converts the text to\nlowercase by default (but you can set lower=False if you do not want\nthat). Now the tokenizer can encode a sentence (or a list of sentences) to a\nlist of character IDs and back, and it tells us how many distinct characters\nthere are and the total number of characters in the text:\n>>> tokenizer.texts_to_sequences([\"First\"]) \n[[20, 6, 9, 8, 3]] \n>>> tokenizer.sequences_to_texts([[20, 6, 9, 8, 3]]) \n['f i r s t'] \n>>> max_id = len(tokenizer.word_index) # number of distinct characters \n>>> dataset_size = tokenizer.document_count # total number of characters\nLet’s encode the full text so each character is represented by its ID (we\nsubtract 1 to get IDs from 0 to 38, rather than from 1 to 39):\n[encoded] = np.array(tokenizer.texts_to_sequences([shakespeare_text])) - \n1\nBefore we continue, we need to split the dataset into a training set, a\nvalidation set, and a test set. We can’t just shuffle all the characters in the\ntext, so how do you split a sequential dataset?\nHow to Split a Sequential Dataset\nIt is very important to avoid any overlap between the training set, the\nvalidation set, and the test set. For example, we can take the first 90% of\nthe text for the training set, then the next 5% for the validation set, and the\nfinal 5% for the test set. It would also be a good idea to leave a gap\nbetween these sets to avoid the risk of a paragraph overlapping over two\nsets.\nWhen dealing with time series, you would in general split across time,: for\nexample, you might take the years 2000 to 2012 for the training set, the\nyears 2013 to 2015 for the validation set, and the years 2016 to 2018 for\nthe test set. However, in some cases you may be able to split along other\ndimensions, which will give you a longer time period to train on. For\nexample, if you have data about the financial health of 10,000 companies\nfrom 2000 to 2018, you might be able to split this data across the different\ncompanies. It’s very likely that many of these companies will be strongly\ncorrelated, though (e.g., whole economic sectors may go up or down\njointly), and if you have correlated companies across the training set and\nthe test set your test set will not be as useful, as its measure of the\ngeneralization error will be optimistically biased.\nSo, it is often safer to split across time—but this implicitly assumes that\nthe patterns the RNN can learn in the past (in the training set) will still\nexist in the future. In other words, we assume that the time series is\nstationary (at least in a wide sense). For many time series this\nassumption is reasonable (e.g., chemical reactions should be fine, since the\nlaws of chemistry don’t change every day), but for many others it is not\n(e.g., financial markets are notoriously not stationary since patterns\ndisappear as soon as traders spot them and start exploiting them). To make\nsure the time series is indeed sufficiently stationary, you can plot the\nmodel’s errors on the validation set across time: if the model performs\nmuch better on the first part of the validation set than on the last part, then\nthe time series may not be stationary enough, and you might be better off\ntraining the model on a shorter time span.\nIn short, splitting a time series into a training set, a validation set, and a\ntest set is not a trivial task, and how it’s done will depend strongly on the\ntask at hand.\nNow back to Shakespeare! Let’s take the first 90% of the text for the\ntraining set (keeping the rest for the validation set and the test set), and\ncreate a tf.data.Dataset that will return each character one by one from\nthis set:\ntrain_size = dataset_size * 90 // 100 \ndataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])\nChopping the Sequential Dataset into Multiple Windows\n3\nThe training set now consists of a single sequence of over a million\ncharacters, so we can’t just train the neural network directly on it: the\nRNN would be equivalent to a deep net with over a million layers, and we\nwould have a single (very long) instance to train it. Instead, we will use\nthe dataset’s window() method to convert this long sequence of characters\ninto many smaller windows of text. Every instance in the dataset will be a\nfairly short substring of the whole text, and the RNN will be unrolled only\nover the length of these substrings. This is called truncated\nbackpropagation through time. Let’s call the window() method to create a\ndataset of short text windows:\nn_steps = 100 \nwindow_length = n_steps + 1 # target = input shifted 1 character ahead \ndataset = dataset.window(window_length, shift=1, drop_remainder=True)\nTIP\nYou can try tuning n_steps: it is easier to train RNNs on shorter input sequences,\nbut of course the RNN will not be able to learn any pattern longer than n_steps, so\ndon’t make it too small.\nBy default, the window() method creates nonoverlapping windows, but to\nget the largest possible training set we use shift=1 so that the first\nwindow contains characters 0 to 100, the second contains characters 1 to\n101, and so on. To ensure that all windows are exactly 101 characters long\n(which will allow us to create batches without having to do any padding),\nwe set drop_remainder=True (otherwise the last 100 windows will\ncontain 100 characters, 99 characters, and so on down to 1 character).\nThe window() method creates a dataset that contains windows, each of\nwhich is also represented as a dataset. It’s a nested dataset, analogous to a\nlist of lists. This is useful when you want to transform each window by\ncalling its dataset methods (e.g., to shuffle them or batch them). However,\nwe cannot use a nested dataset directly for training, as our model will\nexpect tensors as input, not datasets. So, we must call the flat_map()\nmethod: it converts a nested dataset into a flat dataset (one that does not\ncontain datasets). For example, suppose {1, 2, 3} represents a dataset\ncontaining the sequence of tensors 1, 2, and 3. If you flatten the nested\ndataset {{1, 2}, {3, 4, 5, 6}}, you get back the flat dataset {1, 2, 3, 4, 5, 6}.\nMoreover, the flat_map() method takes a function as an argument, which\nallows you to transform each dataset in the nested dataset before\nflattening. For example, if you pass the function lambda ds:\nds.batch(2) to flat_map(), then it will transform the nested dataset {{1,\n2}, {3, 4, 5, 6}} into the flat dataset {[1, 2], [3, 4], [5, 6]}: it’s a dataset of\ntensors of size 2. With that in mind, we are ready to flatten our dataset:\ndataset = dataset.flat_map(lambda window: window.batch(window_length))\nNotice that we call batch(window_length) on each window: since all\nwindows have exactly that length, we will get a single tensor for each of\nthem. Now the dataset contains consecutive windows of 101 characters\neach. Since Gradient Descent works best when the instances in the training\nset are independent and identically distributed (see Chapter 4), we need to\nshuffle these windows. Then we can batch the windows and separate the\ninputs (the first 100 characters) from the target (the last character):\nbatch_size = 32 \ndataset = dataset.shuffle(10000).batch(batch_size) \ndataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\nFigure 16-1 summarizes the dataset preparation steps discussed so far\n(showing windows of length 11 rather than 101, and a batch size of 3\ninstead of 32).\nFigure 16-1. Preparing a dataset of shuffled windows\nAs discussed in Chapter 13, categorical input features should generally be\nencoded, usually as one-hot vectors or as embeddings. Here, we will\nencode each character using a one-hot vector because there are fairly few\ndistinct characters (only 39):\ndataset = dataset.map( \n lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), \nY_batch))\nFinally, we just need to add prefetching:\ndataset = dataset.prefetch(1)\nThat’s it! Preparing the dataset was the hardest part. Now let’s create the\nmodel.\nBuilding and Training the Char-RNN Model\nTo predict the next character based on the previous 100 characters, we can\nuse an RNN with 2 GRU layers of 128 units each and 20% dropout on both\nthe inputs (dropout) and the hidden states (recurrent_dropout). We can\ntweak these hyperparameters later, if needed. The output layer is a time-\ndistributed Dense layer like we saw in Chapter 15. This time this layer\nmust have 39 units (max_id) because there are 39 distinct characters in the\ntext, and we want to output a probability for each possible character (at\neach time step). The output probabilities should sum up to 1 at each time\nstep, so we apply the softmax activation function to the outputs of the\nDense layer. We can then compile this model, using the\n\"sparse_categorical_crossentropy\" loss and an Adam optimizer.\nFinally, we are ready to train the model for several epochs (this may take\nmany hours, depending on your hardware):\nmodel = keras.models.Sequential([ \n keras.layers.GRU(128, return_sequences=True, input_shape=[None, \nmax_id], \n dropout=0.2, recurrent_dropout=0.2), \n keras.layers.GRU(128, return_sequences=True, \n dropout=0.2, recurrent_dropout=0.2), \n keras.layers.TimeDistributed(keras.layers.Dense(max_id, \n \nactivation=\"softmax\")) \n]) \nmodel.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\") \nhistory = model.fit(dataset, epochs=20)\nUsing the Char-RNN Model\nNow we have a model that can predict the next character in text written by\nShakespeare. To feed it some text, we first need to preprocess it like we\ndid earlier, so let’s create a little function for this:\ndef preprocess(texts): \n X = np.array(tokenizer.texts_to_sequences(texts)) - 1 \n return tf.one_hot(X, max_id)\nNow let’s use the model to predict the next letter in some text:\n>>> X_new = preprocess([\"How are yo\"]) \n>>> Y_pred = model.predict_classes(X_new) \n>>> tokenizer.sequences_to_texts(Y_pred + 1)[0][-1] # 1st sentence, last \nchar \n'u'\nSuccess! The model guessed right. Now let’s use this model to generate\nnew text.\nGenerating Fake Shakespearean Text\nTo generate new text using the Char-RNN model, we could feed it some\ntext, make the model predict the most likely next letter, add it at the end of\nthe text, then give the extended text to the model to guess the next letter,\nand so on. But in practice this often leads to the same words being\nrepeated over and over again. Instead, we can pick the next character\nrandomly, with a probability equal to the estimated probability, using\nTensorFlow’s tf.random.categorical() function. This will generate\nmore diverse and interesting text. The categorical() function samples\nrandom class indices, given the class log probabilities (logits). To have\nmore control over the diversity of the generated text, we can divide the\nlogits by a number called the temperature, which we can tweak as we\nwish: a temperature close to 0 will favor the high-probability characters,\nwhile a very high temperature will give all characters an equal probability.\nThe following next_char() function uses this approach to pick the next\ncharacter to add to the input text:\ndef next_char(text, temperature=1): \n X_new = preprocess([text]) \n y_proba = model.predict(X_new)[0, -1:, :] \n rescaled_logits = tf.math.log(y_proba) / temperature \n char_id = tf.random.categorical(rescaled_logits, num_samples=1) + 1 \n return tokenizer.sequences_to_texts(char_id.numpy())[0]\nNext, we can write a small function that will repeatedly call next_char()\nto get the next character and append it to the given text:\ndef complete_text(text, n_chars=50, temperature=1): \n for _ in range(n_chars): \n text += next_char(text, temperature) \n return text\nWe are now ready to generate some text! Let’s try with different\ntemperatures:\n>>> print(complete_text(\"t\", temperature=0.2)) \nthe belly the great and who shall be the belly the \n>>> print(complete_text(\"w\", temperature=1)) \nthing? or why you gremio. \nwho make which the first \n>>> print(complete_text(\"w\", temperature=2)) \nth no cce: \nyeolg-hormer firi. a play asks. \nfol rusb\nApparently our Shakespeare model works best at a temperature close to 1.\nTo generate more convincing text, you could try using more GRU layers and\nmore neurons per layer, train for longer, and add some regularization (for\nexample, you could set recurrent_dropout=0.3 in the GRU layers).\nMoreover, the model is currently incapable of learning patterns longer\nthan n_steps, which is just 100 characters. You could try making this\nwindow larger, but it will also make training harder, and even LSTM and\nGRU cells cannot handle very long sequences. Alternatively, you could use\na stateful RNN.\nStateful RNN\nUntil now, we have used only stateless RNNs: at each training iteration the\nmodel starts with a hidden state full of zeros, then it updates this state at\neach time step, and after the last time step, it throws it away, as it is not\nneeded anymore. What if we told the RNN to preserve this final state after\nprocessing one training batch and use it as the initial state for the next\ntraining batch? This way the model can learn long-term patterns despite\nonly backpropagating through short sequences. This is called a stateful\nRNN. Let’s see how to build one.\nFirst, note that a stateful RNN only makes sense if each input sequence in\na batch starts exactly where the corresponding sequence in the previous\nbatch left off. So the first thing we need to do to build a stateful RNN is to\nuse sequential and nonoverlapping input sequences (rather than the\nshuffled and overlapping sequences we used to train stateless RNNs).\nWhen creating the Dataset, we must therefore use shift=n_steps\n(instead of shift=1) when calling the window() method. Moreover, we\nmust obviously not call the shuffle() method. Unfortunately, batching is\nmuch harder when preparing a dataset for a stateful RNN than it is for a\nstateless RNN. Indeed, if we were to call batch(32), then 32 consecutive\nwindows would be put in the same batch, and the following batch would\nnot continue each of these window where it left off. The first batch would\ncontain windows 1 to 32 and the second batch would contain windows 33\nto 64, so if you consider, say, the first window of each batch (i.e., windows\n1 and 33), you can see that they are not consecutive. The simplest solution\nto this problem is to just use “batches” containing a single window:\ndataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size]) \ndataset = dataset.window(window_length, shift=n_steps, \ndrop_remainder=True) \ndataset = dataset.flat_map(lambda window: window.batch(window_length)) \ndataset = dataset.batch(1) \ndataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:])) \ndataset = dataset.map( \n lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), \nY_batch)) \ndataset = dataset.prefetch(1)\nFigure 16-2 summarizes the first steps.\nFigure 16-2. Preparing a dataset of consecutive sequence fragments for a stateful RNN\nBatching is harder, but it is not impossible. For example, we could chop\nShakespeare’s text into 32 texts of equal length, create one dataset of\nconsecutive input sequences for each of them, and finally use\ntf.train.Dataset.zip(datasets).map(lambda *windows:\ntf.stack(windows)) to create proper consecutive batches, where the n\ninput sequence in a batch starts off exactly where the n input sequence\nended in the previous batch (see the notebook for the full code).\nNow let’s create the stateful RNN. First, we need to set stateful=True\nwhen creating every recurrent layer. Second, the stateful RNN needs to\nknow the batch size (since it will preserve a state for each input sequence\nin the batch), so we must set the batch_input_shape argument in the first\nlayer. Note that we can leave the second dimension unspecified, since the\ninputs could have any length:\nmodel = keras.models.Sequential([ \n keras.layers.GRU(128, return_sequences=True, stateful=True, \n dropout=0.2, recurrent_dropout=0.2, \n batch_input_shape=[batch_size, None, max_id]), \n keras.layers.GRU(128, return_sequences=True, stateful=True, \n dropout=0.2, recurrent_dropout=0.2), \n keras.layers.TimeDistributed(keras.layers.Dense(max_id, \n \nactivation=\"softmax\")) \n])\nth\nth\nAt the end of each epoch, we need to reset the states before we go back to\nthe beginning of the text. For this, we can use a small callback:\nclass ResetStatesCallback(keras.callbacks.Callback): \n def on_epoch_begin(self, epoch, logs): \n self.model.reset_states()\nAnd now we can compile and fit the model (for more epochs, because each\nepoch is much shorter than earlier, and there is only one instance per\nbatch):\nmodel.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\") \nmodel.fit(dataset, epochs=50, callbacks=[ResetStatesCallback()])\nTIP\nAfter this model is trained, it will only be possible to use it to make predictions for\nbatches of the same size as were used during training. To avoid this restriction, create\nan identical stateless model, and copy the stateful model’s weights to this model.\nNow that we have built a character-level model, it’s time to look at word-\nlevel models and tackle a common natural language processing task:\nsentiment analysis. In the process we will learn how to handle sequences\nof variable lengths using masking.\nSentiment Analysis\nIf MNIST is the “hello world” of computer vision, then the IMDb reviews\ndataset is the “hello world” of natural language processing: it consists of\n50,000 movie reviews in English (25,000 for training, 25,000 for testing)\nextracted from the famous Internet Movie Database, along with a simple\nbinary target for each review indicating whether it is negative (0) or\npositive (1). Just like MNIST, the IMDb reviews dataset is popular for\ngood reasons: it is simple enough to be tackled on a laptop in a reasonable\namount of time, but challenging enough to be fun and rewarding. Keras\nprovides a simple function to load it:\n>>> (X_train, y_train), (X_test, y_test) = \nkeras.datasets.imdb.load_data() \n>>> X_train[0][:10] \n[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65]\nWhere are the movie reviews? Well, as you can see, the dataset is already\npreprocessed for you: X_train consists of a list of reviews, each of which\nis represented as a NumPy array of integers, where each integer represents\na word. All punctuation was removed, and then words were converted to\nlowercase, split by spaces, and finally indexed by frequency (so low\nintegers correspond to frequent words). The integers 0, 1, and 2 are\nspecial: they represent the padding token, the start-of-sequence (SSS)\ntoken, and unknown words, respectively. If you want to visualize a review,\nyou can decode it like this:\n>>> word_index = keras.datasets.imdb.get_word_index() \n>>> id_to_word = {id_ + 3: word for word, id_ in word_index.items()} \n>>> for id_, token in enumerate((\"\", \"\", \"\")): \n... id_to_word[id_] = token \n... \n>>> \" \".join([id_to_word[id_] for id_ in X_train[0][:10]]) \n' this film was just brilliant casting location scenery story'\nIn a real project, you will have to preprocess the text yourself. You can do\nthat using the same Tokenizer class we used earlier, but this time setting\nchar_level=False (which is the default). When encoding words, it filters\nout a lot of characters, including most punctuation, line breaks, and tabs\n(but you can change this by setting the filters argument). Most\nimportantly, it uses spaces to identify word boundaries. This is OK for\nEnglish and many other scripts (written languages) that use spaces\nbetween words, but not all scripts use spaces this way. Chinese does not\nuse spaces between words, Vietnamese uses spaces even within words, and\nlanguages such as German often attach multiple words together, without\nspaces. Even in English, spaces are not always the best way to tokenize\ntext: think of “San Francisco” or “#ILoveDeepLearning.”\nFortunately, there are better options! The 2018 paper by Taku Kudo\nintroduced an unsupervised learning technique to tokenize and detokenize\ntext at the subword level in a language-independent way, treating spaces\nlike other characters. With this approach, even if your model encounters a\nword it has never seen before, it can still reasonably guess what it means.\nFor example, it may never have seen the word “smartest” during training,\nbut perhaps it learned the word “smart” and it also learned that the suffix\n“est��� means “the most,” so it can infer the meaning of “smartest.”\nGoogle’s SentencePiece project provides an open source implementation,\ndescribed in a paper by Taku Kudo and John Richardson.\nAnother option was proposed in an earlier paper by Rico Sennrich et al.\nthat explored other ways of creating subword encodings (e.g., using byte\npair encoding). Last but not least, the TensorFlow team released the\nTF.Text library in June 2019, which implements various tokenization\nstrategies, including WordPiece (a variant of byte pair encoding).\nIf you want to deploy your model to a mobile device or a web browser, and\nyou don’t want to have to write a different preprocessing function every\ntime, then you will want to handle preprocessing using only TensorFlow\noperations, so it can be included in the model itself. Let’s see how. First,\nlet’s load the original IMDb reviews, as text (byte strings), using\nTensorFlow Datasets (introduced in Chapter 13):\nimport tensorflow_datasets as tfds \n \ndatasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, \nwith_info=True) \ntrain_size = info.splits[\"train\"].num_examples\nNext, let’s write the preprocessing function:\ndef preprocess(X_batch, y_batch): \n X_batch = tf.strings.substr(X_batch, 0, 300) \n X_batch = tf.strings.regex_replace(X_batch, b\"\", b\" \") \n4\n5\n6\n7\n X_batch = tf.strings.regex_replace(X_batch, b\"[^a-zA-Z']\", b\" \") \n X_batch = tf.strings.split(X_batch) \n return X_batch.to_tensor(default_value=b\"\"), y_batch\nIt starts by truncating the reviews, keeping only the first 300 characters of\neach: this will speed up training, and it won’t impact performance too\nmuch because you can generally tell whether a review is positive or not in\nthe first sentence or two. Then it uses regular expressions to replace tags with spaces, and to replace any characters other than letters and\nquotes with spaces. For example, the text \"Well, I can't
\" will\nbecome \"Well I can't\". Finally, the preprocess() function splits the\nreviews by the spaces, which returns a ragged tensor, and it converts this\nragged tensor to a dense tensor, padding all reviews with the padding\ntoken \"\" so that they all have the same length.\nNext, we need to construct the vocabulary. This requires going through the\nwhole training set once, applying our preprocess() function, and using a\nCounter to count the number of occurrences of each word:\nfrom collections import Counter \nvocabulary = Counter() \nfor X_batch, y_batch in datasets[\"train\"].batch(32).map(preprocess): \n for review in X_batch: \n vocabulary.update(list(review.numpy()))\nLet’s look at the three most common words:\n>>> vocabulary.most_common()[:3] \n[(b'', 215797), (b'the', 61137), (b'a', 38564)]\nGreat! We probably don’t need our model to know all the words in the\ndictionary to get good performance, though, so let’s truncate the\nvocabulary, keeping only the 10,000 most common words:\nvocab_size = 10000 \ntruncated_vocabulary = [ \n word for word, count in vocabulary.most_common()[:vocab_size]]\nNow we need to add a preprocessing step to replace each word with its ID\n(i.e., its index in the vocabulary). Just like we did in Chapter 13, we will\ncreate a lookup table for this, using 1,000 out-of-vocabulary (oov)\nbuckets:\nwords = tf.constant(truncated_vocabulary) \nword_ids = tf.range(len(truncated_vocabulary), dtype=tf.int64) \nvocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids) \nnum_oov_buckets = 1000 \ntable = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)\nWe can then use this table to look up the IDs of a few words:\n>>> table.lookup(tf.constant([b\"This movie was faaaaaantastic\".split()])) \n\nNote that the words “this,” “movie,” and “was” were found in the table, so\ntheir IDs are lower than 10,000, while the word “faaaaaantastic” was not\nfound, so it was mapped to one of the oov buckets, with an ID greater than\nor equal to 10,000.\nTIP\nTF Transform (introduced in Chapter 13) provides some useful functions to handle\nsuch vocabularies. For example, check out the\ntft.compute_and_apply_vocabulary() function: it will go through the dataset to\nfind all distinct words and build the vocabulary, and it will generate the TF\noperations required to encode each word using this vocabulary.\nNow we are ready to create the final training set. We batch the reviews,\nthen convert them to short sequences of words using the preprocess()\nfunction, then encode these words using a simple encode_words()\nfunction that uses the table we just built, and finally prefetch the next\nbatch:\ndef encode_words(X_batch, y_batch): \n return table.lookup(X_batch), y_batch \n \ntrain_set = datasets[\"train\"].batch(32).map(preprocess) \ntrain_set = train_set.map(encode_words).prefetch(1)\nAt last we can create the model and train it:\nembed_size = 128 \nmodel = keras.models.Sequential([ \n keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size, \n input_shape=[None]), \n keras.layers.GRU(128, return_sequences=True), \n keras.layers.GRU(128), \n keras.layers.Dense(1, activation=\"sigmoid\") \n]) \nmodel.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", \n metrics=[\"accuracy\"]) \nhistory = model.fit(train_set, epochs=5)\nThe first layer is an Embedding layer, which will convert word IDs into\nembeddings (introduced in Chapter 13). The embedding matrix needs to\nhave one row per word ID (vocab_size + num_oov_buckets) and one\ncolumn per embedding dimension (this example uses 128 dimensions, but\nthis is a hyperparameter you could tune). Whereas the inputs of the model\nwill be 2D tensors of shape [batch size, time steps], the output of the\nEmbedding layer will be a 3D tensor of shape [batch size, time steps,\nembedding size].\nThe rest of the model is fairly straightforward: it is composed of two GRU\nlayers, with the second one returning only the output of the last time step.\nThe output layer is just a single neuron using the sigmoid activation\nfunction to output the estimated probability that the review expresses a\npositive sentiment regarding the movie. We then compile the model quite\nsimply, and we fit it on the dataset we prepared earlier, for a few epochs.\nMasking\nAs it stands, the model will need to learn that the padding tokens should be\nignored. But we already know that! Why don’t we tell the model to ignore\nthe padding tokens, so that it can focus on the data that actually matters?\nIt’s actually trivial: simply add mask_zero=True when creating the\nEmbedding layer. This means that padding tokens (whose ID is 0) will be\nignored by all downstream layers. That’s all!\nThe way this works is that the Embedding layer creates a mask tensor\nequal to K.not_equal(inputs, 0) (where K = keras.backend): it is a\nBoolean tensor with the same shape as the inputs, and it is equal to False\nanywhere the word IDs are 0, or True otherwise. This mask tensor is then\nautomatically propagated by the model to all subsequent layers, as long as\nthe time dimension is preserved. So in this example, both GRU layers will\nreceive this mask automatically, but since the second GRU layer does not\nreturn sequences (it only returns the output of the last time step), the mask\nwill not be transmitted to the Dense layer. Each layer may handle the mask\ndifferently, but in general they simply ignore masked time steps (i.e., time\nsteps for which the mask is False). For example, when a recurrent layer\nencounters a masked time step, it simply copies the output from the\nprevious time step. If the mask propagates all the way to the output (in\nmodels that output sequences, which is not the case in this example), then\nit will be applied to the losses as well, so the masked time steps will not\ncontribute to the loss (their loss will be 0).\nWARNING\nThe LSTM and GRU layers have an optimized implementation for GPUs, based on\nNvidia’s cuDNN library. However, this implementation does not support masking. If\nyour model uses a mask, then these layers will fall back to the (much slower) default\nimplementation. Note that the optimized implementation also requires you to use the\ndefault values for several hyperparameters: activation, recurrent_activation,\nrecurrent_dropout, unroll, use_bias, and reset_after.\n8\nAll layers that receive the mask must support masking (or else an\nexception will be raised). This includes all recurrent layers, as well as the\nTimeDistributed layer and a few other layers. Any layer that supports\nmasking must have a supports_masking attribute equal to True. If you\nwant to implement your own custom layer with masking support, you\nshould add a mask argument to the call() method (and obviously make\nthe method use the mask somehow). Additionally, you should set\nself.supports_masking = True in the constructor. If your layer does\nnot start with an Embedding layer, you may use the\nkeras.layers.Masking layer instead: it sets the mask to\nK.any(K.not_equal(inputs, 0), axis=-1), meaning that time steps\nwhere the last dimension is full of zeros will be masked out in subsequent\nlayers (again, as long as the time dimension exists).\nUsing masking layers and automatic mask propagation works best for\nsimple Sequential models. It will not always work for more complex\nmodels, such as when you need to mix Conv1D layers with recurrent layers.\nIn such cases, you will need to explicitly compute the mask and pass it to\nthe appropriate layers, using either the Functional API or the Subclassing\nAPI. For example, the following model is identical to the previous model,\nexcept it is built using the Functional API and handles masking manually:\nK = keras.backend \ninputs = keras.layers.Input(shape=[None]) \nmask = keras.layers.Lambda(lambda inputs: K.not_equal(inputs, 0))(inputs) \nz = keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size)\n(inputs) \nz = keras.layers.GRU(128, return_sequences=True)(z, mask=mask) \nz = keras.layers.GRU(128)(z, mask=mask) \noutputs = keras.layers.Dense(1, activation=\"sigmoid\")(z) \nmodel = keras.Model(inputs=[inputs], outputs=[outputs])\nAfter training for a few epochs, this model will become quite good at\njudging whether a review is positive or not. If you use the TensorBoard()\ncallback, you can visualize the embeddings in TensorBoard as they are\nbeing learned: it is fascinating to see words like “awesome” and\n“amazing” gradually cluster on one side of the embedding space, while\nwords like “awful” and “terrible” cluster on the other side. Some words\nare not as positive as you might expect (at least with this model), such as\nthe word “good,” presumably because many negative reviews contain the\nphrase “not good.” It’s impressive that the model is able to learn useful\nword embeddings based on just 25,000 movie reviews. Imagine how good\nthe embeddings would be if we had billions of reviews to train on!\nUnfortunately we don’t, but perhaps we can reuse word embeddings\ntrained on some other large text corpus (e.g., Wikipedia articles), even if it\nis not composed of movie reviews? After all, the word “amazing”\ngenerally has the same meaning whether you use it to talk about movies or\nanything else. Moreover, perhaps embeddings would be useful for\nsentiment analysis even if they were trained on another task: since words\nlike “awesome” and “amazing” have a similar meaning, they will likely\ncluster in the embedding space even for other tasks (e.g., predicting the\nnext word in a sentence). If all positive words and all negative words form\nclusters, then this will be helpful for sentiment analysis. So instead of\nusing so many parameters to learn word embeddings, let’s see if we can’t\njust reuse pretrained embeddings.\nReusing Pretrained Embeddings\nThe TensorFlow Hub project makes it easy to reuse pretrained model\ncomponents in your own models. These model components are called\nmodules. Simply browse the TF Hub repository, find the one you need, and\ncopy the code example into your project, and the module will be\nautomatically downloaded, along with its pretrained weights, and included\nin your model. Easy!\nFor example, let’s use the nnlm-en-dim50 sentence embedding module,\nversion 1, in our sentiment analysis model:\nimport tensorflow_hub as hub \n \nmodel = keras.Sequential([ \n hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-\ndim50/1\", \n dtype=tf.string, input_shape=[], output_shape=[50]), \n keras.layers.Dense(128, activation=\"relu\"), \n keras.layers.Dense(1, activation=\"sigmoid\") \n]) \nmodel.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", \n metrics=[\"accuracy\"])\nThe hub.KerasLayer layer downloads the module from the given URL.\nThis particular module is a sentence encoder: it takes strings as input and\nencodes each one as a single vector (in this case, a 50-dimensional vector).\nInternally, it parses the string (splitting words on spaces) and embeds each\nword using an embedding matrix that was pretrained on a huge corpus: the\nGoogle News 7B corpus (seven billion words long!). Then it computes the\nmean of all the word embeddings, and the result is the sentence\nembedding. We can then add two simple Dense layers to create a good\nsentiment analysis model. By default, a hub.KerasLayer is not trainable,\nbut you can set trainable=True when creating it to change that so that\nyou can fine-tune it for your task.\nWARNING\nNot all TF Hub modules support TensorFlow 2, so make sure you choose a module\nthat does.\nNext, we can just load the IMDb reviews dataset—no need to preprocess it\n(except for batching and prefetching)—and directly train the model:\ndatasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, \nwith_info=True) \ntrain_size = info.splits[\"train\"].num_examples \nbatch_size = 32 \ntrain_set = datasets[\"train\"].batch(batch_size).prefetch(1) \nhistory = model.fit(train_set, epochs=5)\nNote that the last part of the TF Hub module URL specified that we\nwanted version 1 of the model. This versioning ensures that if a new\n9\nmodule version is released, it will not break our model. Conveniently, if\nyou just enter this URL in a web browser, you will get the documentation\nfor this module. By default, TF Hub will cache the downloaded files into\nthe local system’s temporary directory. You may prefer to download them\ninto a more permanent directory to avoid having to download them again\nafter every system cleanup. To do that, set the TFHUB_CACHE_DIR\nenvironment variable to the directory of your choice (e.g.,\nos.environ[\"TFHUB_CACHE_DIR\"] = \"./my_tfhub_cache\").\nSo far, we have looked at time series, text generation using Char-RNN,\nand sentiment analysis using word-level RNN models, training our own\nword embeddings or reusing pretrained embeddings. Let’s now look at\nanother important NLP task: neural machine translation (NMT), first\nusing a pure Encoder–Decoder model, then improving it with attention\nmechanisms, and finally looking the extraordinary Transformer\narchitecture.\nAn Encoder–Decoder Network for Neural\nMachine Translation\nLet’s take a look at a simple neural machine translation model\n that will\ntranslate English sentences to French (see Figure 16-3).\nIn short, the English sentences are fed to the encoder, and the decoder\noutputs the French translations. Note that the French translations are also\nused as inputs to the decoder, but shifted back by one step. In other words,\nthe decoder is given as input the word that it should have output at the\nprevious step (regardless of what it actually output). For the very first\nword, it is given the start-of-sequence (SOS) token. The decoder is\nexpected to end the sentence with an end-of-sequence (EOS) token.\nNote that the English sentences are reversed before they are fed to the\nencoder. For example, “I drink milk” is reversed to “milk drink I.” This\nensures that the beginning of the English sentence will be fed last to the\n10\nencoder, which is useful because that’s generally the first thing that the\ndecoder needs to translate.\nEach word is initially represented by its ID (e.g., 288 for the word “milk”).\nNext, an embedding layer returns the word embedding. These word\nembeddings are what is actually fed to the encoder and the decoder.\nFigure 16-3. A simple machine translation model\nAt each step, the decoder outputs a score for each word in the output\nvocabulary (i.e., French), and then the softmax layer turns these scores\ninto probabilities. For example, at the first step the word “Je” may have a\nprobability of 20%, “Tu” may have a probability of 1%, and so on. The\nword with the highest probability is output. This is very much like a\nregular classification task, so you can train the model using the\n\"sparse_categorical_crossentropy\" loss, much like we did in the\nChar-RNN model.\nNote that at inference time (after training), you will not have the target\nsentence to feed to the decoder. Instead, simply feed the decoder the word\nthat it output at the previous step, as shown in Figure 16-4 (this will\nrequire an embedding lookup that is not shown in the diagram).\nFigure 16-4. Feeding the previous output word as input at inference time\nOK, now you have the big picture. Still, there are a few more details to\nhandle if you implement this model:\nSo far we have assumed that all input sequences (to the encoder\nand to the decoder) have a constant length. But obviously\nsentence lengths vary. Since regular tensors have fixed shapes,\nthey can only contain sentences of the same length. You can use\nmasking to handle this, as discussed earlier. However, if the\nsentences have very different lengths, you can’t just crop them\nlike we did for sentiment analysis (because we want full\ntranslations, not cropped translations). Instead, group sentences\ninto buckets of similar lengths (e.g., a bucket for the 1- to 6-word\nsentences, another for the 7- to 12-word sentences, and so on),\nusing padding for the shorter sequences to ensure all sentences in\na bucket have the same length (check out the\ntf.data.experimental.bucket_by_sequence_length()\nfunction for this). For example, “I drink milk” becomes “\n milk drink I.”\nWe want to ignore any output past the EOS token, so these tokens\nshould not contribute to the loss (they must be masked out). For\nexample, if the model outputs “Je bois du lait oui,” the loss\nfor the last word should be ignored.\nWhen the output vocabulary is large (which is the case here),\noutputting a probability for each and every possible word would\nbe terribly slow. If the target vocabulary contains, say, 50,000\nFrench words, then the decoder would output 50,000-dimensional\nvectors, and then computing the softmax function over such a\nlarge vector would be very computationally intensive. To avoid\nthis, one solution is to look only at the logits output by the model\nfor the correct word and for a random sample of incorrect words,\nthen compute an approximation of the loss based only on these\nlogits. This sampled softmax technique was introduced in 2015 by\nSébastien Jean et al.. In TensorFlow you can use the\ntf.nn.sampled_softmax_loss() function for this during\ntraining and use the normal softmax function at inference time\n(sampled softmax cannot be used at inference time because it\nrequires knowing the target).\nThe TensorFlow Addons project includes many sequence-to-sequence\ntools to let you easily build production-ready Encoder–Decoders. For\nexample, the following code creates a basic Encoder–Decoder model,\nsimilar to the one represented in Figure 16-3:\nimport tensorflow_addons as tfa \n \nencoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32) \ndecoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32) \nsequence_lengths = keras.layers.Input(shape=[], dtype=np.int32) \n \nembeddings = keras.layers.Embedding(vocab_size, embed_size) \nencoder_embeddings = embeddings(encoder_inputs) \ndecoder_embeddings = embeddings(decoder_inputs) \n \nencoder = keras.layers.LSTM(512, return_state=True) \nencoder_outputs, state_h, state_c = encoder(encoder_embeddings) \n11\nencoder_state = [state_h, state_c] \n \nsampler = tfa.seq2seq.sampler.TrainingSampler() \n \ndecoder_cell = keras.layers.LSTMCell(512) \noutput_layer = keras.layers.Dense(vocab_size) \ndecoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell, sampler, \n \noutput_layer=output_layer) \nfinal_outputs, final_state, final_sequence_lengths = decoder( \n decoder_embeddings, initial_state=encoder_state, \n sequence_length=sequence_lengths) \nY_proba = tf.nn.softmax(final_outputs.rnn_output) \n \nmodel = keras.Model(inputs=[encoder_inputs, decoder_inputs, \nsequence_lengths], \n outputs=[Y_proba])\nThe code is mostly self-explanatory, but there are a few points to note.\nFirst, we set return_state=True when creating the LSTM layer so that we\ncan get its final hidden state and pass it to the decoder. Since we are using\nan LSTM cell, it actually returns two hidden states (short term and long\nterm). The TrainingSampler is one of several samplers available in\nTensorFlow Addons: their role is to tell the decoder at each step what it\nshould pretend the previous output was. During inference, this should be\nthe embedding of the token that was actually output. During training, it\nshould be the embedding of the previous target token: this is why we used\nthe TrainingSampler. In practice, it is often a good idea to start training\nwith the embedding of the target of the previous time step and gradually\ntransition to using the embedding of the actual token that was output at the\nprevious step. This idea was introduced in a 2015 paper\n by Samy Bengio\net al. The ScheduledEmbeddingTrainingSampler will randomly choose\nbetween the target or the actual output, with a probability that you can\ngradually change during training.\nBidirectional RNNs\nA each time step, a regular recurrent layer only looks at past and present\ninputs before generating its output. In other words, it is “causal,” meaning\n12\nit cannot look into the future. This type of RNN makes sense when\nforecasting time series, but for many NLP tasks, such as Neural Machine\nTranslation, it is often preferable to look ahead at the next words before\nencoding a given word. For example, consider the phrases “the Queen of\nthe United Kingdom,” “the queen of hearts,” and “the queen bee”: to\nproperly encode the word “queen,” you need to look ahead. To implement\nthis, run two recurrent layers on the same inputs, one reading the words\nfrom left to right and the other reading them from right to left. Then\nsimply combine their outputs at each time step, typically by concatenating\nthem. This is called a bidirectional recurrent layer (see Figure 16-5).\nTo implement a bidirectional recurrent layer in Keras, wrap a recurrent\nlayer in a keras.layers.Bidirectional layer. For example, the\nfollowing code creates a bidirectional GRU layer:\nkeras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))\nNOTE\nThe Bidirectional layer will create a clone of the GRU layer (but in the reverse\ndirection), and it will run both and concatenate their outputs. So although the GRU\nlayer has 10 units, the Bidirectional layer will output 20 values per time step.\nFigure 16-5. A bidirectional recurrent layer\nBeam Search\nSuppose you train an Encoder–Decoder model, and use it to translate the\nFrench sentence “Comment vas-tu?” to English. You are hoping that it will\noutput the proper translation (“How are you?”), but unfortunately it\noutputs “How will you?” Looking at the training set, you notice many\nsentences such as “Comment vas-tu jouer?” which translates to “How will\nyou play?” So it wasn’t absurd for the model to output “How will” after\nseeing “Comment vas.” Unfortunately, in this case it was a mistake, and\nthe model could not go back and fix it, so it tried to complete the sentence\nas best it could. By greedily outputting the most likely word at every step,\nit ended up with a suboptimal translation. How can we give the model a\nchance to go back and fix mistakes it made earlier? One of the most\ncommon solutions is beam search: it keeps track of a short list of the k\nmost promising sentences (say, the top three), and at each decoder step it\ntries to extend them by one word, keeping only the k most likely\nsentences. The parameter k is called the beam width.\nFor example, suppose you use the model to translate the sentence\n“Comment vas-tu?” using beam search with a beam width of 3. At the first\ndecoder step, the model will output an estimated probability for each\npossible word. Suppose the top three words are “How” (75% estimated\nprobability), “What” (3%), and “You” (1%). That’s our short list so far.\nNext, we create three copies of our model and use them to find the next\nword for each sentence. Each model will output one estimated probability\nper word in the vocabulary. The first model will try to find the next word\nin the sentence “How,” and perhaps it will output a probability of 36% for\nthe word “will,” 32% for the word “are,” 16% for the word “do,” and so\non. Note that these are actually conditional probabilities, given that the\nsentence starts with “How.” The second model will try to complete the\nsentence “What”; it might output a conditional probability of 50% for the\nword “are,” and so on. Assuming the vocabulary has 10,000 words, each\nmodel will output 10,000 probabilities.\nNext, we compute the probabilities of each of the 30,000 two-word\nsentences that these models considered (3 × 10,000). We do this by\nmultiplying the estimated conditional probability of each word by the\nestimated probability of the sentence it completes. For example, the\nestimated probability of the sentence “How” was 75%, while the estimated\nconditional probability of the word “will” (given that the first word is\n“How”) was 36%, so the estimated probability of the sentence “How will”\nis 75% × 36% = 27%. After computing the probabilities of all 30,000 two-\nword sentences, we keep only the top 3. Perhaps they all start with the\nword “How”: “How will” (27%), “How are” (24%), and “How do” (12%).\nRight now, the sentence “How will” is winning, but “How are” has not\nbeen eliminated.\nThen we repeat the same process: we use three models to predict the next\nword in each of these three sentences, and we compute the probabilities of\nall 30,000 three-word sentences we considered. Perhaps the top three are\nnow “How are you” (10%), “How do you” (8%), and “How will you”\n(2%). At the next step we may get “How do you do” (7%), “How are you\n” (6%), and “How are you doing” (3%). Notice that “How will” was\neliminated, and we now have three perfectly reasonable translations. We\nboosted our Encoder–Decoder model’s performance without any extra\ntraining, simply by using it more wisely.\nYou can implement beam search fairly easily using TensorFlow Addons:\nbeam_width = 10 \ndecoder = tfa.seq2seq.beam_search_decoder.BeamSearchDecoder( \n cell=decoder_cell, beam_width=beam_width, output_layer=output_layer) \ndecoder_initial_state = tfa.seq2seq.beam_search_decoder.tile_batch( \n encoder_state, multiplier=beam_width) \noutputs, _, _ = decoder( \n embedding_decoder, start_tokens=start_tokens, end_token=end_token, \n initial_state=decoder_initial_state)\nWe first create a BeamSearchDecoder, which wraps all the decoder clones\n(in this case 10 clones). Then we create one copy of the encoder’s final\nstate for each decoder clone, and we pass these states to the decoder, along\nwith the start and end tokens.\nWith all this, you can get good translations for fairly short sentences\n(especially if you use pretrained word embeddings). Unfortunately, this\nmodel will be really bad at translating long sentences. Once again, the\nproblem comes from the limited short-term memory of RNNs. Attention\nmechanisms are the game-changing innovation that addressed this\nproblem.\nAttention Mechanisms\nConsider the path from the word “milk” to its translation “lait” in\nFigure 16-3: it is quite long! This means that a representation of this word\n(along with all the other words) needs to be carried over many steps before\nit is actually used. Can’t we make this path shorter?\nThis was the core idea in a groundbreaking 2014 paper\n by Dzmitry\nBahdanau et al. They introduced a technique that allowed the decoder to\nfocus on the appropriate words (as encoded by the encoder) at each time\nstep. For example, at the time step where the decoder needs to output the\n13\nword “lait,” it will focus its attention on the word “milk.” This means that\nthe path from an input word to its translation is now much shorter, so the\nshort-term memory limitations of RNNs have much less impact. Attention\nmechanisms revolutionized neural machine translation (and NLP in\ngeneral), allowing a significant improvement in the state of the art,\nespecially for long sentences (over 30 words).\nFigure 16-6 shows this model’s architecture (slightly simplified, as we\nwill see). On the left, you have the encoder and the decoder. Instead of just\nsending the encoder’s final hidden state to the decoder (which is still done,\nalthough it is not shown in the figure), we now send all of its outputs to the\ndecoder. At each time step, the decoder’s memory cell computes a\nweighted sum of all these encoder outputs: this determines which words it\nwill focus on at this step. The weight α\n is the weight of the i encoder\noutput at the t decoder time step. For example, if the weight α\n is\nmuch larger than the weights α\n and α\n, then the decoder will pay\nmuch more attention to word number 2 (“milk”) than to the other two\nwords, at least at this time step. The rest of the decoder works just like\nearlier: at each time step the memory cell receives the inputs we just\ndiscussed, plus the hidden state from the previous time step, and finally\n(although it is not represented in the diagram) it receives the target word\nfrom the previous time step (or at inference time, the output from the\nprevious time step).\n14\n(t,i)\nth\nth\n(3,2)\n(3,0)\n(3,1)\nFigure 16-6. Neural machine translation using an Encoder–Decoder network with an attention\nmodel\nBut where do these α\n weights come from? It’s actually pretty simple:\nthey are generated by a type of small neural network called an alignment\nmodel (or an attention layer), which is trained jointly with the rest of the\nEncoder–Decoder model. This alignment model is illustrated on the\nrighthand side of Figure 16-6. It starts with a time-distributed Dense\nlayer\n with a single neuron, which receives as input all the encoder\noutputs, concatenated with the decoder’s previous hidden state (e.g., h\n).\nThis layer outputs a score (or energy) for each encoder output (e.g., e\n):\nthis score measures how well each output is aligned with the decoder’s\nprevious hidden state. Finally, all the scores go through a softmax layer to\nget a final weight for each encoder output (e.g., α\n). All the weights for\na given decoder time step add up to 1 (since the softmax layer is not time-\ndistributed). This particular attention mechanism is called Bahdanau\nattention (named after the paper’s first author). Since it concatenates the\nencoder output with the decoder’s previous hidden state, it is sometimes\ncalled concatenative attention (or additive attention).\n(t,i)\n15\n(2)\n(3, 2)\n(3,2)\nNOTE\nIf the input sentence is n words long, and assuming the output sentence is about as\nlong, then this model will need to compute about n weights. Fortunately, this\nquadratic computational complexity is still tractable because even long sentences\ndon’t have thousands of words.\nAnother common attention mechanism was proposed shortly after, in a\n2015 paper\n by Minh-Thang Luong et al. Because the goal of the\nattention mechanism is to measure the similarity between one of the\nencoder’s outputs and the decoder’s previous hidden state, the authors\nproposed to simply compute the dot product (see Chapter 4) of these two\nvectors, as this is often a fairly good similarity measure, and modern\nhardware can compute it much faster. For this to be possible, both vectors\nmust have the same dimensionality. This is called Luong attention (again,\nafter the paper’s first author), or sometimes multiplicative attention. The\ndot product gives a score, and all the scores (at a given decoder time step)\ngo through a softmax layer to give the final weights, just like in Bahdanau\nattention. Another simplification they proposed was to use the decoder’s\nhidden state at the current time step rather than at the previous time step\n(i.e., h ) rather than h\n), then to use the output of the attention\nmechanism (noted ˜h(t)) directly to compute the decoder’s predictions\n(rather than using it to compute the decoder’s current hidden state). They\nalso proposed a variant of the dot product mechanism where the encoder\noutputs first go through a linear transformation (i.e., a time-distributed\nDense layer without a bias term) before the dot products are computed.\nThis is called the “general” dot product approach. They compared both dot\nproduct approaches to the concatenative attention mechanism (adding a\nrescaling parameter vector v), and they observed that the dot product\nvariants performed better than concatenative attention. For this reason,\nconcatenative attention is much less used now. The equations for these\nthree attention mechanisms are summarized in Equation 16-1.\nEquation 16-1. Attention mechanisms\n2\n16\n(t)\n(t–1)\n˜h(t)\n= ∑\ni\nα(t,i)y(i)\nwith α(t,i)\n=\nand e(t,i)\n=\n⎧\n⎪ \n⎪\n⎨\n⎪ \n⎪\n⎩\nh(t)\n⊺y(i)\ndot\nh(t)\n⊺W y(i)\ngeneral\nv⊺tanh (W [h(t); y(i)])\nconcat\nHere is how you can add Luong attention to an Encoder–Decoder model\nusing TensorFlow Addons:\nattention_mechanism = tfa.seq2seq.attention_wrapper.LuongAttention( \n units, encoder_state, memory_sequence_length=encoder_sequence_length) \nattention_decoder_cell = tfa.seq2seq.attention_wrapper.AttentionWrapper( \n decoder_cell, attention_mechanism, attention_layer_size=n_units)\nWe simply wrap the decoder cell in an AttentionWrapper, and we provide\nthe desired attention mechanism (Luong attention in this example).\nVisual Attention\nAttention mechanisms are now used for a variety of purposes. One of their\nfirst applications beyond NMT was in generating image captions using\nvisual attention:\n a convolutional neural network first processes the\nimage and outputs some feature maps, then a decoder RNN equipped with\nan attention mechanism generates the caption, one word at a time. At each\ndecoder time step (each word), the decoder uses the attention model to\nfocus on just the right part of the image. For example, in Figure 16-7, the\nmodel generated the caption “A woman is throwing a frisbee in a park,”\nand you can see what part of the input image the decoder focused its\nattention on when it was about to output the word “frisbee”: clearly, most\nof its attention was focused on the frisbee.\nexp(e(t,i))\n∑i′ exp(e(t,i′))\n17\nFigure 16-7. Visual attention: an input image (left) and the model’s focus before producing the\nword “frisbee” (right)\nEXPLAINABILITY\nOne extra benefit of attention mechanisms is that they make it easier\nto understand what led the model to produce its output. This is called\nexplainability. It can be especially useful when the model makes a\nmistake: for example, if an image of a dog walking in the snow is\nlabeled as “a wolf walking in the snow,” then you can go back and\ncheck what the model focused on when it output the word “wolf.” You\nmay find that it was paying attention not only to the dog, but also to\nthe snow, hinting at a possible explanation: perhaps the way the model\nlearned to distinguish dogs from wolves is by checking whether or not\nthere’s a lot of snow around. You can then fix this by training the\nmodel with more images of wolves without snow, and dogs with snow.\nThis example comes from a great 2016 paper\n by Marco Tulio\nRibeiro et al. that uses a different approach to explainability: learning\nan interpretable model locally around a classifier’s prediction.\nIn some applications, explainability is not just a tool to debug a\nmodel; it can be a legal requirement (think of a system deciding\nwhether or not it should grant you a loan).\n18\n19\nAttention mechanisms are so powerful that you can actually build state-of-\nthe-art models using only attention mechanisms.\nAttention Is All You Need: The Transformer Architecture\nIn a groundbreaking 2017 paper,\n a team of Google researchers suggested\nthat “Attention Is All You Need.” They managed to create an architecture\ncalled the Transformer, which significantly improved the state of the art in\nNMT without using any recurrent or convolutional layers,\n just attention\nmechanisms (plus embedding layers, dense layers, normalization layers,\nand a few other bits and pieces). As an extra bonus, this architecture was\nalso much faster to train and easier to parallelize, so they managed to train\nit at a fraction of the time and cost of the previous state-of-the-art models.\nThe Transformer architecture is represented in Figure 16-8.\n20\n21\nFigure 16-8. The Transformer architecture\nLet’s walk through this figure:\nThe lefthand part is the encoder. Just like earlier, it takes as input\na batch of sentences represented as sequences of word IDs (the\ninput shape is [batch size, max input sentence length]), and it\nencodes each word into a 512-dimensional representation (so the\nencoder’s output shape is [batch size, max input sentence length,\n512]). Note that the top part of the encoder is stacked N times (in\nthe paper, N = 6).\nThe righthand part is the decoder. During training, it takes the\ntarget sentence as input (also represented as a sequence of word\nIDs), shifted one time step to the right (i.e., a start-of-sequence\ntoken is inserted at the beginning). It also receives the outputs of\nthe encoder (i.e., the arrows coming from the left side). Note that\nthe top part of the decoder is also stacked N times, and the\nencoder stack’s final outputs are fed to the decoder at each of\nthese N levels. Just like earlier, the decoder outputs a probability\nfor each possible next word, at each time step (its output shape is\n[batch size, max output sentence length, vocabulary length]).\nDuring inference, the decoder cannot be fed targets, so we feed it\nthe previously output words (starting with a start-of-sequence\ntoken). So the model needs to be called repeatedly, predicting one\nmore word at every round (which is fed to the decoder at the next\nround, until the end-of-sequence token is output).\nLooking more closely, you can see that you are already familiar\nwith most components: there are two embedding layers, 5 × N\nskip connections, each of them followed by a layer normalization\nlayer, 2 × N “Feed Forward” modules that are composed of two\ndense layers each (the first one using the ReLU activation\nfunction, the second with no activation function), and finally the\noutput layer is a dense layer using the softmax activation\n22\nfunction. All of these layers are time-distributed, so each word is\ntreated independently of all the others. But how can we translate a\nsentence by only looking at one word at a time? Well, that’s where\nthe new components come in:\nThe encoder’s Multi-Head Attention layer encodes each\nword’s relationship with every other word in the same\nsentence, paying more attention to the most relevant\nones. For example, the output of this layer for the word\n“Queen” in the sentence “They welcomed the Queen of\nthe United Kingdom” will depend on all the words in the\nsentence, but it will probably pay more attention to the\nwords “United” and “Kingdom” than to the words “They”\nor “welcomed.” This attention mechanism is called self-\nattention (the sentence is paying attention to itself). We\nwill discuss exactly how it works shortly. The decoder’s\nMasked Multi-Head Attention layer does the same thing,\nbut each word is only allowed to attend to words located\nbefore it. Finally, the decoder’s upper Multi-Head\nAttention layer is where the decoder pays attention to the\nwords in the input sentence. For example, the decoder\nwill probably pay close attention to the word “Queen” in\nthe input sentence when it is about to output this word’s\ntranslation.\nThe positional embeddings are simply dense vectors\n(much like word embeddings) that represent the position\nof a word in the sentence. The n positional embedding\nis added to the word embedding of the n word in each\nsentence. This gives the model access to each word’s\nposition, which is needed because the Multi-Head\nAttention layers do not consider the order or the position\nof the words; they only look at their relationships. Since\nall the other layers are time-distributed, they have no way\nof knowing the position of each word (either relative or\nth\nth\nabsolute). Obviously, the relative and absolute word\npositions are important, so we need to give this\ninformation to the Transformer somehow, and positional\nembeddings are a good way to do this.\nLet’s look a bit closer at both these novel components of the Transformer\narchitecture, starting with the positional embeddings.\nPositional embeddings\nA positional embedding is a dense vector that encodes the position of a\nword within a sentence: the i positional embedding is simply added to\nthe word embedding of the i word in the sentence. These positional\nembeddings can be learned by the model, but in the paper the authors\npreferred to use fixed positional embeddings, defined using the sine and\ncosine functions of different frequencies. The positional embedding\nmatrix P is defined in Equation 16-2 and represented at the bottom of\nFigure 16-9 (transposed), where P is the i component of the embedding\nfor the word located at the p position in the sentence.\nEquation 16-2. Sine/cosine positional embeddings\nPp,2i\n= sin(p/100002i/d)\nPp,2i+1\n= cos(p/100002i/d)\nth\nth\np,i\nth\nth\nFigure 16-9. Sine/cosine positional embedding matrix (transposed, top) with a focus on two\nvalues of i (bottom)\nThis solution gives the same performance as learned positional\nembeddings do, but it can extend to arbitrarily long sentences, which is\nwhy it’s favored. After the positional embeddings are added to the word\nembeddings, the rest of the model has access to the absolute position of\neach word in the sentence because there is a unique positional embedding\nfor each position (e.g., the positional embedding for the word located at\nthe 22nd position in a sentence is represented by the vertical dashed line at\nthe bottom left of Figure 16-9, and you can see that it is unique to that\nposition). Moreover, the choice of oscillating functions (sine and cosine)\nmakes it possible for the model to learn relative positions as well. For\nexample, words located 38 words apart (e.g., at positions p = 22 and\np = 60) always have the same positional embedding values in the\nembedding dimensions i = 100 and i = 101, as you can see in Figure 16-9.\nThis explains why we need both the sine and the cosine for each\nfrequency: if we only used the sine (the blue wave at i = 100), the model\nwould not be able to distinguish positions p = 25 and p = 35 (marked by a\ncross).\nThere is no PositionalEmbedding layer in TensorFlow, but it is easy to\ncreate one. For efficiency reasons, we precompute the positional\nembedding matrix in the constructor (so we need to know the maximum\nsentence length, max_steps, and the number of dimensions for each word\nrepresentation, max_dims). Then the call() method crops this embedding\nmatrix to the size of the inputs, and it adds it to the inputs. Since we added\nan extra first dimension of size 1 when creating the positional embedding\nmatrix, the rules of broadcasting will ensure that the matrix gets added to\nevery sentence in the inputs:\nclass PositionalEncoding(keras.layers.Layer): \n def __init__(self, max_steps, max_dims, dtype=tf.float32, **kwargs): \n super().__init__(dtype=dtype, **kwargs) \n if max_dims % 2 == 1: max_dims += 1 # max_dims must be even \n p, i = np.meshgrid(np.arange(max_steps), np.arange(max_dims // \n2)) \n pos_emb = np.empty((1, max_steps, max_dims)) \n pos_emb[0, :, ::2] = np.sin(p / 10000**(2 * i / max_dims)).T \n pos_emb[0, :, 1::2] = np.cos(p / 10000**(2 * i / max_dims)).T \n self.positional_embedding = \ntf.constant(pos_emb.astype(self.dtype)) \n def call(self, inputs): \n shape = tf.shape(inputs) \n return inputs + self.positional_embedding[:, :shape[-2], \n:shape[-1]]\nThen we can create the first layers of the Transformer:\nembed_size = 512; max_steps = 500; vocab_size = 10000 \nencoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32) \ndecoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32) \nembeddings = keras.layers.Embedding(vocab_size, embed_size) \nencoder_embeddings = embeddings(encoder_inputs) \ndecoder_embeddings = embeddings(decoder_inputs) \npositional_encoding = PositionalEncoding(max_steps, max_dims=embed_size) \nencoder_in = positional_encoding(encoder_embeddings) \ndecoder_in = positional_encoding(decoder_embeddings)\nNow let’s look deeper into the heart of the Transformer model: the Multi-\nHead Attention layer.\nMulti-Head Attention\nTo understand how a Multi-Head Attention layer works, we must first\nunderstand the Scaled Dot-Product Attention layer, which it is based on.\nLet’s suppose the encoder analyzed the input sentence “They played\nchess,” and it managed to understand that the word “They” is the subject\nand the word “played” is the verb, so it encoded this information in the\nrepresentations of these words. Now suppose the decoder has already\ntranslated the subject, and it thinks that it should translate the verb next.\nFor this, it needs to fetch the verb from the input sentence. This is analog\nto a dictionary lookup: it’s as if the encoder created a dictionary\n{“subject”: “They”, “verb”: “played”, …} and the decoder wanted to look\nup the value that corresponds to the key “verb.” However, the model does\nnot have discrete tokens to represent the keys (like “subject” or “verb”); it\nhas vectorized representations of these concepts (which it learned during\ntraining), so the key it will use for the lookup (called the query) will not\nperfectly match any key in the dictionary. The solution is to compute a\nsimilarity measure between the query and each key in the dictionary, and\nthen use the softmax function to convert these similarity scores to weights\nthat add up to 1. If the key that represents the verb is by far the most\nsimilar to the query, then that key’s weight will be close to 1. Then the\nmodel can compute a weighted sum of the corresponding values, so if the\nweight of the “verb” key is close to 1, then the weighted sum will be very\nclose to the representation of the word “played.” In short, you can think of\nthis whole process as a differentiable dictionary lookup. The similarity\nmeasure used by the Transformer is just the dot product, like in Luong\nattention. In fact, the equation is the same as for Luong attention, except\nfor a scaling factor. The equation is shown in Equation 16-3, in a\nvectorized form.\nEquation 16-3. Scaled Dot-Product Attention\nAttention (Q, K, V) = softmax(\n)V\nQK⊺\n√dkeys\nIn this equation:\nQ is a matrix containing one row per query. Its shape is [n\n,\nd\n], where n\n is the number of queries and d\n is the\nnumber of dimensions of each query and each key.\nK is a matrix containing one row per key. Its shape is [n\n,\nd\n], where n\n is the number of keys and values.\nV is a matrix containing one row per value. Its shape is [n\n,\nd\n], where d\n is the number of each value.\nThe shape of Q K is [n\n, n\n]: it contains one similarity\nscore for each query/key pair. The output of the softmax function\nhas the same shape, but all rows sum up to 1. The final output has\na shape of [n\n, d\n]: there is one row per query, where each\nrow represents the query result (a weighted sum of the values).\nThe scaling factor scales down the similarity scores to avoid\nsaturating the softmax function, which would lead to tiny\ngradients.\nIt is possible to mask out some key/value pairs by adding a very\nlarge negative value to the corresponding similarity scores, just\nbefore computing the softmax. This is useful in the Masked\nMulti-Head Attention layer.\nIn the encoder, this equation is applied to every input sentence in the\nbatch, with Q, K, and V all equal to the list of words in the input sentence\n(so each word in the sentence will be compared to every word in the same\nsentence, including itself). Similarly, in the decoder’s masked attention\nlayer, the equation will be applied to every target sentence in the batch,\nwith Q, K, and V all equal to the list of words in the target sentence, but\nthis time using a mask to prevent any word from comparing itself to words\nlocated after it (at inference time the decoder will only have access to the\nwords it already output, not to future words, so during training we must\nmask out future output tokens). In the upper attention layer of the decoder,\nqueries\nkeys\nqueries\nkeys\nkeys\nkeys\nkeys\nkeys\nvalues\nvalues\n⊺\nqueries\nkeys\nqueries\nvalues\nthe keys K and values V are simply the list of word encodings produced by\nthe encoder, and the queries Q are the list of word encodings produced by\nthe decoder.\nThe keras.layers.Attention layer implements Scaled Dot-Product\nAttention, efficiently applying Equation 16-3 to multiple sentences in a\nbatch. Its inputs are just like Q, K, and V, except with an extra batch\ndimension (the first dimension).\nTIP\nIn TensorFlow, if A and B are tensors with more than two dimensions—say, of shape\n[2, 3, 4, 5] and [2, 3, 5, 6] respectively—then tf.matmul(A, B) will treat these\ntensors as 2 × 3 arrays where each cell contains a matrix, and it will multiply the\ncorresponding matrices: the matrix at the i row and j column in A will be\nmultiplied by the matrix at the i row and j column in B. Since the product of a\n4 × 5 matrix with a 5 × 6 matrix is a 4 × 6 matrix, tf.matmul(A, B) will return an\narray of shape [2, 3, 4, 6].\nIf we ignore the skip connections, the layer normalization layers, the Feed\nForward blocks, and the fact that this is Scaled Dot-Product Attention, not\nexactly Multi-Head Attention, then the rest of the Transformer model can\nbe implemented like this:\nZ = encoder_in \nfor N in range(6): \n Z = keras.layers.Attention(use_scale=True)([Z, Z]) \n \nencoder_outputs = Z \nZ = decoder_in \nfor N in range(6): \n Z = keras.layers.Attention(use_scale=True, causal=True)([Z, Z]) \n Z = keras.layers.Attention(use_scale=True)([Z, encoder_outputs]) \n \noutputs = keras.layers.TimeDistributed( \n keras.layers.Dense(vocab_size, activation=\"softmax\"))(Z)\nth\nth\nth\nth\nThe use_scale=True argument creates an additional parameter that lets\nthe layer learn how to properly downscale the similarity scores. This is a\nbit different from the Transformer model, which always downscales the\nsimilarity scores by the same factor (√dkeys). The causal=True argument\nwhen creating the second attention layer ensures that each output token\nonly attends to previous output tokens, not future ones.\nNow it’s time to look at the final piece of the puzzle: what is a Multi-Head\nAttention layer? Its architecture is shown in Figure 16-10.\nFigure 16-10. Multi-Head Attention layer architecture23\nAs you can see, it is just a bunch of Scaled Dot-Product Attention layers,\neach preceded by a linear transformation of the values, keys, and queries\n(i.e., a time-distributed Dense layer with no activation function). All the\noutputs are simply concatenated, and they go through a final linear\ntransformation (again, time-distributed). But why? What is the intuition\nbehind this architecture? Well, consider the word “played” we discussed\nearlier (in the sentence “They played chess”). The encoder was smart\nenough to encode the fact that it is a verb. But the word representation also\nincludes its position in the text, thanks to the positional encodings, and it\nprobably includes many other features that are useful for its translation,\nsuch as the fact that it is in the past tense. In short, the word representation\nencodes many different characteristics of the word. If we just used a single\nScaled Dot-Product Attention layer, we would only be able to query all of\nthese characteristics in one shot. This is why the Multi-Head Attention\nlayer applies multiple different linear transformations of the values, keys,\nand queries: this allows the model to apply many different projections of\nthe word representation into different subspaces, each focusing on a subset\nof the word’s characteristics. Perhaps one of the linear layers will project\nthe word representation into a subspace where all that remains is the\ninformation that the word is a verb, another linear layer will extract just\nthe fact that it is past tense, and so on. Then the Scaled Dot-Product\nAttention layers implement the lookup phase, and finally we concatenate\nall the results and project them back to the original space.\nAt the time of this writing, there is no Transformer class or\nMultiHeadAttention class available for TensorFlow 2. However, you can\ncheck out TensorFlow’s great tutorial for building a Transformer model for\nlanguage understanding. Moreover, the TF Hub team is currently porting\nseveral Transformer-based modules to TensorFlow 2, and they should be\navailable soon. In the meantime, I hope I have demonstrated that it is not\nthat hard to implement a Transformer yourself, and it is certainly a great\nexercise!\nRecent Innovations in Language Models\nThe year 2018 has been called the “ImageNet moment for NLP”: progress\nwas astounding, with larger and larger LSTM and Transformer-based\narchitectures trained on immense datasets. I highly recommend you check\nout the following papers, all published in 2018:\nThe ELMo paper\n by Matthew Peters introduced Embeddings\nfrom Language Models (ELMo): these are contextualized word\nembeddings learned from the internal states of a deep\nbidirectional language model. For example, the word “queen” will\nnot have the same embedding in “Queen of the United Kingdom”\nand in “queen bee.”\nThe ULMFiT paper\n by Jeremy Howard and Sebastian Ruder\ndemonstrated the effectiveness of unsupervised pretraining for\nNLP tasks: the authors trained an LSTM language model using\nself-supervised learning (i.e., generating the labels automatically\nfrom the data) on a huge text corpus, then they fine-tuned it on\nvarious tasks. Their model outperformed the state of the art on six\ntext classification tasks by a large margin (reducing the error rate\nby 18–24% in most cases). Moreover, they showed that by fine-\ntuning the pretrained model on just 100 labeled examples, they\ncould achieve the same performance as a model trained from\nscratch on 10,000 examples.\nThe GPT paper\n by Alec Radford and other OpenAI researchers\nalso demonstrated the effectiveness of unsupervised pretraining,\nbut this time using a Transformer-like architecture. The authors\npretrained a large but fairly simple architecture composed of a\nstack of 12 Transformer modules (using only Masked Multi-Head\nAttention layers) on a large dataset, once again trained using self-\nsupervised learning. Then they fine-tuned it on various language\ntasks, using only minor adaptations for each task. The tasks were\nquite diverse: they included text classification, entailment\n(whether sentence A entails sentence B),\n similarity (e.g., “Nice\n24\n25\n26\n27\nweather today” is very similar to “It is sunny”), and question\nanswering (given a few paragraphs of text giving some context,\nthe model must answer some multiple-choice questions). Just a\nfew months later, in February 2019, Alec Radford, Jeffrey Wu,\nand other OpenAI researchers published the GPT-2 paper,\n which\nproposed a very similar architecture, but larger still (with over 1.5\nbillion parameters!) and they showed that it could achieve good\nperformance on many tasks without any fine-tuning. This is called\nzero-shot learning (ZSL). A smaller version of the GPT-2 model\n(with “just” 117 million parameters) is available at\nhttps://github.com/openai/gpt-2, along with its pretrained\nweights.\nThe BERT paper\n by Jacob Devlin and other Google researchers\nalso demonstrates the effectiveness of self-supervised pretraining\non a large corpus, using a similar architecture to GPT but non-\nmasked Multi-Head Attention layers (like in the Transformer’s\nencoder). This means that the model is naturally bidirectional;\nhence the B in BERT (Bidirectional Encoder Representations\nfrom Transformers). Most importantly, the authors proposed two\npretraining tasks that explain most of the model’s strength:\nMasked language model (MLM)\nEach word in a sentence has a 15% probability of being\nmasked, and the model is trained to predict the masked words.\nFor example, if the original sentence is “She had fun at the\nbirthday party,” then the model may be given the sentence\n“She fun at the party” and it must predict the\nwords “had” and “birthday” (the other outputs will be\nignored). To be more precise, each selected word has an 80%\nchance of being masked, a 10% chance of being replaced by a\nrandom word (to reduce the discrepancy between pretraining\nand fine-tuning, since the model will not see tokens\nduring fine-tuning), and a 10% chance of being left alone (to\nbias the model toward the correct answer).\n28\n29\nNext sentence prediction (NSP)\nThe model is trained to predict whether two sentences are\nconsecutive or not. For example, it should predict that “The\ndog sleeps” and “It snores loudly” are consecutive sentences,\nwhile “The dog sleeps” and “The Earth orbits the Sun” are not\nconsecutive. This is a challenging task, and it significantly\nimproves the performance of the model when it is fine-tuned\non tasks such as question answering or entailment.\nAs you can see, the main innovations in 2018 and 2019 have been better\nsubword tokenization, shifting from LSTMs to Transformers, and\npretraining universal language models using self-supervised learning, then\nfine-tuning them with very few architectural changes (or none at all).\nThings are moving fast; no one can say what architectures will prevail\nnext year. Today, it’s clearly Transformers, but tomorrow it might be CNNs\n(e.g., check out the 2018 paper\n by Maha Elbayad et al., where the\nresearchers use masked 2D convolutional layers for sequence-to-sequence\ntasks). Or it might even be RNNs, if they make a surprise comeback (e.g.,\ncheck out the 2018 paper\n by Shuai Li et al. that shows that by making\nneurons independent of each other in a given RNN layer, it is possible to\ntrain much deeper RNNs capable of learning much longer sequences).\nIn the next chapter we will discuss how to learn deep representations in an\nunsupervised way using autoencoders, and we will use generative\nadversarial networks (GANs) to produce images and more!\nExercises\n1. What are the pros and cons of using a stateful RNN versus a\nstateless RNN?\n2. Why do people use Encoder–Decoder RNNs rather than plain\nsequence-to-sequence RNNs for automatic translation?\n30\n31\n3. How can you deal with variable-length input sequences? What\nabout variable-length output sequences?\n4. What is beam search and why would you use it? What tool can\nyou use to implement it?\n5. What is an attention mechanism? How does it help?\n6. What is the most important layer in the Transformer architecture?\nWhat is its purpose?\n7. When would you need to use sampled softmax?\n8. Embedded Reber grammars were used by Hochreiter and\nSchmidhuber in their paper about LSTMs. They are artificial\ngrammars that produce strings such as “BPBTSXXVPSEPE.”\nCheck out Jenny Orr’s nice introduction to this topic. Choose a\nparticular embedded Reber grammar (such as the one represented\non Jenny Orr’s page), then train an RNN to identify whether a\nstring respects that grammar or not. You will first need to write a\nfunction capable of generating a training batch containing about\n50% strings that respect the grammar, and 50% that don’t.\n9. Train an Encoder–Decoder model that can convert a date string\nfrom one format to another (e.g., from “April 22, 2019” to “2019-\n04-22”).\n10. Go through TensorFlow’s Neural Machine Translation with\nAttention tutorial.\n11. Use one of the recent language models (e.g., BERT) to generate\nmore convincing Shakespearean text.\nSolutions to these exercises are available in Appendix A.\n1 Alan Turing, “Computing Machinery and Intelligence,” Mind 49 (1950): 433–460.\n2 Of course, the word chatbot came much later. Turing called his test the imitation game:\nmachine A and human B chat with human interrogator C via text messages; the interrogator\nasks questions to figure out which one is the machine (A or B). The machine passes the test\nif it can fool the interrogator, while the human B must try to help the interrogator.\n3 By definition, a stationary time series’s mean, variance, and autocorrelations (i.e.,\ncorrelations between values in the time series separated by a given interval) do not change\nover time. This is quite restrictive; for example, it excludes time series with trends or\ncyclical patterns. RNNs are more tolerant in that they can learn trends and cyclical patterns.\n4 Taku Kudo, “Subword Regularization: Improving Neural Network Translation Models\nwith Multiple Subword Candidates,” arXiv preprint arXiv:1804.10959 (2018).\n5 Taku Kudo and John Richardson, “SentencePiece: A Simple and Language Independent\nSubword Tokenizer and Detokenizer for Neural Text Processing,” arXiv preprint\narXiv:1808.06226 (2018).\n6 Rico Sennrich et al., “Neural Machine Translation of Rare Words with Subword Units,”\nProceedings of the 54th Annual Meeting of the Association for Computational Linguistics 1\n(2016): 1715–1725.\n7 Yonghui Wu et al., “Google’s Neural Machine Translation System: Bridging the Gap\nBetween Human and Machine Translation,” arXiv preprint arXiv:1609.08144 (2016).\n8 Their ID is 0 only because they are the most frequent “words” in the dataset. It would\nprobably be a good idea to ensure that the padding tokens are always encoded as 0, even if\nthey are not the most frequent.\n9 To be precise, the sentence embedding is equal to the mean word embedding multiplied\nby the square root of the number of words in the sentence. This compensates for the fact\nthat the mean of n vectors gets shorter as n grows.\n10 Ilya Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” arXiv\npreprint arXiv:1409.3215 (2014).\n11 Sébastien Jean et al., “On Using Very Large Target Vocabulary for Neural Machine\nTranslation,” Proceedings of the 53rd Annual Meeting of the Association for\nComputational Linguistics and the 7th International Joint Conference on Natural\nLanguage Processing of the Asian Federation of Natural Language Processing 1 (2015):\n1–10.\n12 Samy Bengio et al., “Scheduled Sampling for Sequence Prediction with Recurrent Neural\nNetworks,” arXiv preprint arXiv:1506.03099 (2015).\n13 Dzmitry Bahdanau et al., “Neural Machine Translation by Jointly Learning to Align and\nTranslate,” arXiv preprint arXiv:1409.0473 (2014).\n14 The most common metric used in NMT is the BiLingual Evaluation Understudy (BLEU)\nscore, which compares each translation produced by the model with several good\ntranslations produced by humans: it counts the number of n-grams (sequences of n words)\nthat appear in any of the target translations and adjusts the score to take into account the\nfrequency of the produced n-grams in the target translations.\n15 Recall that a time-distributed Dense layer is equivalent to a regular Dense layer that you\napply independently at each time step (only much faster).\n16 Minh-Thang Luong et al., “Effective Approaches to Attention-Based Neural Machine\nTranslation,” Proceedings of the 2015 Conference on Empirical Methods in Natural\nLanguage Processing (2015): 1412–1421.\n17 Kelvin Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual\nAttention,” Proceedings of the 32nd International Conference on Machine Learning\n(2015): 2048–2057.\n18 This is a part of figure 3 from the paper. It is reproduced with the kind authorization of the\nauthors.\n19 Marco Tulio Ribeiro et al., “‘Why Should I Trust You?’: Explaining the Predictions of Any\nClassifier,” Proceedings of the 22nd ACM SIGKDD International Conference on\nKnowledge Discovery and Data Mining (2016): 1135–1144.\n20 Ashish Vaswani et al., “Attention Is All You Need,” Proceedings of the 31st International\nConference on Neural Information Processing Systems (2017): 6000–6010.\n21 Since the Transformer uses time-distributed Dense layers, you could argue that it uses 1D\nconvolutional layers with a kernel size of 1.\n22 This is figure 1 from the paper, reproduced with the kind authorization of the authors.\n23 This is the right part of figure 2 from the paper, reproduced with the kind authorization of\nthe authors.\n24 Matthew Peters et al., “Deep Contextualized Word Representations,” Proceedings of the\n2018 Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies 1 (2018): 2227–2237.\n25 Jeremy Howard and Sebastian Ruder, “Universal Language Model Fine-Tuning for Text\nClassification,” Proceedings of the 56th Annual Meeting of the Association for\nComputational Linguistics 1 (2018): 328–339.\n26 Alec Radford et al., “Improving Language Understanding by Generative Pre-Training”\n(2018).\n27 For example, the sentence “Jane had a lot of fun at her friend’s birthday party” entails\n“Jane enjoyed the party,” but it is contradicted by “Everyone hated the party” and it is\nunrelated to “The Earth is flat.”\n28 Alec Radford et al., “Language Models Are Unsupervised Multitask Learners” (2019).\n29 Jacob Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language\nUnderstanding,” Proceedings of the 2018 Conference of the North American Chapter of the\nAssociation for Computational Linguistics: Human Language Technologies 1 (2019).\n30 Maha Elbayad et al., “Pervasive Attention: 2D Convolutional Neural Networks for\nSequence-to-Sequence Prediction,” arXiv preprint arXiv:1808.03867 (2018).\n31 Shuai Li et al., “Independently Recurrent Neural Network (IndRNN): Building a Longer\nand Deeper RNN,” Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition (2018): 5457–5466." + }, + { + "chapter": 17, + "text": "Chapter 17. Representation\nLearning and Generative\nLearning Using Autoencoders\nand GANs\nAutoencoders are artificial neural networks capable of learning dense\nrepresentations of the input data, called latent representations or codings,\nwithout any supervision (i.e., the training set is unlabeled). These codings\ntypically have a much lower dimensionality than the input data, making\nautoencoders useful for dimensionality reduction (see Chapter 8),\nespecially for visualization purposes. Autoencoders also act as feature\ndetectors, and they can be used for unsupervised pretraining of deep neural\nnetworks (as we discussed in Chapter 11). Lastly, some autoencoders are\ngenerative models: they are capable of randomly generating new data that\nlooks very similar to the training data. For example, you could train an\nautoencoder on pictures of faces, and it would then be able to generate new\nfaces. However, the generated images are usually fuzzy and not entirely\nrealistic.\nIn contrast, faces generated by generative adversarial networks (GANs)\nare now so convincing that it is hard to believe that the people they\nrepresent do not exist. You can judge so for yourself by visiting\nhttps://thispersondoesnotexist.com/, a website that shows faces generated\nby a recent GAN architecture called StyleGAN (you can also check out\nhttps://thisrentaldoesnotexist.com/ to see some generated Airbnb\nbedrooms). GANs are now widely used for super resolution (increasing the\nresolution of an image), colorization, powerful image editing (e.g.,\nreplacing photo bombers with realistic background), turning a simple\nsketch into a photorealistic image, predicting the next frames in a video,\naugmenting a dataset (to train other models), generating other types of\ndata (such as text, audio, and time series), identifying the weaknesses in\nother models and strengthening them, and more.\nAutoencoders and GANs are both unsupervised, they both learn dense\nrepresentations, they can both be used as generative models, and they have\nmany similar applications. However, they work very differently:\nAutoencoders simply learn to copy their inputs to their outputs.\nThis may sound like a trivial task, but we will see that\nconstraining the network in various ways can make it rather\ndifficult. For example, you can limit the size of the latent\nrepresentations, or you can add noise to the inputs and train the\nnetwork to recover the original inputs. These constraints prevent\nthe autoencoder from trivially copying the inputs directly to the\noutputs, which forces it to learn efficient ways of representing the\ndata. In short, the codings are byproducts of the autoencoder\nlearning the identity function under some constraints.\nGANs are composed of two neural networks: a generator that\ntries to generate data that looks similar to the training data, and a\ndiscriminator that tries to tell real data from fake data. This\narchitecture is very original in Deep Learning in that the\ngenerator and the discriminator compete against each other during\ntraining: the generator is often compared to a criminal trying to\nmake realistic counterfeit money, while the discriminator is like\nthe police investigator trying to tell real money from fake.\nAdversarial training (training competing neural networks) is\nwidely considered as one of the most important ideas in recent\nyears. In 2016, Yann LeCun even said that it was “the most\ninteresting idea in the last 10 years in Machine Learning.”\nIn this chapter we will start by exploring in more depth how autoencoders\nwork and how to use them for dimensionality reduction, feature extraction,\nunsupervised pretraining, or as generative models. This will naturally lead\nus to GANs. We will start by building a simple GAN to generate fake\nimages, but we will see that training is often quite difficult. We will\ndiscuss the main difficulties you will encounter with adversarial training,\nas well as some of the main techniques to work around these difficulties.\nLet’s start with autoencoders!\nEfficient Data Representations\nWhich of the following number sequences do you find the easiest to\nmemorize?\n40, 27, 25, 36, 81, 57, 10, 73, 19, 68\n50, 48, 46, 44, 42, 40, 38, 36, 34, 32, 30, 28, 26, 24, 22, 20, 18, 16,\n14\nAt first glance, it would seem that the first sequence should be easier,\nsince it is much shorter. However, if you look carefully at the second\nsequence, you will notice that it is just the list of even numbers from 50\ndown to 14. Once you notice this pattern, the second sequence becomes\nmuch easier to memorize than the first because you only need to\nremember the pattern (i.e., decreasing even numbers) and the starting and\nending numbers (i.e., 50 and 14). Note that if you could quickly and easily\nmemorize very long sequences, you would not care much about the\nexistence of a pattern in the second sequence. You would just learn every\nnumber by heart, and that would be that. The fact that it is hard to\nmemorize long sequences is what makes it useful to recognize patterns,\nand hopefully this clarifies why constraining an autoencoder during\ntraining pushes it to discover and exploit patterns in the data.\nThe relationship between memory, perception, and pattern matching was\nfamously studied by William Chase and Herbert Simon in the early\n1970s. They observed that expert chess players were able to memorize\nthe positions of all the pieces in a game by looking at the board for just\nfive seconds, a task that most people would find impossible. However, this\nwas only the case when the pieces were placed in realistic positions (from\nactual games), not when the pieces were placed randomly. Chess experts\ndon’t have a much better memory than you and I; they just see chess\n1\npatterns more easily, thanks to their experience with the game. Noticing\npatterns helps them store information efficiently.\nJust like the chess players in this memory experiment, an autoencoder\nlooks at the inputs, converts them to an efficient latent representation, and\nthen spits out something that (hopefully) looks very close to the inputs. An\nautoencoder is always composed of two parts: an encoder (or recognition\nnetwork) that converts the inputs to a latent representation, followed by a\ndecoder (or generative network) that converts the internal representation\nto the outputs (see Figure 17-1).\nFigure 17-1. The chess memory experiment (left) and a simple autoencoder (right)\nAs you can see, an autoencoder typically has the same architecture as a\nMulti-Layer Perceptron (MLP; see Chapter 10), except that the number of\nneurons in the output layer must be equal to the number of inputs. In this\nexample, there is just one hidden layer composed of two neurons (the\nencoder), and one output layer composed of three neurons (the decoder).\nThe outputs are often called the reconstructions because the autoencoder\ntries to reconstruct the inputs, and the cost function contains a\nreconstruction loss that penalizes the model when the reconstructions are\ndifferent from the inputs.\nBecause the internal representation has a lower dimensionality than the\ninput data (it is 2D instead of 3D), the autoencoder is said to be\nundercomplete. An undercomplete autoencoder cannot trivially copy its\ninputs to the codings, yet it must find a way to output a copy of its inputs.\nIt is forced to learn the most important features in the input data (and drop\nthe unimportant ones).\nLet’s see how to implement a very simple undercomplete autoencoder for\ndimensionality reduction.\nPerforming PCA with an Undercomplete\nLinear Autoencoder\nIf the autoencoder uses only linear activations and the cost function is the\nmean squared error (MSE), then it ends up performing Principal\nComponent Analysis (PCA; see Chapter 8).\nThe following code builds a simple linear autoencoder to perform PCA on\na 3D dataset, projecting it to 2D:\nfrom tensorflow import keras \n \nencoder = keras.models.Sequential([keras.layers.Dense(2, input_shape=\n[3])]) \ndecoder = keras.models.Sequential([keras.layers.Dense(3, input_shape=\n[2])]) \nautoencoder = keras.models.Sequential([encoder, decoder]) \n \nautoencoder.compile(loss=\"mse\", optimizer=keras.optimizers.SGD(lr=0.1))\nThis code is really not very different from all the MLPs we built in past\nchapters, but there are a few things to note:\nWe organized the autoencoder into two subcomponents: the\nencoder and the decoder. Both are regular Sequential models\nwith a single Dense layer each, and the autoencoder is a\nSequential model containing the encoder followed by the\ndecoder (remember that a model can be used as a layer in another\nmodel).\nThe autoencoder’s number of outputs is equal to the number of\ninputs (i.e., 3).\nTo perform simple PCA, we do not use any activation function\n(i.e., all neurons are linear), and the cost function is the MSE. We\nwill see more complex autoencoders shortly.\nNow let’s train the model on a simple generated 3D dataset and use it to\nencode that same dataset (i.e., project it to 2D):\nhistory = autoencoder.fit(X_train, X_train, epochs=20) \ncodings = encoder.predict(X_train)\nNote that the same dataset, X_train, is used as both the inputs and the\ntargets. Figure 17-2 shows the original 3D dataset (on the left) and the\noutput of the autoencoder’s hidden layer (i.e., the coding layer, on the\nright). As you can see, the autoencoder found the best 2D plane to project\nthe data onto, preserving as much variance in the data as it could (just like\nPCA).\nFigure 17-2. PCA performed by an undercomplete linear autoencoder\nNOTE\nYou can think of autoencoders as a form of self-supervised learning (i.e., using a\nsupervised learning technique with automatically generated labels, in this case\nsimply equal to the inputs).\nStacked Autoencoders\nJust like other neural networks we have discussed, autoencoders can have\nmultiple hidden layers. In this case they are called stacked autoencoders\n(or deep autoencoders). Adding more layers helps the autoencoder learn\nmore complex codings. That said, one must be careful not to make the\nautoencoder too powerful. Imagine an encoder so powerful that it just\nlearns to map each input to a single arbitrary number (and the decoder\nlearns the reverse mapping). Obviously such an autoencoder will\nreconstruct the training data perfectly, but it will not have learned any\nuseful data representation in the process (and it is unlikely to generalize\nwell to new instances).\nThe architecture of a stacked autoencoder is typically symmetrical with\nregard to the central hidden layer (the coding layer). To put it simply, it\nlooks like a sandwich. For example, an autoencoder for MNIST\n(introduced in Chapter 3) may have 784 inputs, followed by a hidden layer\nwith 100 neurons, then a central hidden layer of 30 neurons, then another\nhidden layer with 100 neurons, and an output layer with 784 neurons. This\nstacked autoencoder is represented in Figure 17-3.\nFigure 17-3. Stacked autoencoder\nImplementing a Stacked Autoencoder Using Keras\nYou can implement a stacked autoencoder very much like a regular deep\nMLP. In particular, the same techniques we used in Chapter 11 for training\ndeep nets can be applied. For example, the following code builds a stacked\nautoencoder for Fashion MNIST (loaded and normalized as in Chapter 10),\nusing the SELU activation function:\nstacked_encoder = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dense(100, activation=\"selu\"), \n keras.layers.Dense(30, activation=\"selu\"), \n]) \nstacked_decoder = keras.models.Sequential([ \n keras.layers.Dense(100, activation=\"selu\", input_shape=[30]), \n keras.layers.Dense(28 * 28, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \nstacked_ae = keras.models.Sequential([stacked_encoder, stacked_decoder]) \nstacked_ae.compile(loss=\"binary_crossentropy\", \n optimizer=keras.optimizers.SGD(lr=1.5)) \nhistory = stacked_ae.fit(X_train, X_train, epochs=10, \n validation_data=[X_valid, X_valid])\nLet’s go through this code:\nJust like earlier, we split the autoencoder model into two\nsubmodels: the encoder and the decoder.\nThe encoder takes 28 × 28–pixel grayscale images, flattens them\nso that each image is represented as a vector of size 784, then\nprocesses these vectors through two Dense layers of diminishing\nsizes (100 units then 30 units), both using the SELU activation\nfunction (you may want to add LeCun normal initialization as\nwell, but the network is not very deep so it won’t make a big\ndifference). For each input image, the encoder outputs a vector of\nsize 30.\nThe decoder takes codings of size 30 (output by the encoder) and\nprocesses them through two Dense layers of increasing sizes (100\nunits then 784 units), and it reshapes the final vectors into 28 × 28\narrays so the decoder’s outputs have the same shape as the\nencoder’s inputs.\nWhen compiling the stacked autoencoder, we use the binary\ncross-entropy loss instead of the mean squared error. We are\ntreating the reconstruction task as a multilabel binary\nclassification problem: each pixel intensity represents the\nprobability that the pixel should be black. Framing it this way\n(rather than as a regression problem) tends to make the model\nconverge faster.\nFinally, we train the model using X_train as both the inputs and\nthe targets (and similarly, we use X_valid as both the validation\ninputs and targets).\nVisualizing the Reconstructions\n2\nOne way to ensure that an autoencoder is properly trained is to compare\nthe inputs and the outputs: the differences should not be too significant.\nLet’s plot a few images from the validation set, as well as their\nreconstructions:\ndef plot_image(image): \n plt.imshow(image, cmap=\"binary\") \n plt.axis(\"off\") \n \ndef show_reconstructions(model, n_images=5): \n reconstructions = model.predict(X_valid[:n_images]) \n fig = plt.figure(figsize=(n_images * 1.5, 3)) \n for image_index in range(n_images): \n plt.subplot(2, n_images, 1 + image_index) \n plot_image(X_valid[image_index]) \n plt.subplot(2, n_images, 1 + n_images + image_index) \n plot_image(reconstructions[image_index]) \n \nshow_reconstructions(stacked_ae)\nFigure 17-4 shows the resulting images.\nFigure 17-4. Original images (top) and their reconstructions (bottom)\nThe reconstructions are recognizable, but a bit too lossy. We may need to\ntrain the model for longer, or make the encoder and decoder deeper, or\nmake the codings larger. But if we make the network too powerful, it will\nmanage to make perfect reconstructions without having learned any useful\npatterns in the data. For now, let’s go with this model.\nVisualizing the Fashion MNIST Dataset\nNow that we have trained a stacked autoencoder, we can use it to reduce\nthe dataset’s dimensionality. For visualization, this does not give great\nresults compared to other dimensionality reduction algorithms (such as\nthose we discussed in Chapter 8), but one big advantage of autoencoders is\nthat they can handle large datasets, with many instances and many\nfeatures. So one strategy is to use an autoencoder to reduce the\ndimensionality down to a reasonable level, then use another\ndimensionality reduction algorithm for visualization. Let’s use this\nstrategy to visualize Fashion MNIST. First, we use the encoder from our\nstacked autoencoder to reduce the dimensionality down to 30, then we use\nScikit-Learn’s implementation of the t-SNE algorithm to reduce the\ndimensionality down to 2 for visualization:\nfrom sklearn.manifold import TSNE \n \nX_valid_compressed = stacked_encoder.predict(X_valid) \ntsne = TSNE() \nX_valid_2D = tsne.fit_transform(X_valid_compressed)\nNow we can plot the dataset:\nplt.scatter(X_valid_2D[:, 0], X_valid_2D[:, 1], c=y_valid, s=10, \ncmap=\"tab10\")\nFigure 17-5 shows the resulting scatterplot (beautified a bit by displaying\nsome of the images). The t-SNE algorithm identified several clusters\nwhich match the classes reasonably well (each class is represented with a\ndifferent color).\nFigure 17-5. Fashion MNIST visualization using an autoencoder followed by t-SNE\nSo, autoencoders can be used for dimensionality reduction. Another\napplication is for unsupervised pretraining.\nUnsupervised Pretraining Using Stacked Autoencoders\nAs we discussed in Chapter 11, if you are tackling a complex supervised\ntask but you do not have a lot of labeled training data, one solution is to\nfind a neural network that performs a similar task and reuse its lower\nlayers. This makes it possible to train a high-performance model using\nlittle training data because your neural network won’t have to learn all the\nlow-level features; it will just reuse the feature detectors learned by the\nexisting network.\nSimilarly, if you have a large dataset but most of it is unlabeled, you can\nfirst train a stacked autoencoder using all the data, then reuse the lower\nlayers to create a neural network for your actual task and train it using the\nlabeled data. For example, Figure 17-6 shows how to use a stacked\nautoencoder to perform unsupervised pretraining for a classification\nneural network. When training the classifier, if you really don’t have much\nlabeled training data, you may want to freeze the pretrained layers (at least\nthe lower ones).\nFigure 17-6. Unsupervised pretraining using autoencoders\nNOTE\nHaving plenty of unlabeled data and little labeled data is common. Building a large\nunlabeled dataset is often cheap (e.g., a simple script can download millions of\nimages off the internet), but labeling those images (e.g., classifying them as cute or\nnot) can usually be done reliably only by humans. Labeling instances is time-\nconsuming and costly, so it’s normal to have only a few thousand human-labeled\ninstances.\nThere is nothing special about the implementation: just train an\nautoencoder using all the training data (labeled plus unlabeled), then reuse\nits encoder layers to create a new neural network (see the exercises at the\nend of this chapter for an example).\nNext, let’s look at a few techniques for training stacked autoencoders.\nTying Weights\nWhen an autoencoder is neatly symmetrical, like the one we just built, a\ncommon technique is to tie the weights of the decoder layers to the\nweights of the encoder layers. This halves the number of weights in the\nmodel, speeding up training and limiting the risk of overfitting.\nSpecifically, if the autoencoder has a total of N layers (not counting the\ninput layer), and W represents the connection weights of the L layer\n(e.g., layer 1 is the first hidden layer, layer \n is the coding layer, and\nlayer N is the output layer), then the decoder layer weights can be defined\nsimply as: W\n = W\n (with L = 1, 2, ⋯,\n).\nTo tie weights between layers using Keras, let’s define a custom layer:\nclass DenseTranspose(keras.layers.Layer): \n def __init__(self, dense, activation=None, **kwargs): \n self.dense = dense \n self.activation = keras.activations.get(activation) \n super().__init__(**kwargs) \n def build(self, batch_input_shape): \n self.biases = self.add_weight(name=\"bias\", initializer=\"zeros\", \n shape=[self.dense.input_shape[-1]]) \n super().build(batch_input_shape) \n def call(self, inputs): \n z = tf.matmul(inputs, self.dense.weights[0], transpose_b=True) \n return self.activation(z + self.biases)\nThis custom layer acts like a regular Dense layer, but it uses another\nDense layer’s weights, transposed (setting transpose_b=True is\nequivalent to transposing the second argument, but it’s more efficient as it\nperforms the transposition on the fly within the matmul() operation).\nHowever, it uses its own bias vector. Next, we can build a new stacked\nL\nth\nN\n2\nN–L+1\nL⊺\nN\n2\nautoencoder, much like the previous one, but with the decoder’s Dense\nlayers tied to the encoder’s Dense layers:\ndense_1 = keras.layers.Dense(100, activation=\"selu\") \ndense_2 = keras.layers.Dense(30, activation=\"selu\") \n \ntied_encoder = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n dense_1, \n dense_2 \n]) \n \ntied_decoder = keras.models.Sequential([ \n DenseTranspose(dense_2, activation=\"selu\"), \n DenseTranspose(dense_1, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \n \ntied_ae = keras.models.Sequential([tied_encoder, tied_decoder])\nThis model achieves a very slightly lower reconstruction error than the\nprevious model, with almost half the number of parameters.\nTraining One Autoencoder at a Time\nRather than training the whole stacked autoencoder in one go like we just\ndid, it is possible to train one shallow autoencoder at a time, then stack all\nof them into a single stacked autoencoder (hence the name), as shown in\nFigure 17-7. This technique is not used as much these days, but you may\nstill run into papers that talk about “greedy layerwise training,” so it’s\ngood to know what it means.\nFigure 17-7. Training one autoencoder at a time\nDuring the first phase of training, the first autoencoder learns to\nreconstruct the inputs. Then we encode the whole training set using this\nfirst autoencoder, and this gives us a new (compressed) training set. We\nthen train a second autoencoder on this new dataset. This is the second\nphase of training. Finally, we build a big sandwich using all these\nautoencoders, as shown in Figure 17-7 (i.e., we first stack the hidden\nlayers of each autoencoder, then the output layers in reverse order). This\ngives us the final stacked autoencoder (see the “Training One Autoencoder\nat a Time” section in the notebook for an implementation). We could\neasily train more autoencoders this way, building a very deep stacked\nautoencoder.\nAs we discussed earlier, one of the triggers of the current tsunami of\ninterest in Deep Learning was the discovery in 2006 by Geoffrey Hinton et\nal. that deep neural networks can be pretrained in an unsupervised fashion,\nusing this greedy layerwise approach. They used restricted Boltzmann\nmachines (RBMs; see Appendix E) for this purpose, but in 2007 Yoshua\nBengio et al. showed that autoencoders worked just as well. For several\nyears this was the only efficient way to train deep nets, until many of the\ntechniques introduced in Chapter 11 made it possible to just train a deep\nnet in one shot.\n3\nAutoencoders are not limited to dense networks: you can also build\nconvolutional autoencoders, or even recurrent autoencoders. Let’s look at\nthese now.\nConvolutional Autoencoders\nIf you are dealing with images, then the autoencoders we have seen so far\nwill not work well (unless the images are very small): as we saw in\nChapter 14, convolutional neural networks are far better suited than dense\nnetworks to work with images. So if you want to build an autoencoder for\nimages (e.g., for unsupervised pretraining or dimensionality reduction),\nyou will need to build a convolutional autoencoder. The encoder is a\nregular CNN composed of convolutional layers and pooling layers. It\ntypically reduces the spatial dimensionality of the inputs (i.e., height and\nwidth) while increasing the depth (i.e., the number of feature maps). The\ndecoder must do the reverse (upscale the image and reduce its depth back\nto the original dimensions), and for this you can use transpose\nconvolutional layers (alternatively, you could combine upsampling layers\nwith convolutional layers). Here is a simple convolutional autoencoder for\nFashion MNIST:\nconv_encoder = keras.models.Sequential([ \n keras.layers.Reshape([28, 28, 1], input_shape=[28, 28]), \n keras.layers.Conv2D(16, kernel_size=3, padding=\"same\", \nactivation=\"selu\"), \n keras.layers.MaxPool2D(pool_size=2), \n keras.layers.Conv2D(32, kernel_size=3, padding=\"same\", \nactivation=\"selu\"), \n keras.layers.MaxPool2D(pool_size=2), \n keras.layers.Conv2D(64, kernel_size=3, padding=\"same\", \nactivation=\"selu\"), \n keras.layers.MaxPool2D(pool_size=2) \n]) \nconv_decoder = keras.models.Sequential([ \n keras.layers.Conv2DTranspose(32, kernel_size=3, strides=2, \npadding=\"valid\", \n activation=\"selu\", \n input_shape=[3, 3, 64]), \n keras.layers.Conv2DTranspose(16, kernel_size=3, strides=2, \n4\npadding=\"same\", \n activation=\"selu\"), \n keras.layers.Conv2DTranspose(1, kernel_size=3, strides=2, \npadding=\"same\", \n activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \nconv_ae = keras.models.Sequential([conv_encoder, conv_decoder])\nRecurrent Autoencoders\nIf you want to build an autoencoder for sequences, such as time series or\ntext (e.g., for unsupervised learning or dimensionality reduction), then\nrecurrent neural networks (see Chapter 15) may be better suited than dense\nnetworks. Building a recurrent autoencoder is straightforward: the\nencoder is typically a sequence-to-vector RNN which compresses the input\nsequence down to a single vector. The decoder is a vector-to-sequence\nRNN that does the reverse:\nrecurrent_encoder = keras.models.Sequential([ \n keras.layers.LSTM(100, return_sequences=True, input_shape=[None, \n28]), \n keras.layers.LSTM(30) \n]) \nrecurrent_decoder = keras.models.Sequential([ \n keras.layers.RepeatVector(28, input_shape=[30]), \n keras.layers.LSTM(100, return_sequences=True), \n keras.layers.TimeDistributed(keras.layers.Dense(28, \nactivation=\"sigmoid\")) \n]) \nrecurrent_ae = keras.models.Sequential([recurrent_encoder, \nrecurrent_decoder])\nThis recurrent autoencoder can process sequences of any length, with 28\ndimensions per time step. Conveniently, this means it can process Fashion\nMNIST images by treating each image as a sequence of rows: at each time\nstep, the RNN will process a single row of 28 pixels. Obviously, you could\nuse a recurrent autoencoder for any kind of sequence. Note that we use a\nRepeatVector layer as the first layer of the decoder, to ensure that its\ninput vector gets fed to the decoder at each time step.\nOK, let’s step back for a second. So far we have seen various kinds of\nautoencoders (basic, stacked, convolutional, and recurrent), and we have\nlooked at how to train them (either in one shot or layer by layer). We also\nlooked at a couple applications: data visualization and unsupervised\npretraining.\nUp to now, in order to force the autoencoder to learn interesting features,\nwe have limited the size of the coding layer, making it undercomplete.\nThere are actually many other kinds of constraints that can be used,\nincluding ones that allow the coding layer to be just as large as the inputs,\nor even larger, resulting in an overcomplete autoencoder. Let’s look at\nsome of those approaches now.\nDenoising Autoencoders\nAnother way to force the autoencoder to learn useful features is to add\nnoise to its inputs, training it to recover the original, noise-free inputs.\nThis idea has been around since the 1980s (e.g., it is mentioned in Yann\nLeCun’s 1987 master’s thesis). In a 2008 paper, Pascal Vincent et al.\nshowed that autoencoders could also be used for feature extraction. In a\n2010 paper, Vincent et al. introduced stacked denoising autoencoders.\nThe noise can be pure Gaussian noise added to the inputs, or it can be\nrandomly switched-off inputs, just like in dropout (introduced in\nChapter 11). Figure 17-8 shows both options.\n5\n6\nFigure 17-8. Denoising autoencoders, with Gaussian noise (left) or dropout (right)\nThe implementation is straightforward: it is a regular stacked autoencoder\nwith an additional Dropout layer applied to the encoder’s inputs (or you\ncould use a GaussianNoise layer instead). Recall that the Dropout layer is\nonly active during training (and so is the GaussianNoise layer):\ndropout_encoder = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dropout(0.5), \n keras.layers.Dense(100, activation=\"selu\"), \n keras.layers.Dense(30, activation=\"selu\") \n]) \ndropout_decoder = keras.models.Sequential([ \n keras.layers.Dense(100, activation=\"selu\", input_shape=[30]), \n keras.layers.Dense(28 * 28, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \ndropout_ae = keras.models.Sequential([dropout_encoder, dropout_decoder])\nFigure 17-9 shows a few noisy images (with half the pixels turned off),\nand the images reconstructed by the dropout-based denoising autoencoder.\nNotice how the autoencoder guesses details that are actually not in the\ninput, such as the top of the white shirt (bottom row, fourth image). As you\ncan see, not only can denoising autoencoders be used for data visualization\nor unsupervised pretraining, like the other autoencoders we’ve discussed\nso far, but they can also be used quite simply and efficiently to remove\nnoise from images.\nFigure 17-9. Noisy images (top) and their reconstructions (bottom)\nSparse Autoencoders\nAnother kind of constraint that often leads to good feature extraction is\nsparsity: by adding an appropriate term to the cost function, the\nautoencoder is pushed to reduce the number of active neurons in the\ncoding layer. For example, it may be pushed to have on average only 5%\nsignificantly active neurons in the coding layer. This forces the\nautoencoder to represent each input as a combination of a small number of\nactivations. As a result, each neuron in the coding layer typically ends up\nrepresenting a useful feature (if you could speak only a few words per\nmonth, you would probably try to make them worth listening to).\nA simple approach is to use the sigmoid activation function in the coding\nlayer (to constrain the codings to values between 0 and 1), use a large\ncoding layer (e.g., with 300 units), and add some ℓ regularization to the\ncoding layer’s activations (the decoder is just a regular decoder):\n1\nsparse_l1_encoder = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dense(100, activation=\"selu\"), \n keras.layers.Dense(300, activation=\"sigmoid\"), \n keras.layers.ActivityRegularization(l1=1e-3) \n]) \nsparse_l1_decoder = keras.models.Sequential([ \n keras.layers.Dense(100, activation=\"selu\", input_shape=[300]), \n keras.layers.Dense(28 * 28, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \nsparse_l1_ae = keras.models.Sequential([sparse_l1_encoder, \nsparse_l1_decoder])\nThis ActivityRegularization layer just returns its inputs, but as a side\neffect it adds a training loss equal to the sum of absolute values of its\ninputs (this layer only has an effect during training). Equivalently, you\ncould remove the ActivityRegularization layer and set\nactivity_regularizer=keras.regularizers.l1(1e-3) in the previous\nlayer. This penalty will encourage the neural network to produce codings\nclose to 0, but since it will also be penalized if it does not reconstruct the\ninputs correctly, it will have to output at least a few nonzero values. Using\nthe ℓ norm rather than the ℓ norm will push the neural network to\npreserve the most important codings while eliminating the ones that are\nnot needed for the input image (rather than just reducing all codings).\nAnother approach, which often yields better results, is to measure the\nactual sparsity of the coding layer at each training iteration, and penalize\nthe model when the measured sparsity differs from a target sparsity. We do\nso by computing the average activation of each neuron in the coding layer,\nover the whole training batch. The batch size must not be too small, or else\nthe mean will not be accurate.\nOnce we have the mean activation per neuron, we want to penalize the\nneurons that are too active, or not active enough, by adding a sparsity loss\nto the cost function. For example, if we measure that a neuron has an\naverage activation of 0.3, but the target sparsity is 0.1, it must be\npenalized to activate less. One approach could be simply adding the\nsquared error (0.3 – 0.1) to the cost function, but in practice a better\n1\n2\n2\napproach is to use the Kullback–Leibler (KL) divergence (briefly\ndiscussed in Chapter 4), which has much stronger gradients than the mean\nsquared error, as you can see in Figure 17-10.\nFigure 17-10. Sparsity loss\nGiven two discrete probability distributions P and Q, the KL divergence\nbetween these distributions, noted D\n(P ∥ Q), can be computed using\nEquation 17-1.\nEquation 17-1. Kullback–Leibler divergence\nDKL (P ∥Q) = ∑\ni\nP (i) log\nIn our case, we want to measure the divergence between the target\nprobability p that a neuron in the coding layer will activate and the actual\nprobability q (i.e., the mean activation over the training batch). So the KL\ndivergence simplifies to Equation 17-2.\nEquation 17-2. KL divergence between the target sparsity p and the actual sparsity q\nKL\nP(i)\nQ(i)\nDKL (p ∥q) = p log\n+ (1 −p) log\nOnce we have computed the sparsity loss for each neuron in the coding\nlayer, we sum up these losses and add the result to the cost function. In\norder to control the relative importance of the sparsity loss and the\nreconstruction loss, we can multiply the sparsity loss by a sparsity weight\nhyperparameter. If this weight is too high, the model will stick closely to\nthe target sparsity, but it may not reconstruct the inputs properly, making\nthe model useless. Conversely, if it is too low, the model will mostly\nignore the sparsity objective and will not learn any interesting features.\nWe now have all we need to implement a sparse autoencoder based on the\nKL divergence. First, let’s create a custom regularizer to apply KL\ndivergence regularization:\nK = keras.backend \nkl_divergence = keras.losses.kullback_leibler_divergence \n \nclass KLDivergenceRegularizer(keras.regularizers.Regularizer): \n def __init__(self, weight, target=0.1): \n self.weight = weight \n self.target = target \n def __call__(self, inputs): \n mean_activities = K.mean(inputs, axis=0) \n return self.weight * ( \n kl_divergence(self.target, mean_activities) + \n kl_divergence(1. - self.target, 1. - mean_activities))\nNow we can build the sparse autoencoder, using the\nKLDivergenceRegularizer for the coding layer’s activations:\nkld_reg = KLDivergenceRegularizer(weight=0.05, target=0.1) \nsparse_kl_encoder = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dense(100, activation=\"selu\"), \n keras.layers.Dense(300, activation=\"sigmoid\", \nactivity_regularizer=kld_reg) \n]) \nsparse_kl_decoder = keras.models.Sequential([ \n keras.layers.Dense(100, activation=\"selu\", input_shape=[300]), \np\nq\n1 −p\n1 −q\n keras.layers.Dense(28 * 28, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \nsparse_kl_ae = keras.models.Sequential([sparse_kl_encoder, \nsparse_kl_decoder])\nAfter training this sparse autoencoder on Fashion MNIST, the activations\nof the neurons in the coding layer are mostly close to 0 (about 70% of all\nactivations are lower than 0.1), and all neurons have a mean activation\naround 0.1 (about 90% of all neurons have a mean activation between 0.1\nand 0.2), as shown in Figure 17-11.\nFigure 17-11. Distribution of all the activations in the coding layer (left) and distribution of the\nmean activation per neuron (right)\nVariational Autoencoders\nAnother important category of autoencoders was introduced in 2013 by\nDiederik Kingma and Max Welling and quickly became one of the most\npopular types of autoencoders: variational autoencoders.\nThey are quite different from all the autoencoders we have discussed so\nfar, in these particular ways:\nThey are probabilistic autoencoders, meaning that their outputs\nare partly determined by chance, even after training (as opposed\nto denoising autoencoders, which use randomness only during\ntraining).\nMost importantly, they are generative autoencoders, meaning that\nthey can generate new instances that look like they were sampled\n7\nfrom the training set.\nBoth these properties make them rather similar to RBMs, but they are\neasier to train, and the sampling process is much faster (with RBMs you\nneed to wait for the network to stabilize into a “thermal equilibrium”\nbefore you can sample a new instance). Indeed, as their name suggests,\nvariational autoencoders perform variational Bayesian inference\n(introduced in Chapter 9), which is an efficient way to perform\napproximate Bayesian inference.\nLet’s take a look at how they work. Figure 17-12 (left) shows a variational\nautoencoder. You can recognize the basic structure of all autoencoders,\nwith an encoder followed by a decoder (in this example, they both have\ntwo hidden layers), but there is a twist: instead of directly producing a\ncoding for a given input, the encoder produces a mean coding μ and a\nstandard deviation σ. The actual coding is then sampled randomly from a\nGaussian distribution with mean μ and standard deviation σ. After that the\ndecoder decodes the sampled coding normally. The right part of the\ndiagram shows a training instance going through this autoencoder. First,\nthe encoder produces μ and σ, then a coding is sampled randomly (notice\nthat it is not exactly located at μ), and finally this coding is decoded; the\nfinal output resembles the training instance.\nFigure 17-12. Variational autoencoder (left) and an instance going through it (right)\nAs you can see in the diagram, although the inputs may have a very\nconvoluted distribution, a variational autoencoder tends to produce\ncodings that look as though they were sampled from a simple Gaussian\ndistribution: during training, the cost function (discussed next) pushes the\ncodings to gradually migrate within the coding space (also called the\nlatent space) to end up looking like a cloud of Gaussian points. One great\nconsequence is that after training a variational autoencoder, you can very\neasily generate a new instance: just sample a random coding from the\nGaussian distribution, decode it, and voilà!\nNow, let’s look at the cost function. It is composed of two parts. The first\nis the usual reconstruction loss that pushes the autoencoder to reproduce\nits inputs (we can use cross entropy for this, as discussed earlier). The\nsecond is the latent loss that pushes the autoencoder to have codings that\n8\nlook as though they were sampled from a simple Gaussian distribution: it\nis the KL divergence between the target distribution (i.e., the Gaussian\ndistribution) and the actual distribution of the codings. The math is a bit\nmore complex than with the sparse autoencoder, in particular because of\nthe Gaussian noise, which limits the amount of information that can be\ntransmitted to the coding layer (thus pushing the autoencoder to learn\nuseful features). Luckily, the equations simplify, so the latent loss can be\ncomputed quite simply using Equation 17-3:\nEquation 17-3. Variational autoencoder’s latent loss\nL = −\nK\n∑\ni=1\n1 + log (σi2) −σi2 −μi2\nIn this equation, L is the latent loss, n is the codings’ dimensionality, and\nμ and σ are the mean and standard deviation of the i component of the\ncodings. The vectors μ and σ (which contain all the μ and σ ) are output by\nthe encoder, as shown in Figure 17-12 (left).\nA common tweak to the variational autoencoder’s architecture is to make\nthe encoder output γ = log(σ ) rather than σ. The latent loss can then be\ncomputed as shown in Equation 17-4. This approach is more numerically\nstable and speeds up training.\nEquation 17-4. Variational autoencoder’s latent loss, rewritten using γ = log(σ )\nL = −\nK\n∑\ni=1\n1 + γi −exp (γi) −μi2\nLet’s start building a variational autoencoder for Fashion MNIST (as\nshown in Figure 17-12, but using the γ tweak). First, we will need a\ncustom layer to sample the codings, given μ and γ:\nclass Sampling(keras.layers.Layer): \n def call(self, inputs): \n mean, log_var = inputs \n9\n1\n2\ni\ni\nth\ni\ni\n2\n2\n1\n2\n return K.random_normal(tf.shape(log_var)) * K.exp(log_var / 2) + \nmean\nThis Sampling layer takes two inputs: mean (μ) and log_var (γ). It uses\nthe function K.random_normal() to sample a random vector (of the same\nshape as γ) from the Normal distribution, with mean 0 and standard\ndeviation 1. Then it multiplies it by exp(γ / 2) (which is equal to σ, as you\ncan verify), and finally it adds μ and returns the result. This samples a\ncodings vector from the Normal distribution with mean μ and standard\ndeviation σ.\nNext, we can create the encoder, using the Functional API because the\nmodel is not entirely sequential:\ncodings_size = 10 \n \ninputs = keras.layers.Input(shape=[28, 28]) \nz = keras.layers.Flatten()(inputs) \nz = keras.layers.Dense(150, activation=\"selu\")(z) \nz = keras.layers.Dense(100, activation=\"selu\")(z) \ncodings_mean = keras.layers.Dense(codings_size)(z) # μ \ncodings_log_var = keras.layers.Dense(codings_size)(z) # γ \ncodings = Sampling()([codings_mean, codings_log_var]) \nvariational_encoder = keras.Model( \n inputs=[inputs], outputs=[codings_mean, codings_log_var, codings])\nNote that the Dense layers that output codings_mean (μ) and\ncodings_log_var (γ) have the same inputs (i.e., the outputs of the second\nDense layer). We then pass both codings_mean and codings_log_var to\nthe Sampling layer. Finally, the variational_encoder model has three\noutputs, in case you want to inspect the values of codings_mean and\ncodings_log_var. The only output we will use is the last one (codings).\nNow let’s build the decoder:\ndecoder_inputs = keras.layers.Input(shape=[codings_size]) \nx = keras.layers.Dense(100, activation=\"selu\")(decoder_inputs) \nx = keras.layers.Dense(150, activation=\"selu\")(x) \nx = keras.layers.Dense(28 * 28, activation=\"sigmoid\")(x) \noutputs = keras.layers.Reshape([28, 28])(x) \nvariational_decoder = keras.Model(inputs=[decoder_inputs], outputs=\n[outputs])\nFor this decoder, we could have used the Sequential API instead of the\nFunctional API, since it is really just a simple stack of layers, virtually\nidentical to many of the decoders we have built so far. Finally, let’s build\nthe variational autoencoder model:\n_, _, codings = variational_encoder(inputs) \nreconstructions = variational_decoder(codings) \nvariational_ae = keras.Model(inputs=[inputs], outputs=[reconstructions])\nNote that we ignore the first two outputs of the encoder (we only want to\nfeed the codings to the decoder). Lastly, we must add the latent loss and\nthe reconstruction loss:\nlatent_loss = -0.5 * K.sum( \n 1 + codings_log_var - K.exp(codings_log_var) - \nK.square(codings_mean), \n axis=-1) \nvariational_ae.add_loss(K.mean(latent_loss) / 784.) \nvariational_ae.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")\nWe first apply Equation 17-4 to compute the latent loss for each instance\nin the batch (we sum over the last axis). Then we compute the mean loss\nover all the instances in the batch, and we divide the result by 784 to\nensure it has the appropriate scale compared to the reconstruction loss.\nIndeed, the variational autoencoder’s reconstruction loss is supposed to be\nthe sum of the pixel reconstruction errors, but when Keras computes the\n\"binary_crossentropy\" loss, it computes the mean over all 784 pixels,\nrather than the sum. So, the reconstruction loss is 784 times smaller than\nwe need it to be. We could define a custom loss to compute the sum rather\nthan the mean, but it is simpler to divide the latent loss by 784 (the final\nloss will be 784 times smaller than it should be, but this just means that\nwe should use a larger learning rate).\nNote that we use the RMSprop optimizer, which works well in this case.\nAnd finally we can train the autoencoder!\nhistory = variational_ae.fit(X_train, X_train, epochs=50, batch_size=128, \n validation_data=[X_valid, X_valid])\nGenerating Fashion MNIST Images\nNow let’s use this variational autoencoder to generate images that look\nlike fashion items. All we need to do is sample random codings from a\nGaussian distribution and decode them:\ncodings = tf.random.normal(shape=[12, codings_size]) \nimages = variational_decoder(codings).numpy()\nFigure 17-13 shows the 12 generated images.\nFigure 17-13. Fashion MNIST images generated by the variational autoencoder\nThe majority of these images look fairly convincing, if a bit too fuzzy. The\nrest are not great, but don’t be too harsh on the autoencoder—it only had a\nfew minutes to learn! Give it a bit more fine-tuning and training time, and\nthose images should look better.\nVariational autoencoders make it possible to perform semantic\ninterpolation: instead of interpolating two images at the pixel level (which\nwould look as if the two images were overlaid), we can interpolate at the\ncodings level. We first run both images through the encoder, then we\ninterpolate the two codings we get, and finally we decode the interpolated\ncodings to get the final image. It will look like a regular Fashion MNIST\nimage, but it will be an intermediate between the original images. In the\nfollowing code example, we take the 12 codings we just generated, we\norganize them in a 3 × 4 grid, and we use TensorFlow’s\ntf.image.resize() function to resize this grid to 5 × 7. By default, the\nresize() function will perform bilinear interpolation, so every other row\nand column will contain interpolated codings. We then use the decoder to\nproduce all the images:\ncodings_grid = tf.reshape(codings, [1, 3, 4, codings_size]) \nlarger_grid = tf.image.resize(codings_grid, size=[5, 7]) \ninterpolated_codings = tf.reshape(larger_grid, [-1, codings_size]) \nimages = variational_decoder(interpolated_codings).numpy()\nFigure 17-14 shows the resulting images. The original images are framed,\nand the rest are the result of semantic interpolation between the nearby\nimages. Notice, for example, how the shoe in the fourth row and fifth\ncolumn is a nice interpolation between the two shoes located above and\nbelow it.\nFigure 17-14. Semantic interpolation\nFor several years, variational autoencoders were quite popular, but GANs\neventually took the lead, in particular because they are capable of\ngenerating much more realistic and crisp images. So let’s turn our\nattention to GANs.\nGenerative Adversarial Networks\nGenerative adversarial networks were proposed in a 2014 paper\n by Ian\nGoodfellow et al., and although the idea got researchers excited almost\ninstantly, it took a few years to overcome some of the difficulties of\ntraining GANs. Like many great ideas, it seems simple in hindsight: make\nneural networks compete against each other in the hope that this\ncompetition will push them to excel. As shown in Figure 17-15, a GAN is\ncomposed of two neural networks:\nGenerator\n10\nTakes a random distribution as input (typically Gaussian) and outputs\nsome data—typically, an image. You can think of the random inputs as\nthe latent representations (i.e., codings) of the image to be generated.\nSo, as you can see, the generator offers the same functionality as a\ndecoder in a variational autoencoder, and it can be used in the same\nway to generate new images (just feed it some Gaussian noise, and it\noutputs a brand-new image). However, it is trained very differently, as\nwe will soon see.\nDiscriminator\nTakes either a fake image from the generator or a real image from the\ntraining set as input, and must guess whether the input image is fake or\nreal.\nFigure 17-15. A generative adversarial network\nDuring training, the generator and the discriminator have opposite goals:\nthe discriminator tries to tell fake images from real images, while the\ngenerator tries to produce images that look real enough to trick the\ndiscriminator. Because the GAN is composed of two networks with\ndifferent objectives, it cannot be trained like a regular neural network.\nEach training iteration is divided into two phases:\nIn the first phase, we train the discriminator. A batch of real\nimages is sampled from the training set and is completed with an\nequal number of fake images produced by the generator. The\nlabels are set to 0 for fake images and 1 for real images, and the\ndiscriminator is trained on this labeled batch for one step, using\nthe binary cross-entropy loss. Importantly, backpropagation only\noptimizes the weights of the discriminator during this phase.\nIn the second phase, we train the generator. We first use it to\nproduce another batch of fake images, and once again the\ndiscriminator is used to tell whether the images are fake or real.\nThis time we do not add real images in the batch, and all the\nlabels are set to 1 (real): in other words, we want the generator to\nproduce images that the discriminator will (wrongly) believe to\nbe real! Crucially, the weights of the discriminator are frozen\nduring this step, so backpropagation only affects the weights of\nthe generator.\nNOTE\nThe generator never actually sees any real images, yet it gradually learns to produce\nconvincing fake images! All it gets is the gradients flowing back through the\ndiscriminator. Fortunately, the better the discriminator gets, the more information\nabout the real images is contained in these secondhand gradients, so the generator\ncan make significant progress.\nLet’s go ahead and build a simple GAN for Fashion MNIST.\nFirst, we need to build the generator and the discriminator. The generator\nis similar to an autoencoder’s decoder, and the discriminator is a regular\nbinary classifier (it takes an image as input and ends with a Dense layer\ncontaining a single unit and using the sigmoid activation function). For the\nsecond phase of each training iteration, we also need the full GAN model\ncontaining the generator followed by the discriminator:\ncodings_size = 30 \n \ngenerator = keras.models.Sequential([ \n keras.layers.Dense(100, activation=\"selu\", input_shape=\n[codings_size]), \n keras.layers.Dense(150, activation=\"selu\"), \n keras.layers.Dense(28 * 28, activation=\"sigmoid\"), \n keras.layers.Reshape([28, 28]) \n]) \ndiscriminator = keras.models.Sequential([ \n keras.layers.Flatten(input_shape=[28, 28]), \n keras.layers.Dense(150, activation=\"selu\"), \n keras.layers.Dense(100, activation=\"selu\"), \n keras.layers.Dense(1, activation=\"sigmoid\") \n]) \ngan = keras.models.Sequential([generator, discriminator])\nNext, we need to compile these models. As the discriminator is a binary\nclassifier, we can naturally use the binary cross-entropy loss. The\ngenerator will only be trained through the gan model, so we do not need to\ncompile it at all. The gan model is also a binary classifier, so it can use the\nbinary cross-entropy loss. Importantly, the discriminator should not be\ntrained during the second phase, so we make it non-trainable before\ncompiling the gan model:\ndiscriminator.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\") \ndiscriminator.trainable = False \ngan.compile(loss=\"binary_crossentropy\", optimizer=\"rmsprop\")\nNOTE\nThe trainable attribute is taken into account by Keras only when compiling a\nmodel, so after running this code, the discriminator is trainable if we call its\nfit() method or its train_on_batch() method (which we will be using), while it\nis not trainable when we call these methods on the gan model.\nSince the training loop is unusual, we cannot use the regular fit()\nmethod. Instead, we will write a custom training loop. For this, we first\nneed to create a Dataset to iterate through the images:\nbatch_size = 32 \ndataset = tf.data.Dataset.from_tensor_slices(X_train).shuffle(1000) \ndataset = dataset.batch(batch_size, drop_remainder=True).prefetch(1)\nWe are now ready to write the training loop. Let’s wrap it in a\ntrain_gan() function:\ndef train_gan(gan, dataset, batch_size, codings_size, n_epochs=50): \n generator, discriminator = gan.layers \n for epoch in range(n_epochs): \n for X_batch in dataset: \n # phase 1 - training the discriminator \n noise = tf.random.normal(shape=[batch_size, codings_size]) \n generated_images = generator(noise) \n X_fake_and_real = tf.concat([generated_images, X_batch], \naxis=0) \n y1 = tf.constant([[0.]] * batch_size + [[1.]] * batch_size) \n discriminator.trainable = True \n discriminator.train_on_batch(X_fake_and_real, y1) \n # phase 2 - training the generator \n noise = tf.random.normal(shape=[batch_size, codings_size]) \n y2 = tf.constant([[1.]] * batch_size) \n discriminator.trainable = False \n gan.train_on_batch(noise, y2) \n \ntrain_gan(gan, dataset, batch_size, codings_size)\nAs discussed earlier, you can see the two phases at each iteration:\nIn phase one we feed Gaussian noise to the generator to produce\nfake images, and we complete this batch by concatenating an\nequal number of real images. The targets y1 are set to 0 for fake\nimages and 1 for real images. Then we train the discriminator on\nthis batch. Note that we set the discriminator’s trainable\nattribute to True: this is only to get rid of a warning that Keras\ndisplays when it notices that trainable is now False but was\nTrue when the model was compiled (or vice versa).\nIn phase two, we feed the GAN some Gaussian noise. Its\ngenerator will start by producing fake images, then the\ndiscriminator will try to guess whether these images are fake or\nreal. We want the discriminator to believe that the fake images are\nreal, so the targets y2 are set to 1. Note that we set the trainable\nattribute to False, once again to avoid a warning.\nThat’s it! If you display the generated images (see Figure 17-16), you will\nsee that at the end of the first epoch, they already start to look like (very\nnoisy) Fashion MNIST images.\nUnfortunately, the images never really get much better than that, and you\nmay even find epochs where the GAN seems to be forgetting what it\nlearned. Why is that? Well, it turns out that training a GAN can be\nchallenging. Let’s see why.\nFigure 17-16. Images generated by the GAN after one epoch of training\nThe Difficulties of Training GANs\nDuring training, the generator and the discriminator constantly try to\noutsmart each other, in a zero-sum game. As training advances, the game\nmay end up in a state that game theorists call a Nash equilibrium, named\nafter the mathematician John Nash: this is when no player would be better\noff changing their own strategy, assuming the other players do not change\ntheirs. For example, a Nash equilibrium is reached when everyone drives\non the left side of the road: no driver would be better off being the only\none to switch sides. Of course, there is a second possible Nash\nequilibrium: when everyone drives on the right side of the road. Different\ninitial states and dynamics may lead to one equilibrium or the other. In\nthis example, there is a single optimal strategy once an equilibrium is\nreached (i.e., driving on the same side as everyone else), but a Nash\nequilibrium can involve multiple competing strategies (e.g., a predator\nchases its prey, the prey tries to escape, and neither would be better off\nchanging their strategy).\nSo how does this apply to GANs? Well, the authors of the paper\ndemonstrated that a GAN can only reach a single Nash equilibrium: that’s\nwhen the generator produces perfectly realistic images, and the\ndiscriminator is forced to guess (50% real, 50% fake). This fact is very\nencouraging: it would seem that you just need to train the GAN for long\nenough, and it will eventually reach this equilibrium, giving you a perfect\ngenerator. Unfortunately, it’s not that simple: nothing guarantees that the\nequilibrium will ever be reached.\nThe biggest difficulty is called mode collapse: this is when the generator’s\noutputs gradually become less diverse. How can this happen? Suppose that\nthe generator gets better at producing convincing shoes than any other\nclass. It will fool the discriminator a bit more with shoes, and this will\nencourage it to produce even more images of shoes. Gradually, it will\nforget how to produce anything else. Meanwhile, the only fake images that\nthe discriminator will see will be shoes, so it will also forget how to\ndiscriminate fake images of other classes. Eventually, when the\ndiscriminator manages to discriminate the fake shoes from the real ones,\nthe generator will be forced to move to another class. It may then become\ngood at shirts, forgetting about shoes, and the discriminator will follow.\nThe GAN may gradually cycle across a few classes, never really becoming\nvery good at any of them.\nMoreover, because the generator and the discriminator are constantly\npushing against each other, their parameters may end up oscillating and\nbecoming unstable. Training may begin properly, then suddenly diverge\nfor no apparent reason, due to these instabilities. And since many factors\naffect these complex dynamics, GANs are very sensitive to the\nhyperparameters: you may have to spend a lot of effort fine-tuning them.\nThese problems have kept researchers very busy since 2014: many papers\nwere published on this topic, some proposing new cost functions (though\na 2018 paper\n by Google researchers questions their efficiency) or\ntechniques to stabilize training or to avoid the mode collapse issue. For\nexample, a popular technique called experience replay consists in storing\nthe images produced by the generator at each iteration in a replay buffer\n(gradually dropping older generated images) and training the\ndiscriminator using real images plus fake images drawn from this buffer\n(rather than just fake images produced by the current generator). This\nreduces the chances that the discriminator will overfit the latest\ngenerator’s outputs. Another common technique is called mini-batch\ndiscrimination: it measures how similar images are across the batch and\nprovides this statistic to the discriminator, so it can easily reject a whole\nbatch of fake images that lack diversity. This encourages the generator to\nproduce a greater variety of images, reducing the chance of mode collapse.\nOther papers simply propose specific architectures that happen to perform\nwell.\nIn short, this is still a very active field of research, and the dynamics of\nGANs are still not perfectly understood. But the good news is that great\nprogress has been made, and some of the results are truly astounding! So\nlet’s look at some of the most successful architectures, starting with deep\nconvolutional GANs, which were the state of the art just a few years ago.\nThen we will look at two more recent (and more complex) architectures.\n11\n12\nDeep Convolutional GANs\nThe original GAN paper in 2014 experimented with convolutional layers,\nbut only tried to generate small images. Soon after, many researchers tried\nto build GANs based on deeper convolutional nets for larger images. This\nproved to be tricky, as training was very unstable, but Alec Radford et al.\nfinally succeeded in late 2015, after experimenting with many different\narchitectures and hyperparameters. They called their architecture deep\nconvolutional GANs (DCGANs).\n Here are the main guidelines they\nproposed for building stable convolutional GANs:\nReplace any pooling layers with strided convolutions (in the\ndiscriminator) and transposed convolutions (in the generator).\nUse Batch Normalization in both the generator and the\ndiscriminator, except in the generator’s output layer and the\ndiscriminator’s input layer.\nRemove fully connected hidden layers for deeper architectures.\nUse ReLU activation in the generator for all layers except the\noutput layer, which should use tanh.\nUse leaky ReLU activation in the discriminator for all layers.\nThese guidelines will work in many cases, but not always, so you may still\nneed to experiment with different hyperparameters (in fact, just changing\nthe random seed and training the same model again will sometimes work).\nFor example, here is a small DCGAN that works reasonably well with\nFashion MNIST:\ncodings_size = 100 \n \ngenerator = keras.models.Sequential([ \n keras.layers.Dense(7 * 7 * 128, input_shape=[codings_size]), \n keras.layers.Reshape([7, 7, 128]), \n keras.layers.BatchNormalization(), \n keras.layers.Conv2DTranspose(64, kernel_size=5, strides=2, \npadding=\"same\", \n activation=\"selu\"), \n13\n keras.layers.BatchNormalization(), \n keras.layers.Conv2DTranspose(1, kernel_size=5, strides=2, \npadding=\"same\", \n activation=\"tanh\") \n]) \ndiscriminator = keras.models.Sequential([ \n keras.layers.Conv2D(64, kernel_size=5, strides=2, padding=\"same\", \n activation=keras.layers.LeakyReLU(0.2), \n input_shape=[28, 28, 1]), \n keras.layers.Dropout(0.4), \n keras.layers.Conv2D(128, kernel_size=5, strides=2, padding=\"same\", \n activation=keras.layers.LeakyReLU(0.2)), \n keras.layers.Dropout(0.4), \n keras.layers.Flatten(), \n keras.layers.Dense(1, activation=\"sigmoid\") \n]) \ngan = keras.models.Sequential([generator, discriminator])\nThe generator takes codings of size 100, and it projects them to 6272\ndimensions (7 * 7 * 128), and reshapes the result to get a 7 × 7 × 128\ntensor. This tensor is batch normalized and fed to a transposed\nconvolutional layer with a stride of 2, which upsamples it from 7 × 7 to 14\n× 14 and reduces its depth from 128 to 64. The result is batch normalized\nagain and fed to another transposed convolutional layer with a stride of 2,\nwhich upsamples it from 14 × 14 to 28 × 28 and reduces the depth from 64\nto 1. This layer uses the tanh activation function, so the outputs will range\nfrom –1 to 1. For this reason, before training the GAN, we need to rescale\nthe training set to that same range. We also need to reshape it to add the\nchannel dimension:\nX_train = X_train.reshape(-1, 28, 28, 1) * 2. - 1. # reshape and rescale\nThe discriminator looks much like a regular CNN for binary classification,\nexcept instead of using max pooling layers to downsample the image, we\nuse strided convolutions (strides=2). Also note that we use the leaky\nReLU activation function.\nOverall, we respected the DCGAN guidelines, except we replaced the\nBatchNormalization layers in the discriminator with Dropout layers\n(otherwise training was unstable in this case) and we replaced ReLU with\nSELU in the generator. Feel free to tweak this architecture: you will see\nhow sensitive it is to the hyperparameters (especially the relative learning\nrates of the two networks).\nLastly, to build the dataset, then compile and train this model, we use the\nexact same code as earlier. After 50 epochs of training, the generator\nproduces images like those shown in Figure 17-17. It’s still not perfect, but\nmany of these images are pretty convincing.\nFigure 17-17. Images generated by the DCGAN after 50 epochs of training\nIf you scale up this architecture and train it on a large dataset of faces, you\ncan get fairly realistic images. In fact, DCGANs can learn quite\nmeaningful latent representations, as you can see in Figure 17-18: many\nimages were generated, and nine of them were picked manually (top left),\nincluding three representing men with glasses, three men without glasses,\nand three women without glasses. For each of these categories, the codings\nthat were used to generate the images were averaged, and an image was\ngenerated based on the resulting mean codings (lower left). In short, each\nof the three lower-left images represents the mean of the three images\nlocated above it. But this is not a simple mean computed at the pixel level\n(this would result in three overlapping faces), it is a mean computed in the\nlatent space, so the images still look like normal faces. Amazingly, if you\ncompute men with glasses, minus men without glasses, plus women\nwithout glasses—where each term corresponds to one of the mean codings\n—and you generate the image that corresponds to this coding, you get the\nimage at the center of the 3 × 3 grid of faces on the right: a woman with\nglasses! The eight other images around it were generated based on the\nsame vector plus a bit of noise, to illustrate the semantic interpolation\ncapabilities of DCGANs. Being able to do arithmetic on faces feels like\nscience fiction!\nFigure 17-18. Vector arithmetic for visual concepts (part of figure 7 from the DCGAN paper)\nTIP\nIf you add each image’s class as an extra input to both the generator and the\ndiscriminator, they will both learn what each class looks like, and thus you will be\nable to control the class of each image produced by the generator. This is called a\nconditional GAN\n (CGAN).\nDCGANs aren’t perfect, though. For example, when you try to generate\nvery large images using DCGANs, you often end up with locally\nconvincing features but overall inconsistencies (such as shirts with one\nsleeve much longer than the other). How can you fix this?\n14\n15\nProgressive Growing of GANs\nAn important technique was proposed in a 2018 paper\n by Nvidia\nresearchers Tero Karras et al.: they suggested generating small images at\nthe beginning of training, then gradually adding convolutional layers to\nboth the generator and the discriminator to produce larger and larger\nimages (4 × 4, 8 × 8, 16 × 16, …, 512 × 512, 1,024 × 1,024). This approach\nresembles greedy layer-wise training of stacked autoencoders. The extra\nlayers get added at the end of the generator and at the beginning of the\ndiscriminator, and previously trained layers remain trainable.\nFor example, when growing the generator’s outputs from 4 × 4 to 8 × 8\n(see Figure 17-19), an upsampling layer (using nearest neighbor filtering)\nis added to the existing convolutional layer, so it outputs 8 × 8 feature\nmaps, which are then fed to the new convolutional layer (which uses\n\"same\" padding and strides of 1, so its outputs are also 8 × 8). This new\nlayer is followed by a new output convolutional layer: this is a regular\nconvolutional layer with kernel size 1 that projects the outputs down to the\ndesired number of color channels (e.g., 3). To avoid breaking the trained\nweights of the first convolutional layer when the new convolutional layer\nis added, the final output is a weighted sum of the original output layer\n(which now outputs 8 × 8 feature maps) and the new output layer. The\nweight of the new outputs is α, while the weight of the original outputs is\n1 – α, and α is slowly increased from 0 to 1. In other words, the new\nconvolutional layers (represented with dashed lines in Figure 17-19) are\ngradually faded in, while the original output layer is gradually faded out.\nA similar fade-in/fade-out technique is used when a new convolutional\nlayer is added to the discriminator (followed by an average pooling layer\nfor downsampling).\n16\nFigure 17-19. Progressively growing GAN: a GAN generator outputs 4 × 4 color images (left);\nwe extend it to output 8 × 8 images (right)\nThe paper also introduced several other techniques aimed at increasing the\ndiversity of the outputs (to avoid mode collapse) and making training\nmore stable:\nMinibatch standard deviation layer\nAdded near the end of the discriminator. For each position in the\ninputs, it computes the standard deviation across all channels and all\ninstances in the batch (S = tf.math.reduce_std(inputs, axis=[0,\n-1])). These standard deviations are then averaged across all points to\nget a single value (v = tf.reduce_ mean(S)). Finally, an extra feature\nmap is added to each instance in the batch and filled with the\ncomputed value (tf.concat([inputs, tf.fill([batch_size,\nheight, width, 1], v)], axis=-1)). How does this help? Well, if\nthe generator produces images with little variety, then there will be a\nsmall standard deviation across feature maps in the discriminator.\nThanks to this layer, the discriminator will have easy access to this\nstatistic, making it less likely to be fooled by a generator that produces\ntoo little diversity. This will encourage the generator to produce more\ndiverse outputs, reducing the risk of mode collapse.\nEqualized learning rate\nInitializes all weights using a simple Gaussian distribution with mean\n0 and standard deviation 1 rather than using He initialization.\nHowever, the weights are scaled down at runtime (i.e., every time the\nlayer is executed) by the same factor as in He initialization: they are\ndivided by √2/ninputs, where n\n is the number of inputs to the\nlayer. The paper demonstrated that this technique significantly\nimproved the GAN’s performance when using RMSProp, Adam, or\nother adaptive gradient optimizers. Indeed, these optimizers normalize\nthe gradient updates by their estimated standard deviation (see\nChapter 11), so parameters that have a larger dynamic range\n will\ntake longer to train, while parameters with a small dynamic range may\nbe updated too quickly, leading to instabilities. By rescaling the\nweights as part of the model itself rather than just rescaling them upon\ninitialization, this approach ensures that the dynamic range is the same\nfor all parameters, throughout training, so they all learn at the same\nspeed. This both speeds up and stabilizes training.\nPixelwise normalization layer\nAdded after each convolutional layer in the generator. It normalizes\neach activation based on all the activations in the same image and at\nthe same location, but across all channels (dividing by the square root\nof the mean squared activation). In TensorFlow code, this is inputs /\ntf.sqrt(tf.reduce_mean(tf.square(X), axis=-1,\nkeepdims=True) + 1e-8) (the smoothing term 1e-8 is needed to\navoid division by zero). This technique avoids explosions in the\nactivations due to excessive competition between the generator and the\ndiscriminator.\ninputs\n17\nThe combination of all these techniques allowed the authors to generate\nextremely convincing high-definition images of faces. But what exactly do\nwe call “convincing”? Evaluation is one of the big challenges when\nworking with GANs: although it is possible to automatically evaluate the\ndiversity of the generated images, judging their quality is a much trickier\nand subjective task. One technique is to use human raters, but this is costly\nand time-consuming. So the authors proposed to measure the similarity\nbetween the local image structure of the generated images and the training\nimages, considering every scale. This idea led them to another\ngroundbreaking innovation: StyleGANs.\nStyleGANs\nThe state of the art in high-resolution image generation was advanced once\nagain by the same Nvidia team in a 2018 paper\n that introduced the\npopular StyleGAN architecture. The authors used style transfer techniques\nin the generator to ensure that the generated images have the same local\nstructure as the training images, at every scale, greatly improving the\nquality of the generated images. The discriminator and the loss function\nwere not modified, only the generator. Let’s take a look at the StyleGAN.\nIt is composed of two networks (see Figure 17-20):\nMapping network\nAn eight-layer MLP that maps the latent representations z (i.e., the\ncodings) to a vector w. This vector is then sent through multiple affine\ntransformations (i.e., Dense layers with no activation functions,\nrepresented by the “A” boxes in Figure 17-20), which produces\nmultiple vectors. These vectors control the style of the generated\nimage at different levels, from fine-grained texture (e.g., hair color) to\nhigh-level features (e.g., adult or child). In short, the mapping network\nmaps the codings to multiple style vectors.\nSynthesis network\n18\nResponsible for generating the images. It has a constant learned input\n(to be clear, this input will be constant after training, but during\ntraining it keeps getting tweaked by backpropagation). It processes this\ninput through multiple convolutional and upsampling layers, as earlier,\nbut there are two twists: first, some noise is added to the input and to\nall the outputs of the convolutional layers (before the activation\nfunction). Second, each noise layer is followed by an Adaptive\nInstance Normalization (AdaIN) layer: it standardizes each feature\nmap independently (by subtracting the feature map’s mean and\ndividing by its standard deviation), then it uses the style vector to\ndetermine the scale and offset of each feature map (the style vector\ncontains one scale and one bias term for each feature map).\nFigure 17-20. StyleGAN’s generator architecture (part of figure 1 from the StyleGAN paper)\nThe idea of adding noise independently from the codings is very\nimportant. Some parts of an image are quite random, such as the exact\nposition of each freckle or hair. In earlier GANs, this randomness had to\neither come from the codings or be some pseudorandom noise produced by\nthe generator itself. If it came from the codings, it meant that the\ngenerator had to dedicate a significant portion of the codings’\nrepresentational power to store noise: this is quite wasteful. Moreover, the\n19\nnoise had to be able to flow through the network and reach the final layers\nof the generator: this seems like an unnecessary constraint that probably\nslowed down training. And finally, some visual artifacts may appear\nbecause the same noise was used at different levels. If instead the\ngenerator tried to produce its own pseudorandom noise, this noise might\nnot look very convincing, leading to more visual artifacts. Plus, part of the\ngenerator’s weights would be dedicated to generating pseudorandom noise,\nwhich again seems wasteful. By adding extra noise inputs, all these issues\nare avoided; the GAN is able to use the provided noise to add the right\namount of stochasticity to each part of the image.\nThe added noise is different for each level. Each noise input consists of a\nsingle feature map full of Gaussian noise, which is broadcast to all feature\nmaps (of the given level) and scaled using learned per-feature scaling\nfactors (this is represented by the “B” boxes in Figure 17-20) before it is\nadded.\nFinally, StyleGAN uses a technique called mixing regularization (or style\nmixing), where a percentage of the generated images are produced using\ntwo different codings. Specifically, the codings c and c are sent through\nthe mapping network, giving two style vectors w and w . Then the\nsynthesis network generates an image based on the styles w for the first\nlevels and the styles w for the remaining levels. The cutoff level is picked\nrandomly. This prevents the network from assuming that styles at adjacent\nlevels are correlated, which in turn encourages locality in the GAN,\nmeaning that each style vector only affects a limited number of traits in\nthe generated image.\nThere is such a wide variety of GANs out there that it would require a\nwhole book to cover them all. Hopefully this introduction has given you\nthe main ideas, and most importantly the desire to learn more. If you’re\nstruggling with a mathematical concept, there are probably blog posts out\nthere that will help you understand it better. Then go ahead and implement\nyour own GAN, and do not get discouraged if it has trouble learning at\nfirst: unfortunately, this is normal, and it will require quite a bit of\npatience before it works, but the result is worth it. If you’re struggling\n1\n2\n1\n2\n1\n2\nwith an implementation detail, there are plenty of Keras or TensorFlow\nimplementations that you can look at. In fact, if all you want is to get\nsome amazing results quickly, then you can just use a pretrained model\n(e.g., there are pretrained StyleGAN models available for Keras).\nIn the next chapter we will move to an entirely different branch of Deep\nLearning: Deep Reinforcement Learning.\nExercises\n1. What are the main tasks that autoencoders are used for?\n2. Suppose you want to train a classifier, and you have plenty of\nunlabeled training data but only a few thousand labeled instances.\nHow can autoencoders help? How would you proceed?\n3. If an autoencoder perfectly reconstructs the inputs, is it\nnecessarily a good autoencoder? How can you evaluate the\nperformance of an autoencoder?\n4. What are undercomplete and overcomplete autoencoders? What is\nthe main risk of an excessively undercomplete autoencoder? What\nabout the main risk of an overcomplete autoencoder?\n5. How do you tie weights in a stacked autoencoder? What is the\npoint of doing so?\n6. What is a generative model? Can you name a type of generative\nautoencoder?\n7. What is a GAN? Can you name a few tasks where GANs can\nshine?\n8. What are the main difficulties when training GANs?\n9. Try using a denoising autoencoder to pretrain an image classifier.\nYou can use MNIST (the simplest option), or a more complex\nimage dataset such as CIFAR10 if you want a bigger challenge.\nRegardless of the dataset you’re using, follow these steps:\nSplit the dataset into a training set and a test set. Train a\ndeep denoising autoencoder on the full training set.\nCheck that the images are fairly well reconstructed.\nVisualize the images that most activate each neuron in\nthe coding layer.\nBuild a classification DNN, reusing the lower layers of\nthe autoencoder. Train it using only 500 images from the\ntraining set. Does it perform better with or without\npretraining?\n10. Train a variational autoencoder on the image dataset of your\nchoice, and use it to generate images. Alternatively, you can try to\nfind an unlabeled dataset that you are interested in and see if you\ncan generate new samples.\n11. Train a DCGAN to tackle the image dataset of your choice, and\nuse it to generate images. Add experience replay and see if this\nhelps. Turn it into a conditional GAN where you can control the\ngenerated class.\nSolutions to these exercises are available in Appendix A.\n1 William G. Chase and Herbert A. Simon, “Perception in Chess,” Cognitive Psychology 4,\nno. 1 (1973): 55–81.\n2 You might be tempted to use the accuracy metric, but it would not work properly, since\nthis metric expects the labels to be either 0 or 1 for each pixel. You can easily work around\nthis problem by creating a custom metric that computes the accuracy after rounding the\ntargets and predictions to 0 or 1.\n3 Yoshua Bengio et al., “Greedy Layer-Wise Training of Deep Networks,” Proceedings of\nthe 19th International Conference on Neural Information Processing Systems (2006): 153–\n160.\n4 Jonathan Masci et al., “Stacked Convolutional Auto-Encoders for Hierarchical Feature\nExtraction,” Proceedings of the 21st International Conference on Artificial Neural\nNetworks 1 (2011): 52–59.\n5 Pascal Vincent et al., “Extracting and Composing Robust Features with Denoising\nAutoencoders,” Proceedings of the 25th International Conference on Machine Learning\n(2008): 1096–1103.\n6 Pascal Vincent et al., “Stacked Denoising Autoencoders: Learning Useful Representations\nin a Deep Network with a Local Denoising Criterion,” Journal of Machine Learning\nResearch 11 (2010): 3371–3408.\n7 Diederik Kingma and Max Welling, “Auto-Encoding Variational Bayes,” arXiv preprint\narXiv:1312.6114 (2013).\n8 Variational autoencoders are actually more general; the codings are not limited to\nGaussian distributions.\n9 For more mathematical details, check out the original paper on variational autoencoders,\nor Carl Doersch’s great tutorial (2016).\n10 Ian Goodfellow et al., “Generative Adversarial Nets,” Proceedings of the 27th\nInternational Conference on Neural Information Processing Systems 2 (2014): 2672–2680.\n11 For a nice comparison of the main GAN losses, check out this great GitHub project by\nHwalsuk Lee.\n12 Mario Lucic et al., “Are GANs Created Equal? A Large-Scale Study,” Proceedings of the\n32nd International Conference on Neural Information Processing Systems (2018): 698–\n707.\n13 Alec Radford et al., “Unsupervised Representation Learning with Deep Convolutional\nGenerative Adversarial Networks,” arXiv preprint arXiv:1511.06434 (2015).\n14 Reproduced with the kind authorization of the authors.\n15 Mehdi Mirza and Simon Osindero, “Conditional Generative Adversarial Nets,” arXiv\npreprint arXiv:1411.1784 (2014).\n16 Tero Karras et al., “Progressive Growing of GANs for Improved Quality, Stability, and\nVariation,” Proceedings of the International Conference on Learning Representations\n(2018).\n17 The dynamic range of a variable is the ratio between the highest and the lowest value it\nmay take.\n18 Tero Karras et al., “A Style-Based Generator Architecture for Generative Adversarial\nNetworks,” arXiv preprint arXiv:1812.04948 (2018).\n19 Reproduced with the kind authorization of the authors." + }, + { + "chapter": 18, + "text": "Chapter 18. Reinforcement\nLearning\nReinforcement Learning (RL) is one of the most exciting fields of Machine\nLearning today, and also one of the oldest. It has been around since the 1950s,\nproducing many interesting applications over the years, particularly in games\n(e.g., TD-Gammon, a Backgammon-playing program) and in machine control,\nbut seldom making the headline news. But a revolution took place in 2013, when\nresearchers from a British startup called DeepMind demonstrated a system that\ncould learn to play just about any Atari game from scratch, eventually\noutperforming humans in most of them, using only raw pixels as inputs and\nwithout any prior knowledge of the rules of the games. This was the first of a\nseries of amazing feats, culminating in March 2016 with the victory of their\nsystem AlphaGo against Lee Sedol, a legendary professional player of the game\nof Go, and in May 2017 against Ke Jie, the world champion. No program had\never come close to beating a master of this game, let alone the world champion.\nToday the whole field of RL is boiling with new ideas, with a wide range of\napplications. DeepMind was bought by Google for over $500 million in 2014.\nSo how did DeepMind achieve all this? With hindsight it seems rather simple:\nthey applied the power of Deep Learning to the field of Reinforcement Learning,\nand it worked beyond their wildest dreams. In this chapter we will first explain\nwhat Reinforcement Learning is and what it’s good at, then present two of the\nmost important techniques in Deep Reinforcement Learning: policy gradients\nand deep Q-networks (DQNs), including a discussion of Markov decision\nprocesses (MDPs). We will use these techniques to train models to balance a\npole on a moving cart; then I’ll introduce the TF-Agents library, which uses\nstate-of-the-art algorithms that greatly simplify building powerful RL systems,\nand we will use the library to train an agent to play Breakout, the famous Atari\ngame. I’ll close the chapter by taking a look at some of the latest advances in the\nfield.\nLearning to Optimize Rewards\n1\n2\n3\n4\nIn Reinforcement Learning, a software agent makes observations and takes\nactions within an environment, and in return it receives rewards. Its objective is\nto learn to act in a way that will maximize its expected rewards over time. If you\ndon’t mind a bit of anthropomorphism, you can think of positive rewards as\npleasure, and negative rewards as pain (the term “reward” is a bit misleading in\nthis case). In short, the agent acts in the environment and learns by trial and\nerror to maximize its pleasure and minimize its pain.\nThis is quite a broad setting, which can apply to a wide variety of tasks. Here are\na few examples (see Figure 18-1):\n1. The agent can be the program controlling a robot. In this case, the\nenvironment is the real world, the agent observes the environment\nthrough a set of sensors such as cameras and touch sensors, and its\nactions consist of sending signals to activate motors. It may be\nprogrammed to get positive rewards whenever it approaches the target\ndestination, and negative rewards whenever it wastes time or goes in the\nwrong direction.\n2. The agent can be the program controlling Ms. Pac-Man. In this case, the\nenvironment is a simulation of the Atari game, the actions are the nine\npossible joystick positions (upper left, down, center, and so on), the\nobservations are screenshots, and the rewards are just the game points.\n3. Similarly, the agent can be the program playing a board game such as\nGo.\n4. The agent does not have to control a physically (or virtually) moving\nthing. For example, it can be a smart thermostat, getting positive\nrewards whenever it is close to the target temperature and saves energy,\nand negative rewards when humans need to tweak the temperature, so\nthe agent must learn to anticipate human needs.\n5. The agent can observe stock market prices and decide how much to buy\nor sell every second. Rewards are obviously the monetary gains and\nlosses.\nFigure 18-1. Reinforcement Learning examples: (a) robotics, (b) Ms. Pac-Man, (c) Go player, (d)\nthermostat, (e) automatic trader\nNote that there may not be any positive rewards at all; for example, the agent\nmay move around in a maze, getting a negative reward at every time step, so it\nhad better find the exit as quickly as possible! There are many other examples of\ntasks to which Reinforcement Learning is well suited, such as self-driving cars,\nrecommender systems, placing ads on a web page, or controlling where an image\nclassification system should focus its attention.\nPolicy Search\nThe algorithm a software agent uses to determine its actions is called its policy.\nThe policy could be a neural network taking observations as inputs and\noutputting the action to take (see Figure 18-2).\n5\nFigure 18-2. Reinforcement Learning using a neural network policy\nThe policy can be any algorithm you can think of, and it does not have to be\ndeterministic. In fact, in some cases it does not even have to observe the\nenvironment! For example, consider a robotic vacuum cleaner whose reward is\nthe amount of dust it picks up in 30 minutes. Its policy could be to move forward\nwith some probability p every second, or randomly rotate left or right with\nprobability 1 – p. The rotation angle would be a random angle between –r and\n+r. Since this policy involves some randomness, it is called a stochastic policy.\nThe robot will have an erratic trajectory, which guarantees that it will eventually\nget to any place it can reach and pick up all the dust. The question is, how much\ndust will it pick up in 30 minutes?\nHow would you train such a robot? There are just two policy parameters you can\ntweak: the probability p and the angle range r. One possible learning algorithm\ncould be to try out many different values for these parameters, and pick the\ncombination that performs best (see Figure 18-3). This is an example of policy\nsearch, in this case using a brute force approach. When the policy space is too\nlarge (which is generally the case), finding a good set of parameters this way is\nlike searching for a needle in a gigantic haystack.\nAnother way to explore the policy space is to use genetic algorithms. For\nexample, you could randomly create a first generation of 100 policies and try\nthem out, then “kill” the 80 worst policies and make the 20 survivors produce 4\noffspring each. An offspring is a copy of its parent plus some random variation.\nThe surviving policies plus their offspring together constitute the second\ngeneration. You can continue to iterate through generations this way until you\nfind a good policy.\n6\n7\n8\nFigure 18-3. Four points in policy space (left) and the agent’s corresponding behavior (right)\nYet another approach is to use optimization techniques, by evaluating the\ngradients of the rewards with regard to the policy parameters, then tweaking\nthese parameters by following the gradients toward higher rewards. We will\ndiscuss this approach, is called policy gradients (PG), in more detail later in this\nchapter. Going back to the vacuum cleaner robot, you could slightly increase p\nand evaluate whether doing so increases the amount of dust picked up by the\nrobot in 30 minutes; if it does, then increase p some more, or else reduce p. We\nwill implement a popular PG algorithm using TensorFlow, but before we do, we\nneed to create an environment for the agent to live in —so it’s time to introduce\nOpenAI Gym.\nIntroduction to OpenAI Gym\nOne of the challenges of Reinforcement Learning is that in order to train an\nagent, you first need to have a working environment. If you want to program an\nagent that will learn to play an Atari game, you will need an Atari game\nsimulator. If you want to program a walking robot, then the environment is the\nreal world, and you can directly train your robot in that environment, but this has\nits limits: if the robot falls off a cliff, you can’t just click Undo. You can’t speed\nup time either; adding more computing power won’t make the robot move any\nfaster. And it’s generally too expensive to train 1,000 robots in parallel. In short,\ntraining is hard and slow in the real world, so you generally need a simulated\n9\nenvironment at least for bootstrap training. For example, you may use a library\nlike PyBullet or MuJoCo for 3D physics simulation.\nOpenAI Gym is a toolkit that provides a wide variety of simulated\nenvironments (Atari games, board games, 2D and 3D physical simulations, and\nso on), so you can train agents, compare them, or develop new RL algorithms.\nBefore installing the toolkit, if you created an isolated environment using\nvirtualenv, you first need to activate it:\n$ cd $ML_PATH # Your ML working directory (e.g., $HOME/ml) \n$ source my_env/bin/activate # on Linux or MacOS \n$ .\\my_env\\Scripts\\activate # on Windows \nNext, install OpenAI Gym (if you are not using a virtual environment, you will\nneed to add the --user option, or have administrator rights):\n$ python3 -m pip install --upgrade gym \nDepending on your system, you may also need to install the Mesa OpenGL\nUtility (GLU) library (e.g., on Ubuntu 18.04 you need to run apt install\nlibglu1-mesa). This library will be needed to render the first environment.\nNext, open up a Python shell or a Jupyter notebook and create an environment\nwith make():\n>>> import gym \n>>> env = gym.make(\"CartPole-v1\") \n>>> obs = env.reset() \n>>> obs \narray([-0.01258566, -0.00156614, 0.04207708, -0.00180545])\nHere, we’ve created a CartPole environment. This is a 2D simulation in which a\ncart can be accelerated left or right in order to balance a pole placed on top of it\n(see Figure 18-4). You can get the list of all available environments by running\ngym.envs.registry.all(). After the environment is created, you must\ninitialize it using the reset() method. This returns the first observation.\nObservations depend on the type of environment. For the CartPole environment,\neach observation is a 1D NumPy array containing four floats: these floats\nrepresent the cart’s horizontal position (0.0 = center), its velocity (positive\n10\nmeans right), the angle of the pole (0.0 = vertical), and its angular velocity\n(positive means clockwise).\nNow let’s display this environment by calling its render() method (see\nFigure 18-4). On Windows, this requires first installing an X Server, such as\nVcXsrv or Xming:\n>>> env.render() \nTrue\nFigure 18-4. The CartPole environment\nTIP\nIf you are using a headless server (i.e., without a screen), such as a virtual machine on the\ncloud, rendering will fail. The only way to avoid this is to use a fake X server such as Xvfb\nor Xdummy. For example, you can install Xvfb (apt install xvfb on Ubuntu or Debian)\nand start Python using the following command: xvfb-run -s \"-screen 0 1400x900x24\"\npython3. Alternatively, install Xvfb and the pyvirtualdisplay library (which wraps\nXvfb) and run pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()\nat the beginning of your program.\nIf you want render() to return the rendered image as a NumPy array, you can\nset mode=\"rgb_array\" (oddly, this environment will render the environment to\nscreen as well):\n>>> img = env.render(mode=\"rgb_array\") \n>>> img.shape # height, width, channels (3 = Red, Green, Blue) \n(800, 1200, 3)\nLet’s ask the environment what actions are possible:\n>>> env.action_space \nDiscrete(2)\nDiscrete(2) means that the possible actions are integers 0 and 1, which\nrepresent accelerating left (0) or right (1). Other environments may have\nadditional discrete actions, or other kinds of actions (e.g., continuous). Since the\npole is leaning toward the right (obs[2] > 0), let’s accelerate the cart toward\nthe right:\n>>> action = 1 # accelerate right \n>>> obs, reward, done, info = env.step(action) \n>>> obs \narray([-0.01261699, 0.19292789, 0.04204097, -0.28092127]) \n>>> reward \n1.0 \n>>> done \nFalse \n>>> info \n{}\nThe step() method executes the given action and returns four values:\nobs\nThis is the new observation. The cart is now moving toward the right\n(obs[1] > 0). The pole is still tilted toward the right (obs[2] > 0), but its\nangular velocity is now negative (obs[3] < 0), so it will likely be tilted\ntoward the left after the next step.\nreward\nIn this environment, you get a reward of 1.0 at every step, no matter what\nyou do, so the goal is to keep the episode running as long as possible.\ndone\nThis value will be True when the episode is over. This will happen when the\npole tilts too much, or goes off the screen, or after 200 steps (in this last\ncase, you have won). After that, the environment must be reset before it can\nbe used again.\ninfo\nThis environment-specific dictionary can provide some extra information\nthat you may find useful for debugging or for training. For example, in some\ngames it may indicate how many lives the agent has.\nTIP\nOnce you have finished using an environment, you should call its close() method to free\nresources.\nLet’s hardcode a simple policy that accelerates left when the pole is leaning\ntoward the left and accelerates right when the pole is leaning toward the right.\nWe will run this policy to see the average rewards it gets over 500 episodes:\ndef basic_policy(obs): \n angle = obs[2] \n return 0 if angle < 0 else 1 \n \ntotals = [] \nfor episode in range(500): \n episode_rewards = 0 \n obs = env.reset() \n for step in range(200): \n action = basic_policy(obs) \n obs, reward, done, info = env.step(action) \n episode_rewards += reward \n if done: \n break \n totals.append(episode_rewards)\nThis code is hopefully self-explanatory. Let’s look at the result:\n>>> import numpy as np \n>>> np.mean(totals), np.std(totals), np.min(totals), np.max(totals) \n(41.718, 8.858356280936096, 24.0, 68.0)\nEven with 500 tries, this policy never managed to keep the pole upright for more\nthan 68 consecutive steps. Not great. If you look at the simulation in the Jupyter\nnotebooks, you will see that the cart oscillates left and right more and more\nstrongly until the pole tilts too much. Let’s see if a neural network can come up\nwith a better policy.\nNeural Network Policies\nLet’s create a neural network policy. Just like with the policy we hardcoded\nearlier, this neural network will take an observation as input, and it will output\nthe action to be executed. More precisely, it will estimate a probability for each\naction, and then we will select an action randomly, according to the estimated\nprobabilities (see Figure 18-5). In the case of the CartPole environment, there\nare just two possible actions (left or right), so we only need one output neuron. It\nwill output the probability p of action 0 (left), and of course the probability of\naction 1 (right) will be 1 – p. For example, if it outputs 0.7, then we will pick\naction 0 with 70% probability, or action 1 with 30% probability.\nFigure 18-5. Neural network policy\nYou may wonder why we are picking a random action based on the probabilities\ngiven by the neural network, rather than just picking the action with the highest\nscore. This approach lets the agent find the right balance between exploring new\nactions and exploiting the actions that are known to work well. Here’s an\nanalogy: suppose you go to a restaurant for the first time, and all the dishes look\nequally appealing, so you randomly pick one. If it turns out to be good, you can\nincrease the probability that you’ll order it next time, but you shouldn’t increase\nthat probability up to 100%, or else you will never try out the other dishes, some\nof which may be even better than the one you tried.\nAlso note that in this particular environment, the past actions and observations\ncan safely be ignored, since each observation contains the environment’s full\nstate. If there were some hidden state, then you might need to consider past\nactions and observations as well. For example, if the environment only revealed\nthe position of the cart but not its velocity, you would have to consider not only\nthe current observation but also the previous observation in order to estimate the\ncurrent velocity. Another example is when the observations are noisy; in that\ncase, you generally want to use the past few observations to estimate the most\nlikely current state. The CartPole problem is thus as simple as can be; the\nobservations are noise-free, and they contain the environment’s full state.\nHere is the code to build this neural network policy using tf.keras:\nimport tensorflow as tf \nfrom tensorflow import keras \n \nn_inputs = 4 # == env.observation_space.shape[0] \n \nmodel = keras.models.Sequential([ \n keras.layers.Dense(5, activation=\"elu\", input_shape=[n_inputs]), \n keras.layers.Dense(1, activation=\"sigmoid\"), \n])\nAfter the imports, we use a simple Sequential model to define the policy\nnetwork. The number of inputs is the size of the observation space (which in the\ncase of CartPole is 4), and we have just five hidden units because it’s a simple\nproblem. Finally, we want to output a single probability (the probability of going\nleft), so we have a single output neuron using the sigmoid activation function. If\nthere were more than two possible actions, there would be one output neuron per\naction, and we would use the softmax activation function instead.\nOK, we now have a neural network policy that will take observations and output\naction probabilities. But how do we train it?\nEvaluating Actions: The Credit Assignment\nProblem\nIf we knew what the best action was at each step, we could train the neural\nnetwork as usual, by minimizing the cross entropy between the estimated\nprobability distribution and the target probability distribution. It would just be\nregular supervised learning. However, in Reinforcement Learning the only\nguidance the agent gets is through rewards, and rewards are typically sparse and\ndelayed. For example, if the agent manages to balance the pole for 100 steps,\nhow can it know which of the 100 actions it took were good, and which of them\nwere bad? All it knows is that the pole fell after the last action, but surely this\nlast action is not entirely responsible. This is called the credit assignment\nproblem: when the agent gets a reward, it is hard for it to know which actions\nshould get credited (or blamed) for it. Think of a dog that gets rewarded hours\nafter it behaved well; will it understand what it is being rewarded for?\nTo tackle this problem, a common strategy is to evaluate an action based on the\nsum of all the rewards that come after it, usually applying a discount factor γ\n(gamma) at each step. This sum of discounted rewards is called the action’s\nreturn. Consider the example in Figure 18-6). If an agent decides to go right\nthree times in a row and gets +10 reward after the first step, 0 after the second\nstep, and finally –50 after the third step, then assuming we use a discount factor\nγ = 0.8, the first action will have a return of 10 + γ × 0 + γ × (–50) = –22. If the\ndiscount factor is close to 0, then future rewards won’t count for much compared\nto immediate rewards. Conversely, if the discount factor is close to 1, then\nrewards far into the future will count almost as much as immediate rewards.\nTypical discount factors vary from 0.9 to 0.99. With a discount factor of 0.95,\nrewards 13 steps into the future count roughly for half as much as immediate\nrewards (since 0.95 ≈ 0.5), while with a discount factor of 0.99, rewards 69\nsteps into the future count for half as much as immediate rewards. In the\nCartPole environment, actions have fairly short-term effects, so choosing a\ndiscount factor of 0.95 seems reasonable.\n2\n13\nFigure 18-6. Computing an action’s return: the sum of discounted future rewards\nOf course, a good action may be followed by several bad actions that cause the\npole to fall quickly, resulting in the good action getting a low return (similarly, a\ngood actor may sometimes star in a terrible movie). However, if we play the\ngame enough times, on average good actions will get a higher return than bad\nones. We want to estimate how much better or worse an action is, compared to\nthe other possible actions, on average. This is called the action advantage. For\nthis, we must run many episodes and normalize all the action returns (by\nsubtracting the mean and dividing by the standard deviation). After that, we can\nreasonably assume that actions with a negative advantage were bad while actions\nwith a positive advantage were good. Perfect—now that we have a way to\nevaluate each action, we are ready to train our first agent using policy gradients.\nLet’s see how.\nPolicy Gradients\nAs discussed earlier, PG algorithms optimize the parameters of a policy by\nfollowing the gradients toward higher rewards. One popular class of PG\nalgorithms, called REINFORCE algorithms, was introduced back in 1992 by\nRonald Williams. Here is one common variant:\n1. First, let the neural network policy play the game several times, and at\neach step, compute the gradients that would make the chosen action\neven more likely—but don’t apply these gradients yet.\n2. Once you have run several episodes, compute each action’s advantage\n(using the method described in the previous section).\n3. If an action’s advantage is positive, it means that the action was\nprobably good, and you want to apply the gradients computed earlier to\nmake the action even more likely to be chosen in the future. However, if\nthe action’s advantage is negative, it means the action was probably bad,\nand you want to apply the opposite gradients to make this action\nslightly less likely in the future. The solution is simply to multiply each\ngradient vector by the corresponding action’s advantage.\n4. Finally, compute the mean of all the resulting gradient vectors, and use\nit to perform a Gradient Descent step.\nLet’s use tf.keras to implement this algorithm. We will train the neural network\npolicy we built earlier so that it learns to balance the pole on the cart. First, we\nneed a function that will play one step. We will pretend for now that whatever\naction it takes is the right one so that we can compute the loss and its gradients\n(these gradients will just be saved for a while, and we will modify them later\ndepending on how good or bad the action turned out to be):\ndef play_one_step(env, obs, model, loss_fn): \n with tf.GradientTape() as tape: \n left_proba = model(obs[np.newaxis]) \n action = (tf.random.uniform([1, 1]) > left_proba) \n y_target = tf.constant([[1.]]) - tf.cast(action, tf.float32) \n loss = tf.reduce_mean(loss_fn(y_target, left_proba)) \n grads = tape.gradient(loss, model.trainable_variables) \n obs, reward, done, info = env.step(int(action[0, 0].numpy())) \n return obs, reward, done, grads\nLet’s walk though this function:\nWithin the GradientTape block (see Chapter 12), we start by calling\nthe model, giving it a single observation (we reshape the observation so\n11\nit becomes a batch containing a single instance, as the model expects a\nbatch). This outputs the probability of going left.\nNext, we sample a random float between 0 and 1, and we check whether\nit is greater than left_proba. The action will be False with\nprobability left_proba, or True with probability 1 - left_proba.\nOnce we cast this Boolean to a number, the action will be 0 (left) or 1\n(right) with the appropriate probabilities.\nNext, we define the target probability of going left: it is 1 minus the\naction (cast to a float). If the action is 0 (left), then the target\nprobability of going left will be 1. If the action is 1 (right), then the\ntarget probability will be 0.\nThen we compute the loss using the given loss function, and we use the\ntape to compute the gradient of the loss with regard to the model’s\ntrainable variables. Again, these gradients will be tweaked later, before\nwe apply them, depending on how good or bad the action turned out to\nbe.\nFinally, we play the selected action, and we return the new observation,\nthe reward, whether the episode is ended or not, and of course the\ngradients that we just computed.\nNow let’s create another function that will rely on the play_one_step()\nfunction to play multiple episodes, returning all the rewards and gradients for\neach episode and each step:\ndef play_multiple_episodes(env, n_episodes, n_max_steps, model, loss_fn): \n all_rewards = [] \n all_grads = [] \n for episode in range(n_episodes): \n current_rewards = [] \n current_grads = [] \n obs = env.reset() \n for step in range(n_max_steps): \n obs, reward, done, grads = play_one_step(env, obs, model, loss_fn) \n current_rewards.append(reward) \n current_grads.append(grads) \n if done: \n break \n all_rewards.append(current_rewards) \n all_grads.append(current_grads) \n return all_rewards, all_grads\nThis code returns a list of reward lists (one reward list per episode, containing\none reward per step), as well as a list of gradient lists (one gradient list per\nepisode, each containing one tuple of gradients per step and each tuple\ncontaining one gradient tensor per trainable variable).\nThe algorithm will use the play_multiple_episodes() function to play the\ngame several times (e.g., 10 times), then it will go back and look at all the\nrewards, discount them, and normalize them. To do that, we need a couple more\nfunctions: the first will compute the sum of future discounted rewards at each\nstep, and the second will normalize all these discounted rewards (returns) across\nmany episodes by subtracting the mean and dividing by the standard deviation:\ndef discount_rewards(rewards, discount_factor): \n discounted = np.array(rewards) \n for step in range(len(rewards) - 2, -1, -1): \n discounted[step] += discounted[step + 1] * discount_factor \n return discounted \n \ndef discount_and_normalize_rewards(all_rewards, discount_factor): \n all_discounted_rewards = [discount_rewards(rewards, discount_factor) \n for rewards in all_rewards] \n flat_rewards = np.concatenate(all_discounted_rewards) \n reward_mean = flat_rewards.mean() \n reward_std = flat_rewards.std() \n return [(discounted_rewards - reward_mean) / reward_std \n for discounted_rewards in all_discounted_rewards]\nLet’s check that this works:\n>>> discount_rewards([10, 0, -50], discount_factor=0.8) \narray([-22, -40, -50]) \n>>> discount_and_normalize_rewards([[10, 0, -50], [10, 20]], \n... discount_factor=0.8) \n... \n[array([-0.28435071, -0.86597718, -1.18910299]), \n array([1.26665318, 1.0727777 ])]\nThe call to discount_rewards() returns exactly what we expect (see Figure 18-\n6). You can verify that the function discount_and_normalize_rewards() does\nindeed return the normalized action advantages for each action in both episodes.\nNotice that the first episode was much worse than the second, so its normalized\nadvantages are all negative; all actions from the first episode would be\nconsidered bad, and conversely all actions from the second episode would be\nconsidered good.\nWe are almost ready to run the algorithm! Now let’s define the hyperparameters.\nWe will run 150 training iterations, playing 10 episodes per iteration, and each\nepisode will last at most 200 steps. We will use a discount factor of 0.95:\nn_iterations = 150 \nn_episodes_per_update = 10 \nn_max_steps = 200 \ndiscount_factor = 0.95\nWe also need an optimizer and the loss function. A regular Adam optimizer with\nlearning rate 0.01 will do just fine, and we will use the binary cross-entropy loss\nfunction because we are training a binary classifier (there are two possible\nactions: left or right):\noptimizer = keras.optimizers.Adam(lr=0.01) \nloss_fn = keras.losses.binary_crossentropy\nWe are now ready to build and run the training loop!\nfor iteration in range(n_iterations): \n all_rewards, all_grads = play_multiple_episodes( \n env, n_episodes_per_update, n_max_steps, model, loss_fn) \n all_final_rewards = discount_and_normalize_rewards(all_rewards, \n discount_factor) \n all_mean_grads = [] \n for var_index in range(len(model.trainable_variables)): \n mean_grads = tf.reduce_mean( \n [final_reward * all_grads[episode_index][step][var_index] \n for episode_index, final_rewards in enumerate(all_final_rewards) \n for step, final_reward in enumerate(final_rewards)], axis=0) \n all_mean_grads.append(mean_grads) \n optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))\nLet’s walk through this code:\nAt each training iteration, this loop calls the\nplay_multiple_episodes() function, which plays the game 10 times\nand returns all the rewards and gradients for every episode and step.\nThen we call the discount_and_normalize_rewards() to compute\neach action’s normalized advantage (which in the code we call the\nfinal_reward). This provides a measure of how good or bad each\naction actually was, in hindsight.\nNext, we go through each trainable variable, and for each of them we\ncompute the weighted mean of the gradients for that variable over all\nepisodes and all steps, weighted by the final_reward.\nFinally, we apply these mean gradients using the optimizer: the model’s\ntrainable variables will be tweaked, and hopefully the policy will be a\nbit better.\nAnd we’re done! This code will train the neural network policy, and it will\nsuccessfully learn to balance the pole on the cart (you can try it out in the\n“Policy Gradients” section of the Jupyter notebook). The mean reward per\nepisode will get very close to 200 (which is the maximum by default with this\nenvironment). Success!\nTIP\nResearchers try to find algorithms that work well even when the agent initially knows\nnothing about the environment. However, unless you are writing a paper, you should not\nhesitate to inject prior knowledge into the agent, as it will speed up training dramatically. For\nexample, since you know that the pole should be as vertical as possible, you could add\nnegative rewards proportional to the pole’s angle. This will make the rewards much less\nsparse and speed up training. Also, if you already have a reasonably good policy (e.g.,\nhardcoded), you may want to train the neural network to imitate it before using policy\ngradients to improve it.\nThe simple policy gradients algorithm we just trained solved the CartPole task,\nbut it would not scale well to larger and more complex tasks. Indeed, it is highly\nsample inefficient, meaning it needs to explore the game for a very long time\nbefore it can make significant progress. This is due to the fact that it must run\nmultiple episodes to estimate the advantage of each action, as we have seen.\nHowever, it is the foundation of more powerful algorithms, such as Actor-Critic\nalgorithms (which we will discuss briefly at the end of this chapter).\nWe will now look at another popular family of algorithms. Whereas PG\nalgorithms directly try to optimize the policy to increase rewards, the algorithms\nwe will look at now are less direct: the agent learns to estimate the expected\nreturn for each state, or for each action in each state, then it uses this knowledge\nto decide how to act. To understand these algorithms, we must first introduce\nMarkov decision processes.\nMarkov Decision Processes\nIn the early 20th century, the mathematician Andrey Markov studied stochastic\nprocesses with no memory, called Markov chains. Such a process has a fixed\nnumber of states, and it randomly evolves from one state to another at each step.\nThe probability for it to evolve from a state s to a state s′ is fixed, and it depends\nonly on the pair (s, s′), not on past states (this is why we say that the system has\nno memory).\nFigure 18-7 shows an example of a Markov chain with four states.\nFigure 18-7. Example of a Markov chain\nSuppose that the process starts in state s , and there is a 70% chance that it will\nremain in that state at the next step. Eventually it is bound to leave that state and\n0\nnever come back because no other state points back to s . If it goes to state s , it\nwill then most likely go to state s (90% probability), then immediately back to\nstate s (with 100% probability). It may alternate a number of times between\nthese two states, but eventually it will fall into state s and remain there forever\n(this is a terminal state). Markov chains can have very different dynamics, and\nthey are heavily used in thermodynamics, chemistry, statistics, and much more.\nMarkov decision processes were first described in the 1950s by Richard\nBellman. They resemble Markov chains but with a twist: at each step, an agent\ncan choose one of several possible actions, and the transition probabilities\ndepend on the chosen action. Moreover, some state transitions return some\nreward (positive or negative), and the agent’s goal is to find a policy that will\nmaximize reward over time.\nFor example, the MDP represented in Figure 18-8 has three states (represented\nby circles) and up to three possible discrete actions at each step (represented by\ndiamonds).\nFigure 18-8. Example of a Markov decision process\nIf it starts in state s , the agent can choose between actions a , a , or a . If it\nchooses action a , it just remains in state s with certainty, and without any\nreward. It can thus decide to stay there forever if it wants to. But if it chooses\naction a , it has a 70% probability of gaining a reward of +10 and remaining in\nstate s . It can then try again and again to gain as much reward as possible, but at\none point it is going to end up instead in state s . In state s it has only two\n0\n1\n2\n1\n3\n12\n0\n0\n1\n2\n1\n0\n0\n0\n1\n1\npossible actions: a or a . It can choose to stay put by repeatedly choosing action\na , or it can choose to move on to state s and get a negative reward of –50\n(ouch). In state s it has no other choice than to take action a , which will most\nlikely lead it back to state s , gaining a reward of +40 on the way. You get the\npicture. By looking at this MDP, can you guess which strategy will gain the most\nreward over time? In state s it is clear that action a is the best option, and in\nstate s the agent has no choice but to take action a , but in state s it is not\nobvious whether the agent should stay put (a ) or go through the fire (a ).\nBellman found a way to estimate the optimal state value of any state s, noted V*\n(s), which is the sum of all discounted future rewards the agent can expect on\naverage after it reaches a state s, assuming it acts optimally. He showed that if\nthe agent acts optimally, then the Bellman Optimality Equation applies (see\nEquation 18-1). This recursive equation says that if the agent acts optimally,\nthen the optimal value of the current state is equal to the reward it will get on\naverage after taking one optimal action, plus the expected optimal value of all\npossible next states that this action can lead to.\nEquation 18-1. Bellman Optimality Equation\nV ∗(s) = maxa ∑s T (s, a, s') [R(s, a, s') + γ ⋅V ∗(s')]\nfor all s\nIn this equation:\nT(s, a, s′) is the transition probability from state s to state s′, given that\nthe agent chose action a. For example, in Figure 18-8, T(s , a , s ) = 0.8.\nR(s, a, s′) is the reward that the agent gets when it goes from state s to\nstate s′, given that the agent chose action a. For example, in Figure 18-8,\nR(s , a , s ) = +40.\nγ is the discount factor.\nThis equation leads directly to an algorithm that can precisely estimate the\noptimal state value of every possible state: you first initialize all the state value\nestimates to zero, and then you iteratively update them using the Value Iteration\nalgorithm (see Equation 18-2). A remarkable result is that, given enough time,\nthese estimates are guaranteed to converge to the optimal state values,\ncorresponding to the optimal policy.\nEquation 18-2. Value Iteration algorithm\n0\n2\n0\n2\n2\n1\n0\n0\n0\n2\n1\n1\n0\n2\n2\n1\n0\n2\n1\n0\nVk+1 (s) ←max\na\n∑\ns'\nT (s, a, s') [R(s, a, s') + γ ⋅Vk (s')]\nfor all s\nIn this equation, V (s) is the estimated value of state s at the k iteration of the\nalgorithm.\nNOTE\nThis algorithm is an example of Dynamic Programming, which breaks down a complex\nproblem into tractable subproblems that can be tackled iteratively.\nKnowing the optimal state values can be useful, in particular to evaluate a\npolicy, but it does not give us the optimal policy for the agent. Luckily, Bellman\nfound a very similar algorithm to estimate the optimal state-action values,\ngenerally called Q-Values (Quality Values). The optimal Q-Value of the state-\naction pair (s, a), noted Q*(s, a), is the sum of discounted future rewards the\nagent can expect on average after it reaches the state s and chooses action a, but\nbefore it sees the outcome of this action, assuming it acts optimally after that\naction.\nHere is how it works: once again, you start by initializing all the Q-Value\nestimates to zero, then you update them using the Q-Value Iteration algorithm\n(see Equation 18-3).\nEquation 18-3. Q-Value Iteration algorithm\nQk+1 (s, a) ←∑\ns'\nT (s, a, s') [R(s, a, s') + γ ⋅max\na'\nQk (s', a')]\nfor all (s'a)\nOnce you have the optimal Q-Values, defining the optimal policy, noted π*(s), is\ntrivial: when the agent is in state s, it should choose the action with the highest\nQ-Value for that state: π∗(s) = argmax\na\nQ∗(s, a).\nLet’s apply this algorithm to the MDP represented in Figure 18-8. First, we need\nto define the MDP:\ntransition_probabilities = [ # shape=[s, a, s'] \n [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]], \n [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]], \nk\nth\n [None, [0.8, 0.1, 0.1], None]] \nrewards = [ # shape=[s, a, s'] \n [[+10, 0, 0], [0, 0, 0], [0, 0, 0]], \n [[0, 0, 0], [0, 0, 0], [0, 0, -50]], \n [[0, 0, 0], [+40, 0, 0], [0, 0, 0]]] \npossible_actions = [[0, 1, 2], [0, 2], [1]]\nFor example, to know the transition probability from s to s after playing action\na , we will look up transition_probabilities[2][1][0] (which is 0.8).\nSimilarly, to get the corresponding reward, we will look up rewards[2][1][0]\n(which is +40). And to get the list of possible actions in s , we will look up\npossible_actions[2] (in this case, only action a is possible). Next, we must\ninitialize all the Q-Values to 0 (except for the the impossible actions, for which\nwe set the Q-Values to –∞):\nQ_values = np.full((3, 3), -np.inf) # -np.inf for impossible actions \nfor state, actions in enumerate(possible_actions): \n Q_values[state, actions] = 0.0 # for all possible actions\nNow let’s run the Q-Value Iteration algorithm. It applies Equation 18-3\nrepeatedly, to all Q-Values, for every state and every possible action:\ngamma = 0.90 # the discount factor \n \nfor iteration in range(50): \n Q_prev = Q_values.copy() \n for s in range(3): \n for a in possible_actions[s]: \n Q_values[s, a] = np.sum([ \n transition_probabilities[s][a][sp] \n * (rewards[s][a][sp] + gamma * np.max(Q_prev[sp])) \n for sp in range(3)])\nThat’s it! The resulting Q-Values look like this:\n>>> Q_values \narray([[18.91891892, 17.02702702, 13.62162162], \n [ 0. , -inf, -4.87971488], \n [ -inf, 50.13365013, -inf]])\nFor example, when the agent is in state s and it chooses action a , the expected\nsum of discounted future rewards is approximately 17.0.\n2\n0\n1\n2\n1\n0\n1\nFor each state, let’s look at the action that has the highest Q-Value:\n>>> np.argmax(Q_values, axis=1) # optimal action for each state \narray([0, 0, 1])\nThis gives us the optimal policy for this MDP, when using a discount factor of\n0.90: in state s choose action a ; in state s choose action a (i.e., stay put); and\nin state s choose action a (the only possible action). Interestingly, if we\nincrease the discount factor to 0.95, the optimal policy changes: in state s the\nbest action becomes a (go through the fire!). This makes sense because the\nmore you value future rewards, the more you are willing to put up with some\npain now for the promise of future bliss.\nTemporal Difference Learning\nReinforcement Learning problems with discrete actions can often be modeled as\nMarkov decision processes, but the agent initially has no idea what the transition\nprobabilities are (it does not know T(s, a, s′)), and it does not know what the\nrewards are going to be either (it does not know R(s, a, s′)). It must experience\neach state and each transition at least once to know the rewards, and it must\nexperience them multiple times if it is to have a reasonable estimate of the\ntransition probabilities.\nThe Temporal Difference Learning (TD Learning) algorithm is very similar to\nthe Value Iteration algorithm, but tweaked to take into account the fact that the\nagent has only partial knowledge of the MDP. In general we assume that the\nagent initially knows only the possible states and actions, and nothing more. The\nagent uses an exploration policy—for example, a purely random policy—to\nexplore the MDP, and as it progresses, the TD Learning algorithm updates the\nestimates of the state values based on the transitions and rewards that are\nactually observed (see Equation 18-4).\nEquation 18-4. TD Learning algorithm\nVk+1(s) ←(1 −α)Vk(s) + α(r + γ ⋅Vk(s'))\nor, equivalently: \nVk+1(s) ←Vk(s) + α ⋅δk(s, r, s')\nwith δk(s, r, s') = r + γ ⋅Vk(s') −Vk(s)\n0\n0\n1\n0\n2\n1\n1\n2\nIn this equation:\nα is the learning rate (e.g., 0.01).\nr + γ · V (s′) is called the TD target.\nδ (s, r, s′) is called the TD error.\nA more concise way of writing the first form of this equation is to use the\nnotation a ←\nα b, which means a\n ← (1 – α) · a + α ·b . So, the first line of\nEquation 18-4 can be rewritten like this: V (s) ←\nα r + γ ⋅V (s').\nTIP\nTD Learning has many similarities with Stochastic Gradient Descent, in particular the fact\nthat it handles one sample at a time. Moreover, just like Stochastic GD, it can only truly\nconverge if you gradually reduce the learning rate (otherwise it will keep bouncing around\nthe optimum Q-Values).\nFor each state s, this algorithm simply keeps track of a running average of the\nimmediate rewards the agent gets upon leaving that state, plus the rewards it\nexpects to get later (assuming it acts optimally).\nQ-Learning\nSimilarly, the Q-Learning algorithm is an adaptation of the Q-Value Iteration\nalgorithm to the situation where the transition probabilities and the rewards are\ninitially unknown (see Equation 18-5). Q-Learning works by watching an agent\nplay (e.g., randomly) and gradually improving its estimates of the Q-Values.\nOnce it has accurate Q-Value estimates (or close enough), then the optimal\npolicy is choosing the action that has the highest Q-Value (i.e., the greedy\npolicy).\nEquation 18-5. Q-Learning algorithm\nQ(s, a) ←\nα r + γ ⋅max\na'\n Q(s', a')\nk\nk\nk+1\nk\nk\nFor each state-action pair (s, a), this algorithm keeps track of a running average\nof the rewards r the agent gets upon leaving the state s with action a, plus the\nsum of discounted future rewards it expects to get. To estimate this sum, we take\nthe maximum of the Q-Value estimates for the next state s′, since we assume that\nthe target policy would act optimally from then on.\nLet’s implement the Q-Learning algorithm. First, we will need to make an agent\nexplore the environment. For this, we need a step function so that the agent can\nexecute one action and get the resulting state and reward:\ndef step(state, action): \n probas = transition_probabilities[state][action] \n next_state = np.random.choice([0, 1, 2], p=probas) \n reward = rewards[state][action][next_state] \n return next_state, reward\nNow let’s implement the agent’s exploration policy. Since the state space is\npretty small, a simple random policy will be sufficient. If we run the algorithm\nfor long enough, the agent will visit every state many times, and it will also try\nevery possible action many times:\ndef exploration_policy(state): \n return np.random.choice(possible_actions[state])\nNext, after we initialize the Q-Values just like earlier, we are ready to run the Q-\nLearning algorithm with learning rate decay (using power scheduling,\nintroduced in Chapter 11):\nalpha0 = 0.05 # initial learning rate \ndecay = 0.005 # learning rate decay \ngamma = 0.90 # discount factor \nstate = 0 # initial state \n \nfor iteration in range(10000): \n action = exploration_policy(state) \n next_state, reward = step(state, action) \n next_value = np.max(Q_values[next_state]) \n alpha = alpha0 / (1 + iteration * decay) \n Q_values[state, action] *= 1 - alpha \n Q_values[state, action] += alpha * (reward + gamma * next_value) \n state = next_state\nThis algorithm will converge to the optimal Q-Values, but it will take many\niterations, and possibly quite a lot of hyperparameter tuning. As you can see in\nFigure 18-9, the Q-Value Iteration algorithm (left) converges very quickly, in\nfewer than 20 iterations, while the Q-Learning algorithm (right) takes about\n8,000 iterations to converge. Obviously, not knowing the transition probabilities\nor the rewards makes finding the optimal policy significantly harder!\nFigure 18-9. The Q-Value Iteration algorithm (left) versus the Q-Learning algorithm (right)\nThe Q-Learning algorithm is called an off-policy algorithm because the policy\nbeing trained is not necessarily the one being executed: in the previous code\nexample, the policy being executed (the exploration policy) is completely\nrandom, while the policy being trained will always choose the actions with the\nhighest Q-Values. Conversely, the Policy Gradients algorithm is an on-policy\nalgorithm: it explores the world using the policy being trained. It is somewhat\nsurprising that Q-Learning is capable of learning the optimal policy by just\nwatching an agent act randomly (imagine learning to play golf when your\nteacher is a drunk monkey). Can we do better?\nExploration Policies\nOf course, Q-Learning can work only if the exploration policy explores the MDP\nthoroughly enough. Although a purely random policy is guaranteed to eventually\nvisit every state and every transition many times, it may take an extremely long\ntime to do so. Therefore, a better option is to use the ε-greedy policy (ε is\nepsilon): at each step it acts randomly with probability ε, or greedily with\nprobability 1–ε (i.e., choosing the action with the highest Q-Value). The\nadvantage of the ε-greedy policy (compared to a completely random policy) is\nthat it will spend more and more time exploring the interesting parts of the\nenvironment, as the Q-Value estimates get better and better, while still spending\nsome time visiting unknown regions of the MDP. It is quite common to start\nwith a high value for ε (e.g., 1.0) and then gradually reduce it (e.g., down to\n0.05).\nAlternatively, rather than relying only on chance for exploration, another\napproach is to encourage the exploration policy to try actions that it has not tried\nmuch before. This can be implemented as a bonus added to the Q-Value\nestimates, as shown in Equation 18-6.\nEquation 18-6. Q-Learning using an exploration function\nQ(s, a) ←\nα r + γ ⋅max\na'\n f(Q(s', a'), N(s', a'))\nIn this equation:\nN(s′, a′) counts the number of times the action a′ was chosen in state s′.\nf(Q, N) is an exploration function, such as f(Q, N) = Q + κ/(1 + N),\nwhere κ is a curiosity hyperparameter that measures how much the\nagent is attracted to the unknown.\nApproximate Q-Learning and Deep Q-Learning\nThe main problem with Q-Learning is that it does not scale well to large (or\neven medium) MDPs with many states and actions. For example, suppose you\nwanted to use Q-Learning to train an agent to play Ms. Pac-Man (see Figure 18-\n1). There are about 150 pellets that Ms. Pac-Man can eat, each of which can be\npresent or absent (i.e., already eaten). So, the number of possible states is greater\nthan 2\n ≈ 10 . And if you add all the possible combinations of positions for\nall the ghosts and Ms. Pac-Man, the number of possible states becomes larger\nthan the number of atoms in our planet, so there’s absolutely no way you can\nkeep track of an estimate for every single Q-Value.\nThe solution is to find a function Qθ(s, a) that approximates the Q-Value of any\nstate-action pair (s, a) using a manageable number of parameters (given by the\nparameter vector θ). This is called Approximate Q-Learning. For years it was\nrecommended to use linear combinations of handcrafted features extracted from\n150\n45\nthe state (e.g., distance of the closest ghosts, their directions, and so on) to\nestimate Q-Values, but in 2013, DeepMind showed that using deep neural\nnetworks can work much better, especially for complex problems, and it does\nnot require any feature engineering. A DNN used to estimate Q-Values is called\na Deep Q-Network (DQN), and using a DQN for Approximate Q-Learning is\ncalled Deep Q-Learning.\nNow, how can we train a DQN? Well, consider the approximate Q-Value\ncomputed by the DQN for a given state-action pair (s, a). Thanks to Bellman, we\nknow we want this approximate Q-Value to be as close as possible to the reward\nr that we actually observe after playing action a in state s, plus the discounted\nvalue of playing optimally from then on. To estimate this sum of future\ndiscounted rewards, we can simply execute the DQN on the next state s′ and for\nall possible actions a′. We get an approximate future Q-Value for each possible\naction. We then pick the highest (since we assume we will be playing optimally)\nand discount it, and this gives us an estimate of the sum of future discounted\nrewards. By summing the reward r and the future discounted value estimate, we\nget a target Q-Value y(s, a) for the state-action pair (s, a), as shown in Equation\n18-7.\nEquation 18-7. Target Q-Value\nQtarget(s, a) = r + γ ⋅max\na'\n Qθ(s', a')\nWith this target Q-Value, we can run a training step using any Gradient Descent\nalgorithm. Specifically, we generally try to minimize the squared error between\nthe estimated Q-Value Q(s, a) and the target Q-Value (or the Huber loss to\nreduce the algorithm’s sensitivity to large errors). And that’s all for the basic\nDeep Q-Learning algorithm! Let’s see how to implement it to solve the CartPole\nenvironment.\nImplementing Deep Q-Learning\nThe first thing we need is a Deep Q-Network. In theory, you need a neural net\nthat takes a state-action pair and outputs an approximate Q-Value, but in practice\nit’s much more efficient to use a neural net that takes a state and outputs one\napproximate Q-Value for each possible action. To solve the CartPole\nenvironment, we do not need a very complicated neural net; a couple of hidden\nlayers will do:\nenv = gym.make(\"CartPole-v0\") \ninput_shape = [4] # == env.observation_space.shape \nn_outputs = 2 # == env.action_space.n \n \nmodel = keras.models.Sequential([ \n keras.layers.Dense(32, activation=\"elu\", input_shape=input_shape), \n keras.layers.Dense(32, activation=\"elu\"), \n keras.layers.Dense(n_outputs) \n])\nTo select an action using this DQN, we pick the action with the largest predicted\nQ-Value. To ensure that the agent explores the environment, we will use an ε-\ngreedy policy (i.e., we will choose a random action with probability ε):\ndef epsilon_greedy_policy(state, epsilon=0): \n if np.random.rand() < epsilon: \n return np.random.randint(2) \n else: \n Q_values = model.predict(state[np.newaxis]) \n return np.argmax(Q_values[0])\nInstead of training the DQN based only on the latest experiences, we will store\nall experiences in a replay buffer (or replay memory), and we will sample a\nrandom training batch from it at each training iteration. This helps reduce the\ncorrelations between the experiences in a training batch, which tremendously\nhelps training. For this, we will just use a deque list:\nfrom collections import deque \n \nreplay_buffer = deque(maxlen=2000)\nTIP\nA deque is a linked list, where each element points to the next one and to the previous one.\nIt makes inserting and deleting items very fast, but the longer the deque is, the slower\nrandom access will be. If you need a very large replay buffer, use a circular buffer; see the\n“Deque vs Rotating List” section of the notebook for an implementation.\nEach experience will be composed of five elements: a state, the action the agent\ntook, the resulting reward, the next state it reached, and finally a Boolean\nindicating whether the episode ended at that point (done). We will need a small\nfunction to sample a random batch of experiences from the replay buffer. It will\nreturn five NumPy arrays corresponding to the five experience elements:\ndef sample_experiences(batch_size): \n indices = np.random.randint(len(replay_buffer), size=batch_size) \n batch = [replay_buffer[index] for index in indices] \n states, actions, rewards, next_states, dones = [ \n np.array([experience[field_index] for experience in batch]) \n for field_index in range(5)] \n return states, actions, rewards, next_states, dones\nLet’s also create a function that will play a single step using the ε-greedy policy,\nthen store the resulting experience in the replay buffer:\ndef play_one_step(env, state, epsilon): \n action = epsilon_greedy_policy(state, epsilon) \n next_state, reward, done, info = env.step(action) \n replay_buffer.append((state, action, reward, next_state, done)) \n return next_state, reward, done, info\nFinally, let’s create one last function that will sample a batch of experiences\nfrom the replay buffer and train the DQN by performing a single Gradient\nDescent step on this batch:\nbatch_size = 32 \ndiscount_factor = 0.95 \noptimizer = keras.optimizers.Adam(lr=1e-3) \nloss_fn = keras.losses.mean_squared_error \n \ndef training_step(batch_size): \n experiences = sample_experiences(batch_size) \n states, actions, rewards, next_states, dones = experiences \n next_Q_values = model.predict(next_states) \n max_next_Q_values = np.max(next_Q_values, axis=1) \n target_Q_values = (rewards + \n (1 - dones) * discount_factor * max_next_Q_values) \n mask = tf.one_hot(actions, n_outputs) \n with tf.GradientTape() as tape: \n all_Q_values = model(states) \n Q_values = tf.reduce_sum(all_Q_values * mask, axis=1, keepdims=True) \n loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values)) \n grads = tape.gradient(loss, model.trainable_variables) \n optimizer.apply_gradients(zip(grads, model.trainable_variables))\nLet’s go through this code:\nFirst we define some hyperparameters, and we create the optimizer and\nthe loss function.\nThen we create the training_step() function. It starts by sampling a\nbatch of experiences, then it uses the DQN to predict the Q-Value for\neach possible action in each experience’s next state. Since we assume\nthat the agent will be playing optimally, we only keep the maximum Q-\nValue for each next state. Next, we use Equation 18-7 to compute the\ntarget Q-Value for each experience’s state-action pair.\nNext, we want to use the DQN to compute the Q-Value for each\nexperienced state-action pair. However, the DQN will also output the Q-\nValues for the other possible actions, not just for the action that was\nactually chosen by the agent. So we need to mask out all the Q-Values\nwe do not need. The tf.one_hot() function makes it easy to convert an\narray of action indices into such a mask. For example, if the first three\nexperiences contain actions 1, 1, 0, respectively, then the mask will start\nwith [[0, 1], [0, 1], [1, 0],...]. We can then multiply the\nDQN’s output with this mask, and this will zero out all the Q-Values we\ndo not want. We then sum over axis 1 to get rid of all the zeros, keeping\nonly the Q-Values of the experienced state-action pairs. This gives us\nthe Q_values tensor, containing one predicted Q-Value for each\nexperience in the batch.\nThen we compute the loss: it is the mean squared error between the\ntarget and predicted Q-Values for the experienced state-action pairs.\nFinally, we perform a Gradient Descent step to minimize the loss with\nregard to the model’s trainable variables.\nThis was the hardest part. Now training the model is straightforward:\nfor episode in range(600): \n obs = env.reset() \n for step in range(200): \n epsilon = max(1 - episode / 500, 0.01) \n obs, reward, done, info = play_one_step(env, obs, epsilon) \n if done: \n break \n if episode > 50: \n training_step(batch_size)\nWe run 600 episodes, each for a maximum of 200 steps. At each step, we first\ncompute the epsilon value for the ε-greedy policy: it will go from 1 down to\n0.01, linearly, in a bit under 500 episodes. Then we call the play_one_step()\nfunction, which will use the ε-greedy policy to pick an action, then execute it\nand record the experience in the replay buffer. If the episode is done, we exit the\nloop. Finally, if we are past the 50th episode, we call the training_step()\nfunction to train the model on one batch sampled from the replay buffer. The\nreason we play 50 episodes without training is to give the replay buffer some\ntime to fill up (if we don’t wait enough, then there will not be enough diversity\nin the replay buffer). And that’s it, we just implemented the Deep Q-Learning\nalgorithm!\nFigure 18-10 shows the total rewards the agent got during each episode.\nFigure 18-10. Learning curve of the Deep Q-Learning algorithm\nAs you can see, the algorithm made no apparent progress at all for almost 300\nepisodes (in part because ε was very high at the beginning), then its performance\nsuddenly skyrocketed up to 200 (which is the maximum possible performance in\nthis environment). That’s great news: the algorithm worked fine, and it actually\nran much faster than the Policy Gradient algorithm! But wait… just a few\nepisodes later, it forgot everything it knew, and its performance dropped below\n25! This is called catastrophic forgetting, and it is one of the big problems\nfacing virtually all RL algorithms: as the agent explores the environment, it\nupdates its policy, but what it learns in one part of the environment may break\nwhat it learned earlier in other parts of the environment. The experiences are\nquite correlated, and the learning environment keeps changing—this is not ideal\nfor Gradient Descent! If you increase the size of the replay buffer, the algorithm\nwill be less subject to this problem. Reducing the learning rate may also help.\nBut the truth is, Reinforcement Learning is hard: training is often unstable, and\nyou may need to try many hyperparameter values and random seeds before you\nfind a combination that works well. For example, if you try changing the number\nof neurons per layer in the preceding from 32 to 30 or 34, the performance will\nnever go above 100 (the DQN may be more stable with one hidden layer instead\nof two).\nNOTE\nReinforcement Learning is notoriously difficult, largely because of the training instabilities\nand the huge sensitivity to the choice of hyperparameter values and random seeds.\n As the\nresearcher Andrej Karpathy put it: “[Supervised learning] wants to work. […] RL must be\nforced to work.” You will need time, patience, perseverance, and perhaps a bit of luck too.\nThis is a major reason RL is not as widely adopted as regular Deep Learning (e.g.,\nconvolutional nets). But there are a few real-world applications, beyond AlphaGo and Atari\ngames: for example, Google uses RL to optimize its datacenter costs, and it is used in some\nrobotics applications, for hyperparameter tuning, and in recommender systems.\nYou might wonder why we didn’t plot the loss. It turns out that loss is a poor\nindicator of the model’s performance. The loss might go down, yet the agent\nmight perform worse (e.g., this can happen when the agent gets stuck in one\nsmall region of the environment, and the DQN starts overfitting this region).\nConversely, the loss could go up, yet the agent might perform better (e.g., if the\nDQN was underestimating the Q-Values, and it starts correctly increasing its\npredictions, the agent will likely perform better, getting more rewards, but the\nloss might increase because the DQN also sets the targets, which will be larger\ntoo).\nThe basic Deep Q-Learning algorithm we’ve been using so far would be too\nunstable to learn to play Atari games. So how did DeepMind do it? Well, they\n13\ntweaked the algorithm!\nDeep Q-Learning Variants\nLet’s look at a few variants of the Deep Q-Learning algorithm that can stabilize\nand speed up training.\nFixed Q-Value Targets\nIn the basic Deep Q-Learning algorithm, the model is used both to make\npredictions and to set its own targets. This can lead to a situation analogous to a\ndog chasing its own tail. This feedback loop can make the network unstable: it\ncan diverge, oscillate, freeze, and so on. To solve this problem, in their 2013\npaper the DeepMind researchers used two DQNs instead of one: the first is the\nonline model, which learns at each step and is used to move the agent around,\nand the other is the target model used only to define the targets. The target\nmodel is just a clone of the online model:\ntarget = keras.models.clone_model(model) \ntarget.set_weights(model.get_weights())\nThen, in the training_step() function, we just need to change one line to use\nthe target model instead of the online model when computing the Q-Values of\nthe next states:\nnext_Q_values = target.predict(next_states)\nFinally, in the training loop, we must copy the weights of the online model to the\ntarget model, at regular intervals (e.g., every 50 episodes):\nif episode % 50 == 0: \n target.set_weights(model.get_weights())\nSince the target model is updated much less often than the online model, the Q-\nValue targets are more stable, the feedback loop we discussed earlier is\ndampened, and its effects are less severe. This approach was one of the\nDeepMind researchers’ main contributions in their 2013 paper, allowing agents\nto learn to play Atari games from raw pixels. To stabilize training, they used a\ntiny learning rate of 0.00025, they updated the target model only every 10,000\nsteps (instead of the 50 in the previous code example), and they used a very\nlarge replay buffer of 1 million experiences. They decreased epsilon very\nslowly, from 1 to 0.1 in 1 million steps, and they let the algorithm run for 50\nmillion steps.\nLater in this chapter, we will use the TF-Agents library to train a DQN agent to\nplay Breakout using these hyperparameters, but before we get there, let’s take a\nlook at another DQN variant that managed to beat the state of the art once more.\nDouble DQN\nIn a 2015 paper, DeepMind researchers tweaked their DQN algorithm,\nincreasing its performance and somewhat stabilizing training. They called this\nvariant Double DQN. The update was based on the observation that the target\nnetwork is prone to overestimating Q-Values. Indeed, suppose all actions are\nequally good: the Q-Values estimated by the target model should be identical,\nbut since they are approximations, some may be slightly greater than others, by\npure chance. The target model will always select the largest Q-Value, which will\nbe slightly greater than the mean Q-Value, most likely overestimating the true\nQ-Value (a bit like counting the height of the tallest random wave when\nmeasuring the depth of a pool). To fix this, they proposed using the online model\ninstead of the target model when selecting the best actions for the next states,\nand using the target model only to estimate the Q-Values for these best actions.\nHere is the updated training_step() function:\ndef training_step(batch_size): \n experiences = sample_experiences(batch_size) \n states, actions, rewards, next_states, dones = experiences \n next_Q_values = model.predict(next_states) \n best_next_actions = np.argmax(next_Q_values, axis=1) \n next_mask = tf.one_hot(best_next_actions, n_outputs).numpy() \n next_best_Q_values = (target.predict(next_states) * next_mask).sum(axis=1) \n target_Q_values = (rewards + \n (1 - dones) * discount_factor * next_best_Q_values) \n mask = tf.one_hot(actions, n_outputs) \n [...] # the rest is the same as earlier\nJust a few months later, another improvement to the DQN algorithm was\nproposed.\n14\nPrioritized Experience Replay\nInstead of sampling experiences uniformly from the replay buffer, why not\nsample important experiences more frequently? This idea is called importance\nsampling (IS) or prioritized experience replay (PER), and it was introduced in a\n2015 paper\n by DeepMind researchers (once again!).\nMore specifically, experiences are considered “important” if they are likely to\nlead to fast learning progress. But how can we estimate this? One reasonable\napproach is to measure the magnitude of the TD error δ = r + γ·V(s′) – V(s). A\nlarge TD error indicates that a transition (s, r, s′) is very surprising, and thus\nprobably worth learning from. When an experience is recorded in the replay\nbuffer, its priority is set to a very large value, to ensure that it gets sampled at\nleast once. However, once it is sampled (and every time it is sampled), the TD\nerror δ is computed, and this experience’s priority is set to p = |δ| (plus a small\nconstant to ensure that every experience has a non-zero probability of being\nsampled). The probability P of sampling an experience with priority p is\nproportional to p , where ζ is a hyperparameter that controls how greedy we want\nimportance sampling to be: when ζ = 0, we just get uniform sampling, and when\nζ = 1, we get full-blown importance sampling. In the paper, the authors used ζ =\n0.6, but the optimal value will depend on the task.\nThere’s one catch, though: since the samples will be biased toward important\nexperiences, we must compensate for this bias during training by downweighting\nthe experiences according to their importance, or else the model will just overfit\nthe important experiences. To be clear, we want important experiences to be\nsampled more often, but this also means we must give them a lower weight\nduring training. To do this, we define each experience’s training weight as w = (n\nP) , where n is the number of experiences in the replay buffer, and β is a\nhyperparameter that controls how much we want to compensate for the\nimportance sampling bias (0 means not at all, while 1 means entirely). In the\npaper, the authors used β = 0.4 at the beginning of training and linearly increased\nit to β = 1 by the end of training. Again, the optimal value will depend on the\ntask, but if you increase one, you will usually want to increase the other as well.\nNow let’s look at one last important variant of the DQN algorithm.\nDueling DQN\n15\n16\nζ\n–β\nThe Dueling DQN algorithm (DDQN, not to be confused with Double DQN,\nalthough both techniques can easily be combined) was introduced in yet another\n2015 paper\n by DeepMind researchers. To understand how it works, we must\nfirst note that the Q-Value of a state-action pair (s, a) can be expressed as Q(s, a)\n= V(s) + A(s, a), where V(s) is the value of state s and A(s, a) is the advantage of\ntaking the action a in state s, compared to all other possible actions in that state.\nMoreover, the value of a state is equal to the Q-Value of the best action a for\nthat state (since we assume the optimal policy will pick the best action), so V(s)\n= Q(s, a ), which implies that A(s, a ) = 0. In a Dueling DQN, the model\nestimates both the value of the state and the advantage of each possible action.\nSince the best action should have an advantage of 0, the model subtracts the\nmaximum predicted advantage from all predicted advantages. Here is a simple\nDueling DQN model, implemented using the Functional API:\nK = keras.backend \ninput_states = keras.layers.Input(shape=[4]) \nhidden1 = keras.layers.Dense(32, activation=\"elu\")(input_states) \nhidden2 = keras.layers.Dense(32, activation=\"elu\")(hidden1) \nstate_values = keras.layers.Dense(1)(hidden2) \nraw_advantages = keras.layers.Dense(n_outputs)(hidden2) \nadvantages = raw_advantages - K.max(raw_advantages, axis=1, keepdims=True) \nQ_values = state_values + advantages \nmodel = keras.Model(inputs=[input_states], outputs=[Q_values])\nThe rest of the algorithm is just the same as earlier. In fact, you can build a\nDouble Dueling DQN and combine it with prioritized experience replay! More\ngenerally, many RL techniques can be combined, as DeepMind demonstrated in\na 2017 paper. The paper’s authors combined six different techniques into an\nagent called Rainbow, which largely outperformed the state of the art.\nUnfortunately, implementing all of these techniques, debugging them, fine-\ntuning them, and of course training the models can require a huge amount of\nwork. So instead of reinventing the wheel, it is often best to reuse scalable and\nwell-tested libraries, such as TF-Agents.\nThe TF-Agents Library\nThe TF-Agents library is a Reinforcement Learning library based on TensorFlow,\ndeveloped at Google and open sourced in 2018. Just like OpenAI Gym, it\nprovides many off-the-shelf environments (including wrappers for all OpenAI\n17\n*\n*\n*\n18\nGym environments), plus it supports the PyBullet library (for 3D physics\nsimulation), DeepMind’s DM Control library (based on MuJoCo’s physics\nengine), and Unity’s ML-Agents library (simulating many 3D environments). It\nalso implements many RL algorithms, including REINFORCE, DQN, and\nDDQN, as well as various RL components such as efficient replay buffers and\nmetrics. It is fast, scalable, easy to use, and customizable: you can create your\nown environments and neural nets, and you can customize pretty much any\ncomponent. In this section we will use TF-Agents to train an agent to play\nBreakout, the famous Atari game (see Figure 18-11 ), using the DQN algorithm\n(you can easily switch to another algorithm if you prefer).\n19\nFigure 18-11. The famous Breakout game\nInstalling TF-Agents\nLet’s start by installing TF-Agents. This can be done using pip (as always, if you\nare using a virtual environment, make sure to activate it first; if not, you will\nneed to use the --user option, or have administrator rights):\n$ python3 -m pip install --upgrade tf-agents \nWARNING\nAt the time of this writing, TF-Agents is still quite new and improving every day, so the API\nmay change a bit by the time you read this—but the big picture should remain the same, as\nwell as most of the code. If anything breaks, I will update the Jupyter notebook accordingly,\nso make sure to check it out.\nNext, let’s create a TF-Agents environment that will just wrap OpenAI GGym’s\nBreakout environment. For this, you must first install OpenAI Gym’s Atari\ndependencies:\n$ python3 -m pip install --upgrade 'gym[atari]' \nAmong other libraries, this command will install atari-py, which is a Python\ninterface for the Arcade Learning Environment (ALE), a framework built on top\nof the Atari 2600 emulator Stella.\nTF-Agents Environments\nIf everything went well, you should be able to import TF-Agents and create a\nBreakout environment:\n>>> from tf_agents.environments import suite_gym \n>>> env = suite_gym.load(\"Breakout-v4\") \n>>> env \n\nThis is just a wrapper around an OpenAI Gym environment, which you can\naccess through the gym attribute:\n>>> env.gym \n\nTF-Agents environments are very similar to OpenAI Gym environments, but\nthere are a few differences. First, the reset() method does not return an\nobservation; instead it returns a TimeStep object that wraps the observation, as\nwell as some extra information:\n>>> env.reset() \nTimeStep(step_type=array(0, dtype=int32), \n reward=array(0., dtype=float32), \n discount=array(1., dtype=float32), \n observation=array([[[0., 0., 0.], [0., 0., 0.],...]]], dtype=float32))\nThe step() method returns a TimeStep object as well:\n>>> env.step(1) # Fire \nTimeStep(step_type=array(1, dtype=int32), \n reward=array(0., dtype=float32), \n discount=array(1., dtype=float32), \n observation=array([[[0., 0., 0.], [0., 0., 0.],...]]], dtype=float32))\nThe reward and observation attributes are self-explanatory, and they are the\nsame as for OpenAI Gym (except the reward is represented as a NumPy array).\nThe step_type attribute is equal to 0 for the first time step in the episode, 1 for\nintermediate time steps, and 2 for the final time step. You can call the time\nstep’s is_last() method to check whether it is the final one or not. Lastly, the\ndiscount attribute indicates the discount factor to use at this time step. In this\nexample it is equal to 1, so there will be no discount at all. You can define the\ndiscount factor by setting the discount parameter when loading the\nenvironment.\nNOTE\nAt any time, you can access the environment’s current time step by calling its\ncurrent_time_step() method.\nEnvironment Specifications\nConveniently, a TF-Agents environment provides the specifications of the\nobservations, actions, and time steps, including their shapes, data types, and\nnames, as well as their minimum and maximum values:\n>>> env.observation_spec() \nBoundedArraySpec(shape=(210, 160, 3), dtype=dtype('float32'), name=None, \n minimum=[[[0. 0. 0.], [0. 0. 0.],...]], \n maximum=[[[255., 255., 255.], [255., 255., 255.], ...]]) \n>>> env.action_spec() \nBoundedArraySpec(shape=(), dtype=dtype('int64'), name=None, \n minimum=0, maximum=3) \n>>> env.time_step_spec() \nTimeStep(step_type=ArraySpec(shape=(), dtype=dtype('int32'), name='step_type'), \n reward=ArraySpec(shape=(), dtype=dtype('float32'), name='reward'), \n discount=BoundedArraySpec(shape=(), ..., minimum=0.0, maximum=1.0), \n observation=BoundedArraySpec(shape=(210, 160, 3), ...))\nAs you can see, the observations are simply screenshots of the Atari screen,\nrepresented as NumPy arrays of shape [210, 160, 3]. To render an environment,\nyou can call env.render(mode=\"human\"), and if you want to get back the image\nin the form of a NumPy array, just call env.render(mode=\"rgb_array\")\n(unlike in OpenAI Gym, this is the default mode).\nThere are four actions available. Gym’s Atari environments have an extra\nmethod that you can call to know what each action corresponds to:\n>>> env.gym.get_action_meanings() \n['NOOP', 'FIRE', 'RIGHT', 'LEFT']\nTIP\nSpecs can be instances of a specification class, nested lists, or dictionaries of specs. If the\nspecification is nested, then the specified object must match the specification’s nested\nstructure. For example, if the observation spec is {\"sensors\": ArraySpec(shape=[2]),\n\"camera\": ArraySpec(shape=[100, 100])}, then a valid observation would be\n{\"sensors\": np.array([1.5, 3.5]), \"camera\": np.array(...)}. The tf.nest\npackage provides tools to handle such nested structures (a.k.a. nests).\nThe observations are quite large, so we will downsample them and also convert\nthem to grayscale. This will speed up training and use less RAM. For this, we\ncan use an environment wrapper.\nEnvironment Wrappers and Atari Preprocessing\nTF-Agents provides several environment wrappers in the\ntf_agents.environments.wrappers package. As their name suggests, they\nwrap an environment, forwarding every call to it, but also adding some extra\nfunctionality. Here are some of the available wrappers:\nActionClipWrapper\nClips the actions to the action spec.\nActionDiscretizeWrapper\nQuantizes a continuous action space to a discrete action space. For example,\nif the original environment’s action space is the continuous range from –1.0\nto +1.0, but you want to use an algorithm that only supports discrete action\nspaces, such as a DQN, then you can wrap the environment using\ndiscrete_env = ActionDiscretizeWrapper(env, num_actions=5), and\nthe new discrete_env will have a discrete action space with five possible\nactions: 0, 1, 2, 3, 4. These actions correspond to the actions –1.0, –0.5, 0.0,\n0.5, and 1.0 in the original environment.\nActionRepeat\nRepeats each action over n steps, while accumulating the rewards. In many\nenvironments, this can speed up training significantly.\nRunStats\nRecords environment statistics such as the number of steps and the number\nof episodes.\nTimeLimit\nInterrupts the environment if it runs for longer than a maximum number of\nsteps.\nVideoWrapper\nRecords a video of the environment.\nTo create a wrapped environment, you must create a wrapper, passing the\nwrapped environment to the constructor. That’s all! For example, the following\ncode will wrap our environment in an ActionRepeat wrapper so that every\naction is repeated four times:\nfrom tf_agents.environments.wrappers import ActionRepeat \n \nrepeating_env = ActionRepeat(env, times=4)\nOpenAI Gym has some environment wrappers of its own in the gym.wrappers\npackage. They are meant to wrap Gym environments, though, not TF-Agents\nenvironments, so to use them you must first wrap the Gym environment with a\nGym wrapper, then wrap the resulting environment with a TF-Agents wrapper.\nThe suite_gym.wrap_env() function will do this for you, provided you give it a\nGym environment and a list of Gym wrappers and/or a list of TF-Agents\nwrappers. Alternatively, the suite_gym.load() function will both create the\nGym environment and wrap it for you, if you give it some wrappers. Each\nwrapper will be created without any arguments, so if you want to set some\narguments, you must pass a lambda. For example, the following code creates a\nBreakout environment that will run for a maximum of 10,000 steps during each\nepisode, and each action will be repeated four times:\nfrom gym.wrappers import TimeLimit \n \nlimited_repeating_env = suite_gym.load( \n \"Breakout-v4\", \n gym_env_wrappers=[lambda env: TimeLimit(env, max_episode_steps=10000)], \n env_wrappers=[lambda env: ActionRepeat(env, times=4)])\nFor Atari environments, some standard preprocessing steps are applied in most\npapers that use them, so TF-Agents provides a handy AtariPreprocessing\nwrapper that implements them. Here is the list of preprocessing steps it\nsupports:\nGrayscale and downsampling\nObservations are converted to grayscale and downsampled (by default to 84\n× 84 pixels).\nMax pooling\nThe last two frames of the game are max-pooled using a 1 × 1 filter. This is\nto remove the flickering that occurs in some Atari games due to the limited\nnumber of sprites that the Atari 2600 could display in each frame.\nFrame skipping\nThe agent only gets to see every n frames of the game (by default n = 4), and\nits actions are repeated for each frame, collecting all the rewards. This\neffectively speeds up the game from the perspective of the agent, and it also\nspeeds up training because rewards are less delayed.\nEnd on life lost\nIn some games, the rewards are just based on the score, so the agent gets no\nimmediate penalty for losing a life. One solution is to end the game\nimmediately whenever a life is lost. There is some debate over the actual\nbenefits of this strategy, so it is off by default.\nSince the default Atari environment already applies random frame skipping and\nmax pooling, we will need to load the raw, nonskipping variant called\n\"BreakoutNoFrameskip-v4\". Moreover, a single frame from the Breakout game\nis insufficient to know the direction and speed of the ball, which will make it\nvery difficult for the agent to play the game properly (unless it is an RNN agent,\nwhich preserves some internal state between steps). One way to handle this is to\nuse an environment wrapper that will output observations composed of multiple\nframes stacked on top of each other along the channels dimension. This strategy\nis implemented by the FrameStack4 wrapper, which returns stacks of four\nframes. Let’s create the wrapped Atari environment!\nfrom tf_agents.environments import suite_atari \nfrom tf_agents.environments.atari_preprocessing import AtariPreprocessing \nfrom tf_agents.environments.atari_wrappers import FrameStack4 \n \nmax_episode_steps = 27000 # <=> 108k ALE frames since 1 step = 4 frames \nenvironment_name = \"BreakoutNoFrameskip-v4\" \n \nenv = suite_atari.load( \n environment_name, \n max_episode_steps=max_episode_steps, \n gym_env_wrappers=[AtariPreprocessing, FrameStack4])\nThe result of all this preprocessing is shown in Figure 18-12. You can see that\nthe resolution is much lower, but sufficient to play the game. Moreover, frames\nare stacked along the channels dimension, so red represents the frame from three\nsteps ago, green is two steps ago, blue is the previous frame, and pink is the\ncurrent frame. From this single observation, the agent can see that the ball is\ngoing toward the lower-left corner, and that it should continue to move the\npaddle to the left (as it did in the previous steps).\n20\nFigure 18-12. Preprocessed Breakout observation\nLastly, we can wrap the environment inside a TFPyEnvironment:\nfrom tf_agents.environments.tf_py_environment import TFPyEnvironment \n \ntf_env = TFPyEnvironment(env)\nThis will make the environment usable from within a TensorFlow graph (under\nthe hood, this class relies on tf.py_function(), which allows a graph to call\narbitrary Python code). Thanks to the TFPyEnvironment class, TF-Agents\nsupports both pure Python environments and TensorFlow-based environments.\nMore generally, TF-Agents supports and provides both pure Python and\nTensorFlow-based components (agents, replay buffers, metrics, and so on).\nNow that we have a nice Breakout environment, with all the appropriate\npreprocessing and TensorFlow support, we must create the DQN agent and the\nother components we will need to train it. Let’s look at the architecture of the\nsystem we will build.\nTraining Architecture\nA TF-Agents training program is usually split into two parts that run in parallel,\nas you can see in Figure 18-13: on the left, a driver explores the environment\nusing a collect policy to choose actions, and it collects trajectories (i.e.,\nexperiences), sending them to an observer, which saves them to a replay buffer;\non the right, an agent pulls batches of trajectories from the replay buffer and\ntrains some networks, which the collect policy uses. In short, the left part\nexplores the environment and collects trajectories, while the right part learns\nand updates the collect policy.\nFigure 18-13. A typical TF-Agents training architecture\nThis figure begs a few questions, which I’ll attempt to answer here:\nWhy are there multiple environments? Instead of exploring a single\nenvironment, you generally want the driver to explore multiple copies\nof the environment in parallel, taking advantage of the power of all your\nCPU cores, keeping the training GPUs busy, and providing less-\ncorrelated trajectories to the training algorithm.\nWhat is a trajectory? It is a concise representation of a transition from\none time step to the next, or a sequence of consecutive transitions from\ntime step n to time step n + t. The trajectories collected by the driver are\npassed to the observer, which saves them in the replay buffer, and they\nare later sampled by the agent and used for training.\nWhy do we need an observer? Can’t the driver save the trajectories\ndirectly? Indeed, it could, but this would make the architecture less\nflexible. For example, what if you don’t want to use a replay buffer?\nWhat if you want to use the trajectories for something else, like\ncomputing metrics? In fact, an observer is just any function that takes a\ntrajectory as an argument. You can use an observer to save the\ntrajectories to a replay buffer, or to save them to a TFRecord file (see\nChapter 13), or to compute metrics, or for anything else. Moreover, you\ncan pass multiple observers to the driver, and it will broadcast the\ntrajectories to all of them.\nTIP\nAlthough this architecture is the most common, you can customize it as you please, and\neven replace some components with your own. In fact, unless you are researching new RL\nalgorithms, you will most likely want to use a custom environment for your task. For this,\nyou just need to create a custom class that inherits from the PyEnvironment class in the\ntf_agents.environments.py_environment package and overrides the appropriate\nmethods, such as action_spec(), observation_spec(), _reset(), and _step() (see\nthe “Creating a Custom TF_Agents Environment” section of the notebook for an example).\nNow we will create all these components: first the Deep Q-Network, then the\nDQN agent (which will take care of creating the collect policy), then the replay\nbuffer and the observer to write to it, then a few training metrics, then the driver,\nand finally the dataset. Once we have all the components in place, we will\npopulate the replay buffer with some initial trajectories, then we will run the\nmain training loop. So, let’s start by creating the Deep Q-Network.\nCreating the Deep Q-Network\nThe TF-Agents library provides many networks in the tf_agents.networks\npackage and its subpackages. We will use the\ntf_agents.networks.q_network.QNetwork class:\nfrom tf_agents.networks.q_network import QNetwork \n \npreprocessing_layer = keras.layers.Lambda( \n lambda obs: tf.cast(obs, np.float32) / 255.) \nconv_layer_params=[(32, (8, 8), 4), (64, (4, 4), 2), (64, (3, 3), 1)] \nfc_layer_params=[512] \n \nq_net = QNetwork( \n tf_env.observation_spec(), \n tf_env.action_spec(), \n preprocessing_layers=preprocessing_layer, \n conv_layer_params=conv_layer_params, \n fc_layer_params=fc_layer_params)\nThis QNetwork takes an observation as input and outputs one Q-Value per action,\nso we must give it the specifications of the observations and the actions. It starts\nwith a preprocessing layer: a simple Lambda layer that casts the observations to\n32-bit floats and normalizes them (the values will range from 0.0 to 1.0). The\nobservations contain unsigned bytes, which use 4 times less space than 32-bit\nfloats, which is why we did not cast the observations to 32-bit floats earlier; we\nwant to save RAM in the replay buffer. Next, the network applies three\nconvolutional layers: the first has 32 8 × 8 filters and uses a stride of 4, the\nsecond has 64 4 × 4 filters and a stride of 2, and the third has 64 3 × 3 filters and\na stride of 1. Lastly, it applies a dense layer with 512 units, followed by a dense\noutput layer with 4 units, one per Q-Value to output (i.e., one per action). All\nconvolutional layers and all dense layers except the output layer use the ReLU\nactivation function by default (you can change this by setting the\nactivation_fn argument). The output layer does not use any activation\nfunction.\nUnder the hood, a QNetwork is composed of two parts: an encoding network that\nprocesses the observations, followed by a dense output layer that outputs one Q-\nValue per action. TF-Agent’s EncodingNetwork class implements a neural\nnetwork architecture found in various agents (see Figure 18-14).\nIt may have one or more inputs. For example, if each observation is composed of\nsome sensor data plus an image from a camera, you will have two inputs. Each\ninput may require some preprocessing steps, in which case you can specify a list\nof Keras layers via the preprocessing_layers argument, with one\npreprocessing layer per input, and the network will apply each layer to the\ncorresponding input (if an input requires multiple layers of preprocessing, you\ncan pass a whole model, since a Keras model can always be used as a layer). If\nthere are two inputs or more, you must also pass an extra layer via the\npreprocessing_combiner argument, to combine the outputs from the\npreprocessing layers into a single output.\nNext, the encoding network will optionally apply a list of convolutions\nsequentially, provided you specify their parameters via the conv_layer_params\nargument. This must be a list composed of 3-tuples (one per convolutional layer)\nindicating the number of filters, the kernel size, and the stride. After these\nconvolutional layers, the encoding network will optionally apply a sequence of\ndense layers, if you set the fc_layer_params argument: it must be a list\ncontaining the number of neurons for each dense layer. Optionally, you can also\npass a list of dropout rates (one per dense layer) via the dropout_layer_params\nargument if you want to apply dropout after each dense layer. The QNetwork\ntakes the output of this encoding network and passes it to the dense output layer\n(with one unit per action).\nFigure 18-14. Architecture of an encoding network\nNOTE\nThe QNetwork class is flexible enough to build many different architectures, but you can\nalways build your own network class if you need extra flexibility: extend the\ntf_agents.networks.Network class and implement it like a regular custom Keras layer.\nThe tf_agents.networks.Network class is a subclass of the keras.layers.Layer class\nthat adds some functionality required by some agents, such as the possibility to easily create\nshallow copies of the network (i.e., copying the network’s architecture, but not its weights).\nFor example, the DQNAgent uses this to create a copy of the online model.\nNow that we have the DQN, we are ready to build the DQN agent.\nCreating the DQN Agent\nThe TF-Agents library implements many types of agents, located in the\ntf_agents .agents package and its subpackages. We will use the\ntf_agents.agents .dqn.dqn_agent.DqnAgent class:\nfrom tf_agents.agents.dqn.dqn_agent import DqnAgent \n \ntrain_step = tf.Variable(0) \nupdate_period = 4 # train the model every 4 steps \noptimizer = keras.optimizers.RMSprop(lr=2.5e-4, rho=0.95, momentum=0.0, \n epsilon=0.00001, centered=True) \nepsilon_fn = keras.optimizers.schedules.PolynomialDecay( \n initial_learning_rate=1.0, # initial ε \n decay_steps=250000 // update_period, # <=> 1,000,000 ALE frames \n end_learning_rate=0.01) # final ε \nagent = DqnAgent(tf_env.time_step_spec(), \n tf_env.action_spec(), \n q_network=q_net, \n optimizer=optimizer, \n target_update_period=2000, # <=> 32,000 ALE frames \n td_errors_loss_fn=keras.losses.Huber(reduction=\"none\"), \n gamma=0.99, # discount factor \n train_step_counter=train_step, \n epsilon_greedy=lambda: epsilon_fn(train_step)) \nagent.initialize()\nLet’s walk through this code:\nWe first create a variable that will count the number of training steps.\nThen we build the optimizer, using the same hyperparameters as in the\n2015 DQN paper.\nNext, we create a PolynomialDecay object that will compute the ε\nvalue for the ε-greedy collect policy, given the current training step (it\nis normally used to decay the learning rate, hence the names of the\narguments, but it will work just fine to decay any other value). It will go\nfrom 1.0 down to 0.01 (the value used during in the 2015 DQN paper) in\n1 million ALE frames, which corresponds to 250,000 steps, since we\nuse frame skipping with a period of 4. Moreover, we will train the agent\nevery 4 steps (i.e., 16 ALE frames), so ε will actually decay over 62,500\ntraining steps.\nWe then build the DQNAgent, passing it the time step and action specs,\nthe QNetwork to train, the optimizer, the number of training steps\nbetween target model updates, the loss function to use, the discount\nfactor, the train_step variable, and a function that returns the ε value\n(it must take no argument, which is why we need a lambda to pass the\ntrain_step).\nNote that the loss function must return an error per instance, not the\nmean error, which is why we set reduction=\"none\".\nLastly, we initialize the agent.\nNext, let’s build the replay buffer and the observer that will write to it.\nCreating the Replay Buffer and the Corresponding Observer\nThe TF-Agents library provides various replay buffer implementations in the\ntf_agents.replay_buffers package. Some are purely written in Python (their\nmodule names start with py_), and others are written based on TensorFlow (their\nmodule names start with tf_). We will use the TFUniformReplayBuffer class in\nthe tf_agents.replay_buffers.tf_uniform_replay_buffer package. It\nprovides a high-performance implementation of a replay buffer with uniform\nsampling:\nfrom tf_agents.replay_buffers import tf_uniform_replay_buffer \n \nreplay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer( \n data_spec=agent.collect_data_spec, \n batch_size=tf_env.batch_size, \n max_length=1000000)\nLet’s look at each of these arguments:\ndata_spec\nThe specification of the data that will be saved in the replay buffer. The DQN\nagent knowns what the collected data will look like, and it makes the data\nspec available via its collect_data_spec attribute, so that’s what we give\nthe replay buffer.\nbatch_size\nThe number of trajectories that will be added at each step. In our case, it will\nbe one, since the driver will just execute one action per step and collect one\ntrajectory. If the environment were a batched environment, meaning an\nenvironment that takes a batch of actions at each step and returns a batch of\nobservations, then the driver would have to save a batch of trajectories at\neach step. Since we are using a TensorFlow replay buffer, it needs to know\nthe size of the batches it will handle (to build the computation graph). An\nexample of a batched environment is the ParallelPyEnvironment (from the\ntf_agents.environments.parallel_py_environment package): it runs\nmultiple environments in parallel in separate processes (they can be different\n21\nas long as they have the same action and observation specs), and at each step\nit takes a batch of actions and executes them in the environments (one action\nper environment), then it returns all the resulting observations.\nmax_length\nThe maximum size of the replay buffer. We created a large replay buffer that\ncan store one million trajectories (as was done in the 2015 DQN paper). This\nwill require a lot of RAM.\nTIP\nWhen we store two consecutive trajectories, they contain two consecutive observations with\nfour frames each (since we used the FrameStack4 wrapper), and unfortunately three of the\nfour frames in the second observation are redundant (they are already present in the first\nobservation). In other words, we are using about four times more RAM than necessary. To\navoid this, you can instead use a PyHashedReplayBuffer from the\ntf_agents.replay_buffers.py_hashed_replay_buffer package: it deduplicates data\nin the stored trajectories along the last axis of the observations.\nNow we can create the observer that will write the trajectories to the replay\nbuffer. An observer is just a function (or a callable object) that takes a trajectory\nargument, so we can directly use the add_method() method (bound to the\nreplay_buffer object) as our observer:\nreplay_buffer_observer = replay_buffer.add_batch\nIf you wanted to create your own observer, you could write any function with a\ntrajectory argument. If it must have a state, you can write a class with a\n__call__(self, trajectory) method. For example, here is a simple observer\nthat will increment a counter every time it is called (except when the trajectory\nrepresents a boundary between two episodes, which does not count as a step),\nand every 100 increments it displays the progress up to a given total (the\ncarriage return \\r along with end=\"\" ensures that the displayed counter remains\non the same line):\nclass ShowProgress: \n def __init__(self, total): \n self.counter = 0 \n self.total = total \n def __call__(self, trajectory): \n if not trajectory.is_boundary(): \n self.counter += 1 \n if self.counter % 100 == 0: \n print(\"\\r{}/{}\".format(self.counter, self.total), end=\"\")\nNow let’s create a few training metrics.\nCreating Training Metrics\nTF-Agents implements several RL metrics in the tf_agents.metrics package,\nsome purely in Python and some based on TensorFlow. Let’s create a few of\nthem in order to count the number of episodes, the number of steps taken, and\nmost importantly the average return per episode and the average episode length:\nfrom tf_agents.metrics import tf_metrics \n \ntrain_metrics = [ \n tf_metrics.NumberOfEpisodes(), \n tf_metrics.EnvironmentSteps(), \n tf_metrics.AverageReturnMetric(), \n tf_metrics.AverageEpisodeLengthMetric(), \n]\nNOTE\nDiscounting the rewards makes sense for training or to implement a policy, as it makes it\npossible to balance the importance of immediate rewards with future rewards. However,\nonce an episode is over, we can evaluate how good it was overalls by summing the\nundiscounted rewards. For this reason, the AverageReturnMetric computes the sum of\nundiscounted rewards for each episode, and it keeps track of the streaming mean of these\nsums over all the episodes it encounters.\nAt any time, you can get the value of each of these metrics by calling its\nresult() method (e.g., train_metrics[0].result()). Alternatively, you can\nlog all metrics by calling log_metrics(train_metrics) (this function is\nlocated in the tf_agents.eval.metric_utils package):\n>>> from tf_agents.eval.metric_utils import log_metrics \n>>> import logging \n>>> logging.get_logger().set_level(logging.INFO) \n>>> log_metrics(train_metrics) \n[...] \nNumberOfEpisodes = 0 \nEnvironmentSteps = 0 \nAverageReturn = 0.0 \nAverageEpisodeLength = 0.0\nNext, let’s create the collect driver.\nCreating the Collect Driver\nAs we explored in Figure 18-13, a driver is an object that explores an\nenvironment using a given policy, collects experiences, and broadcasts them to\nsome observers. At each step, the following things happen:\nThe driver passes the current time step to the collect policy, which uses\nthis time step to choose an action and returns an action step object\ncontaining the action.\nThe driver then passes the action to the environment, which returns the\nnext time step.\nFinally, the driver creates a trajectory object to represent this transition\nand broadcasts it to all the observers.\nSome policies, such as RNN policies, are stateful: they choose an action based\non both the given time step and their own internal state. Stateful policies return\ntheir own state in the action step, along with the chosen action. The driver will\nthen pass this state back to the policy at the next time step. Moreover, the driver\nsaves the policy state to the trajectory (in the policy_info field), so it ends up\nin the replay buffer. This is essential when training a stateful policy: when the\nagent samples a trajectory, it must set the policy’s state to the state it was in at\nthe time of the sampled time step.\nAlso, as discussed earlier, the environment may be a batched environment, in\nwhich case the driver passes a batched time step to the policy (i.e., a time step\nobject containing a batch of observations, a batch of step types, a batch of\nrewards, and a batch of discounts, all four batches of the same size). The driver\nalso passes a batch of previous policy states. The policy then returns a batched\naction step containing a batch of actions and a batch of policy states. Finally, the\ndriver creates a batched trajectory (i.e., a trajectory containing a batch of step\ntypes, a batch of observations, a batch of actions, a batch of rewards, and more\ngenerally a batch for each trajectory attribute, with all batches of the same size).\nThere are two main driver classes: DynamicStepDriver and\nDynamicEpisodeDriver. The first one collects experiences for a given number\nof steps, while the second collects experiences for a given number of episodes.\nWe want to collect experiences for four steps for each training iteration (as was\ndone in the 2015 DQN paper), so let’s create a DynamicStepDriver:\nfrom tf_agents.drivers.dynamic_step_driver import DynamicStepDriver \n \ncollect_driver = DynamicStepDriver( \n tf_env, \n agent.collect_policy, \n observers=[replay_buffer_observer] + training_metrics, \n num_steps=update_period) # collect 4 steps for each training iteration\nWe give it the environment to play with, the agent’s collect policy, a list of\nobservers (including the replay buffer observer and the training metrics), and\nfinally the number of steps to run (in this case, four). We could now run it by\ncalling its run() method, but it’s best to warm up the replay buffer with\nexperiences collected using a purely random policy. For this, we can use the\nRandomTFPolicy class and create a second driver that will run this policy for\n20,000 steps (which is equivalent to 80,000 simulator frames, as was done in the\n2015 DQN paper). We can use our ShowProgress observer to display the\nprogress:\nfrom tf_agents.policies.random_tf_policy import RandomTFPolicy \n \ninitial_collect_policy = RandomTFPolicy(tf_env.time_step_spec(), \n tf_env.action_spec()) \ninit_driver = DynamicStepDriver( \n tf_env, \n initial_collect_policy, \n observers=[replay_buffer.add_batch, ShowProgress(20000)], \n num_steps=20000) # <=> 80,000 ALE frames \nfinal_time_step, final_policy_state = init_driver.run()\nWe’re almost ready to run the training loop! We just need one last component:\nthe dataset.\nCreating the Dataset\nTo sample a batch of trajectories from the replay buffer, call its get_next()\nmethod. This returns the batch of trajectories plus a BufferInfo object that\ncontains the sample identifiers and their sampling probabilities (this may be\nuseful for some algorithms, such as PER). For example, the following code will\nsample a small batch of two trajectories (subepisodes), each containing three\nconsecutive steps. These subepisodes are shown in Figure 18-15 (each row\ncontains three consecutive steps from an episode):\n>>> trajectories, buffer_info = replay_buffer.get_next( \n... sample_batch_size=2, num_steps=3) \n... \n>>> trajectories._fields \n('step_type', 'observation', 'action', 'policy_info', \n 'next_step_type', 'reward', 'discount') \n>>> trajectories.observation.shape \nTensorShape([2, 3, 84, 84, 4]) \n>>> trajectories.step_type.numpy() \narray([[1, 1, 1], \n [1, 1, 1]], dtype=int32)\nThe trajectories object is a named tuple, with seven fields. Each field\ncontains a tensor whose first two dimensions are 2 and 3 (since there are two\ntrajectories, each with three steps). This explains why the shape of the\nobservation field is [2, 3, 84, 84, 4]: that’s two trajectories, each with three\nsteps, and each step’s observation is 84 × 84 × 4. Similarly, the step_type tensor\nhas a shape of [2, 3]: in this example, both trajectories contain three consecutive\nsteps in the middle on an episode (types 1, 1, 1). In the second trajectory, you\ncan barely see the ball at the lower left of the first observation, and it disappears\nin the next two observations, so the agent is about to lose a life, but the episode\nwill not end immediately because it still has several lives left.\nFigure 18-15. Two trajectories containing three consecutive steps each\nEach trajectory is a concise representation of a sequence of consecutive time\nsteps and action steps, designed to avoid redundancy. How so? Well, as you can\nsee in Figure 18-16, transition n is composed of time step n, action step n, and\ntime step n + 1, while transition n + 1 is composed of time step n + 1, action step\nn + 1, and time step n + 2. If we just stored these two transitions directly in the\nreplay buffer, the time step n + 1 would be duplicated. To avoid this duplication,\nthe n trajectory step includes only the type and observation from time step n\n(not its reward and discount), and it does not contain the observation from time\nstep n + 1 (however, it does contain a copy of the next time step’s type; that’s the\nonly duplication).\nth\nFigure 18-16. Trajectories, transitions, time steps, and action steps\nSo if you have a batch of trajectories where each trajectory has t + 1 steps (from\ntime step n to time step n + t), then it contains all the data from time step n to\ntime step n + t, except for the reward and discount from time step n (but it\ncontains the reward and discount of time step n + t + 1). This represents t\ntransitions (n to n + 1, n + 1 to n + 2, …, n + t – 1 to n + t).\nThe to_transition() function in the tf_agents.trajectories.trajectory\nmodule converts a batched trajectory into a list containing a batched time_step,\na batched action_step, and a batched next_time_step. Notice that the second\ndimension is 2 instead of 3, since there are t transitions between t + 1 time steps\n(don’t worry if you’re a bit confused; you’ll get the hang of it):\n>>> from tf_agents.trajectories.trajectory import to_transition \n>>> time_steps, action_steps, next_time_steps = to_transition(trajectories) \n>>> time_steps.observation.shape \nTensorShape([2, 2, 84, 84, 4]) # 3 time steps = 2 transitions\nNOTE\nA sampled trajectory may actually overlap two (or more) episodes! In this case, it will\ncontain boundary transitions, meaning transitions with a step_type equal to 2 (end) and a\nnext_step_type equal to 0 (start). Of course, TF-Agents properly handles such trajectories\n(e.g., by resetting the policy state when encountering a boundary). The trajectory’s\nis_boundary() method returns a tensor indicating whether each step is a boundary or not.\nFor our main training loop, instead of calling the get_next() method, we will\nuse a tf.data.Dataset. This way, we can benefit from the power of the Data\nAPI (e.g., parallelism and prefetching). For this, we call the replay buffer’s\nas_dataset() method:\ndataset = replay_buffer.as_dataset( \n sample_batch_size=64, \n num_steps=2, \n num_parallel_calls=3).prefetch(3)\nWe will sample batches of 64 trajectories at each training step (as in the 2015\nDQN paper), each with 2 steps (i.e., 2 steps = 1 full transition, including the next\nstep’s observation). This dataset will process three elements in parallel, and\nprefetch three batches.\nNOTE\nFor on-policy algorithms such as Policy Gradients, each experience should be sampled once,\nused from training, and then discarded. In this case, you can still use a replay buffer, but\ninstead of using a Dataset, you would call the replay buffer’s gather_all() method at\neach training iteration to get a tensor containing all the trajectories recorded so far, then use\nthem to perform a training step, and finally clear the replay buffer by calling its clear()\nmethod.\nNow that we have all the components in place, we are ready to train the model!\nCreating the Training Loop\nTo speed up training, we will convert the main functions to TensorFlow\nFunctions. For this we will use the tf_agents.utils.common.function()\nfunction, which wraps tf.function(), with some extra experimental options:\nfrom tf_agents.utils.common import function \n \ncollect_driver.run = function(collect_driver.run) \nagent.train = function(agent.train)\nLet’s create a small function that will run the main training loop for\nn_iterations:\ndef train_agent(n_iterations): \n time_step = None \n policy_state = agent.collect_policy.get_initial_state(tf_env.batch_size) \n iterator = iter(dataset) \n for iteration in range(n_iterations): \n time_step, policy_state = collect_driver.run(time_step, policy_state) \n trajectories, buffer_info = next(iterator) \n train_loss = agent.train(trajectories) \n print(\"\\r{} loss:{:.5f}\".format( \n iteration, train_loss.loss.numpy()), end=\"\") \n if iteration % 1000 == 0: \n log_metrics(train_metrics)\nThe function first asks the collect policy for its initial state (given the\nenvironment batch size, which is 1 in this case). Since the policy is stateless, this\nreturns an empty tuple (so we could have written policy_state = ()). Next,\nwe create an iterator over the dataset, and we run the training loop. At each\niteration, we call the driver’s run() method, passing it the current time step\n(initially None) and the current policy state. It will run the collect policy and\ncollect experience for four steps (as we configured earlier), broadcasting the\ncollected trajectories to the replay buffer and the metrics. Next, we sample one\nbatch of trajectories from the dataset, and we pass it to the agent’s train()\nmethod. It returns a train_loss object which may vary depending on the type\nof agent. Next, we display the iteration number and the training loss, and every\n1,000 iterations we log all the metrics. Now you can just call train_agent() for\nsome number of iterations, and see the agent gradually learn to play Breakout!\ntrain_agent(10000000)\nThis will take a lot of computing power and a lot of patience (it may take hours,\nor even days, depending on your hardware), plus you may need to run the\nalgorithm several times with different random seeds to get good results, but\nonce it’s done, the agent will be superhuman (at least at Breakout). You can also\ntry training this DQN agent on other Atari games: it can achieve superhuman\nskill at most action games, but it is not so good at games with long-running\nstorylines.\nOverview of Some Popular RL Algorithms\nBefore we finish this chapter, let’s take a quick look at a few popular RL\nalgorithms:\nActor-Critic algorithms\nA family of RL algorithms that combine Policy Gradients with Deep Q-\nNetworks. An Actor-Critic agent contains two neural networks: a policy net\nand a DQN. The DQN is trained normally, by learning from the agent’s\nexperiences. The policy net learns differently (and much faster) than in\nregular PG: instead of estimating the value of each action by going through\nmultiple episodes, then summing the future discounted rewards for each\naction, and finally normalizing them, the agent (actor) relies on the action\nvalues estimated by the DQN (critic). It’s a bit like an athlete (the agent)\nlearning with the help of a coach (the DQN).\nAsynchronous Advantage Actor-Critic\n (A3C)\nAn important Actor-Critic variant introduced by DeepMind researchers in\n2016, where multiple agents learn in parallel, exploring different copies of\nthe environment. At regular intervals, but asynchronously (hence the name),\neach agent pushes some weight updates to a master network, then it pulls the\nlatest weights from that network. Each agent thus contributes to improving\nthe master network and benefits from what the other agents have learned.\nMoreover, instead of estimating the Q-Values, the DQN estimates the\nadvantage of each action (hence the second A in the name), which stabilizes\ntraining.\nAdvantage Actor-Critic (A2C)\nA variant of the A3C algorithm that removes the asynchronicity. All model\nupdates are synchronous, so gradient updates are performed over larger\nbatches, which allows the model to better utilize the power of the GPU.\n22\n23\nSoft Actor-Critic\n (SAC)\nAn Actor-Critic variant proposed in 2018 by Tuomas Haarnoja and other UC\nBerkeley researchers. It learns not only rewards, but also to maximize the\nentropy of its actions. In other words, it tries to be as unpredictable as\npossible while still getting as many rewards as possible. This encourages the\nagent to explore the environment, which speeds up training, and makes it\nless likely to repeatedly execute the same action when the DQN produces\nimperfect estimates. This algorithm has demonstrated an amazing sample\nefficiency (contrary to all the previous algorithms, which learn very slowly).\nSAC is available in TF-Agents.\nProximal Policy Optimization (PPO)\nAn algorithm based on A2C that clips the loss function to avoid excessively\nlarge weight updates (which often lead to training instabilities). PPO is a\nsimplification of the previous Trust Region Policy Optimization (TRPO)\nalgorithm, also by John Schulman and other OpenAI researchers. OpenAI\nmade the news in April 2019 with their AI called OpenAI Five, based on the\nPPO algorithm, which defeated the world champions at the multiplayer game\nDota 2. PPO is also available in TF-Agents.\nCuriosity-based exploration\nA recurring problem in RL is the sparsity of the rewards, which makes\nlearning very slow and inefficient. Deepak Pathak and other UC Berkeley\nresearchers have proposed an exciting way to tackle this issue: why not\nignore the rewards, and just make the agent extremely curious to explore the\nenvironment? The rewards thus become intrinsic to the agent, rather than\ncoming from the environment. Similarly, stimulating curiosity in a child is\nmore likely to give good results than purely rewarding the child for getting\ngood grades. How does this work? The agent continuously tries to predict the\noutcome of its actions, and it seeks situations where the outcome does not\nmatch its predictions. In other words, it wants to be surprised. If the outcome\nis predictable (boring), it goes elsewhere. However, if the outcome is\nunpredictable but the agent notices that it has no control over it, it also gets\nbored after a while. With only curiosity, the authors succeeded in training an\nagent at many video games: even though the agent gets no penalty for losing,\nthe game starts over, which is boring so it learns to avoid it.\n24\n25\n26\n27\nWe covered many topics in this chapter: Policy Gradients, Markov chains,\nMarkov decision processes, Q-Learning, Approximate Q-Learning, and Deep Q-\nLearning and its main variants (fixed Q-Value targets, Double DQN, Dueling\nDQN, and prioritized experience replay). We discussed how to use TF-Agents to\ntrain agents at scale, and finally we took a quick look at a few other popular\nalgorithms. Reinforcement Learning is a huge and exciting field, with new ideas\nand algorithms popping out every day, so I hope this chapter sparked your\ncuriosity: there is a whole world to explore!\nExercises\n1. How would you define Reinforcement Learning? How is it different\nfrom regular supervised or unsupervised learning?\n2. Can you think of three possible applications of RL that were not\nmentioned in this chapter? For each of them, what is the environment?\nWhat is the agent? What are some possible actions? What are the\nrewards?\n3. What is the discount factor? Can the optimal policy change if you\nmodify the discount factor?\n4. How do you measure the performance of a Reinforcement Learning\nagent?\n5. What is the credit assignment problem? When does it occur? How can\nyou alleviate it?\n6. What is the point of using a replay buffer?\n7. What is an off-policy RL algorithm?\n8. Use policy gradients to solve OpenAI Gym’s LunarLander-v2\nenvironment. You will need to install the Box2D dependencies\n(python3 -m pip install gym[box2d]).\n9. Use TF-Agents to train an agent that can achieve a superhuman level at\nSpaceInvaders-v4 using any of the available algorithms.\n10. If you have about $100 to spare, you can purchase a Raspberry Pi 3 plus\nsome cheap robotics components, install TensorFlow on the Pi, and go\nwild! For an example, check out this fun post by Lukas Biewald, or take\na look at GoPiGo or BrickPi. Start with simple goals, like making the\nrobot turn around to find the brightest angle (if it has a light sensor) or\nthe closest object (if it has a sonar sensor), and move in that direction.\nThen you can start using Deep Learning: for example, if the robot has a\ncamera, you can try to implement an object detection algorithm so it\ndetects people and moves toward them. You can also try to use RL to\nmake the agent learn on its own how to use the motors to achieve that\ngoal. Have fun!\nSolutions to these exercises are available in Appendix A.\n1 For more details, be sure to check out Richard Sutton and Andrew Barto’s book on RL,\nReinforcement Learning: An Introduction (MIT Press).\n2 Volodymyr Mnih et al., “Playing Atari with Deep Reinforcement Learning,” arXiv preprint\narXiv:1312.5602 (2013).\n3 Volodymyr Mnih et al., “Human-Level Control Through Deep Reinforcement Learning,” Nature\n518 (2015): 529–533.\n4 Check out the videos of DeepMind’s system learning to play Space Invaders, Breakout, and other\nvideo games at https://homl.info/dqn3.\n5 Image (a) is from NASA (public domain). (b) is a screenshot from the Ms. Pac-Man game,\ncopyright Atari (fair use in this chapter). Images (c) and (d) are reproduced from Wikipedia. (c)\nwas created by user Stevertigo and released under Creative Commons BY-SA 2.0. (d) is in the\npublic domain. (e) was reproduced from Pixabay, released under Creative Commons CC0.\n6 It is often better to give the poor performers a slight chance of survival, to preserve some diversity\nin the “gene pool.”\n7 If there is a single parent, this is called asexual reproduction. With two (or more) parents, it is\ncalled sexual reproduction. An offspring’s genome (in this case a set of policy parameters) is\nrandomly composed of parts of its parents’ genomes.\n8 One interesting example of a genetic algorithm used for Reinforcement Learning is the\nNeuroEvolution of Augmenting Topologies (NEAT) algorithm.\n9 This is called Gradient Ascent. It’s just like Gradient Descent but in the opposite direction:\nmaximizing instead of minimizing.\n10 OpenAI is an artificial intelligence research company, funded in part by Elon Musk. Its stated goal\nis to promote and develop friendly AIs that will benefit humanity (rather than exterminate it).\n11 Ronald J. Williams, “Simple Statistical Gradient-Following Algorithms for Connectionist\nReinforcement Leaning,” Machine Learning 8 (1992) : 229–256.\n12 Richard Bellman, “A Markovian Decision Process,” Journal of Mathematics and Mechanics 6,\nno. 5 (1957): 679–684.\n13 A great 2018 post by Alex Irpan nicely lays out RL’s biggest difficulties and limitations.\n14 Hado van Hasselt et al., “Deep Reinforcement Learning with Double Q-Learning,” Proceedings of\nthe 30th AAAI Conference on Artificial Intelligence (2015): 2094–2100.\n15 Tom Schaul et al., “Prioritized Experience Replay,” arXiv preprint arXiv:1511.05952 (2015).\n16 It could also just be that the rewards are noisy, in which case there are better methods for\nestimating an experience’s importance (see the paper for some examples).\n17 Ziyu Wang et al., “Dueling Network Architectures for Deep Reinforcement Learning,” arXiv\npreprint arXiv:1511.06581 (2015).\n18 Matteo Hessel et al., “Rainbow: Combining Improvements in Deep Reinforcement Learning,”\narXiv preprint arXiv:1710.02298 (2017): 3215–3222.\n19 If you don’t know this game, it’s simple: a ball bounces around and breaks bricks when it touches\nthem. You control a paddle near the bottom of the screen. The paddle can go left or right, and you\nmust get the ball to break every brick, while preventing it from touching the bottom of the screen.\n20 Since there are only three primary colors, you cannot just display an image with four color\nchannels. For this reason, I combined the last channel with the first three to get the RGB image\nrepresented here. Pink is actually a mix of blue and red, but the agent sees four independent\nchannels.\n21 At the time of this writing, there is no prioritized experience replay buffer yet, but one will likely\nbe open sourced soon.\n22 For a comparison of this algorithm’s performance on various Atari games, see figure 3 in\nDeepMind’s 2015 paper.\n23 Volodymyr Mnih et al., “Asynchonous Methods for Deep Reinforcement Learning,” Proceedings\nof the 33rd International Conference on Machine Learning (2016): 1928–1937.\n24 Tuomas Haarnoja et al., “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement\nLearning with a Stochastic Actor,” Proceedings of the 35th International Conference on Machine\nLearning (2018): 1856–1865.\n25 John Schulman et al., “Proximal Policy Optimization Algorithms,” arXiv preprint\narXiv:1707.06347 (2017).\n26 John Schulman et al., “Trust Region Policy Optimization,” Proceedings of the 32nd International\nConference on Machine Learning (2015): 1889–1897.\n27 Deepak Pathak et al., “Curiosity-Driven Exploration by Self-Supervised Prediction,” Proceedings\nof the 34th International Conference on Machine Learning (2017): 2778–2787." + }, + { + "chapter": 19, + "text": "Chapter 19. Training and\nDeploying TensorFlow Models\nat Scale\nOnce you have a beautiful model that makes amazing predictions, what do\nyou do with it? Well, you need to put it in production! This could be as\nsimple as running the model on a batch of data and perhaps writing a\nscript that runs this model every night. However, it is often much more\ninvolved. Various parts of your infrastructure may need to use this model\non live data, in which case you probably want to wrap your model in a web\nservice: this way, any part of your infrastructure can query your model at\nany time using a simple REST API (or some other protocol), as we\ndiscussed in Chapter 2. But as time passes, you need to regularly retrain\nyour model on fresh data and push the updated version to production. You\nmust handle model versioning, gracefully transition from one model to the\nnext, possibly roll back to the previous model in case of problems, and\nperhaps run multiple different models in parallel to perform A/B\nexperiments. If your product becomes successful, your service may start\nto get plenty of queries per second (QPS), and it must scale up to support\nthe load. A great solution to scale up your service, as we will see in this\nchapter, is to use TF Serving, either on your own hardware infrastructure\nor via a cloud service such as Google Cloud AI Platform. It will take care\nof efficiently serving your model, handle graceful model transitions, and\nmore. If you use the cloud platform, you will also get many extra features,\nsuch as powerful monitoring tools.\nMoreover, if you have a lot of training data, and compute-intensive\nmodels, then training time may be prohibitively long. If your product\nneeds to adapt to changes quickly, then a long training time can be a\nshowstopper (e.g., think of a news recommendation system promoting\n1\nnews from last week). Perhaps even more importantly, a long training time\nwill prevent you from experimenting with new ideas. In Machine Learning\n(as in many other fields), it is hard to know in advance which ideas will\nwork, so you should try out as many as possible, as fast as possible. One\nway to speed up training is to use hardware accelerators such as GPUs or\nTPUs. To go even faster, you can train a model across multiple machines,\neach equipped with multiple hardware accelerators. TensorFlow’s simple\nyet powerful Distribution Strategies API makes this easy, as we will see.\nIn this chapter we will look at how to deploy models, first to TF Serving,\nthen to Google Cloud AI Platform. We will also take a quick look at\ndeploying models to mobile apps, embedded devices, and web apps.\nLastly, we will discuss how to speed up computations using GPUs and how\nto train models across multiple devices and servers using the Distribution\nStrategies API. That’s a lot of topics to discuss, so let’s get started!\nServing a TensorFlow Model\nOnce you have trained a TensorFlow model, you can easily use it in any\nPython code: if it’s a tf.keras model, just call its predict() method! But\nas your infrastructure grows, there comes a point where it is preferable to\nwrap your model in a small service whose sole role is to make predictions\nand have the rest of the infrastructure query it (e.g., via a REST or gRPC\nAPI). This decouples your model from the rest of the infrastructure,\nmaking it possible to easily switch model versions or scale the service up\nas needed (independently from the rest of your infrastructure), perform\nA/B experiments, and ensure that all your software components rely on the\nsame model versions. It also simplifies testing and development, and\nmore. You could create your own microservice using any technology you\nwant (e.g., using the Flask library), but why reinvent the wheel when you\ncan just use TF Serving?\nUsing TensorFlow Serving\n2\nTF Serving is a very efficient, battle-tested model server that’s written in\nC++. It can sustain a high load, serve multiple versions of your models and\nwatch a model repository to automatically deploy the latest versions, and\nmore (see Figure 19-1).\nFigure 19-1. TF Serving can serve multiple models and automatically deploy the latest version of\neach model\nSo let’s suppose you have trained an MNIST model using tf.keras, and you\nwant to deploy it to TF Serving. The first thing you have to do is export\nthis model to TensorFlow’s SavedModel format.\nExporting SavedModels\nTensorFlow provides a simple tf.saved_model.save() function to export\nmodels to the SavedModel format. All you need to do is give it the model,\nspecifying its name and version number, and the function will save the\nmodel’s computation graph and its weights:\nmodel = keras.models.Sequential([...]) \nmodel.compile([...]) \nhistory = model.fit([...]) \n \nmodel_version = \"0001\" \nmodel_name = \"my_mnist_model\" \nmodel_path = os.path.join(model_name, model_version) \ntf.saved_model.save(model, model_path)\nIt’s usually a good idea to include all the preprocessing layers in the final\nmodel you export so that it can ingest data in its natural form once it is\ndeployed to production. This avoids having to take care of preprocessing\nseparately within the application that uses the model. Bundling the\npreprocessing steps within the model also makes it simpler to update them\nlater on and limits the risk of mismatch between a model and the\npreprocessing steps it requires.\nWARNING\nSince a SavedModel saves the computation graph, it can only be used with models\nthat are based exclusively on TensorFlow operations, excluding the\ntf.py_function() operation (which wraps arbitrary Python code). It also excludes\ndynamic tf.keras models (see Appendix G), since these models cannot be converted\nto computation graphs. Dynamic models need to be served using other tools (e.g.,\nFlask).\nA SavedModel represents a version of your model. It is stored as a\ndirectory containing a saved_model.pb file, which defines the computation\ngraph (represented as a serialized protocol buffer), and a variables\nsubdirectory containing the variable values. For models containing a large\nnumber of weights, these variable values may be split across multiple\nfiles. A SavedModel also includes an assets subdirectory that may contain\nadditional data, such as vocabulary files, class names, or some example\ninstances for this model. The directory structure is as follows (in this\nexample, we don’t use assets):\nmy_mnist_model \n└── 0001 \n ├── assets \n ├── saved_model.pb \n └── variables \n ├── variables.data-00000-of-00001 \n └── variables.index\nAs you might expect, you can load a SavedModel using the\ntf.saved_model.load() function. However, the returned object is not a\nKeras model: it represents the SavedModel, including its computation\ngraph and variable values. You can use it like a function, and it will make\npredictions (make sure to pass the inputs as tensors, and you must also set\nthe training argument, generally to False):\nsaved_model = tf.saved_model.load(model_path) \ny_pred = saved_model(X_new, training=False)\nAlternatively, you can wrap this SavedModel’s prediction function in a\nKeras model:\ninputs = keras.layers.Input(shape=...) \noutputs = saved_model(inputs, training=False) \nmodel = keras.models.Model(inputs=[inputs], outputs=[outputs]) \ny_pred = model.predict(X_new)\nTensorFlow also comes with a small saved_model_cli command-line tool\nto inspect SavedModels:\n$ export ML_PATH=\"$HOME/ml\" # point to this project, wherever it is \n$ cd $ML_PATH \n$ saved_model_cli show --dir my_mnist_model/0001 --all \nMetaGraphDef with tag-set: 'serve' contains the following SignatureDefs: \nsignature_def['__saved_model_init_op']: \n [...] \n \nsignature_def['serving_default']: \n The given SavedModel SignatureDef contains the following input(s): \n inputs['flatten_input'] tensor_info: \n dtype: DT_FLOAT \n shape: (-1, 28, 28) \n name: serving_default_flatten_input:0 \n The given SavedModel SignatureDef contains the following output(s): \n outputs['dense_1'] tensor_info: \n dtype: DT_FLOAT \n shape: (-1, 10) \n name: StatefulPartitionedCall:0 \n Method name is: tensorflow/serving/predict \nA SavedModel contains one or more metagraphs. A metagraph is a\ncomputation graph plus some function signature definitions (including\ntheir input and output names, types, and shapes). Each metagraph is\nidentified by a set of tags. For example, you may want to have a\nmetagraph containing the full computation graph, including the training\noperations (this one may be tagged \"train\", for example), and another\nmetagraph containing a pruned computation graph with only the prediction\noperations, including some GPU-specific operations (this metagraph may\nbe tagged \"serve\", \"gpu\"). However, when you pass a tf.keras model to\nthe tf.saved_model.save() function, by default the function saves a\nmuch simpler SavedModel: it saves a single metagraph tagged \"serve\",\nwhich contains two signature definitions, an initialization function (called\n__saved_model_init_op, which you do not need to worry about) and a\ndefault serving function (called serving_default). When saving a\ntf.keras model, the default serving function corresponds to the model’s\ncall() function, which of course makes predictions.\nThe saved_model_cli tool can also be used to make predictions (for\ntesting, not really for production). Suppose you have a NumPy array\n(X_new) containing three images of handwritten digits that you want to\nmake predictions for. You first need to export them to NumPy’s npy\nformat:\nnp.save(\"my_mnist_tests.npy\", X_new)\nNext, use the saved_model_cli command like this:\n$ saved_model_cli run --dir my_mnist_model/0001 --tag_set serve \\ \n --signature_def serving_default \\ \n --inputs flatten_input=my_mnist_tests.npy \n[...] Result for output key dense_1: \n[[1.1739199e-04 1.1239604e-07 6.0210604e-04 [...] 3.9471846e-04] \n [1.2294615e-03 2.9207937e-05 9.8599273e-01 [...] 1.1113169e-07] \n [6.4066830e-05 9.6359509e-01 9.0598064e-03 [...] 4.2495009e-04]] \nThe tool’s output contains the 10 class probabilities of each of the 3\ninstances. Great! Now that you have a working SavedModel, the next step\nis to install TF Serving.\nInstalling TensorFlow Serving\nThere are many ways to install TF Serving: using a Docker image, using\nthe system’s package manager, installing from source, and more. Let’s use\nthe Docker option, which is highly recommended by the TensorFlow team\nas it is simple to install, it will not mess with your system, and it offers\nhigh performance. You first need to install Docker. Then download the\nofficial TF Serving Docker image:\n$ docker pull tensorflow/serving \nNow you can create a Docker container to run this image:\n$ docker run -it --rm -p 8500:8500 -p 8501:8501 \\ \n -v \"$ML_PATH/my_mnist_model:/models/my_mnist_model\" \\ \n -e MODEL_NAME=my_mnist_model \\ \n tensorflow/serving \n[...] \n2019-06-01 [...] loaded servable version {name: my_mnist_model version: \n1} \n2019-06-01 [...] Running gRPC ModelServer at 0.0.0.0:8500 ... \n2019-06-01 [...] Exporting HTTP/REST API at:localhost:8501 ... \n[evhttp_server.cc : 237] RAW: Entering the event loop ... \nThat’s it! TF Serving is running. It loaded our MNIST model (version 1),\nand it is serving it through both gRPC (on port 8500) and REST (on port\n8501). Here is what all the command-line options mean:\n-it\nMakes the container interactive (so you can press Ctrl-C to stop it) and\ndisplays the server’s output.\n--rm\n3\nDeletes the container when you stop it (no need to clutter your\nmachine with interrupted containers). However, it does not delete the\nimage.\n-p 8500:8500\nMakes the Docker engine forward the host’s TCP port 8500 to the\ncontainer’s TCP port 8500. By default, TF Serving uses this port to\nserve the gRPC API.\n-p 8501:8501\nForwards the host’s TCP port 8501 to the container’s TCP port 8501.\nBy default, TF Serving uses this port to serve the REST API.\n-v \"$ML_PATH/my_mnist_model:/models/my_mnist_model\"\nMakes the host’s $ML_PATH/my_mnist_model directory available to the\ncontainer at the path /models/mnist_model. On Windows, you may\nneed to replace / with \\ in the host path (but not in the container path).\n-e MODEL_NAME=my_mnist_model\nSets the container’s MODEL_NAME environment variable, so TF Serving\nknows which model to serve. By default, it will look for models in the\n/models directory, and it will automatically serve the latest version it\nfinds.\ntensorflow/serving\nThis is the name of the image to run.\nNow let’s go back to Python and query this server, first using the REST\nAPI, then the gRPC API.\nQuerying TF Serving through the REST API\nLet’s start by creating the query. It must contain the name of the function\nsignature you want to call, and of course the input data:\nimport json \n \ninput_data_json = json.dumps({ \n \"signature_name\": \"serving_default\", \n \"instances\": X_new.tolist(), \n})\nNote that the JSON format is 100% text-based, so the X_new NumPy array\nhad to be converted to a Python list and then formatted as JSON:\n>>> input_data_json \n'{\"signature_name\": \"serving_default\", \"instances\": [[[0.0, 0.0, 0.0, \n[...] \n0.3294117647058824, 0.725490196078431, [...very long], 0.0, 0.0, 0.0, \n0.0]]]}'\nNow let’s send the input data to TF Serving by sending an HTTP POST\nrequest. This can be done easily using the requests library (it is not part\nof Python’s standard library, so you will need to install it first, e.g., using\npip):\nimport requests \n \nSERVER_URL = 'http://localhost:8501/v1/models/my_mnist_model:predict' \nresponse = requests.post(SERVER_URL, data=input_data_json) \nresponse.raise_for_status() # raise an exception in case of error \nresponse = response.json()\nThe response is a dictionary containing a single \"predictions\" key. The\ncorresponding value is the list of predictions. This list is a Python list, so\nlet’s convert it to a NumPy array and round the floats it contains to the\nsecond decimal:\n>>> y_proba = np.array(response[\"predictions\"]) \n>>> y_proba.round(2) \narray([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. ], \n [0. , 0. , 0.99, 0.01, 0. , 0. , 0. , 0. , 0. , 0. ], \n [0. , 0.96, 0.01, 0. , 0. , 0. , 0. , 0.01, 0.01, 0. ]])\nHurray, we have the predictions! The model is close to 100% confident\nthat the first image is a 7, 99% confident that the second image is a 2, and\n96% confident that the third image is a 1.\nThe REST API is nice and simple, and it works well when the input and\noutput data are not too large. Moreover, just about any client application\ncan make REST queries without additional dependencies, whereas other\nprotocols are not always so readily available. However, it is based on\nJSON, which is text-based and fairly verbose. For example, we had to\nconvert the NumPy array to a Python list, and every float ended up\nrepresented as a string. This is very inefficient, both in terms of\nserialization/deserialization time (to convert all the floats to strings and\nback) and in terms of payload size: many floats end up being represented\nusing over 15 characters, which translates to over 120 bits for 32-bit\nfloats! This will result in high latency and bandwidth usage when\ntransferring large NumPy arrays. So let’s use gRPC instead.\nTIP\nWhen transferring large amounts of data, it is much better to use the gRPC API (if\nthe client supports it), as it is based on a compact binary format and an efficient\ncommunication protocol (based on HTTP/2 framing).\nQuerying TF Serving through the gRPC API\nThe gRPC API expects a serialized PredictRequest protocol buffer as\ninput, and it outputs a serialized PredictResponse protocol buffer. These\nprotobufs are part of the tensorflow-serving-api library, which you\nmust install (e.g., using pip). First, let’s create the request:\nfrom tensorflow_serving.apis.predict_pb2 import PredictRequest \n \nrequest = PredictRequest() \nrequest.model_spec.name = model_name \nrequest.model_spec.signature_name = \"serving_default\" \ninput_name = model.input_names[0] \nrequest.inputs[input_name].CopyFrom(tf.make_tensor_proto(X_new))\n4\nThis code creates a PredictRequest protocol buffer and fills in the\nrequired fields, including the model name (defined earlier), the signature\nname of the function we want to call, and finally the input data, in the\nform of a Tensor protocol buffer. The tf.make_tensor_proto() function\ncreates a Tensor protocol buffer based on the given tensor or NumPy\narray, in this case X_new.\nNext, we’ll send the request to the server and get its response (for this you\nwill need the grpcio library, which you can install using pip):\nimport grpc \nfrom tensorflow_serving.apis import prediction_service_pb2_grpc \n \nchannel = grpc.insecure_channel('localhost:8500') \npredict_service = \nprediction_service_pb2_grpc.PredictionServiceStub(channel) \nresponse = predict_service.Predict(request, timeout=10.0)\nThe code is quite straightforward: after the imports, we create a gRPC\ncommunication channel to localhost on TCP port 8500, then we create a\ngRPC service over this channel and use it to send a request, with a 10-\nsecond timeout (not that the call is synchronous: it will block until it\nreceives the response or the timeout period expires). In this example the\nchannel is insecure (no encryption, no authentication), but gRPC and\nTensorFlow Serving also support secure channels over SSL/TLS.\nNext, let’s convert the PredictResponse protocol buffer to a tensor:\noutput_name = model.output_names[0] \noutputs_proto = response.outputs[output_name] \ny_proba = tf.make_ndarray(outputs_proto)\nIf you run this code and print y_proba.numpy().round(2), you will get\nthe exact same estimated class probabilities as earlier. And that’s all there\nis to it: in just a few lines of code, you can now access your TensorFlow\nmodel remotely, using either REST or gRPC.\nDeploying a new model version\nNow let’s create a new model version and export a SavedModel to the\nmy_mnist_model/0002 directory, just like earlier:\nmodel = keras.models.Sequential([...]) \nmodel.compile([...]) \nhistory = model.fit([...]) \n \nmodel_version = \"0002\" \nmodel_name = \"my_mnist_model\" \nmodel_path = os.path.join(model_name, model_version) \ntf.saved_model.save(model, model_path)\nAt regular intervals (the delay is configurable), TensorFlow Serving\nchecks for new model versions. If it finds one, it will automatically handle\nthe transition gracefully: by default, it will answer pending requests (if\nany) with the previous model version, while handling new requests with\nthe new version. As soon as every pending request has been answered, the\nprevious model version is unloaded. You can see this at work in the\nTensorFlow Serving logs:\n[...] \nreserved resources to load servable {name: my_mnist_model version: 2} \n[...] \nReading SavedModel from: /models/my_mnist_model/0002 \nReading meta graph with tags { serve } \nSuccessfully loaded servable version {name: my_mnist_model version: 2} \nQuiescing servable version {name: my_mnist_model version: 1} \nDone quiescing servable version {name: my_mnist_model version: 1} \nUnloading servable version {name: my_mnist_model version: 1}\nThis approach offers a smooth transition, but it may use too much RAM\n(especially GPU RAM, which is generally the most limited). In this case,\nyou can configure TF Serving so that it handles all pending requests with\nthe previous model version and unloads it before loading and using the\nnew model version. This configuration will avoid having two model\nversions loaded at the same time, but the service will be unavailable for a\nshort period.\n5\nAs you can see, TF Serving makes it quite simple to deploy new models.\nMoreover, if you discover that version 2 does not work as well as you\nexpected, then rolling back to version 1 is as simple as removing the\nmy_mnist_model/0002 directory.\nTIP\nAnother great feature of TF Serving is its automatic batching capability, which you\ncan activate using the --enable_batching option upon startup. When TF Serving\nreceives multiple requests within a short period of time (the delay is configurable), it\nwill automatically batch them together before using the model. This offers a\nsignificant performance boost by leveraging the power of the GPU. Once the model\nreturns the predictions, TF Serving dispatches each prediction to the right client. You\ncan trade a bit of latency for a greater throughput by increasing the batching delay\n(see the --batching_parameters_file option).\nIf you expect to get many queries per second, you will want to deploy TF\nServing on multiple servers and load-balance the queries (see Figure 19-\n2). This will require deploying and managing many TF Serving containers\nacross these servers. One way to handle that is to use a tool such as\nKubernetes, which is an open source system for simplifying container\norchestration across many servers. If you do not want to purchase,\nmaintain, and upgrade all the hardware infrastructure, you will want to use\nvirtual machines on a cloud platform such as Amazon AWS, Microsoft\nAzure, Google Cloud Platform, IBM Cloud, Alibaba Cloud, Oracle Cloud,\nor some other Platform-as-a-Service (PaaS). Managing all the virtual\nmachines, handling container orchestration (even with the help of\nKubernetes), taking care of TF Serving configuration, tuning and\nmonitoring—all of this can be a full-time job. Fortunately, some service\nproviders can take care of all this for you. In this chapter we will use\nGoogle Cloud AI Platform because it’s the only platform with TPUs today,\nit supports TensorFlow 2, it offers a nice suite of AI services (e.g.,\nAutoML, Vision API, Natural Language API), and it is the one I have the\nmost experience with. But there are several other providers in this space,\nsuch as Amazon AWS SageMaker and Microsoft AI Platform, which are\nalso capable of serving TensorFlow models.\nFigure 19-2. Scaling up TF Serving with load balancing\nNow let’s see how to serve our wonderful MNIST model on the cloud!\nCreating a Prediction Service on GCP AI Platform\nBefore you can deploy a model, there’s a little bit of setup to take care of:\n1. Log in to your Google account, and then go to the Google Cloud\nPlatform (GCP) console (see Figure 19-3). If you don’t have a\nGoogle account, you’ll have to create one.\n2. If it is your first time using GCP, you will have to read and accept\nthe terms and conditions. Click Tour Console if you want. At the\ntime of this writing, new users are offered a free trial, including\n$300 worth of GCP credit that you can use over the course of 12\nmonths. You will only need a small portion of that to pay for the\nservices you will use in this chapter. Upon signing up for the free\ntrial, you will still need to create a payment profile and enter your\ncredit card number: it is used for verification purposes (probably\nto avoid people using the free trial multiple times), but you will\nnot be billed. Activate and upgrade your account if requested.\nFigure 19-3. Google Cloud Platform console\n3. If you have used GCP before and your free trial has expired, then\nthe services you will use in this chapter will cost you some\nmoney. It should not be too much, especially if you remember to\nturn off the services when you do not need them anymore. Make\nsure you understand and agree to the pricing conditions before\nyou run any service. I hereby decline any responsibility if services\nend up costing more than you expected! Also make sure your\nbilling account is active. To check, open the navigation menu on\nthe left and click Billing, and make sure you have set up a\npayment method and that the billing account is active.\n4. Every resource in GCP belongs to a project. This includes all the\nvirtual machines you may use, the files you store, and the training\njobs you run. When you create an account, GCP automatically\ncreates a project for you, called “My First Project.” If you want,\nyou can change its display name by going to the project settings:\nin the navigation menu (on the left of the screen), select IAM &\nadmin → Settings, change the project’s display name, and click\nSave. Note that the project also has a unique ID and number. You\ncan choose the project ID when you create a project, but you\ncannot change it later. The project number is automatically\ngenerated and cannot be changed. If you want to create a new\nproject, click the project name at the top of the page, then click\nNew Project and enter the project ID. Make sure billing is active\nfor this new project.\nWARNING\nAlways set an alarm to remind yourself to turn services off when you\nknow you will only need them for a few hours, or else you might leave\nthem running for days or months, incurring potentially significant costs.\n5. Now that you have a GCP account with billing activated, you can\nstart using the services. The first one you will need is Google\nCloud Storage (GCS): this is where you will put the SavedModels,\nthe training data, and more. In the navigation menu, scroll down\nto the Storage section, and click Storage → Browser. All your\nfiles will go in one or more buckets. Click Create Bucket and\nchoose the bucket name (you may need to activate the Storage\nAPI first). GCS uses a single worldwide namespace for buckets,\nso simple names like “machine-learning” will most likely not be\navailable. Make sure the bucket name conforms to DNS naming\nconventions, as it may be used in DNS records. Moreover, bucket\nnames are public, so do not put anything private in there. It is\ncommon to use your domain name or your company name as a\nprefix to ensure uniqueness, or simply use a random number as\npart of the name. Choose the location where you want the bucket\nto be hosted, and the rest of the options should be fine by default.\nThen click Create.\n6. Upload the my_mnist_model folder you created earlier (including\none or more versions) to your bucket. To do this, just go to the\nGCS Browser, click the bucket, then drag and drop the\nmy_mnist_model folder from your system to the bucket (see\nFigure 19-4). Alternatively, you can click “Upload folder” and\nselect the my_mnist_model folder to upload. By default, the\nmaximum size for a SavedModel is 250 MB, but it is possible to\nrequest a higher quota.\nFigure 19-4. Uploading a SavedModel to Google Cloud Storage\n7. Now you need to configure AI Platform (formerly known as ML\nEngine) so that it knows which models and versions you want to\nuse. In the navigation menu, scroll down to the Artificial\nIntelligence section, and click AI Platform → Models. Click\nActivate API (it takes a few minutes), then click “Create model.”\nFill in the model details (see Figure 19-5) and click Create.\nFigure 19-5. Creating a new model on Google Cloud AI Platform\n8. Now that you have a model on AI Platform, you need to create a\nmodel version. In the list of models, click the model you just\ncreated, then click “Create version” and fill in the version details\n(see Figure 19-6): set the name, description, Python version (3.5\nor above), framework (TensorFlow), framework version (2.0 if\navailable, or 1.13), ML runtime version (2.0, if available or\n1.13), machine type (choose “Single core CPU” for now), model\npath on GCS (this is the full path to the actual version folder, e.g.,\ngs://my-mnist-model-bucket/my_mnist_model/0002/), scaling\n(choose automatic), and minimum number of TF Serving\ncontainers to have running at all times (leave this field empty).\nThen click Save.\n6\nFigure 19-6. Creating a new model version on Google Cloud AI Platform\nCongratulations, you have deployed your first model on the cloud!\nBecause you selected automatic scaling, AI Platform will start more TF\nServing containers when the number of queries per second increases, and\nit will load-balance the queries between them. If the QPS goes down, it\nwill stop containers automatically. The cost is therefore directly linked to\nthe QPS (as well as the type of machine you choose and the amount of data\nyou store on GCS). This pricing model is particularly useful for occasional\nusers and for services with important usage spikes, as well as for startups:\nthe price remains low until the startup actually starts up.\nNOTE\nIf you do not use the prediction service, AI Platform will stop all containers. This\nmeans you will only pay for the amount of storage you use (a few cents per gigabyte\nper month). Note that when you query the service, AI Platform will need to start up a\nTF Serving container, which will take a few seconds. If this delay is unacceptable,\nyou will have to set the minimum number of TF Serving containers to 1 when\ncreating the model version. Of course, this means at least one machine will run\nconstantly, so the monthly fee will be higher.\nNow let’s query this prediction service!\nUsing the Prediction Service\nUnder the hood, AI Platform just runs TF Serving, so in principle you\ncould use the same code as earlier, if you knew which URL to query.\nThere’s just one problem: GCP also takes care of encryption and\nauthentication. Encryption is based on SSL/TLS, and authentication is\ntoken-based: a secret authentication token must be sent to the server in\nevery request. So before your code can use the prediction service (or any\nother GCP service), it must obtain a token. We will see how to do this\nshortly, but first you need to configure authentication and give your\napplication the appropriate access rights on GCP. You have two options for\nauthentication:\nYour application (i.e., the client code that will query the\nprediction service) could authenticate using user credentials with\nyour own Google login and password. Using user credentials\nwould give your application the exact same rights as on GCP,\nwhich is certainly way more than it needs. Moreover, you would\nhave to deploy your credentials in your application, so anyone\nwith access could steal your credentials and fully access your\nGCP account. In short, do not choose this option; it is only needed\nin very rare cases (e.g., when your application needs to access its\nuser’s GCP account).\nThe client code can authenticate with a service account. This is an\naccount that represents an application, not a user. It is generally\ngiven very restricted access rights: strictly what it needs, and no\nmore. This is the recommended option.\nSo, let’s create a service account for your application: in the navigation\nmenu, go to IAM & admin → Service accounts, then click Create Service\nAccount, fill in the form (service account name, ID, description), and click\nCreate (see Figure 19-7). Next, you must give this account some access\nrights. Select the ML Engine Developer role: this will allow the service\naccount to make predictions, and not much more. Optionally, you can\ngrant some users access to the service account (this is useful when your\nGCP user account is part of an organization, and you wish to authorize\nother users in the organization to deploy applications that will be based on\nthis service account or to manage the service account itself). Next, click\nCreate Key to export the service account’s private key, choose JSON, and\nclick Create. This will download the private key in the form of a JSON\nfile. Make sure to keep it private!\nFigure 19-7. Creating a new service account in Google IAM\nGreat! Now let’s write a small script that will query the prediction service.\nGoogle provides several libraries to simplify access to its services:\nGoogle API Client Library\nThis is a fairly thin layer on top of OAuth 2.0 (for the authentication)\nand REST. You can use it with all GCP services, including AI Platform.\nYou can install it using pip: the library is called google-api-python-\nclient.\nGoogle Cloud Client Libraries\nThese are a bit more high-level: each one is dedicated to a particular\nservice, such as GCS, Google BigQuery, Google Cloud Natural\nLanguage, and Google Cloud Vision. All these libraries can be\ninstalled using pip (e.g., the GCS Client Library is called google-\ncloud-storage). When a client library is available for a given service,\nit is recommended to use it rather than the Google API Client Library,\nas it implements all the best practices and will often use gRPC rather\nthan REST, for better performance.\nAt the time of this writing there is no client library for AI Platform, so we\nwill use the Google API Client Library. It will need to use the service\naccount’s private key; you can tell it where it is by setting the\nGOOGLE_APPLICATION_CREDENTIALS environment variable, either before\nstarting the script or within the script like this:\nimport os \n \nos.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \n\"my_service_account_key.json\"\nNOTE\nIf you deploy your application to a virtual machine on Google Cloud Engine (GCE),\nor within a container using Google Cloud Kubernetes Engine, or as a web\napplication on Google Cloud App Engine, or as a microservice on Google Cloud\nFunctions, and if the GOOGLE_APPLICATION_CREDENTIALS environment variable is\nnot set, then the library will use the default service account for the host service (e.g.,\nthe default GCE service account, if your application runs on GCE).\nNext, you must create a resource object that wraps access to the prediction\nservice:\nimport googleapiclient.discovery \n \nproject_id = \"onyx-smoke-242003\" # change this to your project ID \nmodel_id = \"my_mnist_model\" \nmodel_path = \"projects/{}/models/{}\".format(project_id, model_id) \nml_resource = googleapiclient.discovery.build(\"ml\", \"v1\").projects()\nNote that you can append /versions/0001 (or any other version number)\nto the model_path to specify the version you want to query: this can be\nuseful for A/B testing or for testing a new version on a small group of\nusers before releasing it widely (this is called a canary). Next, let’s write a\n7\nsmall function that will use the resource object to call the prediction\nservice and get the predictions back:\ndef predict(X): \n input_data_json = {\"signature_name\": \"serving_default\", \n \"instances\": X.tolist()} \n request = ml_resource.predict(name=model_path, body=input_data_json) \n response = request.execute() \n if \"error\" in response: \n raise RuntimeError(response[\"error\"]) \n return np.array([pred[output_name] for pred in \nresponse[\"predictions\"]])\nThe function takes a NumPy array containing the input images and\nprepares a dictionary that the client library will convert to the JSON\nformat (as we did earlier). Then it prepares a prediction request, and\nexecutes it; it raises an exception if the response contains an error, or else\nit extracts the predictions for each instance and bundles them in a NumPy\narray. Let’s see if it works:\n>>> Y_probas = predict(X_new) \n>>> np.round(Y_probas, 2) \narray([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. ], \n [0. , 0. , 0.99, 0.01, 0. , 0. , 0. , 0. , 0. , 0. ], \n [0. , 0.96, 0.01, 0. , 0. , 0. , 0. , 0.01, 0.01, 0. ]])\nYes! You now have a nice prediction service running on the cloud that can\nautomatically scale up to any number of QPS, plus you can query it from\nanywhere securely. Moreover, it costs you close to nothing when you don’t\nuse it: you’ll pay just a few cents per month per gigabyte used on GCS.\nAnd you can also get detailed logs and metrics using Google Stackdriver.\nBut what if you want to deploy your model to a mobile app? Or to an\nembedded device?\nDeploying a Model to a Mobile or Embedded\nDevice\nIf you need to deploy your model to a mobile or embedded device, a large\nmodel may simply take too long to download and use too much RAM and\nCPU, all of which will make your app unresponsive, heat the device, and\ndrain its battery. To avoid this, you need to make a mobile-friendly,\nlightweight, and efficient model, without sacrificing too much of its\naccuracy. The TFLite library provides several tools to help you deploy\nyour models to mobile and embedded devices, with three main objectives:\nReduce the model size, to shorten download time and reduce\nRAM usage.\nReduce the amount of computations needed for each prediction, to\nreduce latency, battery usage, and heating.\nAdapt the model to device-specific constraints.\nTo reduce the model size, TFLite’s model converter can take a\nSavedModel and compress it to a much lighter format based on\nFlatBuffers. This is an efficient cross-platform serialization library (a bit\nlike Protocol Buffers) initially created by Google for gaming. It is\ndesigned so you can load FlatBuffers straight to RAM without any\npreprocessing: this reduces the loading time and memory footprint. Once\nthe model is loaded into a mobile or embedded device, the TFLite\ninterpreter will execute it to make predictions. Here is how you can\nconvert a SavedModel to a FlatBuffer and save it to a .tflite file:\nconverter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path) \ntflite_model = converter.convert() \nwith open(\"converted_model.tflite\", \"wb\") as f: \n f.write(tflite_model)\nTIP\nYou can also save a tf.keras model directly to a FlatBuffer using\nfrom_keras_model().\n8\nThe converter also optimizes the model, both to shrink it and to reduce its\nlatency. It prunes all the operations that are not needed to make predictions\n(such as training operations), and it optimizes computations whenever\npossible; for example, 3×a + 4×a + 5×a will be converted to (3 + 4 + 5)×a.\nIt also tries to fuse operations whenever possible. For example, Batch\nNormalization layers end up folded into the previous layer’s addition and\nmultiplication operations, whenever possible. To get a good idea of how\nmuch TFLite can optimize a model, download one of the pretrained TFLite\nmodels, unzip the archive, then open the excellent Netron graph\nvisualization tool and upload the .pb file to view the original model. It’s a\nbig, complex graph, right? Next, open the optimized .tflite model and\nmarvel at its beauty!\nAnother way you can reduce the model size (other than simply using\nsmaller neural network architectures) is by using smaller bit-widths: for\nexample, if you use half-floats (16 bits) rather than regular floats (32 bits),\nthe model size will shrink by a factor of 2, at the cost of a (generally\nsmall) accuracy drop. Moreover, training will be faster, and you will use\nroughly half the amount of GPU RAM.\nTFLite’s converter can go further than that, by quantizing the model\nweights down to fixed-point, 8-bit integers! This leads to a fourfold size\nreduction compared to using 32-bit floats. The simplest approach is called\npost-training quantization: it just quantizes the weights after training,\nusing a fairly basic but efficient symmetrical quantization technique. It\nfinds the maximum absolute weight value, m, then it maps the floating-\npoint range –m to +m to the fixed-point (integer) range –127 to +127. For\nexample (see Figure 19-8), if the weights range from –1.5 to +0.8, then the\nbytes –127, 0, and +127 will correspond to the floats –1.5, 0.0, and +1.5,\nrespectively. Note that 0.0 always maps to 0 when using symmetrical\nquantization (also note that the byte values +68 to +127 will not be used,\nsince they map to floats greater than +0.8).\nFigure 19-8. From 32-bit floats to 8-bit integers, using symmetrical quantization\nTo perform this post-training quantization, simply add\nOPTIMIZE_FOR_SIZE to the list of converter optimizations before calling\nthe convert() method:\nconverter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]\nThis technique dramatically reduces the model’s size, so it’s much faster\nto download and store. However, at runtime the quantized weights get\nconverted back to floats before they are used (these recovered floats are\nnot perfectly identical to the original floats, but not too far off, so the\naccuracy loss is usually acceptable). To avoid recomputing them all the\ntime, the recovered floats are cached, so there is no reduction of RAM\nusage. And there is no reduction either in compute speed.\nThe most effective way to reduce latency and power consumption is to\nalso quantize the activations so that the computations can be done entirely\nwith integers, without the need for any floating-point operations. Even\nwhen using the same bit-width (e.g., 32-bit integers instead of 32-bit\nfloats), integer computations use less CPU cycles, consume less energy,\nand produce less heat. And if you also reduce the bit-width (e.g., down to\n8-bit integers), you can get huge speedups. Moreover, some neural\nnetwork accelerator devices (such as the Edge TPU) can only process\nintegers, so full quantization of both weights and activations is\ncompulsory. This can be done post-training; it requires a calibration step\nto find the maximum absolute value of the activations, so you need to\nprovide a representative sample of training data to TFLite (it does not\nneed to be huge), and it will process the data through the model and\nmeasure the activation statistics required for quantization (this step is\ntypically fast).\nThe main problem with quantization is that it loses a bit of accuracy: it is\nequivalent to adding noise to the weights and activations. If the accuracy\ndrop is too severe, then you may need to use quantization-aware training.\nThis means adding fake quantization operations to the model so it can\nlearn to ignore the quantization noise during training; the final weights\nwill then be more robust to quantization. Moreover, the calibration step\ncan be taken care of automatically during training, which simplifies the\nwhole process.\nI have explained the core concepts of TFLite, but going all the way to\ncoding a mobile app or an embedded program would require a whole other\nbook. Fortunately, one exists: if you want to learn more about building\nTensorFlow applications for mobile and embedded devices, check out the\nO’Reilly book TinyML: Machine Learning with TensorFlow on Arduino\nand Ultra-Low Power Micro-Controllers, by Pete Warden (who leads the\nTFLite team) and Daniel Situnayake.\nTENSORFLOW IN THE BROWSER\nWhat if you want to use your model in a website, running directly in\nthe user’s browser? This can be useful in many scenarios, such as:\nWhen your web application is often used in situations where\nthe user’s connectivity is intermittent or slow (e.g., a website\nfor hikers), so running the model directly on the client side is\nthe only way to make your website reliable.\nWhen you need the model’s responses to be as fast as possible\n(e.g., for an online game). Removing the need to query the\nserver to make predictions will definitely reduce the latency\nand make the website much more responsive.\nWhen your web service makes predictions based on some\nprivate user data, and you want to protect the user’s privacy\nby making the predictions on the client side so that the private\ndata never has to leave the user’s machine.\nFor all these scenarios, you can export your model to a special format\nthat can be loaded by the TensorFlow.js JavaScript library. This library\ncan then use your model to make predictions directly in the user’s\nbrowser. The TensorFlow.js project includes a\ntensorflowjs_converter tool that can convert a TensorFlow\nSavedModel or a Keras model file to the TensorFlow.js Layers format:\nthis is a directory containing a set of sharded weight files in binary\nformat and a model.json file that describes the model’s architecture\nand links to the weight files. This format is optimized to be\ndownloaded efficiently on the web. Users can then download the\nmodel and run predictions in the browser using the TensorFlow.js\nlibrary. Here is a code snippet to give you an idea of what the\nJavaScript API looks like:\nimport * as tf from '@tensorflow/tfjs'; \nconst model = await \n9\ntf.loadLayersModel('https://example.com/tfjs/model.json'); \nconst image = tf.fromPixels(webcamElement); \nconst prediction = model.predict(image);\nOnce again, doing justice to this topic would require a whole book. If\nyou want to learn more about TensorFlow.js, check out the O’Reilly\nbook Practical Deep Learning for Cloud, Mobile, and Edge, by\nAnirudh Koul, Siddha Ganju, and Meher Kasam.\nNext, we will see how to use GPUs to speed up computations!\nUsing GPUs to Speed Up Computations\nIn Chapter 11 we discussed several techniques that can considerably speed\nup training: better weight initialization, Batch Normalization,\nsophisticated optimizers, and so on. But even with all of these techniques,\ntraining a large neural network on a single machine with a single CPU can\ntake days or even weeks.\nIn this section we will look at how to speed up your models by using\nGPUs. We will also see how to split the computations across multiple\ndevices, including the CPU and multiple GPU devices (see Figure 19-9).\nFor now we will run everything on a single machine, but later in this\nchapter we will discuss how to distribute computations across multiple\nservers.\nFigure 19-9. Executing a TensorFlow graph across multiple devices in parallel\nThanks to GPUs, instead of waiting for days or weeks for a training\nalgorithm to complete, you may end up waiting for just a few minutes or\nhours. Not only does this save an enormous amount of time, but it also\nmeans that you can experiment with various models much more easily and\nfrequently retrain your models on fresh data.\nTIP\nYou can often get a major performance boost simply by adding GPU cards to a\nsingle machine. In fact, in many cases this will suffice; you won’t need to use\nmultiple machines at all. For example, you can typically train a neural network just\nas fast using four GPUs on a single machine rather than eight GPUs across multiple\nmachines, due to the extra delay imposed by network communications in a\ndistributed setup. Similarly, using a single powerful GPU is often preferable to using\nmultiple slower GPUs.\nThe first step is to get your hands on a GPU. There are two options for\nthis: you can either purchase your own GPU(s), or you can use GPU-\nequipped virtual machines on the cloud. Let’s start with the first option.\nGetting Your Own GPU\nIf you choose to purchase a GPU card, then take some time to make the\nright choice. Tim Dettmers wrote an excellent blog post to help you\nchoose, and he updates it regularly: I encourage you to read it carefully. At\nthe time of this writing, TensorFlow only supports Nvidia cards with\nCUDA Compute Capability 3.5+ (as well as Google’s TPUs, of course),\nbut it may extend its support to other manufacturers. Moreover, although\nTPUs are currently only available on GCP, it is highly likely that TPU-like\ncards will be available for sale in the near future, and TensorFlow may\nsupport them. In short, make sure to check TensorFlow’s documentation to\nsee what devices are supported at this point.\nIf you go for an Nvidia GPU card, you will need to install the appropriate\nNvidia drivers and several Nvidia libraries.\n These include the Compute\nUnified Device Architecture library (CUDA), which allows developers to\nuse CUDA-enabled GPUs for all sorts of computations (not just graphics\nacceleration), and the CUDA Deep Neural Network library (cuDNN), a\nGPU-accelerated library of primitives for DNNs. cuDNN provides\noptimized implementations of common DNN computations such as\nactivation layers, normalization, forward and backward convolutions, and\npooling (see Chapter 14). It is part of Nvidia’s Deep Learning SDK (note\nthat you’ll need to create an Nvidia developer account in order to\ndownload it). TensorFlow uses CUDA and cuDNN to control the GPU\ncards and accelerate computations (see Figure 19-10).\n10\nFigure 19-10. TensorFlow uses CUDA and cuDNN to control GPUs and boost DNNs\nOnce you have installed the GPU card(s) and all the required drivers and\nlibraries, you can use the nvidia-smi command to check that CUDA is\nproperly installed. It lists the available GPU cards, as well as processes\nrunning on each card:\n$ nvidia-smi \nSun Jun 2 10:05:22 2019 \n+------------------------------------------------------------------------\n-----+ \n| NVIDIA-SMI 418.67 Driver Version: 410.79 CUDA Version: 10.0 \n| \n|-------------------------------+----------------------+-----------------\n-----+ \n| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. \nECC | \n| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util \nCompute M. | \n|===============================+======================+=================\n=====| \n| 0 Tesla T4 Off | 00000000:00:04.0 Off | \n0 | \n| N/A 61C P8 17W / 70W | 0MiB / 15079MiB | 0% \nDefault | \n+-------------------------------+----------------------+-----------------\n-----+ \n \n+------------------------------------------------------------------------\n-----+ \n| Processes: GPU \nMemory | \n| GPU PID Type Process name Usage \n| \n|========================================================================\n=====| \n| No running processes found \n| \n+------------------------------------------------------------------------\n-----+ \nAt the time of this writing, you’ll also need to install the GPU version of\nTensorFlow (i.e., the tensorflow-gpu library); however, there is ongoing\nwork to have a unified installation procedure for both CPU-only and GPU\nmachines, so please check the installation documentation to see which\nlibrary you should install. In any case, since installing every required\nlibrary correctly is a bit long and tricky (and all hell breaks loose if you do\nnot install the correct library versions), TensorFlow provides a Docker\nimage with everything you need inside. However, in order for the Docker\ncontainer to have access to the GPU, you will still need to install the\nNvidia drivers on the host machine.\nTo check that TensorFlow actually sees the GPUs, run the following tests:\n>>> import tensorflow as tf \n>>> tf.test.is_gpu_available() \nTrue \n>>> tf.test.gpu_device_name() \n'/device:GPU:0' \n>>> tf.config.experimental.list_physical_devices(device_type='GPU') \n[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\nThe is_gpu_available() function checks whether at least one GPU is\navailable. The gpu_device_name() function gives the first GPU’s name:\nby default, operations will run on this GPU. The\nlist_physical_devices() function returns the list of all available GPU\ndevices (just one in this example).\nNow, what if you don’t want to invest time and money in getting your own\nGPU card? Just use a GPU VM on the cloud!\nUsing a GPU-Equipped Virtual Machine\nAll major cloud platforms now offer GPU VMs, some preconfigured with\nall the drivers and libraries you need (including TensorFlow). Google\nCloud Platform enforces various GPU quotas, both worldwide and per\nregion: you cannot just create thousands of GPU VMs without prior\nauthorization from Google.\n By default, the worldwide GPU quota is\nzero, so you cannot use any GPU VMs. Therefore, the very first thing you\nneed to do is to request a higher worldwide quota. In the GCP console,\nopen the navigation menu and go to IAM & admin → Quotas. Click\nMetric, click None to uncheck all locations, then search for “GPU” and\nselect “GPUs (all regions)” to see the corresponding quota. If this quota’s\nvalue is zero (or just insufficient for your needs), then check the box next\nto it (it should be the only selected one) and click “Edit quotas.” Fill in the\nrequested information, then click “Submit request.” It may take a few\nhours (or up to a few days) for your quota request to be processed and\n(generally) accepted. By default, there is also a quota of one GPU per\nregion and per GPU type. You can request to increase these quotas too:\nclick Metric, select None to uncheck all metrics, search for “GPU,” and\nselect the type of GPU you want (e.g., NVIDIA P4 GPUs). Then click the\nLocation drop-down menu, click None to uncheck all metrics, and click\nthe location you want; check the boxes next to the quota(s) you want to\nchange, and click “Edit quotas” to file a request.\nOnce your GPU quota requests are approved, you can in no time create a\nVM equipped with one or more GPUs by using Google Cloud AI\n11\n12\nPlatform’s Deep Learning VM Images: go to https://homl.info/dlvm, click\nView Console, then click “Launch on Compute Engine” and fill in the VM\nconfiguration form. Note that some locations do not have all types of\nGPUs, and some have no GPUs at all (change the location to see the types\nof GPUs available, if any). Make sure to select TensorFlow 2.0 as the\nframework, and check “Install NVIDIA GPU driver automatically on first\nstartup.” It is also a good idea to check “Enable access to JupyterLab via\nURL instead of SSH”: this will make it very easy to start a Jupyter\nnotebook running on this GPU VM, powered by JupyterLab (this is an\nalternative web interface to run Jupyter notebooks). Once the VM is\ncreated, scroll down the navigation menu to the Artificial Intelligence\nsection, then click AI Platform → Notebooks. Once the Notebook instance\nappears in the list (this may take a few minutes, so click Refresh once in a\nwhile until it appears), click its Open JupyterLab link. This will run\nJupyterLab on the VM and connect your browser to it. You can create\nnotebooks and run any code you want on this VM, and benefit from its\nGPUs!\nBut if you just want to run some quick tests or easily share notebooks with\nyour colleagues, then you should try Colaboratory.\nColaboratory\nThe simplest and cheapest way to access a GPU VM is to use\nColaboratory (or Colab, for short). It’s free! Just go to\nhttps://colab.research.google.com/ and create a new Python 3 notebook:\nthis will create a Jupyter notebook on your Google Drive (alternatively,\nyou can open any notebook on GitHub, or on Google Drive, or you can\neven upload your own notebooks). Colab’s user interface is similar to\nJupyter’s, except you can share and use the notebooks like regular Google\nDocs, and there are a few other minor differences (e.g., you can create\nhandy widgets using special comments in your code).\nWhen you open a Colab notebook, it runs on a free Google VM dedicated\nto you, called a Colab Runtime. By default the Runtime is CPU-only, but\nyou can change this by going to Runtime → “Change runtime type,”\nselecting GPU in the “Hardware accelerator” drop-down menu, then\nclicking Save. In fact, you could even select TPU! (Yes, you can actually\nuse a TPU for free; we will talk about TPUs later in this chapter, though,\nso for now just select GPU.)\nIf you run multiple Colab notebooks using the same runtime type (see\nFigure 19-11), they will use the same Colab Runtime. So if one writes to a\nfile, the others will be able to read that file. It’s important to understand\nthe security implications of this: if you run an untrusted Colab notebook\nwritten by a nasty hacker, it may read private data produced by the other\nnotebooks and then leak this data back to the hacker. If this includes\nprivate access keys for some resources, the hacker will gain access to\nthose resources. Moreover, if you install a library in the Colab Runtime,\nthe other notebooks will also have that library. Depending on what you\nwant to do, this might be great or annoying (e.g., it means you cannot\neasily use different versions of the same library in different Colab\nnotebooks).\nFigure 19-11. Colab Runtimes and notebooks\nColab does have some restrictions: as the FAQ states, “Colaboratory is\nintended for interactive use. Long-running background computations,\nparticularly on GPUs, may be stopped. Please do not use Colaboratory for\ncryptocurrency mining.” The web interface will automatically disconnect\nfrom the Colab Runtime if you leave it unattended for a while (~30\nminutes). When you reconnect to the Colab Runtime, it may have been\nreset, so make sure you always download any data you care about. Even if\nyou never disconnect, the Colab Runtime will automatically shut down\nafter 12 hours, as it is not meant for long-running computations. Despite\nthese limitations, it’s a fantastic tool to run tests easily, get quick results,\nand collaborate with your colleagues.\nManaging the GPU RAM\nBy default TensorFlow automatically grabs all the RAM in all available\nGPUs the first time you run a computation. It does this to limit GPU RAM\nfragmentation. This means that if you try to start a second TensorFlow\nprogram (or any program that requires the GPU), it will quickly run out of\nRAM. This does not happen as often as you might think, as you will most\noften have a single TensorFlow program running on a machine: usually a\ntraining script, a TF Serving node, or a Jupyter notebook. If you need to\nrun multiple programs for some reason (e.g., to train two different models\nin parallel on the same machine), then you will need to split the GPU\nRAM between these processes more evenly.\nIf you have multiple GPU cards on your machine, a simple solution is to\nassign each of them to a single process. To do this, you can set the\nCUDA_VISIBLE_DEVICES environment variable so that each process only\nsees the appropriate GPU card(s). Also set the CUDA_DEVICE_ORDER\nenvironment variable to PCI_BUS_ID to ensure that each ID always refers\nto the same GPU card. For example, if you have four GPU cards, you\ncould start two programs, assigning two GPUs to each of them, by\nexecuting commands like the following in two separate terminal windows:\n$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python3 \nprogram_1.py \n# and in another terminal: \n$ CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=3,2 python3 \nprogram_2.py \nProgram 1 will then only see GPU cards 0 and 1, named /gpu:0 and\n/gpu:1 respectively, and program 2 will only see GPU cards 2 and 3,\nnamed /gpu:1 and /gpu:0 respectively (note the order). Everything will\nwork fine (see Figure 19-12). Of course, you can also define these\nenvironment variables in Python by setting\nos.environ[\"CUDA_DEVICE_ORDER\"] and\nos.environ[\"CUDA_VISIBLE_DEVICES\"], as long as you do so before\nusing TensorFlow.\nFigure 19-12. Each program gets two GPUs\nAnother option is to tell TensorFlow to grab only a specific amount of\nGPU RAM. This must be done immediately after importing TensorFlow.\nFor example, to make TensorFlow grab only 2 GiB of RAM on each GPU,\nyou must create a virtual GPU device (also called a logical GPU device)\nfor each physical GPU device and set its memory limit to 2 GiB (i.e.,\n2,048 MiB):\nfor gpu in tf.config.experimental.list_physical_devices(\"GPU\"): \n tf.config.experimental.set_virtual_device_configuration( \n gpu, \n \n[tf.config.experimental.VirtualDeviceConfiguration(memory_limit=2048)])\nNow (supposing you have four GPUs, each with at least 4 GiB of RAM)\ntwo programs like this one can run in parallel, each using all four GPU\ncards (see Figure 19-13).\nFigure 19-13. Each program gets all four GPUs, but with only 2 GiB of RAM on each GPU\nIf you run the nvidia-smi command while both programs are running,\nyou should see that each process holds 2 GiB of RAM on each card:\n$ nvidia-smi \n[...] \n+------------------------------------------------------------------------\n-----+ \n| Processes: GPU \nMemory | \n| GPU PID Type Process name Usage \n| \n|========================================================================\n=====| \n| 0 2373 C /usr/bin/python3 \n2241MiB | \n| 0 2533 C /usr/bin/python3 \n2241MiB | \n| 1 2373 C /usr/bin/python3 \n2241MiB | \n| 1 2533 C /usr/bin/python3 \n2241MiB | \n[...]\nYet another option is to tell TensorFlow to grab memory only when it\nneeds it (this also must be done immediately after importing TensorFlow):\nfor gpu in tf.config.experimental.list_physical_devices(\"GPU\"): \n tf.config.experimental.set_memory_growth(gpu, True)\nAnother way to do this is to set the TF_FORCE_GPU_ALLOW_GROWTH\nenvironment variable to true. With this option, TensorFlow will never\nrelease memory once it has grabbed it (again, to avoid memory\nfragmentation), except of course when the program ends. It can be harder\nto guarantee deterministic behavior using this option (e.g., one program\nmay crash because another program’s memory usage went through the\nroof), so in production you’ll probably want to stick with one of the\nprevious options. However, there are some cases where it is very useful:\nfor example, when you use a machine to run multiple Jupyter notebooks,\nseveral of which use TensorFlow. This is why the\nTF_FORCE_GPU_ALLOW_GROWTH environment variable is set to true in\nColab Runtimes.\nLastly, in some cases you may want to split a GPU into two or more\nvirtual GPUs—for example, if you want to test a distribution algorithm\n(this is a handy way to try out the code examples in the rest of this chapter\neven if you have a single GPU, such as in a Colab Runtime). The following\ncode splits the first GPU into two virtual devices, with 2 GiB of RAM each\n(again, this must be done immediately after importing TensorFlow):\nphysical_gpus = tf.config.experimental.list_physical_devices(\"GPU\") \ntf.config.experimental.set_virtual_device_configuration( \n physical_gpus[0], \n \n[tf.config.experimental.VirtualDeviceConfiguration(memory_limit=2048), \n \ntf.config.experimental.VirtualDeviceConfiguration(memory_limit=2048)])\nThese two virtual devices will then be called /gpu:0 and /gpu:1, and you\ncan place operations and variables on each of them as if they were really\ntwo independent GPUs. Now let’s see how TensorFlow decides which\ndevices it should place variables and execute operations on.\nPlacing Operations and Variables on Devices\nThe TensorFlow whitepaper\n presents a friendly dynamic placer\nalgorithm that automagically distributes operations across all available\ndevices, taking into account things like the measured computation time in\nprevious runs of the graph, estimations of the size of the input and output\ntensors for each operation, the amount of RAM available in each device,\ncommunication delay when transferring data into and out of devices, and\nhints and constraints from the user. In practice this algorithm turned out to\nbe less efficient than a small set of placement rules specified by the user,\nso the TensorFlow team ended up dropping the dynamic placer.\nThat said, tf.keras and tf.data generally do a good job of placing operations\nand variables where they belong (e.g., heavy computations on the GPU,\nand data preprocessing on the CPU). But you can also place operations and\nvariables manually on each device, if you want more control:\nAs just mentioned, you generally want to place the data\npreprocessing operations on the CPU, and place the neural\nnetwork operations on the GPUs.\nGPUs usually have a fairly limited communication bandwidth, so\nit is important to avoid unnecessary data transfers in and out of\nthe GPUs.\nAdding more CPU RAM to a machine is simple and fairly cheap,\nso there’s usually plenty of it, whereas the GPU RAM is baked\ninto the GPU: it is an expensive and thus limited resource, so if a\nvariable is not needed in the next few training steps, it should\nprobably be placed on the CPU (e.g., datasets generally belong on\nthe CPU).\nBy default, all variables and all operations will be placed on the first GPU\n(named /gpu:0), except for variables and operations that don’t have a\nGPU kernel:\n these are placed on the CPU (named /cpu:0). A tensor or\nvariable’s device attribute tells you which device it was placed on:\n>>> a = tf.Variable(42.0) \n>>> a.device \n13\n14\n15\n'/job:localhost/replica:0/task:0/device:GPU:0' \n>>> b = tf.Variable(42) \n>>> b.device \n'/job:localhost/replica:0/task:0/device:CPU:0'\nYou can safely ignore the prefix /job:localhost/replica:0/task:0 for\nnow (it allows you to place operations on other machines when using a\nTensorFlow cluster; we will talk about jobs, replicas, and tasks later in this\nchapter). As you can see, the first variable was placed on GPU 0, which is\nthe default device. However, the second variable was placed on the CPU:\nthis is because there are no GPU kernels for integer variables (or for\noperations involving integer tensors), so TensorFlow fell back to the CPU.\nIf you want to place an operation on a different device than the default\none, use a tf.device() context:\n>>> with tf.device(\"/cpu:0\"): \n... c = tf.Variable(42.0) \n... \n>>> c.device \n'/job:localhost/replica:0/task:0/device:CPU:0'\nNOTE\nThe CPU is always treated as a single device (/cpu:0), even if your machine has\nmultiple CPU cores. Any operation placed on the CPU may run in parallel across\nmultiple cores if it has a multithreaded kernel.\nIf you explicitly try to place an operation or variable on a device that does\nnot exist or for which there is no kernel, then you will get an exception.\nHowever, in some cases you may prefer to fall back to the CPU; for\nexample, if your program may run both on CPU-only machines and on\nGPU machines, you may want TensorFlow to ignore your\ntf.device(\"/gpu:*\") on CPU-only machines. To do this, you can call\ntf.config.set_soft_device_placement(True) just after importing\nTensorFlow: when a placement request fails, TensorFlow will fall back to\nits default placement rules (i.e., GPU 0 by default if it exists and there is a\nGPU kernel, and CPU 0 otherwise).\nNow how exactly will TensorFlow execute all these operations across\nmultiple devices?\nParallel Execution Across Multiple Devices\nAs we saw in Chapter 12, one of the benefits of using TF Functions is\nparallelism. Let’s look at this a bit more closely. When TensorFlow runs a\nTF Function, it starts by analyzing its graph to find the list of operations\nthat need to be evaluated, and it counts how many dependencies each of\nthem has. TensorFlow then adds each operation with zero dependencies\n(i.e., each source operation) to the evaluation queue of this operation’s\ndevice (see Figure 19-14). Once an operation has been evaluated, the\ndependency counter of each operation that depends on it is decremented.\nOnce an operation’s dependency counter reaches zero, it is pushed to the\nevaluation queue of its device. And once all the nodes that TensorFlow\nneeds have been evaluated, it returns their outputs.\nFigure 19-14. Parallelized execution of a TensorFlow graph\nOperations in the CPU’s evaluation queue are dispatched to a thread pool\ncalled the inter-op thread pool. If the CPU has multiple cores, then these\noperations will effectively be evaluated in parallel. Some operations have\nmultithreaded CPU kernels: these kernels split their tasks into multiple\nsuboperations, which are placed in another evaluation queue and\ndispatched to a second thread pool called the intra-op thread pool (shared\nby all multithreaded CPU kernels). In short, multiple operations and\nsuboperations may be evaluated in parallel on different CPU cores.\nFor the GPU, things are a bit simpler. Operations in a GPU’s evaluation\nqueue are evaluated sequentially. However, most operations have\nmultithreaded GPU kernels, typically implemented by libraries that\nTensorFlow depends on, such as CUDA and cuDNN. These\nimplementations have their own thread pools, and they typically exploit as\nmany GPU threads as they can (which is the reason why there is no need\nfor an inter-op thread pool in GPUs: each operation already floods most\nGPU threads).\nFor example, in Figure 19-14, operations A, B, and C are source ops, so\nthey can immediately be evaluated. Operations A and B are placed on the\nCPU, so they are sent to the CPU’s evaluation queue, then they are\ndispatched to the inter-op thread pool and immediately evaluated in\nparallel. Operation A happens to have a multithreaded kernel; its\ncomputations are split into three parts, which are executed in parallel by\nthe intra-op thread pool. Operation C goes to GPU 0’s evaluation queue,\nand in this example its GPU kernel happens to use cuDNN, which manages\nits own intra-op thread pool and runs the operation across many GPU\nthreads in parallel. Suppose C finishes first. The dependency counters of D\nand E are decremented and they reach zero, so both operations are pushed\nto GPU 0’s evaluation queue, and they are executed sequentially. Note that\nC only gets evaluated once, even though both D and E depend on it.\nSuppose B finishes next. Then F’s dependency counter is decremented\nfrom 4 to 3, and since that’s not 0, it does not run yet. Once A, D, and E are\nfinished, then F’s dependency counter reaches 0, and it is pushed to the\nCPU’s evaluation queue and evaluated. Finally, TensorFlow returns the\nrequested outputs.\nAn extra bit of magic that TensorFlow performs is when the TF Function\nmodifies a stateful resource, such as a variable: it ensures that the order of\nexecution matches the order in the code, even if there is no explicit\ndependency between the statements. For example, if your TF Function\ncontains v.assign_add(1) followed by v.assign(v * 2), TensorFlow\nwill ensure that these operations are executed in that order.\nTIP\nYou can control the number of threads in the inter-op thread pool by calling\ntf.config.threading.set_inter_op_parallelism_threads(). To set the\nnumber of intra-op threads, use\ntf.config.threading.set_intra_op_parallelism_threads(). This is useful\nif you want do not want TensorFlow to use all the CPU cores or if you want it to be\nsingle-threaded.\nWith that, you have all you need to run any operation on any device, and\nexploit the power of your GPUs! Here are some of the things you could do:\nYou could train several models in parallel, each on its own GPU:\njust write a training script for each model and run them in\nparallel, setting CUDA_DEVICE_ORDER and\nCUDA_VISIBLE_DEVICES so that each script only sees a single\nGPU device. This is great for hyperparameter tuning, as you can\ntrain in parallel multiple models with different hyperparameters.\nIf you have a single machine with two GPUs, and it takes one\nhour to train one model on one GPU, then training two models in\nparallel, each on its own dedicated GPU, will take just one hour.\nSimple!\nYou could train a model on a single GPU and perform all the\npreprocessing in parallel on the CPU, using the dataset’s\nprefetch() method\n to prepare the next few batches in advance\nso that they are ready when the GPU needs them (see Chapter 13).\nIf your model takes two images as input and processes them using\ntwo CNNs before joining their outputs, then it will probably run\nmuch faster if you place each CNN on a different GPU.\nYou can create an efficient ensemble: just place a different trained\nmodel on each GPU so that you can get all the predictions much\nfaster to produce the ensemble’s final prediction.\n16\n17\nBut what if you want to train a single model across multiple GPUs?\nTraining Models Across Multiple Devices\nThere are two main approaches to training a single model across multiple\ndevices: model parallelism, where the model is split across the devices,\nand data parallelism, where the model is replicated across every device,\nand each replica is trained on a subset of the data. Let’s look at these two\noptions closely before we train a model on multiple GPUs.\nModel Parallelism\nSo far we have trained each neural network on a single device. What if we\nwant to train a single neural network across multiple devices? This\nrequires chopping the model into separate chunks and running each chunk\non a different device. Unfortunately, such model parallelism turns out to\nbe pretty tricky, and it really depends on the architecture of your neural\nnetwork. For fully connected networks, there is generally not much to be\ngained from this approach (see Figure 19-15). Intuitively, it may seem that\nan easy way to split the model is to place each layer on a different device,\nbut this does not work because each layer needs to wait for the output of\nthe previous layer before it can do anything. So perhaps you can slice it\nvertically—for example, with the left half of each layer on one device, and\nthe right part on another device? This is slightly better, since both halves\nof each layer can indeed work in parallel, but the problem is that each half\nof the next layer requires the output of both halves, so there will be a lot of\ncross-device communication (represented by the dashed arrows). This is\nlikely to completely cancel out the benefit of the parallel computation,\nsince cross-device communication is slow (especially when the devices\nare located on different machines).\nFigure 19-15. Splitting a fully connected neural network\nSome neural network architectures, such as convolutional neural networks\n(see Chapter 14), contain layers that are only partially connected to the\nlower layers, so it is much easier to distribute chunks across devices in an\nefficient way (Figure 19-16).\nFigure 19-16. Splitting a partially connected neural network\nDeep recurrent neural networks (see Chapter 15) can be split a bit more\nefficiently across multiple GPUs. If you split the network horizontally by\nplacing each layer on a different device, and you feed the network with an\ninput sequence to process, then at the first time step only one device will\nbe active (working on the sequence’s first value), at the second step two\nwill be active (the second layer will be handling the output of the first\nlayer for the first value, while the first layer will be handling the second\nvalue), and by the time the signal propagates to the output layer, all\ndevices will be active simultaneously (Figure 19-17). There is still a lot of\ncross-device communication going on, but since each cell may be fairly\ncomplex, the benefit of running multiple cells in parallel may (in theory)\noutweigh the communication penalty. However, in practice a regular stack\nof LSTM layers running on a single GPU actually runs much faster.\nFigure 19-17. Splitting a deep recurrent neural network\nIn short, model parallelism may speed up running or training some types\nof neural networks, but not all, and it requires special care and tuning,\nsuch as making sure that devices that need to communicate the most run\non the same machine.\n Let’s look at a much simpler and generally more\nefficient option: data parallelism.\nData Parallelism\nAnother way to parallelize the training of a neural network is to replicate\nit on every device and run each training step simultaneously on all\nreplicas, using a different mini-batch for each. The gradients computed by\neach replica are then averaged, and the result is used to update the model\nparameters. This is called data parallelism. There are many variants of\nthis idea, so let’s look at the most important ones.\nData parallelism using the mirrored strategy\nArguably the simplest approach is to completely mirror all the model\nparameters across all the GPUs and always apply the exact same\nparameter updates on every GPU. This way, all replicas always remain\nperfectly identical. This is called the mirrored strategy, and it turns out to\nbe quite efficient, especially when using a single machine (see Figure 19-\n18).\n18\nFigure 19-18. Data parallelism using the mirrored strategy\nThe tricky part when using this approach is to efficiently compute the\nmean of all the gradients from all the GPUs and distribute the result across\nall the GPUs. This can be done using an AllReduce algorithm, a class of\nalgorithms where multiple nodes collaborate to efficiently perform a\nreduce operation (such as computing the mean, sum, and max), while\nensuring that all nodes obtain the same final result. Fortunately, there are\noff-the-shelf implementations of such algorithms, as we will see.\nData parallelism with centralized parameters\nAnother approach is to store the model parameters outside of the GPU\ndevices performing the computations (called workers), for example on the\nCPU (see Figure 19-19). In a distributed setup, you may place all the\nparameters on one or more CPU-only servers called parameter servers,\nwhose only role is to host and update the parameters.\nFigure 19-19. Data parallelism with centralized parameters\nWhereas the mirrored strategy imposes synchronous weight updates across\nall GPUs, this centralized approach allows either synchronous or\nasynchronous updates. Let’s see the pros and cons of both options.\nSynchronous updates\nWith synchronous updates, the aggregator waits until all gradients are\navailable before it computes the average gradients and passes them to the\noptimizer, which will update the model parameters. Once a replica has\nfinished computing its gradients, it must wait for the parameters to be\nupdated before it can proceed to the next mini-batch. The downside is that\nsome devices may be slower than others, so all other devices will have to\nwait for them at every step. Moreover, the parameters will be copied to\nevery device almost at the same time (immediately after the gradients are\napplied), which may saturate the parameter servers’ bandwidth.\nTIP\nTo reduce the waiting time at each step, you could ignore the gradients from the\nslowest few replicas (typically ~10%). For example, you could run 20 replicas, but\nonly aggregate the gradients from the fastest 18 replicas at each step, and just ignore\nthe gradients from the last 2. As soon as the parameters are updated, the first 18\nreplicas can start working again immediately, without having to wait for the 2\nslowest replicas. This setup is generally described as having 18 replicas plus 2 spare\nreplicas.\nAsynchronous updates\nWith asynchronous updates, whenever a replica has finished computing\nthe gradients, it immediately uses them to update the model parameters.\nThere is no aggregation (it removes the “mean” step in Figure 19-19) and\nno synchronization. Replicas work independently of the other replicas.\nSince there is no waiting for the other replicas, this approach runs more\ntraining steps per minute. Moreover, although the parameters still need to\nbe copied to every device at every step, this happens at different times for\neach replica, so the risk of bandwidth saturation is reduced.\nData parallelism with asynchronous updates is an attractive choice\nbecause of its simplicity, the absence of synchronization delay, and a better\nuse of the bandwidth. However, although it works reasonably well in\npractice, it is almost surprising that it works at all! Indeed, by the time a\nreplica has finished computing the gradients based on some parameter\nvalues, these parameters will have been updated several times by other\nreplicas (on average N – 1 times, if there are N replicas), and there is no\nguarantee that the computed gradients will still be pointing in the right\ndirection (see Figure 19-20). When gradients are severely out-of-date, they\nare called stale gradients: they can slow down convergence, introducing\nnoise and wobble effects (the learning curve may contain temporary\noscillations), or they can even make the training algorithm diverge.\n19\nFigure 19-20. Stale gradients when using asynchronous updates\nThere are a few ways you can reduce the effect of stale gradients:\nReduce the learning rate.\nDrop stale gradients or scale them down.\nAdjust the mini-batch size.\nStart the first few epochs using just one replica (this is called the\nwarmup phase). Stale gradients tend to be more damaging at the\nbeginning of training, when gradients are typically large and the\nparameters have not settled into a valley of the cost function yet,\nso different replicas may push the parameters in quite different\ndirections.\nA paper published by the Google Brain team in 2016\n benchmarked\nvarious approaches and found that using synchronous updates with a few\nspare replicas was more efficient than using asynchronous updates, not\nonly converging faster but also producing a better model. However, this is\nstill an active area of research, so you should not rule out asynchronous\nupdates just yet.\n20\nBandwidth saturation\nWhether you use synchronous or asynchronous updates, data parallelism\nwith centralized parameters still requires communicating the model\nparameters from the parameter servers to every replica at the beginning of\neach training step, and the gradients in the other direction at the end of\neach training step. Similarly, when using the mirrored strategy, the\ngradients produced by each GPU will need to be shared with every other\nGPU. Unfortunately, there always comes a point where adding an extra\nGPU will not improve performance at all because the time spent moving\nthe data into and out of GPU RAM (and across the network in a distributed\nsetup) will outweigh the speedup obtained by splitting the computation\nload. At that point, adding more GPUs will just worsen the bandwidth\nsaturation and actually slow down training.\nTIP\nFor some models, typically relatively small and trained on a very large training set,\nyou are often better off training the model on a single machine with a single\npowerful GPU with a large memory bandwidth.\nSaturation is more severe for large dense models, since they have a lot of\nparameters and gradients to transfer. It is less severe for small models (but\nthe parallelization gain is limited) and for large sparse models, where the\ngradients are typically mostly zeros and so can be communicated\nefficiently. Jeff Dean, initiator and lead of the Google Brain project,\nreported typical speedups of 25–40× when distributing computations\nacross 50 GPUs for dense models, and a 300× speedup for sparser models\ntrained across 500 GPUs. As you can see, sparse models really do scale\nbetter. Here are a few concrete examples:\nNeural machine translation: 6× speedup on 8 GPUs\nInception/ImageNet: 32× speedup on 50 GPUs\nRankBrain: 300× speedup on 500 GPUs\nBeyond a few dozen GPUs for a dense model or few hundred GPUs for a\nsparse model, saturation kicks in and performance degrades. There is\nplenty of research going on to solve this problem (exploring peer-to-peer\narchitectures rather than centralized parameter servers, using lossy model\ncompression, optimizing when and what the replicas need to\ncommunicate, and so on), so there will likely be a lot of progress in\nparallelizing neural networks in the next few years.\nIn the meantime, to reduce the saturation problem, you probably want to\nuse a few powerful GPUs rather than plenty of weak GPUs, and you should\nalso group your GPUs on few and very well interconnected servers. You\ncan also try dropping the float precision from 32 bits (tf.float32) to 16\nbits (tf.bfloat16). This will cut in half the amount of data to transfer,\noften without much impact on the convergence rate or the model’s\nperformance. Lastly, if you are using centralized parameters, you can\nshard (split) the parameters across multiple parameter servers: adding\nmore parameter servers will reduce the network load on each server and\nlimit the risk of bandwidth saturation.\nOK, now let’s train a model across multiple GPUs!\nTraining at Scale Using the Distribution Strategies API\nMany models can be trained quite well on a single GPU, or even on a CPU.\nBut if training is too slow, you can try distributing it across multiple GPUs\non the same machine. If that’s still too slow, try using more powerful\nGPUs, or add more GPUs to the machine. If your model performs heavy\ncomputations (such as large matrix multiplications), then it will run much\nfaster on powerful GPUs, and you could even try to use TPUs on Google\nCloud AI Platform, which will usually run even faster for such models.\nBut if you can’t fit any more GPUs on the same machine, and if TPUs\naren’t for you (e.g., perhaps your model doesn’t benefit much from TPUs,\nor perhaps you want to use your own hardware infrastructure), then you\ncan try training it across several servers, each with multiple GPUs (if this\nis still not enough, as a last resort you can try adding some model\nparallelism, but this requires a lot more effort). In this section we will see\nhow to train models at scale, starting with multiple GPUs on the same\nmachine (or TPUs) and then moving on to multiple GPUs across multiple\nmachines.\nLuckily, TensorFlow comes with a very simple API that takes care of all\nthe complexity for you: the Distribution Strategies API. To train a Keras\nmodel across all available GPUs (on a single machine, for now) using data\nparallelism with the mirrored strategy, create a MirroredStrategy object,\ncall its scope() method to get a distribution context, and wrap the\ncreation and compilation of your model inside that context. Then call the\nmodel’s fit() method normally:\ndistribution = tf.distribute.MirroredStrategy() \n \nwith distribution.scope(): \n mirrored_model = tf.keras.Sequential([...]) \n mirrored_model.compile([...]) \n \nbatch_size = 100 # must be divisible by the number of replicas \nhistory = mirrored_model.fit(X_train, y_train, epochs=10)\nUnder the hood, tf.keras is distribution-aware, so in this\nMirroredStrategy context it knows that it must replicate all variables\nand operations across all available GPU devices. Note that the fit()\nmethod will automatically split each training batch across all the replicas,\nso it’s important that the batch size be divisible by the number of replicas.\nAnd that’s all! Training will generally be significantly faster than using a\nsingle device, and the code change was really minimal.\nOnce you have finished training your model, you can use it to make\npredictions efficiently: call the predict() method, and it will\nautomatically split the batch across all replicas, making predictions in\nparallel (again, the batch size must be divisible by the number of replicas).\nIf you call the model’s save() method, it will be saved as a regular model,\nnot as a mirrored model with multiple replicas. So when you load it, it will\nrun like a regular model, on a single device (by default GPU 0, or the CPU\nif there are no GPUs). If you want to load a model and run it on all\navailable devices, you must call keras.models.load_model() within a\ndistribution context:\nwith distribution.scope(): \n mirrored_model = keras.models.load_model(\"my_mnist_model.h5\")\nIf you only want to use a subset of all the available GPU devices, you can\npass the list to the MirroredStrategy’s constructor:\ndistribution = tf.distribute.MirroredStrategy([\"/gpu:0\", \"/gpu:1\"])\nBy default, the MirroredStrategy class uses the NVIDIA Collective\nCommunications Library (NCCL) for the AllReduce mean operation, but\nyou can change it by setting the cross_device_ops argument to an\ninstance of the tf.distribute.HierarchicalCopyAllReduce class, or an\ninstance of the tf.distribute.ReductionToOneDevice class. The\ndefault NCCL option is based on the tf.distribute.NcclAllReduce\nclass, which is usually faster, but this depends on the number and types of\nGPUs, so you may want to give the alternatives a try.\nIf you want to try using data parallelism with centralized parameters,\nreplace the MirroredStrategy with the CentralStorageStrategy:\ndistribution = tf.distribute.experimental.CentralStorageStrategy()\nYou can optionally set the compute_devices argument to specify the list\nof devices you want to use as workers (by default it will use all available\nGPUs), and you can optionally set the parameter_device argument to\nspecify the device you want to store the parameters on (by default it will\nuse the CPU, or the GPU if there is just one).\nNow let’s see how to train a model across a cluster of TensorFlow servers!\nTraining a Model on a TensorFlow Cluster\n21\nA TensorFlow cluster is a group of TensorFlow processes running in\nparallel, usually on different machines, and talking to each other to\ncomplete some work—for example, training or executing a neural\nnetwork. Each TF process in the cluster is called a task, or a TF server. It\nhas an IP address, a port, and a type (also called its role or its job). The\ntype can be either \"worker\", \"chief\", \"ps\" (parameter server), or\n\"evaluator\":\nEach worker performs computations, usually on a machine with\none or more GPUs.\nThe chief performs computations as well (it is a worker), but it\nalso handles extra work such as writing TensorBoard logs or\nsaving checkpoints. There is a single chief in a cluster. If no chief\nis specified, then the first worker is the chief.\nA parameter server only keeps track of variable values, and it is\nusually on a CPU-only machine. This type of task is only used\nwith the ParameterServerStrategy.\nAn evaluator obviously takes care of evaluation.\nTo start a TensorFlow cluster, you must first specify it. This means\ndefining each task’s IP address, TCP port, and type. For example, the\nfollowing cluster specification defines a cluster with three tasks (two\nworkers and one parameter server; see Figure 19-21). The cluster spec is a\ndictionary with one key per job, and the values are lists of task addresses\n(IP:port):\ncluster_spec = { \n \"worker\": [ \n \"machine-a.example.com:2222\", # /job:worker/task:0 \n \"machine-b.example.com:2222\" # /job:worker/task:1 \n ], \n \"ps\": [\"machine-a.example.com:2221\"] # /job:ps/task:0 \n}\nFigure 19-21. TensorFlow cluster\nIn general there will be a single task per machine, but as this example\nshows, you can configure multiple tasks on the same machine if you want\n(if they share the same GPUs, make sure the RAM is split appropriately, as\ndiscussed earlier).\nWARNING\nBy default, every task in the cluster may communicate with every other task, so\nmake sure to configure your firewall to authorize all communications between these\nmachines on these ports (it’s usually simpler if you use the same port on every\nmachine).\nWhen you start a task, you must give it the cluster spec, and you must also\ntell it what its type and index are (e.g., worker 0). The simplest way to\nspecify everything at once (both the cluster spec and the current task’s\ntype and index) is to set the TF_CONFIG environment variable before\nstarting TensorFlow. It must be a JSON-encoded dictionary containing a\ncluster specification (under the \"cluster\" key) and the type and index of\nthe current task (under the \"task\" key). For example, the following\nTF_CONFIG environment variable uses the cluster we just defined and\nspecifies that the task to start is the first worker:\nimport os \nimport json \n \nos.environ[\"TF_CONFIG\"] = json.dumps({ \n \"cluster\": cluster_spec, \n \"task\": {\"type\": \"worker\", \"index\": 0} \n})\nTIP\nIn general you want to define the TF_CONFIG environment variable outside of\nPython, so the code does not need to include the current task’s type and index (this\nmakes it possible to use the same code across all workers).\nNow let’s train a model on a cluster! We will start with the mirrored\nstrategy—it’s surprisingly simple! First, you need to set the TF_CONFIG\nenvironment variable appropriately for each task. There should be no\nparameter server (remove the “ps” key in the cluster spec), and in general\nyou will want a single worker per machine. Make extra sure you set a\ndifferent task index for each task. Finally, run the following training code\non every worker:\ndistribution = tf.distribute.experimental.MultiWorkerMirroredStrategy() \n \nwith distribution.scope(): \n mirrored_model = tf.keras.Sequential([...]) \n mirrored_model.compile([...]) \n \nbatch_size = 100 # must be divisible by the number of replicas \nhistory = mirrored_model.fit(X_train, y_train, epochs=10)\nYes, that’s exactly the same code we used earlier, except this time we are\nusing the MultiWorkerMirroredStrategy (in future versions, the\nMirroredStrategy will probably handle both the single machine and\nmultimachine cases). When you start this script on the first workers, they\nwill remain blocked at the AllReduce step, but as soon as the last worker\nstarts up training will begin, and you will see them all advancing at\nexactly the same rate (since they synchronize at each step).\nYou can choose from two AllReduce implementations for this distribution\nstrategy: a ring AllReduce algorithm based on gRPC for the network\ncommunications, and NCCL’s implementation. The best algorithm to use\ndepends on the number of workers, the number and types of GPUs, and the\nnetwork. By default, TensorFlow will apply some heuristics to select the\nright algorithm for you, but if you want to force one algorithm, pass\nCollectiveCommunication.RING or CollectiveCommunication.NCCL\n(from tf.distribute.experimental) to the strategy’s constructor.\nIf you prefer to implement asynchronous data parallelism with parameter\nservers, change the strategy to ParameterServerStrategy, add one or\nmore parameter servers, and configure TF_CONFIG appropriately for each\ntask. Note that although the workers will work asynchronously, the\nreplicas on each worker will work synchronously.\nLastly, if you have access to TPUs on Google Cloud, you can create a\nTPUStrategy like this (then use it like the other strategies):\nresolver = tf.distribute.cluster_resolver.TPUClusterResolver() \ntf.tpu.experimental.initialize_tpu_system(resolver) \ntpu_strategy = tf.distribute.experimental.TPUStrategy(resolver)\nTIP\nIf you are a researcher, you may be eligible to use TPUs for free; see\nhttps://tensorflow.org/tfrc for more details.\nYou can now train models across multiple GPUs and multiple servers: give\nyourself a pat on the back! If you want to train a large model, you will\nneed many GPUs, across many servers, which will require either buying a\nlot of hardware or managing a lot of cloud VMs. In many cases, it’s going\nto be less hassle and less expensive to use a cloud service that takes care\nof provisioning and managing all this infrastructure for you, just when you\nneed it. Let’s see how to do that on GCP.\nRunning Large Training Jobs on Google Cloud AI\nPlatform\nIf you decide to use Google AI Platform, you can deploy a training job\nwith the same training code as you would run on your own TF cluster, and\nthe platform will take care of provisioning and configuring as many GPU\nVMs as you desire (within your quotas).\nTo start the job, you will need the gcloud command-line tool, which is\npart of the Google Cloud SDK. You can either install the SDK on your own\nmachine, or just use the Google Cloud Shell on GCP. This is a terminal\nyou can use directly in your web browser; it runs on a free Linux VM\n(Debian), with the SDK already installed and preconfigured for you. The\nCloud Shell is available anywhere in GCP: just click the Activate Cloud\nShell icon at the top right of the page (see Figure 19-22).\nFigure 19-22. Activating the Google Cloud Shell\nIf you prefer to install the SDK on your machine, once you have installed\nit, you need to initialize it by running gcloud init: you will need to log\nin to GCP and grant access to your GCP resources, then select the GCP\nproject you want to use (if you have more than one), as well as the region\nwhere you want the job to run. The gcloud command gives you access to\nevery GCP feature, including the ones we used earlier. You don’t have to\ngo through the web interface every time; you can write scripts that start or\nstop VMs for you, deploy models, or perform any other GCP action.\nBefore you can run the training job, you need to write the training code,\nexactly like you did earlier for a distributed setup (e.g., using the\nParameterServerStrategy). AI Platform will take care of setting\nTF_CONFIG for you on each VM. Once that’s done, you can deploy it and\nrun it on a TF cluster with a command line like this:\n$ gcloud ai-platform jobs submit training my_job_20190531_164700 \\ \n --region asia-southeast1 \\ \n --scale-tier PREMIUM_1 \\ \n --runtime-version 2.0 \\ \n --python-version 3.5 \\ \n --package-path /my_project/src/trainer \\ \n --module-name trainer.task \\ \n --staging-bucket gs://my-staging-bucket \\ \n --job-dir gs://my-mnist-model-bucket/trained_model \\ \n -- \n --my-extra-argument1 foo --my-extra-argument2 bar \nLet’s go through these options. The command will start a training job\nnamed my_job_20190531_164700, in the asia-southeast1 region, using\na PREMIUM_1 scale tier: this corresponds to 20 workers (including a chief)\nand 11 parameter servers (check out the other available scale tiers). All\nthese VMs will be based on AI Platform’s 2.0 runtime (a VM\nconfiguration that includes TensorFlow 2.0 and many other packages)\nand Python 3.5. The training code is located in the /my_project/src/trainer\ndirectory, and the gcloud command will automatically bundle it into a pip\npackage and upload it to GCS at gs://my-staging-bucket. Next, AI Platform\nwill start several VMs, deploy the package to them, and run the\ntrainer.task module. Lastly, the --job-dir argument and the extra\narguments (i.e., all the arguments located after the -- separator) will be\npassed to the training program: the chief task will usually use the --job-\ndir argument to find out where to save the final model on GCS, in this\ncase at gs://my-mnist-model-bucket/trained_model. And that’s it! In the\nGCP console, you can then open the navigation menu, scroll down to the\n22\nArtificial Intelligence section, and open AI Platform → Jobs. You should\nsee your job running, and if you click it you will see graphs showing the\nCPU, GPU, and RAM utilization for every task. You can click View Logs\nto access the detailed logs using Stackdriver.\nNOTE\nIf you place the training data on GCS, you can create a tf.data.TextLineDataset\nor tf.data.TFRecordDataset to access it: just use the GCS paths as the filenames\n(e.g., gs://my-data-bucket/my_data_001.csv). These datasets rely on the\ntf.io.gfile package to access files: it supports both local files and GCS files (but\nmake sure the service account you use has access to GCS).\nIf you want to explore a few hyperparameter values, you can simply run\nmultiple jobs and specify the hyperparameter values using the extra\narguments for your tasks. However, if you want to explore many\nhyperparameters efficiently, it’s a good idea to use AI Platform’s\nhyperparameter tuning service instead.\nBlack Box Hyperparameter Tuning on AI Platform\nAI Platform provides a powerful Bayesian optimization hyperparameter\ntuning service called Google Vizier.\n To use it, you need to pass a YAML\nconfiguration file when creating the job (--config tuning.yaml). For\nexample, it may look like this:\ntrainingInput: \n hyperparameters: \n goal: MAXIMIZE \n hyperparameterMetricTag: accuracy \n maxTrials: 10 \n maxParallelTrials: 2 \n params: \n - parameterName: n_layers \n type: INTEGER \n minValue: 10 \n maxValue: 100 \n scaleType: UNIT_LINEAR_SCALE \n23\nyp\n_\n_\n - parameterName: momentum \n type: DOUBLE \n minValue: 0.1 \n maxValue: 1.0 \n scaleType: UNIT_LOG_SCALE\nThis tells AI Platform that we want to maximize the metric named\n\"accuracy\", the job will run a maximum of 10 trials (each trial will run\nour training code to train the model from scratch), and it will run a\nmaximum of 2 trials in parallel. We want it to tune two hyperparameters:\nthe n_layers hyperparameter (an integer between 10 and 100) and the\nmomentum hyperparameter (a float between 0.1 and 1.0). The scaleType\nargument specifies the prior for the hyperparameter value:\nUNIT_LINEAR_SCALE means a flat prior (i.e., no a priori preference), while\nUNIT_LOG_SCALE says we have a prior belief that the optimal value lies\ncloser to the max value (the other possible prior is\nUNIT_REVERSE_LOG_SCALE, when we believe the optimal value to be close\nto the min value).\nThe n_layers and momentum arguments will be passed as command-line\narguments to the training code, and of course it is expected to use them.\nThe question is, how will the training code communicate the metric back\nto the AI Platform so that it can decide which hyperparameter values to\nuse during the next trial? Well, AI Platform just monitors the output\ndirectory (specified via --job-dir) for any event file (introduced in\nChapter 10) containing summaries for a metric named \"accuracy\" (or\nwhatever metric name is specified as the hyperparameterMetricTag),\nand it reads those values. So your training code simply has to use the\nTensorBoard() callback (which you will want to do anyway for\nmonitoring), and you’re good to go!\nOnce the job is finished, all the hyperparameter values used in each trial\nand the resulting accuracy will be available in the job’s output (available\nvia the AI Platform → Jobs page).\nNOTE\nAI Platform jobs can also be used to efficiently execute your model on large\namounts of data: each worker can read part of the data from GCS, make predictions,\nand save them to GCS.\nNow you have all the tools and knowledge you need to create state-of-the-\nart neural net architectures and train them at scale using various\ndistribution strategies, on your own infrastructure or on the cloud—and\nyou can even perform powerful Bayesian optimization to fine-tune the\nhyperparameters!\nExercises\n1. What does a SavedModel contain? How do you inspect its\ncontent?\n2. When should you use TF Serving? What are its main features?\nWhat are some tools you can use to deploy it?\n3. How do you deploy a model across multiple TF Serving\ninstances?\n4. When should you use the gRPC API rather than the REST API to\nquery a model served by TF Serving?\n5. What are the different ways TFLite reduces a model’s size to\nmake it run on a mobile or embedded device?\n6. What is quantization-aware training, and why would you need it?\n7. What are model parallelism and data parallelism? Why is the\nlatter generally recommended?\n8. When training a model across multiple servers, what distribution\nstrategies can you use? How do you choose which one to use?\n9. Train a model (any model you like) and deploy it to TF Serving or\nGoogle Cloud AI Platform. Write the client code to query it using\nthe REST API or the gRPC API. Update the model and deploy the\nnew version. Your client code will now query the new version.\nRoll back to the first version.\n10. Train any model across multiple GPUs on the same machine\nusing the MirroredStrategy (if you do not have access to GPUs,\nyou can use Colaboratory with a GPU Runtime and create two\nvirtual GPUs). Train the model again using the\nCentralStorageStrategy and compare the training time.\n11. Train a small model on Google Cloud AI Platform, using black\nbox hyperparameter tuning.\nThank You!\nBefore we close the last chapter of this book, I would like to thank you for\nreading it up to the last paragraph. I truly hope that you had as much\npleasure reading this book as I had writing it, and that it will be useful for\nyour projects, big or small.\nIf you find errors, please send feedback. More generally, I would love to\nknow what you think, so please don’t hesitate to contact me via O’Reilly,\nthrough the ageron/handson-ml2 GitHub project, or on Twitter at\n@aureliengeron.\nGoing forward, my best advice to you is to practice and practice: try going\nthrough all the exercises (if you have not done so already), play with the\nJupyter notebooks, join Kaggle.com or some other ML community, watch\nML courses, read papers, attend conferences, and meet experts. It also\nhelps tremendously to have a concrete project to work on, whether it is for\nwork or for fun (ideally for both), so if there’s anything you have always\ndreamt of building, give it a shot! Work incrementally; don’t shoot for the\nmoon right away, but stay focused on your project and build it piece by\npiece. It will require patience and perseverance, but when you have a\nwalking robot, or a working chatbot, or whatever else you fancy to build, it\nwill be immensely rewarding.\nMy greatest hope is that this book will inspire you to build a wonderful\nML application that will benefit all of us! What will it be?\n—Aurélien Géron, June 17, 2019\n1 An A/B experiment consists in testing two different versions of your product on different\nsubsets of users in order to check which version works best and get other insights.\n2 A REST (or RESTful) API is an API that uses standard HTTP verbs, such as GET, POST,\nPUT, and DELETE, and uses JSON inputs and outputs. The gRPC protocol is more\ncomplex but more efficient. Data is exchanged using Protocol Buffers (see Chapter 13).\n3 If you are not familiar with Docker, it allows you to easily download a set of applications\npackaged in a Docker image (including all their dependencies and usually some good\ndefault configuration) and then run them on your system using a Docker engine. When you\nrun an image, the engine creates a Docker container that keeps the applications well\nisolated from your own system (but you can give it some limited access if you want). It is\nsimilar to a virtual machine, but much faster and more lightweight, as the container relies\ndirectly on the host’s kernel. This means that the image does not need to include or run its\nown kernel.\n4 To be fair, this can be mitigated by serializing the data first and encoding it to Base64\nbefore creating the REST request. Moreover, REST requests can be compressed using gzip,\nwhich reduces the payload size significantly.\n5 If the SavedModel contains some example instances in the assets/extra directory, you can\nconfigure TF Serving to execute the model on these instances before starting to serve new\nrequests with it. This is called model warmup: it will ensure that everything is properly\nloaded, avoiding long response times for the first requests.\n6 At the time of this writing, TensorFlow version 2 is not available yet on AI Platform, but\nthat’s OK: you can use 1.13, and it will run your TF 2 SavedModels just fine.\n7 If you get an error saying that module google.appengine was not found, set\ncache_discovery=False in the call to the build() method; see\nhttps://stackoverflow.com/q/55561354.\n8 Also check out TensorFlow’s Graph Transform Tools for modifying and optimizing\ncomputational graphs.\n9 If you’re interested in this topic, check out federated learning.\n10 Please check the docs for detailed and up-to-date installation instructions, as they change\nquite often.\n11 Many code examples in this chapter use experimental APIs. They are very likely to be\nmoved to the core API in future versions. So if an experimental function fails, try simply\nremoving the word experimental, and hopefully it will work. If not, then perhaps the API\nhas changed a bit; please check the Jupyter notebook, as I will ensure it contains the correct\ncode.\n12 Presumably, these quotas are meant to stop bad guys who might be tempted to use GCP\nwith stolen credit cards to mine cryptocurrencies.\n13 Martín Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous\nDistributed Systems” Google Research whitepaper (2015).\n14 As we saw in Chapter 12, a kernel is a variable or operation’s implementation for a\nspecific data type and device type. For example, there is a GPU kernel for the float32\ntf.matmul() operation, but there is no GPU kernel for int32 tf.matmul() (only a CPU\nkernel).\n15 You can also use tf.debugging.set_log_device_placement(True) to log all device\nplacements.\n16 This can be useful if you want to guarantee perfect reproducibility, as I explain in this\nvideo, based on TF 1.\n17 At the time of this writing it only prefetches the data to the CPU RAM, but you can use\ntf.data.experimental.prefetch_to_device() to make it prefetch the data and push\nit to the device of your choice so that the GPU does not waste time waiting for the data to\nbe transferred.\n18 If you are interested in going further with model parallelism, check out Mesh TensorFlow.\n19 This name is slightly confusing because it sounds like some replicas are special, doing\nnothing. In reality, all replicas are equivalent: they all work hard to be among the fastest at\neach training step, and the losers vary at every step (unless some devices are really slower\nthan others). However, it does mean that if a server crashes, training will continue just fine.\n20 Jianmin Chen et al., “Revisiting Distributed Synchronous SGD,” arXiv preprint\narXiv:1604.00981 (2016).\n21 For more details on AllReduce algorithms, read this great post by Yuichiro Ueno, and this\npage on scaling with NCCL.\n22 At the time of this writing, the 2.0 runtime is not yet available, but it should be ready by\nthe time you read this. Check out the list of available runtimes.\n23 Daniel Golovin et al., “Google Vizier: A Service for Black-Box Optimization,”\nProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery\nand Data Mining (2017): 1487–1495." + } + ] + } +] \ No newline at end of file