```python
from IPython.core.display import HTML
HTML("<style>.container { width:95% !important; }</style>")
```

# Lecture 11, Solution methods for multiobjective optimization

## Reminder:

### Mathematical formulation of multiobjective optimization problems

Multiobjective optimization problems are often formulated as
$$
\begin{align} \
\min \quad &\{f_1(x),\ldots,f_k(x)\}\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_q(x) = 0\text{ for all }q=1,\ldots,Q\\
&a_i\leq x_i\leq b_i\text{ for all } i=1,\ldots,n\\
&x\in \mathbb R^n,
\end{align}
$$
where $$f_1,\ldots,f_k:\{x\in\mathbb R^n: g_j(x) \geq 0 \text{ for all }j=1,\ldots,J \text{ and } h_q(x) = 0\text{ for all }q=1,\ldots,Q\}\mapsto\mathbb R$$ are the objective functions.

## Pareto optimality
A feasible solution $x_1$ is Pareto optimal to the multiobjective optimization problem, if there does not exist a feasible solution $x_2$, $x_1\neq x_2$, such that
$$
\left\{
\begin{align}
&f_i(x_2)\leq f_i(x_1)\text{ for all }i\in \{1,\ldots,k\}\\
&f_j(x_2)<f_j(x_1)\text{ for some }j\in \{1,\ldots,k\}.\\
\end{align}
\right.
$$

### Basic concepts

* There is no single optimal solution, instead we have a set of solutions called pareto optimal set.
* All the objectives don’t have the same optimal solution → optimality needs to be modified

### PARETO OPTIMALITY (PO)
* A solution is pareto optimal if none of the objectives can be improved without impairing at least one of the others

It means:
$$
\text{“Take from Sami to pay Anna”}
$$

* Optimal solutions are located at the boundary to the down & left (for minimization problems)*

* There are to spaces connected to the problem: the space $\mathbb R^n$ is called the decision space and $\mathbb R^k$ is called the objective space.

1. **Decision space**: includes the **Pareto optimal solution set**
2. **Objective space**: consists of the image of Pareto optimal solutions (**Pareto frontier**)

## Some more concepts:

In addition to Pareto optimality, two more concepts are important, which are called the ideal and the nadir vector.

* **Ideal objective vector $𝒛^∗$:** best values for each objective (when optimized independently)
* **Nadir objective vector $𝒛^𝑛𝑎𝑑$:** worst values for each objective in PO set

Mathematically the ideal vector $z^{ideal}$ can be defined as having
$$
z^{ideal}_i = \begin{align} \
\min \quad &f_i(x)\\
\text{s.t.} \quad &x\text{ is feasible}
\end{align}
$$
for all $i=1,\ldots,k$ (i.e., by **solving single-objective optimization problems, one for each objective**).

The nadir vector $z^{nadir}$ on the other hand has
$$
z^{nadir}_i =
\begin{align}
\max \quad &f_i(x)\\
\text{s.t.} \quad &x\text{ is Pareto optimal},
\end{align}
$$
for all $i=1,\ldots,k$ (**not as straightforward as calculating the ideal points for more than two objectives**).

## Optimization problem formulation
* By optimizing only one criterion, the rest are not considered
* Objective vs. constraint
* Summation of the objectives
   * adding apples and oranges
* Converting the objectives (e.g. as costs)
   * not easy, includes uncertainteis
* Multiobjective formulation reveals interdependences between the objectives

## Example

Consider multiobjective optimization problem
$$
\min \{f_1(x,y)=x^2+y,\quad f_2(x,y)=1-x\}\\
\text{s.t. }x\in[0,1], y\geq0.
$$

#### Pareto optimal solutions
Now, the set of Pareto optimal solutions is

$$
\{(x,0):x\in[0,1]\}.
$$

How to show this?

Let's show that $(x',0)$ is Pareto optimal for all $x'\in[0,1]$. *The idea of the proof: assume that $(x',0)$ is not Pareto optimal and then deduce a contradiction.*

Let's assume $(x,y)$ with $x\in[0,1]$ and $y\geq0$ such that

$$
\left\{
\begin{align}
f_1(x,y)=x^2+y\leq x'^2=f_1(x',0),\textbf{ and}\\
f_2(x,y)=1-x\leq 1-x'=f_2(x',0).
\end{align}
\right.
$$

and

$$
\left\{
\begin{align}
f_1(x,y)=x^2+y< x'^2 =f_1(x',0)\textbf{ or}\\
f_2(x,y)=1-x< 1-x'=f_2(x',0).
\end{align}
\right.
$$

Second inequality in the first system of inequalities gives $x\geq x'$. This yields from the first inequality in that same system of inequalities

$$
y\leq x'^2-x^2\leq 0.
$$

Thus, $y=0$. This means that $x=x'$ using again the first inequality.

This means that the solution cannot satisfy the second system of strict inequalities. We have a contradiction and, therefore, $(x',0)$ has to be Pareto optimal.

Now, we show that any other feasible solution can not be Pareto optimal. Let's assume a solution $(x,y)$, where $x\in[0,1]$ and $y>0$ and show that this is not Pareto optimal:

By choosing solution $(x,0)$, we have

$$
\left\{
\begin{align}
f_1(x,0)=x^2<x^2+y=f_1(x,y) ,\text{ and}\\
f_2(x,0)=1-x\leq 1-x=f_2(x,y).
\end{align}
\right.
$$

Thus, the solution $(x,y)$ cannot be Pareto optimal.

#### Ideal
Now

$$
\begin{align}
\min x^2+y\\
\text{s.t. }x\in[0,1],\ y\geq0
\end{align}
= 0
$$

and

$$
\begin{align}
\min 1-x\\
\text{s.t. }x\in[0,1],\ y\geq0
\end{align}
= 0.
$$

Thus, the ideal is

$$
z^{ideal} = (0,0)^T
$$

#### Nadir
Now,

$$
\begin{align}
\max x^2+y\\
\text{s.t. }x\in[0,1],\ y=0
\end{align}
= 1
$$

and

$$
\begin{align}
\max 1-x\\
\text{s.t. }x\in[0,1],\ y=0
\end{align}
= 1.
$$

Thus,

$$
z^{nadir}=(1,1)^T.
$$

# What means solving a multiobjective optimization problem?

* **Find all Pareto optimal solutions**
  * As we learned in the previous lecture, there can be infinitely many Pareto optimal solutions for problems having real valued variables $\rightarrow$ extremely difficult and possible only in some simple special cases

* **Find a set of solutions that approximate the set of all Pareto optimal solutions**
  * How to evaluate the goodness of the approximation? (closeness, spread, ...)
  * The number of solutions required for a good approximation grows exponentially with the number of objectives!
  * Works well with two objectives and, in some cases, for three objectives

* **Find a solution/solutions that best satisfies the preferences of a decision maker**
  * Usually in practical problems one solution has to be finally selected for further analysis
  * Sometimes, more than one (but not that many) are needed $\rightarrow$ e.g. choose best design for different types of cars to be manufactured (small and economic sedan, spacious vagon, efficient sports model, etc.)
  * does not depend on the number of objectives

If you want to know more about the topic of this lecture, I urge you to read Professor Miettinen's book Nonlinear Multiobjective Optimization

## Scalarization
* One way to consider a multiobjective optimization problem is to convert it to a single objective subproblem whose solution is Pareto optimal for the original problem
* The subproblem is called a *scalarization* and it can be solved by using a suitable single objective optimization method
* By changing the values of the parameters in the scalarization, different (Pareto optimal) solutions can be computed

## Classification of methods

Methods for multiobjective optimization are often characterized by the involvement of the decision maker in the process.

The types of methods are
* **no preference methods**, where the decision maker does not play a role,
* **a priori methods**, where the decision maker gives his/her preference information at first and then the optimization method finds the best match to that preference information,
* **a posteriori methods**, where the optimization methods try to characterize all/find a good representation of the Pareto optimal solutions and the decision maker chooses the most preferred one of those,
* **interactive methods**, where the optimization method and the decision maker alternate in iterative search for the most preferred solution.

## Multiple Criteria Decision Making (MCDM)
* The related research field is called multiple criteria decision making
* More information in the website of the International Society on MCDM

##  Our example problem for this lecture

We study a hypothetical decision problem of buying a car, when you can choose to have a car with power between (denoted by $p$) 50 and 200 kW and average consumption (denoted by $c$) per 100 km between 3 and 10 l. However, in addition to the average consumption and power, you need to decide the volume of the cylinders (v), which may be between 1000 $cm^3$ and 4000 $cm^3$. Finally, the price of the car follows now a function

$$
\left(\sqrt{\frac{p-50}{50}}\\
+\left(\frac{p-50}{50}\right)^2+0.3(10-c)\\ +10^{-5}\left(v-\left(1000+3000\frac{p-50}{150}\right)\right)^2\right)10000\\+5000
$$

in euros. This problem can be formulated as a multiobjective optimization problem

$$
\begin{align}
\min \quad & \{c,-p,P\},\\
\text{s.t. }\quad
&50\leq p\leq 200\\
&3\leq c\leq 10\\
&1000\leq v\leq 4000,\\
\text{where }\quad&P = \left(\sqrt{\frac{p-50}{50}}+\left(\frac{p-50}{50}\right)^2+0.3(10-c)\right.\\
& \left.+ 10^{-5}\left(v-\left(1000+3000\frac{p-50}{150}\right)\right)^2\right)10000+5000
\end{align}
$$

```python
#Let us define a Python function which returns the value of this
import math
def car_problem(c,p,v):
#    import pdb; pdb.set_trace()
    return [#Objective function values
        c,-p,
        (math.sqrt((p-50.)/50.)+((p-50.)/50.)**2+
        0.3*(10.-c)+0.00001*(v-(1000.+3000.*(p-50.)/150.))**2)*10000.
        +5000.]
```

```python
print("Car with 3 l/100km consumption, 50kW and 1000cm^3 engine would cost "
      +str(car_problem(3,50,1000)[2])+"€")
print("Car with 3 l/100km consumption, 100kW and 2000cm^3 engine would cost "
      +str(car_problem(3,100,2000)[2])+"€")
print("Car with 3 l/100km consumption, 100kW and 1000cm^3 engine would cost "
      +str(car_problem(3,100,1000)[2])+"€")
```

    Car with 3 l/100km consumption, 50kW and 1000cm^3 engine would cost 26000.0€
    Car with 3 l/100km consumption, 100kW and 2000cm^3 engine would cost 46000.0€
    Car with 3 l/100km consumption, 100kW and 1000cm^3 engine would cost 146000.0€

## Normalization of the objectives

**In many of the methods, the normalization of the objectives is necessary.**

We can normalize the objectives using the nadir and ideal and setting the normalized objective as
$$ \tilde f_i = \frac{f_i-z_i^{ideal}}{z_i^{nadir}-z_i^{ideal}}$$

## Calculating the ideal

**Finding the ideal for problems is usually easy, if you can optimize the objective functions separately.**

For the car problem, ideal can be computed easily using the script:

```python
#Calculating the ideal
from scipy.optimize import minimize
import ad
def calc_ideal(f):
    ideal = [0]*3 #Because three objectives
    solutions = [] #list for storing the actual solutions, which give the ideal
    bounds = ((3,10),(50,200),(1000,4000)) #Bounds of the problem
    starting_point = [3,50,1000]
    for i in range(3):
        res=minimize(
            #Minimize each objective at the time
            lambda x: f(x[0],x[1],x[2])[i], starting_point, method='SLSQP'
            #Jacobian using automatic differentiation (note: SLSQP can estimate gradiants itself with some extra function evaluations)
            #,jac=ad.gh(lambda x: f(x[0],x[1],x[2])[i])[0]
            #bounds given above
            ,bounds = bounds
            ,options = {'disp':True, 'ftol': 1e-20, 'maxiter': 1000})
        solutions.append(f(res.x[0],res.x[1],res.x[2]))
        ideal[i]=res.fun
    return ideal,solutions
```

```python
ideal, solutions= calc_ideal(car_problem)
print ("ideal is "+str(ideal))
```

    Optimization terminated successfully    (Exit mode 0)
                Current function value: 3.0
                Iterations: 1
                Function evaluations: 4
                Gradient evaluations: 1
    Optimization terminated successfully    (Exit mode 0)
                Current function value: -200.0
                Iterations: 5
                Function evaluations: 20
                Gradient evaluations: 5
    Optimization terminated successfully    (Exit mode 0)
                Current function value: 5000.0
                Iterations: 6
                Function evaluations: 8
                Gradient evaluations: 2
    ideal is [3.0, -200.0, 5000.0]

## Pay-off table method

**Finding the nadir value is however, usually much harder.**

Usually, the nadir value is estimated using the so-called pay-off table method.

The pay-off table method does not guarantee to find the exact nadir for problems with more than two objectives.

The method is, however, a generally accepted way of approximating the nadir vector.

In the pay-off table method:
1. the objective values for attaining the individual minima are added in table
2. the nadir is estimated by each objectives maxima in the table.
3. the ideal values are located in the diagonal of the pay-off table

### $x^{(*,i)} =$ optimal solution for $f_i$

### The nadir for the car selection problem
The table now becomes by using the *solutions* that we returned while calculating the ideal

```python
for solution in solutions:
    print(solution)
```

    [3.0, -50.0, 26000.0]
    [3.0, -200.0, 1033320.5080756888]
    [10.0, -50.0, 5000.0]

Thus, the esimation of the nadir vector is
$$(10,-50,1033320.5080756888)$$

This is actually the real Nadir vector for this problem.

### Normalized car problem

```python
#Let us define a Python function which returns the value of this
import math
def car_problem_normalized(c,p,v):
    z_ideal = [3.0, -200.0, 5000]
    z_nadir = [10,-50,1033320.5080756888]
    z = car_problem(c,p,v)
    return [(zi-zideali)/(znadiri-zideali) for
            (zi,zideali,znadiri) in zip(z,z_ideal,z_nadir)]
```

the zip function in Python

```python
print("Normalized value of the car problem at (3,50,1000) is "
      +str(car_problem_normalized(3,50,1000)))
print("Normalized value of the car problem at (3,125,2500) is "
      +str(car_problem_normalized(3,125,2500)))
print("Normalized value of the car problem at (10,100,1000) is "
      +str(car_problem_normalized(10,100,1000)))
```

    Normalized value of the car problem at (3,50,1000) is [0.0, 1.0, 0.020421648537670038]
    Normalized value of the car problem at (3,125,2500) is [0.0, 0.5, 0.054212133547970276]
    Normalized value of the car problem at (10,100,1000) is [1.0, 0.6666666666666666, 0.11669513450097163]

**So, value 1 now indicates the worst value on the Pareto frontier and value 0 indicates the best values**

Let's set the ideal and nadir for later reference:

```python
z_ideal = [3.0, -200.0, 5000]
z_nadir = [10.,-50,1033320.5080756888]
```

**From now on, we will deal with the normalized problem, although, we write just $f$.** The aim of this is to simplify presentation.

Lecture 11, Methods for multiobjective optimization.ipynb


For Loop

Iterate over the elements of a sequence (such as a string, tuple, list or array) or other iterable object

- [Overview](#Overview)
- [Indentation](#Indentation)
- [Examples](#Examples)
    - [Summing](#Summing)
    - [Taylor series](#Taylor-series)
    - [Factorial](#Factorial)
    - [Fibonacci series](#Fibonacci-series)
- [Looping with indices](#Looping-with-indices)
- [Nested loops](#Nested-loops)
- [Debugging](#Debugging)
- [Glossary](#Glossary)
- [Exercises](#Exercises)

## Overview

Computers are often used to automate repetitive tasks since repeating identical or similar tasks without making errors is something that computers do well and people do poorly. In a computer program, repetition is also called iteration.

The `for` statement is used to iterate over (step through) the elements of a sequence (such as a string, tuple, list or array) or other iterable object. In other words the `for` statement can be used to repeat a block of code a predetermined number of times (once for each element in the sequence). The framework of the `for` statement is:

where `sequence` is some iterable object like a string, tuple, list, or array and `var` is a variable name that gets updated each time the loop repeats. The syntax of a `for` statement is similar to a function definition. It has a header that ends with a colon and an indented body. The body can contain any number of statements. This type of flow is called a loop because the third step loops back around to the top.

 Take Note

Note the double colon (`:`) at the end of the `for` statement and the indentation infront of the body. This tells *Python* where the header ends and which program statements (lines of code) belong to the body that will be repeated.

 More Information

You can get more information about the `for` statement by executing `help("for")` in a *Code Cell*

```python
help("for")
```

Consider the following example:

```python
print("Begin")
for val in [20, 30, 10, 0]:
    print("val =", val)
print("End")
```

A `for` statement is also called a loop because the flow of execution runs through the body and then loops back to the top. Each time through the loop, the next number in the list is assigned to the variable name `val`. The loop continues until it has gone through all numbers in the list.

This example could also have be written as:

```python
%load_ext nbtutor
```

```python
%%nbtutor -rf
print("Begin")
seq = [20, 30, 10, 0]
for val in seq:
    print("val =", val)
print("End")
```

Execute the *Code Cell* and step through the code execution line-by-line. Executing the *Code Cell* above does the following:
- Line 2: Creates the objects `20`, `30`, `10`, and `0` in memory as well as a `list` object with the name `seq`.
- Line 3: Steps through the objects in the list `seq` one at a time and assigns the name `val` to each object
    - The first time the loop executes `val` gets assigned to the first object in the list `seq`, which is `20`
    - The second time the loop executes `val` gets assigned to the second object in the list `seq`, which is `30`
    - The third time the loop executes `val` gets assigned to the third object in the list `seq`, which is `10`
    - Etc.
    - Once the loop has stepped through all objects in the list `seq` then the loop terminates
- Line 4: Gets repeated for each object in the list `seq`

## Indentation

Indentation means the amount of white space placed in front of your code. *Python* uses 4 spaces as 1 indentation and it is this indentation that "tells" *Python* which statements must get repeated as they belong inside the `for` statement.

Consider the following example:

```python
for var in range(3):
    print("I’m inside the for loop")
print("I’m outside the for loop")
```

From this you should note that line 2 is executed 3 times and after the loop has terminated only then is line 3 executed. You should also note that the variable `var` is not used inside the loop. The variable `var` will still get assigned to each value in the list as the `for` statement iterates through the list. Place a `print` statement after line 1 to print out the variable `var` and verify this.

 Take Note

Be very careful and aware of indentation when typing your programs, as this can sometimes lead to unexpected results and logic errors.

There is a big difference between the following two examples. Make sure you know the difference in the results of these two examples and why they are different !!!

```python
foo = 0
for var in range(5):
    print("foo =", foo)
    foo = foo + var
print("foo =", foo)
```

```python
foo = 0
for var in range(5):
    print("foo =", foo)
foo = foo + var
print("foo =", foo)
```

 More Information

Most editors (like *Jupyter Notebook*) will automatically add 4 spaces when you push the `tab` key. And `Shift + tab` will automatically remove 4 spaces. This can also be used if you have highlighted several lines of code. Try it !!

## Examples

Many of the examples that follow can be solved using lists and/or arrays alone without a `for` statement. I recommend, as a means of practice, trying to solve each of the following examples in as many different ways as you can.

### Summing

As a first example let us create a function that computes the sum of the first `N` integers:

$$
    \sum^{N}_{k=1} k = 1 + 2 + 3 + \dots + N
$$

```python
def sum_ints(N):
    summation = 0
    for term in range(1, N+1):
        summation = summation + term
    return summation
```

```python
print(sum_ints(100))
```

The `range` object is used to create the sequence of terms from `1` up to (and including) `N`. The `for` statement is used to step through each term in this sequence. The `summation` variable is updated each time trough the loop by taking the current `summation` value plus the `term` value.

### Taylor series

A Taylor series is an approximation to any function using a series of polynomial terms. In the limit of using an infinite number of terms, the approximation is exact. Although a Taylor series does not intuitively require a `for`  statement, it is useful to illustrate the `for` statement.

A feature of a Taylor series approximation is that each new term contributes less to the function approximation than the terms before. So, at some point the additional Taylor terms don’t contribute much to the overall function value and we might as well stop adding additional terms. Therefore, when we use a computer to compute a function
value using a Taylor series approximation, we will always use a finite number of terms.

As an example, the value of $\pi$ can be computed from the following Taylor series:

$$
    \pi = 4 \sum^{\infty}_{k=0}
    \frac{
        \left( -1 \right)^k
    }{
        2k + 1
    } = 4 \left[
        1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \frac{1}{9} - \dots
    \right]
$$

Suppose that we want to write a function that approximates the value of $\pi$ by using the above Taylor series and `N` number of terms:

```python
def taylor_pi(N):
    pi = 0
    for k in range(N):
        term = (-1)**k / (2*k + 1)
        pi = pi + term
    return 4 * pi
```

```python
import numpy as np

pi = taylor_pi(100)
print(pi)
print(np.pi)
```

Compared to the true value of $\pi = 3.14159\dots$, the 1st 100 terms of this Taylor series approximation is only accurate to 2 digits. Note that since we start counting at $k = 0$, we only need to repeat the loop up to $k = 99$ to add up the first 100 terms. Also note that we must initialise the name `pi` to zero outside the `for` statement. If we do not do this, *Python* will produce an error message upon executing line 5 for the first time: we’re instructing *Python* to add the value $(-1)^k/(2k+1)$ to the name `pi` (whose value is unknown). If we did not assign zero to the name `pi` outside the `for` statement, we cannot expect *Python* to perform the required computation.

If we want to compute $\pi$ more accurately, we have to sum more terms in this Taylor series. If we compute the sum of the first 1000 terms we get

```python
import numpy as np

pi = taylor_pi(1000)
print(pi)
print(np.pi)
```

Even the first 1000 terms in the Taylor series produces an approximate value for $\pi$ that is only accurate to 3 digits.

Before you decide that Taylor series approximations are of no use, consider the approximation to the exponential function:

$$
e^x = \sum^{\infty}_{k=0} \frac{x^k}{k!}
    = 1 + x + \frac{1}{2}x^2 + \frac{1}{6}x^3 + \dots
$$

Let us use this Taylor series approximation to compute the value of $e$ (i.e. compute $e^x$ for $x = 1$):

```python
import numpy as np

def taylor_exp(x, N):
    exp = 0
    for k in range(N):
        exp = exp + x**k / np.math.factorial(k)
    return exp
```

```python
import numpy as np

e = taylor_exp(1, 18)
print(e)
print(np.exp(1))
```

which is accurate to 15 decimals when only summing the first 18 terms. Here we have an example where a few terms in this Taylor series produces a very accurate approximation, whereas the 1000 term approximation to $\pi$ was still inaccurate.

### Factorial

Let us create a function that computes the factorial of an integer:

$$
N! = \prod_{k=1}^{N} k = 1 \times 2 \times 3 \times \dots \times N
$$

```python
def factorial(N):
    prod = 1
    for term in range(1, N+1):
        prod = prod * term
    return prod
```

```python
print(factorial(10))
```

Again the `range` object is used to create the sequence of terms from `1` up to (and including) `N` and the `for` statement is used to step through each term in this sequence. The `prod` variable is updated each time the loop execute by taking the current `prod` value times the `term` value.

Would this function work if we initialize `prod = 0`? What would the result be?

 More Information

Remember that you can use the `numpy.math.factorial(N)` function to verify the result of this example.

### Fibonacci series

A very famous series was proposed by the mathematician Fibonacci: The first two terms of the series are given by $L_0 = 0$ and $L_1 = 1$. Hereafter, each new term in the series is given by the sum of the previous two terms in the series:

$$
    L_k = L_{k−1} + L_{k−2} \qquad \text{for} \quad k = 2, 3, 4, \dots
$$

Using the above formula, the following sequence is generated: $0, 1, 1, 2, 3, 5, 8, 13, \dots$

Let us first do this problem step by step using *Python* as a calculator. Let us compute the fifth term ($k = 4$):

```python
L0 = 0
L1 = 1
L2 = L0 + L1
L3 = L1 + L2
L4 = L2 + L3
print(L4)
```

Let us pretend we are only interested in the fifth term ($k = 4$). The solution above requires five variables with five values. If we require the twentieth term ($k = 19$), we would have twenty variables using this approach.

In computing the next term we only need the previous two terms, which means we can get away with only using three variables regardless of how many terms we need to compute. Let us do that by shifting objects. The computed $k$ th term in the $k$ th iteration becomes the ($k − 1$) th term when we increment $k$ by 1. Similarly the ($k − 1$) th term becomes the ($k − 2$) th term. Let us rewrite the above code to use only three variables:

```python
previous2 = 0
previous1 = 1
new = previous1 + previous2  # k = 2

previous2 = previous1
previous1 = new
new = previous1 + previous2  # k = 3

previous2 = previous1
previous1 = new
new = previous1 + previous2  # k = 4

print(new)
```

Let us see how we would go about computing the $N$ th term ($k = N$) of the Fibonacci series. We know how many terms we want to compute and we have a pattern of what we would like to repeat. So, let us compute the $N$ th term using the `for` statement:

```python
def fibo(N):
    previous2 = 0
    previous1 = 1
    for k in range(2, N+1):
        new = previous1 + previous2
        # shift variables for next loop
        previous2 = previous1
        previous1 = new
    return new
```

```python
print(fibo(7))
```

It is important to note that when we shift the objects we do it in such a way that we don’t end up with all names bound to the same object. We therefore start with the `previous2` name in line 7, since its current value was used to compute the current $k$ th Fibonacci term and it is not used to compute the next Fibonacci term.
We therefore bind the name term `previous2` to the same object bound by `previous1` and similarly we bind the name `previous1` to the same object bound by `new` that we just computed.
Now we are ready to increment $k$ to compute the next term of the Fibonacci series. When $k$ is incremented, and the loop repeats, the name `new` will get bound to a new object resulting from `previous1 + previous2`.

Let us go one step further and collect the Fibonacci terms into a list:

```python
def fibo(N):
    previous2 = 0
    previous1 = 1
    store = [previous2, previous1]
    for k in range(2, N+1):
        new = previous1 + previous2
        store.append(new)
        # shift variables for next loop
        previous2 = previous1
        previous1 = new
    return store
```

```python
print(fibo(7))
```

If it were required of us to store the Fibonacci terms in a list, from the beginning, then this list object and negative indexing makes the idea of taking the previous two terms and adding them together to get the next term a lot easier to follow:

```python
def fibo(N):
    store = [0, 1]
    for k in range(2, N+1):
        new = store[-1] + store[-2]  # add last two elements in store
        store.append(new)
    return store
```

```python
print(fibo(7))
```

Try modifiy this last example to use only positive indexing instead of negative indexing. You may need a new variable or would you be able to use `k`.

## Looping with indices

Again consider the example of an object falling from a height above the ground.
The equations describing the motion of this object, assuming constant acceleration, are:

$$
\begin{align}
    a(t) &= g \: \text{(constant)} \\
    v(t) &= \int a(t) \: \mathrm{d}t  =  v_0 + gt \\
    s(t) &= \int v(t) \: \mathrm{d}t = s_0 + v_0t + 0.5 gt^2
\end{align}
$$

where $g$ is the gravitationaly acceleration $[m/s^2]$, $v_0$ the initial object velocity $[m/s]$, $s_0$ the initial object height $[m]$ above the ground, and $t$ is the time $[s]$.

The time it takes for this object to hit the ground ($t_e$) can be computed using:

$$ t_e = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} $$

where
$$
\begin{align}
    a &= 0.5g = 0.5 (-9.81) \\
    b &= v_0 \\
    c &= s_0
\end{align}
$$

Let us calculate $t_e$ for the following initial values:

- $s_0=100$ and $v_0 = 50$
- $s_0=200$ and $v_0 = 40$
- $s_0=300$ and $v_0 = 30$
- $s_0=400$ and $v_0 = 20$
- $s_0=500$ and $v_0 = 10$

```python
def end_time(s0, v0):
    a = -0.5 * 9.81
    b, c = v0, s0
    return (-b - (b**2 - 4*a*c)**0.5) / (2*a)
```

```python
#       0    1    2    3    4
s0 = [100, 200, 300, 400, 500]
v0 = [ 50,  40,  30,  20,  10]

for i in range(5):
    te = end_time(s0[i], v0[i])
    print(i, s0[i], v0[i], te)
```

You should note in this example we need to repeat the computation of $t_e$ 5 times, once for each pair of initial values. The `range` object is used to create the sequence of index numbers from `0` up to `4` in increments of `1` and the `for` statement is used to step through this sequence one index at a time. The loop starts with `i` equal to `0` and ends with `i` equal to `4`. The `s0` and `v0` lists are indexed with `i` so when `i` is `0`, for example, this indexing yields `100` and `50` respectively.

## Nested loops

Let us now calculate $t_e$ for the following initial values:

- $s_0=100$ and $v_0 = [10, 20, 30, 40]$
- $s_0=200$ and $v_0 = [10, 20, 30, 40]$
- $s_0=300$ and $v_0 = [10, 20, 30, 40]$

Notice that $v_0$ repeats for each $s_0$ value (4 $v_0$ values for each $s_0$ value) and in total there should be 12 solutions for $t_e$

```python
s0 = [100, 200, 300]
v0 = [10, 20, 30, 40]

for s in s0:
    print("Start Inner")
    for v in v0:
        te = end_time(s, v)
        print(s, v, te)
    print("End Inner")
```

You should note that the outer loop (lines 4 through 9) repeats 3 times (once for each value in `s0`), which means the inner loop (lines 6 through 8) repeats $4 \times 3 = 12$ times (once for each value in `v0` repeated for each value in `s0`).

 Take Note

Notice the indentation for this nested loop example above. Lines 5 to 9 require 4 spaces in front of the program statements to "tell" *Python* that they are inside the first (outer) `for` loop. Lines 7 and 8 require 4+4 spaces in front of the program statements to "tell" *Python* that they are inside the second (inner) `for` loop.

Make sure you understand the behaviour and output of the following three examples and why they are different!!

```python
a = 0.0
for i in range(10):
    a += 10 * 1
    for j in range(5):
        a /= 2
    a *= 10
print("a = ", a)
```

```python
a = 0.0
for i in range(10):
    a += 10 * 1
    for j in range(5):
        a /= 2
        a *= 10
print("a = ", a)
```

```python
a = 0.0
for i in range(10):
    a += 10 * 1
for j in range(5):
    a /= 2
a *= 10
print("a = ", a)
```

## Debugging

As you start writing bigger programs, you might find yourself spending more time debugging. More code means more chances to make an error and more places for bugs to hide.

One way to cut your debugging time is “debugging by bisection”. For example, if there are 100 lines in your program and you check them one at a time, it would take 100 steps.

Instead, try to break the problem in half. Look at the middle of the program, or near it, for an intermediate value you can check. Add a print statement (or something else that has a verifiable effect) and run the program.

If the mid-point check is incorrect, there must be a problem in the first half of the program. If it is correct, the problem is in the second half.

Every time you perform a check like this, you halve the number of lines you have to search. After six steps (which is fewer than 100), you would be down to one or two lines of code, at least in theory.

In practice it is not always clear what the “middle of the program” is and not always possible to check it. It doesn’t make sense to count lines and find the exact midpoint. Instead, think about places in the program where there might be errors and places where it is easy to put a check. Then choose a spot where you think the chances are about the same that the bug is before or after the check.

When you use indices to traverse the values in a sequence, it is tricky to get the beginning and end of the traversal right. Here is a function that is supposed to compare two words and return True if one of the words is the reverse of the other, but it contains one error:

```python
def is_reversed(one, two):
    check = []
    j = len(two)
    for i in range(len(one)):
        check.append(one[i] == two[j])
        j = j-1
    return all(check)
```

`i` and `j` are indices: `i` traverses `one` forward while `j` traverses `two` backward. If we find two numbers that match we append `True` else we append `False` to the list `check`. The `all` function is used to return `True` if all entries in `check` are `True` else it will return `False`.

If we test this function with the lists `[1, 2, 3, 4]` and `[4, 3, 2, 1]`, we expect the return value `True` but we get an `IndexError`:

```python
is_reversed([1, 2, 3, 4], [4, 3, 2, 1])
```

For debugging this kind of error, my first move is to print the values of the indices immediately before the line where the error appears.

```python
def is_reversed(one, two):
    check = []
    j = len(two)
    for i in range(len(one)):
        print(i, j)
        check.append(one[i] == two[j])
        j = j-1
    return all(check)
```

Now when I run the program again, I get more information:

```python
is_reversed([1, 2, 3, 4], [4, 3, 2, 1])
```

The first time through the loop, the value of `j` is 4, which is out of range for the list `two`. The index of the last number is `3`, so the initial value for `j` should be `len(two) - 1`.

If I fix that error and run the program again, I get:

```python
def is_reversed(one, two):
    check = []
    j = len(two) - 1
    for i in range(len(one)):
        print(i, j)
        check.append(one[i] == two[j])
        j = j-1
    return all(check)
```

```python
is_reversed([1, 2, 3, 4], [4, 3, 2, 1])
```

This time we get the right answer

## Glossary

**decrement**: An update that decreases the value of a variable.

**increment**: An update that increases the value of a variable (often by one).

**index**: An integer value used to select an item in a sequence, such as a number in a list. In *Python* indices start from 0.

**initialization**: An assignment that gives an initial value to a variable that will be updated.

**iteration**: Repeated execution of a set of statements using either a recursive function call or a loop.

**reassignment**: Assigning a new value to a variable that already exists.

**sequence**: An ordered collection of objects where each value is identified by an integer index.

**traverse**: To iterate through the items in a sequence, performing a similar operation on each.

**update**: An assignment where the new value of the variable depends on the old.

## Exercises

Many of the exercises that follow can be solved using lists and/or arrays alone without a `for` statement. Again I recommend, as a means of practice, trying to solve each of the following exercises in as many different ways as you can.

1) Write a function (called `add`) that takes two positive integers as input. This function must multiply the two positive integers by using the addition (`+`) operator. Use the multiplication (`*`) operator only to verify your answer. Recall that you can write

$$
    7\times4 =
    7 + 7 + 7 + 7 =
    4 + 4 + 4 + 4 + 4 + 4 + 4
$$

2) Write a function (called `die_throws`) that takes a positive integer `N` as input. This function must simulate the throwing of a die `N` times and must return a list of the `N` die values.

*Hint*: You can use `np.random.randint` to generate a random integer between 1 to 6.

3) An alternating series is a series where the terms alternate signs. Write a function that takes a positive integer `N` as input. This function must return the sum of the first `N` terms of the following series:

$$
\sum^{100}_{n=1} (-1)^{n+1} \frac{1}{n} = 1 - \frac{1}{2} + \frac{1}{3} - \frac{1}{4} + \dots -  \frac{1}{100}
$$

4) Find the pattern of the following series. Write a function that takes a positive integer `N` as input. This function must return the sum of the first `N` terms of the following series:

$$
1 + \frac{1}{2} - \frac{1}{4} + \frac{1}{8} - \frac{1}{16} + \frac{1}{32} - \dots
$$

5) The Basel problem was first posed by Pietro Mengoli in 1644 and was solved by Leonhard Euler in 1735 which brought Leonhard Euler immediate fame at the age of 28. Euler showed that the sequence

$$
1 + \frac{1}{4} + \frac{1}{9} + \frac{1}{16} + \frac{1}{25} + \frac{1}{36} + \dots
$$

converges to $\pi^2/6$. Write a function that takes a positive integer `N` as input. This function must return the sum of the first `N` terms of this series. Compare the accuracy to $\pi^2/6$ for different inputs of `N`.

6) A very famous series was proposed by the mathematician Fibonacci: The first two terms of the series are given by $L_0 = 0$ and $L_1 = 1$. Hereafter, each new term in the series is given by the sum of the previous two terms in the series i.e.

$$
L_k = L_{k-2} + L_{k-1} \qquad \text{for} \quad k = 2, 3, 4, \dots
$$

Using the above formula, the following sequence is generated: $0, 1, 1, 2, 3, 5, 8, 13, \dots$
Write a program that computes the first 20 terms of the Fibonacci sequence and displays it on the screen.

7) The sine function can be approximated by the following infinite series:

$$
\sin(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \dots
$$

Write a function (called `sine`) that takes `x` and a positive integer `N` as input. This function must return the sum of the first `N` terms of this series. Compare the accuracy of the output for different inputs of `N`.

8) The mathematician Srinivasa Ramanujan found an infinite series that can be used to generate a numerical approximation of $1/\pi$:

$$
\frac{1}{\pi} = \frac{2\sqrt{2}}{9801} \sum_{k=0}^{\infty} \frac{\left(4k\right)!\left(1103 + 26390k\right)}{\left(k!\right)^4 396^{4k}}
$$

Write a function (called `estimate_pi`) that takes a positive integer `N` as input. This function must return the sum of the first `N` terms of this series. Compare the accuracy of the output for different inputs of `N`.

9) Write a function (called `cum_sum`) that takes a list (of positive numbers) as input. This function must return the cumulative sum of values from the input list. For example if `[1, 2, 3, 4, 5]` is given as input, the `cum_sum` function should return `[1, 3, 6, 10, 15]`.

10) Write a function (called `sum_pairs`) that takes a list (of positive numbers) as input. This function should return the sum of consecutive pairs of numbers from the input list. For example if `[1, 2, 3, 4, 5]` is give as input, this `sum_pairs` function should return `[3, 5, 7, 9]`.

11) The wind chill factor (WCF) indicates the perceived air temperature to exposed skin and is given by:

$$
WCF = 13.12 + 0.6215 T_a - 11.37 v^{0.16} + 0.3965 T_a v^{0.16}
$$

where $T_a$ the air temperature in degrees Celcius and $v$ the air speed in $km/h$.

Write a program that displays a collection of WCF’s using nested for loop statements. The temperature ($T_a$) must range from `-20` to `55` degrees Celcius in steps of `5` and wind speed ($v$) must range from `0` to `100` $km/h$ in increments of `10`.

```python

```


# Predicting water solubility - Part II?

> Feature selection

- toc: true
- badges: true
- comments: true
- categories: [fastpages, jupyter]
- image: images/fruit.jpg

# Requirements

 - rdkit >= 2020.09.1
 - pandas >= 1.1.3
 - seaborn
 - matplotlib
 - fastcore (!conda install fastcore)

In this tutorial we'll use a dataset compiled by [Sorkun et al (2019)](https://www.nature.com/articles/s41597-019-0151-1) from multiple projects to predict water solubility. You can download the original dataset from [here](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OVHAW8).

> ```At the end of the notebook I added a class to process the original dataset in order to remove salts, mixtures, neutralize charges and generate canonical SMILES. I highly recommend checking each structure before modeling. ```

In this notebook we'll cover three main topics:

   - **Featurization of molecules**
   - **What's the correlation between features?**
   - **Feature selection methods**

# Background

Drug solubility is a critical factor in drug development. If a drug is not soluble enough or doesn't dissolve readily its intestinal absoption will be compromised, leading to low concentration in the blood circulation and reduced (or none) biological activity.

Crystal formation of low soluble drugs may also lead to toxicity. In practical terms, poor solubility is one of the factors that lead to fail in drug discovery projects. Therefore, medicinal chemists work hard to design molecules tha have the intended bioactivity and that can display the desired effect in vivo.

Despite being conceptually easy to understand solubility, its estimation isn't easy. That's because the intrinsic solubility of a molecule depends on many factors, including its size, shape, the ability to make intermolecular interactions and crystal packing.

# Import modules

```python
#collapse
%reload_ext autoreload
%autoreload 2
%matplotlib inline
from IPython.display import Image
import matplotlib.pyplot as plt
from matplotlib.ticker import FormatStrFormatter
import seaborn as sns
import pandas as pd
import numpy as np

from sklearn.preprocessing import Normalizer, normalize, RobustScaler
from sklearn.ensemble import RandomForestRegressor
from xgboost import XGBRegressor
from sklearn.metrics import make_scorer, mean_squared_error

from sklearn.feature_selection import mutual_info_regression, RFECV,SelectFromModel,RFE,VarianceThreshold
from functools import partial
from pathlib import Path
from joblib import load, dump

from scipy import stats
from scipy.stats import norm
from statsmodels.graphics.gofplots import qqplot
from rdkit.Chem import Draw
from rdkit.Chem import MolFromSmiles, MolToSmiles

from descriptastorus.descriptors.DescriptorGenerator import MakeGenerator

```

    WARNING:root:No normalization for BCUT2D_MWHI
    WARNING:root:No normalization for BCUT2D_MWLOW
    WARNING:root:No normalization for BCUT2D_CHGHI
    WARNING:root:No normalization for BCUT2D_CHGLO
    WARNING:root:No normalization for BCUT2D_LOGPHI
    WARNING:root:No normalization for BCUT2D_LOGPLOW
    WARNING:root:No normalization for BCUT2D_MRHI
    WARNING:root:No normalization for BCUT2D_MRLOW

```python
np.random.seed(5)
```

```python
sns.set(rc={'figure.figsize': (16, 16)})
sns.set_style('whitegrid')
sns.set_context('paper',font_scale=1.5)
```

# Load Data

Let's load the dataset without outliers from the last [post](https://marcossantanaioc.github.io/fiocruzcheminformatics/jupyter/2021/05/31/third.html).

```python
data = pd.read_csv('../_data/water_solubility_nooutliers.csv')
```

```python
#collapse_output
data.head()
```

      ID
      Name
      InChI
      InChIKey
      SMILES
      Solubility
      SD
      Ocurrences
      Group
      MolWt
      ...
      NumAromaticRings
      NumSaturatedRings
      NumAliphaticRings
      RingCount
      TPSA
      LabuteASA
      BalabanJ
      BertzCT
      processed_smiles
      class

      0
      A-3
      N,N,N-trimethyloctadecan-1-aminium bromide
      InChI=1S/C21H46N.BrH/c1-5-6-7-8-9-10-11-12-13-...
      SZEMGTQCPRNXEG-UHFFFAOYSA-M
      [Br-].CCCCCCCCCCCCCCCCCC[N+](C)(C)C
      -3.616127
      0.0
      1
      G1
      392.510
      ...
      0.0
      0.0
      0.0
      0.0
      0.00
      158.520601
      0.000000
      210.377334
      CCCCCCCCCCCCCCCCCC[N+](C)(C)C
      slightly soluble

      1
      A-4
      Benzo[cd]indol-2(1H)-one
      InChI=1S/C11H7NO/c13-11-8-5-1-3-7-4-2-6-9(12-1...
      GPYLCFQEKPUWLD-UHFFFAOYSA-N
      O=C1Nc2cccc3cccc1c23
      -3.254767
      0.0
      1
      G1
      169.183
      ...
      2.0
      0.0
      1.0
      3.0
      29.10
      75.183563
      2.582996
      511.229248
      O=C1Nc2cccc3cccc1c23
      slightly soluble

      2
      A-5
      4-chlorobenzaldehyde
      InChI=1S/C7H5ClO/c8-7-3-1-6(5-9)2-4-7/h1-5H
      AVPYQKSLYISFPO-UHFFFAOYSA-N
      Clc1ccc(C=O)cc1
      -2.177078
      0.0
      1
      G1
      140.569
      ...
      1.0
      0.0
      0.0
      1.0
      17.07
      58.261134
      3.009782
      202.661065
      O=Cc1ccc(Cl)cc1
      slightly soluble

      3
      A-10
      vinyltoluene
      InChI=1S/C9H10/c1-3-9-6-4-5-8(2)7-9/h3-7H,1H2,2H3
      JZHGRUMIRATHIU-UHFFFAOYSA-N
      Cc1cccc(C=C)c1
      -3.123150
      0.0
      1
      G1
      118.179
      ...
      1.0
      0.0
      0.0
      1.0
      0.00
      55.836626
      3.070761
      211.033225
      C=Cc1cccc(C)c1
      slightly soluble

      4
      A-11
      3-(3-ethylcyclopentyl)propanoic acid
      InChI=1S/C10H18O2/c1-2-8-3-4-9(7-8)5-6-10(11)1...
      WVRFSLWCFASCIS-UHFFFAOYSA-N
      CCC1CCC(CCC(O)=O)C1
      -3.286116
      0.0
      1
      G1
      170.252
      ...
      0.0
      1.0
      1.0
      1.0
      37.30
      73.973655
      2.145839
      153.917569
      CCC1CCC(CCC(=O)O)C1
      slightly soluble

5 rows × 28 columns

# Featurization

In this dataset 17 features were already calculated for each molecule. These features include a range of physichochemical properties (e.g. MolWt, MolLogP and MolMR), atomic counts (e.g. HeavyAtomCount, NumHDonors and Acceptors) and more abstract topological descriptors (e.g. BalabanJ, BertzCT).

```python
descriptors = ['MolWt', 'MolLogP', 'MolMR', 'HeavyAtomCount',
       'NumHAcceptors', 'NumHDonors', 'NumHeteroatoms', 'NumRotatableBonds',
       'NumValenceElectrons', 'NumAromaticRings', 'NumSaturatedRings',
       'NumAliphaticRings', 'RingCount', 'TPSA', 'LabuteASA', 'BalabanJ',
       'BertzCT']

var =  ['Solubility']+ descriptors
```

```python
print(len(descriptors))
```

    17

We can check the correlation between each feature and solubility using seaborn's regplot:

```python
nr_rows = 6
nr_cols = 3
target = 'Solubility'
```

```python
#hide_input
fig, axs = plt.subplots(nr_rows, nr_cols, figsize=(nr_cols*3.5,nr_rows*3))

for r in range(0,nr_rows):

    for c in range(0,nr_cols):
        i = r*nr_cols+c

        if i < len(descriptors):

            sns.regplot(x=data[descriptors[i]], y=data[target], ax = axs[r][c])

            stp = stats.pearsonr(data[descriptors[i]], data[target])

            str_title = "r = " + "{0:.2f}".format(stp[0]) + "      " "p = " + "{0:.2f}".format(stp[1])
            axs[r][c].set_title(str_title,fontsize=11)
fig.delaxes(axs[-1,-1]) #The indexing is zero-based here
plt.tight_layout()
plt.show()
```

Most features have a negative correlation with solubility. In fact, the available features reflect more or less the size/complexity of the molecule (e.g. MolWt, RingCount, MolMR) and it's capacity to make inter and intramolecular interactions (e.g., NumValenceElectrons, NumHAcceptors, NumHDonors, MolLogP), which are known to influence water solubility. For instance, bigger molecules tend to be less soluble in water (high MolMW and HeavyAtomCount); the same applies for very liphophilic (high MolLogP) molecules. On the other hand, more polar molecules, capable of making hydrogen bonds with water, are usually more soluble.

Based on this preliminary analysis we can start thinking which features to include when training a regression model. For instance, molecular weight and liphophilicity are the features most correlated with the target variable. These features have a strong influence in solubility as demonstrated experimentally and by theorical calculations. As the molecular size and liphophilicity grows, its much harder to dissolve a molecule in water because the solute needs to disrupt a great number interactions within solvent molecules and force its way into bulk water, which demands a great amount of energy. The [influence of lipophilicity](https://pubs.acs.org/doi/10.1021/jo01265a071) is so well known that its part of many predictive models, such as the famous [ESOL (estimated solubility)](https://pubs.acs.org/doi/10.1021/ci034243x), which estimates water solubility based only on the molecular structure.

```Back in the day (2003!) the author of ESOL didn't even use the robust machine learning algorithms we have today to derive the following equation:```

$$logS_{w} = 0.16 - 0.63logP - 0.0062MolWt + 0.066Rot - -0.74AP$$

where $logS_{w}$ is the log of the water solubility, $logP$ is the lipophilicity, $MolWt$ is the molecular weight, $Rot$ is the number of rotatable bonds (e.g. single bonds) and $AP$ is the proportion of heavy atoms that are part of aromatic rings.

# Understanding the features

Let's take a look at what those features mean.

## 1) LogP

The logP is the partition coefficient given by:

$$logP = \frac{C_{n-octanol}}{C_{water}}$$

where $C_{n-octanol}$ and $C_{water}$ are the solute concentration in n-octanol and water, respectively. Thus, higher logP means that a smaller concentration of the molecule is available in the water phase of a system, which makes it more lipophilic or hydrophobic.

## 2) Molecular weight

Molecular weight (MolWt) is simply the total mass of a [mole](https://en.wikipedia.org/wiki/Mole_(unit)) of a compound. The MolWt can be calculated by the summing the mass contribution of each atom in a molecule. For example, the molar weight of water is:

$$H_{2}O = 2H * (1.01 g/mol) + 1O * (16 g/mol) = 18.01 g/mol$$

## 3) Molar refractivity

Molar refractivity (MolMR) is the [refractivity](https://en.wikipedia.org/wiki/Molar_refractivity) of a mole of a substance and is a measure of polarizability. It's given by the formula:

$$MolMR = \frac{n^{2}-1}{n^{2}+2}*\frac{MolWt}{d}$$

where $MolWt$ is the molar weight, $d$ is the density and $n$ is the refractivity index. The right hand side of the equation ($\frac{MolWt}{d}$) is the volume. Thus, the molar refractivity encodes the molecular volume and is a way to estimate the steric bulk.

## 4) Topological polar surface area (TPSA) and LabuteASA

 - **TPSA**: The polar surface area is the sum of the surfaces of polar atoms in a molecule, such as oxygen and nitrogen, and sometimes sulphur and its hydrogen. The total polar surface area is a useful feature to encode both the polarity and the hydrogen bonding capacity of a molecule.

 - [**LabuteASA**](https://www.sciencedirect.com/science/article/abs/pii/S1093326300000681): the accessible surface area (ASA) is the area of a molecule that is accessible to the solvent (e.g. water). The ASA is calculated by summing the surface area of each atom in a molecule. We can also think about ASA as the area a water molecule can touch as we roll it on the surface of the solute.

**Figure 1**. Representation of the polar surface area, showing the contributions of each polar atom in the molecule.

**Figure 2**. Representation of the polar surface area, showing the contributions of each polar atom in the molecule.

## 5) Atomic counts

Atomics counts represent how many times an atom or group appears in a molecule. Its possible to count specific substructures (e.g. number of aromatic rings), atoms and atoms types, hydrogen bond donors and acceptors etc.

## 6) Topological descriptors

Topological descriptors in this dataset include [BalabanJ](http://publications.iupac.org/pac/55/2/0199/index.html) and BertzCT and represent more abstract definition of a molecule. For the BalabanJ, the feature value is calculated from the distance matrix, a n x n matrix where n is the number of atoms and the entries represent the distances between each atom in the molecular graph. The BalabanJ descriptor can thus capture topological information such as branching, distance between substructures and the molecular size.
The BertzCT descriptor encodes the molecular complexity by taking into account bond connectivity and atom types.

**Now let's get to work!**

# In depth analysis of feature correlation

First, let's calculate the Pearson's R coefficient for each pair of features and visualize the results using a heatmap.

The largest diagonal corresponds to self correlation for each feature (e.g. MolWt x MolWt), and is always 1. The most interesting part are the other values, showing the correlation between each pair of features and target variable.

As we can see from the heatmap, some features have a high negative linear correlation with solubility, such as MolWt (-0.61), LabuteASA (-0.62), MolLogP (-0.78) and MolMR (-0.64), HeavyAtomCount (-0.57).

We can also investigate the correlations between features. For the BalabanJ descriptor, the highest correlation was with RingCount (-0.64). For BertzCT, the highest correlations were with features that encode the molecular complexity/size, such as molecular weight, molar refractivity and number of heavy atoms, which is consistent with its definition.

```python
corr = data[var].corr('pearson')
```

```python
#collapse_output
sns.heatmap(corr, annot=True)
```

We can also see that the number of hydrogen bond donors have a weak positive correlation with solubility, while the number of H-bond acceptors have a very weak negative correlation. This is interesting because it shows us that increasing the H-bond potential might not be the best strategy to increase solubility.

In general, [increasing H-bond donors and acceptors also increases solubility](https://www.sciencedirect.com/science/article/abs/pii/S0022354915508666) because it alters the capacity to make H-bonds with water molecules. However, if the molecule can both donate and accept H-bonds, a decrease in solubility might happen due to intermolecular interactions between molecules of the same kind, leading to reduced ability to be solvated by water molecules.
The intuition is that a molecule needs to make better interactions with water than with others of its kind, otherwise it will be poorly solvated.

To summarize this initial analysis, the heatmap give us an idea of what types of features to include in a model. If we were to cherry pick, features like **MolLogP**, **MolMR** and **LabuteASA** would probably be selected. In addition, we must avoid highly correlated features (i.e. that encode the same kind of information) because that would only make our model more complex without any benefit in performance.

But no need to rush! We'll systematically check each feature for better correlations with the target variable and with each other. Remember that our goal is to develop a model with high predictive power, and to do that we need to select the right set of features.

# Training a baseline model

Before we dive into feature selection, let's try a baseline model and see how it performs when training using all features. For this task, we will use the Random Forest algorithm. First, split the data into training and testing sets.

```python
from sklearn.model_selection import train_test_split
```

```python
trainset, testset = train_test_split(data, test_size = 0.30, random_state=42)
```

Since our features are in different scales, let's normalize them to have zero mean and unit std. This is an important step so that one feature with large std doesn't dominate the others during training, which could lead to overfitting.

> Random forest is robust to the scale of features so this step isn't necessary in this case. We will normalize anyway because it's best practice, especially in models that depends on coefficients and intercepts (e.g. linear regression).

```python
xtrain, xtest = trainset[descriptors].values, testset[descriptors].values
ytrain, ytest = trainset['Solubility'].values, testset['Solubility'].values
```

We will do the preprocessing in a more compact way by using sklearn pipeline, which will contain the preprocessing steps and a fit call to the training algorithm.

```python
from sklearn.pipeline import Pipeline
```

```python
pipe = Pipeline(steps=[
    ('scaler',RobustScaler()),
    ('estimator', RandomForestRegressor(n_estimators=2000, n_jobs=-1))
                      ]
               )
```

We will validate our model using 5-fold cross-validation.

```python
from sklearn.model_selection import cross_val_score
```

```python
metric = make_scorer(mean_squared_error, squared=False) # RMSE
cross_score = cross_val_score(estimator=pipe, X=xtrain,scoring=metric, y=ytrain, cv=5, n_jobs=-1)
```

```python
print(f'Mean 5-fold RMSE = {cross_score.mean():.4f}')
```

    Mean 5-fold RMSE = 0.9983

Now the test set:

```python
pipe.fit(xtrain,ytrain)
```

    Pipeline(steps=[('scaler', RobustScaler()),
                    ('estimator',
                     RandomForestRegressor(n_estimators=2000, n_jobs=-1))])

```python
def get_preds(estimator, x):
    preds = estimator.predict(x)
    return preds
```

```python
preds = pipe.predict(xtest)
```

```python
print(f'RMSE test set = {mean_squared_error(preds, ytest, squared=False):.3f}')
```

    RMSE test set = 0.963

Not bad! This is our baseline. Let's see if we can beat it or simplify our data a little bit without compromising performance.

### The perils of non-informative features

What happens if we only have non-informative features in the dataset? What performance could we expect? Intuitively, one would say RMSE >> 0. Let's test that by randomizing the rows of the target variable vector, which will make every feature **non-informative**.

```python
yrandom = np.random.choice(ytrain,ytrain.shape[0])
```

```python
yrandom.shape,ytrain.shape
```

    ((5967,), (5967,))

```python
yrandom, ytrain
```

    (array([-1.7       ,  1.00830886, -4.45042772, ..., -2.56313048,
            -1.063     , -2.7813    ]),
     array([-5.05611784, -1.02322129, -8.7546501 , ..., -0.7956    ,
            -1.11610783, -4.561     ]))

```python
pipe.fit(xtrain,yrandom)
preds_random = pipe.predict(xtest)
```

```python
print(f'RMSE (randomized target) test set = {mean_squared_error(preds_random, ytest, squared=False)}')
```

    RMSE (randomized target) test set = 2.3878056084158237

As you can see, the RMSE on the test set does increase. In a data set with dozens or even hundreds of features, which is relatively common in QSAR applications, we would expect much higher errors. That's why it's so important to analyze your features carefully in order to remove anything that could hamper performance, including missing or constant values and highly correlated features.

# Feature selection

In our solubility dataset we have 17 features but we could have more. There are hundreds of molecular descriptors available in literature. So, how does one select the best set of features to include in a model? Let's investigate some feature selection methods!

## 1) Filter methods

Filter methods are the easiest to implement because they are model-agnostic, which means we don't need any fancy learning algorithm. This class of methods consists of statistical approaches that use the distribution of the dataset to remove features that don't have much information or correlation with the target variable. Since we don't need a model, filter methods are very fast and can give an initial guess of what types of features to keep.

Despite being fast, filter methods also come with some dangerous drawbacks. Since most methods are **univariate**, important correlations between features may be missed; or worse redudant features could be selected. In addition, most metrics (e.g. Pearson's correlation coefficient) used to select the features are subjective and not always correlate with the metric that will be used to access performance.

> While filter methods tend to be simple and fast, there is a subjective nature to the procedure. Most scoring methods have no obvious cut point to declare which predictors are important enough to go into the model. Even in the case of statistical hypothesis tests, the user must still select the confidence level to apply to the results. In practice, finding an appropriate value for the confidence value α may require several evaluations until acceptable performance is achieved

> ```Applied Predictive Modeling, p.499.```

### 1.1) Basic approaches

Basic approaches are purely statistical and consits of removing constant or quasi-constant features from the dataset.

#### 1.1.1) Remove constant features using variance threshold

The first method we will use consists of removing constant or quasi-constant features. In this situation, all samples have the same value or almost the same. The intuition to remove this kind of feature is that it doesn't add any useful information to training because most values are not present or are repeated; so we can't differentiate between data points.

```python
from sklearn.feature_selection import VarianceThreshold, mutual_info_regression
```

```python
selector = VarianceThreshold()
```

```python
scaler = RobustScaler()
xnorm = scaler.fit(xtrain).transform(xtrain)

```

```python
reducer = VarianceThreshold()
reducer.fit(xnorm)
```

    VarianceThreshold()

We can get the final number of features by using the ```get_support``` method from a fitted reducer.

```python
sum(reducer.get_support())
```

    17

Well, it seems we don't have any feature with 0 variance. That's a good thing! But what about features with less than 1% variance?

```python
reducer = VarianceThreshold(threshold=0.01)
reducer.fit(xnorm)
```

    VarianceThreshold(threshold=0.01)

```python
sum(reducer.get_support())
```

    17

Again no feature was removed. That's a good start. You can play around with different threshold values depending on the dataset.

Let's try more robust methods!

### 1.2) Univariate selection methods

#### Mutual information

[Mutual information](https://thuijskens.github.io/2017/10/07/feature-selection/) is a *linear* approach that quantifies the amount of information obtained about one random variable using another random variable. The mutual information is given by:

$$
\begin{align}
I(X; Y) = \int_X \int_Y p(x, y) \log \frac{p(x, y)}{p(x) p(y)} dx dy
\end{align}
$$

If the joint distribution $p(x,y)$ of variables $X$ and $Y$ equals the individual probabilities $p(x)$ and $p(y)$, the variables are considered independent and the integral is 0. Therefore, our goal is to find variables that are somehow correlated with the dependent variable logS that with maximum information content.

```python
from sklearn.feature_selection import mutual_info_regression
```

```python
importances = mutual_info_regression(xtrain, ytrain)
```

```python
df_importances = pd.DataFrame(importances,index=descriptors)
```

```python
df_importances.sort_values(0,ascending=False).head(5)
```

      0

      MolLogP
      0.618195

      MolWt
      0.451355

      MolMR
      0.444390

      LabuteASA
      0.423057

      NumValenceElectrons
      0.366864

It seems that the top-5 most important features makes sense; as mentioned before logP and the molecular weight are important parameters to estimate water solubility. LabuteASA and NumValenceElectrons are also good estimates of the molecular shape and complexity. In fact, the top-5 features are what we expect based on the Pearson's R heatmap! That's very good, we can see the feature selection methods are showing some consistence and now we are bit more confident that we should include logP and some feature that encode structural information about the molecules.

For the bottom 5 features we can see that the are mostly related to atomic or fragment counts, which does tells us something about the structure but are not determinant for solubility per se.

```python
df_importances.sort_values(0,ascending=False).tail(5)
```

      0

      NumRotatableBonds
      0.080012

      NumHeteroatoms
      0.070472

      NumHDonors
      0.059317

      NumAliphaticRings
      0.024100

      NumSaturatedRings
      0.019100

We can also use the mutual information as metric in the sklearn [SelectKBest](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest) class.

```python
from sklearn.feature_selection import SelectKBest
```

```python
kbest = SelectKBest(score_func=mutual_info_regression)
xkbest = kbest.fit_transform(xtrain, ytrain)
```

The ```get_support``` method returns a bool array that we can use to index the list of descriptors.

```python
np.array(descriptors)[kbest.get_support()]
```

    array(['MolWt', 'MolLogP', 'MolMR', 'HeavyAtomCount',
           'NumValenceElectrons', 'NumAromaticRings', 'RingCount',
           'LabuteASA', 'BalabanJ', 'BertzCT'], dtype='<U19')

In general, the most used approach is to use SelectKBest and select a scoring method, which will depend on the kind of variable we have. For continuous variables in regression problems, the [f_regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression) and [mutual_info_regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html#sklearn.feature_selection.mutual_info_regression) scoring functions are available on scikit-learn. For categorical variables in a classification problem, one can use Fischer [chi-squared](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2), [mutual information for classification](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif) (mutual_info_classif) and the [ANOVA F-value (f_classif)](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif)

## 2) Wrapper methods

Wrapper methods consists of using models to decide if a feature should be added or removed. The main idea is to find a subset of features that maximize the performance. In this class of methods we have forward/backward selection and recursive feature elimination (RFE). Another way to understand wrapper methods is to think of them as search algorithms, with the features representing the search space and model performance as the target metric.

> Wrapper methods evaluate multiple models using procedures that add and/or remove predictors to find the optimal combination that maximizes model performance. In essence, wrapper methods are search algorithms that treat the predictors as the inputs and utilize model performance as the output to be optimized.

> ```Applied Predictive Modeling, p.499.```

### 2.1) Forward selection

In forward selection we start with 0 features and then add one at a time to evaluate the performance. If the performance on iteration $i+1$ is better than the previous $i^{th}$ iteration, we keep the new feature. The algorithm stops when the addition of new features does not lead to an increase in performance.

Let's implement a naive approach to forward selection.

```python
#hide_input
from sklearn.base import clone
from tqdm.notebook import tqdm

class ForwardSelection():
    def __init__(self, estimator, X, y, scoring, test_size):
        self.estimator = clone(estimator)

        self.X = X
        self.dim = X.shape[1]
        self.y = y
        self.scoring = scoring
        self.test_size = test_size

    def fit_predict(self):

        index = tuple(range(self.dim))

        self.subset = []

        Xtrain, Xtest, Ytrain, Ytest = train_test_split(self.X,self.y,test_size=self.test_size)

        self.scores = [self._calc_score(self._no_features(Xtrain), Ytrain, Xtest, Ytest)]

        for i in index:

            Xtransform, Xtest_transform = self.transform(Xtrain,i), self.transform(Xtest,i)

            score = self._calc_score(Xtransform, Ytrain, Xtest_transform, Ytest)

            if score < np.min(self.scores):
                print(f'Found import feature {descriptors[i]} with new min RMSE of {score:.3f}')
                self.subset.append(i)
                self.scores.append(score)

    def transform(self, X, i):
        return X[:, self.subset + [i]].reshape(-1, len(self.subset + [i]))

    def _calc_score(self, Xtrain, ytrain, Xtest, y_test):

        self.estimator.fit(Xtrain, ytrain)
        preds = self.estimator.predict(Xtest)

        score = self.scoring(preds, y_test)

        return score

    def get_support(self):

        return self.subset

    def _no_features(self, x):
        return np.zeros(x.shape)
```

```python
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.svm import SVR
from sklearn.neighbors import KNeighborsRegressor
```

```python
pipe_ffs = Pipeline(steps=[('scaler',RobustScaler()), ('estimator', LinearRegression())])
```

```python
ffs = ForwardSelection(pipe_ffs, xtrain, ytrain,
                       scoring = partial(mean_squared_error, squared=False),
                       test_size=0.25)
```

```python
ffs.fit_predict()
```

    Found import feature MolWt with new min RMSE of 1.756
    Found import feature MolLogP with new min RMSE of 1.426
    Found import feature MolMR with new min RMSE of 1.411
    Found import feature HeavyAtomCount with new min RMSE of 1.396
    Found import feature NumHeteroatoms with new min RMSE of 1.393
    Found import feature NumRotatableBonds with new min RMSE of 1.340
    Found import feature TPSA with new min RMSE of 1.339
    Found import feature LabuteASA with new min RMSE of 1.338
    Found import feature BalabanJ with new min RMSE of 1.338

```python
selected_features = np.array(descriptors)[ffs.get_support()]
print(f'Number of selected features = {len(selected_features)}')
```

    Number of selected features = 9

Our naive forward selector found 13 important features. If we go back to our heatmap, we can see that the selected features do have some correlation with solubility! Furthermore, it's basically the same features selected by the mutual information metric in the previous section.

Now let's try the faster and reliable scikit-learn implementation of forward selection, the [```SequentialFeatureSelector```](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html) class.

We need to define a number of features to select so let's use 10 and compare the results with the approches so far.

```python
from sklearn.feature_selection import SequentialFeatureSelector
```

```python
ssf = SequentialFeatureSelector(estimator=pipe_ffs,n_features_to_select=12,cv=5, direction='forward',scoring='neg_mean_squared_error')
```

>It seems SequentialFeatureSelector tries to **maximize** the scoring function. If we use the ```make_scorer(mean_squared_error)``` the selector will output features that increase the RMSE. Thus, we'll use the negative mean squared error, neg_mean_squared_error.

```python
ssf.fit(xtrain, ytrain)
```

    SequentialFeatureSelector(estimator=Pipeline(steps=[('scaler', RobustScaler()),
                                                        ('estimator',
                                                         LinearRegression())]),
                              n_features_to_select=12,
                              scoring='neg_mean_squared_error')

```python
selected_features_ssf = np.array(descriptors)[ssf.get_support()]
```

```python
selected_features_ssf
```

    array(['MolLogP', 'MolMR', 'HeavyAtomCount', 'NumHAcceptors',
           'NumHeteroatoms', 'NumRotatableBonds', 'NumValenceElectrons',
           'NumSaturatedRings', 'RingCount', 'LabuteASA', 'BalabanJ',
           'BertzCT'], dtype='<U19')

There we go! Our naive approache gives similar results to sklearn implementation. As always, MolLogP was selected as an informative feature.

### 2.2) Backward selection

We can also do backward selection with [```SequentialFeatureSelector```](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html). In this method, we start with the whole set of features and remove at each step the ones that do not improve the model.

```python
ssf_backward = SequentialFeatureSelector(estimator=pipe_ffs,n_features_to_select=0.5,cv=5, direction='backward',scoring='neg_mean_squared_error')
```

```python
ssf_backward.fit(xtrain, ytrain)
```

    SequentialFeatureSelector(direction='backward',
                              estimator=Pipeline(steps=[('scaler', RobustScaler()),
                                                        ('estimator',
                                                         LinearRegression())]),
                              n_features_to_select=0.5,
                              scoring='neg_mean_squared_error')

```python
selected_features_ssf = np.array(descriptors)[ssf_backward.get_support()]
print(selected_features_ssf)
```

    ['MolLogP' 'HeavyAtomCount' 'NumHeteroatoms' 'NumRotatableBonds'
     'NumValenceElectrons' 'LabuteASA' 'BalabanJ' 'BertzCT']

### 2.3) Recursive feature elimination

Recursive feature elimination is another backward selection method but with the benefit of not fitting models using the same data multiple times. The first step consists of training the model on the full set of features. Then, the performance is calculated and the features are ranked in order of importance. The method iteratively removes features that are not important and rebuilds the model to estimate its performance and ranking of the remaining features. The process stops when a predefined number of features is reached.

In scikit-learn, we can use ```RFE``` as is or coupled with cross-validation (```RFECV```), in which case the mean cross-validated scored is returned.

```python
from sklearn.feature_selection import RFE, RFECV
```

```python
xnorm = scaler.fit_transform(xtrain)
rfe = RFECV(estimator=LinearRegression(), cv=5, step=1)
```

```python
rfe.fit(xnorm, ytrain)
```

    RFECV(cv=5, estimator=LinearRegression())

```python
np.array(descriptors)[rfe.support_], rfe.ranking_
```

    (array(['MolWt', 'MolLogP', 'MolMR', 'HeavyAtomCount', 'NumHDonors',
            'NumHeteroatoms', 'NumRotatableBonds', 'NumValenceElectrons',
            'RingCount', 'TPSA', 'LabuteASA', 'BalabanJ', 'BertzCT'],
           dtype='<U19'),
     array([1, 1, 1, 1, 5, 1, 1, 1, 1, 2, 4, 3, 1, 1, 1, 1, 1]))

Again, consistent results with the previous selection methods.

## 3) Intrinsic methods - selecting directly from a model

In scikit-learn, models that returns ```coef_``` or ```feature_importances_``` assigns importances to the features and we can use these models to select an optimal subset of features for training.

### 3.1) [Random Forest](https://scikit-learn.org/stable/modules/ensemble.html#forest)

Random forest is a learning algorithm that uses the average response of an ensemble of decision trees to make a prediction. Each tree considers a random subset of the features in order to make its predictions. At each node, the algorithm tries to minimize the **impurity** of a split or maximize its **gain**, which is similar to reducing the number of wrong predictions. We can use the average decrease in impurity of the trees at each split to rank the features the model considered most important to make its decision. In scikit-learn, the importances of fitted random forest can be accessed via the ```feature_importance_``` attribute.

```python
from sklearn.ensemble import RandomForestRegressor
```

```python
rf = RandomForestRegressor(n_estimators=2000, n_jobs=-1)
```

```python
rf.fit(xtrain, ytrain)
```

    RandomForestRegressor(n_estimators=2000, n_jobs=-1)

```python
importances = rf.feature_importances_.reshape(1,-1)
```

```python
def rf_feat_importance(model, feats_names):
    return pd.DataFrame({'cols':feats_names, 'imp':model.feature_importances_}).sort_values('imp',ascending=False)
```

```python
fi = rf_feat_importance(rf, descriptors)
```

```python
#collapse_output
sns.barplot(y='cols', x='imp',data=fi)
```

The relative importance is normalized to sum up to 1.0. Features with higher importances are assigned higher values. Thus, MolLogP was the most relevant feature for random forest, followed by BertzCT and BalabanJ (both are topological descriptors). Features based on counts of atoms or fragments were assiged very small importances, which means that we could probably remove than from the data and without impacting performance so much.

### 3.2) [LASSO](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso)

LASSO is a linear model trained with L1 regularization in order to reduce model complexity. By "reduce complexity" I mean make the model simpler to gain a boost in generalization. LASSO achieves regularization by using the L1 norm:

$$\min_{w} { \frac{1}{2n_{\text{samples}}} ||X w - y||_2 ^ 2 + \alpha ||w||_1}$$

The L1 norm consists of adding the sum of the absolute values of the weights or coefficients to the cost function (e.g. sum of squared errors in a regression problem). In practice regularization forces the model to learn smaller weights, which is a way of making it simpler. Under L1 norm, most weights will be zero and as a result the weight vector will be sparse. Thus, the sparcitiy of the feature vector makes LASSO a great tool for feature selection since most unimportant features will be zeroed out.

Let's try scikit-learn implementation of LASSO.

```python
from sklearn.linear_model import Lasso
```

The alpha hyperparameter is a penalization term. If alpha = 0, LASSO becomes ordinary least square (e.g. LinearRegression). The default value is 1.0 but we will reduce it in order to make the model more flexible and not zero out all features.

```python
pipe_lasso = Pipeline(steps=[('scaler',RobustScaler()), ('estimator', Lasso(alpha=0.1))])
```

```python
pipe_lasso.fit(xtrain, ytrain)
```

    Pipeline(steps=[('scaler', RobustScaler()), ('estimator', Lasso(alpha=0.1))])

We can access the coefficients of the lasso estimator via the ```coef_``` attribute of the pipeline.

```python
feature_coef = pipe_lasso.named_steps.estimator.coef_
```

```python
feature_coef
```

    array([-0.0024517 , -1.46516244, -0.        , -0.        , -0.        ,
            0.        , -0.05151828,  0.09117961, -0.        , -0.        ,
           -0.        , -0.        , -0.        , -0.        , -0.        ,
            0.        , -0.56361338])

That's how the magic happens! The LASSO esimator zeroed out most features. It seems only 5 features survived the ordeal.

```python
np.array(descriptors)[np.where(feature_coef!=0.)[0]]
```

    array(['MolWt', 'MolLogP', 'NumHeteroatoms', 'NumRotatableBonds',
           'BertzCT'], dtype='<U19')

I'm not surprised at all! We can see features that were also selected by some of the previous methods.

# Final selection of features

Let's review which features were selected by the different approaches. Overall, the feature selection methods seems to agree that MolLogP and MolWt are important for predicting solubility; so let's keep them. The same applies for the number of rotatable bounds; in general flexibility reduce solubility because it makes more difficult for molecules to pack together and establish intermolecular interactions.

LabuteASA might also be useful because it gives information about the shape and size of the molecule. However, it's highly correlated with MolWt and MolMR, besides giving similar information about molecular size to these features. Furthermore, if we consider the importances returned by random forest and LASSO, we can see that LabuteASA have a very low importance.

At least one topological descriptor, BalabanJ and BertzCT, was selected in most methods. The information they encode have a relative overlap, so it could be worth investigating which is more informative for us. We could also keep both features since they don't have a high collinearity.

Only RFE and random forest recognized the importance of TPSA to predict solubility. In fact, TPSA encodes the H-bonding potential of a molecule because it includes all nitrogen, oxygen and sulphur atoms, making count-based descriptors not so important (e.g. NumHBondDonors and NumHBondAcceptors). Thus, we could consider including TPSA since it can give us implicit information about intermolecular interactions and charge distribution.

Let's rebuild the model using the reduced set of features.

```python
descriptors_reduced = ['MolWt','MolLogP','TPSA','BalabanJ','BertzCT','MolMR','NumRotatableBonds']
xtrain_reduced = trainset[descriptors_reduced].values
xtest_reduced = testset[descriptors_reduced].values
```

```python
rf.fit(xtrain_reduced, ytrain)
```

    RandomForestRegressor(n_estimators=2000, n_jobs=-1)

```python
preds_reduced = rf.predict(xtest_reduced)
print(f'RMSE (reduced set) test set = {mean_squared_error(preds_reduced, ytest, squared=False)}')
```

    RMSE (reduced set) test set = 0.979472446585308

As we can see, the performance is almost the same as the baseline trained with all features.

# What method should I use?

In this post we used different feature selection methods and learned about their advantages and drawbacks. So, which method is the best? The answer to this is a good o' "it depends". The best method depends on the type of feature and problem that you have. The image below will help you select a method and the reader is referred to the feature selection [post](https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/) at machine learning mastery.

> **You don't need to use all feature selection methods on your dataset!**

**Figure 3.**  Feature selection strategies

# Conclusion

We learned how to inspect the correlation between features using heatmap and select the most informative subset of features using selection methods. The most important takeaway is that irrelevant features must be removed before training a model. If a feature do not contain relevant information or is highly correlated with other features, it will only make the model more complex, less intepretable and you risk reducing the generalization potential of the model.

In the next post we will train our solubility prediction model using the selected features. Can we achieve state-of-the art performance? Stay tuned for the next posts!

# References

**Molecular descriptors**

https://northstar-www.dartmouth.edu/doc/MOE/Documentation/quasar/descr.htm#KH

http://www.codessa-pro.com/descriptors/index.htm

**Feature selection methods**

https://www.kaggle.com/prashant111/comprehensive-guide-on-feature-selection#5.-How-to-choose-the-right-feature-selection-method-

https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/

Kuhn, Max., and Kjell Johnson. [Applied Predictive Modeling](https://www.amazon.com/Applied-Predictive-Modeling-Max-Kuhn/dp/1461468485/ref=as_li_ss_tl?dchild=1&keywords=Applied+Predictive+Modeling&qid=1588722749&s=books&sr=1-1&linkCode=sl1&tag=inspiredalgor-20&linkId=45aa03c24c87a9e32f5611baa5e287ff&language=en_US). New York: Springer, 2013.

https://towardsdatascience.com/mistakes-in-applying-univariate-feature-selection-methods-34c43ce8b93d

https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection

https://scikit-learn.org/stable/modules/feature_selection.html

# **Fin**