LinearKernel

"LinearKernel"

**************

class numpy_ml.utils.kernels.LinearKernel(c0=0)

   The linear (i.e., dot-product) kernel.

   -[ Notes ]-

   For input vectors \mathbf{x} and \mathbf{y}, the linear kernel is:

      k(\mathbf{x}, \mathbf{y}) = \mathbf{x}^\top \mathbf{y} + c_0

   Parameters:

      **c0** (*float*) -- An "inhomogeneity" parameter. When *c0* = 0,

      the kernel is said to be homogenous. Default is 1.

   set_params(summary_dict)

      Set the model parameters and hyperparameters using the settings

      in *summary_dict*.

      Parameters:

         **summary_dict** (*dict*) -- A dictionary with keys

         'parameters' and 'hyperparameters', structured as would be

         returned by the "summary()" method. If a particular

         (hyper)parameter is not included in this dict, the current

         value will be used.

      Returns:

         **new_kernel** (Kernel instance) -- A kernel with parameters

         and hyperparameters adjusted to those specified in

         *summary_dict*.

   summary()

      Return the dictionary of model parameters, hyperparameters, and

      ID


GELU

"GELU"

******

class numpy_ml.neural_nets.activations.GELU(approximate=True)

   A Gaussian error linear unit (GELU). [*]

   -[ Notes ]-

   A ReLU alternative. GELU weights inputs by their value, rather than

   gates inputs by their sign, as in vanilla ReLUs.

   -[ References ]-

   [*] Hendrycks, D., & Gimpel, K. (2016). "Bridging nonlinearities

       and stochastic regularizers with Gaussian error linear units."

       *CoRR*.

   Parameters:

      **approximate** (*bool*) -- Whether to use a faster but less

      precise approximation to the Gauss error function when

      calculating the unit activation and gradient. Default is True.

   fn(z)

      Compute the GELU function on the elements of input *z*.

         \text{GELU}(z_i) = z_i P(Z \leq z_i) = z_i \Phi(z_i)     =

         z_i \cdot \frac{1}{2}(1 + \text{erf}(x/\sqrt{2}))

   grad(x)

      Evaluate the first derivative of the GELU function on the

      elements of input *x*.

         \frac{\partial \text{GELU}}{\partial x_i}  =     \frac{1}{2}

         + \frac{1}{2}\left(\text{erf}(\frac{x}{\sqrt{2}}) +

         \frac{x + \text{erf}'(\frac{x}{\sqrt{2}})}{\sqrt{2}}\right)

      where \text{erf}'(x) = \frac{2}{\sqrt{\pi}} \cdot \exp\{-x^2\}.

   grad2(x)

      Evaluate the second derivative of the GELU function on the

      elements of input *x*.

         \frac{\partial^2 \text{GELU}}{\partial x_i^2} =

         \frac{1}{2\sqrt{2}} \left\[

         \text{erf}'(\frac{x}{\sqrt{2}}) +         \frac{1}{\sqrt{2}}

         \text{erf}''(\frac{x}{\sqrt{2}})     \right]

      where \text{erf}'(x) = \frac{2}{\sqrt{\pi}} \cdot \exp\{-x^2\}

      and \text{erf}''(x) = \frac{-4x}{\sqrt{\pi}} \cdot \exp\{-x^2\}.


LeakyReLU

"LeakyReLU"

***********

class numpy_ml.neural_nets.activations.LeakyReLU(alpha=0.3)

   'Leaky' version of a rectified linear unit (ReLU).

   -[ Notes ]-

   Leaky ReLUs [*] are designed to address the vanishing gradient

   problem in ReLUs by allowing a small non-zero gradient when *x* is

   negative.

   Parameters:

      **alpha** (*float*) -- Activation slope when x < 0. Default is

      0.3.

   -[ References ]-

   [*] Mass, L. M., Hannun, A. Y, & Ng, A. Y. (2013). "Rectifier

       nonlinearities improve neural network acoustic models."

       *Proceedings of the 30th International Conference of Machine

       Learning, 30*.

   Initialize the ActivationBase object

   fn(z)

      Evaluate the leaky ReLU function on the elements of input *z*.

         \text{LeakyReLU}(z_i)     &=  z_i \ \ \ \ &&\text{if } z_i >

         0 \\     &=  \alpha z_i \ \ \ \ &&\text{otherwise}

   grad(x)

      Evaluate the first derivative of the leaky ReLU function on the

      elements of input *x*.

         \frac{\partial \text{LeakyReLU}}{\partial x_i}     &=  1 \ \

         \ \ &&\text{if }x_i > 0 \\     &=  \alpha \ \ \ \

         &&\text{otherwise}

   grad2(x)

      Evaluate the second derivative of the leaky ReLU function on the

      elements of input *x*.

         \frac{\partial^2 \text{LeakyReLU}}{\partial x_i^2}  =  0


Conv1D

"Conv1D"

********

class numpy_ml.neural_nets.layers.Conv1D(out_ch, kernel_width, pad=0, stride=1, dilation=0, act_fn=None, init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   Apply a one-dimensional convolution kernel over an input volume.

   -[ Notes ]-

   Equations:

      out = act_fn(pad(X) * W + b)

      out_dim = floor(1 + (n_rows_in + pad_left + pad_right - kernel_width) / stride)

   where '***' denotes the cross-correlation operation with stride *s*

   and dilation *d*.

   Parameters:

      * **out_ch** (*int*) -- The number of filters/kernels to compute

        in the current layer

      * **kernel_width** (*int*) -- The width of a single 1D

        filter/kernel in the current layer

      * **act_fn** (str, Activation object, or None) -- The activation

        function for computing "Y[t]". If None, use the identity

        function f(x) = x by default. Default is None.

      * **pad** (*int**, **tuple**, or **{'same'**, **'causal'}*) --

        The number of rows/columns to zero-pad the input with. If

        *'same'*, calculate padding to ensure the output length

        matches in the input length. If *'causal'* compute padding

        such that the output both has the same length as the input AND

        "output[t]" does not depend on "input[t + 1:]". Default is 0.

      * **stride** (*int*) -- The stride/hop of the convolution

        kernels as they move over the input volume. Default is 1.

      * **dilation** (*int*) -- Number of pixels inserted between

        kernel elements. Effective kernel shape after dilation is:

        "[kernel_rows * (d + 1) - d, kernel_cols * (d + 1) - d]".

        Default is 0.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output given input volume *X*.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, l_in, in_ch)*) -- The

           input volume consisting of *n_ex* examples, each of length

           *l_in* and with *in_ch* input channels

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, l_out, out_ch)*) -- The

         layer output.

   backward(dLdy, retain_grads=True)

      Compute the gradient of the loss with respect to the layer

      parameters.

      -[ Notes ]-

      Relies on "im2col()" and "col2im()" to vectorize the gradient

      calculation.  See the private method "_backward_naive()" for a

      more straightforward implementation.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, l_out, out_ch)* or

           list of arrays) -- The gradient(s) of the loss with respect

           to the layer output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, l_in, in_ch)*) -- The

         gradient of the loss with respect to the layer input volume.


SquaredError

"SquaredError"

**************

class numpy_ml.neural_nets.losses.SquaredError

   A squared-error / *L2* loss.

   -[ Notes ]-

   For real-valued target **y** and predictions \hat{\mathbf{y}}, the

   squared error is

      \mathcal{L}(\mathbf{y}, \hat{\mathbf{y}})     = 0.5

      ||\hat{\mathbf{y}} - \mathbf{y}||_2^2

   static loss(y, y_pred)

      Compute the squared error between *y* and *y_pred*.

      Parameters:

         * **y** ("ndarray" of shape (n, m)) -- Ground truth values

           for each of *n* examples

         * **y_pred** ("ndarray" of shape (n, m)) -- Predictions for

           the *n* examples in the batch.

      Returns:

         **loss** (*float*) -- The sum of the squared error across

         dimensions and examples.

   static grad(y, y_pred, z, act_fn)

      Gradient of the squared error loss with respect to the pre-

      nonlinearity input, *z*.

      -[ Notes ]-

      The current method computes the gradient \frac{\partial

      \mathcal{L}}{\partial \mathbf{z}}, where

         \mathcal{L}(\mathbf{z})     &=

         \text{squared_error}(\mathbf{y}, g(\mathbf{z})) \\

         g(\mathbf{z})     &=  \text{act_fn}(\mathbf{z})

      The gradient with respect to \mathbf{z} is then

         \frac{\partial \mathcal{L}}{\partial \mathbf{z}}     =

         (g(\mathbf{z}) - \mathbf{y}) \left(         \frac{\partial

         g}{\partial \mathbf{z}} \right)

      Parameters:

         * **y** ("ndarray" of shape (n, m)) -- Ground truth values

           for each of *n* examples.

         * **y_pred** ("ndarray" of shape (n, m)) -- Predictions for

           the *n* examples in the batch.

         * **act_fn** (Activation object) -- The activation function

           for the output layer of the network.

      Returns:

         **grad** ("ndarray" of shape (n, m)) -- The gradient of the

         squared error loss with respect to *z*.


DotProductAttention

"DotProductAttention"

*********************

class numpy_ml.neural_nets.layers.DotProductAttention(scale=True, dropout_p=0, init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   A single "attention head" layer using a dot-product for the scoring

   function.

   -[ Notes ]-

   The equations for a dot product attention layer are:

      \mathbf{Z}  &=  \mathbf{K Q}^\\top \ \ \ \ &&\text{if scale =

      False} \\             &=  \mathbf{K Q}^\top / \sqrt{d_k} \ \ \ \

      &&\text{if scale = True} \\ \mathbf{Y}  &=

      \text{dropout}(\text{softmax}(\mathbf{Z})) \mathbf{V}

   Parameters:

      * **scale** (*bool*) -- Whether to scale the the key-query dot

        product by the square root of the key/query vector

        dimensionality before applying the Softmax. This is useful,

        since the scale of dot product will otherwise increase as

        query / key dimensions grow. Default is True.

      * **dropout_p** (*float in** [**0**, **1**)*) -- The dropout

        propbability during training, applied to the output of the

        softmax. If 0, no dropout is applied. Default is 0.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

        Unused.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None. Unused.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Unused

      * **parameters** (*dict*) -- Unused

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   freeze()

      Freeze the layer parameters at their current values so they can

      no longer be updated.

   unfreeze()

      Unfreeze the layer parameters so they can be updated.

   forward(Q, K, V, retain_derived=True)

      Compute the attention-weighted output of a collection of keys,

      values, and queries.

      -[ Notes ]-

      In the most abstract (ie., hand-wave-y) sense:

         * Query vectors ask questions

         * Key vectors advertise their relevancy to questions

         * Value vectors give possible answers to questions

         * The dot product between Key and Query vectors provides

           scores for each of the the *n_ex* different Value vectors

      For a single query and *n* key-value pairs, dot-product

      attention (with scaling) is:

         w0 = dropout(softmax( (query @ key[0]) / sqrt(d_k) ))

         w1 = dropout(softmax( (query @ key[1]) / sqrt(d_k) ))

                                 ...

         wn = dropout(softmax( (query @ key[n]) / sqrt(d_k) ))

         y = np.array([w0, ..., wn]) @ values

                   (1 × n_ex)      (n_ex × d_v)

      In words, keys and queries are combined via dot-product to

      produce a score, which is then passed through a softmax to

      produce a weight on each value vector in Values. We elementwise

      multiply each value vector by its weight, and then take the

      elementwise sum of each weighted value vector to get the 1

      \times d_v output for the current example.

      In vectorized form,

         \mathbf{Y} = \text{dropout}(

         \text{softmax}(\mathbf{KQ}^\top / \sqrt{d_k}) ) \mathbf{V}

      Parameters:

         * **Q** ("ndarray" of shape *(n_ex, *, d_k)*) -- A set of

           *n_ex* query vectors packed into a single matrix. Optional

           middle dimensions can be used to specify, e.g., the number

           of parallel attention heads.

         * **K** ("ndarray" of shape *(n_ex, *, d_k)*) -- A set of

           *n_ex* key vectors packed into a single matrix. Optional

           middle dimensions can be used to specify, e.g., the number

           of parallel attention heads.

         * **V** ("ndarray" of shape *(n_ex, *, d_v)*) -- A set of

           *n_ex* value vectors packed into a single matrix. Optional

           middle dimensions can be used to specify, e.g., the number

           of parallel attention heads.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, *, d_v)*) -- The attention-

         weighted output values

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, *, d_v)*) -- The

           gradient of the loss wrt. the layer output *Y*

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         * **dQ** ("ndarray" of shape *(n_ex, *, d_k)* or list of

           arrays) -- The gradient of the loss wrt. the layer query

           matrix/matrices *Q*.

         * **dK** ("ndarray" of shape *(n_ex, *, d_k)* or list of

           arrays) -- The gradient of the loss wrt. the layer key

           matrix/matrices *K*.

         * **dV** ("ndarray" of shape *(n_ex, *, d_v)* or list of

           arrays) -- The gradient of the loss wrt. the layer value

           matrix/matrices *V*.


Pool2D

"Pool2D"

********

class numpy_ml.neural_nets.layers.Pool2D(kernel_shape, stride=1, pad=0, mode='max', optimizer=None)

   Bases: "LayerBase"

   A single two-dimensional pooling layer.

   Parameters:

      * **kernel_shape** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the current layer

      * **stride** (*int*) -- The stride/hop of the convolution

        kernels as they move over the input volume. Default is 1.

      * **pad** (*int**, **tuple**, or **'same'*) -- The number of

        rows/columns of 0's to pad the input. Default is 0.

      * **mode** (*{"max"**, **"average"}*) -- The pooling function to

        apply.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output given input volume *X*.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The input volume consisting of *n_ex* examples,

           each with dimension (*in_rows*,`in_cols`, *in_ch*)

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, out_rows, out_cols,

         out_ch)*) -- The layer output.

   backward(dLdY, retain_grads=True)

      Backprop from layer outputs to inputs

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The gradient of the loss wrt. the layer output

           *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- The gradient of the loss wrt. the layer input *X*.


GaussianNBClassifier

"GaussianNBClassifier"

**********************

class numpy_ml.linear_models.GaussianNBClassifier(eps=1e-06)

   A naive Bayes classifier for real-valued data.

   -[ Notes ]-

   The naive Bayes model assumes the features of each training example

   \mathbf{x} are mutually independent given the example label *y*:

      P(\mathbf{x}_i \mid y_i) = \prod_{j=1}^M P(x_{i,j} \mid y_i)

   where M is the rank of the i^{th} example \mathbf{x}_i and y_i is

   the label associated with the i^{th} example.

   Combining the conditional independence assumption with a simple

   application of Bayes' theorem gives the naive Bayes classification

   rule:

      \hat{y} &= \arg \max_y P(y \mid \mathbf{x}) \\         &= \arg

      \max_y  P(y) P(\mathbf{x} \mid y) \\         &= \arg \max_y

      P(y) \prod_{j=1}^M P(x_j \mid y)

   In the final expression, the prior class probability P(y) can be

   specified in advance or estimated empirically from the training

   data.

   In the Gaussian version of the naive Bayes model, the feature

   likelihood is assumed to be normally distributed for each class:

      \mathbf{x}_i \mid y_i = c, \theta \sim \mathcal{N}(\mu_c,

      \Sigma_c)

   where \theta is the set of model parameters: \{\mu_1, \Sigma_1,

   \ldots, \mu_K, \Sigma_K\}, K is the total number of unique classes

   present in the data, and the parameters for the Gaussian associated

   with class c, \mu_c and \Sigma_c (where 1 \leq c \leq K), are

   estimated via MLE from the set of training examples with label c.

   Parameters:

      **eps** (*float*) -- A value added to the variance to prevent

      numerical error. Default is 1e-6.

   Variables:

      * **parameters** (*dict*) -- Dictionary of model parameters:

        "mean", the *(K, M)* array of feature means under each class,

        "sigma", the *(K, M)* array of feature variances under each

        class, and "prior", the *(K,)* array of empirical prior

        probabilities for each class label.

      * **hyperparameters** (*dict*) -- Dictionary of model

        hyperparameters

      * **labels** ("ndarray" of shape *(K,)*) -- An array containing

        the unique class labels for the training examples.

   fit(X, y)

      Fit the model parameters via maximum likelihood.

      -[ Notes ]-

      The model parameters are stored in the "parameters" attribute.

      The following keys are present:

         "mean": "ndarray" of shape *(K, M)*

            Feature means for each of the *K* label classes

         "sigma": "ndarray" of shape *(K, M)*

            Feature variances for each of the *K* label classes

         "prior": "ndarray" of shape *(K,)*

            Prior probability of each of the *K* label classes,

            estimated empirically from the training data

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- A dataset consisting

           of *N* examples, each of dimension *M*

         * **y** ("ndarray" of shape *(N,)*) -- The class label for

           each of the *N* examples in *X*

      Returns:

         **self** ("GaussianNBClassifier" instance)

   predict(X)

      Use the trained classifier to predict the class label for each

      example in **X**.

      Parameters:

         **X** ("ndarray" of shape *(N, M)*) -- A dataset of *N*

         examples, each of dimension *M*

      Returns:

         **labels** ("ndarray" of shape *(N)*) -- The predicted class

         labels for each example in *X*


RBFKernel

"RBFKernel"

***********

class numpy_ml.utils.kernels.RBFKernel(sigma=None)

   Radial basis function (RBF) / squared exponential kernel.

   -[ Notes ]-

   For input vectors \mathbf{x} and \mathbf{y}, the radial basis

   function kernel is:

      k(\mathbf{x}, \mathbf{y}) = \exp \left\{ -0.5     \left\lVert

      \frac{\mathbf{x} -         \mathbf{y}}{\sigma} \right\rVert_2^2

      \right\}

   The RBF kernel decreases with distance and ranges between zero (in

   the limit) to one (when **x** = **y**). Notably, the implied

   feature space of the kernel has an infinite number of dimensions.

   Parameters:

      **sigma** (float or array of shape *(C,)* or None) -- A scaling

      parameter for the vectors **x** and **y**, producing an

      isotropic kernel if a float, or an anistropic kernel if an array

      of length *C*.  Larger values result in higher resolution /

      greater smoothing. If None, defaults to \sqrt(C / 2). Sometimes

      referred to as the kernel 'bandwidth'. Default is None.

   set_params(summary_dict)

      Set the model parameters and hyperparameters using the settings

      in *summary_dict*.

      Parameters:

         **summary_dict** (*dict*) -- A dictionary with keys

         'parameters' and 'hyperparameters', structured as would be

         returned by the "summary()" method. If a particular

         (hyper)parameter is not included in this dict, the current

         value will be used.

      Returns:

         **new_kernel** (Kernel instance) -- A kernel with parameters

         and hyperparameters adjusted to those specified in

         *summary_dict*.

   summary()

      Return the dictionary of model parameters, hyperparameters, and

      ID


LSTMCell

"LSTMCell"

**********

class numpy_ml.neural_nets.layers.LSTMCell(n_out, act_fn='Tanh', gate_fn='Sigmoid', init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   A single step of a long short-term memory (LSTM) RNN.

   -[ Notes ]-

   Notation:

   * "Z[t]"  is the input to each of the gates at timestep *t*

   * "A[t]"  is the value of the hidden state at timestep *t*

   * "Cc[t]" is the value of the *candidate* cell/memory state at

     timestep *t*

   * "C[t]"  is the value of the *final* cell/memory state at timestep

     *t*

   * "Gf[t]" is the output of the forget gate at timestep *t*

   * "Gu[t]" is the output of the update gate at timestep *t*

   * "Go[t]" is the output of the output gate at timestep *t*

   Equations:

      Z[t]  = stack([A[t-1], X[t]])

      Gf[t] = gate_fn(Wf @ Z[t] + bf)

      Gu[t] = gate_fn(Wu @ Z[t] + bu)

      Go[t] = gate_fn(Wo @ Z[t] + bo)

      Cc[t] = act_fn(Wc @ Z[t] + bc)

      C[t]  = Gf[t] * C[t-1] + Gu[t] * Cc[t]

      A[t]  = Go[t] * act_fn(C[t])

   where *@* indicates dot/matrix product, and '*' indicates

   elementwise multiplication.

   Parameters:

      * **n_out** (*int*) -- The dimension of a single hidden state /

        output on a given timestep.

      * **act_fn** (str, Activation object, or None) -- The activation

        function for computing "A[t]". Default is *'Tanh'*.

      * **gate_fn** (str, Activation object, or None) -- The gate

        function for computing the update, forget, and output gates.

        Default is *'Sigmoid'*.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(Xt)

      Compute the layer output for a single timestep.

      Parameters:

         **Xt** ("ndarray" of shape *(n_ex, n_in)*) -- Input at

         timestep t consisting of *n_ex* examples each of

         dimensionality *n_in*.

      Returns:

         * **At** ("ndarray" of shape *(n_ex, n_out)*) -- The value of

           the hidden state at timestep *t* for each of the *n_ex*

           examples.

         * **Ct** ("ndarray" of shape *(n_ex, n_out)*) -- The value of

           the cell/memory state at timestep *t* for each of the

           *n_ex* examples.

   backward(dLdAt)

      Backprop for a single timestep.

      Parameters:

         **dLdAt** ("ndarray" of shape *(n_ex, n_out)*) -- The

         gradient of the loss wrt. the layer outputs (ie., hidden

         states) at timestep *t*.

      Returns:

         **dLdXt** ("ndarray" of shape *(n_ex, n_in)*) -- The gradient

         of the loss wrt. the layer inputs at timestep *t*.

   flush_gradients()

      Erase all the layer's derived variables and gradients.


DFT

"DFT"

*****

numpy_ml.preprocessing.dsp.DFT(frame, positive_only=True)

   A naive O(N^2) implementation of the 1D discrete Fourier transform

   (DFT).

   -[ Notes ]-

   The Fourier transform decomposes a signal into a linear combination

   of sinusoids (ie., basis elements in the space of continuous

   periodic functions).  For a sequence \mathbf{x} = [x_1, \ldots,

   x_N] of N evenly spaced samples, the *k* th DFT coefficient is

   given by:

      c_k = \sum_{n=0}^{N-1} x_n \exp(-2 \pi i k n / N)

   where *i* is the imaginary unit, *k* is an index ranging from *0,

   ..., N-1*, and X_k is the complex coefficient representing the

   phase (imaginary part) and amplitude (real part) of the *k* th

   sinusoid in the DFT spectrum. The frequency of the *k* th sinusoid

   is (k 2 \pi / N) radians per sample.

   When applied to a real-valued input, the negative frequency terms

   are the complex conjugates of the positive-frequency terms and the

   overall spectrum is symmetric (excluding the first index, which

   contains the zero-frequency / intercept term).

   Parameters:

      * **frame** ("ndarray" of shape *(N,)*) -- A signal frame

        consisting of N samples

      * **positive_only** (*bool*) -- Whether to only return the

        coefficients for the positive frequency terms. Default is

        True.

   Returns:

      **spectrum** ("ndarray" of shape *(N,)* or *(N // 2 + 1,)* if

      *real_only*) -- The coefficients of the frequency spectrum for

      *frame*, including imaginary components.


SoftPlus

"SoftPlus"

**********

class numpy_ml.neural_nets.activations.SoftPlus

   A softplus activation function.

   -[ Notes ]-

   In contrast to "ReLU", the softplus activation is differentiable

   everywhere (including 0). It is, however, less computationally

   efficient to compute.

   The derivative of the softplus activation is the logistic sigmoid.

   fn(z)

      Evaluate the softplus activation on the elements of input *z*.

         \text{SoftPlus}(z_i) = \log(1 + e^{z_i})

   grad(x)

      Evaluate the first derivative of the softplus activation on the

      elements of input *x*.

         \frac{\partial \text{SoftPlus}}{\partial x_i} =

         \frac{e^{x_i}}{1 + e^{x_i}}

   grad2(x)

      Evaluate the second derivative of the softplus activation on the

      elements of input *x*.

         \frac{\partial^2 \text{SoftPlus}}{\partial x_i^2} =

         \frac{e^{x_i}}{(1 + e^{x_i})^2}


GeneralizedLinearModel

"GeneralizedLinearModel"

************************

class numpy_ml.linear_models.GeneralizedLinearModel(link, fit_intercept=True, tol=1e-05, max_iter=100)

   A generalized linear model with maximum likelihood fit via

   iteratively reweighted least squares (IRLS).

   -[ Notes ]-

   The generalized linear model (GLM) [7] [8] assumes that each

   target/dependent variable y_i in target vector \mathbf{y} = (y_1,

   \ldots, y_n), has been drawn independently from a pre-specified

   distribution in the exponential family [11] with unknown mean

   \mu_i. The GLM models a (one-to-one, continuous, differentiable)

   function, *g*, of this mean value as a linear combination of the

   model parameters \mathbf{b} and observed covariates, \mathbf{x}_i:

      g(\mathbb{E}[y_i \mid \mathbf{x}_i]) =     g(\mu_i) =

      \mathbf{b}^\top \mathbf{x}_i

   where *g* is known as the "link function" associated with the GLM.

   The choice of link function is informed by the instance of the

   exponential family the target is drawn from. Common examples:

   +---------------------------+----------------------+--------------------------------+

   | Distribution              | Link                 | Formula                        |

   |===========================|======================|================================|

   | Normal                    | Identity             | g(x) = x                       |

   +---------------------------+----------------------+--------------------------------+

   | Bernoulli                 | Logit                | g(x) = \log(x) - \log(1 - x)   |

   +---------------------------+----------------------+--------------------------------+

   | Binomial                  | Logit                | g(x) = \log(x) - \log(n - x)   |

   +---------------------------+----------------------+--------------------------------+

   | Poisson                   | Log                  | g(x) = \log(x)                 |

   +---------------------------+----------------------+--------------------------------+

   An iteratively re-weighted least squares (IRLS) algorithm [9] can

   be employed to find the maximum likelihood estimate for the model

   parameters \beta in any instance of the generalized linear model.

   IRLS is equivalent to Fisher scoring [10], which itself is a slight

   modification of classic Newton-Raphson for finding the zeros of the

   first derivative of the model log-likelihood.

   -[ References ]-

   [7] Nelder, J., & Wedderburn, R. (1972). Generalized linear models.

       *Journal of the Royal Statistical Society, Series A (General),

       135(3)*: 370–384.

   [8] https://en.wikipedia.org/wiki/Generalized_linear_model

   [9] https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squ

       ares

   [10] https://en.wikipedia.org/wiki/Scoring_algorithm

   [11] https://en.wikipedia.org/wiki/Exponential_family

   Parameters:

      * **link** (*{'identity'**, **'logit'**, **'log'}*) -- The link

        function to use during modeling.

      * **fit_intercept** (*bool*) -- Whether to fit an intercept term

        in addition to the model coefficients. Default is True.

      * **tol** (*float*) -- The minimum difference between successive

        iterations of IRLS Default is 1e-5.

      * **max_iter** (*int*) -- The maximum number of iteratively

        reweighted least squares iterations to run during fitting.

        Default is 100.

   Variables:

      **beta** ("ndarray" of shape *(M, 1)* or None) -- Fitted model

      coefficients.

   fit(X, y)

      Find the maximum likelihood GLM coefficients via IRLS.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- A dataset consisting

           of *N* examples, each of dimension *M*.

         * **y** ("ndarray" of shape *(N,)*) -- The targets for each

           of the *N* examples in *X*.

      Returns:

         **self** ("GeneralizedLinearModel" instance)

   predict(X)

      Use the trained model to generate predictions for the

      distribution means, \mu, associated with the collection of data

      points in **X**.

      Parameters:

         **X** ("ndarray" of shape *(Z, M)*) -- A dataset consisting

         of *Z* new examples, each of dimension *M*.

      Returns:

         **mu_pred** ("ndarray" of shape *(Z,)*) -- The model

         predictions for the expected value of the target associated

         with each item in *X*.


AdditiveNGram

"AdditiveNGram"

***************

class numpy_ml.ngram.AdditiveNGram(N, K=1, unk=True, filter_stopwords=True, filter_punctuation=True)

   An N-Gram model with smoothed probabilities calculated via additive

   / Lidstone smoothing.

   -[ Notes ]-

   The resulting estimates correspond to the expected value of the

   posterior, *p(ngram_prob | counts)*, when using a symmetric

   Dirichlet prior on counts with parameter *K*.

   Parameters:

      * **N** (*int*) -- The maximum length (in words) of the context-

        window to use in the langauge model. Model will compute all

        n-grams from 1, ..., N

      * **K** (*float*) -- The pseudocount to add to each observation.

        Larger values allocate more probability toward unseen events.

        When *K* = 1, the model is known as Laplace smoothing.  When

        *K* = 0.5, the model is known as expected likelihood

        estimation (ELE) or the Jeffreys-Perks law. Default is 1.

      * **unk** (*bool*) -- Whether to include the "<unk>" (unknown)

        token in the LM. Default is True.

      * **filter_stopwords** (*bool*) -- Whether to remove stopwords

        before training. Default is True.

      * **filter_punctuation** (*bool*) -- Whether to remove

        punctuation before training. Default is True.

   log_prob(words, N)

      Compute the smoothed log probability of a sequence of words

      under the *N*-gram language model with additive smoothing.

      -[ Notes ]-

      For a bigram, additive smoothing amounts to:

         P(w_i \mid w_{i-1}) = \frac{A + K}{B + KV}

      where

         A  &=  \text{Count}(w_{i-1}, w_i) \\ B  &=  \sum_j

         \text{Count}(w_{i-1}, w_j) \\ V  &= |\{ w_j \ : \

         \text{Count}(w_{i-1}, w_j) > 0 \}|

      This is equivalent to pretending we've seen every possible

      *N*-gram sequence at least *K* times.

      Additive smoothing can be problematic, as it:

         * Treats each predicted word in the same way

         * Can assign too much probability mass to unseen *N*-grams

      Parameters:

         * **words** (*list** of **strings*) -- A sequence of words.

         * **N** (*int*) -- The gram-size of the language model to use

           when calculating the log probabilities of the sequence.

      Returns:

         **total_prob** (*float*) -- The total log-probability of the

         sequence *words* under the *N*-gram language model.

   completions(words, N)

      Return the distribution over proposed next words under the

      *N*-gram language model.

      Parameters:

         * **words** (*list** or **tuple** of **strings*) -- The

           initial sequence of words

         * **N** (*int*) -- The gram-size of the language model to use

           to generate completions

      Returns:

         **probs** (*list of (word, log_prob) tuples*) -- The list of

         possible next words and their log probabilities under the

         *N*-gram language model (unsorted)

   cross_entropy(words, N)

      Calculate the model cross-entropy on a sequence of words against

      the empirical distribution of words in a sample.

      -[ Notes ]-

      Model cross-entropy, *H*, is defined as

         H(W) = -\frac{\log p(W)}{n}

      where W = [w_1, \ldots, w_k] is a sequence of words, and *n* is

      the number of *N*-grams in *W*.

      The model cross-entropy is proportional (not equal, since we use

      base *e*) to the average number of bits necessary to encode *W*

      under the model distribution.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to calculate

           cross-entropy on.

         * **words** (*list** or **tuple** of **strings*) -- The

           sequence of words to compute cross-entropy on.

      Returns:

         **H** (*float*) -- The model cross-entropy for the words in

         *words*.

   generate(N, seed_words=['<bol>'], n_sentences=5)

      Use the *N*-gram language model to generate sentences.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to generate

           from

         * **seed_words** (*list** of **strs*) -- A list of seed words

           to use to condition the initial sentence generation.

           Default is "["<bol>"]".

         * **sentences** (*int*) -- The number of sentences to

           generate from the *N*-gram model. Default is 50.

      Returns:

         **sentences** (*str*) -- Samples from the *N*-gram model,

         joined by white spaces, with individual sentences separated

         by newlines.

   perplexity(words, N)

      Calculate the model perplexity on a sequence of words.

      -[ Notes ]-

      Perplexity, *PP*, is defined as

         PP(W)  =  \left( \frac{1}{p(W)} \right)^{1 / n}

      or simply

         PP(W)  &=  \exp(-\log p(W) / n) \\        &=  \exp(H(W))

      where W = [w_1, \ldots, w_k] is a sequence of words, *H(w)* is

      the cross-entropy of *W* under the current model, and *n* is the

      number of *N*-grams in *W*.

      Minimizing perplexity is equivalent to maximizing the

      probability of *words* under the *N*-gram model. It may also be

      interpreted as the average branching factor when predicting the

      next word under the language model.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to calculate

           perplexity with.

         * **words** (*list** or **tuple** of **strings*) -- The

           sequence of words to compute perplexity on.

      Returns:

         **perplexity** (*float*) -- The model perlexity for the words

         in *words*.

   train(corpus_fp, vocab=None, encoding=None)

      Compile the n-gram counts for the text(s) in *corpus_fp*.

      -[ Notes ]-

      After running *train*, the "self.counts" attribute will store

      dictionaries of the *N*, *N-1*, ..., 1-gram counts.

      Parameters:

         * **corpus_fp** (*str*) -- The path to a newline-separated

           text corpus file.

         * **vocab** ("Vocabulary" instance or None) -- If not None,

           only the words in *vocab* will be used to construct the

           language model; all out-of-vocabulary words will either be

           mappend to "<unk>" (if "self.unk = True") or removed (if

           "self.unk = False"). Default is None.

         * **encoding** (*str** or **None*) -- Specifies the text

           encoding for corpus. Common entries are 'utf-8',

           'utf-8-sig', 'utf-16'. Default is None.


GradientBoostedDecisionTree

"GradientBoostedDecisionTree"

*****************************

class numpy_ml.trees.GradientBoostedDecisionTree(n_iter, max_depth=None, classifier=True, learning_rate=1, loss='crossentropy', step_size='constant')

   A gradient boosted ensemble of decision trees.

   -[ Notes ]-

   Gradient boosted machines (GBMs) fit an ensemble of *m* weak

   learners such that:

      f_m(X) = b(X) + \eta w_1 g_1 + \ldots + \eta w_m g_m

   where *b* is a fixed initial estimate for the targets, \eta is a

   learning rate parameter, and w_{\cdot} and g_{\cdot} denote the

   weights and learner predictions for subsequent fits.

   We fit each *w* and *g* iteratively using a greedy strategy so that

   at each iteration *i*,

      w_i, g_i = \arg \min_{w_i, g_i} L(Y, f_{i-1}(X) + w_i g_i)

   On each iteration we fit a new weak learner to predict the negative

   gradient of the loss with respect to the previous prediction,

   f_{i-1}(X). We then use the element-wise product of the predictions

   of this weak learner, g_i, with a weight, w_i, to compute the

   amount to adjust the predictions of our model at the previous

   iteration, f_{i-1}(X):

      f_i(X) := f_{i-1}(X) + w_i g_i

   Parameters:

      * **n_iter** (*int*) -- The number of iterations / weak

        estimators to use when fitting each dimension / class of *Y*.

      * **max_depth** (*int*) -- The maximum depth of each decision

        tree weak estimator. Default is None.

      * **classifier** (*bool*) -- Whether *Y* contains class labels

        or real-valued targets. Default is True.

      * **learning_rate** (*float*) -- Value in [0, 1] controlling the

        amount each weak estimator contributes to the overall model

        prediction. Sometimes known as the *shrinkage parameter* in

        the GBM literature. Default is 1.

      * **loss** (*{'crossentropy'**, **'mse'}*) -- The loss to

        optimize for the GBM. Default is 'crossentropy'.

      * **step_size** (*{"constant"**, **"adaptive"}*) -- How to

        choose the weight for each weak learner. If "constant", use a

        fixed weight of 1 for each learner. If "adaptive", use a step

        size computed via line-search on the current iteration's loss.

        Default is 'constant'.

   fit(X, Y)

      Fit the gradient boosted decision trees on a dataset.

      Parameters:

         * **X** ("ndarray" of shape (N, M)) -- The training data of

           *N* examples, each with *M* features

         * **Y** ("ndarray" of shape (N,)) -- An array of integer

           class labels for each example in *X* if "self.classifier =

           True", otherwise the set of target values for each example

           in *X*.

   predict(X)

      Use the trained model to classify or predict the examples in

      *X*.

      Parameters:

         **X** ("ndarray" of shape *(N, M)*) -- The training data of

         *N* examples, each with *M* features

      Returns:

         **preds** ("ndarray" of shape *(N,)*) -- The integer class

         labels predicted for each example in *X* if "self.classifier

         = True", otherwise the predicted target values.


BatchNorm1D

"BatchNorm1D"

*************

class numpy_ml.neural_nets.layers.BatchNorm1D(momentum=0.9, epsilon=1e-05, optimizer=None)

   Bases: "LayerBase"

   A batch normalization layer for 1D inputs.

   -[ Notes ]-

   BatchNorm is an attempt address the problem of internal covariate

   shift (ICS) during training by normalizing layer inputs.

   ICS refers to the change in the distribution of layer inputs during

   training as a result of the changing parameters of the previous

   layer(s). ICS can make it difficult to train models with saturating

   nonlinearities, and in general can slow training by requiring a

   lower learning rate.

   Equations [train]:

      Y = scaler * norm(X) + intercept

      norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

   Equations [test]:

      Y = scaler * running_norm(X) + intercept

      running_norm(X) = (X - running_mean) / sqrt(running_var + epsilon)

   In contrast to "LayerNorm1D", the BatchNorm layer calculates the

   mean and var across the *batch* rather than the output features.

   This has two disadvantages:

      1. It is highly affected by batch size: smaller mini-batch sizes

      increase the variance of the estimates for the global mean and

      variance.

      2. It is difficult to apply in RNNs -- one must fit a separate

      BatchNorm layer for *each* time-step.

   Parameters:

      * **momentum** (*float*) -- The momentum term for the running

        mean/running std calculations. The closer this is to 1, the

        less weight will be given to the mean/std of the current batch

        (i.e., higher smoothing). Default is 0.9.

      * **epsilon** (*float*) -- A small smoothing constant to use

        during computation of "norm(X)" to avoid divide-by-zero

        errors. Default is 1e-5.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   reset_running_stats()

      Reset the running mean and variance estimates to 0 and 1.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, n_in)*) -- Layer input,

           representing the *n_in*-dimensional features for a

           minibatch of *n_ex* examples.

         * **retain_derived** (*bool*) -- Whether to use the current

           intput to adjust the running mean and running_var

           computations. Setting this to True is the same as freezing

           the layer for the current input. Default is True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_in)*) -- Layer output for

         each of the *n_ex* examples

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, n_in)*) -- The

           gradient of the loss wrt. the layer output *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, n_in)*) -- The gradient of

         the loss wrt. the layer input *X*.


LinearRegression

"LinearRegression"

******************

class numpy_ml.linear_models.LinearRegression(fit_intercept=True)

   A weighted linear least-squares regression model.

   -[ Notes ]-

   In weighted linear least-squares regression [1], a real-valued

   target vector, **y**, is modeled as a linear combination of

   covariates, **X**, and model coefficients, \beta:

      y_i = \beta^\top \mathbf{x}_i + \epsilon_i

   In this equation \epsilon_i \sim \mathcal{N}(0, \sigma^2_i) is the

   error term associated with example i, and \sigma^2_i is the

   variance of the corresponding example.

   Under this model, the maximum-likelihood estimate for the

   regression coefficients, \beta, is:

      \hat{\beta} = \Sigma^{-1} \mathbf{X}^\top \mathbf{Wy}

   where \Sigma^{-1} = (\mathbf{X}^\top \mathbf{WX})^{-1} and **W** is

   a diagonal matrix of weights, with each entry inversely

   proportional to the variance of the corresponding measurement. When

   **W** is the identity matrix the examples are weighted equally and

   the model reduces to standard linear least squares [2].

   -[ References ]-

   [1] https://en.wikipedia.org/wiki/Weighted_least_squares

   [2] https://en.wikipedia.org/wiki/General_linear_model

   Parameters:

      **fit_intercept** (*bool*) -- Whether to fit an intercept term

      in addition to the model coefficients. Default is True.

   Variables:

      * **beta** ("ndarray" of shape *(M, K)* or None) -- Fitted model

        coefficients.

      * **sigma_inv** ("ndarray" of shape *(N, N)* or None) -- Inverse

        of the data covariance matrix.

   update(X, y, weights=None)

      Incrementally update the linear least-squares coefficients for a

      set of new examples.

      -[ Notes ]-

      The recursive least-squares algorithm [3] [4] is used to

      efficiently update the regression parameters as new examples

      become available. For a single new example (\mathbf{x}_{t+1},

      \mathbf{y}_{t+1}), the parameter updates are

         \beta_{t+1} = \left(     \mathbf{X}_{1:t}^\top

         \mathbf{X}_{1:t} +

         \mathbf{x}_{t+1}\mathbf{x}_{t+1}^\top \right)^{-1}

         \mathbf{X}_{1:t}^\top \mathbf{Y}_{1:t} +

         \mathbf{x}_{t+1}^\top \mathbf{y}_{t+1}

      where \beta_{t+1} are the updated regression coefficients,

      \mathbf{X}_{1:t} and \mathbf{Y}_{1:t} are the set of examples

      observed from timestep 1 to *t*.

      In the single-example case, the RLS algorithm uses the Sherman-

      Morrison formula [5] to avoid re-inverting the covariance matrix

      on each new update. In the multi-example case (i.e., where

      \mathbf{X}_{t+1} and \mathbf{y}_{t+1} are matrices of *N*

      examples each), we use the generalized Woodbury matrix identity

      [6] to update the inverse covariance. This comes at a

      performance cost, but is still more performant than doing

      multiple single-example updates if *N* is large.

      -[ References ]-

      [3] Gauss, C. F. (1821) *Theoria combinationis observationum

          erroribus minimis obnoxiae*, Werke, 4. Gottinge

      [4] https://en.wikipedia.org/wiki/Recursive_least_squares_filter

      [5] https://en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_form

          ula

      [6] https://en.wikipedia.org/wiki/Woodbury_matrix_identity

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- A dataset consisting

           of *N* examples, each of dimension *M*

         * **y** ("ndarray" of shape *(N, K)*) -- The targets for each

           of the *N* examples in *X*, where each target has dimension

           *K*

         * **weights** ("ndarray" of shape *(N,)* or None) -- Weights

           associated with the examples in *X*. Examples with larger

           weights exert greater influence on model fit.  When *y* is

           a vector (i.e., *K = 1*), weights should be set to the

           reciporical of the variance for each measurement (i.e., w_i

           = 1/\sigma^2_i). When *K > 1*, it is assumed that all

           columns of *y* share the same weight w_i. If None, examples

           are weighted equally, resulting in the standard linear

           least squares update.  Default is None.

      Returns:

         **self** ("LinearRegression" instance)

   fit(X, y, weights=None)

      Fit regression coefficients via maximum likelihood.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- A dataset consisting

           of *N* examples, each of dimension *M*.

         * **y** ("ndarray" of shape *(N, K)*) -- The targets for each

           of the *N* examples in *X*, where each target has dimension

           *K*.

         * **weights** ("ndarray" of shape *(N,)* or None) -- Weights

           associated with the examples in *X*. Examples with larger

           weights exert greater influence on model fit.  When *y* is

           a vector (i.e., *K = 1*), weights should be set to the

           reciporical of the variance for each measurement (i.e., w_i

           = 1/\sigma^2_i). When *K > 1*, it is assumed that all

           columns of *y* share the same weight w_i. If None, examples

           are weighted equally, resulting in the standard linear

           least squares update.  Default is None.

      Returns:

         **self** ("LinearRegression" instance)

   predict(X)

      Use the trained model to generate predictions on a new

      collection of data points.

      Parameters:

         **X** ("ndarray" of shape *(Z, M)*) -- A dataset consisting

         of *Z* new examples, each of dimension *M*.

      Returns:

         **y_pred** ("ndarray" of shape *(Z, K)*) -- The model

         predictions for the items in *X*.


ELU

"ELU"

*****

class numpy_ml.neural_nets.activations.ELU(alpha=1.0)

   An exponential linear unit (ELU).

   -[ Notes ]-

   ELUs are intended to address the fact that ReLUs are strictly

   nonnegative and thus have an average activation > 0, increasing the

   chances of internal covariate shift and slowing down learning. ELU

   units address this by (1) allowing negative values when x < 0,

   which (2) are bounded by a value -\alpha. Similar to "LeakyReLU",

   the negative activation values help to push the average unit

   activation towards 0. Unlike "LeakyReLU", however, the boundedness

   of the negative activation allows for greater robustness in the

   face of large negative values, allowing the function to avoid

   conveying the *degree* of "absence" (negative activation) in the

   input. [*]

   Parameters:

      **alpha** (*float*) -- Slope of negative segment. Default is 1.

   -[ References ]-

   [*] Clevert, D. A., Unterthiner, T., Hochreiter, S. (2016). "Fast

       and accurate deep network learning by exponential linear units

       (ELUs)". *4th International Conference on Learning

       Representations*.

   fn(z)

      Evaluate the ELU activation on the elements of input *z*.

         \text{ELU}(z_i)     &=  z_i \ \ \ \ &&\text{if }z_i > 0 \\

         &=  \alpha (e^{z_i} - 1) \ \ \ \ &&\text{otherwise}

   grad(x)

      Evaluate the first derivative of the ELU activation on the

      elements of input *x*.

         \frac{\partial \text{ELU}}{\partial x_i}     &=  1 \ \ \ \

         &&\text{if } x_i > 0 \\     &=  \alpha e^{x_i} \ \ \ \

         &&\text{otherwise}

   grad2(x)

      Evaluate the second derivative of the ELU activation on the

      elements of input *x*.

         \frac{\partial^2 \text{ELU}}{\partial x_i^2}     &=  0 \ \ \

         \ &&\text{if } x_i > 0 \\     &=  \alpha e^{x_i} \ \ \ \

         &&\text{otherwise}


WavenetResidualModule

"WavenetResidualModule"

***********************

class numpy_ml.neural_nets.modules.WavenetResidualModule(ch_residual, ch_dilation, dilation, kernel_width, optimizer=None, init='glorot_uniform')

   A WaveNet-like residual block with causal dilated convolutions.

      *Skip path in* >-------------------------------------------> + ---> *Skip path out*

                        Causal      |--> Tanh --|                  |

      *Main    |--> Dilated Conv1D -|           * --> 1x1 Conv1D --|

       path >--|                    |--> Sigm --|                  |

       in*     |-------------------------------------------------> + ---> *Main path out*

                                   *Residual path*

   On the final block, the output of the skip path is further

   processed to produce the network predictions.

   -[ References ]-

   [1] van den Oord et al. (2016). "Wavenet: a generative model for

       raw audio". https://arxiv.org/pdf/1609.03499.pdf

   Parameters:

      * **ch_residual** (*int*) -- The number of output channels for

        the 1x1 "Conv1D" layer in the main path.

      * **ch_dilation** (*int*) -- The number of output channels for

        the causal dilated "Conv1D" layer in the main path.

      * **dilation** (*int*) -- The dilation rate for the causal

        dilated "Conv1D" layer in the main path.

      * **kernel_width** (*int*) -- The width of the causal dilated

        "Conv1D" kernel in the main path.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is 'glorot_uniform'.

      * **optimizer** (str or Optimizer object or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   property parameters

      A dictionary of the module parameters.

   property hyperparameters

      A dictionary of the module hyperparameters

   property derived_variables

      A dictionary of intermediate values computed during the

      forward/backward passes.

   property gradients

      A dictionary of the module parameter gradients.

   forward(X_main, X_skip=None)

      Compute the module output on a single minibatch.

      Parameters:

         * **X_main** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The input volume consisting of *n_ex* examples,

           each with dimension (*in_rows*, *in_cols*, *in_ch*).

         * **X_skip** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*, or None) -- The output of the preceding skip-

           connection if this is not the first module in the network.

      Returns:

         * **Y_main** ("ndarray" of shape *(n_ex, out_rows, out_cols,

           out_ch)*) -- The output of the main pathway.

         * **Y_skip** ("ndarray" of shape *(n_ex, out_rows, out_cols,

           out_ch)*) -- The output of the skip-connection pathway.

   backward(dY_skip, dY_main=None)