BallTree

"BallTree"

**********

class numpy_ml.utils.data_structures.BallTree(leaf_size=40, metric=None)

   A ball tree data structure.

   -[ Notes ]-

   A ball tree is a binary tree in which every node defines a

   *D*-dimensional hypersphere ("ball") containing a subset of the

   points to be searched. Each internal node of the tree partitions

   the data points into two disjoint sets which are associated with

   different balls. While the balls themselves may intersect, each

   point is assigned to one or the other ball in the partition

   according to its distance from the ball's center. Each leaf node in

   the tree defines a ball and enumerates all data points inside that

   ball.

   Parameters:

      * **leaf_size** (*int*) -- The maximum number of datapoints at

        each leaf. Default is 40.

      * **metric** (Distance metric or None) -- The distance metric to

        use for computing nearest neighbors. If None, use the

        "euclidean()" metric. Default is None.

   -[ References ]-

   [1] Omohundro, S. M. (1989). "Five balltree construction

       algorithms". *ICSI Technical Report TR-89-063*.

   [2] Liu, T., Moore, A., & Gray A. (2006). "New algorithms for

       efficient high-dimensional nonparametric classification". *J.

       Mach. Learn. Res., 7*, 1135-1158.

   fit(X, y=None)

      Build a ball tree recursively using the O(M log N) *k*-d

      construction algorithm.

      -[ Notes ]-

      Recursively divides data into nodes defined by a centroid *C*

      and radius *r* such that each point below the node lies within

      the hyper-sphere defined by *C* and *r*.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- An array of *N*

           examples each with *M* features.

         * **y** ("ndarray" of shape *(N, *)* or None) -- An array of

           target values / labels associated with the entries in *X*.

           Default is None.

   nearest_neighbors(k, x)

      Find the *k* nearest neighbors in the ball tree to a query

      vector *x* using the KNS1 algorithm.

      Parameters:

         * **k** (*int*) -- The number of closest points in *X* to

           return

         * **x** ("ndarray" of shape *(1, M)*) -- The query vector.

      Returns:

         **nearest** (list of "PQNode" s of length *k*) -- List of the

         *k* points in *X* to closest to the query vector. The "key"

         attribute of each "PQNode" contains the point itself, the

         "val" attribute contains its target, and the "distance"

         attribute contains its distance to the query vector.


BatchNorm2D

"BatchNorm2D"

*************

class numpy_ml.neural_nets.layers.BatchNorm2D(momentum=0.9, epsilon=1e-05, optimizer=None)

   Bases: "LayerBase"

   A batch normalization layer for two-dimensional inputs with an

   additional channel dimension.

   -[ Notes ]-

   BatchNorm is an attempt address the problem of internal covariate

   shift (ICS) during training by normalizing layer inputs.

   ICS refers to the change in the distribution of layer inputs during

   training as a result of the changing parameters of the previous

   layer(s). ICS can make it difficult to train models with saturating

   nonlinearities, and in general can slow training by requiring a

   lower learning rate.

   Equations [train]:

      Y = scaler * norm(X) + intercept

      norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

   Equations [test]:

      Y = scaler * running_norm(X) + intercept

      running_norm(X) = (X - running_mean) / sqrt(running_var + epsilon)

   In contrast to "LayerNorm2D", the BatchNorm layer calculates the

   mean and var across the *batch* rather than the output features.

   This has two disadvantages:

      1. It is highly affected by batch size: smaller mini-batch sizes

      increase the variance of the estimates for the global mean and

      variance.

      2. It is difficult to apply in RNNs -- one must fit a separate

      BatchNorm layer for *each* time-step.

   Parameters:

      * **momentum** (*float*) -- The momentum term for the running

        mean/running std calculations. The closer this is to 1, the

        less weight will be given to the mean/std of the current batch

        (i.e., higher smoothing). Default is 0.9.

      * **epsilon** (*float*) -- A small smoothing constant to use

        during computation of "norm(X)" to avoid divide-by-zero

        errors. Default is 1e-5.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   reset_running_stats()

      Reset the running mean and variance estimates to 0 and 1.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      -[ Notes ]-

      Equations [train]:

         Y = scaler * norm(X) + intercept

         norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

      Equations [test]:

         Y = scaler * running_norm(X) + intercept

         running_norm(X) = (X - running_mean) / sqrt(running_var + epsilon)

      In contrast to "LayerNorm2D", the BatchNorm layer calculates the

      mean and var across the *batch* rather than the output features.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- Input volume containing the *in_rows* x

           *in_cols*-dimensional features for a minibatch of *n_ex*

           examples.

         * **retain_derived** (*bool*) -- Whether to use the current

           intput to adjust the running mean and running_var

           computations. Setting this to False is the same as freezing

           the layer for the current input. Default is True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- Layer output for each of the *n_ex* examples.

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The gradient of the loss wrt. the layer output

           *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- The gradient of the loss wrt. the layer input *X*.


RandomForest

"RandomForest"

**************

class numpy_ml.trees.RandomForest(n_trees, max_depth, n_feats, classifier=True, criterion='entropy')

   An ensemble (forest) of decision trees where each split is

   calculated using a random subset of the features in the input.

   Parameters:

      * **n_trees** (*int*) -- The number of individual decision trees

        to use within the ensemble.

      * **max_depth** (*int** or **None*) -- The depth at which to

        stop growing each decision tree. If None, grow each tree until

        the leaf nodes are pure.

      * **n_feats** (*int*) -- The number of features to sample on

        each split.

      * **classifier** (*bool*) -- Whether *Y* contains class labels

        or real-valued targets. Default is True.

      * **criterion** (*{'entropy'**, **'gini'**, **'mse'}*) -- The

        error criterion to use when calculating splits for each weak

        learner. When "classifier = False", valid entries are {'mse'}.

        When "classifier = True", valid entries are {'entropy',

        'gini'}. Default is 'entropy'.

   fit(X, Y)

      Create *n_trees*-worth of bootstrapped samples from the training

      data and use each to fit a separate decision tree.

   predict(X)

      Predict the target value for each entry in *X*.

      Parameters:

         **X** ("ndarray" of shape *(N, M)*) -- The training data of

         *N* examples, each with *M* features.

      Returns:

         **y_pred** ("ndarray" of shape *(N,)*) -- Model predictions

         for each entry in *X*.


MLENGram

"MLENGram"

**********

class numpy_ml.ngram.MLENGram(N, unk=True, filter_stopwords=True, filter_punctuation=True)

   A simple, unsmoothed N-gram model.

   Parameters:

      * **N** (*int*) -- The maximum length (in words) of the context-

        window to use in the langauge model. Model will compute all

        n-grams from 1, ..., N.

      * **unk** (*bool*) -- Whether to include the "<unk>" (unknown)

        token in the LM. Default is True.

      * **filter_stopwords** (*bool*) -- Whether to remove stopwords

        before training. Default is True.

      * **filter_punctuation** (*bool*) -- Whether to remove

        punctuation before training. Default is True.

   log_prob(words, N)

      Compute the log probability of a sequence of words under the

      unsmoothed, maximum-likelihood *N*-gram language model.

      Parameters:

         * **words** (*list** of **strings*) -- A sequence of words

         * **N** (*int*) -- The gram-size of the language model to use

           when calculating the log probabilities of the sequence

      Returns:

         **total_prob** (*float*) -- The total log-probability of the

         sequence *words* under the *N*-gram language model

   completions(words, N)

      Return the distribution over proposed next words under the

      *N*-gram language model.

      Parameters:

         * **words** (*list** or **tuple** of **strings*) -- The

           initial sequence of words

         * **N** (*int*) -- The gram-size of the language model to use

           to generate completions

      Returns:

         **probs** (*list of (word, log_prob) tuples*) -- The list of

         possible next words and their log probabilities under the

         *N*-gram language model (unsorted)

   cross_entropy(words, N)

      Calculate the model cross-entropy on a sequence of words against

      the empirical distribution of words in a sample.

      -[ Notes ]-

      Model cross-entropy, *H*, is defined as

         H(W) = -\frac{\log p(W)}{n}

      where W = [w_1, \ldots, w_k] is a sequence of words, and *n* is

      the number of *N*-grams in *W*.

      The model cross-entropy is proportional (not equal, since we use

      base *e*) to the average number of bits necessary to encode *W*

      under the model distribution.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to calculate

           cross-entropy on.

         * **words** (*list** or **tuple** of **strings*) -- The

           sequence of words to compute cross-entropy on.

      Returns:

         **H** (*float*) -- The model cross-entropy for the words in

         *words*.

   generate(N, seed_words=['<bol>'], n_sentences=5)

      Use the *N*-gram language model to generate sentences.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to generate

           from

         * **seed_words** (*list** of **strs*) -- A list of seed words

           to use to condition the initial sentence generation.

           Default is "["<bol>"]".

         * **sentences** (*int*) -- The number of sentences to

           generate from the *N*-gram model. Default is 50.

      Returns:

         **sentences** (*str*) -- Samples from the *N*-gram model,

         joined by white spaces, with individual sentences separated

         by newlines.

   perplexity(words, N)

      Calculate the model perplexity on a sequence of words.

      -[ Notes ]-

      Perplexity, *PP*, is defined as

         PP(W)  =  \left( \frac{1}{p(W)} \right)^{1 / n}

      or simply

         PP(W)  &=  \exp(-\log p(W) / n) \\        &=  \exp(H(W))

      where W = [w_1, \ldots, w_k] is a sequence of words, *H(w)* is

      the cross-entropy of *W* under the current model, and *n* is the

      number of *N*-grams in *W*.

      Minimizing perplexity is equivalent to maximizing the

      probability of *words* under the *N*-gram model. It may also be

      interpreted as the average branching factor when predicting the

      next word under the language model.

      Parameters:

         * **N** (*int*) -- The gram-size of the model to calculate

           perplexity with.

         * **words** (*list** or **tuple** of **strings*) -- The

           sequence of words to compute perplexity on.

      Returns:

         **perplexity** (*float*) -- The model perlexity for the words

         in *words*.

   train(corpus_fp, vocab=None, encoding=None)

      Compile the n-gram counts for the text(s) in *corpus_fp*.

      -[ Notes ]-

      After running *train*, the "self.counts" attribute will store

      dictionaries of the *N*, *N-1*, ..., 1-gram counts.

      Parameters:

         * **corpus_fp** (*str*) -- The path to a newline-separated

           text corpus file.

         * **vocab** ("Vocabulary" instance or None) -- If not None,

           only the words in *vocab* will be used to construct the

           language model; all out-of-vocabulary words will either be

           mappend to "<unk>" (if "self.unk = True") or removed (if

           "self.unk = False"). Default is None.

         * **encoding** (*str** or **None*) -- Specifies the text

           encoding for corpus. Common entries are 'utf-8',

           'utf-8-sig', 'utf-16'. Default is None.


BidirectionalLSTM

"BidirectionalLSTM"

*******************

class numpy_ml.neural_nets.modules.BidirectionalLSTM(n_out, act_fn=None, gate_fn=None, merge_mode='concat', init='glorot_uniform', optimizer=None)

   A single bidirectional long short-term memory (LSTM) layer.

   Parameters:

      * **n_out** (*int*) -- The dimension of a single hidden state /

        output on a given timestep

      * **act_fn** (Activation object or None) -- The activation

        function for computing "A[t]". If not specified, use "Tanh" by

        default.

      * **gate_fn** (Activation object or None) -- The gate function

        for computing the update, forget, and output gates. If not

        specified, use "Sigmoid" by default.

      * **merge_mode** (*{"sum"**, **"multiply"**, **"concat"**,

        **"average"}*) -- Mode by which outputs of the forward and

        backward LSTMs will be combined. Default is 'concat'.

      * **optimizer** (str or Optimizer object or None) -- The

        optimization strategy to use when performing gradient updates

        within the *update* method.  If None, use the "SGD" optimizer

        with default parameters. Default is None.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is 'glorot_uniform'.

   forward(X)

      Run a forward pass across all timesteps in the input.

      Parameters:

         **X** ("ndarray" of shape *(n_ex, n_in, n_t)*) -- Input

         consisting of *n_ex* examples each of dimensionality *n_in*

         and extending for *n_t* timesteps.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_out, n_t)*) -- The value

         of the hidden state for each of the *n_ex* examples across

         each of the *n_t* timesteps.

   backward(dLdA)

      Run a backward pass across all timesteps in the input.

      Parameters:

         **dLdA** ("ndarray" of shape *(n_ex, n_out, n_t)*) -- The

         gradient of the loss with respect to the layer output for

         each of the *n_ex* examples across all *n_t* timesteps.

      Returns:

         **dLdX** ("ndarray" of shape *(n_ex, n_in, n_t)*) -- The

         value of the hidden state for each of the *n_ex* examples

         across each of the *n_t* timesteps.

   property derived_variables

      A dictionary of intermediate values computed during the

      forward/backward passes.

   property gradients

      A dictionary of the accumulated module parameter gradients.

   property parameters

      A dictionary of the module parameters.

   property hyperparameters

      A dictionary of the module hyperparameters.


KNN

"KNN"

*****

class numpy_ml.nonparametric.KNN(k=5, leaf_size=40, classifier=True, metric=None, weights='uniform')

   A *k*-nearest neighbors (kNN) model relying on a ball tree for

   efficient computation.

   Parameters:

      * **k** (*int*) -- The number of neighbors to use during

        prediction. Default is 5.

      * **leaf_size** (*int*) -- The maximum number of datapoints at

        each leaf in the ball tree. Default is 40.

      * **classifier** (*bool*) -- Whether to treat the values in Y as

        class labels (classifier = True) or real-valued targets

        (classifier = False). Default is True.

      * **metric** (Distance metric or None) -- The distance metric to

        use for computing nearest neighbors. If None, use the

        "euclidean()" metric by default. Default is None.

      * **weights** (*{'uniform'**, **'distance'}*) -- How to weight

        the predictions from each neighbors. 'uniform' assigns uniform

        weights to each neighbor, while 'distance' assigns weights

        proportional to the inverse of the distance from the query

        point. Default is 'uniform'.

   fit(X, y)

      Fit the model to the data and targets in *X* and *y*

      Parameters:

         * **X** (numpy array of shape *(N, M)*) -- An array of *N*

           examples to generate predictions on.

         * **y** (numpy array of shape *(N, *)*) -- Targets for the

           *N* rows in *X*.

   predict(X)

      Generate predictions for the targets associated with the rows in

      *X*.

      Parameters:

         **X** (numpy array of shape *(N', M')*) -- An array of *N'*

         examples to generate predictions on.

      Returns:

         **y** (numpy array of shape *(N', *)*) -- Predicted targets

         for the *N'* rows in *X*.


Sigmoid

"Sigmoid"

*********

class numpy_ml.neural_nets.activations.Sigmoid

   A logistic sigmoid activation function.

   fn(z)

      Evaluate the logistic sigmoid, \sigma, on the elements of input

      *z*.

         \sigma(x_i) = \frac{1}{1 + e^{-x_i}}

   grad(x)

      Evaluate the first derivative of the logistic sigmoid on the

      elements of *x*.

         \frac{\partial \sigma}{\partial x_i} = \sigma(x_i) (1 -

         \sigma(x_i))

   grad2(x)

      Evaluate the second derivative of the logistic sigmoid on the

      elements of *x*.

         \frac{\partial^2 \sigma}{\partial x_i^2} =     \frac{\partial

         \sigma}{\partial x_i} (1 - 2 \sigma(x_i))


SkipConnectionIdentityModule

"SkipConnectionIdentityModule"

******************************

class numpy_ml.neural_nets.modules.SkipConnectionIdentityModule(out_ch, kernel_shape1, kernel_shape2, stride1=1, stride2=1, act_fn=None, epsilon=1e-05, momentum=0.9, optimizer=None, init='glorot_uniform')

   A ResNet-like "identity" shortcut module.

   -[ Notes ]-

   The identity module enforces *same* padding during each convolution

   to ensure module output has same dims as its input.

      X -> Conv2D -> Act_fn -> BatchNorm2D -> Conv2D -> BatchNorm2D -> + -> Act_fn

       \______________________________________________________________/

   -[ References ]-

   [1] He et al. (2015). "Deep residual learning for image

       recognition." https://arxiv.org/pdf/1512.03385.pdf

   Parameters:

      * **out_ch** (*int*) -- The number of filters/kernels to compute

        in the first convolutional layer.

      * **kernel_shape1** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the first convolutional layer.

      * **kernel_shape2** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the second convolutional layer.

      * **stride1** (*int*) -- The stride/hop of the convolution

        kernels in the first convolutional layer. Default is 1.

      * **stride2** (*int*) -- The stride/hop of the convolution

        kernels in the second convolutional layer. Default is 1.

      * **act_fn** (Activation object or None) -- The activation

        function for computing Y[t]. If None, use the identity f(x) =

        x by default. Default is None.

      * **epsilon** (*float*) -- A small smoothing constant to use

        during "BatchNorm2D" computation to avoid divide-by-zero

        errors. Default is 1e-5.

      * **momentum** (*float*) -- The momentum term for the running

        mean/running std calculations in the "BatchNorm2D" layers.

        The closer this is to 1, the less weight will be given to the

        mean/std of the current batch (i.e., higher smoothing).

        Default is 0.9.

      * **optimizer** (str or Optimizer object or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is 'glorot_uniform'.

   property parameters

      A dictionary of the module parameters.

   property hyperparameters

      A dictionary of the module hyperparameters.

   property derived_variables

      A dictionary of intermediate values computed during the

      forward/backward passes.

   property gradients

      A dictionary of the accumulated module parameter gradients.

   forward(X, retain_derived=True)

      Compute the module output given input volume *X*.

      Parameters:

         * **X** ("ndarray" of shape (n_ex, in_rows, in_cols, in_ch))

           -- The input volume consisting of *n_ex* examples, each

           with dimension (*in_rows*, *in_cols*, *in_ch*).

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape (n_ex, out_rows, out_cols, out_ch))

         -- The module output volume.

   backward(dLdY, retain_grads=True)

      Compute the gradient of the loss with respect to the layer

      parameters.

      Parameters:

         * **dLdy** ("ndarray" of shape (*n_ex, out_rows, out_cols,

           out_ch*) or list of arrays) -- The gradient(s) of the loss

           with respect to the module output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape (n_ex, in_rows, in_cols, in_ch))

         -- The gradient of the loss with respect to the module input

         volume.


FullyConnected

"FullyConnected"

****************

class numpy_ml.neural_nets.layers.FullyConnected(n_out, act_fn=None, init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   A fully-connected (dense) layer.

   -[ Notes ]-

   A fully connected layer computes the function

      \mathbf{Y} = f( \mathbf{WX} + \mathbf{b} )

   where *f* is the activation nonlinearity, **W** and **b** are

   parameters of the layer, and **X** is the minibatch of input

   examples.

   Parameters:

      * **n_out** (*int*) -- The dimensionality of the layer output

      * **act_fn** (str, Activation object, or None) -- The element-

        wise output nonlinearity used in computing *Y*. If None, use

        the identity function f(X) = X. Default is None.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, n_in)*) -- Layer input,

           representing the *n_in*-dimensional features for a

           minibatch of *n_ex* examples.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_out)*) -- Layer output

         for each of the *n_ex* examples.

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, n_out)* or list of

           arrays) -- The gradient(s) of the loss wrt. the layer

           output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dLdX** ("ndarray" of shape *(n_ex, n_in)* or list of

         arrays) -- The gradient of the loss wrt. the layer input(s)

         *X*.


ReLU

"ReLU"

******

class numpy_ml.neural_nets.activations.ReLU

   A rectified linear activation function.

   -[ Notes ]-

   "ReLU units can be fragile during training and can "die". For

   example, a large gradient flowing through a ReLU neuron could cause

   the weights to update in such a way that the neuron will never

   activate on any datapoint again. If this happens, then the gradient

   flowing through the unit will forever be zero from that point on.

   That is, the ReLU units can irreversibly die during training since

   they can get knocked off the data manifold.

   For example, you may find that as much as 40% of your network can

   be "dead" (i.e. neurons that never activate across the entire

   training dataset) if the learning rate is set too high. With a

   proper setting of the learning rate this is less frequently an

   issue." [*]

   -[ References ]-

   [*] Karpathy, A. "CS231n: Convolutional neural networks for visual

       recognition."

   Initialize the ActivationBase object

   fn(z)

      Evaulate the ReLU function on the elements of input *z*.

         \text{ReLU}(z_i)     &=  z_i \ \ \ \ &&\text{if }z_i > 0 \\

         &=  0 \ \ \ \ &&\text{otherwise}

   grad(x)

      Evaulate the first derivative of the ReLU function on the

      elements of input *x*.

         \frac{\partial \text{ReLU}}{\partial x_i}     &=  1 \ \ \ \

         &&\text{if }x_i > 0 \\     &=  0   \ \ \ \ &&\text{otherwise}

   grad2(x)

      Evaulate the second derivative of the ReLU function on the

      elements of input *x*.

         \frac{\partial^2 \text{ReLU}}{\partial x_i^2}  =  0


manhattan

"manhattan"

***********

numpy_ml.utils.distance_metrics.manhattan(x, y)

   Compute the Manhattan (*L1*) distance between two real vectors

   -[ Notes ]-

   The Manhattan distance between two vectors **x** and **y** is

      d(\mathbf{x}, \mathbf{y}) = \sum_i |x_i - y_i|

   Parameters:

      * **x** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

      * **y** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

   Returns:

      **d** (*float*) -- The L1 distance between **x** and **y**.


Tanh

"Tanh"

******

class numpy_ml.neural_nets.activations.Tanh

   A hyperbolic tangent activation function.

   fn(z)

      Compute the tanh function on the elements of input *z*.

   grad(x)

      Evaluate the first derivative of the tanh function on the

      elements of input *x*.

         \frac{\partial \tanh}{\partial x_i}  =  1 - \tanh(x)^2

   grad2(x)

      Evaluate the second derivative of the tanh function on the

      elements of input *x*.

         \frac{\partial^2 \tanh}{\partial x_i^2} =     -2 \tanh(x)

         \left(\frac{\partial \tanh}{\partial x_i}\right)


Standardizer

"Standardizer"

**************

class numpy_ml.preprocessing.general.Standardizer(with_mean=True, with_std=True)

   Feature-wise standardization for vector inputs.

   -[ Notes ]-

   Due to the sensitivity of empirical mean and standard deviation

   calculations to extreme values, *Standardizer* cannot guarantee

   balanced feature scales in the presence of outliers. In particular,

   note that because outliers for each feature can have different

   magnitudes, the spread of the transformed data on each feature can

   be very different.

   Similar to sklearn, *Standardizer* uses a biased estimator for the

   standard deviation: "numpy.std(x, ddof=0)".

   Parameters:

      * **with_mean** (*bool*) -- Whether to scale samples to have 0

        mean during transformation. Default is True.

      * **with_std** (*bool*) -- Whether to scale samples to have unit

        variance during transformation. Default is True.

   property hyperparameters

   property parameters

   fit(X)

      Store the feature-wise mean and standard deviation across the

      samples in *X* for future scaling.

      Parameters:

         **X** ("ndarray" of shape *(N, C)*) -- An array of N samples,

         each with dimensionality *C*

   transform(X)

      Standardize features by removing the mean and scaling to unit

      variance.

      For a sample *x*, the standardized score is calculated as:

         z = (x - u) / s

      where *u* is the mean of the training samples or zero if

      *with_mean* is False, and *s* is the standard deviation of the

      training samples or 1 if *with_std* is False.

      Parameters:

         **X** ("ndarray" of shape *(N, C)*) -- An array of N samples,

         each with dimensionality *C*.

      Returns:

         **Z** ("ndarray" of shape *(N, C)*) -- The feature-wise

         standardized version of *X*.

   inverse_transform(Z)

      Convert a collection of standardized features back into the

      original feature space.

      For a standardized sample *z*, the unstandardized score is

      calculated as:

         x = z s + u

      where *u* is the mean of the training samples or zero if

      *with_mean* is False, and *s* is the standard deviation of the

      training samples or 1 if *with_std* is False.

      Parameters:

         **Z** ("ndarray" of shape *(N, C)*) -- An array of *N*

         standardized samples, each with dimensionality *C*.

      Returns:

         **X** ("ndarray" of shape *(N, C)*) -- The unstandardixed

         samples from *Z*.


DCT

"DCT"

*****

numpy_ml.preprocessing.dsp.DCT(frame, orthonormal=True)

   A naive O(N^2) implementation of the 1D discrete cosine transform-

   II (DCT-II).

   -[ Notes ]-

   For a signal \mathbf{x} = [x_1, \ldots, x_N] consisting of *N*

   samples, the *k* th DCT coefficient, c_k, is

      c_k = 2 \sum_{n=0}^{N-1} x_n \cos(\pi k (2 n + 1) / (2 N))

   where *k* ranges from 0, \ldots, N-1.

   The DCT is highly similar to the DFT -- whereas in a DFT the basis

   functions are sinusoids, in a DCT they are restricted solely to

   cosines. A signal's DCT representation tends to have more of its

   energy concentrated in a smaller number of coefficients when

   compared to the DFT, and is thus commonly used for signal

   compression. [1]

   [1] Smoother signals can be accurately approximated using fewer DFT

       / DCT coefficients, resulting in a higher compression ratio.

       The DCT naturally yields a continuous extension at the signal

       boundaries due its use of even basis functions (cosine). This

       in turn produces a smoother extension in comparison to DFT or

       DCT approximations, resulting in a higher compression.

   Parameters:

      * **frame** ("ndarray" of shape *(N,)*) -- A signal frame

        consisting of N samples

      * **orthonormal** (*bool*) -- Scale to ensure the coefficient

        vector is orthonormal. Default is True.

   Returns:

      **dct** ("ndarray" of shape *(N,)*) -- The discrete cosine

      transform of the samples in *frame*.


Multiply

"Multiply"

**********

class numpy_ml.neural_nets.layers.Multiply(act_fn=None, optimizer=None)

   Bases: "LayerBase"

   A multiplication layer that returns the *elementwise* product of

   its inputs, passed through an optional nonlinearity.

   Parameters:

      * **act_fn** (str, Activation object, or None) -- The element-

        wise output nonlinearity used in computing the final output.

        If None, use the identity function f(x) = x. Default is None.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Unused

      * **parameters** (*dict*) -- Unused

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** (list of length *n_inputs*) -- A list of tensors, all

           of the same shape.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, *)*) -- The product over

         the *n_ex* examples.

   backward(dLdY, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, *)*) -- The gradient

           of the loss wrt. the layer output *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** (list of length *n_inputs*) -- The gradient of the

         loss wrt. each input in *X*.


TFIDFEncoder

"TFIDFEncoder"

**************

class numpy_ml.preprocessing.nlp.TFIDFEncoder(vocab=None, lowercase=True, min_count=0, smooth_idf=True, max_tokens=None, input_type='files', filter_stopwords=True, filter_punctuation=True, tokenizer='words')

   An object for compiling and encoding the term-frequency inverse-

   document-frequency (TF-IDF) representation of the tokens in a text

   corpus.

   -[ Notes ]-

   TF-IDF is intended to reflect how important a word is to a document

   in a collection or corpus. For a word token *w* in a document *d*,

   and a corpus, D = \{d_1, \ldots, d_N\}, we have:

      \text{TF}(w, d)  &=  \text{num. occurences of }w \text{ in

      document }d \\ \text{IDF}(w, D)  &=  \log \frac{|D|}{|\{ d \in

      D: t \in d \}|}

   Parameters:

      * **vocab** ("Vocabulary" object or list-like) -- An existing

        vocabulary to filter the tokens in the corpus against. Default

        is None.

      * **lowercase** (*bool*) -- Whether to convert each string to

        lowercase before tokenization. Default is True.

      * **min_count** (*int*) -- Minimum number of times a token must

        occur in order to be included in vocab. Default is 0.

      * **smooth_idf** (*bool*) -- Whether to add 1 to the denominator

        of the IDF calculation to avoid divide-by-zero errors. Default

        is True.

      * **max_tokens** (*int*) -- Only add the *max_tokens* most

        frequent tokens that occur more than *min_count* to the

        vocabulary.  If None, add all tokens greater that occur more

        than than *min_count*. Default is None.

      * **input_type** (*{'files'**, **'strings'}*) -- If 'files', the

        sequence input to *fit* is expected to be a list of filepaths.

        If 'strings', the input is expected to be a list of lists,

        each sublist containing the raw strings for a single document

        in the corpus. Default is 'filename'.

      * **filter_stopwords** (*bool*) -- Whether to remove stopwords

        before encoding the words in the corpus. Default is True.

      * **filter_punctuation** (*bool*) -- Whether to remove

        punctuation before encoding the words in the corpus. Default

        is True.

      * **tokenizer** (*{'whitespace'**, **'words'**,

        **'characters'**, **'bytes'}*) -- Strategy to follow when

        mapping strings to tokens. The *'whitespace'* tokenizer splits

        strings at whitespace characters. The *'words'* tokenizer

        splits strings using a "word" regex. The *'characters'*

        tokenizer splits strings into individual characters. The

        *'bytes'* tokenizer splits strings into a collection of

        individual bytes.

   fit(corpus_seq, encoding='utf-8-sig')

      Compute term-frequencies and inverse document frequencies on a

      collection of documents.

      Parameters:

         * **corpus_seq** (*str** or **list** of **strs*) -- The

           filepath / list of filepaths / raw string contents of the

           document(s) to be encoded, in accordance with the

           *input_type* parameter passed to the "__init__()" method.

           Each document is expected to be a string of tokens

           separated by whitespace.

         * **encoding** (*str*) -- Specifies the text encoding for

           corpus if *input_type* is *files*. Common entries are

           either 'utf-8' (no header byte), or 'utf-8-sig' (header

           byte). Default is 'utf-8-sig'.

      Returns:

         *self*

   transform(ignore_special_chars=True)

      Generate the term-frequency inverse-document-frequency encoding

      of a text corpus.

      Parameters:

         **ignore_special_chars** (*bool*) -- Whether to drop columns

         corresponding to "<eol>", "<bol>", and "<unk>" tokens from

         the final tfidf encoding. Default is True.

      Returns:

         **tfidf** (numpy array of shape *(D, M [- 3])*) -- The

         encoded corpus, with each row corresponding to a single

         document, and each column corresponding to a token id. The

         mapping between column numbers and tokens is stored in the

         *idx2token* attribute IFF *ignore_special_chars* is False.

         Otherwise, the mappings are not accurate.


hamming

"hamming"

*********

numpy_ml.utils.distance_metrics.hamming(x, y)

   Compute the Hamming distance between two integer-valued vectors.

   -[ Notes ]-

   The Hamming distance between two vectors **x** and **y** is

      d(\mathbf{x}, \mathbf{y}) = \frac{1}{N} \sum_i \mathbb{1}_{x_i

      \neq y_i}

   Parameters:

      * **x** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between. Both vectors should be integer-

        valued.

      * **y** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between. Both vectors should be integer-

        valued.

   Returns:

      **d** (*float*) -- The Hamming distance between **x** and **y**.


Conv2D

"Conv2D"

********

class numpy_ml.neural_nets.layers.Conv2D(out_ch, kernel_shape, pad=0, stride=1, dilation=0, act_fn=None, optimizer=None, init='glorot_uniform')

   Bases: "LayerBase"

   Apply a two-dimensional convolution kernel over an input volume.

   -[ Notes ]-

   Equations:

      out = act_fn(pad(X) * W + b)

      n_rows_out = floor(1 + (n_rows_in + pad_left + pad_right - filter_rows) / stride)

      n_cols_out = floor(1 + (n_cols_in + pad_top + pad_bottom - filter_cols) / stride)

   where *'*'* denotes the cross-correlation operation with stride *s*

   and dilation *d*.

   Parameters:

      * **out_ch** (*int*) -- The number of filters/kernels to compute

        in the current layer

      * **kernel_shape** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the current layer

      * **act_fn** (str, Activation object, or None) -- The activation

        function for computing "Y[t]". If None, use the identity

        function f(X) = X by default. Default is None.

      * **pad** (*int**, **tuple**, or **'same'*) -- The number of

        rows/columns to zero-pad the input with. Default is 0.

      * **stride** (*int*) -- The stride/hop of the convolution

        kernels as they move over the input volume. Default is 1.

      * **dilation** (*int*) -- Number of pixels inserted between

        kernel elements. Effective kernel shape after dilation is:

        "[kernel_rows * (d + 1) - d, kernel_cols * (d + 1) - d]".

        Default is 0.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      A dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output given input volume *X*.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The input volume consisting of *n_ex* examples,

           each with dimension (*in_rows*, *in_cols*, *in_ch*).

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, out_rows, out_cols,

         out_ch)*) -- The layer output.

   backward(dLdy, retain_grads=True)

      Compute the gradient of the loss with respect to the layer

      parameters.

      -[ Notes ]-

      Relies on "im2col()" and "col2im()" to vectorize the gradient

      calculation.

      See the private method "_backward_naive()" for a more

      straightforward implementation.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, out_rows, out_cols,

           out_ch)* or list of arrays) -- The gradient(s) of the loss

           with respect to the layer output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- The gradient of the loss with respect to the layer input

         volume.


LayerNorm1D

"LayerNorm1D"

*************

class numpy_ml.neural_nets.layers.LayerNorm1D(epsilon=1e-05, optimizer=None)

   Bases: "LayerBase"

   A layer normalization layer for 1D inputs.

   -[ Notes ]-

   In contrast to "BatchNorm1D", the LayerNorm layer calculates the

   mean and variance across *features* rather than examples in the

   batch ensuring that the mean and variance estimates are independent

   of batch size and permitting straightforward application in RNNs.

   Equations [train & test]:

      Y = scaler * norm(X) + intercept

      norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

   Also in contrast to "BatchNorm1D", *scaler* and *intercept* are

   applied *elementwise* to "norm(X)".

   Parameters:

      * **epsilon** (*float*) -- A small smoothing constant to use

        during computation of "norm(X)" to avoid divide-by-zero

        errors. Default is 1e-5.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, n_in)*) -- Layer input,

           representing the *n_in*-dimensional features for a

           minibatch of *n_ex* examples.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_in)*) -- Layer output for

         each of the *n_ex* examples.

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, n_in)*) -- The

           gradient of the loss wrt. the layer output *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, n_in)*) -- The gradient of

         the loss wrt. the layer input *X*.


SkipConnectionConvModule

"SkipConnectionConvModule"

**************************

class numpy_ml.neural_nets.modules.SkipConnectionConvModule(out_ch1, out_ch2, kernel_shape1, kernel_shape2, kernel_shape_skip, pad1=0, pad2=0, stride1=1, stride2=1, act_fn=None, epsilon=1e-05, momentum=0.9, stride_skip=1, optimizer=None, init='glorot_uniform')

   A ResNet-like "convolution" shortcut module.

   -[ Notes ]-

   In contrast to "SkipConnectionIdentityModule", the additional

   *conv2d_skip* and *batchnorm_skip* layers in the shortcut path

   allow adjusting the dimensions of *X* to match the output of the

   main set of convolutions.

      X -> Conv2D -> Act_fn -> BatchNorm2D -> Conv2D -> BatchNorm2D -> + -> Act_fn

       \_____________________ Conv2D -> Batchnorm2D __________________/

   -[ References ]-

   [1] He et al. (2015). "Deep residual learning for image

       recognition." https://arxiv.org/pdf/1512.03385.pdf

   Parameters:

      * **out_ch1** (*int*) -- The number of filters/kernels to

        compute in the first convolutional layer.

      * **out_ch2** (*int*) -- The number of filters/kernels to

        compute in the second convolutional layer.

      * **kernel_shape1** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the first convolutional layer.

      * **kernel_shape2** (*2-tuple*) -- The dimension of a single 2D

        filter/kernel in the second convolutional layer.

      * **kernel_shape_skip** (*2-tuple*) -- The dimension of a single

        2D filter/kernel in the "skip" convolutional layer.

      * **stride1** (*int*) -- The stride/hop of the convolution

        kernels in the first convolutional layer. Default is 1.

      * **stride2** (*int*) -- The stride/hop of the convolution

        kernels in the second convolutional layer. Default is 1.

      * **stride_skip** (*int*) -- The stride/hop of the convolution

        kernels in the "skip" convolutional layer. Default is 1.

      * **pad1** (*int**, **tuple**, or **'same'*) -- The number of

        rows/columns of 0's to pad the input to the first

        convolutional layer with. Default is 0.

      * **pad2** (*int**, **tuple**, or **'same'*) -- The number of

        rows/columns of 0's to pad the input to the second

        convolutional layer with. Default is 0.

      * **act_fn** (Activation object or None) -- The activation

        function for computing "Y[t]". If None, use the identity f(x)

        = x by default. Default is None.

      * **epsilon** (*float*) -- A small smoothing constant to use

        during "BatchNorm2D" computation to avoid divide-by-zero

        errors. Default is 1e-5.

      * **momentum** (*float*) -- The momentum term for the running

        mean/running std calculations in the "BatchNorm2D" layers.

        The closer this is to 1, the less weight will be given to the

        mean/std of the current batch (i.e., higher smoothing).

        Default is 0.9.

      * **init** (*str*) -- The weight initialization strategy. Valid

        entries are {'glorot_normal', 'glorot_uniform', 'he_normal',

        'he_uniform'}.

      * **optimizer** (str or Optimizer object) -- The optimization

        strategy to use when performing gradient updates within the

        "update" method.  If None, use the "SGD" optimizer with

        default parameters. Default is None.

   property parameters

      A dictionary of the module parameters.

   property hyperparameters

      A dictionary of the module hyperparameters.

   property derived_variables

      A dictionary of intermediate values computed during the

      forward/backward passes.

   property gradients

      A dictionary of the accumulated module parameter gradients.

   forward(X, retain_derived=True)

      Compute the layer output given input volume *X*.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The input volume consisting of *n_ex* examples,

           each with dimension (*in_rows*, *in_cols*, *in_ch*).

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, out_rows, out_cols,

         out_ch)*) -- The module output volume.

   backward(dLdY, retain_grads=True)

      Compute the gradient of the loss with respect to the module

      parameters.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, out_rows, out_cols,

           out_ch)*) --

         * **arrays** (*or list of*) -- The gradient(s) of the loss

           with respect to the module output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- The gradient of the loss with respect to the module input

         volume.


WGAN_GPLoss

"WGAN_GPLoss"

*************

class numpy_ml.neural_nets.losses.WGAN_GPLoss(lambda_=10)

   The loss function for a Wasserstein GAN [*] [†] with gradient

   penalty.

   -[ Notes ]-

   Assuming an optimal critic, minimizing this quantity wrt. the

   generator parameters corresponds to minimizing the Wasserstein-1

   (earth-mover) distance between the fake and real data

   distributions.

   The formula for the WGAN-GP critic loss is

      \text{WGANLoss}     &=  \sum_{x \in X_{real}} p(x) D(x)

      - \sum_{x' \in X_{fake}} p(x') D(x') \\ \text{WGANLossGP}     &=

      \text{WGANLoss} + \lambda         (||\nabla_{X_{interp}}

      D(X_{interp})||_2 - 1)^2

   where

      X_{fake}  &=   \text{Generator}(\mathbf{z}) \\ X_{interp}   &=

      \alpha X_{real} + (1 - \alpha) X_{fake} \\

   and

      \mathbf{z}  &\sim  \mathcal{N}(0, \mathbb{1}) \\ \alpha  &\sim

      \text{Uniform}(0, 1)

   -[ References ]-

   [*] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., &

       Courville, A. (2017) "Improved training of Wasserstein GANs"

       *Advances in Neural Information Processing Systems, 31*:

       5769-5779.

   [†] Goodfellow, I. J, Abadie, P. A., Mirza, M., Xu, B., Farley, D.

       W., Ozair, S., Courville, A., & Bengio, Y. (2014) "Generative

       adversarial nets" *Advances in Neural Information Processing

       Systems, 27*: 2672-2680.

   Parameters:

      **lambda** (*float*) -- The gradient penalty coefficient.

      Default is 10.

   loss(Y_fake, module, Y_real=None, gradInterp=None)

      Computes the generator and critic loss using the WGAN-GP value

      function.

      Parameters:

         * **Y_fake** ("ndarray" of shape (n_ex,)) -- The output of

           the critic for *X_fake*.

         * **module** (*{'C'**, **'G'}*) -- Whether to calculate the

           loss for the critic ('C') or the generator ('G'). If

           calculating loss for the critic, *Y_real* and *gradInterp*

           must not be None.

         * **Y_real** ("ndarray" of shape *(n_ex,)* or None) -- The

           output of the critic for *X_real*. Default is None.

         * **gradInterp** ("ndarray" of shape *(n_ex, n_feats)* or

           None) -- The gradient of the critic output for *X_interp*

           wrt. *X_interp*. Default is None.

      Returns:

         **loss** (*float*) -- Depending on the setting for *module*,

         either the critic or generator loss, averaged over examples

         in the minibatch.

   grad(Y_fake, module, Y_real=None, gradInterp=None)

      Computes the gradient of the generator or critic loss with

      regard to its inputs.

      Parameters:

         * **Y_fake** ("ndarray" of shape *(n_ex,)*) -- The output of

           the critic for *X_fake*.

         * **module** (*{'C'**, **'G'}*) -- Whether to calculate the

           gradient for the critic loss ('C') or the generator loss

           ('G'). If calculating grads for the critic, *Y_real* and

           *gradInterp* must not be None.

         * **Y_real** ("ndarray" of shape *(n_ex,)* or None) -- The

           output of the critic for *X_real*. Default is None.

         * **gradInterp** ("ndarray" of shape *(n_ex, n_feats)* or

           None) -- The gradient of the critic output on *X_interp*

           wrt. *X_interp*. Default is None.

      Returns:

         **grads** (*tuple*) -- If *module* == 'C', returns a 3-tuple

         containing the gradient of the critic loss with regard to

         (*Y_fake*, *Y_real*, *gradInterp*). If *module* == 'G',

         returns the gradient of the generator with regard to

         *Y_fake*.


chebyshev

"chebyshev"

***********

numpy_ml.utils.distance_metrics.chebyshev(x, y)

   Compute the Chebyshev (L_\infty) distance between two real vectors

   -[ Notes ]-

   The Chebyshev distance between two vectors **x** and **y** is

      d(\mathbf{x}, \mathbf{y}) = \max_i |x_i - y_i|

   Parameters:

      * **x** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

      * **y** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

   Returns:

      **d** (*float*) -- The Chebyshev distance between **x** and

      **y**.


DiGraph

"DiGraph"

*********

class numpy_ml.utils.graphs.DiGraph(V, E)

   Bases: "Graph"

   A generic directed graph object.

   Parameters:

      * **V** (*list*) -- A list of vertex IDs.

      * **E** (list of "Edge" objects) -- A list of directed edges

        connecting pairs of vertices in "V".

   reverse()

      Reverse the direction of all edges in the graph

   topological_ordering()

      Returns a (non-unique) topological sort / linearization of the

      nodes IFF the graph is acyclic, otherwise returns None.

      -[ Notes ]-

      A topological sort is an ordering on the nodes in *G* such that

      for every directed edge u \rightarrow v in the graph, *u*

      appears before *v* in the ordering.  The topological ordering is

      produced by ordering the nodes in *G* by their DFS "last visit

      time," from greatest to smallest.

      This implementation follows a recursive, DFS-based approach [1]

      which may break if the graph is very large. For an iterative

      version, see Khan's algorithm [2] .

      -[ References ]-

      [1] Tarjan, R. (1976), Edge-disjoint spanning trees and depth-

          first search, *Acta Informatica, 6 (2)*: 171–185.

      [2] Kahn, A. (1962), Topological sorting of large networks,

          *Communications of the ACM, 5 (11)*: 558–562.

      Returns:

         **ordering** (*list or None*) -- A topoligical ordering of

         the vertex indices if the graph is a DAG, otherwise None.

   is_acyclic()

      Check whether the graph contains cycles


Add

"Add"

*****

class numpy_ml.neural_nets.layers.Add(act_fn=None, optimizer=None)

   Bases: "LayerBase"

   An "addition" layer that returns the sum of its inputs, passed

   through an optional nonlinearity.

   Parameters:

      * **act_fn** (str, Activation object, or None) -- The element-

        wise output nonlinearity used in computing the final output.

        If None, use the identity function f(x) = x. Default is None.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Unused

      * **parameters** (*dict*) -- Unused

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** (list of length *n_inputs*) -- A list of tensors, all

           of the same shape.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, *)*) -- The sum over the

         *n_ex* examples.

   backward(dLdY, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, *)*) -- The gradient

           of the loss wrt. the layer output *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** (list of length *n_inputs*) -- The gradient of the

         loss wrt. each input in *X*.


DecisionTree

"DecisionTree"

**************

class numpy_ml.trees.DecisionTree(classifier=True, max_depth=None, n_feats=None, criterion='entropy', seed=None)

   A decision tree model for regression and classification problems.

   Parameters:

      * **classifier** (*bool*) -- Whether to treat target values as

        categorical (classifier = True) or continuous (classifier =

        False). Default is True.

      * **max_depth** (*int** or **None*) -- The depth at which to

        stop growing the tree. If None, grow the tree until all leaves

        are pure. Default is None.

      * **n_feats** (*int*) -- Specifies the number of features to

        sample on each split. If None, use all features on each split.

        Default is None.

      * **criterion** (*{'mse'**, **'entropy'**, **'gini'}*) -- The

        error criterion to use when calculating splits. When

        *classifier* is False, valid entries are {'mse'}. When

        *classifier* is True, valid entries are {'entropy', 'gini'}.

        Default is 'entropy'.

      * **seed** (*int** or **None*) -- Seed for the random number

        generator. Default is None.

   fit(X, Y)

      Fit a binary decision tree to a dataset.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- The training data of

           *N* examples, each with *M* features

         * **Y** ("ndarray" of shape *(N,)*) -- An array of integer

           class labels for each example in *X* if self.classifier =

           True, otherwise the set of target values for each example

           in *X*.

   predict(X)

      Use the trained decision tree to classify or predict the

      examples in *X*.

      Parameters:

         **X** ("ndarray" of shape *(N, M)*) -- The training data of

         *N* examples, each with *M* features

      Returns:

         **preds** ("ndarray" of shape *(N,)*) -- The integer class

         labels predicted for each example in *X* if self.classifier =

         True, otherwise the predicted target values.

   predict_class_probs(X)

      Use the trained decision tree to return the class probabilities

      for the examples in *X*.

      Parameters:

         **X** ("ndarray" of shape *(N, M)*) -- The training data of

         *N* examples, each with *M* features

      Returns:

         **preds** ("ndarray" of shape *(N, n_classes)*) -- The class

         probabilities predicted for each example in *X*.


PolynomialKernel

"PolynomialKernel"

******************

class numpy_ml.utils.kernels.PolynomialKernel(d=3, gamma=None, c0=1)

   The degree-*d* polynomial kernel.

   -[ Notes ]-

   For input vectors \mathbf{x} and \mathbf{y}, the polynomial kernel

   is:

      k(\mathbf{x}, \mathbf{y}) = (\gamma \mathbf{x}^\top \mathbf{y} +

      c_0)^d

   In contrast to the linear kernel, the polynomial kernel also

   computes similarities *across* dimensions of the **x** and **y**

   vectors, allowing it to account for interactions between features.

   As an instance of the dot product family of kernels, the polynomial

   kernel is invariant to a rotation of the coordinates about the

   origin, but *not* to translations.

   Parameters:

      * **d** (*int*) -- Degree of the polynomial kernel. Default is

        3.

      * **gamma** (*float** or **None*) -- A scaling parameter for the

        dot product between *x* and *y*, determining the amount of

        smoothing/resonlution of the kernel. Larger values result in

        greater smoothing. If None, defaults to 1 / *C*.  Sometimes

        referred to as the kernel bandwidth.  Default is None.

      * **c0** (*float*) -- Parameter trading off the influence of

        higher-order versus lower-order terms in the polynomial. If

        *c0* = 0, the kernel is said to be homogenous. Default is 1.

   set_params(summary_dict)

      Set the model parameters and hyperparameters using the settings

      in *summary_dict*.

      Parameters:

         **summary_dict** (*dict*) -- A dictionary with keys

         'parameters' and 'hyperparameters', structured as would be

         returned by the "summary()" method. If a particular

         (hyper)parameter is not included in this dict, the current

         value will be used.

      Returns:

         **new_kernel** (Kernel instance) -- A kernel with parameters

         and hyperparameters adjusted to those specified in

         *summary_dict*.

   summary()

      Return the dictionary of model parameters, hyperparameters, and

      ID


VAELoss

"VAELoss"

*********

class numpy_ml.neural_nets.losses.VAELoss

   The variational lower bound for a variational autoencoder with

   Bernoulli units.

   -[ Notes ]-

   The VLB to the sum of the binary cross entropy between the true

   input and the predicted output (the "reconstruction loss") and the

   KL divergence between the learned variational distribution q and

   the prior, p, assumed to be a unit Gaussian.

      \text{VAELoss} =     \text{cross_entropy}(\mathbf{y},

      \hat{\mathbf{y}})         + \mathbb{KL}[q \ || \ p]

   where \mathbb{KL}[q \ || \ p] is the Kullback-Leibler divergence

   between the distributions q and p.

   -[ References ]-

   [1] Kingma, D. P. & Welling, M. (2014). "Auto-encoding variational

       Bayes". *arXiv preprint arXiv:1312.6114.*

       https://arxiv.org/pdf/1312.6114.pdf

   static loss(y, y_pred, t_mean, t_log_var)

      Variational lower bound for a Bernoulli VAE.

      Parameters:

         * **y** ("ndarray" of shape *(n_ex, N)*) -- The original

           images.

         * **y_pred** ("ndarray" of shape *(n_ex, N)*) -- The VAE

           reconstruction of the images.

         * **t_mean** ("ndarray" of shape *(n_ex, T)*) -- Mean of the

           variational distribution q(t \mid x).

         * **t_log_var** ("ndarray" of shape *(n_ex, T)*) -- Log of

           the variance vector of the variational distribution q(t

           \mid x).

      Returns:

         **loss** (*float*) -- The VLB, averaged across the batch.

   static grad(y, y_pred, t_mean, t_log_var)

      Compute the gradient of the VLB with regard to the network

      parameters.

      Parameters:

         * **y** ("ndarray" of shape *(n_ex, N)*) -- The original

           images.

         * **y_pred** ("ndarray" of shape *(n_ex, N)*) -- The VAE

           reconstruction of the images.

         * **t_mean** ("ndarray" of shape *(n_ex, T)*) -- Mean of the

           variational distribution q(t | x).

         * **t_log_var** ("ndarray" of shape *(n_ex, T)*) -- Log of

           the variance vector of the variational distribution q(t |

           x).

      Returns:

         * **dY_pred** ("ndarray" of shape *(n_ex, N)*) -- The

           gradient of the VLB with regard to *y_pred*.

         * **dLogVar** ("ndarray" of shape *(n_ex, T)*) -- The

           gradient of the VLB with regard to *t_log_var*.

         * **dMean** ("ndarray" of shape *(n_ex, T)*) -- The gradient

           of the VLB with regard to *t_mean*.


MultiHeadedAttentionModule

"MultiHeadedAttentionModule"

****************************

class numpy_ml.neural_nets.modules.MultiHeadedAttentionModule(n_heads=8, dropout_p=0, init='glorot_uniform', optimizer=None)

   A mutli-headed attention module.

   -[ Notes ]-

   Multi-head attention allows a model to jointly attend to

   information from different representation subspaces at different

   positions. With a single head, this information would get averaged

   away when the attention weights are combined with the value

      \text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V})     =

      [\text{head}_1; ...; \text{head}_h] \mathbf{W}^{(O)}

   where

      \text{head}_i = \text{SDP_attention}(     \mathbf{Q W}_i^{(Q)},

      \mathbf{K W}_i^{(K)}, \mathbf{V W}_i^{(V)})

   and the projection weights are parameter matrices:

      \mathbf{W}_i^{(Q)}  &\in     \mathbb{R}^{(\text{kqv_dim} \

      \times \ \text{latent_dim})} \\ \mathbf{W}_i^{(K)}  &\in

      \mathbb{R}^{(\text{kqv_dim} \ \times \ \text{latent_dim})} \\

      \mathbf{W}_i^{(V)}  &\in     \mathbb{R}^{(\text{kqv_dim} \

      \times \ \text{latent_dim})} \\ \mathbf{W}^{(O)}  &\in

      \mathbb{R}^{(\text{n_heads} \cdot \text{latent_dim} \ \times \

      \text{kqv_dim})}

   Importantly, the current module explicitly assumes that

      \text{kqv_dim} = \text{dim(query)} = \text{dim(keys)} =

      \text{dim(values)}

   and that

      \text{latent_dim} = \text{kqv_dim / n_heads}

   **[MH Attention Head h]**:

      K --> W_h^(K) ------\

      V --> W_h^(V) ------- > DP_Attention --> head_h

      Q --> W_h^(Q) ------/

   The full **[MultiHeadedAttentionModule]** then becomes

            -----------------

      K --> | [Attn Head 1] | --> head_1 --\

      V --> | [Attn Head 2] | --> head_2 --\

      Q --> |      ...      |      ...       --> Concat --> W^(O) --> MH_out

            | [Attn Head Z] | --> head_Z --/

            -----------------

   Due to the reduced dimension of each head, the total computational

   cost is similar to that of a single attention head with full (i.e.,

   kqv_dim) dimensionality.

   Parameters:

      * **n_heads** (*int*) -- The number of simultaneous attention

        heads to use. Note that the larger *n_heads*, the smaller the

        dimensionality of any single head, since "latent_dim = kqv_dim

        / n_heads". Default is 8.

      * **dropout_p** (*float in** [**0**, **1**)*) -- The dropout

        propbability during training, applied to the output of the

        softmax in each dot-product attention head. If 0, no dropout

        is applied. Default is 0.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is 'glorot_uniform'.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   forward(Q, K, V)

   backward(dLdy)

   property derived_variables

      A dictionary of intermediate values computed during the

      forward/backward passes.

   property gradients

      A dictionary of the accumulated module parameter gradients.

   property parameters

      A dictionary of the module parameters.

   property hyperparameters

      A dictionary of the module hyperparameters.


HuffmanEncoder

"HuffmanEncoder"

****************

class numpy_ml.preprocessing.nlp.HuffmanEncoder

   fit(text)

      Build a Huffman tree for the tokens in *text* and compute each

      token's binary encoding.

      -[ Notes ]-

      In a Huffman code, tokens that occur more frequently are

      (generally) represented using fewer bits. Huffman codes produce

      the minimum expected codeword length among all methods for

      encoding tokens individually.

      Huffman codes correspond to paths through a binary tree, with 1

      corresponding to "move right" and 0 corresponding to "move

      left". In contrast to standard binary trees, the Huffman tree is

      constructed from the bottom up. Construction begins by

      initializing a min-heap priority queue consisting of each token

      in the corpus, with priority corresponding to the token

      frequency. At each step, the two most infrequent tokens in the

      corpus are removed and become the children of a parent

      pseudotoken whose "frequency" is the sum of the frequencies of

      its children. This new parent pseudotoken is added to the

      priority queue and the process is repeated recursively until no

      tokens remain.

      Parameters:

         **text** (list of strs or "Vocabulary" instance) -- The

         tokenized text or a pretrained "Vocabulary" object to use for

         building the Huffman code.

   transform(text)

      Transform the words in *text* into their Huffman-code

      representations.

      Parameters:

         **text** (list of *N* strings) -- The list of words to encode

      Returns:

         **codes** (list of *N* binary strings) -- The encoded words

         in *text*

   inverse_transform(codes)

      Transform an encoded sequence of bit-strings back into words.

      Parameters:

         **codes** (list of *N* binary strings) -- A list of encoded

         bit-strings, represented as strings.

      Returns:

         **text** (list of *N* strings) -- The decoded text.

   property tokens

      A list the unique tokens in *text*

   property codes

      A list with the Huffman code for each unique token in *text*


GPRegression

"GPRegression"

**************

class numpy_ml.nonparametric.GPRegression(kernel='RBFKernel', alpha=1e-10)

   A Gaussian Process (GP) regression model.

      y \mid X, f  &\sim  \mathcal{N}( [f(x_1), \ldots, f(x_n)],

      \alpha I ) \\ f \mid X     &\sim  \text{GP}(0, K)

   for data D = \{(x_1, y_1), \ldots, (x_n, y_n) \} and a covariance

   matrix K_{ij} = \text{kernel}(x_i, x_j) for all i, j \in \{1,

   \ldots, n \}.

   Parameters:

      * **kernel** (*str*) -- The kernel to use in fitting the GP

        prior. Default is 'RBFKernel'.

      * **alpha** (*float*) -- An isotropic noise term for the

        diagonal in the GP covariance, *K*. Larger values correspond

        to the expectation of greater noise in the observed data

        points. Default is 1e-10.

   fit(X, y)

      Fit the GP prior to the training data.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- A training dataset

           of *N* examples, each with dimensionality *M*.

         * **y** ("ndarray" of shape *(N, O)*) -- A collection of

           real-valued training targets for the examples in *X*, each

           with dimension *O*.

   predict(X, conf_interval=0.95, return_cov=False)

      Return the MAP estimate for y^*, corresponding the mean/mode of

      the posterior predictive distribution, p(y^* \mid x^*, X, y).

      -[ Notes ]-

      Under the GP regression model, the posterior predictive

      distribution is

         y^* \mid x^*, X, y \sim \mathcal{N}(\mu^*, \text{cov}^*)

      where

         \mu^*  &=  K^* (K + \alpha I)^{-1} y \\ \text{cov}^*  &=

         K^{**} - K^{*'} (K + \alpha I)^{-1} K^*

      and

         K  &=  \text{kernel}(X, X) \\ K^*  &=  \text{kernel}(X, X^*)

         \\ K^{**}  &=  \text{kernel}(X^*, X^*)

      NB. This implementation uses the inefficient but general purpose

      *np.linalg.inv* routine to invert (K + \alpha I). A more

      efficient way is to rely on the fact that *K* (and hence also K

      + \alpha I) is symmetric positive (semi-)definite and take the

      inner product of the inverse of its (lower) Cholesky

      decompositions:

         Q^{-1} = \text{cholesky}(Q)^{-1 \top} \text{cholesky}(Q)^{-1}

      For more details on a production-grade implementation, see

      Algorithm 2.1 in Rasmussen & Williams (2006).

      Parameters:

         * **X** ("ndarray" of shape (N, M)) -- The collection of

           datapoints to generate predictions on

         * **conf_interval** (*float in** (**0**, **1**)*) -- The

           percentage confidence bound to return for each prediction.

           If the scipy package is not available, this value is always

           set to 0.95. Default is 0.95.

         * **return_cov** (*bool*) -- If True, also return the

           covariance (*cov**) of the posterior predictive

           distribution for the points in *X*. Default is False.

      Returns:

         * **y_pred** ("ndarray" of shape *(N, O)*) -- The predicted

           values for each point in *X*, each with dimensionality *O*.

         * **conf** ("ndarray" of shape *(N, O)*) -- The %

           conf_interval confidence bound for each *y_pred*. The conf

           % confidence interval for the *i*'th prediction is "[y[i] -

           conf[i], y[i] + conf[i]]".

         * **cov** ("ndarray" of shape *(N, N)*) -- The covariance

           (*cov**) of the posterior predictive distribution for *X*.

           Only returned if *return_cov* is True.

   marginal_log_likelihood(kernel_params=None)

      Compute the log of the marginal likelihood (i.e., the log model

      evidence), p(y \mid X, \text{kernel_params}).

      -[ Notes ]-

      Under the GP regression model, the marginal likelihood is

      normally distributed:

         y | X, \theta  \sim  \mathcal{N}(0, K + \alpha I)

      Hence,

         \log p(y \mid X, \theta) =     -0.5 \log \det(K + \alpha I) -

         0.5 y^\top (K + \alpha I)^{-1} y + \frac{n}{2} \log 2 \pi

      where K = \text{kernel}(X, X), \theta is the set of kernel

      parameters, and *n* is the number of dimensions in *K*.

      Parameters:

         **kernel_params** (*dict*) -- Parameters for the kernel

         function. If None, calculate the marginal likelihood under

         the kernel parameters defined at model initialization.

         Default is None.

      Returns:

         **marginal_log_likelihood** (*float*) -- The log likelihood

         of the training targets given the kernel parameterized by

         *kernel_params* and the training inputs, marginalized over

         all functions *f*.

   sample(X, n_samples=1, dist='posterior_predictive')

      Sample functions from the GP prior or posterior predictive

      distribution.

      Parameters:

         * **X** ("ndarray" of shape *(N, M)*) -- The collection of

           datapoints to generate predictions on. Only used if *dist*

           = 'posterior_predictive'.

         * **n_samples** (*int*) -- The number of samples to generate.

           Default is 1.

         * **dist** (*{"posterior_predictive"**, **"prior"}*) -- The

           distribution to draw samples from. Default is

           "posterior_predictive".

      Returns:

         **samples** ("ndarray" of shape *(n_samples, O, N)*) -- The

         generated samples for the points in *X*.


CrossEntropy

"CrossEntropy"

**************

class numpy_ml.neural_nets.losses.CrossEntropy

   A cross-entropy loss.

   -[ Notes ]-

   For a one-hot target **y** and predicted class probabilities

   \hat{\mathbf{y}}, the cross entropy is

      \mathcal{L}(\mathbf{y}, \hat{\mathbf{y}})     = \sum_i y_i \log

      \hat{y}_i

   static loss(y, y_pred)

      Compute the cross-entropy (log) loss.

      -[ Notes ]-

      This method returns the sum (not the average!) of the losses for

      each sample.

      Parameters:

         * **y** ("ndarray" of shape (n, m)) -- Class labels (one-hot

           with *m* possible classes) for each of *n* examples.

         * **y_pred** ("ndarray" of shape (n, m)) -- Probabilities of

           each of *m* classes for the *n* examples in the batch.

      Returns:

         **loss** (*float*) -- The sum of the cross-entropy across

         classes and examples.

   static grad(y, y_pred)

      Compute the gradient of the cross entropy loss with regard to

      the softmax input, *z*.

      -[ Notes ]-

      The gradient for this method goes through both the cross-entropy

      loss AND the softmax non-linearity to return \frac{\partial

      \mathcal{L}}{\partial \mathbf{z}} (rather than \frac{\partial

      \mathcal{L}}{\partial \text{softmax}(\mathbf{z})}).

      In particular, let:

         \mathcal{L}(\mathbf{z})     =

         \text{cross_entropy}(\text{softmax}(\mathbf{z})).

      The current method computes:

         \frac{\partial \mathcal{L}}{\partial \mathbf{z}}     &=

         \text{softmax}(\mathbf{z}) - \mathbf{y} \\     &=

         \hat{\mathbf{y}} - \mathbf{y}

      Parameters:

         * **y** ("ndarray" of shape *(n, m)*) -- A one-hot encoding

           of the true class labels. Each row constitues a training

           example, and each column is a different class.

         * **y_pred** ("ndarray" of shape *(n, m)*) -- The network

           predictions for the probability of each of *m* class labels

           on each of *n* examples in a batch.

      Returns:

         **grad** ("ndarray" of shape (n, m)) -- The gradient of the

         cross-entropy loss with respect to the *input* to the softmax

         function.


RNNCell

"RNNCell"

*********

class numpy_ml.neural_nets.layers.RNNCell(n_out, act_fn='Tanh', init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   A single step of a vanilla (Elman) RNN.

   -[ Notes ]-

   At timestep *t*, the vanilla RNN cell computes

      \mathbf{Z}^{(t)}  &=     \mathbf{W}_{ax} \mathbf{X}^{(t)} +

      \mathbf{b}_{ax} +         \mathbf{W}_{aa} \mathbf{A}^{(t-1)} +

      \mathbf{b}_{aa} \\ \mathbf{A}^{(t)}  &=  f(\mathbf{Z}^{(t)})

   where

   * \mathbf{X}^{(t)} is the input at time *t*

   * \mathbf{A}^{(t)} is the hidden state at timestep *t*

   * *f* is the layer activation function

   * \mathbf{W}_{ax} and \mathbf{b}_{ax} are the weights and bias for

     the input to hidden layer

   * \mathbf{W}_{aa} and \mathbf{b}_{aa} are the weights and biases

     for the hidden to hidden layer

   Parameters:

      * **n_out** (*int*) -- The dimension of a single hidden state /

        output on a given timestep

      * **act_fn** (str, Activation object, or None) -- The activation

        function for computing "A[t]". Default is *'Tanh'*.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(Xt)

      Compute the network output for a single timestep.

      Parameters:

         **Xt** ("ndarray" of shape *(n_ex, n_in)*) -- Input at

         timestep *t* consisting of *n_ex* examples each of

         dimensionality *n_in*.

      Returns:

         **At** ("ndarray" of shape *(n_ex, n_out)*) -- The value of

         the hidden state at timestep *t* for each of the *n_ex*

         examples.

   backward(dLdAt)

      Backprop for a single timestep.

      Parameters:

         **dLdAt** ("ndarray" of shape *(n_ex, n_out)*) -- The

         gradient of the loss wrt. the layer outputs (ie., hidden

         states) at timestep *t*.

      Returns:

         **dLdXt** ("ndarray" of shape *(n_ex, n_in)*) -- The gradient

         of the loss wrt. the layer inputs at timestep *t*.

   flush_gradients()

      Erase all the layer's derived variables and gradients.


minkowski

"minkowski"

***********

numpy_ml.utils.distance_metrics.minkowski(x, y, p)

   Compute the Minkowski-*p* distance between two real vectors.

   -[ Notes ]-

   The Minkowski-*p* distance between two vectors **x** and **y** is

      d(\mathbf{x}, \mathbf{y}) = \left( \sum_i |x_i - y_i|^p

      \right)^{1/p}

   Parameters:

      * **x** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

      * **y** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

      * **p** (*float > 1*) -- The parameter of the distance function.

        When *p = 1*, this is the *L1* distance, and when *p=2*, this

        is the *L2* distance. For *p < 1*, Minkowski-*p* does not

        satisfy the triangle inequality and hence is not a valid

        distance metric.

   Returns:

      **d** (*float*) -- The Minkowski-*p* distance between **x** and

      **y**.


Embedding

"Embedding"

***********

class numpy_ml.neural_nets.layers.Embedding(n_out, vocab_size, pool=None, init='glorot_uniform', optimizer=None)

   Bases: "LayerBase"

   An embedding layer.

   -[ Notes ]-

   Equations:

      Y = W[x]

   NB. This layer must be the first in a neural network as the

   gradients do not get passed back through to the inputs.

   Parameters:

      * **n_out** (*int*) -- The dimensionality of the embeddings

      * **vocab_size** (*int*) -- The total number of items in the

        vocabulary. All integer indices are expected to range between

        0 and *vocab_size - 1*.

      * **pool** (*{'sum'**, **'mean'**, **None}*) -- If not None,

        apply this function to the collection of *n_in* encodings in

        each example to produce a single, pooled embedding. Default is

        None.

      * **init** (*{'glorot_normal'**, **'glorot_uniform'**,

        **'he_normal'**, **'he_uniform'}*) -- The weight

        initialization strategy. Default is *'glorot_uniform'*.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   lookup(ids)

      Return the embeddings associated with the IDs in *ids*.

      Parameters:

         **word_ids** ("ndarray" of shape (*M*,)) -- An array of *M*

         IDs to retrieve embeddings for.

      Returns:

         **embeddings** ("ndarray" of shape (*M*, *n_out*)) -- The

         embedding vectors for each of the *M* IDs.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      -[ Notes ]-

      Equations:

         Y = W[x]

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, n_in)* or list of length

           *n_ex*) -- Layer input, representing a minibatch of *n_ex*

           examples. If "self.pool" is None, each example must consist

           of exactly *n_in* integer token IDs. Otherwise, *X* can be

           a ragged array, with each example consisting of a variable

           number of token IDs.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through with regard to this input.

           Default is True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_in, n_out)*) --

         Embeddings for each coordinate of each of the *n_ex* examples

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to embedding weights.

      -[ Notes ]-

      Because the items in *X* are interpreted as indices, we cannot

      compute the gradient of the layer output wrt. *X*.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, n_in, n_out)* or list

           of arrays) -- The gradient(s) of the loss wrt. the layer

           output(s)

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.


LayerNorm2D

"LayerNorm2D"

*************

class numpy_ml.neural_nets.layers.LayerNorm2D(epsilon=1e-05, optimizer=None)

   Bases: "LayerBase"

   A layer normalization layer for 2D inputs with an additional

   channel dimension.

   -[ Notes ]-

   In contrast to "BatchNorm2D", the LayerNorm layer calculates the

   mean and variance across *features* rather than examples in the

   batch ensuring that the mean and variance estimates are independent

   of batch size and permitting straightforward application in RNNs.

   Equations [train & test]:

      Y = scaler * norm(X) + intercept

      norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

   Also in contrast to "BatchNorm2D", *scaler* and *intercept* are

   applied *elementwise* to "norm(X)".

   Parameters:

      * **epsilon** (*float*) -- A small smoothing constant to use

        during computation of "norm(X)" to avoid divide-by-zero

        errors. Default is 1e-5.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      -[ Notes ]-

      Equations [train & test]:

         Y = scaler * norm(X) + intercept

         norm(X) = (X - mean(X)) / sqrt(var(X) + epsilon)

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- Input volume containing the *in_rows* by

           *in_cols*-dimensional features for a minibatch of *n_ex*

           examples.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- Layer output for each of the *n_ex* examples.

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdY** ("ndarray" of shape *(n_ex, in_rows, in_cols,

           in_ch)*) -- The gradient of the loss wrt. the layer output

           *Y*.

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dX** ("ndarray" of shape *(n_ex, in_rows, in_cols, in_ch)*)

         -- The gradient of the loss wrt. the layer input *X*.


SELU

"SELU"

******

class numpy_ml.neural_nets.activations.SELU

   A scaled exponential linear unit (SELU).

   -[ Notes ]-

   SELU units, when used in conjunction with proper weight

   initialization and regularization techniques, encourage neuron

   activations to converge to zero-mean and unit variance without

   explicit use of e.g., batchnorm.

   For SELU units, the \alpha and \text{scale} values are constants

   chosen so that the mean and variance of the inputs are preserved

   between consecutive layers. As such the authors propose weights be

   initialized using Lecun-Normal initialization: w_{ij} \sim

   \mathcal{N}(0, 1 / \text{fan_in}), and to use the dropout variant

   \alpha-dropout during regularization. [*]

   See the reference for more information (especially the appendix ;-)

   ).

   -[ References ]-

   [*] Klambauer, G., Unterthiner, T., & Hochreiter, S. (2017). "Self-

       normalizing neural networks." *Advances in Neural Information

       Processing Systems, 30.*

   Initialize the ActivationBase object

   fn(z)

      Evaluate the SELU activation on the elements of input *z*.

         \text{SELU}(z_i)  =  \text{scale} \times \text{ELU}(z_i,

         \alpha)

      which is simply

         \text{SELU}(z_i)     &= \text{scale} \times z_i \ \ \ \

         &&\text{if }z_i > 0 \\     &= \text{scale} \times \alpha

         (e^{z_i} - 1) \ \ \ \ &&\text{otherwise}

   grad(x)

      Evaluate the first derivative of the SELU activation on the

      elements of input *x*.

         \frac{\partial \text{SELU}}{\partial x_i}     &=

         \text{scale} \ \ \ \ &&\text{if } x_i > 0 \\     &=

         \text{scale} \times \alpha e^{x_i} \ \ \ \ &&\text{otherwise}

   grad2(x)

      Evaluate the second derivative of the SELU activation on the

      elements of input *x*.

         \frac{\partial^2 \text{SELU}}{\partial x_i^2}     &=  0 \ \ \

         \ &&\text{if } x_i > 0 \\     &=  \text{scale} \times \alpha

         e^{x_i} \ \ \ \ &&\text{otherwise}


Softmax

"Softmax"

*********

class numpy_ml.neural_nets.layers.Softmax(dim=-1, optimizer=None)

   Bases: "LayerBase"

   A softmax nonlinearity layer.

   -[ Notes ]-

   This is implemented as a layer rather than an activation primarily

   because it requires retaining the layer input in order to compute

   the softmax gradients properly. In other words, in contrast to

   other simple activations, the softmax function and its gradient are

   not computed elementwise, and thus are more easily expressed as a

   layer.

   The softmax function computes:

      y_i = \frac{e^{x_i}}{\sum_j e^{x_j}}

   where x_i is the *i* th element of input example **x**.

   Parameters:

      * **dim** (*int*) -- The dimension in *X* along which the

        softmax will be computed. Default is -1.

      * **optimizer** (str, Optimizer object, or None) -- The

        optimization strategy to use when performing gradient updates

        within the "update()" method.  If None, use the "SGD"

        optimizer with default parameters. Default is None. Unused for

        this layer.

   Variables:

      * **X** (*list*) -- Running list of inputs to the "forward"

        method since the last call to "update". Only updated if the

        *retain_derived* argument was set to True.

      * **gradients** (*dict*) -- Dictionary of loss gradients with

        regard to the layer parameters

      * **parameters** (*dict*) -- Dictionary of layer parameters

      * **hyperparameters** (*dict*) -- Dictionary of layer

        hyperparameters

      * **derived_variables** (*dict*) -- Dictionary of any

        intermediate values computed during forward/backward

        propagation.

   property hyperparameters

      Return a dictionary containing the layer hyperparameters.

   forward(X, retain_derived=True)

      Compute the layer output on a single minibatch.

      Parameters:

         * **X** ("ndarray" of shape *(n_ex, n_in)*) -- Layer input,

           representing the *n_in*-dimensional features for a

           minibatch of *n_ex* examples.

         * **retain_derived** (*bool*) -- Whether to retain the

           variables calculated during the forward pass for use later

           during backprop. If False, this suggests the layer will not

           be expected to backprop through wrt. this input. Default is

           True.

      Returns:

         **Y** ("ndarray" of shape *(n_ex, n_out)*) -- Layer output

         for each of the *n_ex* examples.

   backward(dLdy, retain_grads=True)

      Backprop from layer outputs to inputs.

      Parameters:

         * **dLdy** ("ndarray" of shape *(n_ex, n_out)* or list of

           arrays) -- The gradient(s) of the loss wrt. the layer

           output(s).

         * **retain_grads** (*bool*) -- Whether to include the

           intermediate parameter gradients computed during the

           backward pass in the final parameter update. Default is

           True.

      Returns:

         **dLdX** ("ndarray" of shape *(n_ex, n_in)*) -- The gradient

         of the loss wrt. the layer input *X*.


euclidean

"euclidean"

***********

numpy_ml.utils.distance_metrics.euclidean(x, y)

   Compute the Euclidean (*L2*) distance between two real vectors

   -[ Notes ]-

   The Euclidean distance between two vectors **x** and **y** is

      d(\mathbf{x}, \mathbf{y}) = \sqrt{ \sum_i (x_i - y_i)^2  }

   Parameters:

      * **x** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

      * **y** ("ndarray" s of shape *(N,)*) -- The two vectors to

        compute the distance between

   Returns:

      **d** (*float*) -- The L2 distance between **x** and **y**.