Instruction:
How would you code a function in Python 3 to
model selection for optimal variational hyperparameters. Given the full training set (parameterized by `observations` and `observation_index_points`), compute the optimal variational location and scale for the VGP. This is based of the method suggested in [Titsias, 2009][1]. Args: kernel: `PositiveSemidefiniteKernel`-like instance representing the GP's covariance function. inducing_index_points: `float` `Tensor` of locations of inducing points in the index set. Shape has the form `[b1, ..., bB, e2, f1, ..., fF]`, just like `observation_index_points`. The batch shape components needn't be identical to those of `observation_index_points`, but must be broadcast compatible with them. observation_index_points: `float` `Tensor` representing finite (batch of) vector(s) of points where observations are defined. Shape has the form `[b1, ..., bB, e1, f1, ..., fF]` where `F` is the number of feature dimensions and must equal `kernel.feature_ndims` and `e1` is the number (size) of index points in each batch (we denote it `e1` to distinguish it from the numer of inducing index points, denoted `e2` below). observations: `float` `Tensor` representing collection, or batch of collections, of observations corresponding to `observation_index_points`. Shape has the form `[b1, ..., bB, e]`, which must be brodcastable with the batch and example shapes of `observation_index_points`. The batch shape `[b1, ..., bB]` must be broadcastable with the shapes of all other batched parameters (`kernel.batch_shape`, `observation_index_points`, etc.). observation_noise_variance: `float` `Tensor` representing the variance of the noise in the Normal likelihood distribution of the model. May be batched, in which case the batch shape must be broadcastable with the shapes of all other batched parameters (`kernel.batch_shape`, `index_points`, etc.). Default value: `0.` mean_fn: Python `callable` that acts on index points to produce a (batch of) vector(s) of mean values at those index points. Takes a `Tensor` of shape `[b1, ..., bB, f1, ..., fF]` and returns a `Tensor` whose shape is (broadcastable with) `[b1, ..., bB]`. Default value: `None` implies constant zero function. jitter: `float` scalar `Tensor` added to the diagonal of the covariance matrix to ensure positive definiteness of the covariance matrix. Default value: `1e-6`. name: Python `str` name prefixed to Ops created by this class. Default value: "optimal_variational_posterior". Returns: loc, scale: Tuple representing the variational location and scale. Raises: ValueError: if `mean_fn` is not `None` and is not callable. #### References [1]: Titsias, M. "Variational Model Selection for Sparse Gaussian Process Regression", 2009. http://proceedings.mlr.press/v5/titsias09a/titsias09a.pdf

Response:
def optimal_variational_posterior(
      kernel,
      inducing_index_points,
      observation_index_points,
      observations,
      observation_noise_variance,
      mean_fn=None,
      jitter=1e-6,
      name=None):
    """Model selection for optimal variational hyperparameters.

    Given the full training set (parameterized by `observations` and
    `observation_index_points`), compute the optimal variational
    location and scale for the VGP. This is based of the method suggested
    in [Titsias, 2009][1].

    Args:
      kernel: `PositiveSemidefiniteKernel`-like instance representing the
        GP's covariance function.
      inducing_index_points: `float` `Tensor` of locations of inducing points in
        the index set. Shape has the form `[b1, ..., bB, e2, f1, ..., fF]`, just
        like `observation_index_points`. The batch shape components needn't be
        identical to those of `observation_index_points`, but must be broadcast
        compatible with them.
      observation_index_points: `float` `Tensor` representing finite (batch of)
        vector(s) of points where observations are defined. Shape has the
        form `[b1, ..., bB, e1, f1, ..., fF]` where `F` is the number of feature
        dimensions and must equal `kernel.feature_ndims` and `e1` is the number
        (size) of index points in each batch (we denote it `e1` to distinguish
        it from the numer of inducing index points, denoted `e2` below).
      observations: `float` `Tensor` representing collection, or batch of
        collections, of observations corresponding to
        `observation_index_points`. Shape has the form `[b1, ..., bB, e]`, which
        must be brodcastable with the batch and example shapes of
        `observation_index_points`. The batch shape `[b1, ..., bB]` must be
        broadcastable with the shapes of all other batched parameters
        (`kernel.batch_shape`, `observation_index_points`, etc.).
      observation_noise_variance: `float` `Tensor` representing the variance
        of the noise in the Normal likelihood distribution of the model. May be
        batched, in which case the batch shape must be broadcastable with the
        shapes of all other batched parameters (`kernel.batch_shape`,
        `index_points`, etc.).
        Default value: `0.`
      mean_fn: Python `callable` that acts on index points to produce a (batch
        of) vector(s) of mean values at those index points. Takes a `Tensor` of
        shape `[b1, ..., bB, f1, ..., fF]` and returns a `Tensor` whose shape is
        (broadcastable with) `[b1, ..., bB]`. Default value: `None` implies
        constant zero function.
      jitter: `float` scalar `Tensor` added to the diagonal of the covariance
        matrix to ensure positive definiteness of the covariance matrix.
        Default value: `1e-6`.
      name: Python `str` name prefixed to Ops created by this class.
        Default value: "optimal_variational_posterior".
    Returns:
      loc, scale: Tuple representing the variational location and scale.
    Raises:
      ValueError: if `mean_fn` is not `None` and is not callable.

    #### References

    [1]: Titsias, M. "Variational Model Selection for Sparse Gaussian Process
         Regression", 2009.
         http://proceedings.mlr.press/v5/titsias09a/titsias09a.pdf
    """

    with tf.name_scope(name or 'optimal_variational_posterior'):
      dtype = dtype_util.common_dtype(
          [inducing_index_points,
           observation_index_points,
           observations,
           observation_noise_variance,
           jitter], tf.float32)

      inducing_index_points = tf.convert_to_tensor(
          value=inducing_index_points,
          dtype=dtype, name='inducing_index_points')
      observation_index_points = tf.convert_to_tensor(
          value=observation_index_points, dtype=dtype,
          name='observation_index_points')
      observations = tf.convert_to_tensor(
          value=observations, dtype=dtype, name='observations')
      observation_noise_variance = tf.convert_to_tensor(
          value=observation_noise_variance,
          dtype=dtype,
          name='observation_noise_variance')
      jitter = tf.convert_to_tensor(
          value=jitter, dtype=dtype, name='jitter')

      # Default to a constant zero function.
      if mean_fn is None:
        mean_fn = lambda x: tf.zeros([1], dtype=dtype)
      else:
        if not callable(mean_fn):
          raise ValueError('`mean_fn` must be a Python callable')

      # z are the inducing points and x are the observation index points.
      kzz = kernel.matrix(inducing_index_points, inducing_index_points)
      kzx = kernel.matrix(inducing_index_points, observation_index_points)

      noise_var_inv = tf.math.reciprocal(observation_noise_variance)

      sigma_inv = _add_diagonal_shift(
          kzz + noise_var_inv * tf.matmul(kzx, kzx, adjoint_b=True),
          jitter)

      chol_sigma_inv = tf.linalg.cholesky(sigma_inv)

      kzx_lin_op = tf.linalg.LinearOperatorFullMatrix(kzx)
      kzx_obs = kzx_lin_op.matvec(
          observations - mean_fn(observation_index_points))
      kzz_lin_op = tf.linalg.LinearOperatorFullMatrix(kzz)
      loc = (mean_fn(inducing_index_points) +
             noise_var_inv * kzz_lin_op.matvec(
                 _solve_cholesky_factored_system_vec(chol_sigma_inv, kzx_obs)))

      chol_sigma_inv_lin_op = tf.linalg.LinearOperatorLowerTriangular(
          chol_sigma_inv)
      scale = chol_sigma_inv_lin_op.solve(kzz)

      return loc, scale


Instruction:
Can you create a Python 3 function that
builds utility method to compute whether the season is changing.

Response:
def build_is_last_day_of_season(num_steps_per_season):
  """Build utility method to compute whether the season is changing."""
  num_steps_per_cycle = np.sum(num_steps_per_season)
  changepoints = np.cumsum(np.ravel(num_steps_per_season)) - 1
  def is_last_day_of_season(t):
    t_ = dist_util.maybe_get_static_value(t)
    if t_ is not None:  # static case
      step_in_cycle = t_ % num_steps_per_cycle
      return any(step_in_cycle == changepoints)
    else:
      step_in_cycle = tf.math.floormod(t, num_steps_per_cycle)
      return tf.reduce_any(
          input_tensor=tf.equal(step_in_cycle, changepoints))
  return is_last_day_of_season


Instruction:
Given the following Python 3 function, write the documentation
def build_effects_to_residuals_matrix(num_seasons, dtype):

  # Build the matrix that converts effects `e_i` into differences from the mean
  # effect `(e_i - sum(e_i)) / num_seasons`, with the mean effect in the last
  # row so that the transformation is invertible.
  effects_to_residuals_fullrank = np.eye(num_seasons) - 1./num_seasons
  effects_to_residuals_fullrank[-1, :] = 1./num_seasons  # compute mean effect
  residuals_to_effects_fullrank = np.linalg.inv(effects_to_residuals_fullrank)

  # Drop the final dimension, effectively setting the mean effect to zero.
  effects_to_residuals = effects_to_residuals_fullrank[:-1, :]
  residuals_to_effects = residuals_to_effects_fullrank[:, :-1]

  # Return Tensor values of the specified dtype.
  effects_to_residuals = tf.cast(
      effects_to_residuals, dtype=dtype, name='effects_to_residuals')
  residuals_to_effects = tf.cast(
      residuals_to_effects, dtype=dtype, name='residuals_to_effects')

  return effects_to_residuals, residuals_to_effects

Response: Builds the change - of - basis matrices for constrained seasonal effects.


Instruction:
Write a Python 3 function that can
build a function computing the transition matrix for a seasonal effect model.

Response:
def build_seasonal_transition_matrix(
    num_seasons, is_last_day_of_season, dtype,
    basis_change_matrix=None, basis_change_matrix_inv=None):
  """Build a function computing transitions for a seasonal effect model."""

  with tf.compat.v1.name_scope('build_seasonal_transition_matrix'):
    # If the season is changing, the transition matrix permutes the latent
    # state to shift all seasons up by a dimension, and sends the current
    # season's effect to the bottom.
    seasonal_permutation = np.concatenate(
        [np.arange(1, num_seasons), [0]], axis=0)
    seasonal_permutation_matrix = tf.constant(
        np.eye(num_seasons)[seasonal_permutation], dtype=dtype)

    # Optionally transform the transition matrix into a reparameterized space,
    # enforcing the zero-sum constraint for ConstrainedSeasonalStateSpaceModel.
    if basis_change_matrix is not None:
      seasonal_permutation_matrix = tf.matmul(
          basis_change_matrix,
          tf.matmul(seasonal_permutation_matrix, basis_change_matrix_inv))

    identity_matrix = tf.eye(
        tf.shape(input=seasonal_permutation_matrix)[-1], dtype=dtype)

    def seasonal_transition_matrix(t):
      return tf.linalg.LinearOperatorFullMatrix(
          matrix=dist_util.pick_scalar_condition(
              is_last_day_of_season(t),
              seasonal_permutation_matrix,
              identity_matrix))

  return seasonal_transition_matrix


Instruction:
Here you have a function in Python 3, explain what it does
def build_seasonal_transition_noise(
    drift_scale, num_seasons, is_last_day_of_season):
  """Build the transition noise model for a SeasonalStateSpaceModel."""

  # If the current season has just ended, increase the variance of its effect
  # following drift_scale. (the just-ended seasonal effect will always be the
  # bottom element of the vector). Otherwise, do nothing.
  drift_scale_diag = tf.stack(
      [tf.zeros_like(drift_scale)] * (num_seasons - 1) + [drift_scale],
      axis=-1)
  def seasonal_transition_noise(t):
    noise_scale_diag = dist_util.pick_scalar_condition(
        is_last_day_of_season(t),
        drift_scale_diag,
        tf.zeros_like(drift_scale_diag))
    return tfd.MultivariateNormalDiag(
        loc=tf.zeros(num_seasons, dtype=drift_scale.dtype),
        scale_diag=noise_scale_diag)
  return seasonal_transition_noise

Response: Builds the transition noise model for a SeasonalStateSpaceModel.


Instruction:
Can you generate the documentation for the following Python 3 function
def build_constrained_seasonal_transition_noise(
    drift_scale, num_seasons, is_last_day_of_season):
  """Build transition noise distribution for a ConstrainedSeasonalSSM."""

  # Conceptually, this method takes the noise covariance on effects L @ L'
  # computed by `build_seasonal_transition_noise`, with scale factor
  #       L = [ 0, 0, ..., 0
  #             ...
  #             0, 0, ..., drift_scale],
  # and transforms it to act on the constrained-residual representation.
  #
  # The resulting noise covariance M @ M' is equivalent to
  #    M @ M' = effects_to_residuals @ LL' @ residuals_to_effects
  # where `@` is matrix multiplication. However because this matrix is
  # rank-deficient, we can't take its Cholesky decomposition directly, so we'll
  # construct its lower-triangular scale factor `M` by hand instead.
  #
  # Concretely, let `M = P @ R @ L` be the scale factor in the
  # transformed space, with matrices `R`, `P` applying the reparameterization
  # and zero-mean constraint respectively as defined in the
  # "Mathematical Details" section of `ConstrainedSeasonalStateSpaceModel`. It's
  # easy to see (*) that the implied covariance
  # `M @ M' = P @ R @ L @ L' @ R' @ P'` is just the constant matrix
  #  `M @ M' = [ 1, 1, ..., 1, 0
  #              1, 1, ..., 1, 0
  #              ...
  #              1, 1, ..., 1, 0
  #              0, 0, ..., 0, 0] * (drift_scale / num_seasons)**2`
  # with zeros in the final row and column. So we can directly construct
  # the lower-triangular factor
  #  `Q = [ 1, 0, ...  0
  #         1, 0, ..., 0
  #         ...
  #         1, 0, ..., 0
  #         0, 0, ..., 0 ] * drift_scale/num_seasons`
  # such that Q @ Q' = M @ M'. In practice, we don't reify the final row and
  # column full of zeroes, i.e., we construct
  # `Q[:num_seasons-1, :num_seasons-1]` as the scale-TriL covariance factor.
  #
  # (*) Argument: `L` is zero everywhere but the last column, so `R @ L` will be
  # too. Since the last column of `R` is the constant `-1/num_seasons`, `R @ L`
  # is simply the matrix with constant `-drift_scale/num_seasons` in the final
  # column (except the final row, which is negated) and zero in all other
  # columns, and `M = P @ R @ L` additionally zeroes out the final row. Then
  # M @ M' is just the outer product of that final column with itself (since all
  # other columns are zero), which gives the matrix shown above.

  drift_scale_tril_nonzeros = tf.concat([
      tf.ones([num_seasons - 1, 1], dtype=drift_scale.dtype),
      tf.zeros([num_seasons - 1, num_seasons - 2], dtype=drift_scale.dtype)],
                                        axis=-1)
  drift_scale_tril = (drift_scale_tril_nonzeros *
                      drift_scale[..., tf.newaxis, tf.newaxis] / num_seasons)

  # Inject transition noise iff it is the last day of the season.
  def seasonal_transition_noise(t):
    noise_scale_tril = dist_util.pick_scalar_condition(
        is_last_day_of_season(t),
        drift_scale_tril,
        tf.zeros_like(drift_scale_tril))
    return tfd.MultivariateNormalTriL(
        loc=tf.zeros(num_seasons-1, dtype=drift_scale.dtype),
        scale_tril=noise_scale_tril)
  return seasonal_transition_noise

Response: Builds a constrained Seasonal State Space Model transition noise distribution for a Seasonal State Space model.


Instruction:
Here you have a function in Python 3, explain what it does
def _is_empty_observation_data(
    feature_ndims, observation_index_points, observations):
  """Returns `True` if given observation data is empty.

  Emptiness means either
    1. Both `observation_index_points` and `observations` are `None`, or
    2. the "number of observations" shape is 0. The shape of
    `observation_index_points` is `[..., N, f1, ..., fF]`, where `N` is the
    number of observations and the `f`s are feature dims. Thus, we look at the
    shape element just to the left of the leftmost feature dim. If that shape is
    zero, we consider the data empty.

  We don't check the shape of observations; validations are checked elsewhere in
  the calling code, to ensure these shapes are consistent.

  Args:
    feature_ndims: the number of feature dims, as reported by the GP kernel.
    observation_index_points: the observation data locations in the index set.
    observations: the observation data.

  Returns:
    is_empty: True if the data were deemed to be empty.
  """
  # If both input locations and observations are `None`, we consider this
  # "empty" observation data.
  if observation_index_points is None and observations is None:
    return True
  num_obs = tf.compat.dimension_value(
      observation_index_points.shape[-(feature_ndims + 1)])
  if num_obs is not None and num_obs == 0:
    return True
  return False

Response: Returns True if given observation data is empty.


Instruction:
Can you write a function in Python 3 where it
checks that the observation data and locations have consistent shapes.

Response:
def _validate_observation_data(
    kernel, observation_index_points, observations):
  """Ensure that observation data and locations have consistent shapes.

  This basically means that the batch shapes are broadcastable. We can only
  ensure this when those shapes are fully statically defined.

  Args:
    kernel: The GP kernel.
    observation_index_points: the observation data locations in the index set.
    observations: the observation data.

  Raises:
    ValueError: if the observations' batch shapes are not broadcastable.
  """
  # Check that observation index points and observation counts broadcast.
  ndims = kernel.feature_ndims
  if (tensorshape_util.is_fully_defined(
      observation_index_points.shape[:-ndims]) and
      tensorshape_util.is_fully_defined(observations.shape)):
    index_point_count = observation_index_points.shape[:-ndims]
    observation_count = observations.shape
    try:
      tf.broadcast_static_shape(index_point_count, observation_count)
    except ValueError:
      # Re-raise with our own more contextual error message.
      raise ValueError(
          'Observation index point and observation counts are not '
          'broadcastable: {} and {}, respectively.'.format(
              index_point_count, observation_count))


Instruction:
How would you explain what the following Python 3 function does
def _kl_gamma_gamma(g0, g1, name=None):
  with tf.name_scope(name or "kl_gamma_gamma"):
    # Result from:
    #   http://www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps
    # For derivation see:
    #   http://stats.stackexchange.com/questions/11646/kullback-leibler-divergence-between-two-gamma-distributions   pylint: disable=line-too-long
    return (((g0.concentration - g1.concentration) *
             tf.math.digamma(g0.concentration)) +
            tf.math.lgamma(g1.concentration) -
            tf.math.lgamma(g0.concentration) +
            g1.concentration * tf.math.log(g0.rate) -
            g1.concentration * tf.math.log(g1.rate) + g0.concentration *
            (g1.rate / g0.rate - 1.))

Response: Calculate the batched KL divergence between two Gamma distributions.


Instruction:
Write a Python 3 function for
adding a learning rate scheduler to the contained schedules

Response:
def add(self, scheduler, max_iteration, bigdl_type="float"):
        """
        Add a learning rate scheduler to the contained `schedules`

        :param scheduler: learning rate scheduler to be add
        :param max_iteration: iteration numbers this scheduler will run
        """
        return callBigDlFunc(bigdl_type, "addScheduler", self.value, scheduler, max_iteration)


Instruction:
Create a Python 3 function for
saving the current object to a file.

Response:
def save(self, path, overWrite):
        """
        save OptimMethod
        :param path      path
        :param overWrite whether to overwrite
        """
        method=self.value
        return callBigDlFunc(self.bigdl_type, "saveOptimMethod", method, path, overWrite)


Instruction:
Make a summary of the following Python 3 code
def set_checkpoint(self, checkpoint_trigger,
                       checkpoint_path, isOverWrite=True):
        """
        Configure checkpoint settings.

        :param checkpoint_trigger: the interval to write snapshots
        :param checkpoint_path: the path to write snapshots into
        :param isOverWrite: whether to overwrite existing snapshots in path.default is True
        """
        if not os.path.exists(checkpoint_path):
            mkpath(checkpoint_path)
        callBigDlFunc(self.bigdl_type, "setCheckPoint", self.value,
                      checkpoint_trigger, checkpoint_path, isOverWrite)

Response:
Configure checkpoint settings.

        :param checkpoint_trigger: the interval to write snapshots
        :param checkpoint_path: the path to write snapshots into
        :param isOverWrite: whether to overwrite existing snapshots in path.default is True


Instruction:
Write a Python 3 function for
configuring constant clipping settings.

Response:
def set_gradclip_const(self, min_value, max_value):
        """
        Configure constant clipping settings.

        :param min_value: the minimum value to clip by
        :param max_value: the maxmimum value to clip by
        """
        callBigDlFunc(self.bigdl_type, "setConstantClip", self.value, min_value, max_value)


Instruction:
Given the following Python 3 function, write the documentation
def set_train_summary(self, summary):
        callBigDlFunc(self.bigdl_type, "setTrainSummary", self.value,
                      summary)
        return self

Response: Set the train summary.


Instruction:
Write a Python 3 function that can
set the validation summary for the current object.

Response:
def set_val_summary(self, summary):
        """
        Set validation summary. A ValidationSummary object contains information
        necessary for the optimizer to know how often the logs are recorded,
        where to store the logs and how to retrieve them, etc. For details,
        refer to the docs of ValidationSummary.

        :param summary: a ValidationSummary object

        """
        callBigDlFunc(self.bigdl_type, "setValSummary", self.value,
                      summary)
        return self


Instruction:
Can you generate a brief explanation for the following Python 3 code
def create(model,
               training_set,
               criterion,
               end_trigger=None,
               batch_size=32,
               optim_method=None,
               cores=None,
               bigdl_type="float"):
        """
        Create an optimizer.
        Depend on the input type, the returning optimizer can be a local optimizer \
        or a distributed optimizer.

        :param model: the neural net model
        :param training_set: (features, label) for local mode. RDD[Sample] for distributed mode.
        :param criterion: the loss function
        :param optim_method: the algorithm to use for optimization,
           e.g. SGD, Adagrad, etc. If optim_method is None, the default algorithm is SGD.
        :param end_trigger: when to end the optimization. default value is MapEpoch(1)
        :param batch_size: training batch size
        :param cores: This is for local optimizer only and use total physical cores as the default value
        """
        if not end_trigger:
            end_trigger = MaxEpoch(1)
        if not optim_method:
            optim_method = SGD()
        if isinstance(training_set, RDD) or isinstance(training_set, DataSet):
            return DistriOptimizer(model=model,
                                   training_rdd=training_set,
                                   criterion=criterion,
                                   end_trigger=end_trigger,
                                   batch_size=batch_size,
                                   optim_method=optim_method,
                                   bigdl_type=bigdl_type)
        elif isinstance(training_set, tuple) and len(training_set) == 2:
            x, y = training_set
            return LocalOptimizer(X=x,
                                  Y=y,
                                  model=model,
                                  criterion=criterion,
                                  end_trigger=end_trigger,
                                  batch_size=batch_size,
                                  optim_method=optim_method,
                                  cores=cores,
                                  bigdl_type="float")
        else:
            raise Exception("Not supported training set: %s" % type(training_set))

Response: Create an optimizer object.


Instruction:
Write a Python 3 function that can
configure the validation settings for the current object.

Response:
def set_validation(self, batch_size, val_rdd, trigger, val_method=None):
        """
        Configure validation settings.

        :param batch_size: validation batch size
        :param val_rdd: validation dataset
        :param trigger: validation interval
        :param val_method: the ValidationMethod to use,e.g. "Top1Accuracy", "Top5Accuracy", "Loss"
        """
        if val_method is None:
            val_method = [Top1Accuracy()]
        func_name = "setValidation"
        if isinstance(val_rdd, DataSet):
            func_name = "setValidationFromDataSet"
        callBigDlFunc(self.bigdl_type, func_name, self.value, batch_size,
                      trigger, val_rdd, to_list(val_method))


Instruction:
Can you create a Python 3 function that
sets new training dataset for optimizer reuse the training dataset

Response:
def set_traindata(self, training_rdd, batch_size):
        """
        Set new training dataset, for optimizer reuse

        :param training_rdd: the training dataset
        :param batch_size: training batch size
        :return:
        """
        callBigDlFunc(self.bigdl_type, "setTrainData", self.value,
                     training_rdd, batch_size)


Instruction:
Implement a function in Python 3 to
configure the validation settings for the current log - likelihood dataset.

Response:
def set_validation(self, batch_size, X_val, Y_val, trigger, val_method=None):
        """
        Configure validation settings.

        :param batch_size: validation batch size
        :param X_val: features of validation dataset
        :param Y_val: label of validation dataset
        :param trigger: validation interval
        :param val_method: the ValidationMethod to use,e.g. "Top1Accuracy", "Top5Accuracy", "Loss"
        """
        if val_method is None:
            val_method = [Top1Accuracy()]
        callBigDlFunc(self.bigdl_type, "setValidation", self.value, batch_size,
                      trigger, [JTensor.from_ndarray(X) for X in to_list(X_val)],
                      JTensor.from_ndarray(Y_val), to_list(val_method))


Instruction:
Can you implement a function in Python 3 that
sets the interval of recording for each indicator.

Response:
def set_summary_trigger(self, name, trigger):
        """
        Set the interval of recording for each indicator.

        :param tag: tag name. Supported tag names are "LearningRate", "Loss","Throughput", "Parameters". "Parameters" is an umbrella tag thatincludes weight, bias, gradWeight, gradBias, and some running status(eg. runningMean and runningVar in BatchNormalization). If youdidn't set any triggers, we will by default record Loss and Throughputin each iteration, while *NOT* recording LearningRate and Parameters,as recording parameters may introduce substantial overhead when themodel is very big, LearningRate is not a public attribute for allOptimMethod.
        :param trigger: trigger
        """
        return callBigDlFunc(self.bigdl_type, "summarySetTrigger", self.value,
                             name, trigger)


Instruction:
How would you explain what the following Python 3 function does
def read_data_sets(train_dir, data_type="train"):
    TRAIN_IMAGES = 'train-images-idx3-ubyte.gz'
    TRAIN_LABELS = 'train-labels-idx1-ubyte.gz'
    TEST_IMAGES = 't10k-images-idx3-ubyte.gz'
    TEST_LABELS = 't10k-labels-idx1-ubyte.gz'

    if data_type == "train":
        local_file = base.maybe_download(TRAIN_IMAGES, train_dir,
                                         SOURCE_URL + TRAIN_IMAGES)
        with open(local_file, 'rb') as f:
            train_images = extract_images(f)

        local_file = base.maybe_download(TRAIN_LABELS, train_dir,
                                         SOURCE_URL + TRAIN_LABELS)
        with open(local_file, 'rb') as f:
            train_labels = extract_labels(f)
        return train_images, train_labels

    else:
        local_file = base.maybe_download(TEST_IMAGES, train_dir,
                                         SOURCE_URL + TEST_IMAGES)
        with open(local_file, 'rb') as f:
            test_images = extract_images(f)

        local_file = base.maybe_download(TEST_LABELS, train_dir,
                                         SOURCE_URL + TEST_LABELS)
        with open(local_file, 'rb') as f:
            test_labels = extract_labels(f)
        return test_images, test_labels

Response: Parse or download mnist data sets.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_news20(source_dir="./data/news20/"):
    news_dir = download_news20(source_dir)
    texts = []  # list of text samples
    label_id = 0
    for name in sorted(os.listdir(news_dir)):
        path = os.path.join(news_dir, name)
        label_id += 1
        if os.path.isdir(path):
            for fname in sorted(os.listdir(path)):
                if fname.isdigit():
                    fpath = os.path.join(path, fname)
                    if sys.version_info < (3,):
                        f = open(fpath)
                    else:
                        f = open(fpath, encoding='latin-1')
                    content = f.read()
                    texts.append((content, label_id))
                    f.close()

    print('Found %s texts.' % len(texts))
    return texts

Response: Parse or download news20 and return a list of tokens and labels.


Instruction:
Implement a function in Python 3 to
parse or download the pre - trained glove word2vec file and return a dict mapping from word to vector.

Response:
def get_glove_w2v(source_dir="./data/news20/", dim=100):
    """
    Parse or download the pre-trained glove word2vec if source_dir is empty.

    :param source_dir: The directory storing the pre-trained word2vec
    :param dim: The dimension of a vector
    :return: A dict mapping from word to vector
    """
    w2v_dir = download_glove_w2v(source_dir)
    w2v_path = os.path.join(w2v_dir, "glove.6B.%sd.txt" % dim)
    if sys.version_info < (3,):
        w2v_f = open(w2v_path)
    else:
        w2v_f = open(w2v_path, encoding='latin-1')
    pre_w2v = {}
    for line in w2v_f.readlines():
        items = line.split(" ")
        pre_w2v[items[0]] = [float(i) for i in items[1:]]
    w2v_f.close()
    return pre_w2v


Instruction:
How would you code a function in Python 3 to
compile the learning process.

Response:
def compile(self, optimizer, loss, metrics=None):
        """
        Configures the learning process. Must be called before fit or evaluate.

        # Arguments
        optimizer: Optimization method to be used. One can alternatively pass in the corresponding
                   string representation, such as 'sgd'.
        loss: Criterion to be used. One can alternatively pass in the corresponding string
              representation, such as 'mse'.
        metrics: List of validation methods to be used. Default is None. One can alternatively use ['accuracy'].
        """
        if isinstance(optimizer, six.string_types):
            optimizer = self.__convert_optim_method(optimizer)
        if isinstance(loss, six.string_types):
            loss = self.__convert_criterion(loss)
        if all(isinstance(metric, six.string_types) for metric in metrics):
            metrics = self.__convert_metrics(metrics)
        callBigDlFunc(self.bigdl_type, "compile",
                      self.value,
                      optimizer,
                      loss,
                      metrics)


Instruction:
Can you generate the documentation for the following Python 3 function
def fit(self, x, y=None, batch_size=32, nb_epoch=10, validation_data=None, distributed=True):
        if distributed:
            if isinstance(x, np.ndarray) and isinstance(y, np.ndarray):
                training_data = to_sample_rdd(x, y)
                if validation_data:
                    validation_data = to_sample_rdd(*validation_data)
            elif (isinstance(x, RDD) or isinstance(x, DataSet)) and not y:
                training_data = x
            else:
                raise TypeError("Unsupported training data type: %s" % type(x))
            callBigDlFunc(self.bigdl_type, "fit",
                          self.value,
                          training_data,
                          batch_size,
                          nb_epoch,
                          validation_data)
        else:
            if validation_data:
                val_x = [JTensor.from_ndarray(x) for x in to_list(validation_data[0])]
                val_y = JTensor.from_ndarray(validation_data[1])
            else:
                val_x, val_y = None, None
            callBigDlFunc(self.bigdl_type, "fit",
                          self.value,
                          [JTensor.from_ndarray(x) for x in to_list(x)],
                          JTensor.from_ndarray(y),
                          batch_size,
                          nb_epoch,
                          val_x,
                          val_y,
                          multiprocessing.cpu_count())

Response: Train a model for a fixed number of epochs on a dataset.


Instruction:
Implement a Python 3 function for
evaluating a model on a given dataset in distributed mode.

Response:
def evaluate(self, x, y=None, batch_size=32):
        """
        Evaluate a model on a given dataset in distributed mode.

        # Arguments
        x: Input data. A Numpy array or RDD of Sample.
        y: Labels. A Numpy array. Default is None if x is already RDD of Sample.
        batch_size: Number of samples per gradient update.
        """
        if isinstance(x, np.ndarray) and isinstance(y, np.ndarray):
            evaluation_data = to_sample_rdd(x, y)
        elif isinstance(x, RDD) and not y:
            evaluation_data = x
        else:
            raise TypeError("Unsupported evaluation data type: %s" % type(x))
        return callBigDlFunc(self.bigdl_type, "evaluate",
                             self.value,
                             evaluation_data,
                             batch_size)


Instruction:
Write a Python 3 script for
using a model to predict the class of the user s income.

Response:
def predict(self, x, distributed=True):
        """
        Use a model to do prediction.

        # Arguments
        x: Input data. A Numpy array or RDD of Sample.
        distributed: Boolean. Whether to do prediction in distributed mode or local mode.
                     Default is True. In local mode, x must be a Numpy array.
        """
        if is_distributed:
            if isinstance(x, np.ndarray):
                features = to_sample_rdd(x, np.zeros([x.shape[0]]))
            elif isinstance(x, RDD):
                features = x
            else:
                raise TypeError("Unsupported prediction data type: %s" % type(x))
            return self.predict_distributed(features)
        else:
            if isinstance(x, np.ndarray):
                return self.predict_local(x)
            else:
                raise TypeError("Unsupported prediction data type: %s" % type(x))


Instruction:
Can you create a Python 3 function that
creates a Python Model from a Java value.

Response:
def from_jvalue(jvalue, bigdl_type="float"):
        """
        Create a Python Model base on the given java value
        :param jvalue: Java object create by Py4j
        :return: A Python Model
        """
        model = Sequential(jvalue=jvalue)
        model.value = jvalue
        return model


Instruction:
How would you code a function in Python 3 to
create a Python Model from a Java value.

Response:
def from_jvalue(jvalue, bigdl_type="float"):
        """
        Create a Python Model base on the given java value
        :param jvalue: Java object create by Py4j
        :return: A Python Model
        """
        model = Model([], [], jvalue=jvalue)
        model.value = jvalue
        return model


Instruction:
Write a Python 3 script to
get mnist dataset and parallelize into RDDs.

Response:
def get_mnist(sc, data_type="train", location="/tmp/mnist"):
    """
    Get mnist dataset and parallelize into RDDs.
    Data would be downloaded automatically if it doesn't present at the specific location.

    :param sc: SparkContext.
    :param data_type: "train" for training data and "test" for testing data.
    :param location: Location to store mnist dataset.
    :return: RDD of (features: ndarray, label: ndarray).
    """
    (images, labels) = mnist.read_data_sets(location, data_type)
    images = sc.parallelize(images)
    labels = sc.parallelize(labels + 1)  # Target start from 1 in BigDL
    record = images.zip(labels)
    return record


Instruction:
Can you tell what is the following Python 3 function doing
def get_end_trigger(options):
    if options.endTriggerType.lower() == "epoch":
        return MaxEpoch(options.endTriggerNum)
    else:
        return MaxIteration(options.endTriggerNum)

Response: Returns the end trigger number for the current optimization based on input option.


Instruction:
Here you have a function in Python 3, explain what it does
def validate_optimizer(optimizer, test_data, options):
    optimizer.set_validation(
        batch_size=options.batchSize,
        val_rdd=test_data,
        trigger=EveryEpoch(),
        val_method=[Top1Accuracy()]
    )
    optimizer.set_checkpoint(EveryEpoch(), options.checkpointPath)

Response: Set validation and checkpoint for distributed optimizer.


Instruction:
Write a Python 3 function that can
set the value of batchSize.

Response:
def setBatchSize(self, val):
        """
        Sets the value of :py:attr:`batchSize`.
        """
        self._paramMap[self.batchSize] = val
        pythonBigDL_method_name = "setBatchSize" + self.__class__.__name__
        callBigDlFunc(self.bigdl_type, pythonBigDL_method_name, self.value, val)
        return self


Instruction:
Write a Python 3 function for
returning the broadcasted value of the attribute.

Response:
def value(self):
        """ Return the broadcasted value
        """
        if not hasattr(self, "_value") and self._path is not None:
            self._value = self._load(self._path)
        return self._value


Instruction:
Can you tell what is the following Python 3 function doing
def to_sample_rdd(x, y, numSlices=None):
    sc = get_spark_context()
    from bigdl.util.common import Sample
    x_rdd = sc.parallelize(x, numSlices)
    y_rdd = sc.parallelize(y, numSlices)
    return x_rdd.zip(y_rdd).map(lambda item: Sample.from_ndarray(item[0], item[1]))

Response: Conver x and y into RDD [ Sample ]


Instruction:
Create a Python 3 function to
get the current SparkContext and create one if no active instance

Response:
def get_spark_context(conf=None):
    """
    Get the current active spark context and create one if no active instance
    :param conf: combining bigdl configs into spark conf
    :return: SparkContext
    """
    if hasattr(SparkContext, "getOrCreate"):
        with SparkContext._lock:
            if SparkContext._active_spark_context is None:
                spark_conf = create_spark_conf() if conf is None else conf
                return SparkContext.getOrCreate(spark_conf)
            else:
                return SparkContext.getOrCreate()

    else:
        # Might have threading issue but we cann't add _lock here
        # as it's not RLock in spark1.5;
        if SparkContext._active_spark_context is None:
            spark_conf = create_spark_conf() if conf is None else conf
            return SparkContext(conf=spark_conf)
        else:
            return SparkContext._active_spark_context


Instruction:
Can you write a function in Python 3 where it
calls API in PythonBigDL

Response:
def callBigDlFunc(bigdl_type, name, *args):
    """ Call API in PythonBigDL """
    gateway = _get_gateway()
    error = Exception("Cannot find function: %s" % name)
    for jinvoker in JavaCreator.instance(bigdl_type, gateway).value:
        # hasattr(jinvoker, name) always return true here,
        # so you need to invoke the method to check if it exist or not
        try:
            api = getattr(jinvoker, name)
            result = callJavaFunc(api, *args)
        except Exception as e:
            error = e
            if "does not exist" not in str(e):
                raise e
        else:
            return result
    raise error


Instruction:
Write a Python 3 function for
returning a JavaRDD of Object by unpickling .

Response:
def _to_java_object_rdd(rdd):
    """ Return a JavaRDD of Object by unpickling

    It will convert each Python object into Java object by Pyrolite, whenever
    the RDD is serialized in batch or not.
    """
    rdd = rdd._reserialize(AutoBatchedSerializer(PickleSerializer()))
    return \
        rdd.ctx._jvm.org.apache.spark.bigdl.api.python.BigDLSerDe.pythonToJava(
            rdd._jrdd, True)


Instruction:
How would you code a function in Python 3 to
convert Python object into Java object.

Response:
def _py2java(gateway, obj):
    """ Convert Python object into Java """
    if isinstance(obj, RDD):
        obj = _to_java_object_rdd(obj)
    elif isinstance(obj, DataFrame):
        obj = obj._jdf
    elif isinstance(obj, SparkContext):
        obj = obj._jsc
    elif isinstance(obj, (list, tuple)):
        obj = ListConverter().convert([_py2java(gateway, x) for x in obj],
                                      gateway._gateway_client)
    elif isinstance(obj, dict):
        result = {}
        for (key, value) in obj.items():
            result[key] = _py2java(gateway, value)
        obj = MapConverter().convert(result, gateway._gateway_client)
    elif isinstance(obj, JavaValue):
        obj = obj.value
    elif isinstance(obj, JavaObject):
        pass
    elif isinstance(obj, (int, long, float, bool, bytes, unicode)):
        pass
    else:
        data = bytearray(PickleSerializer().dumps(obj))
        obj = gateway.jvm.org.apache.spark.bigdl.api.python.BigDLSerDe.loads(data)
    return obj


Instruction:
How would you explain what the following Python 3 function does
def get_activation_by_name(activation_name, activation_id=None):
    import bigdl.nn.layer as BLayer
    activation = None
    activation_name = activation_name.lower()
    if activation_name == "tanh":
        activation = BLayer.Tanh()
    elif activation_name == "sigmoid":
        activation = BLayer.Sigmoid()
    elif activation_name == "hard_sigmoid":
        activation = BLayer.HardSigmoid()
    elif activation_name == "relu":
        activation = BLayer.ReLU()
    elif activation_name == "softmax":
        activation = BLayer.SoftMax()
    elif activation_name == "softplus":
        activation = BLayer.SoftPlus(beta=1.0)
    elif activation_name == "softsign":
        activation = BLayer.SoftSign()
    elif activation_name == "linear":
        activation = BLayer.Identity()
    else:
        raise Exception("Unsupported activation type: %s" % activation_name)
    if not activation_id:
        activation.set_name(activation_id)
    return activation

Response: Convert a activation name to a bigdl activation layer


Instruction:
Can you generate a brief explanation for the following Python 3 code
def from_ndarray(cls, a_ndarray, bigdl_type="float"):
        if a_ndarray is None:
            return None
        assert isinstance(a_ndarray, np.ndarray), \
            "input should be a np.ndarray, not %s" % type(a_ndarray)
        return cls(a_ndarray,
                   a_ndarray.shape if a_ndarray.shape else (a_ndarray.size),
                   bigdl_type)

Response: Convert a numpy array to a DenseTensor which would be used in Java side.


Instruction:
Can you generate the documentation for the following Python 3 function
def sparse(cls, a_ndarray, i_ndarray, shape, bigdl_type="float"):
        if a_ndarray is None:
            return None
        assert isinstance(a_ndarray, np.ndarray), \
            "values array should be a np.ndarray, not %s" % type(a_ndarray)
        assert isinstance(i_ndarray, np.ndarray), \
            "indices array should be a np.ndarray, not %s" % type(a_ndarray)
        assert i_ndarray.size == a_ndarray.size * shape.size, \
            "size of values and indices should match."
        return cls(a_ndarray,
                   shape,
                   bigdl_type,
                   i_ndarray)

Response: Convert a three ndarray to SparseTensor which would be used in Java side.


Instruction:
Explain what the following Python 3 code does
def to_ndarray(self):
        assert self.indices is None, "sparseTensor to ndarray is not supported"
        return np.array(self.storage, dtype=get_dtype(self.bigdl_type)).reshape(self.shape)

Response: Transfer JTensor to ndarray


Instruction:
Here you have a function in Python 3, explain what it does
def from_ndarray(cls, features, labels, bigdl_type="float"):
        if isinstance(features, np.ndarray):
            features = [features]
        else:
            assert all(isinstance(feature, np.ndarray) for feature in features), \
                "features should be a list of np.ndarray, not %s" % type(features)
        if np.isscalar(labels):  # in case labels is a scalar.
            labels = [np.array(labels)]
        elif isinstance(labels, np.ndarray):
            labels = [labels]
        else:
            assert all(isinstance(label, np.ndarray) for label in labels), \
                "labels should be a list of np.ndarray, not %s" % type(labels)
        return cls(
            features=[JTensor.from_ndarray(feature) for feature in features],
            labels=[JTensor.from_ndarray(label) for label in labels],
            bigdl_type=bigdl_type)

Response: Convert a numpy ndarray of features and labels to Sample which would be used in Java side.


Instruction:
Write a Python 3 script to
get label as ndarray from ImageFeature

Response:
def get_label(self):
        """
        get label as ndarray from ImageFeature
        """
        label = callBigDlFunc(self.bigdl_type, "imageFeatureToLabelTensor", self.value)
        return label.to_ndarray()


Instruction:
How would you code a function in Python 3 to
read images from HDFS or local file system.

Response:
def read(cls, path, sc=None, min_partitions=1, bigdl_type="float"):
        """
        Read images as Image Frame
        if sc is defined, Read image as DistributedImageFrame from local file system or HDFS
        if sc is null, Read image as LocalImageFrame from local file system
        :param path path to read images
        if sc is defined, path can be local or HDFS. Wildcard character are supported.
        if sc is null, path is local directory/image file/image file with wildcard character
        :param sc SparkContext
        :param min_partitions A suggestion value of the minimal splitting number for input data.
        :return ImageFrame
        """
        return ImageFrame(jvalue=callBigDlFunc(bigdl_type, "read", path, sc, min_partitions))


Instruction:
Explain what the following Python 3 code does
def read_parquet(cls, path, sc, bigdl_type="float"):
        return DistributedImageFrame(jvalue=callBigDlFunc(bigdl_type, "readParquet", path, sc))

Response: Read a DistributedImageFrame from a parquet file.


Instruction:
Implement a Python 3 function for
writing ImageFrame as parquet file

Response:
def write_parquet(cls, path, output, sc, partition_num = 1, bigdl_type="float"):
        """
        write ImageFrame as parquet file
        """
        return callBigDlFunc(bigdl_type, "writeParquet", path, output, sc, partition_num)


Instruction:
Write a Python 3 function that can
transform the image frame of this object.

Response:
def transform(self, transformer, bigdl_type="float"):
        """
        transformImageFrame
        """
        self.value = callBigDlFunc(bigdl_type,
                                 "transformImageFrame", transformer, self.value)
        return self


Instruction:
Here you have a function in Python 3, explain what it does
def get_image(self, float_key="floats", to_chw=True):
        return self.image_frame.get_image(float_key, to_chw)

Response: get image from ImageFrame


Instruction:
Can you generate the documentation for the following Python 3 function
def random_split(self, weights):
        jvalues =  self.image_frame.random_split(weights)
        return [ImageFrame(jvalue) for jvalue in jvalues]

Response: Random split imageframes according to weights


Instruction:
How would you explain what the following Python 3 function does
def get_image(self, float_key="floats", to_chw=True):
        tensors = callBigDlFunc(self.bigdl_type,
                                   "localImageFrameToImageTensor", self.value, float_key, to_chw)
        return map(lambda tensor: tensor.to_ndarray(), tensors)

Response: get list of images from ImageFrame


Instruction:
How would you explain what the following Python 3 function does
def get_label(self):
        tensor_rdd = callBigDlFunc(self.bigdl_type, "distributedImageFrameToLabelTensorRdd", self.value)
        return tensor_rdd.map(lambda tensor: tensor.to_ndarray())

Response: get label rdd from ImageFrame


Instruction:
How would you implement a function in Python 3 that
gets prediction rdd from ImageFrame

Response:
def get_predict(self, key="predict"):
        """
        get prediction rdd from ImageFrame
        """
        predicts = callBigDlFunc(self.bigdl_type, "distributedImageFrameToPredict", self.value, key)
        return predicts.map(lambda predict: (predict[0], predict[1].to_ndarray()) if predict[1] else (predict[0], None))


Instruction:
Explain what the following Python 3 code does
def files_to_image_frame(cls,
                             url,
                             sc,
                             class_num,
                             partition_num=-1,
                             bigdl_type="float"):
        """
        Extract hadoop sequence files from an HDFS path as ImageFrame
        :param url: sequence files folder path
        :param sc: spark context
        :param class_num: class number of data
        :param partition_num: partition number, default: Engine.nodeNumber() * Engine.coreNumber()
        """
        jvalue = callBigDlFunc(bigdl_type,
                               "seqFilesToImageFrame",
                               url,
                               sc,
                               class_num,
                               partition_num)
        return ImageFrame(jvalue=jvalue)

Response: This function extracts the hadoop sequence files from an HDFS path as ImageFrame.


Instruction:
How would you code a function in Python 3 to
evaluate a model by the given metrics.

Response:
def evaluate(self, x, y, batch_size=32, sample_weight=None, is_distributed=False):
        """
        Evaluate a model by the given metrics.
        :param x: ndarray or list of ndarray for local mode.
                  RDD[Sample] for distributed mode
        :param y: ndarray or list of ndarray for local mode and would be None for cluster mode.
        :param batch_size
        :param is_distributed: run in local mode or distributed mode.
               NB: if is_distributed=true, x should be RDD[Sample] and y should be None
        :return:
        """
        if sample_weight:
            unsupport_exp("sample_weight")
        if is_distributed:
            if isinstance(x, np.ndarray):
                input = to_sample_rdd(x, y)
            elif isinstance(x, RDD):
                input = x
            if self.metrics:
                sc = get_spark_context()
                return [r.result for r in
                        self.bmodel.evaluate(input, batch_size, self.metrics)]
            else:
                raise Exception("No Metrics found.")
        else:
            raise Exception("We only support evaluation in distributed mode")


Instruction:
Can you generate the documentation for the following Python 3 function
def predict(self, x, batch_size=None, verbose=None, is_distributed=False):
        if batch_size or verbose:
            raise Exception("we don't support batch_size or verbose for now")
        if is_distributed:
            if isinstance(x, np.ndarray):
                input = to_sample_rdd(x, np.zeros([x.shape[0]]))
            #  np.asarray(self.bmodel.predict(x_rdd).collect())
            elif isinstance(x, RDD):
                input = x
            return self.bmodel.predict(input)
        else:
            if isinstance(x, np.ndarray):
                return self.bmodel.predict_local(x)
        raise Exception("not supported type: %s" % x)

Response: Generates output predictions for the input samples in a batched way.


Instruction:
Explain what the following Python 3 code does
def fit(self, x, y=None, batch_size=32, nb_epoch=10, verbose=1, callbacks=None,
            validation_split=0., validation_data=None, shuffle=True,
            class_weight=None, sample_weight=None, initial_epoch=0, is_distributed=False):
        """Optimize the model by the given options

        :param x: ndarray or list of ndarray for local mode.
                  RDD[Sample] for distributed mode
        :param y: ndarray or list of ndarray for local mode and would be None for cluster mode.
            is_distributed: used to control run in local or cluster. the default value is False.
            NB: if is_distributed=true, x should be RDD[Sample] and y should be None
        :param is_distributed: Whether to train in local mode or distributed mode
        :return:
            A Numpy array or RDD[Sample] of predictions.
        """
        if callbacks:
            raise Exception("We don't support callbacks in fit for now")
        if class_weight:
            unsupport_exp("class_weight")
        if sample_weight:
            unsupport_exp("sample_weight")
        if initial_epoch != 0:
            unsupport_exp("initial_epoch")
        if shuffle != True:
            unsupport_exp("shuffle")
        if validation_split != 0.:
            unsupport_exp("validation_split")
        bopt = self.__create_optimizer(x=x,
                                       y=y,
                                       batch_size=batch_size,
                                       nb_epoch=nb_epoch,
                                       validation_data=validation_data,
                                       is_distributed=is_distributed)
        bopt.optimize()

Response: Optimize the model by the given options


Instruction:
Can you tell what is the following Python 3 function doing
def transform(self, dataset):
        self._transfer_params_to_java()
        return callBigDlFunc(self.bigdl_type, "dlImageTransform", self.value, dataset)

Response: Apply the transformer to the images in inputCol and store the transformed result in outputCols


Instruction:
Create a Python 3 function for
saving a Keras model definition to JSON with given path

Response:
def save_keras_definition(keras_model, path):
    """
    Save a Keras model definition to JSON with given path
    """
    model_json = keras_model.to_json()
    with open(path, "w") as json_file:
        json_file.write(model_json)


Instruction:
Write a Python 3 function for
downloading or load MNIST dataset to the specified path.

Response:
def get_mnist(sc, data_type="train", location="/tmp/mnist"):
    """
    Download or load MNIST dataset to/from the specified path.
    Normalize and transform input data into an RDD of Sample
    """
    from bigdl.dataset import mnist
    from bigdl.dataset.transformer import normalizer
    (images, labels) = mnist.read_data_sets(location, data_type)
    images = images.reshape((images.shape[0], ) + input_shape)
    images = sc.parallelize(images)
    labels = sc.parallelize(labels + 1)  # Target start from 1 in BigDL
    record = images.zip(labels).map(lambda rec_tuple: (normalizer(rec_tuple[0], mnist.TRAIN_MEAN, mnist.TRAIN_STD),
                                    rec_tuple[1])) \
                               .map(lambda t: Sample.from_ndarray(t[0], t[1]))
    return record


Instruction:
Write a Python 3 function that can
define a convnet model in Keras 1. 2. 2

Response:
def build_keras_model():
    """
    Define a convnet model in Keras 1.2.2
    """
    from keras.models import Sequential
    from keras.layers import Dense, Dropout, Activation, Flatten
    from keras.layers import Convolution2D, MaxPooling2D

    keras_model = Sequential()
    keras_model.add(Convolution2D(32, 3, 3, border_mode='valid',
                                  input_shape=input_shape))
    keras_model.add(Activation('relu'))
    keras_model.add(Convolution2D(32, 3, 3))
    keras_model.add(Activation('relu'))
    keras_model.add(MaxPooling2D(pool_size=(2, 2)))
    keras_model.add(Dropout(0.25))
    keras_model.add(Flatten())
    keras_model.add(Dense(128))
    keras_model.add(Activation('relu'))
    keras_model.add(Dropout(0.5))
    keras_model.add(Dense(10))
    keras_model.add(Activation('softmax'))
    return keras_model


Instruction:
Can you create a Python 3 function that
loads a pre - trained Bigdl model.

Response:
def load(path, bigdl_type="float"):
        """
        Load a pre-trained Bigdl model.

        :param path: The path containing the pre-trained model.
        :return: A pre-trained model.
        """
        jmodel = callBigDlFunc(bigdl_type, "loadBigDL", path)
        return Layer.of(jmodel)


Instruction:
Create a Python 3 function for
creating a Python Layer base on the given java value and the real type.

Response:
def of(jvalue, bigdl_type="float"):
        """
        Create a Python Layer base on the given java value and the real type.
        :param jvalue: Java object create by Py4j
        :return: A Python Layer
        """
        def get_py_name(jclass_name):
            if jclass_name == "StaticGraph" or jclass_name == "DynamicGraph":
                return "Model"
            elif jclass_name == "Input":
                return "Layer"
            else:
                return jclass_name

        jname = callBigDlFunc(bigdl_type,
                                      "getRealClassNameOfJValue",
                                      jvalue)

        jpackage_name = ".".join(jname.split(".")[:-1])
        pclass_name = get_py_name(jname.split(".")[-1])

        if "com.intel.analytics.bigdl.nn.keras.Model" == jname or \
                        "com.intel.analytics.bigdl.nn.keras.Sequential" == jname:
            base_module = importlib.import_module('bigdl.nn.keras.topology')
        elif "com.intel.analytics.bigdl.nn.keras" == jpackage_name:
            base_module = importlib.import_module('bigdl.nn.keras.layer')
        else:
            base_module = importlib.import_module('bigdl.nn.layer')

        realClassName = "Layer" # The top base class
        if pclass_name in dir(base_module):
            realClassName = pclass_name
        module = getattr(base_module, realClassName)
        jvalue_creator = getattr(module, "from_jvalue")
        model = jvalue_creator(jvalue, bigdl_type)
        return model


Instruction:
Implement a Python 3 function for
setting the running mean of the layer.

Response:
def set_running_mean(self, running_mean):
        """
        Set the running mean of the layer.
        Only use this method for a BatchNormalization layer.
        :param running_mean: a Numpy array.
        """
        callBigDlFunc(self.bigdl_type, "setRunningMean",
                      self.value, JTensor.from_ndarray(running_mean))
        return self


Instruction:
Can you generate the documentation for the following Python 3 function
def set_running_std(self, running_std):
        callBigDlFunc(self.bigdl_type, "setRunningStd",
                      self.value, JTensor.from_ndarray(running_std))
        return self

Response: Set the running variance of the layer.


Instruction:
Implement a Python 3 function for
creating a Python Model from a Java value.

Response:
def from_jvalue(jvalue, bigdl_type="float"):
        """
        Create a Python Model base on the given java value
        :param jvalue: Java object create by Py4j
        :return: A Python Model
        """
        model = Layer(jvalue=jvalue, bigdl_type=bigdl_type)
        model.value = jvalue
        return model


Instruction:
Write a Python 3 function that can
check input for valid input for the KATCP version.

Response:
def check_input(input):
        """
        :param input: ndarray or list of ndarray or JTensor or list of JTensor.
        :return: (list of JTensor, isTable)
        """
        def to_jtensor(i):
            if isinstance(i, np.ndarray):
                return JTensor.from_ndarray(i)
            elif isinstance(i, JTensor):
                return i
            else:
                raise Exception("Error unknown input type %s" % type(i))
        if type(input) is list:
            if len(input) == 0:
                raise Exception('Error when checking: empty input')
            return list(map(lambda i: to_jtensor(i), input)), True
        else:
            return [to_jtensor(input)], False


Instruction:
Make a summary of the following Python 3 code
def backward(self, input, grad_output):
        jinput, input_is_table = self.check_input(input)
        jgrad_output, grad_output_is_table = self.check_input(grad_output)
        output = callBigDlFunc(self.bigdl_type,
                               "modelBackward",
                               self.value,
                               jinput,
                               input_is_table,
                               jgrad_output,
                               grad_output_is_table)
        return self.convert_output(output)

Response: This method performs a backward propagation step through the module with respect to the input.


Instruction:
Explain what the following Python 3 code does
def parameters(self):
        name_to_params = callBigDlFunc(self.bigdl_type,
                                       "modelGetParameters",
                                       self.value)

        def to_ndarray(params):
            return dict((param_name,
                         np.array(values[0], dtype=self.get_dtype()).reshape(
                             values[1])) for param_name, values in
                        params.items())

        return dict((layer_name, to_ndarray(params)) for layer_name, params in
                name_to_params.items())

Response: Get the model parameters which contains weight bias gradWeight gradBias gradWeight


Instruction:
How would you code a function in Python 3 to
evaluate the model for the current layer.

Response:
def evaluate(self, *args):
        """
        No argument passed in:
        Evaluate the model to set train = false, useful when doing test/forward
        :return: layer itself

        Three arguments passed in:
        A method to benchmark the model quality.

        :param dataset: the input data
        :param batch_size: batch size
        :param val_methods: a list of validation methods. i.e: Top1Accuracy,Top5Accuracy and Loss.
        :return: a list of the metrics result
        """
        if len(args) == 0:
            callBigDlFunc(self.bigdl_type,
                          "evaluate", self.value)
            return self
        elif len(args) == 3:
            dataset, batch_size, val_methods = args
            if (isinstance(dataset, ImageFrame)):
                return callBigDlFunc(self.bigdl_type,
                                    "modelEvaluateImageFrame",
                                    self.value,
                                    dataset, batch_size, val_methods)
            else:
                return callBigDlFunc(self.bigdl_type,
                                     "modelEvaluate",
                                     self.value,
                                     dataset, batch_size, val_methods)
        else:
            raise Exception("Error when calling evaluate(): it takes no argument or exactly three arguments only")


Instruction:
Can you implement a function in Python 3 that
predicts the local values of the model with the given data.

Response:
def predict_local(self, X, batch_size = -1):
        """
        :param X: X can be a ndarray or list of ndarray if the model has multiple inputs.
                  The first dimension of X should be batch.
        :param batch_size: total batch size of prediction.
        :return: a ndarray as the prediction result.
        """

        jresults = callBigDlFunc(self.bigdl_type,
                             "predictLocal",
                               self.value,
                               self._to_jtensors(X),
                               batch_size)

        return np.stack([j.to_ndarray()for j in jresults])


Instruction:
Can you generate a brief explanation for the following Python 3 code
def predict(self, features, batch_size = -1):
        if isinstance(features, RDD):
            return self.predict_distributed(features, batch_size)
        else:
            return self.predict_local(features, batch_size)

Response: Predict the model for the given features.


Instruction:
Here you have a function in Python 3, explain what it does
def predict_class(self, features):
        if isinstance(features, RDD):
            return self.predict_class_distributed(features)
        else:
            return self.predict_class_local(features)

Response: Model inference base on the given data which returning label


Instruction:
How would you implement a function in Python 3 that
predicts the distributed version of the data_rdd.

Response:
def predict_distributed(self, data_rdd, batch_size = -1):
        """
        Model inference base on the given data.
        You need to invoke collect() to trigger those action \
        as the returning result is an RDD.

        :param data_rdd: the data to be predict.
        :param batch_size: total batch size of prediction.
        :return: An RDD represent the predict result.
        """
        result = callBigDlFunc(self.bigdl_type,
                               "modelPredictRDD", self.value, data_rdd, batch_size)
        return result.map(lambda data: data.to_ndarray())


Instruction:
Make a summary of the following Python 3 code
def predict_class_distributed(self, data_rdd):
        result = callBigDlFunc(self.bigdl_type,
                               "modelPredictClass", self.value, data_rdd)
        return result

Response: Predict the class for the given data RDD.


Instruction:
Can you tell what is the following Python 3 function doing
def predict_image(self, image_frame, output_layer=None, share_buffer=False,
                      batch_per_partition=4, predict_key="predict"):
        """
        model predict images, return imageFrame with predicted tensor
        :param image_frame imageFrame that contains images
        :param output_layer if output_layer is not null, the output of layer that matches
        output_layer will be used as predicted output
        :param share_buffer whether to share same memory for each batch predict results
        :param batch_per_partition batch size per partition, default is 4
        :param predict_key key to store predicted results
        """

        image_frame = callBigDlFunc(self.bigdl_type, "modelPredictImage", self.value,
                             image_frame,
                             output_layer,
                             share_buffer,
                             batch_per_partition,
                             predict_key)
        return ImageFrame(image_frame)

Response: model predict images return imageFrame with predicted tensor


Instruction:
Write a Python 3 script to
set the weights for this layer.

Response:
def set_weights(self, weights):
        """
        Set weights for this layer

        :param weights: a list of numpy arrays which represent weight and bias
        :return:

        >>> linear = Linear(3,2)
        creating: createLinear
        >>> linear.set_weights([np.array([[1,2,3],[4,5,6]]), np.array([7,8])])
        >>> weights = linear.get_weights()
        >>> weights[0].shape == (2,3)
        True
        >>> np.testing.assert_allclose(weights[0][0], np.array([1., 2., 3.]))
        >>> np.testing.assert_allclose(weights[1], np.array([7., 8.]))
        >>> relu = ReLU()
        creating: createReLU
        >>> from py4j.protocol import Py4JJavaError
        >>> try:
        ...     relu.set_weights([np.array([[1,2,3],[4,5,6]]), np.array([7,8])])
        ... except Py4JJavaError as err:
        ...     print(err.java_exception)
        ...
        java.lang.IllegalArgumentException: requirement failed: this layer does not have weight/bias
        >>> relu.get_weights()
        The layer does not have weight/bias
        >>> add = Add(2)
        creating: createAdd
        >>> try:
        ...     add.set_weights([np.array([7,8]), np.array([1,2])])
        ... except Py4JJavaError as err:
        ...     print(err.java_exception)
        ...
        java.lang.IllegalArgumentException: requirement failed: the number of input weight/bias is not consistant with number of weight/bias of this layer, number of input 1, number of output 2
        >>> cAdd = CAdd([4, 1])
        creating: createCAdd
        >>> cAdd.set_weights(np.ones([4, 1]))
        >>> (cAdd.get_weights()[0] == np.ones([4, 1])).all()
        True
        """
        tensors = [JTensor.from_ndarray(param, self.bigdl_type) for param in to_list(weights)]
        callBigDlFunc(self.bigdl_type, "setWeights", self.value, tensors)


Instruction:
How would you explain what the following Python 3 function does
def get_weights(self):
        tensorWeights = callBigDlFunc(self.bigdl_type,
                              "getWeights", self.value)
        if tensorWeights is not None:
            return [tensor.to_ndarray() for tensor in tensorWeights]
        else:
            print("The layer does not have weight/bias")
            return None

Response: Get weights for this layer


Instruction:
Create a Python 3 function for
saving a model to protobuf files.

Response:
def save_tensorflow(self, inputs, path, byte_order="little_endian", data_format="nhwc"):
        """
        Save a model to protobuf files so that it can be used in tensorflow inference.

        When saving the model, placeholders will be added to the tf model as input nodes. So
        you need to pass in the names and shapes of the placeholders. BigDL model doesn't have
        such information. The order of the placeholder information should be same as the inputs
        of the graph model.
        :param inputs: placeholder information, should be an array of tuples (input_name, shape)
                       where 'input_name' is a string and shape is an array of integer
        :param path: the path to be saved to
        :param byte_order: model byte order
        :param data_format: model data format, should be "nhwc" or "nchw"
        """
        callBigDlFunc(self.bigdl_type, "saveTF", self.value, inputs, path, byte_order, data_format)


Instruction:
Implement a function in Python 3 to
freeze the current module.

Response:
def freeze(self, names=None):
        """
        freeze module, if names is not None, set an array of layers that match given names
        to be freezed
        :param names: an array of layer names
        :return:
        """
        callBigDlFunc(self.bigdl_type, "freeze", self.value, names)
        return self


Instruction:
Can you write a function in Python 3 where it
sets this layer in training mode or in predition mode.

Response:
def training(self, is_training=True):
        '''
        Set this layer in the training mode or in predition mode if is_training=False
        '''
        if is_training:
            callJavaFunc(self.value.training)
        else:
            callJavaFunc(self.value.evaluate)
        return self


Instruction:
Can you tell what is the following Python 3 function doing
def quantize(self):
        '''
        Clone self and quantize it, at last return a new quantized model.
        :return: A new quantized model.

        >>> fc = Linear(4, 2)
        creating: createLinear
        >>> fc.set_weights([np.ones((2, 4)), np.ones((2,))])
        >>> input = np.ones((2, 4))
        >>> output = fc.forward(input)
        >>> expected_output = np.array([[5., 5.], [5., 5.]])
        >>> np.testing.assert_allclose(output, expected_output)
        >>> quantized_fc = fc.quantize()
        >>> quantized_output = quantized_fc.forward(input)
        >>> expected_quantized_output = np.array([[5., 5.], [5., 5.]])
        >>> np.testing.assert_allclose(quantized_output, expected_quantized_output)

        >>> assert("quantized.Linear" in quantized_fc.__str__())
        >>> conv = SpatialConvolution(1, 2, 3, 3)
        creating: createSpatialConvolution
        >>> conv.set_weights([np.ones((2, 1, 3, 3)), np.zeros((2,))])
        >>> input = np.ones((2, 1, 4, 4))
        >>> output = conv.forward(input)
        >>> expected_output = np.array([[[[9., 9.], [9., 9.]], [[9., 9.], [9., 9.]]], [[[9., 9.], [9., 9.]], [[9., 9.], [9., 9.]]]])
        >>> np.testing.assert_allclose(output, expected_output)
        >>> quantized_conv = conv.quantize()
        >>> quantized_output = quantized_conv.forward(input)
        >>> expected_quantized_output = np.array([[[[9., 9.], [9., 9.]], [[9., 9.], [9., 9.]]], [[[9., 9.], [9., 9.]], [[9., 9.], [9., 9.]]]])
        >>> np.testing.assert_allclose(quantized_output, expected_quantized_output)
        >>> assert("quantized.SpatialConvolution" in quantized_conv.__str__())
        >>> seq = Sequential()
        creating: createSequential
        >>> seq = seq.add(conv)
        >>> seq = seq.add(Reshape([8, 4], False))
        creating: createReshape
        >>> seq = seq.add(fc)
        >>> input = np.ones([1, 1, 6, 6])
        >>> output = seq.forward(input)
        >>> expected_output = np.array([[37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.]])
        >>> np.testing.assert_allclose(output, expected_output)
        >>> quantized_seq = seq.quantize()
        >>> quantized_output = quantized_seq.forward(input)
        >>> expected_quantized_output = np.array([[37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.], [37., 37.]])
        >>> np.testing.assert_allclose(quantized_output, expected_quantized_output)
        >>> assert("quantized.Linear" in quantized_seq.__str__())
        >>> assert("quantized.SpatialConvolution" in quantized_seq.__str__())
        '''
        quantized_model = callBigDlFunc(self.bigdl_type, "quantize", self.value)
        return Layer.of(quantized_model)

Response: Clone self and quantize it at last return a new quantized model.


Instruction:
Explain what the following Python 3 code does
def loadModel(modelPath, weightPath =None, bigdl_type="float"):
        jmodel = callBigDlFunc(bigdl_type, "loadBigDLModule", modelPath, weightPath)
        return Layer.of(jmodel)

Response: Load a pre - trained Bigdl model.


Instruction:
Explain what the following Python 3 code does
def load_torch(path, bigdl_type="float"):
        jmodel = callBigDlFunc(bigdl_type, "loadTorch", path)
        return Layer.of(jmodel)

Response: Load a pre - trained Torch model.


Instruction:
Can you write a function in Python 3 where it
loads a pre - trained Keras model.

Response:
def load_keras(json_path=None, hdf5_path=None, by_name=False):
        """
        Load a pre-trained Keras model.

        :param json_path: The json path containing the keras model definition.
        :param hdf5_path: The HDF5 path containing the pre-trained keras model weights with or without the model architecture.
        :return: A bigdl model.
        """
        import os
        try:
            import tensorflow
        except ImportError:
            os.environ['KERAS_BACKEND'] = "theano"
            try:
                # Make theano backend compatible with Python3
                from theano import ifelse
            except ImportError:
                raise Exception("No backend is found for Keras. "
                                "Please install either tensorflow or theano.")
        from bigdl.keras.converter import DefinitionLoader, WeightLoader
        if json_path and not hdf5_path:
            return DefinitionLoader.from_json_path(json_path)
        elif json_path and hdf5_path:
            return WeightLoader.load_weights_from_json_hdf5(json_path, hdf5_path, by_name=by_name)
        elif hdf5_path and not json_path:
            kmodel, bmodel = DefinitionLoader.from_hdf5_path(hdf5_path)
            WeightLoader.load_weights_from_kmodel(bmodel, kmodel)
            return bmodel


Instruction:
Can you generate the documentation for the following Python 3 function
def load_caffe(model, defPath, modelPath, match_all=True, bigdl_type="float"):
        jmodel = callBigDlFunc(bigdl_type, "loadCaffe", model, defPath, modelPath, match_all)
        return Layer.of(jmodel)

Response: Load a pre - trained caffe model.


Instruction:
Create a Python 3 function to
load a pre - trained Caffe model.

Response:
def load_caffe_model(defPath, modelPath, bigdl_type="float"):
        """
        Load a pre-trained Caffe model.

        :param defPath: The path containing the caffe model definition.
        :param modelPath: The path containing the pre-trained caffe model.
        :return: A pre-trained model.
        """
        jmodel = callBigDlFunc(bigdl_type, "loadCaffeModel", defPath, modelPath)
        return Layer.of(jmodel)


Instruction:
Make a summary of the following Python 3 code
def load_tensorflow(path, inputs, outputs, byte_order = "little_endian",
                        bin_file = None, generated_backward = True, bigdl_type = "float"):
        """
        Load a pre-trained Tensorflow model.
        :param path: The path containing the pre-trained model.
        :param inputs: The input node of this graph
        :param outputs: The output node of this graph
        :param byte_order: byte_order of the file, `little_endian` or `big_endian`
        :param bin_file: the optional bin file produced by bigdl dump_model util function to store the weights
        :param generated_backward: if generate backward graph
        :return: A pre-trained model.
        """
        jmodel = callBigDlFunc(bigdl_type, "loadTF", path, inputs, outputs,
                               byte_order, bin_file, generated_backward)
        return Model.of(jmodel)

Response: Load a pre - trained Tensorflow model.


Instruction:
Explain what the following Python 3 code does
def stop_gradient(self, stop_layers, bigdl_type="float"):
        callBigDlFunc(bigdl_type, "setStopGradient", self.value, stop_layers)
        return self

Response: Stop the gradient of the input of the given set of layers that depend on them.


Instruction:
Write a Python 3 function that can
return the corresponding node has the given name.

Response:
def node(self, name, bigdl_type="float"):
        """
        Return the corresponding node has the given name. If the given name doesn't match any node,
        an exception will be thrown
        :param name: node name
        :param bigdl_type:
        :return:
        """
        jnode = callBigDlFunc(bigdl_type, "findGraphNode", self.value, name)
        return Node.of(jnode)


Instruction:
Write a Python 3 script to
save the current model topology to a folder.

Response:
def save_graph_topology(self, log_path, bigdl_type="float"):
        """
        save current model graph to a folder, which can be display in tensorboard by running
            tensorboard --logdir logPath
        :param log_path: path to save the model graph
        :param bigdl_type:
        :return:
        """
        callBigDlFunc(bigdl_type, "saveGraphTopology", self.value, log_path)
        return self


Instruction:
Can you tell what is the following Python 3 function doing
def forward(self, input, target):
        jinput, input_is_table = Layer.check_input(input)
        jtarget, target_is_table = Layer.check_input(target)
        output = callBigDlFunc(self.bigdl_type,
                               "criterionForward",
                               self.value,
                               jinput,
                               input_is_table,
                               jtarget,
                               target_is_table)
        return output

Response: This method computes the loss of the criterion from input to target.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def of(cls, jcriterion, bigdl_type="float"):
        criterion = Criterion(bigdl_type, jcriterion)
        criterion.value = jcriterion
        criterion.bigdl_type = bigdl_type
        return criterion

Response: Create a python Criterion by a java criterion object.


Instruction:
Can you generate the documentation for the following Python 3 function
def readImages(path, sc=None, minParitions = 1, bigdl_type="float"):
        df = callBigDlFunc(bigdl_type, "dlReadImage", path, sc, minParitions)
        df._sc._jsc = sc._jsc
        return df

Response: Read the directory of images into a DataFrame.


Instruction:
Given the following Python 3 function, write the documentation
def load_weights_from_json_hdf5(def_json, weights_hdf5, by_name=False):
        bmodel = DefinitionLoader.from_json_path(def_json)
        def_value = BCommon.text_from_path(def_json)
        kmodel = model_from_json(def_value)
        WeightLoader.load_weights_from_hdf5(bmodel, kmodel, weights_hdf5, by_name)
        return bmodel

Response: Loads a WeightModel from a json file.


Instruction:
Given the following Python 3 function, write the documentation
def load_weights_from_hdf5(bmodel, kmodel, filepath, by_name=False):
        '''Loads all layer weights from a HDF5 save file.
        filepath can be stored in a local file system, HDFS, S3,
        or any Hadoop-supported file system.
        If `by_name` is False (default) weights are loaded
        based on the network's execution order topology,
        meaning layers in the execution seq should be exactly the same
        the architecture

        If `by_name` is True, weights are loaded into layers
        only if they share the same name. This is useful
        for fine-tuning or transfer-learning models where
        some of the layers have changed.
        '''
        local_file_path = BCommon.get_local_file(filepath)
        kmodel.load_weights(filepath=local_file_path, by_name=by_name)
        WeightLoader.load_weights_from_kmodel(bmodel, kmodel)

Response: Loads all layer weights from a HDF5 save file.


Instruction:
Can you implement a function in Python 3 that
converts KModel s weights to bigdl format.

Response:
def get_weights_from_kmodel(kmodel):
        """
        Convert kmodel's weights to bigdl format.
        We are supposing the order is the same as the execution order.
        :param kmodel: keras model
        :return: list of ndarray
        """
        layers_with_weights = [layer for layer in kmodel.layers if layer.weights]
        bweights = []
        for klayer in layers_with_weights:
            # bws would be [weights, bias] or [weights]
            bws = WeightsConverter.get_bigdl_weights_from_klayer(klayer)
            for w in bws:
                bweights.append(w)
        return bweights


Instruction:
Can you generate the documentation for the following Python 3 function
def from_hdf5_path(cls, hdf5_path):
        from keras.models import load_model
        hdf5_local_path = BCommon.get_local_file(hdf5_path)
        kmodel = load_model(hdf5_local_path)
        return kmodel, DefinitionLoader.from_kmodel(kmodel)

Response: Loads a BigDL model from an hdf5 file.


Instruction:
Can you write a function in Python 3 where it
loads a BigDL model from a JSON file.

Response:
def from_json_path(cls, json_path):
        """
        :param json_path: definition path which can be stored in a local file system, HDFS, S3, or any Hadoop-supported file system.
        :return: BigDL Model
        """
        json_str = BCommon.text_from_path(json_path)
        return DefinitionLoader.from_json_str(json_str)


Instruction:
How would you explain what the following Python 3 function does
def build_keras_model():
    from keras.models import Sequential
    from keras.layers import Dense, Dropout, Activation
    from keras.layers import Embedding
    from keras.layers import LSTM
    from keras.layers import Convolution1D, MaxPooling1D
    keras_model = Sequential()
    keras_model.add(Embedding(20000, 128, input_length=100))
    keras_model.add(Dropout(0.25))
    keras_model.add(Convolution1D(nb_filter=64,
                                  filter_length=5,
                                  border_mode='valid',
                                  activation='relu',
                                  subsample_length=1))
    keras_model.add(MaxPooling1D(pool_length=4))
    keras_model.add(LSTM(70))
    keras_model.add(Dense(1))
    keras_model.add(Activation('sigmoid'))
    return keras_model

Response: Define a keras model in Keras 1. 2. 2


Instruction:
How would you implement a function in Python 3 that
returns a list of shape tuples if there are multiple inputs.

Response:
def get_input_shape(self):
        """
        Return a list of shape tuples if there are multiple inputs.
        Return one shape tuple otherwise.
        """
        input = callBigDlFunc(self.bigdl_type, "getInputShape",
                              self.value)
        return self.__process_shape(input)


Instruction:
Can you tell what is the following Python 3 function doing
def get_output_shape(self):
        output = callBigDlFunc(self.bigdl_type, "getOutputShape",
                               self.value)
        return self.__process_shape(output)

Response: Return a list of shape tuples if there are multiple outputs.


Instruction:
Can you tell what is the following Python 3 function doing
def get_mnist(data_type="train", location="/tmp/mnist"):
    X, Y = mnist.read_data_sets(location, data_type)
    return X, Y + 1

Response: Get mnist dataset with features and label as ndarray.


Instruction:
Implement a Python 3 function for
parsing or download movielens 1m data and return a 2D numpy array with user index and item index in each row

Response:
def read_data_sets(data_dir):
    """
    Parse or download movielens 1m  data if train_dir is empty.

    :param data_dir: The directory storing the movielens data
    :return: a 2D numpy array with user index and item index in each row
    """
    WHOLE_DATA = 'ml-1m.zip'
    local_file = base.maybe_download(WHOLE_DATA, data_dir, SOURCE_URL + WHOLE_DATA)
    zip_ref = zipfile.ZipFile(local_file, 'r')
    extracted_to = os.path.join(data_dir, "ml-1m")
    if not os.path.exists(extracted_to):
        print("Extracting %s to %s" % (local_file, data_dir))
        zip_ref.extractall(data_dir)
        zip_ref.close()
    rating_files = os.path.join(extracted_to,"ratings.dat")

    rating_list = [i.strip().split("::") for i in open(rating_files,"r").readlines()]
    movielens_data = np.array(rating_list).astype(int)
    return movielens_data


Instruction:
Write a Python 3 function that can
get and return the jar path for bigdl if exists.

Response:
def get_bigdl_classpath():
    """
    Get and return the jar path for bigdl if exists.
    """
    if os.getenv("BIGDL_CLASSPATH"):
        return os.environ["BIGDL_CLASSPATH"]
    jar_dir = os.path.abspath(__file__ + "/../../")
    jar_paths = glob.glob(os.path.join(jar_dir, "share/lib/*.jar"))
    if jar_paths:
        assert len(jar_paths) == 1, "Expecting one jar: %s" % len(jar_paths)
        return jar_paths[0]
    return ""


Instruction:
How would you explain what the following Python 3 function does
def is_spark_below_2_2():
    import pyspark
    if(hasattr(pyspark,"version")):
        full_version = pyspark.version.__version__
        # We only need the general spark version (eg, 1.6, 2.2).
        parts = full_version.split(".")
        spark_version = parts[0] + "." + parts[1]
        if(compare_version(spark_version, "2.2")>=0):
            return False
    return True

Response: Check if spark version is below 2. 2


Instruction:
Can you generate a brief explanation for the following Python 3 code
def compare_version(version1, version2):
    v1Arr = version1.split(".")
    v2Arr = version2.split(".")
    len1 = len(v1Arr)
    len2 = len(v2Arr)
    lenMax = max(len1, len2)
    for x in range(lenMax):
        v1Token = 0
        if x < len1:
            v1Token = int(v1Arr[x])
        v2Token = 0
        if x < len2:
            v2Token = int(v2Arr[x])
        if v1Token < v2Token:
            return -1
        if v1Token > v2Token:
            return 1
    return 0

Response: Compare two version strings.


Instruction:
Can you generate the documentation for the following Python 3 function
def convert(input_ops, output_ops, byte_order, bigdl_type):

    input_names = map(lambda x: x.name.split(":")[0], input_ops)
    output_names = map(lambda x: x.name.split(":")[0], output_ops)
    temp = tempfile.mkdtemp()

    dump_model(path=temp)
    model_path = temp + '/model.pb'
    bin_path = temp + '/model.bin'

    model = Model.load_tensorflow(model_path, input_names, output_names,
                                  byte_order, bin_path, bigdl_type)

    try:
        shutil.rmtree(temp)
    except OSError as e:
        if e.errno != errno.ENOENT:
            raise

    return model

Response: Convert tensorflow model to bigdl model


Instruction:
Implement a Python 3 function for
exporting variable tensors from the checkpoint files.

Response:
def export_checkpoint(checkpoint_path):
    """
    Export variable tensors from the checkpoint files.

    :param checkpoint_path: tensorflow checkpoint path
    :return: dictionary of tensor. The key is the variable name and the value is the numpy
    """
    reader = tf.train.NewCheckpointReader(checkpoint_path)

    # Get tensor name list
    tensor_names = filter(lambda n: n!='global_step',
                          reader.get_variable_to_shape_map().keys())
    # Prepare key-value dictionary
    tensors = {}
    for tn in tensor_names:
        tensors[tn] = reader.get_tensor(tn)

    return tensors


Instruction:
Write a Python 3 function that can
save a variable dictionary to a Java object file.

Response:
def save_variable_bigdl(tensors, target_path, bigdl_type="float"):
    """
    Save a variable dictionary to a Java object file, so it can be read by BigDL

    :param tensors: tensor dictionary
    :param target_path: where is the Java object file store
    :param bigdl_type: model variable numeric type
    :return: nothing
    """
    import numpy as np
    jtensors = {}
    for tn in tensors.keys():
        if not isinstance(tensors[tn], np.ndarray):
            value = np.array(tensors[tn])
        else:
            value = tensors[tn]
        jtensors[tn] = JTensor.from_ndarray(value)

    callBigDlFunc(bigdl_type, "saveTensorDictionary", jtensors, target_path)


Instruction:
Can you tell what is the following Python 3 function doing
def dump_model(path, graph=None, sess=None, ckpt_file=None, bigdl_type="float"):
    if not os.path.isdir(path):
        raise ValueError("Folder " + path + " does not exist")

    temp = None
    if ckpt_file is None:
        if sess is None:
            sess = tf.Session()
            init = tf.global_variables_initializer()
            sess.run(init)
            temp = tempfile.mkdtemp()
            ckpt_file = temp
        # dump checkpoint to temp files
        saver = tf.train.Saver()
        saver.save(sess, ckpt_file)

    # generate bin files
    tensors = export_checkpoint(ckpt_file)
    save_variable_bigdl(tensors, path + "/model.bin", bigdl_type)

    # dump grap to pb file
    graph = sess.graph if graph is None else graph
    with gfile.GFile(path + "/model.pb", "wb") as f:
        f.write(graph.as_graph_def().SerializeToString())
    if temp is not None:
        try:
            shutil.rmtree(temp)
        except OSError as e:
            if e.errno != errno.ENOENT:
                raise

Response: Dump a tensorflow model to files.


Instruction:
Make a summary of the following Python 3 code
def merge_checkpoint(input_graph,
                     checkpoint,
                     output_node_names,
                     output_graph,
                     sess):
    """
    Get the variable values from the checkpoint file, and merge them to the GraphDef file
    Args:
        input_graph: the GraphDef file, doesn't contain variable values
        checkpoint: the checkpoint file
        output_node_names: A list of string, the output names
        output_graph: String of the location and the name of the
            output graph
    """
    restore_op_name = "save/restore_all"
    filename_tensor_name = "save/Const:0"

    input_graph_def = graph_pb2.GraphDef()
    with gfile.FastGFile(input_graph, "r") as f:
        text_format.Merge(f.read().decode("utf-8"), input_graph_def)

    for node in input_graph_def.node:
        node.device = ""

    importer.import_graph_def(input_graph_def, name="")

    sess.run([restore_op_name], {filename_tensor_name: checkpoint})
    output_graph_def = graph_util.convert_variables_to_constants(
        sess,
        input_graph_def,
        output_node_names,
        variable_names_blacklist=""
    )

    with gfile.GFile(output_graph, "wb") as f:
        f.write(output_graph_def.SerializeToString())

Response: Merge the checkpoint file into the output graph


Instruction:
Implement a function in Python 3 to
process utterances batch through skills filter and skills processor.

Response:
def _call(self, utterances_batch: list, utterances_ids: Optional[list]=None) -> list:
        """
        Processes batch of utterances and returns corresponding responses batch.

        Each call of Agent passes incoming utterances batch through skills filter,
        agent skills, skills processor. Batch of dialog IDs can be provided, in
        other case utterances indexes in incoming batch are used as dialog IDs.

        Args:
            utterances_batch: Batch of incoming utterances.
            utterances_ids: Batch of dialog IDs corresponding to incoming utterances.

        Returns:
            responses: A batch of responses corresponding to the
                utterance batch received by agent.
        """
        batch_size = len(utterances_batch)
        ids = utterances_ids or list(range(batch_size))
        batch_history = [self.history[utt_id] for utt_id in ids]
        responses = []

        filtered = self.skills_filter(utterances_batch, batch_history)

        for skill_i, (filtered_utterances, skill) in enumerate(zip(filtered, self.wrapped_skills)):
            skill_i_utt_indexes = [utt_index for utt_index, utt_filter in enumerate(filtered_utterances) if utt_filter]

            if skill_i_utt_indexes:
                skill_i_utt_batch = [utterances_batch[i] for i in skill_i_utt_indexes]
                skill_i_utt_ids = [ids[i] for i in skill_i_utt_indexes]
                res = [(None, 0.)] * batch_size
                predicted, confidence = skill(skill_i_utt_batch, skill_i_utt_ids)

                for i, predicted, confidence in zip(skill_i_utt_indexes, predicted, confidence):
                    res[i] = (predicted, confidence)

                responses.append(res)

        responses = self.skills_processor(utterances_batch, batch_history, *responses)

        return responses


Instruction:
Write a Python 3 script for
expanding and tile tensor along given axis.

Response:
def expand_tile(units, axis):
    """
    Expand and tile tensor along given axis

    Args:
        units: tf tensor with dimensions [batch_size, time_steps, n_input_features]
        axis: axis along which expand and tile. Must be 1 or 2

    """
    assert axis in (1, 2)
    n_time_steps = K.int_shape(units)[1]
    repetitions = [1, 1, 1, 1]
    repetitions[axis] = n_time_steps
    if axis == 1:
        expanded = Reshape(target_shape=( (1,) + K.int_shape(units)[1:] ))(units)
    else:
        expanded = Reshape(target_shape=(K.int_shape(units)[1:2] + (1,) + K.int_shape(units)[2:]))(units)
    return K.tile(expanded, repetitions)


Instruction:
How would you implement a function in Python 3 that
computes additive self attention for time series of vectors with batch dimension.

Response:
def additive_self_attention(units, n_hidden=None, n_output_features=None, activation=None):
    """
    Compute additive self attention for time series of vectors (with batch dimension)
            the formula: score(h_i, h_j) = <v, tanh(W_1 h_i + W_2 h_j)>
            v is a learnable vector of n_hidden dimensionality,
            W_1 and W_2 are learnable [n_hidden, n_input_features] matrices

    Args:
        units: tf tensor with dimensionality [batch_size, time_steps, n_input_features]
        n_hidden: number of2784131 units in hidden representation of similarity measure
        n_output_features: number of features in output dense layer
        activation: activation at the output

    Returns:
        output: self attended tensor with dimensionality [batch_size, time_steps, n_output_features]
        """
    n_input_features = K.int_shape(units)[2]
    if n_hidden is None:
        n_hidden = n_input_features
    if n_output_features is None:
        n_output_features = n_input_features
    exp1 = Lambda(lambda x: expand_tile(x, axis=1))(units)
    exp2 = Lambda(lambda x: expand_tile(x, axis=2))(units)
    units_pairs = Concatenate(axis=3)([exp1, exp2])
    query = Dense(n_hidden, activation="tanh")(units_pairs)
    attention = Dense(1, activation=lambda x: softmax(x, axis=2))(query)
    attended_units = Lambda(lambda x: K.sum(attention * x, axis=2))(exp1)
    output = Dense(n_output_features, activation=activation)(attended_units)
    return output