Instruction:
Write a Python 3 function that can
give a series of anomaly scores, compute the likelihood for each score. This function should be called once on a bunch of historical anomaly scores for an initial estimate of the distribution. It should be called again every so often (say every 50 records) to update the estimate. :param anomalyScores: a list of records. Each record is a list with the following three elements: [timestamp, value, score] Example:: [datetime.datetime(2013, 8, 10, 23, 0), 6.0, 1.0] For best results, the list should be between 1000 and 10,000 records :param averagingWindow: integer number of records to average over :param skipRecords: integer specifying number of records to skip when estimating distributions. If skip records are >= len(anomalyScores), a very broad distribution is returned that makes everything pretty likely. :param verbosity: integer controlling extent of printouts for debugging 0 = none 1 = occasional information 2 = print every record :returns: 3-tuple consisting of: - likelihoods numpy array of likelihoods, one for each aggregated point - avgRecordList list of averaged input records - params a small JSON dict that contains the state of the estimator

Response:
def estimateAnomalyLikelihoods(anomalyScores,
                               averagingWindow=10,
                               skipRecords=0,
                               verbosity=0):
  """
  Given a series of anomaly scores, compute the likelihood for each score. This
  function should be called once on a bunch of historical anomaly scores for an
  initial estimate of the distribution. It should be called again every so often
  (say every 50 records) to update the estimate.

  :param anomalyScores: a list of records. Each record is a list with the
                        following three elements: [timestamp, value, score]

                        Example::

                            [datetime.datetime(2013, 8, 10, 23, 0), 6.0, 1.0]

                        For best results, the list should be between 1000
                        and 10,000 records
  :param averagingWindow: integer number of records to average over
  :param skipRecords: integer specifying number of records to skip when
                      estimating distributions. If skip records are >=
                      len(anomalyScores), a very broad distribution is returned
                      that makes everything pretty likely.
  :param verbosity: integer controlling extent of printouts for debugging

                      0 = none
                      1 = occasional information
                      2 = print every record

  :returns: 3-tuple consisting of:

            - likelihoods

              numpy array of likelihoods, one for each aggregated point

            - avgRecordList

              list of averaged input records

            - params

              a small JSON dict that contains the state of the estimator

  """
  if verbosity > 1:
    print("In estimateAnomalyLikelihoods.")
    print("Number of anomaly scores:", len(anomalyScores))
    print("Skip records=", skipRecords)
    print("First 20:", anomalyScores[0:min(20, len(anomalyScores))])

  if len(anomalyScores) == 0:
    raise ValueError("Must have at least one anomalyScore")

  # Compute averaged anomaly scores
  aggRecordList, historicalValues, total =  _anomalyScoreMovingAverage(
    anomalyScores,
    windowSize = averagingWindow,
    verbosity = verbosity)
  s = [r[2] for r in aggRecordList]
  dataValues = numpy.array(s)

  # Estimate the distribution of anomaly scores based on aggregated records
  if len(aggRecordList) <= skipRecords:
    distributionParams = nullDistribution(verbosity = verbosity)
  else:
    distributionParams = estimateNormal(dataValues[skipRecords:])

    # HACK ALERT! The HTMPredictionModel currently does not handle constant
    # metric values very well (time of day encoder changes sometimes lead to
    # unstable SDR's even though the metric is constant). Until this is
    # resolved, we explicitly detect and handle completely flat metric values by
    # reporting them as not anomalous.
    s = [r[1] for r in aggRecordList]
    # Only do this if the values are numeric
    if all([isinstance(r[1], numbers.Number) for r in aggRecordList]):
      metricValues = numpy.array(s)
      metricDistribution = estimateNormal(metricValues[skipRecords:],
                                          performLowerBoundCheck=False)

      if metricDistribution["variance"] < 1.5e-5:
        distributionParams = nullDistribution(verbosity = verbosity)

  # Estimate likelihoods based on this distribution
  likelihoods = numpy.array(dataValues, dtype=float)
  for i, s in enumerate(dataValues):
    likelihoods[i] = tailProbability(s, distributionParams)

  # Filter likelihood values
  filteredLikelihoods = numpy.array(
    _filterLikelihoods(likelihoods) )

  params = {
    "distribution":       distributionParams,
    "movingAverage": {
      "historicalValues": historicalValues,
      "total":            total,
      "windowSize":       averagingWindow,
    },
    "historicalLikelihoods":
      list(likelihoods[-min(averagingWindow, len(likelihoods)):]),
  }

  if verbosity > 1:
    print("Discovered params=")
    print(params)
    print("Number of likelihoods:", len(likelihoods))
    print("First 20 likelihoods:", (
      filteredLikelihoods[0:min(20, len(filteredLikelihoods))] ))
    print("leaving estimateAnomalyLikelihoods")

  return (filteredLikelihoods, aggRecordList, params)


Instruction:
Write a Python 3 function for
updating the anomaly likelihoods for a set of anomaly scores using the given parameters.

Response:
def updateAnomalyLikelihoods(anomalyScores,
                             params,
                             verbosity=0):
  """
  Compute updated probabilities for anomalyScores using the given params.

  :param anomalyScores: a list of records. Each record is a list with the
                        following three elements: [timestamp, value, score]

                        Example::

                            [datetime.datetime(2013, 8, 10, 23, 0), 6.0, 1.0]

  :param params: the JSON dict returned by estimateAnomalyLikelihoods
  :param verbosity: integer controlling extent of printouts for debugging
  :type verbosity: int

  :returns: 3-tuple consisting of:

            - likelihoods

              numpy array of likelihoods, one for each aggregated point

            - avgRecordList

              list of averaged input records

            - params

              an updated JSON object containing the state of this metric.

  """
  if verbosity > 3:
    print("In updateAnomalyLikelihoods.")
    print("Number of anomaly scores:", len(anomalyScores))
    print("First 20:", anomalyScores[0:min(20, len(anomalyScores))])
    print("Params:", params)

  if len(anomalyScores) == 0:
    raise ValueError("Must have at least one anomalyScore")

  if not isValidEstimatorParams(params):
    raise ValueError("'params' is not a valid params structure")

  # For backward compatibility.
  if "historicalLikelihoods" not in params:
    params["historicalLikelihoods"] = [1.0]

  # Compute moving averages of these new scores using the previous values
  # as well as likelihood for these scores using the old estimator
  historicalValues  = params["movingAverage"]["historicalValues"]
  total             = params["movingAverage"]["total"]
  windowSize        = params["movingAverage"]["windowSize"]

  aggRecordList = numpy.zeros(len(anomalyScores), dtype=float)
  likelihoods = numpy.zeros(len(anomalyScores), dtype=float)
  for i, v in enumerate(anomalyScores):
    newAverage, historicalValues, total = (
      MovingAverage.compute(historicalValues, total, v[2], windowSize)
    )
    aggRecordList[i] = newAverage
    likelihoods[i]   = tailProbability(newAverage, params["distribution"])

  # Filter the likelihood values. First we prepend the historical likelihoods
  # to the current set. Then we filter the values.  We peel off the likelihoods
  # to return and the last windowSize values to store for later.
  likelihoods2 = params["historicalLikelihoods"] + list(likelihoods)
  filteredLikelihoods = _filterLikelihoods(likelihoods2)
  likelihoods[:] = filteredLikelihoods[-len(likelihoods):]
  historicalLikelihoods = likelihoods2[-min(windowSize, len(likelihoods2)):]

  # Update the estimator
  newParams = {
    "distribution": params["distribution"],
    "movingAverage": {
      "historicalValues": historicalValues,
      "total": total,
      "windowSize": windowSize,
    },
    "historicalLikelihoods": historicalLikelihoods,
  }

  assert len(newParams["historicalLikelihoods"]) <= windowSize

  if verbosity > 3:
    print("Number of likelihoods:", len(likelihoods))
    print("First 20 likelihoods:", likelihoods[0:min(20, len(likelihoods))])
    print("Leaving updateAnomalyLikelihoods.")

  return (likelihoods, aggRecordList, newParams)


Instruction:
Can you create a Python 3 function that
filters the likelihoods in a list of floats so that we only preserve the likelihoods in likelihood.

Response:
def _filterLikelihoods(likelihoods,
                       redThreshold=0.99999, yellowThreshold=0.999):
  """
  Filter the list of raw (pre-filtered) likelihoods so that we only preserve
  sharp increases in likelihood. 'likelihoods' can be a numpy array of floats or
  a list of floats.

  :returns: A new list of floats likelihoods containing the filtered values.
  """
  redThreshold    = 1.0 - redThreshold
  yellowThreshold = 1.0 - yellowThreshold

  # The first value is untouched
  filteredLikelihoods = [likelihoods[0]]

  for i, v in enumerate(likelihoods[1:]):

    if v <= redThreshold:
      # Value is in the redzone

      if likelihoods[i] > redThreshold:
        # Previous value is not in redzone, so leave as-is
        filteredLikelihoods.append(v)
      else:
        filteredLikelihoods.append(yellowThreshold)

    else:
      # Value is below the redzone, so leave as-is
      filteredLikelihoods.append(v)

  return filteredLikelihoods


Instruction:
Write a Python 3 script for
giving a list of anomaly scores return a list of averaged records. anomalyScores is assumed to be a list of records of the form: [datetime.datetime(2013, 8, 10, 23, 0), 6.0, 1.0] Each record in the returned list list contains: [datetime, value, averagedScore] *Note:* we only average the anomaly score.

Response:
def _anomalyScoreMovingAverage(anomalyScores,
                               windowSize=10,
                               verbosity=0,
                              ):
  """
  Given a list of anomaly scores return a list of averaged records.
  anomalyScores is assumed to be a list of records of the form:
                [datetime.datetime(2013, 8, 10, 23, 0), 6.0, 1.0]

  Each record in the returned list list contains:
      [datetime, value, averagedScore]

  *Note:* we only average the anomaly score.
  """

  historicalValues = []
  total = 0.0
  averagedRecordList = []    # Aggregated records
  for record in anomalyScores:

    # Skip (but log) records without correct number of entries
    if not isinstance(record, (list, tuple)) or len(record) != 3:
      if verbosity >= 1:
        print("Malformed record:", record)
      continue

    avg, historicalValues, total = (
      MovingAverage.compute(historicalValues, total, record[2], windowSize)
      )

    averagedRecordList.append( [record[0], record[1], avg] )

    if verbosity > 2:
      print("Aggregating input record:", record)
      print("Result:", [record[0], record[1], avg])

  return averagedRecordList, historicalValues, total


Instruction:
Write a Python 3 function for
estimating the parameters of a normal distribution based on the sampleData.

Response:
def estimateNormal(sampleData, performLowerBoundCheck=True):
  """
  :param sampleData:
  :type sampleData: Numpy array.
  :param performLowerBoundCheck:
  :type performLowerBoundCheck: bool
  :returns: A dict containing the parameters of a normal distribution based on
      the ``sampleData``.
  """
  params = {
    "name": "normal",
    "mean": numpy.mean(sampleData),
    "variance": numpy.var(sampleData),
  }

  if performLowerBoundCheck:
    # Handle edge case of almost no deviations and super low anomaly scores. We
    # find that such low anomaly means can happen, but then the slightest blip
    # of anomaly score can cause the likelihood to jump up to red.
    if params["mean"] < 0.03:
      params["mean"] = 0.03

    # Catch all for super low variance to handle numerical precision issues
    if params["variance"] < 0.0003:
      params["variance"] = 0.0003

  # Compute standard deviation
  if params["variance"] > 0:
    params["stdev"] = math.sqrt(params["variance"])
  else:
    params["stdev"] = 0

  return params


Instruction:
Explain what the following Python 3 code does
def tailProbability(x, distributionParams):
  if "mean" not in distributionParams or "stdev" not in distributionParams:
    raise RuntimeError("Insufficient parameters to specify the distribution.")

  if x < distributionParams["mean"]:
    # Gaussian is symmetrical around mean, so flip to get the tail probability
    xp = 2 * distributionParams["mean"] - x
    return tailProbability(xp, distributionParams)

  # Calculate the Q function with the complementary error function, explained
  # here: http://www.gaussianwaves.com/2012/07/q-function-and-error-functions
  z = (x - distributionParams["mean"]) / distributionParams["stdev"]
  return 0.5 * math.erfc(z/1.4142)

Response:
Calculates the tail probability of getting samples further
  from the mean and standard deviation of the get - tail - distribution.


Instruction:
Can you write a function in Python 3 where it
checks if the estimator params are valid.

Response:
def isValidEstimatorParams(p):
  """
  :returns: ``True`` if ``p`` is a valid estimator params as might be returned
    by ``estimateAnomalyLikelihoods()`` or ``updateAnomalyLikelihoods``,
    ``False`` otherwise.  Just does some basic validation.
  """
  if not isinstance(p, dict):
    return False
  if "distribution" not in p:
    return False
  if "movingAverage" not in p:
    return False
  dist = p["distribution"]
  if not ("mean" in dist and "name" in dist
          and "variance" in dist and "stdev" in dist):
    return False

  return True


Instruction:
Can you tell what is the following Python 3 function doing
def _calcSkipRecords(numIngested, windowSize, learningPeriod):
    numShiftedOut = max(0, numIngested - windowSize)
    return min(numIngested, max(0, learningPeriod - numShiftedOut))

Response:
Calculates the value of skipRecords for passing to estimateAnomalyLikelihoods
   .


Instruction:
Can you generate the documentation for the following Python 3 function
def read(cls, proto):
    # pylint: disable=W0212
    anomalyLikelihood = object.__new__(cls)
    anomalyLikelihood._iteration = proto.iteration

    anomalyLikelihood._historicalScores = collections.deque(
      maxlen=proto.historicWindowSize)
    for i, score in enumerate(proto.historicalScores):
      anomalyLikelihood._historicalScores.append((i, score.value,
                                                  score.anomalyScore))
    if proto.distribution.name: # is "" when there is no distribution.
      anomalyLikelihood._distribution = dict()
      anomalyLikelihood._distribution['distribution'] = dict()
      anomalyLikelihood._distribution['distribution']["name"] = proto.distribution.name
      anomalyLikelihood._distribution['distribution']["mean"] = proto.distribution.mean
      anomalyLikelihood._distribution['distribution']["variance"] = proto.distribution.variance
      anomalyLikelihood._distribution['distribution']["stdev"] = proto.distribution.stdev

      anomalyLikelihood._distribution["movingAverage"] = {}
      anomalyLikelihood._distribution["movingAverage"]["windowSize"] = proto.distribution.movingAverage.windowSize
      anomalyLikelihood._distribution["movingAverage"]["historicalValues"] = []
      for value in proto.distribution.movingAverage.historicalValues:
        anomalyLikelihood._distribution["movingAverage"]["historicalValues"].append(value)
      anomalyLikelihood._distribution["movingAverage"]["total"] = proto.distribution.movingAverage.total

      anomalyLikelihood._distribution["historicalLikelihoods"] = []
      for likelihood in proto.distribution.historicalLikelihoods:
        anomalyLikelihood._distribution["historicalLikelihoods"].append(likelihood)
    else:
      anomalyLikelihood._distribution = None

    anomalyLikelihood._probationaryPeriod = proto.probationaryPeriod
    anomalyLikelihood._learningPeriod = proto.learningPeriod
    anomalyLikelihood._reestimationPeriod = proto.reestimationPeriod
    # pylint: enable=W0212

    return anomalyLikelihood

Response: This method is used to deserialize the anomaly likelihood object from the given capnp object.


Instruction:
Make a summary of the following Python 3 code
def write(self, proto):

    proto.iteration = self._iteration

    pHistScores = proto.init('historicalScores', len(self._historicalScores))
    for i, score in enumerate(list(self._historicalScores)):
      _, value, anomalyScore = score
      record = pHistScores[i]
      record.value = float(value)
      record.anomalyScore = float(anomalyScore)

    if self._distribution:
      proto.distribution.name = self._distribution["distribution"]["name"]
      proto.distribution.mean = float(self._distribution["distribution"]["mean"])
      proto.distribution.variance = float(self._distribution["distribution"]["variance"])
      proto.distribution.stdev = float(self._distribution["distribution"]["stdev"])

      proto.distribution.movingAverage.windowSize = float(self._distribution["movingAverage"]["windowSize"])

      historicalValues = self._distribution["movingAverage"]["historicalValues"]
      pHistValues = proto.distribution.movingAverage.init(
        "historicalValues", len(historicalValues))
      for i, value in enumerate(historicalValues):
        pHistValues[i] = float(value)

      #proto.distribution.movingAverage.historicalValues = self._distribution["movingAverage"]["historicalValues"]
      proto.distribution.movingAverage.total = float(self._distribution["movingAverage"]["total"])

      historicalLikelihoods = self._distribution["historicalLikelihoods"]
      pHistLikelihoods = proto.distribution.init("historicalLikelihoods",
                                                 len(historicalLikelihoods))
      for i, likelihood in enumerate(historicalLikelihoods):
        pHistLikelihoods[i] = float(likelihood)

    proto.probationaryPeriod = self._probationaryPeriod
    proto.learningPeriod = self._learningPeriod
    proto.reestimationPeriod = self._reestimationPeriod
    proto.historicWindowSize = self._historicalScores.maxlen

Response: Writes the anomaly likelihood object to the given capnp object.


Instruction:
Make a summary of the following Python 3 code
def anomalyProbability(self, value, anomalyScore, timestamp=None):
    if timestamp is None:
      timestamp = self._iteration

    dataPoint = (timestamp, value, anomalyScore)
    # We ignore the first probationaryPeriod data points
    if self._iteration < self._probationaryPeriod:
      likelihood = 0.5
    else:
      # On a rolling basis we re-estimate the distribution
      if ( (self._distribution is None) or
           (self._iteration % self._reestimationPeriod == 0) ):

        numSkipRecords = self._calcSkipRecords(
          numIngested=self._iteration,
          windowSize=self._historicalScores.maxlen,
          learningPeriod=self._learningPeriod)

        _, _, self._distribution = estimateAnomalyLikelihoods(
          self._historicalScores,
          skipRecords=numSkipRecords)

      likelihoods, _, self._distribution = updateAnomalyLikelihoods(
        [dataPoint],
        self._distribution)

      likelihood = 1.0 - likelihoods[0]

    # Before we exit update historical scores and iteration
    self._historicalScores.append(dataPoint)
    self._iteration += 1

    return likelihood

Response:
Compute the probability that the current value plus anomaly score represents
    anomaly score.


Instruction:
How would you implement a function in Python 3 that
creates and returns the _IterationPhaseLearnOnly instance corresponding to this phase specification model

Response:
def _getImpl(self, model):
    """ Creates and returns the _IterationPhase-based instance corresponding
    to this phase specification

    model:          Model instance
    """
    impl = _IterationPhaseLearnOnly(model=model,
                                    nIters=self.__nIters)
    return impl


Instruction:
Implement a function in Python 3 to
create and returns the _IterationPhaseInferOnly instance corresponding to this phase specification model

Response:
def _getImpl(self, model):
    """ Creates and returns the _IterationPhase-based instance corresponding
    to this phase specification

    model:          Model instance
    """
    impl = _IterationPhaseInferOnly(model=model,
                                    nIters=self.__nIters,
                                    inferenceArgs=self.__inferenceArgs)
    return impl


Instruction:
Can you create a Python 3 function that
replaces the Iteration Cycle phases with the given sequence of IterationPhaseSpecXXXXX elements.

Response:
def replaceIterationCycle(self, phaseSpecs):
    """ Replaces the Iteration Cycle phases

    :param phaseSpecs: Iteration cycle description consisting of a sequence of
                  IterationPhaseSpecXXXXX elements that are performed in the
                  given order
    """

    # -----------------------------------------------------------------------
    # Replace our phase manager
    #
    self.__phaseManager = _PhaseManager(
      model=self.__model,
      phaseSpecs=phaseSpecs)

    return


Instruction:
Can you tell what is the following Python 3 function doing
def handleInputRecord(self, inputRecord):
    assert inputRecord, "Invalid inputRecord: %r" % inputRecord

    results = self.__phaseManager.handleInputRecord(inputRecord)
    metrics = self.__metricsMgr.update(results)

    # Execute task-postIter callbacks
    for cb in self.__userCallbacks['postIter']:
      cb(self.__model)

    results.metrics = metrics

    # Return the input and predictions for this record
    return results

Response:
Processes the given record according to the current iteration cycle phase
    Updates the metrics and returns the input and predictions for this record.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def __advancePhase(self):
    self.__currentPhase = self.__phaseCycler.next()
    self.__currentPhase.enterPhase()

    return

Response: Advance to the next iteration cycle phase


Instruction:
How would you explain what the following Python 3 function does
def handleInputRecord(self, inputRecord):

    results = self.__model.run(inputRecord)

    shouldContinue = self.__currentPhase.advance()
    if not shouldContinue:
      self.__advancePhase()

    return results

Response: Processes the given record according to the current phase and returns the results.


Instruction:
Implement a Python 3 function for
initializing the iterator that is necessary upon entry to the phase.

Response:
def enterPhase(self):
    """
    Performs initialization that is necessary upon entry to the phase. Must
    be called before handleInputRecord() at the beginning of each phase
    """

    self.__iter = iter(xrange(self.__nIters))

    # Prime the iterator
    self.__iter.next()


Instruction:
How would you code a function in Python 3 to
advance the iteration ; returns True if more iterations remain ; False otherwise.

Response:
def advance(self):
    """ Advances the iteration;

    Returns:      True if more iterations remain; False if this is the final
                  iteration.
    """
    hasMore = True
    try:
      self.__iter.next()
    except StopIteration:
      self.__iter = None
      hasMore = False

    return hasMore


Instruction:
Make a summary of the following Python 3 code
def enterPhase(self):
    super(_IterationPhaseLearnOnly, self).enterPhase()
    self.__model.enableLearning()
    self.__model.disableInference()
    return

Response: Initializes the learning and inference states for the current iteration.


Instruction:
Create a Python 3 function for
initializing the inference method that is necessary upon entry to the phase.

Response:
def enterPhase(self):
    """ [_IterationPhase method implementation]
    Performs initialization that is necessary upon entry to the phase. Must
    be called before handleInputRecord() at the beginning of each phase
    """
    super(_IterationPhaseInferCommon, self).enterPhase()
    self._model.enableInference(inferenceArgs=self._inferenceArgs)
    return


Instruction:
Can you generate a brief explanation for the following Python 3 code
def write(self, proto):
    super(PreviousValueModel, self).writeBaseToProto(proto.modelBase)

    proto.fieldNames = self._fieldNames
    proto.fieldTypes = self._fieldTypes
    if self._predictedField:
      proto.predictedField = self._predictedField
    proto.predictionSteps = self._predictionSteps

Response:
Serialize via capnp

    :param proto: capnp PreviousValueModelProto message builder


Instruction:
Here you have a function in Python 3, explain what it does
def read(cls, proto):
    instance = object.__new__(cls)
    super(PreviousValueModel, instance).__init__(proto=proto.modelBase)

    instance._logger = opf_utils.initLogger(instance)
    if len(proto.predictedField):
      instance._predictedField = proto.predictedField
    else:
      instance._predictedField = None
    instance._fieldNames = list(proto.fieldNames)
    instance._fieldTypes = list(proto.fieldTypes)
    instance._predictionSteps = list(proto.predictionSteps)

    return instance

Response:
Deserialize via capnp

    :param proto: capnp PreviousValueModelProto message reader

    :returns: new instance of PreviousValueModel deserialized from the given
              proto


Instruction:
Here you have a function in Python 3, explain what it does
def lscsum(lx, epsilon=None):
  lx = numpy.asarray(lx)
  base = lx.max()

  # If the input is the log of 0's, catch this condition before we generate
  #  an exception, and return the log(0)
  if numpy.isinf(base):
    return base

  # If the user specified an epsilon and we are below it, return epsilon
  if (epsilon is not None) and (base < epsilon):
    return epsilon

  x = numpy.exp(lx - base)
  ssum = x.sum()

  result = numpy.log(ssum) + base
  # try:
  #   conventional = numpy.log(numpy.exp(lx).sum())
  #   if not similar(result, conventional):
  #     if numpy.isinf(conventional).any() and not numpy.isinf(result).any():
  #       # print "Scaled log sum avoided underflow or overflow."
  #       pass
  #     else:
  #       import sys
  #       print >>sys.stderr, "Warning: scaled log sum did not match."
  #       print >>sys.stderr, "Scaled log result:"
  #       print >>sys.stderr, result
  #       print >>sys.stderr, "Conventional result:"
  #       print >>sys.stderr, conventional
  # except FloatingPointError, e:
  #   # print "Scaled log sum avoided underflow or overflow."
  #   pass

  return result

Response: Generates a log - sum of the log - values.


Instruction:
Write a Python 3 script for
taking a log - values as input exponentiates them sums down the rows and returns the result.

Response:
def lscsum0(lx):
  """
  Accepts log-values as input, exponentiates them, sums down the rows
  (first dimension), then converts the sum back to log-space and returns the result.
  Handles underflow by rescaling so that the largest values is exactly 1.0.
  """
  # rows = lx.shape[0]
  # columns = numpy.prod(lx.shape[1:])
  # lx = lx.reshape(rows, columns)
  # bases = lx.max(1).reshape(rows, 1)
  # bases = lx.max(0).reshape((1,) + lx.shape[1:])
  lx = numpy.asarray(lx)
  bases = lx.max(0) # Don't need to reshape in the case of 0.
  x = numpy.exp(lx - bases)
  ssum = x.sum(0)

  result = numpy.log(ssum) + bases
  try:
    conventional = numpy.log(numpy.exp(lx).sum(0))

    if not similar(result, conventional):
      if numpy.isinf(conventional).any() and not numpy.isinf(result).any():
        # print "Scaled log sum down axis 0 avoided underflow or overflow."
        pass
      else:
        import sys
        print >>sys.stderr, "Warning: scaled log sum down axis 0 did not match."
        print >>sys.stderr, "Scaled log result:"
        print >>sys.stderr, result
        print >>sys.stderr, "Conventional result:"
        print >>sys.stderr, conventional
  except FloatingPointError, e:
    # print "Scaled log sum down axis 0 avoided underflow or overflow."
    pass

  return result


Instruction:
Can you generate the documentation for the following Python 3 function
def normalize(lx):
  lx = numpy.asarray(lx)
  base = lx.max()
  x = numpy.exp(lx - base)
  result = x / x.sum()

  conventional = (numpy.exp(lx) / numpy.exp(lx).sum())
  assert similar(result, conventional)

  return result

Response: Normalizes the log - values in input and returns the result.


Instruction:
Can you implement a function in Python 3 that
returns the sum of the values of the first dimension of the log - values.

Response:
def nsum0(lx):
  """
  Accepts log-values as input, exponentiates them, sums down the rows
  (first dimension), normalizes and returns the result.
  Handles underflow by rescaling so that the largest values is exactly 1.0.
  """
  lx = numpy.asarray(lx)
  base = lx.max()
  x = numpy.exp(lx - base)
  ssum = x.sum(0)
  result = ssum / ssum.sum()

  conventional = (numpy.exp(lx).sum(0) / numpy.exp(lx).sum())
  assert similar(result, conventional)

  return result


Instruction:
Create a Python 3 function to
return log sum of exp of A and B

Response:
def logSumExp(A, B, out=None):
  """ returns log(exp(A) + exp(B)). A and B are numpy arrays"""

  if out is None:
    out = numpy.zeros(A.shape)

  indicator1 = A >= B
  indicator2 = numpy.logical_not(indicator1)
  out[indicator1] = A[indicator1] + numpy.log1p(numpy.exp(B[indicator1]-A[indicator1]))
  out[indicator2]  = B[indicator2] + numpy.log1p(numpy.exp(A[indicator2]-B[indicator2]))

  return out


Instruction:
Can you create a Python 3 function that
returns log - exp - exp of A - B

Response:
def logDiffExp(A, B, out=None):
  """ returns log(exp(A) - exp(B)). A and B are numpy arrays. values in A should be
  greater than or equal to corresponding values in B"""

  if out is None:
    out = numpy.zeros(A.shape)

  indicator1 = A >= B
  assert indicator1.all(), "Values in the first array should be greater than the values in the second"
  out[indicator1] = A[indicator1] + numpy.log(1 - numpy.exp(B[indicator1]-A[indicator1]))

  return out


Instruction:
Here you have a function in Python 3, explain what it does
def _getFieldIndexBySpecial(fields, special):
  for i, field in enumerate(fields):
    if field.special == special:
      return i
  return None

Response: Return the index of the field with the given meta special value.


Instruction:
Can you tell what is the following Python 3 function doing
def encode(self, inputRow):

    # Create the return dict
    result = dict(zip(self._fieldNames, inputRow))

    # Add in the special fields
    if self._categoryFieldIndex is not None:
      # category value can be an int or a list
      if isinstance(inputRow[self._categoryFieldIndex], int):
        result['_category'] = [inputRow[self._categoryFieldIndex]]
      else:
        result['_category'] = (inputRow[self._categoryFieldIndex]
                               if inputRow[self._categoryFieldIndex]
                               else [None])
    else:
      result['_category'] = [None]

    if self._resetFieldIndex is not None:
      result['_reset'] = int(bool(inputRow[self._resetFieldIndex]))
    else:
      result['_reset'] = 0

    if self._learningFieldIndex is not None:
      result['_learning'] = int(bool(inputRow[self._learningFieldIndex]))

    result['_timestampRecordIdx'] = None
    if self._timestampFieldIndex is not None:
      result['_timestamp'] = inputRow[self._timestampFieldIndex]
      # Compute the record index based on timestamp
      result['_timestampRecordIdx'] = self._computeTimestampRecordIdx(
        inputRow[self._timestampFieldIndex])
    else:
      result['_timestamp'] = None

    # -----------------------------------------------------------------------
    # Figure out the sequence ID
    hasReset = self._resetFieldIndex is not None
    hasSequenceId = self._sequenceFieldIndex is not None
    if hasReset and not hasSequenceId:
      # Reset only
      if result['_reset']:
        self._sequenceId += 1
      sequenceId = self._sequenceId

    elif not hasReset and hasSequenceId:
      sequenceId = inputRow[self._sequenceFieldIndex]
      result['_reset'] = int(sequenceId != self._sequenceId)
      self._sequenceId = sequenceId

    elif hasReset and hasSequenceId:
      sequenceId = inputRow[self._sequenceFieldIndex]

    else:
      sequenceId = 0

    if sequenceId is not None:
      result['_sequenceId'] = hash(sequenceId)
    else:
      result['_sequenceId'] = None

    return result

Response: Encodes the given input row as a dictionary.


Instruction:
Make a summary of the following Python 3 code
def _computeTimestampRecordIdx(self, recordTS):

    if self._aggregationPeriod is None:
      return None

    # Base record index on number of elapsed months if aggregation is in
    #  months
    if self._aggregationPeriod['months'] > 0:
      assert self._aggregationPeriod['seconds'] == 0
      result = int(
        (recordTS.year * 12 + (recordTS.month-1)) /
        self._aggregationPeriod['months'])

    # Base record index on elapsed seconds
    elif self._aggregationPeriod['seconds'] > 0:
      delta = recordTS - datetime.datetime(year=1, month=1, day=1)
      deltaSecs = delta.days * 24 * 60 * 60   \
                + delta.seconds               \
                + delta.microseconds / 1000000.0
      result = int(deltaSecs / self._aggregationPeriod['seconds'])

    else:
      result = None

    return result

Response: Given the timestamp of a record compute the record s timestamp index.


Instruction:
Can you write a function in Python 3 where it
returns the next available data record as a dict.

Response:
def getNextRecordDict(self):
    """Returns next available data record from the storage as a dict, with the
    keys being the field names. This also adds in some meta fields:

      - ``_category``: The value from the category field (if any)
      - ``_reset``: True if the reset field was True (if any)
      - ``_sequenceId``: the value from the sequenceId field (if any)

    """

    values = self.getNextRecord()
    if values is None:
      return None

    if not values:
      return dict()

    if self._modelRecordEncoder is None:
      self._modelRecordEncoder = ModelRecordEncoder(
        fields=self.getFields(),
        aggregationPeriod=self.getAggregationMonthsAndSeconds())

    return self._modelRecordEncoder.encode(values)


Instruction:
Write a Python 3 script to
return the current minimum value for the given field name.

Response:
def getFieldMin(self, fieldName):
    """
    If underlying implementation does not support min/max stats collection,
    or if a field type does not support min/max (non scalars), the return
    value will be None.

    :param fieldName: (string) name of field to get min
    :returns: current minimum value for the field ``fieldName``.
    """
    stats = self.getStats()
    if stats == None:
      return None
    minValues = stats.get('min', None)
    if minValues == None:
      return None
    index = self.getFieldNames().index(fieldName)
    return minValues[index]


Instruction:
Write a Python 3 function for
returning the maximum value for the given field name.

Response:
def getFieldMax(self, fieldName):
    """
    If underlying implementation does not support min/max stats collection,
    or if a field type does not support min/max (non scalars), the return
    value will be None.

    :param fieldName: (string) name of field to get max
    :returns: current maximum value for the field ``fieldName``.
    """
    stats = self.getStats()
    if stats == None:
      return None
    maxValues = stats.get('max', None)
    if maxValues == None:
      return None
    index = self.getFieldNames().index(fieldName)
    return maxValues[index]


Instruction:
Create a Python 3 function to
log msg % args with severity DEBUG.

Response:
def debug(self, msg, *args, **kwargs):
    """
    Log 'msg % args' with severity 'DEBUG'.

    To pass exception information, use the keyword argument exc_info with
    a true value, e.g.

    logger.debug("Houston, we have a %s", "thorny problem", exc_info=1)
    """
    self._baseLogger.debug(self, self.getExtendedMsg(msg), *args, **kwargs)


Instruction:
Create a Python 3 function for
logging msg % args with severity INFO.

Response:
def info(self, msg, *args, **kwargs):
    """
    Log 'msg % args' with severity 'INFO'.

    To pass exception information, use the keyword argument exc_info with
    a true value, e.g.

    logger.info("Houston, we have a %s", "interesting problem", exc_info=1)
    """
    self._baseLogger.info(self, self.getExtendedMsg(msg), *args, **kwargs)


Instruction:
Implement a function in Python 3 to
log msg % args with severity WARNING.

Response:
def warning(self, msg, *args, **kwargs):
    """
    Log 'msg % args' with severity 'WARNING'.

    To pass exception information, use the keyword argument exc_info with
    a true value, e.g.

    logger.warning("Houston, we have a %s", "bit of a problem", exc_info=1)
    """
    self._baseLogger.warning(self, self.getExtendedMsg(msg), *args, **kwargs)


Instruction:
Can you implement a function in Python 3 that
logs msg % args with severity ERROR.

Response:
def error(self, msg, *args, **kwargs):
    """
    Log 'msg % args' with severity 'ERROR'.

    To pass exception information, use the keyword argument exc_info with
    a true value, e.g.

    logger.error("Houston, we have a %s", "major problem", exc_info=1)
    """
    self._baseLogger.error(self, self.getExtendedMsg(msg), *args, **kwargs)


Instruction:
Given the following Python 3 function, write the documentation
def critical(self, msg, *args, **kwargs):
    self._baseLogger.critical(self, self.getExtendedMsg(msg), *args, **kwargs)

Response: Log msg % args with severity CRITICAL.


Instruction:
Create a Python 3 function for
logging msg % args with the integer severity level.

Response:
def log(self, level, msg, *args, **kwargs):
      """
      Log 'msg % args' with the integer severity 'level'.

      To pass exception information, use the keyword argument exc_info with
      a true value, e.g.

      logger.log(level, "We have a %s", "mysterious problem", exc_info=1)
      """
      self._baseLogger.log(self, level, self.getExtendedMsg(msg), *args,
                           **kwargs)


Instruction:
Write a Python 3 function that can
initialize the internal filter variables for further processing.

Response:
def initFilter(input, filterInfo = None):
  """ Initializes internal filter variables for further processing.
  Returns a tuple (function to call,parameters for the filter call)

  The filterInfo is a dict. Here is an example structure:
    {fieldName: {'min': x,
                 'max': y,
                 'type': 'category', # or 'number'
                 'acceptValues': ['foo', 'bar'],
                 }
    }

  This returns the following:
    (filterFunc, ((fieldIdx, fieldFilterFunc, filterDict),
                  ...)

  Where fieldIdx is the index of the field within each record
        fieldFilterFunc returns True if the value is "OK" (within min, max or
           part of acceptValues)
        fieldDict is a dict containing 'type', 'min', max', 'acceptValues'
  """

  if filterInfo is None:
    return None

  # Build an array of index/func to call on record[index]
  filterList = []
  for i, fieldName in enumerate(input.getFieldNames()):
    fieldFilter = filterInfo.get(fieldName, None)
    if fieldFilter == None:
      continue

    var = dict()
    var['acceptValues'] = None
    min = fieldFilter.get('min', None)
    max = fieldFilter.get('max', None)
    var['min'] = min
    var['max'] = max

    if fieldFilter['type'] == 'category':
      var['acceptValues'] = fieldFilter['acceptValues']
      fp = lambda x: (x['value'] != SENTINEL_VALUE_FOR_MISSING_DATA and \
                      x['value'] in x['acceptValues'])

    elif fieldFilter['type'] == 'number':

      if min != None and max != None:
        fp = lambda x: (x['value'] != SENTINEL_VALUE_FOR_MISSING_DATA and \
                        x['value'] >= x['min'] and x['value'] <= x['max'])
      elif min != None:
        fp = lambda x: (x['value'] != SENTINEL_VALUE_FOR_MISSING_DATA and \
                        x['value'] >= x['min'])
      else:
        fp = lambda x: (x['value'] != SENTINEL_VALUE_FOR_MISSING_DATA and \
                        x['value'] <= x['max'])

    filterList.append((i, fp, var))

  return (_filterRecord, filterList)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _filterRecord(filterList, record):

  for (fieldIdx, fp, params) in filterList:
    x = dict()
    x['value'] = record[fieldIdx]
    x['acceptValues'] = params['acceptValues']
    x['min'] = params['min']
    x['max'] = params['max']
    if not fp(x):
      return False

  # None of the field filters triggered, accept the record as a good one
  return True

Response: Takes a record and returns true if it meets the filter criteria and false otherwise.


Instruction:
Here you have a function in Python 3, explain what it does
def _aggr_sum(inList):
  aggrMean = _aggr_mean(inList)
  if aggrMean == None:
    return None

  aggrSum = 0
  for elem in inList:
    if elem != SENTINEL_VALUE_FOR_MISSING_DATA:
      aggrSum += elem
    else:
      aggrSum += aggrMean

  return aggrSum

Response:
Returns the sum of the elements in the list. Missing items are replaced with
 athe.


Instruction:
Can you tell what is the following Python 3 function doing
def _aggr_mean(inList):
  aggrSum = 0
  nonNone = 0
  for elem in inList:
    if elem != SENTINEL_VALUE_FOR_MISSING_DATA:
      aggrSum += elem
      nonNone += 1
  if nonNone != 0:
    return aggrSum / nonNone
  else:
    return None

Response:
Calculates the average of non - None elements of the list
.


Instruction:
Can you generate the documentation for the following Python 3 function
def _aggr_mode(inList):

  valueCounts = dict()
  nonNone = 0

  for elem in inList:
    if elem == SENTINEL_VALUE_FOR_MISSING_DATA:
      continue

    nonNone += 1
    if elem in valueCounts:
      valueCounts[elem] += 1
    else:
      valueCounts[elem] = 1

  # Get the most common one
  if nonNone == 0:
    return None

  # Sort by counts
  sortedCounts = valueCounts.items()
  sortedCounts.sort(cmp=lambda x,y: x[1] - y[1], reverse=True)
  return sortedCounts[0][0]

Response:
Returns the most common value seen in the non - None elements of the list
 """


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _aggr_weighted_mean(inList, params):
  assert(len(inList) == len(params))

  # If all weights are 0, then the value is not defined, return None (missing)
  weightsSum = sum(params)
  if weightsSum == 0:
    return None

  weightedMean = 0
  for i, elem in enumerate(inList):
    weightedMean += elem * params[i]

  return weightedMean / weightsSum

Response: Calculates the weighted mean of inList based on the given parameters.


Instruction:
Make a summary of the following Python 3 code
def generateDataset(aggregationInfo, inputFilename, outputFilename=None):

  # Create the input stream
  inputFullPath = resource_filename("nupic.datafiles", inputFilename)
  inputObj = FileRecordStream(inputFullPath)

  # Instantiate the aggregator
  aggregator = Aggregator(aggregationInfo=aggregationInfo,
                          inputFields=inputObj.getFields())

  # Is it a null aggregation? If so, just return the input file unmodified
  if aggregator.isNullAggregation():
    return inputFullPath

  # ------------------------------------------------------------------------
  # If we were not given an output filename, create one based on the
  #  aggregation settings
  if outputFilename is None:
    outputFilename = 'agg_%s' % \
                        os.path.splitext(os.path.basename(inputFullPath))[0]
    timePeriods = 'years months weeks days '\
                  'hours minutes seconds milliseconds microseconds'
    for k in timePeriods.split():
      if aggregationInfo.get(k, 0) > 0:
        outputFilename += '_%s_%d' % (k, aggregationInfo[k])

    outputFilename += '.csv'
    outputFilename = os.path.join(os.path.dirname(inputFullPath), outputFilename)

  # ------------------------------------------------------------------------
  # If some other process already started creating this file, simply
  #   wait for it to finish and return without doing anything
  lockFilePath = outputFilename + '.please_wait'
  if os.path.isfile(outputFilename) or \
     os.path.isfile(lockFilePath):
    while os.path.isfile(lockFilePath):
      print 'Waiting for %s to be fully written by another process' % \
            lockFilePath
      time.sleep(1)
    return outputFilename

  # Create the lock file
  lockFD = open(lockFilePath, 'w')

  # -------------------------------------------------------------------------
  # Create the output stream
  outputObj = FileRecordStream(streamID=outputFilename, write=True,
                               fields=inputObj.getFields())

  # -------------------------------------------------------------------------
  # Write all aggregated records to the output
  while True:
    inRecord = inputObj.getNextRecord()

    (aggRecord, aggBookmark) = aggregator.next(inRecord, None)

    if aggRecord is None and inRecord is None:
      break

    if aggRecord is not None:
      outputObj.appendRecord(aggRecord)

  return outputFilename

Response: Generates a dataset of aggregated values from the given file.


Instruction:
Write a Python 3 function for
generating the filename for aggregated dataset .

Response:
def getFilename(aggregationInfo, inputFile):
  """Generate the filename for aggregated dataset

  The filename is based on the input filename and the
  aggregation period.

  Returns the inputFile if no aggregation required (aggregation
  info has all 0's)
  """

  # Find the actual file, with an absolute path
  inputFile = resource_filename("nupic.datafiles", inputFile)

  a = defaultdict(lambda: 0, aggregationInfo)
  outputDir = os.path.dirname(inputFile)
  outputFile = 'agg_%s' % os.path.splitext(os.path.basename(inputFile))[0]
  noAggregation = True
  timePeriods = 'years months weeks days '\
                'hours minutes seconds milliseconds microseconds'
  for k in timePeriods.split():
    if a[k] > 0:
      noAggregation = False
      outputFile += '_%s_%d' % (k, a[k])

  if noAggregation:
    return inputFile
  outputFile += '.csv'
  outputFile = os.path.join(outputDir, outputFile)

  return outputFile


Instruction:
Can you generate the documentation for the following Python 3 function
def _getEndTime(self, t):

    assert isinstance(t, datetime.datetime)
    if self._aggTimeDelta:
      return t + self._aggTimeDelta
    else:
      year = t.year + self._aggYears + (t.month - 1 + self._aggMonths) / 12
      month = (t.month - 1 + self._aggMonths) % 12 + 1
      return t.replace(year=year, month=month)

Response: Add the aggregation period to the input time t and return a datetime object.


Instruction:
Can you tell what is the following Python 3 function doing
def _getFuncPtrAndParams(self, funcName):

    params = None
    if isinstance(funcName, basestring):
      if funcName == 'sum':
        fp = _aggr_sum
      elif funcName == 'first':
        fp = _aggr_first
      elif funcName == 'last':
        fp = _aggr_last
      elif funcName == 'mean':
        fp = _aggr_mean
      elif funcName == 'max':
        fp = max
      elif funcName == 'min':
        fp = min
      elif funcName == 'mode':
        fp = _aggr_mode
      elif funcName.startswith('wmean:'):
        fp = _aggr_weighted_mean
        paramsName = funcName[6:]
        params = [f[0] for f in self._inputFields].index(paramsName)
    else:
      fp = funcName

    return (fp, params)

Response: Returns the function pointer and param for the given aggregation function.


Instruction:
How would you explain what the following Python 3 function does
def _createAggregateRecord(self):

    record = []

    for i, (fieldIdx, aggFP, paramIdx) in enumerate(self._fields):
      if aggFP is None: # this field is not supposed to be aggregated.
        continue

      values = self._slice[i]
      refIndex = None
      if paramIdx is not None:
        record.append(aggFP(values, self._slice[paramIdx]))
      else:
        record.append(aggFP(values))

    return record

Response: Generates the aggregated output record for this object.


Instruction:
How would you explain what the following Python 3 function does
def next(self, record, curInputBookmark):

    # This will hold the aggregated record we return
    outRecord = None

    # This will hold the bookmark of the last input used within the
    #  aggregated record we return.
    retInputBookmark = None

    if record is not None:

      # Increment input count
      self._inIdx += 1

      #print self._inIdx, record

      # Apply the filter, ignore the record if any field is unacceptable
      if self._filter != None and not self._filter[0](self._filter[1], record):
        return (None, None)

      # If no aggregation info just return as-is
      if self._nullAggregation:
        return (record, curInputBookmark)

      # ----------------------------------------------------------------------
      # Do aggregation

      #
      # Remember the very first record time stamp - it will be used as
      # the timestamp for all first records in all sequences to align
      # times for the aggregation/join of sequences.
      #
      # For a set of aggregated records, it will use the beginning of the time
      # window as a timestamp for the set
      #
      t = record[self._timeFieldIdx]

      if self._firstSequenceStartTime == None:
        self._firstSequenceStartTime = t

      # Create initial startTime and endTime if needed
      if self._startTime is None:
        self._startTime = t
      if self._endTime is None:
        self._endTime = self._getEndTime(t)
        assert self._endTime > t

      #print 'Processing line:', i, t, endTime
      #from dbgp.client import brk; brk(port=9011)

      # ----------------------------------------------------------------------
      # Does this record have a reset signal or sequence Id associated with it?
      # If so, see if we've reached a sequence boundary
      if self._resetFieldIdx is not None:
        resetSignal = record[self._resetFieldIdx]
      else:
        resetSignal = None

      if self._sequenceIdFieldIdx is not None:
        currSequenceId = record[self._sequenceIdFieldIdx]
      else:
        currSequenceId = None

      newSequence = (resetSignal == 1 and self._inIdx > 0) \
                      or self._sequenceId != currSequenceId \
                      or self._inIdx == 0

      if newSequence:
        self._sequenceId = currSequenceId

      # --------------------------------------------------------------------
      # We end the aggregation chunk if we go past the end time
      # -OR- we get an out of order record (t < startTime)
      sliceEnded = (t >= self._endTime or t < self._startTime)

      # -------------------------------------------------------------------
      # Time to generate a new output record?
      if (newSequence or sliceEnded) and len(self._slice) > 0:
        # Create aggregated record
        # print 'Creating aggregate record...'

        # Make first record timestamp as the beginning of the time period,
        # in case the first record wasn't falling on the beginning of the period
        for j, f in enumerate(self._fields):
          index = f[0]
          if index == self._timeFieldIdx:
            self._slice[j][0] = self._startTime
            break

        # Generate the aggregated record
        outRecord = self._createAggregateRecord()
        retInputBookmark = self._aggrInputBookmark

        # Reset the slice
        self._slice = defaultdict(list)

      # --------------------------------------------------------------------
      # Add current record to slice (Note keeping slices in memory). Each
      # field in the slice is a list of field values from all the sliced
      # records
      for j, f in enumerate(self._fields):
        index = f[0]
        # append the parsed field value to the proper aggregated slice field.
        self._slice[j].append(record[index])
        self._aggrInputBookmark = curInputBookmark

      # --------------------------------------------------------------------
      # If we've encountered a new sequence, start aggregation over again
      if newSequence:
        # TODO: May use self._firstSequenceStartTime as a start for the new
        # sequence (to align all sequences)
        self._startTime = t
        self._endTime = self._getEndTime(t)

      # --------------------------------------------------------------------
      # If a slice just ended, re-compute the start and end time for the
      #  next aggregated record
      if sliceEnded:
        # Did we receive an out of order record? If so, go back and iterate
        #   till we get to the next end time boundary.
        if t < self._startTime:
          self._endTime = self._firstSequenceStartTime
        while t >= self._endTime:
          self._startTime = self._endTime
          self._endTime = self._getEndTime(self._endTime)

      # If we have a record to return, do it now
      if outRecord is not None:
        return (outRecord, retInputBookmark)

    # ---------------------------------------------------------------------
    # Input reached EOF
    # Aggregate one last time in the end if necessary
    elif self._slice:

      # Make first record timestamp as the beginning of the time period,
      # in case the first record wasn't falling on the beginning of the period
      for j, f in enumerate(self._fields):
        index = f[0]
        if index == self._timeFieldIdx:
          self._slice[j][0] = self._startTime
          break

      outRecord = self._createAggregateRecord()
      retInputBookmark = self._aggrInputBookmark

      self._slice = defaultdict(list)

    # Return aggregated record
    return (outRecord, retInputBookmark)

Response: Returns the next aggregated record in the sequence of records.


Instruction:
Explain what the following Python 3 code does
def processClubAttendance(f, clubs):
  try:
    # Skip as many empty lines as necessary (file format inconsistent)
    line = f.next()
    while line == ',,,,,,,,,,,,,,,,,,,\n':
      line = f.next()

    # The first non-empty line should have the name as the first field
    name = line.split(',')[0]

    # Create a new club object if needed
    if name not in clubs:
      clubs[name] = Club(name)

    # Get the named club
    c = clubs[name]

    c.processAttendance(f)
    return True
  except StopIteration:
    return False

Response: Process the attendance data of one club


Instruction:
Can you tell what is the following Python 3 function doing
def processClubConsumption(f, clubs):
  try:
    # Skip header line
    line = f.next()
    assert line.endswith('"   ","SITE_LOCATION_NAME","TIMESTAMP","TOTAL_KWH"\n')

    valid_times = range(24)
    t = 0 # used to track time
    club = None
    clubName = None
    lastDate = None
    while True:
      assert t in valid_times
      consumption = 0
      for x in range(4):
        # Read the line and get rid of the newline character
        line = f.next()[:-1]
        fields = line.split(',')
        assert len(fields) == 4
        for i, field in enumerate(fields):
          # Strip the redundant double quotes
          assert field[0] == '"' and field[-1] == '"'
          fields[i] = field[1:-1]

        # Ignoring field 0, which is just a running count

        # Get the club name
        name = fields[1]

        # Hack to fix inconsistent club names like: "Melbourne CBD - Melbourne Central" vs. "Melbourne Central"
        partialNames = ('Melbourne Central', 'North Sydney', 'Park St', 'Pitt St')
        for pn in partialNames:
          if pn in name:
            name = pn

        # Locate the club if needed (maybe )
        if name != clubName:
          clubName = name
          club = clubs[name]

        # Split the date (time is counted using the t variable)
        tokens = fields[2].split()

        # Verify that t == 0 and consumption == 0 when there is no time in the file
        if len(tokens) == 1:
          assert consumption == 0 and t == 0

        # The first (and sometimes only) token is the date
        date = tokens[0]

        # Aggregate the consumption
        consumption += float(fields[3])

      # Update the Club object after aggregating the consumption of 4 lines
      club.updateRecord(date, t, consumption)

      # Increment time
      t += 1
      t %= 24
  except StopIteration:
    return

Response: Process the consumption a clubConsumption a clubVersion is a bit more efficient than reading the consumption a clubVersion is a bit more efficient than reading the clubVersion and parsing the line until the next line.


Instruction:
Write a Python 3 function that can
run one iteration of this model.

Response:
def run(self, inputRecord):
    """
    Run one iteration of this model.

    :param inputRecord: (object)
           A record object formatted according to
           :meth:`~nupic.data.record_stream.RecordStreamIface.getNextRecord` or
           :meth:`~nupic.data.record_stream.RecordStreamIface.getNextRecordDict`
           result format.
    :returns: (:class:`~nupic.frameworks.opf.opf_utils.ModelResult`)
             An ModelResult namedtuple. The contents of ModelResult.inferences
             depends on the the specific inference type of this model, which
             can be queried by :meth:`.getInferenceType`.
    """
    # 0-based prediction index for ModelResult
    predictionNumber = self._numPredictions
    self._numPredictions += 1
    result = opf_utils.ModelResult(predictionNumber=predictionNumber,
                                   rawInput=inputRecord)
    return result


Instruction:
Can you generate the documentation for the following Python 3 function
def _getModelCheckpointFilePath(checkpointDir):
    path = os.path.join(checkpointDir, "model.data")
    path = os.path.abspath(path)
    return path

Response: Returns the absolute path of the model s checkpoint file.


Instruction:
How would you code a function in Python 3 to
serialize model using capnproto and writes data to checkpointDir

Response:
def writeToCheckpoint(self, checkpointDir):
    """Serializes model using capnproto and writes data to ``checkpointDir``"""
    proto = self.getSchema().new_message()

    self.write(proto)

    checkpointPath = self._getModelCheckpointFilePath(checkpointDir)

    # Clean up old saved state, if any
    if os.path.exists(checkpointDir):
      if not os.path.isdir(checkpointDir):
        raise Exception(("Existing filesystem entry <%s> is not a model"
                         " checkpoint -- refusing to delete (not a directory)") \
                          % checkpointDir)
      if not os.path.isfile(checkpointPath):
        raise Exception(("Existing filesystem entry <%s> is not a model"
                         " checkpoint -- refusing to delete"\
                         " (%s missing or not a file)") % \
                          (checkpointDir, checkpointPath))

      shutil.rmtree(checkpointDir)

    # Create a new directory for saving state
    self.__makeDirectoryFromAbsolutePath(checkpointDir)

    with open(checkpointPath, 'wb') as f:
      proto.write(f)


Instruction:
Given the following Python 3 function, write the documentation
def readFromCheckpoint(cls, checkpointDir):
    checkpointPath = cls._getModelCheckpointFilePath(checkpointDir)

    with open(checkpointPath, 'r') as f:
      proto = cls.getSchema().read(f,
                                   traversal_limit_in_words=_TRAVERSAL_LIMIT_IN_WORDS)

    model = cls.read(proto)
    return model

Response: Deserializes model from checkpointDir using capnproto


Instruction:
Can you tell what is the following Python 3 function doing
def writeBaseToProto(self, proto):
    inferenceType = self.getInferenceType()
    # lower-case first letter to be compatible with capnproto enum naming
    inferenceType = inferenceType[:1].lower() + inferenceType[1:]
    proto.inferenceType = inferenceType

    proto.numPredictions = self._numPredictions

    proto.learningEnabled = self.__learningEnabled
    proto.inferenceEnabled = self.__inferenceEnabled
    proto.inferenceArgs = json.dumps(self.__inferenceArgs)

Response: Save the state maintained by the Model base class to the given capnp. ModelProto message builder.


Instruction:
Explain what the following Python 3 code does
def save(self, saveModelDir):
    logger = self._getLogger()
    logger.debug("(%s) Creating local checkpoint in %r...",
                       self, saveModelDir)

    modelPickleFilePath = self._getModelPickleFilePath(saveModelDir)

    # Clean up old saved state, if any
    if os.path.exists(saveModelDir):
      if not os.path.isdir(saveModelDir):
        raise Exception(("Existing filesystem entry <%s> is not a model"
                         " checkpoint -- refusing to delete (not a directory)") \
                          % saveModelDir)
      if not os.path.isfile(modelPickleFilePath):
        raise Exception(("Existing filesystem entry <%s> is not a model"
                         " checkpoint -- refusing to delete"\
                         " (%s missing or not a file)") % \
                          (saveModelDir, modelPickleFilePath))

      shutil.rmtree(saveModelDir)

    # Create a new directory for saving state
    self.__makeDirectoryFromAbsolutePath(saveModelDir)

    with open(modelPickleFilePath, 'wb') as modelPickleFile:
      logger.debug("(%s) Pickling Model instance...", self)

      pickle.dump(self, modelPickleFile, protocol=pickle.HIGHEST_PROTOCOL)

      logger.debug("(%s) Finished pickling Model instance", self)

    # Tell the model to save extra data, if any, that's too big for pickling
    self._serializeExtraData(extraDataDir=self._getModelExtraDataDir(saveModelDir))

    logger.debug("(%s) Finished creating local checkpoint", self)

    return

Response: Save the current state of the current model in the given directory.


Instruction:
Can you tell what is the following Python 3 function doing
def load(cls, savedModelDir):
    logger = opf_utils.initLogger(cls)
    logger.debug("Loading model from local checkpoint at %r...", savedModelDir)

    # Load the model
    modelPickleFilePath = Model._getModelPickleFilePath(savedModelDir)

    with open(modelPickleFilePath, 'rb') as modelPickleFile:
      logger.debug("Unpickling Model instance...")

      model = pickle.load(modelPickleFile)

      logger.debug("Finished unpickling Model instance")

    # Tell the model to load extra data, if any, that was too big for pickling
    model._deSerializeExtraData(
        extraDataDir=Model._getModelExtraDataDir(savedModelDir))

    logger.debug("Finished Loading model from local checkpoint")

    return model

Response: Loads the saved model.


Instruction:
Create a Python 3 function to
return the absolute path of the model s pickle file.

Response:
def _getModelPickleFilePath(saveModelDir):
    """ Return the absolute path of the model's pickle file.

    :param saveModelDir: (string)
           Directory of where the experiment is to be or was saved
    :returns: (string) An absolute path.
    """
    path = os.path.join(saveModelDir, "model.pkl")
    path = os.path.abspath(path)
    return path


Instruction:
Explain what the following Python 3 code does
def _getModelExtraDataDir(saveModelDir):
    path = os.path.join(saveModelDir, "modelextradata")
    path = os.path.abspath(path)

    return path

Response:
Returns the absolute path to the directory where the model s own
    extra data are stored.


Instruction:
Write a Python 3 script to
run an experiment on the given list of arguments and returns the model instance that was created.

Response:
def runExperiment(args, model=None):
  """
  Run a single OPF experiment.

  .. note:: The caller is responsible for initializing python logging before
     calling this function (e.g., import :mod:`nupic.support`;
     :meth:`nupic.support.initLogging`)

  See also: :meth:`.initExperimentPrng`.

  :param args: (string) Experiment command-line args list. Too see all options,
      run with ``--help``:

      .. code-block:: text

        Options:
          -h, --help           show this help message and exit
          -c <CHECKPOINT>      Create a model and save it under the given <CHECKPOINT>
                               name, but don't run it
          --listCheckpoints    List all available checkpoints
          --listTasks          List all task labels in description.py
          --load=<CHECKPOINT>  Load a model from the given <CHECKPOINT> and run it.
                               Run with --listCheckpoints flag for more details.
          --newSerialization   Use new capnproto serialization
          --tasks              Run the tasks with the given TASK LABELS in the order
                               they are given.  Either end of arg-list, or a
                               standalone dot ('.') arg or the next short or long
                               option name (-a or --blah) terminates the list. NOTE:
                               FAILS TO RECOGNIZE task label names with one or more
                               leading dashes. [default: run all of the tasks in
                               description.py]
          --testMode           Reduce iteration count for testing
          --noCheckpoint       Don't checkpoint the model after running each task.

  :param model: (:class:`~nupic.frameworks.opf.model.Model`) For testing, may
      pass in an existing OPF Model to use instead of creating a new one.

  :returns: (:class:`~nupic.frameworks.opf.model.Model`)
    reference to OPF Model instance that was constructed (this
    is provided to aid with debugging) or None, if none was
    created.
  """
  # Parse command-line options
  opt = _parseCommandLineOptions(args)

  #print "runExperiment: Parsed Command Options: ", opt

  model = _runExperimentImpl(opt, model)

  return model


Instruction:
Can you tell what is the following Python 3 function doing
def _parseCommandLineOptions(args):
  usageStr = (
    "%prog [options] descriptionPyDirectory\n"
    "This script runs a single OPF Model described by description.py "
    "located in the given directory."
    )

  parser = optparse.OptionParser(usage=usageStr)

  parser.add_option("-c",
                    help="Create a model and save it under the given "
                         "<CHECKPOINT> name, but don't run it",
                    dest="createCheckpointName",
                    action="store", type="string", default="",
                    metavar="<CHECKPOINT>")

  parser.add_option("--listCheckpoints",
                    help="List all available checkpoints",
                    dest="listAvailableCheckpoints",
                    action="store_true", default=False)

  parser.add_option("--listTasks",
                    help="List all task labels in description.py",
                    dest="listTasks",
                    action="store_true", default=False)

  parser.add_option("--load",
                    help="Load a model from the given <CHECKPOINT> and run it. "
                         "Run with --listCheckpoints flag for more details. ",
                    dest="runCheckpointName",
                    action="store", type="string", default="",
                    metavar="<CHECKPOINT>")

  parser.add_option("--newSerialization",
                    help="Use new capnproto serialization",
                    dest="newSerialization",
                    action="store_true", default=False)

  #parser.add_option("--reuseDatasets",
  #                  help="Keep existing generated/aggregated datasets",
  #                  dest="reuseDatasets", action="store_true",
  #                  default=False)

  parser.add_option("--tasks",
                    help="Run the tasks with the given TASK LABELS "
                         "in the order they are given.  Either end of "
                         "arg-list, or a standalone dot ('.') arg or "
                         "the next short or long option name (-a or "
                         "--blah) terminates the list. NOTE: FAILS "
                         "TO RECOGNIZE task label names with one or more "
                         "leading dashes. [default: run all of the tasks in "
                         "description.py]",
                    dest="taskLabels", default=[],
                    action="callback", callback=reapVarArgsCallback,
                    metavar="TASK_LABELS")

  parser.add_option("--testMode",
                    help="Reduce iteration count for testing",
                    dest="testMode", action="store_true",
                    default=False)

  parser.add_option("--noCheckpoint",
                    help="Don't checkpoint the model after running each task.",
                    dest="checkpointModel", action="store_false",
                    default=True)

  options, experiments = parser.parse_args(args)

  # Validate args
  mutuallyExclusiveOptionCount = sum([bool(options.createCheckpointName),
                                      options.listAvailableCheckpoints,
                                      options.listTasks,
                                      bool(options.runCheckpointName)])
  if mutuallyExclusiveOptionCount > 1:
    _reportCommandLineUsageErrorAndExit(
        parser,
        "Options: -c, --listCheckpoints, --listTasks, and --load are "
        "mutually exclusive. Please select only one")

  mutuallyExclusiveOptionCount = sum([bool(not options.checkpointModel),
                                      bool(options.createCheckpointName)])
  if mutuallyExclusiveOptionCount > 1:
    _reportCommandLineUsageErrorAndExit(
        parser,
        "Options: -c and --noCheckpoint are "
        "mutually exclusive. Please select only one")

  if len(experiments) != 1:
    _reportCommandLineUsageErrorAndExit(
        parser,
        "Exactly ONE experiment must be specified, but got %s (%s)" % (
            len(experiments), experiments))

  # Done with parser
  parser.destroy()

  # Prepare results

  # Directory path of the experiment (that contain description.py)
  experimentDir = os.path.abspath(experiments[0])

  # RunExperiment.py's private options (g_parsedPrivateCommandLineOptionsSchema)
  privateOptions = dict()
  privateOptions['createCheckpointName'] = options.createCheckpointName
  privateOptions['listAvailableCheckpoints'] = options.listAvailableCheckpoints
  privateOptions['listTasks'] = options.listTasks
  privateOptions['runCheckpointName'] = options.runCheckpointName
  privateOptions['newSerialization'] = options.newSerialization
  privateOptions['testMode'] = options.testMode
  #privateOptions['reuseDatasets']  = options.reuseDatasets
  privateOptions['taskLabels'] = options.taskLabels
  privateOptions['checkpointModel'] = options.checkpointModel

  result = ParseCommandLineOptionsResult(experimentDir=experimentDir,
                                         privateOptions=privateOptions)
  return result

Response: Parse command line options and return a namedtuple with the parsed options.


Instruction:
How would you explain what the following Python 3 function does
def reapVarArgsCallback(option, optStr, value, parser):
  newValues = []

  # Reap the args, taking care to stop before the next option or '.'
  gotDot = False
  for arg in parser.rargs:
    # Stop on --longname options
    if arg.startswith("--") and len(arg) > 2:
      break

    # Stop on -b options
    if arg.startswith("-") and len(arg) > 1:
      break

    if arg == ".":
      gotDot = True
      break

    newValues.append(arg)

  if not newValues:
    raise optparse.OptionValueError(
      ("Empty arg list for option %r expecting one or more args "
       "(remaining tokens: %r)") % (optStr, parser.rargs))

  del parser.rargs[:len(newValues) + int(gotDot)]

  # Retrieve the existing arg accumulator, if any
  value = getattr(parser.values, option.dest, [])
  #print "Previous value: %r" % value
  if value is None:
    value = []

  # Append the new args to the existing ones and save to the parser
  value.extend(newValues)
  setattr(parser.values, option.dest, value)

Response: This callback is used to reap the variable number of option args. It will reap the args and save the value to the parser. values attribute.


Instruction:
Implement a Python 3 function for
reporting usage error and exit program with error indication.

Response:
def _reportCommandLineUsageErrorAndExit(parser, message):
  """Report usage error and exit program with error indication."""
  print parser.get_usage()
  print message
  sys.exit(1)


Instruction:
Write a Python 3 function that can
create and runs the experiment from the given options and model.

Response:
def _runExperimentImpl(options, model=None):
  """Creates and runs the experiment

  Args:
    options: namedtuple ParseCommandLineOptionsResult
    model: For testing: may pass in an existing OPF Model instance
        to use instead of creating a new one.

  Returns: reference to OPFExperiment instance that was constructed (this
      is provided to aid with debugging) or None, if none was
      created.
  """
  json_helpers.validate(options.privateOptions,
                        schemaDict=g_parsedPrivateCommandLineOptionsSchema)

  # Load the experiment's description.py module
  experimentDir = options.experimentDir
  descriptionPyModule = helpers.loadExperimentDescriptionScriptFromDir(
      experimentDir)
  expIface = helpers.getExperimentDescriptionInterfaceFromModule(
      descriptionPyModule)

  # Handle "list checkpoints" request
  if options.privateOptions['listAvailableCheckpoints']:
    _printAvailableCheckpoints(experimentDir)
    return None

  # Load experiment tasks
  experimentTasks = expIface.getModelControl().get('tasks', [])

  # If the tasks list is empty, and this is a nupic environment description
  # file being run from the OPF, convert it to a simple OPF description file.
  if (len(experimentTasks) == 0 and
      expIface.getModelControl()['environment'] == OpfEnvironment.Nupic):
    expIface.convertNupicEnvToOPF()
    experimentTasks = expIface.getModelControl().get('tasks', [])

  # Ensures all the source locations are either absolute paths or relative to
  # the nupic.datafiles package_data location.
  expIface.normalizeStreamSources()

  # Extract option
  newSerialization = options.privateOptions['newSerialization']

  # Handle listTasks
  if options.privateOptions['listTasks']:
    print "Available tasks:"

    for label in [t['taskLabel'] for t in experimentTasks]:
      print "\t", label

    return None

  # Construct the experiment instance
  if options.privateOptions['runCheckpointName']:

    assert model is None

    checkpointName = options.privateOptions['runCheckpointName']

    model = ModelFactory.loadFromCheckpoint(
          savedModelDir=_getModelCheckpointDir(experimentDir, checkpointName),
          newSerialization=newSerialization)

  elif model is not None:
    print "Skipping creation of OPFExperiment instance: caller provided his own"
  else:
    modelDescription = expIface.getModelDescription()
    model = ModelFactory.create(modelDescription)

  # Handle "create model" request
  if options.privateOptions['createCheckpointName']:
    checkpointName = options.privateOptions['createCheckpointName']
    _saveModel(model=model,
               experimentDir=experimentDir,
               checkpointLabel=checkpointName,
               newSerialization=newSerialization)

    return model

  # Build the task list

  # Default task execution index list is in the natural list order of the tasks
  taskIndexList = range(len(experimentTasks))

  customTaskExecutionLabelsList = options.privateOptions['taskLabels']
  if customTaskExecutionLabelsList:
    taskLabelsList = [t['taskLabel'] for t in experimentTasks]
    taskLabelsSet = set(taskLabelsList)

    customTaskExecutionLabelsSet = set(customTaskExecutionLabelsList)

    assert customTaskExecutionLabelsSet.issubset(taskLabelsSet), \
           ("Some custom-provided task execution labels don't correspond "
            "to actual task labels: mismatched labels: %r; actual task "
            "labels: %r.") % (customTaskExecutionLabelsSet - taskLabelsSet,
                              customTaskExecutionLabelsList)

    taskIndexList = [taskLabelsList.index(label) for label in
                     customTaskExecutionLabelsList]

    print "#### Executing custom task list: %r" % [taskLabelsList[i] for
                                                   i in taskIndexList]

  # Run all experiment tasks
  for taskIndex in taskIndexList:

    task = experimentTasks[taskIndex]

    # Create a task runner and run it!
    taskRunner = _TaskRunner(model=model,
                             task=task,
                             cmdOptions=options)
    taskRunner.run()
    del taskRunner

    if options.privateOptions['checkpointModel']:
      _saveModel(model=model,
                 experimentDir=experimentDir,
                 checkpointLabel=task['taskLabel'],
                 newSerialization=newSerialization)

  return model


Instruction:
Given the following Python 3 function, write the documentation
def _saveModel(model, experimentDir, checkpointLabel, newSerialization=False):
  checkpointDir = _getModelCheckpointDir(experimentDir, checkpointLabel)
  if newSerialization:
    model.writeToCheckpoint(checkpointDir)
  else:
    model.save(saveModelDir=checkpointDir)

Response: Save a model to checkpoint.


Instruction:
Make a summary of the following Python 3 code
def _getModelCheckpointDir(experimentDir, checkpointLabel):
  checkpointDir = os.path.join(getCheckpointParentDir(experimentDir),
                               checkpointLabel + g_defaultCheckpointExtension)
  checkpointDir = os.path.abspath(checkpointDir)

  return checkpointDir

Response: Creates the directory for serialization of the model


Instruction:
Given the following Python 3 function, write the documentation
def getCheckpointParentDir(experimentDir):
  baseDir = os.path.join(experimentDir, "savedmodels")
  baseDir = os.path.abspath(baseDir)

  return baseDir

Response:
Get the absolute path to the base serialization directory within which
      model checkpoints are created


Instruction:
Can you generate the documentation for the following Python 3 function
def _checkpointLabelFromCheckpointDir(checkpointDir):
  assert checkpointDir.endswith(g_defaultCheckpointExtension)

  lastSegment = os.path.split(checkpointDir)[1]

  checkpointLabel = lastSegment[0:-len(g_defaultCheckpointExtension)]

  return checkpointLabel

Response: Returns a checkpoint label string for the given checkpoint directory.


Instruction:
Write a Python 3 script for
returning true iff checkpointDir appears to be a checkpoint directory.

Response:
def _isCheckpointDir(checkpointDir):
  """Return true iff checkpointDir appears to be a checkpoint directory."""
  lastSegment = os.path.split(checkpointDir)[1]
  if lastSegment[0] == '.':
    return False

  if not checkpointDir.endswith(g_defaultCheckpointExtension):
    return False

  if not os.path.isdir(checkpointDir):
    return False

  return True


Instruction:
Can you tell what is the following Python 3 function doing
def _printAvailableCheckpoints(experimentDir):
  checkpointParentDir = getCheckpointParentDir(experimentDir)

  if not os.path.exists(checkpointParentDir):
    print "No available checkpoints."
    return

  checkpointDirs = [x for x in os.listdir(checkpointParentDir)
                    if _isCheckpointDir(os.path.join(checkpointParentDir, x))]
  if not checkpointDirs:
    print "No available checkpoints."
    return

  print "Available checkpoints:"
  checkpointList = [_checkpointLabelFromCheckpointDir(x)
                    for x in checkpointDirs]

  for checkpoint in sorted(checkpointList):
    print "\t", checkpoint

  print
  print "To start from a checkpoint:"
  print "  python run_opf_experiment.py experiment --load <CHECKPOINT>"
  print "For example, to start from the checkpoint \"MyCheckpoint\":"
  print "  python run_opf_experiment.py experiment --load MyCheckpoint"

Response: List available checkpoints for the specified experiment.


Instruction:
Given the following Python 3 function, write the documentation
def run(self):
    self.__logger.debug("run(): Starting task <%s>", self.__task['taskLabel'])

    # Set up the task

    # Create our main loop-control iterator
    if self.__cmdOptions.privateOptions['testMode']:
      numIters = 10
    else:
      numIters = self.__task['iterationCount']

    if numIters >= 0:
      iterTracker = iter(xrange(numIters))
    else:
      iterTracker = iter(itertools.count())

    # Initialize periodic activities
    periodic = PeriodicActivityMgr(
      requestedActivities=self._createPeriodicActivities())

    # Reset sequence states in the model, so it starts looking for a new
    # sequence
    # TODO: should this be done in OPFTaskDriver.setup(), instead?  Is it always
    #       desired in Nupic?
    self.__model.resetSequenceStates()

    # Have Task Driver perform its initial setup activities, including setup
    # callbacks
    self.__taskDriver.setup()

    # Run it!
    while True:
      # Check controlling iterator first
      try:
        next(iterTracker)
      except StopIteration:
        break

      # Read next input record
      try:
        inputRecord = self.__datasetReader.next()
      except StopIteration:
        break

      # Process input record
      result = self.__taskDriver.handleInputRecord(inputRecord=inputRecord)

      if InferenceElement.encodings in result.inferences:
        result.inferences.pop(InferenceElement.encodings)
      self.__predictionLogger.writeRecord(result)

      # Run periodic activities
      periodic.tick()

    # Dump the experiment metrics at the end of the task
    self._getAndEmitExperimentMetrics(final=True)

    # Have Task Driver perform its final activities
    self.__taskDriver.finalize()

    # Reset sequence states in the model, so it starts looking for a new
    # sequence
    # TODO: should this be done in OPFTaskDriver.setup(), instead?  Is it always
    #       desired in Nupic?
    self.__model.resetSequenceStates()

Response: Runs a single experiment task


Instruction:
Write a Python 3 script for
creating and returns a list of PeriodicActivityRequest elements for this TaskRunner instance

Response:
def _createPeriodicActivities(self):
    """Creates and returns a list of activites for this TaskRunner instance

    Returns: a list of PeriodicActivityRequest elements
    """
    # Initialize periodic activities
    periodicActivities = []

    # Metrics reporting
    class MetricsReportCb(object):
      def __init__(self, taskRunner):
        self.__taskRunner = taskRunner
        return

      def __call__(self):
        self.__taskRunner._getAndEmitExperimentMetrics()

    reportMetrics = PeriodicActivityRequest(
      repeating=True,
      period=1000,
      cb=MetricsReportCb(self))

    periodicActivities.append(reportMetrics)

    # Iteration progress
    class IterationProgressCb(object):
      PROGRESS_UPDATE_PERIOD_TICKS = 1000

      def __init__(self, taskLabel, requestedIterationCount, logger):
        self.__taskLabel = taskLabel
        self.__requestedIterationCount = requestedIterationCount
        self.__logger = logger

        self.__numIterationsSoFar = 0

      def __call__(self):
        self.__numIterationsSoFar += self.PROGRESS_UPDATE_PERIOD_TICKS
        self.__logger.debug("%s: ITERATION PROGRESS: %s of %s" % (
                              self.__taskLabel,
                              self.__numIterationsSoFar,
                              self.__requestedIterationCount))

    iterationProgressCb = IterationProgressCb(
      taskLabel=self.__task['taskLabel'],
      requestedIterationCount=self.__task['iterationCount'],
      logger=self.__logger)
    iterationProgressReporter = PeriodicActivityRequest(
      repeating=True,
      period=IterationProgressCb.PROGRESS_UPDATE_PERIOD_TICKS,
      cb=iterationProgressCb)

    periodicActivities.append(iterationProgressReporter)

    return periodicActivities


Instruction:
Make a summary of the following Python 3 code
def _generateFile(filename, data):

  # Create the file
  print "Creating %s..." % (filename)
  numRecords, numFields = data.shape

  fields = [('field%d'%(i+1), 'float', '') for i in range(numFields)]
  outFile = File(filename, fields)

  for i in xrange(numRecords):
    outFile.write(data[i].tolist())

  outFile.close()

Response: Generates the file for the sequence number.


Instruction:
Can you tell what is the following Python 3 function doing
def corruptVector(v1, noiseLevel, numActiveCols):
  size = len(v1)
  v2 = np.zeros(size, dtype="uint32")
  bitsToSwap = int(noiseLevel * numActiveCols)
  # Copy the contents of v1 into v2
  for i in range(size):
    v2[i] = v1[i]
  for _ in range(bitsToSwap):
    i = random.randrange(size)
    if v2[i] == 1:
      v2[i] = 0
    else:
      v2[i] = 1
  return v2

Response: Corrupts a copy of a binary vector by inverting noiseLevel percent of its bits.


Instruction:
Write a Python 3 function that can
show predictions of the TM when presented with the characters A B C D X and Y without any contextual information that is embedded within a sequence.

Response:
def showPredictions():
  """
  Shows predictions of the TM when presented with the characters A, B, C, D, X, and
  Y without any contextual information, that is, not embedded within a sequence.
  """
  for k in range(6):
    tm.reset()
    print "--- " + "ABCDXY"[k] + " ---"
    tm.compute(set(seqT[k][:].nonzero()[0].tolist()), learn=False)
    activeColumnsIndices = [tm.columnForCell(i) for i in tm.getActiveCells()]
    predictedColumnIndices = [tm.columnForCell(i) for i in tm.getPredictiveCells()]
    currentColumns = [1 if i in activeColumnsIndices else 0 for i in range(tm.numberOfColumns())]
    predictedColumns = [1 if i in predictedColumnIndices else 0 for i in range(tm.numberOfColumns())]
    print("Active cols: " + str(np.nonzero(currentColumns)[0]))
    print("Predicted cols: " + str(np.nonzero(predictedColumns)[0]))
    print ""


Instruction:
Implement a function in Python 3 to
train the TM with given sequence for a given number of time steps and level of input

Response:
def trainTM(sequence, timeSteps, noiseLevel):
  """
  Trains the TM with given sequence for a given number of time steps and level of input
  corruption

  @param sequence   (array) array whose rows are the input characters
  @param timeSteps  (int)   number of time steps in which the TM will be presented with sequence
  @param noiseLevel (float) amount of noise to be applied on the characters in the sequence
  """
  currentColumns = np.zeros(tm.numberOfColumns(), dtype="uint32")
  predictedColumns = np.zeros(tm.numberOfColumns(), dtype="uint32")
  ts = 0
  for t in range(timeSteps):
    tm.reset()
    for k in range(4):
      v = corruptVector(sequence[k][:], noiseLevel, sparseCols)
      tm.compute(set(v[:].nonzero()[0].tolist()), learn=True)
      activeColumnsIndices = [tm.columnForCell(i) for i in tm.getActiveCells()]
      predictedColumnIndices = [tm.columnForCell(i) for i in tm.getPredictiveCells()]
      currentColumns = [1 if i in activeColumnsIndices else 0 for i in range(tm.numberOfColumns())]
      acc = accuracy(currentColumns, predictedColumns)
      x.append(ts)
      y.append(acc)
      ts += 1
      predictedColumns = [1 if i in predictedColumnIndices else 0 for i in range(tm.numberOfColumns())]


Instruction:
Here you have a function in Python 3, explain what it does
def encodeIntoArray(self, inputVal, outputVal):
    if len(inputVal) != len(outputVal):
      raise ValueError("Different input (%i) and output (%i) sizes." % (
          len(inputVal), len(outputVal)))

    if self.w is not None and sum(inputVal) != self.w:
      raise ValueError("Input has %i bits but w was set to %i." % (
          sum(inputVal), self.w))

    outputVal[:] = inputVal[:]

    if self.verbosity >= 2:
      print "input:", inputVal, "output:", outputVal
      print "decoded:", self.decodedToStr(self.decode(outputVal))

Response: Encodes inputVal into outputVal.


Instruction:
Explain what the following Python 3 code does
def decode(self, encoded, parentFieldName=""):

    if parentFieldName != "":
      fieldName = "%s.%s" % (parentFieldName, self.name)
    else:
      fieldName = self.name

    return ({fieldName: ([[0, 0]], "input")}, [fieldName])

Response: Decode the encoded field into a dictionary.


Instruction:
How would you code a function in Python 3 to
see the function description in base. py

Response:
def getBucketInfo(self, buckets):
    """See the function description in base.py"""
    return [EncoderResult(value=0, scalar=0, encoding=numpy.zeros(self.n))]


Instruction:
Can you tell what is the following Python 3 function doing
def topDownCompute(self, encoded):
    return EncoderResult(value=0, scalar=0,
                         encoding=numpy.zeros(self.n))

Response: Compute the top down encoding of a single element.


Instruction:
Write a Python 3 function for
doing a bitwise compare of the two bitmaps and returns a fractonal value between 0 and 1 of how similar they are.

Response:
def closenessScores(self, expValues, actValues, **kwargs):
    """
    Does a bitwise compare of the two bitmaps and returns a fractonal
    value between 0 and 1 of how similar they are.

    - ``1`` => identical
    - ``0`` => no overlaping bits

    ``kwargs`` will have the keyword "fractional", which is assumed by this
    encoder.
    """
    ratio = 1.0
    esum = int(expValues.sum())
    asum = int(actValues.sum())
    if asum > esum:
      diff = asum - esum
      if diff < esum:
        ratio = 1 - diff/float(esum)
      else:
        ratio = 1/float(diff)

    olap = expValues & actValues
    osum = int(olap.sum())
    if esum == 0:
      r = 0.0
    else:
      r = osum/float(esum)
    r = r * ratio

    return numpy.array([r])


Instruction:
Can you generate the documentation for the following Python 3 function
def getCallerInfo(depth=2):
  f = sys._getframe(depth)
  method_name = f.f_code.co_name
  filename = f.f_code.co_filename

  arg_class = None
  args = inspect.getargvalues(f)
  if len(args[0]) > 0:
    arg_name = args[0][0] # potentially the 'self' arg if its a method
    arg_class = args[3][arg_name].__class__.__name__
  return (method_name, filename, arg_class)

Response: Utility function to get information about the caller s caller


Instruction:
Can you implement a function in Python 3 that
gets the argument names default values and argument descriptions for a function.

Response:
def getArgumentDescriptions(f):
  """
  Get the arguments, default values, and argument descriptions for a function.

  Parses the argument descriptions out of the function docstring, using a
  format something lke this:

  ::

    [junk]
    argument_name:     description...
      description...
      description...
    [junk]
    [more arguments]

  It will find an argument as long as the exact argument name starts the line.
  It will then strip a trailing colon, if present, then strip the rest of the
  line and use it to start the description. It will then strip and append any
  subsequent lines with a greater indent level than the original argument name.

  :param f: (function) to inspect
  :returns: (list of tuples) (``argName``, ``argDescription``, ``defaultValue``)
    If an argument has no default value, the tuple is only two elements long (as
    ``None`` cannot be used, since it could be a default value itself).
  """

  # Get the argument names and default values
  argspec = inspect.getargspec(f)

  # Scan through the docstring to extract documentation for each argument as
  # follows:
  #   Check the first word of the line, stripping a colon if one is present.
  #   If it matches an argument name:
  #    Take the rest of the line, stripping leading whitespeace
  #    Take each subsequent line if its indentation level is greater than the
  #      initial indentation level
  #    Once the indentation level is back to the original level, look for
  #      another argument
  docstring = f.__doc__
  descriptions = {}
  if docstring:
    lines = docstring.split('\n')
    i = 0
    while i < len(lines):
      stripped = lines[i].lstrip()
      if not stripped:
        i += 1
        continue
      # Indentation level is index of the first character
      indentLevel = lines[i].index(stripped[0])
      # Get the first word and remove the colon, if present
      firstWord = stripped.split()[0]
      if firstWord.endswith(':'):
        firstWord = firstWord[:-1]
      if firstWord in argspec.args:
        # Found an argument
        argName = firstWord
        restOfLine = stripped[len(firstWord)+1:].strip()
        argLines = [restOfLine]
        # Take the next lines as long as they are indented more
        i += 1
        while i < len(lines):
          stripped = lines[i].lstrip()
          if not stripped:
            # Empty line - stop
            break
          if lines[i].index(stripped[0]) <= indentLevel:
            # No longer indented far enough - stop
            break
          # This line counts too
          argLines.append(lines[i].strip())
          i += 1
        # Store this description
        descriptions[argName] = ' '.join(argLines)
      else:
        # Not an argument
        i += 1

  # Build the list of (argName, description, defaultValue)
  args = []
  if argspec.defaults:
    defaultCount = len(argspec.defaults)
  else:
    defaultCount = 0
  nonDefaultArgCount = len(argspec.args) - defaultCount
  for i, argName in enumerate(argspec.args):
    if i >= nonDefaultArgCount:
      defaultValue = argspec.defaults[i - nonDefaultArgCount]
      args.append((argName, descriptions.get(argName, ""), defaultValue))
    else:
      args.append((argName, descriptions.get(argName, "")))

  return args


Instruction:
Make a summary of the following Python 3 code
def initLogging(verbose=False, console='stdout', consoleLevel='DEBUG'):

  # NOTE: If you call this twice from the same process there seems to be a
  # bug - logged messages don't show up for loggers that you do another
  # logging.getLogger() on.
  global gLoggingInitialized
  if gLoggingInitialized:
    if verbose:
      print >> sys.stderr, "Logging already initialized, doing nothing."
    return

  consoleStreamMappings = {
    'stdout'  : 'stdoutConsoleHandler',
    'stderr'  : 'stderrConsoleHandler',
  }

  consoleLogLevels = ['DEBUG', 'INFO', 'WARNING', 'WARN', 'ERROR', 'CRITICAL',
                      'FATAL']

  assert console is None or console in consoleStreamMappings.keys(), (
    'Unexpected console arg value: %r') % (console,)

  assert consoleLevel in consoleLogLevels, (
    'Unexpected consoleLevel arg value: %r') % (consoleLevel)

  # -----------------------------------------------------------------------
  # Setup logging. Look for the nupic-logging.conf file, first in the
  #   NTA_CONFIG_DIR path (if defined), then in a subdirectory of the nupic
  #   module
  configFilename = 'nupic-logging.conf'
  configFilePath = resource_filename("nupic.support", configFilename)

  configLogDir = os.environ.get('NTA_LOG_DIR', None)

  # Load in the logging configuration file
  if verbose:
    print >> sys.stderr, (
      "Using logging configuration file: %s") % (configFilePath)

  # This dict will hold our replacement strings for logging configuration
  replacements = dict()

  def makeKey(name):
    """ Makes replacement key """
    return "$$%s$$" % (name)

  platform = sys.platform.lower()
  if platform.startswith('java'):
    # Jython
    import java.lang
    platform = java.lang.System.getProperty("os.name").lower()
    if platform.startswith('mac os x'):
      platform = 'darwin'

  if platform.startswith('darwin'):
    replacements[makeKey('SYSLOG_HANDLER_ADDRESS')] = '"/var/run/syslog"'
  elif platform.startswith('linux'):
    replacements[makeKey('SYSLOG_HANDLER_ADDRESS')] = '"/dev/log"'
  elif platform.startswith('win'):
    replacements[makeKey('SYSLOG_HANDLER_ADDRESS')] = '"log"'
  else:
    raise RuntimeError("This platform is neither darwin, win32, nor linux: %s" % (
      sys.platform,))

  # Nupic logs go to file
  replacements[makeKey('PERSISTENT_LOG_HANDLER')] = 'fileHandler'
  if platform.startswith('win'):
    replacements[makeKey('FILE_HANDLER_LOG_FILENAME')] = '"NUL"'
  else:
    replacements[makeKey('FILE_HANDLER_LOG_FILENAME')] = '"/dev/null"'

  # Set up log file path for the default file handler and configure handlers
  handlers = list()

  if configLogDir is not None:
    logFilePath = _genLoggingFilePath()
    makeDirectoryFromAbsolutePath(os.path.dirname(logFilePath))
    replacements[makeKey('FILE_HANDLER_LOG_FILENAME')] = repr(logFilePath)

    handlers.append(replacements[makeKey('PERSISTENT_LOG_HANDLER')])

  if console is not None:
    handlers.append(consoleStreamMappings[console])

  replacements[makeKey('ROOT_LOGGER_HANDLERS')] = ", ".join(handlers)

  # Set up log level for console handlers
  replacements[makeKey('CONSOLE_LOG_LEVEL')] = consoleLevel

  customConfig = StringIO()

  # Using pkg_resources to get the logging file, which should be packaged and
  # associated with this source file name.
  loggingFileContents = resource_string(__name__, configFilename)

  for lineNum, line in enumerate(loggingFileContents.splitlines()):
    if "$$" in line:
      for (key, value) in replacements.items():
        line = line.replace(key, value)

    # If there is still a replacement string in the line, we're missing it
    #  from our replacements dict
    if "$$" in line and "$$<key>$$" not in line:
      raise RuntimeError(("The text %r, found at line #%d of file %r, "
                          "contains a string not found in our replacement "
                          "dict.") % (line, lineNum, configFilePath))

    customConfig.write("%s\n" % line)

  customConfig.seek(0)
  if python_version()[:3] >= '2.6':
    logging.config.fileConfig(customConfig, disable_existing_loggers=False)
  else:
    logging.config.fileConfig(customConfig)

  gLoggingInitialized = True

Response: Initializes logging for the NuPic.