Instruction:
Can you implement a function in Python 3 that
processes one input sample. This method is called by outer loop code outside the nupic-engine. We use this instead of the nupic engine compute() because our inputs and outputs aren't fixed size vectors of reals. :param recordNum: Record number of this input pattern. Record numbers normally increase sequentially by 1 each time unless there are missing records in the dataset. Knowing this information insures that we don't get confused by missing records. :param patternNZ: List of the active indices from the output below. When the input is from TemporalMemory, this list should be the indices of the active cells. :param classification: Dict of the classification information where: - bucketIdx: list of indices of the encoder bucket - actValue: list of actual values going into the encoder Classification could be None for inference mode. :param learn: (bool) if true, learn this sample :param infer: (bool) if true, perform inference :return: Dict containing inference results, there is one entry for each step in self.steps, where the key is the number of steps, and the value is an array containing the relative likelihood for each bucketIdx starting from bucketIdx 0. There is also an entry containing the average actual value to use for each bucket. The key is 'actualValues'. for example: .. code-block:: python {1 : [0.1, 0.3, 0.2, 0.7], 4 : [0.2, 0.4, 0.3, 0.5], 'actualValues': [1.5, 3,5, 5,5, 7.6], }

Response:
def compute(self, recordNum, patternNZ, classification, learn, infer):
    """
    Process one input sample.

    This method is called by outer loop code outside the nupic-engine. We
    use this instead of the nupic engine compute() because our inputs and
    outputs aren't fixed size vectors of reals.

    :param recordNum: Record number of this input pattern. Record numbers
      normally increase sequentially by 1 each time unless there are missing
      records in the dataset. Knowing this information insures that we don't get
      confused by missing records.

    :param patternNZ: List of the active indices from the output below. When the
      input is from TemporalMemory, this list should be the indices of the
      active cells.

    :param classification: Dict of the classification information where:

      - bucketIdx: list of indices of the encoder bucket
      - actValue: list of actual values going into the encoder

      Classification could be None for inference mode.
    :param learn: (bool) if true, learn this sample
    :param infer: (bool) if true, perform inference

    :return:    Dict containing inference results, there is one entry for each
                step in self.steps, where the key is the number of steps, and
                the value is an array containing the relative likelihood for
                each bucketIdx starting from bucketIdx 0.

                There is also an entry containing the average actual value to
                use for each bucket. The key is 'actualValues'.

                for example:

                .. code-block:: python

                   {1 :             [0.1, 0.3, 0.2, 0.7],
                     4 :             [0.2, 0.4, 0.3, 0.5],
                     'actualValues': [1.5, 3,5, 5,5, 7.6],
                   }
    """
    if self.verbosity >= 1:
      print "  learn:", learn
      print "  recordNum:", recordNum
      print "  patternNZ (%d):" % len(patternNZ), patternNZ
      print "  classificationIn:", classification

    # ensures that recordNum increases monotonically
    if len(self._patternNZHistory) > 0:
      if recordNum < self._patternNZHistory[-1][0]:
        raise ValueError("the record number has to increase monotonically")

    # Store pattern in our history if this is a new record
    if len(self._patternNZHistory) == 0 or \
                    recordNum > self._patternNZHistory[-1][0]:
      self._patternNZHistory.append((recordNum, patternNZ))

    # To allow multi-class classification, we need to be able to run learning
    # without inference being on. So initialize retval outside
    # of the inference block.
    retval = {}

    # Update maxInputIdx and augment weight matrix with zero padding
    if max(patternNZ) > self._maxInputIdx:
      newMaxInputIdx = max(patternNZ)
      for nSteps in self.steps:
        self._weightMatrix[nSteps] = numpy.concatenate((
          self._weightMatrix[nSteps],
          numpy.zeros(shape=(newMaxInputIdx-self._maxInputIdx,
                             self._maxBucketIdx+1))), axis=0)
      self._maxInputIdx = int(newMaxInputIdx)

    # Get classification info
    if classification is not None:
      if type(classification["bucketIdx"]) is not list:
        bucketIdxList = [classification["bucketIdx"]]
        actValueList = [classification["actValue"]]
        numCategory = 1
      else:
        bucketIdxList = classification["bucketIdx"]
        actValueList = classification["actValue"]
        numCategory = len(classification["bucketIdx"])
    else:
      if learn:
        raise ValueError("classification cannot be None when learn=True")
      actValueList = None
      bucketIdxList = None
    # ------------------------------------------------------------------------
    # Inference:
    # For each active bit in the activationPattern, get the classification
    # votes
    if infer:
      retval = self.infer(patternNZ, actValueList)

    if learn and classification["bucketIdx"] is not None:
      for categoryI in range(numCategory):
        bucketIdx = bucketIdxList[categoryI]
        actValue = actValueList[categoryI]

        # Update maxBucketIndex and augment weight matrix with zero padding
        if bucketIdx > self._maxBucketIdx:
          for nSteps in self.steps:
            self._weightMatrix[nSteps] = numpy.concatenate((
              self._weightMatrix[nSteps],
              numpy.zeros(shape=(self._maxInputIdx+1,
                                 bucketIdx-self._maxBucketIdx))), axis=1)

          self._maxBucketIdx = int(bucketIdx)

        # Update rolling average of actual values if it's a scalar. If it's
        # not, it must be a category, in which case each bucket only ever
        # sees one category so we don't need a running average.
        while self._maxBucketIdx > len(self._actualValues) - 1:
          self._actualValues.append(None)
        if self._actualValues[bucketIdx] is None:
          self._actualValues[bucketIdx] = actValue
        else:
          if (isinstance(actValue, int) or
                isinstance(actValue, float) or
                isinstance(actValue, long)):
            self._actualValues[bucketIdx] = ((1.0 - self.actValueAlpha)
                                             * self._actualValues[bucketIdx]
                                             + self.actValueAlpha * actValue)
          else:
            self._actualValues[bucketIdx] = actValue

      for (learnRecordNum, learnPatternNZ) in self._patternNZHistory:
        error = self._calculateError(recordNum, bucketIdxList)

        nSteps = recordNum - learnRecordNum
        if nSteps in self.steps:
          for bit in learnPatternNZ:
            self._weightMatrix[nSteps][bit, :] += self.alpha * error[nSteps]

    # ------------------------------------------------------------------------
    # Verbose print
    if infer and self.verbosity >= 1:
      print "  inference: combined bucket likelihoods:"
      print "    actual bucket values:", retval["actualValues"]
      for (nSteps, votes) in retval.items():
        if nSteps == "actualValues":
          continue
        print "    %d steps: " % (nSteps), _pFormatArray(votes)
        bestBucketIdx = votes.argmax()
        print ("      most likely bucket idx: "
               "%d, value: %s" % (bestBucketIdx,
                                  retval["actualValues"][bestBucketIdx]))
      print

    return retval


Instruction:
How would you implement a function in Python 3 that
infers the actual value of a pattern from one input sample.

Response:
def infer(self, patternNZ, actValueList):
    """
    Return the inference value from one input sample. The actual
    learning happens in compute().

    :param patternNZ: list of the active indices from the output below
    :param classification: dict of the classification information:
                    bucketIdx: index of the encoder bucket
                    actValue:  actual value going into the encoder

    :return:    dict containing inference results, one entry for each step in
                self.steps. The key is the number of steps, the value is an
                array containing the relative likelihood for each bucketIdx
                starting from bucketIdx 0.

                for example:

                .. code-block:: python

                   {'actualValues': [0.0, 1.0, 2.0, 3.0]
                     1 : [0.1, 0.3, 0.2, 0.7]
                     4 : [0.2, 0.4, 0.3, 0.5]}
    """

    # Return value dict. For buckets which we don't have an actual value
    # for yet, just plug in any valid actual value. It doesn't matter what
    # we use because that bucket won't have non-zero likelihood anyways.

    # NOTE: If doing 0-step prediction, we shouldn't use any knowledge
    #  of the classification input during inference.
    if self.steps[0] == 0 or actValueList is None:
      defaultValue = 0
    else:
      defaultValue = actValueList[0]
    actValues = [x if x is not None else defaultValue
                 for x in self._actualValues]
    retval = {"actualValues": actValues}

    for nSteps in self.steps:
      predictDist = self.inferSingleStep(patternNZ, self._weightMatrix[nSteps])

      retval[nSteps] = predictDist

    return retval


Instruction:
Given the following Python 3 function, write the documentation
def inferSingleStep(self, patternNZ, weightMatrix):
    outputActivation = weightMatrix[patternNZ].sum(axis=0)

    # softmax normalization
    outputActivation = outputActivation - numpy.max(outputActivation)
    expOutputActivation = numpy.exp(outputActivation)
    predictDist = expOutputActivation / numpy.sum(expOutputActivation)
    return predictDist

Response: Perform inference for a single step. Given an SDR input and a weight matrix return a predicted class label distribution.


Instruction:
Write a Python 3 function for
calculating the error at the output layer.

Response:
def _calculateError(self, recordNum, bucketIdxList):
    """
    Calculate error signal

    :param bucketIdxList: list of encoder buckets

    :return: dict containing error. The key is the number of steps
             The value is a numpy array of error at the output layer
    """
    error = dict()
    targetDist = numpy.zeros(self._maxBucketIdx + 1)
    numCategories = len(bucketIdxList)
    for bucketIdx in bucketIdxList:
      targetDist[bucketIdx] = 1.0/numCategories

    for (learnRecordNum, learnPatternNZ) in self._patternNZHistory:
      nSteps = recordNum - learnRecordNum
      if nSteps in self.steps:
        predictDist = self.inferSingleStep(learnPatternNZ,
                                           self._weightMatrix[nSteps])
        error[nSteps] = targetDist - predictDist

    return error


Instruction:
Can you write a function in Python 3 where it
sorts a potentially big file by a list of field names.

Response:
def sort(filename, key, outputFile, fields=None, watermark=1024 * 1024 * 100):
  """Sort a potentially big file

  filename - the input file (standard File format)
  key - a list of field names to sort by
  outputFile - the name of the output file
  fields - a list of fields that should be included (all fields if None)
  watermark - when available memory goes bellow the watermark create a new chunk

  sort() works by reading as records from the file into memory
  and calling _sortChunk() on each chunk. In the process it gets
  rid of unneeded fields if any. Once all the chunks have been sorted and
  written to chunk files it calls _merge() to merge all the chunks into a
  single sorted file.

  Note, that sort() gets a key that contains field names, which it converts
  into field indices for _sortChunk() becuase _sortChunk() doesn't need to know
  the field name.

  sort() figures out by itself how many chunk files to use by reading records
  from the file until the low watermark value of availabel memory is hit and
  then it sorts the current records, generates a chunk file, clears the sorted
  records and starts on a new chunk.

  The key field names are turned into indices
  """
  if fields is not None:
    assert set(key).issubset(set([f[0] for f in fields]))

  with FileRecordStream(filename) as f:

    # Find the indices of the requested fields
    if fields:
      fieldNames = [ff[0] for ff in fields]
      indices = [f.getFieldNames().index(name) for name in fieldNames]
      assert len(indices) == len(fields)
    else:
      fileds = f.getFields()
      fieldNames = f.getFieldNames()
      indices = None

    # turn key fields to key indices
    key = [fieldNames.index(name) for name in key]

    chunk = 0
    records = []
    for i, r in enumerate(f):
      # Select requested fields only
      if indices:
        temp = []
        for i in indices:
          temp.append(r[i])
        r = temp
      # Store processed record
      records.append(r)

      # Check memory
      available_memory = psutil.avail_phymem()

      # If bellow the watermark create a new chunk, reset and keep going
      if available_memory < watermark:
        _sortChunk(records, key, chunk, fields)
        records = []
        chunk += 1

    # Sort and write the remainder
    if len(records) > 0:
      _sortChunk(records, key, chunk, fields)
      chunk += 1

    # Marge all the files
    _mergeFiles(key, chunk, outputFile, fields)


Instruction:
Create a Python 3 function to
sort in memory chunk of records by chunkIndex

Response:
def _sortChunk(records, key, chunkIndex, fields):
  """Sort in memory chunk of records

  records - a list of records read from the original dataset
  key - a list of indices to sort the records by
  chunkIndex - the index of the current chunk

  The records contain only the fields requested by the user.

  _sortChunk() will write the sorted records to a standard File
  named "chunk_<chunk index>.csv" (chunk_0.csv, chunk_1.csv,...).
  """
  title(additional='(key=%s, chunkIndex=%d)' % (str(key), chunkIndex))

  assert len(records) > 0

  # Sort the current records
  records.sort(key=itemgetter(*key))

  # Write to a chunk file
  if chunkIndex is not None:
    filename = 'chunk_%d.csv' % chunkIndex
    with FileRecordStream(filename, write=True, fields=fields) as o:
      for r in records:
        o.appendRecord(r)

    assert os.path.getsize(filename) > 0

  return records


Instruction:
How would you explain what the following Python 3 function does
def _mergeFiles(key, chunkCount, outputFile, fields):
  title()

  # Open all chun files
  files = [FileRecordStream('chunk_%d.csv' % i) for i in range(chunkCount)]

  # Open output file
  with FileRecordStream(outputFile, write=True, fields=fields) as o:
    # Open all chunk files
    files = [FileRecordStream('chunk_%d.csv' % i) for i in range(chunkCount)]
    records = [f.getNextRecord() for f in files]

    # This loop will run until all files are exhausted
    while not all(r is None for r in records):
      # Cleanup None values (files that were exhausted)
      indices = [i for i,r in enumerate(records) if r is not None]
      records = [records[i] for i in indices]
      files = [files[i] for i in indices]

      # Find the current record
      r = min(records, key=itemgetter(*key))
      # Write it to the file
      o.appendRecord(r)

      # Find the index of file that produced the current record
      index = records.index(r)
      # Read a new record from the file
      records[index] = files[index].getNextRecord()

  # Cleanup chunk files
  for i, f in enumerate(files):
    f.close()
    os.remove('chunk_%d.csv' % i)

Response: Merge sorted chunk files into a sorted output file


Instruction:
Write a Python 3 script to
compute the state of the temporal memory.

Response:
def compute(self, activeColumns, learn=True):
    """
    Feeds input record through TM, performing inference and learning.
    Updates member variables with new state.

    @param activeColumns (set) Indices of active columns in `t`
    """
    bottomUpInput = numpy.zeros(self.numberOfCols, dtype=dtype)
    bottomUpInput[list(activeColumns)] = 1
    super(TemporalMemoryShim, self).compute(bottomUpInput,
                                            enableLearn=learn,
                                            enableInference=True)

    predictedState = self.getPredictedState()
    self.predictiveCells = set(numpy.flatnonzero(predictedState))


Instruction:
Can you generate the documentation for the following Python 3 function
def read(cls, proto):
    tm = super(TemporalMemoryShim, cls).read(proto.baseTM)
    tm.predictiveCells = set(proto.predictedState)
    tm.connections = Connections.read(proto.conncetions)

Response: Deserialize from proto instance.


Instruction:
Can you implement a function in Python 3 that
populates serialization proto instance.

Response:
def write(self, proto):
    """Populate serialization proto instance.

    :param proto: (TemporalMemoryShimProto) the proto instance to populate
    """
    super(TemporalMemoryShim, self).write(proto.baseTM)
    proto.connections.write(self.connections)
    proto.predictiveCells = self.predictiveCells


Instruction:
Create a Python 3 function to
print a message to the console.

Response:
def cPrint(self, level, message, *args, **kw):
    """Print a message to the console.

    Prints only if level <= self.consolePrinterVerbosity
    Printing with level 0 is equivalent to using a print statement,
    and should normally be avoided.

    :param level: (int) indicating the urgency of the message with
           lower values meaning more urgent (messages at level 0  are the most
           urgent and are always printed)

    :param message: (string) possibly with format specifiers

    :param args: specifies the values for any format specifiers in message

    :param kw: newline is the only keyword argument. True (default) if a newline
           should be printed
    """

    if level > self.consolePrinterVerbosity:
      return

    if len(kw) > 1:
      raise KeyError("Invalid keywords for cPrint: %s" % str(kw.keys()))

    newline = kw.get("newline", True)
    if len(kw) == 1 and 'newline' not in kw:
      raise KeyError("Invalid keyword for cPrint: %s" % kw.keys()[0])

    if len(args) == 0:
      if newline:
        print message
      else:
        print message,
    else:
      if newline:
        print message % args
      else:
        print message % args,


Instruction:
How would you explain what the following Python 3 function does
def profileTM(tmClass, tmDim, nRuns):

  # create TM instance to measure
  tm = tmClass(numberOfCols=tmDim)

  # generate input data
  data = numpy.random.randint(0, 2, [tmDim, nRuns]).astype('float32')

  for i in xrange(nRuns):
    # new data every time, this is the worst case performance
    # real performance would be better, as the input data would not be completely random
    d = data[:,i]

    # the actual function to profile!
    tm.compute(d, True)

Response: profile performance of a TemporalMemory


Instruction:
Write a Python 3 script for
generating a simple dataset.

Response:
def _generateCategory(filename="simple.csv", numSequences=2, elementsPerSeq=1,
                    numRepeats=10, resets=False):
  """ Generate a simple dataset. This contains a bunch of non-overlapping
  sequences.

  Parameters:
  ----------------------------------------------------
  filename:       name of the file to produce, including extension. It will
                  be created in a 'datasets' sub-directory within the
                  directory containing this script.
  numSequences:   how many sequences to generate
  elementsPerSeq: length of each sequence
  numRepeats:     how many times to repeat each sequence in the output
  resets:         if True, turn on reset at start of each sequence
  """

  # Create the output file
  scriptDir = os.path.dirname(__file__)
  pathname = os.path.join(scriptDir, 'datasets', filename)
  print "Creating %s..." % (pathname)
  fields = [('reset', 'int', 'R'), ('category', 'int', 'C'),
            ('field1', 'string', '')]
  outFile = FileRecordStream(pathname, write=True, fields=fields)

  # Create the sequences
  sequences = []
  for i in range(numSequences):
    seq = [x for x in range(i*elementsPerSeq, (i+1)*elementsPerSeq)]
    sequences.append(seq)

  # Write out the sequences in random order
  seqIdxs = []
  for i in range(numRepeats):
    seqIdxs += range(numSequences)
  random.shuffle(seqIdxs)

  for seqIdx in seqIdxs:
    reset = int(resets)
    seq = sequences[seqIdx]
    for x in seq:
      outFile.appendRecord([reset, str(seqIdx), str(x)])
      reset = 0

  outFile.close()


Instruction:
Implement a function in Python 3 to
encode the SDR into a numpy array.

Response:
def encodeIntoArray(self, inputData, output):
    """
    See `nupic.encoders.base.Encoder` for more information.

    :param: inputData (tuple) Contains speed (float), longitude (float),
                             latitude (float), altitude (float)
    :param: output (numpy.array) Stores encoded SDR in this numpy array
    """
    altitude = None
    if len(inputData) == 4:
      (speed, longitude, latitude, altitude) = inputData
    else:
      (speed, longitude, latitude) = inputData
    coordinate = self.coordinateForPosition(longitude, latitude, altitude)
    radius = self.radiusForSpeed(speed)
    super(GeospatialCoordinateEncoder, self).encodeIntoArray(
     (coordinate, radius), output)


Instruction:
Can you implement a function in Python 3 that
returns the coordinate that maps the given GPS position.

Response:
def coordinateForPosition(self, longitude, latitude, altitude=None):
    """
    Returns coordinate for given GPS position.

    :param: longitude (float) Longitude of position
    :param: latitude (float) Latitude of position
    :param: altitude (float) Altitude of position
    :returns: (numpy.array) Coordinate that the given GPS position
                          maps to
    """
    coords = PROJ(longitude, latitude)

    if altitude is not None:
      coords = transform(PROJ, geocentric, coords[0], coords[1], altitude)

    coordinate = numpy.array(coords)
    coordinate = coordinate / self.scale
    return coordinate.astype(int)


Instruction:
Can you write a function in Python 3 where it
returns the radius of the atom in the order of the readings in the given speed.

Response:
def radiusForSpeed(self, speed):
    """
    Returns radius for given speed.

    Tries to get the encodings of consecutive readings to be
    adjacent with some overlap.

    :param: speed (float) Speed (in meters per second)
    :returns: (int) Radius for given speed
    """
    overlap = 1.5
    coordinatesPerTimestep = speed * self.timestep / self.scale
    radius = int(round(float(coordinatesPerTimestep) / 2 * overlap))
    minRadius = int(math.ceil((math.sqrt(self.w) - 1) / 2))
    return max(radius, minRadius)


Instruction:
Here you have a function in Python 3, explain what it does
def getSearch(rootDir):

  # Form the stream definition
  dataPath = os.path.abspath(os.path.join(rootDir, 'datasets', 'scalar_1.csv'))
  streamDef = dict(
    version = 1,
    info = "testSpatialClassification",
    streams = [
      dict(source="file://%s" % (dataPath),
           info="scalar_1.csv",
           columns=["*"],
           ),
      ],
  )

  # Generate the experiment description
  expDesc = {
    "environment": 'nupic',
    "inferenceArgs":{
      "predictedField":"classification",
      "predictionSteps": [0],
    },
    "inferenceType":  "MultiStep",
    "streamDef": streamDef,
    "includedFields": [
      { "fieldName": "field1",
        "fieldType": "float",
      },
      { "fieldName": "classification",
        "fieldType": "string",
      },
      { "fieldName": "randomData",
        "fieldType": "float",
      },
    ],
    "iterationCount": -1,
  }

  return expDesc

Response: This method returns the search description for the given dictionary


Instruction:
Can you write a function in Python 3 where it
encodes value into output array.

Response:
def encodeIntoArray(self, value, output):
    """ See method description in base.py """
    denseInput = numpy.zeros(output.shape)
    try:
      denseInput[value] = 1
    except IndexError:
      if isinstance(value, numpy.ndarray):
        raise ValueError(
            "Numpy array must have integer dtype but got {}".format(
                value.dtype))
      raise
    super(SparsePassThroughEncoder, self).encodeIntoArray(denseInput, output)


Instruction:
Implement a Python 3 function for
reading serialized object from file.

Response:
def readFromFile(cls, f, packed=True):
    """
    Read serialized object from file.

    :param f: input file
    :param packed: If true, will assume content is packed
    :return: first-class instance initialized from proto obj
    """
    # Get capnproto schema from instance
    schema = cls.getSchema()

    # Read from file
    if packed:
      proto = schema.read_packed(f)
    else:
      proto = schema.read(f)

    # Return first-class instance initialized from proto obj
    return cls.read(proto)


Instruction:
Given the following Python 3 function, write the documentation
def writeToFile(self, f, packed=True):
    # Get capnproto schema from instance
    schema = self.getSchema()

    # Construct new message, otherwise refered to as `proto`
    proto = schema.new_message()

    # Populate message w/ `write()` instance method
    self.write(proto)

    # Finally, write to file
    if packed:
      proto.write_packed(f)
    else:
      proto.write(f)

Response: Writes serialized object to file.


Instruction:
How would you code a function in Python 3 to
read a TwoGramModel from a proto message.

Response:
def read(cls, proto):
    """
    :param proto: capnp TwoGramModelProto message reader
    """
    instance = object.__new__(cls)
    super(TwoGramModel, instance).__init__(proto=proto.modelBase)

    instance._logger = opf_utils.initLogger(instance)

    instance._reset = proto.reset
    instance._hashToValueDict = {x.hash: x.value
                                 for x in proto.hashToValueDict}
    instance._learningEnabled = proto.learningEnabled
    instance._encoder = encoders.MultiEncoder.read(proto.encoder)
    instance._fieldNames = instance._encoder.getScalarNames()
    instance._prevValues = list(proto.prevValues)
    instance._twoGramDicts = [dict() for _ in xrange(len(proto.twoGramDicts))]
    for idx, field in enumerate(proto.twoGramDicts):
      for entry in field:
        prev = None if entry.value == -1 else entry.value
        instance._twoGramDicts[idx][prev] = collections.defaultdict(int)
        for bucket in entry.buckets:
          instance._twoGramDicts[idx][prev][bucket.index] = bucket.count

    return instance


Instruction:
Write a Python 3 script to
write the model to the proto message builder

Response:
def write(self, proto):
    """
    :param proto: capnp TwoGramModelProto message builder
    """
    super(TwoGramModel, self).writeBaseToProto(proto.modelBase)

    proto.reset = self._reset
    proto.learningEnabled = self._learningEnabled
    proto.prevValues = self._prevValues
    self._encoder.write(proto.encoder)
    proto.hashToValueDict = [{"hash": h, "value": v}
                             for h, v in self._hashToValueDict.items()]

    twoGramDicts = []
    for items in self._twoGramDicts:
      twoGramArr = []
      for prev, values in items.iteritems():
        buckets = [{"index": index, "count": count}
                   for index, count in values.iteritems()]
        if prev is None:
          prev = -1
        twoGramArr.append({"value": prev, "buckets": buckets})

      twoGramDicts.append(twoGramArr)

    proto.twoGramDicts = twoGramDicts


Instruction:
Make a summary of the following Python 3 code
def requireAnomalyModel(func):
  @wraps(func)
  def _decorator(self, *args, **kwargs):
    if not self.getInferenceType() == InferenceType.TemporalAnomaly:
      raise RuntimeError("Method required a TemporalAnomaly model.")
    if self._getAnomalyClassifier() is None:
      raise RuntimeError("Model does not support this command. Model must"
          "be an active anomalyDetector model.")
    return func(self, *args, **kwargs)
  return _decorator

Response: Decorator for functions that require anomaly models.


Instruction:
Can you write a function in Python 3 where it
removes labels from the anomaly classifier within this model.

Response:
def anomalyRemoveLabels(self, start, end, labelFilter):
    """
    Remove labels from the anomaly classifier within this model. Removes all
    records if ``labelFilter==None``, otherwise only removes the labels equal to
    ``labelFilter``.

    :param start: (int) index to start removing labels
    :param end: (int) index to end removing labels
    :param labelFilter: (string) If specified, only removes records that match
    """
    self._getAnomalyClassifier().getSelf().removeLabels(start, end, labelFilter)


Instruction:
Can you write a function in Python 3 where it
adds a label to the anomaly classifier within this model.

Response:
def anomalyAddLabel(self, start, end, labelName):
    """
    Add labels from the anomaly classifier within this model.

    :param start: (int) index to start label
    :param end: (int) index to end label
    :param labelName: (string) name of label
    """
    self._getAnomalyClassifier().getSelf().addLabel(start, end, labelName)


Instruction:
How would you explain what the following Python 3 function does
def anomalyGetLabels(self, start, end):
    return self._getAnomalyClassifier().getSelf().getLabels(start, end)

Response: Get the labels from the anomaly classifier within this model.


Instruction:
Write a Python 3 function for
returning a SensorInput object which represents the parsed representation of the input record

Response:
def _getSensorInputRecord(self, inputRecord):
    """
    inputRecord - dict containing the input to the sensor

    Return a 'SensorInput' object, which represents the 'parsed'
    representation of the input record
    """
    sensor = self._getSensorRegion()
    dataRow = copy.deepcopy(sensor.getSelf().getOutputValues('sourceOut'))
    dataDict = copy.deepcopy(inputRecord)
    inputRecordEncodings = sensor.getSelf().getOutputValues('sourceEncodings')
    inputRecordCategory = int(sensor.getOutputData('categoryOut')[0])
    resetOut = sensor.getOutputData('resetOut')[0]

    return SensorInput(dataRow=dataRow,
                       dataDict=dataDict,
                       dataEncodings=inputRecordEncodings,
                       sequenceReset=resetOut,
                       category=inputRecordCategory)


Instruction:
Create a Python 3 function to
return a ClassifierInput object which contains the value of the predicted field in the inputRecord

Response:
def _getClassifierInputRecord(self, inputRecord):
    """
    inputRecord - dict containing the input to the sensor

    Return a 'ClassifierInput' object, which contains the mapped
    bucket index for input Record
    """
    absoluteValue = None
    bucketIdx = None

    if self._predictedFieldName is not None and self._classifierInputEncoder is not None:
      absoluteValue = inputRecord[self._predictedFieldName]
      bucketIdx = self._classifierInputEncoder.getBucketIndices(absoluteValue)[0]

    return ClassifierInput(dataRow=absoluteValue,
                           bucketIndex=bucketIdx)


Instruction:
Here you have a function in Python 3, explain what it does
def _anomalyCompute(self):
    inferenceType = self.getInferenceType()

    inferences = {}
    sp = self._getSPRegion()
    score = None
    if inferenceType == InferenceType.NontemporalAnomaly:
      score = sp.getOutputData("anomalyScore")[0] #TODO move from SP to Anomaly ?

    elif inferenceType == InferenceType.TemporalAnomaly:
      tm = self._getTPRegion()

      if sp is not None:
        activeColumns = sp.getOutputData("bottomUpOut").nonzero()[0]
      else:
        sensor = self._getSensorRegion()
        activeColumns = sensor.getOutputData('dataOut').nonzero()[0]

      if not self._predictedFieldName in self._input:
        raise ValueError(
          "Expected predicted field '%s' in input row, but was not found!"
          % self._predictedFieldName
        )
      # Calculate the anomaly score using the active columns
      # and previous predicted columns.
      score = tm.getOutputData("anomalyScore")[0]

      # Calculate the classifier's output and use the result as the anomaly
      # label. Stores as string of results.

      # TODO: make labels work with non-SP models
      if sp is not None:
        self._getAnomalyClassifier().setParameter(
            "activeColumnCount", len(activeColumns))
        self._getAnomalyClassifier().prepareInputs()
        self._getAnomalyClassifier().compute()

        labels = self._getAnomalyClassifier().getSelf().getLabelResults()
        inferences[InferenceElement.anomalyLabel] = "%s" % labels

    inferences[InferenceElement.anomalyScore] = score
    return inferences

Response: Compute anomaly score for the current anomaly in the current region.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _handleSDRClassifierMultiStep(self, patternNZ,
                                    inputTSRecordIdx,
                                    rawInput):
    """ Handle the CLA Classifier compute logic when implementing multi-step
    prediction. This is where the patternNZ is associated with one of the
    other fields from the dataset 0 to N steps in the future. This method is
    used by each type of network (encoder only, SP only, SP +TM) to handle the
    compute logic through the CLA Classifier. It fills in the inference dict with
    the results of the compute.

    Parameters:
    -------------------------------------------------------------------
    patternNZ: The input to the CLA Classifier as a list of active input indices
    inputTSRecordIdx: The index of the record as computed from the timestamp
                  and aggregation interval. This normally increments by 1
                  each time unless there are missing records. If there is no
                  aggregation interval or timestamp in the data, this will be
                  None.
    rawInput:   The raw input to the sensor, as a dict.
    """
    inferenceArgs = self.getInferenceArgs()
    predictedFieldName = inferenceArgs.get('predictedField', None)
    if predictedFieldName is None:
      raise ValueError(
        "No predicted field was enabled! Did you call enableInference()?"
      )
    self._predictedFieldName = predictedFieldName

    classifier = self._getClassifierRegion()
    if not self._hasCL or classifier is None:
      # No classifier so return an empty dict for inferences.
      return {}

    sensor = self._getSensorRegion()
    minLikelihoodThreshold = self._minLikelihoodThreshold
    maxPredictionsPerStep = self._maxPredictionsPerStep
    needLearning = self.isLearningEnabled()
    inferences = {}

    # Get the classifier input encoder, if we don't have it already
    if self._classifierInputEncoder is None:
      if predictedFieldName is None:
        raise RuntimeError("This experiment description is missing "
              "the 'predictedField' in its config, which is required "
              "for multi-step prediction inference.")

      encoderList = sensor.getSelf().encoder.getEncoderList()
      self._numFields = len(encoderList)

      # This is getting index of predicted field if being fed to CLA.
      fieldNames = sensor.getSelf().encoder.getScalarNames()
      if predictedFieldName in fieldNames:
        self._predictedFieldIdx = fieldNames.index(predictedFieldName)
      else:
        # Predicted field was not fed into the network, only to the classifier
        self._predictedFieldIdx = None

      # In a multi-step model, the classifier input encoder is separate from
      #  the other encoders and always disabled from going into the bottom of
      # the network.
      if sensor.getSelf().disabledEncoder is not None:
        encoderList = sensor.getSelf().disabledEncoder.getEncoderList()
      else:
        encoderList = []
      if len(encoderList) >= 1:
        fieldNames = sensor.getSelf().disabledEncoder.getScalarNames()
        self._classifierInputEncoder = encoderList[fieldNames.index(
                                                        predictedFieldName)]
      else:
        # Legacy multi-step networks don't have a separate encoder for the
        #  classifier, so use the one that goes into the bottom of the network
        encoderList = sensor.getSelf().encoder.getEncoderList()
        self._classifierInputEncoder = encoderList[self._predictedFieldIdx]

    # Get the actual value and the bucket index for this sample. The
    # predicted field may not be enabled for input to the network, so we
    # explicitly encode it outside of the sensor
    # TODO: All this logic could be simpler if in the encoder itself
    if not predictedFieldName in rawInput:
      raise ValueError("Input row does not contain a value for the predicted "
                       "field configured for this model. Missing value for '%s'"
                       % predictedFieldName)
    absoluteValue = rawInput[predictedFieldName]
    bucketIdx = self._classifierInputEncoder.getBucketIndices(absoluteValue)[0]

    # Convert the absolute values to deltas if necessary
    # The bucket index should be handled correctly by the underlying delta encoder
    if isinstance(self._classifierInputEncoder, DeltaEncoder):
      # Make the delta before any values have been seen 0 so that we do not mess up the
      # range for the adaptive scalar encoder.
      if not hasattr(self,"_ms_prevVal"):
        self._ms_prevVal = absoluteValue
      prevValue = self._ms_prevVal
      self._ms_prevVal = absoluteValue
      actualValue = absoluteValue - prevValue
    else:
      actualValue = absoluteValue

    if isinstance(actualValue, float) and math.isnan(actualValue):
      actualValue = SENTINEL_VALUE_FOR_MISSING_DATA

    # Pass this information to the classifier's custom compute method
    # so that it can assign the current classification to possibly
    # multiple patterns from the past and current, and also provide
    # the expected classification for some time step(s) in the future.
    classifier.setParameter('inferenceMode', True)
    classifier.setParameter('learningMode', needLearning)
    classificationIn = {'bucketIdx': bucketIdx,
                        'actValue': actualValue}

    # Handle missing records
    if inputTSRecordIdx is not None:
      recordNum = inputTSRecordIdx
    else:
      recordNum = self.__numRunCalls
    clResults = classifier.getSelf().customCompute(recordNum=recordNum,
                                           patternNZ=patternNZ,
                                           classification=classificationIn)

    # ---------------------------------------------------------------
    # Get the prediction for every step ahead learned by the classifier
    predictionSteps = classifier.getParameter('steps')
    predictionSteps = [int(x) for x in predictionSteps.split(',')]

    # We will return the results in this dict. The top level keys
    # are the step number, the values are the relative likelihoods for
    # each classification value in that time step, represented as
    # another dict where the keys are the classification values and
    # the values are the relative likelihoods.
    inferences[InferenceElement.multiStepPredictions] = dict()
    inferences[InferenceElement.multiStepBestPredictions] = dict()
    inferences[InferenceElement.multiStepBucketLikelihoods] = dict()

    # ======================================================================
    # Plug in the predictions for each requested time step.
    for steps in predictionSteps:
      # From the clResults, compute the predicted actual value. The
      # SDRClassifier classifies the bucket index and returns a list of
      # relative likelihoods for each bucket. Let's find the max one
      # and then look up the actual value from that bucket index
      likelihoodsVec = clResults[steps]
      bucketValues = clResults['actualValues']

      # Create a dict of value:likelihood pairs. We can't simply use
      #  dict(zip(bucketValues, likelihoodsVec)) because there might be
      #  duplicate bucketValues (this happens early on in the model when
      #  it doesn't have actual values for each bucket so it returns
      #  multiple buckets with the same default actual value).
      likelihoodsDict = dict()
      bestActValue = None
      bestProb = None
      for (actValue, prob) in zip(bucketValues, likelihoodsVec):
        if actValue in likelihoodsDict:
          likelihoodsDict[actValue] += prob
        else:
          likelihoodsDict[actValue] = prob
        # Keep track of best
        if bestProb is None or likelihoodsDict[actValue] > bestProb:
          bestProb = likelihoodsDict[actValue]
          bestActValue = actValue

      # Remove entries with 0 likelihood or likelihood less than
      # minLikelihoodThreshold, but don't leave an empty dict.
      likelihoodsDict = HTMPredictionModel._removeUnlikelyPredictions(
          likelihoodsDict, minLikelihoodThreshold, maxPredictionsPerStep)

      # calculate likelihood for each bucket
      bucketLikelihood = {}
      for k in likelihoodsDict.keys():
        bucketLikelihood[self._classifierInputEncoder.getBucketIndices(k)[0]] = (
                                                                likelihoodsDict[k])

      # ---------------------------------------------------------------------
      # If we have a delta encoder, we have to shift our predicted output value
      #  by the sum of the deltas
      if isinstance(self._classifierInputEncoder, DeltaEncoder):
        # Get the prediction history for this number of timesteps.
        # The prediction history is a store of the previous best predicted values.
        # This is used to get the final shift from the current absolute value.
        if not hasattr(self, '_ms_predHistories'):
          self._ms_predHistories = dict()
        predHistories = self._ms_predHistories
        if not steps in predHistories:
          predHistories[steps] = deque()
        predHistory = predHistories[steps]

        # Find the sum of the deltas for the steps and use this to generate
        # an offset from the current absolute value
        sumDelta = sum(predHistory)
        offsetDict = dict()
        for (k, v) in likelihoodsDict.iteritems():
          if k is not None:
            # Reconstruct the absolute value based on the current actual value,
            # the best predicted values from the previous iterations,
            # and the current predicted delta
            offsetDict[absoluteValue+float(k)+sumDelta] = v

        # calculate likelihood for each bucket
        bucketLikelihoodOffset = {}
        for k in offsetDict.keys():
          bucketLikelihoodOffset[self._classifierInputEncoder.getBucketIndices(k)[0]] = (
                                                                            offsetDict[k])

        # Push the current best delta to the history buffer for reconstructing the final delta
        if bestActValue is not None:
          predHistory.append(bestActValue)
        # If we don't need any more values in the predictionHistory, pop off
        # the earliest one.
        if len(predHistory) >= steps:
          predHistory.popleft()

        # Provide the offsetDict as the return value
        if len(offsetDict)>0:
          inferences[InferenceElement.multiStepPredictions][steps] = offsetDict
          inferences[InferenceElement.multiStepBucketLikelihoods][steps] = bucketLikelihoodOffset
        else:
          inferences[InferenceElement.multiStepPredictions][steps] = likelihoodsDict
          inferences[InferenceElement.multiStepBucketLikelihoods][steps] = bucketLikelihood

        if bestActValue is None:
          inferences[InferenceElement.multiStepBestPredictions][steps] = None
        else:
          inferences[InferenceElement.multiStepBestPredictions][steps] = (
            absoluteValue + sumDelta + bestActValue)

      # ---------------------------------------------------------------------
      # Normal case, no delta encoder. Just plug in all our multi-step predictions
      #  with likelihoods as well as our best prediction
      else:
        # The multiStepPredictions element holds the probabilities for each
        #  bucket
        inferences[InferenceElement.multiStepPredictions][steps] = (
                                                      likelihoodsDict)
        inferences[InferenceElement.multiStepBestPredictions][steps] = (
                                                      bestActValue)
        inferences[InferenceElement.multiStepBucketLikelihoods][steps] = (
                                                      bucketLikelihood)

    return inferences

Response: This method handles the CLA Classifier multi - step inference logic when implementing multi - step prediction.


Instruction:
Given the following Python 3 function, write the documentation
def _removeUnlikelyPredictions(cls, likelihoodsDict, minLikelihoodThreshold,
                                 maxPredictionsPerStep):
    """Remove entries with 0 likelihood or likelihood less than
    minLikelihoodThreshold, but don't leave an empty dict.
    """
    maxVal = (None, None)
    for (k, v) in likelihoodsDict.items():
      if len(likelihoodsDict) <= 1:
        break
      if maxVal[0] is None or v >= maxVal[1]:
        if maxVal[0] is not None and maxVal[1] < minLikelihoodThreshold:
          del likelihoodsDict[maxVal[0]]
        maxVal = (k, v)
      elif v < minLikelihoodThreshold:
        del likelihoodsDict[k]
    # Limit the number of predictions to include.
    likelihoodsDict = dict(sorted(likelihoodsDict.iteritems(),
                                  key=itemgetter(1),
                                  reverse=True)[:maxPredictionsPerStep])
    return likelihoodsDict

Response: Remove entries with 0 likelihood or likelihood less than minLikelihoodThreshold but don t leave an empty dict.


Instruction:
How would you explain what the following Python 3 function does
def getRuntimeStats(self):
    ret = {"numRunCalls" : self.__numRunCalls}

    #--------------------------------------------------
    # Query temporal network stats
    temporalStats = dict()
    if self._hasTP:
      for stat in self._netInfo.statsCollectors:
        sdict = stat.getStats()
        temporalStats.update(sdict)

    ret[InferenceType.getLabel(InferenceType.TemporalNextStep)] = temporalStats

    return ret

Response: Returns a dictionary of stats for a stat called numRunCalls.


Instruction:
Write a Python 3 script to
return the Classifier region in the network

Response:
def _getClassifierRegion(self):
    """
    Returns reference to the network's Classifier region
    """
    if (self._netInfo.net is not None and
        "Classifier" in self._netInfo.net.regions):
      return self._netInfo.net.regions["Classifier"]
    else:
      return None


Instruction:
Can you write a function in Python 3 where it
creates a CLA network and return it.

Response:
def __createHTMNetwork(self, sensorParams, spEnable, spParams, tmEnable,
                         tmParams, clEnable, clParams, anomalyParams):
    """ Create a CLA network and return it.

    description:  HTMPredictionModel description dictionary (TODO: define schema)
    Returns:      NetworkInfo instance;
    """

    #--------------------------------------------------
    # Create the network
    n = Network()

    #--------------------------------------------------
    # Add the Sensor
    n.addRegion("sensor", "py.RecordSensor", json.dumps(dict(verbosity=sensorParams['verbosity'])))
    sensor = n.regions['sensor'].getSelf()

    enabledEncoders = copy.deepcopy(sensorParams['encoders'])
    for name, params in enabledEncoders.items():
      if params is not None:
        classifierOnly = params.pop('classifierOnly', False)
        if classifierOnly:
          enabledEncoders.pop(name)

    # Disabled encoders are encoders that are fed to SDRClassifierRegion but not
    # SP or TM Regions. This is to handle the case where the predicted field
    # is not fed through the SP/TM. We typically just have one of these now.
    disabledEncoders = copy.deepcopy(sensorParams['encoders'])
    for name, params in disabledEncoders.items():
      if params is None:
        disabledEncoders.pop(name)
      else:
        classifierOnly = params.pop('classifierOnly', False)
        if not classifierOnly:
          disabledEncoders.pop(name)

    encoder = MultiEncoder(enabledEncoders)

    sensor.encoder = encoder
    sensor.disabledEncoder = MultiEncoder(disabledEncoders)
    sensor.dataSource = DataBuffer()

    prevRegion = "sensor"
    prevRegionWidth = encoder.getWidth()

    # SP is not enabled for spatial classification network
    if spEnable:
      spParams = spParams.copy()
      spParams['inputWidth'] = prevRegionWidth
      self.__logger.debug("Adding SPRegion; spParams: %r" % spParams)
      n.addRegion("SP", "py.SPRegion", json.dumps(spParams))

      # Link SP region
      n.link("sensor", "SP", "UniformLink", "")
      n.link("sensor", "SP", "UniformLink", "", srcOutput="resetOut",
             destInput="resetIn")

      n.link("SP", "sensor", "UniformLink", "", srcOutput="spatialTopDownOut",
             destInput="spatialTopDownIn")
      n.link("SP", "sensor", "UniformLink", "", srcOutput="temporalTopDownOut",
             destInput="temporalTopDownIn")

      prevRegion = "SP"
      prevRegionWidth = spParams['columnCount']

    if tmEnable:
      tmParams = tmParams.copy()
      if prevRegion == 'sensor':
        tmParams['inputWidth'] = tmParams['columnCount'] = prevRegionWidth
      else:
        assert tmParams['columnCount'] == prevRegionWidth
        tmParams['inputWidth'] = tmParams['columnCount']

      self.__logger.debug("Adding TMRegion; tmParams: %r" % tmParams)
      n.addRegion("TM", "py.TMRegion", json.dumps(tmParams))

      # Link TM region
      n.link(prevRegion, "TM", "UniformLink", "")
      if prevRegion != "sensor":
        n.link("TM", prevRegion, "UniformLink", "", srcOutput="topDownOut",
           destInput="topDownIn")
      else:
        n.link("TM", prevRegion, "UniformLink", "", srcOutput="topDownOut",
           destInput="temporalTopDownIn")
      n.link("sensor", "TM", "UniformLink", "", srcOutput="resetOut",
         destInput="resetIn")

      prevRegion = "TM"
      prevRegionWidth = tmParams['inputWidth']

    if clEnable and clParams is not None:
      clParams = clParams.copy()
      clRegionName = clParams.pop('regionName')
      self.__logger.debug("Adding %s; clParams: %r" % (clRegionName,
                                                      clParams))
      n.addRegion("Classifier", "py.%s" % str(clRegionName), json.dumps(clParams))

      # SDR Classifier-specific links
      if str(clRegionName) == "SDRClassifierRegion":
        n.link("sensor", "Classifier", "UniformLink", "", srcOutput="actValueOut",
               destInput="actValueIn")
        n.link("sensor", "Classifier", "UniformLink", "", srcOutput="bucketIdxOut",
               destInput="bucketIdxIn")

      # This applies to all (SDR and KNN) classifiers
      n.link("sensor", "Classifier", "UniformLink", "", srcOutput="categoryOut",
             destInput="categoryIn")

      n.link(prevRegion, "Classifier", "UniformLink", "")

    if self.getInferenceType() == InferenceType.TemporalAnomaly:
      anomalyClParams = dict(
          trainRecords=anomalyParams.get('autoDetectWaitRecords', None),
          cacheSize=anomalyParams.get('anomalyCacheRecords', None)
      )
      self._addAnomalyClassifierRegion(n, anomalyClParams, spEnable, tmEnable)

    #--------------------------------------------------
    # NuPIC doesn't initialize the network until you try to run it
    # but users may want to access components in a setup callback
    n.initialize()

    return NetworkInfo(net=n, statsCollectors=[])


Instruction:
Here you have a function in Python 3, explain what it does
def write(self, proto):
    super(HTMPredictionModel, self).writeBaseToProto(proto.modelBase)

    proto.numRunCalls = self.__numRunCalls
    proto.minLikelihoodThreshold = self._minLikelihoodThreshold
    proto.maxPredictionsPerStep = self._maxPredictionsPerStep

    self._netInfo.net.write(proto.network)
    proto.spLearningEnabled = self.__spLearningEnabled
    proto.tpLearningEnabled = self.__tpLearningEnabled
    if self._predictedFieldIdx is None:
      proto.predictedFieldIdx.none = None
    else:
      proto.predictedFieldIdx.value = self._predictedFieldIdx
    if self._predictedFieldName is None:
      proto.predictedFieldName.none = None
    else:
      proto.predictedFieldName.value = self._predictedFieldName
    if self._numFields is None:
      proto.numFields.none = None
    else:
      proto.numFields.value = self._numFields
    proto.trainSPNetOnlyIfRequested = self.__trainSPNetOnlyIfRequested
    proto.finishedLearning = self.__finishedLearning

Response: writes the current state of this object to the specified proto message builder


Instruction:
Can you generate a brief explanation for the following Python 3 code
def read(cls, proto):
    obj = object.__new__(cls)
    # model.capnp
    super(HTMPredictionModel, obj).__init__(proto=proto.modelBase)

    # HTMPredictionModelProto.capnp
    obj._minLikelihoodThreshold = round(proto.minLikelihoodThreshold,
                                        EPSILON_ROUND)
    obj._maxPredictionsPerStep = proto.maxPredictionsPerStep

    network = Network.read(proto.network)
    obj._hasSP = ("SP" in network.regions)
    obj._hasTP = ("TM" in network.regions)
    obj._hasCL = ("Classifier" in network.regions)
    obj._netInfo = NetworkInfo(net=network, statsCollectors=[])

    obj.__spLearningEnabled = bool(proto.spLearningEnabled)
    obj.__tpLearningEnabled = bool(proto.tpLearningEnabled)
    obj.__numRunCalls = proto.numRunCalls

    obj._classifierInputEncoder = None
    if proto.predictedFieldIdx.which() == "none":
      obj._predictedFieldIdx = None
    else:
      obj._predictedFieldIdx = proto.predictedFieldIdx.value
    if proto.predictedFieldName.which() == "none":
      obj._predictedFieldName = None
    else:
      obj._predictedFieldName = proto.predictedFieldName.value
    obj._numFields = proto.numFields
    if proto.numFields.which() == "none":
      obj._numFields = None
    else:
      obj._numFields = proto.numFields.value
    obj.__trainSPNetOnlyIfRequested = proto.trainSPNetOnlyIfRequested
    obj.__finishedLearning = proto.finishedLearning
    obj._input = None
    sensor = network.regions['sensor'].getSelf()
    sensor.dataSource = DataBuffer()
    network.initialize()

    obj.__logger = initLogger(obj)
    obj.__logger.debug("Instantiating %s." % obj.__myClassName)

    # Mark end of restoration from state
    obj.__restoringFromState = False
    obj.__restoringFromV1 = False

    return obj

Response: Reads an HTMPredictionModel from a proto message.


Instruction:
How would you explain what the following Python 3 function does
def _deSerializeExtraData(self, extraDataDir):
    assert self.__restoringFromState

    #--------------------------------------------------
    # Check to make sure that our Network member wasn't restored from
    # serialized data
    assert (self._netInfo.net is None), "Network was already unpickled"

    #--------------------------------------------------
    # Restore the network
    stateDir = self.__getNetworkStateDirectory(extraDataDir=extraDataDir)

    self.__logger.debug(
      "(%s) De-serializing network...", self)

    self._netInfo.net = Network(stateDir)

    self.__logger.debug(
      "(%s) Finished de-serializing network", self)

    # NuPIC doesn't initialize the network until you try to run it
    # but users may want to access components in a setup callback
    self._netInfo.net.initialize()

    # Used for backwards compatibility for anomaly classification models.
    # Previous versions used the HTMPredictionModelClassifierHelper class for utilizing
    # the KNN classifier. Current version uses KNNAnomalyClassifierRegion to
    # encapsulate all the classifier functionality.
    if self.getInferenceType() == InferenceType.TemporalAnomaly:
      classifierType = self._getAnomalyClassifier().getSelf().__class__.__name__
      if classifierType is 'KNNClassifierRegion':

        anomalyClParams = dict(
          trainRecords=self._classifier_helper._autoDetectWaitRecords,
          cacheSize=self._classifier_helper._history_length,
        )

        spEnable = (self._getSPRegion() is not None)
        tmEnable = True

        # Store original KNN region
        knnRegion = self._getAnomalyClassifier().getSelf()

        # Add new KNNAnomalyClassifierRegion
        self._addAnomalyClassifierRegion(self._netInfo.net, anomalyClParams,
                                         spEnable, tmEnable)

        # Restore state
        self._getAnomalyClassifier().getSelf()._iteration = self.__numRunCalls
        self._getAnomalyClassifier().getSelf()._recordsCache = (
            self._classifier_helper.saved_states)
        self._getAnomalyClassifier().getSelf().saved_categories = (
            self._classifier_helper.saved_categories)
        self._getAnomalyClassifier().getSelf()._knnclassifier = knnRegion

        # Set TM to output neccessary information
        self._getTPRegion().setParameter('anomalyMode', True)

        # Remove old classifier_helper
        del self._classifier_helper

        self._netInfo.net.initialize()

    #--------------------------------------------------
    # Mark end of restoration from state
    self.__restoringFromState = False

    self.__logger.debug("(%s) Finished restoring from state", self)

    return

Response: This method de - serialize the extra data of the current model into a new NuPIC structure.


Instruction:
Explain what the following Python 3 code does
def _addAnomalyClassifierRegion(self, network, params, spEnable, tmEnable):

    allParams = copy.deepcopy(params)
    knnParams = dict(k=1,
                     distanceMethod='rawOverlap',
                     distanceNorm=1,
                     doBinarization=1,
                     replaceDuplicates=0,
                     maxStoredPatterns=1000)
    allParams.update(knnParams)

    # Set defaults if not set
    if allParams['trainRecords'] is None:
      allParams['trainRecords'] = DEFAULT_ANOMALY_TRAINRECORDS

    if allParams['cacheSize'] is None:
      allParams['cacheSize'] = DEFAULT_ANOMALY_CACHESIZE

    # Remove current instance if already created (used for deserializing)
    if self._netInfo is not None and self._netInfo.net is not None \
              and self._getAnomalyClassifier() is not None:
      self._netInfo.net.removeRegion('AnomalyClassifier')

    network.addRegion("AnomalyClassifier",
                      "py.KNNAnomalyClassifierRegion",
                      json.dumps(allParams))

    # Attach link to SP
    if spEnable:
      network.link("SP", "AnomalyClassifier", "UniformLink", "",
          srcOutput="bottomUpOut", destInput="spBottomUpOut")
    else:
      network.link("sensor", "AnomalyClassifier", "UniformLink", "",
          srcOutput="dataOut", destInput="spBottomUpOut")

    # Attach link to TM
    if tmEnable:
      network.link("TM", "AnomalyClassifier", "UniformLink", "",
              srcOutput="topDownOut", destInput="tpTopDownOut")
      network.link("TM", "AnomalyClassifier", "UniformLink", "",
              srcOutput="lrnActiveStateT", destInput="tpLrnActiveStateT")
    else:
      raise RuntimeError("TemporalAnomaly models require a TM region.")

Response: Adds an AnomalyClassifier region to the network.


Instruction:
Make a summary of the following Python 3 code
def __getNetworkStateDirectory(self, extraDataDir):
    if self.__restoringFromV1:
      if self.getInferenceType() == InferenceType.TemporalNextStep:
        leafName = 'temporal'+ "-network.nta"
      else:
        leafName = 'nonTemporal'+ "-network.nta"
    else:
      leafName = InferenceType.getLabel(self.getInferenceType()) + "-network.nta"
    path = os.path.join(extraDataDir, leafName)
    path = os.path.abspath(path)
    return path

Response: Returns the absolute path to the network state directory for saving CLA Network.


Instruction:
Can you tell what is the following Python 3 function doing
def __manglePrivateMemberName(self, privateMemberName, skipCheck=False):

    assert privateMemberName.startswith("__"), \
           "%r doesn't start with __" % privateMemberName
    assert not privateMemberName.startswith("___"), \
           "%r starts with ___" % privateMemberName
    assert not privateMemberName.endswith("__"), \
           "%r ends with more than one underscore" % privateMemberName

    realName = "_" + (self.__myClassName).lstrip("_") + privateMemberName

    if not skipCheck:
      # This will throw an exception if the member is missing
      getattr(self, realName)

    return realName

Response: Mangles the given private member name into a single unmangled member name.


Instruction:
Create a Python 3 function to
set the radius resolution and range of the encoder.

Response:
def _setEncoderParams(self):
    """
    Set the radius, resolution and range. These values are updated when minval
    and/or maxval change.
    """

    self.rangeInternal = float(self.maxval - self.minval)

    self.resolution = float(self.rangeInternal) / (self.n - self.w)
    self.radius = self.w * self.resolution
    self.range = self.rangeInternal + self.resolution

    # nInternal represents the output area excluding the possible padding on each side
    self.nInternal = self.n - 2 * self.padding

    # Invalidate the bucket values cache so that they get recomputed
    self._bucketValues = None


Instruction:
How would you explain what the following Python 3 function does
def setFieldStats(self, fieldName, fieldStats):
    #If the stats are not fully formed, ignore.
    if fieldStats[fieldName]['min'] == None or \
      fieldStats[fieldName]['max'] == None:
        return
    self.minval = fieldStats[fieldName]['min']
    self.maxval = fieldStats[fieldName]['max']
    if self.minval == self.maxval:
      self.maxval+=1
    self._setEncoderParams()

Response: Sets the min and max values of the fieldStats for the specified fieldName.


Instruction:
Write a Python 3 script for
overriding this method to set the min and max values for the next bucket.

Response:
def getBucketIndices(self, input, learn=None):
    """
    [overrides nupic.encoders.scalar.ScalarEncoder.getBucketIndices]
    """

    self.recordNum +=1
    if learn is None:
      learn = self._learningEnabled

    if type(input) is float and math.isnan(input):
      input = SENTINEL_VALUE_FOR_MISSING_DATA

    if input == SENTINEL_VALUE_FOR_MISSING_DATA:
      return [None]
    else:
      self._setMinAndMax(input, learn)
      return super(AdaptiveScalarEncoder, self).getBucketIndices(input)


Instruction:
Here you have a function in Python 3, explain what it does
def encodeIntoArray(self, input, output,learn=None):

    self.recordNum +=1
    if learn is None:
      learn = self._learningEnabled
    if input == SENTINEL_VALUE_FOR_MISSING_DATA:
        output[0:self.n] = 0
    elif not math.isnan(input):
      self._setMinAndMax(input, learn)

    super(AdaptiveScalarEncoder, self).encodeIntoArray(input, output)

Response: Override encodeIntoArray to set the min and max values for the next record.


Instruction:
Implement a Python 3 function for
overriding this method to return encoderResult for multi - valued buckets.

Response:
def getBucketInfo(self, buckets):
    """
    [overrides nupic.encoders.scalar.ScalarEncoder.getBucketInfo]
    """

    if self.minval is None or self.maxval is None:
      return [EncoderResult(value=0, scalar=0,
                           encoding=numpy.zeros(self.n))]

    return super(AdaptiveScalarEncoder, self).getBucketInfo(buckets)


Instruction:
Can you tell what is the following Python 3 function doing
def topDownCompute(self, encoded):

    if self.minval is None or self.maxval is None:
      return [EncoderResult(value=0, scalar=0,
                           encoding=numpy.zeros(self.n))]
    return super(AdaptiveScalarEncoder, self).topDownCompute(encoded)

Response: Compute the top down of the encoder.


Instruction:
Can you generate the documentation for the following Python 3 function
def recordDataPoint(self, swarmId, generation, errScore):
    terminatedSwarms = []

    # Append score to existing swarm.
    if swarmId in self.swarmScores:
      entry = self.swarmScores[swarmId]
      assert(len(entry) == generation)
      entry.append(errScore)

      entry = self.swarmBests[swarmId]
      entry.append(min(errScore, entry[-1]))

      assert(len(self.swarmBests[swarmId]) == len(self.swarmScores[swarmId]))
    else:
      # Create list of scores for a new swarm
      assert (generation == 0)
      self.swarmScores[swarmId] = [errScore]
      self.swarmBests[swarmId] = [errScore]

    # If the current swarm hasn't completed at least MIN_GENERATIONS, it should
    # not be candidate for maturation or termination. This prevents the initial
    # allocation of particles in PSO from killing off a field combination too
    # early.
    if generation + 1 < self.MATURITY_WINDOW:
      return terminatedSwarms

    # If the swarm has completed more than MAX_GENERATIONS, it should be marked
    # as mature, regardless of how its value is changing.
    if self.MAX_GENERATIONS is not None and generation > self.MAX_GENERATIONS:
      self._logger.info(
          'Swarm %s has matured (more than %d generations). Stopping' %
          (swarmId, self.MAX_GENERATIONS))
      terminatedSwarms.append(swarmId)

    if self._isTerminationEnabled:
      terminatedSwarms.extend(self._getTerminatedSwarms(generation))

    # Return which swarms to kill when we've reached maturity
    # If there is no change in the swarm's best for some time,
    # Mark it dead
    cumulativeBestScores = self.swarmBests[swarmId]
    if cumulativeBestScores[-1] == cumulativeBestScores[-self.MATURITY_WINDOW]:
      self._logger.info('Swarm %s has matured (no change in %d generations).'
                        'Stopping...'% (swarmId, self.MATURITY_WINDOW))
      terminatedSwarms.append(swarmId)

    self.terminatedSwarms = self.terminatedSwarms.union(terminatedSwarms)
    return terminatedSwarms

Response: Record the best score for a given swarm. Returns a list of swarmIds to terminate when we reach the given generation index.


Instruction:
Write a Python 3 script to
return a dictionary of all the state of the current object.

Response:
def getState(self):
    """See comments in base class."""
    return dict(_position = self._position,
                position = self.getPosition(),
                velocity = self._velocity,
                bestPosition = self._bestPosition,
                bestResult = self._bestResult)


Instruction:
Can you write a function in Python 3 where it
sets the state of the internal state of the internal state of the internal state.

Response:
def setState(self, state):
    """See comments in base class."""
    self._position = state['_position']
    self._velocity = state['velocity']
    self._bestPosition = state['bestPosition']
    self._bestResult = state['bestResult']


Instruction:
Can you generate the documentation for the following Python 3 function
def getPosition(self):
    if self.stepSize is None:
      return self._position

    # Find nearest step
    numSteps = (self._position - self.min)  / self.stepSize
    numSteps = int(round(numSteps))
    position = self.min + (numSteps * self.stepSize)
    position = max(self.min, position)
    position = min(self.max, position)
    return position

Response: Returns the position of the next step in the sequence.


Instruction:
How would you explain what the following Python 3 function does
def agitate(self):
    # Increase velocity enough that it will be higher the next time
    # newPosition() is called. We know that newPosition multiplies by inertia,
    # so take that into account.
    self._velocity *= 1.5 / self._inertia

    # Clip velocity
    maxV = (self.max - self.min)/2
    if self._velocity > maxV:
      self._velocity = maxV
    elif self._velocity < -maxV:
      self._velocity = -maxV

    # if we at the max or min, reverse direction
    if self._position == self.max and self._velocity > 0:
      self._velocity *= -1
    if self._position == self.min and self._velocity < 0:
      self._velocity *= -1

Response: This method is used to agitate the current time series.


Instruction:
Make a summary of the following Python 3 code
def newPosition(self, globalBestPosition, rng):
    # First, update the velocity. The new velocity is given as:
    # v = (inertia * v)  + (cogRate * r1 * (localBest-pos))
    #                    + (socRate * r2 * (globalBest-pos))
    #
    # where r1 and r2 are random numbers between 0 and 1.0
    lb=float(Configuration.get("nupic.hypersearch.randomLowerBound"))
    ub=float(Configuration.get("nupic.hypersearch.randomUpperBound"))

    self._velocity = (self._velocity * self._inertia + rng.uniform(lb, ub) *
                      self._cogRate * (self._bestPosition - self.getPosition()))
    if globalBestPosition is not None:
      self._velocity += rng.uniform(lb, ub) * self._socRate * (
          globalBestPosition - self.getPosition())

    # update position based on velocity
    self._position += self._velocity

    # Clip it
    self._position = max(self.min, self._position)
    self._position = min(self.max, self._position)

    # Return it
    return self.getPosition()

Response: Update the position based on the current position and the global best position.


Instruction:
Write a Python 3 function that can
push a particle from another set of positions.

Response:
def pushAwayFrom(self, otherPositions, rng):
    """See comments in base class."""
    # If min and max are the same, nothing to do
    if self.max == self.min:
      return

    # How many potential other positions to evaluate?
    numPositions = len(otherPositions) * 4
    if numPositions == 0:
      return

    # Assign a weight to each potential position based on how close it is
    # to other particles.
    stepSize = float(self.max-self.min) / numPositions
    positions = numpy.arange(self.min, self.max + stepSize, stepSize)

    # Get rid of duplicates.
    numPositions = len(positions)
    weights = numpy.zeros(numPositions)

    # Assign a weight to each potential position, based on a gaussian falloff
    # from each existing variable. The weight of a variable to each potential
    # position is given as:
    #    e ^ -(dist^2/stepSize^2)
    maxDistanceSq = -1 * (stepSize ** 2)
    for pos in otherPositions:
      distances = pos - positions
      varWeights = numpy.exp(numpy.power(distances, 2) / maxDistanceSq)
      weights += varWeights

    # Put this particle at the position with smallest weight.
    positionIdx = weights.argmin()
    self._position = positions[positionIdx]

    # Set its best position to this.
    self._bestPosition = self.getPosition()

    # Give it a random direction.
    self._velocity *= rng.choice([1, -1])


Instruction:
Can you implement a function in Python 3 that
resets the velocity of the current node to a random value.

Response:
def resetVelocity(self, rng):
    """See comments in base class."""
    maxVelocity = (self.max - self.min) / 5.0
    self._velocity = maxVelocity #min(abs(self._velocity), maxVelocity)
    self._velocity *= rng.choice([1, -1])


Instruction:
Explain what the following Python 3 code does
def getPosition(self):
    position = super(PermuteInt, self).getPosition()
    position =  int(round(position))
    return position

Response: Returns the position of the permute integer.


Instruction:
How would you explain what the following Python 3 function does
def getState(self):
    return dict(_position = self.getPosition(),
                position = self.getPosition(),
                velocity = None,
                bestPosition = self.choices[self._bestPositionIdx],
                bestResult = self._bestResult)

Response: Returns a dictionary of state of the current page.


Instruction:
Here you have a function in Python 3, explain what it does
def setState(self, state):
    self._positionIdx = self.choices.index(state['_position'])
    self._bestPositionIdx = self.choices.index(state['bestPosition'])
    self._bestResult = state['bestResult']

Response: Sets the state of the current selection.


Instruction:
Here you have a function in Python 3, explain what it does
def newPosition(self, globalBestPosition, rng):
    # Compute the mean score per choice.
    numChoices = len(self.choices)
    meanScorePerChoice = []
    overallSum = 0
    numResults = 0

    for i in range(numChoices):
      if len(self._resultsPerChoice[i]) > 0:
        data = numpy.array(self._resultsPerChoice[i])
        meanScorePerChoice.append(data.mean())
        overallSum += data.sum()
        numResults += data.size
      else:
        meanScorePerChoice.append(None)

    if numResults == 0:
      overallSum = 1.0
      numResults = 1

    # For any choices we don't have a result for yet, set to the overall mean.
    for i in range(numChoices):
      if meanScorePerChoice[i] is None:
        meanScorePerChoice[i] = overallSum / numResults

    # Now, pick a new choice based on the above probabilities. Note that the
    #  best result is the lowest result. We want to make it more likely to
    #  pick the choice that produced the lowest results. So, we need to invert
    #  the scores (someLargeNumber - score).
    meanScorePerChoice = numpy.array(meanScorePerChoice)

    # Invert meaning.
    meanScorePerChoice = (1.1 * meanScorePerChoice.max()) - meanScorePerChoice

    # If you want the scores to quickly converge to the best choice, raise the
    # results to a power. This will cause lower scores to become lower
    # probability as you see more results, until it eventually should
    # assymptote to only choosing the best choice.
    if self._fixEarly:
      meanScorePerChoice **= (numResults * self._fixEarlyFactor / numChoices)
    # Normalize.
    total = meanScorePerChoice.sum()
    if total == 0:
      total = 1.0
    meanScorePerChoice /= total

    # Get distribution and choose one based on those probabilities.
    distribution = meanScorePerChoice.cumsum()
    r = rng.random() * distribution[-1]
    choiceIdx = numpy.where(r <= distribution)[0][0]

    self._positionIdx = choiceIdx
    return self.getPosition()

Response: This method is used to set the best position of the new choice.


Instruction:
Explain what the following Python 3 code does
def pushAwayFrom(self, otherPositions, rng):
    # Get the count of how many in each position
    positions = [self.choices.index(x) for x in otherPositions]

    positionCounts = [0] * len(self.choices)
    for pos in positions:
      positionCounts[pos] += 1

    self._positionIdx = numpy.array(positionCounts).argmin()
    self._bestPositionIdx = self._positionIdx

Response: Pushes a way from another position to the current one.


Instruction:
Implement a function in Python 3 to
return a dict that can be used to construct this encoder.

Response:
def getDict(self, encoderName, flattenedChosenValues):
    """ Return a dict that can be used to construct this encoder. This dict
    can be passed directly to the addMultipleEncoders() method of the
    multi encoder.

    Parameters:
    ----------------------------------------------------------------------
    encoderName:            name of the encoder
    flattenedChosenValues:  dict of the flattened permutation variables. Any
                              variables within this dict whose key starts
                              with encoderName will be substituted for
                              encoder constructor args which are being
                              permuted over.
    """
    encoder = dict(fieldname=self.fieldName,
                   name=self.name)

    # Get the position of each encoder argument
    for encoderArg, value in self.kwArgs.iteritems():
      # If a permuted variable, get its chosen value.
      if isinstance(value, PermuteVariable):
        value = flattenedChosenValues["%s:%s" % (encoderName, encoderArg)]

      encoder[encoderArg] = value

    # Special treatment for DateEncoder timeOfDay and dayOfWeek stuff. In the
    #  permutations file, the class can be one of:
    #    DateEncoder.timeOfDay
    #    DateEncoder.dayOfWeek
    #    DateEncoder.season
    # If one of these, we need to intelligently set the constructor args.
    if '.' in self.encoderClass:
      (encoder['type'], argName) = self.encoderClass.split('.')
      argValue = (encoder['w'], encoder['radius'])
      encoder[argName] = argValue
      encoder.pop('w')
      encoder.pop('radius')
    else:
      encoder['type'] = self.encoderClass

    return encoder


Instruction:
How would you explain what the following Python 3 function does
def _translateMetricsToJSON(self, metrics, label):

    # Transcode the MetricValueElement values into JSON-compatible
    # structure
    metricsDict = metrics

    # Convert the structure to a display-friendly JSON string
    def _mapNumpyValues(obj):
      """
      """
      import numpy

      if isinstance(obj, numpy.float32):
        return float(obj)

      elif isinstance(obj, numpy.bool_):
        return bool(obj)

      elif isinstance(obj, numpy.ndarray):
        return obj.tolist()

      else:
        raise TypeError("UNEXPECTED OBJ: %s; class=%s" % (obj, obj.__class__))

    jsonString = json.dumps(metricsDict, indent=4, default=_mapNumpyValues)

    return jsonString

Response: Translate the given metrics value to a JSON string.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def __openDatafile(self, modelResult):

    # Write reset bit
    resetFieldMeta = FieldMetaInfo(
      name="reset",
      type=FieldMetaType.integer,
      special = FieldMetaSpecial.reset)

    self.__outputFieldsMeta.append(resetFieldMeta)

    # -----------------------------------------------------------------------
    # Write each of the raw inputs that go into the encoders
    rawInput = modelResult.rawInput
    rawFields = rawInput.keys()
    rawFields.sort()
    for field in rawFields:
      if field.startswith('_') or field == 'reset':
        continue
      value = rawInput[field]
      meta = FieldMetaInfo(name=field, type=FieldMetaType.string,
                           special=FieldMetaSpecial.none)
      self.__outputFieldsMeta.append(meta)
      self._rawInputNames.append(field)

    # -----------------------------------------------------------------------
    # Handle each of the inference elements
    for inferenceElement, value in modelResult.inferences.iteritems():
      inferenceLabel = InferenceElement.getLabel(inferenceElement)

      # TODO: Right now we assume list inferences are associated with
      # The input field metadata
      if type(value) in (list, tuple):
        # Append input and prediction field meta-info
        self.__outputFieldsMeta.extend(self.__getListMetaInfo(inferenceElement))

      elif isinstance(value, dict):
          self.__outputFieldsMeta.extend(self.__getDictMetaInfo(inferenceElement,
                                                                value))
      else:

        if InferenceElement.getInputElement(inferenceElement):
          self.__outputFieldsMeta.append(FieldMetaInfo(name=inferenceLabel+".actual",
                type=FieldMetaType.string, special = ''))
        self.__outputFieldsMeta.append(FieldMetaInfo(name=inferenceLabel,
                type=FieldMetaType.string, special = ''))

    if self.__metricNames:
      for metricName in self.__metricNames:
        metricField = FieldMetaInfo(
          name = metricName,
          type = FieldMetaType.float,
          special = FieldMetaSpecial.none)

        self.__outputFieldsMeta.append(metricField)

    # Create the inference directory for our experiment
    inferenceDir = _FileUtils.createExperimentInferenceDir(self.__experimentDir)

    # Consctruct the prediction dataset file path
    filename = (self.__label + "." +
                opf_utils.InferenceType.getLabel(self.__inferenceType) +
               ".predictionLog.csv")
    self.__datasetPath = os.path.join(inferenceDir, filename)

    # Create the output dataset
    print "OPENING OUTPUT FOR PREDICTION WRITER AT: %r" % self.__datasetPath
    print "Prediction field-meta: %r" % ([tuple(i) for i in self.__outputFieldsMeta],)
    self.__dataset = FileRecordStream(streamID=self.__datasetPath, write=True,
                                     fields=self.__outputFieldsMeta)

    # Copy data from checkpoint cache
    if self.__checkpointCache is not None:
      self.__checkpointCache.seek(0)

      reader = csv.reader(self.__checkpointCache, dialect='excel')

      # Skip header row
      try:
        header = reader.next()
      except StopIteration:
        print "Empty record checkpoint initializer for %r" % (self.__datasetPath,)
      else:
        assert tuple(self.__dataset.getFieldNames()) == tuple(header), \
          "dataset.getFieldNames(): %r; predictionCheckpointFieldNames: %r" % (
          tuple(self.__dataset.getFieldNames()), tuple(header))

      # Copy the rows from checkpoint
      numRowsCopied = 0
      while True:
        try:
          row = reader.next()
        except StopIteration:
          break

        #print "DEBUG: restoring row from checkpoint: %r" % (row,)

        self.__dataset.appendRecord(row)
        numRowsCopied += 1

      self.__dataset.flush()

      print "Restored %d rows from checkpoint for %r" % (
        numRowsCopied, self.__datasetPath)

      # Dispose of our checkpoint cache
      self.__checkpointCache.close()
      self.__checkpointCache = None

    return

Response: Open the data file and write the header row


Instruction:
Here you have a function in Python 3, explain what it does
def setLoggedMetrics(self, metricNames):
    if metricNames is None:
      self.__metricNames = set([])
    else:
      self.__metricNames = set(metricNames)

Response: Tells the writer which metrics should be written


Instruction:
Implement a function in Python 3 to
get the field meta information for the list inferences that are of list type

Response:
def __getListMetaInfo(self, inferenceElement):
    """ Get field metadata information for inferences that are of list type
    TODO: Right now we assume list inferences are associated with the input field
    metadata
    """
    fieldMetaInfo = []
    inferenceLabel = InferenceElement.getLabel(inferenceElement)

    for inputFieldMeta in self.__inputFieldsMeta:
      if InferenceElement.getInputElement(inferenceElement):
        outputFieldMeta = FieldMetaInfo(
          name=inputFieldMeta.name + ".actual",
          type=inputFieldMeta.type,
          special=inputFieldMeta.special
        )

      predictionField = FieldMetaInfo(
        name=inputFieldMeta.name + "." + inferenceLabel,
        type=inputFieldMeta.type,
        special=inputFieldMeta.special
      )

      fieldMetaInfo.append(outputFieldMeta)
      fieldMetaInfo.append(predictionField)

    return fieldMetaInfo


Instruction:
Here you have a function in Python 3, explain what it does
def __getDictMetaInfo(self, inferenceElement, inferenceDict):
    fieldMetaInfo = []
    inferenceLabel = InferenceElement.getLabel(inferenceElement)

    if InferenceElement.getInputElement(inferenceElement):
      fieldMetaInfo.append(FieldMetaInfo(name=inferenceLabel+".actual",
                                         type=FieldMetaType.string,
                                         special = ''))

    keys = sorted(inferenceDict.keys())
    for key in keys:
      fieldMetaInfo.append(FieldMetaInfo(name=inferenceLabel+"."+str(key),
                                         type=FieldMetaType.string,
                                         special=''))

    return fieldMetaInfo

Response: Get field metadate information for the given inference element and dict


Instruction:
Make a summary of the following Python 3 code
def append(self, modelResult):

    #print "DEBUG: _BasicPredictionWriter: writing modelResult: %r" % (modelResult,)

    # If there are no inferences, don't write anything
    inferences = modelResult.inferences
    hasInferences = False
    if inferences is not None:
      for value in inferences.itervalues():
        hasInferences = hasInferences or (value is not None)

    if not hasInferences:
      return

    if self.__dataset is None:
      self.__openDatafile(modelResult)

    inputData = modelResult.sensorInput

    sequenceReset = int(bool(inputData.sequenceReset))
    outputRow = [sequenceReset]

    # -----------------------------------------------------------------------
    # Write out the raw inputs
    rawInput = modelResult.rawInput
    for field in self._rawInputNames:
      outputRow.append(str(rawInput[field]))

    # -----------------------------------------------------------------------
    # Write out the inference element info
    for inferenceElement, outputVal in inferences.iteritems():
      inputElement = InferenceElement.getInputElement(inferenceElement)
      if inputElement:
        inputVal = getattr(inputData, inputElement)
      else:
        inputVal = None

      if type(outputVal) in (list, tuple):
        assert type(inputVal) in (list, tuple, None)

        for iv, ov in zip(inputVal, outputVal):
          # Write actual
          outputRow.append(str(iv))

          # Write inferred
          outputRow.append(str(ov))
      elif isinstance(outputVal, dict):
        if inputVal is not None:
          # If we have a predicted field, include only that in the actuals
          if modelResult.predictedFieldName is not None:
            outputRow.append(str(inputVal[modelResult.predictedFieldName]))
          else:
            outputRow.append(str(inputVal))
        for key in sorted(outputVal.keys()):
          outputRow.append(str(outputVal[key]))
      else:
        if inputVal is not None:
          outputRow.append(str(inputVal))
        outputRow.append(str(outputVal))

    metrics = modelResult.metrics
    for metricName in self.__metricNames:
      outputRow.append(metrics.get(metricName, 0.0))

    #print "DEBUG: _BasicPredictionWriter: writing outputRow: %r" % (outputRow,)

    self.__dataset.appendRecord(outputRow)

    self.__dataset.flush()

    return

Response:
Emits a single prediction as input versus
    predicted.


Instruction:
Write a Python 3 script to
save a checkpoint of the prediction output.

Response:
def checkpoint(self, checkpointSink, maxRows):
    """ [virtual method override] Save a checkpoint of the prediction output
    stream. The checkpoint comprises up to maxRows of the most recent inference
    records.

    Parameters:
    ----------------------------------------------------------------------
    checkpointSink:     A File-like object where predictions checkpoint data, if
                        any, will be stored.
    maxRows:            Maximum number of most recent inference rows
                        to checkpoint.
    """

    checkpointSink.truncate()

    if self.__dataset is None:
      if self.__checkpointCache is not None:
        self.__checkpointCache.seek(0)
        shutil.copyfileobj(self.__checkpointCache, checkpointSink)
        checkpointSink.flush()
        return
      else:
        # Nothing to checkpoint
        return

    self.__dataset.flush()
    totalDataRows = self.__dataset.getDataRowCount()

    if totalDataRows == 0:
      # Nothing to checkpoint
      return

    # Open reader of prediction file (suppress missingValues conversion)
    reader = FileRecordStream(self.__datasetPath, missingValues=[])

    # Create CSV writer for writing checkpoint rows
    writer = csv.writer(checkpointSink)

    # Write the header row to checkpoint sink -- just field names
    writer.writerow(reader.getFieldNames())

    # Determine number of rows to checkpoint
    numToWrite = min(maxRows, totalDataRows)

    # Skip initial rows to get to the rows that we actually need to checkpoint
    numRowsToSkip = totalDataRows - numToWrite
    for i in xrange(numRowsToSkip):
      reader.next()

    # Write the data rows to checkpoint sink
    numWritten = 0
    while True:
      row = reader.getNextRecord()
      if row is None:
        break;

      row =  [str(element) for element in row]

      #print "DEBUG: _BasicPredictionWriter: checkpointing row: %r" % (row,)

      writer.writerow(row)

      numWritten +=1

    assert numWritten == numToWrite, \
      "numWritten (%s) != numToWrite (%s)" % (numWritten, numToWrite)

    checkpointSink.flush()

    return


Instruction:
Write a Python 3 function for
creating the inference output directory for the given experiment

Response:
def createExperimentInferenceDir(cls, experimentDir):
    """ Creates the inference output directory for the given experiment

    experimentDir:  experiment directory path that contains description.py

    Returns:  path of the inference output directory
    """
    path = cls.getExperimentInferenceDirPath(experimentDir)

    cls.makeDirectory(path)

    return path


Instruction:
Can you create a Python 3 function that
generates the initial first order transition sequence and second order transition sequence for model0.

Response:
def _generateModel0(numCategories):
  """ Generate the initial, first order, and second order transition
  probabilities for 'model0'. For this model, we generate the following
  set of sequences:

  1-2-3   (4X)
  1-2-4   (1X)
  5-2-3   (1X)
  5-2-4   (4X)

  Parameters:
  ----------------------------------------------------------------------
  numCategories:      Number of categories
  retval: (initProb, firstOrder, secondOrder, seqLen)
            initProb:     Initial probability for each category. This is a vector
                            of length len(categoryList).
            firstOrder:   A dictionary of the 1st order probabilities. The key
                            is the 1st element of the sequence, the value is
                            the probability of each 2nd element given the first.
            secondOrder:  A dictionary of the 2nd order probabilities. The key
                            is the first 2 elements of the sequence, the value is
                            the probability of each possible 3rd element given the
                            first two.
            seqLen:       Desired length of each sequence. The 1st element will
                          be generated using the initProb, the 2nd element by the
                          firstOrder table, and the 3rd and all successive
                          elements by the secondOrder table.

  Here is an example of some return values:
  initProb:         [0.7, 0.2, 0.1]

  firstOrder:       {'[0]': [0.3, 0.3, 0.4],
                     '[1]': [0.3, 0.3, 0.4],
                     '[2]': [0.3, 0.3, 0.4]}

  secondOrder:      {'[0,0]': [0.3, 0.3, 0.4],
                     '[0,1]': [0.3, 0.3, 0.4],
                     '[0,2]': [0.3, 0.3, 0.4],
                     '[1,0]': [0.3, 0.3, 0.4],
                     '[1,1]': [0.3, 0.3, 0.4],
                     '[1,2]': [0.3, 0.3, 0.4],
                     '[2,0]': [0.3, 0.3, 0.4],
                     '[2,1]': [0.3, 0.3, 0.4],
                     '[2,2]': [0.3, 0.3, 0.4]}
  """

  # ===============================================================
  # Let's model the following:
  #  a-b-c (4X)
  #  a-b-d (1X)
  #  e-b-c (1X)
  #  e-b-d (4X)

  # --------------------------------------------------------------------
  # Initial probabilities, 'a' and 'e' equally likely
  initProb = numpy.zeros(numCategories)
  initProb[0] = 0.5
  initProb[4] = 0.5

  # --------------------------------------------------------------------
  # 1st order transitions
  # both 'a' and 'e' should lead to 'b'
  firstOrder = dict()
  for catIdx in range(numCategories):
    key = str([catIdx])
    probs = numpy.ones(numCategories) / numCategories
    if catIdx == 0 or catIdx == 4:
      probs.fill(0)
      probs[1] = 1.0    # lead only to b
    firstOrder[key] = probs

  # --------------------------------------------------------------------
  # 2nd order transitions
  # a-b should lead to c 80% and d 20%
  # e-b should lead to c 20% and d 80%
  secondOrder = dict()
  for firstIdx in range(numCategories):
    for secondIdx in range(numCategories):
      key = str([firstIdx, secondIdx])
      probs = numpy.ones(numCategories) / numCategories
      if key == str([0,1]):
        probs.fill(0)
        probs[2] = 0.80   # 'ab' leads to 'c' 80% of the time
        probs[3] = 0.20   # 'ab' leads to 'd' 20% of the time
      elif key == str([4,1]):
        probs.fill(0)
        probs[2] = 0.20   # 'eb' leads to 'c' 20% of the time
        probs[3] = 0.80   # 'eb' leads to 'd' 80% of the time

      secondOrder[key] = probs

  return (initProb, firstOrder, secondOrder, 3)


Instruction:
Can you tell what is the following Python 3 function doing
def _generateModel1(numCategories):

  # --------------------------------------------------------------------
  # Initial probabilities, 0 and 1 equally likely
  initProb = numpy.zeros(numCategories)
  initProb[0] = 0.5
  initProb[1] = 0.5

  # --------------------------------------------------------------------
  # 1st order transitions
  # both 0 and 1 should lead to 10,11,12,13,14 with equal probability
  firstOrder = dict()
  for catIdx in range(numCategories):
    key = str([catIdx])
    probs = numpy.ones(numCategories) / numCategories
    if catIdx == 0 or catIdx == 1:
      indices = numpy.array([10,11,12,13,14])
      probs.fill(0)
      probs[indices] = 1.0    # lead only to b
      probs /= probs.sum()
    firstOrder[key] = probs

  # --------------------------------------------------------------------
  # 2nd order transitions
  # 0-10 should lead to 15
  # 0-11 to 16
  # ...
  # 1-10 should lead to 20
  # 1-11 shold lean to 21
  # ...
  secondOrder = dict()
  for firstIdx in range(numCategories):
    for secondIdx in range(numCategories):
      key = str([firstIdx, secondIdx])
      probs = numpy.ones(numCategories) / numCategories
      if key == str([0,10]):
        probs.fill(0)
        probs[15] = 1
      elif key == str([0,11]):
        probs.fill(0)
        probs[16] = 1
      elif key == str([0,12]):
        probs.fill(0)
        probs[17] = 1
      elif key == str([0,13]):
        probs.fill(0)
        probs[18] = 1
      elif key == str([0,14]):
        probs.fill(0)
        probs[19] = 1

      elif key == str([1,10]):
        probs.fill(0)
        probs[20] = 1
      elif key == str([1,11]):
        probs.fill(0)
        probs[21] = 1
      elif key == str([1,12]):
        probs.fill(0)
        probs[22] = 1
      elif key == str([1,13]):
        probs.fill(0)
        probs[23] = 1
      elif key == str([1,14]):
        probs.fill(0)
        probs[24] = 1

      secondOrder[key] = probs

  return (initProb, firstOrder, secondOrder, 3)

Response: Generates the initial first order transition sequence and second order transition sequence for model1.


Instruction:
Can you tell what is the following Python 3 function doing
def _generateModel2(numCategories, alpha=0.25):

  # --------------------------------------------------------------------
  # All initial probabilities, are equally likely
  initProb = numpy.ones(numCategories)/numCategories

  def generatePeakedProbabilities(lastIdx,
				  numCategories=numCategories,
				  alpha=alpha):
    probs = numpy.random.dirichlet(alpha=[alpha]*numCategories)
    probs[lastIdx] = 0.0
    probs /= probs.sum()
    return probs

  # --------------------------------------------------------------------
  # 1st order transitions
  firstOrder = dict()
  for catIdx in range(numCategories):
    key = str([catIdx])
    probs = generatePeakedProbabilities(catIdx)
    firstOrder[key] = probs

  # --------------------------------------------------------------------
  # 2nd order transitions
  secondOrder = dict()
  for firstIdx in range(numCategories):
    for secondIdx in range(numCategories):
      key = str([firstIdx, secondIdx])
      probs = generatePeakedProbabilities(secondIdx)
      secondOrder[key] = probs

  return (initProb, firstOrder, secondOrder, None)

Response: Generates the initial first order second order transition probabilities for the model2.