Instruction:
Can you generate a brief explanation for the following Python 3 code
def _genLoggingFilePath():
  appName = os.path.splitext(os.path.basename(sys.argv[0]))[0] or 'UnknownApp'
  appLogDir = os.path.abspath(os.path.join(
    os.environ['NTA_LOG_DIR'],
    'numenta-logs-%s' % (os.environ['USER'],),
    appName))
  appLogFileName = '%s-%s-%s.log' % (
    appName, long(time.mktime(time.gmtime())), os.getpid())
  return os.path.join(appLogDir, appLogFileName)

Response: Generate a filepath for the calling app


Instruction:
Write a Python 3 script for
returning the number of months and seconds from an aggregation dict that represents a date and time. Interval is a dict that contain one or more of the following keys: 'years', 'months', 'weeks', 'days', 'hours', 'minutes', seconds', 'milliseconds', 'microseconds'. For example: :: aggregationMicroseconds({'years': 1, 'hours': 4, 'microseconds':42}) == {'months':12, 'seconds':14400.000042} :param interval: (dict) The aggregation interval representing a date and time :returns: (dict) number of months and seconds in the interval: ``{months': XX, 'seconds': XX}``. The seconds is a floating point that can represent resolutions down to a microsecond.

Response:
def aggregationToMonthsSeconds(interval):
  """
  Return the number of months and seconds from an aggregation dict that
  represents a date and time.

  Interval is a dict that contain one or more of the following keys: 'years',
  'months', 'weeks', 'days', 'hours', 'minutes', seconds', 'milliseconds',
  'microseconds'.

  For example:

  ::

    aggregationMicroseconds({'years': 1, 'hours': 4, 'microseconds':42}) ==
        {'months':12, 'seconds':14400.000042}

  :param interval: (dict) The aggregation interval representing a date and time
  :returns: (dict) number of months and seconds in the interval:
            ``{months': XX, 'seconds': XX}``. The seconds is
            a floating point that can represent resolutions down to a
            microsecond.

  """

  seconds = interval.get('microseconds', 0) * 0.000001
  seconds += interval.get('milliseconds', 0) * 0.001
  seconds += interval.get('seconds', 0)
  seconds += interval.get('minutes', 0) * 60
  seconds += interval.get('hours', 0) * 60 * 60
  seconds += interval.get('days', 0) * 24 * 60 * 60
  seconds += interval.get('weeks', 0) * 7 * 24 * 60 * 60

  months = interval.get('months', 0)
  months += 12 * interval.get('years', 0)

  return {'months': months, 'seconds': seconds}


Instruction:
Can you tell what is the following Python 3 function doing
def aggregationDivide(dividend, divisor):

  # Convert each into microseconds
  dividendMonthSec = aggregationToMonthsSeconds(dividend)
  divisorMonthSec = aggregationToMonthsSeconds(divisor)

  # It is a usage error to mix both months and seconds in the same operation
  if (dividendMonthSec['months'] != 0 and divisorMonthSec['seconds'] != 0) \
    or (dividendMonthSec['seconds'] != 0 and divisorMonthSec['months'] != 0):
    raise RuntimeError("Aggregation dicts with months/years can only be "
      "inter-operated with other aggregation dicts that contain "
      "months/years")

  if dividendMonthSec['months'] > 0:
    return float(dividendMonthSec['months']) / divisor['months']

  else:
    return float(dividendMonthSec['seconds']) / divisorMonthSec['seconds']

Response: This function takes two dicts representing a date and time and returns the result from dividing the two dicts representing a date and time.


Instruction:
Can you tell what is the following Python 3 function doing
def validateOpfJsonValue(value, opfJsonSchemaFilename):

  # Create a path by joining the filename with our local json schema root
  jsonSchemaPath = os.path.join(os.path.dirname(__file__),
                                "jsonschema",
                                opfJsonSchemaFilename)

  # Validate
  jsonhelpers.validate(value, schemaPath=jsonSchemaPath)

  return

Response: Validate a python object against an OPF json schema file.


Instruction:
Can you tell what is the following Python 3 function doing
def matchPatterns(patterns, keys):
  results = []
  if patterns:
    for pattern in patterns:
      prog = re.compile(pattern)
      for key in keys:
        if prog.match(key):
          results.append(key)
  else:
    return None

  return results

Response:
Returns a subset of the keys that match any of the given patterns

.


Instruction:
Explain what the following Python 3 code does
def _getScaledValue(self, inpt):
    if inpt == SENTINEL_VALUE_FOR_MISSING_DATA:
      return None
    else:
      val = inpt
      if val < self.minval:
        val = self.minval
      elif val > self.maxval:
        val = self.maxval

      scaledVal = math.log10(val)
      return scaledVal

Response: Convert the input which is in normal space into log10


Instruction:
Given the following Python 3 function, write the documentation
def getBucketIndices(self, inpt):

    # Get the scaled value
    scaledVal = self._getScaledValue(inpt)

    if scaledVal is None:
      return [None]
    else:
      return self.encoder.getBucketIndices(scaledVal)

Response: Returns the indices of the bucket in the input value.


Instruction:
Write a Python 3 function for
encoding the input into an array.

Response:
def encodeIntoArray(self, inpt, output):
    """
    See the function description in base.py
    """

    # Get the scaled value
    scaledVal = self._getScaledValue(inpt)

    if scaledVal is None:
      output[0:] = 0
    else:
      self.encoder.encodeIntoArray(scaledVal, output)

      if self.verbosity >= 2:
        print "input:", inpt, "scaledVal:", scaledVal, "output:", output
        print "decoded:", self.decodedToStr(self.decode(output))


Instruction:
How would you explain what the following Python 3 function does
def decode(self, encoded, parentFieldName=''):

    # Get the scalar values from the underlying scalar encoder
    (fieldsDict, fieldNames) = self.encoder.decode(encoded)
    if len(fieldsDict) == 0:
      return (fieldsDict, fieldNames)

    # Expect only 1 field
    assert(len(fieldsDict) == 1)

    # Convert each range into normal space
    (inRanges, inDesc) = fieldsDict.values()[0]
    outRanges = []
    for (minV, maxV) in inRanges:
      outRanges.append((math.pow(10, minV),
                        math.pow(10, maxV)))

    # Generate a text description of the ranges
    desc = ""
    numRanges = len(outRanges)
    for i in xrange(numRanges):
      if outRanges[i][0] != outRanges[i][1]:
        desc += "%.2f-%.2f" % (outRanges[i][0], outRanges[i][1])
      else:
        desc += "%.2f" % (outRanges[i][0])
      if i < numRanges-1:
        desc += ", "

    # Return result
    if parentFieldName != '':
      fieldName = "%s.%s" % (parentFieldName, self.name)
    else:
      fieldName = self.name
    return ({fieldName: (outRanges, desc)}, [fieldName])

Response: Decode the encoded string into a single object.


Instruction:
Write a Python 3 function for
returning a list of all the bucket values in the current language.

Response:
def getBucketValues(self):
    """
    See the function description in base.py
    """

    # Need to re-create?
    if self._bucketValues is None:
      scaledValues = self.encoder.getBucketValues()
      self._bucketValues = []
      for scaledValue in scaledValues:
        value = math.pow(10, scaledValue)
        self._bucketValues.append(value)

    return self._bucketValues


Instruction:
Write a Python 3 script for
returning encoderResult for the given buckets.

Response:
def getBucketInfo(self, buckets):
    """
    See the function description in base.py
    """

    scaledResult = self.encoder.getBucketInfo(buckets)[0]
    scaledValue = scaledResult.value
    value = math.pow(10, scaledValue)

    return [EncoderResult(value=value, scalar=value,
                         encoding = scaledResult.encoding)]


Instruction:
Implement a Python 3 function for
computing the top down encoding of the encoded sequence.

Response:
def topDownCompute(self, encoded):
    """
    See the function description in base.py
    """

    scaledResult = self.encoder.topDownCompute(encoded)[0]
    scaledValue = scaledResult.value
    value = math.pow(10, scaledValue)

    return EncoderResult(value=value, scalar=value,
                         encoding = scaledResult.encoding)


Instruction:
How would you explain what the following Python 3 function does
def closenessScores(self, expValues, actValues, fractional=True):

    # Compute the percent error in log space
    if expValues[0] > 0:
      expValue = math.log10(expValues[0])
    else:
      expValue = self.minScaledValue

    if actValues  [0] > 0:
      actValue = math.log10(actValues[0])
    else:
      actValue = self.minScaledValue

    if fractional:
      err = abs(expValue - actValue)
      pctErr = err / (self.maxScaledValue - self.minScaledValue)
      pctErr = min(1.0, pctErr)
      closeness = 1.0 - pctErr
    else:
      err = abs(expValue - actValue)
      closeness = err

    #print "log::", "expValue:", expValues[0], "actValue:", actValues[0], \
    #      "closeness", closeness
    #import pdb; pdb.set_trace()
    return numpy.array([closeness])

Response: Compute the closeness scores of the given values.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def export(self):
    graph = nx.MultiDiGraph()

    # Add regions to graph as nodes, annotated by name
    regions = self.network.getRegions()

    for idx in xrange(regions.getCount()):
      regionPair = regions.getByIndex(idx)
      regionName = regionPair[0]
      graph.add_node(regionName, label=regionName)

    # Add links between regions to graph as edges, annotate by input-output
    # name pairs
    for linkName, link in self.network.getLinks():
      graph.add_edge(link.getSrcRegionName(),
                     link.getDestRegionName(),
                     src=link.getSrcOutputName(),
                     dest=link.getDestInputName())

    return graph

Response: Exports a network as a networkx MultiDiGraph intermediate representation of the network.


Instruction:
Create a Python 3 function for
returning a string representing a numpy array of 0 s and 1 s

Response:
def bitsToString(arr):
  """Returns a string representing a numpy array of 0's and 1's"""
  s = array('c','.'*len(arr))
  for i in xrange(len(arr)):
    if arr[i] == 1:
      s[i]='*'
  return s


Instruction:
Write a Python 3 function that can
compute the percentage of overlap between vectors x1 and x2.

Response:
def percentOverlap(x1, x2, size):
  """
  Computes the percentage of overlap between vectors x1 and x2.

  @param x1   (array) binary vector
  @param x2   (array) binary vector
  @param size (int)   length of binary vectors

  @return percentOverlap (float) percentage overlap between x1 and x2
  """
  nonZeroX1 = np.count_nonzero(x1)
  nonZeroX2 = np.count_nonzero(x2)
  minX1X2 = min(nonZeroX1, nonZeroX2)
  percentOverlap = 0
  if minX1X2 > 0:
    percentOverlap = float(np.dot(x1, x2))/float(minX1X2)
  return percentOverlap


Instruction:
Write a Python 3 script for
resetting the contents of vector x1 into vector x2.

Response:
def resetVector(x1, x2):
  """
  Copies the contents of vector x1 into vector x2.

  @param x1 (array) binary vector to be copied
  @param x2 (array) binary vector where x1 is copied
  """
  size = len(x1)
  for i in range(size):
    x2[i] = x1[i]


Instruction:
How would you code a function in Python 3 to
poll CPU usage make predictions and plot the results. Runs forever.

Response:
def runCPU():
  """Poll CPU usage, make predictions, and plot the results. Runs forever."""
  # Create the model for predicting CPU usage.
  model = ModelFactory.create(model_params.MODEL_PARAMS)
  model.enableInference({'predictedField': 'cpu'})
  # The shifter will align prediction and actual values.
  shifter = InferenceShifter()
  # Keep the last WINDOW predicted and actual values for plotting.
  actHistory = deque([0.0] * WINDOW, maxlen=60)
  predHistory = deque([0.0] * WINDOW, maxlen=60)

  # Initialize the plot lines that we will update with each new record.
  actline, = plt.plot(range(WINDOW), actHistory)
  predline, = plt.plot(range(WINDOW), predHistory)
  # Set the y-axis range.
  actline.axes.set_ylim(0, 100)
  predline.axes.set_ylim(0, 100)

  while True:
    s = time.time()

    # Get the CPU usage.
    cpu = psutil.cpu_percent()

    # Run the input through the model and shift the resulting prediction.
    modelInput = {'cpu': cpu}
    result = shifter.shift(model.run(modelInput))

    # Update the trailing predicted and actual value deques.
    inference = result.inferences['multiStepBestPredictions'][5]
    if inference is not None:
      actHistory.append(result.rawInput['cpu'])
      predHistory.append(inference)

    # Redraw the chart with the new data.
    actline.set_ydata(actHistory)  # update the data
    predline.set_ydata(predHistory)  # update the data
    plt.draw()
    plt.legend( ('actual','predicted') )

    # Make sure we wait a total of 2 seconds per iteration.
    try:
      plt.pause(SECONDS_PER_STEP)
    except:
      pass


Instruction:
Create a Python 3 function for
returning a dictionary of the args dictionary from the calling method

Response:
def _extractCallingMethodArgs():
  """
  Returns args dictionary from the calling method
  """
  import inspect
  import copy

  callingFrame = inspect.stack()[1][0]

  argNames, _, _, frameLocalVarDict = inspect.getargvalues(callingFrame)

  argNames.remove("self")

  args = copy.copy(frameLocalVarDict)

  for varName in frameLocalVarDict:
    if varName not in argNames:
      args.pop(varName)

  return args


Instruction:
Can you create a Python 3 function that
populates serialization proto instance.

Response:
def write(self, proto):
    """Populate serialization proto instance.

    :param proto: (BacktrackingTMCppProto) the proto instance to populate
    """
    # Write base class to proto.baseTM (BacktrackingTMProto)
    super(BacktrackingTMCPP, self).write(proto.baseTM)
    self.cells4.write(proto.cells4)
    proto.makeCells4Ephemeral = self.makeCells4Ephemeral
    proto.seed = self.seed
    proto.checkSynapseConsistency = self.checkSynapseConsistency
    proto.initArgs = json.dumps(self._initArgsDict)


Instruction:
Can you tell what is the following Python 3 function doing
def read(cls, proto):
    # Use base class to create initial class from proto.baseTM
    # (BacktrackingTMProto)
    obj = BacktrackingTM.read(proto.baseTM)
    obj.__class__ = cls

    # Additional CPP-specific deserialization
    newCells4 = Cells4.read(proto.cells4)
    print newCells4
    obj.cells4 = newCells4
    obj.makeCells4Ephemeral = proto.makeCells4Ephemeral
    obj.seed = proto.seed
    obj.checkSynapseConsistency = proto.checkSynapseConsistency
    obj._initArgsDict = json.loads(proto.initArgs)
    # Convert unicode to str
    obj._initArgsDict["outputType"] = str(obj._initArgsDict["outputType"])

    # Initialize ephemeral attributes
    obj.allocateStatesInCPP = False
    obj.retrieveLearningStates = False
    obj._setStatePointers()

    return obj

Response: Deserialize from proto instance.


Instruction:
Can you generate the documentation for the following Python 3 function
def _getEphemeralMembers(self):
    e = BacktrackingTM._getEphemeralMembers(self)
    if self.makeCells4Ephemeral:
      e.extend(['cells4'])
    return e

Response: Returns a list of our member variables that we don t need to be saved


Instruction:
Can you generate the documentation for the following Python 3 function
def _initEphemerals(self):
    BacktrackingTM._initEphemerals(self)
    #---------------------------------------------------------------------------------
    # cells4 specific initialization

    # If True, let C++ allocate memory for activeState, predictedState, and
    # learnState. In this case we can retrieve copies of these states but can't
    # set them directly from Python. If False, Python can allocate them as
    # numpy arrays and we can pass pointers to the C++ using setStatePointers
    self.allocateStatesInCPP = False

    # Set this to true for debugging or accessing learning states
    self.retrieveLearningStates = False

    if self.makeCells4Ephemeral:
      self._initCells4()

Response: Initialize all ephemeral members after being restored to a pickled state.


Instruction:
Here you have a function in Python 3, explain what it does
def compute(self, bottomUpInput, enableLearn, enableInference=None):
    # The C++ TM takes 32 bit floats as input. uint32 works as well since the
    # code only checks whether elements are non-zero
    assert (bottomUpInput.dtype == numpy.dtype('float32')) or \
           (bottomUpInput.dtype == numpy.dtype('uint32')) or \
           (bottomUpInput.dtype == numpy.dtype('int32'))

    self.iterationIdx = self.iterationIdx + 1

    # As a speed optimization for now (until we need online learning), skip
    #  computing the inference output while learning
    if enableInference is None:
      if enableLearn:
        enableInference = False
      else:
        enableInference = True

    # ====================================================================
    # Run compute and retrieve selected state and member variables
    self._setStatePointers()
    y = self.cells4.compute(bottomUpInput, enableInference, enableLearn)
    self.currentOutput = y.reshape((self.numberOfCols, self.cellsPerColumn))
    self.avgLearnedSeqLength = self.cells4.getAvgLearnedSeqLength()
    self._copyAllocatedStates()

    # ========================================================================
    # Update the prediction score stats
    # Learning always includes inference
    if self.collectStats:
      activeColumns = bottomUpInput.nonzero()[0]
      if enableInference:
        predictedState = self.infPredictedState['t-1']
      else:
        predictedState = self.lrnPredictedState['t-1']
      self._updateStatsInferEnd(self._internalStats,
                                activeColumns,
                                predictedState,
                                self.colConfidence['t-1'])

    # Finally return the TM output
    output = self._computeOutput()

    # Print diagnostic information based on the current verbosity level
    self.printComputeEnd(output, learn=enableLearn)

    self.resetCalled = False
    return output

Response: Compute the current state of the C ++ TM.


Instruction:
Given the following Python 3 function, write the documentation
def _copyAllocatedStates(self):

    # Get learn states if we need to print them out
    if self.verbosity > 1 or self.retrieveLearningStates:
      (activeT, activeT1, predT, predT1) = self.cells4.getLearnStates()
      self.lrnActiveState['t-1'] = activeT1.reshape((self.numberOfCols, self.cellsPerColumn))
      self.lrnActiveState['t'] = activeT.reshape((self.numberOfCols, self.cellsPerColumn))
      self.lrnPredictedState['t-1'] = predT1.reshape((self.numberOfCols, self.cellsPerColumn))
      self.lrnPredictedState['t'] = predT.reshape((self.numberOfCols, self.cellsPerColumn))

    if self.allocateStatesInCPP:
      assert False
      (activeT, activeT1, predT, predT1, colConfidenceT, colConfidenceT1, confidenceT,
       confidenceT1) = self.cells4.getStates()
      self.cellConfidence['t'] = confidenceT.reshape((self.numberOfCols, self.cellsPerColumn))
      self.cellConfidence['t-1'] = confidenceT1.reshape((self.numberOfCols, self.cellsPerColumn))
      self.colConfidence['t'] = colConfidenceT.reshape(self.numberOfCols)
      self.colConfidence['t-1'] = colConfidenceT1.reshape(self.numberOfCols)
      self.infActiveState['t-1'] = activeT1.reshape((self.numberOfCols, self.cellsPerColumn))
      self.infActiveState['t'] = activeT.reshape((self.numberOfCols, self.cellsPerColumn))
      self.infPredictedState['t-1'] = predT1.reshape((self.numberOfCols, self.cellsPerColumn))
      self.infPredictedState['t'] = predT.reshape((self.numberOfCols, self.cellsPerColumn))

Response: Copy the allocated states into our numpy arrays.


Instruction:
Can you write a function in Python 3 where it
sets the state pointers for the cells4 object.

Response:
def _setStatePointers(self):
    """If we are having CPP use numpy-allocated buffers, set these buffer
    pointers. This is a relatively fast operation and, for safety, should be
    done before every call to the cells4 compute methods.  This protects us
    in situations where code can cause Python or numpy to create copies."""
    if not self.allocateStatesInCPP:
      self.cells4.setStatePointers(
          self.infActiveState["t"], self.infActiveState["t-1"],
          self.infPredictedState["t"], self.infPredictedState["t-1"],
          self.colConfidence["t"], self.colConfidence["t-1"],
          self.cellConfidence["t"], self.cellConfidence["t-1"])


Instruction:
Write a Python 3 script to
return a new sequence of cells with only the given synapses.

Response:
def trimSegments(self, minPermanence=None, minNumSyns=None):
    """
    Overrides :meth:`nupic.algorithms.backtracking_tm.BacktrackingTM.trimSegments`.
    """
    # Fill in defaults
    if minPermanence is None:
      minPermanence = 0.0
    if minNumSyns is None:
      minNumSyns = 0

    # Print all cells if verbosity says to
    if self.verbosity >= 5:
      print "Cells, all segments:"
      self.printCells(predictedOnly=False)

    return self.cells4.trimSegments(minPermanence=minPermanence, minNumSyns=minNumSyns)


Instruction:
Can you write a function in Python 3 where it
prints the segment updates.

Response:
def printSegmentUpdates(self):
    """
    Overrides :meth:`nupic.algorithms.backtracking_tm.BacktrackingTM.printSegmentUpdates`.
    """
    # TODO: need to add C++ accessors to implement this method
    assert False
    print "=== SEGMENT UPDATES ===, Num = ", len(self.segmentUpdates)
    for key, updateList in self.segmentUpdates.iteritems():
      c,i = key[0],key[1]
      print c,i,updateList


Instruction:
Make a summary of the following Python 3 code
def printCell(self, c, i, onlyActiveSegments=False):
    nSegs = self.cells4.nSegmentsOnCell(c,i)
    if nSegs > 0:
      segList = self.cells4.getNonEmptySegList(c,i)
      gidx = c * self.cellsPerColumn + i
      print "Column", c, "Cell", i, "(%d)"%(gidx),":", nSegs, "segment(s)"
      for k,segIdx in enumerate(segList):
        seg = self.cells4.getSegment(c, i, segIdx)
        isActive = self._slowIsSegmentActive(seg, 't')
        if onlyActiveSegments and not isActive:
          continue
        isActiveStr = "*" if isActive else " "
        print "  %sSeg #%-3d" % (isActiveStr, segIdx),
        print seg.size(),
        print seg.isSequenceSegment(), "%9.7f" % (seg.dutyCycle(
              self.cells4.getNLrnIterations(), False, True)),

        # numPositive/totalActivations
        print "(%4d/%-4d)" % (seg.getPositiveActivations(),
                           seg.getTotalActivations()),
        # Age
        print "%4d" % (self.cells4.getNLrnIterations()
                       - seg.getLastActiveIteration()),

        numSyn = seg.size()
        for s in xrange(numSyn):
          sc, si = self.getColCellIdx(seg.getSrcCellIdx(s))
          print "[%d,%d]%4.2f"%(sc, si, seg.getPermanence(s)),
        print

Response: Prints the contents of a single cell.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def getColCellIdx(self, idx):
    c = idx//self.cellsPerColumn
    i = idx - c*self.cellsPerColumn
    return c,i

Response: Get column and cell within column from a global index.


Instruction:
Write a Python 3 function for
returning a list of segments that are on the cell i.

Response:
def getSegmentOnCell(self, c, i, segIdx):
    """
    Overrides :meth:`nupic.algorithms.backtracking_tm.BacktrackingTM.getSegmentOnCell`.
    """
    segList = self.cells4.getNonEmptySegList(c,i)
    seg = self.cells4.getSegment(c, i, segList[segIdx])
    numSyn = seg.size()
    assert numSyn != 0

    # Accumulate segment information
    result = []
    result.append([int(segIdx), bool(seg.isSequenceSegment()),
                   seg.getPositiveActivations(),
                   seg.getTotalActivations(), seg.getLastActiveIteration(),
                   seg.getLastPosDutyCycle(),
                   seg.getLastPosDutyCycleIteration()])

    for s in xrange(numSyn):
      sc, si = self.getColCellIdx(seg.getSrcCellIdx(s))
      result.append([int(sc), int(si), seg.getPermanence(s)])

    return result


Instruction:
How would you implement a function in Python 3 that
returns a list of all the information for each segment in the current cell.

Response:
def getSegmentInfo(self, collectActiveData = False):
    """
    Overrides :meth:`nupic.algorithms.backtracking_tm.BacktrackingTM.getSegmentInfo`.
    """
    # Requires appropriate accessors in C++ cells4 (currently unimplemented)
    assert collectActiveData == False

    nSegments, nSynapses = self.getNumSegments(), self.cells4.nSynapses()
    distSegSizes, distNSegsPerCell = {}, {}
    nActiveSegs, nActiveSynapses = 0, 0
    distPermValues = {}   # Num synapses with given permanence values

    numAgeBuckets = 20
    distAges = []
    ageBucketSize = int((self.iterationIdx+20) / 20)
    for i in range(numAgeBuckets):
      distAges.append(['%d-%d' % (i*ageBucketSize, (i+1)*ageBucketSize-1), 0])

    for c in xrange(self.numberOfCols):
      for i in xrange(self.cellsPerColumn):

        # Update histogram counting cell sizes
        nSegmentsThisCell = self.getNumSegmentsInCell(c,i)
        if nSegmentsThisCell > 0:
          if distNSegsPerCell.has_key(nSegmentsThisCell):
            distNSegsPerCell[nSegmentsThisCell] += 1
          else:
            distNSegsPerCell[nSegmentsThisCell] = 1

          # Update histogram counting segment sizes.
          segList = self.cells4.getNonEmptySegList(c,i)
          for segIdx in xrange(nSegmentsThisCell):
            seg = self.getSegmentOnCell(c, i, segIdx)
            nSynapsesThisSeg = len(seg) - 1
            if nSynapsesThisSeg > 0:
              if distSegSizes.has_key(nSynapsesThisSeg):
                distSegSizes[nSynapsesThisSeg] += 1
              else:
                distSegSizes[nSynapsesThisSeg] = 1

              # Accumulate permanence value histogram
              for syn in seg[1:]:
                p = int(syn[2]*10)
                if distPermValues.has_key(p):
                  distPermValues[p] += 1
                else:
                  distPermValues[p] = 1

            segObj = self.cells4.getSegment(c, i, segList[segIdx])
            age = self.iterationIdx - segObj.getLastActiveIteration()
            ageBucket = int(age/ageBucketSize)
            distAges[ageBucket][1] += 1

    return (nSegments, nSynapses, nActiveSegs, nActiveSynapses, \
            distSegSizes, distNSegsPerCell, distPermValues, distAges)


Instruction:
Given the following Python 3 function, write the documentation
def main(argv):

  parser = OptionParser(helpString)

  parser.add_option("--jobID", action="store", type="int", default=None,
        help="jobID of the job within the dbTable [default: %default].")

  parser.add_option("--modelID", action="store", type="str", default=None,
        help=("Tell worker to re-run this model ID. When specified, jobID "
         "must also be specified [default: %default]."))

  parser.add_option("--workerID", action="store", type="str", default=None,
        help=("workerID of the scheduler's SlotAgent (GenericWorker) that "
          "hosts this SpecializedWorker [default: %default]."))

  parser.add_option("--params", action="store", default=None,
        help="Create and execute a new hypersearch request using this JSON " \
        "format params string. This is helpful for unit tests and debugging. " \
        "When specified jobID must NOT be specified. [default: %default].")

  parser.add_option("--clearModels", action="store_true", default=False,
        help="clear out the models table before starting [default: %default].")

  parser.add_option("--resetJobStatus", action="store_true", default=False,
        help="Reset the job status before starting  [default: %default].")

  parser.add_option("--logLevel", action="store", type="int", default=None,
        help="override default log level. Pass in an integer value that "
        "represents the desired logging level (10=logging.DEBUG, "
        "20=logging.INFO, etc.) [default: %default].")

  # Evaluate command line arguments
  (options, args) = parser.parse_args(argv[1:])
  if len(args) != 0:
    raise RuntimeError("Expected no command line arguments, but got: %s" % \
                        (args))

  if (options.jobID and options.params):
    raise RuntimeError("--jobID and --params can not be used at the same time")

  if (options.jobID is None and options.params is None):
    raise RuntimeError("Either --jobID or --params must be specified.")

  initLogging(verbose=True)

  # Instantiate the HypersearchWorker and run it
  hst = HypersearchWorker(options, argv[1:])

  # Normal use. This is one of among a number of workers. If we encounter
  #  an exception at the outer loop here, we fail the entire job.
  if options.params is None:
    try:
      jobID = hst.run()

    except Exception, e:
      jobID = options.jobID
      msg = StringIO.StringIO()
      print >>msg, "%s: Exception occurred in Hypersearch Worker: %r" % \
         (ErrorCodes.hypersearchLogicErr, e)
      traceback.print_exc(None, msg)

      completionReason = ClientJobsDAO.CMPL_REASON_ERROR
      completionMsg = msg.getvalue()
      hst.logger.error(completionMsg)

      # If no other worker already marked the job as failed, do so now.
      jobsDAO = ClientJobsDAO.get()
      workerCmpReason = jobsDAO.jobGetFields(options.jobID,
          ['workerCompletionReason'])[0]
      if workerCmpReason == ClientJobsDAO.CMPL_REASON_SUCCESS:
        jobsDAO.jobSetFields(options.jobID, fields=dict(
            cancel=True,
            workerCompletionReason = ClientJobsDAO.CMPL_REASON_ERROR,
            workerCompletionMsg = completionMsg),
            useConnectionID=False,
            ignoreUnchanged=True)

  # Run just 1 worker for the entire job. Used for unit tests that run in
  # 1 process
  else:
    jobID = None
    completionReason = ClientJobsDAO.CMPL_REASON_SUCCESS
    completionMsg = "Success"

    try:
      jobID = hst.run()
    except Exception, e:
      jobID = hst._options.jobID
      completionReason = ClientJobsDAO.CMPL_REASON_ERROR
      completionMsg = "ERROR: %s" % (e,)
      raise
    finally:
      if jobID is not None:
        cjDAO = ClientJobsDAO.get()
        cjDAO.jobSetCompleted(jobID=jobID,
                              completionReason=completionReason,
                              completionMsg=completionMsg)

  return jobID

Response:
This is the main function of the HypersearchWorker script. It is used by unit test code
                when calling this function.


Instruction:
Write a Python 3 script for
running this worker. Parameters: ---------------------------------------------------------------------- retval: jobID of the job we ran. This is used by unit test code when calling this working using the --params command line option (which tells this worker to insert the job itself).

Response:
def run(self):
    """ Run this worker.

    Parameters:
    ----------------------------------------------------------------------
    retval:     jobID of the job we ran. This is used by unit test code
                  when calling this working using the --params command
                  line option (which tells this worker to insert the job
                  itself).
    """
    # Easier access to options
    options = self._options

    # ---------------------------------------------------------------------
    # Connect to the jobs database
    self.logger.info("Connecting to the jobs database")
    cjDAO = ClientJobsDAO.get()

    # Get our worker ID
    self._workerID = cjDAO.getConnectionID()

    if options.clearModels:
      cjDAO.modelsClearAll()

    # -------------------------------------------------------------------------
    # if params were specified on the command line, insert a new job using
    #  them.
    if options.params is not None:
      options.jobID = cjDAO.jobInsert(client='hwTest', cmdLine="echo 'test mode'",
                  params=options.params, alreadyRunning=True,
                  minimumWorkers=1, maximumWorkers=1,
                  jobType = cjDAO.JOB_TYPE_HS)
    if options.workerID is not None:
      wID = options.workerID
    else:
      wID = self._workerID

    buildID = Configuration.get('nupic.software.buildNumber', 'N/A')
    logPrefix = '<BUILDID=%s, WORKER=HW, WRKID=%s, JOBID=%s> ' % \
                (buildID, wID, options.jobID)
    ExtendedLogger.setLogPrefix(logPrefix)

    # ---------------------------------------------------------------------
    # Get the search parameters
    # If asked to reset the job status, do that now
    if options.resetJobStatus:
      cjDAO.jobSetFields(options.jobID,
           fields={'workerCompletionReason': ClientJobsDAO.CMPL_REASON_SUCCESS,
                   'cancel': False,
                   #'engWorkerState': None
                   },
           useConnectionID=False,
           ignoreUnchanged=True)
    jobInfo = cjDAO.jobInfo(options.jobID)
    self.logger.info("Job info retrieved: %s" % (str(clippedObj(jobInfo))))

    # ---------------------------------------------------------------------
    # Instantiate the Hypersearch object, which will handle the logic of
    #  which models to create when we need more to evaluate.
    jobParams = json.loads(jobInfo.params)

    # Validate job params
    jsonSchemaPath = os.path.join(os.path.dirname(__file__),
                                  "jsonschema",
                                  "jobParamsSchema.json")
    validate(jobParams, schemaPath=jsonSchemaPath)

    hsVersion = jobParams.get('hsVersion', None)
    if hsVersion == 'v2':
      self._hs = HypersearchV2(searchParams=jobParams, workerID=self._workerID,
              cjDAO=cjDAO, jobID=options.jobID, logLevel=options.logLevel)
    else:
      raise RuntimeError("Invalid Hypersearch implementation (%s) specified" \
                          % (hsVersion))

    # =====================================================================
    # The main loop.
    try:
      exit = False
      numModelsTotal = 0
      print >>sys.stderr, "reporter:status:Evaluating first model..."
      while not exit:

        # ------------------------------------------------------------------
        # Choose a model to evaluate
        batchSize = 10              # How many to try at a time.
        modelIDToRun = None
        while modelIDToRun is None:

          if options.modelID is None:
            # -----------------------------------------------------------------
            # Get the latest results on all running models and send them to
            #  the Hypersearch implementation
            # This calls cjDAO.modelsGetUpdateCounters(), compares the
            # updateCounters with what we have cached, fetches the results for the
            # changed and new models, and sends those to the Hypersearch
            # implementation's self._hs.recordModelProgress() method.
            self._processUpdatedModels(cjDAO)

            # --------------------------------------------------------------------
            # Create a new batch of models
            (exit, newModels) = self._hs.createModels(numModels = batchSize)
            if exit:
              break

            # No more models left to create, just loop. The _hs is waiting for
            #   all remaining running models to complete, and may pick up on an
            #  orphan if it detects one.
            if len(newModels) == 0:
              continue

            # Try and insert one that we will run
            for (modelParams, modelParamsHash, particleHash) in newModels:
              jsonModelParams = json.dumps(modelParams)
              (modelID, ours) = cjDAO.modelInsertAndStart(options.jobID,
                                  jsonModelParams, modelParamsHash, particleHash)

              # Some other worker is already running it, tell the Hypersearch object
              #  so that it doesn't try and insert it again
              if not ours:
                mParamsAndHash = cjDAO.modelsGetParams([modelID])[0]
                mResult = cjDAO.modelsGetResultAndStatus([modelID])[0]
                results = mResult.results
                if results is not None:
                  results = json.loads(results)

                modelParams = json.loads(mParamsAndHash.params)
                particleHash = cjDAO.modelsGetFields(modelID,
                                  ['engParticleHash'])[0]
                particleInst = "%s.%s" % (
                          modelParams['particleState']['id'],
                          modelParams['particleState']['genIdx'])
                self.logger.info("Adding model %d to our internal DB " \
                      "because modelInsertAndStart() failed to insert it: " \
                      "paramsHash=%s, particleHash=%s, particleId='%s'", modelID,
                      mParamsAndHash.engParamsHash.encode('hex'),
                      particleHash.encode('hex'), particleInst)
                self._hs.recordModelProgress(modelID = modelID,
                      modelParams = modelParams,
                      modelParamsHash = mParamsAndHash.engParamsHash,
                      results = results,
                      completed = (mResult.status == cjDAO.STATUS_COMPLETED),
                      completionReason = mResult.completionReason,
                      matured = mResult.engMatured,
                      numRecords = mResult.numRecords)
              else:
                modelIDToRun = modelID
                break

          else:
            # A specific modelID was passed on the command line
            modelIDToRun = int(options.modelID)
            mParamsAndHash = cjDAO.modelsGetParams([modelIDToRun])[0]
            modelParams = json.loads(mParamsAndHash.params)
            modelParamsHash = mParamsAndHash.engParamsHash

            # Make us the worker
            cjDAO.modelSetFields(modelIDToRun,
                                     dict(engWorkerConnId=self._workerID))
            if False:
              # Change the hash and params of the old entry so that we can
              #  create a new model with the same params
              for attempt in range(1000):
                paramsHash = hashlib.md5("OrphanParams.%d.%d" % (modelIDToRun,
                                                                 attempt)).digest()
                particleHash = hashlib.md5("OrphanParticle.%d.%d" % (modelIDToRun,
                                                                  attempt)).digest()
                try:
                  cjDAO.modelSetFields(modelIDToRun,
                                           dict(engParamsHash=paramsHash,
                                                engParticleHash=particleHash))
                  success = True
                except:
                  success = False
                if success:
                  break
              if not success:
                raise RuntimeError("Unexpected failure to change paramsHash and "
                                   "particleHash of orphaned model")

              (modelIDToRun, ours) = cjDAO.modelInsertAndStart(options.jobID,
                                  mParamsAndHash.params, modelParamsHash)

            # ^^^ end while modelIDToRun ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

        # ---------------------------------------------------------------
        # We have a model, evaluate it now
        # All done?
        if exit:
          break

        # Run the model now
        self.logger.info("RUNNING MODEL GID=%d, paramsHash=%s, params=%s",
              modelIDToRun, modelParamsHash.encode('hex'), modelParams)

        # ---------------------------------------------------------------------
        # Construct model checkpoint GUID for this model:
        # jobParams['persistentJobGUID'] contains the client's (e.g., API Server)
        # persistent, globally-unique model identifier, which is what we need;
        persistentJobGUID = jobParams['persistentJobGUID']
        assert persistentJobGUID, "persistentJobGUID: %r" % (persistentJobGUID,)

        modelCheckpointGUID = jobInfo.client + "_" + persistentJobGUID + (
          '_' + str(modelIDToRun))

        self._hs.runModel(modelID=modelIDToRun, jobID = options.jobID,
                          modelParams=modelParams, modelParamsHash=modelParamsHash,
                          jobsDAO=cjDAO, modelCheckpointGUID=modelCheckpointGUID)

        # TODO: don't increment for orphaned models
        numModelsTotal += 1

        self.logger.info("COMPLETED MODEL GID=%d; EVALUATED %d MODELs",
          modelIDToRun, numModelsTotal)
        print >>sys.stderr, "reporter:status:Evaluated %d models..." % \
                                    (numModelsTotal)
        print >>sys.stderr, "reporter:counter:HypersearchWorker,numModels,1"

        if options.modelID is not None:
          exit = True
        # ^^^ end while not exit

    finally:
      # Provide Hypersearch instance an opportunity to clean up temporary files
      self._hs.close()

    self.logger.info("FINISHED. Evaluated %d models." % (numModelsTotal))
    print >>sys.stderr, "reporter:status:Finished, evaluated %d models" % (numModelsTotal)
    return options.jobID


Instruction:
Can you generate the documentation for the following Python 3 function
def getBucketIndices(self, x):

    if ((isinstance(x, float) and math.isnan(x)) or
        x == SENTINEL_VALUE_FOR_MISSING_DATA):
      return [None]

    if self._offset is None:
      self._offset = x

    bucketIdx = (
        (self._maxBuckets/2) + int(round((x - self._offset) / self.resolution))
    )

    if bucketIdx < 0:
      bucketIdx = 0
    elif bucketIdx >= self._maxBuckets:
      bucketIdx = self._maxBuckets-1

    return [bucketIdx]

Response: Returns the indices of the bucket that contains x.


Instruction:
Explain what the following Python 3 code does
def mapBucketIndexToNonZeroBits(self, index):
    if index < 0:
      index = 0

    if index >= self._maxBuckets:
      index = self._maxBuckets-1

    if not self.bucketMap.has_key(index):
      if self.verbosity >= 2:
        print "Adding additional buckets to handle index=", index
      self._createBucket(index)
    return self.bucketMap[index]

Response: Given a bucket index return the list of non - zero bits.


Instruction:
How would you code a function in Python 3 to
encode x into a base. py array.

Response:
def encodeIntoArray(self, x, output):
    """ See method description in base.py """

    if x is not None and not isinstance(x, numbers.Number):
      raise TypeError(
          "Expected a scalar input but got input of type %s" % type(x))

    # Get the bucket index to use
    bucketIdx = self.getBucketIndices(x)[0]

    # None is returned for missing value in which case we return all 0's.
    output[0:self.n] = 0
    if bucketIdx is not None:
      output[self.mapBucketIndexToNonZeroBits(bucketIdx)] = 1


Instruction:
Explain what the following Python 3 code does
def _createBucket(self, index):
    if index < self.minIndex:
      if index == self.minIndex - 1:
        # Create a new representation that has exactly w-1 overlapping bits
        # as the min representation
        self.bucketMap[index] = self._newRepresentation(self.minIndex,
                                                        index)
        self.minIndex = index
      else:
        # Recursively create all the indices above and then this index
        self._createBucket(index+1)
        self._createBucket(index)
    else:
      if index == self.maxIndex + 1:
        # Create a new representation that has exactly w-1 overlapping bits
        # as the max representation
        self.bucketMap[index] = self._newRepresentation(self.maxIndex,
                                                        index)
        self.maxIndex = index
      else:
        # Recursively create all the indices below and then this index
        self._createBucket(index-1)
        self._createBucket(index)

Response:
Create the given bucket index. Recursively create as many in - between
    bucket indices as necessary.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _newRepresentation(self, index, newIndex):
    newRepresentation = self.bucketMap[index].copy()

    # Choose the bit we will replace in this representation. We need to shift
    # this bit deterministically. If this is always chosen randomly then there
    # is a 1 in w chance of the same bit being replaced in neighboring
    # representations, which is fairly high
    ri = newIndex % self.w

    # Now we choose a bit such that the overlap rules are satisfied.
    newBit = self.random.getUInt32(self.n)
    newRepresentation[ri] = newBit
    while newBit in self.bucketMap[index] or \
          not self._newRepresentationOK(newRepresentation, newIndex):
      self.numTries += 1
      newBit = self.random.getUInt32(self.n)
      newRepresentation[ri] = newBit

    return newRepresentation

Response:
Return a new representation for newIndex that overlaps with the
    representation at index.


Instruction:
Can you create a Python 3 function that
returns True if this new candidate representation satisfies all our overlap rules.

Response:
def _newRepresentationOK(self, newRep, newIndex):
    """
    Return True if this new candidate representation satisfies all our overlap
    rules. Since we know that neighboring representations differ by at most
    one bit, we compute running overlaps.
    """
    if newRep.size != self.w:
      return False
    if (newIndex < self.minIndex-1) or (newIndex > self.maxIndex+1):
      raise ValueError("newIndex must be within one of existing indices")

    # A binary representation of newRep. We will use this to test containment
    newRepBinary = numpy.array([False]*self.n)
    newRepBinary[newRep] = True

    # Midpoint
    midIdx = self._maxBuckets/2

    # Start by checking the overlap at minIndex
    runningOverlap = self._countOverlap(self.bucketMap[self.minIndex], newRep)
    if not self._overlapOK(self.minIndex, newIndex, overlap=runningOverlap):
      return False

    # Compute running overlaps all the way to the midpoint
    for i in range(self.minIndex+1, midIdx+1):
      # This is the bit that is going to change
      newBit = (i-1)%self.w

      # Update our running overlap
      if newRepBinary[self.bucketMap[i-1][newBit]]:
        runningOverlap -= 1
      if newRepBinary[self.bucketMap[i][newBit]]:
        runningOverlap += 1

      # Verify our rules
      if not self._overlapOK(i, newIndex, overlap=runningOverlap):
        return False

    # At this point, runningOverlap contains the overlap for midIdx
    # Compute running overlaps all the way to maxIndex
    for i in range(midIdx+1, self.maxIndex+1):
      # This is the bit that is going to change
      newBit = i%self.w

      # Update our running overlap
      if newRepBinary[self.bucketMap[i-1][newBit]]:
        runningOverlap -= 1
      if newRepBinary[self.bucketMap[i][newBit]]:
        runningOverlap += 1

      # Verify our rules
      if not self._overlapOK(i, newIndex, overlap=runningOverlap):
        return False

    return True


Instruction:
Create a Python 3 function for
returning the number of overlap between bucket indices i and j

Response:
def _countOverlapIndices(self, i, j):
    """
    Return the overlap between bucket indices i and j
    """
    if self.bucketMap.has_key(i) and self.bucketMap.has_key(j):
      iRep = self.bucketMap[i]
      jRep = self.bucketMap[j]
      return self._countOverlap(iRep, jRep)
    else:
      raise ValueError("Either i or j don't exist")


Instruction:
Can you write a function in Python 3 where it
counts the number of overlap between two representations.

Response:
def _countOverlap(rep1, rep2):
    """
    Return the overlap between two representations. rep1 and rep2 are lists of
    non-zero indices.
    """
    overlap = 0
    for e in rep1:
      if e in rep2:
        overlap += 1
    return overlap


Instruction:
Here you have a function in Python 3, explain what it does
def _overlapOK(self, i, j, overlap=None):
    if overlap is None:
      overlap = self._countOverlapIndices(i, j)
    if abs(i-j) < self.w:
      if overlap == (self.w - abs(i-j)):
        return True
      else:
        return False
    else:
      if overlap <= self._maxOverlap:
        return True
      else:
        return False

Response: Returns True if the given overlap between bucket indices i and j are acceptable.


Instruction:
Write a Python 3 script to
initialize the bucket map assuming the given number of maxBuckets.

Response:
def _initializeBucketMap(self, maxBuckets, offset):
    """
    Initialize the bucket map assuming the given number of maxBuckets.
    """
    # The first bucket index will be _maxBuckets / 2 and bucket indices will be
    # allowed to grow lower or higher as long as they don't become negative.
    # _maxBuckets is required because the current SDR Classifier assumes bucket
    # indices must be non-negative. This normally does not need to be changed
    # but if altered, should be set to an even number.
    self._maxBuckets = maxBuckets
    self.minIndex = self._maxBuckets / 2
    self.maxIndex = self._maxBuckets / 2

    # The scalar offset used to map scalar values to bucket indices. The middle
    # bucket will correspond to numbers in the range
    # [offset-resolution/2, offset+resolution/2).
    # The bucket index for a number x will be:
    #     maxBuckets/2 + int( round( (x-offset)/resolution ) )
    self._offset = offset

    # This dictionary maps a bucket index into its bit representation
    # We initialize the class with a single bucket with index 0
    self.bucketMap = {}

    def _permutation(n):
      r = numpy.arange(n, dtype=numpy.uint32)
      self.random.shuffle(r)
      return r

    self.bucketMap[self.minIndex] = _permutation(self.n)[0:self.w]

    # How often we need to retry when generating valid encodings
    self.numTries = 0


Instruction:
How would you code a function in Python 3 to
return a decorator that returns a closure suitable for use as a decorator for retrySQL.

Response:
def retrySQL(timeoutSec=60*5, logger=None):
  """ Return a closure suitable for use as a decorator for
  retrying a pymysql DAO function on certain failures that warrant retries (
  e.g., RDS/MySQL server down temporarily, transaction deadlock, etc.).
  We share this function across multiple scripts (e.g., ClientJobsDAO,
  StreamMgr) for consitent behavior.

  .. note:: Please ensure that the operation being retried is idempotent.

  .. note:: logging must be initialized *before* any loggers are created, else
     there will be no output; see nupic.support.initLogging()

  Usage Example:

  .. code-block:: python

    @retrySQL()
    def jobInfo(self, jobID):
        ...

  :param timeoutSec:       How many seconds from time of initial call to stop retrying
                     (floating point)
  :param logger:           User-supplied logger instance.

  """

  if logger is None:
    logger = logging.getLogger(__name__)

  def retryFilter(e, args, kwargs):

    if isinstance(e, (pymysql.InternalError, pymysql.OperationalError)):
      if e.args and e.args[0] in _ALL_RETRIABLE_ERROR_CODES:
        return True

    elif isinstance(e, pymysql.Error):
      if (e.args and
          inspect.isclass(e.args[0]) and issubclass(e.args[0], socket_error)):
        return True

    return False

  retryExceptions = tuple([
    pymysql.InternalError,
    pymysql.OperationalError,
    pymysql.Error,
  ])

  return make_retry_decorator(
    timeoutSec=timeoutSec, initialRetryDelaySec=0.1, maxRetryDelaySec=10,
    retryExceptions=retryExceptions, retryFilter=retryFilter,
    logger=logger)


Instruction:
Can you generate the documentation for the following Python 3 function
def create(*args, **kwargs):
    impl = kwargs.pop('implementation', None)
    if impl is None:
      impl = Configuration.get('nupic.opf.sdrClassifier.implementation')
    if impl == 'py':
      return SDRClassifier(*args, **kwargs)
    elif impl == 'cpp':
      return FastSDRClassifier(*args, **kwargs)
    elif impl == 'diff':
      return SDRClassifierDiff(*args, **kwargs)
    else:
      raise ValueError('Invalid classifier implementation (%r). Value must be '
                       '"py", "cpp" or "diff".' % impl)

Response: Create a new SDR classifier factory.


Instruction:
How would you code a function in Python 3 to
read the object containing the next set of elements in the sequence.

Response:
def read(proto):
    """
    :param proto: SDRClassifierRegionProto capnproto object
    """
    impl = proto.implementation
    if impl == 'py':
      return SDRClassifier.read(proto.sdrClassifier)
    elif impl == 'cpp':
      return FastSDRClassifier.read(proto.sdrClassifier)
    elif impl == 'diff':
      return SDRClassifierDiff.read(proto.sdrClassifier)
    else:
      raise ValueError('Invalid classifier implementation (%r). Value must be '
                       '"py", "cpp" or "diff".' % impl)


Instruction:
Write a Python 3 function that can
cross list of sequences.

Response:
def cross_list(*sequences):
  """
  From: http://book.opensourceproject.org.cn/lamp/python/pythoncook2/opensource/0596007973/pythoncook2-chp-19-sect-9.html
  """
  result = [[ ]]
  for seq in sequences:
    result = [sublist+[item] for sublist in result for item in seq]
  return result


Instruction:
Write a Python 3 function for
crossing - product the given sequence.

Response:
def cross(*sequences):
  """
  From: http://book.opensourceproject.org.cn/lamp/python/pythoncook2/opensource/0596007973/pythoncook2-chp-19-sect-9.html
  """
  # visualize an odometer, with "wheels" displaying "digits"...:
  wheels = map(iter, sequences)
  digits = [it.next( ) for it in wheels]
  while True:
    yield tuple(digits)
    for i in range(len(digits)-1, -1, -1):
      try:
        digits[i] = wheels[i].next( )
        break
      except StopIteration:
        wheels[i] = iter(sequences[i])
        digits[i] = wheels[i].next( )
    else:
      break


Instruction:
Here you have a function in Python 3, explain what it does
def dcross(**keywords):
  keys = keywords.keys()
  # Could use keywords.values(), but unsure whether the order
  # the values come out in is guaranteed to be the same as that of keys
  # (appears to be anecdotally true).
  sequences = [keywords[key] for key in keys]

  wheels = map(iter, sequences)
  digits = [it.next( ) for it in wheels]
  while True:
    yield dict(zip(keys, digits))
    for i in range(len(digits)-1, -1, -1):
      try:
        digits[i] = wheels[i].next( )
        break
      except StopIteration:
        wheels[i] = iter(sequences[i])
        digits[i] = wheels[i].next( )
    else:
      break

Response: Similar to cross but generates output dictionaries instead of tuples.


Instruction:
Make a summary of the following Python 3 code
def mmGetMetricFromTrace(self, trace):
    return Metric.createFromTrace(trace.makeCountsTrace(),
                                  excludeResets=self.mmGetTraceResets())

Response: Returns a metric over an indices trace excluding resets.


Instruction:
How would you implement a function in Python 3 that
returns a metric that counts the number of predicted active cells per column for each sequence.

Response:
def mmGetMetricSequencesPredictedActiveCellsPerColumn(self):
    """
    Metric for number of predicted => active cells per column for each sequence

    @return (Metric) metric
    """
    self._mmComputeTransitionTraces()

    numCellsPerColumn = []

    for predictedActiveCells in (
        self._mmData["predictedActiveCellsForSequence"].values()):
      cellsForColumn = self.mapCellsToColumns(predictedActiveCells)
      numCellsPerColumn += [len(x) for x in cellsForColumn.values()]

    return Metric(self,
                  "# predicted => active cells per column for each sequence",
                  numCellsPerColumn)


Instruction:
Write a Python 3 function for
returning a metric that counts how many sequences each predicted active cell appears in.

Response:
def mmGetMetricSequencesPredictedActiveCellsShared(self):
    """
    Metric for number of sequences each predicted => active cell appears in

    Note: This metric is flawed when it comes to high-order sequences.

    @return (Metric) metric
    """
    self._mmComputeTransitionTraces()

    numSequencesForCell = defaultdict(lambda: 0)

    for predictedActiveCells in (
          self._mmData["predictedActiveCellsForSequence"].values()):
      for cell in predictedActiveCells:
        numSequencesForCell[cell] += 1

    return Metric(self,
                  "# sequences each predicted => active cells appears in",
                  numSequencesForCell.values())


Instruction:
Explain what the following Python 3 code does
def mmPrettyPrintConnections(self):
    text = ""

    text += ("Segments: (format => "
             "(#) [(source cell=permanence ...),       ...]\n")
    text += "------------------------------------\n"

    columns = range(self.numberOfColumns())

    for column in columns:
      cells = self.cellsForColumn(column)

      for cell in cells:
        segmentDict = dict()

        for seg in self.connections.segmentsForCell(cell):
          synapseList = []

          for synapse in self.connections.synapsesForSegment(seg):
            synapseData = self.connections.dataForSynapse(synapse)
            synapseList.append(
                (synapseData.presynapticCell, synapseData.permanence))

          synapseList.sort()
          synapseStringList = ["{0:3}={1:.2f}".format(sourceCell, permanence) for
                               sourceCell, permanence in synapseList]
          segmentDict[seg] = "({0})".format(" ".join(synapseStringList))

        text += ("Column {0:3} / Cell {1:3}:\t({2}) {3}\n".format(
          column, cell,
          len(segmentDict.values()),
          "[{0}]".format(",       ".join(segmentDict.values()))))

      if column < len(columns) - 1:  # not last
        text += "\n"

    text += "------------------------------------\n"

    return text

Response: Pretty print the connections in the temporal memory.


Instruction:
Here you have a function in Python 3, explain what it does
def createTemporalAnomaly(recordParams, spatialParams=_SP_PARAMS,
                          temporalParams=_TM_PARAMS,
                          verbosity=_VERBOSITY):

  """Generates a Network with connected RecordSensor, SP, TM.

  This function takes care of generating regions and the canonical links.
  The network has a sensor region reading data from a specified input and
  passing the encoded representation to an SPRegion.
  The SPRegion output is passed to a TMRegion.

  Note: this function returns a network that needs to be initialized. This
  allows the user to extend the network by adding further regions and
  connections.

  :param recordParams: a dict with parameters for creating RecordSensor region.
  :param spatialParams: a dict with parameters for creating SPRegion.
  :param temporalParams: a dict with parameters for creating TMRegion.
  :param verbosity: an integer representing how chatty the network will be.
  """
  inputFilePath = recordParams["inputFilePath"]
  scalarEncoderArgs = recordParams["scalarEncoderArgs"]
  dateEncoderArgs = recordParams["dateEncoderArgs"]

  scalarEncoder = ScalarEncoder(**scalarEncoderArgs)
  dateEncoder = DateEncoder(**dateEncoderArgs)

  encoder = MultiEncoder()
  encoder.addEncoder(scalarEncoderArgs["name"], scalarEncoder)
  encoder.addEncoder(dateEncoderArgs["name"], dateEncoder)

  network = Network()

  network.addRegion("sensor", "py.RecordSensor",
                    json.dumps({"verbosity": verbosity}))

  sensor = network.regions["sensor"].getSelf()
  sensor.encoder = encoder
  sensor.dataSource = FileRecordStream(streamID=inputFilePath)

  # Create the spatial pooler region
  spatialParams["inputWidth"] = sensor.encoder.getWidth()
  network.addRegion("spatialPoolerRegion", "py.SPRegion",
                    json.dumps(spatialParams))

  # Link the SP region to the sensor input
  network.link("sensor", "spatialPoolerRegion", "UniformLink", "")
  network.link("sensor", "spatialPoolerRegion", "UniformLink", "",
               srcOutput="resetOut", destInput="resetIn")
  network.link("spatialPoolerRegion", "sensor", "UniformLink", "",
               srcOutput="spatialTopDownOut", destInput="spatialTopDownIn")
  network.link("spatialPoolerRegion", "sensor", "UniformLink", "",
               srcOutput="temporalTopDownOut", destInput="temporalTopDownIn")

  # Add the TPRegion on top of the SPRegion
  network.addRegion("temporalPoolerRegion", "py.TMRegion",
                    json.dumps(temporalParams))

  network.link("spatialPoolerRegion", "temporalPoolerRegion", "UniformLink", "")
  network.link("temporalPoolerRegion", "spatialPoolerRegion", "UniformLink", "",
               srcOutput="topDownOut", destInput="topDownIn")

  spatialPoolerRegion = network.regions["spatialPoolerRegion"]

  # Make sure learning is enabled
  spatialPoolerRegion.setParameter("learningMode", True)
  # We want temporal anomalies so disable anomalyMode in the SP. This mode is
  # used for computing anomalies in a non-temporal model.
  spatialPoolerRegion.setParameter("anomalyMode", False)

  temporalPoolerRegion = network.regions["temporalPoolerRegion"]

  # Enable topDownMode to get the predicted columns output
  temporalPoolerRegion.setParameter("topDownMode", True)
  # Make sure learning is enabled (this is the default)
  temporalPoolerRegion.setParameter("learningMode", True)
  # Enable inference mode so we get predictions
  temporalPoolerRegion.setParameter("inferenceMode", True)
  # Enable anomalyMode to compute the anomaly score.
  temporalPoolerRegion.setParameter("anomalyMode", True)

  return network

Response: This function creates a Network with connected RecordSensor and a temporal anomaly.


Instruction:
Can you write a function in Python 3 where it
runs the network and write output to writer. Run the network and write output to writer.

Response:
def runNetwork(network, writer):
  """Run the network and write output to writer.

  :param network: a Network instance to run
  :param writer: a csv.writer instance to write output to
  """
  sensorRegion = network.regions["sensor"]
  temporalPoolerRegion = network.regions["temporalPoolerRegion"]

  for i in xrange(_NUM_RECORDS):
    # Run the network for a single iteration
    network.run(1)

    # Write out the anomaly score along with the record number and consumption
    # value.
    anomalyScore = temporalPoolerRegion.getOutputData("anomalyScore")[0]
    consumption = sensorRegion.getOutputData("sourceOut")[0]
    writer.writerow((i, consumption, anomalyScore))


Instruction:
Can you implement a function in Python 3 that
appends the given list of PeriodicActivityRequest elements to the internal list of Activations.

Response:
def __appendActivities(self, periodicActivities):
    """
    periodicActivities: A sequence of PeriodicActivityRequest elements
    """

    for req in periodicActivities:
      act =   self.Activity(repeating=req.repeating,
                            period=req.period,
                            cb=req.cb,
                            iteratorHolder=[iter(xrange(req.period-1))])
      self.__activities.append(act)

    return


Instruction:
Implement a function in Python 3 to
add a value over a range of rows.

Response:
def add(reader, writer, column, start, stop, value):
  """Adds a value over a range of rows.

  Args:
    reader: A FileRecordStream object with input data.
    writer: A FileRecordStream object to write output data to.
    column: The column of data to modify.
    start: The first row in the range to modify.
    end: The last row in the range to modify.
    value: The value to add.
  """
  for i, row in enumerate(reader):
    if i >= start and i <= stop:
      row[column] = type(value)(row[column]) + value
    writer.appendRecord(row)


Instruction:
Make a summary of the following Python 3 code
def scale(reader, writer, column, start, stop, multiple):
  for i, row in enumerate(reader):
    if i >= start and i <= stop:
      row[column] = type(multiple)(row[column]) * multiple
    writer.appendRecord(row)

Response: Multiply a value over a range of rows.


Instruction:
How would you implement a function in Python 3 that
copies a range of values to a new location in the data set.

Response:
def copy(reader, writer, start, stop, insertLocation=None, tsCol=None):
  """Copies a range of values to a new location in the data set.

  Args:
    reader: A FileRecordStream object with input data.
    writer: A FileRecordStream object to write output data to.
    start: The first row in the range to copy.
    stop: The last row in the range to copy.
    insertLocation: The location to insert the copied range. If not specified,
        the range is inserted immediately following itself.
  """
  assert stop >= start
  startRows = []
  copyRows = []
  ts = None
  inc = None
  if tsCol is None:
    tsCol = reader.getTimestampFieldIdx()
  for i, row in enumerate(reader):
    # Get the first timestamp and the increment.
    if ts is None:
      ts = row[tsCol]
    elif inc is None:
      inc = row[tsCol] - ts
    # Keep a list of all rows and a list of rows to copy.
    if i >= start and i <= stop:
      copyRows.append(row)
    startRows.append(row)
  # Insert the copied rows.
  if insertLocation is None:
    insertLocation = stop + 1
  startRows[insertLocation:insertLocation] = copyRows
  # Update the timestamps.
  for row in startRows:
    row[tsCol] = ts
    writer.appendRecord(row)
    ts += inc


Instruction:
Write a Python 3 script for
sampling n rows from the specified range.

Response:
def sample(reader, writer, n, start=None, stop=None, tsCol=None,
           writeSampleOnly=True):
  """Samples n rows.

  Args:
    reader: A FileRecordStream object with input data.
    writer: A FileRecordStream object to write output data to.
    n: The number of elements to sample.
    start: The first row in the range to sample from.
    stop: The last row in the range to sample from.
    tsCol: If specified, the timestamp column to update.
    writeSampleOnly: If False, the rows before start are written before the
        sample and the rows after stop are written after the sample.
  """
  rows = list(reader)
  if tsCol is not None:
    ts = rows[0][tsCol]
    inc = rows[1][tsCol] - ts
  if start is None:
    start = 0
  if stop is None:
    stop = len(rows) - 1
  initialN = stop - start + 1
  # Select random rows in the sample range to delete until the desired number
  # of rows are left.
  numDeletes =  initialN - n
  for i in xrange(numDeletes):
    delIndex = random.randint(start, stop - i)
    del rows[delIndex]
  # Remove outside rows if specified.
  if writeSampleOnly:
    rows = rows[start:start + n]
  # Rewrite columns if tsCol is given.
  if tsCol is not None:
    ts = rows[0][tsCol]
  # Write resulting rows.
  for row in rows:
    if tsCol is not None:
      row[tsCol] = ts
      ts += inc
    writer.appendRecord(row)


Instruction:
How would you code a function in Python 3 to
initialize the encoder for the current version of the class.

Response:
def _initEncoder(self, w, minval, maxval, n, radius, resolution):
    """ (helper function)  There are three different ways of thinking about the representation.
     Handle each case here."""
    if n != 0:
      if (radius !=0 or resolution != 0):
        raise ValueError("Only one of n/radius/resolution can be specified for a ScalarEncoder")
      assert n > w
      self.n = n

      if (minval is not None and maxval is not None):
        if not self.periodic:
          self.resolution = float(self.rangeInternal) / (self.n - self.w)
        else:
          self.resolution = float(self.rangeInternal) / (self.n)

        self.radius = self.w * self.resolution

        if self.periodic:
          self.range = self.rangeInternal
        else:
          self.range = self.rangeInternal + self.resolution

    else:
      if radius != 0:
        if (resolution != 0):
          raise ValueError("Only one of radius/resolution can be specified for a ScalarEncoder")
        self.radius = radius
        self.resolution = float(self.radius) / w
      elif resolution != 0:
        self.resolution = float(resolution)
        self.radius = self.resolution * self.w
      else:
        raise Exception("One of n, radius, resolution must be specified for a ScalarEncoder")

      if (minval is not None and maxval is not None):
        if self.periodic:
          self.range = self.rangeInternal
        else:
          self.range = self.rangeInternal + self.resolution

        nfloat = self.w * (self.range / self.radius) + 2 * self.padding
        self.n = int(math.ceil(nfloat))


Instruction:
Can you create a Python 3 function that
returns the bit offset of the first bit to be set in the encoder output. For periodic encoders, this can be a negative number when the encoded output wraps around.

Response:
def _getFirstOnBit(self, input):
    """ Return the bit offset of the first bit to be set in the encoder output.
    For periodic encoders, this can be a negative number when the encoded output
    wraps around. """

    if input == SENTINEL_VALUE_FOR_MISSING_DATA:
      return [None]

    else:
      if input < self.minval:
        # Don't clip periodic inputs. Out-of-range input is always an error
        if self.clipInput and not self.periodic:
          if self.verbosity > 0:
            print "Clipped input %s=%.2f to minval %.2f" % (self.name, input,
                                                            self.minval)
          input = self.minval
        else:
          raise Exception('input (%s) less than range (%s - %s)' %
                          (str(input), str(self.minval), str(self.maxval)))

      if self.periodic:
        # Don't clip periodic inputs. Out-of-range input is always an error
        if input >= self.maxval:
          raise Exception('input (%s) greater than periodic range (%s - %s)' %
                          (str(input), str(self.minval), str(self.maxval)))
      else:
        if input > self.maxval:
          if self.clipInput:
            if self.verbosity > 0:
              print "Clipped input %s=%.2f to maxval %.2f" % (self.name, input,
                                                              self.maxval)
            input = self.maxval
          else:
            raise Exception('input (%s) greater than range (%s - %s)' %
                            (str(input), str(self.minval), str(self.maxval)))

      if self.periodic:
        centerbin = int((input - self.minval) * self.nInternal / self.range) \
                      + self.padding
      else:
        centerbin = int(((input - self.minval) + self.resolution/2) \
                          / self.resolution ) + self.padding

      # We use the first bit to be set in the encoded output as the bucket index
      minbin = centerbin - self.halfwidth
      return [minbin]


Instruction:
Can you tell what is the following Python 3 function doing
def getBucketIndices(self, input):

    if type(input) is float and math.isnan(input):
      input = SENTINEL_VALUE_FOR_MISSING_DATA

    if input == SENTINEL_VALUE_FOR_MISSING_DATA:
      return [None]

    minbin = self._getFirstOnBit(input)[0]

    # For periodic encoders, the bucket index is the index of the center bit
    if self.periodic:
      bucketIdx = minbin + self.halfwidth
      if bucketIdx < 0:
        bucketIdx += self.n

    # for non-periodic encoders, the bucket index is the index of the left bit
    else:
      bucketIdx = minbin

    return [bucketIdx]

Response: Returns the bucket indices of the input bit.


Instruction:
Can you generate the documentation for the following Python 3 function
def encodeIntoArray(self, input, output, learn=True):

    if input is not None and not isinstance(input, numbers.Number):
      raise TypeError(
          "Expected a scalar input but got input of type %s" % type(input))

    if type(input) is float and math.isnan(input):
      input = SENTINEL_VALUE_FOR_MISSING_DATA

    # Get the bucket index to use
    bucketIdx = self._getFirstOnBit(input)[0]

    if bucketIdx is None:
      # None is returned for missing value
      output[0:self.n] = 0  #TODO: should all 1s, or random SDR be returned instead?

    else:
      # The bucket index is the index of the first bit to set in the output
      output[:self.n] = 0
      minbin = bucketIdx
      maxbin = minbin + 2*self.halfwidth
      if self.periodic:
        # Handle the edges by computing wrap-around
        if maxbin >= self.n:
          bottombins = maxbin - self.n + 1
          output[:bottombins] = 1
          maxbin = self.n - 1
        if minbin < 0:
          topbins = -minbin
          output[self.n - topbins:self.n] = 1
          minbin = 0

      assert minbin >= 0
      assert maxbin < self.n
      # set the output (except for periodic wraparound)
      output[minbin:maxbin + 1] = 1

    # Debug the decode() method
    if self.verbosity >= 2:
      print
      print "input:", input
      print "range:", self.minval, "-", self.maxval
      print "n:", self.n, "w:", self.w, "resolution:", self.resolution, \
            "radius", self.radius, "periodic:", self.periodic
      print "output:",
      self.pprint(output)
      print "input desc:", self.decodedToStr(self.decode(output))

Response: Encode the input into an array of integers.


Instruction:
How would you code a function in Python 3 to
see the function description in base.py

Response:
def decode(self, encoded, parentFieldName=''):
    """ See the function description in base.py
    """

    # For now, we simply assume any top-down output greater than 0
    #  is ON. Eventually, we will probably want to incorporate the strength
    #  of each top-down output.
    tmpOutput = numpy.array(encoded[:self.n] > 0).astype(encoded.dtype)
    if not tmpOutput.any():
      return (dict(), [])

    # ------------------------------------------------------------------------
    # First, assume the input pool is not sampled 100%, and fill in the
    #  "holes" in the encoded representation (which are likely to be present
    #  if this is a coincidence that was learned by the SP).

    # Search for portions of the output that have "holes"
    maxZerosInARow = self.halfwidth
    for i in xrange(maxZerosInARow):
      searchStr = numpy.ones(i + 3, dtype=encoded.dtype)
      searchStr[1:-1] = 0
      subLen = len(searchStr)

      # Does this search string appear in the output?
      if self.periodic:
        for j in xrange(self.n):
          outputIndices = numpy.arange(j, j + subLen)
          outputIndices %= self.n
          if numpy.array_equal(searchStr, tmpOutput[outputIndices]):
            tmpOutput[outputIndices] = 1

      else:
        for j in xrange(self.n - subLen + 1):
          if numpy.array_equal(searchStr, tmpOutput[j:j + subLen]):
            tmpOutput[j:j + subLen] = 1

    if self.verbosity >= 2:
      print "raw output:", encoded[:self.n]
      print "filtered output:", tmpOutput

    # ------------------------------------------------------------------------
    # Find each run of 1's.
    nz = tmpOutput.nonzero()[0]
    runs = []     # will be tuples of (startIdx, runLength)
    run = [nz[0], 1]
    i = 1
    while (i < len(nz)):
      if nz[i] == run[0] + run[1]:
        run[1] += 1
      else:
        runs.append(run)
        run = [nz[i], 1]
      i += 1
    runs.append(run)

    # If we have a periodic encoder, merge the first and last run if they
    #  both go all the way to the edges
    if self.periodic and len(runs) > 1:
      if runs[0][0] == 0 and runs[-1][0] + runs[-1][1] == self.n:
        runs[-1][1] += runs[0][1]
        runs = runs[1:]

    # ------------------------------------------------------------------------
    # Now, for each group of 1's, determine the "left" and "right" edges, where
    #  the "left" edge is inset by halfwidth and the "right" edge is inset by
    #  halfwidth.
    # For a group of width w or less, the "left" and "right" edge are both at
    #   the center position of the group.
    ranges = []
    for run in runs:
      (start, runLen) = run
      if runLen <= self.w:
        left = right = start + runLen / 2
      else:
        left = start + self.halfwidth
        right = start + runLen - 1 - self.halfwidth

      # Convert to input space.
      if not self.periodic:
        inMin = (left - self.padding) * self.resolution + self.minval
        inMax = (right - self.padding) * self.resolution + self.minval
      else:
        inMin = (left - self.padding) * self.range / self.nInternal + self.minval
        inMax = (right - self.padding) * self.range / self.nInternal + self.minval
      # Handle wrap-around if periodic
      if self.periodic:
        if inMin >= self.maxval:
          inMin -= self.range
          inMax -= self.range

      # Clip low end
      if inMin < self.minval:
        inMin = self.minval
      if inMax < self.minval:
        inMax = self.minval

      # If we have a periodic encoder, and the max is past the edge, break into
      #  2 separate ranges
      if self.periodic and inMax >= self.maxval:
        ranges.append([inMin, self.maxval])
        ranges.append([self.minval, inMax - self.range])
      else:
        if inMax > self.maxval:
          inMax = self.maxval
        if inMin > self.maxval:
          inMin = self.maxval
        ranges.append([inMin, inMax])

    desc = self._generateRangeDescription(ranges)
    # Return result
    if parentFieldName != '':
      fieldName = "%s.%s" % (parentFieldName, self.name)
    else:
      fieldName = self.name

    return ({fieldName: (ranges, desc)}, [fieldName])


Instruction:
Can you generate the documentation for the following Python 3 function
def _generateRangeDescription(self, ranges):
    desc = ""
    numRanges = len(ranges)
    for i in xrange(numRanges):
      if ranges[i][0] != ranges[i][1]:
        desc += "%.2f-%.2f" % (ranges[i][0], ranges[i][1])
      else:
        desc += "%.2f" % (ranges[i][0])
      if i < numRanges - 1:
        desc += ", "
    return desc

Response: generate description from a text description of the ranges


Instruction:
Can you write a function in Python 3 where it
returns the interal _topDownMappingM matrix used for handling the bucketInfo and topDownCompute methods. This matrix is used for handling the bucketInfo and topDownCompute methods. This matrix is used for handling the bucketInfo and topDownCompute methods. This matrix is used for handling the bucketInfo and topDownCompute methods.

Response:
def _getTopDownMapping(self):
    """ Return the interal _topDownMappingM matrix used for handling the
    bucketInfo() and topDownCompute() methods. This is a matrix, one row per
    category (bucket) where each row contains the encoded output for that
    category.
    """

    # Do we need to build up our reverse mapping table?
    if self._topDownMappingM is None:

      # The input scalar value corresponding to each possible output encoding
      if self.periodic:
        self._topDownValues = numpy.arange(self.minval + self.resolution / 2.0,
                                           self.maxval,
                                           self.resolution)
      else:
        #Number of values is (max-min)/resolutions
        self._topDownValues = numpy.arange(self.minval,
                                           self.maxval + self.resolution / 2.0,
                                           self.resolution)

      # Each row represents an encoded output pattern
      numCategories = len(self._topDownValues)
      self._topDownMappingM = SM32(numCategories, self.n)

      outputSpace = numpy.zeros(self.n, dtype=GetNTAReal())
      for i in xrange(numCategories):
        value = self._topDownValues[i]
        value = max(value, self.minval)
        value = min(value, self.maxval)
        self.encodeIntoArray(value, outputSpace, learn=False)
        self._topDownMappingM.setRowFromDense(i, outputSpace)

    return self._topDownMappingM


Instruction:
Can you generate a brief explanation for the following Python 3 code
def getBucketValues(self):

    # Need to re-create?
    if self._bucketValues is None:
      topDownMappingM = self._getTopDownMapping()
      numBuckets = topDownMappingM.nRows()
      self._bucketValues = []
      for bucketIdx in range(numBuckets):
        self._bucketValues.append(self.getBucketInfo([bucketIdx])[0].value)

    return self._bucketValues

Response: Returns a list of all the bucket values in the current project.


Instruction:
Make a summary of the following Python 3 code
def getBucketInfo(self, buckets):

    # Get/generate the topDown mapping table
    #NOTE: although variable topDownMappingM is unused, some (bad-style) actions
    #are executed during _getTopDownMapping() so this line must stay here
    topDownMappingM = self._getTopDownMapping()

    # The "category" is simply the bucket index
    category = buckets[0]
    encoding = self._topDownMappingM.getRow(category)

    # Which input value does this correspond to?
    if self.periodic:
      inputVal = (self.minval + (self.resolution / 2.0) +
                  (category * self.resolution))
    else:
      inputVal = self.minval + (category * self.resolution)

    return [EncoderResult(value=inputVal, scalar=inputVal, encoding=encoding)]

Response: Returns the encoder result for the given buckets.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def topDownCompute(self, encoded):

    # Get/generate the topDown mapping table
    topDownMappingM = self._getTopDownMapping()

    # See which "category" we match the closest.
    category = topDownMappingM.rightVecProd(encoded).argmax()

    # Return that bucket info
    return self.getBucketInfo([category])

Response: Compute the topDown mapping table for a given encoded word.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def closenessScores(self, expValues, actValues, fractional=True):

    expValue = expValues[0]
    actValue = actValues[0]
    if self.periodic:
      expValue = expValue % self.maxval
      actValue = actValue % self.maxval

    err = abs(expValue - actValue)
    if self.periodic:
      err = min(err, self.maxval - err)
    if fractional:
      pctErr = float(err) / (self.maxval - self.minval)
      pctErr = min(1.0, pctErr)
      closeness = 1.0 - pctErr
    else:
      closeness = err

    return numpy.array([closeness])

Response: Compute closeness scores for the given expValues and actValues.


Instruction:
Can you implement a function in Python 3 that
initializes all ephemeral members after being restored to a pickled state.

Response:
def _initEphemerals(self):
    """
    Initialize all ephemeral members after being restored to a pickled state.
    """
    ## We store the lists of segments updates, per cell, so that they can be
    # applied later during learning, when the cell gets bottom-up activation.
    # We store one list per cell. The lists are identified with a hash key which
    # is a tuple (column index, cell index).
    self.segmentUpdates = {}

    # Allocate and reset all stats
    self.resetStats()

    # NOTE: We don't use the same backtrack buffer for inference and learning
    # because learning has a different metric for determining if an input from
    # the past is potentially useful again for backtracking.
    #
    # Our inference backtrack buffer. This keeps track of up to
    # maxInfBacktrack of previous input. Each entry is a list of active column
    # inputs.
    self._prevInfPatterns = []

    # Our learning backtrack buffer. This keeps track of up to maxLrnBacktrack
    # of previous input. Each entry is a list of active column inputs
    self._prevLrnPatterns = []

    # Keep integers rather than bools. Float?
    stateShape = (self.numberOfCols, self.cellsPerColumn)

    self.lrnActiveState = {}
    self.lrnActiveState["t"] = numpy.zeros(stateShape, dtype="int8")
    self.lrnActiveState["t-1"] = numpy.zeros(stateShape, dtype="int8")

    self.lrnPredictedState = {}
    self.lrnPredictedState["t"] = numpy.zeros(stateShape, dtype="int8")
    self.lrnPredictedState["t-1"] = numpy.zeros(stateShape, dtype="int8")

    self.infActiveState = {}
    self.infActiveState["t"] = numpy.zeros(stateShape, dtype="int8")
    self.infActiveState["t-1"] = numpy.zeros(stateShape, dtype="int8")
    self.infActiveState["backup"] = numpy.zeros(stateShape, dtype="int8")
    self.infActiveState["candidate"] = numpy.zeros(stateShape, dtype="int8")

    self.infPredictedState = {}
    self.infPredictedState["t"] = numpy.zeros(stateShape, dtype="int8")
    self.infPredictedState["t-1"] = numpy.zeros(stateShape, dtype="int8")
    self.infPredictedState["backup"] = numpy.zeros(stateShape, dtype="int8")
    self.infPredictedState["candidate"] = numpy.zeros(stateShape, dtype="int8")

    self.cellConfidence = {}
    self.cellConfidence["t"] = numpy.zeros(stateShape, dtype="float32")
    self.cellConfidence["t-1"] = numpy.zeros(stateShape, dtype="float32")
    self.cellConfidence["candidate"] = numpy.zeros(stateShape, dtype="float32")

    self.colConfidence = {}
    self.colConfidence["t"] = numpy.zeros(self.numberOfCols, dtype="float32")
    self.colConfidence["t-1"] = numpy.zeros(self.numberOfCols, dtype="float32")
    self.colConfidence["candidate"] = numpy.zeros(self.numberOfCols,
                                                  dtype="float32")


Instruction:
How would you explain what the following Python 3 function does
def write(self, proto):
    proto.version = TM_VERSION
    self._random.write(proto.random)
    proto.numberOfCols = self.numberOfCols
    proto.cellsPerColumn = self.cellsPerColumn
    proto.initialPerm = float(self.initialPerm)
    proto.connectedPerm = float(self.connectedPerm)
    proto.minThreshold = self.minThreshold
    proto.newSynapseCount = self.newSynapseCount
    proto.permanenceInc = float(self.permanenceInc)
    proto.permanenceDec = float(self.permanenceDec)
    proto.permanenceMax = float(self.permanenceMax)
    proto.globalDecay = float(self.globalDecay)
    proto.activationThreshold = self.activationThreshold
    proto.doPooling = self.doPooling
    proto.segUpdateValidDuration = self.segUpdateValidDuration
    proto.burnIn = self.burnIn
    proto.collectStats = self.collectStats
    proto.verbosity = self.verbosity
    proto.pamLength = self.pamLength
    proto.maxAge = self.maxAge
    proto.maxInfBacktrack = self.maxInfBacktrack
    proto.maxLrnBacktrack = self.maxLrnBacktrack
    proto.maxSeqLength = self.maxSeqLength
    proto.maxSegmentsPerCell = self.maxSegmentsPerCell
    proto.maxSynapsesPerSegment = self.maxSynapsesPerSegment
    proto.outputType = self.outputType

    proto.activeColumns = self.activeColumns

    cellListProto = proto.init("cells", len(self.cells))
    for i, columnSegments in enumerate(self.cells):
      columnSegmentsProto = cellListProto.init(i, len(columnSegments))
      for j, cellSegments in enumerate(columnSegments):
        cellSegmentsProto = columnSegmentsProto.init(j, len(cellSegments))
        for k, segment in enumerate(cellSegments):
          segment.write(cellSegmentsProto[k])

    proto.lrnIterationIdx = self.lrnIterationIdx
    proto.iterationIdx = self.iterationIdx
    proto.segID = self.segID
    if self.currentOutput is None:
      proto.currentOutput.none = None
    else:
      proto.currentOutput.list = self.currentOutput.tolist()
    proto.pamCounter = self.pamCounter
    proto.collectSequenceStats = self.collectSequenceStats
    proto.resetCalled = self.resetCalled
    # In case of None, use negative value as placeholder for serialization
    proto.avgInputDensity = self.avgInputDensity or -1.0
    proto.learnedSeqLength = self.learnedSeqLength
    proto.avgLearnedSeqLength = self.avgLearnedSeqLength

    proto.prevLrnPatterns = self._prevLrnPatterns
    proto.prevInfPatterns = self._prevInfPatterns

    segmentUpdatesListProto = proto.init("segmentUpdates",
                                         len(self.segmentUpdates))
    for i, (key, updates) in enumerate(self.segmentUpdates.iteritems()):
      cellSegmentUpdatesProto = segmentUpdatesListProto[i]
      cellSegmentUpdatesProto.columnIdx = key[0]
      cellSegmentUpdatesProto.cellIdx = key[1]
      segmentUpdatesProto = cellSegmentUpdatesProto.init("segmentUpdates",
                                                         len(updates))
      for j, (lrnIterationIdx, segmentUpdate) in enumerate(updates):
        segmentUpdateWrapperProto = segmentUpdatesProto[j]
        segmentUpdateWrapperProto.lrnIterationIdx = lrnIterationIdx
        segmentUpdate.write(segmentUpdateWrapperProto.segmentUpdate)

    # self.cellConfidence
    proto.cellConfidenceT = self.cellConfidence["t"].tolist()
    proto.cellConfidenceT1 = self.cellConfidence["t-1"].tolist()
    proto.cellConfidenceCandidate = self.cellConfidence["candidate"].tolist()

    # self.colConfidence
    proto.colConfidenceT = self.colConfidence["t"].tolist()
    proto.colConfidenceT1 = self.colConfidence["t-1"].tolist()
    proto.colConfidenceCandidate = self.colConfidence["candidate"].tolist()

    # self.lrnActiveState
    proto.lrnActiveStateT = self.lrnActiveState["t"].tolist()
    proto.lrnActiveStateT1 = self.lrnActiveState["t-1"].tolist()

    # self.infActiveState
    proto.infActiveStateT = self.infActiveState["t"].tolist()
    proto.infActiveStateT1 = self.infActiveState["t-1"].tolist()
    proto.infActiveStateBackup = self.infActiveState["backup"].tolist()
    proto.infActiveStateCandidate = self.infActiveState["candidate"].tolist()

    # self.lrnPredictedState
    proto.lrnPredictedStateT = self.lrnPredictedState["t"].tolist()
    proto.lrnPredictedStateT1 = self.lrnPredictedState["t-1"].tolist()

    # self.infPredictedState
    proto.infPredictedStateT = self.infPredictedState["t"].tolist()
    proto.infPredictedStateT1 = self.infPredictedState["t-1"].tolist()
    proto.infPredictedStateBackup = self.infPredictedState["backup"].tolist()
    proto.infPredictedStateCandidate = self.infPredictedState["candidate"].tolist()

    proto.consolePrinterVerbosity = self.consolePrinterVerbosity

Response: Populates the contents of the given proto instance.


Instruction:
Implement a Python 3 function for
resetting the internal state of all cells and resets the internal state of all cells.

Response:
def reset(self,):
    """
    Reset the state of all cells.

    This is normally used between sequences while training. All internal states
    are reset to 0.
    """
    if self.verbosity >= 3:
      print "\n==== RESET ====="

    self.lrnActiveState['t-1'].fill(0)
    self.lrnActiveState['t'].fill(0)
    self.lrnPredictedState['t-1'].fill(0)
    self.lrnPredictedState['t'].fill(0)

    self.infActiveState['t-1'].fill(0)
    self.infActiveState['t'].fill(0)
    self.infPredictedState['t-1'].fill(0)
    self.infPredictedState['t'].fill(0)

    self.cellConfidence['t-1'].fill(0)
    self.cellConfidence['t'].fill(0)

    # Flush the segment update queue
    self.segmentUpdates = {}

    self._internalStats['nInfersSinceReset'] = 0

    #To be removed
    self._internalStats['curPredictionScore'] = 0
    #New prediction score
    self._internalStats['curPredictionScore2']   = 0
    self._internalStats['curFalseNegativeScore'] = 0
    self._internalStats['curFalsePositiveScore'] = 0

    self._internalStats['curMissing'] = 0
    self._internalStats['curExtra'] = 0

    # When a reset occurs, set prevSequenceSignature to the signature of the
    # just-completed sequence and start accumulating histogram for the next
    # sequence.
    self._internalStats['prevSequenceSignature'] = None
    if self.collectSequenceStats:
      if self._internalStats['confHistogram'].sum() > 0:
        sig = self._internalStats['confHistogram'].copy()
        sig.reshape(self.numberOfCols * self.cellsPerColumn)
        self._internalStats['prevSequenceSignature'] = sig
      self._internalStats['confHistogram'].fill(0)

    self.resetCalled = True

    # Clear out input history
    self._prevInfPatterns = []
    self._prevLrnPatterns = []


Instruction:
Here you have a function in Python 3, explain what it does
def _updateStatsInferEnd(self, stats, bottomUpNZ, predictedState,
                           colConfidence):
    """
    Called at the end of learning and inference, this routine will update
    a number of stats in our _internalStats dictionary, including our computed
    prediction score.

    :param stats            internal stats dictionary
    :param bottomUpNZ       list of the active bottom-up inputs
    :param predictedState   The columns we predicted on the last time step (should
                            match the current bottomUpNZ in the best case)
    :param colConfidence    Column confidences we determined on the last time step
    """
    # Return if not collecting stats
    if not self.collectStats:
      return
    stats['nInfersSinceReset'] += 1

    # Compute the prediction score, how well the prediction from the last
    # time step predicted the current bottom-up input
    (numExtra2, numMissing2, confidences2) = self._checkPrediction(
        patternNZs=[bottomUpNZ], output=predictedState,
        colConfidence=colConfidence)
    predictionScore, positivePredictionScore, negativePredictionScore = (
        confidences2[0])

    # Store the stats that don't depend on burn-in
    stats['curPredictionScore2'] = float(predictionScore)
    stats['curFalseNegativeScore'] = 1.0 - float(positivePredictionScore)
    stats['curFalsePositiveScore'] = float(negativePredictionScore)

    stats['curMissing'] = numMissing2
    stats['curExtra'] = numExtra2

    # If we are passed the burn-in period, update the accumulated stats
    # Here's what various burn-in values mean:
    #   0: try to predict the first element of each sequence and all subsequent
    #   1: try to predict the second element of each sequence and all subsequent
    #   etc.
    if stats['nInfersSinceReset'] <= self.burnIn:
      return

    # Burn-in related stats
    stats['nPredictions'] += 1
    numExpected = max(1.0, float(len(bottomUpNZ)))

    stats['totalMissing'] += numMissing2
    stats['totalExtra'] += numExtra2
    stats['pctExtraTotal'] += 100.0 * numExtra2 / numExpected
    stats['pctMissingTotal'] += 100.0 * numMissing2 / numExpected
    stats['predictionScoreTotal2'] += float(predictionScore)
    stats['falseNegativeScoreTotal'] += 1.0 - float(positivePredictionScore)
    stats['falsePositiveScoreTotal'] += float(negativePredictionScore)

    if self.collectSequenceStats:
      # Collect cell confidences for every cell that correctly predicted current
      # bottom up input. Normalize confidence across each column
      cc = self.cellConfidence['t-1'] * self.infActiveState['t']
      sconf = cc.sum(axis=1)
      for c in range(self.numberOfCols):
        if sconf[c] > 0:
          cc[c, :] /= sconf[c]

      # Update cell confidence histogram: add column-normalized confidence
      # scores to the histogram
      self._internalStats['confHistogram'] += cc

Response: This routine is called at the end of learning and inference of a single active bottom - up input. It updates the internal stats dictionary with the number of stats that depend on the current bottom - up input.


Instruction:
Implement a function in Python 3 to
print an integer array that is the same shape as activeState.

Response:
def printState(self, aState):
    """
    Print an integer array that is the same shape as activeState.

    :param aState: TODO: document
    """
    def formatRow(var, i):
      s = ''
      for c in range(self.numberOfCols):
        if c > 0 and c % 10 == 0:
          s += ' '
        s += str(var[c, i])
      s += ' '
      return s

    for i in xrange(self.cellsPerColumn):
      print formatRow(aState, i)


Instruction:
How would you explain what the following Python 3 function does
def printConfidence(self, aState, maxCols = 20):
    def formatFPRow(var, i):
      s = ''
      for c in range(min(maxCols, self.numberOfCols)):
        if c > 0 and c % 10 == 0:
          s += '   '
        s += ' %5.3f' % var[c, i]
      s += ' '
      return s

    for i in xrange(self.cellsPerColumn):
      print formatFPRow(aState, i)

Response: Print a floating point array that is the same shape as activeState.


Instruction:
Make a summary of the following Python 3 code
def printColConfidence(self, aState, maxCols = 20):
    def formatFPRow(var):
      s = ''
      for c in range(min(maxCols, self.numberOfCols)):
        if c > 0 and c % 10 == 0:
          s += '   '
        s += ' %5.3f' % var[c]
      s += ' '
      return s

    print formatFPRow(aState)

Response: Print the col confidence of a flat floating point array.


Instruction:
Can you implement a function in Python 3 that
prints the current state of the current language.

Response:
def printStates(self, printPrevious = True, printLearnState = True):
    """
    TODO: document

    :param printPrevious:
    :param printLearnState:
    :return:
    """
    def formatRow(var, i):
      s = ''
      for c in range(self.numberOfCols):
        if c > 0 and c % 10 == 0:
          s += ' '
        s += str(var[c, i])
      s += ' '
      return s

    print "\nInference Active state"
    for i in xrange(self.cellsPerColumn):
      if printPrevious:
        print formatRow(self.infActiveState['t-1'], i),
      print formatRow(self.infActiveState['t'], i)

    print "Inference Predicted state"
    for i in xrange(self.cellsPerColumn):
      if printPrevious:
        print formatRow(self.infPredictedState['t-1'], i),
      print formatRow(self.infPredictedState['t'], i)

    if printLearnState:
      print "\nLearn Active state"
      for i in xrange(self.cellsPerColumn):
        if printPrevious:
          print formatRow(self.lrnActiveState['t-1'], i),
        print formatRow(self.lrnActiveState['t'], i)

      print "Learn Predicted state"
      for i in xrange(self.cellsPerColumn):
        if printPrevious:
          print formatRow(self.lrnPredictedState['t-1'], i),
        print formatRow(self.lrnPredictedState['t'], i)


Instruction:
Here you have a function in Python 3, explain what it does
def printOutput(self, y):
    print "Output"
    for i in xrange(self.cellsPerColumn):
      for c in xrange(self.numberOfCols):
        print int(y[c, i]),
      print

Response: Print the output of the sequence of documents.


Instruction:
Explain what the following Python 3 code does
def printInput(self, x):
    print "Input"
    for c in xrange(self.numberOfCols):
      print int(x[c]),
    print

Response: Print the input of the sequence.