Instruction:
How would you explain what the following Python 3 function does
def jobsGetFields(self, jobIDs, fields, requireAll=True):
    assert isinstance(jobIDs, self._SEQUENCE_TYPES)
    assert len(jobIDs) >=1

    rows = self._getMatchingRowsWithRetries(
      self._jobs, dict(job_id=jobIDs),
      ['job_id'] + [self._jobs.pubToDBNameDict[x] for x in fields])

    if requireAll and len(rows) < len(jobIDs):
      # NOTE: this will also trigger if the jobIDs list included duplicates
      raise RuntimeError("jobIDs %s not found within the jobs table" % (
        (set(jobIDs) - set(r[0] for r in rows)),))

    return [(r[0], list(r[1:])) for r in rows]

Response: Fetch the values of 1 or more fields from a sequence of job records.


Instruction:
Write a Python 3 script for
changing the values of one or more fields in a job.

Response:
def jobSetFields(self, jobID, fields, useConnectionID=True,
                   ignoreUnchanged=False):
    """ Change the values of 1 or more fields in a job. Here, 'fields' is a
    dict with the name/value pairs to change. The names are the public names of
    the fields (camelBack, not the lower_case_only form as stored in the DB).
    This method is for private use by the ClientJobManager only.

    Parameters:
    ----------------------------------------------------------------
    jobID:     jobID of the job record

    fields:    dictionary of fields to change

    useConnectionID: True if the connection id of the calling function
    must be the same as the connection that created the job. Set
    to False for hypersearch workers

    ignoreUnchanged: The default behavior is to throw a
    RuntimeError if no rows are affected. This could either be
    because:
      1) Because there was not matching jobID
      2) or if the data to update matched the data in the DB exactly.

    Set this parameter to True if you expect case 2 and wish to
    supress the error.
    """

    # Form the sequecce of key=value strings that will go into the
    #  request
    assignmentExpressions = ','.join(
      ["%s=%%s" % (self._jobs.pubToDBNameDict[f],) for f in fields.iterkeys()])
    assignmentValues = fields.values()

    query = 'UPDATE %s SET %s ' \
            '          WHERE job_id=%%s' \
            % (self.jobsTableName, assignmentExpressions,)
    sqlParams = assignmentValues + [jobID]

    if useConnectionID:
      query += ' AND _eng_cjm_conn_id=%s'
      sqlParams.append(self._connectionID)

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      result = conn.cursor.execute(query, sqlParams)

    if result != 1 and not ignoreUnchanged:
      raise RuntimeError(
        "Tried to change fields (%r) of jobID=%s conn_id=%r), but an error " \
        "occurred. result=%r; query=%r" % (
          assignmentExpressions, jobID, self._connectionID, result, query))


Instruction:
Implement a Python 3 function for
changing the value of 1 field in a job to 'newValue', but only if the current value matches 'curValue'. The 'fieldName' is the public name of the field (camelBack, not the lower_case_only form as stored in the DB). This method is used for example by HypersearcWorkers to update the engWorkerState field periodically. By qualifying on curValue, it insures that only 1 worker at a time is elected to perform the next scheduled periodic sweep of the models. Parameters: ---------------------------------------------------------------- jobID: jobID of the job record to modify fieldName: public field name of the field newValue: new value of the field to set curValue: current value to qualify against retval: True if we successfully modified the field False if curValue did not match

Response:
def jobSetFieldIfEqual(self, jobID, fieldName, newValue, curValue):
    """ Change the value of 1 field in a job to 'newValue', but only if the
    current value matches 'curValue'. The 'fieldName' is the public name of
    the field (camelBack, not the lower_case_only form as stored in the DB).

    This method is used for example by HypersearcWorkers to update the
    engWorkerState field periodically. By qualifying on curValue, it insures
    that only 1 worker at a time is elected to perform the next scheduled
    periodic sweep of the models.

    Parameters:
    ----------------------------------------------------------------
    jobID:        jobID of the job record to modify
    fieldName:    public field name of the field
    newValue:     new value of the field to set
    curValue:     current value to qualify against

    retval:       True if we successfully modified the field
                  False if curValue did not match
    """

    # Get the private field name and string form of the value
    dbFieldName = self._jobs.pubToDBNameDict[fieldName]

    conditionValue = []
    if isinstance(curValue, bool):
      conditionExpression = '%s IS %s' % (
        dbFieldName, {True:'TRUE', False:'FALSE'}[curValue])
    elif curValue is None:
      conditionExpression = '%s is NULL' % (dbFieldName,)
    else:
      conditionExpression = '%s=%%s' % (dbFieldName,)
      conditionValue.append(curValue)

    query = 'UPDATE %s SET _eng_last_update_time=UTC_TIMESTAMP(), %s=%%s ' \
            '          WHERE job_id=%%s AND %s' \
            % (self.jobsTableName, dbFieldName, conditionExpression)
    sqlParams = [newValue, jobID] + conditionValue

    with ConnectionFactory.get() as conn:
      result = conn.cursor.execute(query, sqlParams)

    return (result == 1)


Instruction:
Here you have a function in Python 3, explain what it does
def jobIncrementIntField(self, jobID, fieldName, increment=1,
                           useConnectionID=False):
    """ Incremet the value of 1 field in a job by increment. The 'fieldName' is
    the public name of the field (camelBack, not the lower_case_only form as
    stored in the DB).

    This method is used for example by HypersearcWorkers to update the
    engWorkerState field periodically. By qualifying on curValue, it insures
    that only 1 worker at a time is elected to perform the next scheduled
    periodic sweep of the models.

    Parameters:
    ----------------------------------------------------------------
    jobID:        jobID of the job record to modify
    fieldName:    public field name of the field
    increment:    increment is added to the current value of the field
    """
    # Get the private field name and string form of the value
    dbFieldName = self._jobs.pubToDBNameDict[fieldName]

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      query = 'UPDATE %s SET %s=%s+%%s ' \
              '          WHERE job_id=%%s' \
              % (self.jobsTableName, dbFieldName, dbFieldName)
      sqlParams = [increment, jobID]

      if useConnectionID:
        query += ' AND _eng_cjm_conn_id=%s'
        sqlParams.append(self._connectionID)

      result = conn.cursor.execute(query, sqlParams)

    if result != 1:
      raise RuntimeError(
        "Tried to increment the field (%r) of jobID=%s (conn_id=%r), but an " \
        "error occurred. result=%r; query=%r" % (
          dbFieldName, jobID, self._connectionID, result, query))

Response: Increment the value of a field in a job record by increment.


Instruction:
Explain what the following Python 3 code does
def jobUpdateResults(self, jobID, results):
    with ConnectionFactory.get() as conn:
      query = 'UPDATE %s SET _eng_last_update_time=UTC_TIMESTAMP(), ' \
              '              results=%%s ' \
              '          WHERE job_id=%%s' % (self.jobsTableName,)
      conn.cursor.execute(query, [results, jobID])

Response: Update the results string and last - update - time fields of a single job.


Instruction:
Given the following Python 3 function, write the documentation
def modelsClearAll(self):
    self._logger.info('Deleting all rows from models table %r',
                      self.modelsTableName)
    with ConnectionFactory.get() as conn:
      query = 'DELETE FROM %s' % (self.modelsTableName)
      conn.cursor.execute(query)

Response: Delete all models from the models table


Instruction:
Given the following Python 3 function, write the documentation
def modelInsertAndStart(self, jobID, params, paramsHash, particleHash=None):
    # Fill in default particleHash
    if particleHash is None:
      particleHash = paramsHash

    # Normalize hashes
    paramsHash = self._normalizeHash(paramsHash)
    particleHash = self._normalizeHash(particleHash)

    def findExactMatchNoRetries(conn):
      return self._getOneMatchingRowNoRetries(
        self._models, conn,
        {'job_id':jobID, '_eng_params_hash':paramsHash,
         '_eng_particle_hash':particleHash},
        ['model_id', '_eng_worker_conn_id'])

    @g_retrySQL
    def findExactMatchWithRetries():
      with ConnectionFactory.get() as conn:
        return findExactMatchNoRetries(conn)

    # Check if the model is already in the models table
    #
    # NOTE: with retries of mysql transient failures, we can't always tell
    #  whether the row was already inserted (e.g., comms failure could occur
    #  after insertion into table, but before arrival or response), so the
    #  need to check before attempting to insert a new row
    #
    # TODO: if we could be assured that the caller already verified the
    #  model's absence before calling us, we could skip this check here
    row = findExactMatchWithRetries()
    if row is not None:
      return (row[0], False)

    @g_retrySQL
    def insertModelWithRetries():
      """ NOTE: it's possible that another process on some machine is attempting
      to insert the same model at the same time as the caller """
      with ConnectionFactory.get() as conn:
        # Create a new job entry
        query = 'INSERT INTO %s (job_id, params, status, _eng_params_hash, ' \
                '  _eng_particle_hash, start_time, _eng_last_update_time, ' \
                '  _eng_worker_conn_id) ' \
                '  VALUES (%%s, %%s, %%s, %%s, %%s, UTC_TIMESTAMP(), ' \
                '          UTC_TIMESTAMP(), %%s) ' \
                % (self.modelsTableName,)
        sqlParams = (jobID, params, self.STATUS_RUNNING, paramsHash,
                     particleHash, self._connectionID)
        try:
          numRowsAffected = conn.cursor.execute(query, sqlParams)
        except Exception, e:
          # NOTE: We have seen instances where some package in the calling
          #  chain tries to interpret the exception message using unicode.
          #  Since the exception message contains binary data (the hashes), this
          #  can in turn generate a Unicode translation exception. So, we catch
          #  ALL exceptions here and look for the string "Duplicate entry" in
          #  the exception args just in case this happens. For example, the
          #  Unicode exception we might get is:
          #   (<type 'exceptions.UnicodeDecodeError'>, UnicodeDecodeError('utf8', "Duplicate entry '1000-?.\x18\xb1\xd3\xe0CO\x05\x8b\xf80\xd7E5\xbb' for key 'job_id'", 25, 26, 'invalid start byte'))
          #
          #  If it weren't for this possible Unicode translation error, we
          #  could watch for only the exceptions we want, like this:
          #  except pymysql.IntegrityError, e:
          #    if e.args[0] != mysqlerrors.DUP_ENTRY:
          #      raise
          if "Duplicate entry" not in str(e):
            raise

          # NOTE: duplicate entry scenario: however, we can't discern
          # whether it was inserted by another process or this one, because an
          # intermittent failure may have caused us to retry
          self._logger.info('Model insert attempt failed with DUP_ENTRY: '
                            'jobID=%s; paramsHash=%s OR particleHash=%s; %r',
                            jobID, paramsHash.encode('hex'),
                            particleHash.encode('hex'), e)
        else:
          if numRowsAffected == 1:
            # NOTE: SELECT LAST_INSERT_ID() returns 0 after re-connection
            conn.cursor.execute('SELECT LAST_INSERT_ID()')
            modelID = conn.cursor.fetchall()[0][0]
            if modelID != 0:
              return (modelID, True)
            else:
              self._logger.warn(
                'SELECT LAST_INSERT_ID for model returned 0, implying loss of '
                'connection: jobID=%s; paramsHash=%r; particleHash=%r',
                jobID, paramsHash, particleHash)
          else:
            self._logger.error(
              'Attempt to insert model resulted in unexpected numRowsAffected: '
              'expected 1, but got %r; jobID=%s; paramsHash=%r; '
              'particleHash=%r',
              numRowsAffected, jobID, paramsHash, particleHash)

        # Look up the model and discern whether it is tagged with our conn id
        row = findExactMatchNoRetries(conn)
        if row is not None:
          (modelID, connectionID) = row
          return (modelID, connectionID == self._connectionID)

        # This set of params is already in the table, just get the modelID
        query = 'SELECT (model_id) FROM %s ' \
                '                  WHERE job_id=%%s AND ' \
                '                        (_eng_params_hash=%%s ' \
                '                         OR _eng_particle_hash=%%s) ' \
                '                  LIMIT 1 ' \
                % (self.modelsTableName,)
        sqlParams = [jobID, paramsHash, particleHash]
        numRowsFound = conn.cursor.execute(query, sqlParams)
        assert numRowsFound == 1, (
          'Model not found: jobID=%s AND (paramsHash=%r OR particleHash=%r); '
          'numRowsFound=%r') % (jobID, paramsHash, particleHash, numRowsFound)
        (modelID,) = conn.cursor.fetchall()[0]
        return (modelID, False)

    return insertModelWithRetries()

Response: Insert a new model into the model table and return the modelID of the new model.


Instruction:
Implement a function in Python 3 to
get ALL info for a set of models

Response:
def modelsInfo(self, modelIDs):
    """ Get ALL info for a set of models

    WARNING!!!: The order of the results are NOT necessarily in the same order as
    the order of the model IDs passed in!!!

    Parameters:
    ----------------------------------------------------------------
    modelIDs:    list of model IDs
    retval:      list of nametuples containing all the fields stored for each
                    model.
    """
    assert isinstance(modelIDs, self._SEQUENCE_TYPES), (
      "wrong modelIDs type: %s") % (type(modelIDs),)
    assert modelIDs, "modelIDs is empty"

    rows = self._getMatchingRowsWithRetries(
      self._models, dict(model_id=modelIDs),
      [self._models.pubToDBNameDict[f]
       for f in self._models.modelInfoNamedTuple._fields])

    results = [self._models.modelInfoNamedTuple._make(r) for r in rows]

    # NOTE: assetion will also fail if modelIDs contains duplicates
    assert len(results) == len(modelIDs), "modelIDs not found: %s" % (
      set(modelIDs) - set(r.modelId for r in results))

    return results


Instruction:
Implement a Python 3 function for
returning the values of 1 or more fields from a sequence of models.

Response:
def modelsGetFields(self, modelIDs, fields):
    """ Fetch the values of 1 or more fields from a sequence of model records.
    Here, 'fields' is a list with the names of the fields to fetch. The names
    are the public names of the fields (camelBack, not the lower_case_only form
    as stored in the DB).

    WARNING!!!: The order of the results are NOT necessarily in the same order
    as the order of the model IDs passed in!!!

    Parameters:
    ----------------------------------------------------------------
    modelIDs:      A single modelID or sequence of modelIDs
    fields:        A list  of fields to return

    Returns:  If modelIDs is a sequence:
                a list of tuples->(modelID, [field1, field2,...])
              If modelIDs is a single modelID:
                a list of field values->[field1, field2,...]
    """
    assert len(fields) >= 1, 'fields is empty'

    # Form the sequence of field name strings that will go into the
    #  request
    isSequence = isinstance(modelIDs, self._SEQUENCE_TYPES)

    if isSequence:
      assert len(modelIDs) >=1, 'modelIDs is empty'
    else:
      modelIDs = [modelIDs]

    rows = self._getMatchingRowsWithRetries(
      self._models, dict(model_id=modelIDs),
      ['model_id'] + [self._models.pubToDBNameDict[f] for f in fields])

    if len(rows) < len(modelIDs):
      raise RuntimeError("modelIDs not found within the models table: %s" % (
        (set(modelIDs) - set(r[0] for r in rows)),))

    if not isSequence:
      return list(rows[0][1:])

    return [(r[0], list(r[1:])) for r in rows]


Instruction:
Can you tell what is the following Python 3 function doing
def modelsGetFieldsForJob(self, jobID, fields, ignoreKilled=False):

    assert len(fields) >= 1, 'fields is empty'

    # Form the sequence of field name strings that will go into the
    #  request
    dbFields = [self._models.pubToDBNameDict[x] for x in fields]
    dbFieldsStr = ','.join(dbFields)

    query = 'SELECT model_id, %s FROM %s ' \
              '          WHERE job_id=%%s ' \
              % (dbFieldsStr, self.modelsTableName)
    sqlParams = [jobID]

    if ignoreKilled:
      query += ' AND (completion_reason IS NULL OR completion_reason != %s)'
      sqlParams.append(self.CMPL_REASON_KILLED)

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      conn.cursor.execute(query, sqlParams)
      rows = conn.cursor.fetchall()

    if rows is None:
      # fetchall is defined to return a (possibly-empty) sequence of
      # sequences; however, we occasionally see None returned and don't know
      # why...
      self._logger.error("Unexpected None result from cursor.fetchall; "
                         "query=%r; Traceback=%r",
                         query, traceback.format_exc())

    return [(r[0], list(r[1:])) for r in rows]

Response: Returns the specified fields for all the models for a single job.


Instruction:
How would you explain what the following Python 3 function does
def modelsGetFieldsForCheckpointed(self, jobID, fields):

    assert len(fields) >= 1, "fields is empty"

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      dbFields = [self._models.pubToDBNameDict[f] for f in fields]
      dbFieldStr = ", ".join(dbFields)

      query = 'SELECT model_id, {fields} from {models}' \
              '   WHERE job_id=%s AND model_checkpoint_id IS NOT NULL'.format(
        fields=dbFieldStr, models=self.modelsTableName)

      conn.cursor.execute(query, [jobID])
      rows = conn.cursor.fetchall()

    return [(r[0], list(r[1:])) for r in rows]

Response: Returns a list of tuples that are used to figure out whether or not a new model should be checkpointed.


Instruction:
How would you code a function in Python 3 to
change the values of one or more fields in a model.

Response:
def modelSetFields(self, modelID, fields, ignoreUnchanged = False):
    """ Change the values of 1 or more fields in a model. Here, 'fields' is a
    dict with the name/value pairs to change. The names are the public names of
    the fields (camelBack, not the lower_case_only form as stored in the DB).

    Parameters:
    ----------------------------------------------------------------
    jobID:     jobID of the job record

    fields:    dictionary of fields to change

    ignoreUnchanged: The default behavior is to throw a
    RuntimeError if no rows are affected. This could either be
    because:
      1) Because there was no matching modelID
      2) or if the data to update matched the data in the DB exactly.

    Set this parameter to True if you expect case 2 and wish to
    supress the error.
    """

    # Form the sequence of key=value strings that will go into the
    #  request
    assignmentExpressions = ','.join(
      '%s=%%s' % (self._models.pubToDBNameDict[f],) for f in fields.iterkeys())
    assignmentValues = fields.values()

    query = 'UPDATE %s SET %s, update_counter = update_counter+1 ' \
            '          WHERE model_id=%%s' \
            % (self.modelsTableName, assignmentExpressions)
    sqlParams = assignmentValues + [modelID]

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      numAffectedRows = conn.cursor.execute(query, sqlParams)
      self._logger.debug("Executed: numAffectedRows=%r, query=%r, sqlParams=%r",
                         numAffectedRows, query, sqlParams)

    if numAffectedRows != 1 and not ignoreUnchanged:
      raise RuntimeError(
        ("Tried to change fields (%r) of model %r (conn_id=%r), but an error "
         "occurred. numAffectedRows=%r; query=%r; sqlParams=%r") % (
          fields, modelID, self._connectionID, numAffectedRows, query,
          sqlParams,))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def modelsGetParams(self, modelIDs):
    assert isinstance(modelIDs, self._SEQUENCE_TYPES), (
      "Wrong modelIDs type: %r") % (type(modelIDs),)
    assert len(modelIDs) >= 1, "modelIDs is empty"

    rows = self._getMatchingRowsWithRetries(
      self._models, {'model_id' : modelIDs},
      [self._models.pubToDBNameDict[f]
       for f in self._models.getParamsNamedTuple._fields])

    # NOTE: assertion will also fail when modelIDs contains duplicates
    assert len(rows) == len(modelIDs), "Didn't find modelIDs: %r" % (
      (set(modelIDs) - set(r[0] for r in rows)),)

    # Return the params and params hashes as a namedtuple
    return [self._models.getParamsNamedTuple._make(r) for r in rows]

Response: Returns the params and paramsHash for a set of models.


Instruction:
Write a Python 3 function that can
get the results string and other status fields for a set of models.

Response:
def modelsGetResultAndStatus(self, modelIDs):
    """ Get the results string and other status fields for a set of models.

    WARNING!!!: The order of the results are NOT necessarily in the same order
    as the order of the model IDs passed in!!!

    For each model, this returns a tuple containing:
     (modelID, results, status, updateCounter, numRecords, completionReason,
         completionMsg, engParamsHash

    Parameters:
    ----------------------------------------------------------------
    modelIDs:    list of model IDs
    retval:      list of result tuples. Each tuple contains:
                    (modelID, results, status, updateCounter, numRecords,
                      completionReason, completionMsg, engParamsHash)
    """
    assert isinstance(modelIDs, self._SEQUENCE_TYPES), (
      "Wrong modelIDs type: %r") % type(modelIDs)
    assert len(modelIDs) >= 1, "modelIDs is empty"

    rows = self._getMatchingRowsWithRetries(
      self._models, {'model_id' : modelIDs},
      [self._models.pubToDBNameDict[f]
       for f in self._models.getResultAndStatusNamedTuple._fields])

    # NOTE: assertion will also fail when modelIDs contains duplicates
    assert len(rows) == len(modelIDs), "Didn't find modelIDs: %r" % (
      (set(modelIDs) - set(r[0] for r in rows)),)

    # Return the results as a list of namedtuples
    return [self._models.getResultAndStatusNamedTuple._make(r) for r in rows]


Instruction:
How would you explain what the following Python 3 function does
def modelsGetUpdateCounters(self, jobID):
    rows = self._getMatchingRowsWithRetries(
      self._models, {'job_id' : jobID},
      [self._models.pubToDBNameDict[f]
       for f in self._models.getUpdateCountersNamedTuple._fields])

    # Return the results as a list of namedtuples
    return [self._models.getUpdateCountersNamedTuple._make(r) for r in rows]

Response:
Returns info on all of the models that are in already in the models
    table for a given job.


Instruction:
Write a Python 3 script for
updating the results string and numRecords fields of a model.

Response:
def modelUpdateResults(self, modelID, results=None, metricValue =None,
                         numRecords=None):
    """ Update the results string, and/or num_records fields of
    a model. This will fail if the model does not currently belong to this
    client (connection_id doesn't match).

    Parameters:
    ----------------------------------------------------------------
    modelID:      model ID of model to modify
    results:      new results, or None to ignore
    metricValue:  the value of the metric being optimized, or None to ignore
    numRecords:   new numRecords, or None to ignore
    """

    assignmentExpressions = ['_eng_last_update_time=UTC_TIMESTAMP()',
                             'update_counter=update_counter+1']
    assignmentValues = []

    if results is not None:
      assignmentExpressions.append('results=%s')
      assignmentValues.append(results)

    if numRecords is not None:
      assignmentExpressions.append('num_records=%s')
      assignmentValues.append(numRecords)

    # NOTE1: (metricValue==metricValue) tests for Nan
    # NOTE2: metricValue is being passed as numpy.float64
    if metricValue is not None and (metricValue==metricValue):
      assignmentExpressions.append('optimized_metric=%s')
      assignmentValues.append(float(metricValue))

    query = 'UPDATE %s SET %s ' \
            '          WHERE model_id=%%s and _eng_worker_conn_id=%%s' \
                % (self.modelsTableName, ','.join(assignmentExpressions))
    sqlParams = assignmentValues + [modelID, self._connectionID]

    # Get a database connection and cursor
    with ConnectionFactory.get() as conn:
      numRowsAffected = conn.cursor.execute(query, sqlParams)

    if numRowsAffected != 1:
      raise InvalidConnectionException(
        ("Tried to update the info of modelID=%r using connectionID=%r, but "
         "this model belongs to some other worker or modelID not found; "
         "numRowsAffected=%r") % (modelID,self._connectionID, numRowsAffected,))


Instruction:
Explain what the following Python 3 code does
def modelSetCompleted(self, modelID, completionReason, completionMsg,
                        cpuTime=0, useConnectionID=True):
    """ Mark a model as completed, with the given completionReason and
    completionMsg. This will fail if the model does not currently belong to this
    client (connection_id doesn't match).

    Parameters:
    ----------------------------------------------------------------
    modelID:             model ID of model to modify
    completionReason:    completionReason string
    completionMsg:       completionMsg string
    cpuTime:             amount of CPU time spent on this model
    useConnectionID:     True if the connection id of the calling function
                          must be the same as the connection that created the
                          job. Set to True for hypersearch workers, which use
                          this mechanism for orphaned model detection.
    """
    if completionMsg is None:
      completionMsg = ''

    query = 'UPDATE %s SET status=%%s, ' \
              '            completion_reason=%%s, ' \
              '            completion_msg=%%s, ' \
              '            end_time=UTC_TIMESTAMP(), ' \
              '            cpu_time=%%s, ' \
              '            _eng_last_update_time=UTC_TIMESTAMP(), ' \
              '            update_counter=update_counter+1 ' \
              '        WHERE model_id=%%s' \
              % (self.modelsTableName,)
    sqlParams = [self.STATUS_COMPLETED, completionReason, completionMsg,
                 cpuTime, modelID]

    if useConnectionID:
      query += " AND _eng_worker_conn_id=%s"
      sqlParams.append(self._connectionID)

    with ConnectionFactory.get() as conn:
      numRowsAffected = conn.cursor.execute(query, sqlParams)

    if numRowsAffected != 1:
      raise InvalidConnectionException(
        ("Tried to set modelID=%r using connectionID=%r, but this model "
         "belongs to some other worker or modelID not found; "
         "numRowsAffected=%r") % (modelID, self._connectionID, numRowsAffected))

Response: Mark a model as completed.


Instruction:
Write a Python 3 script to
adopt next orphaned model in the models table.

Response:
def modelAdoptNextOrphan(self, jobId, maxUpdateInterval):
    """ Look through the models table for an orphaned model, which is a model
    that is not completed yet, whose _eng_last_update_time is more than
    maxUpdateInterval seconds ago.

    If one is found, change its _eng_worker_conn_id to the current worker's
    and return the model id.

    Parameters:
    ----------------------------------------------------------------
    retval:    modelId of the model we adopted, or None if none found
    """

    @g_retrySQL
    def findCandidateModelWithRetries():
      modelID = None
      with ConnectionFactory.get() as conn:
        # TODO: may need a table index on job_id/status for speed
        query = 'SELECT model_id FROM %s ' \
                '   WHERE  status=%%s ' \
                '          AND job_id=%%s ' \
                '          AND TIMESTAMPDIFF(SECOND, ' \
                '                            _eng_last_update_time, ' \
                '                            UTC_TIMESTAMP()) > %%s ' \
                '   LIMIT 1 ' \
                % (self.modelsTableName,)
        sqlParams = [self.STATUS_RUNNING, jobId, maxUpdateInterval]
        numRows = conn.cursor.execute(query, sqlParams)
        rows = conn.cursor.fetchall()

      assert numRows <= 1, "Unexpected numRows: %r" % numRows
      if numRows == 1:
        (modelID,) = rows[0]

      return modelID

    @g_retrySQL
    def adoptModelWithRetries(modelID):
      adopted = False
      with ConnectionFactory.get() as conn:
        query = 'UPDATE %s SET _eng_worker_conn_id=%%s, ' \
                  '            _eng_last_update_time=UTC_TIMESTAMP() ' \
                  '        WHERE model_id=%%s ' \
                  '              AND status=%%s' \
                  '              AND TIMESTAMPDIFF(SECOND, ' \
                  '                                _eng_last_update_time, ' \
                  '                                UTC_TIMESTAMP()) > %%s ' \
                  '        LIMIT 1 ' \
                  % (self.modelsTableName,)
        sqlParams = [self._connectionID, modelID, self.STATUS_RUNNING,
                     maxUpdateInterval]
        numRowsAffected = conn.cursor.execute(query, sqlParams)

        assert numRowsAffected <= 1, 'Unexpected numRowsAffected=%r' % (
          numRowsAffected,)

        if numRowsAffected == 1:
          adopted = True
        else:
          # Discern between transient failure during update and someone else
          # claiming this model
          (status, connectionID) = self._getOneMatchingRowNoRetries(
            self._models, conn, {'model_id':modelID},
            ['status', '_eng_worker_conn_id'])
          adopted = (status == self.STATUS_RUNNING and
                     connectionID == self._connectionID)
      return adopted

    adoptedModelID = None
    while True:
      modelID = findCandidateModelWithRetries()
      if modelID is None:
        break
      if adoptModelWithRetries(modelID):
        adoptedModelID = modelID
        break

    return adoptedModelID


Instruction:
Can you write a function in Python 3 where it
profiles the performance of a SpatialPooler

Response:
def profileSP(spClass, spDim, nRuns):
  """
  profiling performance of SpatialPooler (SP)
  using the python cProfile module and ordered by cumulative time,
  see how to run on command-line above.

  @param spClass implementation of SP (cpp, py, ..)
  @param spDim number of columns in SP (in 1D, for 2D see colDim in code)
  @param nRuns number of calls of the profiled code (epochs)
  """
  # you can change dimensionality here, eg to 2D
  inDim = [10000, 1, 1]
  colDim = [spDim, 1, 1]

  # create SP instance to measure
  # changing the params here affects the performance
  sp = spClass(
        inputDimensions=inDim,
        columnDimensions=colDim,
        potentialRadius=3,
        potentialPct=0.5,
        globalInhibition=False,
        localAreaDensity=-1.0,
        numActiveColumnsPerInhArea=3,
        stimulusThreshold=1,
        synPermInactiveDec=0.01,
        synPermActiveInc=0.1,
        synPermConnected=0.10,
        minPctOverlapDutyCycle=0.1,
        dutyCyclePeriod=10,
        boostStrength=10.0,
        seed=42,
        spVerbosity=0)

  # generate input data
  dataDim = inDim
  dataDim.append(nRuns)
  data = numpy.random.randint(0, 2, dataDim).astype('float32')

  for i in xrange(nRuns):
    # new data every time, this is the worst case performance
    # real performance would be better, as the input data would not be completely random
    d = data[:,:,:,i]
    activeArray = numpy.zeros(colDim)

    # the actual function to profile!
    sp.compute(d, True, activeArray)


Instruction:
Explain what the following Python 3 code does
def getSpec(cls):
    ns = dict(
        description=KNNClassifierRegion.__doc__,
        singleNodeOnly=True,
        inputs=dict(
          categoryIn=dict(
            description='Vector of zero or more category indices for this input'
                         'sample. -1 implies no category.',
            dataType='Real32',
            count=0,
            required=True,
            regionLevel=True,
            isDefaultInput=False,
            requireSplitterMap=False),

          bottomUpIn=dict(
            description='Belief values over children\'s groups',
            dataType='Real32',
            count=0,
            required=True,
            regionLevel=False,
            isDefaultInput=True,
            requireSplitterMap=False),

          partitionIn=dict(
            description='Partition ID of the input sample',
            dataType='Real32',
            count=0,
            required=True,
            regionLevel=True,
            isDefaultInput=False,
            requireSplitterMap=False),

          auxDataIn=dict(
            description='Auxiliary data from the sensor',
            dataType='Real32',
            count=0,
            required=False,
            regionLevel=True,
            isDefaultInput=False,
            requireSplitterMap=False)
        ),

        outputs=dict(
          categoriesOut=dict(
          description='A vector representing, for each category '
                      'index, the likelihood that the input to the node belongs '
                      'to that category based on the number of neighbors of '
                      'that category that are among the nearest K.',
          dataType='Real32',
          count=0,
          regionLevel=True,
          isDefaultOutput=True),

          bestPrototypeIndices=dict(
          description='A vector that lists, in descending order of '
                      'the match, the positions of the prototypes '
                      'that best match the input pattern.',
          dataType='Real32',
          count=0,
          regionLevel=True,
          isDefaultOutput=False),

          categoryProbabilitiesOut=dict(
          description='A vector representing, for each category '
                      'index, the probability that the input to the node belongs '
                      'to that category based on the distance to the nearest '
                      'neighbor of each category.',
          dataType='Real32',
          count=0,
          regionLevel=True,
          isDefaultOutput=True),

        ),

        parameters=dict(
          learningMode=dict(
            description='Boolean (0/1) indicating whether or not a region '
                        'is in learning mode.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=1,
            accessMode='ReadWrite'),

          inferenceMode=dict(
            description='Boolean (0/1) indicating whether or not a region '
                        'is in inference mode.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='ReadWrite'),

          acceptanceProbability=dict(
            description='During learning, inputs are learned with '
                        'probability equal to this parameter. '
                        'If set to 1.0, the default, '
                        'all inputs will be considered '
                        '(subject to other tests).',
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=1.0,
            #accessMode='Create'),
            accessMode='ReadWrite'), # and Create too

          confusion=dict(
            description='Confusion matrix accumulated during inference. '
                        'Reset with reset(). This is available to Python '
                        'client code only.',
            dataType='Handle',
            count=2,
            constraints='',
            defaultValue=None,
            accessMode='Read'),

          activeOutputCount=dict(
            description='The number of active elements in the '
                        '"categoriesOut" output.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0,
            accessMode='Read'),

          categoryCount=dict(
            description='An integer indicating the number of '
                        'categories that have been learned',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=None,
            accessMode='Read'),

          patternCount=dict(
            description='Number of patterns learned by the classifier.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=None,
            accessMode='Read'),

          patternMatrix=dict(
            description='The actual patterns learned by the classifier, '
                        'returned as a matrix.',
            dataType='Handle',
            count=1,
            constraints='',
            defaultValue=None,
            accessMode='Read'),

          k=dict(
            description='The number of nearest neighbors to use '
                        'during inference.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=1,
            accessMode='Create'),

          maxCategoryCount=dict(
            description='The maximal number of categories the '
                        'classifier will distinguish between.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=2,
            accessMode='Create'),

          distanceNorm=dict(
            description='The norm to use for a distance metric (i.e., '
                        'the "p" in Lp-norm)',
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=2.0,
            accessMode='ReadWrite'),
            #accessMode='Create'),

          distanceMethod=dict(
            description='Method used to compute distances between inputs and'
              'prototypes. Possible options are norm, rawOverlap, '
              'pctOverlapOfLarger, and pctOverlapOfProto',
            dataType="Byte",
            count=0,
            constraints='enum: norm, rawOverlap, pctOverlapOfLarger, '
              'pctOverlapOfProto, pctOverlapOfInput',
            defaultValue='norm',
            accessMode='ReadWrite'),

          outputProbabilitiesByDist=dict(
            description='If True, categoryProbabilitiesOut is the probability of '
              'each category based on the distance to the nearest neighbor of '
              'each category. If False, categoryProbabilitiesOut is the '
              'percentage of neighbors among the top K that are of each category.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          distThreshold=dict(
            description='Distance Threshold.  If a pattern that '
                        'is less than distThreshold apart from '
                        'the input pattern already exists in the '
                        'KNN memory, then the input pattern is '
                        'not added to KNN memory.',
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=0.0,
            accessMode='ReadWrite'),

          inputThresh=dict(
            description='Input binarization threshold, used if '
                        '"doBinarization" is True.',
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=0.5,
            accessMode='Create'),

          doBinarization=dict(
            description='Whether or not to binarize the input vectors.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          useSparseMemory=dict(
            description='A boolean flag that determines whether or '
                        'not the KNNClassifier will use sparse Memory',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=1,
            accessMode='Create'),

          minSparsity=dict(
            description="If useSparseMemory is set, only vectors with sparsity"
                        " >= minSparsity will be stored during learning. A value"
                        " of 0.0 implies all vectors will be stored. A value of"
                        " 0.1 implies only vectors with at least 10% sparsity"
                        " will be stored",
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=0.0,
            accessMode='ReadWrite'),

          sparseThreshold=dict(
            description='If sparse memory is used, input variables '
                        'whose absolute value is less than this '
                        'threshold  will be stored as zero',
            dataType='Real32',
            count=1,
            constraints='',
            defaultValue=0.0,
            accessMode='Create'),

          relativeThreshold=dict(
            description='Whether to multiply sparseThreshold by max value '
                        ' in input',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          winnerCount=dict(
            description='Only this many elements of the input are '
                       'stored. All elements are stored if 0.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0,
            accessMode='Create'),

          doSphering=dict(
            description='A boolean indicating whether or not data should'
              'be "sphered" (i.e. each dimension should be normalized such'
              'that its mean and variance are zero and one, respectively.) This'
              ' sphering normalization would be performed after all training '
              'samples had been received but before inference was performed. '
              'The dimension-specific normalization constants would then '
              ' be applied to all future incoming vectors prior to performing '
              ' conventional NN inference.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          SVDSampleCount=dict(
            description='If not 0, carries out SVD transformation after '
                          'that many samples have been seen.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0,
            accessMode='Create'),

          SVDDimCount=dict(
            description='Number of dimensions to keep after SVD if greater '
                        'than 0. If set to -1 it is considered unspecified. '
                        'If set to 0 it is consider "adaptive" and the number '
                        'is chosen automatically.',
            dataType='Int32',
            count=1,
            constraints='',
            defaultValue=-1,
            accessMode='Create'),

          fractionOfMax=dict(
            description='The smallest singular value which is retained '
                        'as a fraction of the largest singular value. This is '
                        'used only if SVDDimCount==0 ("adaptive").',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0,
            accessMode='Create'),

          useAuxiliary=dict(
            description='Whether or not the classifier should use auxiliary '
                        'input data.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          justUseAuxiliary=dict(
            description='Whether or not the classifier should ONLUY use the '
                        'auxiliary input data.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=0,
            accessMode='Create'),

          verbosity=dict(
            description='An integer that controls the verbosity level, '
                        '0 means no verbose output, increasing integers '
                        'provide more verbosity.',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0 ,
            accessMode='ReadWrite'),

          keepAllDistances=dict(
            description='Whether to store all the protoScores in an array, '
                        'rather than just the ones for the last inference. '
                        'When this parameter is changed from True to False, '
                        'all the scores are discarded except for the most '
                        'recent one.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=None,
            accessMode='ReadWrite'),

          replaceDuplicates=dict(
            description='A boolean flag that determines whether or'
                        'not the KNNClassifier should replace duplicates'
                        'during learning. This should be on when online'
                        'learning.',
            dataType='UInt32',
            count=1,
            constraints='bool',
            defaultValue=None,
            accessMode='ReadWrite'),

          cellsPerCol=dict(
            description='If >= 1, we assume the input is organized into columns, '
                        'in the same manner as the temporal memory AND '
                        'whenever we store a new prototype, we only store the '
                        'start cell (first cell) in any column which is bursting.'
              'colum ',
            dataType='UInt32',
            count=1,
            constraints='',
            defaultValue=0,
            accessMode='Create'),

          maxStoredPatterns=dict(
            description='Limits the maximum number of the training patterns '
                        'stored. When KNN learns in a fixed capacity mode, '
                        'the unused patterns are deleted once the number '
                        'of stored patterns is greater than maxStoredPatterns'
                        'columns. [-1 is no limit] ',
            dataType='Int32',
            count=1,
            constraints='',
            defaultValue=-1,
            accessMode='Create'),
      ),
      commands=dict()
    )

    return ns

Response: Returns a dictionary of all the possible KNN classifiers for this region.


Instruction:
Write a Python 3 function for
initializing the ephemerals of the current object.

Response:
def _initEphemerals(self):
    """
    Initialize attributes that are not saved with the checkpoint.
    """
    self._firstComputeCall = True
    self._accuracy = None
    self._protoScores = None
    self._categoryDistances = None

    self._knn = knn_classifier.KNNClassifier(**self.knnParams)

    for x in ('_partitions', '_useAuxiliary', '_doSphering',
              '_scanInfo', '_protoScores'):
      if not hasattr(self, x):
        setattr(self, x, None)


Instruction:
Given the following Python 3 function, write the documentation
def getParameter(self, name, index=-1):
    if name == "patternCount":
      return self._knn._numPatterns
    elif name == "patternMatrix":
      return self._getPatternMatrix()
    elif name == "k":
      return self._knn.k
    elif name == "distanceNorm":
      return self._knn.distanceNorm
    elif name == "distanceMethod":
      return self._knn.distanceMethod
    elif name == "distThreshold":
      return self._knn.distThreshold
    elif name == "inputThresh":
      return self._knn.binarizationThreshold
    elif name == "doBinarization":
      return self._knn.doBinarization
    elif name == "useSparseMemory":
      return self._knn.useSparseMemory
    elif name == "sparseThreshold":
      return self._knn.sparseThreshold
    elif name == "winnerCount":
      return self._knn.numWinners
    elif name == "relativeThreshold":
      return self._knn.relativeThreshold
    elif name == "SVDSampleCount":
      v = self._knn.numSVDSamples
      return v if v is not None else 0
    elif name == "SVDDimCount":
      v = self._knn.numSVDDims
      return v if v is not None else 0
    elif name == "fractionOfMax":
      v = self._knn.fractionOfMax
      return v if v is not None else 0
    elif name == "useAuxiliary":
      return self._useAuxiliary
    elif name == "justUseAuxiliary":
      return self._justUseAuxiliary
    elif name == "doSphering":
      return self._doSphering
    elif name == "cellsPerCol":
      return self._knn.cellsPerCol
    elif name == "maxStoredPatterns":
      return self.maxStoredPatterns
    elif name == 'categoryRecencyList':
      return self._knn._categoryRecencyList
    else:
      # If any spec parameter name is the same as an attribute, this call
      # will get it automatically, e.g. self.learningMode
      return PyRegion.getParameter(self, name, index)

Response: Returns the value of the parameter with the given name.


Instruction:
Write a Python 3 function that can
enable the tap file for the specified path.

Response:
def enableTap(self, tapPath):
    """
    Begin writing output tap files.

    :param tapPath: (string) base name of the output tap files to write.
    """

    self._tapFileIn = open(tapPath + '.in', 'w')
    self._tapFileOut = open(tapPath + '.out', 'w')


Instruction:
How would you code a function in Python 3 to
disable writing of input tap files.

Response:
def disableTap(self):
    """
    Disable writing of output tap files.
    """

    if self._tapFileIn is not None:
      self._tapFileIn.close()
      self._tapFileIn = None
    if self._tapFileOut is not None:
      self._tapFileOut.close()
      self._tapFileOut = None


Instruction:
Here you have a function in Python 3, explain what it does
def handleLogInput(self, inputs):

    if self._tapFileIn is not None:
      for input in inputs:
        for k in range(len(input)):
          print >> self._tapFileIn, input[k],
        print >> self._tapFileIn

Response: Write input to output tap file.


Instruction:
Write a Python 3 script for
writing outputs to output tap file.

Response:
def handleLogOutput(self, output):
    """
    Write outputs to output tap file.

    :param outputs: (iter) some outputs.
    """
    #raise Exception('MULTI-LINE DUMMY\nMULTI-LINE DUMMY')
    if self._tapFileOut is not None:
      for k in range(len(output)):
        print >> self._tapFileOut, output[k],
      print >> self._tapFileOut


Instruction:
Write a Python 3 script to
store a training sample vector and associated category label.

Response:
def _storeSample(self, inputVector, trueCatIndex, partition=0):
    """
    Store a training sample and associated category label
    """

    # If this is the first sample, then allocate a numpy array
    # of the appropriate size in which to store all samples.
    if self._samples is None:
      self._samples = numpy.zeros((0, len(inputVector)), dtype=RealNumpyDType)
      assert self._labels is None
      self._labels = []

    # Add the sample vector and category lable
    self._samples = numpy.concatenate((self._samples, numpy.atleast_2d(inputVector)), axis=0)
    self._labels += [trueCatIndex]

    # Add the partition ID
    if self._partitions is None:
      self._partitions = []
    if partition is None:
      partition = 0
    self._partitions += [partition]


Instruction:
Here you have a function in Python 3, explain what it does
def compute(self, inputs, outputs):

    #raise Exception('MULTI-LINE DUMMY\nMULTI-LINE DUMMY')
    #For backward compatibility
    if self._useAuxiliary is None:
      self._useAuxiliary = False

    # If the first time being called, then print potential warning messsages
    if self._firstComputeCall:
      self._firstComputeCall = False
      if self._useAuxiliary:
        #print "\n  Auxiliary input stream from Image Sensor enabled."
        if self._justUseAuxiliary == True:
          print "  Warning: You have chosen to ignore the image data and instead just use the auxiliary data stream."

    # Format inputs
    #childInputs = [x.wvector(0) for x in inputs["bottomUpIn"]]
    #inputVector = numpy.concatenate([x.array() for x in childInputs])
    inputVector = inputs['bottomUpIn']

    # Look for auxiliary input
    if self._useAuxiliary==True:
      #auxVector = inputs['auxDataIn'][0].wvector(0).array()
      auxVector = inputs['auxDataIn']
      if auxVector.dtype != numpy.float32:
        raise RuntimeError, "KNNClassifierRegion expects numpy.float32 for the auxiliary data vector"
      if self._justUseAuxiliary == True:
        #inputVector = inputs['auxDataIn'][0].wvector(0).array()
        inputVector = inputs['auxDataIn']
      else:
        #inputVector = numpy.concatenate([inputVector, inputs['auxDataIn'][0].wvector(0).array()])
        inputVector = numpy.concatenate([inputVector, inputs['auxDataIn']])

    # Logging
    #self.handleLogInput(childInputs)
    self.handleLogInput([inputVector])

    # Read the category.
    assert "categoryIn" in inputs, "No linked category input."
    categories = inputs['categoryIn']

    # Read the partition ID.
    if "partitionIn" in inputs:
      assert len(inputs["partitionIn"]) == 1, "Must have exactly one link to partition input."
      partInput = inputs['partitionIn']
      assert len(partInput) == 1, "Partition input element count must be exactly 1."
      partition = int(partInput[0])
    else:
      partition = None

    # ---------------------------------------------------------------------
    # Inference (can be done simultaneously with learning)
    if self.inferenceMode:
      categoriesOut = outputs['categoriesOut']
      probabilitiesOut = outputs['categoryProbabilitiesOut']

      # If we are sphering, then apply normalization
      if self._doSphering:
        inputVector = (inputVector + self._normOffset) * self._normScale

      nPrototypes = 0
      if "bestPrototypeIndices" in outputs:
        #bestPrototypeIndicesOut = outputs["bestPrototypeIndices"].wvector()
        bestPrototypeIndicesOut = outputs["bestPrototypeIndices"]
        nPrototypes = len(bestPrototypeIndicesOut)

      winner, inference, protoScores, categoryDistances = \
                  self._knn.infer(inputVector, partitionId=partition)

      if not self.keepAllDistances:
        self._protoScores = protoScores
      else:
        # Keep all prototype scores in an array
        if self._protoScores is None:
          self._protoScores = numpy.zeros((1, protoScores.shape[0]),
                                          protoScores.dtype)
          self._protoScores[0,:] = protoScores#.reshape(1, protoScores.shape[0])
          self._protoScoreCount = 1
        else:
          if self._protoScoreCount == self._protoScores.shape[0]:
            # Double the size of the array
            newProtoScores = numpy.zeros((self._protoScores.shape[0] * 2,
                                          self._protoScores.shape[1]),
                                          self._protoScores.dtype)
            newProtoScores[:self._protoScores.shape[0],:] = self._protoScores
            self._protoScores = newProtoScores
          # Store the new prototype score
          self._protoScores[self._protoScoreCount,:] = protoScores
          self._protoScoreCount += 1
      self._categoryDistances = categoryDistances

      # --------------------------------------------------------------------
      # Compute the probability of each category
      if self.outputProbabilitiesByDist:
        scores = 1.0 - self._categoryDistances
      else:
        scores = inference

      # Probability is simply the scores/scores.sum()
      total = scores.sum()
      if total == 0:
        numScores = len(scores)
        probabilities = numpy.ones(numScores) / numScores
      else:
        probabilities = scores / total

      # -------------------------------------------------------------------
      # Fill the output vectors with our results
      nout = min(len(categoriesOut), len(inference))
      categoriesOut.fill(0)
      categoriesOut[0:nout] = inference[0:nout]

      probabilitiesOut.fill(0)
      probabilitiesOut[0:nout] = probabilities[0:nout]

      if self.verbosity >= 1:
        print "KNNRegion: categoriesOut: ", categoriesOut[0:nout]
        print "KNNRegion: probabilitiesOut: ", probabilitiesOut[0:nout]

      if self._scanInfo is not None:
        self._scanResults = [tuple(inference[:nout])]

      # Update the stored confusion matrix.
      for category in categories:
        if category >= 0:
          dims = max(int(category)+1, len(inference))
          oldDims = len(self.confusion)
          if oldDims < dims:
            confusion = numpy.zeros((dims, dims))
            confusion[0:oldDims, 0:oldDims] = self.confusion
            self.confusion = confusion
          self.confusion[inference.argmax(), int(category)] += 1

      # Calculate the best prototype indices
      if nPrototypes > 1:
        bestPrototypeIndicesOut.fill(0)
        if categoryDistances is not None:
          indices = categoryDistances.argsort()
          nout = min(len(indices), nPrototypes)
          bestPrototypeIndicesOut[0:nout] = indices[0:nout]
      elif nPrototypes == 1:
        if (categoryDistances is not None) and len(categoryDistances):
          bestPrototypeIndicesOut[0] = categoryDistances.argmin()
        else:
          bestPrototypeIndicesOut[0] = 0

      # Logging
      self.handleLogOutput(inference)

    # ---------------------------------------------------------------------
    # Learning mode
    if self.learningMode:
      if (self.acceptanceProbability < 1.0) and \
            (self._rgen.getReal64() > self.acceptanceProbability):
        pass

      else:
        # Accept the input
        for category in categories:
          if category >= 0:
            # category values of -1 are to be skipped (they are non-categories)
            if self._doSphering:
              # If we are sphering, then we can't provide the data to the KNN
              # library until we have computed per-dimension normalization
              # constants. So instead, we'll just store each training sample.
              self._storeSample(inputVector, category, partition)
            else:
              # Pass the raw training sample directly to the KNN library.
              self._knn.learn(inputVector, category, partition)

    self._epoch += 1

Response: This method computes the KNN classifier for one input sample.


Instruction:
Can you tell what is the following Python 3 function doing
def _finishLearning(self):
    if self._doSphering:
      self._finishSphering()

    self._knn.finishLearning()

    # Compute leave-one-out validation accuracy if
    # we actually received non-trivial partition info
    self._accuracy = None

Response: Finishes learning and returns None.


Instruction:
Can you tell what is the following Python 3 function doing
def _finishSphering(self):

    # If we are sphering our data, we need to compute the
    # per-dimension normalization constants
    # First normalize the means (to zero)
    self._normOffset = self._samples.mean(axis=0) * -1.0
    self._samples += self._normOffset
    # Now normalize the variances (to one).  However, we need to be
    # careful because the variance could conceivably be zero for one
    # or more dimensions.
    variance = self._samples.var(axis=0)
    variance[numpy.where(variance == 0.0)] = 1.0
    self._normScale  = 1.0 / numpy.sqrt(variance)
    self._samples *= self._normScale

    # Now feed each "sphered" sample into the SVM library
    for sampleIndex in range(len(self._labels)):
      self._knn.learn(self._samples[sampleIndex],
                      self._labels[sampleIndex],
                      self._partitions[sampleIndex])

Response: Finish sphering the SVM model.


Instruction:
Here you have a function in Python 3, explain what it does
def getOutputElementCount(self, name):
    if name == 'categoriesOut':
      return self.maxCategoryCount
    elif name == 'categoryProbabilitiesOut':
      return self.maxCategoryCount
    elif name == 'bestPrototypeIndices':
      return self._bestPrototypeIndexCount if self._bestPrototypeIndexCount else 0
    else:
      raise Exception('Unknown output: ' + name)

Response: Returns the number of elements in the output sequence.


Instruction:
How would you explain what the following Python 3 function does
def generateStats(filename, statsInfo, maxSamples = None, filters=[], cache=True):

  # Sanity checking
  if not isinstance(statsInfo, dict):
    raise RuntimeError("statsInfo must be a dict -- "
                       "found '%s' instead" % type(statsInfo))

  filename = resource_filename("nupic.datafiles", filename)

  if cache:
    statsFilename = getStatsFilename(filename, statsInfo, filters)
    # Use cached stats if found AND if it has the right data
    if os.path.exists(statsFilename):
      try:
        r = pickle.load(open(statsFilename, "rb"))
      except:
        # Ok to ignore errors -- we will just re-generate the file
        print "Warning: unable to load stats for %s -- " \
              "will regenerate" % filename
        r = dict()
      requestedKeys = set([s for s in statsInfo])
      availableKeys = set(r.keys())
      unavailableKeys = requestedKeys.difference(availableKeys)
      if len(unavailableKeys ) == 0:
        return r
      else:
        print "generateStats: re-generating stats file %s because " \
              "keys %s are not available" %  \
              (filename, str(unavailableKeys))
        os.remove(filename)

  print "Generating statistics for file '%s' with filters '%s'" % (filename, filters)
  sensor = RecordSensor()
  sensor.dataSource = FileRecordStream(filename)
  sensor.preEncodingFilters = filters

  # Convert collector description to collector object
  stats = []
  for field in statsInfo:
    # field = key from statsInfo
    if statsInfo[field] == "number":
      # This wants a field name e.g. consumption and the field type as the value
      statsInfo[field] = NumberStatsCollector()
    elif statsInfo[field] == "category":
      statsInfo[field] = CategoryStatsCollector()
    else:
      raise RuntimeError("Unknown stats type '%s' for field '%s'" % (statsInfo[field], field))

  # Now collect the stats
  if maxSamples is None:
    maxSamples = 500000
  for i in xrange(maxSamples):
    try:
      record = sensor.getNextRecord()
    except StopIteration:
      break
    for (name, collector) in statsInfo.items():
      collector.add(record[name])

  del sensor

  # Assemble the results and return
  r = dict()
  for (field, collector) in statsInfo.items():
    stats = collector.getStats()
    if field not in r:
      r[field] = stats
    else:
      r[field].update(stats)

  if cache:
    f = open(statsFilename, "wb")
    pickle.dump(r, f)
    f.close()
    # caller may need to know name of cached file
    r["_filename"] = statsFilename

  return r

Response: Generate requested statistics for a dataset.


Instruction:
Given the following Python 3 function, write the documentation
def getScalarMetricWithTimeOfDayAnomalyParams(metricData,
                                              minVal=None,
                                              maxVal=None,
                                              minResolution=None,
                                              tmImplementation = "cpp"):
  """
  Return a dict that can be used to create an anomaly model via
  :meth:`nupic.frameworks.opf.model_factory.ModelFactory.create`.

  Example:

  .. code-block:: python

    from nupic.frameworks.opf.model_factory import ModelFactory
    from nupic.frameworks.opf.common_models.cluster_params import (
      getScalarMetricWithTimeOfDayAnomalyParams)

    params = getScalarMetricWithTimeOfDayAnomalyParams(
      metricData=[0],
      tmImplementation="cpp",
      minVal=0.0,
      maxVal=100.0)

    model = ModelFactory.create(modelConfig=params["modelConfig"])
    model.enableLearning()
    model.enableInference(params["inferenceArgs"])

  :param metricData: numpy array of metric data. Used to calculate ``minVal``
    and ``maxVal`` if either is unspecified

  :param minVal: minimum value of metric. Used to set up encoders. If ``None``
    will be derived from ``metricData``.

  :param maxVal: maximum value of metric. Used to set up input encoders. If
    ``None`` will be derived from ``metricData``

  :param minResolution: minimum resolution of metric. Used to set up
    encoders.  If ``None``, will use default value of ``0.001``.

  :param tmImplementation: (string) specifying type of temporal memory
    implementation. Valid strings : ``["cpp", "tm_cpp"]``

  :returns: (dict) containing ``modelConfig`` and ``inferenceArgs`` top-level
    properties. The value of the ``modelConfig`` property is for passing to
    :meth:`~nupic.frameworks.opf.model_factory.ModelFactory.create` method as
    the ``modelConfig`` parameter. The ``inferenceArgs`` property is for passing
    to the resulting model's
    :meth:`~nupic.frameworks.opf.model.Model.enableInference` method as the
    ``inferenceArgs`` parameter.

    .. note:: The timestamp field corresponds to input ``c0``; the predicted
       field corresponds to input ``c1``.

  """
  # Default values
  if minResolution is None:
    minResolution = 0.001

  # Compute min and/or max from the data if not specified
  if minVal is None or maxVal is None:
    compMinVal, compMaxVal = _rangeGen(metricData)
    if minVal is None:
      minVal = compMinVal
    if maxVal is None:
      maxVal = compMaxVal

  # Handle the corner case where the incoming min and max are the same
  if minVal == maxVal:
    maxVal = minVal + 1

  # Load model parameters and update encoder params
  if (tmImplementation is "cpp"):
    paramFileRelativePath = os.path.join(
      "anomaly_params_random_encoder",
      "best_single_metric_anomaly_params_cpp.json")
  elif (tmImplementation is "tm_cpp"):
    paramFileRelativePath = os.path.join(
      "anomaly_params_random_encoder",
      "best_single_metric_anomaly_params_tm_cpp.json")
  else:
    raise ValueError("Invalid string for tmImplementation. Try cpp or tm_cpp")

  with resource_stream(__name__, paramFileRelativePath) as infile:
    paramSet = json.load(infile)

  _fixupRandomEncoderParams(paramSet, minVal, maxVal, minResolution)

  return paramSet

Response: Returns a dict that can be used to create an anomaly model for a set of time - of - day metric data.


Instruction:
Create a Python 3 function for
returning reasonable min and max values to use given the data.

Response:
def _rangeGen(data, std=1):
  """
  Return reasonable min/max values to use given the data.
  """
  dataStd = np.std(data)
  if dataStd == 0:
    dataStd = 1
  minval = np.min(data) -  std * dataStd
  maxval = np.max(data) +  std * dataStd
  return minval, maxval


Instruction:
How would you explain what the following Python 3 function does
def _fixupRandomEncoderParams(params, minVal, maxVal, minResolution):
  encodersDict = (
    params["modelConfig"]["modelParams"]["sensorParams"]["encoders"]
  )

  for encoder in encodersDict.itervalues():
    if encoder is not None:
      if encoder["type"] == "RandomDistributedScalarEncoder":
        resolution = max(minResolution,
                         (maxVal - minVal) / encoder.pop("numBuckets")
                        )
        encodersDict["c1"]["resolution"] = resolution

Response: Given model params figure out the correct parameters for RandomDistributed encoder. Modifies params in place.


Instruction:
Write a Python 3 function for
initializing the TemporalMemory state and returns it.

Response:
def read(cls, proto):
    """
    Intercepts TemporalMemory deserialization request in order to initialize
    `TemporalMemoryMonitorMixin` state

    @param proto (DynamicStructBuilder) Proto object

    @return (TemporalMemory) TemporalMemory shim instance
    """
    tm = super(TemporalMemoryMonitorMixin, cls).read(proto)

    # initialize `TemporalMemoryMonitorMixin` attributes
    tm.mmName = None
    tm._mmTraces = None
    tm._mmData = None
    tm.mmClearHistory()
    tm._mmResetActive = True
    return tm


Instruction:
How would you explain what the following Python 3 function does
def read(cls, proto):
    tm = super(TMShimMixin, cls).read(proto)
    tm.infActiveState = {"t": None}
    return tm

Response: Deserializes the object into a TemporalMemory shim instance.


Instruction:
Given the following Python 3 function, write the documentation
def topDownCompute(self, topDownIn=None):
    output = numpy.zeros(self.numberOfColumns())
    columns = [self.columnForCell(idx) for idx in self.getPredictiveCells()]
    output[columns] = 1
    return output

Response: Top - down compute - generate expected input given output of the TM


Instruction:
Make a summary of the following Python 3 code
def read(cls, proto):
    tm = super(MonitoredTMShim, cls).read(proto)
    tm.infActiveState = {"t": None}
    return tm

Response: Deserializes the object into a MonitoredTMShim instance.


Instruction:
Here you have a function in Python 3, explain what it does
def compute(self, bottomUpInput, enableLearn, computeInfOutput=None):
    super(MonitoredTMShim, self).compute(set(bottomUpInput.nonzero()[0]),
                                         learn=enableLearn)
    numberOfCells = self.numberOfCells()

    activeState = numpy.zeros(numberOfCells)
    activeState[self.getActiveCells()] = 1
    self.infActiveState["t"] = activeState

    output = numpy.zeros(numberOfCells)
    output[self.getPredictiveCells() + self.getActiveCells()] = 1
    return output

Response: This method computes the current state of the MonitoredTMShim.


Instruction:
Can you generate the documentation for the following Python 3 function
def pickByDistribution(distribution, r=None):

  if r is None:
    r = random

  x = r.uniform(0, sum(distribution))
  for i, d in enumerate(distribution):
    if x <= d:
      return i
    x -= d

Response: Pick a value according to the provided distribution.


Instruction:
Write a Python 3 function for
returning an array of length size and type dtype that is everywhere 0, except in the index in pos. :param pos: (int) specifies the position of the one entry that will be set. :param size: (int) The total size of the array to be returned. :param dtype: The element type (compatible with NumPy array()) of the array to be returned. :returns: (list) of length ``size`` and element type ``dtype``.

Response:
def Indicator(pos, size, dtype):
  """
  Returns an array of length size and type dtype that is everywhere 0,
  except in the index in pos.

  :param pos: (int) specifies the position of the one entry that will be set.
  :param size: (int) The total size of the array to be returned.
  :param dtype: The element type (compatible with NumPy array())
         of the array to be returned.
  :returns: (list) of length ``size`` and element type ``dtype``.
  """
  x = numpy.zeros(size, dtype=dtype)
  x[pos] = 1
  return x


Instruction:
Write a Python 3 script to
get tuple (actually a generator) of indices where the max value of array x occurs. Requires that x have a max() method, as x.max() (in the case of NumPy) is much faster than max(x). For a simpler, faster argmax when there is only a single maximum entry, or when knowing only the first index where the maximum occurs, call argmax() on a NumPy array. :param x: Any sequence that has a max() method. :returns: Generator with the indices where the max value occurs.

Response:
def MultiArgMax(x):
  """
  Get tuple (actually a generator) of indices where the max value of
  array x occurs. Requires that x have a max() method, as x.max()
  (in the case of NumPy) is much faster than max(x).
  For a simpler, faster argmax when there is only a single maximum entry,
  or when knowing only the first index where the maximum occurs,
  call argmax() on a NumPy array.

  :param x: Any sequence that has a max() method.
  :returns: Generator with the indices where the max value occurs.
  """
  m = x.max()
  return (i for i, v in enumerate(x) if v == m)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def Any(sequence):
  return bool(reduce(lambda x, y: x or y, sequence, False))

Response: Tests much faster than bool.


Instruction:
Here you have a function in Python 3, explain what it does
def All(sequence):
  return bool(reduce(lambda x, y: x and y, sequence, True))

Response: Returns True if all elements of the given sequence satisfy True and False otherwise.


Instruction:
Write a Python 3 function that can
return an array of length size and type dtype that is everywhere 0, except in the indices listed in sequence pos. :param pos: A single integer or sequence of integers that specify the position of ones to be set. :param size: The total size of the array to be returned. :param dtype: The element type (compatible with NumPy array()) of the array to be returned. :returns: An array of length size and element type dtype.

Response:
def MultiIndicator(pos, size, dtype):
  """
  Returns an array of length size and type dtype that is everywhere 0,
  except in the indices listed in sequence pos.

  :param pos:   A single integer or sequence of integers that specify
         the position of ones to be set.
  :param size:  The total size of the array to be returned.
  :param dtype: The element type (compatible with NumPy array())
         of the array to be returned.
  :returns: An array of length size and element type dtype.
  """
  x = numpy.zeros(size, dtype=dtype)
  if hasattr(pos, '__iter__'):
    for i in pos: x[i] = 1
  else: x[pos] = 1
  return x


Instruction:
Can you tell what is the following Python 3 function doing
def Distribution(pos, size, counts, dtype):
  x = numpy.zeros(size, dtype=dtype)
  if hasattr(pos, '__iter__'):
    # calculate normalization constant
    total = 0
    for i in pos:
      total += counts[i]
    total = float(total)
    # set included positions to normalized probability
    for i in pos:
      x[i] = counts[i]/total
  # If we don't have a set of positions, assume there's only one position
  else: x[pos] = 1
  return x

Response: Returns an array of length size and type dtype that is everywhere 0.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def grow(self, rows, cols):
    if not self.hist_:
      self.hist_ = SparseMatrix(rows, cols)
      self.rowSums_ = numpy.zeros(rows, dtype=dtype)
      self.colSums_ = numpy.zeros(cols, dtype=dtype)
      self.hack_ = None
    else:
      oldRows = self.hist_.nRows()
      oldCols = self.hist_.nCols()
      nextRows = max(oldRows, rows)
      nextCols = max(oldCols, cols)
      if (oldRows < nextRows) or (oldCols < nextCols):
        self.hist_.resize(nextRows, nextCols)
        if oldRows < nextRows:
          oldSums = self.rowSums_
          self.rowSums_ = numpy.zeros(nextRows, dtype=dtype)
          self.rowSums_[0:len(oldSums)] = oldSums
          self.hack_ = None
        if oldCols < nextCols:
          oldSums = self.colSums_
          self.colSums_ = numpy.zeros(nextCols, dtype=dtype)
          self.colSums_[0:len(oldSums)] = oldSums
          self.hack_ = None

Response: Grow the histogram to have rows and cols columns.


Instruction:
Write a Python 3 function for
adding a distribution to a row.

Response:
def updateRow(self, row, distribution):
    """
    Add distribution to row row.
    Distribution should be an array of probabilities or counts.

    :param row:   Integer index of the row to add to.
                  May be larger than the current number of rows, in which case
                  the histogram grows.
    :param distribution: Array of length equal to the number of columns.
    """
    self.grow(row+1, len(distribution))
    self.hist_.axby(row, 1, 1, distribution)
    self.rowSums_[row] += distribution.sum()
    self.colSums_ += distribution
    self.hack_ = None


Instruction:
Can you generate a brief explanation for the following Python 3 code
def inferRowCompat(self, distribution):
    if self.hack_ is None:
      self.clean_outcpd()
    return self.hack_.vecMaxProd(distribution)

Response: Equivalent to the category inference of zeta1. TopLevel.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def clean_outcpd(self):
    m = self.hist_.toDense()
    for j in xrange(m.shape[1]): # For each column.
      cmax = m[:,j].max()
      if cmax:
        m[:,j] = numpy.array(m[:,j] == cmax, dtype=dtype)
    self.hack_ = SparseMatrix(0, self.hist_.nCols())
    for i in xrange(m.shape[0]):
      self.hack_.addRow(m[i,:])

Response: This function is used to clean outcpd on zeta1. TopLevelNode.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def importAndRunFunction(
    path,
    moduleName,
    funcName,
    **keywords
  ):
  """
  Run a named function specified by a filesystem path, module name
  and function name.

  Returns the value returned by the imported function.

  Use this when access is needed to code that has
  not been added to a package accessible from the ordinary Python
  path. Encapsulates the multiple lines usually needed to
  safely manipulate and restore the Python path.

  Parameters
  ----------
  path: filesystem path
  Path to the directory where the desired module is stored.
  This will be used to temporarily augment the Python path.

  moduleName: basestring
  Name of the module, without trailing extension, where the desired
  function is stored. This module should be in the directory specified
  with path.

  funcName: basestring
  Name of the function to import and call.

  keywords:
  Keyword arguments to be passed to the imported function.
  """
  import sys
  originalPath = sys.path
  try:
    augmentedPath = [path] + sys.path
    sys.path = augmentedPath
    func = getattr(__import__(moduleName, fromlist=[funcName]), funcName)
    sys.path = originalPath
  except:
    # Restore the original path in case of an exception.
    sys.path = originalPath
    raise
  return func(**keywords)

Response: Imports a named function from the specified module and runs it.


Instruction:
Create a Python 3 function for
transferring coincidences from one element to another element.

Response:
def transferCoincidences(network, fromElementName, toElementName):
  """
  Gets the coincidence matrix from one element and sets it on
  another element
  (using locked handles, a la nupic.bindings.research.lockHandle).

  TODO: Generalize to more node types, parameter name pairs, etc.

  Does not work across processes.
  """
  coincidenceHandle = getLockedHandle(
      runtimeElement=network.getElement(fromElementName),
      # TODO: Re-purpose for use with nodes other than PMXClassifierNode.
      expression="self._cd._W"
    )

  network.getElement(toElementName).setParameter("coincidencesAbove",
      coincidenceHandle)


Instruction:
Write a Python 3 script for
instancing method wrapper around compute.

Response:
def next(self, newValue):
    """Instance method wrapper around compute."""
    newAverage, self.slidingWindow, self.total = self.compute(
        self.slidingWindow, self.total, newValue, self.windowSize)
    return newAverage


Instruction:
Write a Python 3 script for
returning an appropriate metric module for the given metricSpec.

Response:
def getModule(metricSpec):
  """
  Factory method to return an appropriate :class:`MetricsIface` module.

  - ``rmse``: :class:`MetricRMSE`
  - ``nrmse``: :class:`MetricNRMSE`
  - ``aae``: :class:`MetricAAE`
  - ``acc``: :class:`MetricAccuracy`
  - ``avg_err``: :class:`MetricAveError`
  - ``trivial``: :class:`MetricTrivial`
  - ``two_gram``: :class:`MetricTwoGram`
  - ``moving_mean``: :class:`MetricMovingMean`
  - ``moving_mode``: :class:`MetricMovingMode`
  - ``neg_auc``: :class:`MetricNegAUC`
  - ``custom_error_metric``: :class:`CustomErrorMetric`
  - ``multiStep``: :class:`MetricMultiStep`
  - ``ms_aae``: :class:`MetricMultiStepAAE`
  - ``ms_avg_err``: :class:`MetricMultiStepAveError`
  - ``passThruPrediction``: :class:`MetricPassThruPrediction`
  - ``altMAPE``: :class:`MetricAltMAPE`
  - ``MAPE``: :class:`MetricMAPE`
  - ``multi``: :class:`MetricMulti`
  - ``negativeLogLikelihood``: :class:`MetricNegativeLogLikelihood`

  :param metricSpec: (:class:`MetricSpec`) metric to find module for.
         ``metricSpec.metric`` must be in the list above.

  :returns: (:class:`AggregateMetric`) an appropriate metric module
  """

  metricName = metricSpec.metric

  if metricName == 'rmse':
    return MetricRMSE(metricSpec)
  if metricName == 'nrmse':
    return MetricNRMSE(metricSpec)
  elif metricName == 'aae':
    return MetricAAE(metricSpec)
  elif metricName == 'acc':
    return MetricAccuracy(metricSpec)
  elif metricName == 'avg_err':
    return MetricAveError(metricSpec)
  elif metricName == 'trivial':
    return MetricTrivial(metricSpec)
  elif metricName == 'two_gram':
    return MetricTwoGram(metricSpec)
  elif metricName == 'moving_mean':
    return MetricMovingMean(metricSpec)
  elif metricName == 'moving_mode':
    return MetricMovingMode(metricSpec)
  elif metricName == 'neg_auc':
    return MetricNegAUC(metricSpec)
  elif metricName == 'custom_error_metric':
    return CustomErrorMetric(metricSpec)
  elif metricName == 'multiStep':
    return MetricMultiStep(metricSpec)
  elif metricName == 'multiStepProbability':
    return MetricMultiStepProbability(metricSpec)
  elif metricName == 'ms_aae':
    return MetricMultiStepAAE(metricSpec)
  elif metricName == 'ms_avg_err':
    return MetricMultiStepAveError(metricSpec)
  elif metricName == 'passThruPrediction':
    return MetricPassThruPrediction(metricSpec)
  elif metricName == 'altMAPE':
    return MetricAltMAPE(metricSpec)
  elif metricName == 'MAPE':
    return MetricMAPE(metricSpec)
  elif metricName == 'multi':
    return MetricMulti(metricSpec)
  elif metricName == 'negativeLogLikelihood':
    return MetricNegativeLogLikelihood(metricSpec)
  else:
    raise Exception("Unsupported metric type: %s" % metricName)


Instruction:
Given the following Python 3 function, write the documentation
def getLabel(self, inferenceType=None):
    result = []
    if inferenceType is not None:
      result.append(InferenceType.getLabel(inferenceType))
    result.append(self.inferenceElement)
    result.append(self.metric)

    params = self.params
    if params is not None:

      sortedParams= params.keys()
      sortedParams.sort()
      for param in sortedParams:
        # Don't include the customFuncSource - it is too long an unwieldy
        if param in ('customFuncSource', 'customFuncDef', 'customExpr'):
          continue
        value = params[param]
        if isinstance(value, str):
          result.extend(["%s='%s'"% (param, value)])
        else:
          result.extend(["%s=%s"% (param, value)])

    if self.field:
      result.append("field=%s"% (self.field) )

    return self._LABEL_SEPARATOR.join(result)

Response: Returns a unique label for a metricSpec.


Instruction:
Explain what the following Python 3 code does
def getInferenceTypeFromLabel(cls, label):
    infType, _, _= label.partition(cls._LABEL_SEPARATOR)

    if not InferenceType.validate(infType):
      return None

    return infType

Response: Extracts the PredictionKind from the given label.


Instruction:
How would you explain what the following Python 3 function does
def addInstance(self, groundTruth, prediction, record = None, result = None):
    self.value = self.avg(prediction)

Response: Compute and store metric value


Instruction:
Explain what the following Python 3 code does
def mostLikely(self, pred):
    if len(pred) == 1:
      return pred.keys()[0]

    mostLikelyOutcome = None
    maxProbability = 0

    for prediction, probability in pred.items():
      if probability > maxProbability:
        mostLikelyOutcome = prediction
        maxProbability = probability

    return mostLikelyOutcome

Response: Helper function to return a scalar value representing the most likely outcome given a probability distribution


Instruction:
Can you generate a brief explanation for the following Python 3 code
def expValue(self, pred):
    if len(pred) == 1:
      return pred.keys()[0]

    return sum([x*p for x,p in pred.items()])

Response:
Helper function to return a scalar value representing the expected
        value of a probability distribution.


Instruction:
Given the following Python 3 function, write the documentation
def getScalarNames(self, parentFieldName=''):
    names = []

    if self.encoders is not None:
      for (name, encoder, offset) in self.encoders:
        subNames = encoder.getScalarNames(parentFieldName=name)
        if parentFieldName != '':
          subNames = ['%s.%s' % (parentFieldName, name) for name in subNames]
        names.extend(subNames)
    else:
      if parentFieldName != '':
        names.append(parentFieldName)
      else:
        names.append(self.name)

    return names

Response: Returns the field names for each of the scalar values returned by this object.


Instruction:
Explain what the following Python 3 code does
def getDecoderOutputFieldTypes(self):
    if hasattr(self, '_flattenedFieldTypeList') and \
          self._flattenedFieldTypeList is not None:
      return self._flattenedFieldTypeList

    fieldTypes = []

    # NOTE: we take care of the composites, but leaf encoders must override
    #       this method and return a list of one field_meta.FieldMetaType.XXXX
    #       element corresponding to the encoder's decoder output field type
    for (name, encoder, offset) in self.encoders:
      subTypes = encoder.getDecoderOutputFieldTypes()
      fieldTypes.extend(subTypes)

    self._flattenedFieldTypeList = fieldTypes
    return fieldTypes

Response: Returns a list of field types corresponding to the elements in the decoder output field array.


Instruction:
Can you tell what is the following Python 3 function doing
def _getInputValue(self, obj, fieldName):
    if isinstance(obj, dict):
      if not fieldName in obj:
        knownFields = ", ".join(
          key for key in obj.keys() if not key.startswith("_")
        )
        raise ValueError(
          "Unknown field name '%s' in input record. Known fields are '%s'.\n"
          "This could be because input headers are mislabeled, or because "
          "input data rows do not contain a value for '%s'." % (
            fieldName, knownFields, fieldName
          )
        )
      return obj[fieldName]
    else:
      return getattr(obj, fieldName)

Response: Gets the value of a given field from the input record


Instruction:
Implement a function in Python 3 to
return a list of all sub - encoders in this encoder

Response:
def getEncoderList(self):
    """
    :return: a reference to each sub-encoder in this encoder. They are
             returned in the same order as they are for :meth:`.getScalarNames`
             and :meth:`.getScalars`.

    """
    if hasattr(self, '_flattenedEncoderList') and \
        self._flattenedEncoderList is not None:

      return self._flattenedEncoderList

    encoders = []

    if self.encoders is not None:
      for (name, encoder, offset) in self.encoders:
        subEncoders = encoder.getEncoderList()
        encoders.extend(subEncoders)
    else:
      encoders.append(self)

    self._flattenedEncoderList = encoders
    return encoders


Instruction:
How would you implement a function in Python 3 that
returns the scalar values for each sub - field of the inputData.

Response:
def getScalars(self, inputData):
    """
    Returns a numpy array containing the sub-field scalar value(s) for
    each sub-field of the ``inputData``. To get the associated field names for
    each of the scalar values, call :meth:`.getScalarNames()`.

    For a simple scalar encoder, the scalar value is simply the input unmodified.
    For category encoders, it is the scalar representing the category string
    that is passed in. For the datetime encoder, the scalar value is the
    the number of seconds since epoch.

    The intent of the scalar representation of a sub-field is to provide a
    baseline for measuring error differences. You can compare the scalar value
    of the inputData with the scalar value returned from :meth:`.topDownCompute`
    on a top-down representation to evaluate prediction accuracy, for example.

    :param inputData: The data from the source. This is typically an object with
                 members
    :return: array of scalar values
    """

    retVals = numpy.array([])

    if self.encoders is not None:
      for (name, encoder, offset) in self.encoders:
        values = encoder.getScalars(self._getInputValue(inputData, name))
        retVals = numpy.hstack((retVals, values))
    else:
      retVals = numpy.hstack((retVals, inputData))

    return retVals


Instruction:
How would you explain what the following Python 3 function does
def getEncodedValues(self, inputData):

    retVals = []

    if self.encoders is not None:
      for name, encoders, offset in self.encoders:
        values = encoders.getEncodedValues(self._getInputValue(inputData, name))

        if _isSequence(values):
          retVals.extend(values)
        else:
          retVals.append(values)
    else:
      if _isSequence(inputData):
        retVals.extend(inputData)
      else:
        retVals.append(inputData)

    return tuple(retVals)

Response: Returns the input data in the same format as is returned by the topDownCompute method.


Instruction:
Write a Python 3 function for
returning the indices of each sub - field bucket of the inputData.

Response:
def getBucketIndices(self, inputData):
    """
    Returns an array containing the sub-field bucket indices for each sub-field
    of the inputData. To get the associated field names for each of the buckets,
    call :meth:`.getScalarNames`.

    :param inputData: The data from the source. This is typically an object with
                 members.
    :return: array of bucket indices
    """

    retVals = []

    if self.encoders is not None:
      for (name, encoder, offset) in self.encoders:
        values = encoder.getBucketIndices(self._getInputValue(inputData, name))
        retVals.extend(values)
    else:
      assert False, "Should be implemented in base classes that are not " \
        "containers for other encoders"

    return retVals


Instruction:
Make a summary of the following Python 3 code
def scalarsToStr(self, scalarValues, scalarNames=None):

    if scalarNames is None:
      scalarNames = self.getScalarNames()

    desc = ''
    for (name, value) in zip(scalarNames, scalarValues):
      if len(desc) > 0:
        desc += ", %s:%.2f" % (name, value)
      else:
        desc += "%s:%.2f" % (name, value)

    return desc

Response: Returns a pretty print string representing the return values from the given scalar values.


Instruction:
Explain what the following Python 3 code does
def getFieldDescription(self, fieldName):

    # Find which field it's in
    description = self.getDescription() + [("end", self.getWidth())]
    for i in xrange(len(description)):
      (name, offset) = description[i]
      if (name == fieldName):
        break

    if i >= len(description)-1:
      raise RuntimeError("Field name %s not found in this encoder" % fieldName)

    # Return the offset and width
    return (offset, description[i+1][1] - offset)

Response: Returns the offset and length of a given field within the encoded output.


Instruction:
Make a summary of the following Python 3 code
def encodedBitDescription(self, bitOffset, formatted=False):

    # Find which field it's in
    (prevFieldName, prevFieldOffset) = (None, None)
    description = self.getDescription()
    for i in xrange(len(description)):
      (name, offset) = description[i]
      if formatted:
        offset = offset + i
        if bitOffset == offset-1:
          prevFieldName = "separator"
          prevFieldOffset = bitOffset
          break

      if bitOffset < offset:
        break
      (prevFieldName, prevFieldOffset) = (name, offset)

    # Return the field name and offset within the field
    # return (fieldName, bitOffset - fieldOffset)
    width = self.getDisplayWidth() if formatted else self.getWidth()

    if prevFieldOffset is None or bitOffset > self.getWidth():
      raise IndexError("Bit is outside of allowable range: [0 - %d]" % width)

    return (prevFieldName, bitOffset - prevFieldOffset)

Response: Return a description of the given bit in the encoded output.


Instruction:
Given the following Python 3 function, write the documentation
def decode(self, encoded, parentFieldName=''):

    fieldsDict = dict()
    fieldsOrder = []

    # What is the effective parent name?
    if parentFieldName == '':
      parentName = self.name
    else:
      parentName = "%s.%s" % (parentFieldName, self.name)

    if self.encoders is not None:
      # Merge decodings of all child encoders together
      for i in xrange(len(self.encoders)):

        # Get the encoder and the encoded output
        (name, encoder, offset) = self.encoders[i]
        if i < len(self.encoders)-1:
          nextOffset = self.encoders[i+1][2]
        else:
          nextOffset = self.width
        fieldOutput = encoded[offset:nextOffset]
        (subFieldsDict, subFieldsOrder) = encoder.decode(fieldOutput,
                                              parentFieldName=parentName)

        fieldsDict.update(subFieldsDict)
        fieldsOrder.extend(subFieldsOrder)

    return (fieldsDict, fieldsOrder)

Response: This method decodes the encoded output into a tuple of the name of the key and the value of each entry in the dictionary.


Instruction:
Explain what the following Python 3 code does
def decodedToStr(self, decodeResults):

    (fieldsDict, fieldsOrder) = decodeResults

    desc = ''
    for fieldName in fieldsOrder:
      (ranges, rangesStr) = fieldsDict[fieldName]
      if len(desc) > 0:
        desc += ", %s:" % (fieldName)
      else:
        desc += "%s:" % (fieldName)

      desc += "[%s]" % (rangesStr)

    return desc

Response:
Return a pretty print string representing the return value from
   .


Instruction:
How would you code a function in Python 3 to
return a list of namedtuples describing the inputs for each sub - field that correspond to the bucket indices passed in buckets.

Response:
def getBucketInfo(self, buckets):
    """
    Returns a list of :class:`.EncoderResult` namedtuples describing the inputs
    for each sub-field that correspond to the bucket indices passed in
    ``buckets``. To get the associated field names for each of the values, call
    :meth:`.getScalarNames`.

    :param buckets: The list of bucket indices, one for each sub-field encoder.
                   These bucket indices for example may have been retrieved
                   from the :meth:`.getBucketIndices` call.
    :return: A list of :class:`.EncoderResult`.
    """
    # Fall back topdown compute
    if self.encoders is None:
      raise RuntimeError("Must be implemented in sub-class")

    # Concatenate the results from bucketInfo on each child encoder
    retVals = []
    bucketOffset = 0
    for i in xrange(len(self.encoders)):
      (name, encoder, offset) = self.encoders[i]

      if encoder.encoders is not None:
        nextBucketOffset = bucketOffset + len(encoder.encoders)
      else:
        nextBucketOffset = bucketOffset + 1
      bucketIndices = buckets[bucketOffset:nextBucketOffset]
      values = encoder.getBucketInfo(bucketIndices)

      retVals.extend(values)

      bucketOffset = nextBucketOffset

    return retVals


Instruction:
Create a Python 3 function for
computing the top - down best guess inputs for each sub - field given the encoded output.

Response:
def topDownCompute(self, encoded):
    """
    Returns a list of :class:`.EncoderResult` namedtuples describing the
    top-down best guess inputs for each sub-field given the encoded output.
    These are the values which are most likely to generate the given encoded
    output. To get the associated field names for each of the values, call
    :meth:`.getScalarNames`.

    :param encoded: The encoded output. Typically received from the topDown
                    outputs from the spatial pooler just above us.

    :return: A list of :class:`.EncoderResult`
    """
    # Fallback topdown compute
    if self.encoders is None:
      raise RuntimeError("Must be implemented in sub-class")

    # Concatenate the results from topDownCompute on each child encoder
    retVals = []
    for i in xrange(len(self.encoders)):
      (name, encoder, offset) = self.encoders[i]

      if i < len(self.encoders)-1:
        nextOffset = self.encoders[i+1][2]
      else:
        nextOffset = self.width

      fieldOutput = encoded[offset:nextOffset]
      values = encoder.topDownCompute(fieldOutput)

      if _isSequence(values):
        retVals.extend(values)
      else:
        retVals.append(values)

    return retVals


Instruction:
Here you have a function in Python 3, explain what it does
def closenessScores(self, expValues, actValues, fractional=True):
    # Fallback closenss is a percentage match
    if self.encoders is None:
      err = abs(expValues[0] - actValues[0])
      if fractional:
        denom = max(expValues[0], actValues[0])
        if denom == 0:
          denom = 1.0
        closeness = 1.0 - float(err)/denom
        if closeness < 0:
          closeness = 0
      else:
        closeness = err

      return numpy.array([closeness])

    # Concatenate the results from closeness scores on each child encoder
    scalarIdx = 0
    retVals = numpy.array([])
    for (name, encoder, offset) in self.encoders:
      values = encoder.closenessScores(expValues[scalarIdx:], actValues[scalarIdx:],
                                       fractional=fractional)
      scalarIdx += len(values)
      retVals = numpy.hstack((retVals, values))

    return retVals

Response: Compute closeness scores between the expected and actual scalar values.