Instruction:
Can you create a Python 3 function that
sees Also : searchResponse

Response:
def search(self, queryType, query=None, vendorSpecific=None, **kwargs):
        """See Also: searchResponse()

        Args:
          queryType:
          query:
          vendorSpecific:
          **kwargs:

        Returns:

        """
        response = self.searchResponse(queryType, query, vendorSpecific, **kwargs)
        return self._read_dataone_type_response(response, 'ObjectList')


Instruction:
Here you have a function in Python 3, explain what it does
def queryResponse(self, queryEngine, query=None, vendorSpecific=None, **kwargs):
        return self.GET(
            ['query', queryEngine, query], headers=vendorSpecific, query=kwargs
        )

Response: Query the set of available data for a given query.


Instruction:
How would you code a function in Python 3 to
see Also : queryResponse

Response:
def query(self, queryEngine, query=None, vendorSpecific=None, **kwargs):
        """See Also: queryResponse()

        Args:
          queryEngine:
          query:
          vendorSpecific:
          **kwargs:

        Returns:

        """
        response = self.queryResponse(queryEngine, query, vendorSpecific, **kwargs)
        return self._read_stream_response(response)


Instruction:
Write a Python 3 function for
setting the rights holder for a user.

Response:
def setRightsHolderResponse(self, pid, userId, serialVersion, vendorSpecific=None):
        """CNAuthorization.setRightsHolder(session, pid, userId, serialVersion)

        → Identifier https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNAuthorization.setRightsHolder.

        Args:
          pid:
          userId:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {'userId': userId, 'serialVersion': str(serialVersion)}
        return self.PUT(['owner', pid], headers=vendorSpecific, fields=mmp_dict)


Instruction:
Given the following Python 3 function, write the documentation
def setRightsHolder(self, pid, userId, serialVersion, vendorSpecific=None):
        response = self.setRightsHolderResponse(
            pid, userId, serialVersion, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: See Also : setRightsHolderResponse


Instruction:
How would you implement a function in Python 3 that
sets the access policy for a specific resource.

Response:
def setAccessPolicyResponse(
        self, pid, accessPolicy, serialVersion, vendorSpecific=None
    ):
        """CNAuthorization.setAccessPolicy(session, pid, accessPolicy, serialVersion) →
        boolean https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNAuthorization.setAccessPolicy.

        Args:
          pid:
          accessPolicy:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {
            'serialVersion': str(serialVersion),
            'accessPolicy': ('accessPolicy.xml', accessPolicy.toxml('utf-8')),
        }
        return self.PUT(['accessRules', pid], fields=mmp_dict, headers=vendorSpecific)


Instruction:
Can you generate the documentation for the following Python 3 function
def setAccessPolicy(self, pid, accessPolicy, serialVersion, vendorSpecific=None):
        response = self.setAccessPolicyResponse(
            pid, accessPolicy, serialVersion, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: Sets the access policy of a specific object.


Instruction:
Can you generate the documentation for the following Python 3 function
def registerAccountResponse(self, person, vendorSpecific=None):
        mmp_dict = {'person': ('person.xml', person.toxml('utf-8'))}
        return self.POST('accounts', fields=mmp_dict, headers=vendorSpecific)

Response: CNIdentity. registerAccount - Creates a new CN identity and returns the response.


Instruction:
Can you implement a function in Python 3 that
sees Also : registerAccountResponse

Response:
def registerAccount(self, person, vendorSpecific=None):
        """See Also: registerAccountResponse()

        Args:
          person:
          vendorSpecific:

        Returns:

        """
        response = self.registerAccountResponse(person, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
Can you generate the documentation for the following Python 3 function
def updateAccountResponse(self, subject, person, vendorSpecific=None):
        mmp_dict = {'person': ('person.xml', person.toxml('utf-8'))}
        return self.PUT(['accounts', subject], fields=mmp_dict, headers=vendorSpecific)

Response: CNIdentity. updateAccount - Updates the NX - API for the given subject and person.


Instruction:
How would you code a function in Python 3 to
see Also : updateAccountResponse

Response:
def updateAccount(self, subject, person, vendorSpecific=None):
        """See Also: updateAccountResponse()

        Args:
          subject:
          person:
          vendorSpecific:

        Returns:

        """
        response = self.updateAccountResponse(subject, person, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
Can you generate the documentation for the following Python 3 function
def verifyAccount(self, subject, vendorSpecific=None):
        response = self.verifyAccountResponse(subject, vendorSpecific)
        return self._read_boolean_response(response)

Response: See Also : verifyAccountResponse


Instruction:
How would you explain what the following Python 3 function does
def getSubjectInfo(self, subject, vendorSpecific=None):
        response = self.getSubjectInfoResponse(subject, vendorSpecific)
        return self._read_dataone_type_response(response, 'SubjectInfo')

Response: See Also : getSubjectInfoResponse


Instruction:
Can you tell what is the following Python 3 function doing
def listSubjectsResponse(
        self, query, status=None, start=None, count=None, vendorSpecific=None
    ):
        """CNIdentity.listSubjects(session, query, status, start, count) → SubjectList
        https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNIdentity.listSubjects.

        Args:
          query:
          status:
          start:
          count:
          vendorSpecific:

        Returns:

        """
        url_query = {'status': status, 'start': start, 'count': count, 'query': query}
        return self.GET('accounts', query=url_query, headers=vendorSpecific)

Response: CNIdentity. listSubjects - Returns a list of Subjects for the given resource


Instruction:
Can you create a Python 3 function that
sees Also : listSubjectsResponse

Response:
def listSubjects(
        self, query, status=None, start=None, count=None, vendorSpecific=None
    ):
        """See Also: listSubjectsResponse()

        Args:
          query:
          status:
          start:
          count:
          vendorSpecific:

        Returns:

        """
        response = self.listSubjectsResponse(
            query, status, start, count, vendorSpecific
        )
        return self._read_dataone_type_response(response, 'SubjectInfo')


Instruction:
Given the following Python 3 function, write the documentation
def mapIdentityResponse(
        self, primarySubject, secondarySubject, vendorSpecific=None
    ):
        """CNIdentity.mapIdentity(session, subject) → boolean
        https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNIdentity.mapIdentity.

        Args:
          primarySubject:
          secondarySubject:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {
            'primarySubject': primarySubject.toxml('utf-8'),
            'secondarySubject': secondarySubject.toxml('utf-8'),
        }
        return self.POST(['accounts', 'map'], fields=mmp_dict, headers=vendorSpecific)

Response: This is a helper method that returns the response from the CN Identity API.


Instruction:
Here you have a function in Python 3, explain what it does
def mapIdentity(self, primarySubject, secondarySubject, vendorSpecific=None):
        response = self.mapIdentityResponse(
            primarySubject, secondarySubject, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: See Also : mapIdentityResponse


Instruction:
Here you have a function in Python 3, explain what it does
def removeMapIdentity(self, subject, vendorSpecific=None):
        response = self.removeMapIdentityResponse(subject, vendorSpecific)
        return self._read_boolean_response(response)

Response: See Also : removeMapIdentityResponse


Instruction:
How would you code a function in Python 3 to
see Also : denyMapIdentityResponse

Response:
def denyMapIdentity(self, subject, vendorSpecific=None):
        """See Also: denyMapIdentityResponse()

        Args:
          subject:
          vendorSpecific:

        Returns:

        """
        response = self.denyMapIdentityResponse(subject, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
How would you implement a function in Python 3 that
requests Map Identity Response

Response:
def requestMapIdentityResponse(self, subject, vendorSpecific=None):
        """CNIdentity.requestMapIdentity(session, subject) → boolean
        https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNIdentity.requestMapIdentity.

        Args:
          subject:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {'subject': subject.toxml('utf-8')}
        return self.POST('accounts', fields=mmp_dict, headers=vendorSpecific)


Instruction:
Explain what the following Python 3 code does
def requestMapIdentity(self, subject, vendorSpecific=None):
        response = self.requestMapIdentityResponse(subject, vendorSpecific)
        return self._read_boolean_response(response)

Response: See Also : requestMapIdentityResponse


Instruction:
How would you code a function in Python 3 to
see Also : confirmMapIdentityResponse

Response:
def confirmMapIdentity(self, subject, vendorSpecific=None):
        """See Also: confirmMapIdentityResponse()

        Args:
          subject:
          vendorSpecific:

        Returns:

        """
        response = self.confirmMapIdentityResponse(subject, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
Can you tell what is the following Python 3 function doing
def createGroupResponse(self, group, vendorSpecific=None):
        mmp_dict = {'group': ('group.xml', group.toxml('utf-8'))}
        return self.POST('groups', fields=mmp_dict, headers=vendorSpecific)

Response: CNIdentity. createGroup - Creates a new group


Instruction:
Can you generate a brief explanation for the following Python 3 code
def createGroup(self, group, vendorSpecific=None):
        response = self.createGroupResponse(group, vendorSpecific)
        return self._read_boolean_response(response)

Response: See Also : createGroupResponse


Instruction:
Write a Python 3 script for
updating the group response.

Response:
def updateGroupResponse(self, group, vendorSpecific=None):
        """CNIdentity.addGroupMembers(session, groupName, members) → boolean
        https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_APIs.html#CNIdentity.addGroupMembers.

        Args:
          group:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {'group': ('group.xml', group.toxml('utf-8'))}
        return self.PUT('groups', fields=mmp_dict, headers=vendorSpecific)


Instruction:
Can you create a Python 3 function that
sees Also : updateGroupResponse

Response:
def updateGroup(self, group, vendorSpecific=None):
        """See Also: updateGroupResponse()

        Args:
          group:
          vendorSpecific:

        Returns:

        """
        response = self.updateGroupResponse(group, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
Can you generate the documentation for the following Python 3 function
def setReplicationStatusResponse(
        self, pid, nodeRef, status, dataoneError=None, vendorSpecific=None
    ):
        """CNReplication.setReplicationStatus(session, pid, nodeRef, status, failure) →
        boolean https://releases.dataone.org/online/api-documentatio
        n-v2.0.1/apis/CN_APIs.html#CNReplication.setReplicationStatus.

        Args:
          pid:
          nodeRef:
          status:
          dataoneError:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {'nodeRef': nodeRef, 'status': status}  # .toxml('utf-8'),
        if dataoneError is not None:
            mmp_dict['failure'] = ('failure.xml', dataoneError.serialize_to_transport())
        return self.PUT(
            ['replicaNotifications', pid], fields=mmp_dict, headers=vendorSpecific
        )

Response: Set replication status for a specific resource.


Instruction:
Write a Python 3 function for
setting the replication status of a specific resource.

Response:
def setReplicationStatus(
        self, pid, nodeRef, status, dataoneError=None, vendorSpecific=None
    ):
        """See Also: setReplicationStatusResponse()

        Args:
          pid:
          nodeRef:
          status:
          dataoneError:
          vendorSpecific:

        Returns:

        """
        response = self.setReplicationStatusResponse(
            pid, nodeRef, status, dataoneError, vendorSpecific
        )
        return self._read_boolean_response(response)


Instruction:
Explain what the following Python 3 code does
def updateReplicationMetadataResponse(
        self, pid, replicaMetadata, serialVersion, vendorSpecific=None
    ):
        """CNReplication.updateReplicationMetadata(session, pid, replicaMetadata,
        serialVersion) → boolean https://releases.dataone.org/online/api-
        documentation-v2.0.1/apis/CN_AP Is.html#CNReplication.updateReplicationMetadata
        Not implemented.

        Args:
          pid:
          replicaMetadata:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {
            'replicaMetadata': ('replicaMetadata.xml', replicaMetadata.toxml('utf-8')),
            'serialVersion': str(serialVersion),
        }
        return self.PUT(
            ['replicaMetadata', pid], fields=mmp_dict, headers=vendorSpecific
        )

Response: Updates the replica metadata for a specific resource.


Instruction:
Explain what the following Python 3 code does
def updateReplicationMetadata(
        self, pid, replicaMetadata, serialVersion, vendorSpecific=None
    ):
        """See Also: updateReplicationMetadataResponse()

        Args:
          pid:
          replicaMetadata:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        response = self.updateReplicationMetadataResponse(
            pid, replicaMetadata, serialVersion, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: Update the replica metadata for a specific object.


Instruction:
Explain what the following Python 3 code does
def setReplicationPolicy(self, pid, policy, serialVersion, vendorSpecific=None):
        response = self.setReplicationPolicyResponse(
            pid, policy, serialVersion, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: Sets the replication policy for a specific object.


Instruction:
Can you tell what is the following Python 3 function doing
def isNodeAuthorizedResponse(self, targetNodeSubject, pid, vendorSpecific=None):
        query_dict = {'targetNodeSubject': targetNodeSubject}
        return self.GET(
            ['replicaAuthorizations', pid], query=query_dict, headers=vendorSpecific
        )

Response: CNReplication. isNodeAuthorized - Returns a boolean response if the user is authorized to replicate the given node.


Instruction:
Create a Python 3 function for
seeing Also : isNodeAuthorizedResponse

Response:
def isNodeAuthorized(self, targetNodeSubject, pid, vendorSpecific=None):
        """See Also: isNodeAuthorizedResponse()

        Args:
          targetNodeSubject:
          pid:
          vendorSpecific:

        Returns:

        """
        response = self.isNodeAuthorizedResponse(targetNodeSubject, pid, vendorSpecific)
        return self._read_boolean_401_response(response)


Instruction:
Make a summary of the following Python 3 code
def deleteReplicationMetadataResponse(
        self, pid, nodeId, serialVersion, vendorSpecific=None
    ):
        """CNReplication.deleteReplicationMetadata(session, pid, policy, serialVersion)

        → boolean https://releases.dataone.org/online/api-docume
        ntation-v2.0.1/apis/CN_APIs.html#CNReplication.deleteReplicationMetadat a.

        Args:
          pid:
          nodeId:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        mmp_dict = {'nodeId': nodeId, 'serialVersion': str(serialVersion)}
        return self.PUT(
            ['removeReplicaMetadata', pid], fields=mmp_dict, headers=vendorSpecific
        )

Response: Delete replication metadata response.


Instruction:
Explain what the following Python 3 code does
def deleteReplicationMetadata(
        self, pid, nodeId, serialVersion, vendorSpecific=None
    ):
        """See Also: deleteReplicationMetadataResponse()

        Args:
          pid:
          nodeId:
          serialVersion:
          vendorSpecific:

        Returns:

        """
        response = self.deleteReplicationMetadataResponse(
            pid, nodeId, serialVersion, vendorSpecific
        )
        return self._read_boolean_response(response)

Response: Delete the replication metadata for a node.


Instruction:
How would you explain what the following Python 3 function does
def updateNodeCapabilitiesResponse(self, nodeId, node, vendorSpecific=None):
        mmp_dict = {'node': ('node.xml', node.toxml('utf-8'))}
        return self.PUT(['node', nodeId], fields=mmp_dict, headers=vendorSpecific)

Response: CNRegister. updateNodeCapabilities - Updates node capabilities


Instruction:
Can you create a Python 3 function that
sees Also : updateNodeCapabilitiesResponse

Response:
def updateNodeCapabilities(self, nodeId, node, vendorSpecific=None):
        """See Also: updateNodeCapabilitiesResponse()

        Args:
          nodeId:
          node:
          vendorSpecific:

        Returns:

        """
        response = self.updateNodeCapabilitiesResponse(nodeId, node, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
How would you implement a function in Python 3 that
sees Also : registerResponse

Response:
def register(self, node, vendorSpecific=None):
        """See Also: registerResponse()

        Args:
          node:
          vendorSpecific:

        Returns:

        """
        response = self.registerResponse(node, vendorSpecific)
        return self._read_boolean_response(response)


Instruction:
How would you explain what the following Python 3 function does
def valid(self, instance, schema):
        try:
            jsonschema.validate(instance, schema)
        except jsonschema.exceptions.ValidationError as ex:
            self.stderr.write("    VALIDATION ERROR: {}".format(instance['name'] if 'name' in instance else ''))
            self.stderr.write("        path:       {}".format(ex.path))
            self.stderr.write("        message:    {}".format(ex.message))
            self.stderr.write("        validator:  {}".format(ex.validator))
            self.stderr.write("        val. value: {}".format(ex.validator_value))
            return False

        try:
            # Check that default values fit field schema.
            for field in ['input', 'output', 'schema']:
                for schema, _, path in iterate_schema({}, instance.get(field, {})):
                    if 'default' in schema:
                        validate_schema({schema['name']: schema['default']}, [schema])
        except ValidationError:
            self.stderr.write("    VALIDATION ERROR: {}".format(instance['name']))
            self.stderr.write("        Default value of field '{}' is not valid.". format(path))
            return False

        return True

Response: Validate the given instance.


Instruction:
Create a Python 3 function to
find descriptor schemas in given path.

Response:
def find_descriptor_schemas(self, schema_file):
        """Find descriptor schemas in given path."""
        if not schema_file.lower().endswith(('.yml', '.yaml')):
            return []

        with open(schema_file) as fn:
            schemas = yaml.load(fn, Loader=yaml.FullLoader)
        if not schemas:
            self.stderr.write("Could not read YAML file {}".format(schema_file))
            return []

        descriptor_schemas = []
        for schema in schemas:
            if 'schema' not in schema:
                continue

            descriptor_schemas.append(schema)

        return descriptor_schemas


Instruction:
Implement a Python 3 function for
finding schemas in a directory that match filters.

Response:
def find_schemas(self, schema_path, schema_type=SCHEMA_TYPE_PROCESS, verbosity=1):
        """Find schemas in packages that match filters."""
        schema_matches = []

        if not os.path.isdir(schema_path):
            if verbosity > 0:
                self.stdout.write("Invalid path {}".format(schema_path))
            return

        if schema_type not in [SCHEMA_TYPE_PROCESS, SCHEMA_TYPE_DESCRIPTOR]:
            raise ValueError('Invalid schema type')

        for root, _, files in os.walk(schema_path):
            for schema_file in [os.path.join(root, fn) for fn in files]:
                schemas = None
                if schema_type == SCHEMA_TYPE_DESCRIPTOR:
                    # Discover descriptors.
                    schemas = self.find_descriptor_schemas(schema_file)
                elif schema_type == SCHEMA_TYPE_PROCESS:
                    # Perform process discovery for all supported execution engines.
                    schemas = []
                    for execution_engine in manager.execution_engines.values():
                        schemas.extend(execution_engine.discover_process(schema_file))

                for schema in schemas:
                    schema_matches.append(schema)

        return schema_matches


Instruction:
Write a Python 3 function that can
read and register processes.

Response:
def register_processes(self, process_schemas, user, force=False, verbosity=1):
        """Read and register processors."""
        log_processors = []
        log_templates = []

        for p in process_schemas:
            # TODO: Remove this when all processes are migrated to the
            #       new syntax.
            if 'flow_collection' in p:
                if 'entity' in p:
                    self.stderr.write(
                        "Skip processor {}: only one of 'flow_collection' and 'entity' fields "
                        "allowed".format(p['slug'])
                    )
                    continue

                p['entity'] = {'type': p.pop('flow_collection')}

            if p['type'][-1] != ':':
                p['type'] += ':'

            if 'category' in p and not p['category'].endswith(':'):
                p['category'] += ':'

            for field in ['input', 'output']:
                for schema, _, _ in iterate_schema({}, p[field] if field in p else {}):
                    if not schema['type'][-1].endswith(':'):
                        schema['type'] += ':'
            # TODO: Check if schemas validate with our JSON meta schema and Processor model docs.

            if not self.valid(p, PROCESSOR_SCHEMA):
                continue

            if 'entity' in p:
                if 'type' not in p['entity']:
                    self.stderr.write(
                        "Skip process {}: 'entity.type' required if 'entity' defined".format(p['slug'])
                    )
                    continue

                p['entity_type'] = p['entity']['type']
                p['entity_descriptor_schema'] = p['entity'].get('descriptor_schema', p['entity_type'])
                p['entity_input'] = p['entity'].get('input', None)
                p.pop('entity')

                if not DescriptorSchema.objects.filter(slug=p['entity_descriptor_schema']).exists():
                    self.stderr.write(
                        "Skip processor {}: Unknown descriptor schema '{}' used in 'entity' "
                        "field.".format(p['slug'], p['entity_descriptor_schema'])
                    )
                    continue

            if 'persistence' in p:
                persistence_mapping = {
                    'RAW': Process.PERSISTENCE_RAW,
                    'CACHED': Process.PERSISTENCE_CACHED,
                    'TEMP': Process.PERSISTENCE_TEMP,
                }

                p['persistence'] = persistence_mapping[p['persistence']]

            if 'scheduling_class' in p:
                scheduling_class_mapping = {
                    'interactive': Process.SCHEDULING_CLASS_INTERACTIVE,
                    'batch': Process.SCHEDULING_CLASS_BATCH
                }

                p['scheduling_class'] = scheduling_class_mapping[p['scheduling_class']]

            if 'input' in p:
                p['input_schema'] = p.pop('input')

            if 'output' in p:
                p['output_schema'] = p.pop('output')

            slug = p['slug']

            if 'run' in p:
                # Set default language to 'bash' if not set.
                p['run'].setdefault('language', 'bash')

                # Transform output schema using the execution engine.
                try:
                    execution_engine = manager.get_execution_engine(p['run']['language'])
                    extra_output_schema = execution_engine.get_output_schema(p)
                    if extra_output_schema:
                        p.setdefault('output_schema', []).extend(extra_output_schema)
                except InvalidEngineError:
                    self.stderr.write("Skip processor {}: execution engine '{}' not supported".format(
                        slug, p['run']['language']
                    ))
                    continue

            # Validate if container image is allowed based on the configured pattern.
            # NOTE: This validation happens here and is not deferred to executors because the idea
            #       is that this will be moved to a "container" requirement independent of the
            #       executor.
            if hasattr(settings, 'FLOW_CONTAINER_VALIDATE_IMAGE'):
                try:
                    container_image = dict_dot(p, 'requirements.executor.docker.image')
                    if not re.match(settings.FLOW_CONTAINER_VALIDATE_IMAGE, container_image):
                        self.stderr.write("Skip processor {}: container image does not match '{}'".format(
                            slug, settings.FLOW_CONTAINER_VALIDATE_IMAGE,
                        ))
                        continue
                except KeyError:
                    pass

            version = p['version']
            int_version = convert_version_string_to_int(version, VERSION_NUMBER_BITS)

            # `latest version` is returned as `int` so it has to be compared to `int_version`
            latest_version = Process.objects.filter(slug=slug).aggregate(Max('version'))['version__max']
            if latest_version is not None and latest_version > int_version:
                self.stderr.write("Skip processor {}: newer version installed".format(slug))
                continue

            previous_process_qs = Process.objects.filter(slug=slug)
            if previous_process_qs.exists():
                previous_process = previous_process_qs.latest()
            else:
                previous_process = None

            process_query = Process.objects.filter(slug=slug, version=version)
            if process_query.exists():
                if not force:
                    if verbosity > 0:
                        self.stdout.write("Skip processor {}: same version installed".format(slug))
                    continue

                process_query.update(**p)
                log_processors.append("Updated {}".format(slug))
            else:
                process = Process.objects.create(contributor=user, **p)
                assign_contributor_permissions(process)
                if previous_process:
                    copy_permissions(previous_process, process)
                log_processors.append("Inserted {}".format(slug))

        if verbosity > 0:
            if log_processors:
                self.stdout.write("Processor Updates:")
                for log in log_processors:
                    self.stdout.write("  {}".format(log))

            if log_templates:
                self.stdout.write("Default Template Updates:")
                for log in log_templates:
                    self.stdout.write("  {}".format(log))


Instruction:
Write a Python 3 function that can
read and register descriptors.

Response:
def register_descriptors(self, descriptor_schemas, user, force=False, verbosity=1):
        """Read and register descriptors."""
        log_descriptors = []

        for descriptor_schema in descriptor_schemas:
            for schema, _, _ in iterate_schema({}, descriptor_schema.get('schema', {})):
                if not schema['type'][-1].endswith(':'):
                    schema['type'] += ':'

            if 'schema' not in descriptor_schema:
                descriptor_schema['schema'] = []

            if not self.valid(descriptor_schema, DESCRIPTOR_SCHEMA):
                continue

            slug = descriptor_schema['slug']
            version = descriptor_schema.get('version', '0.0.0')
            int_version = convert_version_string_to_int(version, VERSION_NUMBER_BITS)

            # `latest version` is returned as `int` so it has to be compared to `int_version`
            latest_version = DescriptorSchema.objects.filter(slug=slug).aggregate(Max('version'))['version__max']
            if latest_version is not None and latest_version > int_version:
                self.stderr.write("Skip descriptor schema {}: newer version installed".format(slug))
                continue

            previous_descriptor_qs = DescriptorSchema.objects.filter(slug=slug)
            if previous_descriptor_qs.exists():
                previous_descriptor = previous_descriptor_qs.latest()
            else:
                previous_descriptor = None

            descriptor_query = DescriptorSchema.objects.filter(slug=slug, version=version)
            if descriptor_query.exists():
                if not force:
                    if verbosity > 0:
                        self.stdout.write("Skip descriptor schema {}: same version installed".format(slug))
                    continue

                descriptor_query.update(**descriptor_schema)
                log_descriptors.append("Updated {}".format(slug))
            else:
                descriptor = DescriptorSchema.objects.create(contributor=user, **descriptor_schema)
                assign_contributor_permissions(descriptor)
                if previous_descriptor:
                    copy_permissions(previous_descriptor, descriptor)
                log_descriptors.append("Inserted {}".format(slug))

        if log_descriptors and verbosity > 0:
            self.stdout.write("Descriptor schemas Updates:")
            for log in log_descriptors:
                self.stdout.write("  {}".format(log))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def retire(self, process_schemas):
        process_slugs = set(ps['slug'] for ps in process_schemas)

        # Processes that are in DB but not in the code
        retired_processes = Process.objects.filter(~Q(slug__in=process_slugs))

        # Remove retired processes which do not have data
        retired_processes.filter(data__exact=None).delete()

        # Remove non-latest processes which do not have data
        latest_version_processes = Process.objects.order_by('slug', '-version').distinct('slug')
        Process.objects.filter(data__exact=None).difference(latest_version_processes).delete()

        # Deactivate retired processes which have data
        retired_processes.update(is_active=False)

Response:
Retire obsolete processes.

        Remove old process versions without data. Find processes that have been
        registered but do not exist in the code anymore, then:

        - If they do not have data: remove them
        - If they have data: flag them not active (``is_active=False``)


Instruction:
Can you tell what is the following Python 3 function doing
def handle(self, *args, **options):
        force = options.get('force')
        retire = options.get('retire')
        verbosity = int(options.get('verbosity'))

        users = get_user_model().objects.filter(is_superuser=True).order_by('date_joined')

        if not users.exists():
            self.stderr.write("Admin does not exist: create a superuser")
            exit(1)

        process_paths, descriptor_paths = [], []
        process_schemas, descriptor_schemas = [], []

        for finder in get_finders():
            process_paths.extend(finder.find_processes())
            descriptor_paths.extend(finder.find_descriptors())

        for proc_path in process_paths:
            process_schemas.extend(
                self.find_schemas(proc_path, schema_type=SCHEMA_TYPE_PROCESS, verbosity=verbosity))

        for desc_path in descriptor_paths:
            descriptor_schemas.extend(
                self.find_schemas(desc_path, schema_type=SCHEMA_TYPE_DESCRIPTOR, verbosity=verbosity))

        user_admin = users.first()
        self.register_descriptors(descriptor_schemas, user_admin, force, verbosity=verbosity)
        # NOTE: Descriptor schemas must be registered first, so
        #       processes can validate 'entity_descriptor_schema' field.
        self.register_processes(process_schemas, user_admin, force, verbosity=verbosity)

        if retire:
            self.retire(process_schemas)

        if verbosity > 0:
            self.stdout.write("Running executor post-registration hook...")
        manager.get_executor().post_register_hook(verbosity=verbosity)

Response: Register processes and descriptors.


Instruction:
Can you implement a function in Python 3 that
returns True if the DID is valid for create and update.

Response:
def is_valid_pid_for_create(did):
    """Return True if ``did`` is the PID of an object that can be created with
    MNStorage.create() or MNStorage.update().

    To be valid for create() and update(), the DID:

    - Must not be the PID of an object that exists on this MN
    - Must not be a known SID known to this MN
    - Must not have been accepted for replication by this MN.
    - Must not be referenced as obsoletes or obsoletedBy in an object that exists on
      this MN

    In addition, if the DID exists in a resource map:

    - If RESOURCE_MAP_CREATE = 'reserve':

      - The DataONE subject that is making the call must have write or changePermission
        on the resource map.

    """
    # logger.debug('existing: {}'.format(is_existing_object(did)))
    # logger.debug('sid: {}'.format(is_sid(did)))
    # logger.debug('local_replica: {}'.format(is_local_replica(did)))
    # logger.debug('revision: {}'.format(d1_gmn.app.revision.is_revision(did)))
    return (
        not is_existing_object(did)
        and not is_sid(did)
        and not is_local_replica(did)
        and not d1_gmn.app.revision.is_revision(did)
        and d1_gmn.app.resource_map.is_sciobj_valid_for_create()
    )


Instruction:
Implement a function in Python 3 to
return True if the given object is a valid PID to be updated.

Response:
def is_valid_pid_to_be_updated(did):
    """Return True if ``did`` is the PID of an object that can be updated (obsoleted)
    with MNStorage.update()"""
    return (
        is_existing_object(did)
        and not is_local_replica(did)
        and not is_archived(did)
        and not is_obsoleted(did)
    )


Instruction:
Implement a Python 3 function for
returning True if sid can be assigned to the single object or to the chain to which pid belongs.

Response:
def is_valid_sid_for_chain(pid, sid):
    """Return True if ``sid`` can be assigned to the single object ``pid`` or to the
    chain to which ``pid`` belongs.

    - If the chain does not have a SID, the new SID must be previously unused.
    - If the chain already has a SID, the new SID must match the existing SID.

    All known PIDs are associated with a chain.

    Preconditions:
    - ``pid`` is verified to exist. E.g., with
      d1_gmn.app.views.asserts.is_existing_object().
    - ``sid`` is None or verified to be a SID

    """
    if _is_unused_did(sid):
        return True
    existing_sid = d1_gmn.app.revision.get_sid_by_pid(pid)
    if existing_sid is None:
        return False
    return existing_sid == sid


Instruction:
Write a Python 3 function that can
return True if PID is for an object for which science bytes are stored locally.

Response:
def is_existing_object(did):
    """Return True if PID is for an object for which science bytes are stored locally.

    This excludes SIDs and PIDs for unprocessed replica requests, remote or non-existing
    revisions of local replicas and objects aggregated in Resource Maps.

    """
    return d1_gmn.app.models.ScienceObject.objects.filter(pid__did=did).exists()


Instruction:
Create a Python 3 function to
return a text fragment classifying the DID.

Response:
def classify_identifier(did):
    """Return a text fragment classifying the ``did``

    Return <UNKNOWN> if the DID could not be classified. This should not normally happen
    and may indicate that the DID was orphaned in the database.

    """
    if _is_unused_did(did):
        return 'unused on this Member Node'
    elif is_sid(did):
        return 'a Series ID (SID) of a revision chain'
    elif is_local_replica(did):
        return 'a Persistent ID (PID) of a local replica'
    elif is_unprocessed_local_replica(did):
        return (
            'a Persistent ID (PID) of an accepted but not yet processed local replica'
        )
    elif is_archived(did):
        return 'a Persistent ID (PID) of a previously archived local object'
    elif is_obsoleted(did):
        return 'a Persistent ID (PID) of a previously updated (obsoleted) local object'
    elif is_resource_map_db(did):
        return 'a Persistent ID (PID) of a local resource map'
    elif is_existing_object(did):
        return 'a Persistent ID (PID) of an existing local object'
    elif is_revision_chain_placeholder(did):
        return (
            'a Persistent ID (PID) of a remote or non-existing revision of a local '
            'replica'
        )
    elif is_resource_map_member(did):
        return (
            'a Persistent ID (PID) of a remote or non-existing object aggregated in '
            'a local Resource Map'
        )
    logger.warning('Unable to classify known identifier. did="{}"'.format(did))
    return '<UNKNOWN>'


Instruction:
How would you implement a function in Python 3 that
returns whether the given PID is a local replica.

Response:
def is_local_replica(pid):
    """Includes unprocessed replication requests."""
    return d1_gmn.app.models.LocalReplica.objects.filter(pid__did=pid).exists()


Instruction:
Write a Python 3 script to
am local replica with status queued?

Response:
def is_unprocessed_local_replica(pid):
    """Is local replica with status "queued"."""
    return d1_gmn.app.models.LocalReplica.objects.filter(
        pid__did=pid, info__status__status='queued'
    ).exists()


Instruction:
Can you tell what is the following Python 3 function doing
def is_revision_chain_placeholder(pid):
    return d1_gmn.app.models.ReplicaRevisionChainReference.objects.filter(
        pid__did=pid
    ).exists()

Response:
Returns whether the given PID is reserved for use by
    other replicas.


Instruction:
Here you have a function in Python 3, explain what it does
def _is_did(did):
    return d1_gmn.app.models.IdNamespace.objects.filter(did=did).exists()

Response: Return True if the given DID is recorded in a local context.


Instruction:
Can you tell what is the following Python 3 function doing
def prepare_connection():
    elasticsearch_host = getattr(settings, 'ELASTICSEARCH_HOST', 'localhost')
    elasticsearch_port = getattr(settings, 'ELASTICSEARCH_PORT', 9200)
    connections.create_connection(hosts=['{}:{}'.format(elasticsearch_host, elasticsearch_port)])

Response: Prepare connection for ElasticSearch.


Instruction:
Make a summary of the following Python 3 code
def _log(pid, request, event, timestamp=None):
    # Support logging events that are not associated with an object.
    sciobj_model = None
    if pid is not None:
        try:
            sciobj_model = d1_gmn.app.models.ScienceObject.objects.filter(pid__did=pid)[
                0
            ]
        except IndexError:
            raise d1_common.types.exceptions.ServiceFailure(
                0,
                'Attempted to create event log for non-existing object. pid="{}"'.format(
                    pid
                ),
            )

    event_log_model = create_log_entry(
        sciobj_model,
        event,
        request.META['REMOTE_ADDR'],
        request.META.get('HTTP_USER_AGENT', '<not provided>'),
        request.primary_subject_str,
    )

    # The datetime is an optional parameter. If it is not provided, a
    # "auto_now_add=True" value in the the model defaults it to Now. The
    # disadvantage to this approach is that we have to update the timestamp in a
    # separate step if we want to set it to anything other than Now.
    if timestamp is not None:
        event_log_model.timestamp = timestamp
        event_log_model.save()

Response: Log an operation that was performed on a sciobj.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _is_ignored_read_event(request):
    if (
        django.conf.settings.LOG_IGNORE_TRUSTED_SUBJECT
        and d1_gmn.app.auth.is_trusted_subject(request)
    ):
        return True
    if (
        django.conf.settings.LOG_IGNORE_NODE_SUBJECT
        and d1_gmn.app.auth.is_client_side_cert_subject(request)
    ):
        return True
    if _has_regex_match(
        request.META['REMOTE_ADDR'], django.conf.settings.LOG_IGNORE_IP_ADDRESS
    ):
        return True
    if _has_regex_match(
        request.META.get('HTTP_USER_AGENT', '<not provided>'),
        django.conf.settings.LOG_IGNORE_USER_AGENT,
    ):
        return True
    if _has_regex_match(
        request.primary_subject_str, django.conf.settings.LOG_IGNORE_SUBJECT
    ):
        return True
    return False

Response: Return True if this read event was generated by an automated process and if the user has specified the LOG_IGNORE settings.


Instruction:
Given the following Python 3 function, write the documentation
def action_to_level(action):
    try:
        return ACTION_LEVEL_MAP[action]
    except LookupError:
        raise d1_common.types.exceptions.InvalidRequest(
            0, 'Unknown action. action="{}"'.format(action)
        )

Response: Map action name to action level.


Instruction:
Can you generate the documentation for the following Python 3 function
def level_to_action(level):
    try:
        return LEVEL_ACTION_MAP[level]
    except LookupError:
        raise d1_common.types.exceptions.InvalidRequest(
            0, 'Unknown action level. level="{}"'.format(level)
        )

Response: Map action level to action name.


Instruction:
Write a Python 3 function for
getting set of subjects that have unlimited access to all SciObj and APIs on this node.

Response:
def get_trusted_subjects():
    """Get set of subjects that have unlimited access to all SciObj and APIs on this
    node."""
    cert_subj = _get_client_side_certificate_subject()
    return (
        d1_gmn.app.node_registry.get_cn_subjects()
        | django.conf.settings.DATAONE_TRUSTED_SUBJECTS
        | {cert_subj}
        if cert_subj is not None
        else set()
    )


Instruction:
Can you implement a function in Python 3 that
determines if calling subject is fully trusted.

Response:
def is_trusted_subject(request):
    """Determine if calling subject is fully trusted."""
    logging.debug('Active subjects: {}'.format(', '.join(request.all_subjects_set)))
    logging.debug('Trusted subjects: {}'.format(', '.join(get_trusted_subjects())))
    return not request.all_subjects_set.isdisjoint(get_trusted_subjects())


Instruction:
Here you have a function in Python 3, explain what it does
def _get_client_side_certificate_subject():
    subject = django.core.cache.cache.get('client_side_certificate_subject')
    if subject is not None:
        return subject
    cert_pem = _get_client_side_certificate_pem()
    if cert_pem is None:
        return None
    subject = _extract_subject_from_pem(cert_pem)
    django.core.cache.cache.set('client_side_certificate_subject', subject)
    return subject

Response:
Return the DN from the client side certificate as a D1 subject if a client side
    cert has been configured. Otherwise return None.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def is_allowed(request, level, pid):
    if is_trusted_subject(request):
        return True
    return d1_gmn.app.models.Permission.objects.filter(
        sciobj__pid__did=pid,
        subject__subject__in=request.all_subjects_set,
        level__gte=level,
    ).exists()

Response: Check if one or more subjects are allowed to perform action level on object.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def assert_create_update_delete_permission(request):
    if not has_create_update_delete_permission(request):
        raise d1_common.types.exceptions.NotAuthorized(
            0,
            'Access allowed only for subjects with Create/Update/Delete '
            'permission. active_subjects="{}"'.format(format_active_subjects(request)),
        )

Response:
Access only by subjects with Create Update and Delete permission and by trusted
    infrastructure ( CNs ).


Instruction:
Here you have a function in Python 3, explain what it does
def assert_allowed(request, level, pid):
    if not d1_gmn.app.models.ScienceObject.objects.filter(pid__did=pid).exists():
        raise d1_common.types.exceptions.NotFound(
            0,
            'Attempted to perform operation on non-existing object. pid="{}"'.format(
                pid
            ),
        )
    if not is_allowed(request, level, pid):
        raise d1_common.types.exceptions.NotAuthorized(
            0,
            'Operation is denied. level="{}", pid="{}", active_subjects="{}"'.format(
                level_to_action(level), pid, format_active_subjects(request)
            ),
        )

Response: Assert that one or more subjects are allowed to perform action on object.


Instruction:
Explain what the following Python 3 code does
def format_active_subjects(request):
    decorated_subject_list = [request.primary_subject_str + ' (primary)']
    for subject in request.all_subjects_set:
        if subject != request.primary_subject_str:
            decorated_subject_list.append(subject)
    return ', '.join(decorated_subject_list)

Response:
Create a string listing active subjects for this connection suitable for
    appending to authentication error messages.


Instruction:
Can you create a Python 3 function that
removes unused imports Unsafe! Only tested on our codebase, which uses simple absolute imports on the form, "import a.b.c".

Response:
def main():
    """Remove unused imports Unsafe!

    Only tested on our codebase, which uses simple absolute imports on the form, "import
    a.b.c".

    """
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("path", nargs="+", help="File or directory path")
    parser.add_argument("--exclude", nargs="+", help="Exclude glob patterns")
    parser.add_argument(
        "--no-recursive",
        dest="recursive",
        action="store_false",
        help="Search directories recursively",
    )
    parser.add_argument(
        "--ignore-invalid", action="store_true", help="Ignore invalid paths"
    )
    parser.add_argument(
        "--pycharm", action="store_true", help="Enable PyCharm integration"
    )
    parser.add_argument(
        "--diff",
        dest="show_diff",
        action="store_true",
        help="Show diff and do not modify any files",
    )
    parser.add_argument(
        "--dry-run", action="store_true", help="Process files but do not write results"
    )
    parser.add_argument("--debug", action="store_true", help="Debug level logging")

    args = parser.parse_args()
    d1_common.util.log_setup(args.debug)

    repo_path = d1_dev.util.find_repo_root_by_path(__file__)
    repo = git.Repo(repo_path)

    specified_file_path_list = get_specified_file_path_list(args)
    # tracked_path_list = list(d1_dev.util.get_tracked_files(repo))
    # format_path_list = sorted(
    #     set(specified_file_path_list).intersection(tracked_path_list)
    # )
    format_path_list = specified_file_path_list
    for format_path in format_path_list:
        comment_unused_imports(args, format_path)


Instruction:
Implement a Python 3 function for
commenting out import for unused_dot_list.

Response:
def comment_import(r, unused_dot_list):
    """Comment out import for {dot_str}."""
    unused_dot_str = ".".join(unused_dot_list)
    for n in r("ImportNode"):
        if n.names()[0] == unused_dot_str:
            # The "!" is inserted so that this line doesn't show up when searching for
            # the comment pattern in code.
            n.replace("#{}# {}".format("!", str(n)))
            break


Instruction:
Implement a Python 3 function for
capturing only the leading dotted name list.

Response:
def get_atomtrailer_list(r):
    """Capture only the leading dotted name list.

    A full sequence typically includes function calls and parameters.
    pkga.pkgb.pkgc.one_call(arg1, arg2, arg3=4)

    """
    dot_set = set()
    for n in r.find_all(("atomtrailers",)):
        name_list = []
        for x in n.value:
            if x.type != "name":
                break
            name_list.append(x.value)
        if name_list:
            dot_set.add(tuple(name_list))
    return sorted(dot_set)


Instruction:
Make a summary of the following Python 3 code
def custom_filter_text(self, value, search):
        if isinstance(value, list):
            value = ' '.join(value)

        should = [
            Q('match', slug={'query': value, 'operator': 'and', 'boost': 10.0}),
            Q('match', **{'slug.ngrams': {'query': value, 'operator': 'and', 'boost': 5.0}}),
            Q('match', name={'query': value, 'operator': 'and', 'boost': 10.0}),
            Q('match', **{'name.ngrams': {'query': value, 'operator': 'and', 'boost': 5.0}}),
            Q('match', contributor_name={'query': value, 'operator': 'and', 'boost': 5.0}),
            Q('match', **{'contributor_name.ngrams': {'query': value, 'operator': 'and', 'boost': 2.0}}),
            Q('match', owner_names={'query': value, 'operator': 'and', 'boost': 5.0}),
            Q('match', **{'owner_names.ngrams': {'query': value, 'operator': 'and', 'boost': 2.0}}),
            Q('match', descriptor_data={'query': value, 'operator': 'and'}),
        ]

        # Add registered text extensions.
        for extension in composer.get_extensions(self):
            if hasattr(extension, 'text_filter'):
                should += extension.text_filter(value)

        search = search.query('bool', should=should)

        return search

Response: Support general query using the text attribute.


Instruction:
Can you generate the documentation for the following Python 3 function
def set_content_permissions(self, user, obj, payload):
        for entity in obj.entity_set.all():
            if user.has_perm('share_entity', entity):
                update_permission(entity, payload)

        # Data doesn't have "ADD" permission, so it has to be removed
        payload = remove_permission(payload, 'add')

        for data in obj.data.all():
            if user.has_perm('share_data', data):
                update_permission(data, payload)

Response: Apply permissions to data objects and entities in Collection.


Instruction:
Make a summary of the following Python 3 code
def create(self, request, *args, **kwargs):
        if not request.user.is_authenticated:
            raise exceptions.NotFound

        return super().create(request, *args, **kwargs)

Response: Only authenticated usesr can create new collections.


Instruction:
Write a Python 3 function for
destroying a model instance.

Response:
def destroy(self, request, *args, **kwargs):
        """Destroy a model instance.

        If ``delete_content`` flag is set in query parameters, also all
        Data objects and Entities, on which user has ``EDIT``
        permission, contained in collection will be deleted.
        """
        obj = self.get_object()
        user = request.user

        if strtobool(request.query_params.get('delete_content', 'false')):
            for entity in obj.entity_set.all():
                if user.has_perm('edit_entity', entity):
                    entity.delete()

            for data in obj.data.all():
                if user.has_perm('edit_data', data):
                    data.delete()

        return super().destroy(request, *args, **kwargs)


Instruction:
How would you code a function in Python 3 to
add data to collection.

Response:
def add_data(self, request, pk=None):
        """Add data to collection."""
        collection = self.get_object()

        if 'ids' not in request.data:
            return Response({"error": "`ids`parameter is required"}, status=status.HTTP_400_BAD_REQUEST)

        missing = []
        for data_id in request.data['ids']:
            if not Data.objects.filter(pk=data_id).exists():
                missing.append(data_id)

        if missing:
            return Response(
                {"error": "Data objects with following ids are missing: {}".format(', '.join(missing))},
                status=status.HTTP_400_BAD_REQUEST)

        for data_id in request.data['ids']:
            collection.data.add(data_id)

        return Response()


Instruction:
Write a Python 3 function for
removing data from collection.

Response:
def remove_data(self, request, pk=None):
        """Remove data from collection."""
        collection = self.get_object()

        if 'ids' not in request.data:
            return Response({"error": "`ids`parameter is required"}, status=status.HTTP_400_BAD_REQUEST)

        for data_id in request.data['ids']:
            collection.data.remove(data_id)

        return Response()


Instruction:
Create a Python 3 function for
duplicating ( make copy of ) Collection models.

Response:
def duplicate(self, request, *args, **kwargs):
        """Duplicate (make copy of) ``Collection`` models."""
        if not request.user.is_authenticated:
            raise exceptions.NotFound

        ids = self.get_ids(request.data)
        queryset = get_objects_for_user(request.user, 'view_collection', Collection.objects.filter(id__in=ids))
        actual_ids = queryset.values_list('id', flat=True)
        missing_ids = list(set(ids) - set(actual_ids))
        if missing_ids:
            raise exceptions.ParseError(
                "Collections with the following ids not found: {}".format(', '.join(map(str, missing_ids)))
            )

        duplicated = queryset.duplicate(contributor=request.user)

        serializer = self.get_serializer(duplicated, many=True)
        return Response(serializer.data)


Instruction:
Can you write a function in Python 3 where it
checks if an object with pid has successfully synced to CN.

Response:
async def is_object_synced_to_cn(self, client, pid):
        """Check if object with {pid} has successfully synced to the CN.

        CNRead.describe() is used as it's a light-weight HTTP HEAD request.

        This assumes that the call is being made over a connection that has been
        authenticated and has read or better access on the given object if it exists.

        """
        try:
            await client.describe(pid)
        except d1_common.types.exceptions.DataONEException:
            return False
        return True


Instruction:
Can you tell what is the following Python 3 function doing
def apply(self, search, field, value):
        return search.query(
            self.query_type(  # pylint: disable=not-callable
                **{
                    field: {
                        self.operator: self.get_value_query(value)
                    }
                }
            )
        )

Response: Apply lookup expression to search query.


Instruction:
Given the following Python 3 function, write the documentation
def apply(self, search, field, value):
        if not isinstance(value, list):
            value = [x for x in value.strip().split(',') if x]

        filters = [Q('match', **{field: item}) for item in value]
        return search.query('bool', should=filters)

Response: Apply lookup expression to search query.


Instruction:
Given the following Python 3 function, write the documentation
def apply(self, search, field, value):
        # We assume that the field in question has a "raw" counterpart.
        return search.query('match', **{'{}.raw'.format(field): value})

Response: Apply lookup expression to search query.


Instruction:
Write a Python 3 function for
looking up a lookup.

Response:
def get_lookup(self, operator):
        """Look up a lookup.

        :param operator: Name of the lookup operator
        """
        try:
            return self._lookups[operator]
        except KeyError:
            raise NotImplementedError("Lookup operator '{}' is not supported".format(operator))


Instruction:
Can you generate the documentation for the following Python 3 function
def build(self, search, raw_query):
        unmatched_items = {}
        for expression, value in raw_query.items():
            # Parse query expression into tokens.
            tokens = expression.split(TOKEN_SEPARATOR)

            field = tokens[0]
            tail = tokens[1:]

            if field not in self.fields:
                unmatched_items[expression] = value
                continue

            # Map field alias to final field.
            field = self.fields_map.get(field, field)

            # Parse lookup expression. Currently only no token or a single token is allowed.
            if tail:
                if len(tail) > 1:
                    raise NotImplementedError("Nested lookup expressions are not supported")

                lookup = self.get_lookup(tail[0])
                search = lookup.apply(search, field, value)
            else:
                # Default lookup.
                custom_filter = getattr(self.custom_filter_object, 'custom_filter_{}'.format(field), None)
                if custom_filter is not None:
                    search = custom_filter(value, search)
                elif isinstance(value, list):
                    # Default is 'should' between matches. If you need anything else,
                    # a custom filter for this field should be implemented.
                    filters = [Q('match', **{field: item}) for item in value]
                    search = search.query('bool', should=filters)
                else:
                    search = search.query('match', **{field: {'query': value, 'operator': 'and'}})

        return (search, unmatched_items)

Response:
Build query.

        :param search: Search query instance
        :param raw_query: Raw query arguments dictionary


Instruction:
Implement a function in Python 3 to
view handler decorator that adds SID resolve and PID validation. - For v1 calls, assume that ``did`` is a pid and raise NotFound exception if it's not valid. - For v2 calls, if DID is a valid PID, return it. If not, try to resolve it as a SID and, if successful, return the new PID. Else, raise NotFound exception.

Response:
def resolve_sid(f):
    """View handler decorator that adds SID resolve and PID validation.

    - For v1 calls, assume that ``did`` is a pid and raise NotFound exception if it's
      not valid.
    - For v2 calls, if DID is a valid PID, return it. If not, try to resolve it as a
      SID and, if successful, return the new PID. Else, raise NotFound exception.

    """

    @functools.wraps(f)
    def wrapper(request, did, *args, **kwargs):
        pid = resolve_sid_func(request, did)
        return f(request, pid, *args, **kwargs)

    return wrapper


Instruction:
Can you generate the documentation for the following Python 3 function
def decode_did(f):

    @functools.wraps(f)
    def wrapper(request, did, *args, **kwargs):
        return f(request, decode_path_segment(did), *args, **kwargs)

    return wrapper

Response:
Decorator that decodes a DID from URL
    path segment by Django.


Instruction:
Can you tell what is the following Python 3 function doing
def trusted_permission(f):

    @functools.wraps(f)
    def wrapper(request, *args, **kwargs):
        trusted(request)
        return f(request, *args, **kwargs)

    return wrapper

Response: Access only by D1 infrastructure.


Instruction:
Explain what the following Python 3 code does
def list_objects_access(f):

    @functools.wraps(f)
    def wrapper(request, *args, **kwargs):
        if not django.conf.settings.PUBLIC_OBJECT_LIST:
            trusted(request)
        return f(request, *args, **kwargs)

    return wrapper

Response: Decorator to allow access to listObjects() controlled by settings. PUBLIC_OBJECT_LIST.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_log_records_access(f):

    @functools.wraps(f)
    def wrapper(request, *args, **kwargs):
        if not django.conf.settings.PUBLIC_LOG_RECORDS:
            trusted(request)
        return f(request, *args, **kwargs)

    return wrapper

Response: Decorator to allow access to the log records of the current page.


Instruction:
Write a Python 3 function that can
access only by subjects with Create/Update/Delete permission and by trusted infrastructure (CNs).

Response:
def assert_create_update_delete_permission(f):
    """Access only by subjects with Create/Update/Delete permission and by trusted
    infrastructure (CNs)."""

    @functools.wraps(f)
    def wrapper(request, *args, **kwargs):
        d1_gmn.app.auth.assert_create_update_delete_permission(request)
        return f(request, *args, **kwargs)

    return wrapper


Instruction:
Can you implement a function in Python 3 that
accesses only with a valid session.

Response:
def authenticated(f):
    """Access only with a valid session."""

    @functools.wraps(f)
    def wrapper(request, *args, **kwargs):
        if d1_common.const.SUBJECT_AUTHENTICATED not in request.all_subjects_set:
            raise d1_common.types.exceptions.NotAuthorized(
                0,
                'Access allowed only for authenticated subjects. Please reconnect with '
                'a valid DataONE session certificate. active_subjects="{}"'.format(
                    d1_gmn.app.auth.format_active_subjects(request)
                ),
            )
        return f(request, *args, **kwargs)

    return wrapper


Instruction:
Here you have a function in Python 3, explain what it does
def required_permission(f, level):

    @functools.wraps(f)
    def wrapper(request, pid, *args, **kwargs):
        d1_gmn.app.auth.assert_allowed(request, level, pid)
        return f(request, pid, *args, **kwargs)

    return wrapper

Response: Decorator that asserts that subject has access at given level or higher for object.


Instruction:
Can you generate the documentation for the following Python 3 function
def _read_and_deserialize_dataone_type(self, response):
        try:
            return d1_common.xml.deserialize(response.content)
        except ValueError as e:
            self._raise_service_failure_invalid_dataone_type(response, e)

Response: Given a response body try to create an instance of a DataONE type.


Instruction:
Implement a function in Python 3 to
initiate a MNRead. get. Return a Requests Response object from which the object bytes can be retrieved.

Response:
def get(self, pid, stream=False, vendorSpecific=None):
        """Initiate a MNRead.get(). Return a Requests Response object from which the
        object bytes can be retrieved.

        When ``stream`` is False, Requests buffers the entire object in memory before
        returning the Response. This can exhaust available memory on the local machine
        when retrieving large science objects. The solution is to set ``stream`` to
        True, which causes the returned Response object to contain a a stream. However,
        see note below.

        When ``stream`` = True, the Response object will contain a stream which can be
        processed without buffering the entire science object in memory. However,
        failure to read all data from the stream can cause connections to be blocked.
        Due to this, the ``stream`` parameter is False by default.

        Also see:

        - http://docs.python-requests.org/en/master/user/advanced/body-content-workflow
        - get_and_save() in this module.

        """
        response = self.getResponse(pid, stream, vendorSpecific)
        return self._read_stream_response(response)


Instruction:
Can you implement a function in Python 3 that
likes MNRead. get but also retrieve the object bytes and store them in aSciObjStream.

Response:
def get_and_save(
        self, pid, sciobj_stream, create_missing_dirs=False, vendorSpecific=None
    ):
        """Like MNRead.get(), but also retrieve the object bytes and store them in a
        stream. This method does not have the potential issue with excessive memory
        usage that get() with ``stream``=False has.

        Also see MNRead.get().

        """
        response = self.get(pid, stream=True, vendorSpecific=vendorSpecific)
        try:
            if create_missing_dirs:
                d1_common.utils.filesystem.create_missing_directories_for_file(
                    sciobj_stream
                )
            for chunk_str in response.iter_content(
                chunk_size=d1_common.const.DEFAULT_CHUNK_SIZE
            ):
                if chunk_str:
                    sciobj_stream.write(chunk_str)
        finally:
            response.close()
        return response


Instruction:
Can you generate a brief explanation for the following Python 3 code
def describe(self, pid, vendorSpecific=None):
        response = self.describeResponse(pid, vendorSpecific=vendorSpecific)
        return self._read_header_response(response)

Response: Returns a dictionary of the metadata for the specified resource ID.


Instruction:
Create a Python 3 function to
return True if user is allowed to perform action on pid else False.

Response:
def isAuthorized(self, pid, action, vendorSpecific=None):
        """Return True if user is allowed to perform ``action`` on ``pid``, else
        False."""
        response = self.isAuthorizedResponse(pid, action, vendorSpecific)
        return self._read_boolean_401_response(response)


Instruction:
Can you tell what is the following Python 3 function doing
def fields(self):
        fields = super().fields
        return apply_subfield_projection(self, copy.copy(fields))

Response: Filter fields based on request query parameters.


Instruction:
How would you code a function in Python 3 to
iterate over all field values sub - fields.

Response:
def iterate_fields(fields, schema, path_prefix=None):
    """Iterate over all field values sub-fields.

    This will iterate over all field values. Some fields defined in the schema
    might not be visited.

    :param fields: field values to iterate over
    :type fields: dict
    :param schema: schema to iterate over
    :type schema: dict
    :return: (field schema, field value)
    :rtype: tuple

    """
    if path_prefix is not None and path_prefix != '' and path_prefix[-1] != '.':
        path_prefix += '.'

    schema_dict = {val['name']: val for val in schema}
    for field_id, properties in fields.items():
        path = '{}{}'.format(path_prefix, field_id) if path_prefix is not None else None
        if field_id not in schema_dict:
            raise KeyError("Field definition ({}) missing in schema".format(field_id))
        if 'group' in schema_dict[field_id]:
            for rvals in iterate_fields(properties, schema_dict[field_id]['group'], path):
                yield rvals if path_prefix is not None else rvals[:2]
        else:
            rvals = (schema_dict[field_id], fields, path)
            yield rvals if path_prefix is not None else rvals[:2]


Instruction:
Create a Python 3 function to
iterate over all schema sub - fields.

Response:
def iterate_schema(fields, schema, path_prefix=''):
    """Iterate over all schema sub-fields.

    This will iterate over all field definitions in the schema. Some field v
    alues might be None.

    :param fields: field values to iterate over
    :type fields: dict
    :param schema: schema to iterate over
    :type schema: dict
    :param path_prefix: dot separated path prefix
    :type path_prefix: str
    :return: (field schema, field value, field path)
    :rtype: tuple

    """
    if path_prefix and path_prefix[-1] != '.':
        path_prefix += '.'

    for field_schema in schema:
        name = field_schema['name']
        if 'group' in field_schema:
            for rvals in iterate_schema(fields[name] if name in fields else {},
                                        field_schema['group'], '{}{}'.format(path_prefix, name)):
                yield rvals
        else:
            yield (field_schema, fields, '{}{}'.format(path_prefix, name))


Instruction:
Can you generate the documentation for the following Python 3 function
def iterate_dict(container, exclude=None, path=None):
    if path is None:
        path = []

    for key, value in container.items():
        if callable(exclude) and exclude(key, value):
            continue

        if isinstance(value, collections.Mapping):
            for inner_path, inner_key, inner_value in iterate_dict(value, exclude=exclude, path=path + [key]):
                yield inner_path, inner_key, inner_value

        yield path, key, value

Response: Iterate over a nested dictionary.


Instruction:
Given the following Python 3 function, write the documentation
def get_subjects(request):
    if _is_certificate_provided(request):
        try:
            return get_authenticated_subjects(request.META['SSL_CLIENT_CERT'])
        except Exception as e:
            raise d1_common.types.exceptions.InvalidToken(
                0,
                'Error extracting session from certificate. error="{}"'.format(str(e)),
            )
    else:
        return d1_common.const.SUBJECT_PUBLIC, set()

Response: Get all subjects in the certificate.


Instruction:
Explain what the following Python 3 code does
def get_authenticated_subjects(cert_pem):
    if isinstance(cert_pem, str):
        cert_pem = cert_pem.encode('utf-8')
    return d1_common.cert.subjects.extract_subjects(cert_pem)

Response: Return primary subject and set of equivalents authenticated by certificate.


Instruction:
Can you tell what is the following Python 3 function doing
def get_serializer_class(self):
        base_class = super().get_serializer_class()

        class SerializerWithPermissions(base_class):
            """Augment serializer class."""

            def get_fields(serializer_self):  # pylint: disable=no-self-argument
                """Return serializer's fields."""
                fields = super().get_fields()
                fields['current_user_permissions'] = CurrentUserPermissionsSerializer(read_only=True)
                return fields

            def to_representation(serializer_self, instance):  # pylint: disable=no-self-argument
                """Object serializer."""
                data = super().to_representation(instance)

                if ('fields' not in self.request.query_params
                        or 'current_user_permissions' in self.request.query_params['fields']):
                    data['current_user_permissions'] = get_object_perms(instance, self.request.user)

                return data

        return SerializerWithPermissions

Response: Augment base serializer class.


Instruction:
Can you tell what is the following Python 3 function doing
def detail_permissions(self, request, pk=None):
        obj = self.get_object()

        if request.method == 'POST':
            content_type = ContentType.objects.get_for_model(obj)
            payload = request.data
            share_content = strtobool(payload.pop('share_content', 'false'))
            user = request.user
            is_owner = user.has_perm('owner_{}'.format(content_type), obj=obj)

            allow_owner = is_owner or user.is_superuser
            check_owner_permission(payload, allow_owner)
            check_public_permissions(payload)
            check_user_permissions(payload, request.user.pk)

            with transaction.atomic():
                update_permission(obj, payload)

                owner_count = UserObjectPermission.objects.filter(
                    object_pk=obj.id,
                    content_type=content_type,
                    permission__codename__startswith='owner_'
                ).count()

                if not owner_count:
                    raise exceptions.ParseError('Object must have at least one owner.')

            if share_content:
                self.set_content_permissions(user, obj, payload)

        return Response(get_object_perms(obj))

Response: Get or set permissions API endpoint.


Instruction:
Create a Python 3 function for
performing descriptor validation and save object.

Response:
def save(self, *args, **kwargs):
        """Perform descriptor validation and save object."""
        if self.descriptor_schema:
            try:
                validate_schema(self.descriptor, self.descriptor_schema.schema)  # pylint: disable=no-member
                self.descriptor_dirty = False
            except DirtyError:
                self.descriptor_dirty = True
        elif self.descriptor and self.descriptor != {}:
            raise ValueError("`descriptor_schema` must be defined if `descriptor` is given")

        super().save()


Instruction:
Can you tell what is the following Python 3 function doing
def duplicate(self, contributor=None):
        duplicate = Collection.objects.get(id=self.id)
        duplicate.pk = None
        duplicate.slug = None
        duplicate.name = 'Copy of {}'.format(self.name)
        duplicate.duplicated = now()
        if contributor:
            duplicate.contributor = contributor

        duplicate.save(force_insert=True)

        assign_contributor_permissions(duplicate)

        # Fields to inherit from original data object.
        duplicate.created = self.created
        duplicate.save()

        # Duplicate collection's entities.
        entities = get_objects_for_user(contributor, 'view_entity', self.entity_set.all())  # pylint: disable=no-member
        duplicated_entities = entities.duplicate(contributor=contributor)
        duplicate.entity_set.add(*duplicated_entities)

        # Add duplicated data objects to collection.
        for duplicated_entity in duplicate.entity_set.all():
            duplicate.data.add(*duplicated_entity.data.all())

        return duplicate

Response: Duplicate the current instance.


Instruction:
Can you tell what is the following Python 3 function doing
def _scalar2array(d):
    da = {}
    for k, v in d.items():
        if '_' not in k:
            da[k] = v
        else:
            name = ''.join(k.split('_')[:-1])
            ind = k.split('_')[-1]
            dim = len(ind)
            if name not in da:
                shape = tuple(3 for i in range(dim))
                da[name] = np.empty(shape, dtype=complex)
                da[name][:] = np.nan
            da[name][tuple(int(i) - 1 for i in ind)] = v
    return da

Response: Convert a dictionary with scalar elements and string indices _1234 to a dictionary of arrays. Unspecified entries are np. nan.


Instruction:
Here you have a function in Python 3, explain what it does
def _symm_current(C):
    nans = np.isnan(C)
    C[nans] = np.einsum('klij', C)[nans]
    return C

Response:
To get rid of NaNs produced by _scalar2array symmetrize operators
    where C_ijkl = C_klij


Instruction:
Can you generate the documentation for the following Python 3 function
def _antisymm_12(C):
    nans = np.isnan(C)
    C[nans] = -np.einsum('jikl', C)[nans]
    return C

Response: Antisymmetrize the first two indices of operators where C_ijkl = - C_jikl


Instruction:
Write a Python 3 function for
undoing the scaling applied in scale_dict_wet.

Response:
def unscale_dict_wet(C):
    """Undo the scaling applied in `scale_dict_wet`."""
    return {k: _scale_dict[k] * v for k, v in C.items()}


Instruction:
Explain what the following Python 3 code does
def write_reqs(req_path, req_list):
    with open(req_path, 'w') as f:
        f.write('\n'.join(req_list) + "\n")

Response: Writes a list of requests to a file


Instruction:
Write a Python 3 script for
submitting a new process locally.

Response:
def submit(self, data, runtime_dir, argv):
        """Run process locally.

        For details, see
        :meth:`~resolwe.flow.managers.workload_connectors.base.BaseConnector.submit`.
        """
        logger.debug(__(
            "Connector '{}' running for Data with id {} ({}).",
            self.__class__.__module__,
            data.id,
            repr(argv)
        ))
        subprocess.Popen(
            argv,
            cwd=runtime_dir,
            stdin=subprocess.DEVNULL
        ).wait()


Instruction:
How would you explain what the following Python 3 function does
def create_checksum_object_from_stream(
    f, algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM
):
    """Calculate the checksum of a stream.

    Args:
      f: file-like object
        Only requirement is a ``read()`` method that returns ``bytes``.

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

    Returns:
      Populated Checksum PyXB object.

    """
    checksum_str = calculate_checksum_on_stream(f, algorithm)
    checksum_pyxb = d1_common.types.dataoneTypes.checksum(checksum_str)
    checksum_pyxb.algorithm = algorithm
    return checksum_pyxb

Response: Calculate the checksum of a file - like object.


Instruction:
Write a Python 3 script for
calculating the checksum of an iterator.

Response:
def create_checksum_object_from_iterator(
    itr, algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM
):
    """Calculate the checksum of an iterator.

    Args:
      itr: iterable
        Object which supports the iterator protocol.

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

    Returns:
      Populated Checksum PyXB object.

    """
    checksum_str = calculate_checksum_on_iterator(itr, algorithm)
    checksum_pyxb = d1_common.types.dataoneTypes.checksum(checksum_str)
    checksum_pyxb.algorithm = algorithm
    return checksum_pyxb


Instruction:
Write a Python 3 script to
calculate the checksum of the bytes b and return a PyXB Checksum object.

Response:
def create_checksum_object_from_bytes(
    b, algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM
):
    """Calculate the checksum of ``bytes``.

    Warning:
      This method requires the entire object to be buffered in (virtual) memory, which
      should normally be avoided in production code.

    Args:
      b: bytes
        Raw bytes

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

    Returns:
      Populated PyXB Checksum object.

    """
    checksum_str = calculate_checksum_on_bytes(b, algorithm)
    checksum_pyxb = d1_common.types.dataoneTypes.checksum(checksum_str)
    checksum_pyxb.algorithm = algorithm
    return checksum_pyxb


Instruction:
Given the following Python 3 function, write the documentation
def calculate_checksum_on_stream(
    f,
    algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM,
    chunk_size=DEFAULT_CHUNK_SIZE,
):
    """Calculate the checksum of a stream.

    Args:
      f: file-like object
        Only requirement is a ``read()`` method that returns ``bytes``.

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

      chunk_size : int
        Number of bytes to read from the file and add to the checksum at a time.

    Returns:
      str : Checksum as a hexadecimal string, with length decided by the algorithm.

    """
    checksum_calc = get_checksum_calculator_by_dataone_designator(algorithm)
    while True:
        chunk = f.read(chunk_size)
        if not chunk:
            break
        checksum_calc.update(chunk)
    return checksum_calc.hexdigest()

Response: Calculate the checksum of a file - like object.


Instruction:
Implement a Python 3 function for
calculating the checksum of an iterator.

Response:
def calculate_checksum_on_iterator(
    itr, algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM
):
    """Calculate the checksum of an iterator.

    Args:
      itr: iterable
        Object which supports the iterator protocol.

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

    Returns:
      str : Checksum as a hexadecimal string, with length decided by the algorithm.

    """
    checksum_calc = get_checksum_calculator_by_dataone_designator(algorithm)
    for chunk in itr:
        checksum_calc.update(chunk)
    return checksum_calc.hexdigest()


Instruction:
Can you tell what is the following Python 3 function doing
def calculate_checksum_on_bytes(
    b, algorithm=d1_common.const.DEFAULT_CHECKSUM_ALGORITHM
):
    """Calculate the checksum of ``bytes``.

    Warning: This method requires the entire object to be buffered in (virtual) memory,
    which should normally be avoided in production code.

    Args:
      b: bytes
        Raw bytes

      algorithm: str
        Checksum algorithm, ``MD5`` or ``SHA1`` / ``SHA-1``.

    Returns:
      str : Checksum as a hexadecimal string, with length decided by the algorithm.

    """
    checksum_calc = get_checksum_calculator_by_dataone_designator(algorithm)
    checksum_calc.update(b)
    return checksum_calc.hexdigest()

Response: Calculate the checksum of the given bytes.


Instruction:
Create a Python 3 function for
returning True if checksums are equal.

Response:
def are_checksums_equal(checksum_a_pyxb, checksum_b_pyxb):
    """Determine if checksums are equal.

    Args:
       checksum_a_pyxb, checksum_b_pyxb: PyXB Checksum objects to compare.

    Returns:
      bool
        - **True**: The checksums contain the same hexadecimal values calculated with
          the same algorithm. Identical checksums guarantee (for all practical
          purposes) that the checksums were calculated from the same sequence of bytes.
        - **False**: The checksums were calculated with the same algorithm but the
          hexadecimal values are different.

    Raises:
      ValueError
        The checksums were calculated with different algorithms, hence cannot be
        compared.

    """
    if checksum_a_pyxb.algorithm != checksum_b_pyxb.algorithm:
        raise ValueError(
            'Cannot compare checksums calculated with different algorithms. '
            'a="{}" b="{}"'.format(checksum_a_pyxb.algorithm, checksum_b_pyxb.algorithm)
        )
    return checksum_a_pyxb.value().lower() == checksum_b_pyxb.value().lower()


Instruction:
Write a Python 3 function that can
return a string representation of a PyXB Checksum object.

Response:
def format_checksum(checksum_pyxb):
    """Create string representation of a PyXB Checksum object.

    Args:
      PyXB Checksum object

    Returns:
      str : Combined hexadecimal value and algorithm name.

    """
    return '{}/{}'.format(
        checksum_pyxb.algorithm.upper().replace('-', ''), checksum_pyxb.value().lower()
    )


Instruction:
Explain what the following Python 3 code does
def handle(self, *args, **kwargs):
        listener = ExecutorListener(redis_params=getattr(settings, 'FLOW_MANAGER', {}).get('REDIS_CONNECTION', {}))

        def _killer(signum, frame):
            """Kill the listener on receipt of a signal."""
            listener.terminate()
        signal(SIGINT, _killer)
        signal(SIGTERM, _killer)

        async def _runner():
            """Run the listener instance."""
            if kwargs['clear_queue']:
                await listener.clear_queue()
            async with listener:
                pass

        loop = asyncio.new_event_loop()
        loop.run_until_complete(_runner())
        loop.close()

Response: Run the executor listener. This method never returns.


Instruction:
Implement a function in Python 3 to
apply a projection to a single resource.

Response:
def apply_subfield_projection(field, value, deep=False):
    """Apply projection from request context.

    The passed dictionary may be mutated.

    :param field: An instance of `Field` or `Serializer`
    :type field: `Field` or `Serializer`
    :param value: Dictionary to apply the projection to
    :type value: dict
    :param deep: Also process all deep projections
    :type deep: bool
    """
    # Discover the root manually. We cannot use either `self.root` or `self.context`
    # due to a bug with incorrect caching (see DRF issue #5087).
    prefix = []
    root = field
    while root.parent is not None:
        # Skip anonymous serializers (e.g., intermediate ListSerializers).
        if root.field_name:
            prefix.append(root.field_name)
        root = root.parent
    prefix = prefix[::-1]

    context = getattr(root, '_context', {})

    # If there is no request, we cannot perform filtering.
    request = context.get('request')
    if request is None:
        return value

    filtered = set(request.query_params.get('fields', '').split(FIELD_SEPARATOR))
    filtered.discard('')
    if not filtered:
        # If there are no fields specified in the filter, return all fields.
        return value

    # Extract projection for current and deeper levels.
    current_level = len(prefix)
    current_projection = []
    for item in filtered:
        item = item.split(FIELD_DEREFERENCE)
        if len(item) <= current_level:
            continue

        if item[:current_level] == prefix:
            if deep:
                current_projection.append(item[current_level:])
            else:
                current_projection.append([item[current_level]])

    if deep and not current_projection:
        # For deep projections, an empty projection means that all fields should
        # be returned without any projection.
        return value

    # Apply projection.
    return apply_projection(current_projection, value)


Instruction:
Write a Python 3 script for
applying a projection to the value.

Response:
def apply_projection(projection, value):
    """Apply projection."""
    if isinstance(value, Sequence):
        # Apply projection to each item in the list.
        return [
            apply_projection(projection, item)
            for item in value
        ]
    elif not isinstance(value, Mapping):
        # Non-dictionary values are simply ignored.
        return value

    # Extract projection for current level.
    try:
        current_projection = [p[0] for p in projection]
    except IndexError:
        return value

    # Apply projection.
    for name in list(value.keys()):
        if name not in current_projection:
            value.pop(name)
        elif isinstance(value[name], dict):
            # Apply projection recursively.
            value[name] = apply_projection(
                [p[1:] for p in projection if p[0] == name],
                value[name]
            )

    return value


Instruction:
Write a Python 3 script to
create Relation object and add partitions of Entities.

Response:
def create(self, validated_data):
        """Create ``Relation`` object and add partitions of ``Entities``."""
        # `partitions` field is renamed to `relationpartition_set` based on source of nested serializer
        partitions = validated_data.pop('relationpartition_set')

        with transaction.atomic():
            instance = Relation.objects.create(**validated_data)
            self._create_partitions(instance, partitions)

        return instance


Instruction:
How would you implement a function in Python 3 that
updates the relation with the given data.

Response:
def update(self, instance, validated_data):
        """Update ``Relation``."""
        # `partitions` field is renamed to `relationpartition_set` based on source of nested serializer
        partitions = validated_data.pop('relationpartition_set', None)

        with transaction.atomic():
            instance = super().update(instance, validated_data)

            if partitions is not None:
                # TODO: Apply the diff instead of recreating all objects.
                instance.relationpartition_set.all().delete()
                self._create_partitions(instance, partitions)

        return instance


Instruction:
Here you have a function in Python 3, explain what it does
def output(file_like_object, path, verbose=False):
    if not path:
        for line in file_like_object:
            if verbose:
                print_info(line.rstrip())
            else:
                print(line.rstrip())
    else:
        try:
            object_file = open(os.path.expanduser(path), "w", encoding="utf-8")
            shutil.copyfileobj(file_like_object, object_file)
            object_file.close()
        except EnvironmentError as xxx_todo_changeme:
            (errno, strerror) = xxx_todo_changeme.args
            error_line_list = [
                "Could not write to object_file: {}".format(path),
                "I/O error({}): {}".format(errno, strerror),
            ]
            error_message = "\n".join(error_line_list)
            raise d1_cli.impl.exceptions.CLIError(error_message)

Response: Display or save file like object.


Instruction:
Explain what the following Python 3 code does
def _print_level(level, msg):
    for l in str(msg.rstrip()).split("\n"):
        print("{0:<9s}{1}".format(level, str(l)))

Response: Print the information in Unicode safe manner.


Instruction:
Write a Python 3 script for
finding the first line of process definition.

Response:
def get_process_definition_start(fname, slug):
    """Find the first line of process definition.

    The first line of process definition is the line with a slug.

    :param str fname: Path to filename with processes
    :param string slug: process slug
    :return: line where the process definiton starts
    :rtype: int

    """
    with open(fname) as file_:
        for i, line in enumerate(file_):
            if re.search(r'slug:\s*{}'.format(slug), line):
                return i + 1
    # In case starting line is not found just return first line
    return 1


Instruction:
Explain what the following Python 3 code does
def get_processes(process_dir, base_source_uri):
    global PROCESS_CACHE  # pylint: disable=global-statement
    if PROCESS_CACHE is not None:
        return PROCESS_CACHE

    all_process_files = []
    process_file_extensions = ['*.yaml', '*.yml']
    for root, _, filenames in os.walk(process_dir):
        for extension in process_file_extensions:
            for filename in fnmatch.filter(filenames, extension):
                all_process_files.append(os.path.join(root, filename))

    def read_yaml_file(fname):
        """Read the yaml file."""
        with open(fname) as f:
            return yaml.load(f, Loader=yaml.FullLoader)

    processes = []
    for process_file in all_process_files:
        processes_in_file = read_yaml_file(process_file)
        for process in processes_in_file:
            # This section finds the line in file where the
            # defintion of the process starts. (there are
            # multiple process definition in some files).
            startline = get_process_definition_start(process_file, process['slug'])

            # Put together URL to starting line of process definition.
            process['source_uri'] = base_source_uri + process_file[len(process_dir) + 1:] + '#L' + str(startline)

            if 'category' not in process:
                process['category'] = 'uncategorized'

            processes.append(process)

    PROCESS_CACHE = processes
    return processes

Response: Find processes in a directory and return a dictionary of processes where keys are URLs pointing to processes source code and values are processes definitions parsed from YAML files.


Instruction:
Explain what the following Python 3 code does
def setup(app):
    app.add_config_value('autoprocess_process_dir', '', 'env')
    app.add_config_value('autoprocess_source_base_url', '', 'env')
    app.add_config_value('autoprocess_definitions_uri', '', 'env')

    app.add_directive('autoprocess', AutoProcessDirective)
    app.add_directive('autoprocesscategory', AutoProcessCategoryDirective)
    app.add_directive('autoprocesstype', AutoProcessTypesDirective)

    # The setup() function can return a dictionary. This is treated by
    # Sphinx as metadata of the extension:
    return {'version': '0.2'}

Response:
Register directives.

    When sphinx loads the extension (= imports the extension module) it
    also executes the setup() function. Setup is the way extension
    informs Sphinx about everything that the extension enables: which
    config_values are introduced, which custom nodes/directives/roles
    and which events are defined in extension.

    In this case, only one new directive is created. All used nodes are
    constructed from already existing nodes in docutils.nodes package.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def make_field(self, field_name, field_body):
        name = nodes.field_name()
        name += nodes.Text(field_name)

        paragraph = nodes.paragraph()
        if isinstance(field_body, str):
            # This is the case when field_body is just a string:
            paragraph += nodes.Text(field_body)
        else:
            # This is the case when field_body is a complex node:
            # useful when constructing nested field lists
            paragraph += field_body

        body = nodes.field_body()
        body += paragraph

        field = nodes.field()
        field.extend([name, body])
        return field

Response: Fill content into nodes. field instance with given name and body.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def make_properties_list(self, field):
        properties_list = nodes.field_list()

        # changing the order of elements in this list affects
        # the order in which they are displayed
        property_names = ['label', 'type', 'description', 'required',
                          'disabled', 'hidden', 'default', 'placeholder',
                          'validate_regex', 'choices', 'collapse', 'group']

        for name in property_names:
            if name not in field:
                continue

            value = field[name]

            # Value should be formatted in code-style (=literal) mode
            if name in ['type', 'default', 'placeholder', 'validate_regex']:
                literal_node = nodes.literal(str(value), str(value))
                properties_list += self.make_field(name, literal_node)

            # Special formating of ``value`` is needed if name == 'choices'
            elif name == 'choices':
                bullet_list = nodes.bullet_list()
                for choice in value:
                    label = nodes.Text(choice['label'] + ': ')
                    val = nodes.literal(choice['value'], choice['value'])

                    paragraph = nodes.paragraph()
                    paragraph += label
                    paragraph += val
                    list_item = nodes.list_item()
                    list_item += paragraph
                    bullet_list += list_item

                properties_list += self.make_field(name, bullet_list)

            else:
                properties_list += self.make_field(name, str(value))

        return properties_list

Response: Fill the given field into a properties list and return it.


Instruction:
Write a Python 3 function that can
generate a process definition header.

Response:
def make_process_header(self, slug, typ, version, source_uri, description, inputs):
        """Generate a process definition header.

        :param str slug: process' slug
        :param str typ: process' type
        :param str version:  process' version
        :param str source_uri: url to the process definition
        :param str description: process' description
        :param dict inputs: process' inputs

        """
        node = addnodes.desc()
        signode = addnodes.desc_signature(slug, '')
        node.append(signode)

        node['objtype'] = node['desctype'] = typ

        signode += addnodes.desc_annotation(typ, typ, classes=['process-type'])
        signode += addnodes.desc_addname('', '')
        signode += addnodes.desc_name(slug + ' ', slug + ' ')

        paramlist = addnodes.desc_parameterlist()

        for field_schema, _, _ in iterate_schema({}, inputs, ''):
            field_type = field_schema['type']
            field_name = field_schema['name']

            field_default = field_schema.get('default', None)
            field_default = '' if field_default is None else '={}'.format(field_default)

            param = addnodes.desc_parameter('', '', noemph=True)
            param += nodes.emphasis(field_type, field_type, classes=['process-type'])
            # separate by non-breaking space in the output
            param += nodes.strong(text='\xa0\xa0' + field_name)

            paramlist += param

        signode += paramlist
        signode += nodes.reference('', nodes.Text('[Source: v{}]'.format(version)),
                                   refuri=source_uri, classes=['viewcode-link'])

        desc = nodes.paragraph()
        desc += nodes.Text(description, description)

        return [node, desc]