Instruction:
Write a Python 3 function that can
generate aligned sequences for the texts t1 and t2 based on the n - gram ngram.

Response:
def _generate_sequences_for_ngram(self, t1, t2, ngram, covered_spans):
        """Generates aligned sequences for the texts `t1` and `t2`, based
        around `ngram`.

        Does not generate sequences that occur within `covered_spans`.

        :param t1: text content of first witness
        :type t1: `str`
        :param t2: text content of second witness
        :type t2: `str`
        :param ngram: n-gram to base sequences on
        :type ngram: `str`
        :param covered_spans: lists of start and end indices for parts
                              of the texts already covered by a sequence
        :type covered_spans: `list` of two `list`s of 2-`tuple` of `int`

        """
        self._logger.debug('Generating sequences for n-gram "{}"'.format(
            ngram))
        pattern = re.compile(re.escape(ngram))
        context_length = len(ngram)
        t1_spans = [match.span() for match in pattern.finditer(t1)]
        t2_spans = [match.span() for match in pattern.finditer(t2)]
        sequences = []
        self._logger.debug(t1)
        for t1_span in t1_spans:
            for t2_span in t2_spans:
                if self._is_inside(t1_span, t2_span, covered_spans):
                    self._logger.debug(
                        'Skipping match due to existing coverage')
                    continue
                sequence = self._generate_sequence(
                    t1, t1_span, t2, t2_span, context_length, covered_spans)
                if sequence:
                    sequences.append(sequence)
        return sequences


Instruction:
Create a Python 3 function to
generate and outputs aligned sequences for the texts t1 and t2 from ngrams.

Response:
def _generate_sequences_for_texts(self, l1, t1, l2, t2, ngrams):
        """Generates and outputs aligned sequences for the texts `t1` and `t2`
        from `ngrams`.

        :param l1: label of first witness
        :type l1: `str`
        :param t1: text content of first witness
        :type t1: `str`
        :param l2: label of second witness
        :type l2: `str`
        :param t2: text content of second witness
        :type t2: `str`
        :param ngrams: n-grams to base sequences on
        :type ngrams: `list` of `str`

        """
        # self._subsitutes is gradually populated as each text is
        # processed, so this needs to be regenerated each time.
        self._reverse_substitutes = dict((v, k) for k, v in
                                         self._substitutes.items())
        sequences = []
        # Keep track of spans within each text that have been covered
        # by an aligned sequence, to ensure that they aren't reported
        # more than once. The first sub-list contains span indices for
        # text t1, the second for t2.
        covered_spans = [[], []]
        for ngram in ngrams:
            sequences.extend(self._generate_sequences_for_ngram(
                t1, t2, ngram, covered_spans))
        if sequences:
            sequences.sort(key=lambda x: x.start_index)
            context = {'l1': l1, 'l2': l2, 'sequences': sequences}
            report_name = '{}-{}.html'.format(l1, l2)
            os.makedirs(self._output_dir, exist_ok=True)
            self._write(context, self._output_dir, report_name)


Instruction:
Make a summary of the following Python 3 code
def _get_text(self, text):
        tokens = text.get_tokens()
        for i, token in enumerate(tokens):
            if len(token) > 1:
                char = chr(self._char_code)
                substitute = self._substitutes.setdefault(token, char)
                if substitute == char:
                    self._char_code += 1
                tokens[i] = substitute
        return self._tokenizer.joiner.join(tokens)

Response:
Returns the text content of text with all multi - character tokens
        replaced with a single character. Substitutions are recorded
xid in self. _substitutes.


Instruction:
Implement a Python 3 function for
returning the text encompassed by span plus context_length characters before and after the text.

Response:
def _get_text_sequence(self, text, span, context_length):
        """Returns the subset of `text` encompassed by `span`, plus
        `context_length` characters before and after.

        :param text: text to extract the sequence from
        :type text: `str`
        :param span: start and end indices within `text`
        :type span: 2-`tuple` of `int`
        :param context_length: length of context on either side of
                               `span` to include in the extract
        :type context_length: `int`

        """
        start = max(0, span[0] - context_length)
        end = min(len(text), span[1] + context_length)
        return text[start:end], (start, end)


Instruction:
Write a Python 3 script to
return True if both span1 and span2 fall within covered_spans.

Response:
def _is_inside(self, span1, span2, covered_spans):
        """Returns True if both `span1` and `span2` fall within
        `covered_spans`.

        :param span1: start and end indices of a span
        :type span1: 2-`tuple` of `int`
        :param span2: start and end indices of a span
        :type span2: 2-`tuple` of `int`
        :param covered_spans: lists of start and end indices for parts
                              of the texts already covered by a sequence
        :type covered_spans: `list` of two `list`s of 2-`tuple` of `int`
        :rtype: `bool`

        """
        if self._is_span_inside(span1, covered_spans[0]) and \
           self._is_span_inside(span2, covered_spans[1]):
            return True
        return False


Instruction:
Here you have a function in Python 3, explain what it does
def _is_span_inside(self, span, covered_spans):
        start = span[0]
        end = span[1]
        for c_start, c_end in covered_spans:
            if start >= c_start and end <= c_end:
                return True
        return False

Response: Returns True if span falls within covered_spans.


Instruction:
How would you explain what the following Python 3 function does
def add_corpus_arguments(parser):
    add_tokenizer_argument(parser)
    parser.add_argument('corpus', help=constants.DB_CORPUS_HELP,
                        metavar='CORPUS')

Response:
Adds common arguments for commands making use of a corpus to
    parser.


Instruction:
How would you explain what the following Python 3 function does
def add_db_arguments(parser, db_option=False):
    parser.add_argument('-m', '--memory', action='store_true',
                        help=constants.DB_MEMORY_HELP)
    parser.add_argument('-r', '--ram', default=3, help=constants.DB_RAM_HELP,
                        type=int)
    if db_option:
        parser.add_argument('-d', '--db', help=constants.DB_DATABASE_HELP,
                            metavar='DATABASE', required=True)
    else:
        parser.add_argument('db', help=constants.DB_DATABASE_HELP,
                            metavar='DATABASE')

Response: Adds common arguments for the database sub - commands to parser.


Instruction:
How would you code a function in Python 3 to
add common arguments for supplied query sub - commands to parser.

Response:
def add_supplied_query_arguments(parser):
    """Adds common arguments for supplied query sub-commands to
    `parser`."""
    parser.add_argument('-l', '--labels', help=constants.SUPPLIED_LABELS_HELP,
                        nargs='+', required=True)
    parser.add_argument('-s', '--supplied',
                        help=constants.SUPPLIED_RESULTS_HELP,
                        metavar='RESULTS', nargs='+', required=True)


Instruction:
Make a summary of the following Python 3 code
def configure_logging(verbose, logger):
    if not verbose:
        log_level = logging.WARNING
    elif verbose == 1:
        log_level = logging.INFO
    else:
        log_level = logging.DEBUG
    logger.setLevel(log_level)
    ch = colorlog.StreamHandler()
    ch.setLevel(log_level)
    formatter = colorlog.ColoredFormatter(
        '%(log_color)s%(asctime)s %(name)s %(levelname)s: %(message)s')
    ch.setFormatter(formatter)
    logger.addHandler(ch)

Response: Configures the logging used.


Instruction:
Write a Python 3 function that can
return a tacl. Catalogue object.

Response:
def get_catalogue(args):
    """Returns a `tacl.Catalogue`."""
    catalogue = tacl.Catalogue()
    catalogue.load(args.catalogue)
    return catalogue


Instruction:
Write a Python 3 function that can
return a tacl. Corpus object.

Response:
def get_corpus(args):
    """Returns a `tacl.Corpus`."""
    tokenizer = get_tokenizer(args)
    return tacl.Corpus(args.corpus, tokenizer)


Instruction:
Implement a function in Python 3 to
return a tacl. DataStore.

Response:
def get_data_store(args):
    """Returns a `tacl.DataStore`."""
    return tacl.DataStore(args.db, args.memory, args.ram)


Instruction:
Can you tell what is the following Python 3 function doing
def get_ngrams(path):
    with open(path, encoding='utf-8') as fh:
        ngrams = [ngram.strip() for ngram in fh.readlines()]
    return ngrams

Response: Returns a list of n - grams read from the file at path.


Instruction:
How would you explain what the following Python 3 function does
def excise(self, ngrams, replacement):
        content = self.get_token_content()
        ngrams.sort(key=len, reverse=True)
        for ngram in ngrams:
            content = content.replace(ngram, replacement)
        return content

Response: Returns the token content of this text with every occurrence of each n - gram in ngrams replaced with replacement.


Instruction:
How would you explain what the following Python 3 function does
def get_ngrams(self, minimum, maximum, skip_sizes=None):
        skip_sizes = skip_sizes or []
        tokens = self.get_tokens()
        for size in range(minimum, maximum + 1):
            if size not in skip_sizes:
                ngrams = collections.Counter(self._ngrams(tokens, size))
                yield (size, ngrams)

Response: Returns a generator that yields the n - grams for this text.


Instruction:
Can you tell what is the following Python 3 function doing
def _ngrams(self, sequence, degree):
        count = max(0, len(sequence) - degree + 1)
        # The extra split and join are due to having to handle
        # whitespace within a CBETA token (eg, [(禾*尤)\n/上/日]).
        return [self._tokenizer.joiner.join(
            self._tokenizer.joiner.join(sequence[i:i+degree]).split())
                for i in range(count)]

Response: Returns the n - grams generated from the sequence.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_filter_ngrams_pattern(filter_ngrams):
        return re.compile('|'.join([re.escape(ngram) for ngram in
                                    filter_ngrams]))

Response:
Returns a compiled regular expression matching on any of the
        n - grams in filter_ngrams.


Instruction:
Can you implement a function in Python 3 that
returns a generator that yields the n - grams of the text that are contained in the generated n - grams.

Response:
def get_ngrams(self, minimum, maximum, filter_ngrams):
        """Returns a generator supplying the n-grams (`minimum` <= n
        <= `maximum`) for this text.

        Each iteration of the generator supplies a tuple consisting of
        the size of the n-grams and a `collections.Counter` of the
        n-grams.

        :param minimum: minimum n-gram size
        :type minimum: `int`
        :param maximum: maximum n-gram size
        :type maximum: `int`
        :param filter_ngrams: n-grams that must be contained by the generated
                              n-grams
        :type filter_ngrams: `list`
        :rtype: `generator`

        """
        tokens = self.get_tokens()
        filter_pattern = self.get_filter_ngrams_pattern(filter_ngrams)
        for size in range(minimum, maximum + 1):
            ngrams = collections.Counter(
                self._ngrams(tokens, size, filter_pattern))
            yield (size, ngrams)


Instruction:
How would you explain what the following Python 3 function does
def set_relation_many_to_many(self, obj, field_name, objs):
        "Set a many-to-many field on an object"
        relation = getattr(obj, field_name)
        if hasattr(relation, 'set'):
            relation.set(objs) # Django 2.x
        else:
            setattr(obj, field_name, objs)

Response: Set a many - to - many field on an object


Instruction:
Can you generate a brief explanation for the following Python 3 code
def detect_missing_relations(self, obj, exc):
        missing = defaultdict(set)
        for name, err in exc.error_dict.items():
            # check if it was a relationship that doesnt exist locally
            pattern = r".+ with id (\d+) does not exist.+"
            m = re.match(pattern, str(err))
            if m:
                field = obj._meta.get_field(name)
                res = self.get_resource(field.related_model)
                missing[res].add(int(m.group(1)))
        return missing

Response: Parse the error messages and collect the missing - relationship errors that are not present locally.


Instruction:
Explain what the following Python 3 code does
def detect_uniqueness_error(self, exc):
        pattern = r"(\w+) with this (\w+) already exists"

        fields = []
        if isinstance(exc, IntegrityError):
            return self._detect_integrity_error(exc)
        assert isinstance(exc, ValidationError), TypeError

        for name, err in exc.error_dict.items():
            if re.search(pattern, str(err)):
                fields.append(name)
        return fields or None

Response: Detects uniqueness constraint errors and returns the corresponding fields.


Instruction:
Write a Python 3 script for
yielding the TLS metadata for each flow in the input_stream.

Response:
def tls_meta_data(self):
        """Pull out the TLS metadata for each flow in the input_stream"""

        # For each flow process the contents
        for flow in self.input_stream:

            # Just TCP for now
            if flow['protocol'] != 'TCP':
                continue

            # Client to Server
            if flow['direction'] == 'CTS':

                # Try to process the payload as a set of TLS records
                try:
                    tls_records, bytes_consumed = dpkt.ssl.tls_multi_factory(flow['payload'])
                    if bytes_consumed != len(flow['payload']):
                        logger.warning('Incomplete TLS record at the end...')

                    # Process the TLS records
                    flow['tls'] = {'type':'TLS_CTS', 'data':{'tls_records': tls_records, 'uri':None, 'headers':None}}
                except (dpkt.dpkt.NeedData, dpkt.dpkt.UnpackError, dpkt.ssl.SSL3Exception):
                    flow['tls'] = None

            # Server to Client
            else:
                try:
                    tls_records, bytes_consumed = dpkt.ssl.tls_multi_factory(flow['payload'])
                    if bytes_consumed != len(flow['payload']):
                        logger.warning('Incomplete TLS record at the end...')

                    # Process the TLS records
                    flow['tls'] = {'type':'TLS_STC', 'data':{'tls_records': tls_records, 'uri':None, 'headers':None}}
                except (dpkt.dpkt.NeedData, dpkt.dpkt.UnpackError, dpkt.ssl.SSL3Exception):
                    flow['tls'] = None

            # All done
            yield flow


Instruction:
How would you explain what the following Python 3 function does
def make_dict(obj):

    # Recursion base case
    if is_builtin(obj) or isinstance(obj, OrderedDict):
        return obj

    output_dict = {}
    for key in dir(obj):
        if not key.startswith('__') and not callable(getattr(obj, key)):
            attr = getattr(obj, key)
            if isinstance(attr, list):
                output_dict[key] = []
                for item in attr:
                    output_dict[key].append(make_dict(item))
            else:
                output_dict[key] = make_dict(attr)

    # All done
    return output_dict

Response: This method creates a dictionary out of a non - builtin object


Instruction:
Implement a function in Python 3 to
follow the dot notation to get the proper field then perform the action

Response:
def get_value(data, key):
    """Follow the dot notation to get the proper field, then perform the action

       Args:
           data: the data as a dictionary (required to be a dictionary)
           key: the key (as dot notation) into the data that gives the field (IP.src)

        Returns:
           the value of the field(subfield) if it exist, otherwise None
    """
    ref = data
    try:
        for subkey in key.split('.'):
            if isinstance(ref, dict):
                ref = ref[subkey]
            else:
                print('CRITICAL: Cannot use subkey %s on non-dictionary element' % subkey)
                return None
        return ref

    # In general KeyErrors are expected
    except KeyError:
        return None


Instruction:
Write a Python 3 script to
write the report data to fh in CSV format and returns it.

Response:
def csv(self, fh):
        """Writes the report data to `fh` in CSV format and returns it.

        :param fh: file to write data to
        :type fh: file object
        :rtype: file object

        """
        self._stats.to_csv(fh, encoding='utf-8', index=False)
        return fh


Instruction:
Write a Python 3 function that can
replace result rows with summary statistics about the results.

Response:
def generate_statistics(self):
        """Replaces result rows with summary statistics about the results.

        These statistics give the filename, total matching tokens,
        percentage of matching tokens and label for each witness in
        the results.

        """
        matches = self._matches
        witness_fields = [constants.WORK_FIELDNAME, constants.SIGLUM_FIELDNAME,
                          constants.LABEL_FIELDNAME]
        witnesses = matches[witness_fields].drop_duplicates()
        rows = []
        for index, (work, siglum, label) in witnesses.iterrows():
            witness = self._corpus.get_witness(work, siglum)
            witness_matches = matches[
                (matches[constants.WORK_FIELDNAME] == work) &
                (matches[constants.SIGLUM_FIELDNAME] == siglum)]
            total_count, matching_count = self._process_witness(
                witness, witness_matches)
            percentage = matching_count / total_count * 100
            rows.append({constants.WORK_FIELDNAME: work,
                         constants.SIGLUM_FIELDNAME: siglum,
                         constants.COUNT_TOKENS_FIELDNAME: matching_count,
                         constants.TOTAL_TOKENS_FIELDNAME: total_count,
                         constants.PERCENTAGE_FIELDNAME: percentage,
                         constants.LABEL_FIELDNAME: label})
        self._stats = pd.DataFrame(
            rows, columns=constants.STATISTICS_FIELDNAMES)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _generate_text_from_slices(self, full_text, slices):
        sliced_text = []
        for start, end in slices:
            sliced_text.append(full_text[start:end])
        return self._tokenizer.joiner.join(sliced_text)

Response:
Returns a single string consisting of the parts specified in
        slices joined together by the tokenizer s joining string.


Instruction:
How would you explain what the following Python 3 function does
def _merge_slices(match_slices):
        # Sort by earliest range, then by largest range.
        match_slices.sort(key=lambda x: (x[0], -x[1]))
        merged_slices = [match_slices.pop(0)]
        for slice_indices in match_slices:
            last_end = merged_slices[-1][1]
            if slice_indices[0] <= last_end:
                if slice_indices[1] > last_end:
                    merged_slices[-1][1] = slice_indices[1]
            else:
                merged_slices.append(slice_indices)
        return merged_slices

Response: Return a list of slice indices lists derived from match_slices with no overlaps.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _process_witness(self, witness, matches):
        # In order to provide a correct count of matched tokens,
        # avoiding the twin dangers of counting the same token
        # multiple times due to being part of multiple n-grams (which
        # can happen even in reduced results) and not counting tokens
        # due to an n-gram overlapping with itself or another n-gram,
        # a bit of work is required.
        #
        # Using regular expressions, get the slice indices for all
        # matches (including overlapping ones) for all matching
        # n-grams. Merge these slices together (without overlap) and
        # create a Text using that text, which can then be tokenised
        # and the tokens counted.
        tokens = witness.get_tokens()
        full_text = witness.get_token_content()
        fields = [constants.NGRAM_FIELDNAME, constants.SIZE_FIELDNAME]
        match_slices = []
        for index, (ngram, size) in matches[fields].iterrows():
            pattern = re.compile(re.escape(ngram))
            # Because the same n-gram may overlap itself ("heh" in the
            # string "heheh"), re.findall cannot be used.
            start = 0
            while True:
                match = pattern.search(full_text, start)
                if match is None:
                    break
                match_slices.append([match.start(), match.end()])
                start = match.start() + 1
        merged_slices = self._merge_slices(match_slices)
        match_content = self._generate_text_from_slices(
            full_text, merged_slices)
        match_text = Text(match_content, self._tokenizer)
        return len(tokens), len(match_text.get_tokens())

Response: Return the counts of total tokens and matching tokens in witness.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_witnesses(self, source_tree):
        witnesses = []
        witness_elements = source_tree.xpath(
            '/tei:*/tei:teiHeader/tei:fileDesc/tei:sourceDesc/'
            'tei:listWit/tei:witness', namespaces=constants.NAMESPACES)
        for witness_element in witness_elements:
            witnesses.append((witness_element.text,
                              witness_element.get(constants.XML + 'id')))
        if not witnesses:
            witnesses = [(constants.BASE_WITNESS, constants.BASE_WITNESS_ID)]
        return witnesses

Response:
Returns a list of all witnesses of variant readings in
        source_tree along with their XML ids.


Instruction:
Create a Python 3 function to
run the Simple Packet Printer Example

Response:
def run(iface_name=None, bpf=None, summary=None, max_packets=50):
    """Run the Simple Packet Printer Example"""

    # Create the classes
    streamer = packet_streamer.PacketStreamer(iface_name=iface_name, bpf=bpf, max_packets=max_packets)
    meta = packet_meta.PacketMeta()
    rdns = reverse_dns.ReverseDNS()
    tags = packet_tags.PacketTags()
    tmeta = transport_meta.TransportMeta()
    printer = packet_summary.PacketSummary()

    # Set up the chain
    meta.link(streamer)
    rdns.link(meta)
    tags.link(rdns)
    tmeta.link(tags)
    printer.link(tmeta)

    # Pull the chain
    printer.pull()


Instruction:
Make a summary of the following Python 3 code
def generate(self, output_dir, catalogue, results, label):
        data = results.get_raw_data()
        labels = catalogue.ordered_labels
        ngrams = self._generate_results(output_dir, labels, data)
        ngram_table = self._generate_ngram_table(output_dir, labels, data)
        corpus_table = self._generate_corpus_table(labels, ngrams)
        context = {'corpus_table': corpus_table, 'focus_label': label,
                   'labels': labels, 'ngram_table': ngram_table, 'sep': os.sep}
        report_name = 'lifetime-{}.html'.format(label)
        self._write(context, output_dir, report_name)

Response: Generates the report and writes it to output_dir.


Instruction:
Can you generate the documentation for the following Python 3 function
def _generate_corpus_table(self, labels, ngrams):
        html = []
        for label in labels:
            html.append(self._render_corpus_row(label, ngrams))
        return '\n'.join(html)

Response: Returns an HTML table containing data on each corpus n - grams.


Instruction:
Explain what the following Python 3 code does
def _generate_ngram_table(self, output_dir, labels, results):
        html = []
        grouped = results.groupby(constants.NGRAM_FIELDNAME)
        row_template = self._generate_ngram_row_template(labels)
        for name, group in grouped:
            html.append(self._render_ngram_row(name, group, row_template,
                                               labels))
        return '\n'.join(html)

Response:
Returns an HTML table containing data on each n - gram in
        results.


Instruction:
Explain what the following Python 3 code does
def _generate_ngram_row_template(self, labels):
        cells = ['{ngram}	']
        for label in labels:
            cells.append('{{{}}}	'.format(label))
        return '\n'.join(cells)

Response: Returns the HTML template for a row in the n - gram table.


Instruction:
Can you generate the documentation for the following Python 3 function
def _generate_results(self, output_dir, labels, results):
        ngrams = {}
        for idx, label in enumerate(labels):
            now_results = results[results[constants.LABEL_FIELDNAME] == label]
            earlier_labels = labels[:idx]
            earlier_ngrams = results[results[constants.LABEL_FIELDNAME].isin(
                earlier_labels)][constants.NGRAM_FIELDNAME].values
            later_labels = labels[idx + 1:]
            later_ngrams = results[results[constants.LABEL_FIELDNAME].isin(
                later_labels)][constants.NGRAM_FIELDNAME].values
            first_ngrams = []
            only_ngrams = []
            last_ngrams = []
            for ngram in now_results[constants.NGRAM_FIELDNAME].unique():
                if ngram in earlier_ngrams:
                    if ngram not in later_ngrams:
                        last_ngrams.append(ngram)
                elif ngram in later_ngrams:
                    first_ngrams.append(ngram)
                else:
                    only_ngrams.append(ngram)
            self._save_results(output_dir, label, now_results, first_ngrams,
                               'first')
            self._save_results(output_dir, label, now_results, only_ngrams,
                               'only')
            self._save_results(output_dir, label, now_results, last_ngrams,
                               'last')
            ngrams[label] = {'first': first_ngrams, 'last': last_ngrams,
                             'only': only_ngrams}
        return ngrams

Response: Generates multiple results files in output_dir based on results.


Instruction:
Explain what the following Python 3 code does
def _render_corpus_row(self, label, ngrams):
        row = ('
\n{label}	\n{first}	\n{only}	\n'
               '{last}	\n
')
        cell_data = {'label': label}
        for period in ('first', 'only', 'last'):
            cell_data[period] = ', '.join(ngrams[label][period])
        return row.format(**cell_data)

Response: Returns the HTML for a corpus row.


Instruction:
Implement a Python 3 function for
returning the HTML for an n - gram row.

Response:
def _render_ngram_row(self, ngram, ngram_group, row_template, labels):
        """Returns the HTML for an n-gram row."""
        cell_data = {'ngram': ngram}
        label_data = {}
        for label in labels:
            label_data[label] = []
        work_grouped = ngram_group.groupby(constants.WORK_FIELDNAME)
        for work, group in work_grouped:
            min_count = group[constants.COUNT_FIELDNAME].min()
            max_count = group[constants.COUNT_FIELDNAME].max()
            if min_count == max_count:
                count = min_count
            else:
                count = '{}\N{EN DASH}{}'.format(min_count, max_count)
            label_data[group[constants.LABEL_FIELDNAME].iloc[0]].append(
                '{} ({})'.format(work, count))
        for label, data in label_data.items():
            label_data[label] = '; '.join(data)
        cell_data.update(label_data)
        html = row_template.format(**cell_data)
        return '<tr>\n{}\n</tr>'.format(html)


Instruction:
Can you tell what is the following Python 3 function doing
def _save_results(self, output_dir, label, results, ngrams, type_label):
        path = os.path.join(output_dir, '{}-{}.csv'.format(label, type_label))
        results[results[constants.NGRAM_FIELDNAME].isin(
            ngrams)].to_csv(path, encoding='utf-8', float_format='%d',
                            index=False)

Response: Saves results filtered by label and ngrams to output_dir.


Instruction:
How would you implement a function in Python 3 that
sets my input stream

Response:
def link(self, stream_instance):
        """Set my input stream"""
        if isinstance(stream_instance, collections.Iterable):
            self.input_stream = stream_instance
        elif getattr(stream_instance, 'output_stream', None):
            self.input_stream = stream_instance.output_stream
        else:
            raise RuntimeError('Calling link() with unknown instance type %s' % type(stream_instance))


Instruction:
Can you tell what is the following Python 3 function doing
def generate(self, path, label):
        for filename in os.listdir(path):
            self[filename] = label

Response:
Creates default data from the corpus at path marking all
        works with label.


Instruction:
Implement a function in Python 3 to
return a list of works associated with label.

Response:
def get_works_by_label(self, label):
        """Returns a list of works associated with `label`.

        :param label: label of works to be returned
        :type label: `str`
        :rtype: `list` of `str`

        """
        return [work for work, c_label in self.items() if c_label == label]


Instruction:
Can you write a function in Python 3 where it
loads the data from path into the catalogue.

Response:
def load(self, path):
        """Loads the data from `path` into the catalogue.

        :param path: path to catalogue file
        :type path: `str`

        """
        fieldnames = ['work', 'label']
        with open(path, 'r', encoding='utf-8', newline='') as fh:
            reader = csv.DictReader(fh, delimiter=' ', fieldnames=fieldnames,
                                    skipinitialspace=True)
            for row in reader:
                work, label = row['work'], row['label']
                if label:
                    if label not in self._ordered_labels:
                        self._ordered_labels.append(label)
                    if work in self:
                        raise MalformedCatalogueError(
                            CATALOGUE_WORK_RELABELLED_ERROR.format(work))
                    self[work] = label


Instruction:
Can you tell what is the following Python 3 function doing
def relabel(self, label_map):
        catalogue = copy.deepcopy(self)
        to_delete = set()
        for work, old_label in catalogue.items():
            if old_label in label_map:
                catalogue[work] = label_map[old_label]
            else:
                to_delete.add(catalogue[work])
        for label in to_delete:
            catalogue.remove_label(label)
        return catalogue

Response: Returns a copy of the catalogue with the labels remapped according to label_map.


Instruction:
Write a Python 3 script to
remove a label from the catalogue by removing all works carrying it.

Response:
def remove_label(self, label):
        """Removes `label` from the catalogue, by removing all works carrying
        it.

        :param label: label to remove
        :type label: `str`

        """
        works_to_delete = []
        for work, work_label in self.items():
            if work_label == label:
                works_to_delete.append(work)
        for work in works_to_delete:
            del self[work]
        if self._ordered_labels:
            self._ordered_labels.remove(label)


Instruction:
How would you explain what the following Python 3 function does
def save(self, path):
        writer = csv.writer(open(path, 'w', newline=''), delimiter=' ')
        rows = list(self.items())
        rows.sort(key=lambda x: x[0])
        writer.writerows(rows)

Response: Saves this catalogue s data to path.


Instruction:
Create a Python 3 function to
yield the dns metadata for each packet in the input_stream.

Response:
def dns_meta_data(self):
        """Pull out the dns metadata for packet/transport in the input_stream"""

        # For each packet process the contents
        for packet in self.input_stream:
            # Skip packets without transport info (ARP/ICMP/IGMP/whatever)
            if 'transport' not in packet:
                continue
            try:
                dns_meta = dpkt.dns.DNS(packet['transport']['data'])
                _raw_info = data_utils.make_dict(dns_meta)
                packet['dns'] = self._dns_info_mapper(_raw_info)
                packet['dns']['_raw'] = _raw_info
            except (dpkt.dpkt.NeedData, dpkt.dpkt.UnpackError):
                if 'dns' in packet:
                    del packet['dns']

            # All done
            yield packet


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _dns_weird(self, record):
        weird = {}

        # Zero should always be 0
        if record['flags']['zero'] != 0:
            weird['zero'] = record['flags']['zero']

        # Trucated may indicate an exfil
        if record['flags']['truncated']:
            weird['trucnated'] = True

        # Weird Query Types
        weird_types = set(['DNS_NULL', 'DNS_HINFO', 'DNS_TXT', 'UNKNOWN'])
        for query in record['queries']:
            if query['type'] in weird_types:
                weird['query_type'] = query['type']

        # Weird Query Classes
        weird_classes = set(['DNS_CHAOS', 'DNS_HESIOD', 'DNS_NONE', 'DNS_ANY'])
        for query in record['queries']:
            if query['class'] in weird_classes:
                weird['query_class'] = query['class']

        # Weird Answer Types
        for section_name in ['answers', 'name_servers', 'additional']:
            for answer in record['answers'][section_name]:
                if answer['type'] in weird_types:
                    weird['answer_type'] = answer['type']

        # Weird Answer Classes
        for section_name in ['answers', 'name_servers', 'additional']:
            for answer in record['answers'][section_name]:
                if answer['class'] in weird_classes:
                    weird['answer_class'] = answer['class']

        # Is the subdomain name especially long or have high entropy?
        for query in record['queries']:
            subdomain = '.'.join(query['name'].split('.')[:-2])
            length = len(subdomain)
            entropy = self.entropy(subdomain)
            if length > 35 and entropy > 3.5:
                weird['subdomain_length'] = length
                weird['subdomain'] = subdomain
                weird['subdomain_entropy'] = entropy
                weird['subdomain'] = subdomain

        # Return the weird stuff
        return weird

Response: Look for weird stuff in dns record using a set of criteria to mark the weird stuff


Instruction:
Create a Python 3 function for
returning a list of all of the sigla for work.

Response:
def get_sigla(self, work):
        """Returns a list of all of the sigla for `work`.

        :param work: name of work
        :type work: `str`
        :rtype: `list` of `str`

        """
        return [os.path.splitext(os.path.basename(path))[0]
                for path in glob.glob(os.path.join(self._path, work, '*.txt'))]


Instruction:
Can you generate the documentation for the following Python 3 function
def get_witness(self, work, siglum, text_class=WitnessText):
        filename = os.path.join(work, siglum + '.txt')
        self._logger.debug('Creating WitnessText object from {}'.format(
            filename))
        with open(os.path.join(self._path, filename), encoding='utf-8') \
                as fh:
            content = fh.read()
        return text_class(work, siglum, content, self._tokenizer)

Response: Returns a WitnessText object representing the file associated with the work and siglum.


Instruction:
Can you write a function in Python 3 where it
returns a generator supplying the witnesses for each work in the corpus.

Response:
def get_witnesses(self, name='*'):
        """Returns a generator supplying `WitnessText` objects for each work
        in the corpus.

        :rtype: `generator` of `WitnessText`

        """
        for filepath in glob.glob(os.path.join(self._path, name, '*.txt')):
            if os.path.isfile(filepath):
                name = os.path.split(os.path.split(filepath)[0])[1]
                siglum = os.path.splitext(os.path.basename(filepath))[0]
                yield self.get_witness(name, siglum)


Instruction:
Given the following Python 3 function, write the documentation
def get_works(self):
        return [os.path.split(filepath)[1] for filepath in
                glob.glob(os.path.join(self._path, '*'))
                if os.path.isdir(filepath)]

Response: Returns a list of the names of all works in the corpus.


Instruction:
Explain what the following Python 3 code does
def process_for_rdns(self):

        # For each packet process the contents
        for item in self.input_stream:

            # Do for both the src and dst
            for endpoint in ['src', 'dst']:

                # Sanity check (might be an ARP, whatever... without a src/dst)
                if endpoint not in item['packet']:

                    # Set the domain to None
                    item['packet'][endpoint+self.domain_postfix] = None
                    continue

                # Convert inet_address to str ip_address
                ip_address = net_utils.inet_to_str(item['packet'][endpoint])

                # Is this already in our cache
                if self.ip_lookup_cache.get(ip_address):
                    domain = self.ip_lookup_cache.get(ip_address)

                # Is the ip_address local or special
                elif net_utils.is_internal(ip_address):
                    domain = 'internal'
                elif net_utils.is_special(ip_address):
                    domain = net_utils.is_special(ip_address)

                # Look it up at this point
                else:
                    domain = self._reverse_dns_lookup(ip_address)

                # Set the domain
                item['packet'][endpoint+self.domain_postfix] = domain

                # Cache it
                self.ip_lookup_cache.set(ip_address, domain)

            # All done
            yield item

Response: Process the contents of the input stream and yield the items that are ready to be used for the RDNS lookup.


Instruction:
Can you tell what is the following Python 3 function doing
def hsv_to_rgb(h, s, v):
    c = v * s
    hp = h*6
    x = c * (1 - abs(hp % 2 - 1))
    if hp < 1:
        r, g, b = c, x, 0
    elif hp < 2:
        r, g, b = x, c, 0
    elif hp < 3:
        r, g, b = 0, c, x
    elif hp < 4:
        r, g, b = 0, x, c
    elif hp < 5:
        r, g, b = x, 0, c
    elif hp < 6:
        r, g, b = c, 0, x
    m = v - c
    colour = (r + m, g + m, b + m)
    return 'rgb({}, {}, {})'.format(*[round(value * 255) for value in colour])

Response: Convert a colour specified in HSV to an RGB colour.


Instruction:
Make a summary of the following Python 3 code
def generate_colours(n):
    colours = []
    golden_ratio_conjugate = 0.618033988749895
    h = 0.8  # Initial hue
    s = 0.7  # Fixed saturation
    v = 0.95  # Fixed value
    for i in range(n):
        h += golden_ratio_conjugate
        h %= 1
        colours.append(hsv_to_rgb(h, s, v))
    return colours

Response:
Generate a list of n distinct colours each represented as an RGB
    string suitable for use in CSS.


Instruction:
Here you have a function in Python 3, explain what it does
def tag_stuff(self):

        # For each packet in the pcap process the contents
        for item in self.input_stream:

            # Make sure it has a tags field (which is a set)
            if 'tags' not in item:
                item['tags'] = set()

            # For each tag_methods run it on the item
            for tag_method in self.tag_methods:
                item['tags'].add(tag_method(item))

            # Not interested in None tags
            if None in item['tags']:
                item['tags'].remove(None)
            # All done
            yield item

Response: Yields the items that are tagged with the fields in the item.


Instruction:
Implement a Python 3 function for
creating a tag based on the direction of the traffic

Response:
def _tag_net_direction(data):
        """Create a tag based on the direction of the traffic"""

        # IP or IPv6
        src = data['packet']['src_domain']
        dst = data['packet']['dst_domain']
        if src == 'internal':
            if dst == 'internal' or 'multicast' in dst or 'broadcast' in dst:
                return 'internal'
            else:
                return 'outgoing'
        elif dst == 'internal':
            return 'incoming'
        else:
            return None


Instruction:
How would you implement a function in Python 3 that
returns a sorted list of all witnesses of readings in source_tree and the elements that bear @wit attributes.

Response:
def get_witnesses(self, source_tree):
        """Returns a sorted list of all witnesses of readings in
        `source_tree`, and the elements that bear @wit attributes.

        :param source_tree: XML tree of source document
        :type source_tree: `etree._ElementTree`
        :rtype: `tuple` of `list`\s

        """
        witnesses = set()
        bearers = source_tree.xpath('//tei:app/tei:*[@wit]',
                                    namespaces=constants.NAMESPACES)
        for bearer in bearers:
            for witness in witnesses_splitter.split(bearer.get('wit')):
                if witness:
                    witnesses.add(witness)
        return sorted(witnesses), bearers


Instruction:
Given the following Python 3 function, write the documentation
def _handle_witnesses(self, root):
        witnesses, bearers = self.get_witnesses(root)
        if not witnesses:
            return root
        source_desc = root.xpath(
            '/tei:teiCorpus/tei:teiHeader/tei:fileDesc/tei:sourceDesc',
            namespaces=constants.NAMESPACES)[0]
        wit_list = etree.SubElement(source_desc, TEI + 'listWit')
        for index, siglum in enumerate(witnesses):
            wit = etree.SubElement(wit_list, TEI + 'witness')
            xml_id = 'wit{}'.format(index+1)
            wit.set(constants.XML + 'id', xml_id)
            wit.text = siglum
            full_siglum = '【{}】'.format(siglum)
            self._update_refs(root, bearers, 'wit', full_siglum, xml_id)
        return root

Response:
Returns root with a witness list added to the TEI header and @wit
        values changed to references.


Instruction:
How would you implement a function in Python 3 that
saves the TEI XML document root at the path work.

Response:
def _output_work(self, work, root):
        """Saves the TEI XML document `root` at the path `work`."""
        output_filename = os.path.join(self._output_dir, work)
        tree = etree.ElementTree(root)
        tree.write(output_filename, encoding='utf-8', pretty_print=True)


Instruction:
Create a Python 3 function to
populate the teiHeader of the teiCorpus with useful information from the teiHeader of the first TEI part.

Response:
def _populate_header(self, root):
        """Populate the teiHeader of the teiCorpus with useful information
        from the teiHeader of the first TEI part."""
        # If this gets more complicated, it should be handled via an XSLT.
        title_stmt = root.xpath(
            'tei:teiHeader/tei:fileDesc/tei:titleStmt',
            namespaces=constants.NAMESPACES)[0]
        # There is no guarantee that a title or author is specified,
        # in which case do nothing.
        try:
            title_stmt[0].text = root.xpath(
                'tei:TEI[1]/tei:teiHeader/tei:fileDesc/tei:titleStmt/'
                'tei:title', namespaces=constants.NAMESPACES)[0].text
        except IndexError:
            pass
        try:
            title_stmt[1].text = root.xpath(
                'tei:TEI[1]/tei:teiHeader/tei:fileDesc/tei:titleStmt/'
                'tei:author', namespaces=constants.NAMESPACES)[0].text
        except IndexError:
            pass
        return root


Instruction:
Here you have a function in Python 3, explain what it does
def _update_refs(self, root, bearers, attribute, ref_text, xml_id):
        ref = ' #{} '.format(xml_id)
        for bearer in bearers:
            attribute_text = bearer.get(attribute).replace(ref_text, ref)
            refs = ' '.join(sorted(attribute_text.strip().split()))
            bearer.set(attribute, refs)

Response: Update the ref_text on the elements bearing attribute with the new text.


Instruction:
Can you generate the documentation for the following Python 3 function
def _extract_work(self, filename):
        basename = os.path.splitext(os.path.basename(filename))[0]
        match = self.work_pattern.search(basename)
        if match is None:
            self._logger.warning('Found an anomalous filename "{}"'.format(
                filename))
            return None, None
        work = '{}{}'.format(match.group('prefix'), match.group('work'))
        return work, match.group('part')

Response: Extracts the name of the work in filename.


Instruction:
Create a Python 3 function for
returning a sorted list of all resps in source_tree and their elements that bear @resp attributes.

Response:
def get_resps(self, source_tree):
        """Returns a sorted list of all resps in `source_tree`, and the
        elements that bear @resp attributes.

        :param source_tree: XML tree of source document
        :type source_tree: `etree._ElementTree`
        :rtype: `tuple` of `lists`

        """
        resps = set()
        bearers = source_tree.xpath('//*[@resp]',
                                    namespaces=constants.NAMESPACES)
        for bearer in bearers:
            for resp in resp_splitter.split(bearer.get('resp')):
                if resp:
                    resps.add(tuple(resp.split('|', maxsplit=1)))
        return sorted(resps), bearers


Instruction:
Create a Python 3 function to
return root with a resp list added to the TEI header and @resp values changed to references.

Response:
def _handle_resps(self, root):
        """Returns `root` with a resp list added to the TEI header and @resp
        values changed to references."""
        resps, bearers = self.get_resps(root)
        if not resps:
            return root
        file_desc = root.xpath(
            '/tei:teiCorpus/tei:teiHeader/tei:fileDesc',
            namespaces=constants.NAMESPACES)[0]
        edition_stmt = etree.Element(TEI + 'editionStmt')
        file_desc.insert(1, edition_stmt)
        for index, (resp_resp, resp_name) in enumerate(resps):
            resp_stmt = etree.SubElement(edition_stmt, TEI + 'respStmt')
            xml_id = 'resp{}'.format(index+1)
            resp_stmt.set(constants.XML + 'id', xml_id)
            resp = etree.SubElement(resp_stmt, TEI + 'resp')
            resp.text = resp_resp
            name = etree.SubElement(resp_stmt, TEI + 'name')
            name.text = resp_name
            resp_data = '{{{}|{}}}'.format(resp_resp, resp_name)
            self._update_refs(root, bearers, 'resp', resp_data, xml_id)
        return root


Instruction:
Create a Python 3 function for
transforming the file at file_path into simpler XML and returns that.

Response:
def _tidy(self, work, file_path):
        """Transforms the file at `file_path` into simpler XML and returns
        that.

        """
        output_file = os.path.join(self._output_dir, work)
        self._logger.info('Tidying file {} into {}'.format(
            file_path, output_file))
        try:
            tei_doc = etree.parse(file_path)
        except etree.XMLSyntaxError as err:
            self._logger.error('XML file "{}" is invalid: {}'.format(
                file_path, err))
            raise
        return self.transform(tei_doc).getroot()


Instruction:
Write a Python 3 function that can
return tuple for flow data

Response:
def flow_tuple(data):
    """Tuple for flow (src, dst, sport, dport, proto)"""
    src = net_utils.inet_to_str(data['packet'].get('src')) if data['packet'].get('src') else None
    dst = net_utils.inet_to_str(data['packet'].get('dst')) if data['packet'].get('dst') else None
    sport = data['transport'].get('sport') if data.get('transport') else None
    dport = data['transport'].get('dport') if data.get('transport') else None
    proto = data['transport'].get('type') if data.get('transport') else data['packet']['type']
    return (src, dst, sport, dport, proto)


Instruction:
Given the following Python 3 function, write the documentation
def add_packet(self, packet):

        # First packet?
        if not self.meta['flow_id']:
            self.meta['flow_id'] = flow_tuple(packet)
            self.meta['src'] = self.meta['flow_id'][0]
            self.meta['dst'] = self.meta['flow_id'][1]
            self.meta['src_domain'] = packet['packet']['src_domain']
            self.meta['dst_domain'] = packet['packet']['dst_domain']
            self.meta['sport'] = self.meta['flow_id'][2]
            self.meta['dport'] = self.meta['flow_id'][3]
            self.meta['protocol'] = self.meta['flow_id'][4]
            self.meta['direction'] = self._cts_or_stc(packet)
            self.meta['start'] = packet['timestamp']
            self.meta['end'] = packet['timestamp']

        # Add the packet
        self.meta['packet_list'].append(packet)
        if packet['timestamp'] < self.meta['start']:
            self.meta['start'] = packet['timestamp']
        if packet['timestamp'] > self.meta['end']:
            self.meta['end'] = packet['timestamp']

        # State of connection/flow
        if self.meta['protocol'] == 'TCP':
            flags = packet['transport']['flags']
            if 'syn' in flags:
                self.meta['state'] = 'partial_syn'
                self.meta['direction'] = 'CTS'
            elif 'fin' in flags:
                # print('--- FIN RECEIVED %s ---'  % str(self.meta['flow_id))
                self.meta['state'] = 'complete' if self.meta['state'] == 'partial_syn' else 'partial'
                self.meta['timeout'] = datetime.now() + timedelta(seconds=1)
            elif 'syn_ack' in flags:
                self.meta['state'] = 'partial_syn'
                self.meta['direction'] = 'STC'
            elif 'fin_ack'in flags:
                # print('--- FIN_ACK RECEIVED %s ---' % str(self.meta['flow_id))
                self.meta['state'] = 'complete' if self.meta['state'] == 'partial_syn' else 'partial'
                self.meta['timeout'] = datetime.now() + timedelta(seconds=1)
            elif 'rst' in flags:
                # print('--- RESET RECEIVED %s ---' % str(self.meta['flow_id))
                self.meta['state'] = 'partial'
                self.meta['timeout'] = datetime.now() + timedelta(seconds=1)

        # Only collect UDP and TCP
        if self.meta['protocol'] not in ['UDP', 'TCP']:
            self.meta['timeout'] = datetime.now()

Response: Add a packet to the flow


Instruction:
Can you implement a function in Python 3 that
reassembles the flow and return all the info and data

Response:
def get_flow(self):
        """Reassemble the flow and return all the info/data"""
        if self.meta['protocol'] == 'TCP':
            self.meta['packet_list'].sort(key=lambda packet: packet['transport']['seq'])
            for packet in self.meta['packet_list']:
                self.meta['payload'] += packet['transport']['data']

        return self.meta


Instruction:
Create a Python 3 function to
return the name of the cts or stc class that corresponds to the data.

Response:
def _cts_or_stc(data):
        """Does the data look like a Client to Server (cts) or Server to Client (stc) traffic?"""

        # UDP/TCP
        if data['transport']:

            # TCP flags
            if data['transport']['type'] == 'TCP':
                flags = data['transport']['flags']

                # Syn/Ack or fin/ack is a server response
                if 'syn_ack' in flags or 'fin_ack' in flags:
                    return 'STC'

                # Syn or fin is a client response
                if 'syn' in flags or 'fin' in flags:
                    return 'CTS'

            # Source Port/Destination Port
            if 'sport' in data['transport']:
                sport = data['transport']['sport']
                dport = data['transport']['dport']

                # High port talking to low port
                if dport < 1024 and sport > dport:
                    return 'CTS'

                # Low port talking to high port
                if sport < 1024 and sport < dport:
                    return 'STC'

                # Wow... guessing
                return 'STC' if sport < dport else 'CTS'

        # Internal/External
        if 'src' in data['packet'] and 'dst' in data['packet']:
            src = net_utils.inet_to_str(data['packet']['src'])
            dst = net_utils.inet_to_str(data['packet']['dst'])

            # Internal talking to external?
            if net_utils.is_internal(src) and not net_utils.is_internal(dst):
                return 'CTS'

            # External talking to internal?
            if net_utils.is_internal(dst) and not net_utils.is_internal(src):
                return 'STC'

        # Okay we have no idea
        return 'CTS'


Instruction:
Make a summary of the following Python 3 code
def _generate_tokens(self, text):
    # This is technically an undocumented API for Python3, but allows us to use the same API as for
    # Python2. See http://stackoverflow.com/a/4952291/328565.
    for index, tok in enumerate(tokenize.generate_tokens(io.StringIO(text).readline)):
      tok_type, tok_str, start, end, line = tok
      yield Token(tok_type, tok_str, start, end, line, index,
                  self._line_numbers.line_to_offset(start[0], start[1]),
                  self._line_numbers.line_to_offset(end[0], end[1]))

Response: Generates tokens for the given text.


Instruction:
Make a summary of the following Python 3 code
def get_token_from_offset(self, offset):
    return self._tokens[bisect.bisect(self._token_offsets, offset) - 1]

Response: Returns the token containing the given character offset.


Instruction:
How would you implement a function in Python 3 that
returns the token containing the given lineno and col_offset.

Response:
def get_token(self, lineno, col_offset):
    """
    Returns the token containing the given (lineno, col_offset) position, or the preceeding token
    if the position is between tokens.
    """
    # TODO: add test for multibyte unicode. We need to translate offsets from ast module (which
    # are in utf8) to offsets into the unicode text. tokenize module seems to use unicode offsets
    # but isn't explicit.
    return self.get_token_from_offset(self._line_numbers.line_to_offset(lineno, col_offset))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_token_from_utf8(self, lineno, col_offset):
    return self.get_token(lineno, self._line_numbers.from_utf8_col(lineno, col_offset))

Response: Get a token from a UTF8 line number and column offset.


Instruction:
Explain what the following Python 3 code does
def next_token(self, tok, include_extra=False):
    i = tok.index + 1
    if not include_extra:
      while is_non_coding_token(self._tokens[i].type):
        i += 1
    return self._tokens[i]

Response: Returns the next token after the given one.


Instruction:
How would you explain what the following Python 3 function does
def find_token(self, start_token, tok_type, tok_str=None, reverse=False):
    t = start_token
    advance = self.prev_token if reverse else self.next_token
    while not match_token(t, tok_type, tok_str) and not token.ISEOF(t.type):
      t = advance(t, include_extra=True)
    return t

Response: Finds the first token that matches tok_type and tok_str. Returns ENDMARKER token if found.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def token_range(self, first_token, last_token, include_extra=False):
    for i in xrange(first_token.index, last_token.index + 1):
      if include_extra or not is_non_coding_token(self._tokens[i].type):
        yield self._tokens[i]

Response: Yields all tokens in order from first_token through and including last_token.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_tokens(self, node, include_extra=False):
    return self.token_range(node.first_token, node.last_token, include_extra=include_extra)

Response: Yields all tokens making up the given node.


Instruction:
Explain what the following Python 3 code does
def get_text_range(self, node):
    if not hasattr(node, 'first_token'):
      return (0, 0)

    start = node.first_token.startpos
    if any(match_token(t, token.NEWLINE) for t in self.get_tokens(node)):
      # Multi-line nodes would be invalid unless we keep the indentation of the first node.
      start = self._text.rfind('\n', 0, start) + 1

    return (start, node.last_token.endpos)

Response: Returns the start and end positions of the source text that correspond to the given node.


Instruction:
Explain what the following Python 3 code does
def get_text(self, node):
    start, end = self.get_text_range(node)
    return self._text[start : end]

Response: Returns the text corresponding to the given node.


Instruction:
How would you explain what the following Python 3 function does
def from_utf8_col(self, line, utf8_column):
    offsets = self._utf8_offset_cache.get(line)
    if offsets is None:
      end_offset = self._line_offsets[line] if line < len(self._line_offsets) else self._text_len
      line_text = self._text[self._line_offsets[line - 1] : end_offset]

      offsets = [i for i,c in enumerate(line_text) for byte in c.encode('utf8')]
      offsets.append(len(line_text))
      self._utf8_offset_cache[line] = offsets

    return offsets[max(0, min(len(offsets)-1, utf8_column))]

Response: Given a 1 - based line number and 0 - based utf8 column returns a 0 - based unicode column.


Instruction:
How would you implement a function in Python 3 that
converts line number and column to character offset into text.

Response:
def line_to_offset(self, line, column):
    """
    Converts 1-based line number and 0-based column to 0-based character offset into text.
    """
    line -= 1
    if line >= len(self._line_offsets):
      return self._text_len
    elif line < 0:
      return 0
    else:
      return min(self._line_offsets[line] + max(0, column), self._text_len)


Instruction:
Can you tell what is the following Python 3 function doing
def offset_to_line(self, offset):
    offset = max(0, min(self._text_len, offset))
    line_index = bisect.bisect_right(self._line_offsets, offset) - 1
    return (line_index + 1, offset - self._line_offsets[line_index])

Response: Converts 0 - based character offset to line and column numbers.


Instruction:
Create a Python 3 function to
yield all tokens in [ first_token last_token ) that do not belong to any children of node.

Response:
def _iter_non_child_tokens(self, first_token, last_token, node):
    """
    Generates all tokens in [first_token, last_token] range that do not belong to any children of
    node. E.g. `foo(bar)` has children `foo` and `bar`, but we would yield the `(`.
    """
    tok = first_token
    for n in self._iter_children(node):
      for t in self._code.token_range(tok, self._code.prev_token(n.first_token)):
        yield t
      if n.last_token.index >= last_token.index:
        return
      tok = self._code.next_token(n.last_token)

    for t in self._code.token_range(tok, last_token):
      yield t


Instruction:
Can you implement a function in Python 3 that
scans tokens in [ first_token last_token ) and adjust first_token to include the closing pair.

Response:
def _expand_to_matching_pairs(self, first_token, last_token, node):
    """
    Scan tokens in [first_token, last_token] range that are between node's children, and for any
    unmatched brackets, adjust first/last tokens to include the closing pair.
    """
    # We look for opening parens/braces among non-child tokens (i.e. tokens between our actual
    # child nodes). If we find any closing ones, we match them to the opens.
    to_match_right = []
    to_match_left = []
    for tok in self._iter_non_child_tokens(first_token, last_token, node):
      tok_info = tok[:2]
      if to_match_right and tok_info == to_match_right[-1]:
        to_match_right.pop()
      elif tok_info in _matching_pairs_left:
        to_match_right.append(_matching_pairs_left[tok_info])
      elif tok_info in _matching_pairs_right:
        to_match_left.append(_matching_pairs_right[tok_info])

    # Once done, extend `last_token` to match any unclosed parens/braces.
    for match in reversed(to_match_right):
      last = self._code.next_token(last_token)
      # Allow for a trailing comma before the closing delimiter.
      if util.match_token(last, token.OP, ','):
        last = self._code.next_token(last)
      # Now check for the actual closing delimiter.
      if util.match_token(last, *match):
        last_token = last

    # And extend `first_token` to match any unclosed opening parens/braces.
    for match in to_match_left:
      first = self._code.prev_token(first_token)
      if util.match_token(first, *match):
        first_token = first

    return (first_token, last_token)


Instruction:
Can you tell what is the following Python 3 function doing
def match_token(token, tok_type, tok_str=None):
  return token.type == tok_type and (tok_str is None or token.string == tok_str)

Response: Returns true if the given token is of the given type and has that string.


Instruction:
Here you have a function in Python 3, explain what it does
def expect_token(token, tok_type, tok_str=None):
  if not match_token(token, tok_type, tok_str):
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
      token_repr(tok_type, tok_str), str(token),
      token.start[0], token.start[1] + 1))

Response: Verifies that the given token is of the expected type.


Instruction:
Create a Python 3 function to
visit a node in the tree.

Response:
def visit_tree(node, previsit, postvisit):
  """
  Scans the tree under the node depth-first using an explicit stack. It avoids implicit recursion
  via the function call stack to avoid hitting 'maximum recursion depth exceeded' error.

  It calls ``previsit()`` and ``postvisit()`` as follows:

  * ``previsit(node, par_value)`` - should return ``(par_value, value)``
        ``par_value`` is as returned from ``previsit()`` of the parent.

  * ``postvisit(node, par_value, value)`` - should return ``value``
        ``par_value`` is as returned from ``previsit()`` of the parent, and ``value`` is as
        returned from ``previsit()`` of this node itself. The return ``value`` is ignored except
        the one for the root node, which is returned from the overall ``visit_tree()`` call.

  For the initial node, ``par_value`` is None. Either ``previsit`` and ``postvisit`` may be None.
  """
  if not previsit:
    previsit = lambda node, pvalue: (None, None)
  if not postvisit:
    postvisit = lambda node, pvalue, value: None

  iter_children = iter_children_func(node)
  done = set()
  ret = None
  stack = [(node, None, _PREVISIT)]
  while stack:
    current, par_value, value = stack.pop()
    if value is _PREVISIT:
      assert current not in done    # protect againt infinite loop in case of a bad tree.
      done.add(current)

      pvalue, post_value = previsit(current, par_value)
      stack.append((current, par_value, post_value))

      # Insert all children in reverse order (so that first child ends up on top of the stack).
      ins = len(stack)
      for n in iter_children(current):
        stack.insert(ins, (n, pvalue, _PREVISIT))
    else:
      ret = postvisit(current, par_value, value)
  return ret


Instruction:
Can you generate a brief explanation for the following Python 3 code
def walk(node):
  iter_children = iter_children_func(node)
  done = set()
  stack = [node]
  while stack:
    current = stack.pop()
    assert current not in done    # protect againt infinite loop in case of a bad tree.
    done.add(current)

    yield current

    # Insert all children in reverse order (so that first child ends up on top of the stack).
    # This is faster than building a list and reversing it.
    ins = len(stack)
    for c in iter_children(current):
      stack.insert(ins, c)

Response: Recursively yield all descendant nodes in the tree starting at node.


Instruction:
How would you explain what the following Python 3 function does
def replace(text, replacements):
  p = 0
  parts = []
  for (start, end, new_text) in sorted(replacements):
    parts.append(text[p:start])
    parts.append(new_text)
    p = end
  parts.append(text[p:])
  return ''.join(parts)

Response: Replaces multiple slices of text with new values.


Instruction:
Here you have a function in Python 3, explain what it does
def get(self, obj, cls):
    method = self._cache.get(cls)
    if not method:
      name = "visit_" + cls.__name__.lower()
      method = getattr(obj, name, obj.visit_default)
      self._cache[cls] = method
    return method

Response: Returns the method that corresponds to the given object.


Instruction:
Write a Python 3 function for
returning a string representation of a dataframe value.

Response:
def _cellrepr(value, allow_formulas):
    """
    Get a string representation of dataframe value.

    :param :value: the value to represent
    :param :allow_formulas: if True, allow values starting with '='
            to be interpreted as formulas; otherwise, escape
            them with an apostrophe to avoid formula interpretation.
    """
    if pd.isnull(value) is True:
        return ""
    if isinstance(value, float):
        value = repr(value)
    else:
        value = str(value)
    if (not allow_formulas) and value.startswith('='):
        value = "'%s" % value
    return value


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _resize_to_minimum(worksheet, rows=None, cols=None):
    # get the current size
    current_cols, current_rows = (
        worksheet.col_count,
        worksheet.row_count
        )
    if rows is not None and rows <= current_rows:
        rows = None
    if cols is not None and cols <= current_cols:
        cols = None

    if cols is not None or rows is not None:
        worksheet.resize(rows, cols)

Response: Resize the current size of the node in the worksheet to guarantee a minimum size.


Instruction:
Create a Python 3 function for
returning the contents of the worksheet as a DataFrame.

Response:
def get_as_dataframe(worksheet,
                     evaluate_formulas=False,
                     **options):
    """
    Returns the worksheet contents as a DataFrame.

    :param worksheet: the worksheet.
    :param evaluate_formulas: if True, get the value of a cell after
            formula evaluation; otherwise get the formula itself if present.
            Defaults to False.
    :param \*\*options: all the options for pandas.io.parsers.TextParser,
            according to the version of pandas that is installed.
            (Note: TextParser supports only the default 'python' parser engine,
            not the C engine.)
    :returns: pandas.DataFrame
    """
    all_values = _get_all_values(worksheet, evaluate_formulas)
    return TextParser(all_values, **options).read()


Instruction:
Can you generate the documentation for the following Python 3 function
def set_with_dataframe(worksheet,
                       dataframe,
                       row=1,
                       col=1,
                       include_index=False,
                       include_column_header=True,
                       resize=False,
                       allow_formulas=True):
    """
    Sets the values of a given DataFrame, anchoring its upper-left corner
    at (row, col). (Default is row 1, column 1.)

    :param worksheet: the gspread worksheet to set with content of DataFrame.
    :param dataframe: the DataFrame.
    :param include_index: if True, include the DataFrame's index as an
            additional column. Defaults to False.
    :param include_column_header: if True, add a header row before data with
            column names. (If include_index is True, the index's name will be
            used as its column's header.) Defaults to True.
    :param resize: if True, changes the worksheet's size to match the shape
            of the provided DataFrame. If False, worksheet will only be
            resized as necessary to contain the DataFrame contents.
            Defaults to False.
    :param allow_formulas: if True, interprets `=foo` as a formula in
            cell values; otherwise all text beginning with `=` is escaped
            to avoid its interpretation as a formula. Defaults to True.
    """
    # x_pos, y_pos refers to the position of data rows only,
    # excluding any header rows in the google sheet.
    # If header-related params are True, the values are adjusted
    # to allow space for the headers.
    y, x = dataframe.shape
    if include_index:
        x += 1
    if include_column_header:
        y += 1
    if resize:
        worksheet.resize(y, x)
    else:
        _resize_to_minimum(worksheet, y, x)

    updates = []

    if include_column_header:
        elts = list(dataframe.columns)
        if include_index:
            elts = [ dataframe.index.name ] + elts
        for idx, val in enumerate(elts):
            updates.append(
                (row,
                 col+idx,
                 _cellrepr(val, allow_formulas))
            )
        row += 1

    values = []
    for value_row, index_value in zip_longest(dataframe.values, dataframe.index):
        if include_index:
            value_row = [index_value] + list(value_row)
        values.append(value_row)
    for y_idx, value_row in enumerate(values):
        for x_idx, cell_value in enumerate(value_row):
            updates.append(
                (y_idx+row,
                 x_idx+col,
                 _cellrepr(cell_value, allow_formulas))
            )

    if not updates:
        logger.debug("No updates to perform on worksheet.")
        return

    cells_to_update = [ Cell(row, col, value) for row, col, value in updates ]
    logger.debug("%d cell updates to send", len(cells_to_update))

    resp = worksheet.update_cells(cells_to_update, value_input_option='USER_ENTERED')
    logger.debug("Cell update response: %s", resp)

Response: Sets the values of a given DataFrame in a new gspread file.


Instruction:
Can you generate the documentation for the following Python 3 function
def primitive(self):
        primitive = {}
        if self.entry_number is not None:
            primitive['entry-number'] = self.entry_number

        if self.item_hash is not None:
            primitive['item-hash'] = self.item_hash

        primitive['timestamp'] = self.timestamp.strftime(fmt)
        return primitive

Response: Entry as Python primitive.


Instruction:
How would you explain what the following Python 3 function does
def config(self, name, suffix):
        "Return config variable value, defaulting to environment"
        var = '%s_%s' % (name, suffix)
        var = var.upper().replace('-', '_')
        if var in self._config:
            return self._config[var]
        return os.environ[var]

Response: Return config variable value defaulting to environment


Instruction:
Make a summary of the following Python 3 code
def index(self, index, field, value):
        "Search for records matching a value in an index service"
        params = {
            "q": value,
            # search index has '_' instead of '-' in field names ..
            "q.options": "{fields:['%s']}" % (field.replace('-', '_'))
        }

        response = self.get(self.config(index, 'search_url'), params=params)
        results = [hit['fields'] for hit in response.json()['hits']['hit']]

        for result in results:
            for key in result:
                result[key.replace('_', '-')] = result.pop(key)

        return results

Response: Search for records matching a value in an index service


Instruction:
Can you generate the documentation for the following Python 3 function
def primitive(self):
        dict = {}
        for key, value in self.__dict__.items():
            if not key.startswith('_'):
                dict[key] = copy(value)

        for key in dict:
            if isinstance(dict[key], (set)):
                dict[key] = sorted(list(dict[key]))

        return dict

Response: Python primitive representation of the object.


Instruction:
Here you have a function in Python 3, explain what it does
def primitive(self, dictionary):
        self.__dict__ = {k: v for k, v in dictionary.items() if v}

Response: Item from Python primitive.


Instruction:
Can you generate the documentation for the following Python 3 function
def primitive(self):
        primitive = copy(self.item.primitive)
        primitive.update(self.entry.primitive)
        return primitive

Response: Record as Python primitive.


Instruction:
Write a Python 3 function that can
record from Python primitive.

Response:
def primitive(self, primitive):
        """Record from Python primitive."""
        self.entry = Entry()
        self.entry.primitive = primitive

        primitive = copy(primitive)
        for field in self.entry.fields:
            del primitive[field]

        self.item = Item()
        self.item.primitive = primitive


Instruction:
Can you write a function in Python 3 where it
reads Items from a stream containing TSV.

Response:
def reader(stream, fieldnames=None):
    """Read Items from a stream containing TSV."""
    if not fieldnames:
        fieldnames = load_line(stream.readline())
    for line in stream:
        values = load_line(line)
        item = Item()
        item.__dict__ = dict(zip(fieldnames, values))
        yield item


Instruction:
Given the following Python 3 function, write the documentation
def dump(self):
    dict = self.primitive
    if not dict:
        return ''
    return dump_line(self.keys) + dump_line(self.values)

Response: Dump the dictionary into a TSV representation.


Instruction:
Write a Python 3 script for
loading the object from CSV representation.

Response:
def load(self, text,
         lineterminator='\r\n',
         quotechar='"',
         delimiter=",",
         escapechar=escapechar,
         quoting=csv.QUOTE_MINIMAL):
    """Item from CSV representation."""

    f = io.StringIO(text)

    if not quotechar:
        quoting = csv.QUOTE_NONE

    reader = csv.DictReader(
        f,
        delimiter=delimiter,
        quotechar=quotechar,
        quoting=quoting,
        lineterminator=lineterminator)

    if reader.fieldnames:
        reader.fieldnames = [field.strip() for field in reader.fieldnames]

    try:
        self.primitive = next(reader)
    except StopIteration:
        self.primitive = {}


Instruction:
Can you tell what is the following Python 3 function doing
def dump(self, **kwargs):

    f = io.StringIO()
    w = Writer(f, self.keys, **kwargs)
    w.write(self)
    text = f.getvalue().lstrip()
    f.close()

    return text

Response: CSV representation of a item.


Instruction:
Explain what the following Python 3 code does
def git_hash(blob):
    head = str("blob " + str(len(blob)) + "\0").encode("utf-8")
    return sha1(head + blob).hexdigest()

Response: Return git - hash compatible SHA - 1 hexdigits for a blob of data.


Instruction:
How would you code a function in Python 3 to
dump the object as a JSON representation.

Response:
def dump(self):
    """Item as a JSON representation."""
    return json.dumps(
        self.primitive,
        sort_keys=True,
        ensure_ascii=False,
        separators=(',', ':'))


Instruction:
How would you explain what the following Python 3 function does
def reader(stream):
    string = stream.read()
    decoder = json.JSONDecoder().raw_decode
    index = START.match(string, 0).end()

    while index < len(string):
        obj, end = decoder(string, index)
        item = Item()
        item.primitive = obj
        yield item
        index = END.match(string, end).end()

Response: Read Items from a stream containing a JSON array.


Instruction:
Make a summary of the following Python 3 code
def reader(stream):
    for line in stream:
        item = Item()
        item.json = line
        yield item

Response: Read Items from a stream containing lines of JSON.


Instruction:
Write a Python 3 function for
calculating statistics for a collection

Response:
def meta(self, total, page=1, page_size=None):
        """
        Calculate statistics for a collection
        return: meta
        """
        if page_size is None or page_size < 0:
            page_size = self.page_size

        meta = {}
        meta['total'] = total
        meta['page_size'] = page_size
        meta['pages'] = math.ceil(meta['total']/page_size)
        meta['page'] = page
        meta['skip'] = page_size * (page-1)
        return meta


Instruction:
How would you explain what the following Python 3 function does
def tag_id(self, name):
        return self._store.get(self.tag_key(name)) or self.reset_tag(name)

Response: Get the unique tag identifier for a given tag.


Instruction:
Implement a function in Python 3 to
reset the tag and return the new tag identifier.

Response:
def reset_tag(self, name):
        """
        Reset the tag and return the new tag identifier.

        :param name: The tag
        :type name: str

        :rtype: str
        """
        id_ = str(uuid.uuid4()).replace('-', '')

        self._store.forever(self.tag_key(name), id_)

        return id_


Instruction:
Write a Python 3 function that can
install log handler on Flask application.

Response:
def install_handler(self, app):
        """Install log handler on Flask application."""
        # Check if directory exists.
        basedir = dirname(app.config['LOGGING_FS_LOGFILE'])
        if not exists(basedir):
            raise ValueError(
                'Log directory {0} does not exists.'.format(basedir))

        handler = RotatingFileHandler(
            app.config['LOGGING_FS_LOGFILE'],
            backupCount=app.config['LOGGING_FS_BACKUPCOUNT'],
            maxBytes=app.config['LOGGING_FS_MAXBYTES'],
            delay=True,
        )
        handler.setFormatter(logging.Formatter(
            '%(asctime)s %(levelname)s: %(message)s '
            '[in %(pathname)s:%(lineno)d]'
        ))
        handler.setLevel(app.config['LOGGING_FS_LEVEL'])

        # Add handler to application logger
        app.logger.addHandler(handler)

        if app.config['LOGGING_FS_PYWARNINGS']:
            self.capture_pywarnings(handler)

        # Add request_id to log record
        app.logger.addFilter(add_request_id_filter)


Instruction:
Can you implement a function in Python 3 that
stores an item in the cache for a given number of minutes.

Response:
def put(self, key, value, minutes):
        """
        Store an item in the cache for a given number of minutes.

        :param key: The cache key
        :type key: str

        :param value: The cache value
        :type value: mixed

        :param minutes: The lifetime in minutes of the cached value
        :type minutes: int or datetime
        """
        minutes = self._get_minutes(minutes)

        if minutes is not None:
            return self._store.put(self.tagged_item_key(key), value, minutes)


Instruction:
Make a summary of the following Python 3 code
def add(self, key, val, minutes):
        if not self.has(key):
            self.put(key, val, minutes)

            return True

        return False

Response: Stores an item in the cache if it does not exist. Returns True if the item was added False otherwise.


Instruction:
Write a Python 3 script for
incrementing the value of an item in the cache.

Response:
def increment(self, key, value=1):
        """
        Increment the value of an item in the cache.

        :param key: The cache key
        :type key: str

        :param value: The increment value
        :type value: int

        :rtype: int or bool
        """
        self._store.increment(self.tagged_item_key(key), value)


Instruction:
Implement a Python 3 function for
storing an item in the cache indefinitely.

Response:
def forever(self, key, value):
        """
        Store an item in the cache indefinitely.

        :param key: The cache key
        :type key: str

        :param value: The value
        :type value: mixed
        """
        self._store.forever(self.tagged_item_key(key), value)


Instruction:
Can you write a function in Python 3 where it
gets an item from the cache or store the default value.

Response:
def remember(self, key, minutes, callback):
        """
        Get an item from the cache, or store the default value.

        :param key: The cache key
        :type key: str

        :param minutes: The lifetime in minutes of the cached value
        :type minutes: int or datetime

        :param callback: The default function
        :type callback: mixed

        :rtype: mixed
        """
        # If the item exists in the cache we will just return this immediately
        # otherwise we will execute the given callback and cache the result
        # of that execution for the given number of minutes in storage.
        val = self.get(key)
        if val is not None:
            return val

        val = value(callback)

        self.put(key, val, minutes)

        return val


Instruction:
Implement a function in Python 3 to
get an item from the cache or store the default value forever.

Response:
def remember_forever(self, key, callback):
        """
        Get an item from the cache, or store the default value forever.

        :param key: The cache key
        :type key: str

        :param callback: The default function
        :type callback: mixed

        :rtype: mixed
        """
        # If the item exists in the cache we will just return this immediately
        # otherwise we will execute the given callback and cache the result
        # of that execution forever.
        val = self.get(key)
        if val is not None:
            return val

        val = value(callback)

        self.forever(key, val)

        return val


Instruction:
Can you generate the documentation for the following Python 3 function
def tagged_item_key(self, key):
        return '%s:%s' % (hashlib.sha1(encode(self._tags.get_namespace())).hexdigest(), key)

Response: Get a fully qualified key for a tagged item.


Instruction:
Write a Python 3 function that can
calculate the number of minutes with the given duration.

Response:
def _get_minutes(self, duration):
        """
        Calculate the number of minutes with the given duration.

        :param duration: The duration
        :type duration: int or datetime

        :rtype: int or None
        """
        if isinstance(duration, datetime.datetime):
            from_now = (duration - datetime.datetime.now()).total_seconds()
            from_now = math.ceil(from_now / 60)

            if from_now > 0:
                return from_now

            return

        return duration


Instruction:
Can you generate the documentation for the following Python 3 function
async def main():
    with aiohttp.ClientSession() as session:
        data = Netdata('localhost', loop, session, data='data')
        await data.get_data('system.cpu')

        print(json.dumps(data.values, indent=4, sort_keys=True))

        # Print the current value of the system's CPU
        print("CPU System:", round(data.values['system'], 2))

    with aiohttp.ClientSession() as session:
        data = Netdata('localhost', loop, session, data='alarms')
        await data.get_alarms()

        print(data.alarms)

    with aiohttp.ClientSession() as session:
        data = Netdata('localhost', loop, session)
        await data.get_allmetrics()

        print(data.metrics)

        # Print the current value for the system's CPU
        print("CPU System:", round(data.metrics['system.cpu']
                                   ['dimensions']['system']['value'], 2))

Response: Get the data from a Netdata instance.


Instruction:
Create a Python 3 function for
executing a command and wait for its completion

Response:
def execute(cmd, shell=False, poll_period=1.0, catch_out=False):
    """Execute UNIX command and wait for its completion

    Args:
        cmd (str or list): command to execute
        shell (bool): invoke inside shell environment
        catch_out (bool): collect process' output

    Returns:
        returncode (int): process return code
        stdout (str): collected process stdout (only if catch_out set to true)
        stderr (str): collected process stderr (only if catch_out set to true)
    """
    # FIXME: move to module level
    log = logging.getLogger(__name__)
    log.debug("Starting: %s", cmd)

    stdout = ""
    stderr = ""

    if not shell and isinstance(cmd, string_types):
        cmd = shlex.split(cmd)

    if catch_out:
        process = subprocess.Popen(
            cmd,
            shell=shell,
            stderr=subprocess.PIPE,
            stdout=subprocess.PIPE,
            close_fds=True)
    else:
        process = subprocess.Popen(cmd, shell=shell, close_fds=True)

    stdout, stderr = process.communicate()
    if stderr:
        log.error("There were errors:\n%s", stderr)

    if stdout:
        log.debug("Process output:\n%s", stdout)
    returncode = process.returncode
    log.debug("Process exit code: %s", returncode)
    return returncode, stdout, stderr


Instruction:
Write a Python 3 script to
return the payload of a cache entry.

Response:
def _get_payload(self, key):
        """
        Retrieve an item and expiry time from the cache by key.

        :param key: The cache key
        :type key: str

        :rtype: dict
        """
        path = self._path(key)

        # If the file doesn't exists, we obviously can't return the cache so we will
        # just return null. Otherwise, we'll get the contents of the file and get
        # the expiration UNIX timestamps from the start of the file's contents.
        if not os.path.exists(path):
            return {'data': None, 'time': None}

        with open(path, 'rb') as fh:
            contents = fh.read()

        expire = int(contents[:10])

        # If the current time is greater than expiration timestamps we will delete
        # the file and return null. This helps clean up the old files and keeps
        # this directory much cleaner for us as old files aren't hanging out.
        if round(time.time()) >= expire:
            self.forget(key)

            return {'data': None, 'time': None}

        data = self.unserialize(contents[10:])

        # Next, we'll extract the number of minutes that are remaining for a cache
        # so that we can properly retain the time for things like the increment
        # operation that may be performed on the cache. We'll round this out.
        time_ = math.ceil((expire - round(time.time())) / 60.)

        return {'data': data, 'time': time_}


Instruction:
Can you write a function in Python 3 where it
stores an item in the cache for a given number of minutes.

Response:
def put(self, key, value, minutes):
        """
        Store an item in the cache for a given number of minutes.

        :param key: The cache key
        :type key: str

        :param value: The cache value
        :type value: mixed

        :param minutes: The lifetime in minutes of the cached value
        :type minutes: int
        """
        value = encode(str(self._expiration(minutes))) + encode(self.serialize(value))

        path = self._path(key)
        self._create_cache_directory(path)

        with open(path, 'wb') as fh:
            fh.write(value)


Instruction:
Create a Python 3 function for
incrementing the value of an item in the cache.

Response:
def increment(self, key, value=1):
        """
        Increment the value of an item in the cache.

        :param key: The cache key
        :type key: str

        :param value: The increment value
        :type value: int

        :rtype: int or bool
        """
        raw = self._get_payload(key)

        integer = int(raw['data']) + value

        self.put(key, integer, int(raw['time']))

        return integer


Instruction:
Create a Python 3 function for
removing an item from the cache.

Response:
def forget(self, key):
        """
        Remove an item from the cache.

        :param key: The cache key
        :type key: str

        :rtype: bool
        """
        path = self._path(key)

        if os.path.exists(path):
            os.remove(path)

            return True

        return False


Instruction:
How would you explain what the following Python 3 function does
def flush(self):
        if os.path.isdir(self._directory):
            for root, dirs, files in os.walk(self._directory, topdown=False):
                for name in files:
                    os.remove(os.path.join(root, name))

                for name in dirs:
                    os.rmdir(os.path.join(root, name))

Response: Remove all items from the cache.


Instruction:
Can you implement a function in Python 3 that
gets the full path for the given cache key.

Response:
def _path(self, key):
        """
        Get the full path for the given cache key.

        :param key: The cache key
        :type key: str

        :rtype: str
        """
        hash_type, parts_count = self._HASHES[self._hash_type]
        h = hash_type(encode(key)).hexdigest()

        parts = [h[i:i+2] for i in range(0, len(h), 2)][:parts_count]

        return os.path.join(self._directory, os.path.sep.join(parts), h)


Instruction:
Write a Python 3 function that can
return the content of a resource file.

Response:
def resource_string(self, path):
        """
        Args:
            path: str, resource file url or resource file absolute/relative path.

        Returns:
            string, file content
        """
        opener = self.get_opener(path)
        filename = opener.get_filename
        try:
            size = os.path.getsize(filename)
            if size > 50 * 1024 * 1024:
                logger.warning(
                    'Reading large resource to memory: %s. Size: %s bytes',
                    filename, size)
        except Exception as exc:
            logger.debug('Unable to check resource size %s. %s', filename, exc)
        with opener(filename, 'r') as resource:
            content = resource.read()
        return content


Instruction:
Implement a function in Python 3 to
return the opener object for the given path.

Response:
def get_opener(self, path):
        """
        Args:
            path: str, resource file url or resource file absolute/relative path.

        Returns:
            file object
        """
        self.path = path
        opener = None
        # FIXME this parser/matcher should use `urlparse` stdlib
        for opener_name, signature in self.openers.items():
            if self.path.startswith(signature[0]):
                opener = signature[1](self.path)
                break
        if not opener:
            opener = FileOpener(self.path)
        return opener


Instruction:
Can you tell what is the following Python 3 function doing
def tags(self, *names):
        if len(names) == 1 and isinstance(names[0], list):
            names = names[0]

        return TaggedCache(self, TagSet(self, names))

Response: Begin executing a new tags operation.


Instruction:
Can you implement a function in Python 3 that
creates public Luna job.

Response:
def create_job(self):
        """ Create public Luna job

        Returns:
            job_id (basestring): Luna job id
        """
        my_user_agent = None
        try:
            my_user_agent = pkg_resources.require('netort')[0].version
        except pkg_resources.DistributionNotFound:
            my_user_agent = 'DistributionNotFound'
        finally:
            headers = {
                "User-Agent": "Uploader/{uploader_ua}, {upward_ua}".format(
                    upward_ua=self.meta.get('user_agent', ''),
                    uploader_ua=my_user_agent
                )
            }
        req = requests.Request(
            'POST',
            "{api_address}{path}".format(
                api_address=self.api_address,
                path=self.create_job_path
            ),
            headers=headers
        )
        req.data = {
            'test_start': self.job.test_start
        }
        prepared_req = req.prepare()
        logger.debug('Prepared create_job request:\n%s', pretty_print(prepared_req))

        response = send_chunk(self.session, prepared_req)
        logger.debug('Luna create job status: %s', response.status_code)
        logger.debug('Answ data: %s', response.content)
        job_id = response.content.decode('utf-8') if isinstance(response.content, bytes) else response.content
        if not job_id:
            self.failed.set()
            raise ValueError('Luna returned answer without jobid: %s', response.content)
        else:
            logger.info('Luna job created: %s', job_id)
            return job_id


Instruction:
Explain what the following Python 3 code does
def new_metric(self, meta):
        type_ = meta.get('type')
        if not type_:
            raise ValueError('Metric type should be defined.')

        if type_ in available_metrics:
            metric_obj = available_metrics[type_](meta, self.routing_queue)  # create metric object
            metric_meta = pd.DataFrame({metric_obj.local_id: meta}).T  # create metric meta
            self.metrics_meta = self.metrics_meta.append(metric_meta)  # register metric meta
            self.metrics[metric_obj.local_id] = metric_obj  # register metric object

            # find subscribers for this metric
            this_metric_subscribers = self.__reversed_filter(self.subscribers, meta)
            if this_metric_subscribers.empty:
                logger.debug('subscriber for metric %s not found', metric_obj.local_id)
            else:
                logger.debug('Found subscribers for this metric, subscribing...: %s', this_metric_subscribers)
                # attach this metric id to discovered subscribers and select id <-> callbacks
                this_metric_subscribers['id'] = metric_obj.local_id
                found_callbacks = this_metric_subscribers[['id', 'callback']].set_index('id')
                # add this metric callbacks to DataManager's callbacks
                self.callbacks = self.callbacks.append(found_callbacks)
            return metric_obj
        else:
            raise NotImplementedError('Unknown metric type: %s' % type_)

Response: Create and register a new metric object and return it


Instruction:
Create a Python 3 function for
creating and register a new tank metric.

Response:
def new_tank_metric(self, _type, name, hostname=None, group=None, source=None, *kw):
        """
        Create and register metric,
        find subscribers for this metric (using meta as filter) and subscribe

        Return:
            metric (available_metrics[0]): one of Metric
        """
        if not _type:
            raise ValueError('Metric type should be defined.')

        if _type in available_metrics:
            metric_obj = available_metrics[_type](None, self.routing_queue)  # create metric object
            metric_info = {'type': _type,
                           'source': source,
                           'name': name,
                           'hostname': hostname,
                           'group': group}
            if kw is not None:
                metric_info.update(kw)
            metric_meta = pd.DataFrame({metric_obj.local_id: metric_info}).T  # create metric meta
            self.metrics_meta = self.metrics_meta.append(metric_meta)  # register metric meta
            self.metrics[metric_obj.local_id] = metric_obj  # register metric object

            # find subscribers for this metric
            this_metric_subscribers = self.__reversed_filter(self.subscribers, metric_info)
            if this_metric_subscribers.empty:
                logger.debug('subscriber for metric %s not found', metric_obj.local_id)
            else:
                logger.debug('Found subscribers for this metric, subscribing...: %s', this_metric_subscribers)
                # attach this metric id to discovered subscribers and select id <-> callbacks
                this_metric_subscribers['id'] = metric_obj.local_id
                found_callbacks = this_metric_subscribers[['id', 'callback']].set_index('id')
                # add this metric callbacks to DataManager's callbacks
                self.callbacks = self.callbacks.append(found_callbacks)
            return metric_obj
        else:
            raise NotImplementedError('Unknown metric type: %s' % _type)


Instruction:
How would you implement a function in Python 3 that
subscribes to the manager and register the callback for each metric that has been discovered.

Response:
def subscribe(self, callback, filter_):
        """
        Create and register metric subscriber,
        find metrics for this subscriber (using filter_) and subscribe

        Args:
            callback (object method): subscriber's callback
            filter_ (dict): filter dict

        filter sample:
            {'type': 'metrics', 'source': 'gun'}
        """
        sub_id = "subscriber_{uuid}".format(uuid=uuid.uuid4())
        # register subscriber in manager
        sub = pd.DataFrame({sub_id: filter_}).T
        sub['callback'] = callback
        self.subscribers = self.subscribers.append(sub)

        # find metrics for subscriber using `filter`
        this_subscriber_metrics = self.__filter(self.metrics_meta, filter_)
        if this_subscriber_metrics.empty:
            logger.debug('Metrics for subscriber %s not found', sub_id)
        else:
            logger.debug('Found metrics for this subscriber, subscribing...: %s', this_subscriber_metrics)
            # attach this sub callback to discovered metrics and select id <-> callbacks
            this_subscriber_metrics['callback'] = callback
            prepared_callbacks = this_subscriber_metrics[['callback']]
            # add this subscriber callbacks to DataManager's callbacks
            self.callbacks = self.callbacks.append(prepared_callbacks)