Instruction:
Can you write a function in Python 3 where it
installs all packages in this set.

Response:
def install(self, install_options, global_options=(), *args, **kwargs):
        """
        Install everything in this set (after having downloaded and unpacked
        the packages)
        """
        to_install = self._to_install()

        if to_install:
            logger.info(
                'Installing collected packages: %s',
                ', '.join([req.name for req in to_install]),
            )

        with indent_log():
            for requirement in to_install:
                if requirement.conflicts_with:
                    logger.info(
                        'Found existing installation: %s',
                        requirement.conflicts_with,
                    )
                    with indent_log():
                        requirement.uninstall(auto_confirm=True)
                try:
                    requirement.install(
                        install_options,
                        global_options,
                        *args,
                        **kwargs
                    )
                except:
                    # if install did not succeed, rollback previous uninstall
                    if (requirement.conflicts_with and not
                            requirement.install_succeeded):
                        requirement.rollback_uninstall()
                    raise
                else:
                    if (requirement.conflicts_with and
                            requirement.install_succeeded):
                        requirement.commit_uninstall()
                requirement.remove_temporary_source()

        self.successfully_installed = to_install


Instruction:
Write a Python 3 function for
returning sorted list of all package namespaces

Response:
def _get_all_ns_packages(self):
        """Return sorted list of all package namespaces"""
        nsp = set()
        for pkg in self.distribution.namespace_packages or []:
            pkg = pkg.split('.')
            while pkg:
                nsp.add('.'.join(pkg))
                pkg.pop()
        return sorted(nsp)


Instruction:
Here you have a function in Python 3, explain what it does
def default(self, obj):
        if isinstance(obj, models.Model):
            return self.encode(model_to_dict(obj))
        elif isinstance(obj, models.query.QuerySet):
            return serializers.serialize('json', obj)
        else:
            return super(JsonResponseEncoder, self).default(obj)

Response: Convert the given object into a list of counter - parts.


Instruction:
Can you implement a function in Python 3 that
takes a list of documents and returns a list of HTML documents that are annotated with the given markup.

Response:
def html_annotate(doclist, markup=default_markup):
    """
    doclist should be ordered from oldest to newest, like::

        >>> version1 = 'Hello World'
        >>> version2 = 'Goodbye World'
        >>> print(html_annotate([(version1, 'version 1'),
        ...                      (version2, 'version 2')]))
        Goodbye World

    The documents must be *fragments* (str/UTF8 or unicode), not
    complete documents

    The markup argument is a function to markup the spans of words.
    This function is called like markup('Hello', 'version 2'), and
    returns HTML.  The first argument is text and never includes any
    markup.  The default uses a span with a title:

        >>> print(default_markup('Some Text', 'by Joe'))
        Some Text
    """
    # The basic strategy we have is to split the documents up into
    # logical tokens (which are words with attached markup).  We then
    # do diffs of each of the versions to track when a token first
    # appeared in the document; the annotation attached to the token
    # is the version where it first appeared.
    tokenlist = [tokenize_annotated(doc, version)
                 for doc, version in doclist]
    cur_tokens = tokenlist[0]
    for tokens in tokenlist[1:]:
        html_annotate_merge_annotations(cur_tokens, tokens)
        cur_tokens = tokens

    # After we've tracked all the tokens, we can combine spans of text
    # that are adjacent and have the same annotation
    cur_tokens = compress_tokens(cur_tokens)
    # And finally add markup
    result = markup_serialize_tokens(cur_tokens, markup)
    return ''.join(result).strip()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def tokenize_annotated(doc, annotation):
    tokens = tokenize(doc, include_hrefs=False)
    for tok in tokens:
        tok.annotation = annotation
    return tokens

Response: Tokenize a document and add an annotation attribute to each token


Instruction:
Explain what the following Python 3 code does
def html_annotate_merge_annotations(tokens_old, tokens_new):
    s = InsensitiveSequenceMatcher(a=tokens_old, b=tokens_new)
    commands = s.get_opcodes()

    for command, i1, i2, j1, j2 in commands:
        if command == 'equal':
            eq_old = tokens_old[i1:i2]
            eq_new = tokens_new[j1:j2]
            copy_annotations(eq_old, eq_new)

Response:
Merge the annotations from tokens_old into tokens_new when the
    tokens in the old document already existed in the new document.


Instruction:
Create a Python 3 function to
copy annotations from the tokens listed in src to the tokens listed in dest.

Response:
def copy_annotations(src, dest):
    """
    Copy annotations from the tokens listed in src to the tokens in dest
    """
    assert len(src) == len(dest)
    for src_tok, dest_tok in zip(src, dest):
        dest_tok.annotation = src_tok.annotation


Instruction:
Can you generate the documentation for the following Python 3 function
def compress_tokens(tokens):
    result = [tokens[0]]
    for tok in tokens[1:]:
        if (not result[-1].post_tags and
            not tok.pre_tags and
            result[-1].annotation == tok.annotation):
            compress_merge_back(result, tok)
        else:
            result.append(tok)
    return result

Response: Combine adjacent tokens when there is no HTML between the tokens and their annotations.


Instruction:
Implement a function in Python 3 to
merge tok into the last element of tokens in - place.

Response:
def compress_merge_back(tokens, tok):
    """ Merge tok into the last element of tokens (modifying the list of
    tokens in-place).  """
    last = tokens[-1]
    if type(last) is not token or type(tok) is not token:
        tokens.append(tok)
    else:
        text = _unicode(last)
        if last.trailing_whitespace:
            text += last.trailing_whitespace
        text += tok
        merged = token(text,
                       pre_tags=last.pre_tags,
                       post_tags=tok.post_tags,
                       trailing_whitespace=tok.trailing_whitespace)
        merged.annotation = last.annotation
        tokens[-1] = merged


Instruction:
Can you generate the documentation for the following Python 3 function
def markup_serialize_tokens(tokens, markup_func):
    for token in tokens:
        for pre in token.pre_tags:
            yield pre
        html = token.html()
        html = markup_func(html, token.annotation)
        if token.trailing_whitespace:
            html += token.trailing_whitespace
        yield html
        for post in token.post_tags:
            yield post

Response: Serialize the list of tokens into a list of text chunks calling markup_func around text to add annotations.


Instruction:
How would you explain what the following Python 3 function does
def htmldiff(old_html, new_html):
    ## FIXME: this should take parsed documents too, and use their body
    ## or other content.
    """ Do a diff of the old and new document.  The documents are HTML
    *fragments* (str/UTF8 or unicode), they are not complete documents
    (i.e., no <html> tag).

    Returns HTML with <ins> and <del> tags added around the
    appropriate text.

    Markup is generally ignored, with the markup from new_html
    preserved, and possibly some markup from old_html (though it is
    considered acceptable to lose some of the old markup).  Only the
    words in the HTML are diffed.  The exception is <img> tags, which
    are treated like words, and the href attribute of <a> tags, which
    are noted inside the tag itself when there are changes.
    """
    old_html_tokens = tokenize(old_html)
    new_html_tokens = tokenize(new_html)
    result = htmldiff_tokens(old_html_tokens, new_html_tokens)
    result = ''.join(result).strip()
    return fixup_ins_del_tags(result)

Response: Do a diff of the old and new HTML document.


Instruction:
Given the following Python 3 function, write the documentation
def htmldiff_tokens(html1_tokens, html2_tokens):
    # There are several passes as we do the differences.  The tokens
    # isolate the portion of the content we care to diff; difflib does
    # all the actual hard work at that point.
    #
    # Then we must create a valid document from pieces of both the old
    # document and the new document.  We generally prefer to take
    # markup from the new document, and only do a best effort attempt
    # to keep markup from the old document; anything that we can't
    # resolve we throw away.  Also we try to put the deletes as close
    # to the location where we think they would have been -- because
    # we are only keeping the markup from the new document, it can be
    # fuzzy where in the new document the old text would have gone.
    # Again we just do a best effort attempt.
    s = InsensitiveSequenceMatcher(a=html1_tokens, b=html2_tokens)
    commands = s.get_opcodes()
    result = []
    for command, i1, i2, j1, j2 in commands:
        if command == 'equal':
            result.extend(expand_tokens(html2_tokens[j1:j2], equal=True))
            continue
        if command == 'insert' or command == 'replace':
            ins_tokens = expand_tokens(html2_tokens[j1:j2])
            merge_insert(ins_tokens, result)
        if command == 'delete' or command == 'replace':
            del_tokens = expand_tokens(html1_tokens[i1:i2])
            merge_delete(del_tokens, result)
    # If deletes were inserted directly as <del> then we'd have an
    # invalid document at this point.  Instead we put in special
    # markers, and when the complete diffed document has been created
    # we try to move the deletes around and resolve any problems.
    result = cleanup_delete(result)

    return result

Response:
Does a diff on the given HTML tokens returning a list of text
    chunks.


Instruction:
Can you create a Python 3 function that
gives a list of tokens return a generator of the chunks of the text for the data in the tokens.

Response:
def expand_tokens(tokens, equal=False):
    """Given a list of tokens, return a generator of the chunks of
    text for the data in the tokens.
    """
    for token in tokens:
        for pre in token.pre_tags:
            yield pre
        if not equal or not token.hide_when_equal:
            if token.trailing_whitespace:
                yield token.html() + token.trailing_whitespace
            else:
                yield token.html()
        for post in token.post_tags:
            yield post


Instruction:
Implement a Python 3 function for
merging the text chunks in ins_chunks into the document doc.

Response:
def merge_insert(ins_chunks, doc):
    """ doc is the already-handled document (as a list of text chunks);
    here we add ins_chunks to the end of that.  """
    # Though we don't throw away unbalanced_start or unbalanced_end
    # (we assume there is accompanying markup later or earlier in the
    # document), we only put  around the balanced portion.
    unbalanced_start, balanced, unbalanced_end = split_unbalanced(ins_chunks)
    doc.extend(unbalanced_start)
    if doc and not doc[-1].endswith(' '):
        # Fix up the case where the word before the insert didn't end with
        # a space
        doc[-1] += ' '
    doc.append('')
    if balanced and balanced[-1].endswith(' '):
        # We move space outside of
        balanced[-1] = balanced[-1][:-1]
    doc.extend(balanced)
    doc.append(' ')
    doc.extend(unbalanced_end)


Instruction:
Implement a Python 3 function for
adding the text chunks in del_chunks to the document doc.

Response:
def merge_delete(del_chunks, doc):
    """ Adds the text chunks in del_chunks to the document doc (another
    list of text chunks) with marker to show it is a delete.
    cleanup_delete later resolves these markers into <del> tags."""
    doc.append(DEL_START)
    doc.extend(del_chunks)
    doc.append(DEL_END)


Instruction:
Make a summary of the following Python 3 code
def cleanup_delete(chunks):
    while 1:
        # Find a pending DEL_START/DEL_END, splitting the document
        # into stuff-preceding-DEL_START, stuff-inside, and
        # stuff-following-DEL_END
        try:
            pre_delete, delete, post_delete = split_delete(chunks)
        except NoDeletes:
            # Nothing found, we've cleaned up the entire doc
            break
        # The stuff-inside-DEL_START/END may not be well balanced
        # markup.  First we figure out what unbalanced portions there are:
        unbalanced_start, balanced, unbalanced_end = split_unbalanced(delete)
        # Then we move the span forward and/or backward based on these
        # unbalanced portions:
        locate_unbalanced_start(unbalanced_start, pre_delete, post_delete)
        locate_unbalanced_end(unbalanced_end, pre_delete, post_delete)
        doc = pre_delete
        if doc and not doc[-1].endswith(' '):
            # Fix up case where the word before us didn't have a trailing space
            doc[-1] += ' '
        doc.append('<del>')
        if balanced and balanced[-1].endswith(' '):
            # We move space outside of </del>
            balanced[-1] = balanced[-1][:-1]
        doc.extend(balanced)
        doc.append('</del> ')
        doc.extend(post_delete)
        chunks = doc
    return chunks

Response: Cleans up any pending DELETE markers in the document and returns a list of the deleted tags.


Instruction:
Can you implement a function in Python 3 that
splits the unbalanced tags into two lists.

Response:
def split_unbalanced(chunks):
    """Return (unbalanced_start, balanced, unbalanced_end), where each is
    a list of text and tag chunks.

    unbalanced_start is a list of all the tags that are opened, but
    not closed in this span.  Similarly, unbalanced_end is a list of
    tags that are closed but were not opened.  Extracting these might
    mean some reordering of the chunks."""
    start = []
    end = []
    tag_stack = []
    balanced = []
    for chunk in chunks:
        if not chunk.startswith('<'):
            balanced.append(chunk)
            continue
        endtag = chunk[1] == '/'
        name = chunk.split()[0].strip('<>/')
        if name in empty_tags:
            balanced.append(chunk)
            continue
        if endtag:
            if tag_stack and tag_stack[-1][0] == name:
                balanced.append(chunk)
                name, pos, tag = tag_stack.pop()
                balanced[pos] = tag
            elif tag_stack:
                start.extend([tag for name, pos, tag in tag_stack])
                tag_stack = []
                end.append(chunk)
            else:
                end.append(chunk)
        else:
            tag_stack.append((name, len(balanced), chunk))
            balanced.append(None)
    start.extend(
        [chunk for name, pos, chunk in tag_stack])
    balanced = [chunk for chunk in balanced if chunk is not None]
    return start, balanced, end


Instruction:
How would you implement a function in Python 3 that
splits a list of delete strings into two lists.

Response:
def split_delete(chunks):
    """ Returns (stuff_before_DEL_START, stuff_inside_DEL_START_END,
    stuff_after_DEL_END).  Returns the first case found (there may be
    more DEL_STARTs in stuff_after_DEL_END).  Raises NoDeletes if
    there's no DEL_START found. """
    try:
        pos = chunks.index(DEL_START)
    except ValueError:
        raise NoDeletes
    pos2 = chunks.index(DEL_END)
    return chunks[:pos], chunks[pos+1:pos2], chunks[pos2+1:]


Instruction:
Implement a Python 3 function for
liking locate_unbalanced_start except handling end tags and handling delete tags and insert them into the document.

Response:
def locate_unbalanced_end(unbalanced_end, pre_delete, post_delete):
    """ like locate_unbalanced_start, except handling end tags and
    possibly moving the point earlier in the document.  """
    while 1:
        if not unbalanced_end:
            # Success
            break
        finding = unbalanced_end[-1]
        finding_name = finding.split()[0].strip('<>/')
        if not pre_delete:
            break
        next = pre_delete[-1]
        if next is DEL_END or not next.startswith('</'):
            # A word or a start tag
            break
        name = next.split()[0].strip('<>/')
        if name == 'ins' or name == 'del':
            # Can't move into an insert or delete
            break
        if name == finding_name:
            unbalanced_end.pop()
            post_delete.insert(0, pre_delete.pop())
        else:
            # Found a tag that doesn't match
            break


Instruction:
Write a Python 3 function for
parsing the given HTML and returns a list of token objects.

Response:
def tokenize(html, include_hrefs=True):
    """
    Parse the given HTML and returns token objects (words with attached tags).

    This parses only the content of a page; anything in the head is
    ignored, and the <head> and <body> elements are themselves
    optional.  The content is then parsed by lxml, which ensures the
    validity of the resulting parsed document (though lxml may make
    incorrect guesses when the markup is particular bad).

    <ins> and <del> tags are also eliminated from the document, as
    that gets confusing.

    If include_hrefs is true, then the href attribute of <a> tags is
    included as a special kind of diffable token."""
    if etree.iselement(html):
        body_el = html
    else:
        body_el = parse_html(html, cleanup=True)
    # Then we split the document into text chunks for each tag, word, and end tag:
    chunks = flatten_el(body_el, skip_tag=True, include_hrefs=include_hrefs)
    # Finally re-joining them into token objects:
    return fixup_chunks(chunks)


Instruction:
Can you create a Python 3 function that
parses an HTML fragment returning an lxml element.

Response:
def parse_html(html, cleanup=True):
    """
    Parses an HTML fragment, returning an lxml element.  Note that the HTML will be
    wrapped in a <div> tag that was not in the original document.

    If cleanup is true, make sure there's no <head> or <body>, and get
    rid of any <ins> and <del> tags.
    """
    if cleanup:
        # This removes any extra markup or structure like <head>:
        html = cleanup_html(html)
    return fragment_fromstring(html, create_parent=True)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def split_trailing_whitespace(word):
    stripped_length = len(word.rstrip())
    return word[0:stripped_length], word[stripped_length:]

Response: This function splits a word into a list of trailing whitespace and the rest of the word.


Instruction:
How would you implement a function in Python 3 that
takes an lxml element el and generates all the text chunks for that tag is not in empty_tags.

Response:
def flatten_el(el, include_hrefs, skip_tag=False):
    """ Takes an lxml element el, and generates all the text chunks for
    that tag.  Each start tag is a chunk, each word is a chunk, and each
    end tag is a chunk.

    If skip_tag is true, then the outermost container tag is
    not returned (just its contents)."""
    if not skip_tag:
        if el.tag == 'img':
            yield ('img', el.get('src'), start_tag(el))
        else:
            yield start_tag(el)
    if el.tag in empty_tags and not el.text and not len(el) and not el.tail:
        return
    start_words = split_words(el.text)
    for word in start_words:
        yield html_escape(word)
    for child in el:
        for item in flatten_el(child, include_hrefs=include_hrefs):
            yield item
    if el.tag == 'a' and el.get('href') and include_hrefs:
        yield ('href', el.get('href'))
    if not skip_tag:
        yield end_tag(el)
        end_words = split_words(el.tail)
        for word in end_words:
            yield html_escape(word)


Instruction:
How would you explain what the following Python 3 function does
def split_words(text):
    if not text or not text.strip():
        return []

    words = split_words_re.findall(text)
    return words

Response:
Splits some text into words. Includes trailing whitespace
    on each word when appropriate.


Instruction:
Create a Python 3 function to
return the text representation of the start tag for a tag.

Response:
def start_tag(el):
    """
    The text representation of the start tag for a tag.
    """
    return '<%s%s>' % (
        el.tag, ''.join([' %s="%s"' % (name, html_escape(value, True))
                         for name, value in el.attrib.items()]))


Instruction:
Can you generate the documentation for the following Python 3 function
def end_tag(el):
    if el.tail and start_whitespace_re.search(el.tail):
        extra = ' '
    else:
        extra = ''
    return '</%s>%s' % (el.tag, extra)

Response: Returns the text representation of an end tag for a tag.


Instruction:
Can you create a Python 3 function that
gives an html string move any ins or del tags inside of any block - level elements e. g. transform <ins > word to <p > word to <p > word to <p > word to <p > word to <p > word to <p > word to <p > word to <p > word to word to word to word to

Response:
def fixup_ins_del_tags(html):
    """ Given an html string, move any  or  tags inside of any
    block-level elements, e.g. transform
word
 to

word
 """
    doc = parse_html(html, cleanup=False)
    _fixup_ins_del_tags(doc)
    html = serialize_html_fragment(doc, skip_outer=True)
    return html


Instruction:
Write a Python 3 script for
serializing a single lxml element as HTML.

Response:
def serialize_html_fragment(el, skip_outer=False):
    """ Serialize a single lxml element as HTML.  The serialized form
    includes the elements tail.

    If skip_outer is true, then don't serialize the outermost tag
    """
    assert not isinstance(el, basestring), (
        "You should pass in an element, not a string like %r" % el)
    html = etree.tostring(el, method="html", encoding=_unicode)
    if skip_outer:
        # Get rid of the extra starting tag:
        html = html[html.find('>')+1:]
        # Get rid of the extra end tag:
        html = html[:html.rfind('<')]
        return html.strip()
    else:
        return html


Instruction:
How would you explain what the following Python 3 function does
def _fixup_ins_del_tags(doc):
    for tag in ['ins', 'del']:
        for el in doc.xpath('descendant-or-self::%s' % tag):
            if not _contains_block_level_tag(el):
                continue
            _move_el_inside_block(el, tag=tag)
            el.drop_tag()

Response: fixup_ins_del_tags that works on an lxml document in - place


Instruction:
Can you generate the documentation for the following Python 3 function
def _contains_block_level_tag(el):
    if el.tag in block_level_tags or el.tag in block_level_container_tags:
        return True
    for child in el:
        if _contains_block_level_tag(child):
            return True
    return False

Response: True if the element contains any block - level elements like p td etc.


Instruction:
Here you have a function in Python 3, explain what it does
def _move_el_inside_block(el, tag):
    for child in el:
        if _contains_block_level_tag(child):
            break
    else:
        import sys
        # No block-level tags in any child
        children_tag = etree.Element(tag)
        children_tag.text = el.text
        el.text = None
        children_tag.extend(list(el))
        el[:] = [children_tag]
        return
    for child in list(el):
        if _contains_block_level_tag(child):
            _move_el_inside_block(child, tag)
            if child.tail:
                tail_tag = etree.Element(tag)
                tail_tag.text = child.tail
                child.tail = None
                el.insert(el.index(child)+1, tail_tag)
        else:
            child_tag = etree.Element(tag)
            el.replace(child, child_tag)
            child_tag.append(child)
    if el.text:
        text_tag = etree.Element(tag)
        text_tag.text = el.text
        el.text = None
        el.insert(0, text_tag)

Response:
helper for _fixup_ins_del_tags ; actually takes the ins etc tags
    and moves them inside any block - level tags.


Instruction:
Can you write a function in Python 3 where it
removes an element and merges its contents into its place.

Response:
def _merge_element_contents(el):
    """
    Removes an element, but merges its contents into its place, e.g.,
    given
Hi there!
, if you remove the  element you get

Hi there!

    """
    parent = el.getparent()
    text = el.text or ''
    if el.tail:
        if not len(el):
            text += el.tail
        else:
            if el[-1].tail:
                el[-1].tail += el.tail
            else:
                el[-1].tail = el.tail
    index = parent.index(el)
    if text:
        if index == 0:
            previous = None
        else:
            previous = parent[index-1]
        if previous is None:
            if parent.text:
                parent.text += text
            else:
                parent.text = text
        else:
            if previous.tail:
                previous.tail += text
            else:
                previous.tail = text
    parent[index:index+1] = el.getchildren()


Instruction:
Implement a Python 3 function for
iterating over the code object.

Response:
def _iter_code(code):

    """Yield '(op,arg)' pair for each operation in code object 'code'"""

    from array import array
    from dis import HAVE_ARGUMENT, EXTENDED_ARG

    bytes = array('b',code.co_code)
    eof = len(code.co_code)

    ptr = 0
    extended_arg = 0

    while ptr<eof:

        op = bytes[ptr]

        if op>=HAVE_ARGUMENT:

            arg = bytes[ptr+1] + bytes[ptr+2]*256 + extended_arg
            ptr += 3

            if op==EXTENDED_ARG:
                extended_arg = arg * compat.long_type(65536)
                continue

        else:
            arg = None
            ptr += 1

        yield op,arg


Instruction:
Given the following Python 3 function, write the documentation
def extract_constant(code, symbol, default=-1):

    if symbol not in code.co_names:
        # name's not there, can't possibly be an assigment
        return None

    name_idx = list(code.co_names).index(symbol)

    STORE_NAME = 90
    STORE_GLOBAL = 97
    LOAD_CONST = 100

    const = default

    for op, arg in _iter_code(code):

        if op==LOAD_CONST:
            const = code.co_consts[arg]
        elif arg==name_idx and (op==STORE_NAME or op==STORE_GLOBAL):
            return const
        else:
            const = default

Response: Extract the constant value of symbol from code.


Instruction:
Given the following Python 3 function, write the documentation
def autolink(el, link_regexes=_link_regexes,
             avoid_elements=_avoid_elements,
             avoid_hosts=_avoid_hosts,
             avoid_classes=_avoid_classes):
    """
    Turn any URLs into links.

    It will search for links identified by the given regular
    expressions (by default mailto and http(s) links).

    It won't link text in an element in avoid_elements, or an element
    with a class in avoid_classes.  It won't link to anything with a
    host that matches one of the regular expressions in avoid_hosts
    (default localhost and 127.0.0.1).

    If you pass in an element, the element's tail will not be
    substituted, only the contents of the element.
    """
    if el.tag in avoid_elements:
        return
    class_name = el.get('class')
    if class_name:
        class_name = class_name.split()
        for match_class in avoid_classes:
            if match_class in class_name:
                return
    for child in list(el):
        autolink(child, link_regexes=link_regexes,
                 avoid_elements=avoid_elements,
                 avoid_hosts=avoid_hosts,
                 avoid_classes=avoid_classes)
        if child.tail:
            text, tail_children = _link_text(
                child.tail, link_regexes, avoid_hosts, factory=el.makeelement)
            if tail_children:
                child.tail = text
                index = el.index(child)
                el[index+1:index+1] = tail_children
    if el.text:
        text, pre_children = _link_text(
            el.text, link_regexes, avoid_hosts, factory=el.makeelement)
        if pre_children:
            el.text = text
            el[:0] = pre_children

Response: Autolinks an element into a tree of links.


Instruction:
Write a Python 3 script for
breaking the text of the element el into words.

Response:
def word_break(el, max_width=40,
               avoid_elements=_avoid_word_break_elements,
               avoid_classes=_avoid_word_break_classes,
               break_character=unichr(0x200b)):
    """
    Breaks any long words found in the body of the text (not attributes).

    Doesn't effect any of the tags in avoid_elements, by default
    ``<textarea>`` and ``<pre>``

    Breaks words by inserting , which is a unicode character
    for Zero Width Space character.  This generally takes up no space
    in rendering, but does copy as a space, and in monospace contexts
    usually takes up space.

    See http://www.cs.tut.fi/~jkorpela/html/nobr.html for a discussion
    """
    # Character suggestion of &#8203 comes from:
    #   http://www.cs.tut.fi/~jkorpela/html/nobr.html
    if el.tag in _avoid_word_break_elements:
        return
    class_name = el.get('class')
    if class_name:
        dont_break = False
        class_name = class_name.split()
        for avoid in avoid_classes:
            if avoid in class_name:
                dont_break = True
                break
        if dont_break:
            return
    if el.text:
        el.text = _break_text(el.text, max_width, break_character)
    for child in el:
        word_break(child, max_width=max_width,
                   avoid_elements=avoid_elements,
                   avoid_classes=avoid_classes,
                   break_character=break_character)
        if child.tail:
            child.tail = _break_text(child.tail, max_width, break_character)


Instruction:
Write a Python 3 function for
killing any conditional comments in the given document.

Response:
def kill_conditional_comments(self, doc):
        """
        IE conditional comments basically embed HTML that the parser
        doesn't normally see.  We can't allow anything like that, so
        we'll kill any comments that could be conditional.
        """
        bad = []
        self._kill_elements(
            doc, lambda el: _conditional_comment_re.search(el.text),
            etree.Comment)


Instruction:
Create a Python 3 function for
checking if the style contains a sneaky javascript.

Response:
def _has_sneaky_javascript(self, style):
        """
        Depending on the browser, stuff like ``e x p r e s s i o n(...)``
        can get interpreted, or ``expre/* stuff */ssion(...)``.  This
        checks for attempt to do stuff like this.

        Typically the response will be to kill the entire style; if you
        have just a bit of Javascript in the style another rule will catch
        that and remove only the Javascript from the style; this catches
        more sneaky attempts.
        """
        style = self._substitute_comments('', style)
        style = style.replace('\\', '')
        style = _substitute_whitespace('', style)
        style = style.lower()
        if 'javascript:' in style:
            return True
        if 'expression(' in style:
            return True
        return False


Instruction:
How would you code a function in Python 3 to
parse a whole document into a string.

Response:
def document_fromstring(html, guess_charset=True, parser=None):
    """Parse a whole document into a string."""
    if not isinstance(html, _strings):
        raise TypeError('string required')

    if parser is None:
        parser = html_parser

    return parser.parse(html, useChardet=guess_charset).getroot()


Instruction:
How would you implement a function in Python 3 that
parses several HTML elements returning a list of elements.

Response:
def fragments_fromstring(html, no_leading_text=False,
                         guess_charset=False, parser=None):
    """Parses several HTML elements, returning a list of elements.

    The first item in the list may be a string.  If no_leading_text is true,
    then it will be an error if there is leading text, and it will always be
    a list of only elements.

    If `guess_charset` is `True` and the text was not unicode but a
    bytestring, the `chardet` library will perform charset guessing on the
    string.
    """
    if not isinstance(html, _strings):
        raise TypeError('string required')

    if parser is None:
        parser = html_parser

    children = parser.parseFragment(html, 'div', useChardet=guess_charset)
    if children and isinstance(children[0], _strings):
        if no_leading_text:
            if children[0].strip():
                raise etree.ParserError('There is leading text: %r' %
                                        children[0])
            del children[0]
    return children


Instruction:
Create a Python 3 function for
parsing a single HTML element and returns a node containing the fragment.

Response:
def fragment_fromstring(html, create_parent=False,
                        guess_charset=False, parser=None):
    """Parses a single HTML element; it is an error if there is more than
    one element, or if anything but whitespace precedes or follows the
    element.

    If create_parent is true (or is a tag name) then a parent node
    will be created to encapsulate the HTML in a single element.  In
    this case, leading or trailing text is allowed.
    """
    if not isinstance(html, _strings):
        raise TypeError('string required')

    accept_leading_text = bool(create_parent)

    elements = fragments_fromstring(
        html, guess_charset=guess_charset, parser=parser,
        no_leading_text=not accept_leading_text)

    if create_parent:
        if not isinstance(create_parent, _strings):
            create_parent = 'div'
        new_root = Element(create_parent)
        if elements:
            if isinstance(elements[0], _strings):
                new_root.text = elements[0]
                del elements[0]
            new_root.extend(elements)
        return new_root

    if not elements:
        raise etree.ParserError('No elements found')
    if len(elements) > 1:
        raise etree.ParserError('Multiple elements found')
    result = elements[0]
    if result.tail and result.tail.strip():
        raise etree.ParserError('Element followed by text: %r' % result.tail)
    result.tail = None
    return result


Instruction:
Can you write a function in Python 3 where it
parses the html string and return a single element or document.

Response:
def fromstring(html, guess_charset=True, parser=None):
    """Parse the html, returning a single element/document.

    This tries to minimally parse the chunk of text, without knowing if it
    is a fragment or a document.

    base_url will set the document's base_url attribute (and the tree's docinfo.URL)
    """
    if not isinstance(html, _strings):
        raise TypeError('string required')
    doc = document_fromstring(html, parser=parser,
                              guess_charset=guess_charset)

    # document starts with doctype or <html>, full document!
    start = html[:50].lstrip().lower()
    if start.startswith('<html') or start.startswith('<!doctype'):
        return doc

    head = _find_tag(doc, 'head')

    # if the head is not empty we have a full document
    if len(head):
        return doc

    body = _find_tag(doc, 'body')

    # The body has just one element, so it was probably a single
    # element passed in
    if (len(body) == 1 and (not body.text or not body.text.strip())
        and (not body[-1].tail or not body[-1].tail.strip())):
        return body[0]

    # Now we have a body which represents a bunch of tags which have the
    # content that was passed in.  We will create a fake container, which
    # is the body tag, except <body> implies too much structure.
    if _contains_block_level_tag(body):
        body.tag = 'div'
    else:
        body.tag = 'span'
    return body


Instruction:
Can you generate a brief explanation for the following Python 3 code
def parse(filename_url_or_file, guess_charset=True, parser=None):
    if parser is None:
        parser = html_parser
    if not isinstance(filename_url_or_file, _strings):
        fp = filename_url_or_file
    elif _looks_like_url(filename_url_or_file):
        fp = urlopen(filename_url_or_file)
    else:
        fp = open(filename_url_or_file, 'rb')
    return parser.parse(fp, useChardet=guess_charset)

Response:
Parse a filename URL or file - like object into an HTML document
   .


Instruction:
Here you have a function in Python 3, explain what it does
def api_accepts(fields):
    def decorator(func):
        @wraps(func)
        def wrapped_func(request, *args, **kwargs):
            if request.method not in ['GET', 'POST']:
                return func(request, *args, **kwargs)

            # The fields dict passed into the type() function is modified, so
            # send in a copy instead.
            form_class = type('ApiForm', (forms.Form,), fields.copy())
            form = form_class(getattr(request, request.method))

            if not form.is_valid():
                if settings.DEBUG:
                    return JsonResponseBadRequest(
                        'failed to validate: %s' % dict(form.errors)
                    )
                else:
                    logger.warn(
                        'input to \'%s\' failed to validate: %s',
                        request.path,
                        dict(form.errors)
                    )
                    return func(request, *args, **kwargs)

            # Clean any models.Model fields, by looking up object based on
            # primary key in request.
            for (field_name, field_instance) in fields.items():
                if isinstance(field_instance, models.Model):
                    field_type = type(field_instance)
                    # TODO: irregular, should we remove?
                    field_id = '%s-id' % field_name
                    if field_id not in request.REQUEST:
                        return JsonResponseBadRequest(
                            'field %s not present' % field_name
                        )
                    field_pk = int(request.REQUEST[field_id])
                    try:
                        field_value = field_type.objects.get(pk=field_pk)
                    except field_type.DoesNotExist:
                        return JsonResponseNotFound(
                            '%s with pk=%d does not exist' % (
                                field_type, field_pk
                            )
                        )
                    form.cleaned_data[field_name] = field_value

            validated_request = ValidatedRequest(request, form)
            return func(validated_request, *args, **kwargs)
        return wrapped_func
    return decorator

Response: Decorator that returns a view function that accepts the given fields.


Instruction:
Can you generate the documentation for the following Python 3 function
def api_returns(return_values):
    def decorator(func):
        @wraps(func)
        def wrapped_func(request, *args, **kwargs):
            return_value = func(request, *args, **kwargs)

            if not isinstance(return_value, JsonResponse):
                if settings.DEBUG:
                    return JsonResponseBadRequest('API did not return JSON')
                else:
                    logger.warn('API did not return JSON')

            accepted_return_codes = return_values.keys()
            # Never block 500s - these should be handled by other
            # reporting mechanisms
            accepted_return_codes.append(500)

            if return_value.status_code not in accepted_return_codes:
                if settings.DEBUG:
                    return JsonResponseBadRequest(
                        'API returned %d instead of acceptable values %s' %
                        (return_value.status_code, accepted_return_codes)
                    )
                else:
                    logger.warn(
                        'API returned %d instead of acceptable values %s',
                        return_value.status_code,
                        accepted_return_codes,
                    )

            return return_value
        return wrapped_func
    return decorator

Response: Decorator for returning a single API resource.


Instruction:
Can you generate the documentation for the following Python 3 function
def api(accept_return_dict):
    def decorator(func):
        @wraps(func)
        def wrapped_func(request, *args, **kwargs):
            @api_accepts(accept_return_dict['accepts'])
            @api_returns(accept_return_dict['returns'])
            def apid_fnc(request, *args, **kwargs):
                return func(request, *args, **kwargs)

            return apid_fnc(request, *args, **kwargs)
        return wrapped_func
    return decorator

Response: Decorator that returns a function that returns a response in sequence.


Instruction:
Write a Python 3 script to
return a decorator that ensures that the request passed to the view function/method has a valid JSON request body with the given required fields. The dict parsed from the JSON is then passed as the second argument to the decorated function/method. For example: @json_request({'name', 'date'}) def view_func(request, request_dict): ...

Response:
def validate_json_request(required_fields):
    """
    Return a decorator that ensures that the request passed to the view
    function/method has a valid JSON request body with the given required
    fields.  The dict parsed from the JSON is then passed as the second
    argument to the decorated function/method.  For example:

    @json_request({'name', 'date'})
    def view_func(request, request_dict):
        ...
    """
    def decorator(func):
        @wraps(func)
        def wrapped_func(request, *args, **kwargs):
            try:
                request_dict = json.loads(request.raw_post_data)
            except ValueError as e:
                return JsonResponseBadRequest('invalid POST JSON: %s' % e)

            for k in required_fields:
                if k not in request_dict:
                    return JsonResponseBadRequest(
                        'POST JSON must contain property \'%s\'' % k)

            return func(request, request_dict, *args, **kwargs)
        return wrapped_func
    return decorator


Instruction:
Here you have a function in Python 3, explain what it does
def getTreeWalker(treeType, implementation=None, **kwargs):

    treeType = treeType.lower()
    if treeType not in treeWalkerCache:
        if treeType in ("dom", "pulldom"):
            name = "%s.%s" % (__name__, treeType)
            __import__(name)
            mod = sys.modules[name]
            treeWalkerCache[treeType] = mod.TreeWalker
        elif treeType == "genshi":
            from . import genshistream
            treeWalkerCache[treeType] = genshistream.TreeWalker
        elif treeType == "lxml":
            from . import lxmletree
            treeWalkerCache[treeType] = lxmletree.TreeWalker
        elif treeType == "etree":
            from . import etree
            if implementation is None:
                implementation = default_etree
            # XXX: NEVER cache here, caching is done in the etree submodule
            return etree.getETreeModule(implementation, **kwargs).TreeWalker
    return treeWalkerCache.get(treeType)

Response: Get a TreeWalker class for various types of tree with built - in support.


Instruction:
How would you explain what the following Python 3 function does
def get_include():
    import os
    lxml_path = __path__[0]
    include_path = os.path.join(lxml_path, 'includes')
    includes = [include_path, lxml_path]

    for name in os.listdir(include_path):
        path = os.path.join(include_path, name)
        if os.path.isdir(path):
            includes.append(path)

    return includes

Response: Returns a list of include paths needed to compile C code against lxml.


Instruction:
Create a Python 3 function for
exporting the svn repository at the url to the destination location

Response:
def export(self, location):
        """Export the svn repository at the url to the destination location"""
        url, rev = self.get_url_rev()
        rev_options = get_rev_options(url, rev)
        logger.info('Exporting svn repository %s to %s', url, location)
        with indent_log():
            if os.path.exists(location):
                # Subversion doesn't like to check out over an existing
                # directory --force fixes this, but was only added in svn 1.5
                rmtree(location)
            self.run_command(
                ['export'] + rev_options + [url, location],
                show_stdout=False)


Instruction:
Can you tell what is the following Python 3 function doing
def get_revision(self, location):
        # Note: taken from setuptools.command.egg_info
        revision = 0

        for base, dirs, files in os.walk(location):
            if self.dirname not in dirs:
                dirs[:] = []
                continue    # no sense walking uncontrolled subdirs
            dirs.remove(self.dirname)
            entries_fn = os.path.join(base, self.dirname, 'entries')
            if not os.path.exists(entries_fn):
                # FIXME: should we warn?
                continue

            dirurl, localrev = self._get_svn_url_rev(base)

            if base == location:
                base_url = dirurl + '/'   # save the root url
            elif not dirurl or not dirurl.startswith(base_url):
                dirs[:] = []
                continue    # not part of the same svn tree, skip it
            revision = max(revision, localrev)
        return revision

Response: Return the maximum revision for all files under a given location.


Instruction:
Here you have a function in Python 3, explain what it does
def name(self):
        if self.import_name == '__main__':
            fn = getattr(sys.modules['__main__'], '__file__', None)
            if fn is None:
                return '__main__'
            return os.path.splitext(os.path.basename(fn))[0]
        return self.import_name

Response: The name of the application.


Instruction:
Can you tell what is the following Python 3 function doing
def propagate_exceptions(self):
        rv = self.config['PROPAGATE_EXCEPTIONS']
        if rv is not None:
            return rv
        return self.testing or self.debug

Response: Returns the value of the PROPAGATE_EXCEPTIONS configuration value in case it s set otherwise a sensible default is returned.


Instruction:
Can you tell what is the following Python 3 function doing
def logger(self):
        if self._logger and self._logger.name == self.logger_name:
            return self._logger
        with _logger_lock:
            if self._logger and self._logger.name == self.logger_name:
                return self._logger
            from flask.logging import create_logger
            self._logger = rv = create_logger(self)
            return rv

Response: A logger object for this application.


Instruction:
Make a summary of the following Python 3 code
def make_config(self, instance_relative=False):
        root_path = self.root_path
        if instance_relative:
            root_path = self.instance_path
        return Config(root_path, self.default_config)

Response: Used to create the config attribute by the Flask constructor.


Instruction:
How would you code a function in Python 3 to
try to find the instance path if it was not provided.

Response:
def auto_find_instance_path(self):
        """Tries to locate the instance path if it was not provided to the
        constructor of the application class.  It will basically calculate
        the path to a folder named ``instance`` next to your main file or
        the package.

        .. versionadded:: 0.8
        """
        prefix, package_path = find_package(self.import_name)
        if prefix is None:
            return os.path.join(package_path, 'instance')
        return os.path.join(prefix, 'var', self.name + '-instance')


Instruction:
Can you tell what is the following Python 3 function doing
def open_instance_resource(self, resource, mode='rb'):
        return open(os.path.join(self.instance_path, resource), mode)

Response: Opens a resource from the application s instance folder.


Instruction:
Implement a function in Python 3 to
create the Jinja2 environment based on the current configuration and the current context processor.

Response:
def create_jinja_environment(self):
        """Creates the Jinja2 environment based on :attr:`jinja_options`
        and :meth:`select_jinja_autoescape`.  Since 0.7 this also adds
        the Jinja2 globals and filters after initialization.  Override
        this function to customize the behavior.

        .. versionadded:: 0.5
        """
        options = dict(self.jinja_options)
        if 'autoescape' not in options:
            options['autoescape'] = self.select_jinja_autoescape
        rv = Environment(self, **options)
        rv.globals.update(
            url_for=url_for,
            get_flashed_messages=get_flashed_messages,
            config=self.config,
            # request, session and g are normally added with the
            # context processor for efficiency reasons but for imported
            # templates we also want the proxies in there.
            request=request,
            session=session,
            g=g
        )
        rv.filters['tojson'] = json.tojson_filter
        return rv


Instruction:
Given the following Python 3 function, write the documentation
def update_template_context(self, context):
        funcs = self.template_context_processors[None]
        reqctx = _request_ctx_stack.top
        if reqctx is not None:
            bp = reqctx.request.blueprint
            if bp is not None and bp in self.template_context_processors:
                funcs = chain(funcs, self.template_context_processors[bp])
        orig_ctx = context.copy()
        for func in funcs:
            context.update(func())
        # make sure the original values win.  This makes it possible to
        # easier add new variables in context processors without breaking
        # existing views.
        context.update(orig_ctx)

Response: Update the template context with some commonly used variables.


Instruction:
Can you tell what is the following Python 3 function doing
def run(self, host=None, port=None, debug=None, **options):
        from werkzeug.serving import run_simple
        if host is None:
            host = '127.0.0.1'
        if port is None:
            server_name = self.config['SERVER_NAME']
            if server_name and ':' in server_name:
                port = int(server_name.rsplit(':', 1)[1])
            else:
                port = 5000
        if debug is not None:
            self.debug = bool(debug)
        options.setdefault('use_reloader', self.debug)
        options.setdefault('use_debugger', self.debug)
        try:
            run_simple(host, port, self, **options)
        finally:
            # reset the first request information if the development server
            # resetted normally.  This makes it possible to restart the server
            # without reloader and that stuff from an interactive shell.
            self._got_first_request = False

Response: Runs the application on a local development server.


Instruction:
Here you have a function in Python 3, explain what it does
def save_session(self, session, response):
        return self.session_interface.save_session(self, session, response)

Response: Saves the session if it needs updates.


Instruction:
Create a Python 3 function for
registering a module with this application.

Response:
def register_module(self, module, **options):
        """Registers a module with this application.  The keyword argument
        of this function are the same as the ones for the constructor of the
        :class:`Module` class and will override the values of the module if
        provided.

        .. versionchanged:: 0.7
           The module system was deprecated in favor for the blueprint
           system.
        """
        assert blueprint_is_module(module), 'register_module requires ' \
            'actual module objects.  Please upgrade to blueprints though.'
        if not self.enable_modules:
            raise RuntimeError('Module support was disabled but code '
                'attempted to register a module named %r' % module)
        else:
            from warnings import warn
            warn(DeprecationWarning('Modules are deprecated.  Upgrade to '
                'using blueprints.  Have a look into the documentation for '
                'more information.  If this module was registered by a '
                'Flask-Extension upgrade the extension or contact the author '
                'of that extension instead.  (Registered %r)' % module),
                stacklevel=2)

        self.register_blueprint(module, **options)


Instruction:
Can you generate the documentation for the following Python 3 function
def add_url_rule(self, rule, endpoint=None, view_func=None, **options):
        if endpoint is None:
            endpoint = _endpoint_from_view_func(view_func)
        options['endpoint'] = endpoint
        methods = options.pop('methods', None)

        # if the methods are not given and the view_func object knows its
        # methods we can use that instead.  If neither exists, we go with
        # a tuple of only `GET` as default.
        if methods is None:
            methods = getattr(view_func, 'methods', None) or ('GET',)
        methods = set(methods)

        # Methods that should always be added
        required_methods = set(getattr(view_func, 'required_methods', ()))

        # starting with Flask 0.8 the view_func object can disable and
        # force-enable the automatic options handling.
        provide_automatic_options = getattr(view_func,
            'provide_automatic_options', None)

        if provide_automatic_options is None:
            if 'OPTIONS' not in methods:
                provide_automatic_options = True
                required_methods.add('OPTIONS')
            else:
                provide_automatic_options = False

        # Add the required methods now.
        methods |= required_methods

        # due to a werkzeug bug we need to make sure that the defaults are
        # None if they are an empty dictionary.  This should not be necessary
        # with Werkzeug 0.7
        options['defaults'] = options.get('defaults') or None

        rule = self.url_rule_class(rule, methods=methods, **options)
        rule.provide_automatic_options = provide_automatic_options

        self.url_map.add(rule)
        if view_func is not None:
            old_func = self.view_functions.get(endpoint)
            if old_func is not None and old_func != view_func:
                raise AssertionError('View function mapping is overwriting an '
                                     'existing endpoint function: %s' % endpoint)
            self.view_functions[endpoint] = view_func

Response: Connects a URL rule to the application.


Instruction:
Can you generate the documentation for the following Python 3 function
def endpoint(self, endpoint):
        def decorator(f):
            self.view_functions[endpoint] = f
            return f
        return decorator

Response: A decorator to register a function as an endpoint.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def template_filter(self, name=None):
        def decorator(f):
            self.add_template_filter(f, name=name)
            return f
        return decorator

Response: A decorator that is used to register custom template filter.


Instruction:
Can you write a function in Python 3 where it
registers a custom template filter. Works exactly like the decorator.

Response:
def add_template_filter(self, f, name=None):
        """Register a custom template filter.  Works exactly like the
        :meth:`template_filter` decorator.

        :param name: the optional name of the filter, otherwise the
                     function name will be used.
        """
        self.jinja_env.filters[name or f.__name__] = f


Instruction:
How would you code a function in Python 3 to
register a custom template global function. Works exactly like the decorator.

Response:
def add_template_global(self, f, name=None):
        """Register a custom template global function. Works exactly like the
        :meth:`template_global` decorator.

        .. versionadded:: 0.10

        :param name: the optional name of the global function, otherwise the
                     function name will be used.
        """
        self.jinja_env.globals[name or f.__name__] = f


Instruction:
Write a Python 3 script for
handling an HTTP exception.

Response:
def handle_http_exception(self, e):
        """Handles an HTTP exception.  By default this will invoke the
        registered error handlers and fall back to returning the
        exception as response.

        .. versionadded:: 0.3
        """
        handlers = self.error_handler_spec.get(request.blueprint)
        # Proxy exceptions don't have error codes.  We want to always return
        # those unchanged as errors
        if e.code is None:
            return e
        if handlers and e.code in handlers:
            handler = handlers[e.code]
        else:
            handler = self.error_handler_spec[None].get(e.code)
        if handler is None:
            return e
        return handler(e)


Instruction:
Write a Python 3 function for
checking if an HTTP exception should be trapped or not.

Response:
def trap_http_exception(self, e):
        """Checks if an HTTP exception should be trapped or not.  By default
        this will return `False` for all exceptions except for a bad request
        key error if ``TRAP_BAD_REQUEST_ERRORS`` is set to `True`.  It
        also returns `True` if ``TRAP_HTTP_EXCEPTIONS`` is set to `True`.

        This is called for all HTTP exceptions raised by a view function.
        If it returns `True` for any exception the error handler for this
        exception is not called and it shows up as regular exception in the
        traceback.  This is helpful for debugging implicitly raised HTTP
        exceptions.

        .. versionadded:: 0.8
        """
        if self.config['TRAP_HTTP_EXCEPTIONS']:
            return True
        if self.config['TRAP_BAD_REQUEST_ERRORS']:
            return isinstance(e, BadRequest)
        return False


Instruction:
Can you generate the documentation for the following Python 3 function
def handle_user_exception(self, e):
        exc_type, exc_value, tb = sys.exc_info()
        assert exc_value is e

        # ensure not to trash sys.exc_info() at that point in case someone
        # wants the traceback preserved in handle_http_exception.  Of course
        # we cannot prevent users from trashing it themselves in a custom
        # trap_http_exception method so that's their fault then.
        if isinstance(e, HTTPException) and not self.trap_http_exception(e):
            return self.handle_http_exception(e)

        blueprint_handlers = ()
        handlers = self.error_handler_spec.get(request.blueprint)
        if handlers is not None:
            blueprint_handlers = handlers.get(None, ())
        app_handlers = self.error_handler_spec[None].get(None, ())
        for typecheck, handler in chain(blueprint_handlers, app_handlers):
            if isinstance(e, typecheck):
                return handler(e)

        reraise(exc_type, exc_value, tb)

Response: This method handles the exception raised by the user.


Instruction:
Can you tell what is the following Python 3 function doing
def handle_exception(self, e):
        exc_type, exc_value, tb = sys.exc_info()

        got_request_exception.send(self, exception=e)
        handler = self.error_handler_spec[None].get(500)

        if self.propagate_exceptions:
            # if we want to repropagate the exception, we can attempt to
            # raise it with the whole traceback in case we can do that
            # (the function was actually called from the except part)
            # otherwise, we just raise the error again
            if exc_value is e:
                reraise(exc_type, exc_value, tb)
            else:
                raise e

        self.log_exception((exc_type, exc_value, tb))
        if handler is None:
            return InternalServerError()
        return handler(e)

Response: Default exception handling that kicks in when an exception occurs that is not caught.


Instruction:
Given the following Python 3 function, write the documentation
def log_exception(self, exc_info):
        self.logger.error('Exception on %s [%s]' % (
            request.path,
            request.method
        ), exc_info=exc_info)

Response:
Logs an exception.  This is called by :meth:`handle_exception`
        if debugging is disabled and right before the handler is called.
        The default implementation logs the exception as error on the
        :attr:`logger`.

        .. versionadded:: 0.8


Instruction:
Write a Python 3 script to
raise the routing exception that is not handled by the routing system.

Response:
def raise_routing_exception(self, request):
        """Exceptions that are recording during routing are reraised with
        this method.  During debug we are not reraising redirect requests
        for non ``GET``, ``HEAD``, or ``OPTIONS`` requests and we're raising
        a different error instead to help debug situations.

        :internal:
        """
        if not self.debug \
           or not isinstance(request.routing_exception, RequestRedirect) \
           or request.method in ('GET', 'HEAD', 'OPTIONS'):
            raise request.routing_exception

        from .debughelpers import FormDataRoutingRedirect
        raise FormDataRoutingRedirect(request)


Instruction:
Here you have a function in Python 3, explain what it does
def dispatch_request(self):
        req = _request_ctx_stack.top.request
        if req.routing_exception is not None:
            self.raise_routing_exception(req)
        rule = req.url_rule
        # if we provide automatic options for this URL and the
        # request came with the OPTIONS method, reply automatically
        if getattr(rule, 'provide_automatic_options', False) \
           and req.method == 'OPTIONS':
            return self.make_default_options_response()
        # otherwise dispatch to the handler for that endpoint
        return self.view_functions[rule.endpoint](**req.view_args)

Response: Does the request dispatching.


Instruction:
Create a Python 3 function to
dispatch the request and on top of that performs request pre and post processing and returns the response.

Response:
def full_dispatch_request(self):
        """Dispatches the request and on top of that performs request
        pre and postprocessing as well as HTTP exception catching and
        error handling.

        .. versionadded:: 0.7
        """
        self.try_trigger_before_first_request_functions()
        try:
            request_started.send(self)
            rv = self.preprocess_request()
            if rv is None:
                rv = self.dispatch_request()
        except Exception as e:
            rv = self.handle_user_exception(e)
        response = self.make_response(rv)
        response = self.process_response(response)
        request_finished.send(self, response=response)
        return response


Instruction:
Can you implement a function in Python 3 that
calls before each request and will ensure that it triggers the before_first_request_funcs and only exactly once per application instance.

Response:
def try_trigger_before_first_request_functions(self):
        """Called before each request and will ensure that it triggers
        the :attr:`before_first_request_funcs` and only exactly once per
        application instance (which means process usually).

        :internal:
        """
        if self._got_first_request:
            return
        with self._before_request_lock:
            if self._got_first_request:
                return
            self._got_first_request = True
            for func in self.before_first_request_funcs:
                func()


Instruction:
Explain what the following Python 3 code does
def make_default_options_response(self):
        adapter = _request_ctx_stack.top.url_adapter
        if hasattr(adapter, 'allowed_methods'):
            methods = adapter.allowed_methods()
        else:
            # fallback for Werkzeug < 0.7
            methods = []
            try:
                adapter.match(method='--')
            except MethodNotAllowed as e:
                methods = e.valid_methods
            except HTTPException as e:
                pass
        rv = self.response_class()
        rv.allow.update(methods)
        return rv

Response: This method is called to create the default OPTIONS response. It is called through subclassing to change the default OPTIONS response behavior of the HTTP response.


Instruction:
Can you create a Python 3 function that
converts the return value from a view function to a real response object that is an instance of self. response_class.

Response:
def make_response(self, rv):
        """Converts the return value from a view function to a real
        response object that is an instance of :attr:`response_class`.

        The following types are allowed for `rv`:

        .. tabularcolumns:: |p{3.5cm}|p{9.5cm}|

        ======================= ===========================================
        :attr:`response_class`  the object is returned unchanged
        :class:`str`            a response object is created with the
                                string as body
        :class:`unicode`        a response object is created with the
                                string encoded to utf-8 as body
        a WSGI function         the function is called as WSGI application
                                and buffered as response object
        :class:`tuple`          A tuple in the form ``(response, status,
                                headers)`` where `response` is any of the
                                types defined here, `status` is a string
                                or an integer and `headers` is a list of
                                a dictionary with header values.
        ======================= ===========================================

        :param rv: the return value from the view function

        .. versionchanged:: 0.9
           Previously a tuple was interpreted as the arguments for the
           response object.
        """
        status = headers = None
        if isinstance(rv, tuple):
            rv, status, headers = rv + (None,) * (3 - len(rv))

        if rv is None:
            raise ValueError('View function did not return a response')

        if not isinstance(rv, self.response_class):
            # When we create a response object directly, we let the constructor
            # set the headers and status.  We do this because there can be
            # some extra logic involved when creating these objects with
            # specific values (like default content type selection).
            if isinstance(rv, (text_type, bytes, bytearray)):
                rv = self.response_class(rv, headers=headers, status=status)
                headers = status = None
            else:
                rv = self.response_class.force_type(rv, request.environ)

        if status is not None:
            if isinstance(status, string_types):
                rv.status = status
            else:
                rv.status_code = status
        if headers:
            rv.headers.extend(headers)

        return rv


Instruction:
Can you create a Python 3 function that
creates a URL adapter for the given request.

Response:
def create_url_adapter(self, request):
        """Creates a URL adapter for the given request.  The URL adapter
        is created at a point where the request context is not yet set up
        so the request is passed explicitly.

        .. versionadded:: 0.6

        .. versionchanged:: 0.9
           This can now also be called without a request object when the
           URL adapter is created for the application context.
        """
        if request is not None:
            return self.url_map.bind_to_environ(request.environ,
                server_name=self.config['SERVER_NAME'])
        # We need at the very least the server name to be set for this
        # to work.
        if self.config['SERVER_NAME'] is not None:
            return self.url_map.bind(
                self.config['SERVER_NAME'],
                script_name=self.config['APPLICATION_ROOT'] or '/',
                url_scheme=self.config['PREFERRED_URL_SCHEME'])


Instruction:
Can you generate a brief explanation for the following Python 3 code
def inject_url_defaults(self, endpoint, values):
        funcs = self.url_default_functions.get(None, ())
        if '.' in endpoint:
            bp = endpoint.rsplit('.', 1)[0]
            funcs = chain(funcs, self.url_default_functions.get(bp, ()))
        for func in funcs:
            func(endpoint, values)

Response: Injects the URL defaults for the given endpoint into the values dictionary passed.


Instruction:
Write a Python 3 function that can
handle a BuildError exception on the url_for endpoint.

Response:
def handle_url_build_error(self, error, endpoint, values):
        """Handle :class:`~werkzeug.routing.BuildError` on :meth:`url_for`.
        """
        exc_type, exc_value, tb = sys.exc_info()
        for handler in self.url_build_error_handlers:
            try:
                rv = handler(error, endpoint, values)
                if rv is not None:
                    return rv
            except BuildError as error:
                pass

        # At this point we want to reraise the exception.  If the error is
        # still the same one we can reraise it with the original traceback,
        # otherwise we raise it from here.
        if error is exc_value:
            reraise(exc_type, exc_value, tb)
        raise error


Instruction:
Write a Python 3 function that can
call before the actual request dispatching and will call every as before_request decorated function.

Response:
def preprocess_request(self):
        """Called before the actual request dispatching and will
        call every as :meth:`before_request` decorated function.
        If any of these function returns a value it's handled as
        if it was the return value from the view and further
        request handling is stopped.

        This also triggers the :meth:`url_value_processor` functions before
        the actual :meth:`before_request` functions are called.
        """
        bp = _request_ctx_stack.top.request.blueprint

        funcs = self.url_value_preprocessors.get(None, ())
        if bp is not None and bp in self.url_value_preprocessors:
            funcs = chain(funcs, self.url_value_preprocessors[bp])
        for func in funcs:
            func(request.endpoint, request.view_args)

        funcs = self.before_request_funcs.get(None, ())
        if bp is not None and bp in self.before_request_funcs:
            funcs = chain(funcs, self.before_request_funcs[bp])
        for func in funcs:
            rv = func()
            if rv is not None:
                return rv


Instruction:
Explain what the following Python 3 code does
def process_response(self, response):
        ctx = _request_ctx_stack.top
        bp = ctx.request.blueprint
        funcs = ctx._after_request_functions
        if bp is not None and bp in self.after_request_funcs:
            funcs = chain(funcs, reversed(self.after_request_funcs[bp]))
        if None in self.after_request_funcs:
            funcs = chain(funcs, reversed(self.after_request_funcs[None]))
        for handler in funcs:
            response = handler(response)
        if not self.session_interface.is_null_session(ctx.session):
            self.save_session(ctx.session, response)
        return response

Response: This method is used to process the response object.


Instruction:
Make a summary of the following Python 3 code
def do_teardown_request(self, exc=None):
        if exc is None:
            exc = sys.exc_info()[1]
        funcs = reversed(self.teardown_request_funcs.get(None, ()))
        bp = _request_ctx_stack.top.request.blueprint
        if bp is not None and bp in self.teardown_request_funcs:
            funcs = chain(funcs, reversed(self.teardown_request_funcs[bp]))
        for func in funcs:
            rv = func(exc)
        request_tearing_down.send(self, exc=exc)

Response:
Called after the request dispatching and will
        call every as a teardown_request decorated function.


Instruction:
Given the following Python 3 function, write the documentation
def do_teardown_appcontext(self, exc=None):
        if exc is None:
            exc = sys.exc_info()[1]
        for func in reversed(self.teardown_appcontext_funcs):
            func(exc)
        appcontext_tearing_down.send(self, exc=exc)

Response: Called when an application context is popped.


Instruction:
How would you implement a function in Python 3 that
yields unique values in iterable preserving order.

Response:
def unique(iterable):
    """
    Yield unique values in iterable, preserving order.
    """
    seen = set()
    for value in iterable:
        if not value in seen:
            seen.add(value)
            yield value


Instruction:
Write a Python 3 function that can
add the runtime requirements from pkg_info into the metadata.

Response:
def handle_requires(metadata, pkg_info, key):
    """
    Place the runtime requirements from pkg_info into metadata.
    """
    may_requires = defaultdict(list)
    for value in pkg_info.get_all(key):
        extra_match = EXTRA_RE.search(value)
        if extra_match:
            groupdict = extra_match.groupdict()
            condition = groupdict['condition']
            extra = groupdict['extra']
            package = groupdict['package']
            if condition.endswith(' and '):
                condition = condition[:-5]
        else:
            condition, extra = None, None
            package = value
        key = MayRequiresKey(condition, extra)
        may_requires[key].append(package)

    if may_requires:
        metadata['run_requires'] = []
        for key, value in may_requires.items():
            may_requirement = {'requires':value}
            if key.extra:
                may_requirement['extra'] = key.extra
            if key.condition:
                may_requirement['environment'] = key.condition
            metadata['run_requires'].append(may_requirement)

        if not 'extras' in metadata:
            metadata['extras'] = []
        metadata['extras'].extend([key.extra for key in may_requires.keys() if key.extra])


Instruction:
Can you implement a function in Python 3 that
converts PKG - INFO file to a prototype Metadata 2. 0 dict.

Response:
def pkginfo_to_dict(path, distribution=None):
    """
    Convert PKG-INFO to a prototype Metadata 2.0 (PEP 426) dict.

    The description is included under the key ['description'] rather than
    being written to a separate file.

    path: path to PKG-INFO file
    distribution: optional distutils Distribution()
    """

    metadata = defaultdict(lambda: defaultdict(lambda: defaultdict(dict)))
    metadata["generator"] = "bdist_wheel (" + wheel.__version__ + ")"
    try:
        unicode
        pkg_info = read_pkg_info(path)
    except NameError:
        pkg_info = email.parser.Parser().parsestr(open(path, 'rb').read().decode('utf-8'))
    description = None

    if pkg_info['Summary']:
        metadata['summary'] = pkginfo_unicode(pkg_info, 'Summary')
        del pkg_info['Summary']

    if pkg_info['Description']:
        description = dedent_description(pkg_info)
        del pkg_info['Description']
    else:
        payload = pkg_info.get_payload()
        if isinstance(payload, bytes):
            # Avoid a Python 2 Unicode error.
            # We still suffer ? glyphs on Python 3.
            payload = payload.decode('utf-8')
        if payload:
            description = payload

    if description:
        pkg_info['description'] = description

    for key in unique(k.lower() for k in pkg_info.keys()):
        low_key = key.replace('-', '_')

        if low_key in SKIP_FIELDS:
            continue

        if low_key in UNKNOWN_FIELDS and pkg_info.get(key) == 'UNKNOWN':
            continue

        if low_key in PLURAL_FIELDS:
            metadata[PLURAL_FIELDS[low_key]] = pkg_info.get_all(key)

        elif low_key == "requires_dist":
            handle_requires(metadata, pkg_info, key)

        elif low_key == 'provides_extra':
            if not 'extras' in metadata:
                metadata['extras'] = []
            metadata['extras'].extend(pkg_info.get_all(key))

        elif low_key == 'home_page':
            metadata['extensions']['python.details']['project_urls'] = {'Home':pkg_info[key]}

        elif low_key == 'keywords':
            metadata['keywords'] = KEYWORDS_RE.split(pkg_info[key])

        else:
            metadata[low_key] = pkg_info[key]

    metadata['metadata_version'] = METADATA_VERSION

    if 'extras' in metadata:
        metadata['extras'] = sorted(set(metadata['extras']))

    # include more information if distribution is available
    if distribution:
        for requires, attr in (('test_requires', 'tests_require'),):
            try:
                requirements = getattr(distribution, attr)
                if isinstance(requirements, list):
                    new_requirements = list(convert_requirements(requirements))
                    metadata[requires] = [{'requires':new_requirements}]
            except AttributeError:
                pass

    # handle contacts
    contacts = []
    for contact_type, role in CONTACT_FIELDS:
        contact = {}
        for key in contact_type:
            if contact_type[key] in metadata:
                contact[key] = metadata.pop(contact_type[key])
        if contact:
            contact['role'] = role
            contacts.append(contact)
    if contacts:
        metadata['extensions']['python.details']['contacts'] = contacts

    # convert entry points to exports
    try:
        with open(os.path.join(os.path.dirname(path), "entry_points.txt"), "r") as ep_file:
            ep_map = pkg_resources.EntryPoint.parse_map(ep_file.read())
        exports = {}
        for group, items in ep_map.items():
            exports[group] = {}
            for item in items.values():
                name, export = str(item).split(' = ', 1)
                exports[group][name] = export
        if exports:
            metadata['extensions']['python.exports'] = exports
    except IOError:
        pass

    # copy console_scripts entry points to commands
    if 'python.exports' in metadata['extensions']:
        for (ep_script, wrap_script) in (('console_scripts', 'wrap_console'),
                                         ('gui_scripts', 'wrap_gui')):
            if ep_script in metadata['extensions']['python.exports']:
                metadata['extensions']['python.commands'][wrap_script] = \
                    metadata['extensions']['python.exports'][ep_script]

    return metadata


Instruction:
Create a Python 3 function to
compose the version predicates for a given requirement in PEP 345 fashion.

Response:
def requires_to_requires_dist(requirement):
    """Compose the version predicates for requirement in PEP 345 fashion."""
    requires_dist = []
    for op, ver in requirement.specs:
        requires_dist.append(op + ver)
    if not requires_dist:
        return ''
    return " (%s)" % ','.join(requires_dist)


Instruction:
Implement a function in Python 3 to
convert egg - info directory with PKG - INFO to the Metadata 1. 3 aka old - draft Metadata 2. 0 format.

Response:
def pkginfo_to_metadata(egg_info_path, pkginfo_path):
    """
    Convert .egg-info directory with PKG-INFO to the Metadata 1.3 aka
    old-draft Metadata 2.0 format.
    """
    pkg_info = read_pkg_info(pkginfo_path)
    pkg_info.replace_header('Metadata-Version', '2.0')
    requires_path = os.path.join(egg_info_path, 'requires.txt')
    if os.path.exists(requires_path):
        requires = open(requires_path).read()
        for extra, reqs in pkg_resources.split_sections(requires):
            condition = ''
            if extra and ':' in extra: # setuptools extra:condition syntax
                extra, condition = extra.split(':', 1)
            if extra:
                pkg_info['Provides-Extra'] = extra
                if condition:
                    condition += " and "
                condition += 'extra == %s' % repr(extra)
            if condition:
                condition = '; ' + condition
            for new_req in convert_requirements(reqs):
                pkg_info['Requires-Dist'] = new_req + condition

    description = pkg_info['Description']
    if description:
        pkg_info.set_payload(dedent_description(pkg_info))
        del pkg_info['Description']

    return pkg_info


Instruction:
How would you explain what the following Python 3 function does
def set_possible(self):
        '''
        break up a module path to its various parts (prefix, module, class, method)

        this uses PEP 8 conventions, so foo.Bar would be foo module with class Bar

        return -- list -- a list of possible interpretations of the module path
            (eg, foo.bar can be bar module in foo module, or bar method in foo module)
        '''
        possible = []
        name = self.name
        logger.debug('Guessing test name: {}'.format(name))

        name_f = self.name.lower()
        filepath = ""
        if name_f.endswith(".py") or ".py:" in name_f:
            # path/something:Class.method
            bits = name.split(":", 1)
            filepath = bits[0]
            logger.debug('Found filepath: {}'.format(filepath))

            name = bits[1] if len(bits) > 1 else ""
            if name:
                logger.debug('Found test name: {} for filepath: {}'.format(name, filepath))

        bits = name.split('.')
        basedir = self.basedir
        method_prefix = self.method_prefix

        # check if the last bit is a Class
        if re.search(r'^\*?[A-Z]', bits[-1]):
            logger.debug('Found class in name: {}'.format(bits[-1]))
            possible.append(PathFinder(basedir, method_prefix, **{
                'class_name': bits[-1],
                'module_name': bits[-2] if len(bits) > 1 else '',
                'prefix': os.sep.join(bits[0:-2]),
                'filepath': filepath,
            }))
        elif len(bits) > 1 and re.search(r'^\*?[A-Z]', bits[-2]):
            logger.debug('Found class in name: {}'.format(bits[-2]))
            possible.append(PathFinder(basedir, method_prefix, **{
                'class_name': bits[-2],
                'method_name': bits[-1],
                'module_name': bits[-3] if len(bits) > 2 else '',
                'prefix': os.sep.join(bits[0:-3]),
                'filepath': filepath,
            }))
        else:
            if self.name:
                if filepath:
                    if len(bits):
                        possible.append(PathFinder(basedir, method_prefix, **{
                            'filepath': filepath,
                            'method_name': bits[0],
                        }))
                    else:
                        possible.append(PathFinder(basedir, method_prefix, **{
                            'filepath': filepath,
                        }))

                else:
                    logger.debug('Test name is ambiguous')
                    possible.append(PathFinder(basedir, method_prefix, **{
                        'module_name': bits[-1],
                        'prefix': os.sep.join(bits[0:-1]),
                        'filepath': filepath,
                    }))
                    possible.append(PathFinder(basedir, method_prefix, **{
                        'method_name': bits[-1],
                        'module_name': bits[-2] if len(bits) > 1 else '',
                        'prefix': os.sep.join(bits[0:-2]),
                        'filepath': filepath,
                    }))
                    possible.append(PathFinder(basedir, method_prefix, **{
                        'prefix': os.sep.join(bits),
                        'filepath': filepath,
                    }))

            else:
                possible.append(PathFinder(basedir, method_prefix, filepath=filepath))

        logger.debug("Found {} possible test names".format(len(possible)))
        self.possible = possible

Response: This method sets the possible test names for the current object.


Instruction:
Implement a Python 3 function for
returning modules that match module_name

Response:
def modules(self):
        """return modules that match module_name"""

        # since the module has to be importable we go ahead and put the
        # basepath as the very first path to check as that should minimize
        # namespace collisions, this is what unittest does also
        sys.path.insert(0, self.basedir)
        for p in self.paths():
            # http://stackoverflow.com/questions/67631/
            try:
                module_name = self.module_path(p)
                logger.debug("Importing {} from path {}".format(module_name, p))
                m = importlib.import_module(module_name)
                yield m

            except Exception as e:
                logger.warning('Caught exception while importing {}: {}'.format(p, e))
                logger.warning(e, exc_info=True)
                error_info = getattr(self, 'error_info', None)
                if not error_info:
                    exc_info = sys.exc_info()
                    #raise e.__class__, e, exc_info[2]
                    #self.error_info = (e, exc_info)
                    self.error_info = exc_info
                continue

        sys.path.pop(0)


Instruction:
Can you tell what is the following Python 3 function doing
def classes(self):
        for module in self.modules():
            cs = inspect.getmembers(module, inspect.isclass)
            class_name = getattr(self, 'class_name', '')
            class_regex = ''
            if class_name:
                if class_name.startswith("*"):
                    class_name = class_name.strip("*")
                    class_regex = re.compile(r'.*?{}'.format(class_name), re.I)
                else:
                    class_regex = re.compile(r'^{}'.format(class_name), re.I)

            for c_name, c in cs:
                can_yield = True
                if class_regex and not class_regex.match(c_name):
                #if class_name and class_name not in c_name:
                    can_yield = False

                if can_yield and issubclass(c, unittest.TestCase):
                    if c is not unittest.TestCase: # ignore actual TestCase class
                        logger.debug('class: {} matches {}'.format(c_name, class_name))
                        yield c

Response: the partial self. class_name will be used to find actual TestCase classes


Instruction:
Implement a Python 3 function for
returning the actual test methods that matched self. method_name

Response:
def method_names(self):
        """return the actual test methods that matched self.method_name"""
        for c in self.classes():
            #ms = inspect.getmembers(c, inspect.ismethod)
            # http://stackoverflow.com/questions/17019949/
            ms = inspect.getmembers(c, lambda f: inspect.ismethod(f) or inspect.isfunction(f))
            method_name = getattr(self, 'method_name', '')
            method_regex = ''
            if method_name:
                if method_name.startswith(self.method_prefix):
                    method_regex = re.compile(r'^{}'.format(method_name), flags=re.I)

                else:

                    if method_name.startswith("*"):
                        method_name = method_name.strip("*")
                        method_regex = re.compile(
                            r'^{}[_]{{0,1}}.*?{}'.format(self.method_prefix, method_name),
                            flags=re.I
                        )
                    else:
                        method_regex = re.compile(
                            r'^{}[_]{{0,1}}{}'.format(self.method_prefix, method_name),
                            flags=re.I
                        )

            for m_name, m in ms:
                if not m_name.startswith(self.method_prefix): continue

                can_yield = True
                if method_regex and not method_regex.match(m_name):
                    can_yield = False

                if can_yield:
                    logger.debug('method: {} matches {}'.format(m_name, method_name))
                    yield c, m_name


Instruction:
Can you write a function in Python 3 where it
searches for a basename in the list of basenames or postfixes

Response:
def _find_basename(self, name, basenames, is_prefix=False):
        """check if name combined with test prefixes or postfixes is found anywhere
        in the list of basenames

        :param name: string, the name you're searching for
        :param basenames: list, a list of basenames to check
        :param is_prefix: bool, True if this is a prefix search, which means it will
            also check if name matches any of the basenames without the prefixes or
            postfixes, if it is False then the prefixes or postfixes must be present
            (ie, the module we're looking for is the actual test module, not the parent
             modules it's contained in)
        :returns: string, the basename if it is found
        """
        ret = ""
        fileroots = [(os.path.splitext(n)[0], n) for n in basenames]
        glob = False
        if name.startswith("*"):
            glob = True
        name = name.strip("*")

        for fileroot, basename in fileroots:
            if name in fileroot or fileroot in name:
                for pf in self.module_postfixes:
                    logger.debug(
                        'Checking if basename {} starts with {} and ends with {}'.format(
                        basename,
                        name,
                        pf
                    ))
                    if glob:
                        if name in fileroot and fileroot.endswith(pf):
                            ret = basename
                            break
                    else:
                        if fileroot.startswith(name) and fileroot.endswith(pf):
                            ret = basename
                            break

                if not ret:
                    for pf in self.module_prefixes:
                        n = pf + name
                        logger.debug('Checking if basename {} starts with {}'.format(basename, n))
                        if glob:
                            if fileroot.startswith(pf) and name in fileroot:
                                ret = basename
                                break
                        else:
                            if fileroot.startswith(n):
                                ret = basename
                                break

                if not ret:
                    if is_prefix:
                        logger.debug('Checking if basename {} starts with {}'.format(basename, name))
                        if basename.startswith(name) or (glob and name in basename):
                            ret = basename

                        else:
                            logger.debug(
                                'Checking if basename {} starts with {} and is a test module'.format(
                                basename,
                                name
                            ))
                            if glob:
                                if name in basename and self._is_module_path(basename):
                                    ret = basename

                            else:
                                if basename.startswith(name) and self._is_module_path(basename):
                                    ret = basename

                if ret:
                    logger.debug('Found basename {}'.format(ret))
                    break

        return ret


Instruction:
Here you have a function in Python 3, explain what it does
def _find_prefix_path(self, basedir, prefix):
        ret = ""
        for ret in self._find_prefix_paths(basedir, prefix):
            break

        if not ret:
            raise IOError("Could not find prefix {} in path {}".format(prefix, basedir))

        return ret

Response: Similar to _find_prefix_paths but only returns the first match


Instruction:
Write a Python 3 function that can
return true if the passed in path is a test module path False otherwise

Response:
def _is_module_path(self, path):
        """Returns true if the passed in path is a test module path

        :param path: string, the path to check, will need to start or end with the
            module test prefixes or postfixes to be considered valid
        :returns: boolean, True if a test module path, False otherwise
        """
        ret = False
        basename = os.path.basename(path)
        fileroot = os.path.splitext(basename)[0]
        for pf in self.module_postfixes:
            if fileroot.endswith(pf):
                ret = True
                break

        if not ret:
            for pf in self.module_prefixes:
                if fileroot.startswith(pf):
                    ret = True
                    break
        return ret


Instruction:
Given the following Python 3 function, write the documentation
def walk(self, basedir):
        system_d = SitePackagesDir()
        filter_system_d = system_d and os.path.commonprefix([system_d, basedir]) != system_d

        for root, dirs, files in os.walk(basedir, topdown=True):
            # ignore dot directories and private directories (start with underscore)
            dirs[:] = [d for d in dirs if d[0] != '.' and d[0] != "_"]

            if filter_system_d:
                dirs[:] = [d for d in dirs if not d.startswith(system_d)]

            yield root, dirs, files

Response: Walk all the directories of basedir except hidden directories


Instruction:
Can you tell what is the following Python 3 function doing
def paths(self):
        '''
        given a basedir, yield all test modules paths recursively found in
        basedir that are test modules

        return -- generator
        '''
        module_name = getattr(self, 'module_name', '')
        module_prefix = getattr(self, 'prefix', '')
        filepath = getattr(self, 'filepath', '')

        if filepath:
            if os.path.isabs(filepath):
                yield filepath

            else:
                yield os.path.join(self.basedir, filepath)

        else:
            if module_prefix:
                basedirs = self._find_prefix_paths(self.basedir, module_prefix)
            else:
                basedirs = [self.basedir]

            for basedir in basedirs:
                try:
                    if module_name:
                        path = self._find_module_path(basedir, module_name)

                    else:
                        path = basedir

                    if os.path.isfile(path):
                        logger.debug('Module path: {}'.format(path))
                        yield path

                    else:
                        seen_paths = set()
                        for root, dirs, files in self.walk(path):
                            for basename in files:
                                if basename.startswith("__init__"):
                                    if self._is_module_path(root):
                                        filepath = os.path.join(root, basename)
                                        if filepath not in seen_paths:
                                            logger.debug('Module package path: {}'.format(filepath))
                                            seen_paths.add(filepath)
                                            yield filepath

                                else:
                                    fileroot = os.path.splitext(basename)[0]
                                    for pf in self.module_postfixes:
                                        if fileroot.endswith(pf):
                                            filepath = os.path.join(root, basename)
                                            if filepath not in seen_paths:
                                                logger.debug('Module postfix path: {}'.format(filepath))
                                                seen_paths.add(filepath)
                                                yield filepath

                                    for pf in self.module_prefixes:
                                        if fileroot.startswith(pf):
                                            filepath = os.path.join(root, basename)
                                            if filepath not in seen_paths:
                                                logger.debug('Module prefix path: {}'.format(filepath))
                                                seen_paths.add(filepath)
                                                yield filepath

                except IOError as e:
                    # we failed to find a suitable path
                    logger.warning(e, exc_info=True)
                    pass

Response:
yields all test modules paths recursively found in
        basedir