Instruction:
Can you implement a function in Python 3 that
finds iso-valued contours in a 2D array for a given level value. Uses the "marching squares" method to compute a the iso-valued contours of the input 2D array for a particular level value. Array values are linearly interpolated to provide better precision for the output contours. Parameters ---------- array : 2D ndarray of double Input data in which to find contours. level : float Value along which to find contours in the array. fully_connected : str, {'low', 'high'} Indicates whether array elements below the given level value are to be considered fully-connected (and hence elements above the value will only be face connected), or vice-versa. (See notes below for details.) positive_orientation : either 'low' or 'high' Indicates whether the output contours will produce positively-oriented polygons around islands of low- or high-valued elements. If 'low' then contours will wind counter- clockwise around elements below the iso-value. Alternately, this means that low-valued elements are always on the left of the contour. (See below for details.) Returns ------- contours : list of (n,2)-ndarrays Each contour is an ndarray of shape ``(n, 2)``, consisting of n ``(row, column)`` coordinates along the contour. Notes ----- The marching squares algorithm is a special case of the marching cubes algorithm [1]_. A simple explanation is available here:: http://www.essi.fr/~lingrand/MarchingCubes/algo.html There is a single ambiguous case in the marching squares algorithm: when a given ``2 x 2``-element square has two high-valued and two low-valued elements, each pair diagonally adjacent. (Where high- and low-valued is with respect to the contour value sought.) In this case, either the high-valued elements can be 'connected together' via a thin isthmus that separates the low-valued elements, or vice-versa. When elements are connected together across a diagonal, they are considered 'fully connected' (also known as 'face+vertex-connected' or '8-connected'). Only high-valued or low-valued elements can be fully-connected, the other set will be considered as 'face-connected' or '4-connected'. By default, low-valued elements are considered fully-connected; this can be altered with the 'fully_connected' parameter. Output contours are not guaranteed to be closed: contours which intersect the array edge will be left open. All other contours will be closed. (The closed-ness of a contours can be tested by checking whether the beginning point is the same as the end point.) Contours are oriented. By default, array values lower than the contour value are to the left of the contour and values greater than the contour value are to the right. This means that contours will wind counter-clockwise (i.e. in 'positive orientation') around islands of low-valued pixels. This behavior can be altered with the 'positive_orientation' parameter. The order of the contours in the output list is determined by the position of the smallest ``x,y`` (in lexicographical order) coordinate in the contour. This is a side-effect of how the input array is traversed, but can be relied upon. .. warning:: Array coordinates/values are assumed to refer to the *center* of the array element. Take a simple example input: ``[0, 1]``. The interpolated position of 0.5 in this array is midway between the 0-element (at ``x=0``) and the 1-element (at ``x=1``), and thus would fall at ``x=0.5``. This means that to find reasonable contours, it is best to find contours midway between the expected "light" and "dark" values. In particular, given a binarized array, *do not* choose to find contours at the low or high value of the array. This will often yield degenerate contours, especially around structures that are a single array element wide. Instead choose a middle value, as above. References ---------- .. [1] Lorensen, William and Harvey E. Cline. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. Computer Graphics (SIGGRAPH 87 Proceedings) 21(4) July 1987, p. 163-170). Examples -------- >>> a = np.zeros((3, 3)) >>> a[0, 0] = 1 >>> a array([[ 1., 0., 0.], [ 0., 0., 0.], [ 0., 0., 0.]]) >>> find_contours(a, 0.5) [array([[ 0. , 0.5], [ 0.5, 0. ]])]

Response:
def find_contours(array, level,
                  fully_connected='low', positive_orientation='low'):
    """Find iso-valued contours in a 2D array for a given level value.

    Uses the "marching squares" method to compute a the iso-valued contours of
    the input 2D array for a particular level value. Array values are linearly
    interpolated to provide better precision for the output contours.

    Parameters
    ----------
    array : 2D ndarray of double
        Input data in which to find contours.
    level : float
        Value along which to find contours in the array.
    fully_connected : str, {'low', 'high'}
         Indicates whether array elements below the given level value are to be
         considered fully-connected (and hence elements above the value will
         only be face connected), or vice-versa. (See notes below for details.)
    positive_orientation : either 'low' or 'high'
         Indicates whether the output contours will produce positively-oriented
         polygons around islands of low- or high-valued elements. If 'low' then
         contours will wind counter- clockwise around elements below the
         iso-value. Alternately, this means that low-valued elements are always
         on the left of the contour. (See below for details.)

    Returns
    -------
    contours : list of (n,2)-ndarrays
        Each contour is an ndarray of shape ``(n, 2)``,
        consisting of n ``(row, column)`` coordinates along the contour.

    Notes
    -----
    The marching squares algorithm is a special case of the marching cubes
    algorithm [1]_.  A simple explanation is available here::

      http://www.essi.fr/~lingrand/MarchingCubes/algo.html

    There is a single ambiguous case in the marching squares algorithm: when
    a given ``2 x 2``-element square has two high-valued and two low-valued
    elements, each pair diagonally adjacent. (Where high- and low-valued is
    with respect to the contour value sought.) In this case, either the
    high-valued elements can be 'connected together' via a thin isthmus that
    separates the low-valued elements, or vice-versa. When elements are
    connected together across a diagonal, they are considered 'fully
    connected' (also known as 'face+vertex-connected' or '8-connected'). Only
    high-valued or low-valued elements can be fully-connected, the other set
    will be considered as 'face-connected' or '4-connected'. By default,
    low-valued elements are considered fully-connected; this can be altered
    with the 'fully_connected' parameter.

    Output contours are not guaranteed to be closed: contours which intersect
    the array edge will be left open. All other contours will be closed. (The
    closed-ness of a contours can be tested by checking whether the beginning
    point is the same as the end point.)

    Contours are oriented. By default, array values lower than the contour
    value are to the left of the contour and values greater than the contour
    value are to the right. This means that contours will wind
    counter-clockwise (i.e. in 'positive orientation') around islands of
    low-valued pixels. This behavior can be altered with the
    'positive_orientation' parameter.

    The order of the contours in the output list is determined by the position
    of the smallest ``x,y`` (in lexicographical order) coordinate in the
    contour.  This is a side-effect of how the input array is traversed, but
    can be relied upon.

    .. warning::

       Array coordinates/values are assumed to refer to the *center* of the
       array element. Take a simple example input: ``[0, 1]``. The interpolated
       position of 0.5 in this array is midway between the 0-element (at
       ``x=0``) and the 1-element (at ``x=1``), and thus would fall at
       ``x=0.5``.

    This means that to find reasonable contours, it is best to find contours
    midway between the expected "light" and "dark" values. In particular,
    given a binarized array, *do not* choose to find contours at the low or
    high value of the array. This will often yield degenerate contours,
    especially around structures that are a single array element wide. Instead
    choose a middle value, as above.

    References
    ----------
    .. [1] Lorensen, William and Harvey E. Cline. Marching Cubes: A High
           Resolution 3D Surface Construction Algorithm. Computer Graphics
           (SIGGRAPH 87 Proceedings) 21(4) July 1987, p. 163-170).

    Examples
    --------
    >>> a = np.zeros((3, 3))
    >>> a[0, 0] = 1
    >>> a
    array([[ 1.,  0.,  0.],
           [ 0.,  0.,  0.],
           [ 0.,  0.,  0.]])
    >>> find_contours(a, 0.5)
    [array([[ 0. ,  0.5],
           [ 0.5,  0. ]])]
    """
    array = np.asarray(array, dtype=np.double)
    if array.ndim != 2:
        raise ValueError('Only 2D arrays are supported.')
    level = float(level)
    if (fully_connected not in _param_options or
            positive_orientation not in _param_options):
        raise ValueError('Parameters "fully_connected" and'
                         ' "positive_orientation" must be either "high" or'
                         ' "low".')
    point_list = _find_contours_cy.iterate_and_store(array, level,
                                                     fully_connected == 'high')
    contours = _assemble_contours(_take_2(point_list))
    if positive_orientation == 'high':
        contours = [c[::-1] for c in contours]
    return contours


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_module_ident_from_ident_hash(ident_hash, cursor):
    try:
        uuid, (mj_ver, mn_ver) = split_ident_hash(
            ident_hash, split_version=True)
    except IdentHashMissingVersion as e:
        uuid, mj_ver, mn_ver = e.id, None, None
    args = [uuid]
    stmt = "SELECT module_ident FROM {} WHERE uuid = %s"
    table_name = 'modules'
    if mj_ver is None:
        table_name = 'latest_modules'
    else:
        args.append(mj_ver)
        stmt += " AND major_version = %s"
    if mn_ver is not None:
        args.append(mn_ver)
        stmt += " AND minor_version = %s"
    stmt = stmt.format(table_name)

    cursor.execute(stmt, args)
    try:
        module_ident = cursor.fetchone()[0]
    except TypeError:  # NoneType
        module_ident = None
    return module_ident

Response: Return the moduleid for a given ident_hash.


Instruction:
Can you implement a function in Python 3 that
returns a JSON representation of the binder tree for ident_hash.

Response:
def get_tree(ident_hash, cursor, as_collated=False):
    """Return a JSON representation of the binder tree for ``ident_hash``."""
    uuid, version = split_ident_hash(ident_hash)
    cursor.execute(SQL['get-tree-by-uuid-n-version'],
                   (uuid, version, as_collated,))
    try:
        tree = cursor.fetchone()[0]
    except TypeError:  # NoneType
        raise ContentNotFound()
    if type(tree) in (type(''), type(u'')):
        return json.loads(tree)
    else:
        return tree


Instruction:
Implement a function in Python 3 to
return collated content for ident_hash.

Response:
def get_collated_content(ident_hash, context_ident_hash, cursor):
    """Return collated content for ``ident_hash``."""
    cursor.execute(SQL['get-collated-content'],
                   (ident_hash, context_ident_hash,))
    try:
        return cursor.fetchone()[0]
    except TypeError:  # NoneType
        return


Instruction:
Write a Python 3 script to
retrieve page uuid from legacy moduleid.

Response:
def get_module_uuid(plpy, moduleid):
    """Retrieve page uuid from legacy moduleid."""
    plan = plpy.prepare("SELECT uuid FROM modules WHERE moduleid = $1;",
                        ('text',))
    result = plpy.execute(plan, (moduleid,), 1)
    if result:
        return result[0]['uuid']


Instruction:
Here you have a function in Python 3, explain what it does
def get_current_module_ident(moduleid, plpy):
    plan = plpy.prepare('''\
        SELECT m.module_ident FROM modules m
        WHERE m.moduleid = $1 ORDER BY revised DESC''', ('text',))
    results = plpy.execute(plan, (moduleid,), 1)
    if results:
        return results[0]['module_ident']

Response: Retrieve the module_ident for a given moduleid.


Instruction:
Here you have a function in Python 3, explain what it does
def get_minor_version(module_ident, plpy):
    # Make sure to always return the max minor version that is already in the
    # database, in case the given module_ident is not the latest version
    plan = plpy.prepare('''\
        WITH t AS (
            SELECT uuid, major_version
                FROM modules
                WHERE module_ident = $1
        )
        SELECT MAX(m.minor_version) AS minor_version
            FROM modules m, t
            WHERE m.uuid = t.uuid AND m.major_version = t.major_version
        ''', ('integer',))
    results = plpy.execute(plan, (module_ident,), 1)
    return results[0]['minor_version']

Response: Retrieve minor version only given module_ident.


Instruction:
Implement a function in Python 3 to
get all the collections that the module is part of.

Response:
def get_collections(module_ident, plpy):
    """Get all the collections that the module is part of."""
    # Make sure to only return one match per collection and only if it is the
    # latest collection (which may not be the same as what is in
    # latest_modules)
    plan = plpy.prepare('''
WITH RECURSIVE t(node, parent, path, document) AS (
        SELECT tr.nodeid, tr.parent_id, ARRAY[tr.nodeid], tr.documentid
        FROM trees tr
        WHERE tr.documentid = $1 and tr.is_collated = 'False'
    UNION ALL
        SELECT c.nodeid, c.parent_id, path || ARRAY[c.nodeid], c.documentid
        FROM trees c JOIN t ON (c.nodeid = t.parent)
        WHERE not c.nodeid = ANY(t.path)
    ),
    latest(module_ident) AS (
    SELECT module_ident FROM (
        SELECT m.module_ident, m.revised,
            MAX(m.revised) OVER (PARTITION BY m.uuid) as latest
        FROM  modules m where m.portal_type = 'Collection'
    ) r
    WHERE r.revised = r.latest
    )
    SELECT module_ident FROM t, latest
        WHERE latest.module_ident = t.document
    ''', ('integer',))
    for i in plpy.execute(plan, (module_ident,)):
        yield i['module_ident']


Instruction:
Create a Python 3 function for
getting all the sub - collections that the module is part of.

Response:
def get_subcols(module_ident, plpy):
    """Get all the sub-collections that the module is part of."""
    plan = plpy.prepare('''
    WITH RECURSIVE t(node, parent, path, document) AS (
        SELECT tr.nodeid, tr.parent_id, ARRAY[tr.nodeid], tr.documentid
        FROM trees tr
        WHERE tr.documentid = $1 and tr.is_collated = 'False'
    UNION ALL
        SELECT c.nodeid, c.parent_id, path || ARRAY[c.nodeid], c.documentid
        FROM trees c JOIN t ON (c.nodeid = t.parent)
        WHERE not c.nodeid = ANY(t.path)
    )
    SELECT DISTINCT m.module_ident
    FROM t JOIN modules m ON (t.document = m.module_ident)
    WHERE m.portal_type  = 'SubCollection'
    ORDER BY m.module_ident
    ''', ('integer',))
    for i in plpy.execute(plan, (module_ident,)):
        yield i['module_ident']


Instruction:
Write a Python 3 function that can
rebuild the tree based on the old collection.

Response:
def rebuild_collection_tree(old_collection_ident, new_document_id_map, plpy):
    """Create a new tree for the collection based on the old tree.

    This uses new document ids, replacing old ones.
    """
    get_tree = plpy.prepare('''
    WITH RECURSIVE t(node, parent, document, title, childorder, latest, path)
        AS (SELECT tr.nodeid, tr.parent_id, tr.documentid, tr.title,
                   tr.childorder, tr.latest, ARRAY[tr.nodeid]
            FROM trees tr
            WHERE tr.documentid = $1 AND tr.is_collated = 'False'
    UNION ALL
        SELECT c.nodeid, c.parent_id, c.documentid, c.title,
               c.childorder, c.latest, path || ARRAY[c.nodeid]
        FROM trees c JOIN t ON (c.parent_id = t.node)
        WHERE not c.nodeid = ANY(t.path)
    )
    SELECT * FROM t
    ''', ('integer',))

    def get_old_tree():
        return plpy.execute(get_tree, (old_collection_ident,))

    tree = {}  # { old_nodeid: {'data': ...}, ...}
    children = {}  # { nodeid: [child_nodeid, ...], child_nodeid: [...]}
    for i in get_old_tree():
        tree[i['node']] = {'data': i, 'new_nodeid': None}
        children.setdefault(i['parent'], [])
        children[i['parent']].append(i['node'])

    insert_tree = plpy.prepare('''
    INSERT INTO trees (nodeid, parent_id, documentid,
        title, childorder, latest)
    VALUES (DEFAULT, $1, $2, $3, $4, $5)
    RETURNING nodeid
    ''', ('integer', 'integer', 'text', 'integer', 'boolean'))

    def execute(fields):
        results = plpy.execute(insert_tree, fields, 1)
        return results[0]['nodeid']

    root_node = children[None][0]

    def build_tree(node, parent):
        data = tree[node]['data']
        new_node = execute([parent, new_document_id_map.get(data['document'],
                            data['document']), data['title'],
                            data['childorder'], data['latest']])
        for i in children.get(node, []):
            build_tree(i, new_node)
    build_tree(root_node, None)


Instruction:
How would you code a function in Python 3 to
republish a new version of the modules in the specified collection.

Response:
def republish_collection(submitter, submitlog, next_minor_version,
                         collection_ident, plpy, revised=None):
    """Insert a new row for collection_ident with a new version.

    Returns the module_ident of the row inserted.
    """
    sql = '''
    INSERT INTO modules (portal_type, moduleid, uuid, version, name, created,
        revised, abstractid, licenseid, doctype, submitter, submitlog,
        stateid, parent, language, authors, maintainers, licensors,
        parentauthors, google_analytics, buylink, print_style,
        major_version, minor_version)
      SELECT m.portal_type, m.moduleid, m.uuid, m.version, m.name, m.created,
        {}, m.abstractid, m.licenseid, m.doctype, $3, $4,
        m.stateid, m.parent, m.language, m.authors, m.maintainers, m.licensors,
        m.parentauthors, m.google_analytics, m.buylink, m.print_style,
        m.major_version, $1
      FROM modules m
      WHERE m.module_ident = $2
    RETURNING module_ident
    '''
    if revised is None:
        sql = sql.format('CURRENT_TIMESTAMP')
        types = ('integer', 'integer', 'text', 'text')
        params = (next_minor_version, collection_ident, submitter, submitlog)
    else:
        sql = sql.format('$5')
        types = ('integer', 'integer', 'text', 'text', 'timestamp')
        params = (next_minor_version, collection_ident, submitter, submitlog,
                  revised)
    plan = plpy.prepare(sql, types)
    new_ident = plpy.execute(plan, params, 1)[0]['module_ident']

    plan = plpy.prepare("""\
        INSERT INTO modulekeywords (module_ident, keywordid)
        SELECT $1, keywordid
        FROM modulekeywords
        WHERE module_ident = $2""", ('integer', 'integer'))
    plpy.execute(plan, (new_ident, collection_ident,))

    plan = plpy.prepare("""\
        INSERT INTO moduletags (module_ident, tagid)
        SELECT $1, tagid
        FROM moduletags
        WHERE module_ident = $2""", ('integer', 'integer'))
    plpy.execute(plan, (new_ident, collection_ident,))

    plan = plpy.prepare("""\
        INSERT INTO module_files (module_ident, fileid, filename)
        SELECT $1, fileid, filename
        FROM module_files
        WHERE module_ident = $2 and filename != 'collection.xml'""",
                        ('integer', 'integer'))
    plpy.execute(plan, (new_ident, collection_ident,))
    return new_ident


Instruction:
Can you create a Python 3 function that
sets the major_version and minor_version of the current object.

Response:
def set_version(portal_type, legacy_version, td):
    """Set the major_version and minor_version if they are not set."""
    modified = 'OK'
    legacy_major, legacy_minor = legacy_version.split('.')

    if portal_type == 'Collection':
        # For collections, both major and minor needs to be set
        modified = 'MODIFY'
        td['new']['major_version'] = int(legacy_minor)
        if td['new']['minor_version'] is None:
            td['new']['minor_version'] = 1

    elif portal_type == 'Module':
        # For modules, major should be set and minor should be None
        # N.B. a very few older modules had major=2 and minor zero-based.
        # The odd math below adds one to the minor for those
        modified = 'MODIFY'
        td['new']['major_version'] = int(legacy_minor)+(int(legacy_major)-1)
        td['new']['minor_version'] = None

    return modified


Instruction:
Make a summary of the following Python 3 code
def republish_module(td, plpy):
    portal_type = td['new']['portal_type']
    modified = 'OK'
    moduleid = td['new']['moduleid']
    legacy_version = td['new']['version']
    submitter = td['new']['submitter']
    submitlog = td['new']['submitlog']

    modified = set_version(portal_type, legacy_version, td)

    current_module_ident = get_current_module_ident(moduleid, plpy)
    if current_module_ident:
        # need to overide autogen uuid to keep it constant per moduleid
        uuid = get_module_uuid(plpy, moduleid)
        td['new']['uuid'] = uuid
        modified = 'MODIFY'
    else:
        # nothing to do if the module/collection is new
        return modified

    if portal_type != 'Module':
        # nothing else to do if something else is being published
        return modified

    # Module is republished
    replace_map = {current_module_ident: td['new']['module_ident']}
    # find the nested subcollections the module is in, and
    # republish them, as well, adding to map, for all collections
    # Note that map is for all subcollections, regardless of what
    # collection they are contained in.
    for sub_id in get_subcols(current_module_ident, plpy):
        minor = next_version(sub_id, plpy)
        new_subcol_ident = republish_collection(submitter, submitlog,
                                                minor, sub_id, plpy)
        replace_map[sub_id] = new_subcol_ident

    # Now do each collection that contains this module
    for collection_id in get_collections(current_module_ident, plpy):
        minor = next_version(collection_id, plpy)
        new_ident = republish_collection(submitter, submitlog, minor,
                                         collection_id, plpy)
        replace_map[collection_id] = new_ident
        rebuild_collection_tree(collection_id, replace_map, plpy)

    return modified

Response: This function is used to republish a module. It will create a new minor version of collections.


Instruction:
How would you code a function in Python 3 to
trigger called from postgres database when republishing a module.

Response:
def republish_module_trigger(plpy, td):
    """Trigger called from postgres database when republishing a module.

    When a module is republished, the versions of the collections that it is
    part of will need to be updated (a minor update).

    e.g. there is a collection c1 v2.1, which contains module m1 v3

    m1 is updated, we have a new row in the modules table with m1 v4

    this trigger will create increment the minor version of c1, so we'll have
    c1 v2.2

    we need to create a collection tree for c1 v2.2 which is exactly the same
    as c1 v2.1, but with m1 v4 instead of m1 v3, and c1 v2.2 instead of c1 v2.2
    """
    # Is this an insert from legacy? Legacy always supplies the version.
    is_legacy_publication = td['new']['version'] is not None
    if not is_legacy_publication:
        # Bail out, because this trigger only applies to legacy publications.
        return "OK"

    plpy.log('Trigger fired on %s' % (td['new']['moduleid'],))

    modified = republish_module(td, plpy)
    plpy.log('modified: {}'.format(modified))
    plpy.log('insert values:\n{}\n'.format('\n'.join([
        '{}: {}'.format(key, value)
        for key, value in td['new'].items()])))

    return modified


Instruction:
Explain what the following Python 3 code does
def assign_moduleid_default_trigger(plpy, td):
    modified_state = "OK"
    portal_type = td['new']['portal_type']
    uuid = td['new']['uuid']
    moduleid = td['new']['moduleid']
    version = td['new']['version']

    # Is this an insert from legacy? Legacy always supplies the version.
    is_legacy_publication = version is not None

    if moduleid is None:
        # If the moduleid is not supplied, it is a new publication.
        if portal_type in ("Collection", "SubCollection"):
            prefix, sequence_name = 'col', "collectionid_seq"
        else:
            prefix, sequence_name = 'm', "moduleid_seq"
        plan = plpy.prepare("SELECT $1 || nextval($2)::text AS moduleid",
                            ['text', 'text'])
        row = plpy.execute(plan, (prefix, sequence_name,), 1)
        moduleid = row[0]['moduleid']
        modified_state = "MODIFY"
        td['new']['moduleid'] = moduleid
    elif is_legacy_publication and moduleid is not None:
        # Set the sequence value based on what legacy gave us.
        plan = plpy.prepare("""\
SELECT setval($1, max(substr(moduleid, $2)::int))
FROM (
  SELECT moduleid from modules where portal_type in ($3,$4)
  UNION ALL
  SELECT $4) AS all_together""", ['text', 'int', 'text', 'text'])
        args = []
        if portal_type == 'Collection':
            args.append('collectionid_seq')
            args.append(4)
            args.extend(('Collection', 'SubCollection'))
        elif portal_type == 'Module':
            args.append('moduleid_seq')
            args.append(2)
            args.extend(('Module', 'CompositeModule'))
        args.append(moduleid)
        if len(args) == 4:
            plpy.execute(plan, args)

    plpy.log("Fixed identifier and version for publication at '{}' "
             "with the following values: {} and {}"
             .format(uuid, moduleid, version))

    return modified_state

Response: Trigger to fill in legacy moduleid when publishing.


Instruction:
Can you implement a function in Python 3 that
assigns the default version to the noton version field.

Response:
def assign_version_default_trigger(plpy, td):
    """Trigger to fill in legacy data fields.

    A compatibilty trigger to fill in legacy data fields that are not
    populated when inserting publications from cnx-publishing.

    If this is not a legacy publication the ``version`` will be set
    based on the ``major_version`` value.
    """
    modified_state = "OK"
    portal_type = td['new']['portal_type']
    version = td['new']['version']
    minor_version = td['new']['minor_version']

    # Set the minor version on collections, because by default it is
    # None/Null, which is the correct default for modules.
    if minor_version is None and portal_type in ('Collection',
                                                 'SubCollection'):
        modified_state = "MODIFY"
        td['new']['minor_version'] = 1

    # Set the legacy version field based on the major version.
    if version is None:
        major_version = td['new']['major_version']
        version = "1.{}".format(major_version)
        modified_state = "MODIFY"
        td['new']['version'] = version

    return modified_state


Instruction:
Create a Python 3 function to
assign default document_controls to the noton_document_controls table.

Response:
def assign_document_controls_default_trigger(plpy, td):
    """Trigger to fill in document_controls when legacy publishes.

    A compatibilty trigger to fill in ``uuid`` and ``licenseid`` columns
    of the ``document_controls`` table that are not
    populated when inserting publications from legacy.

    This uuid default is not on ``modules.uuid`` column itself,
    because the value needs to be loosely associated
    with the ``document_controls`` entry
    to prevent uuid collisions and bridge the pending publications gap.
    """
    modified_state = "OK"
    uuid = td['new']['uuid']

    # Only do the procedure if this is a legacy publication.
    if uuid is None:
        modified_state = "MODIFY"
        plan = plpy.prepare("""\
INSERT INTO document_controls (uuid, licenseid) VALUES (DEFAULT, $1)
RETURNING uuid""", ('integer',))
        uuid_ = plpy.execute(plan, (td['new']['licenseid'],))[0]['uuid']
        td['new']['uuid'] = uuid_

    return modified_state


Instruction:
Can you generate the documentation for the following Python 3 function
def upsert_document_acl_trigger(plpy, td):
    modified_state = "OK"
    uuid_ = td['new']['uuid']
    authors = td['new']['authors'] and td['new']['authors'] or []
    maintainers = td['new']['maintainers'] and td['new']['maintainers'] or []
    is_legacy_publication = td['new']['version'] is not None

    if not is_legacy_publication:
        return modified_state

    # Upsert all authors and maintainers into the ACL
    # to give them publish permission.
    permissibles = []
    permissibles.extend(authors)
    permissibles.extend(maintainers)
    permissibles = set([(uid, 'publish',) for uid in permissibles])

    plan = plpy.prepare("""\
SELECT user_id, permission FROM document_acl WHERE uuid = $1""",
                        ['uuid'])
    existing_permissibles = set([(r['user_id'], r['permission'],)
                                 for r in plpy.execute(plan, (uuid_,))])

    new_permissibles = permissibles.difference(existing_permissibles)

    for uid, permission in new_permissibles:
        plan = plpy.prepare("""\
INSERT INTO document_acl (uuid, user_id, permission)
VALUES ($1, $2, $3)""", ['uuid', 'text', 'permission_type'])
        plpy.execute(plan, (uuid_, uid, permission,))

Response: Trigger for filling in acls when legacy publishes.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def upsert_users_from_legacy_publication_trigger(plpy, td):
    modified_state = "OK"
    authors = td['new']['authors'] and td['new']['authors'] or []
    maintainers = td['new']['maintainers'] and td['new']['maintainers'] or []
    licensors = td['new']['licensors'] and td['new']['licensors'] or []
    is_legacy_publication = td['new']['version'] is not None

    if not is_legacy_publication:
        return modified_state

    # Upsert all roles into the users table.
    users = []
    users.extend(authors)
    users.extend(maintainers)
    users.extend(licensors)
    users = list(set(users))

    plan = plpy.prepare("""\
SELECT username FROM users WHERE username = any($1)""",
                        ['text[]'])
    existing_users = set([r['username'] for r in plpy.execute(plan, (users,))])

    new_users = set(users).difference(existing_users)
    for username in new_users:
        plan = plpy.prepare("""\
INSERT INTO users (username, first_name, last_name, full_name, title)
SELECT personid, firstname, surname, fullname, honorific
FROM persons where personid = $1""", ['text'])
        plpy.execute(plan, (username,))

    return modified_state

Response: A compatibility trigger to upsert users from legacy persons table.


Instruction:
Make a summary of the following Python 3 code
def insert_users_for_optional_roles_trigger(plpy, td):
    modified_state = "OK"
    users = td['new']['personids'] and td['new']['personids'] or []

    plan = plpy.prepare("""\
SELECT username FROM users WHERE username = any($1)""",
                        ['text[]'])
    existing_users = set([r['username'] for r in plpy.execute(plan, (users,))])

    new_users = set(users).difference(existing_users)
    for username in new_users:
        plan = plpy.prepare("""\
INSERT INTO users (username, first_name, last_name, full_name, title)
SELECT personid, firstname, surname, fullname, honorific
FROM persons where personid = $1""", ['text'])
        plpy.execute(plan, (username,))

    return modified_state

Response: Trigger to insert users from moduleoptionalroles.


Instruction:
Implement a Python 3 function for
adding a module file to the internal list of modules.

Response:
def add_module_file(plpy, td):
    """Database trigger for adding a module file.

    When a legacy ``index.cnxml`` is added, this trigger
    transforms it into html and stores it as ``index.cnxml.html``.
    When a cnx-publishing ``index.cnxml.html`` is added, this trigger
    checks if ``index.html.cnxml`` exists before
    transforming it into cnxml and stores it as ``index.html.cnxml``.

    Note, we do not use ``index.html`` over ``index.cnxml.html``, because
    legacy allows users to name files ``index.html``.

    """
    module_ident = td['new']['module_ident']
    filename = td['new']['filename']
    msg = "produce {}->{} for module_ident = {}"

    def check_for(filenames, module_ident):
        """Find filenames associated with module_ident.

        Check for a file at ``filename`` associated with
        module at ``module_ident``.
        """
        stmt = plpy.prepare("""\
SELECT TRUE AS exists FROM module_files
WHERE filename = $1 AND module_ident = $2""", ['text', 'integer'])
        any_exist = False
        for filename in filenames:
            result = plpy.execute(stmt, [filename, module_ident])
            try:
                exists = result[0]['exists']
            except IndexError:
                exists = False
            any_exist = any_exist or exists
        return any_exist

    # Declare the content producer function variable,
    #   because it is possible that it will not be assigned.
    producer_func = None
    if filename == 'index.cnxml':
        new_filenames = ('index.cnxml.html',)
        # Transform content to html.
        other_exists = check_for(new_filenames, module_ident)
        if not other_exists:
            msg = msg.format('cnxml', 'html', module_ident)
            producer_func = produce_html_for_module
    elif filename == 'index.cnxml.html':
        new_filenames = ('index.html.cnxml', 'index.cnxml',)
        # Transform content to cnxml.
        other_exists = check_for(new_filenames, module_ident)
        if not other_exists:
            msg = msg.format('html', 'cnxml', module_ident)
            producer_func = produce_cnxml_for_module
    else:
        # Not one of the special named files.
        return  # skip

    plpy.info(msg)
    if producer_func is not None:
        producer_func(plpy,
                      module_ident,
                      source_filename=filename,
                      destination_filenames=new_filenames)
    _transform_abstract(plpy, module_ident)
    return


Instruction:
Write a Python 3 script for
transforming an abstract using bi - directionally.

Response:
def _transform_abstract(plpy, module_ident):
    """Transform abstract, bi-directionally.

    Transforms an abstract using one of content columns
    ('abstract' or 'html') to determine which direction the transform
    will go (cnxml->html or html->cnxml).
    A transform is done on either one of them to make
    the other value. If no value is supplied, the trigger raises an error.
    If both values are supplied, the trigger will skip.
    """
    plan = plpy.prepare("""\
SELECT a.abstractid, a.abstract, a.html
FROM modules AS m NATURAL JOIN abstracts AS a
WHERE m.module_ident = $1""", ('integer',))
    result = plpy.execute(plan, (module_ident,), 1)[0]
    abstractid, cnxml, html = (
        result['abstractid'], result['abstract'], result['html'])
    if cnxml is not None and html is not None:
        return  # skip
    # TODO Prevent blank abstracts (abstract = null & html = null).

    msg = "produce {}->{} for abstractid={}"
    if cnxml is None:
        # Transform html->cnxml
        msg = msg.format('html', 'cnxml', abstractid)
        content = html
        column = 'abstract'
        transform_func = transform_abstract_to_cnxml
    else:
        # Transform cnxml->html
        msg = msg.format('cnxml', 'html', abstractid)
        content = cnxml
        column = 'html'
        transform_func = transform_abstract_to_html

    content, messages = transform_func(content, module_ident, plpy)
    plan = plpy.prepare(
        "UPDATE abstracts SET {} = $1 WHERE abstractid = $2".format(column),
        ('text', 'integer'))
    plpy.execute(plan, (content, abstractid,))
    return msg


Instruction:
Write a Python 3 function that can
build and retrieve json tree representation of a book.

Response:
def get_collection_tree(collection_ident, cursor):
    """Build and retrieve json tree representation of a book."""
    cursor.execute('''
    WITH RECURSIVE t(node, parent, document, path) AS (
        SELECT tr.nodeid, tr.parent_id, tr.documentid, ARRAY[tr.nodeid]
        FROM trees tr
        WHERE tr.documentid = %s and tr.is_collated = 'False'
    UNION ALL
        SELECT c.nodeid, c.parent_id, c.documentid, path || ARRAY[c.nodeid]
        FROM trees c JOIN t ON c.parent_id = t.node
        WHERE NOT c.nodeid = ANY(t.path)
    )
    SELECT t.document, m.portal_type
    FROM t JOIN modules m
    ON t.document = m.module_ident''', [collection_ident])
    for i in cursor.fetchall():
        yield i


Instruction:
Can you generate the documentation for the following Python 3 function
def get_module_can_publish(cursor, id):
    cursor.execute("""
SELECT DISTINCT user_id
FROM document_acl
WHERE uuid = %s AND permission = 'publish'""", (id,))
    return [i[0] for i in cursor.fetchall()]

Response: Return userids allowed to publish this book.


Instruction:
Explain what the following Python 3 code does
def tree_to_nodes(tree, context=None, metadata=None):
    nodes = []
    for item in tree['contents']:
        if 'contents' in item:
            sub_nodes = tree_to_nodes(item, context=context, metadata=metadata)
            if metadata is None:
                metadata = {}
            else:
                metadata = metadata.copy()
                for key in ('title', 'id', 'shortid',
                            'cnx-archive-uri', 'cnx-archive-shortid'):
                    if key in metadata:
                        metadata.pop(key)

            for key in ('title', 'id', 'shortId'):
                if item.get(key):
                    metadata[key] = item[key]
                    if item[key] != 'subcol':
                        if key == 'id':
                            metadata['cnx-archive-uri'] = item[key]
                        elif key == 'shortId':
                            metadata['cnx-archive-shortid'] = item[key]

            titles = _title_overrides_from_tree(item)
            if item.get('id') is not None:
                tbinder = cnxepub.Binder(item.get('id'),
                                         sub_nodes,
                                         metadata=metadata,
                                         title_overrides=titles)
            else:
                tbinder = cnxepub.TranslucentBinder(sub_nodes,
                                                    metadata=metadata,
                                                    title_overrides=titles)
            nodes.append(tbinder)
        else:
            doc = document_factory(item['id'], context=context)
            for key in ('title', 'id', 'shortId'):
                if item.get(key):
                    doc.metadata[key] = item[key]
                    if key == 'id':
                        doc.metadata['cnx-archive-uri'] = item[key]
                    elif key == 'shortId':
                        doc.metadata['cnx-archive-shortid'] = item[key]
            nodes.append(doc)
    return nodes

Response: Assembles tree nodes into object models.


Instruction:
How would you explain what the following Python 3 function does
def create_epub(ident_hash, file, format='raw'):
    model = factory(ident_hash, baked=(format != 'raw'))
    if isinstance(model, cnxepub.Document):
        model = cnxepub.TranslucentBinder(nodes=[model])
    cnxepub.make_epub(model, file)

Response: Creates an epub from an ident_hash which is output to the given file - like object. Returns None writes to the given file.


Instruction:
Make a summary of the following Python 3 code
def get_export(request):
    settings = get_current_registry().settings
    exports_dirs = settings['exports-directories'].split()
    args = request.matchdict
    ident_hash, type = args['ident_hash'], args['type']
    id, version = split_ident_hash(ident_hash)

    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            try:
                results = get_export_files(cursor, id, version, [type],
                                           exports_dirs, read_file=True)
                if not results:
                    raise httpexceptions.HTTPNotFound()
                filename, mimetype, size, modtime, state, file_content \
                    = results[0]
            except ExportError as e:
                logger.debug(str(e))
                raise httpexceptions.HTTPNotFound()

    if state == 'missing':
        raise httpexceptions.HTTPNotFound()

    encoded_filename = urllib.quote(filename.encode('utf-8'))
    resp = request.response
    resp.status = "200 OK"
    resp.content_type = mimetype
    #  Need both filename and filename* below for various browsers
    #  See: https://fastmail.blog/2011/06/24/download-non-english-filenames/
    resp.content_disposition = "attachment; filename={fname};" \
                               " filename*=UTF-8''{fname}".format(
                                       fname=encoded_filename)
    resp.body = file_content
    #  Remove version and extension from filename, to recover title slug
    slug_title = '-'.join(encoded_filename.split('-')[:-1])
    resp.headerlist.append(
            ('Link', '<https://{}/contents/{}/{}> ;rel="Canonical"'.format(
                           request.host, id, slug_title)))
    return resp

Response: Retrieve an export file.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_export_files(cursor, id, version, types, exports_dirs, read_file=True):
    request = get_current_request()
    type_info = dict(request.registry.settings['_type_info'])
    metadata = get_content_metadata(id, version, cursor)
    legacy_id = metadata['legacy_id']
    legacy_version = metadata['legacy_version']

    reachable_dirs = [dir for dir in exports_dirs if safe_stat(dir)]

    # 1 result per type, in the same order as the given types
    results = []

    for type in list(types):
        if type not in type_info:
            raise ExportError("invalid type '{}' requested.".format(type))
        file_extension = type_info[type]['file_extension']

        # skip module PDFs
        if metadata['mediaType'] == MODULE_MIMETYPE and \
                file_extension == 'pdf':
            continue

        mimetype = type_info[type]['mimetype']
        filename = '{}@{}.{}'.format(id, version, file_extension)
        legacy_filenames = [
            '{}-{}.{}'.format(legacy_id, legacy_version, ext)
            for ext in LEGACY_EXTENSION_MAP[file_extension]
        ]
        slugify_title_filename = u'{}-{}.{}'.format(slugify(metadata['title']),
                                                    version, file_extension)

        for dir in reachable_dirs:
            filepath = os.path.join(dir, filename)
            try:
                if read_file:
                    with open(filepath, 'r') as file:
                        stats = os.fstat(file.fileno())
                        contents = file.read()
                else:
                    stats = os.stat(filepath)
                    contents = None
                modtime = fromtimestamp(int(stats.st_mtime))
                results.append((slugify_title_filename, mimetype,
                                stats.st_size, modtime, 'good', contents))
                break
            except EnvironmentError:
                pass
        else:
            # Let's see if the legacy file's there and make the new link
            legacy_file_found = False
            for dir in reachable_dirs:
                filepath = os.path.join(dir, filename)
                legacy_filepaths = [os.path.join(dir, fn)
                                    for fn in legacy_filenames]
                for legacy_filepath in legacy_filepaths:
                    try:
                        if read_file:
                            with open(legacy_filepath, 'r') as file:
                                stats = os.fstat(file.fileno())
                                contents = file.read()
                        else:
                            stats = os.stat(legacy_filepath)
                            contents = None
                        modtime = fromtimestamp(stats.st_mtime)
                        os.link(legacy_filepath, filepath)
                        results.append((slugify_title_filename, mimetype,
                                        stats.st_size, modtime, 'good',
                                        contents))
                        legacy_file_found = True
                        break
                    except EnvironmentError:
                        pass
                if legacy_file_found:
                    break
            else:
                filenames = [filename] + legacy_filenames
                log_formatted_filenames = '\n'.join([' - {}'.format(x)
                                                     for x in filenames])
                logger.error("Could not find a file for '{}' at version '{}' "
                             "with any of the following file names:\n{}"
                             .format(id, version, log_formatted_filenames))
                # No file, return "missing" state
                results.append((slugify_title_filename, mimetype,
                                0, None, 'missing', None))

    return results

Response: Retrieve files associated with document.


Instruction:
Create a Python 3 function to
return metadata related to the content from the database.

Response:
def get_content_metadata(id, version, cursor):
    """Return metadata related to the content from the database."""
    # Do the module lookup
    args = dict(id=id, version=version)
    # FIXME We are doing two queries here that can hopefully be
    #       condensed into one.
    cursor.execute(SQL['get-module-metadata'], args)
    try:
        result = cursor.fetchone()[0]
        # version is what we want to return, but in the sql we're using
        # current_version because otherwise there's a "column reference is
        # ambiguous" error
        result['version'] = result.pop('current_version')

        # FIXME We currently have legacy 'portal_type' names in the database.
        #       Future upgrades should replace the portal type with a mimetype
        #       of 'application/vnd.org.cnx.(module|collection|folder|<etc>)'.
        #       Until then we will do the replacement here.
        result['mediaType'] = portaltype_to_mimetype(result['mediaType'])

        return result
    except (TypeError, IndexError,):  # None returned
        raise httpexceptions.HTTPNotFound()


Instruction:
Can you generate the documentation for the following Python 3 function
def find_contours_level(density, x, y, level, closed=False):
    if level >= 1 or level <= 0:
        raise ValueError("`level` must be in (0,1), got '{}'!".format(level))
    # level relative to maximum
    level = level * density.max()
    # xy coordinates
    if len(x.shape) == 2:
        assert np.all(x[:, 0] == x[:, 1])
        x = x[:, 0]
    if len(y.shape) == 2:
        assert np.all(y[0, :] == y[1, :])
        y = y[0, :]
    if closed:
        # find closed contours
        density = np.pad(density, ((1, 1), (1, 1)), mode="constant")
        offset = 1
    else:
        # leave contours open at kde boundary
        offset = 0

    conts_idx = find_contours(density, level)
    conts_xy = []

    for cc in conts_idx:
        cx = np.interp(x=cc[:, 0]-offset,
                       xp=range(x.size),
                       fp=x)
        cy = np.interp(x=cc[:, 1]-offset,
                       xp=range(y.size),
                       fp=y)
        conts_xy.append(np.stack((cx, cy), axis=1))

    return conts_xy

Response: Find contours for a given level value in a given density estimate.


Instruction:
Write a Python 3 script for
finding the density level for a given data quantile by iteration.

Response:
def _find_quantile_level(density, x, y, xp, yp, quantile, acc=.01,
                         ret_err=False):
    """Find density level for a given data quantile by iteration

    Parameters
    ----------
    density: 2d ndarray of shape (M, N)
        Kernel density estimate for which to compute the contours
    x: 2d ndarray of shape (M, N) or 1d ndarray of size M
        X-values corresponding to `kde`
    y: 2d ndarray of shape (M, N) or 1d ndarray of size M
        Y-values corresponding to `kde`
    xp: 1d ndarray of size D
        Event x-data from which to compute the quantile
    yp: 1d ndarray of size D
        Event y-data from which to compute the quantile
    quantile: float between 0 and 1
        Quantile along which to find contours in `kde` relative
        to its maximum
    acc: float
        Desired absolute accuracy (stopping criterion) of the
        contours
    ret_err: bool
        If True, also return the absolute error

    Returns
    -------
    level: float
        Contours level corresponding to the given quantile

    Notes
    -----
    A much more faster method (using interpolation) is implemented in
    :func:`get_quantile_levels`.
    NaN-values events in `xp` and `yp` are ignored.

    See Also
    --------
    skimage.measure.find_contours: Contour finding algorithm
    """
    if quantile >= 1 or quantile <= 0:
        raise ValueError("Invalid value for `quantile`: {}".format(quantile))

    # remove bad events
    bad = get_bad_vals(xp, yp)
    xp = xp[~bad]
    yp = yp[~bad]

    # initial guess
    level = quantile
    # error of current iteration
    err = 1
    # iteration factor (guarantees convergence)
    itfac = 1
    # total number of events
    nev = xp.size

    while np.abs(err) > acc:
        # compute contours
        conts = find_contours_level(density, x, y, level, closed=True)
        # compute number of points in contour
        isin = 0
        for ii in range(nev):
            for cc in conts:
                isin += PolygonFilter.point_in_poly((xp[ii], yp[ii]),
                                                    poly=cc)
                break  # no need to check other contours
        err = quantile - (nev - isin) / nev
        level += err * itfac
        itfac *= .9

    if ret_err:
        return level, err
    else:
        return level


Instruction:
Can you generate the documentation for the following Python 3 function
def write(path_or_h5file, data={}, meta={}, logs={}, mode="reset",
          compression=None):
    """Write data to an RT-DC file

    Parameters
    ----------
    path: path or h5py.File
        The path or the hdf5 file object to write to.
    data: dict-like
        The data to store. Each key of `data` must be a valid
        feature name (see `dclab.dfn.feature_names`). The data
        type must be given according to the feature type:

        - scalar feature: 1d ndarray of size `N`, any dtype,
          with the number of events `N`.
        - contour: list of `N` 2d ndarrays of shape `(2,C)`, any dtype,
          with each ndarray containing the x- and y- coordinates
          of `C` contour points in pixels.
        - image: 3d ndarray of shape `(N,A,B)`, is converted to uint8,
          with the image dimensions `(x,y) = (A,B)`
        - mask: 3d ndarray of shape `(N,A,B)`, is converted to bool,
          with the image dimensions `(x,y) = (A,B)`
        - trace: 2d ndarray of shape `(N,T)`, any dtype
          with a globally constant trace length `T`.
    meta: dict of dicts
        The meta data to store (see `dclab.dfn.config_keys`).
        Each key depicts a meta data section name whose data is given
        as a dictionary, e.g.

            meta = {"imaging": {"exposure time": 20,
                                "flash duration": 2,
                                ...
                                },
                    "setup": {"channel width": 20,
                              "chip region": "channel",
                              ...
                              },
                    ...
                    }

        Only section key names and key values therein registered
        in dclab are allowed and are converted to the pre-defined
        dtype.
    logs: dict of lists
        Each key of `logs` refers to a list of strings that contains
        logging information. Each item in the list can be considered to
        be one line in the log file.
    mode: str
        Defines how the input `data` and `logs` are stored:
        - "append": append new data to existing Datasets; the opened
                    `h5py.File` object is returned (used in real-
                    time data storage)
        - "replace": replace keys given by `data` and `logs`; the
                    opened `h5py.File` object is closed and `None`
                    is returned (used for ancillary feature storage)
        - "reset": do not keep any previous data; the opened
                   `h5py.File` object is closed and `None` is returned
                   (default)
    compression: str
        Compression method for "contour", "image", and "trace" data
        as well as logs; one of [None, "lzf", "gzip", "szip"].

    Notes
    -----
    If `data` is an instance of RTDCBase, then `meta` must be set to
    `data.config`, otherwise no meta data will be saved.
    """
    if mode not in ["append", "replace", "reset"]:
        raise ValueError("`mode` must be one of [append, replace, reset]")
    if compression not in [None, "gzip", "lzf", "szip"]:
        raise ValueError("`compression` must be one of [gzip, lzf, szip]")

    if (not hasattr(data, "__iter__") or
        not hasattr(data, "__contains__") or
            not hasattr(data, "__getitem__") or
            isinstance(data, (list, np.ndarray))):
        msg = "`data` must be dict-like"
        raise ValueError(msg)

    # Check meta data
    for sec in meta:
        if sec not in dfn.config_keys:
            # only allow writing of meta data that are not editable
            # by the user (not dclab.dfn.CFG_ANALYSIS)
            msg = "Meta data section not defined in dclab: {}".format(sec)
            raise ValueError(msg)
        for ck in meta[sec]:
            if ck not in dfn.config_keys[sec]:
                msg = "Meta key not defined in dclab: {}:{}".format(sec, ck)
                raise ValueError(msg)

    # Check feature keys
    feat_keys = []
    for kk in data:
        if kk in dfn.feature_names:
            feat_keys.append(kk)
        else:
            raise ValueError("Unknown key '{}'!".format(kk))
        # verify trace names
        if kk == "trace":
            for sk in data["trace"]:
                if sk not in dfn.FLUOR_TRACES:
                    msg = "Unknown trace key: {}".format(sk)
                    raise ValueError(msg)

    # Create file
    # (this should happen after all checks)
    if isinstance(path_or_h5file, h5py.File):
        h5obj = path_or_h5file
    else:
        if mode == "reset":
            h5mode = "w"
        else:
            h5mode = "a"
        h5obj = h5py.File(path_or_h5file, mode=h5mode)

    # update version
    # - if it is not already in the hdf5 file (prevent override)
    # - if it is explicitly given in meta (append to old version string)
    if ("setup:software version" not in h5obj.attrs
            or ("setup" in meta and "software version" in meta["setup"])):
        thisver = "dclab {}".format(version)
        if "setup" in meta and "software version" in meta["setup"]:
            oldver = meta["setup"]["software version"]
            thisver = "{} | {}".format(oldver, thisver)
        if "setup" not in meta:
            meta["setup"] = {}
        meta["setup"]["software version"] = thisver
    # Write meta
    for sec in meta:
        for ck in meta[sec]:
            idk = "{}:{}".format(sec, ck)
            conffunc = dfn.config_funcs[sec][ck]
            h5obj.attrs[idk] = conffunc(meta[sec][ck])

    # Write data
    # create events group
    if "events" not in h5obj:
        h5obj.create_group("events")
    events = h5obj["events"]
    # remove previous data
    if mode == "replace":
        for rk in feat_keys:
            if rk in events:
                del events[rk]
    # store experimental data
    for fk in feat_keys:
        if fk in dfn.scalar_feature_names:
            store_scalar(h5group=events,
                         name=fk,
                         data=data[fk])
        elif fk == "contour":
            store_contour(h5group=events,
                          data=data["contour"],
                          compression=compression)
        elif fk == "image":
            store_image(h5group=events,
                        data=data["image"],
                        compression=compression)
        elif fk == "mask":
            store_mask(h5group=events,
                       data=data["mask"],
                       compression=compression)
        elif fk == "trace":
            store_trace(h5group=events,
                        data=data["trace"],
                        compression=compression)

    # Write logs
    if "logs" not in h5obj:
        h5obj.create_group("logs")
    log_group = h5obj["logs"]
    # remove previous data
    if mode == "replace":
        for rl in logs:
            if rl in log_group:
                del log_group[rl]
    dt = h5py.special_dtype(vlen=hdf5_str)
    for lkey in logs:
        ldata = logs[lkey]
        if isinstance(ldata, (str, hdf5_str)):
            # single event
            ldata = [ldata]
        lnum = len(ldata)
        if lkey not in log_group:
            log_dset = log_group.create_dataset(lkey,
                                                (lnum,),
                                                dtype=dt,
                                                maxshape=(None,),
                                                chunks=True,
                                                fletcher32=True,
                                                compression=compression)
            for ii, line in enumerate(ldata):
                log_dset[ii] = line
        else:
            log_dset = log_group[lkey]
            oldsize = log_dset.shape[0]
            log_dset.resize(oldsize + lnum, axis=0)
            for ii, line in enumerate(ldata):
                log_dset[oldsize + ii] = line

    if mode == "append":
        return h5obj
    else:
        h5obj.close()
        return None

Response: Writes data to an RT - DC file.


Instruction:
How would you code a function in Python 3 to
convert a tuple into a list of tuples.

Response:
def _convert(tup, dictlist):
    """
    :param tup: a list of tuples
    :param di: a dictionary converted from tup
    :return: dictionary
    """
    di = {}
    for a, b in tup:
        di.setdefault(a, []).append(b)
    for key, val in di.items():
        dictlist.append((key, val))
    return dictlist


Instruction:
Implement a function in Python 3 to
return a list of upper - cased strings

Response:
def _upper(val_list):
    """
    :param val_list: a list of strings
    :return: a list of upper-cased strings
    """
    res = []
    for ele in val_list:
        res.append(ele.upper())
    return res


Instruction:
Make a summary of the following Python 3 code
def _build_search(structured_query):
    arguments = {}

    # get text terms and filter out common words
    text_terms = [term for ttype, term in structured_query.terms
                  if ttype == 'text']

    text_terms_wo_stopwords = [term for term in text_terms
                               if term.lower() not in STOPWORDS]
    # sql where clauses
    conditions = {'text_terms': '', 'pubYear': '', 'authorID': '', 'type': '',
                  'keyword': '', 'subject': '', 'language': '', 'title': '',
                  'author': '', 'abstract': ''}

    # if there are other search terms (not type "text") or if the text
    # terms do not only consist of stopwords, then use the text terms
    # without stopwords
    if arguments or text_terms_wo_stopwords:
        text_terms = text_terms_wo_stopwords
    arguments.update({'text_terms': ' '.join(text_terms)})
    # raise Exception(structured_query.filters, structured_query.terms)

    if len(text_terms) > 0:
        conditions['text_terms'] = 'AND module_idx \
                                    @@ plainto_tsquery(%(text_terms)s)'

    # build fulltext keys
    fulltext_key = []
    for term in text_terms_wo_stopwords:
        fulltext_key.append(term + '-::-fulltext')
    arguments.update({'fulltext_key':  ';--;'.join(fulltext_key)})

    idx = 0
    invalid_filters = []
    filters = _convert(structured_query.filters, [])

    while idx < len(filters):
        keyword = filters[idx][0]
        value = filters[idx][1]
        if keyword == 'pubYear':
            conditions['pubYear'] = 'AND extract(year from cm.revised) = \
                                     %(pubYear)s'
            arguments.update({'pubYear': value[0]})
        elif keyword == 'authorID':
            conditions['authorID'] = 'AND ARRAY[%(authorID)s] \
                                      <@ cm.authors'
            arguments.update({'authorID': value[0]})
        elif keyword == 'type':
            value[0] = value[0].lower()

            conditions['type'] = 'AND cm.portal_type = %(type)s'
            if value[0] != 'book' and value[0] != 'collection' and \
                    value[0] != 'page' and value[0] != 'module':
                invalid_filters.append(idx)
            value[0] = 'Collection' if (value[0] == 'book' or
                                        value[0] == 'collection') else 'Module'
            arguments.update({'type': value[0]})
        elif keyword == 'keyword':
            value = _upper(value)
            conditions['keyword'] = 'AND cm.module_ident = \
                                     ANY(WITH target AS ( \
                                     SELECT lm.module_ident AS id, \
                                     array_agg(UPPER(kw.word)) AS akw \
                                     FROM latest_modules lm, \
                                     modulekeywords mk, \
                                     keywords kw \
                                     WHERE lm.module_ident = mk.module_ident \
                                     AND kw.keywordid = mk.keywordid \
                                     GROUP BY id) \
                                     SELECT target.id FROM target WHERE \
                                     target.akw @> %(keyword)s)'
            arguments.update({'keyword': value})
        elif keyword == 'subject':
            conditions['subject'] = 'AND cm.module_ident = \
                                     ANY(WITH sub AS ( \
                                     SELECT module_ident AS id, \
                                     array_agg(tag) AS atag \
                                     FROM latest_modules \
                                     NATURAL JOIN \
                                     moduletags NATURAL JOIN \
                                     tags GROUP BY id) \
                                     SELECT sub.id FROM sub WHERE \
                                     sub.atag @> %(subject)s)'
            arguments.update({'subject': value})
        elif keyword == 'language':
            conditions['language'] = 'AND cm.language = %(language)s'
            arguments.update({'language': value[0]})
        elif keyword == 'title':
            conditions['title'] = 'AND strip_html(cm.name) ~* \
                                   %(title)s'
            arguments.update({'title': value[0]})
        elif keyword == 'author':
            conditions['author'] = 'AND cm.module_ident = \
                                    ANY(WITH name AS ( \
                                    SELECT username FROM users u WHERE \
                                    u.first_name||\' \'||u.last_name \
                                    ~* %(author)s) \
                                    SELECT lm.module_ident \
                                    FROM latest_modules lm \
                                    JOIN name n ON ARRAY[n.username] \
                                    <@ lm.authors)'
            arguments.update({'author': value[0]})
        elif keyword == 'abstract':
            conditions['abstract'] = 'AND cm.module_ident = \
                                      ANY(SELECT lm.module_ident FROM \
                                      latest_modules lm, \
                                      abstracts ab WHERE\
                                      lm.abstractid = ab.abstractid \
                                      AND ab.abstract \
                                      ~* %(abstract)s)'
            arguments.update({'abstract': value[0]})
        else:
            # Invalid filter!
            invalid_filters.append(idx)
        idx += 1

    if len(invalid_filters) == len(structured_query.filters) and \
            len(structured_query.terms) == 0:
        # Either query terms are all invalid filters
        # or we received a null query.
        # Clear the filter list in this case.
        structured_query.filters = []
        return None, None

    for invalid_filter_idx in invalid_filters:
        # Remove invalid filters.
        del structured_query.filters[invalid_filter_idx]

    # Add the arguments for sorting.
    sorts = ['portal_type']
    if structured_query.sorts:
        for sort in structured_query.sorts:
            # These sort values are not the name of the column used
            #   in the database.
            stmt = _transmute_sort(sort)
            sorts.append(stmt)
    sorts.extend(('weight DESC', 'uuid DESC',))
    sorts = ', '.join(sorts)

    statement = SQL_QUICK_SELECT_WRAPPER.format(conditions['pubYear'],
                                                conditions['authorID'],
                                                conditions['type'],
                                                conditions['keyword'],
                                                conditions['subject'],
                                                conditions['text_terms'],
                                                conditions['language'],
                                                conditions['title'],
                                                conditions['author'],
                                                conditions['abstract'],
                                                sorts=sorts)
    return statement, arguments

Response: Construct the search statment for the DBAPI v2 execute method.


Instruction:
Implement a Python 3 function for
searching database using parsed query.

Response:
def search(query, query_type=DEFAULT_QUERY_TYPE):
    """Search database using parsed query.

    Executes a database search query from the given ``query``
    (a ``Query`` object) and optionally accepts a list of search weights.
    By default, the search results are ordered by weight.

    :param query: containing terms, filters, and sorts.
    :type query: Query
    :returns: a sequence of records that match the query conditions
    :rtype: QueryResults (which is a sequence of QueryRecord objects)
    """

    # Build the SQL statement.
    statement, arguments = _build_search(query)

    # Execute the SQL.
    if statement is None and arguments is None:
        return QueryResults([], [], 'AND')
    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            cursor.execute(statement, arguments)
            search_results = cursor.fetchall()
    # Wrap the SQL results.
    return QueryResults(search_results, query, query_type)


Instruction:
Implement a function in Python 3 to
parse a raw string to query. Given a raw string to query. Parse to a structured format and initialize the class.

Response:
def from_raw_query(cls, query_string):
        """Parse raw string to query.

        Given a raw string (typically typed by the user),
        parse to a structured format and initialize the class.
        """
        try:
            node_tree = grammar.parse(query_string)
        except IncompleteParseError:
            query_string = cls.fix_quotes(query_string)
            node_tree = grammar.parse(query_string)

        structured_query = DictFormater().visit(node_tree)

        return cls([t for t in structured_query
                    if t[1].lower() not in STOPWORDS])


Instruction:
Explain what the following Python 3 code does
def highlighted_abstract(self):
        abstract_terms = self.fields.get('abstract', [])
        if abstract_terms:
            sql = _read_sql_file('highlighted-abstract')
        else:
            sql = _read_sql_file('get-abstract')
        arguments = {'id': self['id'],
                     'query': ' & '.join(abstract_terms),
                     }
        with db_connect() as db_connection:
            with db_connection.cursor() as cursor:
                cursor.execute(sql, arguments)
                hl_abstract = cursor.fetchone()
        if hl_abstract:
            return hl_abstract[0]

Response: Highlight the found terms in the abstract text.


Instruction:
Can you implement a function in Python 3 that
highlights the found terms in the fulltext field.

Response:
def highlighted_fulltext(self):
        """Highlight the found terms in the fulltext."""
        terms = self.fields.get('fulltext', [])
        if not terms:
            return None
        arguments = {'id': self['id'],
                     'query': ' & '.join(terms),
                     }
        with db_connect() as db_connection:
            with db_connection.cursor() as cursor:
                cursor.execute(_read_sql_file('highlighted-fulltext'),
                               arguments)
                hl_fulltext = cursor.fetchone()[0]
        return hl_fulltext


Instruction:
Can you generate a brief explanation for the following Python 3 code
def pdf(self, data_predict=None):
        if data_predict is None:
            data_predict = self.data
        else:
            data_predict = _adjust_shape(data_predict, self.k_vars)

        pdf_est = []
        for i in range(np.shape(data_predict)[0]):
            pdf_est.append(gpke(self.bw, data=self.data,
                                data_predict=data_predict[i, :],
                                var_type=self.var_type) / self.nobs)

        pdf_est = np.squeeze(pdf_est)
        return pdf_est

Response: r Evaluate the probability density function at data_predict.


Instruction:
Implement a function in Python 3 to
find the Xenon - GRPC jar - file windows version.

Response:
def find_xenon_grpc_jar():
    """Find the Xenon-GRPC jar-file, windows version."""
    prefix = Path(sys.prefix)

    locations = [
        prefix / 'lib',
        prefix / 'local' / 'lib'
    ]

    for location in locations:
        jar_file = location / 'xenon-grpc-{}-all.jar'.format(
                xenon_grpc_version)

        if not jar_file.exists():
            continue
        else:
            return str(jar_file)

    return None


Instruction:
How would you explain what the following Python 3 function does
def kill_process(process):
    logger = logging.getLogger('xenon')
    logger.info('Terminating Xenon-GRPC server.')
    os.kill(process.pid, signal.SIGINT)
    process.wait()

Response: Kill the process group associated with the given process.


Instruction:
Can you generate the documentation for the following Python 3 function
def start_xenon_server(port=50051, disable_tls=False):
    jar_file = find_xenon_grpc_jar()
    if not jar_file:
        raise RuntimeError("Could not find 'xenon-grpc' jar file.")

    cmd = ['java', '-jar', jar_file, '-p', str(port)]

    if not disable_tls:
        crt_file, key_file = create_self_signed_cert()

        cmd.extend([
            '--server-cert-chain', str(crt_file),
            '--server-private-key', str(key_file),
            '--client-cert-chain', str(crt_file)])
    else:
        crt_file = key_file = None

    process = subprocess.Popen(
        cmd,
        bufsize=1,
        universal_newlines=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)
    return process, crt_file, key_file

Response: Start the xenon - grpc server.


Instruction:
Given the following Python 3 function, write the documentation
def send_keystrokes(ip, keystrokes, wait=False):
        tv_url = 'http://{}:6095/controller?action=keyevent&keycode='.format(ip)

        for keystroke in keystrokes:
            if keystroke == 'wait' or wait is True:

                time.sleep(0.7)
            else:
                request = requests.get(tv_url + keystroke)

                if request.status_code != 200:
                    return False

        return True

Response: Connects to TV and sends keystrokes via HTTP.


Instruction:
Can you generate the documentation for the following Python 3 function
def in_book_search(request):
    results = {}

    args = request.matchdict
    ident_hash = args['ident_hash']

    args['search_term'] = request.params.get('q', '')
    query_type = request.params.get('query_type', '')
    combiner = ''
    if query_type:
        if query_type.lower() == 'or':
            combiner = '_or'

    id, version = split_ident_hash(ident_hash)
    args['uuid'] = id
    args['version'] = version

    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            cursor.execute(SQL['get-collated-state'], args)
            res = cursor.fetchall()
            if res and res[0][0]:
                statement = SQL['get-in-collated-book-search']
            else:
                statement = SQL['get-in-book-search']
            cursor.execute(statement.format(combiner=combiner), args)
            res = cursor.fetchall()

            results['results'] = {'query': [],
                                  'total': len(res),
                                  'items': []}
            results['results']['query'] = {
                'id': ident_hash,
                'search_term': args['search_term'],
            }
            for uuid, version, title, snippet, matches, rank in res:
                results['results']['items'].append({
                    'rank': '{}'.format(rank),
                    'id': '{}@{}'.format(uuid, version),
                    'title': '{}'.format(title),
                    'snippet': '{}'.format(snippet),
                    'matches': '{}'.format(matches),
                })

    resp = request.response
    resp.status = '200 OK'
    resp.content_type = 'application/json'
    resp.body = json.dumps(results)

    return resp

Response: Full text in - book search.


Instruction:
How would you code a function in Python 3 to
return an array of shape nobs k_vars for use with GPG.

Response:
def _adjust_shape(dat, k_vars):
    """ Returns an array of shape (nobs, k_vars) for use with `gpke`."""
    dat = np.asarray(dat)
    if dat.ndim > 2:
        dat = np.squeeze(dat)
    if dat.ndim == 1 and k_vars > 1:  # one obs many vars
        nobs = 1
    elif dat.ndim == 1 and k_vars == 1:  # one obs one var
        nobs = len(dat)
    else:
        if np.shape(dat)[0] == k_vars and np.shape(dat)[1] != k_vars:
            dat = dat.T

        nobs = np.shape(dat)[0]  # ndim >1 so many obs many vars

    dat = np.reshape(dat, (nobs, k_vars))
    return dat


Instruction:
Given the following Python 3 function, write the documentation
def gpke(bw, data, data_predict, var_type, ckertype='gaussian',
         okertype='wangryzin', ukertype='aitchisonaitken', tosum=True):
    r"""
    Returns the non-normalized Generalized Product Kernel Estimator

    Parameters
    ----------
    bw: 1-D ndarray
        The user-specified bandwidth parameters.
    data: 1D or 2-D ndarray
        The training data.
    data_predict: 1-D ndarray
        The evaluation points at which the kernel estimation is performed.
    var_type: str, optional
        The variable type (continuous, ordered, unordered).
    ckertype: str, optional
        The kernel used for the continuous variables.
    okertype: str, optional
        The kernel used for the ordered discrete variables.
    ukertype: str, optional
        The kernel used for the unordered discrete variables.
    tosum : bool, optional
        Whether or not to sum the calculated array of densities.  Default is
        True.

    Returns
    -------
    dens: array-like
        The generalized product kernel density estimator.

    Notes
    -----
    The formula for the multivariate kernel estimator for the pdf is:

    .. math:: f(x)=\frac{1}{nh_{1}...h_{q}}\sum_{i=1}^
                        {n}K\left(\frac{X_{i}-x}{h}\right)

    where

    .. math:: K\left(\frac{X_{i}-x}{h}\right) =
                k\left( \frac{X_{i1}-x_{1}}{h_{1}}\right)\times
                k\left( \frac{X_{i2}-x_{2}}{h_{2}}\right)\times...\times
                k\left(\frac{X_{iq}-x_{q}}{h_{q}}\right)
    """
    kertypes = dict(c=ckertype, o=okertype, u=ukertype)

    Kval = np.empty(data.shape)
    for ii, vtype in enumerate(var_type):
        func = kernel_func[kertypes[vtype]]
        Kval[:, ii] = func(bw[ii], data[:, ii], data_predict[ii])

    iscontinuous = np.array([c == 'c' for c in var_type])
    dens = Kval.prod(axis=1) / np.prod(bw[iscontinuous])
    if tosum:
        return dens.sum(axis=0)
    else:
        return dens

Response: r Returns the non - normalized Generalized Product Kernel Estimator for the pdf.


Instruction:
Write a Python 3 script for
computing the bandwidth of the data.

Response:
def _compute_bw(self, bw):
        """
        Computes the bandwidth of the data.

        Parameters
        ----------
        bw: array_like or str
            If array_like: user-specified bandwidth.
            If a string, should be one of:

                - cv_ml: cross validation maximum likelihood
                - normal_reference: normal reference rule of thumb
                - cv_ls: cross validation least squares

        Notes
        -----
        The default values for bw is 'normal_reference'.
        """
        if bw is None:
            bw = 'normal_reference'

        if not isinstance(bw, string_types):
            self._bw_method = "user-specified"
            res = np.asarray(bw)
        else:
            # The user specified a bandwidth selection method
            self._bw_method = bw
            # Workaround to avoid instance methods in __dict__
            if bw == 'normal_reference':
                bwfunc = self._normal_reference
            elif bw == 'cv_ml':
                bwfunc = self._cv_ml
            else:  # bw == 'cv_ls'
                bwfunc = self._cv_ls
            res = bwfunc()

        return res


Instruction:
Can you generate the documentation for the following Python 3 function
def _set_defaults(self, defaults):
        self.n_res = defaults.n_res
        self.n_sub = defaults.n_sub
        self.randomize = defaults.randomize
        self.return_median = defaults.return_median
        self.efficient = defaults.efficient
        self.return_only_bw = defaults.return_only_bw
        self.n_jobs = defaults.n_jobs

Response: Sets the default values for the efficient estimation


Instruction:
Can you write a function in Python 3 where it
returns package version from setup. cfg

Response:
def get_version():
    """Return package version from setup.cfg"""
    config = RawConfigParser()
    config.read(os.path.join('..', 'setup.cfg'))
    return config.get('metadata', 'version')


Instruction:
Can you tell what is the following Python 3 function doing
def to_string(self):
        root = etree.Element('sitemapindex', nsmap={None: SITEMAP_NS})
        for sitemap in self.sitemaps:
            sm = etree.SubElement(root, 'sitemap')
            etree.SubElement(sm, 'loc').text = sitemap.url
            if hasattr(sitemap.lastmod, 'strftime'):
                etree.SubElement(sm, 'lastmod').text = \
                    sitemap.lastmod.strftime('%Y-%m-%d')
            elif isinstance(sitemap.lastmod, str):
                etree.SubElement(sm, 'lastmod').text = sitemap.lastmod
        return etree.tostring(root, pretty_print=True, xml_declaration=True,
                              encoding='utf-8')

Response: Convert SitemapIndex into a string.


Instruction:
Explain what the following Python 3 code does
def add_url(self, *args, **kwargs):
        if len(args) == 1 and not kwargs and isinstance(args[0], UrlEntry):
            self.urls.append(args[0])
        else:
            self.urls.append(UrlEntry(*args, **kwargs))

Response: Add a new url to the sitemap.


Instruction:
Make a summary of the following Python 3 code
def to_string(self):
        root = etree.Element('urlset', nsmap={None: SITEMAP_NS})
        for url in self.urls:
            url.generate(root)
        return etree.tostring(root, pretty_print=True, xml_declaration=True,
                              encoding='utf-8')

Response: Convert the sitemap into a string.


Instruction:
How would you explain what the following Python 3 function does
def generate(self, root_element=None):
        if root_element is not None:
            url = etree.SubElement(root_element, 'url')
        else:
            url = etree.Element('url')
        etree.SubElement(url, 'loc').text = self.loc
        if self.lastmod:
            if hasattr(self.lastmod, 'strftime'):
                etree.SubElement(url, 'lastmod').text = \
                    self.lastmod.strftime('%Y-%m-%d')
            elif isinstance(self.lastmod, str):
                etree.SubElement(url, 'lastmod').text = self.lastmod
        if self.changefreq and self.changefreq in self.freq_values:
            etree.SubElement(url, 'changefreq').text = self.changefreq
        if self.priority and 0.0 <= self.priority <= 1.0:
            etree.SubElement(url, 'priority').text = str(self.priority)
        return url

Response: Generate the url element under root_element.


Instruction:
Make a summary of the following Python 3 code
def make_input_stream():
    input_queue = Queue()

    def input_stream():
        while True:
            cmd, msg = input_queue.get()
            if cmd == 'end':
                input_queue.task_done()
                return
            elif cmd == 'msg':
                yield msg.encode()
                input_queue.task_done()

    return input_queue, input_stream

Response:
Creates a queue and a co - routine yielding from that
   .


Instruction:
How would you explain what the following Python 3 function does
def notblocked(page):
    for blocked in PAGES_TO_BLOCK:
        if blocked[0] != '*':
            blocked = '*' + blocked
        rx = re.compile(blocked.replace('*', '[^$]*'))
        if rx.match(page):
            return False
    return True

Response: Determine if given url is a page that should be in sitemap.


Instruction:
Make a summary of the following Python 3 code
def sitemap(request):
    xml = Sitemap()
    author = request.matchdict.get('from_id')
    if author is not None:
        match = "AND '{}' = authors[1]".format(author)
    else:
        match = ''

    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            cursor.execute("""SELECT
                    ident_hash(uuid, major_version, minor_version)
                        AS idver,
                    name,
                    revised
                FROM latest_modules
                WHERE portal_type NOT IN ('CompositeModule', 'SubCollection')
                {}
                ORDER BY module_ident DESC""".format(match))
            res = cursor.fetchall()
            for ident_hash, page_name, revised in res:
                #  replace punctuation with whitespace
                page_name = re.sub(NON_WORD, ' ',
                                   page_name.decode('utf-8'), re.UNICODE)
                # remove leading and trailing whitespace
                page_name = page_name.strip().encode('utf-8')
                # replace spaces with dashes
                page_name = re.sub(' +', '-',
                                   page_name.decode('utf-8'), re.UNICODE)
                url = request.route_url('content',
                                        ident_hash=ident_hash,
                                        ignore=u'/{}'.format(page_name))
                if notblocked(url):
                    xml.add_url(url, lastmod=revised)

    resp = request.response
    resp.status = '200 OK'
    resp.content_type = 'text/xml'
    resp.body = xml()
    return resp

Response: Return a sitemap xml file for search engines.


Instruction:
Given the following Python 3 function, write the documentation
def sitemap_index(request):
    sitemaps = []
    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            cursor.execute("""\
                SELECT authors[1], max(revised)
                FROM latest_modules
                WHERE portal_type NOT IN ('CompositeModule', 'SubCollection')
                GROUP BY authors[1]
            """)

            for author, revised in cursor.fetchall():
                sitemaps.append(Sitemap(url=request.route_url(
                    'sitemap', from_id=author),
                    lastmod=revised))

    si = SitemapIndex(sitemaps=sitemaps)
    resp = request.response
    resp.status = '200 OK'
    resp.content_type = 'text/xml'
    resp.body = si()
    return resp

Response: Return a sitemap index xml file for search engines.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_volume(cont, pos_x, pos_y, pix):
    if np.isscalar(pos_x):
        cont = [cont]
        ret_list = False
    else:
        ret_list = True

    # Convert input to 1D arrays
    pos_x = np.atleast_1d(pos_x)
    pos_y = np.atleast_1d(pos_y)

    if pos_x.size != pos_y.size:
        raise ValueError("Size of `pos_x` and `pos_y` must match!")

    if pos_x.size > 1 and len(cont) <= 1:
        raise ValueError("Number of given contours too small!")

    # results are stored in a separate array initialized with nans
    v_avg = np.zeros_like(pos_x, dtype=float)*np.nan

    # v_avg has the shape of `pos_x`. We are iterating over the smallest
    # length for `cont` and `pos_x`.
    for ii in range(min(len(cont), pos_x.shape[0])):
        # If the contour has less than 4 pixels, the computation will fail.
        # In that case, the value np.nan is already assigned.
        cc = cont[ii]
        if cc.shape[0] >= 4:
            # Center contour coordinates with given centroid
            contour_x = cc[:, 0] - pos_x[ii] / pix
            contour_y = cc[:, 1] - pos_y[ii] / pix
            # Make sure contour is counter-clockwise
            contour_x, contour_y = counter_clockwise(contour_x, contour_y)
            # Which points are below the x-axis? (y<0)?
            ind_low = np.where(contour_y < 0)
            # These points will be shifted up to y=0 to build an x-axis
            # (wont contribute to lower volume).
            contour_y_low = np.copy(contour_y)
            contour_y_low[ind_low] = 0
            # Which points are above the x-axis? (y>0)?
            ind_upp = np.where(contour_y > 0)
            # These points will be shifted down to y=0 to build an x-axis
            # (wont contribute to upper volume).
            contour_y_upp = np.copy(contour_y)
            contour_y_upp[ind_upp] = 0
            # Move the contour to the left
            Z = contour_x
            # Last point of the contour has to overlap with the first point
            Z = np.hstack([Z, Z[0]])
            Zp = Z[0:-1]
            dZ = Z[1:]-Zp

            # Last point of the contour has to overlap with the first point
            contour_y_low = np.hstack([contour_y_low, contour_y_low[0]])
            contour_y_upp = np.hstack([contour_y_upp, contour_y_upp[0]])

            vol_low = _vol_helper(contour_y_low, Z, Zp, dZ, pix)
            vol_upp = _vol_helper(contour_y_upp, Z, Zp, dZ, pix)

            v_avg[ii] = (vol_low + vol_upp) / 2

    if not ret_list:
        # Do not return a list if the input contour was not in a list
        v_avg = v_avg[0]

    return v_avg

Response: Calculate the volume of a polygon revolved around an axis.


Instruction:
Here you have a function in Python 3, explain what it does
def counter_clockwise(cx, cy):
    # test orientation
    angles = np.unwrap(np.arctan2(cy, cx))
    grad = np.gradient(angles)
    if np.average(grad) > 0:
        return cx[::-1], cy[::-1]
    else:
        return cx, cy

Response: Put contour coordinates into counter - clockwise order.


Instruction:
Write a Python 3 function that can
add track and sectors to the metadata.

Response:
def add_metadata_sectors(self, vtoc, sector_list, header):
        """Add track/sector list
        """
        tslist = BaseSectorList(header)
        for start in range(0, len(sector_list), header.ts_pairs):
            end = min(start + header.ts_pairs, len(sector_list))
            if _xd: log.debug("ts: %d-%d" % (start, end))
            s = Dos33TSSector(header, sector_list, start, end)
            s.ts_start, s.ts_end = start, end
            tslist.append(s)
        self.num_tslists = len(tslist)
        vtoc.assign_sector_numbers(self, tslist)
        sector_list.extend(tslist)
        self.track, self.sector = header.track_from_sector(tslist[0].sector_num)
        if _xd: log.debug("track/sector lists:\n%s" % str(tslist))


Instruction:
Write a Python 3 script for
returning all subjects in the database except internal scheme.

Response:
def _get_subject_list_generator(cursor):
    """Return all subjects (tags) in the database except "internal" scheme."""
    subject = None
    last_tagid = None
    cursor.execute(SQL['get-subject-list'])
    for s in cursor.fetchall():
        tagid, tagname, portal_type, count = s

        if tagid != last_tagid:
            # It's a new subject, create a new dict and initialize count
            if subject:
                yield subject
            subject = {'id': tagid,
                       'name': tagname,
                       'count': {'module': 0, 'collection': 0}, }
            last_tagid = tagid

        if tagid == last_tagid and portal_type:
            # Just need to update the count
            subject['count'][portal_type.lower()] = count

    if subject:
        yield subject


Instruction:
Here you have a function in Python 3, explain what it does
def extras(request):
    key = request.matchdict.get('key', '').lstrip('/')
    key_map = {
        'languages': _get_available_languages_and_count,
        'subjects': _get_subject_list,
        'featured': _get_featured_links,
        'messages': _get_service_state_messages,
        'licenses': _get_licenses
        }

    with db_connect() as db_connection:
        with db_connection.cursor() as cursor:
            if key:
                proc = key_map[key]
                metadata = {key: proc(cursor)}
            else:
                metadata = {key: proc(cursor)
                            for (key, proc) in key_map.items()}

    resp = request.response
    resp.status = '200 OK'
    resp.content_type = 'application/json'
    resp.body = json.dumps(metadata)
    return resp

Response: Return a dict with archive metadata for webview.


Instruction:
Write a Python 3 function that can
run a function call for delay seconds and raise a RuntimeError if the operation didn t complete.

Response:
def timeout(delay, call, *args, **kwargs):
    """Run a function call for `delay` seconds, and raise a RuntimeError
    if the operation didn't complete."""
    return_value = None

    def target():
        nonlocal return_value
        return_value = call(*args, **kwargs)

    t = Thread(target=target)
    t.start()
    t.join(delay)
    if t.is_alive():
        raise RuntimeError("Operation did not complete within time.")

    return return_value


Instruction:
Make a summary of the following Python 3 code
def create_parser(name, description=None):
    prog = _gen_prog_name(name)
    parser = argparse.ArgumentParser(prog=prog, description=description)
    parser.add_argument('config_uri', help="Configuration INI file.")
    parser.add_argument('--config-name',
                        action='store',
                        default='main',
                        help="Supply a section name in the configuration")
    return parser

Response: Create an argument parser with the given name and description.


Instruction:
Make a summary of the following Python 3 code
def get_app_settings_from_arguments(args):
    config_filepath = os.path.abspath(args.config_uri)
    return get_appsettings(config_filepath, name=args.config_name)

Response: Parse the arguments into app settings.


Instruction:
Implement a function in Python 3 to
check if port is open on host.

Response:
def check_socket(host, port):
    """Checks if port is open on host. This is used to check if the
    Xenon-GRPC server is running."""
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
        return sock.connect_ex((host, port)) == 0


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_secure_channel(crt_file, key_file, port=50051):

    creds = grpc.ssl_channel_credentials(
        root_certificates=open(str(crt_file), 'rb').read(),
        private_key=open(str(key_file), 'rb').read(),
        certificate_chain=open(str(crt_file), 'rb').read())

    address = "{}:{}".format(socket.gethostname(), port)
    channel = grpc.secure_channel(address, creds)
    return channel

Response: Try to connect over a secure channel.


Instruction:
Create a Python 3 function for
finding a free port.

Response:
def find_free_port():
    """Finds a free port."""
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
        sock.bind(('', 0))
        return sock.getsockname()[1]


Instruction:
Create a Python 3 function for
printing stream from file to logger.

Response:
def print_stream(file, name):
    """Print stream from file to logger."""
    logger = logging.getLogger('xenon.{}'.format(name))
    for line in file:
        logger.info('[{}] {}'.format(name, line.strip()))


Instruction:
Implement a function in Python 3 to
start the Xenon GRPC server on the specified port.

Response:
def init(port=None, do_not_exit=False, disable_tls=False, log_level='WARNING'):
    """Start the Xenon GRPC server on the specified port, or, if a service
    is already running on that port, connect to that.

    If no port is given, a random port is selected. This means that, by
    default, every python instance will start its own instance of a xenon-grpc
    process.

    :param port: the port number
    :param do_not_exit: by default the GRPC server is shut down after Python
        exits (through the `atexit` module), setting this value to `True` will
        prevent that from happening."""
    logger = logging.getLogger('xenon')
    logger.setLevel(logging.INFO)

    logger_handler = logging.StreamHandler()
    logger_handler.setFormatter(logging.Formatter(style='{'))
    logger_handler.setLevel(getattr(logging, log_level))
    logger.addHandler(logger_handler)

    if port is None:
        port = find_free_port()

    if __server__.process is not None:
        logger.warning(
            "You tried to run init(), but the server is already running.")
        return __server__

    __server__.port = port
    __server__.disable_tls = disable_tls
    __server__.__enter__()

    if not do_not_exit:
        atexit.register(__server__.__exit__, None, None, None)

    return __server__


Instruction:
Can you tell what is the following Python 3 function doing
def add_cors_headers(request, response):
    response.headerlist.append(('Access-Control-Allow-Origin', '*'))
    response.headerlist.append(
        ('Access-Control-Allow-Methods', 'GET, OPTIONS'))
    response.headerlist.append(
        ('Access-Control-Allow-Headers',
         ','.join(DEFAULT_ACCESS_CONTROL_ALLOW_HEADERS)))

Response: Add CORS headers needed for web app implementation.


Instruction:
Write a Python 3 script for
initializing the trace data

Response:
def trace(self):
        """Initializes the trace data"""
        if self._trace is None:
            self._trace = self.load_trace(self.mname)
        return self._trace


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load_trace(mname):
        tname = TraceColumn.find_trace_file(mname)

        # Initialize empty trace dictionary
        trace = {}

        if tname is None:
            pass
        elif tname.suffix == ".tdms":
            # Again load the measurement tdms file.
            # This might increase memory usage, but it is cleaner
            # when looking at code structure.
            mdata = TdmsFile(str(mname))
            sampleids = mdata.object("Cell Track", "FL1index").data

            # Load the trace data. The traces file is usually larger than the
            # measurement file.
            tdata = TdmsFile(str(tname))
            for trace_key in dfn.FLUOR_TRACES:
                group, ch = naming.tr_data_map[trace_key]
                try:
                    trdat = tdata.object(group, ch).data
                except KeyError:
                    pass
                else:
                    if trdat is not None and trdat.size != 0:
                        # Only add trace if there is actual data.
                        # Split only needs the position of the sections,
                        # so we remove the first (0) index.
                        trace[trace_key] = np.split(trdat, sampleids[1:])
        return trace

Response: Loads the traces and returns them as a dictionary which contains the keys of the actual data.


Instruction:
How would you code a function in Python 3 to
try to find the traces tdms file name Returns None if no trace file is found.

Response:
def find_trace_file(mname):
        """Tries to find the traces tdms file name

        Returns None if no trace file is found.
        """
        mname = pathlib.Path(mname)
        tname = None

        if mname.exists():
            cand = mname.with_name(mname.name[:-5] + "_traces.tdms")
            if cand.exists():
                tname = cand

        return tname