Instruction:
Can you generate the documentation for the following Python 3 function
def update_from_element(self, elem):
        self._elem = elem

        for attrname in self.attributes:
            try:
                delattr(self, attrname)
            except AttributeError:
                pass

        document_url = elem.attrib.get('href')
        if document_url is not None:
            self._url = document_url

        return self

Response:
Reset this Resource instance to represent the values in
        the given XML element.


Instruction:
Write a Python 3 function for
returning a Page of instances of this Resource class from its general collection endpoint.

Response:
def all(cls, **kwargs):
        """Return a `Page` of instances of this `Resource` class from
        its general collection endpoint.

        Only `Resource` classes with specified `collection_path`
        endpoints can be requested with this method. Any provided
        keyword arguments are passed to the API endpoint as query
        parameters.

        """
        url = recurly.base_uri() + cls.collection_path
        if kwargs:
            url = '%s?%s' % (url, urlencode_params(kwargs))
        return Page.page_for_url(url)


Instruction:
Create a Python 3 function for
returning a count of server side resources given a set of keyword arguments in kwargs.

Response:
def count(cls, **kwargs):
        """Return a count of server side resources given
        filtering arguments in kwargs.
        """
        url = recurly.base_uri() + cls.collection_path
        if kwargs:
            url = '%s?%s' % (url, urlencode_params(kwargs))
        return Page.count_for_url(url)


Instruction:
Make a summary of the following Python 3 code
def put(self, url):
        response = self.http_request(url, 'PUT', self, {'Content-Type': 'application/xml; charset=utf-8'})
        if response.status != 200:
            self.raise_http_error(response)

        response_xml = response.read()
        logging.getLogger('recurly.http.response').debug(response_xml)
        self.update_from_element(ElementTree.fromstring(response_xml))

Response:
Sends this Resource instance to the service with a
        PUT request to the given URL.


Instruction:
Here you have a function in Python 3, explain what it does
def post(self, url, body=None):
        response = self.http_request(url, 'POST', body or self, {'Content-Type': 'application/xml; charset=utf-8'})
        if response.status not in (200, 201, 204):
            self.raise_http_error(response)

        self._url = response.getheader('Location')

        if response.status in (200, 201):
            response_xml = response.read()
            logging.getLogger('recurly.http.response').debug(response_xml)
            self.update_from_element(ElementTree.fromstring(response_xml))

Response:
Sends this Resource instance to the service with a
        POST request to the given URL. Takes an optional body.


Instruction:
Can you tell what is the following Python 3 function doing
def delete(self):
        response = self.http_request(self._url, 'DELETE')
        if response.status != 204:
            self.raise_http_error(response)

Response: Submits a DELETE request to the resource instance s URL.


Instruction:
How would you implement a function in Python 3 that
raises a HTTPError of the appropriate subclass of InfrastructureException.

Response:
def raise_http_error(cls, response):
        """Raise a `ResponseError` of the appropriate subclass in
        reaction to the given `http_client.HTTPResponse`."""
        response_xml = response.read()
        logging.getLogger('recurly.http.response').debug(response_xml)
        exc_class = recurly.errors.error_class_for_http_status(response.status)
        raise exc_class(response_xml)


Instruction:
Create a Python 3 function to
serialize this Resource instance to an XML element.

Response:
def to_element(self, root_name=None):
        """Serialize this `Resource` instance to an XML element."""
        if not root_name:
            root_name = self.nodename
        elem = ElementTreeBuilder.Element(root_name)
        for attrname in self.serializable_attributes():
            # Only use values that have been loaded into the internal
            # __dict__. For retrieved objects we look into the XML response at
            # access time, so the internal __dict__ contains only the elements
            # that have been set on the client side.
            try:
                value = self.__dict__[attrname]
            except KeyError:
                continue

            if attrname in self.xml_attribute_attributes:
                elem.attrib[attrname] = six.text_type(value)
            else:
                sub_elem = self.element_for_value(attrname, value)
                elem.append(sub_elem)

        return elem


Instruction:
Can you generate a brief explanation for the following Python 3 code
def bss_eval_sources(reference_sources, estimated_sources,
                     compute_permutation=True):
    """
    BSS Eval v3 bss_eval_sources

    Wrapper to ``bss_eval`` with the right parameters.
    The call to this function is not recommended. See the description for the
    ``bsseval_sources`` parameter of ``bss_eval``.

    """
    (sdr, isr, sir, sar, perm) = \
        bss_eval(
            reference_sources, estimated_sources,
            window=np.inf, hop=np.inf,
            compute_permutation=compute_permutation, filters_len=512,
            framewise_filters=True,
            bsseval_sources_version=True
    )
    return (sdr, sir, sar, perm)

Response: Wrapper to bss_eval with the right parameters.


Instruction:
Can you generate the documentation for the following Python 3 function
def bss_eval_sources_framewise(reference_sources, estimated_sources,
                               window=30 * 44100, hop=15 * 44100,
                               compute_permutation=False):
    """
    BSS Eval v3 bss_eval_sources_framewise

    Wrapper to ``bss_eval`` with the right parameters.
    The call to this function is not recommended. See the description for the
    ``bsseval_sources`` parameter of ``bss_eval``.

    """
    (sdr, isr, sir, sar, perm) = \
        bss_eval(
            reference_sources, estimated_sources,
            window=window, hop=hop,
            compute_permutation=compute_permutation, filters_len=512,
            framewise_filters=True,
            bsseval_sources_version=True)
    return (sdr, sir, sar, perm)

Response: Wrapper for bss_eval that returns the framewise version of the BSS EVAL sources.


Instruction:
Can you write a function in Python 3 where it
wraps for bss_eval that returns a list of images for the given set of reference sources and estimated sources.

Response:
def bss_eval_images(reference_sources, estimated_sources,
                    compute_permutation=True):
    """
    BSS Eval v3 bss_eval_images

    Wrapper to ``bss_eval`` with the right parameters.

    """
    return bss_eval(
        reference_sources, estimated_sources,
        window=np.inf, hop=np.inf,
        compute_permutation=compute_permutation, filters_len=512,
        framewise_filters=True,
        bsseval_sources_version=False)


Instruction:
Can you write a function in Python 3 where it
wraps of bss_eval with the right parameters.

Response:
def bss_eval_images_framewise(reference_sources, estimated_sources,
                              window=30 * 44100, hop=15 * 44100,
                              compute_permutation=False):
    """
    BSS Eval v3 bss_eval_images_framewise

    Framewise computation of bss_eval_images.
    Wrapper to ``bss_eval`` with the right parameters.

    """
    return bss_eval(
        reference_sources, estimated_sources,
        window=window, hop=hop,
        compute_permutation=compute_permutation, filters_len=512,
        framewise_filters=True,
        bsseval_sources_version=False
    )


Instruction:
Can you generate the documentation for the following Python 3 function
def _zeropad(sig, N, axis=0):
    # ensures concatenation dimension is the first
    sig = np.moveaxis(sig, axis, 0)
    # zero pad
    out = np.zeros((sig.shape[0] + N,) + sig.shape[1:])
    out[:sig.shape[0], ...] = sig
    # put back axis in place
    out = np.moveaxis(out, 0, axis)
    return out

Response: pads with N zeros at the end of the signal along given axis


Instruction:
Explain what the following Python 3 code does
def _reshape_G(G):
    G = np.moveaxis(G, (1, 3), (3, 4))
    (nsrc, nchan, filters_len) = G.shape[0:3]
    G = np.reshape(
        G, (nsrc * nchan * filters_len, nsrc * nchan * filters_len)
    )
    return G

Response: From a correlation matrix of sizensrc nsrc nchan X nchan X filters_len X filters_len creates a new one of sizensrc nchan X nchan X filters_len X nsrc X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X nchan X filters_len X X


Instruction:
Write a Python 3 function for
computing the inner products between delayed versions of reference_sources reference is nsrc X nsamp X nchan X filters_len X filters_len

Response:
def _compute_reference_correlations(reference_sources, filters_len):
    """Compute the inner products between delayed versions of reference_sources
    reference is nsrc X nsamp X nchan.
    Returns
    * G, matrix : nsrc X nsrc X nchan X nchan X filters_len X filters_len
    * sf, reference spectra: nsrc X nchan X filters_len"""

    # reshape references as nsrc X nchan X nsampl
    (nsrc, nsampl, nchan) = reference_sources.shape
    reference_sources = np.moveaxis(reference_sources, (1), (2))

    # zero padding and FFT of references
    reference_sources = _zeropad(reference_sources, filters_len - 1, axis=2)
    n_fft = int(2**np.ceil(np.log2(nsampl + filters_len - 1.)))
    sf = scipy.fftpack.fft(reference_sources, n=n_fft, axis=2)

    # compute intercorrelation between sources
    G = np.zeros((nsrc, nsrc, nchan, nchan, filters_len, filters_len))
    for ((i, c1), (j, c2)) in itertools.combinations_with_replacement(
        itertools.product(
            list(range(nsrc)), list(range(nchan))
        ),
        2
    ):

        ssf = sf[j, c2] * np.conj(sf[i, c1])
        ssf = np.real(scipy.fftpack.ifft(ssf))
        ss = toeplitz(
            np.hstack((ssf[0], ssf[-1:-filters_len:-1])),
            r=ssf[:filters_len]
        )
        G[j, i, c2, c1] = ss
        G[i, j, c1, c2] = ss.T
    return G, sf


Instruction:
Create a Python 3 function to
compute the least - squares projection of the estimated source on the subspace spanned by delayed versions of 0 and filters_len - 1

Response:
def _compute_projection_filters(G, sf, estimated_source):
    """Least-squares projection of estimated source on the subspace spanned by
    delayed versions of reference sources, with delays between 0 and
    filters_len-1
    """
    # epsilon
    eps = np.finfo(np.float).eps

    # shapes
    (nsampl, nchan) = estimated_source.shape
    # handles the case where we are calling this with only one source
    # G should be nsrc X nsrc X nchan X nchan X filters_len X filters_len
    # and sf should be nsrc X nchan X filters_len
    if len(G.shape) == 4:
        G = G[None, None, ...]
        sf = sf[None, ...]
    nsrc = G.shape[0]
    filters_len = G.shape[-1]

    # zero pad estimates and put chan in first dimension
    estimated_source = _zeropad(estimated_source.T, filters_len - 1, axis=1)

    # compute its FFT
    n_fft = int(2**np.ceil(np.log2(nsampl + filters_len - 1.)))
    sef = scipy.fftpack.fft(estimated_source, n=n_fft)

    # compute the cross-correlations between sources and estimates
    D = np.zeros((nsrc, nchan, filters_len, nchan))
    for (j, cj, c) in itertools.product(
        list(range(nsrc)), list(range(nchan)), list(range(nchan))
    ):
        ssef = sf[j, cj] * np.conj(sef[c])
        ssef = np.real(scipy.fftpack.ifft(ssef))
        D[j, cj, :, c] = np.hstack((ssef[0], ssef[-1:-filters_len:-1]))

    # reshape matrices to build the filters
    D = D.reshape(nsrc * nchan * filters_len, nchan)
    G = _reshape_G(G)

    # Distortion filters
    try:
        C = np.linalg.solve(G + eps*np.eye(G.shape[0]), D).reshape(
            nsrc, nchan, filters_len, nchan
        )
    except np.linalg.linalg.LinAlgError:
        C = np.linalg.lstsq(G, D)[0].reshape(
            nsrc, nchan, filters_len, nchan
        )

    # if we asked for one single reference source,
    # return just a nchan X filters_len matrix
    if nsrc == 1:
        C = C[0]
    return C


Instruction:
How would you explain what the following Python 3 function does
def _project(reference_sources, C):
    # shapes: ensure that input is 3d (comprising the source index)
    if len(reference_sources.shape) == 2:
        reference_sources = reference_sources[None, ...]
        C = C[None, ...]

    (nsrc, nsampl, nchan) = reference_sources.shape
    filters_len = C.shape[-2]

    # zero pad
    reference_sources = _zeropad(reference_sources, filters_len - 1, axis=1)
    sproj = np.zeros((nchan, nsampl + filters_len - 1))

    for (j, cj, c) in itertools.product(
        list(range(nsrc)), list(range(nchan)), list(range(nchan))
    ):
        sproj[c] += fftconvolve(
            C[j, cj, :, c],
            reference_sources[j, :, cj]
        )[:nsampl + filters_len - 1]
    return sproj.T

Response:
Project images using pre - computed filters C
 reference_sources are nsrc nsampl nchan C is nsrc nsampl nchan C is nsrc nchan C is nsrc nchan C is nsrc nchan nchan


Instruction:
Explain what the following Python 3 code does
def _bss_crit(s_true, e_spat, e_interf, e_artif, bsseval_sources_version):
    # energy ratios
    if bsseval_sources_version:
        s_filt = s_true + e_spat
        energy_s_filt = np.sum(s_filt**2)
        sdr = _safe_db(energy_s_filt,
                       np.sum((e_interf + e_artif)**2))
        isr = np.empty(sdr.shape) * np.nan
        sir = _safe_db(energy_s_filt, np.sum(e_interf**2))
        sar = _safe_db(np.sum((s_filt + e_interf)**2),
                       np.sum(e_artif**2))
    else:
        energy_s_true = np.sum((s_true)**2)
        sdr = _safe_db(energy_s_true,
                       np.sum((e_spat + e_interf + e_artif)**2))
        isr = _safe_db(energy_s_true, np.sum(e_spat**2))
        sir = _safe_db(np.sum((s_true + e_spat)**2), np.sum(e_interf**2))
        sar = _safe_db(np.sum((s_true + e_spat + e_interf)**2),
                       np.sum(e_artif**2))

    return (sdr, isr, sir, sar)

Response:
Measurement of the separation quality for a given source in terms of
    filtered true source interference and artifacts.


Instruction:
How would you explain what the following Python 3 function does
def _safe_db(num, den):
    if den == 0:
        return np.inf
    return 10 * np.log10(num / den)

Response: Properly handle the potential + Inf db SIR instead of raising RuntimeWarning. a


Instruction:
Can you write a function in Python 3 where it
gives a link - value return a dictionary whose keys are link URLs and values are dictionaries whose keys are parameters and values are dictionaries whose keys are parameters and values are lists of link URLs and their associated parameters.

Response:
def parse_link_value(instr):
    """
    Given a link-value (i.e., after separating the header-value on commas),
    return a dictionary whose keys are link URLs and values are dictionaries
    of the parameters for their associated links.

    Note that internationalised parameters (e.g., title*) are
    NOT percent-decoded.

    Also, only the last instance of a given parameter will be included.

    For example,

    >>> parse_link_value('</foo>; rel="self"; title*=utf-8\'de\'letztes%20Kapitel')
    {'/foo': {'title*': "utf-8'de'letztes%20Kapitel", 'rel': 'self'}}

    """
    out = {}
    if not instr:
        return out
    for link in [h.strip() for h in link_splitter.findall(instr)]:
        url, params = link.split(">", 1)
        url = url[1:]
        param_dict = {}
        for param in _splitstring(params, PARAMETER, "\s*;\s*"):
            try:
                a, v = param.split("=", 1)
                param_dict[a.lower()] = _unquotestring(v)
            except ValueError:
                param_dict[param.lower()] = None
        out[url] = param_dict
    return out


Instruction:
How would you implement a function in Python 3 that
constructs the URL for the desired API endpoint.

Response:
def construct_api_url(input, representation, resolvers=None, get3d=False, tautomers=False, xml=True, **kwargs):
    """Return the URL for the desired API endpoint.

    :param string input: Chemical identifier to resolve
    :param string representation: Desired output representation
    :param list(str) resolvers: (Optional) Ordered list of resolvers to use
    :param bool get3d: (Optional) Whether to return 3D coordinates (where applicable)
    :param bool tautomers: (Optional) Whether to return all tautomers
    :param bool xml: (Optional) Whether to return full XML response
    :returns: CIR API URL
    :rtype: str
    """
    # File formats require representation=file and the format in the querystring
    if representation in FILE_FORMATS:
        kwargs['format'] = representation
        representation = 'file'
    # Prepend input with 'tautomers:' to return all tautomers
    if tautomers:
        input = 'tautomers:%s' % input
    url = '%s/%s/%s' % (API_BASE, quote(input), representation)
    if xml:
        url += '/xml'
    if resolvers:
        kwargs['resolver'] = ','.join(resolvers)
    if get3d:
        kwargs['get3d'] = True
    if kwargs:
        url += '?%s' % urlencode(kwargs)
    return url


Instruction:
Given the following Python 3 function, write the documentation
def request(input, representation, resolvers=None, get3d=False, tautomers=False, **kwargs):
    url = construct_api_url(input, representation, resolvers, get3d, tautomers, **kwargs)
    log.debug('Making request: %s', url)
    response = urlopen(url)
    return etree.parse(response).getroot()

Response: Make a request to CIR and return the XML response.


Instruction:
Given the following Python 3 function, write the documentation
def query(input, representation, resolvers=None, get3d=False, tautomers=False, **kwargs):
    tree = request(input, representation, resolvers, get3d, tautomers, **kwargs)
    results = []
    for data in tree.findall('.//data'):
        value = [item.text for item in data.findall('item')]
        result = Result(
            input=tree.attrib['string'],
            representation=tree.attrib['representation'],
            resolver=data.attrib['resolver'],
            input_format=data.attrib['string_class'],
            notation=data.attrib['notation'],
            value=value[0] if len(value) == 1 else value
        )
        results.append(result)
    log.debug('Received %s query results', len(results))
    return results

Response: Query the CIR for all results for resolving input to the specified output representation.


Instruction:
Given the following Python 3 function, write the documentation
def resolve(input, representation, resolvers=None, get3d=False, **kwargs):
    # Take first result from XML query
    results = query(input, representation, resolvers, False, get3d, **kwargs)
    result = results[0].value if results else None
    return result

Response: Resolve input to the specified output representation.


Instruction:
Can you write a function in Python 3 where it
resolves input to a 2D image depiction.

Response:
def resolve_image(input, resolvers=None, fmt='png', width=300, height=300, frame=False, crop=None, bgcolor=None,
                  atomcolor=None, hcolor=None, bondcolor=None, framecolor=None, symbolfontsize=11, linewidth=2,
                  hsymbol='special', csymbol='special', stereolabels=False, stereowedges=True, header=None, footer=None,
                  **kwargs):
    """Resolve input to a 2D image depiction.

    :param string input: Chemical identifier to resolve
    :param list(string) resolvers: (Optional) Ordered list of resolvers to use
    :param string fmt: (Optional) gif or png image format (default png)

    :param int width: (Optional) Image width in pixels (default 300)
    :param int height: (Optional) Image height in pixels (default 300)
    :param bool frame: (Optional) Whether to show border frame (default False)
    :param int crop: (Optional) Crop image with specified padding

    :param int symbolfontsize: (Optional) Atom label font size (default 11)
    :param int linewidth: (Optional) Bond line width (default 2)

    :param string bgcolor: (Optional) Background color
    :param string atomcolor: (Optional) Atom label color
    :param string hcolor: (Optional) Hydrogen atom label color
    :param string bondcolor: (Optional) Bond color
    :param string framecolor: (Optional) Border frame color

    :param bool hsymbol: (Optional) Hydrogens: all, special or none (default special)
    :param bool csymbol: (Optional) Carbons: all, special or none (default special)
    :param bool stereolabels: (Optional) Whether to show stereochemistry labels (default False)
    :param bool stereowedges: (Optional) Whether to show wedge/dash bonds (default True)
    :param string header: (Optional) Header text above structure
    :param string footer: (Optional) Footer text below structure

    """
    # Aggregate all arguments into kwargs
    args, _, _, values = inspect.getargvalues(inspect.currentframe())
    for arg in args:
        if values[arg] is not None:
            kwargs[arg] = values[arg]
    # Turn off anti-aliasing for transparent background
    if kwargs.get('bgcolor') == 'transparent':
        kwargs['antialiasing'] = False
    # Renamed parameters
    if 'stereolabels' in kwargs:
        kwargs['showstereo'] = kwargs.pop('stereolabels')
    if 'fmt' in kwargs:
        kwargs['format'] = kwargs.pop('fmt')
    # Toggle stereo wedges
    if 'stereowedges' in kwargs:
        status = kwargs.pop('stereowedges')
        kwargs.update({'wedges': status, 'dashes': status})
    # Constant values
    kwargs.update({'representation': 'image', 'xml': False})
    url = construct_api_url(**kwargs)
    log.debug('Making image request: %s', url)
    response = urlopen(url)
    return response.read()


Instruction:
Can you tell what is the following Python 3 function doing
def download(input, filename, representation, overwrite=False, resolvers=None, get3d=False, **kwargs):
    result = resolve(input, representation, resolvers, get3d, **kwargs)
    # Just log and return if nothing resolved
    if not result:
        log.debug('No file to download.')
        return
    # Only overwrite an existing file if explicitly instructed to.
    if not overwrite and os.path.isfile(filename):
        raise IOError("%s already exists. Use 'overwrite=True' to overwrite it." % filename)
    # Ensure file ends with a newline
    if not result.endswith('\n'):
        result += '\n'
    with open(filename, 'w') as f:
        f.write(result)

Response: Convenience function to save a CIR response as a file.


Instruction:
Here you have a function in Python 3, explain what it does
def twirl_url(self):
        return construct_api_url(self.input, 'twirl', self.resolvers, False, self.get3d, False, **self.kwargs)

Response: Url of a TwirlyMol 3D viewer.


Instruction:
How would you explain what the following Python 3 function does
def download(self, filename, representation, overwrite=False):
        download(self.input, filename, representation, overwrite, self.resolvers, self.get3d, **self.kwargs)

Response: Download the resolved structure as a file.


Instruction:
Given the following Python 3 function, write the documentation
def validate_token(refresh_url, exceptions=(), callback=None,
                   access_key='access_token', refresh_key='refresh_token'):
    ''' a decorator used to validate the access_token for oauth based
    data sources.
    This decorator should be used on every method in the data source that
    fetches data from the oauth controlled resource, and that relies on a
    valid access_token in order to operate properly.
    If the token is valid, the normal flow continues without any change.
    Otherwise, if any of `exceptions` tuple is raised, the normal flow
    will be preceded by the following steps:

    1. `refresh_url` will be called in order to refresh the token
    2. the newly refreshed token will be saved in the source
    3. the `callback` function will be called

    If the refresh fails for any reason, the user would have to re-grant
    permission for the application

    Parameters
    ----------
    refresh_url : str
        The URL to be called in order to refresh the access token.
    callback : str or callable
        A callback function to be called whenever the access_token is
        validated. The callback function would be called with the refreshed
        token as an argument.
        If the `callback` is not `callable`, but an `str` it will be called
        on `self` (i.e. call a method on your Data Source)
        Defaults to None
    exceptions : tuple
        A list of exceptions that should cause token revalidation
        Defaults to Exception, meaning that all errors will cause token
        refresh
    access_key : str
        The access token key as defined in the source and in the response from
        the refresh URL.
        Defaults to `access_token`
    refresh_key : str
        The refresh token key as defined in the source and in the request to
        the refresh URL.
        Defaults to `refresh_token`
    '''
    def _validate_token(f):
        def wrapper(*args):
            self = args[0]
            try:
                return f(*args)
            except exceptions:
                try:
                    self.log('Revalidating the access token...')
                    self.source[access_key] = None

                    # get a new token from refresh_url
                    token = self.source.get(refresh_key)
                    data = dict(self.options['refresh'],
                                **{refresh_key: token})
                    r = requests.post(refresh_url, data=data)
                    self.source[access_key] = r.json()[access_key]

                    # save the new token in the database
                    changes = {access_key: self.source[access_key]}
                    self.fire('source-change', changes)

                    # notify the callback that a new token was issued
                    if callback:
                        if callable(callback):
                            _callback = callback
                        else:
                            _callback = getattr(self, callback)
                        _callback(self.source.get(access_key))

                    return f(*args)
                except Exception, e:
                    self.log('Error: Access token can\'t be revalidated. '
                             'The user would have to re-authenticate',
                             traceback.format_exc())
                    # raise a non-retryable exception
                    raise PanoplyException(
                        'access token could not be refreshed ({})'
                        .format(str(e)), retryable=False)
        return wrapper
    return _validate_token

Response: Decorator that validates the access_token for OAuth based data sources.


Instruction:
How would you explain what the following Python 3 function does
def progress(self, loaded, total, msg=''):

        self.fire('progress', {
            'loaded': loaded,
            'total': total,
            'msg': msg
        })

Response: Notify about a progress change


Instruction:
Implement a function in Python 3 to
create a raw response object

Response:
def raw(self, tag, raw, metadata):
        """ Create a raw response object """
        raw = base64.b64encode(raw)
        return {
            'type': 'raw',
            'tag': tag,
            'raw': raw,
            'metadata': metadata
        }


Instruction:
Given the following Python 3 function, write the documentation
def assert_looks_like(first, second, msg=None):
    first = _re.sub("\s+", " ", first.strip())
    second = _re.sub("\s+", " ", second.strip())
    if first != second:
        raise AssertionError(msg or "%r does not look like %r" % (first, second))

Response: Assert that two strings look like first.


Instruction:
How would you code a function in Python 3 to
load and parse a feature file.

Response:
def load_feature(fname, language):
    """ Load and parse a feature file. """

    fname = os.path.abspath(fname)
    feat = parse_file(fname, language)
    return feat


Instruction:
Can you write a function in Python 3 where it
runs the given steps in the given language.

Response:
def run_steps(spec, language="en"):
    """ Can be called by the user from within a step definition to execute other steps. """

    # The way this works is a little exotic, but I couldn't think of a better way to work around
    # the fact that this has to be a global function and therefore cannot know about which step
    # runner to use (other than making step runner global)

    # Find the step runner that is currently running and use it to run the given steps
    fr = inspect.currentframe()
    while fr:
        if "self" in fr.f_locals:
            f_self = fr.f_locals['self']
            if isinstance(f_self, StepsRunner):
                return f_self.run_steps_from_string(spec, language)
        fr = fr.f_back


Instruction:
Write a Python 3 function for
calling from within step definitions to run other steps.

Response:
def run_steps_from_string(self, spec, language_name='en'):
        """ Called from within step definitions to run other steps. """

        caller = inspect.currentframe().f_back
        line = caller.f_lineno - 1
        fname = caller.f_code.co_filename

        steps = parse_steps(spec, fname, line, load_language(language_name))
        for s in steps:
            self.run_step(s)


Instruction:
Can you create a Python 3 function that
returns all the synonymns of a word in the requested language .

Response:
def words(self, key):
        """
        Give all the synonymns of a word in the requested language
        (or the default language if no word is available).
        """
        if self.default_mappings is not None and key not in self.mappings:
            return self.default_mappings[key].encode('utf').split("|")
        else:
            return self.mappings[key].encode('utf').split("|")


Instruction:
Can you tell what is the following Python 3 function doing
def simulate_async_event():
    scc.state = 'executing'
    def async_event(result):
        """All other asynchronous events or function calls
        returned from later steps will wait until this
        callback fires."""
        scc.state = result
        return 'some event result'
    deferred = Deferred()
    reactor.callLater(1, deferred.callback, 'done') # pylint: disable=E1101
    deferred.addCallback(async_event)
    return deferred

Response: Simulate an asynchronous event.


Instruction:
Given the following Python 3 function, write the documentation
def hook_decorator(cb_type):
    def decorator_wrapper(*tags_or_func):
        if len(tags_or_func) == 1 and callable(tags_or_func[0]):
            # No tags were passed to this decorator
            func = tags_or_func[0]
            return HookImpl(cb_type, func)
        else:
            # We got some tags, so we need to produce the real decorator
            tags = tags_or_func
            def d(func):
                return HookImpl(cb_type, func, tags)
            return d
    return decorator_wrapper

Response: Decorator to wrap hook definitions in. Registers hook.


Instruction:
Given the following Python 3 function, write the documentation
def load_steps_impl(self, registry, path, module_names=None):

        if not module_names:
            module_names = ['steps']

        path = os.path.abspath(path)

        for module_name in module_names:
            mod = self.modules.get((path, module_name))

            if mod is None:
                #log.debug("Looking for step def module '%s' in %s" % (module_name, path))
                cwd = os.getcwd()
                if cwd not in sys.path:
                    sys.path.append(cwd)

                try:
                    actual_module_name = os.path.basename(module_name)
                    complete_path = os.path.join(path, os.path.dirname(module_name))
                    info = imp.find_module(actual_module_name, [complete_path])
                except ImportError:
                    #log.debug("Did not find step defs module '%s' in %s" % (module_name, path))
                    return

                try:
                    # Modules have to be loaded with unique names or else problems arise
                    mod = imp.load_module("stepdefs_" + str(self.module_counter), *info)
                except:
                    exc = sys.exc_info()
                    raise StepImplLoadException(exc)

                self.module_counter += 1
                self.modules[(path, module_name)] = mod

            for item_name in dir(mod):
                item = getattr(mod, item_name)
                if isinstance(item, StepImpl):
                    registry.add_step(item.step_type, item)
                elif isinstance(item, HookImpl):
                    registry.add_hook(item.cb_type, item)
                elif isinstance(item, NamedTransformImpl):
                    registry.add_named_transform(item)
                elif isinstance(item, TransformImpl):
                    registry.add_transform(item)

Response: Load the step implementations at the given path and add them to the given registry.


Instruction:
Make a summary of the following Python 3 code
def find_step_impl(self, step):
        result = None
        for si in self.steps[step.step_type]:
            matches = si.match(step.match)
            if matches:
                if result:
                    raise AmbiguousStepImpl(step, result[0], si)

                args = [self._apply_transforms(arg, si) for arg in matches.groups()]
                result = si, args

        if not result:
            raise UndefinedStepImpl(step)
        return result

Response: Find the implementation of the given step.


Instruction:
Can you write a function in Python 3 where it
opens a child process and return its exit status and stdout.

Response:
def run_command(cmd):
    """
    Open a child process, and return its exit status and stdout.

    """
    child = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE,
                             stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    out = [s.decode("utf-8").strip() for s in child.stdout]
    err = [s.decode("utf-8").strip() for s in child.stderr]
    w = child.wait()
    return os.WEXITSTATUS(w), out, err


Instruction:
Write a Python 3 script to
transform filters into list of table rows.

Response:
def make_filter_list(filters):
    """Transform filters into list of table rows."""
    filter_list = []
    filter_ids = []
    for f in filters:
        filter_ids.append(f.index)
        fullname = URL_P.sub(r'`<\1>`_', f.fullname)
        filter_list.append((str(f.index + 1),
                            f.name,
                            "{0:.2f}".format(f.msun_vega),
                            "{0:.2f}".format(f.msun_ab),
                            "{0:.1f}".format(f.lambda_eff),
                            fullname))
    sortf = lambda item: int(item[0])
    filter_list.sort(key=sortf)
    return filter_list


Instruction:
Can you write a function in Python 3 where it
codes for this RST-formatted table generator comes from http://stackoverflow.com/a/11350643

Response:
def make_table(data, col_names):
    """Code for this RST-formatted table generator comes from
    http://stackoverflow.com/a/11350643
    """
    n_cols = len(data[0])
    assert n_cols == len(col_names)
    col_sizes = [max(len(r[i]) for r in data) for i in range(n_cols)]
    for i, cname in enumerate(col_names):
        if col_sizes[i] < len(cname):
            col_sizes[i] = len(cname)
    formatter = ' '.join('{:<%d}' % c for c in col_sizes)
    rows = '\n'.join([formatter.format(*row) for row in data])
    header = formatter.format(*col_names)
    divider = formatter.format(*['=' * c for c in col_sizes])
    output = '\n'.join((divider, header, divider, rows, divider))
    return output


Instruction:
Write a Python 3 function that can
merge multiple PDF input files in one output file.

Response:
def pdf_merge(inputs: [str], output: str, delete: bool = False):
    """
    Merge multiple Pdf input files in one output file.
    :param inputs: input files
    :param output: output file
    :param delete: delete input files after completion if true

    """
    writer = PdfFileWriter()
    if os.path.isfile(output):
        ans = input(
            "The file '%s' already exists. "
            "Overwrite? Yes/Abort [Y/a]: " % output
        ).lower()
        if ans == "a":
            return

    outputfile = open(output, "wb")
    try:
        infiles = []
        for filename in inputs:
            f = open(filename, "rb")
            reader = PdfFileReader(f)
            for page in reader.pages:
                writer.addPage(page)
            infiles.append(f)
        writer.write(outputfile)
    except FileNotFoundError as e:
        print(e.strerror + ": " + e.filename)
    finally:
        outputfile.close()
        for f in infiles:
            f.close()
    if delete:
        for filename in inputs:
            os.remove(filename)


Instruction:
Can you create a Python 3 function that
rotates the given pdf files clockwise or counter clockwise.

Response:
def pdf_rotate(
    input: str,
    counter_clockwise: bool = False,
    pages: [str] = None,
    output: str = None,
):
    """
    Rotate the given Pdf files clockwise or counter clockwise.
    :param inputs: pdf files
    :param counter_clockwise: rotate counter clockwise if true else clockwise
    :param pages: list of page numbers to rotate, if None all pages will be
        rotated
    """
    infile = open(input, "rb")
    reader = PdfFileReader(infile)
    writer = PdfFileWriter()

    # get pages from source depending on pages parameter
    if pages is None:
        source_pages = reader.pages
    else:
        pages = parse_rangearg(pages, len(reader.pages))
        source_pages = [reader.getPage(i) for i in pages]

    # rotate pages and add to writer
    for i, page in enumerate(source_pages):
        if pages is None or i in pages:
            if counter_clockwise:
                writer.addPage(page.rotateCounterClockwise(90))
            else:
                writer.addPage(page.rotateClockwise(90))
        else:
            writer.addPage(page)

    # Open output file or temporary file for writing
    if output is None:
        outfile = NamedTemporaryFile(delete=False)
    else:
        if not os.path.isfile(output) or overwrite_dlg(output):
            outfile = open(output, "wb")
        else:
            return

    # Write to file
    writer.write(outfile)
    infile.close()
    outfile.close()

    # If no output defined move temporary file to input
    if output is None:
        if overwrite_dlg(input):
            os.remove(input)
            move(outfile.name, input)
        else:
            os.remove(outfile.name)


Instruction:
Write a Python 3 function for
copying pages from the input file in a new output file.

Response:
def pdf_copy(input: str, output: str, pages: [int], yes_to_all=False):
    """
    Copy pages from the input file in a new output file.
    :param input: name of the input pdf file
    :param output: name of the output pdf file
    :param pages: list containing the page numbers to copy in the new file

    """
    if not os.path.isfile(input):
        print("Error. The file '%s' does not exist." % input)
        return

    if os.path.isfile(output) and not yes_to_all and not overwrite_dlg(output):
        return

    with open(input, "rb") as inputfile:
        reader = PdfFileReader(inputfile)
        outputfile = open(output, "wb")
        writer = PdfFileWriter()
        if pages is None:
            pages = range(len(reader.pages))
        else:
            pages = parse_rangearg(pages, len(reader.pages))
        for pagenr in sorted(pages):
            page = reader.getPage(pagenr)
            writer.addPage(page)
            writer.write(outputfile)
        outputfile.close()


Instruction:
Create a Python 3 function to
split the input file in multiple output files.

Response:
def pdf_split(
    input: str, output: str, stepsize: int = 1, sequence: [int] = None
):
    """
    Split the input file in multiple output files
    :param input: name of the input file
    :param output: name of the output files
    :param stepsize: how many pages per file, only if sequence is None
    :param sequence: list with number of pages per file

    """
    output = output or os.path.splitext(input)[0]
    if not os.path.isfile(input):
        print("Error. The file '%s' does not exist." % input)
        return

    with open(input, "rb") as inputfile:
        reader = PdfFileReader(inputfile)
        pagenr = 0
        outputfile = None
        if sequence is None:
            for i, page in enumerate(reader.pages):
                if not i % stepsize:
                    pagenr += 1
                    outputfile = open(output + "_%i.pdf" % pagenr, "wb")
                    writer = PdfFileWriter()
                writer.addPage(page)
                if not (i + 1) % stepsize:
                    writer.write(outputfile)
                    outputfile.close()
        else:
            sequence = map(int, sequence)
            iter_pages = iter(reader.pages)
            for filenr, pagecount in enumerate(sequence):
                with open(
                    output + "_%i.pdf" % (filenr + 1), "wb"
                ) as outputfile:
                    writer = PdfFileWriter()
                    for i in range(pagecount):
                        try:
                            page = next(iter_pages)
                            writer.addPage(page)
                        except StopIteration:
                            writer.write(outputfile)
                            return

                    writer.write(outputfile)

        if not outputfile.closed:
            writer.write(outputfile)
            outputfile.close()


Instruction:
Can you create a Python 3 function that
zips two PDF files.

Response:
def pdf_zip(
    input1: str,
    input2: str,
    output: str,
    delete: bool = False,
    revert: bool = False,
):
    """
    Zip pages of input1 and input2 in one output file. Useful for putting
    even and odd pages together in one document.
    :param input1: first input file
    :param input2: second input file
    :param output: output file
    :param delete: if true the input files will be deleted after zipping

    """
    if os.path.isfile(output):
        ans = input(
            "The file '%s' already exists. "
            "Overwrite? Yes/Abort [Y/a]: " % output
        ).lower()
        if ans not in ["y", ""]:
            return

    outputfile = open(output, "wb")
    try:
        f1, f2 = open(input1, "rb"), open(input2, "rb")
        r1, r2 = PdfFileReader(f1), PdfFileReader(f2)
        writer = PdfFileWriter()
        pages1 = [page for page in r1.pages]
        pages2 = [page for page in r2.pages]
        if not revert:
            for p1, p2 in zip(pages1, pages2):
                writer.addPage(p1)
                writer.addPage(p2)
        else:
            for p1, p2 in zip(pages1, reversed(pages2)):
                writer.addPage(p1)
                writer.addPage(p2)
        writer.write(outputfile)
        f1.close()
        f2.close()
    except FileNotFoundError as e:
        print(e.strerror + ": " + e.filename)
    finally:
        outputfile.close()

    if delete:
        os.remove(input1)
        os.remove(input2)


Instruction:
Can you write a function in Python 3 where it
inserts pages from one file into another.

Response:
def pdf_insert(
    dest: str,
    source: str,
    pages: [str] = None,
    index: int = None,
    output: str = None,
):
    """
    Insert pages from one file into another.
    :param dest: Destination file
    :param source: Source file
    :param pages: list of page numbers to insert
    :param index: index in destination file where to insert the pages
    :param output: output file

    """
    if output is not None and os.path.isfile(output):
        ans = input(
            "The file '%s' already exists. "
            "Overwrite? Yes/Abort [Y/a]: " % output
        ).lower()
        if ans not in ["y", ""]:
            return

    writer = PdfFileWriter()
    # read pages from file1
    destfile = open(dest, "rb")
    destreader = PdfFileReader(destfile)
    for page in destreader.pages:
        writer.addPage(page)

    # read pages from file2
    srcfile = open(source, "rb")
    srcreader = PdfFileReader(srcfile)

    # if no page numbers are given insert all pages
    index = limit(index - 1, 0, len(destreader.pages))
    if pages is None:
        for i, page in enumerate(srcreader.pages):
            if index is None:
                writer.addPage(page)
            else:
                writer.insertPage(page, index + i)
    else:
        pages = parse_rangearg(pages, len(srcreader.pages))
        for i, pagenr in enumerate(pages):
            page = srcreader.getPage(pagenr)
            if index is None:
                writer.addPage(page)
            else:
                writer.insertPage(page, index + i)

    if output is None:
        # Write into Temporary File first and then overwrite dest file
        ans = input(
            "Overwrite the file '%s'? Yes/Abort [Y/a]: " % dest
        ).lower()
        if ans in ["y", ""]:
            tempfile = NamedTemporaryFile(delete=False)
            writer.write(tempfile)
            tempfile.close()
            move(tempfile.name, dest)
    else:
        with open(output, "wb") as outfile:
            writer.write(outfile)
    destfile.close()
    srcfile.close()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def pdf_remove(source: str, pages: [str], output: str = None):
    if output is not None and os.path.isfile(output):
        if overwrite_dlg(output) is False:
            return

    writer = PdfFileWriter()
    srcfile = open(source, "rb")
    srcreader = PdfFileReader(srcfile)

    # Add pages, leave out removed pages
    pages = parse_rangearg(pages, len(srcreader.pages))
    for pagenr, page in enumerate(srcreader.pages):
        if pagenr not in pages:
            writer.addPage(page)

    # Open output file or temporary file for writing
    if output is None:
        outfile = NamedTemporaryFile(delete=False)
    else:
        outfile = open(output, "wb")

    # Write file and close
    writer.write(outfile)
    srcfile.close()
    outfile.close()

    # Move temporary file to source
    if output is None:
        if overwrite_dlg(source):
            os.remove(source)
            move(outfile.name, source)
        else:
            os.remove(outfile)

Response: Remove pages from a PDF source file.


Instruction:
Can you generate the documentation for the following Python 3 function
def pdf_add(dest: str, source: str, pages: [str], output: str):
    if output is not None and os.path.isfile(output):
        if not overwrite_dlg(output):
            return

    writer = PdfFileWriter()

    # read pages from destination file
    destfile = open(dest, "rb")
    destreader = PdfFileReader(destfile)
    for page in destreader.pages:
        writer.addPage(page)

    # read pages from source file
    srcfile = open(source, "rb")
    srcreader = PdfFileReader(srcfile)

    # if no page numbers are given add all pages from source
    if pages is None:
        for i, page in enumerate(srcreader.pages):
            writer.addPage(page)
    else:
        pages = parse_rangearg(pages, len(srcreader.pages))
        for pagenr in pages:
            page = srcreader.getPage(pagenr)
            writer.addPage(page)

    if output is None:
        # Write into Temporary File first and then overwrite dest file
        if overwrite_dlg(dest):
            tempfile = NamedTemporaryFile(delete=False)
            writer.write(tempfile)
            tempfile.close()
            destfile.close()
            srcfile.close()
            os.remove(dest)
            move(tempfile.name, dest)
    else:
        with open(output, "wb") as outfile:
            writer.write(outfile)
            destfile.close()
            srcfile.close()

Response: Add pages from a source pdf file to an output pdf file.


Instruction:
Explain what the following Python 3 code does
def extract_version():
    with open('pdftools/__init__.py', 'r') as f:
        content = f.read()

    version_match = _version_re.search(content)
    version = str(ast.literal_eval(version_match.group(1)))
    return version

Response: Extract the version from the package.


Instruction:
Can you generate the documentation for the following Python 3 function
def add_pypiper_args(parser, groups=("pypiper", ), args=None,
                     required=None, all_args=False):
    """
    Use this to add standardized pypiper arguments to your python pipeline.

    There are two ways to use `add_pypiper_args`: by specifying argument groups,
    or by specifying individual arguments. Specifying argument groups will add
    multiple arguments to your parser; these convenient argument groupings
    make it easy to add arguments to certain types of pipeline. For example,
    to make a looper-compatible pipeline, use `groups = ["pypiper", "looper"]`.

    :param argparse.ArgumentParser parser: ArgumentParser object from a pipeline
    :param str | Iterable[str] groups: Adds arguments belong to specified group
        of args. Options: pypiper, config, looper, resources, common, ngs, all.
    :param str | Iterable[str] args: You may specify a list of specific arguments one by one.
    :param Iterable[str] required: Arguments to be flagged as 'required' by argparse.
    :param bool all_args: Whether to include all of pypiper's arguments defined here.
    :return argparse.ArgumentParser: A new ArgumentParser object, with selected
        pypiper arguments added
    """
    args_to_add = _determine_args(
        argument_groups=groups, arguments=args, use_all_args=all_args)
    parser = _add_args(parser, args_to_add, required)
    return parser

Response: Adds pypiper arguments to a pipeline.


Instruction:
Explain what the following Python 3 code does
def build_command(chunks):

    if not chunks:
        raise ValueError(
            "No command parts: {} ({})".format(chunks, type(chunks)))

    if isinstance(chunks, str):
        return chunks

    parsed_pieces = []

    for cmd_part in chunks:
        if cmd_part is None:
            continue
        try:
            # Trim just space, not all whitespace.
            # This prevents damage to an option that specifies,
            # say, tab as a delimiter.
            parsed_pieces.append(cmd_part.strip(" "))
        except AttributeError:
            option, argument = cmd_part
            if argument is None or argument == "":
                continue
            option, argument = option.strip(" "), str(argument).strip(" ")
            parsed_pieces.append("{} {}".format(option, argument))

    return " ".join(parsed_pieces)

Response: Builds a single meaningful command from a collection of command parts.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def build_sample_paths(sample):
    for path_name, path in sample.paths.items():
        print("{}: '{}'".format(path_name, path))
        base, ext = os.path.splitext(path)
        if ext:
            print("Skipping file-like: '[}'".format(path))
        elif not os.path.isdir(base):
            os.makedirs(base)

Response: Build the base folder paths for a Sample.


Instruction:
Make a summary of the following Python 3 code
def checkpoint_filename(checkpoint, pipeline_name=None):
    # Allow Stage as type for checkpoint parameter's argument without
    # needing to import here the Stage type from stage.py module.
    try:
        base = checkpoint.checkpoint_name
    except AttributeError:
        base = translate_stage_name(checkpoint)
    if pipeline_name:
        base = "{}{}{}".format(
            pipeline_name, PIPELINE_CHECKPOINT_DELIMITER, base)
    return base + CHECKPOINT_EXTENSION

Response: Translate a checkpoint to a filename.


Instruction:
Write a Python 3 script to
create filepath for indicated checkpoint.

Response:
def checkpoint_filepath(checkpoint, pm):
    """
    Create filepath for indicated checkpoint.

    :param str | pypiper.Stage checkpoint: Pipeline phase/stage or one's name
    :param pypiper.PipelineManager | pypiper.Pipeline pm: manager of a pipeline
        instance, relevant for output folder path.
    :return str: standardized checkpoint name for file, plus extension
    :raise ValueError: if the checkpoint is given as absolute path that does
        not point within pipeline output folder
    """

    # Handle case in which checkpoint is given not just as a string, but
    # as a checkpoint-like filename. Don't worry about absolute path status
    # of a potential filename input, or whether it's in the pipeline's
    # output folder. That's handled upstream. While this isn't a protected
    # function, there's no real reason to call this from outside the package.
    if isinstance(checkpoint, str):
        if os.path.isabs(checkpoint):
            if is_in_file_tree(checkpoint, pm.outfolder):
                return checkpoint
            else:
                raise ValueError(
                    "Absolute checkpoint path '{}' is not in pipeline output "
                    "folder '{}'".format(checkpoint, pm.outfolder))
        _, ext = os.path.splitext(checkpoint)
        if ext == CHECKPOINT_EXTENSION:
            return pipeline_filepath(pm, filename=checkpoint)

    # Allow Pipeline as pm type without importing Pipeline.
    try:
        pm = pm.manager
    except AttributeError:
        pass

    # We want the checkpoint filename itself to become a suffix, with a
    # delimiter intervening between the pipeline name and the checkpoint
    # name + extension. This is to handle the case in which a single, e.g.,
    # sample's output folder is the destination for output from multiple
    # pipelines, and we thus want to be able to distinguish between
    # checkpoint files from different pipelines for that sample that may
    # well define one or more stages with the same name (e.g., trim_reads,
    # align_reads, etc.)
    chkpt_name = checkpoint_filename(checkpoint, pipeline_name=pm.name)
    return pipeline_filepath(pm, filename=chkpt_name)


Instruction:
Given the following Python 3 function, write the documentation
def check_shell(cmd, shell=None):
    if isinstance(shell, bool):
        return shell
    return "|" in cmd or ">" in cmd or r"*" in cmd

Response: Checks whether a command appears to involve a shell process.


Instruction:
Write a Python 3 script for
checking whether a command appears to contain shell redirection symbol outside of curly brackets.

Response:
def check_shell_redirection(cmd):
    """
    Determine whether a command appears to contain shell redirection symbol outside of curly brackets

    :param str cmd: Command to investigate.
    :return bool: Whether the command appears to contain shell redirection.
    """
    curly_brackets = True
    while curly_brackets:
        SRE_match_obj = re.search(r'\{(.*?)}',cmd)
        if SRE_match_obj is not None:
            cmd = cmd[:SRE_match_obj.start()] + cmd[(SRE_match_obj.end()+1):]
            if re.search(r'\{(.*?)}',cmd) is None:
                curly_brackets = False
        else:
            curly_brackets = False
    return ">" in cmd


Instruction:
Can you generate the documentation for the following Python 3 function
def get_proc_name(cmd):
    if isinstance(cmd, Iterable) and not isinstance(cmd, str):
        cmd = " ".join(cmd)

    return cmd.split()[0].replace('(', '').replace(')', '')

Response: Get the representative process name from complex command sequence


Instruction:
Write a Python 3 function for
returning the first value for a particular parameter from the first pool of parameter pools.

Response:
def get_first_value(param, param_pools, on_missing=None, error=True):
    """
    Get the value for a particular parameter from the first pool in the provided
    priority list of parameter pools.

    :param str param: Name of parameter for which to determine/fetch value.
    :param Sequence[Mapping[str, object]] param_pools: Ordered (priority)
        collection of mapping from parameter name to value; this should be
        ordered according to descending priority.
    :param object | function(str) -> object on_missing: default value or
        action to take if the requested parameter is missing from all of the
        pools. If a callable, it should return a value when passed the
        requested parameter as the one and only argument.
    :param bool error: Whether to raise an error if the requested parameter
        is not mapped to a value AND there's no value or strategy provided
        with 'on_missing' with which to handle the case of a request for an
        unmapped parameter.
    :return object: Value to which the requested parameter first mapped in
        the (descending) priority collection of parameter 'pools,' or
        a value explicitly defined or derived with 'on_missing.'
    :raise KeyError: If the requested parameter is unmapped in all of the
        provided pools, and the argument to the 'error' parameter evaluates
        to True.
    """

    # Search for the requested parameter.
    for pool in param_pools:
        if param in pool:
            return pool[param]

    # Raise error if unfound and no strategy or value is provided or handling
    # unmapped parameter requests.
    if error and on_missing is None:
        raise KeyError("Unmapped parameter: '{}'".format(param))

    # Use the value or strategy for handling unmapped parameter case.
    try:
        return on_missing(param)
    except TypeError:
        if hasattr(on_missing, "__call__"):
            raise TypeError(
                "Any callable passed as the action to take when a requested "
                "parameter is missing should accept that parameter and return "
                "a value.")
        return on_missing


Instruction:
Here you have a function in Python 3, explain what it does
def is_in_file_tree(fpath, folder):
    file_folder, _ = os.path.split(fpath)
    other_folder = os.path.join(folder, "")
    return other_folder.startswith(file_folder)

Response: Determines whether a file is in a folder.


Instruction:
Implement a Python 3 function for
determining whether indicated file appears to be gzipped FASTQ.

Response:
def is_gzipped_fastq(file_name):
    """
    Determine whether indicated file appears to be a gzipped FASTQ.

    :param str file_name: Name/path of file to check as gzipped FASTQ.
    :return bool: Whether indicated file appears to be in gzipped FASTQ format.
    """
    _, ext = os.path.splitext(file_name)
    return file_name.endswith(".fastq.gz") or file_name.endswith(".fq.gz")


Instruction:
How would you explain what the following Python 3 function does
def make_lock_name(original_path, path_base_folder):
    def make_name(p):
        return p.replace(path_base_folder, "").replace(os.sep, "__")
    if isinstance(original_path, str):
        return make_name(original_path)
    elif isinstance(original_path, Sequence):
        return [make_name(p) for p in original_path]
    raise TypeError("Neither string nor other sequence type: {} ({})".
                    format(original_path, type(original_path)))

Response: Create a name for a lock file from an absolute path.


Instruction:
Given the following Python 3 function, write the documentation
def is_multi_target(target):
    if target is None or isinstance(target, str):
        return False
    elif isinstance(target, Sequence):
        return len(target) > 1
    else:
        raise TypeError("Could not interpret argument as a target: {} ({})".
                        format(target, type(target)))

Response: Determines if the pipeline manager s run target is multiple.


Instruction:
Can you generate the documentation for the following Python 3 function
def parse_stage_name(stage):
    if isinstance(stage, str):
        return stage
    try:
        return stage.name
    except AttributeError:
        try:
            return stage.__name__
        except AttributeError:
            raise TypeError("Unsupported stage type: {}".format(type(stage)))

Response: Determines the name of a stage.


Instruction:
How would you explain what the following Python 3 function does
def pipeline_filepath(pm, filename=None, suffix=None):

    if filename is None and suffix is None:
        raise TypeError("Provide filename and/or suffix to create "
                        "path to a pipeline file.")

    filename = (filename or pm.name) + (suffix or "")

    # Note that Pipeline and PipelineManager define the same outfolder.
    # In fact, a Pipeline just references its manager's outfolder.
    # So we can handle argument of either type to pm parameter.
    return filename if os.path.isabs(filename) \
        else os.path.join(pm.outfolder, filename)

Response: Derive path to a file within a managed pipeline.


Instruction:
Write a Python 3 script for
translating a stage name into a standardized pipeline phase name.

Response:
def translate_stage_name(stage):
    """
    Account for potential variability in stage/phase name definition.

    Since a pipeline author is free to name his/her processing phases/stages
    as desired, but these choices influence file names, enforce some
    standardization. Specifically, prohibit potentially problematic spaces.

    :param str | pypiper.Stage | function stage: Pipeline stage, its name, or a
        representative function.
    :return str: Standardized pipeline phase/stage name.
    """
    # First ensure that we have text.
    name = parse_stage_name(stage)
    # Cast to string to ensure that indexed stages (ints are handled).
    return str(name).lower().replace(" ", STAGE_NAME_SPACE_REPLACEMENT)


Instruction:
Create a Python 3 function for
determining the arguments to add to a parser (for a pipeline). :param Iterable[str] | str argument_groups: Collection of names of groups of arguments to add to an argument parser. :param Iterable[str] | str arguments: Collection of specific arguments to add to the parser. :param bool use_all_args: Whether to use all arguments defined here. :return set[str]: Collection of (unique) argument names to add to a parser.

Response:
def _determine_args(argument_groups, arguments, use_all_args=False):
    """
    Determine the arguments to add to a parser (for a pipeline).

    :param Iterable[str] | str argument_groups: Collection of names of groups
        of arguments to add to an argument parser.
    :param Iterable[str] | str arguments: Collection of specific arguments to
        add to the parser.
    :param bool use_all_args: Whether to use all arguments defined here.
    :return set[str]: Collection of (unique) argument names to add to a parser.
    """

    if sys.version_info < (3, 3):
        from collections import Iterable
    else:
        from collections.abc import Iterable

    # Define the argument groups.
    args_by_group = {
        "pypiper": ["recover", "new-start", "dirty", "force-follow"],
        "config": ["config"],
        "checkpoint": ["stop-before", "stop-after"],
        "resource": ["mem", "cores"],
        "looper": ["config", "output-parent", "mem", "cores"],
        "common": ["input", "sample-name"],
        "ngs": ["sample-name", "input", "input2", "genome", "single-or-paired"]
    }

    # Handle various types of group specifications.
    groups = None
    if use_all_args:
        groups = args_by_group.keys()
    elif isinstance(argument_groups, str):
        groups = [argument_groups]
    elif isinstance(argument_groups, Iterable):
        groups = argument_groups
    elif argument_groups:
        raise TypeError("arguments must be a str or a list.")

    # Collect the groups of arguments.
    final_args = list()
    if groups:
        for g in groups:
            try:
                this_group_args = args_by_group[g]
            except KeyError:
                print("Skipping undefined pypiper argument group '{}'".format(g))
            else:
                final_args.extend(this_group_args)
                # final_args |= {this_group_args} if \
                #     isinstance(this_group_args, str) else set(this_group_args)

    # Handle various types of specific, individual argument specifications.
    if isinstance(arguments, str):
        final_args.append(arguments)
    elif isinstance(arguments, Iterable):
        final_args.extend(arguments)
    elif arguments:
        raise TypeError("arguments must be a str or a list.")

    return uniqify(final_args)


Instruction:
Given the following Python 3 function, write the documentation
def _add_args(parser, args, required):

    import copy

    required = required or []

    # Determine the default pipeline config file.
    pipeline_script = os.path.basename(sys.argv[0])
    default_config, _ = os.path.splitext(pipeline_script)
    default_config += ".yaml"

    # Define the arguments.
    argument_data = {
        "recover":
            ("-R", {"action": "store_true",
                    "help": "Overwrite locks to recover from previous failed run"}),
        "new-start":
            ("-N", {"action": "store_true",
                    "help": "Overwrite all results to start a fresh run"}),
        "dirty":
            ("-D", {"action": "store_true",
                    "help": "Don't auto-delete intermediate files"}),
        "force-follow":
            ("-F", {"action": "store_true",
                    "help": "Always run 'follow' commands"}),
        "start-point":
            {"help": "Name of pipeline stage at which to begin"},
        "stop-before":
            {"help": "Name of pipeline stage at which to stop "
                     "(exclusive, i.e. not run)"},
        "stop-after":
            {"help": "Name of pipeline stage at which to stop "
                     "(inclusive, i.e. run)"},
        "config":
            ("-C", {"dest": "config_file", "metavar": "CONFIG_FILE",
                    "default": default_config,
                    "help": "Pipeline configuration file (YAML). "
                            "Relative paths are with respect to the "
                            "pipeline script."}),
        "sample-name":
            ("-S", {"metavar": "SAMPLE_NAME",
                    "help": "Name for sample to run"}),
        "output-parent":
            ("-O", {"metavar": "PARENT_OUTPUT_FOLDER",
                    "help": "Parent output directory of project"}),
        "cores":
            ("-P", {"type": int, "default": 1, "metavar": "NUMBER_OF_CORES",
                    "help": "Number of cores for parallelized processes"}),
        "mem":
            ("-M", {"default": "4000", "metavar": "MEMORY_LIMIT",
                    "help": "Memory limit for processes accepting such. "
                    "Default units are megabytes unless specified "
                    "using the suffix [K|M|G|T]."}),
        "input":
            ("-I", {"nargs": "+", "metavar": "INPUT_FILES",
                    "help": "One or more primary input files"}),
        "input2":
            ("-I2", {"nargs": "*", "metavar": "INPUT_FILES2",
                     "help": "Secondary input files, such as read2"}),
        "genome":
            ("-G", {"dest": "genome_assembly",
                    "help": "Identifier for genome assembly"}),
        "single-or-paired":
            ("-Q", {"default": "single",
                    "help": "Single- or paired-end sequencing protocol"})
    }

    if len(required) > 0:
        required_named = parser.add_argument_group('required named arguments')

    # Configure the parser for each argument.
    for arg in args:
        try:
            argdata = copy.deepcopy(argument_data[arg])
        except KeyError:
            print("Skipping undefined pypiper argument: '{}'".format(arg))
            continue
        if isinstance(argdata, dict):
            short_opt = None
        else:
            try:
                short_opt, argdata = argdata
            except ValueError:
                raise TypeError(
                    "Option name must map to dict or two-tuple (short "
                    "name and dict) of argument command-line argument "
                    "specification data.")

        argdata["required"] = arg in required

        long_opt = "--{}".format(arg)
        opts = (short_opt, long_opt) if short_opt else (long_opt, )
        if arg in required:
            required_named.add_argument(*opts, **argdata)
        else:
            parser.add_argument(*opts, **argdata)

    return parser

Response: Add new arguments to an ArgumentParser.


Instruction:
Can you implement a function in Python 3 that
ensures that paths to folders exist.

Response:
def _ensure_folders(self, *paths):
        """
        Ensure that paths to folder(s) exist.

        Some command-line tools will not attempt to create folder(s) needed
        for output path to exist. They instead assume that they already are
        present and will fail if that assumption does not hold.

        :param Iterable[str] paths: Collection of path for which
        """
        for p in paths:
            # Only provide assurance for absolute paths.
            if not p or not os.path.isabs(p):
                continue
            # See if what we're assuring is file- or folder-like.
            fpath, fname = os.path.split(p)
            base, ext = os.path.splitext(fname)
            # If there's no extension, ensure that we have the whole path.
            # Otherwise, just ensure that we have path to file's folder.
            self.make_dir(fpath if ext else p)


Instruction:
Implement a Python 3 function for
checking if command can be called.

Response:
def check_command(self, command):
        """
        Check if command can be called.
        """

        # Use `command` to see if command is callable, store exit code
        code = os.system("command -v {0} >/dev/null 2>&1 || {{ exit 1; }}".format(command))

        # If exit code is not 0, report which command failed and return False, else return True
        if code != 0:
            print("Command is not callable: {0}".format(command))
            return False
        else:
            return True


Instruction:
Can you implement a function in Python 3 that
gets size of all files in string ( space - separated ) in megabytes ( Mb.

Response:
def get_file_size(self, filenames):
        """
        Get size of all files in string (space-separated) in megabytes (Mb).

        :param str filenames: a space-separated string of filenames
        """
        # use (1024 ** 3) for gigabytes
        # equivalent to: stat -Lc '%s' filename

        # If given a list, recurse through it.
        if type(filenames) is list:
            return sum([self.get_file_size(filename) for filename in filenames])

        return round(sum([float(os.stat(f).st_size) for f in filenames.split(" ")]) / (1024 ** 2), 4)


Instruction:
Write a Python 3 function that can
convert BAM file to FASTQ.

Response:
def bam2fastq(self, input_bam, output_fastq,
                  output_fastq2=None, unpaired_fastq=None):
        """
        Create command to convert BAM(s) to FASTQ(s).

        :param str input_bam: Path to sequencing reads file to convert
        :param output_fastq: Path to FASTQ to write
        :param output_fastq2: Path to (R2) FASTQ to write
        :param unpaired_fastq: Path to unpaired FASTQ to write
        :return str: Command to convert BAM(s) to FASTQ(s)
        """
        self._ensure_folders(output_fastq, output_fastq2, unpaired_fastq)
        cmd = self.tools.java + " -Xmx" + self.pm.javamem
        cmd += " -jar " + self.tools.picard + " SamToFastq"
        cmd += " INPUT={0}".format(input_bam)
        cmd += " FASTQ={0}".format(output_fastq)
        if output_fastq2 is not None and unpaired_fastq is not None:
            cmd += " SECOND_END_FASTQ={0}".format(output_fastq2)
            cmd += " UNPAIRED_FASTQ={0}".format(unpaired_fastq)
        return cmd


Instruction:
Can you generate a brief explanation for the following Python 3 code
def bam_to_fastq(self, bam_file, out_fastq_pre, paired_end):
        self.make_sure_path_exists(os.path.dirname(out_fastq_pre))
        cmd = self.tools.java + " -Xmx" + self.pm.javamem
        cmd += " -jar " + self.tools.picard + " SamToFastq"
        cmd += " I=" + bam_file
        cmd += " F=" + out_fastq_pre + "_R1.fastq"
        if paired_end:
            cmd += " F2=" + out_fastq_pre + "_R2.fastq"
        cmd += " INCLUDE_NON_PF_READS=true"
        cmd += " QUIET=true"
        cmd += " VERBOSITY=ERROR"
        cmd += " VALIDATION_STRINGENCY=SILENT"
        return cmd

Response: Build command to convert BAM file to FASTQ file


Instruction:
Can you generate a brief explanation for the following Python 3 code
def bam_to_fastq_awk(self, bam_file, out_fastq_pre, paired_end):
        self.make_sure_path_exists(os.path.dirname(out_fastq_pre))
        fq1 = out_fastq_pre + "_R1.fastq"
        if paired_end:
            fq2 = out_fastq_pre + "_R2.fastq"
            cmd = self.tools.samtools + " view " + bam_file + " | awk '"
            cmd += r'{ if (NR%2==1) print "@"$1"/1\n"$10"\n+\n"$11 > "' + fq1 + '";'
            cmd += r' else print "@"$1"/2\n"$10"\n+\n"$11 > "' + fq2 + '"; }'
            cmd += "'"  # end the awk command
        else:
            fq2 = None
            cmd = self.tools.samtools + " view " + bam_file + " | awk '"
            cmd += r'{ print "@"$1"\n"$10"\n+\n"$11 > "' + fq1 + '"; }'
            cmd += "'"
        return cmd, fq1, fq2

Response: This function converts a BAM file to fastq files using awk.


Instruction:
How would you implement a function in Python 3 that
converts bam to fastq using bedtools

Response:
def bam_to_fastq_bedtools(self, bam_file, out_fastq_pre, paired_end):
        """
        Converts bam to fastq; A version using bedtools
        """
        self.make_sure_path_exists(os.path.dirname(out_fastq_pre))
        fq1 = out_fastq_pre + "_R1.fastq"
        fq2 = None
        cmd = self.tools.bedtools + " bamtofastq -i " + bam_file + " -fq " + fq1 + ".fastq"
        if paired_end:
            fq2 = out_fastq_pre + "_R2.fastq"
            cmd += " -fq2 " + fq2

        return cmd, fq1, fq2


Instruction:
Create a Python 3 function for
getting the extension of the input file. Assumes you re using either. bam or. fastq. gz. fq or. fq. gz. gz.

Response:
def get_input_ext(self, input_file):
        """
        Get the extension of the input_file. Assumes you're using either
        .bam or .fastq/.fq or .fastq.gz/.fq.gz.
        """
        if input_file.endswith(".bam"):
            input_ext = ".bam"
        elif input_file.endswith(".fastq.gz") or input_file.endswith(".fq.gz"):
            input_ext = ".fastq.gz"
        elif input_file.endswith(".fastq") or input_file.endswith(".fq"):
            input_ext = ".fastq"
        else:
            errmsg = "'{}'; this pipeline can only deal with .bam, .fastq, " \
                     "or .fastq.gz files".format(input_file)
            raise UnsupportedFiletypeException(errmsg)
        return input_ext


Instruction:
Make a summary of the following Python 3 code
def merge_or_link(self, input_args, raw_folder, local_base="sample"):
        self.make_sure_path_exists(raw_folder)

        if not isinstance(input_args, list):
            raise Exception("Input must be a list")

        if any(isinstance(i, list) for i in input_args):
            # We have a list of lists. Process each individually.
            local_input_files = list()
            n_input_files = len(filter(bool, input_args))
            print("Number of input file sets:\t\t" + str(n_input_files))

            for input_i, input_arg in enumerate(input_args):
                # Count how many non-null items there are in the list;
                # we only append _R1 (etc.) if there are multiple input files.
                if n_input_files > 1:
                    local_base_extended = local_base + "_R" + str(input_i + 1)
                else:
                    local_base_extended = local_base
                if input_arg:
                    out = self.merge_or_link(
                            input_arg, raw_folder, local_base_extended)

                    print("Local input file: '{}'".format(out))
                    # Make sure file exists:
                    if not os.path.isfile(out):
                        print("Not a file: '{}'".format(out))

                    local_input_files.append(out)

            return local_input_files

        else:
            # We have a list of individual arguments. Merge them.

            if len(input_args) == 1:
                # Only one argument in this list. A single input file; we just link
                # it, regardless of file type:
                # Pull the value out of the list
                input_arg = input_args[0]
                input_ext = self.get_input_ext(input_arg)

                # Convert to absolute path
                if not os.path.isabs(input_arg):
                    input_arg = os.path.abspath(input_arg)

                # Link it to into the raw folder
                local_input_abs = os.path.join(raw_folder, local_base + input_ext)
                self.pm.run(
                    "ln -sf " + input_arg + " " + local_input_abs,
                    target=local_input_abs,
                    shell=True)
                # return the local (linked) filename absolute path
                return local_input_abs

            else:
                # Otherwise, there are multiple inputs.
                # If more than 1 input file is given, then these are to be merged
                # if they are in bam format.
                if all([self.get_input_ext(x) == ".bam" for x in input_args]):
                    sample_merged = local_base + ".merged.bam"
                    output_merge = os.path.join(raw_folder, sample_merged)
                    cmd = self.merge_bams(input_args, output_merge)
                    self.pm.run(cmd, output_merge)
                    cmd2 = self.validate_bam(output_merge)
                    self.pm.run(cmd, output_merge, nofail=True)
                    return output_merge

                # if multiple fastq
                if all([self.get_input_ext(x) == ".fastq.gz" for x in input_args]):
                    sample_merged_gz = local_base + ".merged.fastq.gz"
                    output_merge_gz = os.path.join(raw_folder, sample_merged_gz)
                    #cmd1 = self.ziptool + "-d -c " + " ".join(input_args) + " > " + output_merge
                    #cmd2 = self.ziptool + " " + output_merge
                    #self.pm.run([cmd1, cmd2], output_merge_gz)
                    # you can save yourself the decompression/recompression:
                    cmd = "cat " + " ".join(input_args) + " > " + output_merge_gz
                    self.pm.run(cmd, output_merge_gz)
                    return output_merge_gz

                if all([self.get_input_ext(x) == ".fastq" for x in input_args]):
                    sample_merged = local_base + ".merged.fastq"
                    output_merge = os.path.join(raw_folder, sample_merged)
                    cmd = "cat " + " ".join(input_args) + " > " + output_merge
                    self.pm.run(cmd, output_merge)
                    return output_merge

                # At this point, we don't recognize the input file types or they
                # do not match.
                raise NotImplementedError(
                        "Input files must be of the same type, and can only "
                        "merge bam or fastq.")

Response: This function merges the input files into a local file.


Instruction:
Can you generate the documentation for the following Python 3 function
def input_to_fastq(
        self, input_file, sample_name,
        paired_end, fastq_folder, output_file=None, multiclass=False):
        """
        Builds a command to convert input file to fastq, for various inputs.

        Takes either .bam, .fastq.gz, or .fastq input and returns
        commands that will create the .fastq file, regardless of input type.
        This is useful to made your pipeline easily accept any of these input
        types seamlessly, standardizing you to the fastq which is still the
        most common format for adapter trimmers, etc.

        It will place the output fastq file in given `fastq_folder`.

        :param str input_file: filename of input you want to convert to fastq
        :return str: A command (to be run with PipelineManager) that will ensure
            your fastq file exists.
        """

        fastq_prefix = os.path.join(fastq_folder, sample_name)
        self.make_sure_path_exists(fastq_folder)

        # this expects a list; if it gets a string, convert it to a list.
        if type(input_file) != list:
            input_file = [input_file]

        if len(input_file) > 1:
            cmd = []
            output_file = []
            for in_i, in_arg in enumerate(input_file):
                output = fastq_prefix + "_R" + str(in_i + 1) + ".fastq"
                result_cmd, uf, result_file = \
                    self.input_to_fastq(in_arg, sample_name, paired_end,
                                        fastq_folder, output, multiclass=True)
                cmd.append(result_cmd)
                output_file.append(result_file)

        else:
            # There was only 1 input class.
            # Convert back into a string
            input_file = input_file[0]
            if not output_file:
                output_file = fastq_prefix + "_R1.fastq"
            input_ext = self.get_input_ext(input_file)

            if input_ext == ".bam":
                print("Found .bam file")
                #cmd = self.bam_to_fastq(input_file, fastq_prefix, paired_end)
                cmd, fq1, fq2 = self.bam_to_fastq_awk(input_file, fastq_prefix, paired_end)
                # pm.run(cmd, output_file, follow=check_fastq)
            elif input_ext == ".fastq.gz":
                print("Found .fastq.gz file")
                if paired_end and not multiclass:
                    # For paired-end reads in one fastq file, we must split the file into 2.
                    script_path = os.path.join(
                            self.tools.scripts_dir, "fastq_split.py")
                    cmd = self.tools.python + " -u " + script_path
                    cmd += " -i " + input_file
                    cmd += " -o " + fastq_prefix
                    # Must also return the set of output files
                    output_file = [fastq_prefix + "_R1.fastq", fastq_prefix + "_R2.fastq"]
                else:
                    # For single-end reads, we just unzip the fastq.gz file.
                    # or, paired-end reads that were already split.
                    cmd = self.ziptool + " -d -c " + input_file + " > " + output_file
                    # a non-shell version
                    # cmd1 = "gunzip --force " + input_file
                    # cmd2 = "mv " + os.path.splitext(input_file)[0] + " " + output_file
                    # cmd = [cmd1, cmd2]
            elif input_ext == ".fastq":
                cmd = "ln -sf " + input_file + " " + output_file
                print("Found .fastq file; no conversion necessary")

        return [cmd, fastq_prefix, output_file]

Response: This function converts the input file to fastq.


Instruction:
Implement a function in Python 3 to
return a follow sanity-check function to be run after a fastq conversion. Run following a command that will produce the fastq files. This function will make sure any input files have the same number of reads as the output files.

Response:
def check_fastq(self, input_files, output_files, paired_end):
        """
        Returns a follow sanity-check function to be run after a fastq conversion.
        Run following a command that will produce the fastq files.

        This function will make sure any input files have the same number of reads as the
        output files.
        """

        # Define a temporary function which we will return, to be called by the
        # pipeline.
        # Must define default parameters here based on the parameters passed in. This locks
        # these values in place, so that the variables will be defined when this function
        # is called without parameters as a follow function by pm.run.

        # This is AFTER merge, so if there are multiple files it means the
        # files were split into read1/read2; therefore I must divide by number
        # of files for final reads.
        def temp_func(input_files=input_files, output_files=output_files,
                      paired_end=paired_end):

            if type(input_files) != list:
                input_files = [input_files]
            if type(output_files) != list:
                output_files = [output_files]

            print(input_files)
            print(output_files)

            n_input_files = len(filter(bool, input_files))

            total_reads = sum([int(self.count_reads(input_file, paired_end))
                               for input_file in input_files])
            raw_reads = total_reads / n_input_files
            self.pm.report_result("Raw_reads", str(raw_reads))

            total_fastq_reads = sum(
                [int(self.count_reads(output_file, paired_end))
                 for output_file in output_files])
            fastq_reads = total_fastq_reads / n_input_files

            self.pm.report_result("Fastq_reads", fastq_reads)
            input_ext = self.get_input_ext(input_files[0])
            # We can only assess pass filter reads in bam files with flags.
            if input_ext == ".bam":
                num_failed_filter = sum(
                    [int(self.count_fail_reads(f, paired_end))
                     for f in input_files])
                pf_reads = int(raw_reads) - num_failed_filter
                self.pm.report_result("PF_reads", str(pf_reads))
            if fastq_reads != int(raw_reads):
                raise Exception("Fastq conversion error? Number of reads "
                                "doesn't match unaligned bam")

            return fastq_reads

        return temp_func


Instruction:
Write a Python 3 script to
evaluate read trimming and optionally run fastqc if necessary.

Response:
def check_trim(self, trimmed_fastq, paired_end, trimmed_fastq_R2=None, fastqc_folder=None):
        """
        Build function to evaluate read trimming, and optionally run fastqc.

        This is useful to construct an argument for the 'follow' parameter of
        a PipelineManager's 'run' method.

        :param str trimmed_fastq: Path to trimmed reads file.
        :param bool paired_end: Whether the processing is being done with
            paired-end sequencing data.
        :param str trimmed_fastq_R2: Path to read 2 file for the paired-end case.
        :param str fastqc_folder: Path to folder within which to place fastqc
            output files; if unspecified, fastqc will not be run.
        :return callable: Function to evaluate read trimming and possibly run
            fastqc.
        """

        def temp_func():

            print("Evaluating read trimming")

            if paired_end and not trimmed_fastq_R2:
                print("WARNING: specified paired-end but no R2 file")

            n_trim = float(self.count_reads(trimmed_fastq, paired_end))
            self.pm.report_result("Trimmed_reads", int(n_trim))
            try:
                rr = float(self.pm.get_stat("Raw_reads"))
            except:
                print("Can't calculate trim loss rate without raw read result.")
            else:
                self.pm.report_result(
                    "Trim_loss_rate", round((rr - n_trim) * 100 / rr, 2))

            # Also run a fastqc (if installed/requested)
            if fastqc_folder:
                if fastqc_folder and os.path.isabs(fastqc_folder):
                    self.make_sure_path_exists(fastqc_folder)
                cmd = self.fastqc(trimmed_fastq, fastqc_folder)
                self.pm.run(cmd, lock_name="trimmed_fastqc", nofail=True)
                fname, ext = os.path.splitext(os.path.basename(trimmed_fastq))
                fastqc_html = os.path.join(fastqc_folder, fname + "_fastqc.html")
                self.pm.report_object("FastQC report r1", fastqc_html)

                if paired_end and trimmed_fastq_R2:
                    cmd = self.fastqc(trimmed_fastq_R2, fastqc_folder)
                    self.pm.run(cmd, lock_name="trimmed_fastqc_R2", nofail=True)
                    fname, ext = os.path.splitext(os.path.basename(trimmed_fastq_R2))
                    fastqc_html = os.path.join(fastqc_folder, fname + "_fastqc.html")
                    self.pm.report_object("FastQC report r2", fastqc_html)

        return temp_func


Instruction:
Given the following Python 3 function, write the documentation
def validate_bam(self, input_bam):
        cmd = self.tools.java + " -Xmx" + self.pm.javamem
        cmd += " -jar " + self.tools.picard + " ValidateSamFile"
        cmd += " INPUT=" + input_bam
        return cmd

Response: Wrapper for Picard s ValidateSamFile. validate_bam wrapper for Picard s ValidateSamFile. validate_bam


Instruction:
Here you have a function in Python 3, explain what it does
def merge_bams(self, input_bams, merged_bam, in_sorted="TRUE", tmp_dir=None):
        if not len(input_bams) > 1:
            print("No merge required")
            return 0

        outdir, _ = os.path.split(merged_bam)
        if outdir and not os.path.exists(outdir):
            print("Creating path to merge file's folder: '{}'".format(outdir))
            os.makedirs(outdir)

        # Handle more intuitive boolean argument.
        if in_sorted in [False, True]:
            in_sorted = "TRUE" if in_sorted else "FALSE"

        input_string = " INPUT=" + " INPUT=".join(input_bams)
        cmd = self.tools.java + " -Xmx" + self.pm.javamem
        cmd += " -jar " + self.tools.picard + " MergeSamFiles"
        cmd += input_string
        cmd += " OUTPUT=" + merged_bam
        cmd += " ASSUME_SORTED=" + str(in_sorted)
        cmd += " CREATE_INDEX=TRUE"
        cmd += " VALIDATION_STRINGENCY=SILENT"
        if tmp_dir:
            cmd += " TMP_DIR=" + tmp_dir
        return cmd

Response: Combine multiple bams into one.


Instruction:
Can you generate the documentation for the following Python 3 function
def merge_fastq(self, inputs, output, run=False, remove_inputs=False):
        if remove_inputs and not run:
            raise ValueError("Can't delete files if command isn't run")
        cmd = "cat {} > {}".format(" ".join(inputs), output)
        if run:
            subprocess.check_call(cmd.split(), shell=True)
            if remove_inputs:
                cmd = "rm {}".format(" ".join(inputs))
                subprocess.check_call(cmd.split(), shell=True)
        else:
            return cmd

Response: Merge FASTQ files into one.


Instruction:
Here you have a function in Python 3, explain what it does
def count_lines(self, file_name):
        x = subprocess.check_output("wc -l " + file_name + " | sed -E 's/^[[:space:]]+//' | cut -f1 -d' '", shell=True)
        return x.strip()

Response: Uses the command - line utility wc to count the number of lines in a file. For MacOS this is not a good idea.


Instruction:
Can you tell what is the following Python 3 function doing
def get_chrs_from_bam(self, file_name):
        x = subprocess.check_output(self.tools.samtools + " view -H " + file_name + " | grep '^@SQ' | cut -f2| sed s'/SN://'", shell=True)
        # Chromosomes will be separated by newlines; split into list to return
        return x.split()

Response:
Uses samtools to grab the chromosomes from the header that are contained
        in this bam file.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def count_unique_reads(self, file_name, paired_end):
        if file_name.endswith("sam"):
            param = "-S"
        if file_name.endswith("bam"):
            param = ""
        if paired_end:
            r1 = self.samtools_view(file_name, param=param + " -f64", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
            r2 = self.samtools_view(file_name, param=param + " -f128", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
        else:
            r1 = self.samtools_view(file_name, param=param + "", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
            r2 = 0
        return int(r1) + int(r2)

Response: Counts the number of unique reads in a file.


Instruction:
Given the following Python 3 function, write the documentation
def count_unique_mapped_reads(self, file_name, paired_end):

        _, ext = os.path.splitext(file_name)
        ext = ext.lower()

        if ext == ".sam":
            param = "-S -F4"
        elif ext == "bam":
            param = "-F4"
        else:
            raise ValueError("Not a SAM or BAM: '{}'".format(file_name))

        if paired_end:
            r1 = self.samtools_view(file_name, param=param + " -f64", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
            r2 = self.samtools_view(file_name, param=param + " -f128", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
        else:
            r1 = self.samtools_view(file_name, param=param + "", postpend=" | cut -f1 | sort -k1,1 -u | wc -l | sed -E 's/^[[:space:]]+//'")
            r2 = 0

        return int(r1) + int(r2)

Response: Returns the number of unique mapped reads for a file.


Instruction:
Implement a function in Python 3 to
count the number of reads with the specified flag.

Response:
def count_flag_reads(self, file_name, flag, paired_end):
        """
        Counts the number of reads with the specified flag.

        :param str file_name: name of reads file
        :param str flag: sam flag value to be read
        :param bool paired_end: This parameter is ignored; samtools automatically correctly responds depending
            on the data in the bamfile. We leave the option here just for consistency, since all the other
            counting functions require the parameter. This makes it easier to swap counting functions during
            pipeline development.
        """

        param = " -c -f" + str(flag)
        if file_name.endswith("sam"):
            param += " -S"
        return self.samtools_view(file_name, param=param)


Instruction:
Given the following Python 3 function, write the documentation
def count_uniquelymapping_reads(self, file_name, paired_end):
        param = " -c -F256"
        if file_name.endswith("sam"):
            param += " -S"
        return self.samtools_view(file_name, param=param)

Response: Counts the number of reads that mapped to a unique position.


Instruction:
Write a Python 3 script to
run samtools view on the specified file

Response:
def samtools_view(self, file_name, param, postpend=""):
        """
        Run samtools view, with flexible parameters and post-processing.

        This is used internally to implement the various count_reads functions.

        :param str file_name: file_name
        :param str param: String of parameters to pass to samtools view
        :param str postpend: String to append to the samtools command;
            useful to add cut, sort, wc operations to the samtools view output.
        """
        cmd = "{} view {} {} {}".format(
                self.tools.samtools, param, file_name, postpend)
        return subprocess.check_output(cmd, shell=True)


Instruction:
Can you write a function in Python 3 where it
counts the number of reads in a file.

Response:
def count_reads(self, file_name, paired_end):
        """
        Count reads in a file.

        Paired-end reads count as 2 in this function.
        For paired-end reads, this function assumes that the reads are split
        into 2 files, so it divides line count by 2 instead of 4.
        This will thus give an incorrect result if your paired-end fastq files
        are in only a single file (you must divide by 2 again).

        :param str file_name: Name/path of file whose reads are to be counted.
        :param bool paired_end: Whether the file contains paired-end reads.
        """

        _, ext = os.path.splitext(file_name)
        if not (is_sam_or_bam(file_name) or is_fastq(file_name)):
            # TODO: make this an exception and force caller to handle that
            # rather than relying on knowledge of possibility of negative value.
            return -1

        if is_sam_or_bam(file_name):
            param_text = "-c" if ext == ".bam" else "-c -S"
            return self.samtools_view(file_name, param=param_text)
        else:
            num_lines = self.count_lines_zip(file_name) \
                    if is_gzipped_fastq(file_name) \
                    else self.count_lines(file_name)
            divisor = 2 if paired_end else 4
            return int(num_lines) / divisor


Instruction:
Can you generate the documentation for the following Python 3 function
def count_concordant(self, aligned_bam):
        cmd = self.tools.samtools + " view " + aligned_bam + " | "
        cmd += "grep 'YT:Z:CP'" + " | uniq -u | wc -l | sed -E 's/^[[:space:]]+//'"

        return subprocess.check_output(cmd, shell=True)

Response: Count only reads that aligned concordantly exactly 1 time.


Instruction:
Create a Python 3 function to
count the number of mapped reads for a specific file.

Response:
def count_mapped_reads(self, file_name, paired_end):
        """
        Mapped_reads are not in fastq format, so this one doesn't need to accommodate fastq,
        and therefore, doesn't require a paired-end parameter because it only uses samtools view.
        Therefore, it's ok that it has a default parameter, since this is discarded.

        :param str file_name: File for which to count mapped reads.
        :param bool paired_end: This parameter is ignored; samtools automatically correctly responds depending
            on the data in the bamfile. We leave the option here just for consistency, since all the other
            counting functions require the parameter. This makes it easier to swap counting functions during
            pipeline development.
        :return int: Either return code from samtools view command, or -1 to indicate an error state.
        """
        if file_name.endswith("bam"):
            return self.samtools_view(file_name, param="-c -F4")
        if file_name.endswith("sam"):
            return self.samtools_view(file_name, param="-c -F4 -S")
        return -1