Instruction:
How would you explain what the following Python 3 function does
def plot(
            self, data, bbox=None, plot_type='scatter',
            fig_kwargs=None, bmap_kwargs=None, plot_kwargs=None,
            cbar_kwargs=None):
        """
        Plot an array of data on a map using matplotlib and Basemap,
        automatically matching the data to the Pandana network node positions.

        Keyword arguments are passed to the plotting routine.

        Parameters
        ----------
        data : pandas.Series
            Numeric data with the same length and index as the nodes
            in the network.
        bbox : tuple, optional
            (lat_min, lng_min, lat_max, lng_max)
        plot_type : {'hexbin', 'scatter'}, optional
        fig_kwargs : dict, optional
            Keyword arguments that will be passed to
            matplotlib.pyplot.subplots. Use this to specify things like
            figure size or background color.
        bmap_kwargs : dict, optional
            Keyword arguments that will be passed to the Basemap constructor.
            This can be used to specify a projection or coastline resolution.
        plot_kwargs : dict, optional
            Keyword arguments that will be passed to the matplotlib plotting
            command used. Use this to control plot styles and color maps used.
        cbar_kwargs : dict, optional
            Keyword arguments passed to the Basemap.colorbar method.
            Use this to control color bar location and label.

        Returns
        -------
        bmap : Basemap
        fig : matplotlib.Figure
        ax : matplotlib.Axes

        """
        from mpl_toolkits.basemap import Basemap

        fig_kwargs = fig_kwargs or {}
        bmap_kwargs = bmap_kwargs or {}
        plot_kwargs = plot_kwargs or {}
        cbar_kwargs = cbar_kwargs or {}

        if not bbox:
            bbox = (
                self.nodes_df.y.min(),
                self.nodes_df.x.min(),
                self.nodes_df.y.max(),
                self.nodes_df.x.max())

        fig, ax = plt.subplots(**fig_kwargs)

        bmap = Basemap(
            bbox[1], bbox[0], bbox[3], bbox[2], ax=ax, **bmap_kwargs)
        bmap.drawcoastlines()
        bmap.drawmapboundary()

        x, y = bmap(self.nodes_df.x.values, self.nodes_df.y.values)

        if plot_type == 'scatter':
            plot = bmap.scatter(
                x, y, c=data.values, **plot_kwargs)
        elif plot_type == 'hexbin':
            plot = bmap.hexbin(
                x, y, C=data.values, **plot_kwargs)

        bmap.colorbar(plot, **cbar_kwargs)

        return bmap, fig, ax

Response: Plots an array of data on a map using matplotlib and Basemap.


Instruction:
Can you tell what is the following Python 3 function doing
def set_pois(self, category, maxdist, maxitems, x_col, y_col):
        if category not in self.poi_category_names:
            self.poi_category_names.append(category)

        self.max_pois = maxitems

        node_ids = self.get_node_ids(x_col, y_col)

        self.poi_category_indexes[category] = node_ids.index

        node_idx = self._node_indexes(node_ids)

        self.net.initialize_category(maxdist, maxitems, category.encode('utf-8'), node_idx.values)

Response: This method sets the location of all the pois of this category.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def nearest_pois(self, distance, category, num_pois=1, max_distance=None,
                     imp_name=None, include_poi_ids=False):
        """
        Find the distance to the nearest pois from each source node.  The
        bigger values in this case mean less accessibility.

        Parameters
        ----------
        distance : float
            The maximum distance to look for pois. This will usually be a
            distance unit in meters however if you have customized the
            impedance this could be in other units such as utility or time
            etc.
        category : string
            The name of the category of poi to look for
        num_pois : int
            The number of pois to look for, this also sets the number of
            columns in the DataFrame that gets returned
        max_distance : float, optional
            The value to set the distance to if there is NO poi within the
            specified distance - if not specified, gets set to distance. This
            will usually be a distance unit in meters however if you have
            customized the impedance this could be in other units such as
            utility or time etc.
        imp_name : string, optional
            The impedance name to use for the aggregation on this network.
            Must be one of the impedance names passed in the constructor of
            this object.  If not specified, there must be only one impedance
            passed in the constructor, which will be used.
        include_poi_ids : bool, optional
            If this flag is set to true, the call will add columns to the
            return DataFrame - instead of just returning the distance for
            the nth POI, it will also return the id of that POI.  The names
            of the columns with the poi ids will be poi1, poi2, etc - it
            will take roughly twice as long to include these ids as to not
            include them

        Returns
        -------
        d : Pandas DataFrame
            Like aggregate, this series has an index of all the node ids for
            the network.  Unlike aggregate, this method returns a dataframe
            with the number of columns equal to the distances to the Nth
            closest poi.  For instance, if you ask for the 10 closest poi to
            each node, column d[1] wil be the distance to the 1st closest poi
            of that category while column d[2] will be the distance to the 2nd
            closest poi, and so on.
        """
        if max_distance is None:
            max_distance = distance

        if category not in self.poi_category_names:
            assert 0, "Need to call set_pois for this category"

        if num_pois > self.max_pois:
            assert 0, "Asking for more pois than set in init_pois"

        imp_num = self._imp_name_to_num(imp_name)

        dists, poi_ids = self.net.find_all_nearest_pois(
            distance,
            num_pois,
            category.encode('utf-8'),
            imp_num)
        dists[dists == -1] = max_distance

        df = pd.DataFrame(dists, index=self.node_ids)
        df.columns = list(range(1, num_pois+1))

        if include_poi_ids:
            df2 = pd.DataFrame(poi_ids, index=self.node_ids)
            df2.columns = ["poi%d" % i for i in range(1, num_pois+1)]
            for col in df2.columns:
                # if this is still all working according to plan at this point
                # the great magic trick is now to turn the integer position of
                # the poi, which is painstakingly returned from the c++ code,
                # and turn it into the actual index that was used when it was
                # initialized as a pandas series - this really is pandas-like
                # thinking.  it's complicated on the inside, but quite
                # intuitive to the user I think
                s = df2[col].astype('int')
                df2[col] = self.poi_category_indexes[category].values[s]
                df2.loc[s == -1, col] = np.nan

            df = pd.concat([df, df2], axis=1)

        return df

Response: This method returns a dataframe that contains the distance to the nearest POI from each source node.


Instruction:
Write a Python 3 function for
returning a list of low connectivity nodes in the network that are connected to more than some threshold within a given distance.

Response:
def low_connectivity_nodes(self, impedance, count, imp_name=None):
        """
        Identify nodes that are connected to fewer than some threshold
        of other nodes within a given distance.

        Parameters
        ----------
        impedance : float
            Distance within which to search for other connected nodes. This
            will usually be a distance unit in meters however if you have
            customized the impedance this could be in other units such as
            utility or time etc.
        count : int
            Threshold for connectivity. If a node is connected to fewer
            than this many nodes within `impedance` it will be identified
            as "low connectivity".
        imp_name : string, optional
            The impedance name to use for the aggregation on this network.
            Must be one of the impedance names passed in the constructor of
            this object.  If not specified, there must be only one impedance
            passed in the constructor, which will be used.

        Returns
        -------
        node_ids : array
            List of "low connectivity" node IDs.

        """
        # set a counter variable on all nodes
        self.set(self.node_ids.to_series(), name='counter')

        # count nodes within impedance range
        agg = self.aggregate(
            impedance, type='count', imp_name=imp_name, name='counter')

        return np.array(agg[agg < count].index)


Instruction:
Write a Python 3 script to
create a Pandana network from a bounding box.

Response:
def pdna_network_from_bbox(
        lat_min=None, lng_min=None, lat_max=None, lng_max=None, bbox=None,
        network_type='walk', two_way=True,
        timeout=180, memory=None, max_query_area_size=50 * 1000 * 50 * 1000):
    """
    Make a Pandana network from a bounding lat/lon box
    request to the Overpass API. Distance will be in the default units meters.

    Parameters
    ----------
    lat_min, lng_min, lat_max, lng_max : float
    bbox : tuple
        Bounding box formatted as a 4 element tuple:
        (lng_max, lat_min, lng_min, lat_max)
    network_type : {'walk', 'drive'}, optional
        Specify whether the network will be used for walking or driving.
        A value of 'walk' attempts to exclude things like freeways,
        while a value of 'drive' attempts to exclude things like
        bike and walking paths.
    two_way : bool, optional
        Whether the routes are two-way. If True, node pairs will only
        occur once.
    timeout : int, optional
        the timeout interval for requests and to pass to Overpass API
    memory : int, optional
        server memory allocation size for the query, in bytes.
        If none, server will use its default allocation size
    max_query_area_size : float, optional
        max area for any part of the geometry, in the units the geometry is in

    Returns
    -------
    network : pandana.Network

    """

    nodes, edges = network_from_bbox(lat_min=lat_min, lng_min=lng_min,
                                     lat_max=lat_max, lng_max=lng_max,
                                     bbox=bbox, network_type=network_type,
                                     two_way=two_way, timeout=timeout,
                                     memory=memory,
                                     max_query_area_size=max_query_area_size)

    return Network(
        nodes['x'], nodes['y'],
        edges['from'], edges['to'], edges[['distance']])


Instruction:
Can you tell what is the following Python 3 function doing
def process_node(e):

    uninteresting_tags = {
        'source',
        'source_ref',
        'source:ref',
        'history',
        'attribution',
        'created_by',
        'tiger:tlid',
        'tiger:upload_uuid',
    }

    node = {
        'id': e['id'],
        'lat': e['lat'],
        'lon': e['lon']
    }

    if 'tags' in e:
        for t, v in list(e['tags'].items()):
            if t not in uninteresting_tags:
                node[t] = v

    return node

Response: Process a node element entry into a dict suitable for going into a Pandas DataFrame.


Instruction:
Make a summary of the following Python 3 code
def make_osm_query(query):
    osm_url = 'http://www.overpass-api.de/api/interpreter'
    req = requests.get(osm_url, params={'data': query})
    req.raise_for_status()

    return req.json()

Response: Make a request to OSM and return the parsed JSON.


Instruction:
How would you explain what the following Python 3 function does
def build_node_query(lat_min, lng_min, lat_max, lng_max, tags=None):
    if tags is not None:
        if isinstance(tags, str):
            tags = [tags]
        tags = ''.join('[{}]'.format(t) for t in tags)
    else:
        tags = ''

    query_fmt = (
        '[out:json];'
        '('
        '  node'
        '  {tags}'
        '  ({lat_min},{lng_min},{lat_max},{lng_max});'
        ');'
        'out;')

    return query_fmt.format(
        lat_min=lat_min, lng_min=lng_min, lat_max=lat_max, lng_max=lng_max,
        tags=tags)

Response: Builds a string for a node - based OSM query.


Instruction:
Can you implement a function in Python 3 that
searches for OSM nodes within a bounding box that match given tags.

Response:
def node_query(lat_min, lng_min, lat_max, lng_max, tags=None):
    """
    Search for OSM nodes within a bounding box that match given tags.

    Parameters
    ----------
    lat_min, lng_min, lat_max, lng_max : float
    tags : str or list of str, optional
        Node tags that will be used to filter the search.
        See http://wiki.openstreetmap.org/wiki/Overpass_API/Language_Guide
        for information about OSM Overpass queries
        and http://wiki.openstreetmap.org/wiki/Map_Features
        for a list of tags.

    Returns
    -------
    nodes : pandas.DataFrame
        Will have 'lat' and 'lon' columns, plus other columns for the
        tags associated with the node (these will vary based on the query).
        Index will be the OSM node IDs.

    """
    node_data = make_osm_query(build_node_query(
        lat_min, lng_min, lat_max, lng_max, tags=tags))

    if len(node_data['elements']) == 0:
        raise RuntimeError('OSM query results contain no data.')

    nodes = [process_node(n) for n in node_data['elements']]
    return pd.DataFrame.from_records(nodes, index='id')


Instruction:
Can you generate the documentation for the following Python 3 function
def equal(x, y):
    if PY_3:
        return test_case().assertEqual(x, y) or True

    assert x == y

Response: Shortcut function for unittest. TestCase. assertEqual.


Instruction:
Write a Python 3 function for
returning True if x matches y.

Response:
def matches(x, y, regex_expr=False):
    """
    Tries to match a regular expression value ``x`` against ``y``.
    Aliast``unittest.TestCase.assertEqual()``

    Arguments:
        x (regex|str): regular expression to test.
        y (str): value to match.
        regex_expr (bool): enables regex string based expression matching.

    Raises:
        AssertionError: in case of mismatching.

    Returns:
        bool
    """
    # Parse regex expression, if needed
    x = strip_regex(x) if regex_expr and isregex_expr(x) else x

    # Run regex assertion
    if PY_3:
        # Retrieve original regex pattern
        x = x.pattern if isregex(x) else x
        # Assert regular expression via unittest matchers
        return test_case().assertRegex(y, x) or True

    # Primitive regex matching for Python 2.7
    if isinstance(x, str):
        x = re.compile(x, re.IGNORECASE)

    assert x.match(y) is not None


Instruction:
How would you code a function in Python 3 to
return True if the given expression value is a regular expression with prefix re and suffix re and otherwise False.

Response:
def isregex_expr(expr):
    """
    Returns ``True`` is the given expression value is a regular expression
    like string with prefix ``re/`` and suffix ``/``, otherwise ``False``.

    Arguments:
        expr (mixed): expression value to test.

    Returns:
        bool
    """
    if not isinstance(expr, str):
        return False

    return all([
        len(expr) > 3,
        expr.startswith('re/'),
        expr.endswith('/')
    ])


Instruction:
Create a Python 3 function for
returning True if the input value is a native regular expression object otherwise False.

Response:
def isregex(value):
    """
    Returns ``True`` if the input argument object is a native
    regular expression object, otherwise ``False``.

    Arguments:
        value (mixed): input value to test.

    Returns:
        bool
    """
    if not value:
        return False
    return any((isregex_expr(value), isinstance(value, retype)))


Instruction:
How would you code a function in Python 3 to
compare two values with regular expression matching support.

Response:
def compare(self, value, expectation, regex_expr=False):
        """
        Compares two values with regular expression matching support.

        Arguments:
            value (mixed): value to compare.
            expectation (mixed): value to match.
            regex_expr (bool, optional): enables string based regex matching.

        Returns:
            bool
        """
        return compare(value, expectation, regex_expr=regex_expr)


Instruction:
Given the following Python 3 function, write the documentation
def fluent(fn):
    @functools.wraps(fn)
    def wrapper(self, *args, **kw):
        # Trigger method proxy
        result = fn(self, *args, **kw)
        # Return self instance or method result
        return self if result is None else result
    return wrapper

Response: Decorator allowing easy method chaining.


Instruction:
Explain what the following Python 3 code does
def compare(expr, value, regex_expr=False):
    # Strict equality comparison
    if expr == value:
        return True

    # Infer negate expression to match, if needed
    negate = False
    if isinstance(expr, str):
        negate = expr.startswith(NEGATE)
        expr = strip_negate(expr) if negate else expr

    try:
        # RegExp or strict equality comparison
        test(expr, value, regex_expr=regex_expr)
    except Exception as err:
        if negate:
            return True
        else:
            raise err

    return True

Response: Compares an expression against a given value.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def trigger_methods(instance, args):
    # Start the magic
    for name in sorted(args):
        value = args[name]
        target = instance

        # If response attibutes
        if name.startswith('response_') or name.startswith('reply_'):
            name = name.replace('response_', '').replace('reply_', '')
            # If instance has response attribute, use it
            if hasattr(instance, '_response'):
                target = instance._response

        # Retrieve class member for inspection and future use
        member = getattr(target, name, None)

        # Is attribute
        isattr = name in dir(target)
        iscallable = ismethod(member) and not isfunction(member)

        if not iscallable and not isattr:
            raise PookInvalidArgument('Unsupported argument: {}'.format(name))

        # Set attribute or trigger method
        if iscallable:
            member(value)
        else:
            setattr(target, name, value)

Response: Triggers specific class methods using a simple reflection mechanism based on the input dictionary params.


Instruction:
Create a Python 3 function to
match the given HTTP request instance against the registered sequence of registered trust classes.

Response:
def match(self, request):
        """
        Match the given HTTP request instance against the registered
        matcher functions in the current engine.

        Arguments:
            request (pook.Request): outgoing request to match.

        Returns:
            tuple(bool, list[Exception]): ``True`` if all matcher tests
                passes, otherwise ``False``. Also returns an optional list
                of error exceptions.
        """
        errors = []

        def match(matcher):
            try:
                return matcher.match(request)
            except Exception as err:
                err = '{}: {}'.format(type(matcher).__name__, err)
                errors.append(err)
                return False

        return all([match(matcher) for matcher in self]), errors


Instruction:
Implement a function in Python 3 to
return a matcher instance by class or alias name.

Response:
def get(name):
    """
    Returns a matcher instance by class or alias name.

    Arguments:
        name (str): matcher class name or alias.

    Returns:
        matcher: found matcher instance, otherwise ``None``.
    """
    for matcher in matchers:
        if matcher.__name__ == name or getattr(matcher, 'name', None) == name:
            return matcher


Instruction:
Write a Python 3 function for
initializing a new matcher instance passing variadic arguments to its constructor. Acts as a delegator proxy.

Response:
def init(name, *args):
    """
    Initializes a matcher instance passing variadic arguments to
    its constructor. Acts as a delegator proxy.

    Arguments:
        name (str): matcher class name or alias to execute.
        *args (mixed): variadic argument

    Returns:
        matcher: matcher instance.

    Raises:
        ValueError: if matcher was not found.
    """
    matcher = get(name)
    if not matcher:
        raise ValueError('Cannot find matcher: {}'.format(name))
    return matcher(*args)


Instruction:
Here you have a function in Python 3, explain what it does
def header(self, key, value):
        if type(key) is tuple:
            key, value = str(key[0]), key[1]

        headers = {key: value}
        self._headers.extend(headers)

Response: Defines a new response header.


Instruction:
Implement a function in Python 3 to
define the response body data.

Response:
def body(self, body):
        """
        Defines response body data.

        Arguments:
            body (str|bytes): response body to use.

        Returns:
            self: ``pook.Response`` current instance.
        """
        if isinstance(body, bytes):
            body = body.decode('utf-8')

        self._body = body


Instruction:
Can you generate a brief explanation for the following Python 3 code
def json(self, data):
        self._headers['Content-Type'] = 'application/json'
        if not isinstance(data, str):
            data = json.dumps(data, indent=4)
        self._body = data

Response: Defines the mock response JSON body.


Instruction:
Make a summary of the following Python 3 code
def set(self, key, val):
        key_lower = key.lower()
        new_vals = key, val
        # Keep the common case aka no item present as fast as possible
        vals = self._container.setdefault(key_lower, new_vals)
        if new_vals is not vals:
            self._container[key_lower] = [vals[0], vals[1], val]

Response: Sets a header field with the given value removing previous values.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _append_funcs(target, items):
    [target.append(item) for item in items
     if isfunction(item) or ismethod(item)]

Response: Helper function to append functions into a given list.


Instruction:
Write a Python 3 script for
triggering request mock definition methods dynamically based on input keyword arguments passed to pook. Mock constructor.

Response:
def _trigger_request(instance, request):
    """
    Triggers request mock definition methods dynamically based on input
    keyword arguments passed to `pook.Mock` constructor.

    This is used to provide a more Pythonic interface vs chainable API
    approach.
    """
    if not isinstance(request, Request):
        raise TypeError('request must be instance of pook.Request')

    # Register request matchers
    for key in request.keys:
        if hasattr(instance, key):
            getattr(instance, key)(getattr(request, key))


Instruction:
Create a Python 3 function to
set the mock URL to match.

Response:
def url(self, url):
        """
        Defines the mock URL to match.
        It can be a full URL with path and query params.

        Protocol schema is optional, defaults to ``http://``.

        Arguments:
            url (str): mock URL to match. E.g: ``server.com/api``.

        Returns:
            self: current Mock instance.
        """
        self._request.url = url
        self.add_matcher(matcher('URLMatcher', url))


Instruction:
Explain what the following Python 3 code does
def method(self, method):
        self._request.method = method
        self.add_matcher(matcher('MethodMatcher', method))

Response: Sets the HTTP method to match.


Instruction:
Write a Python 3 script for
defining a URL path to match.

Response:
def path(self, path):
        """
        Defines a URL path to match.

        Only call this method if the URL has no path already defined.

        Arguments:
            path (str): URL path value to match. E.g: ``/api/users``.

        Returns:
            self: current Mock instance.
        """
        url = furl(self._request.rawurl)
        url.path = path
        self._request.url = url.url
        self.add_matcher(matcher('PathMatcher', path))


Instruction:
Make a summary of the following Python 3 code
def header(self, name, value):
        headers = {name: value}
        self._request.headers = headers
        self.add_matcher(matcher('HeadersMatcher', headers))

Response: Defines a header name and value to match.


Instruction:
Explain what the following Python 3 code does
def headers(self, headers=None, **kw):
        headers = kw if kw else headers
        self._request.headers = headers
        self.add_matcher(matcher('HeadersMatcher', headers))

Response: Sets the headers of the current request.


Instruction:
How would you code a function in Python 3 to
define a new header matcher expectation that must be present in the current Mock instance.

Response:
def header_present(self, *names):
        """
        Defines a new header matcher expectation that must be present in the
        outgoing request in order to be satisfied, no matter what value it
        hosts.

        Header keys are case insensitive.

        Arguments:
            *names (str): header or headers names to match.

        Returns:
            self: current Mock instance.

        Example::

            (pook.get('server.com/api')
                .header_present('content-type'))
        """
        for name in names:
            headers = {name: re.compile('(.*)')}
            self.add_matcher(matcher('HeadersMatcher', headers))


Instruction:
Can you tell what is the following Python 3 function doing
def headers_present(self, headers):
        headers = {name: re.compile('(.*)') for name in headers}
        self.add_matcher(matcher('HeadersMatcher', headers))

Response: Sets the matcher for the set of headers that must be present in the current Mock instance.


Instruction:
Write a Python 3 script for
defining the Content - Type outgoing header value to match.

Response:
def content(self, value):
        """
        Defines the ``Content-Type`` outgoing header value to match.

        You can pass one of the following type aliases instead of the full
        MIME type representation:

        - ``json`` = ``application/json``
        - ``xml`` = ``application/xml``
        - ``html`` = ``text/html``
        - ``text`` = ``text/plain``
        - ``urlencoded`` = ``application/x-www-form-urlencoded``
        - ``form`` = ``application/x-www-form-urlencoded``
        - ``form-data`` = ``application/x-www-form-urlencoded``

        Arguments:
            value (str): type alias or header value to match.

        Returns:
            self: current Mock instance.
        """
        header = {'Content-Type': TYPES.get(value, value)}
        self._request.headers = header
        self.add_matcher(matcher('HeadersMatcher', header))


Instruction:
Make a summary of the following Python 3 code
def params(self, params):
        url = furl(self._request.rawurl)
        url = url.add(params)
        self._request.url = url.url
        self.add_matcher(matcher('QueryMatcher', params))

Response: Defines a set of URL query params to match.


Instruction:
Implement a Python 3 function for
setting the body data to match.

Response:
def body(self, body):
        """
        Defines the body data to match.

        ``body`` argument can be a ``str``, ``binary`` or a regular expression.

        Arguments:
            body (str|binary|regex): body data to match.

        Returns:
            self: current Mock instance.
        """
        self._request.body = body
        self.add_matcher(matcher('BodyMatcher', body))


Instruction:
Given the following Python 3 function, write the documentation
def json(self, json):
        self._request.json = json
        self.add_matcher(matcher('JSONMatcher', json))

Response: Sets the JSON body to match.


Instruction:
Write a Python 3 script for
setting the current request s XML body value.

Response:
def xml(self, xml):
        """
        Defines a XML body value to match.

        Arguments:
            xml (str|regex): body XML to match.

        Returns:
            self: current Mock instance.
        """
        self._request.xml = xml
        self.add_matcher(matcher('XMLMatcher', xml))


Instruction:
Write a Python 3 function that can
read the body from a disk file.

Response:
def file(self, path):
        """
        Reads the body to match from a disk file.

        Arguments:
            path (str): relative or absolute path to file to read from.

        Returns:
            self: current Mock instance.
        """
        with open(path, 'r') as f:
            self.body(str(f.read()))


Instruction:
Can you create a Python 3 function that
enables persistent mode for the current mock instance.

Response:
def persist(self, status=None):
        """
        Enables persistent mode for the current mock.

        Returns:
            self: current Mock instance.
        """
        self._persist = status if type(status) is bool else True


Instruction:
Can you generate a brief explanation for the following Python 3 code
def error(self, error):
        self._error = RuntimeError(error) if isinstance(error, str) else error

Response: Defines a simulated exception that will be raised.


Instruction:
Write a Python 3 function that can
define the mock response for this instance.

Response:
def reply(self, status=200, new_response=False, **kw):
        """
        Defines the mock response.

        Arguments:
            status (int, optional): response status code. Defaults to ``200``.
            **kw (dict): optional keyword arguments passed to ``pook.Response``
                constructor.

        Returns:
            pook.Response: mock response definition instance.
        """
        # Use or create a Response mock instance
        res = Response(**kw) if new_response else self._response
        # Define HTTP mandatory response status
        res.status(status or res._status)
        # Expose current mock instance in response for self-reference
        res.mock = self
        # Define mock response
        self._response = res
        # Return response
        return res


Instruction:
Can you write a function in Python 3 where it
matches an incoming HTTP request against the current mock matchers.

Response:
def match(self, request):
        """
        Matches an outgoing HTTP request against the current mock matchers.

        This method acts like a delegator to `pook.MatcherEngine`.

        Arguments:
            request (pook.Request): request instance to match.

        Raises:
            Exception: if the mock has an exception defined.

        Returns:
            tuple(bool, list[Exception]): ``True`` if the mock matches
                the outgoing HTTP request, otherwise ``False``. Also returns
                an optional list of error exceptions.
        """
        # If mock already expired, fail it
        if self._times <= 0:
            raise PookExpiredMock('Mock expired')

        # Trigger mock filters
        for test in self.filters:
            if not test(request, self):
                return False, []

        # Trigger mock mappers
        for mapper in self.mappers:
            request = mapper(request, self)
            if not request:
                raise ValueError('map function must return a request object')

        # Match incoming request against registered mock matchers
        matches, errors = self.matchers.match(request)

        # If not matched, return False
        if not matches:
            return False, errors

        # Register matched request for further inspecion and reference
        self._calls.append(request)

        # Increase mock call counter
        self._matches += 1
        if not self._persist:
            self._times -= 1

        # Raise simulated error
        if self._error:
            raise self._error

        # Trigger callback when matched
        for callback in self.callbacks:
            callback(request, self)

        return True, []


Instruction:
Explain what the following Python 3 code does
def activate_async(fn, _engine):
    @coroutine
    @functools.wraps(fn)
    def wrapper(*args, **kw):
        _engine.activate()
        try:
            if iscoroutinefunction(fn):
                yield from fn(*args, **kw)  # noqa
            else:
                fn(*args, **kw)
        finally:
            _engine.disable()

    return wrapper

Response: A decorator that activates the given function asynchronously.


Instruction:
Implement a function in Python 3 to
set a custom mock interceptor engine that will be used to generate the HTTP traffic for the specified object.

Response:
def set_mock_engine(self, engine):
        """
        Sets a custom mock engine, replacing the built-in one.

        This is particularly useful if you want to replace the built-in
        HTTP traffic mock interceptor engine with your custom one.

        For mock engine implementation details, see `pook.MockEngine`.

        Arguments:
            engine (pook.MockEngine): custom mock engine to use.
        """
        if not engine:
            raise TypeError('engine must be a valid object')

        # Instantiate mock engine
        mock_engine = engine(self)

        # Validate minimum viable interface
        methods = ('activate', 'disable')
        if not all([hasattr(mock_engine, method) for method in methods]):
            raise NotImplementedError('engine must implementent the '
                                      'required methods')

        # Use the custom mock engine
        self.mock_engine = mock_engine

        # Enable mock engine, if needed
        if self.active:
            self.mock_engine.activate()


Instruction:
Write a Python 3 script for
enabling real networking mode for the specified list of hostnames.

Response:
def enable_network(self, *hostnames):
        """
        Enables real networking mode, optionally passing one or multiple
        hostnames that would be used as filter.

        If at least one hostname matches with the outgoing traffic, the
        request will be executed via the real network.

        Arguments:
            *hostnames: optional list of host names to enable real network
                against them. hostname value can be a regular expression.
        """
        def hostname_filter(hostname, req):
            if isregex(hostname):
                return hostname.match(req.url.hostname)
            return req.url.hostname == hostname

        for hostname in hostnames:
            self.use_network_filter(partial(hostname_filter, hostname))

        self.networking = True


Instruction:
How would you code a function in Python 3 to
create and registers a new HTTP mock instance for the current engine.

Response:
def mock(self, url=None, **kw):
        """
        Creates and registers a new HTTP mock in the current engine.

        Arguments:
            url (str): request URL to mock.
            activate (bool): force mock engine activation.
                Defaults to ``False``.
            **kw (mixed): variadic keyword arguments for ``Mock`` constructor.

        Returns:
            pook.Mock: new mock instance.
        """
        # Activate mock engine, if explicitly requested
        if kw.get('activate'):
            kw.pop('activate')
            self.activate()

        # Create the new HTTP mock expectation
        mock = Mock(url=url, **kw)
        # Expose current engine instance via mock
        mock._engine = self
        # Register the mock in the current engine
        self.add_mock(mock)

        # Return it for consumer satisfaction
        return mock


Instruction:
Can you create a Python 3 function that
removes a specific mock instance by object reference.

Response:
def remove_mock(self, mock):
        """
        Removes a specific mock instance by object reference.

        Arguments:
            mock (pook.Mock): mock instance to remove.
        """
        self.mocks = [m for m in self.mocks if m is not mock]


Instruction:
Can you create a Python 3 function that
activates the registered interceptors in the mocking engine.

Response:
def activate(self):
        """
        Activates the registered interceptors in the mocking engine.

        This means any HTTP traffic captures by those interceptors will
        trigger the HTTP mock matching engine in order to determine if a given
        HTTP transaction should be mocked out or not.
        """
        if self.active:
            return None

        # Activate mock engine
        self.mock_engine.activate()
        # Enable engine state
        self.active = True


Instruction:
Can you write a function in Python 3 where it
disables interceptors and stops intercepting any outgoing HTTP traffic.

Response:
def disable(self):
        """
        Disables interceptors and stops intercepting any outgoing HTTP traffic.
        """
        if not self.active:
            return None

        # Disable current mock engine
        self.mock_engine.disable()
        # Disable engine state
        self.active = False


Instruction:
Write a Python 3 function that can
verify if real networking mode should be used for the given HTTP request.

Response:
def should_use_network(self, request):
        """
        Verifies if real networking mode should be used for the given
        request, passing it to the registered network filters.

        Arguments:
            request (pook.Request): outgoing HTTP request to test.

        Returns:
            bool
        """
        return (self.networking and
                all((fn(request) for fn in self.network_filters)))


Instruction:
Write a Python 3 script for
matching a given request instance contract against the registered mock definitions.

Response:
def match(self, request):
        """
        Matches a given Request instance contract against the registered mocks.

        If a mock passes all the matchers, its response will be returned.

        Arguments:
            request (pook.Request): Request contract to match.

        Raises:
            pook.PookNoMatches: if networking is disabled and no mock matches
                with the given request contract.

        Returns:
            pook.Response: the mock response to be used by the interceptor.
        """
        # Trigger engine-level request filters
        for test in self.filters:
            if not test(request, self):
                return False

        # Trigger engine-level request mappers
        for mapper in self.mappers:
            request = mapper(request, self)
            if not request:
                raise ValueError('map function must return a request object')

        # Store list of mock matching errors for further debugging
        match_errors = []

        # Try to match the request against registered mock definitions
        for mock in self.mocks[:]:
            try:
                # Return the first matched HTTP request mock
                matches, errors = mock.match(request.copy())
                if len(errors):
                    match_errors += errors
                if matches:
                    return mock
            except PookExpiredMock:
                # Remove the mock if already expired
                self.mocks.remove(mock)

        # Validate that we have a mock
        if not self.should_use_network(request):
            msg = 'pook error!\n\n'

            msg += (
                '=> Cannot match any mock for the '
                'following request:\n{}'.format(request)
            )

            # Compose unmatch error details, if debug mode is enabled
            if self.debug:
                err = '\n\n'.join([str(err) for err in match_errors])
                if err:
                    msg += '\n\n=> Detailed matching errors:\n{}\n'.format(err)

            # Raise no matches exception
            raise PookNoMatches(msg)

        # Register unmatched request
        self.unmatched_reqs.append(request)


Instruction:
Make a summary of the following Python 3 code
def copy(self):
        req = type(self)()
        req.__dict__ = self.__dict__.copy()
        req._headers = self.headers.copy()
        return req

Response: Returns a copy of the current Request instance for side - effects purposes.


Instruction:
How would you explain what the following Python 3 function does
def activate(fn=None):
    # If not used as decorator, activate the engine and exit
    if not isfunction(fn):
        _engine.activate()
        return None

    # If used as decorator for an async coroutine, wrap it
    if iscoroutinefunction is not None and iscoroutinefunction(fn):
        return activate_async(fn, _engine)

    @functools.wraps(fn)
    def wrapper(*args, **kw):
        _engine.activate()
        try:
            fn(*args, **kw)
        finally:
            _engine.disable()

    return wrapper

Response: Decorator for activate a new object.


Instruction:
Write a Python 3 function for
creating a new isolated mock engine to be used via context manager. Example:: with pook.use() as engine: pook.mock('server.com/foo').reply(404) res = requests.get('server.com/foo') assert res.status_code == 404

Response:
def use(network=False):
    """
    Creates a new isolated mock engine to be used via context manager.

    Example::

        with pook.use() as engine:
            pook.mock('server.com/foo').reply(404)

            res = requests.get('server.com/foo')
            assert res.status_code == 404
    """
    global _engine

    # Create temporal engine
    __engine = _engine
    activated = __engine.active
    if activated:
        __engine.disable()

    _engine = Engine(network=network)
    _engine.activate()

    # Yield enfine to be used by the context manager
    yield _engine

    # Restore engine state
    _engine.disable()
    if network:
        _engine.disable_network()

    # Restore previous engine
    _engine = __engine
    if activated:
        _engine.activate()


Instruction:
Can you tell what is the following Python 3 function doing
def regex(expression, flags=re.IGNORECASE):
    return re.compile(expression, flags=flags)

Response: Returns a regular expression that matches the given string.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def add_interceptor(self, *interceptors):
        for interceptor in interceptors:
            self.interceptors.append(interceptor(self.engine))

Response: Adds one or multiple HTTP traffic interceptors to the current mocking engine.


Instruction:
How would you code a function in Python 3 to
remove an interceptor by name.

Response:
def remove_interceptor(self, name):
        """
        Removes a specific interceptor by name.

        Arguments:
            name (str): interceptor name to disable.

        Returns:
            bool: `True` if the interceptor was disabled, otherwise `False`.
        """
        for index, interceptor in enumerate(self.interceptors):
            matches = (
                type(interceptor).__name__ == name or
                getattr(interceptor, 'name') == name
            )
            if matches:
                self.interceptors.pop(index)
                return True
        return False


Instruction:
Can you tell what is the following Python 3 function doing
def get_setting(connection, key):
    if key in connection.settings_dict:
        return connection.settings_dict[key]
    else:
        return getattr(settings, key)

Response: Get key from connection or default to settings.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def as_sql(self, compiler, connection):
        sql, params = super(DecryptedCol, self).as_sql(compiler, connection)
        sql = self.target.get_decrypt_sql(connection) % (sql, self.target.get_cast_sql())
        return sql, params

Response: Build SQL with decryption and casting.


Instruction:
Here you have a function in Python 3, explain what it does
def pre_save(self, model_instance, add):
        if self.original:
            original_value = getattr(model_instance, self.original)
            setattr(model_instance, self.attname, original_value)

        return super(HashMixin, self).pre_save(model_instance, add)

Response: Save the original value.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_placeholder(self, value=None, compiler=None, connection=None):
        if value is None or value.startswith('\\x'):
            return '%s'

        return self.get_encrypt_sql(connection)

Response: Return the placeholder for the current value.


Instruction:
How would you code a function in Python 3 to
get the decryption for col.

Response:
def get_col(self, alias, output_field=None):
        """Get the decryption for col."""
        if output_field is None:
            output_field = self
        if alias != self.model._meta.db_table or output_field != self:
            return DecryptedCol(
                alias,
                self,
                output_field
            )
        else:
            return self.cached_col


Instruction:
How would you code a function in Python 3 to
tell postgres to encrypt this field using PGP.

Response:
def get_placeholder(self, value=None, compiler=None, connection=None):
        """Tell postgres to encrypt this field using PGP."""
        return self.encrypt_sql.format(get_setting(connection, 'PUBLIC_PGP_KEY'))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def base_regression(Q, slope=None):
    if slope is None:
        slope = (Q[dtavgii] - Q[tavgii]*Q[davgii]/Q[sii]) \
                /(Q[tsqii] - Q[tavgii]**2/Q[sii])
        only_intercept=False
    else:
        only_intercept=True

    intercept = (Q[davgii] - Q[tavgii]*slope)/Q[sii]

    if only_intercept:
        return {'slope':slope, 'intercept':intercept,
                'chisq': 0.5*(Q[dsqii]/Q[sii] - Q[davgii]**2/Q[sii]**2)}

    chisq = 0.5*(Q[dsqii] - Q[davgii]**2/Q[sii]
                - (Q[dtavgii] - Q[davgii]*Q[tavgii]/Q[sii])**2/(Q[tsqii]
                - Q[tavgii]**2/Q[sii]))

    estimator_hessian = np.array([[Q[tsqii], Q[tavgii]], [Q[tavgii], Q[sii]]])

    return {'slope':slope, 'intercept':intercept,
            'chisq':chisq, 'hessian':estimator_hessian,
            'cov':np.linalg.inv(estimator_hessian)}

Response:
This function calculates the regression coefficients for a
    given vector containing averages of tip and branch
    quantities.


Instruction:
Here you have a function in Python 3, explain what it does
def Cov(self):
        # accumulate the covariance matrix by adding 'squares'
        M = np.zeros((self.N, self.N))
        for n in self.tree.find_clades():
            if n == self.tree.root:
                continue
            M[np.meshgrid(n._ii, n._ii)] += self.branch_variance(n)
        return M

Response:
Calculates the covariance matrix of the tips assuming variance
        has accumulated along branches of the tree accoriding to the provided
       .


Instruction:
Can you generate a brief explanation for the following Python 3 code
def CovInv(self):
        self.recurse(full_matrix=True)
        return self.tree.root.cinv

Response: Returns the inverse of the covariance matrix.


Instruction:
Explain what the following Python 3 code does
def recurse(self, full_matrix=False):
        for n in self.tree.get_nonterminals(order='postorder'):
            n_leaves = len(n._ii)
            if full_matrix: M = np.zeros((n_leaves, n_leaves), dtype=float)
            r = np.zeros(n_leaves, dtype=float)
            c_count = 0
            for c in n:
                ssq = self.branch_variance(c)
                nc = len(c._ii)
                if c.is_terminal():
                    if full_matrix:
                        M[c_count, c_count] = 1.0/ssq
                    r[c_count] = 1.0/ssq
                else:
                    if full_matrix:
                        M[c_count:c_count+nc, c_count:c_count+nc] = c.cinv - ssq*np.outer(c.r,c.r)/(1+ssq*c.s)
                    r[c_count:c_count+nc] = c.r/(1+ssq*c.s)
                c_count += nc

            if full_matrix: n.cinv = M
            n.r = r #M.sum(axis=1)
            n.s = n.r.sum()

Response: This function recursively calculates the inverse covariance matrix of the tree.


Instruction:
How would you explain what the following Python 3 function does
def _calculate_averages(self):
        for n in self.tree.get_nonterminals(order='postorder'):
            Q = np.zeros(6, dtype=float)
            for c in n:
                tv = self.tip_value(c)
                bv = self.branch_value(c)
                var = self.branch_variance(c)
                Q += self.propagate_averages(c, tv, bv, var)
            n.Q=Q

        for n in self.tree.find_clades(order='preorder'):
            O = np.zeros(6, dtype=float)
            if n==self.tree.root:
                n.Qtot = n.Q
                continue

            for c in n.up:
                if c==n:
                    continue

                tv = self.tip_value(c)
                bv = self.branch_value(c)
                var = self.branch_variance(c)
                O += self.propagate_averages(c, tv, bv, var)

            if n.up!=self.tree.root:
                c = n.up
                tv = self.tip_value(c)
                bv = self.branch_value(c)
                var = self.branch_variance(c)
                O += self.propagate_averages(c, tv, bv, var, outgroup=True)
            n.O = O

            if not n.is_terminal():
                tv = self.tip_value(n)
                bv = self.branch_value(n)
                var = self.branch_variance(n)
                n.Qtot = n.Q + self.propagate_averages(n, tv, bv, var, outgroup=True)

Response: calculate the weighted sums of the tip and branch values and their second moments.


Instruction:
Explain what the following Python 3 code does
def propagate_averages(self, n, tv, bv, var, outgroup=False):
        if n.is_terminal() and outgroup==False:
            if tv is None or np.isinf(tv) or np.isnan(tv):
                res = np.array([0, 0, 0, 0, 0, 0])
            elif var==0:
                res = np.array([np.inf, np.inf, np.inf, np.inf, np.inf, np.inf])
            else:
                res = np.array([
                    tv/var,
                    bv/var,
                    tv**2/var,
                    bv*tv/var,
                    bv**2/var,
                    1.0/var], dtype=float)
        else:
            tmpQ = n.O if outgroup else n.Q
            denom = 1.0/(1+var*tmpQ[sii])
            res = np.array([
                tmpQ[tavgii]*denom,
                (tmpQ[davgii] + bv*tmpQ[sii])*denom,
                tmpQ[tsqii] - var*tmpQ[tavgii]**2*denom,
                tmpQ[dtavgii] + tmpQ[tavgii]*bv - var*tmpQ[tavgii]*(tmpQ[davgii] + bv*tmpQ[sii])*denom,
                tmpQ[dsqii] + 2*bv*tmpQ[davgii] + bv**2*tmpQ[sii] - var*(tmpQ[davgii]**2 + 2*bv*tmpQ[davgii]*tmpQ[sii] + bv**2*tmpQ[sii]**2)*denom,
                tmpQ[sii]*denom]
            )

        return res

Response: This function operates on the means variance and covariances along a branch.


Instruction:
Create a Python 3 function for
calculating standard explained variance of the root - to - tip distance and time

Response:
def explained_variance(self):
        """calculate standard explained variance

        Returns
        -------
        float
            r-value of the root-to-tip distance and time.
            independent of regression model, but dependent on root choice
        """
        self.tree.root._v=0
        for n in self.tree.get_nonterminals(order='preorder'):
            for c in n:
                c._v = n._v + self.branch_value(c)
        raw = np.array([(self.tip_value(n), n._v) for n in self.tree.get_terminals()
                         if self.tip_value(n) is not None])
        return np.corrcoef(raw.T)[0,1]


Instruction:
Can you write a function in Python 3 where it
regresses tip values against branch values

Response:
def regression(self, slope=None):
        """regress tip values against branch values

        Parameters
        ----------
        slope : None, optional
            if given, the slope isn't optimized

        Returns
        -------
        dict
            regression parameters
        """
        self._calculate_averages()

        clock_model = base_regression(self.tree.root.Q, slope)
        clock_model['r_val'] = self.explained_variance()

        return clock_model


Instruction:
Explain what the following Python 3 code does
def find_best_root(self, force_positive=True, slope=None):
        self._calculate_averages()
        best_root = {"chisq": np.inf}
        for n in self.tree.find_clades():
            if n==self.tree.root:
                continue

            tv = self.tip_value(n)
            bv = self.branch_value(n)
            var = self.branch_variance(n)
            x, chisq = self._optimal_root_along_branch(n, tv, bv, var, slope=slope)

            if (chisq<best_root["chisq"]):
                tmpQ = self.propagate_averages(n, tv, bv*x, var*x) \
                     + self.propagate_averages(n, tv, bv*(1-x), var*(1-x), outgroup=True)
                reg = base_regression(tmpQ, slope=slope)
                if reg["slope"]>=0 or (force_positive==False):
                    best_root = {"node":n, "split":x}
                    best_root.update(reg)

        if 'node' not in best_root:
            print("TreeRegression.find_best_root: No valid root found!", force_positive)
            return None

        if 'hessian' in best_root:
            # calculate differentials with respect to x
            deriv = []
            n = best_root["node"]
            tv = self.tip_value(n)
            bv = self.branch_value(n)
            var = self.branch_variance(n)
            for dx in [-0.001, 0.001]:
                y = min(1.0, max(0.0, best_root["split"]+dx))
                tmpQ = self.propagate_averages(n, tv, bv*y, var*y) \
                     + self.propagate_averages(n, tv, bv*(1-y), var*(1-y), outgroup=True)
                reg = base_regression(tmpQ, slope=slope)
                deriv.append([y,reg['chisq'], tmpQ[tavgii], tmpQ[davgii]])

            estimator_hessian = np.zeros((3,3))
            estimator_hessian[:2,:2] = best_root['hessian']
            estimator_hessian[2,2] = (deriv[0][1] + deriv[1][1] - 2.0*best_root['chisq'])/(deriv[0][0] - deriv[1][0])**2
            # estimator_hessian[2,0] = (deriv[0][2] - deriv[1][2])/(deriv[0][0] - deriv[1][0])
            # estimator_hessian[2,1] = (deriv[0][3] - deriv[1][3])/(deriv[0][0] - deriv[1][0])
            estimator_hessian[0,2] = estimator_hessian[2,0]
            estimator_hessian[1,2] = estimator_hessian[2,1]
            best_root['hessian'] = estimator_hessian
            best_root['cov'] = np.linalg.inv(estimator_hessian)

        return best_root

Response: Find the best root node for the tree.


Instruction:
Implement a Python 3 function for
determining the best root and reroot the tree to this value. Note that this can change the parent child relations of the tree and values associated with branches rather than nodes (e.g. confidence) might need to be re-evaluated afterwards Parameters ---------- force_positive : bool, optional if True, the search for a root will only consider positive rate estimates slope : float, optional if given, it will find the optimal root given a fixed rate. If slope==0, this corresponds to minimal root-to-tip variance rooting (min_dev) Returns ------- dict regression parameters

Response:
def optimal_reroot(self, force_positive=True, slope=None):
        """
        determine the best root and reroot the tree to this value.
        Note that this can change the parent child relations of the tree
        and values associated with branches rather than nodes
        (e.g. confidence) might need to be re-evaluated afterwards

        Parameters
        ----------
        force_positive : bool, optional
            if True, the search for a root will only consider positive rate estimates

        slope : float, optional
            if given, it will find the optimal root given a fixed rate. If slope==0, this
            corresponds to minimal root-to-tip variance rooting (min_dev)

        Returns
        -------
        dict
            regression parameters
        """
        best_root = self.find_best_root(force_positive=force_positive, slope=slope)
        best_node = best_root["node"]

        x = best_root["split"]
        if x<1e-5:
            new_node = best_node
        elif x>1.0-1e-5:
            new_node = best_node.up
        else:
            # create new node in the branch and root the tree to it
            new_node = Phylo.BaseTree.Clade()

            # insert the new node in the middle of the branch
            # by simple re-wiring the links on the both sides of the branch
            # and fix the branch lengths
            new_node.branch_length = best_node.branch_length*(1-x)
            new_node.up = best_node.up
            new_node.clades = [best_node]
            new_node.up.clades = [k if k!=best_node else new_node
                                  for k in best_node.up.clades]

            best_node.branch_length *= x
            best_node.up = new_node

        new_node.rtt_regression = best_root
        self.tree.root_with_outgroup(new_node)

        self.tree.ladderize()
        for n in self.tree.get_nonterminals(order='postorder'):
            for c in n:
                c.up=n
        return best_root


Instruction:
Implement a Python 3 function for
plotting the root - to - tip distance vs time as a basic time - tree diagnostic.

Response:
def clock_plot(self, add_internal=False, ax=None, regression=None,
                   confidence=True, n_sigma = 2, fs=14):
        """Plot root-to-tip distance vs time as a basic time-tree diagnostic

        Parameters
        ----------
        add_internal : bool, optional
            add internal nodes. this will only work if the tree has been dated already
        ax : None, optional
            an matplotlib axis to plot into. if non provided, a new figure is opened
        regression : None, optional
            a dict containing parameters of a root-to-tip vs time regression as
            returned by the function base_regression
        confidence : bool, optional
            add confidence area to the regression line
        n_sigma : int, optional
            number of standard deviations for the confidence area.
        fs : int, optional
            fontsize

        """
        import matplotlib.pyplot as plt
        if ax is None:
            plt.figure()
            ax=plt.subplot(111)

        self.tree.root._v=0
        for n in self.tree.get_nonterminals(order='preorder'):
            for c in n:
                c._v = n._v + self.branch_value(c)

        tips = self.tree.get_terminals()
        internal = self.tree.get_nonterminals()

        # get values of terminals
        xi = np.array([self.tip_value(n) for n in tips])
        yi = np.array([n._v for n in tips])
        ind = np.array([n.bad_branch  if hasattr(n, 'bad_branch') else False for n in tips])
        if add_internal:
            xi_int = np.array([n.numdate for n in internal])
            yi_int = np.array([n._v for n in internal])
            ind_int = np.array([n.bad_branch  if hasattr(n, 'bad_branch') else False  for n in internal])

        if regression:
            # plot regression line
            t_mrca = -regression['intercept']/regression['slope']
            if add_internal:
                time_span = np.max(xi_int[~ind_int]) - np.min(xi_int[~ind_int])
                x_vals = np.array([max(np.min(xi_int[~ind_int]), t_mrca) - 0.1*time_span, np.max(xi[~ind])+0.05*time_span])
            else:
                time_span = np.max(xi[~ind]) - np.min(xi[~ind])
                x_vals = np.array([max(np.min(xi[~ind]), t_mrca) - 0.1*time_span, np.max(xi[~ind]+0.05*time_span)])

            # plot confidence interval
            if confidence and 'cov' in regression:
                x_vals = np.linspace(x_vals[0], x_vals[1], 100)
                y_vals = regression['slope']*x_vals + regression['intercept']
                dev = n_sigma*np.array([np.sqrt(regression['cov'][:2,:2].dot(np.array([x, 1])).dot(np.array([x,1]))) for x in x_vals])
                dev_slope = n_sigma*np.sqrt(regression['cov'][0,0])
                ax.fill_between(x_vals, y_vals-dev, y_vals+dev, alpha=0.2)
                dp = np.array([regression['intercept']/regression['slope']**2,-1./regression['slope']])
                dev_rtt = n_sigma*np.sqrt(regression['cov'][:2,:2].dot(dp).dot(dp))

            else:
                dev_rtt = None
                dev_slope = None

            ax.plot(x_vals, regression['slope']*x_vals + regression['intercept'],
                    label = r"$y=\alpha + \beta t$"+"\n"+
                            r"$\beta=$%1.2e"%(regression["slope"])
                            + ("+/- %1.e"%dev_slope if dev_slope else "") +
                            "\nroot date: %1.1f"%(-regression['intercept']/regression['slope']) +
                            ("+/- %1.2f"%dev_rtt if dev_rtt else ""))

        ax.scatter(xi[~ind], yi[~ind], label=("tips" if add_internal else None))
        if ind.sum():
            try:
                # note: this is treetime specific
                tmp_x = np.array([np.mean(n.raw_date_constraint) if n.raw_date_constraint else None
                                  for n in self.tree.get_terminals()])
                ax.scatter(tmp_x[ind], yi[ind], label="ignored tips", c='r')
            except:
                pass
        if add_internal:
            ax.scatter(xi_int[~ind_int], yi_int[~ind_int], label="internal nodes")

        ax.set_ylabel('root-to-tip distance', fontsize=fs)
        ax.set_xlabel('date', fontsize=fs)
        ax.ticklabel_format(useOffset=False)
        ax.tick_params(labelsize=fs*0.8)
        ax.set_ylim([0, 1.1*np.max(yi)])
        plt.tight_layout()
        plt.legend(fontsize=fs*0.8)


Instruction:
Given the following Python 3 function, write the documentation
def F81(mu=1.0, pi=None, alphabet="nuc", **kwargs):

    if pi is None:
        pi=0.25*np.ones(4, dtype=float)
    num_chars = len(alphabets[alphabet])

    pi = np.array(pi, dtype=float)

    if num_chars != len(pi) :
        pi = np.ones((num_chars, ), dtype=float)
        print ("GTR: Warning!The number of the characters in the alphabet does not match the "
            "shape of the vector of equilibrium frequencies Pi -- assuming equal frequencies for all states.")

    W = np.ones((num_chars,num_chars))
    pi /= (1.0 * np.sum(pi))
    gtr = GTR(alphabet=alphabets[alphabet])
    gtr.assign_rates(mu=mu, pi=pi, W=W)
    return gtr

Response: F90 model for the Seismic Algebraic Tree Algorithm.


Instruction:
Here you have a function in Python 3, explain what it does
def T92(mu=1.0, pi_GC=0.5, kappa=0.1, **kwargs):

    W = _create_transversion_transition_W(kappa)
    # A C G T
    if pi_CG >=1.:
        raise ValueError("The relative CG content specified is larger than 1.0!")
    pi = np.array([(1.-pi_CG)*0.5, pi_CG*0.5, pi_CG*0.5, (1-pi_CG)*0.5])
    gtr = GTR(alphabet=alphabets['nuc_nogap'])
    gtr.assign_rates(mu=mu, pi=pi, W=W)
    return gtr

Response: T92 model for the base class of a base class


Instruction:
Can you tell what is the following Python 3 function doing
def _create_transversion_transition_W(kappa):
    W = np.ones((4,4))
    W[0, 2]=W[1, 3]=W[2, 0]=W[3,1]=kappa
    return W

Response: Create transition matrix for the transversion of a given kappa.


Instruction:
Create a Python 3 function for
initializing the merger model with a coalescent time.

Response:
def set_Tc(self, Tc, T=None):
        '''
        initialize the merger model with a coalescent time

        Args:
            - Tc:   a float or an iterable, if iterable another argument T of same shape is required
            - T:    an array like of same shape as Tc that specifies the time pivots corresponding to Tc
        Returns:
            - None
        '''
        if isinstance(Tc, Iterable):
            if len(Tc)==len(T):
                x = np.concatenate(([-ttconf.BIG_NUMBER], T, [ttconf.BIG_NUMBER]))
                y = np.concatenate(([Tc[0]], Tc, [Tc[-1]]))
                self.Tc = interp1d(x,y)
            else:
                self.logger("need Tc values and Timepoints of equal length",2,warn=True)
                self.Tc = interp1d([-ttconf.BIG_NUMBER, ttconf.BIG_NUMBER], [1e-5, 1e-5])
        else:
            self.Tc = interp1d([-ttconf.BIG_NUMBER, ttconf.BIG_NUMBER],
                               [Tc+ttconf.TINY_NUMBER, Tc+ttconf.TINY_NUMBER])
        self.calc_integral_merger_rate()


Instruction:
Can you generate the documentation for the following Python 3 function
def calc_branch_count(self):
        '''
        calculates an interpolation object that maps time to the number of
        concurrent branches in the tree. The result is stored in self.nbranches
        '''

        # make a list of (time, merger or loss event) by root first iteration
        self.tree_events = np.array(sorted([(n.time_before_present, len(n.clades)-1)
                                for n in self.tree.find_clades() if not n.bad_branch],
                                key=lambda x:-x[0]))

        # collapse multiple events at one time point into sum of changes
        from collections import defaultdict
        dn_branch = defaultdict(int)
        for (t, dn) in self.tree_events:
            dn_branch[t]+=dn
        unique_mergers = np.array(sorted(dn_branch.items(), key = lambda x:-x[0]))

        # calculate the branch count at each point summing the delta branch counts
        nbranches = [[ttconf.BIG_NUMBER, 1], [unique_mergers[0,0]+ttconf.TINY_NUMBER, 1]]
        for ti, (t, dn) in enumerate(unique_mergers[:-1]):
            new_n = nbranches[-1][1]+dn
            next_t = unique_mergers[ti+1,0]+ttconf.TINY_NUMBER
            nbranches.append([t, new_n])
            nbranches.append([next_t, new_n])

        new_n += unique_mergers[-1,1]
        nbranches.append([next_t, new_n])
        nbranches.append([-ttconf.BIG_NUMBER, new_n])
        nbranches=np.array(nbranches)

        self.nbranches = interp1d(nbranches[:,0], nbranches[:,1], kind='linear')

Response: Calculates an interpolation object that maps time to the number of concurrent branches in the tree. The result is stored in self. nbranches.


Instruction:
How would you implement a function in Python 3 that
calculates the integral int_0^t ( k - 1 ) - 1)/2Tc ( t ) dt and stores it as self. integral_merger_rate.

Response:
def calc_integral_merger_rate(self):
        '''
        calculates the integral int_0^t (k(t')-1)/2Tc(t') dt' and stores it as
        self.integral_merger_rate. This differences of this quantity evaluated at
        different times points are the cost of a branch.
        '''
        # integrate the piecewise constant branch count function.
        tvals = np.unique(self.nbranches.x[1:-1])
        rate = self.branch_merger_rate(tvals)
        avg_rate = 0.5*(rate[1:] + rate[:-1])
        cost = np.concatenate(([0],np.cumsum(np.diff(tvals)*avg_rate)))
        # make interpolation objects for the branch count and its integral
        # the latter is scaled by 0.5/Tc
        # need to add extra point at very large time before present to
        # prevent 'out of interpolation range' errors
        self.integral_merger_rate = interp1d(np.concatenate(([-ttconf.BIG_NUMBER], tvals,[ttconf.BIG_NUMBER])),
                                  np.concatenate(([cost[0]], cost,[cost[-1]])), kind='linear')


Instruction:
Write a Python 3 function that can
return the cost associated with a branch starting at t_node with a branch starting at t_node with a branch length of branch_length.

Response:
def cost(self, t_node, branch_length, multiplicity=2.0):
        '''
        returns the cost associated with a branch starting at t_node
        t_node is time before present, the branch goes back in time

        Args:
            - t_node:           time of the node
            - branch_length:    branch length, determines when this branch merges with sister
            - multiplicity:     2 if merger is binary, higher if this is a polytomy
        '''
        merger_time = t_node+branch_length
        return self.integral_merger_rate(merger_time) - self.integral_merger_rate(t_node)\
                 - np.log(self.total_merger_rate(merger_time))*(multiplicity-1.0)/multiplicity


Instruction:
Implement a function in Python 3 to
attach the merger cost to each branch length interpolator in the tree.

Response:
def attach_to_tree(self):
        '''
        attaches the the merger cost to each branch length interpolator in the tree.
        '''
        for clade in self.tree.find_clades():
            if clade.up is not None:
                clade.branch_length_interpolator.merger_cost = self.cost


Instruction:
Here you have a function in Python 3, explain what it does
def optimize_Tc(self):
        '''
        determines the coalescent time scale that optimizes the coalescent likelihood of the tree
        '''
        from scipy.optimize import minimize_scalar
        initial_Tc = self.Tc
        def cost(Tc):
            self.set_Tc(Tc)
            return -self.total_LH()

        sol = minimize_scalar(cost, bounds=[ttconf.TINY_NUMBER,10.0])
        if "success" in sol and sol["success"]:
            self.set_Tc(sol['x'])
        else:
            self.logger("merger_models:optimze_Tc: optimization of coalescent time scale failed: " + str(sol), 0, warn=True)
            self.set_Tc(initial_Tc.y, T=initial_Tc.x)

Response:
This function determines the coalescent time scale that optimizes the coalescent likelihood of the tree
           .


Instruction:
Here you have a function in Python 3, explain what it does
def optimize_skyline(self, n_points=20, stiffness=2.0, method = 'SLSQP',
                         tol=0.03, regularization=10.0, **kwarks):
        '''
        optimize the trajectory of the merger rate 1./T_c to maximize the
        coalescent likelihood.
        parameters:
            n_points    --  number of pivots of the Tc interpolation object
            stiffness   --  penalty for rapid changes in log(Tc)
            methods     --  method used to optimize
            tol         --  optimization tolerance
            regularization --  cost of moving logTc outsize of the range [-100,0]
                               merger rate is measured in branch length units, no
                               plausible rates should never be outside this window
        '''
        self.logger("Coalescent:optimize_skyline:... current LH: %f"%self.total_LH(),2)
        from scipy.optimize import minimize
        initial_Tc = self.Tc
        tvals = np.linspace(self.tree_events[0,0], self.tree_events[-1,0], n_points)
        def cost(logTc):
            # cap log Tc to avoid under or overflow and nan in logs
            self.set_Tc(np.exp(np.maximum(-200,np.minimum(100,logTc))), tvals)
            neglogLH = -self.total_LH() + stiffness*np.sum(np.diff(logTc)**2) \
                       + np.sum((logTc>0)*logTc*regularization)\
                       - np.sum((logTc<-100)*logTc*regularization)
            return neglogLH

        sol = minimize(cost, np.ones_like(tvals)*np.log(self.Tc.y.mean()), method=method, tol=tol)
        if "success" in sol and sol["success"]:
            dlogTc = 0.1
            opt_logTc = sol['x']
            dcost = []
            for ii in range(len(opt_logTc)):
                tmp = opt_logTc.copy()
                tmp[ii]+=dlogTc
                cost_plus = cost(tmp)
                tmp[ii]-=2*dlogTc
                cost_minus = cost(tmp)
                dcost.append([cost_minus, cost_plus])

            dcost = np.array(dcost)
            optimal_cost = cost(opt_logTc)
            self.confidence = -dlogTc/(2*optimal_cost - dcost[:,0] - dcost[:,1])
            self.logger("Coalescent:optimize_skyline:...done. new LH: %f"%self.total_LH(),2)
        else:
            self.set_Tc(initial_Tc.y, T=initial_Tc.x)
            self.logger("Coalescent:optimize_skyline:...failed:"+str(sol),0, warn=True)

Response: Optimize the trajectory of the merger rate 1. / T_c to maximize the coalescent likelihood.


Instruction:
Here you have a function in Python 3, explain what it does
def skyline_empirical(self, gen=1.0, n_points = 20):
        '''
        returns the skyline, i.e., an estimate of the inverse rate of coalesence.
        Here, the skyline is estimated from a sliding window average of the observed
        mergers, i.e., without reference to the coalescence likelihood.
        parameters:
            gen -- number of generations per year.
        '''

        mergers = self.tree_events[:,1]>0
        merger_tvals = self.tree_events[mergers,0]
        nlineages = self.nbranches(merger_tvals-ttconf.TINY_NUMBER)
        expected_merger_density = nlineages*(nlineages-1)*0.5

        nmergers = len(mergers)
        et = merger_tvals
        ev = 1.0/expected_merger_density
        # reduce the window size if there are few events in the tree
        if 2*n_points>len(expected_merger_density):
            n_points = len(ev)//4

        # smoothes with a sliding window over data points
        avg = np.sum(ev)/np.abs(et[0]-et[-1])
        dt = et[0]-et[-1]
        mid_points = np.concatenate(([et[0]-0.5*(et[1]-et[0])],
                                      0.5*(et[1:] + et[:-1]),
                                     [et[-1]+0.5*(et[-1]-et[-2])]))

        # this smoothes the ratio of expected and observed merger rate
        self.Tc_inv = interp1d(mid_points[n_points:-n_points],
                        [np.sum(ev[(et>=l)&(et<u)])/(u-l+dt/nmergers)
                        for u,l in zip(mid_points[:-2*n_points],mid_points[2*n_points:])])

        return interp1d(self.date2dist.to_numdate(self.Tc_inv.x), gen/self.date2dist.clock_rate/self.Tc_inv.y)

Response: This method returns the inverse rate of coalesence of the coalesced tree entry point.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def skyline_inferred(self, gen=1.0, confidence=False):
        '''
        return the skyline, i.e., an estimate of the inverse rate of coalesence.
        This function merely returns the merger rate self.Tc that was set or
        estimated by other means. If it was determined using self.optimize_skyline,
        the returned skyline will maximize the coalescent likelihood.
        parameters:
            gen -- number of generations per year. Unit of time is branch length,
                   hence this needs to be the inverse substitution rate per generation
            confidence -- False, or number of standard deviations of confidence intervals
        '''
        if len(self.Tc.x)<=2:
            print("no skyline has been inferred, returning constant population size")
            return gen/self.date2dist.clock_rate*self.Tc.y[-1]

        skyline = interp1d(self.date2dist.to_numdate(self.Tc.x[1:-1]), gen/self.date2dist.clock_rate*self.Tc.y[1:-1])
        if confidence and hasattr(self, 'confidence'):
            conf = [skyline.y*np.exp(-confidence*self.confidence), skyline.y*np.exp(confidence*self.confidence)]
            return skyline, conf
        else:
            return skyline

Response: return the skyline for the specified generation and confidence intervals


Instruction:
Write a Python 3 function that can
convert a profile to sequence and normalize it across sites.

Response:
def prof2seq(profile, gtr, sample_from_prof=False, normalize=True):
    """
    Convert profile to sequence and normalize profile across sites.

    Parameters
    ----------

     profile : numpy 2D array
        Profile. Shape of the profile should be (L x a), where L - sequence
        length, a - alphabet size.

     gtr : gtr.GTR
        Instance of the GTR class to supply the sequence alphabet

     collapse_prof : bool
        Whether to convert the profile to the delta-function

    Returns
    -------
     seq : numpy.array
        Sequence as numpy array of length L

     prof_values :  numpy.array
        Values of the profile for the chosen sequence characters (length L)

     idx : numpy.array
        Indices chosen from profile as array of length L
    """

    # normalize profile such that probabilities at each site sum to one
    if normalize:
        tmp_profile, pre=normalize_profile(profile, return_offset=False)
    else:
        tmp_profile = profile

    # sample sequence according to the probabilities in the profile
    # (sampling from cumulative distribution over the different states)
    if sample_from_prof:
        cumdis = tmp_profile.cumsum(axis=1).T
        randnum = np.random.random(size=cumdis.shape[1])
        idx = np.argmax(cumdis>=randnum, axis=0)
    else:
        idx = tmp_profile.argmax(axis=1)
    seq = gtr.alphabet[idx]  # max LH over the alphabet

    prof_values = tmp_profile[np.arange(tmp_profile.shape[0]), idx]

    return seq, prof_values, idx


Instruction:
Create a Python 3 function to
return a normalized version of a profile matrix

Response:
def normalize_profile(in_profile, log=False, return_offset = True):
    """return a normalized version of a profile matrix

    Parameters
    ----------
    in_profile : np.array
        shape Lxq, will be normalized to one across each row
    log : bool, optional
        treat the input as log probabilities
    return_offset : bool, optional
        return the log of the scale factor for each row

    Returns
    -------
    tuple
        normalized profile (fresh np object) and offset (if return_offset==True)
    """
    if log:
        tmp_prefactor = in_profile.max(axis=1)
        tmp_prof = np.exp(in_profile.T - tmp_prefactor).T
    else:
        tmp_prefactor = 0.0
        tmp_prof = in_profile

    norm_vector = tmp_prof.sum(axis=1)
    return (np.copy(np.einsum('ai,a->ai',tmp_prof,1.0/norm_vector)),
            (np.log(norm_vector) + tmp_prefactor) if return_offset else None)


Instruction:
Can you implement a function in Python 3 that
prints log message msg to stdout.

Response:
def logger(self, msg, level, warn=False):
        """
        Print log message *msg* to stdout.

        Parameters
        -----------

         msg : str
            String to print on the screen

         level : int
            Log-level. Only the messages with a level higher than the
            current verbose level will be shown.

         warn : bool
            Warning flag. If True, the message will be displayed
            regardless of its log-level.

        """
        if level<self.verbose or (warn and level<=self.verbose):
            dt = time.time() - self.t_start
            outstr = '\n' if level<2 else ''
            outstr += format(dt, '4.2f')+'\t'
            outstr += level*'-'
            outstr += msg
            print(outstr, file=sys.stdout)


Instruction:
How would you code a function in Python 3 to
set the GTR object for this object.

Response:
def gtr(self, value):
        """
        Set a new GTR object

        Parameters
        -----------

         value : GTR
            the new GTR object
        """
        if not (isinstance(value, GTR) or isinstance(value, GTR_site_specific)):
            raise TypeError(" GTR instance expected")
        self._gtr = value


Instruction:
Can you implement a function in Python 3 that
sets the GTR model of the TreeAnc class.

Response:
def set_gtr(self, in_gtr, **kwargs):
        """
        Create new GTR model if needed, and set the model as an attribute of the
        TreeAnc class

        Parameters
        -----------

         in_gtr : str, GTR
            The gtr model to be assigned. If string is passed,
            it is taken as the name of a standard GTR model, and is
            attempted to be created through :code:`GTR.standard()` interface. If a
            GTR instance is passed, it is set directly .

         **kwargs
            Keyword arguments to construct the GTR model. If none are passed, defaults
            are assumed.

        """
        if isinstance(in_gtr, str):
            self._gtr = GTR.standard(model=in_gtr, **kwargs)
            self._gtr.logger = self.logger

        elif isinstance(in_gtr, GTR) or isinstance(in_gtr, GTR_site_specific):
            self._gtr = in_gtr
            self._gtr.logger=self.logger
        else:
            self.logger("TreeAnc.gtr_setter: can't interpret GTR model", 1, warn=True)
            raise TypeError("Cannot set GTR model in TreeAnc class: GTR or "
                "string expected")

        if self._gtr.ambiguous is None:
            self.fill_overhangs=False


Instruction:
Here you have a function in Python 3, explain what it does
def tree(self, in_tree):
        '''
        assigns a tree to the internal self._tree variable. The tree is either
        loaded from file (if in_tree is str) or assigned (if in_tree is a Phylo.tree)
        '''

        from os.path import isfile
        if isinstance(in_tree, Phylo.BaseTree.Tree):
            self._tree = in_tree
        elif type(in_tree) in string_types and isfile(in_tree):
            try:
                self._tree=Phylo.read(in_tree, 'newick')
            except:
                fmt = in_tree.split('.')[-1]
                if fmt in ['nexus', 'nex']:
                    self._tree=Phylo.read(in_tree, 'nexus')
                else:
                    self.logger('TreeAnc: could not load tree, format needs to be nexus or newick! input was '+str(in_tree),1)
                    self._tree = None
                    return ttconf.ERROR
        else:
            self.logger('TreeAnc: could not load tree! input was '+str(in_tree),0)
            self._tree = None
            return ttconf.ERROR

        # remove all existing sequence attributes
        for node in self._tree.find_clades():
            if hasattr(node, "sequence"):
                node.__delattr__("sequence")
            node.original_length = node.branch_length
            node.mutation_length = node.branch_length
        self.prepare_tree()
        return ttconf.SUCCESS

Response: This function assigns a tree to the internal _tree variable.


Instruction:
How would you explain what the following Python 3 function does
def aln(self,in_aln):
        # load alignment from file if necessary
        from os.path import isfile
        from Bio.Align import MultipleSeqAlignment
        self._aln = None
        if in_aln is None:
            return
        elif isinstance(in_aln, MultipleSeqAlignment):
            self._aln = in_aln
        elif type(in_aln) in string_types and isfile(in_aln):
            for fmt in ['fasta', 'phylip-relaxed', 'nexus']:
                try:
                    self._aln=AlignIO.read(in_aln, fmt)
                    break
                except:
                    continue
        elif type(in_aln) in [defaultdict, dict]:  #if is read in from VCF file
            self._aln = in_aln
            self.is_vcf = True

        if self._aln is None:
            self.logger("TreeAnc: loading alignment failed... ",1, warn=True)
            return ttconf.ERROR

        #Convert to uppercase here, rather than in _attach_sequences_to_nodes
        #(which used to do it through seq2array in seq_utils.py)
        #so that it is controlled by param convert_upper. This way for
        #mugration (ancestral reconstruction of non-sequences), you can
        #use upper- and lower case characters for discrete states!
        if (not self.is_vcf) and self.convert_upper:
            self._aln = MultipleSeqAlignment([seq.upper() for seq in self._aln])

        if self.seq_len:
            if self.is_vcf and self.seq_len!=len(self.ref):
                self.logger("TreeAnc.aln: specified sequence length doesn't match reference length, ignoring sequence length.", 1, warn=True)
                self._seq_len = len(self.ref)
            else:
                self.logger("TreeAnc.aln: specified sequence length doesn't match alignment length. Treating difference as constant sites.", 2, warn=True)
                self.additional_constant_sites = max(0, self.seq_len - self.aln.get_alignment_length())
        else:
            if self.is_vcf:
                self.seq_len = len(self.ref)
            else:
                self.seq_len = self.aln.get_alignment_length()

        # check whether the alignment is consistent with a nucleotide alignment.
        likely_alphabet = self._guess_alphabet()
        from .seq_utils import alphabets
        # if likely alignment is not nucleotide but the gtr alignment is, WARN
        if likely_alphabet=='aa' and self.gtr.n_states==len(alphabets['nuc']) and np.all(self.gtr.alphabet==alphabets['nuc']):
            self.logger('WARNING: small fraction of ACGT-N in alignment. Really a nucleotide alignment? if not, rerun with --aa', 1, warn=True)
        # conversely, warn if alignment is consistent with nucleotide but gtr has a long alphabet
        if likely_alphabet=='nuc' and self.gtr.n_states>10:
            self.logger('WARNING: almost exclusively ACGT-N in alignment. Really a protein alignment?', 1, warn=True)

        if hasattr(self, '_tree') and (self.tree is not None):
            self._attach_sequences_to_nodes()
        else:
            self.logger("TreeAnc.aln: sequences not yet attached to tree", 3, warn=True)

Response:
Reads in the alignment and sets tree - related parameters and attaches sequences
        to the tree nodes.


Instruction:
Write a Python 3 function that can
set the length of the uncompressed sequence.

Response:
def seq_len(self,L):
        """set the length of the uncompressed sequence. its inverse 'one_mutation'
        is frequently used as a general length scale. This can't be changed once
        it is set.

        Parameters
        ----------
        L : int
            length of the sequence alignment
        """
        if (not hasattr(self, '_seq_len')) or self._seq_len is None:
            if L:
                self._seq_len = int(L)
        else:
            self.logger("TreeAnc: one_mutation and sequence length can't be reset",1)


Instruction:
Can you tell what is the following Python 3 function doing
def _attach_sequences_to_nodes(self):
        '''
        For each node of the tree, check whether there is a sequence available
        in the alignment and assign this sequence as a character array
        '''
        failed_leaves= 0
        if self.is_vcf:
            # if alignment is specified as difference from ref
            dic_aln = self.aln
        else:
            # if full alignment is specified
            dic_aln = {k.name: seq2array(k.seq, fill_overhangs=self.fill_overhangs,
                                                   ambiguous_character=self.gtr.ambiguous)
                                for k in self.aln} #

        # loop over leaves and assign multiplicities of leaves (e.g. number of identical reads)
        for l in self.tree.get_terminals():
            if l.name in self.seq_multiplicity:
                l.count = self.seq_multiplicity[l.name]
            else:
                l.count = 1.0

        # loop over tree, and assign sequences
        for l in self.tree.find_clades():
            if l.name in dic_aln:
                l.sequence= dic_aln[l.name]
            elif l.is_terminal():
                self.logger("***WARNING: TreeAnc._attach_sequences_to_nodes: NO SEQUENCE FOR LEAF: %s" % l.name, 0, warn=True)
                failed_leaves += 1
                l.sequence = seq2array(self.gtr.ambiguous*self.seq_len, fill_overhangs=self.fill_overhangs,
                                                 ambiguous_character=self.gtr.ambiguous)
                if failed_leaves > self.tree.count_terminals()/3:
                    self.logger("ERROR: At least 30\\% terminal nodes cannot be assigned with a sequence!\n", 0, warn=True)
                    self.logger("Are you sure the alignment belongs to the tree?", 2, warn=True)
                    break
            else: # could not assign sequence for internal node - is OK
                pass

        if failed_leaves:
            self.logger("***WARNING: TreeAnc: %d nodes don't have a matching sequence in the alignment."
                        " POSSIBLE ERROR."%failed_leaves, 0, warn=True)

        # extend profile to contain additional unknown characters
        self.extend_profile()
        return self.make_reduced_alignment()

Response: This function is used to assign sequences to nodes in the tree.


Instruction:
How would you explain what the following Python 3 function does
def make_reduced_alignment(self):

        self.logger("TreeAnc: making reduced alignment...", 1)

        # bind positions in real sequence to that of the reduced (compressed) sequence
        self.full_to_reduced_sequence_map = np.zeros(self.seq_len, dtype=int)

        # bind position in reduced sequence to the array of positions in real (expanded) sequence
        self.reduced_to_full_sequence_map = {}

        #if is a dict, want to be efficient and not iterate over a bunch of const_sites
        #so pre-load alignment_patterns with the location of const sites!
        #and get the sites that we want to iterate over only!
        if self.is_vcf:
            tmp_reduced_aln, alignment_patterns, positions = self.process_alignment_dict()
            seqNames = self.aln.keys() #store seqName order to put back on tree
        elif self.reduce_alignment:
            # transpose real alignment, for ease of iteration
            alignment_patterns = {}
            tmp_reduced_aln = []
            # NOTE the order of tree traversal must be the same as below
            # for assigning the cseq attributes to the nodes.
            seqs = [n.sequence for n in self.tree.find_clades() if hasattr(n, 'sequence')]
            if len(np.unique([len(x) for x in seqs]))>1:
                self.logger("TreeAnc: Sequences differ in in length! ABORTING",0, warn=True)
                aln_transpose = None
                return
            else:
                aln_transpose = np.array(seqs).T
                positions = range(aln_transpose.shape[0])
        else:
            self.multiplicity = np.ones(self.seq_len, dtype=float)
            self.full_to_reduced_sequence_map = np.arange(self.seq_len)
            self.reduced_to_full_sequence_map = {p:np.array([p]) for p in np.arange(self.seq_len)}
            for n in self.tree.find_clades():
                if hasattr(n, 'sequence'):
                    n.original_cseq = np.copy(n.sequence)
                    n.cseq = np.copy(n.sequence)
            return ttconf.SUCCESS

        for pi in positions:
            if self.is_vcf:
                pattern = [ self.aln[k][pi] if pi in self.aln[k].keys()
                            else self.ref[pi] for k,v in self.aln.items() ]
            else:
                pattern = aln_transpose[pi]

            str_pat = "".join(pattern)
            # if the column contains only one state and ambiguous nucleotides, replace
            # those with the state in other strains right away
            unique_letters = list(np.unique(pattern))
            if hasattr(self.gtr, "ambiguous"):
                if len(unique_letters)==2 and self.gtr.ambiguous in unique_letters:
                    other = [c for c in unique_letters if c!=self.gtr.ambiguous][0]
                    str_pat = str_pat.replace(self.gtr.ambiguous, other)
                    unique_letters = [other]
            # if there is a mutation in this column, give it its private pattern
            # this is required when sampling mutations from reconstructed profiles.
            # otherwise, all mutations corresponding to the same pattern will be coupled.
            if len(unique_letters)>1:
                str_pat += '_%d'%pi

            # if the pattern is not yet seen,
            if str_pat not in alignment_patterns:
                # bind the index in the reduced aln, index in sequence to the pattern string
                alignment_patterns[str_pat] = (len(tmp_reduced_aln), [pi])
                # append this pattern to the reduced alignment
                tmp_reduced_aln.append(pattern)
            else:
                # if the pattern is already seen, append the position in the real
                # sequence to the reduced aln<->sequence_pos_indexes map
                alignment_patterns[str_pat][1].append(pi)

        # add constant alignment column not in the alignment. We don't know where they
        # are, so just add them to the end. First, determine sequence composition.
        if self.additional_constant_sites:
            character_counts = {c:np.sum(aln_transpose==c) for c in self.gtr.alphabet
                                if c not in [self.gtr.ambiguous, '-']}
            total = np.sum(list(character_counts.values()))
            additional_columns = [(c,int(np.round(self.additional_constant_sites*n/total)))
                                  for c, n in character_counts.items()]
            columns_left = self.additional_constant_sites
            pi = len(positions)
            for c,n in additional_columns:
                if c==additional_columns[-1][0]:  # make sure all additions add up to the correct number to avoid rounding
                    n = columns_left
                str_pat = c*len(self.aln)
                pos_list = list(range(pi, pi+n))

                if str_pat in alignment_patterns:
                    alignment_patterns[str_pat][1].extend(pos_list)
                else:
                    alignment_patterns[str_pat] = (len(tmp_reduced_aln), pos_list)
                    tmp_reduced_aln.append(np.array(list(str_pat)))
                pi += n
                columns_left -= n

        # count how many times each column is repeated in the real alignment
        self.multiplicity = np.zeros(len(alignment_patterns))
        for p, pos in alignment_patterns.values():
            self.multiplicity[p]=len(pos)

        # create the reduced alignment as np array
        self.reduced_alignment = np.array(tmp_reduced_aln).T

        # create map to compress a sequence
        for p, pos in alignment_patterns.values():
            self.full_to_reduced_sequence_map[np.array(pos)]=p

        # create a map to reconstruct full sequence from the reduced (compressed) sequence
        for p, val in alignment_patterns.items():
            self.reduced_to_full_sequence_map[val[0]]=np.array(val[1], dtype=int)

        # assign compressed sequences to all nodes of the tree, which have sequence assigned
        # for dict we cannot assume this is in the same order, as it does below!
        # so do it explicitly
        #
        # sequences are overwritten during reconstruction and
        # ambiguous sites change. Keep orgininals for reference
        if self.is_vcf:
            seq_reduce_align = {n:self.reduced_alignment[i]
                                for i, n in enumerate(seqNames)}
            for n in self.tree.find_clades():
                if hasattr(n, 'sequence'):
                    n.original_cseq = seq_reduce_align[n.name]
                    n.cseq = np.copy(n.original_cseq)
        else:
            # NOTE the order of tree traversal must be the same as above to catch the
            # index in the reduced alignment correctly
            seq_count = 0
            for n in self.tree.find_clades():
                if hasattr(n, 'sequence'):
                    n.original_cseq = self.reduced_alignment[seq_count]
                    n.cseq = np.copy(n.original_cseq)
                    seq_count+=1

        self.logger("TreeAnc: constructed reduced alignment...", 1)
        return ttconf.SUCCESS

Response: Create the reduced alignment from the full sequences attached to the tree nodes.