Instruction:
Can you generate the documentation for the following Python 3 function
def read_and_write(self, reader, writer, chunk_size):
        data = reader.recv(chunk_size)
        if len(data) == 0:
            return True
        writer.sendall(data)

Response: Read chunk_size from reader and write result to writer.


Instruction:
How would you explain what the following Python 3 function does
def get_local_user():
    # TODO: I don't understand why these lines were added outside the
    # try/except, since presumably it means the attempt at catching ImportError
    # wouldn't work. However, that's how the contributing user committed it.
    # Need an older Windows box to test it out, most likely.
    import getpass

    username = None
    # All Unix and most Windows systems support the getpass module.
    try:
        username = getpass.getuser()
    # Some SaaS platforms raise KeyError, implying there is no real user
    # involved. They get the default value of None.
    except KeyError:
        pass
    # Older (?) Windows systems don't support getpass well; they should
    # have the `win32` module instead.
    except ImportError:  # pragma: nocover
        if win32:
            import win32api
            import win32security  # noqa
            import win32profile  # noqa

            username = win32api.GetUserName()
    return username

Response: Return the local executing username or None if one can t be found.


Instruction:
Explain what the following Python 3 code does
def parameterize(self, call, host):
        debug("Parameterizing {!r} for host {!r}".format(call, host))
        # Generate a custom ConnectionCall that knows how to yield a Connection
        # in its make_context(), specifically one to the host requested here.
        clone = call.clone(into=ConnectionCall)
        # TODO: using bag-of-attrs is mildly gross but whatever, I'll take it.
        clone.host = host
        return clone

Response: Parameterize a Call with its Context set to a per - host Config.


Instruction:
Can you generate the documentation for the following Python 3 function
def mixed_use_of_local_and_run(self):
        cxn = Connection("localhost")
        result = cxn.local("echo foo", hide=True)
        assert result.stdout == "foo\n"
        assert not cxn.is_connected  # meh way of proving it didn't use SSH yet
        result = cxn.run("echo foo", hide=True)
        assert cxn.is_connected  # NOW it's using SSH
        assert result.stdout == "foo\n"

Response: mixed use of local and run command truly over SSH via localhost


Instruction:
Write a Python 3 script to
open an SSH connection to the host and port.

Response:
def open(self):
        """
        Initiate an SSH connection to the host/port this object is bound to.

        This may include activating the configured gateway connection, if one
        is set.

        Also saves a handle to the now-set Transport object for easier access.

        Various connect-time settings (and/or their corresponding :ref:`SSH
        config options <ssh-config>`) are utilized here in the call to
        `SSHClient.connect <paramiko.client.SSHClient.connect>`. (For details,
        see :doc:`the configuration docs </concepts/configuration>`.)

        .. versionadded:: 2.0
        """
        # Short-circuit
        if self.is_connected:
            return
        err = "Refusing to be ambiguous: connect() kwarg '{}' was given both via regular arg and via connect_kwargs!"  # noqa
        # These may not be given, period
        for key in """
            hostname
            port
            username
        """.split():
            if key in self.connect_kwargs:
                raise ValueError(err.format(key))
        # These may be given one way or the other, but not both
        if (
            "timeout" in self.connect_kwargs
            and self.connect_timeout is not None
        ):
            raise ValueError(err.format("timeout"))
        # No conflicts -> merge 'em together
        kwargs = dict(
            self.connect_kwargs,
            username=self.user,
            hostname=self.host,
            port=self.port,
        )
        if self.gateway:
            kwargs["sock"] = self.open_gateway()
        if self.connect_timeout:
            kwargs["timeout"] = self.connect_timeout
        # Strip out empty defaults for less noisy debugging
        if "key_filename" in kwargs and not kwargs["key_filename"]:
            del kwargs["key_filename"]
        # Actually connect!
        self.client.connect(**kwargs)
        self.transport = self.client.get_transport()


Instruction:
How would you explain what the following Python 3 function does
def open_gateway(self):
        # ProxyCommand is faster to set up, so do it first.
        if isinstance(self.gateway, string_types):
            # Leverage a dummy SSHConfig to ensure %h/%p/etc are parsed.
            # TODO: use real SSH config once loading one properly is
            # implemented.
            ssh_conf = SSHConfig()
            dummy = "Host {}\n    ProxyCommand {}"
            ssh_conf.parse(StringIO(dummy.format(self.host, self.gateway)))
            return ProxyCommand(ssh_conf.lookup(self.host)["proxycommand"])
        # Handle inner-Connection gateway type here.
        # TODO: logging
        self.gateway.open()
        # TODO: expose the opened channel itself as an attribute? (another
        # possible argument for separating the two gateway types...) e.g. if
        # someone wanted to piggyback on it for other same-interpreter socket
        # needs...
        # TODO: and the inverse? allow users to supply their own socket/like
        # object they got via $WHEREEVER?
        # TODO: how best to expose timeout param? reuse general connection
        # timeout from config?
        return self.gateway.transport.open_channel(
            kind="direct-tcpip",
            dest_addr=(self.host, int(self.port)),
            # NOTE: src_addr needs to be 'empty but not None' values to
            # correctly encode into a network message. Theoretically Paramiko
            # could auto-interpret None sometime & save us the trouble.
            src_addr=("", 0),
        )

Response: Open a new connection from the gateway.


Instruction:
How would you code a function in Python 3 to
terminate the network connection to the remote end if open.

Response:
def close(self):
        """
        Terminate the network connection to the remote end, if open.

        If no connection is open, this method does nothing.

        .. versionadded:: 2.0
        """
        if self.is_connected:
            self.client.close()
            if self.forward_agent and self._agent_handler is not None:
                self._agent_handler.close()


Instruction:
Write a Python 3 function for
executing a shell command on the remote end of this connection.

Response:
def run(self, command, **kwargs):
        """
        Execute a shell command on the remote end of this connection.

        This method wraps an SSH-capable implementation of
        `invoke.runners.Runner.run`; see its documentation for details.

        .. warning::
            There are a few spots where Fabric departs from Invoke's default
            settings/behaviors; they are documented under
            `.Config.global_defaults`.

        .. versionadded:: 2.0
        """
        runner = self.config.runners.remote(self)
        return self._run(runner, command, **kwargs)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def sudo(self, command, **kwargs):
        runner = self.config.runners.remote(self)
        return self._sudo(runner, command, **kwargs)

Response: Execute a shell command on the remote end.


Instruction:
Can you create a Python 3 function that
executes a shell command on the local system.

Response:
def local(self, *args, **kwargs):
        """
        Execute a shell command on the local system.

        This method is effectively a wrapper of `invoke.run`; see its docs for
        details and call signature.

        .. versionadded:: 2.0
        """
        # Superclass run() uses runners.local, so we can literally just call it
        # straight.
        return super(Connection, self).run(*args, **kwargs)


Instruction:
How would you implement a function in Python 3 that
returns a new instance of the SFTP client.

Response:
def sftp(self):
        """
        Return a `~paramiko.sftp_client.SFTPClient` object.

        If called more than one time, memoizes the first result; thus, any
        given `.Connection` instance will only ever have a single SFTP client,
        and state (such as that managed by
        `~paramiko.sftp_client.SFTPClient.chdir`) will be preserved.

        .. versionadded:: 2.0
        """
        if self._sftp is None:
            self._sftp = self.client.open_sftp()
        return self._sftp


Instruction:
Can you generate the documentation for the following Python 3 function
def forward_local(
        self,
        local_port,
        remote_port=None,
        remote_host="localhost",
        local_host="localhost",
    ):
        """
        Open a tunnel connecting ``local_port`` to the server's environment.

        For example, say you want to connect to a remote PostgreSQL database
        which is locked down and only accessible via the system it's running
        on. You have SSH access to this server, so you can temporarily make
        port 5432 on your local system act like port 5432 on the server::

            import psycopg2
            from fabric import Connection

            with Connection('my-db-server').forward_local(5432):
                db = psycopg2.connect(
                    host='localhost', port=5432, database='mydb'
                )
                # Do things with 'db' here

        This method is analogous to using the ``-L`` option of OpenSSH's
        ``ssh`` program.

        :param int local_port: The local port number on which to listen.

        :param int remote_port:
            The remote port number. Defaults to the same value as
            ``local_port``.

        :param str local_host:
            The local hostname/interface on which to listen. Default:
            ``localhost``.

        :param str remote_host:
            The remote hostname serving the forwarded remote port. Default:
            ``localhost`` (i.e., the host this `.Connection` is connected to.)

        :returns:
            Nothing; this method is only useful as a context manager affecting
            local operating system state.

        .. versionadded:: 2.0
        """
        if not remote_port:
            remote_port = local_port

        # TunnelManager does all of the work, sitting in the background (so we
        # can yield) and spawning threads every time somebody connects to our
        # local port.
        finished = Event()
        manager = TunnelManager(
            local_port=local_port,
            local_host=local_host,
            remote_port=remote_port,
            remote_host=remote_host,
            # TODO: not a huge fan of handing in our transport, but...?
            transport=self.transport,
            finished=finished,
        )
        manager.start()

        # Return control to caller now that things ought to be operational
        try:
            yield
        # Teardown once user exits block
        finally:
            # Signal to manager that it should close all open tunnels
            finished.set()
            # Then wait for it to do so
            manager.join()
            # Raise threading errors from within the manager, which would be
            # one of:
            # - an inner ThreadException, which was created by the manager on
            # behalf of its Tunnels; this gets directly raised.
            # - some other exception, which would thus have occurred in the
            # manager itself; we wrap this in a new ThreadException.
            # NOTE: in these cases, some of the metadata tracking in
            # ExceptionHandlingThread/ExceptionWrapper/ThreadException (which
            # is useful when dealing with multiple nearly-identical sibling IO
            # threads) is superfluous, but it doesn't feel worth breaking
            # things up further; we just ignore it for now.
            wrapper = manager.exception()
            if wrapper is not None:
                if wrapper.type is ThreadException:
                    raise wrapper.value
                else:
                    raise ThreadException([wrapper])

Response: Forward a local entry to the local operating system.


Instruction:
Given the following Python 3 function, write the documentation
def forward_remote(
        self,
        remote_port,
        local_port=None,
        remote_host="127.0.0.1",
        local_host="localhost",
    ):
        """
        Open a tunnel connecting ``remote_port`` to the local environment.

        For example, say you're running a daemon in development mode on your
        workstation at port 8080, and want to funnel traffic to it from a
        production or staging environment.

        In most situations this isn't possible as your office/home network
        probably blocks inbound traffic. But you have SSH access to this
        server, so you can temporarily make port 8080 on that server act like
        port 8080 on your workstation::

            from fabric import Connection

            c = Connection('my-remote-server')
            with c.forward_remote(8080):
                c.run("remote-data-writer --port 8080")
                # Assuming remote-data-writer runs until interrupted, this will
                # stay open until you Ctrl-C...

        This method is analogous to using the ``-R`` option of OpenSSH's
        ``ssh`` program.

        :param int remote_port: The remote port number on which to listen.

        :param int local_port:
            The local port number. Defaults to the same value as
            ``remote_port``.

        :param str local_host:
            The local hostname/interface the forwarded connection talks to.
            Default: ``localhost``.

        :param str remote_host:
            The remote interface address to listen on when forwarding
            connections. Default: ``127.0.0.1`` (i.e. only listen on the remote
            localhost).

        :returns:
            Nothing; this method is only useful as a context manager affecting
            local operating system state.

        .. versionadded:: 2.0
        """
        if not local_port:
            local_port = remote_port
        # Callback executes on each connection to the remote port and is given
        # a Channel hooked up to said port. (We don't actually care about the
        # source/dest host/port pairs at all; only whether the channel has data
        # to read and suchlike.)
        # We then pair that channel with a new 'outbound' socket connection to
        # the local host/port being forwarded, in a new Tunnel.
        # That Tunnel is then added to a shared data structure so we can track
        # & close them during shutdown.
        #
        # TODO: this approach is less than ideal because we have to share state
        # between ourselves & the callback handed into the transport's own
        # thread handling (which is roughly analogous to our self-controlled
        # TunnelManager for local forwarding). See if we can use more of
        # Paramiko's API (or improve it and then do so) so that isn't
        # necessary.
        tunnels = []

        def callback(channel, src_addr_tup, dst_addr_tup):
            sock = socket.socket()
            # TODO: handle connection failure such that channel, etc get closed
            sock.connect((local_host, local_port))
            # TODO: we don't actually need to generate the Events at our level,
            # do we? Just let Tunnel.__init__ do it; all we do is "press its
            # button" on shutdown...
            tunnel = Tunnel(channel=channel, sock=sock, finished=Event())
            tunnel.start()
            # Communication between ourselves & the Paramiko handling subthread
            tunnels.append(tunnel)

        # Ask Paramiko (really, the remote sshd) to call our callback whenever
        # connections are established on the remote iface/port.
        # transport.request_port_forward(remote_host, remote_port, callback)
        try:
            self.transport.request_port_forward(
                address=remote_host, port=remote_port, handler=callback
            )
            yield
        finally:
            # TODO: see above re: lack of a TunnelManager
            # TODO: and/or also refactor with TunnelManager re: shutdown logic.
            # E.g. maybe have a non-thread TunnelManager-alike with a method
            # that acts as the callback? At least then there's a tiny bit more
            # encapsulation...meh.
            for tunnel in tunnels:
                tunnel.finished.set()
                tunnel.join()
            self.transport.cancel_port_forward(
                address=remote_host, port=remote_port
            )

Response: Forward a new connection to the remote server.


Instruction:
Explain what the following Python 3 code does
def dedup_list(l):
    dedup = set()
    return [ x for x in l if not (x in dedup or dedup.add(x))]

Response: Given a list l will remove duplicates from the list and return a list with the original order of the list.


Instruction:
Can you generate the documentation for the following Python 3 function
def _parse(s, g):
    # The CYK table. Indexed with a 2-tuple: (start pos, end pos)
    table = defaultdict(set)
    # Top-level structure is similar to the CYK table. Each cell is a dict from
    # rule name to the best (lightest) tree for that rule.
    trees = defaultdict(dict)
    # Populate base case with existing terminal production rules
    for i, w in enumerate(s):
        for terminal, rules in g.terminal_rules.items():
            if match(terminal, w):
                for rule in rules:
                    table[(i, i)].add(rule)
                    if (rule.lhs not in trees[(i, i)] or
                        rule.weight < trees[(i, i)][rule.lhs].weight):
                        trees[(i, i)][rule.lhs] = RuleNode(rule, [T(w)], weight=rule.weight)

    # Iterate over lengths of sub-sentences
    for l in xrange(2, len(s) + 1):
        # Iterate over sub-sentences with the given length
        for i in xrange(len(s) - l + 1):
            # Choose partition of the sub-sentence in [1, l)
            for p in xrange(i + 1, i + l):
                span1 = (i, p - 1)
                span2 = (p, i + l - 1)
                for r1, r2 in itertools.product(table[span1], table[span2]):
                    for rule in g.nonterminal_rules.get((r1.lhs, r2.lhs), []):
                        table[(i, i + l - 1)].add(rule)
                        r1_tree = trees[span1][r1.lhs]
                        r2_tree = trees[span2][r2.lhs]
                        rule_total_weight = rule.weight + r1_tree.weight + r2_tree.weight
                        if (rule.lhs not in trees[(i, i + l - 1)]
                            or rule_total_weight < trees[(i, i + l - 1)][rule.lhs].weight):
                            trees[(i, i + l - 1)][rule.lhs] = RuleNode(rule, [r1_tree, r2_tree], weight=rule_total_weight)
    return table, trees

Response: Parses a sentence s using CNF grammar g.


Instruction:
Explain what the following Python 3 code does
def get_any_nt_unit_rule(g):
    for rule in g.rules:
        if len(rule.rhs) == 1 and isinstance(rule.rhs[0], NT):
            return rule
    return None

Response: Returns a non - terminal unit rule from g or None if there is none.


Instruction:
Implement a Python 3 function for
removing a rule from g without changing the langugage produced by g.

Response:
def _remove_unit_rule(g, rule):
    """Removes 'rule' from 'g' without changing the langugage produced by 'g'."""
    new_rules = [x for x in g.rules if x != rule]
    refs = [x for x in g.rules if x.lhs == rule.rhs[0]]
    new_rules += [build_unit_skiprule(rule, ref) for ref in refs]
    return Grammar(new_rules)


Instruction:
Write a Python 3 function that can
split a rule into shorter rules.

Response:
def _split(rule):
    """Splits a rule whose len(rhs) > 2 into shorter rules."""
    rule_str = str(rule.lhs) + '__' + '_'.join(str(x) for x in rule.rhs)
    rule_name = '__SP_%s' % (rule_str) + '_%d'
    yield Rule(rule.lhs, [rule.rhs[0], NT(rule_name % 1)], weight=rule.weight, alias=rule.alias)
    for i in xrange(1, len(rule.rhs) - 2):
        yield Rule(NT(rule_name % i), [rule.rhs[i], NT(rule_name % (i + 1))], weight=0, alias='Split')
    yield Rule(NT(rule_name % (len(rule.rhs) - 2)), rule.rhs[-2:], weight=0, alias='Split')


Instruction:
Write a Python 3 script for
applying the TERM rule on g.

Response:
def _term(g):
    """Applies the TERM rule on 'g' (see top comment)."""
    all_t = {x for rule in g.rules for x in rule.rhs if isinstance(x, T)}
    t_rules = {t: Rule(NT('__T_%s' % str(t)), [t], weight=0, alias='Term') for t in all_t}
    new_rules = []
    for rule in g.rules:
        if len(rule.rhs) > 1 and any(isinstance(x, T) for x in rule.rhs):
            new_rhs = [t_rules[x].lhs if isinstance(x, T) else x for x in rule.rhs]
            new_rules.append(Rule(rule.lhs, new_rhs, weight=rule.weight, alias=rule.alias))
            new_rules.extend(v for k, v in t_rules.items() if k in rule.rhs)
        else:
            new_rules.append(rule)
    return Grammar(new_rules)


Instruction:
Write a Python 3 function for
applying the BIN rule to g.

Response:
def _bin(g):
    """Applies the BIN rule to 'g' (see top comment)."""
    new_rules = []
    for rule in g.rules:
        if len(rule.rhs) > 2:
            new_rules += _split(rule)
        else:
            new_rules.append(rule)
    return Grammar(new_rules)


Instruction:
Write a Python 3 function that can
apply the UNIT rule to g.

Response:
def _unit(g):
    """Applies the UNIT rule to 'g' (see top comment)."""
    nt_unit_rule = get_any_nt_unit_rule(g)
    while nt_unit_rule:
        g = _remove_unit_rule(g, nt_unit_rule)
        nt_unit_rule = get_any_nt_unit_rule(g)
    return g


Instruction:
Explain what the following Python 3 code does
def revert_cnf(node):
    if isinstance(node, T):
        return node
    # Reverts TERM rule.
    if node.rule.lhs.name.startswith('__T_'):
        return node.children[0]
    else:
        children = []
        for child in map(revert_cnf, node.children):
            # Reverts BIN rule.
            if isinstance(child, RuleNode) and child.rule.lhs.name.startswith('__SP_'):
                children += child.children
            else:
                children.append(child)
        # Reverts UNIT rule.
        if isinstance(node.rule, UnitSkipRule):
            return unroll_unit_skiprule(node.rule.lhs, node.rule.rhs,
                                    node.rule.skipped_rules, children,
                                    node.rule.weight, node.rule.alias)
        else:
            return RuleNode(node.rule, children)

Response: Reverts a parse tree ( RuleNode to its original non - CNF form ( Node ).


Instruction:
Write a Python 3 function that can
convert a lark rule from a Lark VM to a Rule.

Response:
def _to_rule(self, lark_rule):
        """Converts a lark rule, (lhs, rhs, callback, options), to a Rule."""
        assert isinstance(lark_rule.origin, NT)
        assert all(isinstance(x, Symbol) for x in lark_rule.expansion)
        return Rule(
            lark_rule.origin, lark_rule.expansion,
            weight=lark_rule.options.priority if lark_rule.options and lark_rule.options.priority else 0,
            alias=lark_rule)


Instruction:
Can you tell what is the following Python 3 function doing
def parse(self, tokenized):  # pylint: disable=invalid-name
        table, trees = _parse(tokenized, self.grammar)
        # Check if the parse succeeded.
        if all(r.lhs != self.start for r in table[(0, len(tokenized) - 1)]):
            raise ParseError('Parsing failed.')
        parse = trees[(0, len(tokenized) - 1)][self.start]
        return self._to_tree(revert_cnf(parse))

Response: Parses input which is a list of tokens.


Instruction:
Can you generate the documentation for the following Python 3 function
def _to_tree(self, rule_node):
        orig_rule = self.orig_rules[rule_node.rule.alias]
        children = []
        for child in rule_node.children:
            if isinstance(child, RuleNode):
                children.append(self._to_tree(child))
            else:
                assert isinstance(child.name, Token)
                children.append(child.name)
        t = Tree(orig_rule.origin, children)
        t.rule=orig_rule
        return t

Response: Converts a RuleNode parse tree to a lark Tree.


Instruction:
Can you tell what is the following Python 3 function doing
def pydot__tree_to_png(tree, filename, rankdir="LR"):

    import pydot
    graph = pydot.Dot(graph_type='digraph', rankdir=rankdir)

    i = [0]

    def new_leaf(leaf):
        node = pydot.Node(i[0], label=repr(leaf))
        i[0] += 1
        graph.add_node(node)
        return node

    def _to_pydot(subtree):
        color = hash(subtree.data) & 0xffffff
        color |= 0x808080

        subnodes = [_to_pydot(child) if isinstance(child, Tree) else new_leaf(child)
                    for child in subtree.children]
        node = pydot.Node(i[0], style="filled", fillcolor="#%x"%color, label=subtree.data)
        i[0] += 1
        graph.add_node(node)

        for subnode in subnodes:
            graph.add_edge(pydot.Edge(node, subnode))

        return node

    _to_pydot(tree)
    graph.write_png(filename)

Response: Writes a colorful image that represents the tree.


Instruction:
Implement a Python 3 function for
expanding ( inline ) children at the given indices

Response:
def expand_kids_by_index(self, *indices):
        "Expand (inline) children at the given indices"
        for i in sorted(indices, reverse=True): # reverse so that changing tail won't affect indices
            kid = self.children[i]
            self.children[i:i+1] = kid.children


Instruction:
How would you implement a function in Python 3 that
gives a parser instance and a dictionary mapping some label with the same name and a list of malformed syntax examples it ll return the label for the other examples that bests matches the current error.

Response:
def match_examples(self, parse_fn, examples):
        """ Given a parser instance and a dictionary mapping some label with
            some malformed syntax examples, it'll return the label for the
            example that bests matches the current error.
        """
        assert self.state is not None, "Not supported for this exception"

        candidate = None
        for label, example in examples.items():
            assert not isinstance(example, STRING_TYPE)

            for malformed in example:
                try:
                    parse_fn(malformed)
                except UnexpectedInput as ut:
                    if ut.state == self.state:
                        try:
                            if ut.token == self.token:  # Try exact match first
                                return label
                        except AttributeError:
                            pass
                        if not candidate:
                            candidate = label

        return candidate


Instruction:
Make a summary of the following Python 3 code
def feed(self, token, test_newline=True):
        if test_newline:
            newlines = token.count(self.newline_char)
            if newlines:
                self.line += newlines
                self.line_start_pos = self.char_pos + token.rindex(self.newline_char) + 1

        self.char_pos += len(token)
        self.column = self.char_pos - self.line_start_pos + 1

Response: Consume a token and calculate the new line & column.


Instruction:
Can you create a Python 3 function that
creates an instance of Lark with the grammar given by its filename.

Response:
def open(cls, grammar_filename, rel_to=None, **options):
        """Create an instance of Lark with the grammar given by its filename

        If rel_to is provided, the function will find the grammar filename in relation to it.

        Example:

            >>> Lark.open("grammar_file.lark", rel_to=__file__, parser="lalr")
            Lark(...)

        """
        if rel_to:
            basepath = os.path.dirname(rel_to)
            grammar_filename = os.path.join(basepath, grammar_filename)
        with open(grammar_filename, encoding='utf8') as f:
            return cls(f, **options)


Instruction:
Implement a function in Python 3 to
calculate FOLLOW sets. Adapted from: http://lara.epfl.ch/w/cc09:algorithm_for_first_and_follow_sets

Response:
def calculate_sets(rules):
    """Calculate FOLLOW sets.

    Adapted from: http://lara.epfl.ch/w/cc09:algorithm_for_first_and_follow_sets"""
    symbols = {sym for rule in rules for sym in rule.expansion} | {rule.origin for rule in rules}

    # foreach grammar rule X ::= Y(1) ... Y(k)
    # if k=0 or {Y(1),...,Y(k)} subset of NULLABLE then
    #   NULLABLE = NULLABLE union {X}
    # for i = 1 to k
    #   if i=1 or {Y(1),...,Y(i-1)} subset of NULLABLE then
    #     FIRST(X) = FIRST(X) union FIRST(Y(i))
    #   for j = i+1 to k
    #     if i=k or {Y(i+1),...Y(k)} subset of NULLABLE then
    #       FOLLOW(Y(i)) = FOLLOW(Y(i)) union FOLLOW(X)
    #     if i+1=j or {Y(i+1),...,Y(j-1)} subset of NULLABLE then
    #       FOLLOW(Y(i)) = FOLLOW(Y(i)) union FIRST(Y(j))
    # until none of NULLABLE,FIRST,FOLLOW changed in last iteration

    NULLABLE = set()
    FIRST = {}
    FOLLOW = {}
    for sym in symbols:
        FIRST[sym]={sym} if sym.is_term else set()
        FOLLOW[sym]=set()

    # Calculate NULLABLE and FIRST
    changed = True
    while changed:
        changed = False

        for rule in rules:
            if set(rule.expansion) <= NULLABLE:
                if update_set(NULLABLE, {rule.origin}):
                    changed = True

            for i, sym in enumerate(rule.expansion):
                if set(rule.expansion[:i]) <= NULLABLE:
                    if update_set(FIRST[rule.origin], FIRST[sym]):
                        changed = True

    # Calculate FOLLOW
    changed = True
    while changed:
        changed = False

        for rule in rules:
            for i, sym in enumerate(rule.expansion):
                if i==len(rule.expansion)-1 or set(rule.expansion[i+1:]) <= NULLABLE:
                    if update_set(FOLLOW[sym], FOLLOW[rule.origin]):
                        changed = True

                for j in range(i+1, len(rule.expansion)):
                    if set(rule.expansion[i+1:j]) <= NULLABLE:
                        if update_set(FOLLOW[sym], FIRST[rule.expansion[j]]):
                            changed = True

    return FIRST, FOLLOW, NULLABLE


Instruction:
Can you write a function in Python 3 where it
returns all init_ptrs accessible by rule ( recursive )

Response:
def expand_rule(self, rule):
        "Returns all init_ptrs accessible by rule (recursive)"
        init_ptrs = set()
        def _expand_rule(rule):
            assert not rule.is_term, rule

            for r in self.rules_by_origin[rule]:
                init_ptr = RulePtr(r, 0)
                init_ptrs.add(init_ptr)

                if r.expansion: # if not empty rule
                    new_r = init_ptr.next
                    if not new_r.is_term:
                        yield new_r

        for _ in bfs([rule], _expand_rule):
            pass

        return fzset(init_ptrs)


Instruction:
Implement a function in Python 3 to
return all rules and terminals of grammar prepended by a namespace prefix except for those which are aliased.

Response:
def import_from_grammar_into_namespace(grammar, namespace, aliases):
    """Returns all rules and terminals of grammar, prepended
    with a 'namespace' prefix, except for those which are aliased.
    """

    imported_terms = dict(grammar.term_defs)
    imported_rules = {n:(n,deepcopy(t),o) for n,t,o in grammar.rule_defs}

    term_defs = []
    rule_defs = []

    def rule_dependencies(symbol):
        if symbol.type != 'RULE':
            return []
        try:
            _, tree, _ = imported_rules[symbol]
        except KeyError:
            raise GrammarError("Missing symbol '%s' in grammar %s" % (symbol, namespace))
        return tree.scan_values(lambda x: x.type in ('RULE', 'TERMINAL'))

    def get_namespace_name(name):
        try:
            return aliases[name].value
        except KeyError:
            if name[0] == '_':
                return '_%s__%s' % (namespace, name[1:])
            return '%s__%s' % (namespace, name)

    to_import = list(bfs(aliases, rule_dependencies))
    for symbol in to_import:
        if symbol.type == 'TERMINAL':
            term_defs.append([get_namespace_name(symbol), imported_terms[symbol]])
        else:
            assert symbol.type == 'RULE'
            rule = imported_rules[symbol]
            for t in rule[1].iter_subtrees():
                for i, c in enumerate(t.children):
                    if isinstance(c, Token) and c.type in ('RULE', 'TERMINAL'):
                        t.children[i] = Token(c.type, get_namespace_name(c))
            rule_defs.append((get_namespace_name(symbol), rule[1], rule[2]))

    return term_defs, rule_defs


Instruction:
Write a Python 3 script for
parsing grammar_text verify and create Grammar object. Display nice messages on error.

Response:
def load_grammar(self, grammar_text, grammar_name='<?>'):
        "Parse grammar_text, verify, and create Grammar object. Display nice messages on error."

        try:
            tree = self.canonize_tree.transform( self.parser.parse(grammar_text+'\n') )
        except UnexpectedCharacters as e:
            context = e.get_context(grammar_text)
            raise GrammarError("Unexpected input at line %d column %d in %s: \n\n%s" %
                               (e.line, e.column, grammar_name, context))
        except UnexpectedToken as e:
            context = e.get_context(grammar_text)
            error = e.match_examples(self.parser.parse, {
                'Unclosed parenthesis': ['a: (\n'],
                'Umatched closing parenthesis': ['a: )\n', 'a: [)\n', 'a: (]\n'],
                'Expecting rule or terminal definition (missing colon)': ['a\n', 'a->\n', 'A->\n', 'a A\n'],
                'Alias expects lowercase name': ['a: -> "a"\n'],
                'Unexpected colon': ['a::\n', 'a: b:\n', 'a: B:\n', 'a: "a":\n'],
                'Misplaced operator': ['a: b??', 'a: b(?)', 'a:+\n', 'a:?\n', 'a:*\n', 'a:|*\n'],
                'Expecting option ("|") or a new rule or terminal definition': ['a:a\n()\n'],
                '%import expects a name': ['%import "a"\n'],
                '%ignore expects a value': ['%ignore %import\n'],
            })
            if error:
                raise GrammarError("%s at line %s column %s\n\n%s" % (error, e.line, e.column, context))
            elif 'STRING' in e.expected:
                raise GrammarError("Expecting a value at line %s column %s\n\n%s" % (e.line, e.column, context))
            raise

        tree = PrepareGrammar().transform(tree)

        # Extract grammar items
        defs = classify(tree.children, lambda c: c.data, lambda c: c.children)
        term_defs = defs.pop('term', [])
        rule_defs = defs.pop('rule', [])
        statements = defs.pop('statement', [])
        assert not defs

        term_defs = [td if len(td)==3 else (td[0], 1, td[1]) for td in term_defs]
        term_defs = [(name.value, (t, int(p))) for name, p, t in term_defs]
        rule_defs = [options_from_rule(*x) for x in rule_defs]

        # Execute statements
        ignore, imports = [], {}
        for (stmt,) in statements:
            if stmt.data == 'ignore':
                t ,= stmt.children
                ignore.append(t)
            elif stmt.data == 'import':
                if len(stmt.children) > 1:
                    path_node, arg1 = stmt.children
                else:
                    path_node, = stmt.children
                    arg1 = None

                if isinstance(arg1, Tree):  # Multi import
                    dotted_path = tuple(path_node.children)
                    names = arg1.children
                    aliases = dict(zip(names, names))  # Can't have aliased multi import, so all aliases will be the same as names
                else:  # Single import
                    dotted_path = tuple(path_node.children[:-1])
                    name = path_node.children[-1]  # Get name from dotted path
                    aliases = {name: arg1 or name}  # Aliases if exist

                if path_node.data == 'import_lib':  # Import from library
                    base_paths = []
                else:  # Relative import
                    if grammar_name == '<string>':  # Import relative to script file path if grammar is coded in script
                        try:
                            base_file = os.path.abspath(sys.modules['__main__'].__file__)
                        except AttributeError:
                            base_file = None
                    else:
                        base_file = grammar_name  # Import relative to grammar file path if external grammar file
                    if base_file:
                        base_paths = [os.path.split(base_file)[0]]
                    else:
                        base_paths = [os.path.abspath(os.path.curdir)]

                try:
                    import_base_paths, import_aliases = imports[dotted_path]
                    assert base_paths == import_base_paths, 'Inconsistent base_paths for %s.' % '.'.join(dotted_path)
                    import_aliases.update(aliases)
                except KeyError:
                    imports[dotted_path] = base_paths, aliases

            elif stmt.data == 'declare':
                for t in stmt.children:
                    term_defs.append([t.value, (None, None)])
            else:
                assert False, stmt

        # import grammars
        for dotted_path, (base_paths, aliases) in imports.items():
            grammar_path = os.path.join(*dotted_path) + EXT
            g = import_grammar(grammar_path, base_paths=base_paths)
            new_td, new_rd = import_from_grammar_into_namespace(g, '__'.join(dotted_path), aliases)

            term_defs += new_td
            rule_defs += new_rd

        # Verify correctness 1
        for name, _ in term_defs:
            if name.startswith('__'):
                raise GrammarError('Names starting with double-underscore are reserved (Error at %s)' % name)

        # Handle ignore tokens
        # XXX A slightly hacky solution. Recognition of %ignore TERMINAL as separate comes from the lexer's
        #     inability to handle duplicate terminals (two names, one value)
        ignore_names = []
        for t in ignore:
            if t.data=='expansions' and len(t.children) == 1:
                t2 ,= t.children
                if t2.data=='expansion' and len(t2.children) == 1:
                    item ,= t2.children
                    if item.data == 'value':
                        item ,= item.children
                        if isinstance(item, Token) and item.type == 'TERMINAL':
                            ignore_names.append(item.value)
                            continue

            name = '__IGNORE_%d'% len(ignore_names)
            ignore_names.append(name)
            term_defs.append((name, (t, 1)))

        # Verify correctness 2
        terminal_names = set()
        for name, _ in term_defs:
            if name in terminal_names:
                raise GrammarError("Terminal '%s' defined more than once" % name)
            terminal_names.add(name)

        if set(ignore_names) > terminal_names:
            raise GrammarError("Terminals %s were marked to ignore but were not defined!" % (set(ignore_names) - terminal_names))

        resolve_term_references(term_defs)

        rules = rule_defs

        rule_names = set()
        for name, _x, _o in rules:
            if name.startswith('__'):
                raise GrammarError('Names starting with double-underscore are reserved (Error at %s)' % name)
            if name in rule_names:
                raise GrammarError("Rule '%s' defined more than once" % name)
            rule_names.add(name)

        for name, expansions, _o in rules:
            used_symbols = {t for x in expansions.find_data('expansion')
                              for t in x.scan_values(lambda t: t.type in ('RULE', 'TERMINAL'))}
            for sym in used_symbols:
                if sym.type == 'TERMINAL':
                    if sym not in terminal_names:
                        raise GrammarError("Token '%s' used but not defined (in rule %s)" % (sym, name))
                else:
                    if sym not in rule_names:
                        raise GrammarError("Rule '%s' used but not defined (in rule %s)" % (sym, name))

        return Grammar(rules, term_defs, ignore_names)


Instruction:
Create a Python 3 function for
applying the style rules to the radio group.

Response:
def applyStyleRules(self):
        """
        Conditionally disabled/enables form fields based on the current
        section in the radio group
        """
        for button, widget in zip(self.radioButtons, self.widgets):
            if isinstance(widget, CheckBox):
                widget.hideInput()
            if not button.GetValue(): # not checked
                widget.widget.Disable()
            else:
                widget.widget.Enable()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def handleImplicitCheck(self):
        for button, widget in zip(self.radioButtons, self.widgets):
            if isinstance(widget, CheckBox):
                if button.GetValue(): # checked
                    widget.setValue(True)
                else:
                    widget.setValue(False)

Response: Handles implicit check of the Radio button.


Instruction:
Here you have a function in Python 3, explain what it does
def createWidgets(self):
        from gooey.gui.components import widgets
        return [getattr(widgets, item['type'])(self, item)
                for item in getin(self.widgetInfo, ['data', 'widgets'], [])]

Response: Instantiate the Gooey Widgets that are used within the RadioGroup


Instruction:
Given the following Python 3 function, write the documentation
def onStart(self, *args, **kwarg):
        with transactUI(self):
            config = self.navbar.getActiveConfig()
            config.resetErrors()
            if config.isValid():
                self.clientRunner.run(self.buildCliString())
                self.showConsole()
            else:
                config.displayErrors()
                self.Layout()

Response: Called when the user is ready to start the client.


Instruction:
Can you generate the documentation for the following Python 3 function
def onEdit(self):
        with transactUI(self):
            if self.buildSpec['poll_external_updates']:
                self.fetchExternalUpdates()
            self.showSettings()

Response: Return the user to the settings screen for further editing


Instruction:
Can you generate the documentation for the following Python 3 function
def buildCliString(self):
        config = self.navbar.getActiveConfig()
        group = self.buildSpec['widgets'][self.navbar.getSelectedGroup()]
        positional = config.getPositionalArgs()
        optional = config.getOptionalArgs()
        print(cli.buildCliString(
            self.buildSpec['target'],
            group['command'],
            positional,
            optional
        ))
        return cli.buildCliString(
            self.buildSpec['target'],
            group['command'],
            positional,
            optional
        )

Response: Collect all of the required information from the config screen and build a CLI string which can be used to invoke the client program


Instruction:
Write a Python 3 script to
call when the client is finished.

Response:
def onComplete(self, *args, **kwargs):
        """
        Display the appropriate screen based on the success/fail of the
        host program
        """
        with transactUI(self):
            if self.clientRunner.was_success():
                if self.buildSpec.get('return_to_config', False):
                    self.showSettings()
                else:
                    self.showSuccess()
                    if self.buildSpec.get('show_success_modal', True):
                        wx.CallAfter(modals.showSuccess)
            else:
                if self.clientRunner.wasForcefullyStopped:
                    self.showForceStopped()
                else:
                    self.showError()
                    wx.CallAfter(modals.showFailure)


Instruction:
Here you have a function in Python 3, explain what it does
def fetchExternalUpdates(self):
        seeds = seeder.fetchDynamicProperties(
            self.buildSpec['target'],
            self.buildSpec['encoding']
        )
        for config in self.configs:
            config.seedUI(seeds)

Response: Fetch the seed values from the seeder and update the UI


Instruction:
How would you implement a function in Python 3 that
returns the appropriate layout navigation component based on user preferences

Response:
def buildNavigation(self):
        """
        Chooses the appropriate layout navigation component based on user prefs
        """
        if self.buildSpec['navigation'] == constants.TABBED:
            navigation = Tabbar(self, self.buildSpec, self.configs)
        else:
            navigation = Sidebar(self, self.buildSpec, self.configs)
            if self.buildSpec['navigation'] == constants.HIDDEN:
                navigation.Hide()
        return navigation


Instruction:
Make a summary of the following Python 3 code
def getFontFace(self):
        userFace = self.buildSpec['terminal_font_family'] or self.defaultFont.GetFaceName()
        return (''
                if self.buildSpec['monospace_display']
                else userFace)

Response: Choose the best font face available given the user options


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _forward_stdout(self, process):
        '''
        Reads the stdout of `process` and forwards lines and progress
        to any interested subscribers
        '''
        while True:
            line = process.stdout.readline()
            if not line:
                break
            pub.send_message(events.CONSOLE_UPDATE, msg=line.decode(self.encoding))
            pub.send_message(events.PROGRESS_UPDATE,
                             progress=self._extract_progress(line))
        pub.send_message(events.EXECUTION_COMPLETE)

Response: Reads the stdout of process and forwards lines and progress to any interested subscribers.


Instruction:
Create a Python 3 function to
extract the progress information from the text using the progress_regex and calculate instructions

Response:
def _extract_progress(self, text):
        '''
        Finds progress information in the text using the
        user-supplied regex and calculation instructions
        '''
        # monad-ish dispatch to avoid the if/else soup
        find = partial(re.search, string=text.strip().decode(self.encoding))
        regex = unit(self.progress_regex)
        match = bind(regex, find)
        result = bind(match, self._calculate_progress)
        return result


Instruction:
How would you implement a function in Python 3 that
calculates the final progress value found by the regex match

Response:
def _calculate_progress(self, match):
        '''
        Calculates the final progress value found by the regex
        '''
        if not self.progress_expr:
            return safe_float(match.group(1))
        else:
            return self._eval_progress(match)


Instruction:
Can you implement a function in Python 3 that
hides and show configuration panels based on the currently selected option in the sidebar

Response:
def swapConfigPanels(self, event):
        """Hide/show configuration panels based on the currently selected
         option in the sidebar """
        for id, panel in enumerate(self.configPanels):
            panel.Hide()
        self.activeSelection = event.Selection
        self.configPanels[event.Selection].Show()
        self._parent.Layout()


Instruction:
Here you have a function in Python 3, explain what it does
def updateProgressBar(self, *args, **kwargs):
        '''
         value, disable_animation=False
        :param args:
        :param kwargs:
        :return:
        '''
        value = kwargs.get('progress')
        pb = self.progress_bar
        if value is None:
            return
        if value < 0:
            pb.Pulse()
        else:
            value = min(int(value), pb.GetRange())
            if pb.GetValue() != value:
                # Windows 7 progress bar animation hack
                # http://stackoverflow.com/questions/5332616/disabling-net-progressbar-animation-when-changing-value
                if self.buildSpec['disable_progress_bar_animation'] \
                        and sys.platform.startswith("win"):
                    if pb.GetRange() == value:
                        pb.SetValue(value)
                        pb.SetValue(value - 1)
                    else:
                        pb.SetValue(value + 1)
                pb.SetValue(value)

Response: Updates the progress bar with the new value.


Instruction:
Here you have a function in Python 3, explain what it does
def fetchDynamicProperties(target, encoding):
    cmd = '{} {}'.format(target, 'gooey-seed-ui --ignore-gooey')
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    if proc.returncode != 0:
        out, _ = proc.communicate()
        return json.loads(out.decode(encoding))
    else:
        # TODO: useful feedback
        return {}

Response: Sends a gooey - seed - ui request to the client program it retrieve dynamically generated defaults with which to seed the UI


Instruction:
How would you code a function in Python 3 to
create a group of items and returns a wx. BoxSizer instance that can be used to create the group of items.

Response:
def makeGroup(self, parent, thissizer, group, *args):
        '''
        Messily builds the (potentially) nested and grouped layout

        Note! Mutates `self.reifiedWidgets` in place with the widgets as they're
        instantiated! I cannot figure out how to split out the creation of the
        widgets from their styling without WxPython violently exploding

        TODO: sort out the WX quirks and clean this up.
        '''

        # determine the type of border , if any, the main sizer will use
        if getin(group, ['options', 'show_border'], False):
            boxDetails = wx.StaticBox(parent, -1, group['name'] or '')
            boxSizer = wx.StaticBoxSizer(boxDetails, wx.VERTICAL)
        else:
            boxSizer = wx.BoxSizer(wx.VERTICAL)
            boxSizer.AddSpacer(10)
            if group['name']:
                boxSizer.Add(wx_util.h1(parent, group['name'] or ''), 0, wx.TOP | wx.BOTTOM | wx.LEFT, 8)

        group_description = getin(group, ['description'])
        if group_description:
            description = wx.StaticText(parent, label=group_description)
            boxSizer.Add(description, 0,  wx.EXPAND | wx.LEFT, 10)

        # apply an underline when a grouping border is not specified
        if not getin(group, ['options', 'show_border'], False) and group['name']:
            boxSizer.Add(wx_util.horizontal_rule(parent), 0, wx.EXPAND | wx.LEFT, 10)

        ui_groups = self.chunkWidgets(group)

        for uigroup in ui_groups:
            sizer = wx.BoxSizer(wx.HORIZONTAL)
            for item in uigroup:
                widget = self.reifyWidget(parent, item)
                # !Mutate the reifiedWidgets instance variable in place
                self.reifiedWidgets.append(widget)
                sizer.Add(widget, 1, wx.ALL, 5)
            boxSizer.Add(sizer, 0, wx.ALL | wx.EXPAND, 5)

        # apply the same layout rules recursively for subgroups
        hs = wx.BoxSizer(wx.HORIZONTAL)
        for e, subgroup in enumerate(group['groups']):
            self.makeGroup(parent, hs, subgroup, 1, wx.ALL | wx.EXPAND, 5)
            if e % getin(group, ['options', 'columns'], 2) \
                    or e == len(group['groups']):
                boxSizer.Add(hs, *args)
                hs = wx.BoxSizer(wx.HORIZONTAL)

        thissizer.Add(boxSizer, *args)


Instruction:
Write a Python 3 function that can
chunk the widgets up into groups based on their sizing hints

Response:
def chunkWidgets(self, group):
        ''' chunk the widgets up into groups based on their sizing hints '''
        ui_groups = []
        subgroup = []
        for index, item in enumerate(group['items']):
            if getin(item, ['options', 'full_width'], False):
                ui_groups.append(subgroup)
                ui_groups.append([item])
                subgroup = []
            else:
                subgroup.append(item)
            if len(subgroup) == getin(group, ['options', 'columns'], 2) \
                    or item == group['items'][-1]:
                ui_groups.append(subgroup)
                subgroup = []
        return ui_groups


Instruction:
How would you code a function in Python 3 to
convert a JSON description of a widget into a WxObject

Response:
def reifyWidget(self, parent, item):
        ''' Convert a JSON description of a widget into a WxObject '''
        from gooey.gui.components import widgets
        widgetClass = getattr(widgets, item['type'])
        return widgetClass(parent, item)


Instruction:
Can you implement a function in Python 3 that
returns true if the given action is required.

Response:
def is_required(action):
    '''
    _actions possessing the `required` flag and not implicitly optional
    through `nargs` being '*' or '?'
    '''
    return not isinstance(action, _SubParsersAction) and (
    action.required == True and action.nargs not in ['*', '?'])


Instruction:
Can you create a Python 3 function that
returns True if the given action is standard.

Response:
def is_standard(action):
    """ actions which are general "store" instructions.
    e.g. anything which has an argument style like:
       $ script.py -f myfilename.txt
    """
    boolean_actions = (
        _StoreConstAction, _StoreFalseAction,
        _StoreTrueAction
    )
    return (not action.choices
            and not isinstance(action, _CountAction)
            and not isinstance(action, _HelpAction)
            and type(action) not in boolean_actions)


Instruction:
Can you write a function in Python 3 where it
checks if the given action is a flag.

Response:
def is_flag(action):
    """ _actions which are either storeconst, store_bool, etc.. """
    action_types = [_StoreTrueAction, _StoreFalseAction, _StoreConstAction]
    return any(list(map(lambda Action: isinstance(action, Action), action_types)))


Instruction:
Implement a Python 3 function for
returning the string that will be used to drop down the next value

Response:
def counter(metatdata, value):
    '''
    Returns
      str(option_string * DropDown Value)
      e.g.
      -vvvvv
    '''
    if not str(value).isdigit():
        return None
    arg = str(metatdata['commands'][0]).replace('-', '')
    repeated_args = arg * int(value)
    return '-' + repeated_args


Instruction:
Here you have a function in Python 3, explain what it does
def load(language_dir, filename, encoding):
  ''' Open and return the supplied json file '''
  global _DICTIONARY
  try:
    json_file = filename + '.json'
    with io.open(os.path.join(language_dir, json_file), 'r', encoding=encoding) as f:
      _DICTIONARY = json.load(f)
  except IOError:
    raise IOError('{0} Language file not found at location {1}. '
                  'Make sure that your translation file is in the '
                  'listed language directory'.format(filename.title(), language_dir))

Response: Open and return the supplied json file


Instruction:
Implement a Python 3 function for
returning the value in a nested dict

Response:
def getin(m, path, default=None):
    """returns the value in a nested dict"""
    keynotfound = ':com.gooey-project/not-found'
    result = reduce(lambda acc, val: acc.get(val, {keynotfound: None}), path, m)
    # falsey values like 0 would incorrectly trigger the default to be returned
    # so the keynotfound val is used to signify a miss vs just a falesy val
    if isinstance(result, dict) and keynotfound in result:
        return default
    return result


Instruction:
Can you implement a function in Python 3 that
copies - on - write associates a value in a dict

Response:
def assoc(m, key, val):
    """Copy-on-write associates a value in a dict"""
    cpy = deepcopy(m)
    cpy[key] = val
    return cpy


Instruction:
Write a Python 3 function that can
copy - on - write associates a value in a nested dict

Response:
def associn(m, path, value):
    """ Copy-on-write associates a value in a nested dict """
    def assoc_recursively(m, path, value):
        if not path:
            return value
        p = path[0]
        return assoc(m, p, assoc_recursively(m.get(p,{}), path[1:], value))
    return assoc_recursively(m, path, value)


Instruction:
Write a Python 3 function for
merging all maps left to right

Response:
def merge(*maps):
    """Merge all maps left to right"""
    copies = map(deepcopy, maps)
    return reduce(lambda acc, val: acc.update(val) or acc, copies)


Instruction:
Make a summary of the following Python 3 code
def findfirst(f, coll):
    result = list(dropwhile(f, coll))
    return result[0] if result else None

Response: Return first occurrence matching f otherwise None


Instruction:
Implement a function in Python 3 to
generate the input_tensor that minimizes the weighted losses.

Response:
def visualize_activation_with_losses(input_tensor, losses, wrt_tensor=None,
                                     seed_input=None, input_range=(0, 255),
                                     **optimizer_params):
    """Generates the `input_tensor` that minimizes the weighted `losses`. This function is intended for advanced
    use cases where a custom loss is desired.

    Args:
        input_tensor: An input tensor of shape: `(samples, channels, image_dims...)` if `image_data_format=
            channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.
        wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.
            When None, this is assumed to be the same as `input_tensor` (Default value: None)
        losses: List of ([Loss](vis.losses#Loss), weight) tuples.
        seed_input: Seeds the optimization with a starting image. Initialized with a random value when set to None.
            (Default value = None)
        input_range: Specifies the input range as a `(min, max)` tuple. This is used to rescale the
            final optimized input to the given range. (Default value=(0, 255))
        optimizer_params: The **kwargs for optimizer [params](vis.optimizer#optimizerminimize). Will default to
            reasonable values when required keys are not found.

    Returns:
        The model input that minimizes the weighted `losses`.
    """
    # Default optimizer kwargs.
    optimizer_params = utils.add_defaults_to_kwargs({
        'seed_input': seed_input,
        'max_iter': 200,
        'verbose': False
    }, **optimizer_params)

    opt = Optimizer(input_tensor, losses, input_range, wrt_tensor=wrt_tensor)
    img = opt.minimize(**optimizer_params)[0]

    # If range has integer numbers, cast to 'uint8'
    if isinstance(input_range[0], int) and isinstance(input_range[1], int):
        img = np.clip(img, input_range[0], input_range[1]).astype('uint8')

    if K.image_data_format() == 'channels_first':
        img = np.moveaxis(img, 0, -1)
    return img


Instruction:
Make a summary of the following Python 3 code
def visualize_activation(model, layer_idx, filter_indices=None, wrt_tensor=None,
                         seed_input=None, input_range=(0, 255),
                         backprop_modifier=None, grad_modifier=None,
                         act_max_weight=1, lp_norm_weight=10, tv_weight=10,
                         **optimizer_params):
    """Generates the model input that maximizes the output of all `filter_indices` in the given `layer_idx`.

    Args:
        model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`
            if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if
            `image_data_format=channels_last`.
        layer_idx: The layer index within `model.layers` whose filters needs to be visualized.
        filter_indices: filter indices within the layer to be maximized.
            If None, all filters are visualized. (Default value = None)
            For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.
            If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for
            'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.
        wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.
            When None, this is assumed to be the same as `input_tensor` (Default value: None)
        seed_input: Seeds the optimization with a starting input. Initialized with a random value when set to None.
            (Default value = None)
        input_range: Specifies the input range as a `(min, max)` tuple. This is used to rescale the
            final optimized input to the given range. (Default value=(0, 255))
        backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't
            specify anything, no backprop modification is applied. (Default value = None)
        grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't
            specify anything, gradients are unchanged (Default value = None)
        act_max_weight: The weight param for `ActivationMaximization` loss. Not used if 0 or None. (Default value = 1)
        lp_norm_weight: The weight param for `LPNorm` regularization loss. Not used if 0 or None. (Default value = 10)
        tv_weight: The weight param for `TotalVariation` regularization loss. Not used if 0 or None. (Default value = 10)
        optimizer_params: The **kwargs for optimizer [params](vis.optimizer#optimizerminimize). Will default to
            reasonable values when required keys are not found.

    Example:
        If you wanted to visualize the input image that would maximize the output index 22, say on
        final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer_idx = dense_layer_idx`.

        If `filter_indices = [22, 23]`, then it should generate an input image that shows features of both classes.

    Returns:
        The model input that maximizes the output of `filter_indices` in the given `layer_idx`.
    """
    if backprop_modifier is not None:
        modifier_fn = get(backprop_modifier)
        model = modifier_fn(model)

    losses = [
        (ActivationMaximization(model.layers[layer_idx], filter_indices), act_max_weight),
        (LPNorm(model.input), lp_norm_weight),
        (TotalVariation(model.input), tv_weight)
    ]

    # Add grad_filter to optimizer_params.
    optimizer_params = utils.add_defaults_to_kwargs({
        'grad_modifier': grad_modifier
    }, **optimizer_params)

    return visualize_activation_with_losses(model.input, losses, wrt_tensor,
                                            seed_input, input_range, **optimizer_params)

Response: Generates the model input that maximizes the output of all filter_indices in the given layer_idx.


Instruction:
Create a Python 3 function to
use RMSProp to compute step from gradients.

Response:
def _rmsprop(self, grads, cache=None, decay_rate=0.95):
        """Uses RMSProp to compute step from gradients.

        Args:
            grads: numpy array of gradients.
            cache: numpy array of same shape as `grads` as RMSProp cache
            decay_rate: How fast to decay cache

        Returns:
            A tuple of
                step: numpy array of the same shape as `grads` giving the step.
                    Note that this does not yet take the learning rate into account.
                cache: Updated RMSProp cache.
        """
        if cache is None:
            cache = np.zeros_like(grads)
        cache = decay_rate * cache + (1 - decay_rate) * grads ** 2
        step = -grads / np.sqrt(cache + K.epsilon())
        return step, cache


Instruction:
How would you code a function in Python 3 to
create a random seed_input if None. Otherwise returns a random seed_input.

Response:
def _get_seed_input(self, seed_input):
        """Creates a random `seed_input` if None. Otherwise:
            - Ensures batch_size dim on provided `seed_input`.
            - Shuffle axis according to expected `image_data_format`.
        """
        desired_shape = (1, ) + K.int_shape(self.input_tensor)[1:]
        if seed_input is None:
            return utils.random_array(desired_shape, mean=np.mean(self.input_range),
                                      std=0.05 * (self.input_range[1] - self.input_range[0]))

        # Add batch dim if needed.
        if len(seed_input.shape) != len(desired_shape):
            seed_input = np.expand_dims(seed_input, 0)

        # Only possible if channel idx is out of place.
        if seed_input.shape[-1] != desired_shape[-1] and \
           seed_input.shape[1] != desired_shape[1]:
            seed_input = np.moveaxis(seed_input, -1, 1)
        return seed_input.astype(K.floatx())


Instruction:
Implement a function in Python 3 to
perform gradient descent on the input image with respect to defined losses.

Response:
def minimize(self, seed_input=None, max_iter=200,
                 input_modifiers=None, grad_modifier=None,
                 callbacks=None, verbose=True):
        """Performs gradient descent on the input image with respect to defined losses.

        Args:
            seed_input: An N-dim numpy array of shape: `(samples, channels, image_dims...)` if `image_data_format=
                channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.
                Seeded with random noise if set to None. (Default value = None)
            max_iter: The maximum number of gradient descent iterations. (Default value = 200)
            input_modifiers: A list of [InputModifier](vis.input_modifiers#inputmodifier) instances specifying
                how to make `pre` and `post` changes to the optimized input during the optimization process.
                `pre` is applied in list order while `post` is applied in reverse order. For example,
                `input_modifiers = [f, g]` means that `pre_input = g(f(inp))` and `post_input = f(g(inp))`
            grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't
                specify anything, gradients are unchanged. (Default value = None)
            callbacks: A list of [OptimizerCallback](vis.callbacks#optimizercallback) instances to trigger.
            verbose: Logs individual losses at the end of every gradient descent iteration.
                Very useful to estimate loss weight factor(s). (Default value = True)

        Returns:
            The tuple of `(optimized input, grads with respect to wrt, wrt_value)` after gradient descent iterations.
        """
        seed_input = self._get_seed_input(seed_input)
        input_modifiers = input_modifiers or []
        grad_modifier = _identity if grad_modifier is None else get(grad_modifier)

        callbacks = callbacks or []
        if verbose:
            callbacks.append(_PRINT_CALLBACK)

        cache = None
        best_loss = float('inf')
        best_input = None

        grads = None
        wrt_value = None

        for i in range(max_iter):
            # Apply modifiers `pre` step
            for modifier in input_modifiers:
                seed_input = modifier.pre(seed_input)

            # 0 learning phase for 'test'
            computed_values = self.compute_fn([seed_input, 0])
            losses = computed_values[:len(self.loss_names)]
            named_losses = list(zip(self.loss_names, losses))
            overall_loss, grads, wrt_value = computed_values[len(self.loss_names):]

            # TODO: theano grads shape is inconsistent for some reason. Patch for now and investigate later.
            if grads.shape != wrt_value.shape:
                grads = np.reshape(grads, wrt_value.shape)

            # Apply grad modifier.
            grads = grad_modifier(grads)

            # Trigger callbacks
            for c in callbacks:
                c.callback(i, named_losses, overall_loss, grads, wrt_value)

            # Gradient descent update.
            # It only makes sense to do this if wrt_tensor is input_tensor. Otherwise shapes wont match for the update.
            if self.wrt_tensor_is_input_tensor:
                step, cache = self._rmsprop(grads, cache)
                seed_input += step

            # Apply modifiers `post` step
            for modifier in reversed(input_modifiers):
                seed_input = modifier.post(seed_input)

            if overall_loss < best_loss:
                best_loss = overall_loss.copy()
                best_input = seed_input.copy()

        # Trigger on_end
        for c in callbacks:
            c.on_end()

        return utils.deprocess_input(best_input[0], self.input_range), grads, wrt_value


Instruction:
Here you have a function in Python 3, explain what it does
def order_by_line_nos(objs, line_nos):
    ordering = sorted(range(len(line_nos)), key=line_nos.__getitem__)
    return [objs[i] for i in ordering]

Response: Orders the set of objects by line_nos


Instruction:
Given the following Python 3 function, write the documentation
def to_md_file(string, filename, out_path="."):
    md_file = "%s.md" % filename
    with open(os.path.join(out_path, md_file), "w") as f:
        f.write(string)
    print("wrote {}.".format(md_file))

Response: Write a string with line breaks to a. md file.


Instruction:
Write a Python 3 script to
create a src path string with line info for use as markdown link.

Response:
def get_src_path(self, obj, append_base=True):
        """Creates a src path string with line info for use as markdown link.
        """
        path = getsourcefile(obj)
        if self.src_root not in path:
            # this can happen with e.g.
            # inlinefunc-wrapped functions
            if hasattr(obj, "__module__"):
                path = "%s.%s" % (obj.__module__, obj.__name__)
            else:
                path = obj.__name__
            path = path.replace(".", "/")
        pre, post = path.rsplit(self.src_root + "/", 1)

        lineno = self.get_line_no(obj)
        lineno = "" if lineno is None else "#L{}".format(lineno)

        path = self.src_root + "/" + post + lineno
        if append_base:
            path = os.path.join(self.github_link, path)
        return path


Instruction:
Here you have a function in Python 3, explain what it does
def doc2md(self, func):
        doc = getdoc(func) or ""
        blockindent = 0
        argindent = 1
        out = []

        for line in doc.split("\n"):
            indent = len(line) - len(line.lstrip())
            line = line.lstrip()
            if _RE_BLOCKSTART.match(line):
                # start of a new block
                blockindent = indent
                out.append("\n*{}*\n".format(line))
            elif indent > blockindent:
                if _RE_ARGSTART.match(line):
                    # start of new argument
                    out.append("\n" + " " * blockindent + " - " + _RE_ARGSTART.sub(r"**\1** (\2): \3", line))
                    argindent = indent
                elif _RE_EXCSTART.match(line):
                    # start of an exception-type block
                    out.append("\n" + " " * blockindent + " - " + _RE_EXCSTART.sub(r"**\1**: \2", line))
                    argindent = indent
                elif indent > argindent:
                    out.append("\n" + " " * (blockindent + 2) + line)
                else:
                    out.append("\n" + line)
            else:
                out.append("\n" + line)

        return "".join(out)

Response: Parse the docstring of a function and return a markdown string.


Instruction:
Can you implement a function in Python 3 that
takes a function or method and returns a string that contains the contents of the function and its docstring.

Response:
def func2md(self, func, clsname=None, names=None, depth=3):
        """Takes a function (or method) and documents it.

        Args:
            clsname (str, optional): class name to prepend to funcname.
            depth (int, optional): number of ### to append to function name

        """
        section = "#" * depth
        if names is None:
            names = [func.__name__]

        funcname = ", ".join(names)
        escfuncname = ", ".join(["`%s`" % funcname if funcname.startswith("_") else funcname for funcname in names])
        header = "%s%s" % ("%s." % clsname if clsname else "", escfuncname)

        path = self.get_src_path(func)
        doc = self.doc2md(func)

        args, kwargs = [], []
        spec = getargspec(func)
        vargsname, kwargsname = spec.varargs, spec.keywords
        vargs = list(make_iter(spec.args)) if spec.args else []
        defaults = list(make_iter(spec.defaults)) if spec.defaults else []

        while vargs:
            if vargs and vargs[0] == "self":
                args.append(vargs.pop(0))
            elif len(vargs) > len(defaults):
                args.append(vargs.pop(0))
            else:
                default = defaults.pop(0)
                if isinstance(default, str):
                    default = "\"%s\"" % default
                else:
                    default = "%s" % str(default)

                kwargs.append((vargs.pop(0), default))

        if args:
            args = ", ".join("%s" % arg for arg in args)
        if kwargs:
            kwargs = ", ".join("%s=%s" % kwarg for kwarg in kwargs)
            if args:
                kwargs = ", " + kwargs
        if vargsname:
            vargsname = "*%s" % vargsname
            if args or kwargs:
                vargsname = ", " + vargsname
        if kwargsname:
            kwargsname = "**%s" % kwargsname
            if args or kwargs or vargsname:
                kwargsname = ", " + kwargsname

        _FUNCDEF = "{funcname}({args}{kwargs}{vargs}{vkwargs})"
        funcdef = _FUNCDEF.format(funcname=funcname,
                                  args=args or "",
                                  kwargs=kwargs or "",
                                  vargs=vargsname or "",
                                  vkwargs=kwargsname or "")

        # split the function definition if it is too long
        lmax = 90
        if len(funcdef) > lmax:
            # wrap in the args list
            split = funcdef.split("(", 1)
            # we gradually build the string again
            rest = split[1]
            args = rest.split(", ")

            funcname = "(".join(split[:1]) + "("
            lline = len(funcname)
            parts = []
            for arg in args:
                larg = len(arg)
                if larg > lmax - 5:
                    # not much to do if arg is so long
                    parts.append(arg)
                elif lline + larg > lmax:
                    # the next arg is too long, break the line
                    parts.append("\\\n    " + arg)
                    lline = 0
                else:
                    parts.append(arg)
                lline += len(parts[-1])
            funcdef = funcname + ", ".join(parts)

        # build the signature
        string = FUNC_TEMPLATE.format(section=section,
                                      header=header,
                                      funcdef=funcdef,
                                      path=path,
                                      doc=doc if doc else "*No documentation found.*")
        return string


Instruction:
Can you write a function in Python 3 where it
takes a class and creates markdown text to document its methods and variables.

Response:
def class2md(self, cls, depth=2):
        """Takes a class and creates markdown text to document its methods and variables.
        """

        section = "#" * depth
        subsection = "#" * (depth + 2)
        clsname = cls.__name__
        modname = cls.__module__
        header = clsname
        path = self.get_src_path(cls)
        doc = self.doc2md(cls)

        try:
            init = self.func2md(cls.__init__, clsname=clsname)
        except (ValueError, TypeError):
            # this happens if __init__ is outside the repo
            init = ""

        variables = []
        for name, obj in getmembers(cls, lambda a: not (inspect.isroutine(a) or inspect.ismethod(a))):
            if not name.startswith("_") and type(obj) == property:
                comments = self.doc2md(obj) or inspect.getcomments(obj)
                comments = "\n %s" % comments if comments else ""
                variables.append("\n%s %s.%s%s\n" % (subsection, clsname, name, comments))

        handlers = []
        for name, obj in getmembers(cls, inspect.ismethoddescriptor):
            if not name.startswith("_") and hasattr(obj, "__module__") and obj.__module__ == modname:
                handlers.append("\n%s %s.%s\n *Handler*" % (subsection, clsname, name))

        methods = []
        for name, obj in getmembers(cls, inspect.ismethod):
            if not name.startswith("_") and hasattr(obj,
                                                    "__module__") and obj.__module__ == modname and name not in handlers:
                methods.append(self.func2md(obj, clsname=clsname, depth=depth + 1))

        string = CLASS_TEMPLATE.format(section=section,
                                       header=header,
                                       path=path,
                                       doc=doc if doc else "",
                                       init=init,
                                       variables="".join(variables),
                                       handlers="".join(handlers),
                                       methods="".join(methods))
        return string


Instruction:
How would you code a function in Python 3 to
take an imported module object and creates a Markdown string containing functions and classes and classes.

Response:
def module2md(self, module):
        """Takes an imported module object and create a Markdown string containing functions and classes.
        """
        modname = module.__name__
        path = self.get_src_path(module, append_base=False)
        path = "[{}]({})".format(path, os.path.join(self.github_link, path))
        found = set()

        classes = []
        line_nos = []
        for name, obj in getmembers(module, inspect.isclass):
            # handle classes
            found.add(name)
            if not name.startswith("_") and hasattr(obj, "__module__") and obj.__module__ == modname:
                classes.append(self.class2md(obj))
                line_nos.append(self.get_line_no(obj) or 0)
        classes = order_by_line_nos(classes, line_nos)

        # Since functions can have multiple aliases.
        func2names = defaultdict(list)
        for name, obj in getmembers(module, inspect.isfunction):
            func2names[obj].append(name)

        functions = []
        line_nos = []
        for obj in func2names:
            names = func2names[obj]
            found.update(names)

            # Include if within module or included modules within __init__.py and exclude from global variables
            is_module_within_init = '__init__.py' in path and obj.__module__.startswith(modname)
            if is_module_within_init:
                found.add(obj.__module__.replace(modname + '.', ''))

            if hasattr(obj, "__module__") and (obj.__module__ == modname or is_module_within_init):
                names = list(filter(lambda name: not name.startswith("_"), names))
                if len(names) > 0:
                    functions.append(self.func2md(obj, names=names))
                    line_nos.append(self.get_line_no(obj) or 0)
        functions = order_by_line_nos(functions, line_nos)

        variables = []
        line_nos = []
        for name, obj in module.__dict__.items():
            if not name.startswith("_") and name not in found:
                if hasattr(obj, "__module__") and obj.__module__ != modname:
                    continue
                if hasattr(obj, "__name__") and not obj.__name__.startswith(modname):
                    continue

                comments = inspect.getcomments(obj)
                comments = ": %s" % comments if comments else ""
                variables.append("- **%s**%s" % (name, comments))
                line_nos.append(self.get_line_no(obj) or 0)

        variables = order_by_line_nos(variables, line_nos)
        if variables:
            new_list = ["**Global Variables**", "---------------"]
            new_list.extend(variables)
            variables = new_list

        string = MODULE_TEMPLATE.format(path=path,
                                        global_vars="\n".join(variables) if variables else "",
                                        functions="\n".join(functions) if functions else "",
                                        classes="".join(classes) if classes else "")
        return string


Instruction:
Can you tell what is the following Python 3 function doing
def _find_penultimate_layer(model, layer_idx, penultimate_layer_idx):
    if penultimate_layer_idx is None:
        for idx, layer in utils.reverse_enumerate(model.layers[:layer_idx - 1]):
            if isinstance(layer, Wrapper):
                layer = layer.layer
            if isinstance(layer, (_Conv, _Pooling1D, _Pooling2D, _Pooling3D)):
                penultimate_layer_idx = idx
                break

    if penultimate_layer_idx is None:
        raise ValueError('Unable to determine penultimate `Conv` or `Pooling` '
                         'layer for layer_idx: {}'.format(layer_idx))

    # Handle negative indexing otherwise the next check can fail.
    if layer_idx < 0:
        layer_idx = len(model.layers) + layer_idx
    if penultimate_layer_idx > layer_idx:
        raise ValueError('`penultimate_layer_idx` needs to be before `layer_idx`')

    return model.layers[penultimate_layer_idx]

Response: Searches for the nearest penultimate CNN or Pooling layer.


Instruction:
Write a Python 3 script for
generating an attention heatmap over the seed_input with respect to weighted losses.

Response:
def visualize_saliency_with_losses(input_tensor, losses, seed_input, wrt_tensor=None, grad_modifier='absolute', keepdims=False):
    """Generates an attention heatmap over the `seed_input` by using positive gradients of `input_tensor`
    with respect to weighted `losses`.

    This function is intended for advanced use cases where a custom loss is desired. For common use cases,
    refer to `visualize_class_saliency` or `visualize_regression_saliency`.

    For a full description of saliency, see the paper:
    [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps]
    (https://arxiv.org/pdf/1312.6034v2.pdf)

    Args:
        input_tensor: An input tensor of shape: `(samples, channels, image_dims...)` if `image_data_format=
            channels_first` or `(samples, image_dims..., channels)` if `image_data_format=channels_last`.
        losses: List of ([Loss](vis.losses#Loss), weight) tuples.
        seed_input: The model input for which activation map needs to be visualized.
        wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.
            When None, this is assumed to be the same as `input_tensor` (Default value: None)
        grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). By default `absolute`
            value of gradients are used. To visualize positive or negative gradients, use `relu` and `negate`
            respectively. (Default value = 'absolute')
        keepdims: A boolean, whether to keep the dimensions or not.
            If keepdims is False, the channels axis is deleted.
            If keepdims is True, the grad with same shape as input_tensor is returned. (Default value: False)

    Returns:
        The normalized gradients of `seed_input` with respect to weighted `losses`.
    """
    opt = Optimizer(input_tensor, losses, wrt_tensor=wrt_tensor, norm_grads=False)
    grads = opt.minimize(seed_input=seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)[1]

    if not keepdims:
        channel_idx = 1 if K.image_data_format() == 'channels_first' else -1
        grads = np.max(grads, axis=channel_idx)
    return utils.normalize(grads)[0]


Instruction:
Can you write a function in Python 3 where it
generates a visualised Saliency layer over the seed_input for maximizing filter_indices.

Response:
def visualize_saliency(model, layer_idx, filter_indices, seed_input, wrt_tensor=None,
                       backprop_modifier=None, grad_modifier='absolute', keepdims=False):
    """Generates an attention heatmap over the `seed_input` for maximizing `filter_indices`
    output in the given `layer_idx`.

    Args:
        model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`
            if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if
            `image_data_format=channels_last`.
        layer_idx: The layer index within `model.layers` whose filters needs to be visualized.
        filter_indices: filter indices within the layer to be maximized.
            If None, all filters are visualized. (Default value = None)
            For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.
            If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for
            'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.
        seed_input: The model input for which activation map needs to be visualized.
        wrt_tensor: Short for, with respect to. The gradients of losses are computed with respect to this tensor.
            When None, this is assumed to be the same as `input_tensor` (Default value: None)
        backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't
            specify anything, no backprop modification is applied. (Default value = None)
        grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). By default `absolute`
            value of gradients are used. To visualize positive or negative gradients, use `relu` and `negate`
            respectively. (Default value = 'absolute')
        keepdims: A boolean, whether to keep the dimensions or not.
            If keepdims is False, the channels axis is deleted.
            If keepdims is True, the grad with same shape as input_tensor is returned. (Default value: False)

    Example:
        If you wanted to visualize attention over 'bird' category, say output index 22 on the
        final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer = dense_layer`.

        One could also set filter indices to more than one value. For example, `filter_indices = [22, 23]` should
        (hopefully) show attention map that corresponds to both 22, 23 output categories.

    Returns:
        The heatmap image indicating the `seed_input` regions whose change would most contribute towards
        maximizing the output of `filter_indices`.
    """
    if backprop_modifier is not None:
        modifier_fn = get(backprop_modifier)
        model = modifier_fn(model)

    # `ActivationMaximization` loss reduces as outputs get large, hence negative gradients indicate the direction
    # for increasing activations. Multiply with -1 so that positive gradients indicate increase instead.
    losses = [
        (ActivationMaximization(model.layers[layer_idx], filter_indices), -1)
    ]
    return visualize_saliency_with_losses(model.input, losses, seed_input, wrt_tensor, grad_modifier, keepdims)


Instruction:
Can you tell what is the following Python 3 function doing
def visualize_cam_with_losses(input_tensor, losses, seed_input, penultimate_layer, grad_modifier=None):
    penultimate_output = penultimate_layer.output
    opt = Optimizer(input_tensor, losses, wrt_tensor=penultimate_output, norm_grads=False)
    _, grads, penultimate_output_value = opt.minimize(seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False)

    # For numerical stability. Very small grad values along with small penultimate_output_value can cause
    # w * penultimate_output_value to zero out, even for reasonable fp precision of float32.
    grads = grads / (np.max(grads) + K.epsilon())

    # Average pooling across all feature maps.
    # This captures the importance of feature map (channel) idx to the output.
    channel_idx = 1 if K.image_data_format() == 'channels_first' else -1
    other_axis = np.delete(np.arange(len(grads.shape)), channel_idx)
    weights = np.mean(grads, axis=tuple(other_axis))

    # Generate heatmap by computing weight * output over feature maps
    output_dims = utils.get_img_shape(penultimate_output)[2:]
    heatmap = np.zeros(shape=output_dims, dtype=K.floatx())
    for i, w in enumerate(weights):
        if channel_idx == -1:
            heatmap += w * penultimate_output_value[0, ..., i]
        else:
            heatmap += w * penultimate_output_value[0, i, ...]

    # ReLU thresholding to exclude pattern mismatch information (negative gradients).
    heatmap = np.maximum(heatmap, 0)

    # The penultimate feature map size is definitely smaller than input image.
    input_dims = utils.get_img_shape(input_tensor)[2:]

    # Figure out the zoom factor.
    zoom_factor = [i / (j * 1.0) for i, j in iter(zip(input_dims, output_dims))]
    heatmap = zoom(heatmap, zoom_factor)
    return utils.normalize(heatmap)

Response: Generates a gradient based class activation map by using positive gradients of input_tensor with respect to weighted losses.


Instruction:
Can you implement a function in Python 3 that
generates a gradient based class activation map that maximizes the outputs of filter_indices in layer_idx.

Response:
def visualize_cam(model, layer_idx, filter_indices,
                  seed_input, penultimate_layer_idx=None,
                  backprop_modifier=None, grad_modifier=None):
    """Generates a gradient based class activation map (grad-CAM) that maximizes the outputs of
    `filter_indices` in `layer_idx`.

    Args:
        model: The `keras.models.Model` instance. The model input shape must be: `(samples, channels, image_dims...)`
            if `image_data_format=channels_first` or `(samples, image_dims..., channels)` if
            `image_data_format=channels_last`.
        layer_idx: The layer index within `model.layers` whose filters needs to be visualized.
        filter_indices: filter indices within the layer to be maximized.
            If None, all filters are visualized. (Default value = None)
            For `keras.layers.Dense` layer, `filter_idx` is interpreted as the output index.
            If you are visualizing final `keras.layers.Dense` layer, consider switching 'softmax' activation for
            'linear' using [utils.apply_modifications](vis.utils.utils#apply_modifications) for better results.
        seed_input: The input image for which activation map needs to be visualized.
        penultimate_layer_idx: The pre-layer to `layer_idx` whose feature maps should be used to compute gradients
            wrt filter output. If not provided, it is set to the nearest penultimate `Conv` or `Pooling` layer.
        backprop_modifier: backprop modifier to use. See [backprop_modifiers](vis.backprop_modifiers.md). If you don't
            specify anything, no backprop modification is applied. (Default value = None)
        grad_modifier: gradient modifier to use. See [grad_modifiers](vis.grad_modifiers.md). If you don't
            specify anything, gradients are unchanged (Default value = None)

     Example:
        If you wanted to visualize attention over 'bird' category, say output index 22 on the
        final `keras.layers.Dense` layer, then, `filter_indices = [22]`, `layer = dense_layer`.

        One could also set filter indices to more than one value. For example, `filter_indices = [22, 23]` should
        (hopefully) show attention map that corresponds to both 22, 23 output categories.

    Returns:
        The heatmap image indicating the input regions whose change would most contribute towards
        maximizing the output of `filter_indices`.
    """
    if backprop_modifier is not None:
        modifier_fn = get(backprop_modifier)
        model = modifier_fn(model)

    penultimate_layer = _find_penultimate_layer(model, layer_idx, penultimate_layer_idx)

    # `ActivationMaximization` outputs negative gradient values for increase in activations. Multiply with -1
    # so that positive gradients indicate increase instead.
    losses = [
        (ActivationMaximization(model.layers[layer_idx], filter_indices), -1)
    ]
    return visualize_cam_with_losses(model.input, losses, seed_input, penultimate_layer, grad_modifier)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def modify_model_backprop(model, backprop_modifier):
    # The general strategy is as follows:
    # - Save original model so that upstream callers don't see unexpected results with their models.
    # - Call backend specific function that registers the custom op and loads the model under modified context manager.
    # - Maintain cache to save this expensive process on subsequent calls.
    # - Load model with custom context modifying backprop behavior.
    #
    # The reason for this round about way is because the graph needs to be rebuild when any of its layer builder
    # functions are changed. This is very complicated to do in Keras and makes the implementation very tightly bound
    # with keras internals. By saving and loading models, we dont have to worry about future compatibility.
    #
    # The only exception to this is the way advanced activations are handled which makes use of some keras internal
    # knowledge and might break in the future.
    # ADD on 22 Jul 2018:
    #     In fact, it has broken. Currently, advanced activations are not supported.

    # 0. Retrieve from cache if previously computed.
    modified_model = _MODIFIED_MODEL_CACHE.get((model, backprop_modifier))
    if modified_model is not None:
        return modified_model

    model_path = os.path.join(tempfile.gettempdir(), next(tempfile._get_candidate_names()) + '.h5')
    try:
        # 1. Save original model
        model.save(model_path)

        # 2. Register modifier and load modified model under custom context.
        modifier_fn = _BACKPROP_MODIFIERS.get(backprop_modifier)
        if modifier_fn is None:
            raise ValueError("'{}' modifier is not supported".format(backprop_modifier))
        modifier_fn(backprop_modifier)

        # 3. Create graph under custom context manager.
        with tf.get_default_graph().gradient_override_map({'Relu': backprop_modifier}):
            #  This should rebuild graph with modifications.
            modified_model = load_model(model_path)

            # Cache to improve subsequent call performance.
            _MODIFIED_MODEL_CACHE[(model, backprop_modifier)] = modified_model
            return modified_model
    finally:
        os.remove(model_path)

Response: Creates a copy of model with modified activations.


Instruction:
Explain what the following Python 3 code does
def normalize(input_tensor, output_tensor):
    image_dims = utils.get_img_shape(input_tensor)[1:]
    return output_tensor / np.prod(image_dims)

Response: Normalizes the output_tensor with respect to input_tensor dimensions.


Instruction:
Explain what the following Python 3 code does
def build_loss(self):
        image_dims = K.ndim(self.img) - 2

        # Constructing slice [1:] + [:-1] * (image_dims - 1) and [:-1] * (image_dims)
        start_slice = [slice(1, None, None)] + [slice(None, -1, None) for _ in range(image_dims - 1)]
        end_slice = [slice(None, -1, None) for _ in range(image_dims)]
        samples_channels_slice = [slice(None, None, None), slice(None, None, None)]

        # Compute pixel diffs by rolling slices to the right per image dim.
        tv = None
        for i in range(image_dims):
            ss = tuple(samples_channels_slice + start_slice)
            es = tuple(samples_channels_slice + end_slice)
            diff_square = K.square(self.img[utils.slicer[ss]] - self.img[utils.slicer[es]])
            tv = diff_square if tv is None else tv + diff_square

            # Roll over to next image dim
            start_slice = np.roll(start_slice, 1).tolist()
            end_slice = np.roll(end_slice, 1).tolist()

        tv = K.sum(K.pow(tv, self.beta / 2.))
        return normalize(self.img, tv)

Response:
Implements the N - dim version of function
       $$TV^{ beta } ( x ) = \ sum_{whc } \ right


Instruction:
Given the following Python 3 function, write the documentation
def _find_font_file(query):
    return list(filter(lambda path: query.lower() in os.path.basename(path).lower(), fontman.findSystemFonts()))

Response: Utility to find font file.


Instruction:
Can you generate the documentation for the following Python 3 function
def add_defaults_to_kwargs(defaults, **kwargs):
    defaults = dict(defaults)
    defaults.update(kwargs)
    return defaults

Response: Updates kwargs with dict of defaults