Instruction:
Given the following Python 3 function, write the documentation
def kw_fkg_allowing_type_hints(
        namespace: Optional[str],
        fn: Callable,
        to_str: Callable[[Any], str] = repr) -> Callable[[Any], str]:
    """
    As for :func:`fkg_allowing_type_hints`, but allowing keyword arguments.

    For ``kwargs`` passed in, we will build a ``dict`` of all argname (key) to
    argvalue (values) pairs, including default args from the argspec, and then
    alphabetize the list before generating the key.

    NOTE ALSO that once we have keyword arguments, we should be using
    :func:`repr`, because we need to distinguish

    .. code-block:: python

        kwargs = {'p': 'another', 'q': 'thing'}
        # ... which compat.string_type will make into
        #         p=another q=thing
        # ... from
        kwargs = {'p': 'another q=thing'}

    Also modified to make the cached function unique per INSTANCE for normal
    methods of a class.
    """

    namespace = get_namespace(fn, namespace)

    sig = inspect.signature(fn)
    parameters = list(sig.parameters.values())  # convert from odict_values
    argnames = [p.name for p in parameters
                if p.kind == inspect.Parameter.POSITIONAL_OR_KEYWORD]
    has_self = bool(argnames and argnames[0] in ('self', 'cls'))

    if DEBUG_INTERNALS:
        log.debug(
            "At start of kw_fkg_allowing_type_hints: namespace={namespace},"
            "parameters=[{parameters}], argnames={argnames}, "
            "has_self={has_self}, fn={fn}",
            namespace=namespace,
            parameters=", ".join(repr_parameter(p) for p in parameters),
            argnames=repr(argnames),
            has_self=has_self,
            fn=repr(fn),
        )

    def generate_key(*args: Any, **kwargs: Any) -> str:
        as_kwargs = {}  # type: Dict[str, Any]
        loose_args = []  # type: List[Any]  # those captured by *args
        # 1. args: get the name as well.
        for idx, arg in enumerate(args):
            if idx >= len(argnames):
                # positional argument to be scooped up with *args
                loose_args.append(arg)
            else:
                # normal plain positional argument
                if has_self and idx == 0:  # "self" or "cls" initial argument
                    argvalue = hex(id(arg))
                else:
                    argvalue = arg
                as_kwargs[argnames[idx]] = argvalue
        # 1b. args with no name
        if loose_args:
            as_kwargs['*args'] = loose_args
            # '*args' is guaranteed not to be a parameter name in its own right
        # 2. kwargs
        as_kwargs.update(kwargs)
        # 3. default values
        for param in parameters:
            if param.default != inspect.Parameter.empty:
                if param.name not in as_kwargs:
                    as_kwargs[param.name] = param.default
        # 4. sorted by name
        #    ... but also incorporating the name of the argument, because once
        #    we allow the arbitrary **kwargs format, order is no longer
        #    sufficient to discriminate
        #       fn(p="another", q="thing")
        #    from
        #       fn(r="another", s="thing")
        argument_values = ["{k}={v}".format(k=key, v=to_str(as_kwargs[key]))
                           for key in sorted(as_kwargs.keys())]
        key = namespace + '|' + " ".join(argument_values)
        if DEBUG_INTERNALS:
            log.debug("kw_fkg_allowing_type_hints.generate_key() -> {!r}", key)
        return key

    return generate_key

Response: A function that allows keyword arguments to be used in a cached function.


Instruction:
Write a Python 3 function for
returning the GPG version

Response:
def gpg_version():
    """Returns the GPG version"""
    cmd = flatten([gnupg_bin(), "--version"])
    output = stderr_output(cmd)
    output = output \
        .split('\n')[0] \
        .split(" ")[2] \
        .split('.')
    return tuple([int(x) for x in output])


Instruction:
Explain what the following Python 3 code does
def polite_string(a_string):
    if is_py3() and hasattr(a_string, 'decode'):
        try:
            return a_string.decode('utf-8')
        except UnicodeDecodeError:
            return a_string

    return a_string

Response: Returns a string that should work in both Py3 and Py2


Instruction:
Write a Python 3 function that can
flatten iterables This is super ugly. There must be a cleaner py2 / 3 way to handle this.

Response:
def actually_flatten(iterable):
    """Flatten iterables
    This is super ugly. There must be a cleaner py2/3 way
    of handling this."""
    remainder = iter(iterable)
    while True:
        first = next(remainder)  # pylint: disable=R1708
        # Python 2/3 compat
        is_iter = isinstance(first, collections.Iterable)
        try:
            basestring
        except NameError:
            basestring = str  # pylint: disable=W0622

        if is_py3() and is_iter and not_a_string(first):
            remainder = IT.chain(first, remainder)
        elif (not is_py3()) and is_iter and not isinstance(first, basestring):
            remainder = IT.chain(first, remainder)
        else:
            yield polite_string(first)


Instruction:
Explain what the following Python 3 code does
def passphrase_file(passphrase=None):
    cmd = []
    pass_file = None
    if not passphrase and 'CRYPTORITO_PASSPHRASE_FILE' in os.environ:
        pass_file = os.environ['CRYPTORITO_PASSPHRASE_FILE']
        if not os.path.isfile(pass_file):
            raise CryptoritoError('CRYPTORITO_PASSPHRASE_FILE is invalid')
    elif passphrase:
        tmpdir = ensure_tmpdir()
        pass_file = "%s/p_pass" % tmpdir
        p_handle = open(pass_file, 'w')
        p_handle.write(passphrase)
        p_handle.close()

    if pass_file:
        cmd = cmd + ["--batch", "--passphrase-file", pass_file]

        vsn = gpg_version()
        if vsn[0] >= 2 and vsn[1] >= 1:
            cmd = cmd + ["--pinentry-mode", "loopback"]

    return cmd

Response: Read passphrase from a file.


Instruction:
Create a Python 3 function for
returning appropriate arguments for GNUPGHOME.

Response:
def gnupg_home():
    """Returns appropriate arguments if GNUPGHOME is set"""
    if 'GNUPGHOME' in os.environ:
        gnupghome = os.environ['GNUPGHOME']
        if not os.path.isdir(gnupghome):
            raise CryptoritoError("Invalid GNUPGHOME directory")

        return ["--homedir", gnupghome]
    else:
        return []


Instruction:
Make a summary of the following Python 3 code
def fingerprint_from_keybase(fingerprint, kb_obj):
    if 'public_keys' in kb_obj and \
       'pgp_public_keys' in kb_obj['public_keys']:
        for key in kb_obj['public_keys']['pgp_public_keys']:
            keyprint = fingerprint_from_var(key).lower()
            fingerprint = fingerprint.lower()
            if fingerprint == keyprint or \
               keyprint.startswith(fingerprint) or \
               keyprint.endswith(fingerprint):
                return {
                    'fingerprint': keyprint,
                    'bundle': key
                }

    return None

Response: Extracts a key matching a specific fingerprint from a Keybase API response


Instruction:
Can you write a function in Python 3 where it
looks up a public key from a username

Response:
def key_from_keybase(username, fingerprint=None):
    """Look up a public key from a username"""
    url = keybase_lookup_url(username)
    resp = requests.get(url)
    if resp.status_code == 200:
        j_resp = json.loads(polite_string(resp.content))
        if 'them' in j_resp and len(j_resp['them']) == 1:
            kb_obj = j_resp['them'][0]
            if fingerprint:
                return fingerprint_from_keybase(fingerprint, kb_obj)
            else:
                if 'public_keys' in kb_obj \
                   and 'pgp_public_keys' in kb_obj['public_keys']:
                    key = kb_obj['public_keys']['primary']
                    return massage_key(key)

    return None


Instruction:
Make a summary of the following Python 3 code
def has_gpg_key(fingerprint):
    if len(fingerprint) > 8:
        fingerprint = fingerprint[-8:]

    fingerprint = fingerprint.upper()
    cmd = flatten([gnupg_bin(), gnupg_home(), "--list-public-keys"])
    lines = stderr_output(cmd).split('\n')
    return len([key for key in lines if key.find(fingerprint) > -1]) == 1

Response: Checks to see if we have this gpg key


Instruction:
Can you tell what is the following Python 3 function doing
def fingerprint_from_var(var):
    vsn = gpg_version()
    cmd = flatten([gnupg_bin(), gnupg_home()])
    if vsn[0] >= 2 and vsn[1] < 1:
        cmd.append("--with-fingerprint")

    output = polite_string(stderr_with_input(cmd, var)).split('\n')
    if not output[0].startswith('pub'):
        raise CryptoritoError('probably an invalid gpg key')

    if vsn[0] >= 2 and vsn[1] < 1:
        return output[1] \
            .split('=')[1] \
            .replace(' ', '')

    return output[1].strip()

Response: Extract a fingerprint from a GPG public key


Instruction:
Make a summary of the following Python 3 code
def fingerprint_from_file(filename):
    cmd = flatten([gnupg_bin(), gnupg_home(), filename])
    outp = stderr_output(cmd).split('\n')
    if not outp[0].startswith('pub'):
        raise CryptoritoError('probably an invalid gpg key')

    return outp[1].strip()

Response: Extract a fingerprint from a GPG public key file


Instruction:
Write a Python 3 function that can
generate debug - level appropriate stderr context for returning the stderr handle.

Response:
def stderr_handle():
    """Generate debug-level appropriate stderr context for
    executing things through subprocess. Normally stderr gets
    sent to dev/null but when debugging it is sent to stdout."""
    gpg_stderr = None
    handle = None
    if LOGGER.getEffectiveLevel() > logging.DEBUG:
        handle = open(os.devnull, 'wb')
        gpg_stderr = handle

    return handle, gpg_stderr


Instruction:
How would you explain what the following Python 3 function does
def stderr_output(cmd):

    handle, gpg_stderr = stderr_handle()
    try:
        output = subprocess.check_output(cmd, stderr=gpg_stderr)  # nosec
        if handle:
            handle.close()

        return str(polite_string(output))
    except subprocess.CalledProcessError as exception:
        LOGGER.debug("GPG Command %s", ' '.join(exception.cmd))
        LOGGER.debug("GPG Output %s", exception.output)
        raise CryptoritoError('GPG Execution')

Response: Wraps the execution of check_output in a way that returns the output of the command and closes the stderr handle.


Instruction:
Explain what the following Python 3 code does
def stderr_with_input(cmd, stdin):
    handle, gpg_stderr = stderr_handle()
    LOGGER.debug("GPG command %s", ' '.join(cmd))
    try:
        gpg_proc = subprocess.Popen(cmd,  # nosec
                                    stdout=subprocess.PIPE,
                                    stdin=subprocess.PIPE,
                                    stderr=gpg_stderr)

        output, _err = gpg_proc.communicate(polite_bytes(stdin))

        if handle:
            handle.close()

        return output
    except subprocess.CalledProcessError as exception:
        return gpg_error(exception, 'GPG variable encryption error')
    except OSError as exception:
        raise CryptoritoError("File %s not found" % exception.filename)

Response:
Runs a command passing something in stdin and returning
Taxonomy whatever came out from stdout


Instruction:
How would you explain what the following Python 3 function does
def import_gpg_key(key):
    if not key:
        raise CryptoritoError('Invalid GPG Key')

    key_fd, key_filename = mkstemp("cryptorito-gpg-import")
    key_handle = os.fdopen(key_fd, 'w')

    key_handle.write(polite_string(key))
    key_handle.close()
    cmd = flatten([gnupg_bin(), gnupg_home(), "--import", key_filename])
    output = stderr_output(cmd)
    msg = 'gpg: Total number processed: 1'
    output_bits = polite_string(output).split('\n')
    return len([line for line in output_bits if line == msg]) == 1

Response: Imports a GPG key into a single file.


Instruction:
Can you write a function in Python 3 where it
exports a GPG key and returns it

Response:
def export_gpg_key(key):
    """Exports a GPG key and returns it"""
    cmd = flatten([gnupg_bin(), gnupg_verbose(), gnupg_home(),
                   "--export", key])
    handle, gpg_stderr = stderr_handle()
    try:
        gpg_proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,  # nosec
                                    stderr=gpg_stderr)
        output, _err = gpg_proc.communicate()
        if handle:
            handle.close()

        return portable_b64encode(output)
    except subprocess.CalledProcessError as exception:
        LOGGER.debug("GPG Command %s", ' '.join(exception.cmd))
        LOGGER.debug("GPG Output %s", exception.output)
        raise CryptoritoError('GPG encryption error')


Instruction:
Can you implement a function in Python 3 that
encrypts a file using the given keys

Response:
def encrypt(source, dest, keys):
    """Encrypts a file using the given keys"""
    cmd = flatten([gnupg_bin(), "--armor", "--output", dest, gnupg_verbose(),
                   gnupg_home(), recipients_args(keys),
                   "--encrypt", source])

    stderr_output(cmd)
    return True


Instruction:
Create a Python 3 function for
attempting to encrypt a variable

Response:
def encrypt_var(source, keys):
    """Attempts to encrypt a variable"""
    cmd = flatten([gnupg_bin(), "--armor", "--encrypt", gnupg_verbose(),
                   recipients_args(keys)])
    output = stderr_with_input(cmd, source)
    return output


Instruction:
Write a Python 3 script for
handling the output of subprocess errors

Response:
def gpg_error(exception, message):
    """Handles the output of subprocess errors
    in a way that is compatible with the log level"""
    LOGGER.debug("GPG Command %s", ' '.join([str(x) for x in exception.cmd]))
    LOGGER.debug("GPG Output %s", exception.output)
    raise CryptoritoError(message)


Instruction:
How would you code a function in Python 3 to
attempt to decrypt a variable

Response:
def decrypt_var(source, passphrase=None):
    """Attempts to decrypt a variable"""
    cmd = [gnupg_bin(), "--decrypt", gnupg_home(), gnupg_verbose(),
           passphrase_file(passphrase)]

    return stderr_with_input(flatten(cmd), source)


Instruction:
Can you generate the documentation for the following Python 3 function
def decrypt(source, dest=None, passphrase=None):
    if not os.path.exists(source):
        raise CryptoritoError("Encrypted file %s not found" % source)

    cmd = [gnupg_bin(), gnupg_verbose(), "--decrypt", gnupg_home(),
           passphrase_file(passphrase)]

    if dest:
        cmd.append(["--output", dest])

    cmd.append([source])
    stderr_output(flatten(cmd))
    return True

Response: Attempts to decrypt a file


Instruction:
Can you generate a brief explanation for the following Python 3 code
def is_base64(string):
    return (not re.match('^[0-9]+$', string)) and \
        (len(string) % 4 == 0) and \
        re.match('^[A-Za-z0-9+/]+[=]{0,2}$', string)

Response:
Determines whether or not a string is likely to
    be base64 encoded binary nonsense


Instruction:
Make a summary of the following Python 3 code
def portable_b64encode(thing):
    if is_py3():
        try:
            some_bits = bytes(thing, 'utf-8')
        except TypeError:
            some_bits = thing

        return polite_string(b64encode(some_bits).decode('utf-8'))

    return polite_string(b64encode(thing))

Response: Wrap b64encode for Python 2 & 3


Instruction:
Can you generate the documentation for the following Python 3 function
def download_if_not_exists(url: str, filename: str,
                           skip_cert_verify: bool = True,
                           mkdir: bool = True) -> None:
    """
    Downloads a URL to a file, unless the file already exists.
    """
    if os.path.isfile(filename):
        log.info("No need to download, already have: {}", filename)
        return
    if mkdir:
        directory, basename = os.path.split(os.path.abspath(filename))
        mkdir_p(directory)
    download(url=url,
             filename=filename,
             skip_cert_verify=skip_cert_verify)

Response: Downloads a URL to a file unless the file already exists.


Instruction:
How would you implement a function in Python 3 that
fetches a Git repository and returns a list of the new ones.

Response:
def git_clone(prettyname: str, url: str, directory: str,
              branch: str = None,
              commit: str = None,
              clone_options: List[str] = None,
              run_func: Callable[[List[str]], Any] = None) -> bool:
    """
    Fetches a Git repository, unless we have it already.

    Args:
        prettyname: name to display to user
        url: URL
        directory: destination directory
        branch: repository branch
        commit: repository commit tag
        clone_options: additional options to pass to ``git clone``
        run_func: function to use to call an external command

    Returns:
        did we need to do anything?
    """
    run_func = run_func or subprocess.check_call
    clone_options = clone_options or []  # type: List[str]
    if os.path.isdir(directory):
        log.info("Not re-cloning {} Git repository: using existing source "
                 "in {}".format(prettyname, directory))
        return False
    log.info("Fetching {} source from {} into {}",
             prettyname, url, directory)
    require_executable(GIT)
    gitargs = [GIT, "clone"] + clone_options
    if branch:
        gitargs += ["--branch", branch]
    gitargs += [url, directory]
    run_func(gitargs)
    if commit:
        log.info("Resetting {} local Git repository to commit {}",
                 prettyname, commit)
        run_func([GIT,
                  "-C", directory,
                  "reset", "--hard", commit])
        # Using a Git repository that's not in the working directory:
        # https://stackoverflow.com/questions/1386291/git-git-dir-not-working-as-expected  # noqa
    return True


Instruction:
Can you write a function in Python 3 where it
unpacks a TAR file into a specified directory.

Response:
def untar_to_directory(tarfile: str,
                       directory: str,
                       verbose: bool = False,
                       gzipped: bool = False,
                       skip_if_dir_exists: bool = True,
                       run_func: Callable[[List[str]], Any] = None,
                       chdir_via_python: bool = True) -> None:
    """
    Unpacks a TAR file into a specified directory.

    Args:
        tarfile: filename of the ``.tar`` file
        directory: destination directory
        verbose: be verbose?
        gzipped: is the ``.tar`` also gzipped, e.g. a ``.tar.gz`` file?
        skip_if_dir_exists: don't do anything if the destrination directory
            exists?
        run_func: function to use to call an external command
        chdir_via_python: change directory via Python, not via ``tar``.
            Consider using this via Windows, because Cygwin ``tar`` v1.29 falls
            over when given a Windows path for its ``-C`` (or ``--directory``)
            option.
    """
    if skip_if_dir_exists and os.path.isdir(directory):
        log.info("Skipping extraction of {} as directory {} exists",
                 tarfile, directory)
        return
    log.info("Extracting {} -> {}", tarfile, directory)
    require_executable(TAR)
    mkdir_p(directory)
    args = [TAR, "-x"]  # -x: extract
    if verbose:
        args.append("-v")  # -v: verbose
    if gzipped:
        args.append("-z")  # -z: decompress using gzip
    if platform.system() != "Darwin":  # OS/X tar doesn't support --force-local
        args.append("--force-local")  # allows filenames with colons in (Windows!)  # noqa
    args.extend(["-f", tarfile])  # -f: filename follows
    if chdir_via_python:
        with pushd(directory):
            run_func(args)
    else:
        # chdir via tar
        args.extend(["-C", directory])  # -C: change to directory
        run_func(args)


Instruction:
Here you have a function in Python 3, explain what it does
def make_copy_paste_env(env: Dict[str, str]) -> str:
    windows = platform.system() == "Windows"
    cmd = "set" if windows else "export"
    return (
        "\n".join(
            "{cmd} {k}={v}".format(
                cmd=cmd,
                k=k,
                v=env[k] if windows else subprocess.list2cmdline([env[k]])
            ) for k in sorted(env.keys())
        )
    )

Response: Convert an environment into a set of commands that can be copied and pasted on the build platform to recreate that environment.


Instruction:
Here you have a function in Python 3, explain what it does
def run(args: List[str],
        env: Dict[str, str] = None,
        capture_stdout: bool = False,
        echo_stdout: bool = True,
        capture_stderr: bool = False,
        echo_stderr: bool = True,
        debug_show_env: bool = True,
        encoding: str = sys.getdefaultencoding(),
        allow_failure: bool = False,
        **kwargs) -> Tuple[str, str]:
    """
    Runs an external process, announcing it.

    Optionally, retrieves its ``stdout`` and/or ``stderr`` output (if not
    retrieved, the output will be visible to the user).

    Args:
        args: list of command-line arguments (the first being the executable)

        env: operating system environment to use (if ``None``, the current OS
            environment will be used)

        capture_stdout: capture the command's ``stdout``?

        echo_stdout: allow the command's ``stdout`` to go to ``sys.stdout``?

        capture_stderr: capture the command's ``stderr``?

        echo_stderr: allow the command's ``stderr`` to go to ``sys.stderr``?

        debug_show_env: be verbose and show the environment used before calling

        encoding: encoding to use to translate the command's output

        allow_failure: if ``True``, continues if the command returns a
            non-zero (failure) exit code; if ``False``, raises an error if
            that happens

        kwargs: additional arguments to :func:`teed_call`

    Returns:
        a tuple: ``(stdout, stderr)``. If the output wasn't captured, an empty
        string will take its place in this tuple.
    """
    cwd = os.getcwd()
    # log.debug("External command Python form: {}", args)
    copy_paste_cmd = subprocess.list2cmdline(args)
    csep = "=" * 79
    esep = "-" * 79
    effective_env = env or os.environ
    if debug_show_env:
        log.debug(
            "Environment for the command that follows:\n"
            "{esep}\n"
            "{env}\n"
            "{esep}".format(esep=esep, env=make_copy_paste_env(effective_env))
        )
    log.info(
        "Launching external command:\n"
        "{csep}\n"
        "WORKING DIRECTORY: {cwd}\n"
        "PYTHON ARGS: {args!r}\n"
        "COMMAND: {cmd}\n"
        "{csep}".format(csep=csep, cwd=cwd, cmd=copy_paste_cmd,
                        args=args)
    )
    try:
        with io.StringIO() as out, io.StringIO() as err:
            stdout_targets = []  # type: List[TextIO]
            stderr_targets = []  # type: List[TextIO]
            if capture_stdout:
                stdout_targets.append(out)
            if echo_stdout:
                stdout_targets.append(sys.stdout)
            if capture_stderr:
                stderr_targets.append(err)
            if echo_stderr:
                stderr_targets.append(sys.stderr)
            retcode = teed_call(args,
                                stdout_targets=stdout_targets,
                                stderr_targets=stderr_targets,
                                encoding=encoding,
                                env=env,
                                **kwargs)
            stdout = out.getvalue()
            stderr = err.getvalue()
            if retcode != 0 and not allow_failure:
                # subprocess.check_call() and check_output() raise
                # CalledProcessError if the called process returns a non-zero
                # return code.
                raise subprocess.CalledProcessError(returncode=retcode,
                                                    cmd=args,
                                                    output=stdout,
                                                    stderr=stderr)
        log.debug("\n{csep}\nFINISHED SUCCESSFULLY: {cmd}\n{csep}",
                  cmd=copy_paste_cmd, csep=csep)
        return stdout, stderr
    except FileNotFoundError:
        require_executable(args[0])  # which is missing, so we'll see some help
        raise
    except subprocess.CalledProcessError:
        log.critical(
            "Command that failed:\n"
            "[ENVIRONMENT]\n"
            "{env}\n"
            "\n"
            "[DIRECTORY] {cwd}\n"
            "[PYTHON ARGS] {args}\n"
            "[COMMAND] {cmd}".format(
                cwd=cwd,
                env=make_copy_paste_env(effective_env),
                cmd=copy_paste_cmd,
                args=args
            )
        )
        raise

Response: Runs an external process and returns its output.


Instruction:
Explain what the following Python 3 code does
def fetch(args: List[str], env: Dict[str, str] = None,
          encoding: str = sys.getdefaultencoding()) -> str:
    """
    Run a command and returns its stdout.

    Args:
        args: the command-line arguments
        env: the operating system environment to use
        encoding: the encoding to use for ``stdout``

    Returns:
        the command's ``stdout`` output

    """
    stdout, _ = run(args, env=env, capture_stdout=True,
                    echo_stdout=False, encoding=encoding)
    log.debug(stdout)
    return stdout

Response: Run a command and returns its stdout.


Instruction:
Here you have a function in Python 3, explain what it does
def dump_connection_info(engine: Engine, fileobj: TextIO = sys.stdout) -> None:
    meta = MetaData(bind=engine)
    writeline_nl(fileobj, sql_comment('Database info: {}'.format(meta)))

Response: Dumps some connection info as an SQL comment. Obscures passwords.


Instruction:
Can you write a function in Python 3 where it
sends schema-creating DDL from the metadata to the dump engine. This makes ``CREATE TABLE`` statements. Args: metadata: SQLAlchemy :class:`MetaData` dialect_name: string name of SQL dialect to generate DDL in fileobj: file-like object to send DDL to checkfirst: if ``True``, use ``CREATE TABLE IF NOT EXISTS`` or equivalent.

Response:
def dump_ddl(metadata: MetaData,
             dialect_name: str,
             fileobj: TextIO = sys.stdout,
             checkfirst: bool = True) -> None:
    """
    Sends schema-creating DDL from the metadata to the dump engine.
    This makes ``CREATE TABLE`` statements.

    Args:
        metadata: SQLAlchemy :class:`MetaData`
        dialect_name: string name of SQL dialect to generate DDL in
        fileobj: file-like object to send DDL to
        checkfirst: if ``True``, use ``CREATE TABLE IF NOT EXISTS`` or
            equivalent.
    """
    # http://docs.sqlalchemy.org/en/rel_0_8/faq.html#how-can-i-get-the-create-table-drop-table-output-as-a-string  # noqa
    # http://stackoverflow.com/questions/870925/how-to-generate-a-file-with-ddl-in-the-engines-sql-dialect-in-sqlalchemy  # noqa
    # https://github.com/plq/scripts/blob/master/pg_dump.py
    # noinspection PyUnusedLocal
    def dump(querysql, *multiparams, **params):
        compsql = querysql.compile(dialect=engine.dialect)
        writeline_nl(fileobj, "{sql};".format(sql=compsql))

    writeline_nl(fileobj,
                 sql_comment("Schema (for dialect {}):".format(dialect_name)))
    engine = create_engine('{dialect}://'.format(dialect=dialect_name),
                           strategy='mock', executor=dump)
    metadata.create_all(engine, checkfirst=checkfirst)


Instruction:
Implement a function in Python 3 to
return a new SQLAlchemy mapper for an existing table.

Response:
def quick_mapper(table: Table) -> Type[DeclarativeMeta]:
    """
    Makes a new SQLAlchemy mapper for an existing table.
    See
    http://www.tylerlesmann.com/2009/apr/27/copying-databases-across-platforms-sqlalchemy/

    Args:
        table: SQLAlchemy :class:`Table` object

    Returns:
        a :class:`DeclarativeMeta` class

    """  # noqa
    # noinspection PyPep8Naming
    Base = declarative_base()

    class GenericMapper(Base):
        __table__ = table

    # noinspection PyTypeChecker
    return GenericMapper


Instruction:
Here you have a function in Python 3, explain what it does
def get_literal_query(statement: Union[Query, Executable],
                      bind: Connectable = None) -> str:
    """
    Takes an SQLAlchemy statement and produces a literal SQL version, with
    values filled in.

    As per
    http://stackoverflow.com/questions/5631078/sqlalchemy-print-the-actual-query

    Notes:
    - for debugging purposes *only*
    - insecure; you should always separate queries from their values
    - please also note that this function is quite slow

    Args:
        statement: the SQL statement (a SQLAlchemy object) to use
        bind: if the statement is unbound, you will need to specify an object
            here that supports SQL execution

    Returns:
        a string literal version of the query.

    """  # noqa
    # log.debug("statement: {!r}", statement)
    # log.debug("statement.bind: {!r}", statement.bind)
    if isinstance(statement, Query):
        if bind is None:
            bind = statement.session.get_bind(statement._mapper_zero_or_none())
        statement = statement.statement
    elif bind is None:
        bind = statement.bind
    if bind is None:  # despite all that
        raise ValueError("Attempt to call get_literal_query with an unbound "
                         "statement and no 'bind' parameter")

    # noinspection PyUnresolvedReferences
    dialect = bind.dialect
    compiler = statement._compiler(dialect)

    class LiteralCompiler(compiler.__class__):
        # noinspection PyMethodMayBeStatic
        def visit_bindparam(self,
                            bindparam: BindParameter,
                            within_columns_clause: bool = False,
                            literal_binds: bool = False,
                            **kwargs) -> str:
            return super().render_literal_bindparam(
                bindparam,
                within_columns_clause=within_columns_clause,
                literal_binds=literal_binds,
                **kwargs
            )

        # noinspection PyUnusedLocal
        def render_literal_value(self, value: Any, type_) -> str:
            """Render the value of a bind parameter as a quoted literal.

            This is used for statement sections that do not accept bind
            paramters on the target driver/database.

            This should be implemented by subclasses using the quoting services
            of the DBAPI.
            """
            if isinstance(value, str):
                value = value.replace("'", "''")
                return "'%s'" % value
            elif value is None:
                return "NULL"
            elif isinstance(value, (float, int)):
                return repr(value)
            elif isinstance(value, decimal.Decimal):
                return str(value)
            elif (isinstance(value, datetime.datetime) or
                  isinstance(value, datetime.date) or
                  isinstance(value, datetime.time) or
                  isinstance(value, pendulum.DateTime) or
                  isinstance(value, pendulum.Date) or
                  isinstance(value, pendulum.Time)):
                # All have an isoformat() method.
                return "'{}'".format(value.isoformat())
                # return (
                #     "TO_DATE('%s','YYYY-MM-DD HH24:MI:SS')"
                #     % value.strftime("%Y-%m-%d %H:%M:%S")
                # )
            else:
                raise NotImplementedError(
                    "Don't know how to literal-quote value %r" % value)

    compiler = LiteralCompiler(dialect, statement)
    return compiler.process(statement) + ";"

Response: Returns a literal SQL version of the given SQLAlchemy statement.


Instruction:
Can you implement a function in Python 3 that
dumps the data from the database to the file - like object fileobj.

Response:
def dump_table_as_insert_sql(engine: Engine,
                             table_name: str,
                             fileobj: TextIO,
                             wheredict: Dict[str, Any] = None,
                             include_ddl: bool = False,
                             multirow: bool = False) -> None:
    """
    Reads a table from the database, and writes SQL to replicate the table's
    data to the output ``fileobj``.

    Args:
        engine: SQLAlchemy :class:`Engine`
        table_name: name of the table
        fileobj: file-like object to write to
        wheredict: optional dictionary of ``{column_name: value}`` to use as
            ``WHERE`` filters
        include_ddl: if ``True``, include the DDL to create the table as well
        multirow: write multi-row ``INSERT`` statements
    """
    # http://stackoverflow.com/questions/5631078/sqlalchemy-print-the-actual-query  # noqa
    # http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html
    # http://www.tylerlesmann.com/2009/apr/27/copying-databases-across-platforms-sqlalchemy/  # noqa
    # https://github.com/plq/scripts/blob/master/pg_dump.py
    log.info("dump_data_as_insert_sql: table_name={}", table_name)
    writelines_nl(fileobj, [
        SEP1,
        sql_comment("Data for table: {}".format(table_name)),
        SEP2,
        sql_comment("Filters: {}".format(wheredict)),
    ])
    dialect = engine.dialect
    if not dialect.supports_multivalues_insert:
        multirow = False
    if multirow:
        log.warning("dump_data_as_insert_sql: multirow parameter substitution "
                    "not working yet")
        multirow = False

    # literal_query = make_literal_query_fn(dialect)

    meta = MetaData(bind=engine)
    log.debug("... retrieving schema")
    table = Table(table_name, meta, autoload=True)
    if include_ddl:
        log.debug("... producing DDL")
        dump_ddl(table.metadata, dialect_name=engine.dialect.name,
                 fileobj=fileobj)
    # NewRecord = quick_mapper(table)
    # columns = table.columns.keys()
    log.debug("... fetching records")
    # log.debug("meta: {}", meta)  # obscures password
    # log.debug("table: {}", table)
    # log.debug("table.columns: {!r}", table.columns)
    # log.debug("multirow: {}", multirow)
    query = select(table.columns)
    if wheredict:
        for k, v in wheredict.items():
            col = table.columns.get(k)
            query = query.where(col == v)
    # log.debug("query: {}", query)
    cursor = engine.execute(query)
    if multirow:
        row_dict_list = []
        for r in cursor:
            row_dict_list.append(dict(r))
        # log.debug("row_dict_list: {}", row_dict_list)
        if row_dict_list:
            statement = table.insert().values(row_dict_list)
            # log.debug("statement: {!r}", statement)
            # insert_str = literal_query(statement)
            insert_str = get_literal_query(statement, bind=engine)
            # NOT WORKING FOR MULTIROW INSERTS. ONLY SUBSTITUTES FIRST ROW.
            writeline_nl(fileobj, insert_str)
        else:
            writeline_nl(fileobj, sql_comment("No data!"))
    else:
        found_one = False
        for r in cursor:
            found_one = True
            row_dict = dict(r)
            statement = table.insert(values=row_dict)
            # insert_str = literal_query(statement)
            insert_str = get_literal_query(statement, bind=engine)
            # log.debug("row_dict: {}", row_dict)
            # log.debug("insert_str: {}", insert_str)
            writeline_nl(fileobj, insert_str)
        if not found_one:
            writeline_nl(fileobj, sql_comment("No data!"))
    writeline_nl(fileobj, SEP2)
    log.debug("... done")


Instruction:
How would you implement a function in Python 3 that
writes SQL to replicate the entire database to the output file - like object.

Response:
def dump_database_as_insert_sql(engine: Engine,
                                fileobj: TextIO = sys.stdout,
                                include_ddl: bool = False,
                                multirow: bool = False) -> None:
    """
    Reads an entire database and writes SQL to replicate it to the output
    file-like object.

    Args:
        engine: SQLAlchemy :class:`Engine`
        fileobj: file-like object to write to
        include_ddl: if ``True``, include the DDL to create the table as well
        multirow: write multi-row ``INSERT`` statements
    """
    for tablename in get_table_names(engine):
        dump_table_as_insert_sql(
            engine=engine,
            table_name=tablename,
            fileobj=fileobj,
            include_ddl=include_ddl,
            multirow=multirow
        )


Instruction:
Can you tell what is the following Python 3 function doing
def dump_orm_object_as_insert_sql(engine: Engine,
                                  obj: object,
                                  fileobj: TextIO) -> None:
    """
    Takes a SQLAlchemy ORM object, and writes ``INSERT`` SQL to replicate it
    to the output file-like object.

    Args:
        engine: SQLAlchemy :class:`Engine`
        obj: SQLAlchemy ORM object to write
        fileobj: file-like object to write to
    """
    # literal_query = make_literal_query_fn(engine.dialect)
    insp = inspect(obj)
    # insp: an InstanceState
    # http://docs.sqlalchemy.org/en/latest/orm/internals.html#sqlalchemy.orm.state.InstanceState  # noqa
    # insp.mapper: a Mapper
    # http://docs.sqlalchemy.org/en/latest/orm/mapping_api.html#sqlalchemy.orm.mapper.Mapper  # noqa

    # Don't do this:
    #   table = insp.mapper.mapped_table
    # Do this instead. The method above gives you fancy data types like list
    # and Arrow on the Python side. We want the bog-standard datatypes drawn
    # from the database itself.
    meta = MetaData(bind=engine)
    table_name = insp.mapper.mapped_table.name
    # log.debug("table_name: {}", table_name)
    table = Table(table_name, meta, autoload=True)
    # log.debug("table: {}", table)

    # NewRecord = quick_mapper(table)
    # columns = table.columns.keys()
    query = select(table.columns)
    # log.debug("query: {}", query)
    for orm_pkcol in insp.mapper.primary_key:
        core_pkcol = table.columns.get(orm_pkcol.name)
        pkval = getattr(obj, orm_pkcol.name)
        query = query.where(core_pkcol == pkval)
    # log.debug("query: {}", query)
    cursor = engine.execute(query)
    row = cursor.fetchone()  # should only be one...
    row_dict = dict(row)
    # log.debug("obj: {}", obj)
    # log.debug("row_dict: {}", row_dict)
    statement = table.insert(values=row_dict)
    # insert_str = literal_query(statement)
    insert_str = get_literal_query(statement, bind=engine)
    writeline_nl(fileobj, insert_str)

Response: Dump an object as an INSERT SQL statement.


Instruction:
Can you tell what is the following Python 3 function doing
def bulk_insert_extras(dialect_name: str,
                       fileobj: TextIO,
                       start: bool) -> None:
    """
    Writes bulk ``INSERT`` preamble (start=True) or end (start=False).

    For MySQL, this temporarily switches off autocommit behaviour and index/FK
    checks, for speed, then re-enables them at the end and commits.

    Args:
        dialect_name: SQLAlchemy dialect name (see :class:`SqlaDialectName`)
        fileobj: file-like object to write to
        start: if ``True``, write preamble; if ``False``, write end
    """
    lines = []
    if dialect_name == SqlaDialectName.MYSQL:
        if start:
            lines = [
                "SET autocommit=0;",
                "SET unique_checks=0;",
                "SET foreign_key_checks=0;",
            ]
        else:
            lines = [
                "SET foreign_key_checks=1;",
                "SET unique_checks=1;",
                "COMMIT;",
            ]
    writelines_nl(fileobj, lines)

Response: Writes bulk INSERT preamble or end.


Instruction:
Can you implement a function in Python 3 that
dumps an object and all related objects in a tree to a file - like object.

Response:
def dump_orm_tree_as_insert_sql(engine: Engine,
                                baseobj: object,
                                fileobj: TextIO) -> None:
    """
    Sends an object, and all its relations (discovered via "relationship"
    links) as ``INSERT`` commands in SQL, to ``fileobj``.

    Args:
        engine: SQLAlchemy :class:`Engine`
        baseobj: starting SQLAlchemy ORM object
        fileobj: file-like object to write to

    Problem: foreign key constraints.

    - MySQL/InnoDB doesn't wait to the end of a transaction to check FK
      integrity (which it should):
      http://stackoverflow.com/questions/5014700/in-mysql-can-i-defer-referential-integrity-checks-until-commit  # noqa
    - PostgreSQL can.
    - Anyway, slightly ugly hacks...
      https://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html
    - Not so obvious how we can iterate through the list of ORM objects and
      guarantee correct insertion order with respect to all FKs.
    """  # noqa
    writeline_nl(
        fileobj,
        sql_comment("Data for all objects related to the first below:"))
    bulk_insert_extras(engine.dialect.name, fileobj, start=True)
    for part in walk_orm_tree(baseobj):
        dump_orm_object_as_insert_sql(engine, part, fileobj)
    bulk_insert_extras(engine.dialect.name, fileobj, start=False)


Instruction:
Create a Python 3 function for
fetching next page from scroll API.

Response:
def next(self):
        """Fetch next page from scroll API."""
        d = None
        if self._first_results:
            d = succeed(EsUtils.extract_hits(self._first_results))
            self._first_results = None
        elif self._scroll_id:
            d = self._scroll_next_results()
        else:
            raise StopIteration()
        return d


Instruction:
Can you generate the documentation for the following Python 3 function
def reformat_python_docstrings(top_dirs: List[str],
                               correct_copyright_lines: List[str],
                               show_only: bool = True,
                               rewrite: bool = False,
                               process_only_filenum: int = None) -> None:
    """
    Walk a directory, finding Python files and rewriting them.

    Args:
        top_dirs: list of directories to descend into
        correct_copyright_lines:
            list of lines (without newlines) representing the copyright
            docstring block, including the transition lines of equals
            symbols
        show_only: show results (to stdout) only; don't rewrite
        rewrite: write the changes
        process_only_filenum: only process this file number (1-based index);
            for debugging only
    """
    filenum = 0
    for top_dir in top_dirs:
        for dirpath, dirnames, filenames in walk(top_dir):
            for filename in filenames:
                fullname = join(dirpath, filename)
                extension = splitext(filename)[1]
                if extension != PYTHON_EXTENSION:
                    # log.debug("Skipping non-Python file: {}", fullname)
                    continue

                filenum += 1

                if process_only_filenum and filenum != process_only_filenum:
                    continue

                log.info("Processing file {}: {}", filenum, fullname)
                proc = PythonProcessor(
                    full_path=fullname,
                    top_dir=top_dir,
                    correct_copyright_lines=correct_copyright_lines)
                if show_only:
                    proc.show()
                elif rewrite:
                    proc.rewrite_file()

Response: Walk a directory and reformat the copyright - related Python docstrings.


Instruction:
Create a Python 3 function for
reading the source file and adds the lines to the source_lines list.

Response:
def _read_source(self) -> None:
        """
        Reads the source file.
        """
        with open(self.full_path, "rt") as f:
            for linenum, line_with_nl in enumerate(f.readlines(), start=1):
                line_without_newline = (
                    line_with_nl[:-1] if line_with_nl.endswith(NL)
                    else line_with_nl
                )
                if TAB in line_without_newline:
                    self._warn("Tab character at line {}".format(linenum))
                if CR in line_without_newline:
                    self._warn("Carriage return character at line {} "
                               "(Windows CR+LF endings?)".format(linenum))
                self.source_lines.append(line_without_newline)


Instruction:
Can you create a Python 3 function that
creates an internal representation of the destination file.

Response:
def _create_dest(self) -> None:
        """
        Creates an internal representation of the destination file.

        This is where the thinking happens
        """
        in_body = False
        in_docstring = False
        in_copyright = False
        copyright_done = False
        docstring_done = False
        swallow_blanks_and_filename_in_docstring = False
        for linenum, sl in enumerate(self.source_lines, start=1):
            dl = sl

            if dl.endswith(SPACE):
                self._debug("Line {} ends in whitespace".format(linenum))
                dl = dl.rstrip()

            if not in_body:

                if linenum == 1:
                    # Shebang
                    if not dl.startswith(SHEBANG_START):
                        self._warn("File does not start with shebang; "
                                   "first line was {!r}".format(dl))
                        self._too_risky()
                        return
                    if dl != CORRECT_SHEBANG:
                        self._debug("Rewriting shebang; was {!r}".format(dl))
                    dl = CORRECT_SHEBANG

                if (linenum == 2 and dl.startswith(HASH_SPACE) and
                        dl.endswith(PYTHON_EXTENSION)):
                    self._debug(
                        "Removing filename comment: {!r}".format(dl))
                    dl = None

                elif TRIPLE_DOUBLEQUOTE in dl:
                    if (not dl.startswith(TRIPLE_DOUBLEQUOTE) and
                            not dl.startswith(RAW_TRIPLE_DOUBLEQUOTE)):
                        self._warn(
                            "Triple-quote not at start of line, as follows")
                        self._debug_line(linenum, dl)
                        self._too_risky()
                        return
                    if in_docstring:  # docstring finishing
                        in_docstring = False
                        docstring_done = True
                        in_body = True
                        # ... and keep dl, so we write the end of the
                        # docstring, potentially with e.g. "# noqa" on the end
                    elif not docstring_done:  # docstring starting
                        in_docstring = True
                        # self._critical("adding our new docstring")
                        # Write our new docstring's start
                        tdq = ""  # stops linter moaning
                        if dl.startswith(TRIPLE_DOUBLEQUOTE):
                            tdq = TRIPLE_DOUBLEQUOTE
                        elif dl.startswith(RAW_TRIPLE_DOUBLEQUOTE):
                            tdq = RAW_TRIPLE_DOUBLEQUOTE
                        else:
                            assert "Bug!"
                        self.dest_lines.append(tdq)
                        self.dest_lines.append(self.advertised_filename)
                        self.dest_lines.append(BLANK)
                        self.dest_lines.extend(self.correct_copyright_lines)
                        self.dest_lines.append(BLANK)
                        swallow_blanks_and_filename_in_docstring = True
                        if dl == tdq:
                            dl = None  # don't write another triple-quote line
                        else:
                            dl = dl[len(tdq):]

                elif in_docstring:
                    # Reading within the source docstring

                    if dl == TRANSITION:
                        if in_copyright:  # copyright finishing
                            in_copyright = False
                            copyright_done = True
                            dl = None  # we've already replaced with our own
                        elif not copyright_done:
                            in_copyright = True
                            dl = None  # we've already replaced with our own

                    elif in_copyright:
                        dl = None  # we've already replaced with our own

                    elif dl == RST_COMMENT_LINE:
                        dl = None  # remove these

                    elif swallow_blanks_and_filename_in_docstring:
                        # self._debug_line(linenum, dl)
                        if dl == BLANK or dl == self.advertised_filename:
                            dl = None
                        elif copyright_done:
                            swallow_blanks_and_filename_in_docstring = False

                elif not dl.startswith(HASH) and not dl == BLANK:
                    in_body = True

                    if not docstring_done:
                        # The source file didn't have a docstring!
                        new_docstring_lines = [
                            BLANK,
                            TRIPLE_DOUBLEQUOTE,
                            self.advertised_filename,
                            BLANK,
                        ] + self.correct_copyright_lines + [
                            BLANK,
                            MISSING_RST_TITLE,
                            BLANK,
                            TRIPLE_DOUBLEQUOTE
                        ]
                        self._warn("File had no docstring; adding one. "
                                   "Will need manual edit to add RST title. "
                                   "Search for {!r}".format(MISSING_RST_TITLE))
                        self.dest_lines[1:1] = new_docstring_lines

            if dl is not None:
                # self._debug_line(linenum, dl, "adding ")
                self.dest_lines.append(dl)

        self.needs_rewriting = self.dest_lines != self.source_lines


Instruction:
Make a summary of the following Python 3 code
def _debug_line(linenum: int, line: str, extramsg: str = "") -> None:
        log.critical("{}Line {}: {!r}", extramsg, linenum, line)

Response: Writes a debugging report on a line.


Instruction:
Implement a function in Python 3 to
rewrite the source file.

Response:
def rewrite_file(self) -> None:
        """
        Rewrites the source file.
        """
        if not self.needs_rewriting:
            return
        self._info("Rewriting file")
        with open(self.full_path, "w") as outfile:
            self._write(outfile)


Instruction:
Create a Python 3 function to
write the converted output to a destination.

Response:
def _write(self, destination: TextIO) -> None:
        """
        Writes the converted output to a destination.
        """
        for line in self.dest_lines:
            destination.write(line + NL)


Instruction:
Can you generate the documentation for the following Python 3 function
def contains_duplicates(values: Iterable[Any]) -> bool:
    for v in Counter(values).values():
        if v > 1:
            return True
    return False

Response: Returns True if the iterable contains any duplicate values.


Instruction:
Can you write a function in Python 3 where it
returns a list of indexes of x in order of the data x.

Response:
def index_list_for_sort_order(x: List[Any], key: Callable[[Any], Any] = None,
                              reverse: bool = False) -> List[int]:
    """
    Returns a list of indexes of ``x``, IF ``x`` WERE TO BE SORTED.

    Args:
        x: data
        key: function to be applied to the data to generate a sort key; this
            function is passed as the ``key=`` parameter to :func:`sorted`;
            the default is ``itemgetter(1)``
        reverse: reverse the sort order?

    Returns:
        list of integer index values

    Example:

    .. code-block:: python

        z = ["a", "c", "b"]
        index_list_for_sort_order(z)  # [0, 2, 1]
        index_list_for_sort_order(z, reverse=True)  # [1, 2, 0]
        q = [("a", 9), ("b", 8), ("c", 7)]
        index_list_for_sort_order(q, key=itemgetter(1))

    """
    def key_with_user_func(idx_val: Tuple[int, Any]):
        return key(idx_val[1])
    if key:
        sort_key = key_with_user_func
        # see the simpler version below
    else:
        sort_key = itemgetter(1)
        # enumerate, below, will return tuples of (index, value), so
        # itemgetter(1) means sort by the value
    index_value_list = sorted(enumerate(x), key=sort_key, reverse=reverse)
    return [i for i, _ in index_value_list]


Instruction:
Can you tell what is the following Python 3 function doing
def sort_list_by_index_list(x: List[Any], indexes: List[int]) -> None:
    x[:] = [x[i] for i in indexes]

Response: Re - orders x by the list of indexes of x in place.


Instruction:
Can you generate the documentation for the following Python 3 function
def flatten_list(x: List[Any]) -> List[Any]:
    return [item for sublist in x for item in sublist]

Response: Converts a list of lists into a flat list.


Instruction:
How would you code a function in Python 3 to
return a list of all the unique elements in the input list.

Response:
def unique_list(seq: Iterable[Any]) -> List[Any]:
    """
    Returns a list of all the unique elements in the input list.

    Args:
        seq: input list

    Returns:
        list of unique elements

    As per
    http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-whilst-preserving-order

    """  # noqa
    seen = set()
    seen_add = seen.add
    return [x for x in seq if not (x in seen or seen_add(x))]


Instruction:
Given the following Python 3 function, write the documentation
def chunks(l: List[Any], n: int) -> Iterable[List[Any]]:
    for i in range(0, len(l), n):
        yield l[i:i + n]

Response: Yields n - sized chunks from l.


Instruction:
Make a summary of the following Python 3 code
def escape_newlines(s: str) -> str:
    if not s:
        return s
    s = s.replace("\\", r"\\")  # replace \ with \\
    s = s.replace("\n", r"\n")  # escape \n; note ord("\n") == 10
    s = s.replace("\r", r"\r")  # escape \r; note ord("\r") == 13
    return s

Response: Escapes newlines in a string.


Instruction:
Explain what the following Python 3 code does
def unescape_newlines(s: str) -> str:
    # See also http://stackoverflow.com/questions/4020539
    if not s:
        return s
    d = ""  # the destination string
    in_escape = False
    for i in range(len(s)):
        c = s[i]  # the character being processed
        if in_escape:
            if c == "r":
                d += "\r"
            elif c == "n":
                d += "\n"
            else:
                d += c
            in_escape = False
        else:
            if c == "\\":
                in_escape = True
            else:
                d += c
    return d

Response: Unescapes newlines in a string.


Instruction:
Can you write a function in Python 3 where it
escapes CR LF tab and backslashes.

Response:
def escape_tabs_newlines(s: str) -> str:
    """
    Escapes CR, LF, tab, and backslashes.

    Its counterpart is :func:`unescape_tabs_newlines`.
    """
    if not s:
        return s
    s = s.replace("\\", r"\\")  # replace \ with \\
    s = s.replace("\n", r"\n")  # escape \n; note ord("\n") == 10
    s = s.replace("\r", r"\r")  # escape \r; note ord("\r") == 13
    s = s.replace("\t", r"\t")  # escape \t; note ord("\t") == 9
    return s


Instruction:
Can you create a Python 3 function that
converts a list of integers or hex range strings like 0061 - 007A to a string.

Response:
def _unicode_def_src_to_str(srclist: List[Union[str, int]]) -> str:
    """
    Used to create :data:`UNICODE_CATEGORY_STRINGS`.

    Args:
        srclist: list of integers or hex range strings like ``"0061-007A"``

    Returns:
        a string with all characters described by ``srclist``: either the
        character corresponding to the integer Unicode character number, or
        all characters corresponding to the inclusive range described
    """
    charlist = []  # type: List[str]
    for src in srclist:
        if isinstance(src, int):
            charlist.append(chr(src))
        else:
            # Range like "0041-005A"
            first, last = [int(x, 16) for x in src.split("-")]
            charlist += [chr(x) for x in range(first, last + 1)]
    return "".join(charlist)


Instruction:
Explain what the following Python 3 code does
def report(*arrays, **kwargs):
    name = kwargs.pop("name",None)

    grouped = len(arrays) > 1
    if grouped:
        arr = N.concatenate(arrays)
        components = [PCAOrientation(a)
            for a in arrays]
    else:
        arr = arrays[0]
        components = []

    #r = LinearOrientation(arr)
    pca = PCAOrientation(arr)

    distances = list(distance_from_group(components,pca))

    kwargs = dict(
            levels=[1,2,3],
            alpha=[0.8,0.5,0.2],
            linewidth=2)

    #ellipse=error_ellipse(pca)

    kwargs = dict(n=500,levels=[1,2], ellipse=True)
    stereonet_data = dict(
        main=pca.error_coords(**kwargs),
        components=[i.error_coords(**kwargs)
            for i in components])

    t = env.get_template("report.html")

    return t.render(
        name=name,
        pca=pca,
        stereonet_data=stereonet_data,
        angular_errors=tuple(N.degrees(i)
            for i in pca.angular_errors()[::-1]),
        aligned=plot_aligned(pca),
        distances=distances)

Response: Outputs a standalone HTML report card for a single set of measurements.


Instruction:
Can you write a function in Python 3 where it
checks that stream record is actual tweet with coordinates

Response:
def validate_geotweet(self, record):
        """ check that stream record is actual tweet with coordinates """
        if record and  self._validate('user', record) \
                and self._validate('coordinates', record):
            return True
        return False


Instruction:
Implement a function in Python 3 to
compute the half angle subtended for an elliptical conic from the origin or an arbitrary viewpoint.

Response:
def angle_subtended(ell, **kwargs):
    """
    Compute the half angle subtended (or min and max angles)
    for an offset elliptical conic
    from the origin or an arbitrary viewpoint.

    kwargs:
        tangent  Return tangent instead of angle (default false)
        viewpoint   Defaults to origin
    """
    return_tangent = kwargs.pop('tangent',False)

    con, transform, offset = ell.projection(**kwargs)
    v = N.linalg.norm(N.array(con.major_axes()),axis=1)
    A = N.sort(v)[::-1] # Sort highest values first
    A = N.squeeze(A)
    B = N.linalg.norm(offset)
    if return_tangent: return A/B
    return N.arctan2(A,B)


Instruction:
Given the following Python 3 function, write the documentation
def from_axes(cls,axes):
        ax = list(axes)
        #ax[-1] *= -1  # Not sure what is going on here...
        arr = N.diag(ax + [-1])
        return arr.view(cls)

Response: Get elliptical conic from axis - aligned elliptical conic


Instruction:
Implement a function in Python 3 to
get elliptical conic from axis - aligned elliptical conic

Response:
def from_semiaxes(cls,axes):
        """
        Get axis-aligned elliptical conic from axis lenths
        This can be converted into a hyperbola by getting the dual conic
        """
        ax = list(1/N.array(axes)**2)
        #ax[-1] *= -1  # Not sure what is going on here...
        arr = N.diag(ax + [-1])
        return arr.view(cls)


Instruction:
Can you tell what is the following Python 3 function doing
def contains(ell, p, shell_only=False):
        v = augment(p)
        _ = ell.solve(v)
        return N.allclose(_,0) if shell_only else _ <= 0

Response: Check to see if a point is inside the conic s entry set.


Instruction:
Can you implement a function in Python 3 that
gets the major axes of the ellipsoids

Response:
def major_axes(ell):
        """
        Gets major axes of ellipsoids
        """
        _ = ell[:-1,:-1]
        U,s,V = N.linalg.svd(_)
        scalar = -(ell.sum()-_.sum())
        return N.sqrt(s*scalar)*V


Instruction:
Write a Python 3 function for
translating a conic by a vector

Response:
def translate(conic, vector):
        """
        Translates a conic by a vector
        """
        # Translation matrix
        T = N.identity(len(conic))
        T[:-1,-1] = -vector
        return conic.transform(T)


Instruction:
Can you generate the documentation for the following Python 3 function
def polar_plane(conic, vector):
        pole = augment(vector)
        return dot(conic,pole).view(Plane)

Response: Calculates the polar plane to a given vector.


Instruction:
Can you generate the documentation for the following Python 3 function
def pole(conic, plane):
        v = dot(N.linalg.inv(conic),plane)
        return v[:-1]/v[-1]

Response: Calculates the pole of a polar plane for a given conic section and a given plane.


Instruction:
Explain what the following Python 3 code does
def guess_file_name_stream_type_header(args):
    ftype = None
    fheader = None
    if isinstance(args, (tuple, list)):
        if len(args) == 2:
            fname, fstream = args
        elif len(args) == 3:
            fname, fstream, ftype = args
        else:
            fname, fstream, ftype, fheader = args
    else:
        fname, fstream = guess_filename_stream(args)
        ftype = guess_content_type(fname)

    if isinstance(fstream, (str, bytes, bytearray)):
        fdata = fstream
    else:
        fdata = fstream.read()
    return fname, fdata, ftype, fheader

Response: Guess filename stream file type and file header from args.


Instruction:
Can you create a Python 3 function that
builds the body for a multipart / form - data request.

Response:
def encode_params(self, data=None, files=None, **kwargs):
        """
        Build the body for a multipart/form-data request.
        Will successfully encode files when passed as a dict or a list of
        tuples. Order is retained if data is a list of tuples but arbitrary
        if parameters are supplied as a dict.
        The tuples may be string (filepath), 2-tuples (filename, fileobj), 3-tuples (filename, fileobj, contentype)
        or 4-tuples (filename, fileobj, contentype, custom_headers).
        """
        if isinstance(data, basestring):
            raise ValueError("Data must not be a string.")

        # optional args
        boundary = kwargs.get("boundary", None)
        output_str = kwargs.get("output_str", self.output_str)

        new_fields = []
        fields = to_key_val_list(data or {})
        files = to_key_val_list(files or {})

        for field, value in fields:
            ctype = None
            if isinstance(value, (tuple, list)) and len(value) == 2:
                val, ctype = value
            else:
                val = value

            if isinstance(val, basestring) or not hasattr(val, '__iter__'):
                val = [val]
            for v in val:
                # Don't call str() on bytestrings: in Py3 it all goes wrong.
                if not isinstance(v, bytes):
                    v = to_string(v, lang=output_str)

                field = field.decode('utf-8') if isinstance(field, bytes) else field
                v = v.encode('utf-8') if isinstance(v, str) else v

                rf = RequestField(name=field, data=v)
                rf.make_multipart(content_type=ctype)
                new_fields.append(rf)

        for (k, v) in files:
            fn, fdata, ft, fh = guess_file_name_stream_type_header(v)
            rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
            rf.make_multipart(content_type=ft)
            new_fields.append(rf)

        if boundary is None:
            boundary = self.boundary
        body, content_type = encode_multipart_formdata(new_fields, boundary=boundary)

        return body, content_type


Instruction:
Make a summary of the following Python 3 code
def encode_params(self, data=None, **kwargs):
        collection_format = kwargs.get("collection_format", self.collection_format)
        output_str = kwargs.get("output_str", self.output_str)
        sort = kwargs.get("sort", self.sort)

        if data is None:
            return "", self.content_type
        elif isinstance(data, (str, bytes)):
            return data, self.content_type
        elif hasattr(data, 'read'):
            return data, self.content_type
        elif collection_format == 'multi' and hasattr(data, '__iter__'):
            result = []
            for k, vs in to_key_val_list(data, sort=sort):
                if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):
                    vs = [vs]
                for v in vs:
                    result.append(
                        (k.encode('utf-8') if isinstance(k, str) else k,
                         v.encode('utf-8') if isinstance(v, str) else to_string(v, lang=output_str)))
            return urlencode(result, doseq=True), self.content_type
        elif collection_format == 'encoded' and hasattr(data, '__iter__'):
            return urlencode(data, doseq=False), self.content_type
        elif hasattr(data, '__iter__'):
            results = []
            for k, vs in to_key_val_dict(data).items():
                if isinstance(vs, list):
                    v = self.COLLECTION_SEPARATORS[collection_format].join(quote_plus(e) for e in vs)
                    key = k + '[]'
                else:
                    v = quote_plus(vs)
                    key = k
                results.append("%s=%s" % (key, v))

            return '&'.join(results), self.content_type
        else:
            return data, self.content_type

Response: Encode parameters in a piece of data.


Instruction:
Can you generate the documentation for the following Python 3 function
def encode_params(self, data=None, **kwargs):
        charset = kwargs.get("charset", self.charset)
        collection_format = kwargs.get("collection_format", self.collection_format)
        output_str = kwargs.get("output_str", self.output_str)

        if data is None:
            return "", self.get_content_type(charset)
        elif isinstance(data, (str, bytes)):
            return data, self.get_content_type(charset)
        elif hasattr(data, 'read'):
            return data, self.get_content_type(charset)
        elif collection_format == 'multi' and hasattr(data, '__iter__'):
            result = []
            for k, vs in to_key_val_list(data):
                if isinstance(vs, basestring) or not hasattr(vs, '__iter__'):
                    vs = [vs]
                for v in vs:
                    result.append(b"=".join([self._encode(k, charset), self._encode(v, charset, output_str)]))
            return b'\n'.join(result), self.get_content_type(charset)
        elif collection_format == 'plain' and hasattr(data, '__iter__'):
            results = []
            for k, vs in to_key_val_dict(data).items():
                results.append(b"=".join([self._encode(k, charset), self._encode(vs, charset, output_str)]))

            return b'\n'.join(results), self.get_content_type(charset)
        elif hasattr(data, '__iter__'):
            results = []
            for k, vs in to_key_val_dict(data).items():
                if isinstance(vs, list):
                    v = self.COLLECTION_SEPARATORS[collection_format].join(e for e in vs)
                    key = k + '[]'
                else:
                    v = vs
                    key = k
                results.append(b"=".join([self._encode(key, charset), self._encode(v, charset, output_str)]))

            return b"\n".join(results), self.get_content_type(charset)
        else:
            return str(data).encode(charset) if charset else str(data), self.get_content_type(charset)

Response: Builds the body for a text / plain request.


Instruction:
Can you implement a function in Python 3 that
builds the body for a application / json request.

Response:
def encode_params(self, data=None, **kwargs):
        """
        Build the body for a application/json request.
        """
        if isinstance(data, basestring):
            raise ValueError("Data must not be a string.")
        if data is None:
            return b"", self.content_type

        fields = to_key_val_dict(data or "")
        try:
            body = json.dumps(fields)
        except:
            body = json.dumps(fields, encoding='latin-1')

        return str(body).encode(), self.content_type


Instruction:
Create a Python 3 function for
splitting command - line into pieces.

Response:
def cmdline_split(s: str, platform: Union[int, str] = 'this') -> List[str]:
    """
    As per
    https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex.

    Multi-platform variant of ``shlex.split()`` for command-line splitting.
    For use with ``subprocess``, for ``argv`` injection etc. Using fast REGEX.

    Args:
        s:
            string to split
        platform:
            - ``'this'`` = auto from current platform;
            - ``1`` = POSIX;
            - ``0`` = Windows/CMD
            - (other values reserved)
    """  # noqa
    if platform == 'this':
        platform = (sys.platform != 'win32')  # RNC: includes 64-bit Windows
    if platform == 1:  # POSIX
        re_cmd_lex = r'''"((?:\\["\\]|[^"])*)"|'([^']*)'|(\\.)|(&&?|\|\|?|\d?\>|[<])|([^\s'"\\&|<>]+)|(\s+)|(.)'''  # noqa
    elif platform == 0:  # Windows/CMD
        re_cmd_lex = r'''"((?:""|\\["\\]|[^"])*)"?()|(\\\\(?=\\*")|\\")|(&&?|\|\|?|\d?>|[<])|([^\s"&|<>]+)|(\s+)|(.)'''  # noqa
    else:
        raise AssertionError('unknown platform %r' % platform)

    args = []
    accu = None   # collects pieces of one arg
    for qs, qss, esc, pipe, word, white, fail in re.findall(re_cmd_lex, s):
        if word:
            pass   # most frequent
        elif esc:
            word = esc[1]
        elif white or pipe:
            if accu is not None:
                args.append(accu)
            if pipe:
                args.append(pipe)
            accu = None
            continue
        elif fail:
            raise ValueError("invalid or incomplete shell string")
        elif qs:
            word = qs.replace('\\"', '"').replace('\\\\', '\\')
            if platform == 0:
                word = word.replace('""', '"')
        else:
            word = qss   # may be even empty; must be last

        accu = (accu or '') + word

    if accu is not None:
        args.append(accu)

    return args


Instruction:
Create a Python 3 function to
like scoreatpercentile but can take and return array of percentiles.

Response:
def percentiles(a, pcts, axis=None):
    """Like scoreatpercentile but can take and return array of percentiles.
    Parameters
    ----------
    a : array
        data
    pcts : sequence of percentile values
        percentile or percentiles to find score at
    axis : int or None
        if not None, computes scores over this axis
    Returns
    -------
    scores: array
        array of scores at requested percentiles
        first dimension is length of object passed to ``pcts``
    """
    scores = []
    try:
        n = len(pcts)
    except TypeError:
        pcts = [pcts]
        n = 0
    for i, p in enumerate(pcts):
        if axis is None:
            score = stats.scoreatpercentile(a.ravel(), p)
        else:
            score = N.apply_along_axis(stats.scoreatpercentile, axis, a, p)
        scores.append(score)
    scores = N.asarray(scores)
    if not n:
        scores = scores.squeeze()
    return scores


Instruction:
How would you implement a function in Python 3 that
returns a percentile range from an array of values.

Response:
def ci(a, which=95, axis=None):
    """Return a percentile range from an array of values."""
    p = 50 - which / 2, 50 + which / 2
    return percentiles(a, p, axis)


Instruction:
Write a Python 3 function for
retrieving a string value from a ConfigParser instance

Response:
def get_config_string_option(parser: ConfigParser,
                             section: str,
                             option: str,
                             default: str = None) -> str:
    """
    Retrieves a string value from a parser.

    Args:
        parser: instance of :class:`ConfigParser`
        section: section name within config file
        option: option (variable) name within that section
        default: value to return if option is absent

    Returns:
        string value

    Raises:
        ValueError: if the section is absent

    """
    if not parser.has_section(section):
        raise ValueError("config missing section: " + section)
    return parser.get(section, option, fallback=default)


Instruction:
Explain what the following Python 3 code does
def read_config_string_options(obj: Any,
                               parser: ConfigParser,
                               section: str,
                               options: Iterable[str],
                               default: str = None) -> None:
    """
    Reads config options and writes them as attributes of ``obj``, with
    attribute names as per ``options``.

    Args:
        obj: the object to modify
        parser: instance of :class:`ConfigParser`
        section: section name within config file
        options: option (variable) names within that section
        default: value to use for any missing options

    Returns:

    """
    # enforce_str removed; ConfigParser always returns strings unless asked
    # specifically
    for o in options:
        setattr(obj, o, get_config_string_option(parser, section, o,
                                                 default=default))

Response: Reads config options and writes them as attributes of obj with the names of the options as per options.


Instruction:
Can you create a Python 3 function that
returns a list of strings that are multi - line option values from a parser.

Response:
def get_config_multiline_option(parser: ConfigParser,
                                section: str,
                                option: str,
                                default: List[str] = None) -> List[str]:
    """
    Retrieves a multi-line string value from a parser as a list of strings
    (one per line, ignoring blank lines).

    Args:
        parser: instance of :class:`ConfigParser`
        section: section name within config file
        option: option (variable) name within that section
        default: value to return if option is absent (``None`` is mapped to
            ``[]``)

    Returns:
        list of strings

    Raises:
        ValueError: if the section is absent

    """
    default = default or []
    if not parser.has_section(section):
        raise ValueError("config missing section: " + section)
    try:
        multiline = parser.get(section, option)
        values = [x.strip() for x in multiline.splitlines() if x.strip()]
        return values
    except NoOptionError:
        return default


Instruction:
Create a Python 3 function to
read options from a multiline section.

Response:
def read_config_multiline_options(obj: Any,
                                  parser: ConfigParser,
                                  section: str,
                                  options: Iterable[str]) -> None:
    """
    This is to :func:`read_config_string_options` as
    :func:`get_config_multiline_option` is to :func:`get_config_string_option`.
    """
    for o in options:
        setattr(obj, o, get_config_multiline_option(parser, section, o))


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_config_bool_option(parser: ConfigParser,
                           section: str,
                           option: str,
                           default: bool = None) -> bool:
    """
    Retrieves a boolean value from a parser.

    Args:
        parser: instance of :class:`ConfigParser`
        section: section name within config file
        option: option (variable) name within that section
        default: value to return if option is absent

    Returns:
        string value

    Raises:
        ValueError: if the section is absent

    """
    if not parser.has_section(section):
        raise ValueError("config missing section: " + section)
    return parser.getboolean(section, option, fallback=default)

Response: Retrieves a boolean value from a ConfigParser instance


Instruction:
How would you explain what the following Python 3 function does
def get_config_parameter(config: ConfigParser,
                         section: str,
                         param: str,
                         fn: Callable[[Any], Any],
                         default: Any) -> Any:
    """
    Fetch parameter from ``configparser`` ``.INI`` file.

    Args:
        config: :class:`ConfigParser` object
        section: section name within config file
        param: name of parameter within section
        fn: function to apply to string parameter (e.g. ``int``)
        default: default value

    Returns:
        parameter value, or ``None`` if ``default is None``, or ``fn(default)``
    """
    try:
        value = fn(config.get(section, param))
    except (TypeError, ValueError, NoOptionError):
        log.warning(
            "Configuration variable {} not found or improper in section [{}]; "
            "using default of {!r}", param, section, default)
        if default is None:
            value = default
        else:
            value = fn(default)
    return value

Response: Get a parameter from configparser. INI file.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_config_parameter_boolean(config: ConfigParser,
                                 section: str,
                                 param: str,
                                 default: bool) -> bool:
    """
    Get Boolean parameter from ``configparser`` ``.INI`` file.

    Args:
        config: :class:`ConfigParser` object
        section: section name within config file
        param: name of parameter within section
        default: default value
    Returns:
        parameter value, or default
    """
    try:
        value = config.getboolean(section, param)
    except (TypeError, ValueError, NoOptionError):
        log.warning(
            "Configuration variable {} not found or improper in section [{}]; "
            "using default of {!r}", param, section, default)
        value = default
    return value

Response: Get Boolean parameter from ConfigParser object.


Instruction:
Can you create a Python 3 function that
gets the loglevel parameter from configparser. INI file e. g. logging. DEBUG to logging. INFO or logging. WARNING.

Response:
def get_config_parameter_loglevel(config: ConfigParser,
                                  section: str,
                                  param: str,
                                  default: int) -> int:
    """
    Get ``loglevel`` parameter from ``configparser`` ``.INI`` file, e.g.
    mapping ``'debug'`` to ``logging.DEBUG``.

    Args:
        config: :class:`ConfigParser` object
        section: section name within config file
        param: name of parameter within section
        default: default value
    Returns:
        parameter value, or default
    """
    try:
        value = config.get(section, param).lower()
        if value == "debug":
            return logging.DEBUG  # 10
        elif value == "info":
            return logging.INFO
        elif value in ["warn", "warning"]:
            return logging.WARN
        elif value == "error":
            return logging.ERROR
        elif value in ["critical", "fatal"]:
            return logging.CRITICAL  # 50
        else:
            raise ValueError
    except (TypeError, ValueError, NoOptionError, AttributeError):
        log.warning(
            "Configuration variable {} not found or improper in section [{}]; "
            "using default of {!r}", param, section, default)
        return default


Instruction:
Write a Python 3 script for
getting a multi - line string parameter from configparser. INI file.

Response:
def get_config_parameter_multiline(config: ConfigParser,
                                   section: str,
                                   param: str,
                                   default: List[str]) -> List[str]:
    """
    Get multi-line string parameter from ``configparser`` ``.INI`` file,
    as a list of strings (one per line, ignoring blank lines).

    Args:
        config: :class:`ConfigParser` object
        section: section name within config file
        param: name of parameter within section
        default: default value
    Returns:
        parameter value, or default
    """
    try:
        multiline = config.get(section, param)
        lines = [x.strip() for x in multiline.splitlines()]
        return [line for line in lines if line]
    except (TypeError, ValueError, NoOptionError):
        log.warning(
            "Configuration variable {} not found or improper in section [{}]; "
            "using default of {!r}", param, section, default)
        return default


Instruction:
Here you have a function in Python 3, explain what it does
def is_definition(cursor):
    defn = cursor.get_definition()
    return (defn is not None) and (cursor.location == defn.location)

Response: Test if a cursor refers to a definition


Instruction:
Write a Python 3 script for
getting a cone of asymptotes for hyperbola

Response:
def asymptotes(hyp, n=1000):
    """
    Gets a cone of asymptotes for hyperbola
    """
    assert N.linalg.norm(hyp.center()) == 0

    u = N.linspace(0,2*N.pi,n)
    _ = N.ones(len(u))
    angles = N.array([N.cos(u),N.sin(u),_]).T
    return dot(angles,hyp[:-1,:-1])


Instruction:
Given the following Python 3 function, write the documentation
def generic_service_main(cls: Type[WindowsService], name: str) -> None:
    argc = len(sys.argv)
    if argc == 1:
        try:
            print("Trying to start service directly...")
            evtsrc_dll = os.path.abspath(servicemanager.__file__)
            # noinspection PyUnresolvedReferences
            servicemanager.PrepareToHostSingle(cls)  # <-- sets up the service
            # noinspection PyUnresolvedReferences
            servicemanager.Initialize(name, evtsrc_dll)
            # noinspection PyUnresolvedReferences
            servicemanager.StartServiceCtrlDispatcher()
        except win32service.error as details:
            print("Failed: {}".format(details))
            # print(repr(details.__dict__))
            errnum = details.winerror
            if errnum == winerror.ERROR_FAILED_SERVICE_CONTROLLER_CONNECT:
                win32serviceutil.usage()
    elif argc == 2 and sys.argv[1] == 'debug':
        s = cls()
        s.run_debug()
    else:
        win32serviceutil.HandleCommandLine(cls)

Response: Main function for the generic service.


Instruction:
Implement a Python 3 function for
returning the full name of the process.

Response:
def fullname(self) -> str:
        """
        Description of the process.
        """
        fullname = "Process {}/{} ({})".format(self.procnum, self.nprocs,
                                               self.details.name)
        if self.running:
            fullname += " (PID={})".format(self.process.pid)
        return fullname


Instruction:
Here you have a function in Python 3, explain what it does
def debug(self, msg: str) -> None:
        if self.debugging:
            s = "{}: {}".format(self.fullname, msg)
            log.debug(s)

Response: Write a debug message to the Python disk log.


Instruction:
Given the following Python 3 function, write the documentation
def info(self, msg: str) -> None:
        # noinspection PyUnresolvedReferences
        s = "{}: {}".format(self.fullname, msg)
        servicemanager.LogInfoMsg(s)
        if self.debugging:
            log.info(s)

Response: Write an info message to the Windows Application log.


Instruction:
Explain what the following Python 3 code does
def warning(self, msg: str) -> None:
        # Log messages go to the Windows APPLICATION log.
        # noinspection PyUnresolvedReferences
        s = "{}: {}".format(self.fullname, msg)
        servicemanager.LogWarningMsg(s)
        if self.debugging:
            log.warning(s)

Response: Write a warning message to the Windows Application log.


Instruction:
Can you generate the documentation for the following Python 3 function
def error(self, msg: str) -> None:
        # noinspection PyUnresolvedReferences
        s = "{}: {}".format(self.fullname, msg)
        servicemanager.LogErrorMsg(s)
        if self.debugging:
            log.warning(s)

Response: Write an error message to the Windows Application log.


Instruction:
Write a Python 3 function that can
open Python disk logs.

Response:
def open_logs(self) -> None:
        """
        Open Python disk logs.
        """
        if self.details.logfile_out:
            self.stdout = open(self.details.logfile_out, 'a')
        else:
            self.stdout = None
        if self.details.logfile_err:
            if self.details.logfile_err == self.details.logfile_out:
                self.stderr = subprocess.STDOUT
            else:
                self.stderr = open(self.details.logfile_err, 'a')
        else:
            self.stderr = None


Instruction:
Implement a Python 3 function for
closing Python disk logs.

Response:
def close_logs(self) -> None:
        """
        Close Python disk logs.
        """
        if self.stdout is not None:
            self.stdout.close()
            self.stdout = None
        if self.stderr is not None and self.stderr != subprocess.STDOUT:
            self.stderr.close()
            self.stderr = None


Instruction:
Explain what the following Python 3 code does
def start(self) -> None:
        if self.running:
            return
        self.info("Starting: {} (with logs stdout={}, stderr={})".format(
            self.details.procargs,
            self.details.logfile_out,
            self.details.logfile_err))
        self.open_logs()
        creationflags = CREATE_NEW_PROCESS_GROUP if WINDOWS else 0
        # self.warning("creationflags: {}".format(creationflags))
        self.process = subprocess.Popen(self.details.procargs, stdin=None,
                                        stdout=self.stdout, stderr=self.stderr,
                                        creationflags=creationflags)
        self.running = True

Response: Starts a subprocess. Optionally routes its output to our disk logs.


Instruction:
Make a summary of the following Python 3 code
def stop(self) -> None:
        if not self.running:
            return
        try:
            self.wait(timeout_s=0)
            # If we get here: stopped already
        except subprocess.TimeoutExpired:  # still running
            for kill_level in self.ALL_KILL_LEVELS:
                tried_to_kill = self._terminate(level=kill_level)  # please stop
                if tried_to_kill:
                    try:
                        self.wait(timeout_s=self.kill_timeout_sec)
                        break
                    except subprocess.TimeoutExpired:  # failed to close
                        self.warning("Subprocess didn't stop when asked")
                        pass  # carry on escalating
        self.close_logs()
        self.running = False

Response: Stops a subprocess. Waits. Asks less nicely. Waits.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _terminate(self, level: int) -> bool:
        if not self.running:
            return True

        # Already closed by itself?
        try:
            self.wait(0)
            return True
        except subprocess.TimeoutExpired:  # failed to close
            pass

        # SEE NOTES ABOVE. This is tricky under Windows.

        suffix = " [to child process {}]".format(self.process.pid)
        if level == self.KILL_LEVEL_CTRL_C_OR_SOFT_KILL:
            if WINDOWS:
                success = 0 != ctypes.windll.kernel32.GenerateConsoleCtrlEvent(
                    CTRL_C_EVENT, self.process.pid)
                if success:
                    self.info("Sent CTRL-C to request stop" + suffix)
                    # ... but that doesn't mean it'll stop...
                else:
                    self.info("Failed to send CTRL-C" + suffix)
                return success
            else:
                self.warning("Asking process to stop (SIGTERM)" + suffix)
                self.process.terminate()  # soft kill under POSIX
                return True

        elif level == self.KILL_LEVEL_CTRL_BREAK:
            if not WINDOWS:
                return False
            success = 0 != ctypes.windll.kernel32.GenerateConsoleCtrlEvent(
                CTRL_BREAK_EVENT, self.process.pid)
            if success:
                self.info("Sent CTRL-BREAK to request stop" + suffix)
            else:
                self.info("Failed to send CTRL-BREAK" + suffix)
            return success

        elif level == self.KILL_LEVEL_TASKKILL:
            if not WINDOWS:
                return False
            retcode = self._taskkill(force=False)  # does its own info messages
            return retcode == winerror.ERROR_SUCCESS

        elif level == self.KILL_LEVEL_TASKKILL_FORCE:
            if not WINDOWS:
                return False
            retcode = self._taskkill(force=True)  # does its own info messages
            return retcode == winerror.ERROR_SUCCESS

        elif level == self.KILL_LEVEL_HARD_KILL:
            # Last resort
            self._kill()  # may do TASKKILL /F or some other method
            return True

        else:
            raise ValueError("Bad kill level requested")

Response: Terminate the current process.


Instruction:
Implement a function in Python 3 to
execute a Windows ``TASKKILL /pid PROCESS_ID /t`` command (``/t`` for "tree kill" = "kill all children"). Args: force: also add ``/f`` (forcefully) Returns: return code from ``TASKKILL`` **Test code:** Firstly we need a program that won't let itself be killed. Save this as ``nokill.py``: .. code-block:: python #!/usr/bin/env python import logging import time import os from cardinal_pythonlib.logs import main_only_quicksetup_rootlogger from cardinal_pythonlib.signalfunc import trap_ctrl_c_ctrl_break main_only_quicksetup_rootlogger(level=logging.DEBUG) trap_ctrl_c_ctrl_break() while True: print("Process ID is {}; time is {} s".format(os.getpid(), time.clock())) time.sleep(1) Now run that with ``python nokill.py``. It should resist CTRL-C and CTRL-BREAK. Start another command prompt in which to play with ``TASKKILL``. .. code-block:: bat REM Firstly, avoid this single-ampersand syntax: REM taskkill /im notepad.exe & echo %errorlevel% REM ... as it prints the WRONG (previous?) errorlevel. notepad.exe taskkill /im notepad.exe echo %errorlevel% REM ... 0 for success (Windows 10), e.g. REM 'SUCCESS: Sent termination signal to the process "notepad.exe" with PID 6988.' taskkill /im notepad.exe echo %errorlevel% REM ... 128 for "not found" (Windows 10), e.g. REM 'ERROR: The process "notepad.exe" not found.' REM Now run notepad.exe as Administrator taskkill /im notepad.exe & echo %errorlevel% REM ... 1 for "access denied" (Windows 10) REM Now kill the nokill.py process by its PID (e.g. 11892 here): taskkill /pid 11892 echo %errorlevel% REM ... 1 for "not allowed" (Windows 10), e.g. REM 'ERROR: The process with PID 11892 could not be terminated.' REM 'Reason: This process can only be terminated forcefully (with /F option).' REM Now forcefully: taskkill /pid 11892 /f echo %errorlevel% REM ... 0 for success (Windows 10), e.g. REM 'SUCCESS: The process with PID 11892 has been terminated.'

Response:
def _taskkill(self, force: bool = False) -> int:
        """
        Executes a Windows ``TASKKILL /pid PROCESS_ID /t`` command
        (``/t`` for "tree kill" = "kill all children").

        Args:
            force: also add ``/f`` (forcefully)

        Returns:
            return code from ``TASKKILL``

        **Test code:**

        Firstly we need a program that won't let itself be killed. Save this as
        ``nokill.py``:

        .. code-block:: python

            #!/usr/bin/env python

            import logging
            import time
            import os
            from cardinal_pythonlib.logs import main_only_quicksetup_rootlogger
            from cardinal_pythonlib.signalfunc import trap_ctrl_c_ctrl_break

            main_only_quicksetup_rootlogger(level=logging.DEBUG)
            trap_ctrl_c_ctrl_break()

            while True:
                print("Process ID is {}; time is {} s".format(os.getpid(), time.clock()))
                time.sleep(1)

        Now run that with ``python nokill.py``. It should resist CTRL-C and
        CTRL-BREAK. Start another command prompt in which to play with
        ``TASKKILL``.

        .. code-block:: bat

            REM Firstly, avoid this single-ampersand syntax:
            REM     taskkill /im notepad.exe & echo %errorlevel%
            REM ... as it prints the WRONG (previous?) errorlevel.

            notepad.exe
            taskkill /im notepad.exe
            echo %errorlevel%
            REM ... 0 for success (Windows 10), e.g.
            REM 'SUCCESS: Sent termination signal to the process "notepad.exe" with PID 6988.'

            taskkill /im notepad.exe
            echo %errorlevel%
            REM ... 128 for "not found" (Windows 10), e.g.
            REM 'ERROR: The process "notepad.exe" not found.'

            REM Now run notepad.exe as Administrator
            taskkill /im notepad.exe & echo %errorlevel%
            REM ... 1 for "access denied" (Windows 10)

            REM Now kill the nokill.py process by its PID (e.g. 11892 here):
            taskkill /pid 11892
            echo %errorlevel%
            REM ... 1 for "not allowed" (Windows 10), e.g.
            REM 'ERROR: The process with PID 11892 could not be terminated.'
            REM 'Reason: This process can only be terminated forcefully (with /F option).'

            REM Now forcefully:
            taskkill /pid 11892 /f
            echo %errorlevel%
            REM ... 0 for success (Windows 10), e.g.
            REM 'SUCCESS: The process with PID 11892 has been terminated.'

        """  # noqa
        args = [
            "taskkill",  # built in to Windows XP and higher
            "/pid", str(self.process.pid),
            "/t",  # tree kill: kill all children
        ]
        if force:
            args.append("/f")  # forcefully
        callname = " ".join(args)
        retcode = subprocess.call(args)
        # http://stackoverflow.com/questions/18682681/what-are-exit-codes-from-the-taskkill-utility  # noqa
        if retcode == winerror.ERROR_SUCCESS:  # 0
            self.info("Killed with " + repr(callname))
        elif retcode == winerror.ERROR_INVALID_FUNCTION:  # 1
            self.warning(
                repr(callname) +
                " failed (error code 1 = ERROR_INVALID_FUNCTION; "
                "can mean 'Access denied', or 'This process can only be "
                "terminated forcefully (with /F option)').")
        elif retcode == winerror.ERROR_WAIT_NO_CHILDREN:  # 128
            self.warning(
                repr(callname) +
                " failed (error code 128 = ERROR_WAIT_NO_CHILDREN "
                "= 'There are no child processes to wait for', but also "
                "occurs when the process doesn't exist, and when processes "
                "require a forceful [/F] termination)")
        elif retcode == winerror.ERROR_EA_LIST_INCONSISTENT:  # 255
            self.warning(
                repr(callname) +
                " failed (error code 255 = ERROR_EA_LIST_INCONSISTENT "
                "= 'The extended attributes are inconsistent.')")
        else:
            self.warning(callname + " failed: error code {}".format(retcode))
        return retcode


Instruction:
Can you write a function in Python 3 where it
waits for the child process to finish.

Response:
def wait(self, timeout_s: float = None) -> int:
        """
        Wait for up to ``timeout_s`` for the child process to finish.

        Args:
            timeout_s: maximum time to wait or ``None`` to wait forever

        Returns:
            process return code; or ``0`` if it wasn't running, or ``1`` if
            it managed to exit without a return code

        Raises:
            subprocess.TimeoutExpired: if the process continues to run

        """
        if not self.running:
            return 0
        retcode = self.process.wait(timeout=timeout_s)
        # We won't get further unless the process has stopped.
        if retcode is None:
            self.error("Subprocess finished, but return code was None")
            retcode = 1  # we're promising to return an int
        elif retcode == 0:
            self.info("Subprocess finished cleanly (return code 0).")
        else:
            self.error(
                "Subprocess finished, but FAILED (return code {}). "
                "Logs were: {} (stdout), {} (stderr)".format(
                    retcode,
                    self.details.logfile_out,
                    self.details.logfile_err))
        self.running = False
        return retcode


Instruction:
Can you generate the documentation for the following Python 3 function
def info(self, msg: str) -> None:
        # noinspection PyUnresolvedReferences
        servicemanager.LogInfoMsg(str(msg))
        if self.debugging:
            log.info(msg)

Response: Write an info message to the Windows Application log.


Instruction:
Can you tell what is the following Python 3 function doing
def error(self, msg: str) -> None:
        # noinspection PyUnresolvedReferences
        servicemanager.LogErrorMsg(str(msg))
        if self.debugging:
            log.error(msg)

Response: Write an error message to the Windows Application log.


Instruction:
Create a Python 3 function for
calling when the service is being shut down.

Response:
def SvcStop(self) -> None:
        """
        Called when the service is being shut down.
        """
        # tell the SCM we're shutting down
        # noinspection PyUnresolvedReferences
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        # fire the stop event
        win32event.SetEvent(self.h_stop_event)