Instruction:
Given the following Python 3 function, write the documentation
def create_or_replace_primary_key(self,
                                      table: str,
                                      fieldnames: Sequence[str]) -> int:
        """Make a primary key, or replace it if it exists."""
        # *** create_or_replace_primary_key: Uses code specific to MySQL
        sql = """
            SELECT COUNT(*)
            FROM information_schema.table_constraints
            WHERE table_name=?
            AND table_schema={}
            AND constraint_name='PRIMARY'
        """.format(self.get_current_schema_expr())
        # http://forums.mysql.com/read.php?10,114742,114748#msg-114748
        row = self.fetchone(sql, table)
        has_pk_already = True if row[0] >= 1 else False
        drop_pk_if_exists = " DROP PRIMARY KEY," if has_pk_already else ""
        fieldlist = ",".join([self.delimit(f) for f in fieldnames])
        sql = ("ALTER TABLE " + self.delimit(table) +
               drop_pk_if_exists +
               " ADD PRIMARY KEY(" + fieldlist + ")")
        # http://stackoverflow.com/questions/8859353
        return self.db_exec(sql)

Response: Create a primary key or replace it if it exists.


Instruction:
Write a Python 3 function for
returning the noise covariance of the data.

Response:
def noise_covariance(fit, dof=2, **kw):
    """
    Covariance taking into account the 'noise covariance' of the data.
    This is technically more realistic for continuously sampled data.
    From Faber, 1993
    """
    ev = fit.eigenvalues

    measurement_noise = ev[-1]/(fit.n-dof)
    return 4*ev*measurement_noise


Instruction:
Given the following Python 3 function, write the documentation
def angular_errors(hyp_axes):
    # Not quite sure why this is sqrt but it is empirically correct
    ax = N.sqrt(hyp_axes)
    return tuple(N.arctan2(ax[-1],ax[:-1]))

Response: Returns a tuple of minimum and maximum angular errors corresponding to 1st and 2nd axes of PCA distribution.


Instruction:
Explain what the following Python 3 code does
def statistical_axes(fit, **kw):
    method = kw.pop('method', 'noise')
    confidence_level = kw.pop('confidence_level', 0.95)
    dof = kw.pop('dof',2)

    nominal = fit.eigenvalues

    if method == 'sampling':
        cov = sampling_covariance(fit,**kw)
    elif method == 'noise':
        cov = noise_covariance(fit,**kw)

    if kw.pop('chisq', False):
        # Model the incorrect behaviour of using the
        # Chi2 distribution instead of the Fisher
        # distribution (which is a measure of the
        # ratio between the two).
        z = chi2.ppf(confidence_level,dof)
    else:
        z = fisher_statistic(fit.n,confidence_level,dof=dof)

    # Apply two fisher F parameters (one along each axis)
    # Since we apply to each axis without division,
    # it is as if we are applying N.sqrt(2*F) to the entire
    # distribution, aligning us with (Francq, 2014)
    err = z*N.sqrt(cov)
    return apply_error_scaling(nominal, err, n=fit.n, **kw)

Response: Returns a Hyperbolic error using a statistical process.


Instruction:
Here you have a function in Python 3, explain what it does
def create_view(operations, operation):
    operations.execute("CREATE VIEW %s AS %s" % (
        operation.target.name,
        operation.target.sqltext
    ))

Response:
Implements ``CREATE VIEW``.

    Args:
        operations: instance of ``alembic.operations.base.Operations``
        operation: instance of :class:`.ReversibleOp`

    Returns:
        ``None``


Instruction:
Create a Python 3 function to
implement ``CREATE FUNCTION``. Args: operations: instance of ``alembic.operations.base.Operations`` operation: instance of :class:`.ReversibleOp` Returns: ``None``

Response:
def create_sp(operations, operation):
    """
    Implements ``CREATE FUNCTION``.

    Args:
        operations: instance of ``alembic.operations.base.Operations``
        operation: instance of :class:`.ReversibleOp`

    Returns:
        ``None``
    """
    operations.execute(
        "CREATE FUNCTION %s %s" % (
            operation.target.name, operation.target.sqltext
        )
    )


Instruction:
Implement a function in Python 3 to
return a Python object from an Alembic migration module.

Response:
def _get_object_from_version(cls, operations, ident):
        """
        Returns a Python object from an Alembic migration module (script).

        Args:
            operations: instance of ``alembic.operations.base.Operations``
            ident: string of the format ``version.objname``

        Returns:
            the object whose name is ``objname`` within the Alembic migration
            script identified by ``version``
        """
        version, objname = ident.split(".")

        module_ = operations.get_context().script.get_revision(version).module
        obj = getattr(module_, objname)
        return obj


Instruction:
Here you have a function in Python 3, explain what it does
def iso_string_to_python_datetime(
        isostring: str) -> Optional[datetime.datetime]:
    """
    Takes an ISO-8601 string and returns a ``datetime``.
    """
    if not isostring:
        return None  # if you parse() an empty string, you get today's date
    return dateutil.parser.parse(isostring)

Response: Takes an ISO - 8601 string and returns a datetime. datetime object.


Instruction:
Write a Python 3 function that can
convert a Python datetime to a string literal compatible with SQLite

Response:
def python_utc_datetime_to_sqlite_strftime_string(
        value: datetime.datetime) -> str:
    """
    Converts a Python datetime to a string literal compatible with SQLite,
    including the millisecond field.
    """
    millisec_str = str(round(value.microsecond / 1000)).zfill(3)
    return value.strftime("%Y-%m-%d %H:%M:%S") + "." + millisec_str


Instruction:
Implement a Python 3 function for
converting a Python datetime object that has a timezone to an ISO - 8601 string.

Response:
def python_localized_datetime_to_human_iso(value: datetime.datetime) -> str:
    """
    Converts a Python ``datetime`` that has a timezone to an ISO-8601 string
    with ``:`` between the hours and minutes of the timezone.

    Example:
        >>> import datetime
        >>> import pytz
        >>> x = datetime.datetime.now(pytz.utc)
        >>> python_localized_datetime_to_human_iso(x)
        '2017-08-21T20:47:18.952971+00:00'
    """
    s = value.strftime("%Y-%m-%dT%H:%M:%S.%f%z")
    return s[:29] + ":" + s[29:]


Instruction:
Can you implement a function in Python 3 that
converts a MySQL lookup into a SQL string.

Response:
def isodt_lookup_mysql(lookup, compiler, connection,
                       operator) -> Tuple[str, Any]:
    """
    For a comparison "LHS *op* RHS", transforms the LHS from a column
    containing an ISO-8601 date/time into an SQL ``DATETIME``,
    for MySQL.
    """
    lhs, lhs_params = compiler.compile(lookup.lhs)
    rhs, rhs_params = lookup.process_rhs(compiler, connection)
    params = lhs_params + rhs_params
    return '{lhs} {op} {rhs}'.format(
        lhs=iso_string_to_sql_utcdatetime_mysql(lhs),
        op=operator,
        rhs=rhs,
    ), params


Instruction:
How would you code a function in Python 3 to
take an instance and calculates the arguments to pass to the constructor.

Response:
def deconstruct(self) -> Tuple[str, str, List[Any], Dict[str, Any]]:
        """
        Takes an instance and calculates the arguments to pass to ``__init__``
        to reconstruct it.
        """
        name, path, args, kwargs = super().deconstruct()
        del kwargs['max_length']
        return name, path, args, kwargs


Instruction:
Write a Python 3 function for
converting database value to Python value.

Response:
def from_db_value(self, value, expression, connection, context):
        """
        Convert database value to Python value.
        Called when data is loaded from the database.
        """
        # log.debug("from_db_value: {}, {}", value, type(value))
        if value is None:
            return value
        if value == '':
            return None
        return iso_string_to_python_datetime(value)


Instruction:
Can you generate the documentation for the following Python 3 function
def to_python(self, value: Optional[str]) -> Optional[Any]:
        # https://docs.djangoproject.com/en/1.8/howto/custom-model-fields/
        # log.debug("to_python: {}, {}", value, type(value))
        if isinstance(value, datetime.datetime):
            return value
        if value is None:
            return value
        if value == '':
            return None
        return iso_string_to_python_datetime(value)

Response: Converts value to a Python value.


Instruction:
Write a Python 3 script to
convert Python value to database value for QUERYING.

Response:
def get_prep_value(self, value):
        """
        Convert Python value to database value for QUERYING.
        We query with UTC, so this function converts datetime values to UTC.

        Calls to this function are followed by calls to ``get_db_prep_value()``,
        which is for backend-specific conversions.
        """
        log.debug("get_prep_value: {}, {}", value, type(value))
        if not value:
            return ''
            # For underlying (database) string types, e.g. VARCHAR, this
            # function must always return a string type.
            # https://docs.djangoproject.com/en/1.8/howto/custom-model-fields/
        # Convert to UTC
        return value.astimezone(timezone.utc)


Instruction:
Can you generate the documentation for the following Python 3 function
def get_db_prep_value(self, value, connection, prepared=False):
        log.debug("get_db_prep_value: {}, {}", value, type(value))
        value = super().get_db_prep_value(value, connection, prepared)
        if value is None:
            return value
        # log.debug("connection.settings_dict['ENGINE']: {}",
        #           connection.settings_dict['ENGINE'])
        if connection.settings_dict['ENGINE'] == 'django.db.backends.sqlite3':
            return python_utc_datetime_to_sqlite_strftime_string(value)
        return value

Response: This method is used to convert the value to the database value for QUERYING.


Instruction:
Make a summary of the following Python 3 code
def get_db_prep_save(self, value, connection, prepared=False):
        log.debug("get_db_prep_save: {}, {}", value, type(value))
        if not value:
            return ''
            # For underlying (database) string types, e.g. VARCHAR, this
            # function must always return a string type.
            # https://docs.djangoproject.com/en/1.8/howto/custom-model-fields/
        return python_localized_datetime_to_human_iso(value)

Response: Convert Python value to database value for SAVING.


Instruction:
Here you have a function in Python 3, explain what it does
def list_file_extensions(path: str, reportevery: int = 1) -> List[str]:
    extensions = set()
    count = 0
    for root, dirs, files in os.walk(path):
        count += 1
        if count % reportevery == 0:
            log.debug("Walking directory {}: {!r}", count, root)
        for file in files:
            filename, ext = os.path.splitext(file)
            extensions.add(ext)
    return sorted(list(extensions))

Response: Returns a sorted list of every file extension found in a directory and its subdirectories.


Instruction:
Can you generate the documentation for the following Python 3 function
def main() -> None:
    main_only_quicksetup_rootlogger(level=logging.DEBUG)
    parser = argparse.ArgumentParser()
    parser.add_argument("directory", nargs="?", default=os.getcwd())
    parser.add_argument("--reportevery", default=10000)
    args = parser.parse_args()
    log.info("Extensions in directory {!r}:", args.directory)
    print("\n".join(repr(x) for x in
                    list_file_extensions(args.directory,
                                         reportevery=args.reportevery)))

Response: Command - line processor.


Instruction:
Make a summary of the following Python 3 code
def are_debian_packages_installed(packages: List[str]) -> Dict[str, bool]:
    assert len(packages) >= 1
    require_executable(DPKG_QUERY)
    args = [
        DPKG_QUERY,
        "-W",  # --show
        # "-f='${Package} ${Status} ${Version}\n'",
        "-f=${Package} ${Status}\n",  # --showformat
    ] + packages
    completed_process = subprocess.run(args,
                                       stdout=subprocess.PIPE,
                                       stderr=subprocess.PIPE,
                                       check=False)
    encoding = sys.getdefaultencoding()
    stdout = completed_process.stdout.decode(encoding)
    stderr = completed_process.stderr.decode(encoding)
    present = OrderedDict()
    for line in stdout.split("\n"):
        if line:  # e.g. "autoconf install ok installed"
            words = line.split()
            assert len(words) >= 2
            package = words[0]
            present[package] = "installed" in words[1:]
    for line in stderr.split("\n"):
        if line:  # e.g. "dpkg-query: no packages found matching XXX"
            words = line.split()
            assert len(words) >= 2
            package = words[-1]
            present[package] = False
    log.debug("Debian package presence: {}", present)
    return present

Response: Check which of a list of Debian packages are installed via dpkg - query.


Instruction:
How would you explain what the following Python 3 function does
def require_debian_packages(packages: List[str]) -> None:
    present = are_debian_packages_installed(packages)
    missing_packages = [k for k, v in present.items() if not v]
    if missing_packages:
        missing_packages.sort()
        msg = (
            "Debian packages are missing, as follows. Suggest:\n\n"
            "sudo apt install {}".format(" ".join(missing_packages))
        )
        log.critical(msg)
        raise ValueError(msg)

Response: Ensures that specific packages are installed under Debian.


Instruction:
Here you have a function in Python 3, explain what it does
def validate_pair(ob: Any) -> bool:
    try:
        if len(ob) != 2:
            log.warning("Unexpected result: {!r}", ob)
            raise ValueError()
    except ValueError:
        return False
    return True

Response: Validate that the object is a valid pair of objects.


Instruction:
Can you implement a function in Python 3 that
takes a command (either a single command or list of arguments) and return the environment created after running that command. Note that the command must be a batch (``.bat``) file or ``.cmd`` file, or the changes to the environment will not be captured. If ``initial_env`` is supplied, it is used as the initial environment passed to the child process. (Otherwise, this process's ``os.environ()`` will be used by default.) From https://stackoverflow.com/questions/1214496/how-to-get-environment-from-a-subprocess-in-python, with decoding bug fixed for Python 3. PURPOSE: under Windows, ``VCVARSALL.BAT`` sets up a lot of environment variables to compile for a specific target architecture. We want to be able to read them, not to replicate its work. METHOD: create a composite command that executes the specified command, then echoes an unusual string tag, then prints the environment via ``SET``; capture the output, work out what came from ``SET``. Args: env_cmd: command, or list of command arguments initial_env: optional dictionary containing initial environment Returns: dict: environment created after running the command

Response:
def windows_get_environment_from_batch_command(
        env_cmd: Union[str, List[str]],
        initial_env: Dict[str, str] = None) -> Dict[str, str]:
    """
    Take a command (either a single command or list of arguments) and return
    the environment created after running that command. Note that the command
    must be a batch (``.bat``) file or ``.cmd`` file, or the changes to the
    environment will not be captured.

    If ``initial_env`` is supplied, it is used as the initial environment
    passed to the child process. (Otherwise, this process's ``os.environ()``
    will be used by default.)

    From https://stackoverflow.com/questions/1214496/how-to-get-environment-from-a-subprocess-in-python,
    with decoding bug fixed for Python 3.

    PURPOSE: under Windows, ``VCVARSALL.BAT`` sets up a lot of environment
    variables to compile for a specific target architecture. We want to be able
    to read them, not to replicate its work.

    METHOD: create a composite command that executes the specified command,
    then echoes an unusual string tag, then prints the environment via ``SET``;
    capture the output, work out what came from ``SET``.

    Args:
        env_cmd: command, or list of command arguments
        initial_env: optional dictionary containing initial environment

    Returns:
        dict: environment created after running the command
    """  # noqa
    if not isinstance(env_cmd, (list, tuple)):
        env_cmd = [env_cmd]
    # construct the command that will alter the environment
    env_cmd = subprocess.list2cmdline(env_cmd)
    # create a tag so we can tell in the output when the proc is done
    tag = '+/!+/!+/! Finished command to set/print env +/!+/!+/!'  # RNC
    # construct a cmd.exe command to do accomplish this
    cmd = 'cmd.exe /s /c "{env_cmd} && echo "{tag}" && set"'.format(
        env_cmd=env_cmd, tag=tag)
    # launch the process
    log.info("Fetching environment using command: {}", env_cmd)
    log.debug("Full command: {}", cmd)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=initial_env)
    # parse the output sent to stdout
    encoding = sys.getdefaultencoding()

    def gen_lines() -> Generator[str, None, None]:  # RNC: fix decode problem
        for line in proc.stdout:
            yield line.decode(encoding)

    # define a way to handle each KEY=VALUE line
    def handle_line(line: str) -> Tuple[str, str]:  # RNC: as function
        # noinspection PyTypeChecker
        parts = line.rstrip().split('=', 1)
        # split("=", 1) means "split at '=' and do at most 1 split"
        if len(parts) < 2:
            return parts[0], ""
        return parts[0], parts[1]

    lines = gen_lines()  # RNC
    # consume whatever output occurs until the tag is reached
    consume(itertools.takewhile(lambda l: tag not in l, lines))
    # ... RNC: note that itertools.takewhile() generates values not matching
    #     the condition, but then consumes the condition value itself. So the
    #     tag's already gone. Example:
    #
    #   def gen():
    #       mylist = [1, 2, 3, 4, 5]
    #       for x in mylist:
    #           yield x
    #
    #   g = gen()
    #   list(itertools.takewhile(lambda x: x != 3, g))  # [1, 2]
    #   next(g)  # 4, not 3
    #
    # parse key/values into pairs
    pairs = map(handle_line, lines)
    # make sure the pairs are valid (this also eliminates the tag)
    valid_pairs = filter(validate_pair, pairs)
    # construct a dictionary of the pairs
    result = dict(valid_pairs)  # consumes generator
    # let the process finish
    proc.communicate()
    log.debug("Fetched environment:\n" + pformat(result))
    return result


Instruction:
Write a Python 3 script for
checking if target exists in x outside quotes.

Response:
def contains_unquoted_target(x: str,
                             quote: str = '"', target: str = '&') -> bool:
    """
    Checks if ``target`` exists in ``x`` outside quotes (as defined by
    ``quote``). Principal use: from
    :func:`contains_unquoted_ampersand_dangerous_to_windows`.
    """
    in_quote = False
    for c in x:
        if c == quote:
            in_quote = not in_quote
        elif c == target:
            if not in_quote:
                return True
    return False


Instruction:
Implement a Python 3 function for
returning a message that is wrapped in a message.

Response:
def gut_message(message: Message) -> Message:
    """
    Remove body from a message, and wrap in a message/external-body.
    """
    wrapper = Message()
    wrapper.add_header('Content-Type', 'message/external-body',
                       access_type='x-spam-deleted',
                       expiration=time.strftime("%a, %d %b %Y %H:%M:%S %z"),
                       size=str(len(message.get_payload())))

    message.set_payload('')
    wrapper.set_payload([message])

    return wrapper


Instruction:
Write a Python 3 function for
cleaning a message by all its binary parts and return the message itself for convenience.

Response:
def clean_message(message: Message, topmost: bool = False) -> Message:
    """
    Clean a message of all its binary parts.

    This guts all binary attachments, and returns the message itself for
    convenience.

    """
    if message.is_multipart():
        # Don't recurse in already-deleted attachments
        if message.get_content_type() != 'message/external-body':
            parts = message.get_payload()
            parts[:] = map(clean_message, parts)
    elif message_is_binary(message):
        # Don't gut if this is the topmost message
        if not topmost:
            message = gut_message(message)

    return message


Instruction:
Write a Python 3 script for
hashing a plaintext password using bcrypt.

Response:
def hash_password(plaintextpw: str,
                  log_rounds: int = BCRYPT_DEFAULT_LOG_ROUNDS) -> str:
    """
    Makes a hashed password (using a new salt) using ``bcrypt``.

    The hashed password includes the salt at its start, so no need to store a
    separate salt.
    """
    salt = bcrypt.gensalt(log_rounds)  # optional parameter governs complexity
    hashedpw = bcrypt.hashpw(plaintextpw, salt)
    return hashedpw


Instruction:
Here you have a function in Python 3, explain what it does
def is_password_valid(plaintextpw: str, storedhash: str) -> bool:
    # Upon CamCOPS from MySQL 5.5.34 (Ubuntu) to 5.1.71 (CentOS 6.5), the
    # VARCHAR was retrieved as Unicode. We needed to convert that to a str.
    # For Python 3 compatibility, we just str-convert everything, avoiding the
    # unicode keyword, which no longer exists.
    if storedhash is None:
        storedhash = ""
    storedhash = str(storedhash)
    if plaintextpw is None:
        plaintextpw = ""
    plaintextpw = str(plaintextpw)
    try:
        h = bcrypt.hashpw(plaintextpw, storedhash)
    except ValueError:  # e.g. ValueError: invalid salt
        return False
    return h == storedhash

Response: Checks if a plaintext password matches a stored hash.


Instruction:
Can you tell what is the following Python 3 function doing
def create_sys_dsn(driver: str, **kw) -> bool:
    attributes = []  # type: List[str]
    for attr in kw.keys():
        attributes.append("%s=%s" % (attr, kw[attr]))
    return bool(
        ctypes.windll.ODBCCP32.SQLConfigDataSource(0, ODBC_ADD_SYS_DSN,
                                                   driver,
                                                   nul.join(attributes))
    )

Response: Create a system ODBC data source name.


Instruction:
Make a summary of the following Python 3 code
def create_user_dsn(driver: str, **kw) -> bool:
    attributes = []  # type: List[str]
    for attr in kw.keys():
        attributes.append("%s=%s" % (attr, kw[attr]))
    return bool(
        ctypes.windll.ODBCCP32.SQLConfigDataSource(0, ODBC_ADD_DSN, driver,
                                                   nul.join(attributes))
    )

Response: Create a user ODBC data source name.


Instruction:
Write a Python 3 function that can
register a Microsoft Access database with Microsoft SQL Server.

Response:
def register_access_db(fullfilename: str, dsn: str, description: str) -> bool:
    """
    (Windows only.)
    Registers a Microsoft Access database with ODBC.

    Args:
        fullfilename: filename of the existing database
        dsn: ODBC data source name to create
        description: description of the database

    Returns:
        bool: was the DSN created?

    """
    directory = os.path.dirname(fullfilename)
    return create_sys_dsn(
        access_driver,
        SERVER="",
        DESCRIPTION=description,
        DSN=dsn,
        DBQ=fullfilename,
        DefaultDir=directory
    )


Instruction:
Can you tell what is the following Python 3 function doing
def create_and_register_access97_db(filename: str,
                                    dsn: str,
                                    description: str) -> bool:
    """
    (Windows only.)
    Creates a Microsoft Access 97 database and registers it with ODBC.

    Args:
        filename: filename of the database to create
        dsn: ODBC data source name to create
        description: description of the database

    Returns:
        bool: was the DSN created?
    """
    fullfilename = os.path.abspath(filename)
    create_string = fullfilename + " General"
    # ... filename, space, sort order ("General" for English)
    return (create_user_dsn(access_driver, CREATE_DB3=create_string) and
            register_access_db(filename, dsn, description))

Response: Creates a Microsoft Access 97 database and registers it with ODBC.


Instruction:
Can you tell what is the following Python 3 function doing
def create_and_register_access2000_db(filename: str,
                                      dsn: str,
                                      description: str) -> bool:
    """
    (Windows only.)
    Creates a Microsoft Access 2000 database and registers it with ODBC.

    Args:
        filename: filename of the database to create
        dsn: ODBC data source name to create
        description: description of the database

    Returns:
        bool: was the DSN created?
    """
    fullfilename = os.path.abspath(filename)
    create_string = fullfilename + " General"
    # ... filename, space, sort order ("General" for English)
    return (create_user_dsn(access_driver, CREATE_DB4=create_string) and
            register_access_db(filename, dsn, description))

Response: Creates a Microsoft Access 2000 database and registers it with ODBC.


Instruction:
How would you implement a function in Python 3 that
creates a Microsoft Access database and registers it with ODBC.

Response:
def create_and_register_access_db(filename: str,
                                  dsn: str,
                                  description: str) -> bool:
    """
    (Windows only.)
    Creates a Microsoft Access database and registers it with ODBC.

    Args:
        filename: filename of the database to create
        dsn: ODBC data source name to create
        description: description of the database

    Returns:
        bool: was the DSN created?
    """
    fullfilename = os.path.abspath(filename)
    create_string = fullfilename + " General"
    # ... filename, space, sort order ("General" for English)
    return (create_user_dsn(access_driver, CREATE_DB=create_string) and
            register_access_db(filename, dsn, description))


Instruction:
Explain what the following Python 3 code does
def _finalized(self):
        return \
            len(self._finalized_dependencies) >= self.dependency_count \
            and len(self._enqueued_dependencies) >= self.dependency_count

Response: Return true if all dependencies have been finalized and that they ll have sent at least one batch of data.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def version(self):
        try:
            f = self.func.__call__.__code__
        except AttributeError:
            f = self.func.__code__

        h = md5()
        h.update(f.co_code)
        h.update(str(f.co_names).encode())

        try:
            closure = self.func.__closure__
        except AttributeError:
            return h.hexdigest()

        if closure is None or self.closure_fingerprint is None:
            return h.hexdigest()

        d = dict(
            (name, cell.cell_contents)
            for name, cell in zip(f.co_freevars, closure))
        h.update(self.closure_fingerprint(d).encode())

        return h.hexdigest()

Response: Compute the version identifier for this functional node using the functional node s local names and closure fingerprint.


Instruction:
How would you code a function in Python 3 to
create a table from an SQLAlchemy ORM class.

Response:
def create_table_from_orm_class(engine: Engine,
                                ormclass: DeclarativeMeta,
                                without_constraints: bool = False) -> None:
    """
    From an SQLAlchemy ORM class, creates the database table via the specified
    engine, using a ``CREATE TABLE`` SQL (DDL) statement.

    Args:
        engine: SQLAlchemy :class:`Engine` object
        ormclass: SQLAlchemy ORM class
        without_constraints: don't add foreign key constraints
    """
    table = ormclass.__table__  # type: Table
    log.info("Creating table {} on engine {}{}",
             table.name,
             get_safe_url_from_engine(engine),
             " (omitting constraints)" if without_constraints else "")
    # https://stackoverflow.com/questions/19175311/how-to-create-only-one-table-with-sqlalchemy  # noqa
    if without_constraints:
        include_foreign_key_constraints = []
    else:
        include_foreign_key_constraints = None  # the default
    creator = CreateTable(
        table,
        include_foreign_key_constraints=include_foreign_key_constraints
    )
    creator.execute(bind=engine)


Instruction:
Make a summary of the following Python 3 code
def mapper_init(self):
        self.counties = CachedCountyLookup(precision=GEOHASH_PRECISION)
        self.extractor = WordExtractor()

Response: Initialize the counties and extractor objects


Instruction:
Can you generate the documentation for the following Python 3 function
def run(self):
        logging.info("Starting GeoJSON MongoDB loading process.")
        mongo = dict(uri=self.mongo, db=self.db, collection=self.collection)
        self.load(self.source, **mongo)
        logging.info("Finished loading {0} into MongoDB".format(self.source))

Response: Load the state and county GeoJSON files into MongoDB DB


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load(self, geojson, uri=None, db=None, collection=None):
        logging.info("Mongo URI: {0}".format(uri))
        logging.info("Mongo DB: {0}".format(db))
        logging.info("Mongo Collection: {0}".format(collection))
        logging.info("Geojson File to be loaded: {0}".format(geojson))
        mongo = MongoGeo(db=db, collection=collection, uri=uri)
        GeoJSONLoader().load(geojson, mongo.insert)

Response: Load a geojson file into a mongodb instance


Instruction:
Create a Python 3 function for
returning formatted version string.

Response:
def get_version():
    """
    Return formatted version string.

    Returns:
        str: string with project version or empty string.
    """

    if all([VERSION, UPDATED, any([isinstance(UPDATED, date), isinstance(UPDATED, datetime), ]), ]):

        return FORMAT_STRING.format(**{"version": VERSION, "updated": UPDATED, })
    elif VERSION:

        return VERSION
    elif UPDATED:

        return localize(UPDATED) if any([isinstance(UPDATED, date), isinstance(UPDATED, datetime), ]) else ""
    else:

        return ""


Instruction:
Given the following Python 3 function, write the documentation
def process_file(filename: str,
                 filetypes: List[str],
                 move_to: str,
                 delete_if_not_specified_file_type: bool,
                 show_zip_output: bool) -> None:
    """
    Deals with an OpenXML, including if it is potentially corrupted.

    Args:
        filename: filename to process
        filetypes: list of filetypes that we care about, e.g.
            ``['docx', 'pptx', 'xlsx']``.
        move_to: move matching files to this directory
        delete_if_not_specified_file_type: if ``True``, and the file is **not**
            a type specified in ``filetypes``, then delete the file.
        show_zip_output: show the output from the external ``zip`` tool?
    """
    # log.critical("process_file: start")
    try:
        reader = CorruptedOpenXmlReader(filename,
                                        show_zip_output=show_zip_output)
        if reader.file_type in filetypes:
            log.info("Found {}: {}", reader.description, filename)
            if move_to:
                dest_file = os.path.join(move_to, os.path.basename(filename))
                _, ext = os.path.splitext(dest_file)
                if ext != reader.suggested_extension():
                    dest_file += reader.suggested_extension()
                reader.move_to(destination_filename=dest_file)
        else:
            log.info("Unrecognized or unwanted contents: " + filename)
            if delete_if_not_specified_file_type:
                log.info("Deleting: " + filename)
                os.remove(filename)
    except Exception as e:
        # Must explicitly catch and report errors, since otherwise they vanish
        # into the ether.
        log.critical("Uncaught error in subprocess: {!r}\n{}", e,
                     traceback.format_exc())
        raise

Response: Processes an OpenXML file and returns a list of the ether files that can be used to create the ether tree.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def main() -> None:
    parser = ArgumentParser(
        formatter_class=RawDescriptionHelpFormatter,
        description="""
Tool to recognize and rescue Microsoft Office OpenXML files, even if they have
garbage appended to them.

- Rationale: when you have accidentally deleted files from an NTFS disk, and
  they really matter, you should (a) stop what you're doing; (b) clone the disk
  to an image file using "dd" under Linux; (c) perform all subsequent
  operations on the cloned image (in read-only mode). Those steps might
  include:
    - ntfsundelete, to find files that the filesystem is still aware of;
    - scalpel, to find files based on their contents.

- Scalpel is great at finding stuff efficiently, but it works best when files
  can be defined by both a start (header) signature and an end (footer)
  signature. However, the Microsoft Office OpenXML file format has a
  recognizable header, but no standard footer. In these circumstances, Scalpel
  reads up to a certain limit that you specify in its configuration file. (To
  retrieve large Powerpoint files, this limit needs to be substantial, e.g.
  50 Mb or more, depending on your ways of working with Powerpoint.)

- That means that files emerging from a Scalpel search for DOCX/PPTX/XLSX files
  may be
    - false positives, having nothing to do with Office;
    - clean Office files (the least likely category!);
    - Office files with garbage stuck on the end.

- The OpenXML file format is just a zip file. If you stick too much garbage on
  the end of a zip file, zip readers will see it as corrupt.

- THIS TOOL detects (and optionally moves) potentially corrupted zipfiles based
  on file contents, by unzipping the file and checking for "inner" files with
  names like:

        File type       Contents filename signature (regular expression)
        ----------------------------------------------------------------
        DOCX            {DOCX_CONTENTS_REGEX_STR}
        PPTX            {PPTX_CONTENTS_REGEX_STR}
        XLSX            {XLSX_CONTENTS_REGEX_STR}

- WARNING: it's possible for an OpenXML file to contain more than one of these.
  If so, they may be mis-classified.

- If a file is not immediately readable as a zip, it uses Linux's "zip -FF" to
  repair zip files with corrupted ends, and tries again.

- Having found valid-looking files, you can elect to move them elsewhere.

- As an additional and VERY DANGEROUS operation, you can elect to delete files
  that this tool doesn't recognize. (Why? Because a 450Gb disk might produce
  well in excess of 1.7Tb of candidate files; many will be false positives and
  even the true positives will all be expanded to your file size limit, e.g.
  50 Mb. You may have a problem with available disk space, so running this tool
  regularly allows you to clear up the junk. Use the --run_every option to help
  with this.)

        """.format(
            DOCX_CONTENTS_REGEX_STR=DOCX_CONTENTS_REGEX_STR,
            PPTX_CONTENTS_REGEX_STR=PPTX_CONTENTS_REGEX_STR,
            XLSX_CONTENTS_REGEX_STR=XLSX_CONTENTS_REGEX_STR,
        )
    )
    parser.add_argument(
        "filename", nargs="+",
        help="File(s) to check. You can also specify directores if you use "
             "--recursive"
    )
    parser.add_argument(
        "--recursive", action="store_true",
        help="Allow search to descend recursively into any directories "
             "encountered."
    )
    parser.add_argument(
        "--skip_files", nargs="*", default=[],
        help="File pattern(s) to skip. You can specify wildcards like '*.txt' "
             "(but you will have to enclose that pattern in quotes under "
             "UNIX-like operating systems). The basename of each file will be "
             "tested against these filenames/patterns. Consider including "
             "Scalpel's 'audit.txt'."
    )
    parser.add_argument(
        "--filetypes", nargs="+", default=FILETYPES,
        help="File types to check. Options: {}".format(FILETYPES)
    )
    parser.add_argument(
        "--move_to",
        help="If the file is recognized as one of the specified file types, "
             "move it to the directory specified here."
    )
    parser.add_argument(
        "--delete_if_not_specified_file_type", action="store_true",
        help="If a file is NOT recognized as one of the specified file types, "
             "delete it. VERY DANGEROUS."
    )
    parser.add_argument(
        "--run_repeatedly", type=int,
        help="Run the tool repeatedly with a pause of <run_repeatedly> "
             "seconds between runs. (For this to work well with the move/"
             "delete options, you should specify one or more DIRECTORIES in "
             "the 'filename' arguments, not files, and you will need the "
             "--recursive option.)"
    )
    parser.add_argument(
        "--nprocesses", type=int, default=multiprocessing.cpu_count(),
        help="Specify the number of processes to run in parallel."
    )
    parser.add_argument(
        "--verbose", action="store_true",
        help="Verbose output"
    )
    parser.add_argument(
        "--show_zip_output", action="store_true",
        help="Verbose output from the external 'zip' tool"
    )
    args = parser.parse_args()
    main_only_quicksetup_rootlogger(
        level=logging.DEBUG if args.verbose else logging.INFO,
        with_process_id=True
    )

    # Further argument checks
    if args.move_to:
        if not os.path.isdir(args.move_to):
            raise ValueError("Destination directory {!r} is not a "
                             "directory".format(args.move_to))
    if not args.filetypes:
        raise ValueError("No file type to scan for")
    filetypes = [ft.lower() for ft in args.filetypes]
    if any(ft not in FILETYPES for ft in filetypes):
        raise ValueError("Invalid filetypes; choose from {}".format(FILETYPES))
    assert shutil.which("zip"), "Need 'zip' tool!"

    # Repeated scanning loop
    while True:
        log.info("Starting scan.")
        log.info("- Looking for filetypes {}", filetypes)
        log.info("- Scanning files/directories {!r}{}",
                 args.filename,
                 " recursively" if args.recursive else "")
        log.info("- Skipping files matching {!r}", args.skip_files)
        log.info("- Using {} simultaneous processes", args.nprocesses)
        if args.move_to:
            log.info("- Moving target files to " + args.move_to)
        if args.delete_if_not_specified_file_type:
            log.info("- Deleting non-target files.")

        # Iterate through files
        pool = multiprocessing.Pool(processes=args.nprocesses)
        for filename in gen_filenames(starting_filenames=args.filename,
                                      recursive=args.recursive):
            src_basename = os.path.basename(filename)
            if any(fnmatch.fnmatch(src_basename, pattern)
                   for pattern in args.skip_files):
                log.info("Skipping file as ordered: " + filename)
                continue
            exists, locked = exists_locked(filename)
            if locked or not exists:
                log.info("Skipping currently inaccessible file: " + filename)
                continue
            kwargs = {
                'filename': filename,
                'filetypes': filetypes,
                'move_to': args.move_to,
                'delete_if_not_specified_file_type':
                    args.delete_if_not_specified_file_type,
                'show_zip_output': args.show_zip_output,
            }
            # log.critical("start")
            pool.apply_async(process_file, [], kwargs)
            # result = pool.apply_async(process_file, [], kwargs)
            # result.get()  # will re-raise any child exceptions
            # ... but it waits for the process to complete! That's no help.
            # log.critical("next")
            # ... https://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing  # noqa
        pool.close()
        pool.join()

        log.info("Finished scan.")
        if args.run_repeatedly is None:
            break
        log.info("Sleeping for {} s...", args.run_repeatedly)
        sleep(args.run_repeatedly)

Response: This is the main function for the find_recovered_openxml tool. It is used to find the actual set of files that are not part of the NTFS disk and the actual set of files that are not part of the NTFS disk.


Instruction:
Write a Python 3 function for
moving the file to which this class refers to a new location.

Response:
def move_to(self, destination_filename: str,
                alter_if_clash: bool = True) -> None:
        """
        Move the file to which this class refers to a new location.
        The function will not overwrite existing files (but offers the option
        to rename files slightly to avoid a clash).

        Args:
            destination_filename: filename to move to
            alter_if_clash: if ``True`` (the default), appends numbers to
                the filename if the destination already exists, so that the
                move can proceed.
        """
        if not self.src_filename:
            return
        if alter_if_clash:
            counter = 0
            while os.path.exists(destination_filename):
                root, ext = os.path.splitext(destination_filename)
                destination_filename = "{r}_{c}{e}".format(
                    r=root, c=counter, e=ext)
                counter += 1
            # ... for example, "/a/b/c.txt" becomes "/a/b/c_0.txt", then
            # "/a/b/c_1.txt", and so on.
        else:
            if os.path.exists(destination_filename):
                src = self.rescue_filename or self.src_filename
                log.warning("Destination exists; won't move {!r} to {!r}",
                            src, destination_filename)
                return
        if self.rescue_filename:
            shutil.move(self.rescue_filename, destination_filename)
            os.remove(self.src_filename)
            log.info("Moved recovered file {!r} to {!r} and deleted corrupted "
                     "original {!r}",
                     self.rescue_filename,
                     destination_filename,
                     self.src_filename)
            self.rescue_filename = ""
        else:
            shutil.move(self.src_filename, destination_filename)
            log.info("Moved {!r} to {!r}", self.src_filename,
                     destination_filename)
        self.src_filename = ""


Instruction:
How would you implement a function in Python 3 that
checks if the file should be built or not

Response:
def should_build(self, fpath, meta):
        """
        Checks if the file should be built or not
        Only skips layouts which are tagged as INCREMENTAL
        Rebuilds only those files with mtime changed since previous build
        """
        if meta.get('layout', self.default_template) in self.inc_layout:
            if self.prev_mtime.get(fpath, 0) == os.path.getmtime(fpath):
                return False
            else:
                return True
        return True


Instruction:
Implement a function in Python 3 to
return a dict of queues in cluster and servers running them

Response:
def clusterQueues(self):
        """ Return a dict of queues in cluster and servers running them
        """
        servers = yield self.getClusterServers()

        queues = {}

        for sname in servers:
            qs = yield self.get('rhumba.server.%s.queues' % sname)
            uuid = yield self.get('rhumba.server.%s.uuid' % sname)

            qs = json.loads(qs)

            for q in qs:
                if q not in queues:
                    queues[q] = []

                queues[q].append({'host': sname, 'uuid': uuid})

        defer.returnValue(queues)


Instruction:
Make a summary of the following Python 3 code
def clusterStatus(self):
        servers = yield self.getClusterServers()

        d = {
            'workers': {},
            'crons': {},
            'queues': {}
        }

        now = time.time()

        reverse_map = {}

        for sname in servers:
            last = yield self.get('rhumba.server.%s.heartbeat' % sname)
            status = yield self.get('rhumba.server.%s.status' % sname)
            uuid = yield self.get('rhumba.server.%s.uuid' % sname)

            reverse_map[uuid] = sname

            if not last:
                last = 0

            last = float(last)

            if (status == 'ready') and (now - last > 5):
                status = 'offline'

            if not sname in d['workers']:
                d['workers'][sname] = []

            d['workers'][sname].append({
                'lastseen': last,
                'status': status,
                'id': uuid
            })

        # Crons
        crons = yield self.keys('rhumba\.crons\.*')

        for key in crons:
            segments = key.split('.')

            queue = segments[2]
            if queue not in d['crons']:
                d['crons'][queue] = {'methods': {}}

            if len(segments)==4:
                last = yield self.get(key)
                d['crons'][queue]['methods'][segments[3]] = float(last)
            else:
                uid = yield self.get(key)
                d['crons'][queue]['master'] = '%s:%s' % (uid, reverse_map[uid])

        # Queues
        queue_keys = yield self.keys('rhumba.qstats.*')

        for key in queue_keys:
            qname = key.split('.')[2]
            if qname not in d['queues']:
                qlen = yield self.queueSize(qname)

                stats = yield self.getQueueMessageStats(qname)

                d['queues'][qname] = {
                    'waiting': qlen,
                    'messages': stats
                }

        defer.returnValue(d)

Response: Returns a dict of cluster nodes and their status information


Instruction:
Given the following Python 3 function, write the documentation
def get(self, index, id, fields=None, doc_type=EsConst.ALL_VALUES, **query_params):
        if fields:
            query_params[EsConst.FIELDS] = fields

        path = self._es_parser.make_path(index, doc_type, id)
        result = yield self._perform_request(HttpMethod.GET, path, params=query_params)
        returnValue(result)

Response: Retrieve specific record by id


Instruction:
How would you explain what the following Python 3 function does
def exists(self, index, doc_type, id, **query_params):
        self._es_parser.is_not_empty_params(index, doc_type, id)
        path = self._es_parser.make_path(index, doc_type, id)
        result = yield self._perform_request(HttpMethod.HEAD,
                                             path,
                                             params=query_params)
        returnValue(result)

Response: Check if the document exists in the elastic search


Instruction:
Implement a function in Python 3 to
get the _source field of the document with the specified id

Response:
def get_source(self, index, doc_type, id, **query_params):
        """
        Get the _source of the document
        `<http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html>`_
        :param index: the index name
        :param doc_type: the document type
        :param id: the id of the doc type
        :arg _source: True or false to return the _source field or not, or a
            list of fields to return
        :arg _source_exclude: A list of fields to exclude from the returned
            _source field
        :arg _source_include: A list of fields to extract and return from the
            _source field
        :arg parent: The ID of the parent document
        :arg preference: Specify the node or shard the operation should be
            performed on (default: random)
        :arg realtime: Specify whether to perform the operation in realtime or
            search mode
        :arg refresh: Refresh the shard containing the document before
            performing the operation
        :arg routing: Specific routing value
        :arg version: Explicit version number for concurrency control
        :arg version_type: Specific version type, valid choices are: 'internal',
            'external', 'external_gte', 'force'
        """
        self._es_parser.is_not_empty_params(index, doc_type, id)
        path = self._es_parser.make_path(index, doc_type, id, EsMethods.SOURCE)
        result = yield self._perform_request(HttpMethod.GET,
                                             path,
                                             params=query_params)
        returnValue(result)


Instruction:
Can you tell what is the following Python 3 function doing
def mget(self, body, index=None, doc_type=None, **query_params):
        self._es_parser.is_not_empty_params(body)

        path = self._es_parser.make_path(index,
                                         doc_type,
                                         EsMethods.MULTIPLE_GET)

        result = yield self._perform_request(HttpMethod.GET,
                                             path,
                                             body=body,
                                             params=query_params)
        returnValue(result)

Response: Get multiple document from the same index and doc_type by ids


Instruction:
Write a Python 3 script to
update a document in an index

Response:
def update(self, index, doc_type, id, body=None, **query_params):
        """
        Update a document with the body param or list of ids
        `<http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html>`_

        :param index: the name of the index
        :param doc_type: the document type
        :param id: the id of the doc type
        :param body: the data to update in the index
        :arg consistency: Explicit write consistency setting for the operation,
            valid choices are: 'one', 'quorum', 'all'
        :arg fields: A comma-separated list of fields to return in the response
        :arg lang: The script language (default: groovy)
        :arg parent: ID of the parent document. Is is only used for routing and
            when for the upsert request
        :arg refresh: Refresh the index after performing the operation
        :arg retry_on_conflict: Specify how many times should the operation be
            retried when a conflict occurs (default: 0)
        :arg routing: Specific routing value
        :arg script: The URL-encoded script definition (instead of using request
            body)
        :arg script_id: The id of a stored script
        :arg scripted_upsert: True if the script referenced in script or
            script_id should be called to perform inserts - defaults to false
        :arg timeout: Explicit operation timeout
        :arg timestamp: Explicit timestamp for the document
        :arg ttl: Expiration time for the document
        :arg version: Explicit version number for concurrency control
        :arg version_type: Specific version type, valid choices are: 'internal',
            'force'
        """
        self._es_parser.is_not_empty_params(index, doc_type, id)
        path = self._es_parser.make_path(index, doc_type, id, EsMethods.UPDATE)
        result = yield self._perform_request(HttpMethod.POST, path, body=body, params=query_params)
        returnValue(result)


Instruction:
Write a Python 3 script for
making a search on the elastic search index

Response:
def search(self, index=None, doc_type=None, body=None, **query_params):
        """
        Make a search query on the elastic search
        `<http://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html>`_

        :param index: the index name to query
        :param doc_type: he doc type to search in
        :param body: the query
        :param query_params: params
        :arg _source: True or false to return the _source field or not, or a
            list of fields to return
        :arg _source_exclude: A list of fields to exclude from the returned
            _source field
        :arg _source_include: A list of fields to extract and return from the
            _source field
        :arg allow_no_indices: Whether to ignore if a wildcard indices
            expression resolves into no concrete indices. (This includes `_all`
            string or when no indices have been specified)
        :arg analyze_wildcard: Specify whether wildcard and prefix queries
            should be analyzed (default: false)
        :arg analyzer: The analyzer to use for the query string
        :arg default_operator: The default operator for query string query (AND
            or OR), default 'OR', valid choices are: 'AND', 'OR'
        :arg df: The field to use as default where no field prefix is given in
            the query string
        :arg expand_wildcards: Whether to expand wildcard expression to concrete
            indices that are open, closed or both., default 'open', valid
            choices are: 'open', 'closed', 'none', 'all'
        :arg explain: Specify whether to return detailed information about score
            computation as part of a hit
        :arg fielddata_fields: A comma-separated list of fields to return as the
            field data representation of a field for each hit
        :arg fields: A comma-separated list of fields to return as part of a hit
        :arg from\_: Starting offset (default: 0)
        :arg ignore_unavailable: Whether specified concrete indices should be
            ignored when unavailable (missing or closed)
        :arg lenient: Specify whether format-based query failures (such as
            providing text to a numeric field) should be ignored
        :arg lowercase_expanded_terms: Specify whether query terms should be
            lowercased
        :arg preference: Specify the node or shard the operation should be
            performed on (default: random)
        :arg q: Query in the Lucene query string syntax
        :arg request_cache: Specify if request cache should be used for this
            request or not, defaults to index level setting
        :arg routing: A comma-separated list of specific routing values
        :arg scroll: Specify how long a consistent view of the index should be
            maintained for scrolled search
        :arg search_type: Search operation type, valid choices are:
            'query_then_fetch', 'dfs_query_then_fetch'
        :arg size: Number of hits to return (default: 10)
        :arg sort: A comma-separated list of <field>:<direction> pairs
        :arg stats: Specific 'tag' of the request for logging and statistical
            purposes
        :arg suggest_field: Specify which field to use for suggestions
        :arg suggest_mode: Specify suggest mode, default 'missing', valid
            choices are: 'missing', 'popular', 'always'
        :arg suggest_size: How many suggestions to return in response
        :arg suggest_text: The source text for which the suggestions should be
            returned
        :arg terminate_after: The maximum number of documents to collect for
            each shard, upon reaching which the query execution will terminate
            early.
        :arg timeout: Explicit operation timeout
        :arg track_scores: Whether to calculate and return scores even if they
            are not used for sorting
        :arg version: Specify whether to return document version as part of a
            hit
        """
        path = self._es_parser.make_path(index, doc_type, EsMethods.SEARCH)
        result = yield self._perform_request(HttpMethod.POST, path, body=body, params=query_params)
        returnValue(result)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def explain(self, index, doc_type, id, body=None, **query_params):
        self._es_parser.is_not_empty_params(index, doc_type, id)

        path = self._es_parser.make_path(index,
                                         doc_type,
                                         id,
                                         EsMethods.EXPLAIN)

        result = yield self._perform_request(HttpMethod.GET,
                                             path,
                                             body,
                                             params=query_params)
        returnValue(result)

Response: The explain API computes a score explanation for a specific document and a specific document.


Instruction:
Can you generate the documentation for the following Python 3 function
def delete(self, index, doc_type, id, **query_params):
        path = self._es_parser.make_path(index, doc_type, id)
        result = yield self._perform_request(HttpMethod.DELETE, path, params=query_params)
        returnValue(result)

Response: Delete specific record by id


Instruction:
Can you generate the documentation for the following Python 3 function
def index(self, index, doc_type, body, id=None, **query_params):
        self._es_parser.is_not_empty_params(index, doc_type, body)

        method = HttpMethod.POST if id in NULL_VALUES else HttpMethod.PUT
        path = self._es_parser.make_path(index, doc_type, id)
        result = yield self._perform_request(method, path, body, params=query_params)
        returnValue(result)

Response: Adds or updates a typed JSON document in a specific index.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def create(self, index, doc_type, body, id=None, **query_params):
        query_params['op_type'] = 'create'
        result = yield self.index(index, doc_type, body, id=id, params=query_params)
        returnValue(result)

Response: Adds a typed JSON document in a specific index making it searchable.


Instruction:
Can you generate the documentation for the following Python 3 function
def scan(self, index, doc_type, query=None, scroll='5m', preserve_order=False, size=10, **kwargs):
        if not preserve_order:
            kwargs['search_type'] = 'scan'
        # initial search
        results = yield self.search(index=index,
                                    doc_type=doc_type,
                                    body=query,
                                    size=size,
                                    scroll=scroll,
                                    **kwargs)

        returnValue(Scroller(self, results, scroll, size))

Response: Scans the index for the language of the entry in the index.


Instruction:
How would you implement a function in Python 3 that
executes a query and get the number of matches for that query.

Response:
def count(self, index=None, doc_type=None, body=None, **query_params):
        """
        Execute a query and get the number of matches for that query.
        `<http://www.elastic.co/guide/en/elasticsearch/reference/current/search-count.html>`_
        :param index: A comma-separated list of indices to restrict the results
        :param doc_type: A comma-separated list of types to restrict the results
        :param body: A query to restrict the results specified with the Query DSL
            (optional)
        :arg allow_no_indices: Whether to ignore if a wildcard indices
            expression resolves into no concrete indices. (This includes `_all`
            string or when no indices have been specified)
        :arg analyze_wildcard: Specify whether wildcard and prefix queries
            should be analyzed (default: false)
        :arg analyzer: The analyzer to use for the query string
        :arg default_operator: The default operator for query string query (AND
            or OR), default 'OR', valid choices are: 'AND', 'OR'
        :arg df: The field to use as default where no field prefix is given in
            the query string
        :arg expand_wildcards: Whether to expand wildcard expression to concrete
            indices that are open, closed or both., default 'open', valid
            choices are: 'open', 'closed', 'none', 'all'
        :arg ignore_unavailable: Whether specified concrete indices should be
            ignored when unavailable (missing or closed)
        :arg lenient: Specify whether format-based query failures (such as
            providing text to a numeric field) should be ignored
        :arg lowercase_expanded_terms: Specify whether query terms should be
            lowercased
        :arg min_score: Include only documents with a specific `_score` value in
            the result
        :arg preference: Specify the node or shard the operation should be
            performed on (default: random)
        :arg q: Query in the Lucene query string syntax
        :arg routing: Specific routing value
        """
        if doc_type and not index:
            index = EsConst.ALL_VALUES

        path = self._es_parser.make_path(index, doc_type, EsMethods.COUNT)
        result = yield self._perform_request(HttpMethod.GET, path, body, params=query_params)
        returnValue(result)


Instruction:
Given the following Python 3 function, write the documentation
def bulk(self, body, index=None, doc_type=None, **query_params):
        self._es_parser.is_not_empty_params(body)
        path = self._es_parser.make_path(index, doc_type, EsMethods.BULK)
        result = yield self._perform_request(HttpMethod.POST,
                                             path,
                                             self._bulk_body(body),
                                             params=query_params)
        returnValue(result)

Response: Perform bulk operation on a set of items.


Instruction:
How would you code a function in Python 3 to
execute multiple search requests within the same API.

Response:
def msearch(self, body, index=None, doc_type=None, **query_params):
        """
        Execute several search requests within the same API.
        `<http://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html>`_
        :arg body: The request definitions (metadata-search request definition
            pairs), separated by newlines
        :arg index: A comma-separated list of index names to use as default
        :arg doc_type: A comma-separated list of document types to use as
            default
        :arg search_type: Search operation type, valid choices are:
            'query_then_fetch', 'query_and_fetch', 'dfs_query_then_fetch',
            'dfs_query_and_fetch'
        """
        self._es_parser.is_not_empty_params(body)
        path = self._es_parser.make_path(index,
                                         doc_type,
                                         EsMethods.MULTIPLE_SEARCH)

        result = yield self._perform_request(HttpMethod.GET,
                                             path,
                                             self._bulk_body(body),
                                             params=query_params)
        returnValue(result)


Instruction:
How would you implement a function in Python 3 that
closes all http connections.

Response:
def close(self):
        """
        close all http connections.
        returns a deferred that fires once they're all closed.
        """

        def validate_client(client):
            """
            Validate that the connection is for the current client
            :param client:
            :return:
            """
            host, port = client.addr
            parsed_url = urlparse(self._hostname)
            return host == parsed_url.hostname and port == parsed_url.port

        # read https://github.com/twisted/treq/issues/86
        # to understand the following...
        def _check_fds(_):
            fds = set(reactor.getReaders() + reactor.getReaders())
            if not [fd for fd in fds if isinstance(fd, Client) and validate_client(fd)]:
                return

            return deferLater(reactor, 0, _check_fds, None)

        pool = self._async_http_client_params["pool"]
        return pool.closeCachedConnections().addBoth(_check_fds)


Instruction:
Write a Python 3 function for
removing invalid coordinates from a sequence of coordinates.

Response:
def clean_coordinates(coords, silent=False):
    """
    Removes invalid coordinates (this is generally caused
    by importing coordinates outside of the DEM)
    """
    l1 = len(coords)
    coords = coords[~N.isnan(coords).any(axis=1)] # remove NaN values
    l2 = len(coords)
    if not silent:
        msg = "{0} coordinates".format(l2)
        if l2 < l1:
            msg += " ({0} removed as invalid)".format(l1-l2)
        print(msg)
    return coords


Instruction:
Can you tell what is the following Python 3 function doing
def offset_mask(mask):
    def axis_data(axis):
        """Gets the bounds of a masked area along a certain axis"""
        x = mask.sum(axis)
        trimmed_front = N.trim_zeros(x,"f")
        offset = len(x)-len(trimmed_front)
        size = len(N.trim_zeros(trimmed_front,"b"))
        return offset,size

    xo,xs = axis_data(0)
    yo,ys = axis_data(1)

    array = mask[yo:yo+ys,xo:xo+xs]
    offset = (yo,xo)
    return offset, array

Response: Returns a mask shrunk to the minimum bounding rectangle of the nonzero portion of the previous mask and its offset from the original mask.


Instruction:
Given the following Python 3 function, write the documentation
def extract_line(geom, dem, **kwargs):
    kwargs.setdefault('masked', True)

    coords_in = coords_array(geom)

    # Transform geometry into pixels
    f = lambda *x: ~dem.transform * x
    px = transform(f,geom)

    # Subdivide geometry segments if option is given
    interval = kwargs.pop('subdivide', 1)
    if interval is not None:
        px = subdivide(px, interval=interval)

    # Transform pixels back to geometry
    # to capture subdivisions
    f = lambda *x: dem.transform * (x[0],x[1])
    geom = transform(f,px)

    # Get min and max coords for windowing
    # Does not deal with edge cases where points
    # are outside of footprint of DEM

    coords_px = coords_array(px)
    mins = N.floor(coords_px.min(axis=0))
    maxs = N.ceil(coords_px.max(axis=0))

    window = tuple((int(mn),int(mx))
        for mn,mx in zip(mins[::-1],maxs[::-1]))

    aff = Affine.translation(*(-mins))

    f = lambda *x: aff * x
    px_to_extract = transform(f,px)

    band = dem.read(1, window=window, **kwargs)
    extracted = bilinear(band, px_to_extract)
    coords = coords_array(extracted)

    coords[:,:2] = coords_array(geom)
    return coords

Response: Extract a linear feature from a rasterio geospatial dataset.


Instruction:
How would you code a function in Python 3 to
fork a new node process and returns a Deferred that resolves when the child process finishes.

Response:
def fork(executable, args=(), env={}, path=None, timeout=3600):
    """fork
    Provides a deferred wrapper function with a timeout function

    :param executable: Executable
    :type executable: str.
    :param args: Tupple of arguments
    :type args: tupple.
    :param env: Environment dictionary
    :type env: dict.
    :param timeout: Kill the child process if timeout is exceeded
    :type timeout: int.
    """
    d = defer.Deferred()
    p = ProcessProtocol(d, timeout)
    reactor.spawnProcess(p, executable, (executable,)+tuple(args), env, path)
    return d


Instruction:
How would you explain what the following Python 3 function does
def make_grammar(dialect: str) -> SqlGrammar:
    if dialect == SqlaDialectName.MYSQL:
        return mysql_grammar
    elif dialect == SqlaDialectName.MSSQL:
        return mssql_grammar
    else:
        raise AssertionError("Invalid SQL dialect: {}".format(repr(dialect)))

Response: Returns an instance of the appropriate SqlGrammar class for the given SQL dialect.


Instruction:
Can you generate the documentation for the following Python 3 function
def number_to_dp(number: Optional[float],
                 dp: int,
                 default: Optional[str] = "",
                 en_dash_for_minus: bool = True) -> str:
    """
    Format number to ``dp`` decimal places, optionally using a UTF-8 en dash
    for minus signs.
    """
    if number is None:
        return default
    if number == float("inf"):
        return u"∞"
    if number == float("-inf"):
        s = u"-∞"
    else:
        s = u"{:.{precision}f}".format(number, precision=dp)
    if en_dash_for_minus:
        s = s.replace("-", u"–")  # hyphen becomes en dash for minus sign
    return s

Response: Format number to dp decimal places.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def debug_form_contents(form: cgi.FieldStorage,
                        to_stderr: bool = True,
                        to_logger: bool = False) -> None:
    """
    Writes the keys and values of a CGI form to ``stderr``.
    """
    for k in form.keys():
        text = "{0} = {1}".format(k, form.getvalue(k))
        if to_stderr:
            sys.stderr.write(text)
        if to_logger:
            log.info(text)

Response: Writes the keys and values of a CGI form to stderr.


Instruction:
Given the following Python 3 function, write the documentation
def cgi_method_is_post(environ: Dict[str, str]) -> bool:
    method = environ.get("REQUEST_METHOD", None)
    if not method:
        return False
    return method.upper() == "POST"

Response: Determines if the CGI method is POST.


Instruction:
Given the following Python 3 function, write the documentation
def get_cgi_parameter_str(form: cgi.FieldStorage,
                          key: str,
                          default: str = None) -> str:
    """
    Extracts a string parameter from a CGI form.
    Note: ``key`` is CASE-SENSITIVE.
    """
    paramlist = form.getlist(key)
    if len(paramlist) == 0:
        return default
    return paramlist[0]

Response: Extracts a string parameter from a CGI form.


Instruction:
Can you write a function in Python 3 where it
extracts a string parameter from a CGI form or None if the key doesn t exist.

Response:
def get_cgi_parameter_str_or_none(form: cgi.FieldStorage,
                                  key: str) -> Optional[str]:
    """
    Extracts a string parameter from a CGI form, or ``None`` if the key doesn't
    exist or the string is zero-length.
    """
    s = get_cgi_parameter_str(form, key)
    if s is None or len(s) == 0:
        return None
    return s


Instruction:
Write a Python 3 function that can
extract a list of values all with the same key from a CGI form.

Response:
def get_cgi_parameter_list(form: cgi.FieldStorage, key: str) -> List[str]:
    """
    Extracts a list of values, all with the same key, from a CGI form.
    """
    return form.getlist(key)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_cgi_parameter_bool(form: cgi.FieldStorage, key: str) -> bool:
    return is_1(get_cgi_parameter_str(form, key))

Response: Extracts a boolean parameter from a CGI form.


Instruction:
Write a Python 3 script for
extracting a boolean parameter from a CGI form or returns default if the key is not found.

Response:
def get_cgi_parameter_bool_or_default(form: cgi.FieldStorage,
                                      key: str,
                                      default: bool = None) -> Optional[bool]:
    """
    Extracts a boolean parameter from a CGI form (``"1"`` = ``True``,
    other string = ``False``, absent/zero-length string = default value).
    """
    s = get_cgi_parameter_str(form, key)
    if s is None or len(s) == 0:
        return default
    return is_1(s)


Instruction:
Make a summary of the following Python 3 code
def get_cgi_parameter_bool_or_none(form: cgi.FieldStorage,
                                   key: str) -> Optional[bool]:
    """
    Extracts a boolean parameter from a CGI form (``"1"`` = ``True``,
    other string = False, absent/zero-length string = ``None``).
    """
    return get_cgi_parameter_bool_or_default(form, key, default=None)

Response: Extracts a boolean parameter from a CGI form or returns None if the key is not found.


Instruction:
Can you tell what is the following Python 3 function doing
def get_cgi_parameter_int(form: cgi.FieldStorage, key: str) -> Optional[int]:
    return get_int_or_none(get_cgi_parameter_str(form, key))

Response: Extracts an integer parameter from a CGI form or None if the key is absent or the string value is not convertible to int.


Instruction:
Implement a function in Python 3 to
extract a float parameter from a CGI form or None if the key is absent.

Response:
def get_cgi_parameter_float(form: cgi.FieldStorage,
                            key: str) -> Optional[float]:
    """
    Extracts a float parameter from a CGI form, or None if the key is
    absent or the string value is not convertible to ``float``.
    """
    return get_float_or_none(get_cgi_parameter_str(form, key))


Instruction:
Here you have a function in Python 3, explain what it does
def get_cgi_parameter_datetime(form: cgi.FieldStorage,
                               key: str) -> Optional[datetime.datetime]:
    """
    Extracts a date/time parameter from a CGI form. Applies the LOCAL
    timezone if none specified.
    """
    try:
        s = get_cgi_parameter_str(form, key)
        if not s:
            # if you dateutil.parser.parse() an empty string,
            # you get today's date
            return None
        d = dateutil.parser.parse(s)
        if d.tzinfo is None:  # as it will be
            d = d.replace(tzinfo=dateutil.tz.tzlocal())
        return d
    except ValueError:
        return None

Response:
Extracts a date or time parameter from a CGI form. Applies the LOCAL
CTYPE timezone if none specified.


Instruction:
Here you have a function in Python 3, explain what it does
def get_cgi_parameter_file(form: cgi.FieldStorage,
                           key: str) -> Optional[bytes]:
    """
    Extracts a file's contents from a "file" input in a CGI form, or None
    if no such file was uploaded.
    """
    (filename, filecontents) = get_cgi_parameter_filename_and_file(form, key)
    return filecontents

Response: Extracts a file s contents from a file input in a CGI form.


Instruction:
Can you implement a function in Python 3 that
returns a filename and contents of a file input in a CGI form.

Response:
def get_cgi_parameter_filename_and_file(form: cgi.FieldStorage, key: str) \
        -> Tuple[Optional[str], Optional[bytes]]:
    """
    Extracts a file's name and contents from a "file" input in a CGI form.
    Returns ``(name, contents)``, or ``(None, None)`` if no such file was
    uploaded.
    """
    if not (key in form):
        log.warning('get_cgi_parameter_file: form has no key {}', key)
        return None, None
    fileitem = form[key]  # a nested FieldStorage instance; see
    # http://docs.python.org/2/library/cgi.html#using-the-cgi-module
    if isinstance(fileitem, cgi.MiniFieldStorage):
        log.warning('get_cgi_parameter_file: MiniFieldStorage found - did you '
                    'forget to set enctype="multipart/form-data" in '
                    'your form?')
        return None, None
    if not isinstance(fileitem, cgi.FieldStorage):
        log.warning('get_cgi_parameter_file: no FieldStorage instance with '
                    'key {} found', key)
        return None, None
    if fileitem.filename and fileitem.file:  # can check "file" or "filename"
        return fileitem.filename, fileitem.file.read()
        # as per
        # http://upsilon.cc/~zack/teaching/0607/techweb/02-python-cgi.pdf
        # Alternative:
        # return get_cgi_parameter_str(form, key) # contents of the file
    # Otherwise, information about problems:
    if not fileitem.file:
        log.warning('get_cgi_parameter_file: fileitem has no file')
    elif not fileitem.filename:
        log.warning('get_cgi_parameter_file: fileitem has no filename')
    else:
        log.warning('get_cgi_parameter_file: unknown failure reason')
    return None, None


Instruction:
Can you generate the documentation for the following Python 3 function
def cgi_parameter_exists(form: cgi.FieldStorage, key: str) -> bool:
    s = get_cgi_parameter_str(form, key)
    return s is not None

Response: Does a CGI form contain the key?


Instruction:
Can you write a function in Python 3 where it
returns an environment variable s value CGI - escaped or None.

Response:
def getenv_escaped(key: str, default: str = None) -> Optional[str]:
    """
    Returns an environment variable's value, CGI-escaped, or ``None``.
    """
    value = os.getenv(key, default)
    # noinspection PyDeprecation
    return cgi.escape(value) if value is not None else None


Instruction:
Write a Python 3 script for
returning a CGI - escaped version of the value read from an INI file using .

Response:
def getconfigvar_escaped(config: configparser.ConfigParser,
                         section: str,
                         key: str) -> Optional[str]:
    """
    Returns a CGI-escaped version of the value read from an INI file using
    :class:`ConfigParser`, or ``None``.
    """
    value = config.get(section, key)
    # noinspection PyDeprecation
    return cgi.escape(value) if value is not None else None


Instruction:
Can you generate the documentation for the following Python 3 function
def get_cgi_fieldstorage_from_wsgi_env(
        env: Dict[str, str],
        include_query_string: bool = True) -> cgi.FieldStorage:
    """
    Returns a :class:`cgi.FieldStorage` object from the WSGI environment.
    """
    # http://stackoverflow.com/questions/530526/accessing-post-data-from-wsgi
    post_env = env.copy()
    if not include_query_string:
        post_env['QUERY_STRING'] = ''
    form = cgi.FieldStorage(
        fp=env['wsgi.input'],
        environ=post_env,
        keep_blank_values=True
    )
    return form

Response: Returns a cgi. FieldStorage object from the WSGI environment.


Instruction:
Write a Python 3 function that can
convert a PNG blob into a local URL encapsulating the PNG.

Response:
def get_png_data_url(blob: Optional[bytes]) -> str:
    """
    Converts a PNG blob into a local URL encapsulating the PNG.
    """
    return BASE64_PNG_URL_PREFIX + base64.b64encode(blob).decode('ascii')


Instruction:
Can you implement a function in Python 3 that
converts a PNG blob to an HTML IMG tag with embedded data.

Response:
def get_png_img_html(blob: Union[bytes, memoryview],
                     extra_html_class: str = None) -> str:
    """
    Converts a PNG blob to an HTML IMG tag with embedded data.
    """
    return """<img {}src="{}" />""".format(
        'class="{}" '.format(extra_html_class) if extra_html_class else "",
        get_png_data_url(blob)
    )


Instruction:
How would you explain what the following Python 3 function does
def pdf_result(pdf_binary: bytes,
               extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None,
               filename: str = None) -> WSGI_TUPLE_TYPE:
    """
    Returns ``(contenttype, extraheaders, data)`` tuple for a PDF.
    """
    extraheaders = extraheaders or []
    if filename:
        extraheaders.append(
            ('content-disposition', 'inline; filename="{}"'.format(filename))
        )
    contenttype = 'application/pdf'
    if filename:
        contenttype += '; filename="{}"'.format(filename)
    # log.debug("type(pdf_binary): {}", type(pdf_binary))
    return contenttype, extraheaders, pdf_binary

Response: Returns a WSGI response object for a PDF.


Instruction:
Given the following Python 3 function, write the documentation
def zip_result(zip_binary: bytes,
               extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None,
               filename: str = None) -> WSGI_TUPLE_TYPE:
    """
    Returns ``(contenttype, extraheaders, data)`` tuple for a ZIP.
    """
    extraheaders = extraheaders or []
    if filename:
        extraheaders.append(
            ('content-disposition', 'inline; filename="{}"'.format(filename))
        )
    contenttype = 'application/zip'
    if filename:
        contenttype += '; filename="{}"'.format(filename)
    return contenttype, extraheaders, zip_binary

Response: Returns a WSGI response body for a ZIP file.


Instruction:
How would you code a function in Python 3 to
return a WSGI response tuple for UTF - 8 HTML.

Response:
def html_result(html: str,
                extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None) \
        -> WSGI_TUPLE_TYPE:
    """
    Returns ``(contenttype, extraheaders, data)`` tuple for UTF-8 HTML.
    """
    extraheaders = extraheaders or []
    return 'text/html; charset=utf-8', extraheaders, html.encode("utf-8")


Instruction:
Write a Python 3 function that can
return a WSGI response object for the XML.

Response:
def xml_result(xml: str,
               extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None) \
        -> WSGI_TUPLE_TYPE:
    """
    Returns ``(contenttype, extraheaders, data)`` tuple for UTF-8 XML.
    """
    extraheaders = extraheaders or []
    return 'text/xml; charset=utf-8', extraheaders, xml.encode("utf-8")


Instruction:
Can you create a Python 3 function that
returns WSGI response body for UTF - 8 text.

Response:
def text_result(text: str,
                extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None,
                filename: str = None) -> WSGI_TUPLE_TYPE:
    """
    Returns ``(contenttype, extraheaders, data)`` tuple for UTF-8 text.
    """
    extraheaders = extraheaders or []
    if filename:
        extraheaders.append(
            ('content-disposition', 'inline; filename="{}"'.format(filename))
        )
    contenttype = 'text/plain; charset=utf-8'
    if filename:
        contenttype += '; filename="{}"'.format(filename)
    return contenttype, extraheaders, text.encode("utf-8")


Instruction:
How would you implement a function in Python 3 that
prints the result of a CGI - script to stdout.

Response:
def print_result_for_plain_cgi_script_from_tuple(
        contenttype_headers_content: WSGI_TUPLE_TYPE,
        status: str = '200 OK') -> None:
    """
    Writes HTTP result to stdout.

    Args:
        contenttype_headers_content:
            the tuple ``(contenttype, extraheaders, data)``
        status:
            HTTP status message (default ``"200 OK``)
    """
    contenttype, headers, content = contenttype_headers_content
    print_result_for_plain_cgi_script(contenttype, headers, content, status)


Instruction:
Can you generate the documentation for the following Python 3 function
def print_result_for_plain_cgi_script(contenttype: str,
                                      headers: TYPE_WSGI_RESPONSE_HEADERS,
                                      content: bytes,
                                      status: str = '200 OK') -> None:
    """
    Writes HTTP request result to stdout.
    """
    headers = [
        ("Status", status),
        ("Content-Type", contenttype),
        ("Content-Length", str(len(content))),
    ] + headers
    sys.stdout.write("\n".join([h[0] + ": " + h[1] for h in headers]) + "\n\n")
    sys.stdout.write(content)

Response: Writes HTTP request result to stdout.


Instruction:
Can you tell what is the following Python 3 function doing
def wsgi_simple_responder(
        result: Union[str, bytes],
        handler: Callable[[Union[str, bytes]], WSGI_TUPLE_TYPE],
        start_response: TYPE_WSGI_START_RESPONSE,
        status: str = '200 OK',
        extraheaders: TYPE_WSGI_RESPONSE_HEADERS = None) \
        -> TYPE_WSGI_APP_RESULT:
    """
    Simple WSGI app.

    Args:
        result: the data to be processed by ``handler``
        handler: a function returning a ``(contenttype, extraheaders, data)``
            tuple, e.g. ``text_result``, ``html_result``
        start_response: standard WSGI ``start_response`` function
        status: status code (default ``"200 OK"``)
        extraheaders: optional extra HTTP headers

    Returns:
        WSGI application result

    """
    extraheaders = extraheaders or []
    (contenttype, extraheaders2, output) = handler(result)
    response_headers = [('Content-Type', contenttype),
                        ('Content-Length', str(len(output)))]
    response_headers.extend(extraheaders)
    if extraheaders2 is not None:
        response_headers.extend(extraheaders2)
    # noinspection PyArgumentList
    start_response(status, response_headers)
    return [output]

Response: Simple WSGI response handler.


Instruction:
Can you generate the documentation for the following Python 3 function
def webify(v: Any, preserve_newlines: bool = True) -> str:
    nl = "<br>" if preserve_newlines else " "
    if v is None:
        return ""
    if not isinstance(v, str):
        v = str(v)
    # noinspection PyDeprecation
    return cgi.escape(v).replace("\n", nl).replace("\\n", nl)

Response: Converts a value into an HTML - safe string.


Instruction:
Write a Python 3 function for
returning a HTML - emboldens content unless blank.

Response:
def bold_if_not_blank(x: Optional[str]) -> str:
    """
    HTML-emboldens content, unless blank.
    """
    if x is None:
        return u"{}".format(x)
    return u"<b>{}</b>".format(x)


Instruction:
Here you have a function in Python 3, explain what it does
def make_urls_hyperlinks(text: str) -> str:
    find_url = r'''
        (?x)(              # verbose identify URLs within text
        (http|ftp|gopher)  # make sure we find a resource type
        ://                # ...needs to be followed by colon-slash-slash
        (\w+[:.]?){2,}     # at least two domain groups, e.g. (gnosis.)(cx)
        (/?|               # could be just the domain name (maybe w/ slash)
        [^ \n\r"]+         # or stuff then space, newline, tab, quote
        [\w/])             # resource name ends in alphanumeric or slash
        (?=[\s\.,>)'"\]])  # assert: followed by white or clause ending
        )                  # end of match group
    '''
    replace_url = r'\1'
    find_email = re.compile(r'([.\w\-]+@(\w[\w\-]+\.)+[\w\-]+)')
    # '.' doesn't need escaping inside square brackets
    # https://stackoverflow.com/questions/10397968/escape-dot-in-a-regex-range
    replace_email = r'\1'
    text = re.sub(find_url, replace_url, text)
    text = re.sub(find_email, replace_email, text)
    return text

Response: Makes hyperlinks to the text that appears to contain URLs.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def html_table_from_query(rows: Iterable[Iterable[Optional[str]]],
                          descriptions: Iterable[Optional[str]]) -> str:
    """
    Converts rows from an SQL query result to an HTML table.
    Suitable for processing output from the defunct function
    ``rnc_db.fetchall_with_fieldnames(sql)``.
    """
    html = u"
\n"

    # Header row
    html += u"
"
    for x in descriptions:
        if x is None:
            x = u""
        html += u"{}	".format(webify(x))
    html += u"
\n"

    # Data rows
    for row in rows:
        html += u"
"
        for x in row:
            if x is None:
                x = u""
            html += u"{}	".format(webify(x))
        html += u"
\n"

    html += u"
\n"
    return html

Response: Converts rows from an SQL query result to an HTML table.


Instruction:
Can you implement a function in Python 3 that
underlines a heading for RST files.

Response:
def rst_underline(heading: str, underline_char: str) -> str:
    """
    Underlines a heading for RST files.

    Args:
        heading: text to underline
        underline_char: character to use

    Returns:
        underlined heading, over two lines (without a final terminating
        newline)
    """
    assert "\n" not in heading
    assert len(underline_char) == 1
    return heading + "\n" + (underline_char * len(heading))


Instruction:
Write a Python 3 function for
writing the contents to a file if permitted.

Response:
def write_if_allowed(filename: str,
                     content: str,
                     overwrite: bool = False,
                     mock: bool = False) -> None:
    """
    Writes the contents to a file, if permitted.

    Args:
        filename: filename to write
        content: contents to write
        overwrite: permit overwrites?
        mock: pretend to write, but don't

    Raises:
        RuntimeError: if file exists but overwriting not permitted
    """
    # Check we're allowed
    if not overwrite and exists(filename):
        fail("File exists, not overwriting: {!r}".format(filename))

    # Make the directory, if necessary
    directory = dirname(filename)
    if not mock:
        mkdir_p(directory)

    # Write the file
    log.info("Writing to {!r}", filename)
    if mock:
        log.warning("Skipping writes as in mock mode")
    else:
        with open(filename, "wt") as outfile:
            outfile.write(content)


Instruction:
Can you write a function in Python 3 where it
returns the filename of the target RST file relative to a specified index file. Used to make the index refer to the RST.

Response:
def rst_filename_rel_autodoc_index(self, index_filename: str) -> str:
        """
        Returns the filename of the target RST file, relative to a specified
        index file. Used to make the index refer to the RST.
        """
        index_dir = dirname(abspath(expanduser(index_filename)))
        return relpath(self.target_rst_filename, start=index_dir)


Instruction:
Given the following Python 3 function, write the documentation
def python_module_name(self) -> str:
        if not self.is_python:
            return ""
        filepath = self.source_filename_rel_python_root
        dirs_and_base = splitext(filepath)[0]
        dir_and_file_parts = dirs_and_base.split(sep)
        return ".".join(dir_and_file_parts)

Response: Returns the name of the Python module that this instance refers to.


Instruction:
How would you code a function in Python 3 to
return the code type annotation for Pygments ; e. g. python for Python cpp for C ++ etc.

Response:
def pygments_language(self) -> str:
        """
        Returns the code type annotation for Pygments; e.g. ``python`` for
        Python, ``cpp`` for C++, etc.
        """
        extension = splitext(self.source_filename)[1]
        if extension in self.pygments_language_override:
            return self.pygments_language_override[extension]
        try:
            lexer = get_lexer_for_filename(self.source_filename)  # type: Lexer
            return lexer.name
        except ClassNotFound:
            log.warning("Don't know Pygments code type for extension {!r}",
                        self.source_extension)
            return CODE_TYPE_NONE


Instruction:
Make a summary of the following Python 3 code
def rst_content(self,
                    prefix: str = "",
                    suffix: str = "",
                    heading_underline_char: str = "=",
                    method: AutodocMethod = None) -> str:
        """
        Returns the text contents of an RST file that will automatically
        document our source file.

        Args:
            prefix: prefix, e.g. RST copyright comment
            suffix: suffix, after the part we're creating
            heading_underline_char: RST character to use to underline the
                heading
            method: optional method to override ``self.method``; see
                constructor

        Returns:
            the RST contents
        """
        spacer = "    "
        # Choose our final method
        if method is None:
            method = self.method
        is_python = self.is_python
        if method == AutodocMethod.BEST:
            if is_python:
                method = AutodocMethod.AUTOMODULE
            else:
                method = AutodocMethod.CONTENTS
        elif method == AutodocMethod.AUTOMODULE:
            if not is_python:
                method = AutodocMethod.CONTENTS

        # Write the instruction
        if method == AutodocMethod.AUTOMODULE:
            if self.source_rst_title_style_python:
                title = self.python_module_name
            else:
                title = self.source_filename_rel_project_root
            instruction = ".. automodule:: {modulename}\n    :members:".format(
                modulename=self.python_module_name
            )
        elif method == AutodocMethod.CONTENTS:
            title = self.source_filename_rel_project_root
            # Using ".. include::" with options like ":code: python" doesn't
            # work properly; everything comes out as Python.
            # Instead, see http://www.sphinx-doc.org/en/1.4.9/markup/code.html;
            # we need ".. literalinclude::" with ":language: LANGUAGE".

            instruction = (
                ".. literalinclude:: {filename}\n"
                "{spacer}:language: {language}".format(
                    filename=self.source_filename_rel_rst_file,
                    spacer=spacer,
                    language=self.pygments_language
                )
            )
        else:
            raise ValueError("Bad method!")

        # Create the whole file
        content = """
.. {filename}

{AUTOGENERATED_COMMENT}

{prefix}

{underlined_title}

{instruction}

{suffix}
        """.format(
            filename=self.rst_filename_rel_project_root,
            AUTOGENERATED_COMMENT=AUTOGENERATED_COMMENT,
            prefix=prefix,
            underlined_title=rst_underline(
                title, underline_char=heading_underline_char),
            instruction=instruction,
            suffix=suffix,
        ).strip() + "\n"
        return content

Response: Returns the contents of an RST file that will automatically create a new object.


Instruction:
Implement a function in Python 3 to
write the RST file to our destination RST filename.

Response:
def write_rst(self,
                  prefix: str = "",
                  suffix: str = "",
                  heading_underline_char: str = "=",
                  method: AutodocMethod = None,
                  overwrite: bool = False,
                  mock: bool = False) -> None:
        """
        Writes the RST file to our destination RST filename, making any
        necessary directories.

        Args:
            prefix: as for :func:`rst_content`
            suffix: as for :func:`rst_content`
            heading_underline_char: as for :func:`rst_content`
            method: as for :func:`rst_content`
            overwrite: overwrite the file if it exists already?
            mock: pretend to write, but don't
        """
        content = self.rst_content(
            prefix=prefix,
            suffix=suffix,
            heading_underline_char=heading_underline_char,
            method=method
        )
        write_if_allowed(self.target_rst_filename, content,
                         overwrite=overwrite, mock=mock)


Instruction:
How would you implement a function in Python 3 that
adds source files to the index.

Response:
def add_source_files(
            self,
            source_filenames_or_globs: Union[str, List[str]],
            method: AutodocMethod = None,
            recursive: bool = None,
            source_rst_title_style_python: bool = None,
            pygments_language_override: Dict[str, str] = None) -> None:
        """
        Adds source files to the index.

        Args:
            source_filenames_or_globs: string containing a filename or a
                glob, describing the file(s) to be added, or a list of such
                strings
            method: optional method to override ``self.method``
            recursive: use :func:`glob.glob` in recursive mode? (If ``None``,
                the default, uses the version from the constructor.)
            source_rst_title_style_python: optional to override
                ``self.source_rst_title_style_python``
            pygments_language_override: optional to override
                ``self.pygments_language_override``
        """
        if not source_filenames_or_globs:
            return

        if method is None:
            # Use the default
            method = self.method
        if recursive is None:
            recursive = self.recursive
        if source_rst_title_style_python is None:
            source_rst_title_style_python = self.source_rst_title_style_python
        if pygments_language_override is None:
            pygments_language_override = self.pygments_language_override

        # Get a sorted list of filenames
        final_filenames = self.get_sorted_source_files(
            source_filenames_or_globs,
            recursive=recursive
        )

        # Process that sorted list
        for source_filename in final_filenames:
            self.files_to_index.append(FileToAutodocument(
                source_filename=source_filename,
                project_root_dir=self.project_root_dir,
                python_package_root_dir=self.python_package_root_dir,
                target_rst_filename=self.specific_file_rst_filename(
                    source_filename
                ),
                method=method,
                source_rst_title_style_python=source_rst_title_style_python,
                pygments_language_override=pygments_language_override,
            ))