Instruction:
Can you generate a brief explanation for the following Python 3 code
def notify_mail(title, message, recipient=None, sender=None, smtp_host=None, smtp_port=None,
        **kwargs):
    """
    Mail notification method taking a *title* and a string *message*. *recipient*, *sender*,
    *smtp_host* and *smtp_port* default to the configuration values in the [notifications] section.
    """
    cfg = Config.instance()

    if not recipient:
        recipient = cfg.get_expanded("notifications", "mail_recipient")
    if not sender:
        sender = cfg.get_expanded("notifications", "mail_sender")
    if not smtp_host:
        smtp_host = cfg.get_expanded("notifications", "mail_smtp_host")
    if not smtp_port:
        smtp_port = cfg.get_expanded("notifications", "mail_smtp_port")

    if not recipient or not sender:
        logger.warning("cannot send mail notification, recipient ({}) or sender ({}) empty".format(
            recipient, sender))
        return False

    return send_mail(recipient, sender, title, message, smtp_host=smtp_host, smtp_port=smtp_port)

Response: Send a mail notification to the specified recipient and sender.


Instruction:
Implement a function in Python 3 to
run all patches. This function ensures that a second invocation has no effect.

Response:
def patch_all():
    """
    Runs all patches. This function ensures that a second invocation has no effect.
    """
    global _patched

    if _patched:
        return
    _patched = True

    patch_default_retcodes()
    patch_worker_run_task()
    patch_worker_factory()
    patch_keepalive_run()
    patch_cmdline_parser()

    logger.debug("applied law-specific luigi patches")


Instruction:
Can you generate the documentation for the following Python 3 function
def patch_default_retcodes():
    import luigi.retcodes

    retcode = luigi.retcodes.retcode

    retcode.already_running._default = 10
    retcode.missing_data._default = 20
    retcode.not_run._default = 30
    retcode.task_failed._default = 40
    retcode.scheduling_error._default = 50
    retcode.unhandled_exception._default = 60

Response: Patch luigi. retcodes. retcode with default luigi return codes.


Instruction:
Given the following Python 3 function, write the documentation
def patch_worker_run_task():
    _run_task = luigi.worker.Worker._run_task

    def run_task(self, task_id):
        task = self._scheduled_tasks[task_id]

        task._worker_id = self._id
        task._worker_task = self._first_task

        try:
            _run_task(self, task_id)
        finally:
            task._worker_id = None
            task._worker_task = None

        # make worker disposable when sandboxed
        if os.getenv("LAW_SANDBOX_SWITCHED") == "1":
            self._start_phasing_out()

    luigi.worker.Worker._run_task = run_task

Response:
Patches the luigi. worker. Worker. _run_task method to store the worker id and the id of its
    first task in the task.


Instruction:
Can you generate the documentation for the following Python 3 function
def patch_worker_factory():
    def create_worker(self, scheduler, worker_processes, assistant=False):
        worker = luigi.worker.Worker(scheduler=scheduler, worker_processes=worker_processes,
            assistant=assistant, worker_id=os.getenv("LAW_SANDBOX_WORKER_ID"))
        worker._first_task = os.getenv("LAW_SANDBOX_WORKER_TASK")
        return worker

    luigi.interface._WorkerSchedulerFactory.create_worker = create_worker

Response: Patches the luigi. interface. _WorkerSchedulerFactory to include sandboxing information when creating a worker instance.


Instruction:
How would you explain what the following Python 3 function does
def patch_keepalive_run():
    _run = luigi.worker.KeepAliveThread.run

    def run(self):
        # do not run the keep-alive loop when sandboxed
        if os.getenv("LAW_SANDBOX_SWITCHED") == "1":
            self.stop()
        else:
            _run(self)

    luigi.worker.KeepAliveThread.run = run

Response: Patches the luigi. worker. KeepAliveThread. run to immediately stop the keep - alive thread when running within a sandbox.


Instruction:
Can you generate the documentation for the following Python 3 function
def patch_cmdline_parser():
    # store original functions
    _init = luigi.cmdline_parser.CmdlineParser.__init__

    # patch init
    def __init__(self, cmdline_args):
        _init(self, cmdline_args)
        self.cmdline_args = cmdline_args

    luigi.cmdline_parser.CmdlineParser.__init__ = __init__

Response:
Patches the luigi. cmdline_parser. CmdlineParser to store the original command line arguments
    for later processing in the Law. config. Config.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def notify_slack(title, content, attachment_color="#4bb543", short_threshold=40, token=None,
        channel=None, mention_user=None, **kwargs):
    """
    Sends a slack notification and returns *True* on success. The communication with the slack API
    might have some delays and is therefore handled by a thread. The format of the notification
    depends on *content*. If it is a string, a simple text notification is sent. Otherwise, it
    should be a dictionary whose fields are used to build a message attachment with two-column
    formatting.
    """
    # test import
    import slackclient  # noqa: F401

    cfg = Config.instance()

    # get default token and channel
    if not token:
        token = cfg.get_expanded("notifications", "slack_token")
    if not channel:
        channel = cfg.get_expanded("notifications", "slack_channel")

    if not token or not channel:
        logger.warning("cannot send Slack notification, token ({}) or channel ({}) empty".format(
            token, channel))
        return False

    # append the user to mention to the title
    # unless explicitly set to empty string
    mention_text = ""
    if mention_user is None:
        mention_user = cfg.get_expanded("notifications", "slack_mention_user")
    if mention_user:
        mention_text = " (@{})".format(mention_user)

    # request data for the API call
    request = {
        "channel": channel,
        "as_user": True,
        "parse": "full",
    }

    # standard or attachment content?
    if isinstance(content, six.string_types):
        request["text"] = "{}{}\n\n{}".format(title, mention_text, content)
    else:
        # content is a dict, send its data as an attachment
        request["text"] = "{} {}".format(title, mention_text)
        request["attachments"] = at = {
            "color": attachment_color,
            "fields": [],
            "fallback": "{}{}\n\n".format(title, mention_text),
        }

        # fill the attachment fields and extend the fallback
        for key, value in content.items():
            at["fields"].append({
                "title": key,
                "value": value,
                "short": len(value) <= short_threshold,
            })
            at["fallback"] += "_{}_: {}\n".format(key, value)

    # extend by arbitrary kwargs
    request.update(kwargs)

    # threaded, non-blocking API communication
    thread = threading.Thread(target=_notify_slack, args=(token, request))
    thread.start()

    return True

Response: Send a Slack notification to the specified slack site.


Instruction:
Make a summary of the following Python 3 code
def maybe_wait(func):
    def wrapper(self, *args, **kwargs):
        if self.slow:
            time.sleep(random.randint(5, 15))
        return func(self, *args, **kwargs)

    return wrapper

Response: Wrapper around run method that reads the slow flag to decide whether to wait some seconds.


Instruction:
Make a summary of the following Python 3 code
def parse_query_output(cls, out):
        query_data = {}

        for line in out.strip().split("\n"):
            parts = line.split()
            if len(parts) < 6:
                continue

            job_id = parts[0]
            status_flag = parts[2]

            # map the status
            status = cls.map_status(status_flag)

            # save the result
            query_data[job_id] = cls.job_status_dict(job_id=job_id, status=status)

        return query_data

Response: Parse the output of a query from the server into a dictionary.


Instruction:
Explain what the following Python 3 code does
def setup_logging():
    global console_handler

    # make sure logging is setup only once
    if console_handler:
        return

    # set the handler of the law root logger
    console_handler = logging.StreamHandler()
    console_handler.setFormatter(LogFormatter())
    logging.getLogger("law").addHandler(console_handler)

    # set levels for all loggers
    for name, level in Config.instance().items("logging"):
        level = level.upper()
        if hasattr(logging, level):
            logger = logging.getLogger(name)
            logger.setLevel(getattr(logging, level))
            logger.debug("registered logger with level '{}'".format(level))

Response: Sets up the internal logging mechanism.


Instruction:
How would you code a function in Python 3 to
execute the config subprogram with parsed commandline args.

Response:
def execute(args):
    """
    Executes the *config* subprogram with parsed commandline *args*.
    """
    # just print the file location?
    if args.location:
        print(Config.instance().config_file)
        return

    # every option below requires the name to be set
    if not args.name:
        abort("please give the name of the config in the format <section>[.<option>]")

    # removal
    if args.remove:
        abort("config removal not yet implemented")

    # setting
    if args.value:
        abort("config setting not yet implemented")

    # getting
    print(get_config(args.name, expand=args.expand))


Instruction:
Make a summary of the following Python 3 code
def get_config(name, expand=False):
    cfg = Config.instance()
    only_section = "." not in name

    # when only the section is given, print all keys
    if only_section:
        return "\n".join(cfg.keys(name))

    else:
        section, option = name.split(".", 1)
        func = cfg.get_expanded if expand else cfg.get
        return func(section, option)

Response: Get the config value that corresponds to the given name.


Instruction:
Here you have a function in Python 3, explain what it does
def setup_parser(sub_parsers):
    parser = sub_parsers.add_parser("run", prog="law run", description="Run a task with"
        " configurable parameters. See http://luigi.rtfd.io/en/stable/running_luigi.html for more"
        " info.")

    parser.add_argument("task_family", help="a task family registered in the task database file or"
        " a module and task class in the format <module>.<class>")
    parser.add_argument("parameter", nargs="*", help="task parameters")

Response: Sets up the command line parser for the run subprogram and adds it to sub_parsers.


Instruction:
Make a summary of the following Python 3 code
def execute(args):
    task_family = None
    error = None

    # try to infer the task module from the passed task family and import it
    parts = args.task_family.rsplit(".", 1)
    if len(parts) == 2:
        modid, cls_name = parts
        try:
            mod = __import__(modid, globals(), locals(), [cls_name])
            if hasattr(mod, cls_name):
                task_cls = getattr(mod, cls_name)
                if not issubclass(task_cls, Task):
                    abort("object '{}' is not a Task".format(args.task_family))
                task_family = task_cls.task_family
        except ImportError as e:
            logger.warning("import error in module {}: {}".format(modid, e))
            error = e

    # read task info from the index file and import it
    if task_family is None:
        index_file = Config.instance().get_expanded("core", "index_file")
        if os.path.exists(index_file):
            info = read_task_from_index(args.task_family, index_file)
            if not info:
                abort("task family '{}' not found in index".format(args.task_family))
            modid, task_family, _ = info
            __import__(modid, globals(), locals())

    # complain when no task could be found
    if task_family is None:
        if error:
            raise error
        else:
            abort("task '{}' not found".format(args.task_family))

    # import the module and run luigi
    luigi_run([task_family] + sys.argv[3:])

Response: Executes luigi_run with parsed commandline args.


Instruction:
Create a Python 3 function for
reading the task information from the index file given by index_file.

Response:
def read_task_from_index(task_family, index_file=None):
    """
    Returns module id, task family and space-separated parameters in a tuple for a task given by
    *task_family* from the *index_file*. When *None*, the *index_file* refers to the default as
    defined in :py:mod:`law.config`. Returns *None* when the task could not be found.
    """
    # read task information from the index file given a task family
    if index_file is None:
        index_file = Config.instance().get_expanded("core", "index_file")

    # open and go through lines
    with open(index_file, "r") as f:
        for line in f.readlines():
            line = line.strip()
            if line.count(":") >= 2:
                modid, family, params = line.split(":", 2)
                if family == task_family:
                    return modid, family, params

    return None


Instruction:
Here you have a function in Python 3, explain what it does
def run():
    # setup the main parser and sub parsers
    parser = ArgumentParser(prog="law", description="The law command line tool.")
    sub_parsers = parser.add_subparsers(help="subcommands", dest="command")

    # add main arguments
    parser.add_argument("--version", "-V", action="version", version=law.__version__)

    # setup all progs
    mods = {}
    for prog in progs:
        mods[prog] = import_module("law.cli." + prog)
        mods[prog].setup_parser(sub_parsers)

    # parse args and dispatch execution
    if len(sys.argv) >= 2 and sys.argv[1] in forward_progs:
        args = parser.parse_args(sys.argv[1:3])
    else:
        args = parser.parse_args()

    if args.command:
        mods[args.command].execute(args)
    else:
        parser.print_help()

Response: Entry point to the law cli. Sets up all parsers parses all arguments and executes the requested subprogram.


Instruction:
Can you tell what is the following Python 3 function doing
def setup_parser(sub_parsers):
    parser = sub_parsers.add_parser("software", prog="law software", description="Create or update"
        " the law software cache ({}). This is only required for some sandboxes that need to"
        " forward software into containers.".format(get_sw_dir()))

    parser.add_argument("--remove", "-r", action="store_true", help="remove the software cache"
        " directory and exit")
    parser.add_argument("--location", "-l", action="store_true", help="print the location of the"
        " software cache directory and exit")

Response: Sets up the command line parser for the law software cache subprogram and adds it to sub_parsers.


Instruction:
Can you implement a function in Python 3 that
executes the software subprogram with parsed commandline args.

Response:
def execute(args):
    """
    Executes the *software* subprogram with parsed commandline *args*.
    """
    sw_dir = get_sw_dir()

    # just print the cache location?
    if args.location:
        print(sw_dir)
        return

    # just remove the current software cache?
    if args.remove:
        remove_software_cache(sw_dir)
        return

    # rebuild the software cache
    build_software_cache(sw_dir)


Instruction:
Can you tell what is the following Python 3 function doing
def build_software_cache(sw_dir=None):
    # ensure the cache is empty
    sw_dir = get_sw_dir(sw_dir)
    remove_software_cache(sw_dir)
    os.makedirs(sw_dir)

    # reload dependencies to find the proper module paths
    reload_dependencies(force=True)

    for mod in deps:
        path = os.path.dirname(mod.__file__)
        name, ext = os.path.splitext(os.path.basename(mod.__file__))
        # single file or module?
        if name == "__init__":
            # copy the entire module
            name = os.path.basename(path)
            shutil.copytree(path, os.path.join(sw_dir, name))
        else:
            shutil.copy2(os.path.join(path, name + ".py"), sw_dir)

Response: Builds up the software cache directory at sw_dir.


Instruction:
Create a Python 3 function to
remove the software cache directory at sw_dir.

Response:
def remove_software_cache(sw_dir=None):
    """
    Removes the software cache directory at *sw_dir* which is evaluated with :py:func:`get_sw_dir`.
    """
    sw_dir = get_sw_dir(sw_dir)
    if os.path.exists(sw_dir):
        shutil.rmtree(sw_dir)


Instruction:
Given the following Python 3 function, write the documentation
def reload_dependencies(force=False):
    global _reloaded_deps

    if _reloaded_deps and not force:
        return
    _reloaded_deps = True

    for mod in deps:
        six.moves.reload_module(mod)
        logger.debug("reloaded module '{}'".format(mod))

Response: Reloads all python modules that law depends on.


Instruction:
Here you have a function in Python 3, explain what it does
def use_software_cache(sw_dir=None, reload_deps=False):
    sw_dir = get_sw_dir(sw_dir)
    if os.path.exists(sw_dir):
        sys.path.insert(1, sw_dir)

        if reload_deps:
            reload_dependencies()

Response: Sets sys. path so that the software at sw_dir is used.


Instruction:
Can you create a Python 3 function that
returns the software directory defined in the config. software_dir config.

Response:
def get_sw_dir(sw_dir=None):
    """
    Returns the software directory defined in the ``config.software_dir`` config. When *sw_dir* is
    not *None*, it is expanded and returned instead.
    """
    if sw_dir is None:
        sw_dir = Config.instance().get("core", "software_dir")

    sw_dir = os.path.expandvars(os.path.expanduser(sw_dir))

    return sw_dir


Instruction:
Write a Python 3 script for
creating a lazy created guarded TFile class and returns it.

Response:
def GuardedTFile(*args, **kwargs):
    """
    Factory function that lazily creates the guarded TFile class, and creates and returns an
    instance with all passed *args* and *kwargs*. This is required as we do not want to import ROOT
    in the global scope.
    """
    global guarded_tfile_cls

    if not guarded_tfile_cls:
        import ROOT

        class GuardedTFile(ROOT.TFile):

            def __enter__(self):
                return self

            def __exit__(self, exc_type, exc_value, traceback):
                if self.IsOpen():
                    self.Close()

        guarded_tfile_cls = GuardedTFile

    return guarded_tfile_cls(*args, **kwargs)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def cached_workflow_property(func=None, attr=None, setter=True):
    def wrapper(func):
        _attr = attr or "_workflow_cached_" + func.__name__

        @functools.wraps(func)
        def getter(self):
            wf = self.as_workflow()
            if not hasattr(wf, _attr):
                setattr(wf, _attr, func(wf))
            return getattr(wf, _attr)

        _setter = None
        if setter:
            def _setter(self, value):
                wf = self.as_workflow()
                setattr(wf, _attr, value)

            _setter.__name__ = func.__name__

        return property(fget=getter, fset=_setter)

    return wrapper if not func else wrapper(func)

Response:
Decorator to declare a property that is stored only on a workflow and also cached for
 ArcGIS.


Instruction:
Can you write a function in Python 3 where it
returns the default workflow requirements for this task in an ordered dictionary.

Response:
def requires(self):
        """
        Returns the default workflow requirements in an ordered dictionary, which is updated with
        the return value of the task's *workflow_requires* method.
        """
        reqs = OrderedDict()
        reqs.update(self.task.workflow_requires())
        return reqs


Instruction:
How would you explain what the following Python 3 function does
def output(self):
        if self.task.target_collection_cls is not None:
            cls = self.task.target_collection_cls
        elif self.task.outputs_siblings:
            cls = SiblingFileCollection
        else:
            cls = TargetCollection

        targets = luigi.task.getpaths(self.task.get_branch_tasks())
        collection = cls(targets, threshold=self.threshold(len(targets)))

        return OrderedDict([("collection", collection)])

Response:
Returns the default workflow outputs in an ordered dictionary. At the moment this is just
        the collection of outputs of the branch tasks stored with the key collection.


Instruction:
Can you tell what is the following Python 3 function doing
def threshold(self, n=None):
        if n is None:
            n = len(self.task.branch_map())

        acceptance = self.task.acceptance
        return (acceptance * n) if acceptance <= 1 else acceptance

Response:
Returns the threshold number of tasks that need to be complete in order to consider the
        workflow as being complete itself.


Instruction:
Here you have a function in Python 3, explain what it does
def get_prefixed_config(self, section, option, **kwargs):
        cfg = Config.instance()
        default = cfg.get_expanded(section, option, **kwargs)
        return cfg.get_expanded(section, "{}_{}".format(self.workflow_type, option),
            default=default, **kwargs)

Response: Get the config value for the current object.


Instruction:
Can you tell what is the following Python 3 function doing
def getionimage(p, mz_value, tol=0.1, z=1, reduce_func=sum):
    tol = abs(tol)
    im = np.zeros((p.imzmldict["max count of pixels y"], p.imzmldict["max count of pixels x"]))
    for i, (x, y, z_) in enumerate(p.coordinates):
        if z_ == 0:
            UserWarning("z coordinate = 0 present, if you're getting blank images set getionimage(.., .., z=0)")
        if z_ == z:
            mzs, ints = map(lambda x: np.asarray(x), p.getspectrum(i))
            min_i, max_i = _bisect_spectrum(mzs, mz_value, tol)
            im[y - 1, x - 1] = reduce_func(ints[min_i:max_i+1])
    return im

Response: Get an image representation of the intensity distribution of the ion with specified m / z value.


Instruction:
Explain what the following Python 3 code does
def __readimzmlmeta(self):
        d = {}
        scan_settings_list_elem = self.root.find('%sscanSettingsList' % self.sl)
        instrument_config_list_elem = self.root.find('%sinstrumentConfigurationList' % self.sl)
        supportedparams1 = [("max count of pixels x", int), ("max count of pixels y", int),
                            ("max dimension x", int), ("max dimension y", int), ("pixel size x", float),
                            ("pixel size y", float), ("matrix solution concentration", float)]
        supportedparams2 = [("wavelength", float),
                            ("focus diameter x", float), ("focus diameter y", float), ("pulse energy", float),
                            ("pulse duration", float), ("attenuation", float)]
        supportedaccessions1 = [("IMS:1000042", "value"), ("IMS:1000043", "value"),
                                ("IMS:1000044", "value"), ("IMS:1000045", "value"),
                                ("IMS:1000046", "value"), ("IMS:1000047", "value"), ("MS:1000835", "value")]
        supportedaccessions2 = [("MS:1000843", "value"), ("MS:1000844", "value"),
                                ("MS:1000845", "value"), ("MS:1000846", "value"), ("MS:1000847", "value"),
                                ("MS:1000848", "value")]
        for i in range(len(supportedparams1)):
            acc, attr = supportedaccessions1[i]
            elem = scan_settings_list_elem.find('.//%scvParam[@accession="%s"]' % (self.sl, acc))
            if elem is None:
                break
            name, T = supportedparams1[i]
            try:
                d[name] = T(elem.attrib[attr])
            except ValueError:
                warn(Warning('Wrong data type in XML file. Skipped attribute "%s"' % name))
        for i in range(len(supportedparams2)):
            acc, attr = supportedaccessions2[i]
            elem = instrument_config_list_elem.find('.//%scvParam[@accession="%s"]' % (self.sl, acc))
            if elem is None:
                break
            name, T = supportedparams2[i]
            try:
                d[name] = T(elem.attrib[attr])
            except ValueError:
                warn(Warning('Wrong data type in XML file. Skipped attribute "%s"' % name))
        return d

Response: This method reads the meta information from the imzML file.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_physical_coordinates(self, i):
        try:
            pixel_size_x = self.imzmldict["pixel size x"]
            pixel_size_y = self.imzmldict["pixel size y"]
        except KeyError:
            raise KeyError("Could not find all pixel size attributes in imzML file")
        image_x, image_y = self.coordinates[i][:2]
        return image_x * pixel_size_x, image_y * pixel_size_y

Response: Returns the real - world coordinates of the given pixel index i.


Instruction:
Can you write a function in Python 3 where it
reads the desired spectrum from the. ibd file and returns the mz and intensity arrays corresponding to the desired spectrum.

Response:
def getspectrum(self, index):
        """
        Reads the spectrum at specified index from the .ibd file.

        :param index:
            Index of the desired spectrum in the .imzML file

        Output:

        mz_array: numpy.ndarray
            Sequence of m/z values representing the horizontal axis of the desired mass
            spectrum
        intensity_array: numpy.ndarray
            Sequence of intensity values corresponding to mz_array
        """
        mz_bytes, intensity_bytes = self.get_spectrum_as_string(index)
        mz_array = np.frombuffer(mz_bytes, dtype=self.mzPrecision)
        intensity_array = np.frombuffer(intensity_bytes, dtype=self.intensityPrecision)
        return mz_array, intensity_array


Instruction:
Write a Python 3 script to
read the mz array and intensity array of the desired spectrum at the specified location and returns the mz array and intensity array as a byte string.

Response:
def get_spectrum_as_string(self, index):
        """
        Reads m/z array and intensity array of the spectrum at specified location
        from the binary file as a byte string. The string can be unpacked by the struct
        module. To get the arrays as numbers, use getspectrum

        :param index:
            Index of the desired spectrum in the .imzML file
        :rtype: Tuple[str, str]

        Output:

        mz_string:
            string where each character represents a byte of the mz array of the
            spectrum
        intensity_string:
            string where each character represents a byte of the intensity array of
            the spectrum
        """
        offsets = [self.mzOffsets[index], self.intensityOffsets[index]]
        lengths = [self.mzLengths[index], self.intensityLengths[index]]
        lengths[0] *= self.sizeDict[self.mzPrecision]
        lengths[1] *= self.sizeDict[self.intensityPrecision]
        self.m.seek(offsets[0])
        mz_string = self.m.read(lengths[0])
        self.m.seek(offsets[1])
        intensity_string = self.m.read(lengths[1])
        return mz_string, intensity_string


Instruction:
Can you tell what is the following Python 3 function doing
def _read_mz(self, mz_offset, mz_len, mz_enc_len):
        '''reads a mz array from the currently open ibd file'''
        self.ibd.seek(mz_offset)
        data = self.ibd.read(mz_enc_len)
        self.ibd.seek(0, 2)
        data = self.mz_compression.decompress(data)
        return tuple(np.fromstring(data, dtype=self.mz_dtype))

Response: reads a mz array from the currently open ibd file


Instruction:
Write a Python 3 function for
giving an mz array return the mz_data ( disk location )

Response:
def _get_previous_mz(self, mzs):
        '''given an mz array, return the mz_data (disk location)
        if the mz array was not previously written, write to disk first'''
        mzs = tuple(mzs)  # must be hashable
        if mzs in self.lru_cache:
            return self.lru_cache[mzs]

        # mz not recognized ... check hash
        mz_hash = "%s-%s-%s" % (hash(mzs), sum(mzs), len(mzs))
        if mz_hash in self.hashes:
            for mz_data in self.hashes[mz_hash]:
                test_mz = self._read_mz(*mz_data)
                if mzs == test_mz:
                    self.lru_cache[test_mz] = mz_data
                    return mz_data
        # hash not recognized
        # must be a new mz array ... write it, add it to lru_cache and hashes
        mz_data = self._encode_and_write(mzs, self.mz_dtype, self.mz_compression)
        self.hashes[mz_hash].append(mz_data)
        self.lru_cache[mzs] = mz_data
        return mz_data


Instruction:
Write a Python 3 function for
adding a mass spectrum to the file.

Response:
def addSpectrum(self, mzs, intensities, coords, userParams=[]):
        """
        Add a mass spectrum to the file.

        :param mz:
            mz array
        :param intensities:
            intensity array
        :param coords:

            * 2-tuple of x and y position OR
            * 3-tuple of x, y, and z position

            note some applications want coords to be 1-indexed
        """
        # must be rounded now to allow comparisons to later data
        # but don't waste CPU time in continuous mode since the data will not be used anyway
        if self.mode != "continuous" or self.first_mz is None:
            mzs = self.mz_compression.rounding(mzs)
        intensities = self.intensity_compression.rounding(intensities)

        if self.mode == "continuous":
            if self.first_mz is None:
                self.first_mz = self._encode_and_write(mzs, self.mz_dtype, self.mz_compression)
            mz_data = self.first_mz
        elif self.mode == "processed":
            mz_data = self._encode_and_write(mzs, self.mz_dtype, self.mz_compression)
        elif self.mode == "auto":
            mz_data = self._get_previous_mz(mzs)
        else:
            raise TypeError("Unknown mode: %s" % self.mode)
        mz_offset, mz_len, mz_enc_len = mz_data

        int_offset, int_len, int_enc_len = self._encode_and_write(intensities, self.intensity_dtype, self.intensity_compression)
        mz_min = np.min(mzs)
        mz_max = np.max(mzs)
        ix_max = np.argmax(intensities)
        mz_base = mzs[ix_max]
        int_base = intensities[ix_max]
        int_tic = np.sum(intensities)
        s = _Spectrum(coords, mz_len, mz_offset, mz_enc_len, int_len, int_offset, int_enc_len, mz_min, mz_max, mz_base, int_base, int_tic, userParams)
        self.spectra.append(s)


Instruction:
Write a Python 3 script to
attempt to install the sdist and wheel.

Response:
def install_package(context):
    """Attempts to install the sdist and wheel."""

    if not context.dry_run and build_distributions(context):
        with util.mktmpdir() as tmp_dir:
            venv.create_venv(tmp_dir=tmp_dir)
            for distribution in Path('dist').files():
                try:
                    venv.install(distribution, tmp_dir)
                    log.info('Successfully installed %s', distribution)
                    if context.test_command and verification.run_test_command(context):
                        log.info(
                            'Successfully ran test command: %s', context.test_command
                        )
                except Exception as e:
                    raise Exception(
                        'Error installing distribution %s' % distribution, e
                    )
    else:
        log.info('Dry run, skipping installation')


Instruction:
Implement a Python 3 function for
uploading your project packages to pypi with twine with twine.

Response:
def upload_package(context):
    """Uploads your project packages to pypi with twine."""

    if not context.dry_run and build_distributions(context):
        upload_args = 'twine upload '
        upload_args += ' '.join(Path('dist').files())
        if context.pypi:
            upload_args += ' -r %s' % context.pypi

        upload_result = shell.dry_run(upload_args, context.dry_run)
        if not context.dry_run and not upload_result:
            raise Exception('Error uploading: %s' % upload_result)
        else:
            log.info(
                'Successfully uploaded %s:%s', context.module_name, context.new_version
            )
    else:
        log.info('Dry run, skipping package upload')


Instruction:
How would you implement a function in Python 3 that
attempts to install your package from pypi.

Response:
def install_from_pypi(context):
    """Attempts to install your package from pypi."""

    tmp_dir = venv.create_venv()
    install_cmd = '%s/bin/pip install %s' % (tmp_dir, context.module_name)

    package_index = 'pypi'
    if context.pypi:
        install_cmd += '-i %s' % context.pypi
        package_index = context.pypi

    try:
        result = shell.dry_run(install_cmd, context.dry_run)
        if not context.dry_run and not result:
            log.error(
                'Failed to install %s from %s', context.module_name, package_index
            )
        else:
            log.info(
                'Successfully installed %s from %s', context.module_name, package_index
            )

    except Exception as e:
        error_msg = 'Error installing %s from %s' % (context.module_name, package_index)
        log.exception(error_msg)
        raise Exception(error_msg, e)


Instruction:
Create a Python 3 function to
log the probe result and raises on failure

Response:
def report_and_raise(probe_name, probe_result, failure_msg):
    """Logs the probe result and raises on failure"""
    log.info('%s? %s' % (probe_name, probe_result))
    if not probe_result:
        raise exceptions.ProbeException(failure_msg)
    else:
        return True


Instruction:
Implement a Python 3 function for
checking if a module has metadata

Response:
def has_metadata(python_module):
    """`<module_name>/__init__.py` with `__version__` and `__url__`"""
    init_path = '{}/__init__.py'.format(python_module)
    has_metadata = (
        exists(init_path)
        and attributes.has_attribute(python_module, '__version__')
        and attributes.has_attribute(python_module, '__url__')
    )
    return report_and_raise(
        'Has module metadata',
        has_metadata,
        'Your %s/__init__.py must contain __version__ and __url__ attributes',
    )


Instruction:
Can you write a function in Python 3 where it
checks if the project meets changes requirements.

Response:
def probe_project(python_module):
    """
    Check if the project meets `changes` requirements.
    Complain and exit otherwise.
    """
    log.info('Checking project for changes requirements.')
    return (
        has_tools()
        and has_setup()
        and has_metadata(python_module)
        and has_test_runner()
        and has_readme()
        and has_changelog()
    )


Instruction:
Can you create a Python 3 function that
executes the release process.

Response:
def perform_release(context):
    """Executes the release process."""
    try:
        run_tests()

        if not context.skip_changelog:
            generate_changelog(context)

        increment_version(context)

        build_distributions(context)

        install_package(context)

        upload_package(context)

        install_from_pypi(context)

        publish(context)
    except Exception:
        log.exception('Error releasing')


Instruction:
Implement a function in Python 3 to
extract metatdata property from a module

Response:
def extract_attribute(module_name, attribute_name):
    """Extract metatdata property from a module"""
    with open('%s/__init__.py' % module_name) as input_file:
        for line in input_file:
            if line.startswith(attribute_name):
                return ast.literal_eval(line.split('=')[1].strip())


Instruction:
Here you have a function in Python 3, explain what it does
def replace_attribute(module_name, attribute_name, new_value, dry_run=True):
    init_file = '%s/__init__.py' % module_name
    _, tmp_file = tempfile.mkstemp()

    with open(init_file) as input_file:
        with open(tmp_file, 'w') as output_file:
            for line in input_file:
                if line.startswith(attribute_name):
                    line = "%s = '%s'\n" % (attribute_name, new_value)

                output_file.write(line)

    if not dry_run:
        Path(tmp_file).copy(init_file)
    else:
        log.info(diff(tmp_file, init_file, retcode=None))

Response: Update a metadata attribute in a module


Instruction:
Make a summary of the following Python 3 code
def has_attribute(module_name, attribute_name):
    init_file = '%s/__init__.py' % module_name
    return any(
        [attribute_name in init_line for init_line in open(init_file).readlines()]
    )

Response: Is this attribute present?


Instruction:
Explain what the following Python 3 code does
def choose_labels(alternatives):
    if not alternatives:
        raise ValueError

    if not isinstance(alternatives, list):
        raise TypeError

    choice_map = OrderedDict(
        ('{}'.format(i), value) for i, value in enumerate(alternatives, 1)
    )
    # prepend a termination option
    input_terminator = '0'
    choice_map.update({input_terminator: '<done>'})
    choice_map.move_to_end('0', last=False)

    choice_indexes = choice_map.keys()

    choice_lines = ['{} - {}'.format(*c) for c in choice_map.items()]
    prompt = '\n'.join(
        (
            'Select labels:',
            '\n'.join(choice_lines),
            'Choose from {}'.format(', '.join(choice_indexes)),
        )
    )

    user_choices = set()
    user_choice = None

    while not user_choice == input_terminator:
        if user_choices:
            note('Selected labels: [{}]'.format(', '.join(user_choices)))

        user_choice = click.prompt(
            prompt, type=click.Choice(choice_indexes), default=input_terminator
        )
        done = user_choice == input_terminator
        new_selection = user_choice not in user_choices
        nothing_selected = not user_choices

        if not done and new_selection:
            user_choices.add(choice_map[user_choice])

        if done and nothing_selected:
            error('Please select at least one label')
            user_choice = None

    return user_choices

Response: Prompt the user to select several labels from the provided alternatives.


Instruction:
Write a Python 3 script to
stage a release in a repository.

Response:
def stage(draft, discard, repo_directory, release_name, release_description):
    """
    Stages a release
    """
    with work_in(repo_directory):
        if discard:
            stage_command.discard(release_name, release_description)
        else:
            stage_command.stage(draft, release_name, release_description)


Instruction:
Can you generate the documentation for the following Python 3 function
def generate_changelog(context):

    changelog_content = [
        '\n## [%s](%s/compare/%s...%s)\n\n'
        % (
            context.new_version,
            context.repo_url,
            context.current_version,
            context.new_version,
        )
    ]

    git_log_content = None
    git_log = 'log --oneline --no-merges --no-color'.split(' ')
    try:
        git_log_tag = git_log + ['%s..master' % context.current_version]
        git_log_content = git(git_log_tag)
        log.debug('content: %s' % git_log_content)
    except Exception:
        log.warn('Error diffing previous version, initial release')
        git_log_content = git(git_log)

    git_log_content = replace_sha_with_commit_link(context.repo_url, git_log_content)
    # turn change log entries into markdown bullet points
    if git_log_content:
        [
            changelog_content.append('* %s\n' % line) if line else line
            for line in git_log_content[:-1]
        ]

    write_new_changelog(
        context.repo_url, 'CHANGELOG.md', changelog_content, dry_run=context.dry_run
    )
    log.info('Added content to CHANGELOG.md')
    context.changelog_content = changelog_content

Response: Generates an automatic changelog from your commit messages.


Instruction:
Can you generate the documentation for the following Python 3 function
def extract(dictionary, keys):
    return dict((k, dictionary[k]) for k in keys if k in dictionary)

Response: Extract only the specified keys from a dictionary


Instruction:
Can you generate the documentation for the following Python 3 function
def extract_arguments(arguments, long_keys, key_prefix='--'):
    long_arguments = extract(arguments, long_keys)
    return dict(
        [(key.replace(key_prefix, ''), value) for key, value in long_arguments.items()]
    )

Response: Extracts the arguments from the command line arguments dict.


Instruction:
How would you code a function in Python 3 to
tag your git repo with the new version number

Response:
def tag_and_push(context):
    """Tags your git repo with the new version number"""
    tag_option = '--annotate'
    if probe.has_signing_key(context):
        tag_option = '--sign'

    shell.dry_run(
        TAG_TEMPLATE % (tag_option, context.new_version, context.new_version),
        context.dry_run,
    )

    shell.dry_run('git push --tags', context.dry_run)


Instruction:
Can you generate the documentation for the following Python 3 function
def dry_run(command, dry_run):
    if not dry_run:
        cmd_parts = command.split(' ')
        # http://plumbum.readthedocs.org/en/latest/local_commands.html#run-and-popen
        return local[cmd_parts[0]](cmd_parts[1:])
    else:
        log.info('Dry run of %s, skipping' % command)
    return True

Response: Executes a shell command unless the dry run option is set


Instruction:
Can you generate the documentation for the following Python 3 function
def project_config():
    project_name = curdir

    config_path = Path(join(project_name, PROJECT_CONFIG_FILE))

    if not exists(config_path):
        store_settings(DEFAULTS.copy())
        return DEFAULTS

    return toml.load(io.open(config_path)) or {}

Response: Load the project configuration file.


Instruction:
Here you have a function in Python 3, explain what it does
def increment(version, major=False, minor=False, patch=True):
    version = semantic_version.Version(version)
    if major:
        version.major += 1
        version.minor = 0
        version.patch = 0
    elif minor:
        version.minor += 1
        version.patch = 0
    elif patch:
        version.patch += 1

    return str(version)

Response: Increment a semantic version of a node in the sequence.


Instruction:
Can you implement a function in Python 3 that
increments the __version__ attribute of your module s __init__.

Response:
def increment_version(context):
    """Increments the __version__ attribute of your module's __init__."""

    attributes.replace_attribute(
        context.module_name, '__version__', context.new_version, dry_run=context.dry_run
    )
    log.info(
        'Bumped version from %s to %s' % (context.current_version, context.new_version)
    )


Instruction:
Make a summary of the following Python 3 code
def main(github_token, github_api_url, progress):
    global progress_reporter_cls
    progress_reporter_cls.reportProgress = sys.stdout.isatty() and progress
    if progress_reporter_cls.reportProgress:
        progress_reporter_cls = _progress_bar
    global _github_token_cli_arg
    _github_token_cli_arg = github_token
    global _github_api_url
    _github_api_url = github_api_url

Response: A CLI to easily manage GitHub releases assets and references.


Instruction:
Can you write a function in Python 3 where it
updates the commit associated with a given release tag.

Response:
def _update_release_sha(repo_name, tag_name, new_release_sha, dry_run):
    """Update the commit associated with a given release tag.

    Since updating a tag commit is not directly possible, this function
    does the following steps:
    * set the release tag to ``<tag_name>-tmp`` and associate it
      with ``new_release_sha``.
    * delete tag ``refs/tags/<tag_name>``.
    * update the release tag to ``<tag_name>`` and associate it
      with ``new_release_sha``.
    """
    if new_release_sha is None:
        return
    refs = get_refs(repo_name, tags=True, pattern="refs/tags/%s" % tag_name)
    if not refs:
        return
    assert len(refs) == 1

    # If sha associated with "<tag_name>" is up-to-date, we are done.
    previous_release_sha = refs[0]["object"]["sha"]
    if previous_release_sha == new_release_sha:
        return

    tmp_tag_name = tag_name + "-tmp"

    # If any, remove leftover temporary tag "<tag_name>-tmp"
    refs = get_refs(repo_name, tags=True, pattern="refs/tags/%s" % tmp_tag_name)
    if refs:
        assert len(refs) == 1
        time.sleep(0.1)
        gh_ref_delete(repo_name,
                      "refs/tags/%s" % tmp_tag_name, dry_run=dry_run)

    # Update "<tag_name>" release by associating it with the "<tag_name>-tmp"
    # and "<new_release_sha>". It will create the temporary tag.
    time.sleep(0.1)
    patch_release(repo_name, tag_name,
                  tag_name=tmp_tag_name,
                  target_commitish=new_release_sha,
                  dry_run=dry_run)

    # Now "<tag_name>-tmp" references "<new_release_sha>", remove "<tag_name>"
    time.sleep(0.1)
    gh_ref_delete(repo_name, "refs/tags/%s" % tag_name, dry_run=dry_run)

    # Finally, update "<tag_name>-tmp" release by associating it with the
    # "<tag_name>" and "<new_release_sha>".
    time.sleep(0.1)
    patch_release(repo_name, tmp_tag_name,
                  tag_name=tag_name,
                  target_commitish=new_release_sha,
                  dry_run=dry_run)

    # ... and remove "<tag_name>-tmp"
    time.sleep(0.1)
    gh_ref_delete(repo_name,
                  "refs/tags/%s" % tmp_tag_name, dry_run=dry_run)


Instruction:
Given the following Python 3 function, write the documentation
def approved_funds(pronac, dt):
    funds_df = data.approved_funds_by_projects

    project = (
        funds_df
        .loc[funds_df['PRONAC'] == pronac]
    )
    project = project.to_dict('records')[0]
    info = (
        data
        .approved_funds_agg.to_dict(orient="index")
        [project['idSegmento']]
    )

    mean, std = info.values()

    outlier = gaussian_outlier.is_outlier(project['VlTotalAprovado'],
                                          mean, std)
    maximum_expected_funds = gaussian_outlier.maximum_expected_value(mean, std)

    return {
        'is_outlier': outlier,
        'total_approved_funds': project['VlTotalAprovado'],
        'maximum_expected_funds': maximum_expected_funds
    }

Response:
Verifica se o valor total de um projeto é um
    outlier em relação um cultural


Instruction:
Make a summary of the following Python 3 code
def complexidade(obj):
    indicators = obj.indicator_set.all()
    if not indicators:
        value = 0.0
    else:
        value = indicators.first().value
    return value

Response: Returns a value that indicates project health


Instruction:
Write a Python 3 script for
returning project pronac name indicators with details

Response:
def details(project):
    """
    Project detail endpoint,
    Returns project pronac, name,
    and indicators with details
    """
    indicators = project.indicator_set.all()
    indicators_detail = [(indicator_details(i)
                         for i in indicators)][0]
    if not indicators:
        indicators_detail = [
                        {'FinancialIndicator':
                            {'valor': 0.0,
                             'metrics': default_metrics, }, }]
    indicators_detail = convert_list_into_dict(indicators_detail)

    return {'pronac': project.pronac,
            'nome': project.nome,
            'indicadores': indicators_detail,
            }


Instruction:
How would you explain what the following Python 3 function does
def indicator_details(indicator):
    metrics = format_metrics_json(indicator)

    metrics_list = set(indicator.metrics
                       .filter(name__in=metrics_name_map.keys())
                       .values_list('name', flat=True))
    null_metrics = default_metrics
    for keys in metrics_list:
        null_metrics.pop(metrics_name_map[keys], None)

    metrics.update(null_metrics)

    return {type(indicator).__name__: {
            'valor': indicator.value,
            'metricas': metrics, },
            }

Response: Return a dictionary with all metrics in FinancialIndicator filled with default values


Instruction:
Implement a Python 3 function for
getting the metric for the given pronac number.

Response:
def get_metric(self, pronac, metric):
        """
        Get metric for the project with the given pronac number.

        Usage:
            >>> metrics.get_metric(pronac_id, 'finance.approved_funds')
        """

        assert isinstance(metric, str)
        assert '.' in metric, 'metric must declare a namespace'

        try:
            func = self._metrics[metric]
            return func(pronac, self._data)
        except KeyError:
            raise InvalidMetricError('metric does not exist')


Instruction:
How would you explain what the following Python 3 function does
def execute_project_models_sql_scripts(force_update=False):
    # TODO: Remove except and use ignore_conflicts
    # on bulk_create when django 2.2. is released
    with open(MODEL_FILE, "r") as file_content:
        query = file_content.read()
        db = db_connector()
        query_result = db.execute_pandas_sql_query(query)
        db.close()
        try:
            projects = Project.objects.bulk_create(
                (Project(**vals) for vals in query_result.to_dict("records")),
                # ignore_conflicts=True available on django 2.2.
            )
            indicators = [FinancialIndicator(project=p) for p in projects]
            FinancialIndicator.objects.bulk_create(indicators)
        except IntegrityError:
            # happens when there are duplicated projects
            LOG("Projects bulk_create failed, creating one by one...")
            with transaction.atomic():
                if force_update:
                    for item in query_result.to_dict("records"):
                        p, _ = Project.objects.update_or_create(**item)
                        FinancialIndicator.objects.update_or_create(project=p)
                else:

                    for item in query_result.to_dict("records"):
                        p, _ = Project.objects.get_or_create(**item)
                        FinancialIndicator.objects.update_or_create(project=p)

Response: Execute the MinC project models SQL scripts.


Instruction:
Can you generate the documentation for the following Python 3 function
def create_finance_metrics(metrics: list, pronacs: list):
    missing = missing_metrics(metrics, pronacs)
    print(f"There are {len(missing)} missing metrics!")

    processors = mp.cpu_count()
    print(f"Using {processors} processors to calculate metrics!")

    indicators_qs = FinancialIndicator.objects.filter(
        project_id__in=[p for p, _ in missing]
    )
    indicators = {i.project_id: i for i in indicators_qs}

    pool = mp.Pool(processors)
    results = [
        pool.apply_async(create_metric, args=(indicators, metric_name, pronac))
        for pronac, metric_name in missing
    ]

    calculated_metrics = [p.get() for p in results]
    if calculated_metrics:
        Metric.objects.bulk_create(calculated_metrics)
        print("Bulk completed")

        for indicator in indicators.values():
            indicator.fetch_weighted_complexity()

        print("Finished update indicators!")

    pool.close()
    print("Finished metrics calculation!")

Response: Creates metrics creating Indicator if it doesn t already exists


Instruction:
How would you explain what the following Python 3 function does
def total_receipts(pronac, dt):
    dataframe = data.planilha_comprovacao
    project = dataframe.loc[dataframe['PRONAC'] == pronac]

    segment_id = project.iloc[0]["idSegmento"]
    segments_cache = data.segment_projects_agg
    segments_cache = segments_cache.to_dict(orient="index")

    mean = segments_cache[segment_id]["mean"]
    std = segments_cache[segment_id]["<lambda>"]
    total_receipts = project.shape[0]
    is_outlier = gaussian_outlier.is_outlier(total_receipts, mean, std)
    maximum_expected = gaussian_outlier.maximum_expected_value(mean, std)
    return {
            "is_outlier": is_outlier,
            "valor": total_receipts,
            "maximo_esperado": maximum_expected,
            "minimo_esperado": 0,
        }

Response:
This metric calculates the project total of receipts
            and compare it to projects in the same segment
    and return the result


Instruction:
How would you implement a function in Python 3 that
creates project metrics for financial indicator Updates them if already exists

Response:
def load_project_metrics():
    """
    Create project metrics for financial indicator
    Updates them if already exists
    """
    all_metrics = FinancialIndicator.METRICS
    for key in all_metrics:
        df = getattr(data, key)
        pronac = 'PRONAC'
        if key == 'planilha_captacao':
            pronac = 'Pronac'
        pronacs = df[pronac].unique().tolist()
        create_finance_metrics(all_metrics[key], pronacs)


Instruction:
Given the following Python 3 function, write the documentation
def new_providers(pronac, dt):
    info = data.providers_info
    df = info[info['PRONAC'] == pronac]
    providers_count = data.providers_count.to_dict()[0]

    new_providers = []
    segment_id = None

    for _, row in df.iterrows():
        cnpj = row['nrCNPJCPF']
        cnpj_count = providers_count.get(cnpj, 0)
        segment_id = row['idSegmento']

        if cnpj_count <= 1:
            item_id = row['idPlanilhaAprovacao']
            item_name = row['Item']
            provider_name = row['nmFornecedor']

            new_provider = {
                'nome': provider_name,
                'cnpj': cnpj,
                'itens': {
                    item_id: {
                        'nome': item_name,
                        'tem_comprovante': True
                    }
                }
            }
            new_providers.append(new_provider)

    providers_amount = len(df['nrCNPJCPF'].unique())

    new_providers_amount = len(new_providers)

    new_providers_percentage = new_providers_amount / providers_amount

    averages = data.average_percentage_of_new_providers.to_dict()
    segments_average = averages['segments_average_percentage']
    all_projects_average = list(averages['all_projects_average'].values())[0]
    if new_providers:
        new_providers.sort(key=lambda provider: provider['nome'])

    return {
        'lista_de_novos_fornecedores': new_providers,
        'valor': new_providers_amount,
        'new_providers_percentage': new_providers_percentage,
        'is_outlier': new_providers_percentage > segments_average[segment_id],
        'segment_average_percentage': segments_average[segment_id],
        'all_projects_average_percentage': all_projects_average,
    }

Response:
Return the percentage of providers of a project
    that are new to the providers database.


Instruction:
How would you implement a function in Python 3 that
returns the average percentage of new providers and the average percentage of all projects.

Response:
def average_percentage_of_new_providers(providers_info, providers_count):
    """
    Return the average percentage of new providers
    per segment and the average percentage of all projects.
    """
    segments_percentages = {}
    all_projects_percentages = []
    providers_count = providers_count.to_dict()[0]

    for _, items in providers_info.groupby('PRONAC'):
        cnpj_array = items['nrCNPJCPF'].unique()
        new_providers = 0
        for cnpj in cnpj_array:
            cnpj_count = providers_count.get(cnpj, 0)
            if cnpj_count <= 1:
                new_providers += 1

        segment_id = items.iloc[0]['idSegmento']
        new_providers_percent = new_providers / cnpj_array.size
        segments_percentages.setdefault(segment_id, [])
        segments_percentages[segment_id].append(new_providers_percent)
        all_projects_percentages.append(new_providers_percent)

    segments_average_percentage = {}
    for segment_id, percentages in segments_percentages.items():
        mean = np.mean(percentages)
        segments_average_percentage[segment_id] = mean

    return pd.DataFrame.from_dict({
        'segments_average_percentage': segments_average_percentage,
        'all_projects_average': np.mean(all_projects_percentages)
    })


Instruction:
Make a summary of the following Python 3 code
def providers_count(df):
    providers_count = {}
    cnpj_array = df.values

    for a in cnpj_array:
        cnpj = a[0]
        occurrences = providers_count.get(cnpj, 0)
        providers_count[cnpj] = occurrences + 1

    return pd.DataFrame.from_dict(providers_count, orient='index')

Response:
Returns the total number of occurrences of each provider
    in the database.


Instruction:
Implement a Python 3 function for
returning CPF and CNPJ of all providers in database.

Response:
def all_providers_cnpj(df):
    """
    Return CPF/CNPJ of all providers
    in database.
    """
    cnpj_list = []

    for _, items in df.groupby('PRONAC'):
        unique_cnpjs = items['nrCNPJCPF'].unique()
        cnpj_list += list(unique_cnpjs)

    return pd.DataFrame(cnpj_list)


Instruction:
Here you have a function in Python 3, explain what it does
def get_providers_info(pronac):
    df = data.providers_info
    grouped = df.groupby('PRONAC')

    return grouped.get_group(pronac)

Response:
Return all info about providers of a
    project with the given pronac.


Instruction:
Create a Python 3 function to
aggregate mean and std with the given group.

Response:
def get_info(df, group, info=['mean', 'std']):
    """
    Aggregate mean and std with the given group.
    """
    agg = df.groupby(group).agg(info)
    agg.columns = agg.columns.droplevel(0)
    return agg


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_salic_url(item, prefix, df_values=None):
    url_keys = {
        'pronac': 'idPronac',
        'uf': 'uf',
        'product': 'produto',
        'county': 'idmunicipio',
        'item_id': 'idPlanilhaItem',
        'stage': 'etapa',
    }

    if df_values:
        values = [item[v] for v in df_values]
        url_values = dict(
            zip(url_keys.keys(), values)
        )
    else:
        url_values = {
            "pronac": item["idPronac"],
            "uf": item["UfItem"],
            "product": item["idProduto"],
            "county": item["cdCidade"],
            "item_id": item["idPlanilhaItens"],
            "stage": item["cdEtapa"],
        }

    item_data = [(value, url_values[key]) for key, value in url_keys.items()]
    url = prefix
    for k, v in item_data:
        url += f'/{str(k)}/{str(v)}'

    return url

Response: Mount a salic url for the given item.


Instruction:
Can you tell what is the following Python 3 function doing
def get_cpf_cnpj_by_pronac(pronac):
    df = data.planilha_projetos
    cpf_cnpj = None
    row_df = df[df['PRONAC'].astype(str) == str(pronac)]

    if not row_df.empty:
        cpf_cnpj = row_df.iloc[0]['CgcCpf']

    return str(cpf_cnpj)

Response:
Return the CNPF and CNPJ of the proponent
    of the project with the given pronac.


Instruction:
Here you have a function in Python 3, explain what it does
def has_receipt(item):
    pronac_id = str(item['idPronac'])
    item_id = str(item["idPlanilhaItens"])

    combined_id = f'{pronac_id}/{item_id}'

    return combined_id in data.receipt.index

Response: Verify if a item has a receipt.


Instruction:
Given the following Python 3 function, write the documentation
def get_segment_projects(segment_id):
    df = data.all_items
    return (
        df[df['idSegmento'] == str(segment_id)]
        .drop_duplicates(["PRONAC"])
        .values
    )

Response: Returns all projects from a segment.


Instruction:
Can you tell what is the following Python 3 function doing
def receipt(df):
    mutated_df = df[['IdPRONAC', 'idPlanilhaItem']].astype(str)
    mutated_df['pronac_planilha_itens'] = (
        f"{mutated_df['IdPRONAC']}/{mutated_df['idPlanilhaItem']}"
    )

    return (
        mutated_df
        .set_index(['pronac_planilha_itens'])
    )

Response: Return a dataframe to verify if a item has a receipt.


Instruction:
How would you implement a function in Python 3 that
updates local django db projects and pickles files using salic database

Response:
def update_data(ctx, models=True, pickles=False, f=False):
    """
    Updates local django db projects and pickle files using salic database from
    MinC
    Pickles are saved in /data/raw/ from sql queries in /data/scripts/
    Models are created from /data/scripts/models/
    """
    if pickles:
        save_sql_to_files(f)
    if models:
        if f:
            manage(ctx, 'create_models_from_sql --force True', env={})
        else:
            manage(ctx, 'create_models_from_sql', env={})


Instruction:
Explain what the following Python 3 code does
def update_models(ctx, f=False):
    if f:
        manage(ctx, 'create_models_from_sql --force True', env={})
    else:
        manage(ctx, 'create_models_from_sql', env={})

Response:
Updates local django db projects models using salic database from
    MinC


Instruction:
Create a Python 3 function for
creating a FinancialIndicator object for a project calculating metrics and indicator value

Response:
def create_indicator(self, project, is_valid, metrics_list):
        """
        Creates FinancialIndicator object for a project, calculating
        metrics and indicator value
        """
        project = Project.objects.get(pronac=project)
        indicator, _ = (FinancialIndicator
                        .objects.update_or_create(project=project))
        indicator.is_valid = is_valid
        if indicator.is_valid:
            p_metrics = metrics_calc.get_project(project.pronac)
            for metric_name in metrics_list:
                print("calculando a metrica  ", metric_name)
                x = getattr(p_metrics.finance, metric_name)
                print("do projeto: ", project)
                Metric.objects.create_metric(metric_name, x, indicator)
            indicator.fetch_weighted_complexity()
        return indicator


Instruction:
Can you generate a brief explanation for the following Python 3 code
def save_sql_to_files(overwrite=False):
    ext_size = len(SQL_EXTENSION)
    path = DATA_PATH / 'scripts'
    save_dir = DATA_PATH / "raw"

    for file in os.listdir(path):
        if file.endswith(SQL_EXTENSION):
            file_path = os.path.join(save_dir,
                                     file[:-ext_size] + '.' + FILE_EXTENSION)
            if not os.path.isfile(file_path) or overwrite:
                query_result = make_query(path / file)
                save_dataframe_as_pickle(query_result, file_path)
            else:
                print(("file {} already exists, if you would like to update"
                       " it, use -f flag\n").format(file_path))

Response: Saves all. sql files in the database and all raw files in the database into a single file.


Instruction:
How would you explain what the following Python 3 function does
def fetch_weighted_complexity(self, recalculate_metrics=False):
        # TODO: implment metrics recalculation
        max_total = sum(
            [self.metrics_weights[metric_name] for metric_name in self.metrics_weights]
        )
        total = 0
        if recalculate_metrics:
            self.calculate_indicator_metrics()
        for metric in self.metrics.all():
            if metric.name in self.metrics_weights and metric.is_outlier:
                total += self.metrics_weights[metric.name]

        value = total / max_total

        final_value = "{:.1f}".format(value * 10)

        if final_value[-1] == "0":
            final_value = "{:.0f}".format(value * 10)
            final_value = int(final_value)
        else:
            final_value = float(final_value)
        self.value = float(final_value)
        self.is_valid = True
        self.updated_at = datetime.datetime.now()
        self.save()
        return final_value

Response: Calculates the weighted complexity of the current entry in the database.


Instruction:
Can you tell what is the following Python 3 function doing
def item_prices(pronac, data):
    threshold = 0.1

    outlier_info = get_outliers_percentage(pronac)
    outlier_info['is_outlier'] = outlier_info['percentage'] > threshold
    outlier_info['maximum_expected'] = threshold * outlier_info['total_items']

    return outlier_info

Response: Verify if a project is an outlier compared to another project based on the price of bought items.


Instruction:
Write a Python 3 script to
verify if a item is an outlier compared to the other occurrences of the same item based on his price.

Response:
def is_outlier(df, item_id, segment_id, price):
    """
    Verify if a item is an outlier compared to the
    other occurrences of the same item, based on his price.

    Args:
        item_id: idPlanilhaItens
        segment_id: idSegmento
        price: VlUnitarioAprovado
    """

    if (segment_id, item_id) not in df.index:
        return False

    mean = df.loc[(segment_id, item_id)]['mean']
    std = df.loc[(segment_id, item_id)]['std']

    return gaussian_outlier.is_outlier(
        x=price, mean=mean, standard_deviation=std
    )


Instruction:
Given the following Python 3 function, write the documentation
def aggregated_relevant_items(raw_df):
    df = (
        raw_df[['idSegmento', 'idPlanilhaItens', 'VlUnitarioAprovado']]
        .groupby(by=['idSegmento', 'idPlanilhaItens'])
        .agg([np.mean, lambda x: np.std(x, ddof=0)])
    )
    df.columns = df.columns.droplevel(0)
    return (
        df
        .rename(columns={'<lambda>': 'std'})
    )

Response: Aggragation for calculate mean and std.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def items_with_price(raw_df):
    df = (
        raw_df
        [['PRONAC', 'idPlanilhaAprovacao', 'Item',
            'idPlanilhaItens', 'VlUnitarioAprovado',
            'idSegmento', 'DataProjeto', 'idPronac',
            'UfItem', 'idProduto', 'cdCidade', 'cdEtapa']]
    ).copy()

    df['VlUnitarioAprovado'] = df['VlUnitarioAprovado'].apply(pd.to_numeric)
    return df

Response: Dataframe with price as number.


Instruction:
Write a Python 3 script for
returning the percentage of items that are outliers.

Response:
def get_outliers_percentage(pronac):
    """
    Returns the percentage of items
    of the project that are outliers.
    """
    items = (
        data.items_with_price
        .groupby(['PRONAC'])
        .get_group(pronac)
    )

    df = data.aggregated_relevant_items

    outlier_items = {}
    url_prefix = '/prestacao-contas/analisar/comprovante'

    for _, item in items.iterrows():
        item_id = item['idPlanilhaItens']
        price = item['VlUnitarioAprovado']
        segment_id = item['idSegmento']
        item_name = item['Item']

        if is_outlier(df, item_id, segment_id, price):
            outlier_items[item_id] = {
                'name': item_name,
                'salic_url': get_salic_url(item, url_prefix),
                'has_receipt': has_receipt(item)
            }

    total_items = items.shape[0]
    outliers_amount = len(outlier_items)

    percentage = outliers_amount / total_items

    return {
        'items': outlier_items,
        'valor': outliers_amount,
        'total_items': total_items,
        'percentage': percentage,
        'is_outlier': outliers_amount > 0,
    }


Instruction:
Make a summary of the following Python 3 code
def number_of_items(pronac, dt):
    df = data.items_by_project
    project = df.loc[df['PRONAC'] == pronac]
    seg = project.iloc[0]["idSegmento"]
    info = data.items_by_project_agg.to_dict(orient="index")[seg]
    mean, std = info.values()
    threshold = mean + 1.5 * std
    project_items_count = project.shape[0]
    is_outlier = project_items_count > threshold
    return {
       'is_outlier': is_outlier,
       'valor': project_items_count,
       'maximo_esperado': mean,
       'desvio_padrao': std,
    }

Response:
This metric calculates the project number of items in a segment
WorkItem and compares it to projects in the same segment
WorkItem and returns the result


Instruction:
Write a Python 3 function that can
return the itens that are common in all the segments

Response:
def common_items(df):
    """
    Returns the itens that are common in all the segments,
    in the format | idSegmento | id planilhaItens |.
    """
    percentage = 0.1

    return (
        df
        .groupby(['idSegmento', 'idPlanilhaItens'])
        .count()
        .rename(columns={'PRONAC': 'itemOccurrences'})
        .sort_values('itemOccurrences', ascending=False)
        .reset_index(['idSegmento', 'idPlanilhaItens'])
        .groupby('idSegmento')
        .apply(lambda x: x[None: max(2, int(len(x) * percentage))])
        .reset_index(['idSegmento'], drop=True)
        .set_index(['idSegmento'])
    )


Instruction:
Can you generate the documentation for the following Python 3 function
def common_items_percentage(pronac, seg_common_items):
    if len(seg_common_items) == 0:
        return 0

    project_items = get_project_items(pronac).values[:, 0]
    project_items_amount = len(project_items)

    if project_items_amount == 0:
        return 1

    common_found_items = sum(
        seg_common_items.isin(project_items)['idPlanilhaItens']
    )

    return common_found_items / project_items_amount

Response:
Returns the percentage of items in a project that are
    common in the cultural segment.


Instruction:
Make a summary of the following Python 3 code
def common_items_metrics(all_items, common_items):
    segments = common_items.index.unique()
    metrics = {}
    for seg in segments:
        seg_common_items = segment_common_items(seg)
        projects = get_segment_projects(seg)

        metric_values = []

        for proj in projects:
            pronac = proj[0]
            percentage = common_items_percentage(pronac, seg_common_items)
            metric_values.append(percentage)

        metrics[seg] = {
            'mean': np.mean(metric_values),
            'std': np.std(metric_values)
        }
    return pd.DataFrame.from_dict(metrics, orient='index')

Response:
Calculates the percentage of common items for each project
    in each segment and calculates the mean and std of this percentage of each project
    in each segment and calculates the mean and std of this percentage of each project
    in each segment.


Instruction:
Can you tell what is the following Python 3 function doing
def get_project_items(pronac):
    df = data.all_items
    return (
        df[df['PRONAC'] == pronac]
        .drop(columns=['PRONAC', 'idSegmento'])
    )

Response: Returns all items from a project.


Instruction:
Make a summary of the following Python 3 code
def segment_common_items(segment_id):
    df = data.common_items
    return (
        df
        .loc[str(segment_id)]
        .reset_index(drop=1)
        .drop(columns=["itemOccurrences"])
    )

Response: Returns all the common items in a segment.


Instruction:
Make a summary of the following Python 3 code
def get_uncommon_items(pronac):
    segment_id = get_segment_id(str(pronac))
    seg_common_items = (
        segment_common_items(segment_id)
        .set_index('idPlanilhaItens')
        .index
    )
    project_items = (
        get_project_items(pronac)
        .set_index('idPlanilhaItens')
        .index
    )

    diff = list(project_items.difference(seg_common_items))

    return (
        data.distinct_items
        .loc[diff]
        .to_dict()['Item']
    )

Response: Return all uncommon items of a project.


Instruction:
Can you implement a function in Python 3 that
adds extra info to the uncommon items.

Response:
def add_info_to_uncommon_items(filtered_items, uncommon_items):
    """
    Add extra info to the uncommon items.
    """

    result = uncommon_items
    url_prefix = '/prestacao-contas/analisar/comprovante'

    for _, item in filtered_items.iterrows():
        item_id = item['idPlanilhaItens']
        item_name = uncommon_items[item_id]

        result[item_id] = {
            'name': item_name,
            'salic_url': get_salic_url(item, url_prefix),
            'has_recepit': has_receipt(item)
        }

    return result


Instruction:
Can you generate a brief explanation for the following Python 3 code
def common_items_ratio(pronac, dt):
    segment_id = get_segment_id(str(pronac))
    metrics = data.common_items_metrics.to_dict(orient='index')[segment_id]
    ratio = common_items_percentage(pronac, segment_common_items(segment_id))

    # constant that defines the threshold to verify if a project
    # is an outlier.
    k = 1.5

    threshold = metrics['mean'] - k * metrics['std']
    uncommon_items = get_uncommon_items(pronac)

    pronac_filter = data.all_items['PRONAC'] == pronac
    uncommon_items_filter = (
        data.all_items['idPlanilhaItens']
        .isin(uncommon_items)
    )
    items_filter = (pronac_filter & uncommon_items_filter)

    filtered_items = (
        data
        .all_items[items_filter]
        .drop_duplicates(subset='idPlanilhaItens')
    )
    uncommon_items = add_info_to_uncommon_items(filtered_items, uncommon_items)

    return {
        'is_outlier': ratio < threshold,
        'valor': ratio,
        'maximo_esperado': metrics['mean'],
        'desvio_padrao': metrics['std'],
        'items_incomuns': uncommon_items,
        'items_comuns_que_o_projeto_nao_possui': get_common_items_not_present(pronac),
    }

Response: Calculates the common items on projects in a cultural segment and calculates the uncommon items on projects in a cultural segment and verify if a project is an outlier compared to the other projects in his segment.


Instruction:
Write a Python 3 script for
returning a dictionary of the verified funds for a project.

Response:
def verified_funds(pronac, dt):
    """
    Responsable for detecting anomalies in projects total verified funds.
    """
    dataframe = data.planilha_comprovacao
    project = dataframe.loc[dataframe['PRONAC'] == pronac]
    segment_id = project.iloc[0]["idSegmento"]
    pronac_funds = project[
        ["idPlanilhaAprovacao", "PRONAC", "vlComprovacao", "idSegmento"]
    ]
    funds_grp = pronac_funds.drop(columns=["idPlanilhaAprovacao"]).groupby(
        ["PRONAC"]
    )
    project_funds = funds_grp.sum().loc[pronac]["vlComprovacao"]

    segments_info = data.verified_funds_by_segment_agg.to_dict(orient="index")
    mean = segments_info[segment_id]["mean"]
    std = segments_info[segment_id]["std"]
    is_outlier = gaussian_outlier.is_outlier(project_funds, mean, std)
    maximum_expected_funds = gaussian_outlier.maximum_expected_value(mean, std)
    return {
            "is_outlier": is_outlier,
            "valor": project_funds,
            "maximo_esperado": maximum_expected_funds,
            "minimo_esperado": 0,
    }


Instruction:
How would you implement a function in Python 3 that
raises funds organized by project.

Response:
def raised_funds_by_project(df):
    """
    Raised funds organized by project.
    """
    df['CaptacaoReal'] = df['CaptacaoReal'].apply(
        pd.to_numeric
    )
    return (
        df[['Pronac', 'CaptacaoReal']]
        .groupby(['Pronac'])
        .sum()
    )


Instruction:
Here you have a function in Python 3, explain what it does
def to_verify_funds(pronac, dt):
    project_raised_funds = data.raised_funds_by_project.loc[pronac]['CaptacaoReal']

    dataframe = data.planilha_comprovacao
    project_verified = dataframe.loc[dataframe['PRONAC'] == str(pronac)]
    if project_verified.empty:
        project_verified_funds = 0
    else:
        pronac_funds = project_verified[
            ["idPlanilhaAprovacao", "PRONAC", "vlComprovacao", "idSegmento"]
        ]
        funds_grp = pronac_funds.drop(columns=["idPlanilhaAprovacao"]).groupby(
            ["PRONAC"]
        )
        project_verified_funds = funds_grp.sum().loc[pronac]["vlComprovacao"]

    to_verify_value = project_raised_funds - float(project_verified_funds)
    is_outlier = to_verify_value != 0

    return {
        'is_outlier': is_outlier,
        'valor': to_verify_value,
        'valor_captado': project_raised_funds,
        'valor_comprovado': project_verified_funds,
        'minimo_esperado': 0,
    }

Response: Returns a dictionary of the relevant information about the project that is to be verified


Instruction:
Can you generate the documentation for the following Python 3 function
def proponent_projects(pronac, data):
    cpf_cnpj = get_cpf_cnpj_by_pronac(pronac)

    proponent_submitted_projects = {}
    proponent_analyzed_projects = {}

    if cpf_cnpj:
        submitted_projects = get_proponent_submitted_projects(cpf_cnpj)
        analyzed_projects = get_proponent_analyzed_projects(cpf_cnpj)

        try:
            proponent_submitted_projects = {
                'number_of_projects': submitted_projects['num_pronacs'],
                'pronacs_of_this_proponent': submitted_projects['pronac_list']
            }
        except KeyError:
            pass

        try:
            proponent_analyzed_projects = {
                'number_of_projects': analyzed_projects['num_pronacs'],
                'pronacs_of_this_proponent': analyzed_projects['pronac_list']
            }
        except KeyError:
            pass

    return {
        'cpf_cnpj': cpf_cnpj,
        'valor': len(proponent_submitted_projects),
        'projetos_submetidos': proponent_submitted_projects,
        'projetos_analizados': proponent_analyzed_projects,
    }

Response: Returns the CNPJ and CPF of the project containing the pronac of the project containing the pronac.


Instruction:
Create a Python 3 function to
return all projects that were analyzed.

Response:
def analyzed_projects(raw_df):
    """
    Return all projects that was analyzed.
    """
    df = raw_df[['PRONAC', 'proponenteCgcCpf']]

    analyzed_projects = df.groupby('proponenteCgcCpf')[
        'PRONAC'
    ].agg(['unique', 'nunique'])

    analyzed_projects.columns = ['pronac_list', 'num_pronacs']

    return analyzed_projects


Instruction:
Write a Python 3 function for
returning all projects submitted by the current user.

Response:
def submitted_projects(raw_df):
    """
    Return all submitted projects.
    """
    df = raw_df.astype({'PRONAC': str, 'CgcCpf': str})
    submitted_projects = df.groupby('CgcCpf')[
        'PRONAC'
    ].agg(['unique', 'nunique'])

    submitted_projects.columns = ['pronac_list', 'num_pronacs']

    return submitted_projects


Instruction:
Create a Python 3 function to
return a DataFrame from a. csv file stored in the data folder.

Response:
def read_csv(csv_name, usecols=None):
    """Returns a DataFrame from a .csv file stored in /data/raw/"""
    csv_path = os.path.join(DATA_FOLDER, csv_name)
    csv = pd.read_csv(csv_path, low_memory=False,
                      usecols=usecols, encoding="utf-8")
    return csv


Instruction:
Explain what the following Python 3 code does
def read_csv_with_different_type(csv_name, column_types_dict, usecols=None):
    csv_path = os.path.join(DATA_FOLDER, csv_name)
    csv = pd.read_csv(
        csv_path,
        usecols=usecols,
        encoding="utf-8",
        dtype=column_types_dict,
        engine="python",
    )

    for key_column, val_type in column_types_dict.items():
        if val_type == str:
            csv[key_column] = csv[key_column].str.strip()

    return csv

Response: Reads a. csv file stored in the data folder and returns a DataFrame with the values of the same type.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def read_csv_as_integer(csv_name, integer_columns, usecols=None):
    csv_path = os.path.join(DATA_FOLDER, csv_name)
    csv = pd.read_csv(csv_path, low_memory=False, usecols=usecols)
    for column in integer_columns:
        csv = csv[pd.to_numeric(csv[column], errors="coerce").notnull()]
    csv[integer_columns] = csv[integer_columns].apply(pd.to_numeric)
    return csv

Response: Returns a DataFrame from a. csv file stored in the data folder.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def check_receipts(pronac, dt):
    df = verified_repeated_receipts_for_pronac(pronac)
    comprovantes_cheque = df[df['tpFormaDePagamento'] == 1.0]

    return metric_return(comprovantes_cheque)

Response: Checks how many items in a same receipt are in a same payment type


Instruction:
Here you have a function in Python 3, explain what it does
def transfer_receipts(pronac, dt):
    df = verified_repeated_receipts_for_pronac(pronac)
    comprovantes_transferencia = df[df['tpFormaDePagamento'] == 2.0]

    return metric_return(comprovantes_transferencia)

Response: Returns the list of items that are in a same receipt when payment type is bank


Instruction:
Given the following Python 3 function, write the documentation
def money_receipts(pronac, dt):
    df = verified_repeated_receipts_for_pronac(pronac)
    comprovantes_saque = df[df['tpFormaDePagamento'] == 3.0]

    return metric_return(comprovantes_saque)

Response: Returns a metric of how many items in a same receipt are in a same order as the payment type.


Instruction:
Here you have a function in Python 3, explain what it does
def segment_raised_funds_average(df):
    grouped = df.groupby('Segmento')
    aggregated = grouped.agg(['mean', 'std'])
    aggregated.columns = aggregated.columns.droplevel(0)

    return aggregated

Response: Return some info about raised funds.


Instruction:
Explain what the following Python 3 code does
def get_outlier_info(pronac):
    df = data.planilha_captacao
    raised_funds_averages = data.segment_raised_funds_average.to_dict('index')

    segment_id = df[df['Pronac'] == pronac]['Segmento'].iloc[0]

    mean = raised_funds_averages[segment_id]['mean']
    std = raised_funds_averages[segment_id]['std']

    project_raised_funds = get_project_raised_funds(pronac)

    outlier = gaussian_outlier.is_outlier(project_raised_funds, mean, std)

    return (outlier, mean, std, project_raised_funds)

Response: Return if a project with the given pronac is an outlier based on raised funds.


Instruction:
Explain what the following Python 3 code does
def csv_to_pickle(path=ROOT / "raw", clean=False):

    for file in os.listdir(path):
        base, ext = os.path.splitext(file)
        if ext != ".csv":
            continue

        LOG(f"converting {file} to pickle")
        df = pd.read_csv(path / file, low_memory=True)
        WRITE_DF(df, path / (base + "." + FILE_EXTENSION), **WRITE_DF_OPTS)
        if clean:
            os.remove(path / file)
            LOG(f"removed {file}")

Response: Convert all CSV files in path to pickle.


Instruction:
Create a Python 3 function to
store some arbitrary dataframe in the given location.

Response:
def store(self, loc, df):
        """Store dataframe in the given location.

        Store some arbitrary dataframe:

        >>> data.store('my_data', df)

        Now recover it from the global store.
        >>> data.my_data
        ...
        """

        path = "%s.%s" % (self._root / "processed" / loc, FILE_EXTENSION)
        WRITE_DF(df, path, **WRITE_DF_OPTS)
        self._cache[loc] = df


Instruction:
Write a Python 3 function for
closing all database sessions

Response:
def close_databases(self):
        """ Close all database sessions """
        if self.gc_book:
            self.gc_book.close()
        if self.pricedb_session:
            self.pricedb_session.close()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load_stock_quantity(self, symbol: str) -> Decimal(0):
        book = self.get_gc_book()

        collection = SecuritiesAggregate(book)
        sec = collection.get_aggregate_for_symbol(symbol)
        quantity = sec.get_quantity()
        return quantity

Response: loads stock quantity for a symbol


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_gc_book(self):
        if not self.gc_book:
            gc_db = self.config.get(ConfigKeys.gnucash_book_path)
            if not gc_db:
                raise AttributeError("GnuCash book path not configured.")
            # check if this is the abs file exists
            if not os.path.isabs(gc_db):
                gc_db = resource_filename(
                    Requirement.parse("Asset-Allocation"), gc_db)
                if not os.path.exists(gc_db):
                    raise ValueError(f"Invalid GnuCash book path {gc_db}")

            self.gc_book = open_book(gc_db, open_if_lock=True)
        return self.gc_book

Response: Returns the GnuCash db session


Instruction:
Can you implement a function in Python 3 that
returns a list of all the symbols with positive balances

Response:
def get_symbols_with_positive_balances(self) -> List[str]:
        """ Identifies all the securities with positive balances """
        from gnucash_portfolio import BookAggregate

        holdings = []

        with BookAggregate() as book:
            # query = book.securities.query.filter(Commodity.)
            holding_entities = book.securities.get_all()
            for item in holding_entities:
                # Check holding balance
                agg = book.securities.get_aggregate(item)
                balance = agg.get_num_shares()
                if balance > Decimal(0):
                    holdings.append(f"{item.namespace}:{item.mnemonic}")
                else:
                    self.logger.debug(f"0 balance for {item}")
            # holdings = map(lambda x: , holding_entities)

        return holdings


Instruction:
How would you implement a function in Python 3 that
provides initialization and access to module - level session

Response:
def __get_pricedb_session(self):
        """ Provides initialization and access to module-level session """
        from pricedb import dal

        if not self.pricedb_session:
            self.pricedb_session = dal.get_default_session()
        return self.pricedb_session


Instruction:
Given the following Python 3 function, write the documentation
def add(assetclass: int, symbol: str):
    assert isinstance(symbol, str)
    assert isinstance(assetclass, int)

    symbol = symbol.upper()
    app = AppAggregate()
    new_item = app.add_stock_to_class(assetclass, symbol)
    print(f"Record added: {new_item}.")

Response: Add a stock to an asset class


Instruction:
Write a Python 3 function for
returning the view - friendly output of the aa model

Response:
def format(self, model: AssetAllocationModel, full: bool = False):
        """ Returns the view-friendly output of the aa model """
        self.full = full

        # Header
        output = f"Asset Allocation model, total: {model.currency} {model.total_amount:,.2f}\n"

        # Column Headers
        for column in self.columns:
            name = column['name']
            if not self.full and name == "loc.cur.":
                # Skip local currency if not displaying stocks.
                continue
            width = column["width"]
            output += f"{name:^{width}}"
        output += "\n"
        output += f"-------------------------------------------------------------------------------\n"

        # Asset classes

        view_model = ModelMapper(model).map_to_linear(self.full)
        for row in view_model:
            output += self.__format_row(row) + "\n"

        return output


Instruction:
Here you have a function in Python 3, explain what it does
def __format_row(self, row: AssetAllocationViewModel):
        output = ""
        index = 0

        # Name
        value = row.name
        # Indent according to depth.
        for _ in range(0, row.depth):
            value = f"   {value}"
        output += self.append_text_column(value, index)

        # Set Allocation
        value = ""
        index += 1
        if row.set_allocation > 0:
            value = f"{row.set_allocation:.2f}"
        output += self.append_num_column(value, index)

        # Current Allocation
        value = ""
        index += 1
        if row.curr_allocation > Decimal(0):
            value = f"{row.curr_allocation:.2f}"
        output += self.append_num_column(value, index)

        # Allocation difference, percentage
        value = ""
        index += 1
        if row.alloc_diff_perc.copy_abs() > Decimal(0):
            value = f"{row.alloc_diff_perc:.0f} %"
        output += self.append_num_column(value, index)

        # Allocated value
        index += 1
        value = ""
        if row.set_value:
            value = f"{row.set_value:,.0f}"
        output += self.append_num_column(value, index)

        # Current Value
        index += 1
        value = f"{row.curr_value:,.0f}"
        output += self.append_num_column(value, index)

        # Value in security's currency. Show only if displaying full model, with stocks.
        index += 1
        if self.full:
            value = ""
            if row.curr_value_own_currency:
                value = f"({row.curr_value_own_currency:,.0f}"
                value += f" {row.own_currency}"
                value += ")"
            output += self.append_num_column(value, index)

        # https://en.wikipedia.org/wiki/ANSI_escape_code
        # CSI="\x1B["
        # red = 31, green = 32
        # output += CSI+"31;40m" + "Colored Text" + CSI + "0m"

        # Value diff
        index += 1
        value = ""
        if row.diff_value:
            value = f"{row.diff_value:,.0f}"
            # Color the output
            # value = f"{CSI};40m{value}{CSI};40m"
        output += self.append_num_column(value, index)

        return output

Response: Format one row of Asset Class record


Instruction:
How would you implement a function in Python 3 that
adds value to the output row based on index

Response:
def append_num_column(self, text: str, index: int):
        """ Add value to the output row, width based on index """
        width = self.columns[index]["width"]
        return f"{text:>{width}}"


Instruction:
Given the following Python 3 function, write the documentation
def append_text_column(self, text: str, index: int):
        width = self.columns[index]["width"]
        return f"{text:<{width}}"

Response: Append text to the output row based on index


Instruction:
Write a Python 3 function that can
map data from entity to object

Response:
def map_entity(self, entity: dal.AssetClass):
        """ maps data from entity -> object """
        obj = model.AssetClass()

        obj.id = entity.id
        obj.parent_id = entity.parentid
        obj.name = entity.name
        obj.allocation = entity.allocation
        obj.sort_order = entity.sortorder
        #entity.stock_links
        #entity.diff_adjustment
        if entity.parentid == None:
            obj.depth = 0

        return obj


Instruction:
Can you implement a function in Python 3 that
maps the tree to a linear representation suitable for display

Response:
def map_to_linear(self, with_stocks: bool=False):
        """ Maps the tree to a linear representation suitable for display """
        result = []

        for ac in self.model.classes:
            rows = self.__get_ac_tree(ac, with_stocks)
            result += rows
        return result


Instruction:
Write a Python 3 script to
format the ac tree - entity with child elements

Response:
def __get_ac_tree(self, ac: model.AssetClass, with_stocks: bool):
        """ formats the ac tree - entity with child elements """
        output = []
        output.append(self.__get_ac_row(ac))

        for child in ac.classes:
            output += self.__get_ac_tree(child, with_stocks)

        if with_stocks:
            for stock in ac.stocks:
                row = None
                if isinstance(stock, Stock):
                    row = self.__get_stock_row(stock, ac.depth + 1)
                elif isinstance(stock, CashBalance):
                    row = self.__get_cash_row(stock, ac.depth + 1)
                output.append(row)

        return output


Instruction:
Can you create a Python 3 function that
returns the Asset Class record for the given Asset Class

Response:
def __get_ac_row(self, ac: model.AssetClass) -> AssetAllocationViewModel:
        """ Formats one Asset Class record """
        view_model = AssetAllocationViewModel()

        view_model.depth = ac.depth

        # Name
        view_model.name = ac.name

        view_model.set_allocation = ac.allocation
        view_model.curr_allocation = ac.curr_alloc
        view_model.diff_allocation = ac.alloc_diff
        view_model.alloc_diff_perc = ac.alloc_diff_perc

        # value
        view_model.curr_value = ac.curr_value
        # expected value
        view_model.set_value = ac.alloc_value
        # diff
        view_model.diff_value = ac.value_diff

        return view_model


Instruction:
Create a Python 3 function for
adding a stock link to an asset class

Response:
def add_stock_to_class(self, assetclass_id: int, symbol: str):
        """ Add a stock link to an asset class """
        assert isinstance(symbol, str)
        assert isinstance(assetclass_id, int)

        item = AssetClassStock()
        item.assetclassid = assetclass_id
        item.symbol = symbol

        session = self.open_session()
        session.add(item)
        self.save()

        return item


Instruction:
Can you generate the documentation for the following Python 3 function
def delete(self, id: int):
        assert isinstance(id, int)

        self.open_session()
        to_delete = self.get(id)
        self.session.delete(to_delete)
        self.save()

Response: Delete an asset class by id


Instruction:
Can you generate the documentation for the following Python 3 function
def find_unallocated_holdings(self):
        # Get linked securities
        session = self.open_session()
        linked_entities = session.query(AssetClassStock).all()
        linked = []
        # linked = map(lambda x: f"{x.symbol}", linked_entities)
        for item in linked_entities:
            linked.append(item.symbol)

        # Get all securities with balance > 0.
        from .stocks import StocksInfo

        stocks = StocksInfo()
        stocks.logger = self.logger
        holdings = stocks.get_symbols_with_positive_balances()

        # Find those which are not included in the stock links.
        non_alloc = []
        index = -1
        for item in holdings:
            try:
                index = linked.index(item)
                self.logger.debug(index)
            except ValueError:
                non_alloc.append(item)

        return non_alloc

Response: Finds unallocated holdings in the stock links.


Instruction:
Can you tell what is the following Python 3 function doing
def get(self, id: int) -> AssetClass:
        self.open_session()
        item = self.session.query(AssetClass).filter(
            AssetClass.id == id).first()
        return item

Response: Loads Asset Class by ID


Instruction:
Implement a function in Python 3 to
open a db session and returns it

Response:
def open_session(self):
        """ Opens a db session and returns it """
        from .dal import get_session

        cfg = Config()
        cfg.logger = self.logger
        db_path = cfg.get(ConfigKeys.asset_allocation_database_path)

        self.session = get_session(db_path)
        return self.session


Instruction:
Can you write a function in Python 3 where it
creates and populates the Asset Allocation model.

Response:
def get_asset_allocation(self):
        """ Creates and populates the Asset Allocation model. The main function of the app. """
        # load from db
        # TODO set the base currency
        base_currency = "EUR"

        loader = AssetAllocationLoader(base_currency=base_currency)
        loader.logger = self.logger
        model = loader.load_tree_from_db()

        model.validate()

        # securities
        # read stock links
        loader.load_stock_links()
        # read stock quantities from GnuCash
        loader.load_stock_quantity()
        # Load cash balances
        loader.load_cash_balances()
        # loader.session
        # read prices from Prices database
        loader.load_stock_prices()
        # recalculate stock values into base currency
        loader.recalculate_stock_values_into_base()
        # calculate
        model.calculate_current_value()
        model.calculate_set_values()
        model.calculate_current_allocation()

        # return the model for display
        return model


Instruction:
Make a summary of the following Python 3 code
def get_asset_classes_for_security(self, namespace: str, symbol: str) -> List[AssetClass]:
        full_symbol = symbol
        if namespace:
            full_symbol = f"{namespace}:{symbol}"

        result = (
            self.session.query(AssetClassStock)
            .filter(AssetClassStock.symbol == full_symbol)
            .all()
        )
        return result

Response: Get all asset classes for the security symbol.


Instruction:
Write a Python 3 function for
exporting all used symbols to a text file.

Response:
def export_symbols(self):
        """ Exports all used symbols """
        session = self.open_session()
        links = session.query(AssetClassStock).order_by(
            AssetClassStock.symbol).all()
        output = []
        for link in links:
            output.append(link.symbol + '\n')

        # Save output to a text file.
        with open("symbols.txt", mode='w') as file:
            file.writelines(output)

        print("Symbols exported to symbols.txt")


Instruction:
Explain what the following Python 3 code does
def __read_config(self, file_path: str):
        if not os.path.exists(file_path):
            raise FileNotFoundError("File path not found: %s", file_path)
        # check if file exists
        if not os.path.isfile(file_path):
            log(ERROR, "file not found: %s", file_path)
            raise FileNotFoundError("configuration file not found %s", file_path)

        self.config.read(file_path)

Response: Read the config file