Instruction:
Implement a Python 3 function for
converting top results from mash screen txt output to json format Parameters ---------- mash_output: str this is a string that stores the path to this file, i.e, the name of the file sample_id: str sample name

Response:
def main(mash_output, sample_id):
    '''
    converts top results from mash screen txt output to json format

    Parameters
    ----------
    mash_output: str
        this is a string that stores the path to this file, i.e, the name of
        the file
    sample_id: str
        sample name

    '''
    logger.info("Reading file : {}".format(mash_output))
    read_mash_output = open(mash_output)

    dic = {}
    median_list = []
    filtered_dic = {}

    logger.info("Generating dictionary and list to pre-process the final json")
    for line in read_mash_output:
        tab_split = line.split("\t")
        identity = tab_split[0]
        # shared_hashes = tab_split[1]
        median_multiplicity = tab_split[2]
        # p_value = tab_split[3]
        query_id = tab_split[4]
        # query-comment should not exist here and it is irrelevant

        # here identity is what in fact interests to report to json but
        # median_multiplicity also is important since it gives an rough
        # estimation of the coverage depth for each plasmid.
        # Plasmids should have higher coverage depth due to their increased
        # copy number in relation to the chromosome.
        dic[query_id] = [identity, median_multiplicity]
        median_list.append(float(median_multiplicity))

    output_json = open(" ".join(mash_output.split(".")[:-1]) + ".json", "w")

    # median cutoff is twice the median of all median_multiplicity values
    # reported by mash screen. In the case of plasmids, since the database
    # has 9k entries and reads shouldn't have that many sequences it seems ok...
    if len(median_list) > 0:
        # this statement assures that median_list has indeed any entries
        median_cutoff = median(median_list)
        logger.info("Generating final json to dump to a file")
        for k, v in dic.items():
            # estimated copy number
            copy_number = int(float(v[1]) / median_cutoff)
            # assure that plasmid as at least twice the median coverage depth
            if float(v[1]) > median_cutoff:
                filtered_dic["_".join(k.split("_")[0:3])] = [
                    round(float(v[0]),2),
                    copy_number
                ]
        logger.info(
            "Exported dictionary has {} entries".format(len(filtered_dic)))
    else:
        # if no entries were found raise an error
        logger.error("No matches were found using mash screen for the queried reads")

    output_json.write(json.dumps(filtered_dic))
    output_json.close()

    json_dic = {
        "tableRow": [{
            "sample": sample_id,
            "data": [{
                "header": "Mash Screen",
                "table": "plasmids",
                "patlas_mashscreen": filtered_dic,
                "value": len(filtered_dic)
            }]
        }],
    }

    with open(".report.json", "w") as json_report:
        json_report.write(json.dumps(json_dic, separators=(",", ":")))


Instruction:
Explain what the following Python 3 code does
def procs_dict_parser(procs_dict):

    logger.info(colored_print(
        "\n===== L I S T   O F   P R O C E S S E S =====\n", "green_bold"))

    #Sort to print alphabetically ordered list of processes to ease reading
    procs_dict_ordered = {k: procs_dict[k] for k in sorted(procs_dict)}

    for template, dict_proc_info in procs_dict_ordered.items():
        template_str = "=> {}".format(template)
        logger.info(colored_print(template_str, "blue_bold"))

        for info in dict_proc_info:
            info_str = "{}:".format(info)

            if isinstance(dict_proc_info[info], list):
                if not dict_proc_info[info]:
                    arg_msg = "None"
                else:
                    arg_msg = ", ".join(dict_proc_info[info])
            elif info == "directives":
                # this is used for the "directives", which is a dict
                if not dict_proc_info[info]:
                    # if dict is empty then add None to the message
                    arg_msg = "None"
                else:
                    # otherwise fetch all template names within a component
                    # and all the directives for each template to a list
                    list_msg = ["\n      {}: {}".format(
                        templt,
                        " , ".join(["{}: {}".format(dr, val)
                                    for dr, val in drs.items()]))
                                for templt, drs in dict_proc_info[info].items()
                    ]
                    # write list to a str
                    arg_msg = "".join(list_msg)
            else:
                arg_msg = dict_proc_info[info]

            logger.info("   {} {}".format(
                colored_print(info_str, "white_underline"), arg_msg
            ))

Response: This function handles the dictionary of attributes of each Process class and prints to stdout lists of all the components that are used by the - t flag.


Instruction:
How would you implement a function in Python 3 that
functions that collects all processes currently available and stores a dictionary of process class names and their arguments.

Response:
def proc_collector(process_map, args, pipeline_string):
    """
    Function that collects all processes available and stores a dictionary of
    the required arguments of each process class to be passed to
    procs_dict_parser

    Parameters
    ----------
    process_map: dict
        The dictionary with the Processes currently available in flowcraft
        and their corresponding classes as values
    args: argparse.Namespace
        The arguments passed through argparser that will be access to check the
        type of list to be printed
    pipeline_string: str
        the pipeline string

    """

    arguments_list = []

    # prints a detailed list of the process class arguments
    if args.detailed_list:
        # list of attributes to be passed to proc_collector
        arguments_list += [
            "input_type",
            "output_type",
            "description",
            "dependencies",
            "conflicts",
            "directives"
        ]

    # prints a short list with each process and the corresponding description
    if args.short_list:
        arguments_list += [
            "description"
        ]

    if arguments_list:
        # dict to store only the required entries
        procs_dict = {}
        # loops between all process_map Processes
        for name, cls in process_map.items():

            # instantiates each Process class
            cls_inst = cls(template=name)

            # checks if recipe is provided
            if pipeline_string:
                if name not in pipeline_string:
                    continue

            d = {arg_key: vars(cls_inst)[arg_key] for arg_key in
                 vars(cls_inst) if arg_key in arguments_list}
            procs_dict[name] = d

        procs_dict_parser(procs_dict)

        sys.exit(0)


Instruction:
Given the following Python 3 function, write the documentation
def guess_file_compression(file_path, magic_dict=None):

    if not magic_dict:
        magic_dict = MAGIC_DICT

    max_len = max(len(x) for x in magic_dict)

    with open(file_path, "rb") as f:
        file_start = f.read(max_len)

    logger.debug("Binary signature start: {}".format(file_start))

    for magic, file_type in magic_dict.items():
        if file_start.startswith(magic):
            return file_type

    return None

Response: Guesses the compression of a given file.


Instruction:
Can you tell what is the following Python 3 function doing
def get_qual_range(qual_str):

    vals = [ord(c) for c in qual_str]

    return min(vals), max(vals)

Response: Returns the range of Unicode encode range for a given string of characters.


Instruction:
Can you implement a function in Python 3 that
returns the list of all possible encodings for a given encoding range.

Response:
def get_encodings_in_range(rmin, rmax):
    """ Returns the valid encodings for a given encoding range.

    The encoding ranges are stored in the :py:data:`RANGES` dictionary, with
    the encoding name as a string and a list as a value containing the
    phred score and a tuple with the encoding range. For a given encoding
    range provided via the two first arguments, this function will return
    all possible encodings and phred scores.

    Parameters
    ----------
    rmin : int
        Minimum Unicode code in range.
    rmax : int
        Maximum Unicode code in range.

    Returns
    -------
    valid_encodings : list
        List of all possible encodings for the provided range.
    valid_phred : list
        List of all possible phred scores.

    """

    valid_encodings = []
    valid_phred = []

    for encoding, (phred, (emin, emax)) in RANGES.items():
        if rmin >= emin and rmax <= emax:
            valid_encodings.append(encoding)
            valid_phred.append(phred)

    return valid_encodings, valid_phred


Instruction:
Write a Python 3 function for
parsing a TSV file containing coverage information into objects.

Response:
def parse_coverage_table(coverage_file):
    """Parses a file with coverage information into objects.

    This function parses a TSV file containing coverage results for
    all contigs in a given assembly and will build an ``OrderedDict``
    with the information about their coverage and length.  The length
    information is actually gathered from the contig header using a
    regular expression that assumes the usual header produced by Spades::

        contig_len = int(re.search("length_(.+?)_", line).group(1))

    Parameters
    ----------
    coverage_file : str
        Path to TSV file containing the coverage results.

    Returns
    -------
    coverage_dict : OrderedDict
        Contains the coverage and length information for each contig.
    total_size : int
        Total size of the assembly in base pairs.
    total_cov : int
        Sum of coverage values across all contigs.
    """

    # Stores the correspondence between a contig and the corresponding coverage
    # e.g.: {"contig_1": {"cov": 424} }
    coverage_dict = OrderedDict()
    # Stores the total coverage
    total_cov = 0

    with open(coverage_file) as fh:
        for line in fh:
            # Get contig and coverage
            contig, cov = line.strip().split()
            coverage_dict[contig] = {"cov": int(cov)}
            # Add total coverage
            total_cov += int(cov)
            logger.debug("Processing contig '{}' with coverage '{}'"
                         "".format(contig, cov))

    return coverage_dict, total_cov


Instruction:
Write a Python 3 function that can
generate a filtered assembly file. This function generates a filtered assembly file based on an original assembly and a minimum coverage threshold. Parameters ---------- assembly_file : str Path to original assembly file. minimum_coverage : int or float Minimum coverage required for a contig to pass the filter. coverage_info : OrderedDict or dict Dictionary containing the coverage information for each contig. output_file : str Path where the filtered assembly file will be generated.

Response:
def filter_assembly(assembly_file, minimum_coverage, coverage_info,
                    output_file):
    """Generates a filtered assembly file.

    This function generates a filtered assembly file based on an original
    assembly and a minimum coverage threshold.

    Parameters
    ----------
    assembly_file : str
        Path to original assembly file.
    minimum_coverage : int or float
        Minimum coverage required for a contig to pass the filter.
    coverage_info : OrderedDict or dict
        Dictionary containing the coverage information for each contig.
    output_file : str
        Path where the filtered assembly file will be generated.

    """

    # This flag will determine whether sequence data should be written or
    # ignored because the current contig did not pass the minimum
    # coverage threshold
    write_flag = False

    with open(assembly_file) as fh, open(output_file, "w") as out_fh:

        for line in fh:
            if line.startswith(">"):
                # Reset write_flag
                write_flag = False
                # Get header of contig
                header = line.strip()[1:]
                # Check coverage for current contig
                contig_cov = coverage_info[header]["cov"]
                # If the contig coverage is above the threshold, write to
                # output filtered assembly
                if contig_cov >= minimum_coverage:
                    write_flag = True
                    out_fh.write(line)

            elif write_flag:
                out_fh.write(line)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def filter_bam(coverage_info, bam_file, min_coverage, output_bam):

    # Get list of contigs that will be kept
    contig_list = [x for x, vals in coverage_info.items()
                   if vals["cov"] >= min_coverage]

    cli = [
        "samtools",
        "view",
        "-bh",
        "-F",
        "4",
        "-o",
        output_bam,
        "-@",
        "1",
        bam_file,
    ]

    cli += contig_list

    logger.debug("Runnig samtools view subprocess with command: {}".format(
        cli))

    p = subprocess.Popen(cli, stdout=PIPE, stderr=PIPE)
    stdout, stderr = p.communicate()

    # Attempt to decode STDERR output from bytes. If unsuccessful, coerce to
    # string
    try:
        stderr = stderr.decode("utf8")
        stdout = stdout.decode("utf8")
    except (UnicodeDecodeError, AttributeError):
        stderr = str(stderr)
        stdout = str(stdout)

    logger.info("Finished samtools view subprocess with STDOUT:\\n"
                "======================================\\n{}".format(stdout))
    logger.info("Fished samtools view subprocesswith STDERR:\\n"
                "======================================\\n{}".format(stderr))
    logger.info("Finished samtools view with return code: {}".format(
        p.returncode))

    if not p.returncode:
        # Create index
        cli = [
            "samtools",
            "index",
            output_bam
        ]

        logger.debug("Runnig samtools index subprocess with command: "
                     "{}".format(cli))

        p = subprocess.Popen(cli, stdout=PIPE, stderr=PIPE)
        stdout, stderr = p.communicate()

        try:
            stderr = stderr.decode("utf8")
            stdout = stdout.decode("utf8")
        except (UnicodeDecodeError, AttributeError):
            stderr = str(stderr)
            stdout = str(stdout)

        logger.info("Finished samtools index subprocess with STDOUT:\\n"
                    "======================================\\n{}".format(
            stdout))
        logger.info("Fished samtools index subprocesswith STDERR:\\n"
                    "======================================\\n{}".format(
            stderr))
        logger.info("Finished samtools index with return code: {}".format(
            p.returncode))

Response: This function uses Samtools to filter a BAM file according to minimum coverage.


Instruction:
Can you tell what is the following Python 3 function doing
def check_filtered_assembly(coverage_info, coverage_bp, minimum_coverage,
                            genome_size, contig_size, max_contigs,
                            sample_id):
    """Checks whether a filtered assembly passes a size threshold

    Given a minimum coverage threshold, this function evaluates whether an
    assembly will pass the minimum threshold of ``genome_size * 1e6 * 0.8``,
    which means 80% of the expected genome size or the maximum threshold
    of ``genome_size * 1e6 * 1.5``, which means 150% of the expected genome
    size. It will issue a warning if any of these thresholds is crossed.
    In the case of an expected genome size below 80% it will return False.

    Parameters
    ----------
    coverage_info : OrderedDict or dict
        Dictionary containing the coverage information for each contig.
    coverage_bp : dict
        Dictionary containing the per base coverage information for each
        contig. Used to determine the total number of base pairs in the
        final assembly.
    minimum_coverage : int
        Minimum coverage required for a contig to pass the filter.
    genome_size : int
        Expected genome size.
    contig_size : dict
        Dictionary with the len of each contig. Contig headers as keys and
        the corresponding lenght as values.
    max_contigs : int
        Maximum threshold for contig number. A warning is issued if this
        threshold is crossed.
    sample_id : str
        Id or name of the current sample

    Returns
    -------
    x : bool
        True if the filtered assembly size is higher than 80% of the
        expected genome size.

    """

    # Get size of assembly after filtering contigs below minimum_coverage
    assembly_len = sum([v for k, v in contig_size.items()
                        if coverage_info[k]["cov"] >= minimum_coverage])
    logger.debug("Assembly length after filtering for minimum coverage of"
                 " {}: {}".format(minimum_coverage, assembly_len))
    # Get number of contigs after filtering
    ncontigs = len([x for x in coverage_info.values()
                    if x["cov"] >= minimum_coverage])
    logger.debug("Number of contigs: {}".format(ncontigs))
    # Get number of bp after filtering
    filtered_contigs = [k for k, v in coverage_info.items()
                        if v["cov"] >= minimum_coverage]
    logger.debug("Filtered contigs for minimum coverage of "
                 "{}: {}".format(minimum_coverage, filtered_contigs))
    total_assembled_bp = sum([sum(coverage_bp[x]) for x in filtered_contigs
                              if x in coverage_bp])
    logger.debug("Total number of assembled base pairs:"
                 "{}".format(total_assembled_bp))

    warnings = []
    fails = []
    health = True

    with open(".warnings", "w") as warn_fh, \
            open(".report.json", "w") as json_report:

        logger.debug("Checking assembly size after filtering : {}".format(
            assembly_len))

        # If the filtered assembly size is above the 150% genome size
        # threshold, issue a warning
        if assembly_len > genome_size * 1e6 * 1.5:
            warn_msg = "Assembly size ({}) smaller than the maximum" \
                       " threshold of 150% of expected genome size.".format(
                            assembly_len)
            logger.warning(warn_msg)
            warn_fh.write(warn_msg)
            fails.append("Large_genome_size_({})".format(assembly_len))

        # If the number of contigs in the filtered assembly size crosses the
        # max_contigs threshold, issue a warning
        logger.debug("Checking number of contigs: {}".format(
                len(coverage_info)))
        contig_threshold = max_contigs * genome_size / 1.5
        if ncontigs > contig_threshold:
            warn_msg = "The number of contigs ({}) exceeds the threshold of " \
                       "100 contigs per 1.5Mb ({})".format(
                            ncontigs, round(contig_threshold, 1))
            logger.warning(warn_msg)
            warn_fh.write(warn_msg)
            warnings.append(warn_msg)

        # If the filtered assembly size falls below the 80% genome size
        # threshold, fail this check and return False
        if assembly_len < genome_size * 1e6 * 0.8:
            warn_msg = "Assembly size smaller than the minimum" \
                       " threshold of 80% of expected genome size: {}".format(
                            assembly_len)
            logger.warning(warn_msg)
            warn_fh.write(warn_msg)
            fails.append("Small_genome_size_({})".format(assembly_len))
            assembly_len = sum([v for v in contig_size.values()])
            total_assembled_bp = sum(
                [sum(coverage_bp[x]) for x in coverage_info if x in
                 coverage_bp])
            logger.debug("Assembly length without coverage filtering: "
                         "{}".format(assembly_len))
            logger.debug("Total number of assembled base pairs without"
                         " filtering: {}".format(total_assembled_bp))

            health = False

        json_dic = {
            "plotData": [{
                "sample": sample_id,
                "data": {
                    "sparkline": total_assembled_bp
                }
            }]
        }

        if warnings:
            json_dic["warnings"] = [{
                "sample": sample_id,
                "table": "assembly",
                "value": warnings
            }]
        if fails:
            json_dic["fail"] = [{
                "sample": sample_id,
                "table": "assembly",
                "value": [fails]
            }]

        json_report.write(json.dumps(json_dic, separators=(",", ":")))

    return health

Response: Checks whether a filtered assembly passes a minimum coverage threshold Given a minimum coverage threshold this function evaluates whether the filtered assembly passes the minimum coverage threshold and returns True if the filtered assembly passes False otherwise.


Instruction:
Given the following Python 3 function, write the documentation
def evaluate_min_coverage(coverage_opt, assembly_coverage, assembly_size):

    if coverage_opt == "auto":
        # Get the 1/3 value of the current assembly coverage
        min_coverage = (assembly_coverage / assembly_size) * .3
        logger.info("Minimum assembly coverage automatically set to: "
                    "{}".format(min_coverage))
        # If the 1/3 coverage is lower than 10, change it to the minimum of
        # 10
        if min_coverage < 10:
            logger.info("Minimum assembly coverage cannot be set to lower"
                        " that 10. Setting to 10")
            min_coverage = 10
    else:
        min_coverage = int(coverage_opt)
        logger.info("Minimum assembly coverage manually set to: {}".format(
            min_coverage))

    return min_coverage

Response:
Evaluates the minimum coverage threshold from the value provided in
   .


Instruction:
Create a Python 3 function to
return the number of nucleotides and the size per contig for the entire file.

Response:
def get_assembly_size(assembly_file):
    """Returns the number of nucleotides and the size per contig for the
    provided assembly file path

    Parameters
    ----------
    assembly_file : str
        Path to assembly file.

    Returns
    -------
    assembly_size : int
        Size of the assembly in nucleotides
    contig_size : dict
        Length of each contig (contig name as key and length as value)

    """

    assembly_size = 0
    contig_size = {}
    header = ""

    with open(assembly_file) as fh:
        for line in fh:

            # Skip empty lines
            if line.strip() == "":
                continue

            if line.startswith(">"):
                header = line.strip()[1:]
                contig_size[header] = 0

            else:
                line_len = len(line.strip())
                assembly_size += line_len
                contig_size[header] += line_len

    return assembly_size, contig_size


Instruction:
Make a summary of the following Python 3 code
def main(sample_id, assembly_file, gsize, opts, assembler):

    logger.info("Starting assembly file processing")
    warnings = []
    fails = ""

    min_contig_len, min_kmer_cov, max_contigs = [int(x) for x in opts]
    logger.debug("Setting minimum conting length to: {}".format(
        min_contig_len))
    logger.debug("Setting minimum kmer coverage: {}".format(min_kmer_cov))

    # Parse the spades assembly file and perform the first filtering.
    logger.info("Starting assembly parsing")
    assembly_obj = Assembly(assembly_file, min_contig_len, min_kmer_cov,
                               sample_id)

    with open(".warnings", "w") as warn_fh:
        t_80 = gsize * 1000000 * 0.8
        t_150 = gsize * 1000000 * 1.5
        # Check if assembly size of the first assembly is lower than 80% of the
        # estimated genome size. If True, redo the filtering without the
        # k-mer coverage filter
        assembly_len = assembly_obj.get_assembly_length()
        logger.debug("Checking assembly length: {}".format(assembly_len))

        if assembly_len < t_80:

            logger.warning("Assembly size ({}) smaller than the minimum "
                           "threshold of 80% of expected genome size. "
                           "Applying contig filters without the k-mer "
                           "coverage filter".format(assembly_len))
            assembly_obj.filter_contigs(*[
                ["length", ">=", min_contig_len]
            ])

            assembly_len = assembly_obj.get_assembly_length()
            logger.debug("Checking updated assembly length: "
                         "{}".format(assembly_len))
            if assembly_len < t_80:

                warn_msg = "Assembly size smaller than the minimum" \
                           " threshold of 80% of expected genome size: {}".format(
                                assembly_len)
                logger.warning(warn_msg)
                warn_fh.write(warn_msg)
                fails = warn_msg

        if assembly_len > t_150:

            warn_msg = "Assembly size ({}) larger than the maximum" \
                       " threshold of 150% of expected genome size.".format(
                            assembly_len)
            logger.warning(warn_msg)
            warn_fh.write(warn_msg)
            fails = warn_msg

        logger.debug("Checking number of contigs: {}".format(
            len(assembly_obj.contigs)))
        contig_threshold = (max_contigs * gsize) / 1.5
        if len(assembly_obj.contigs) > contig_threshold:

            warn_msg = "The number of contigs ({}) exceeds the threshold of " \
                       "{} contigs per 1.5Mb ({})".format(
                            len(assembly_obj.contigs),
                            max_contigs,
                            round(contig_threshold, 1))

            logger.warning(warn_msg)
            warn_fh.write(warn_msg)
            warnings.append(warn_msg)

    # Write filtered assembly
    logger.debug("Renaming old assembly file to: {}".format(
        "{}.old".format(assembly_file)))
    assembly_obj.write_assembly("{}_proc.fasta".format(
        os.path.splitext(assembly_file)[0]))
    # Write report
    output_report = "{}.report.csv".format(sample_id)
    assembly_obj.write_report(output_report)
    # Write json report
    with open(".report.json", "w") as json_report:
        json_dic = {
            "tableRow": [{
                "sample": sample_id,
                "data": [
                    {"header": "Contigs ({})".format(assembler),
                     "value": len(assembly_obj.contigs),
                     "table": "assembly",
                     "columnBar": True},
                    {"header": "Assembled BP ({})".format(assembler),
                     "value": assembly_len,
                     "table": "assembly",
                     "columnBar": True}
                ]
            }],
        }

        if warnings:
            json_dic["warnings"] = [{
                "sample": sample_id,
                "table": "assembly",
                "value": warnings
            }]

        if fails:
            json_dic["fail"] = [{
                "sample": sample_id,
                "table": "assembly",
                "value": [fails]
            }]

        json_report.write(json.dumps(json_dic, separators=(",", ":")))

    with open(".status", "w") as status_fh:
        status_fh.write("pass")

Response: Main function for the process_spades template.


Instruction:
Can you tell what is the following Python 3 function doing
def convert_camel_case(name):
    s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name)
    return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower()

Response: Convers a CamelCase string into a snake_case one


Instruction:
Can you write a function in Python 3 where it
collects Process classes and return dict mapping templates to classes This function crawls through the components module and retrieves all classes that inherit from the Process class. Then, it converts the name of the classes (which should be CamelCase) to snake_case, which is used as the template name. Returns ------- dict Dictionary mapping the template name (snake_case) to the corresponding process class.

Response:
def collect_process_map():
    """Collects Process classes and return dict mapping templates to classes

    This function crawls through the components module and retrieves all
    classes that inherit from the Process class. Then, it converts the name
    of the classes (which should be CamelCase) to snake_case, which is used
    as the template name.

    Returns
    -------
    dict
        Dictionary mapping the template name (snake_case) to the corresponding
        process class.
    """

    process_map = {}

    prefix = "{}.".format(components.__name__)
    for importer, modname, _ in pkgutil.iter_modules(components.__path__,
                                                     prefix):

        _module = importer.find_module(modname).load_module(modname)

        _component_classes = [
            cls for cls in _module.__dict__.values() if
            isinstance(cls, type) and cls.__name__ != "Process"
        ]

        for cls in _component_classes:
            process_map[convert_camel_case(cls.__name__)] = cls

    return process_map


Instruction:
Make a summary of the following Python 3 code
def main(newick):

    logger.info("Starting newick file processing")

    print(newick)

    tree = dendropy.Tree.get(file=open(newick, 'r'), schema="newick")

    tree.reroot_at_midpoint()

    to_write=tree.as_string("newick").strip().replace("[&R] ", '').replace(' ', '_').replace("'", "")

    with open(".report.json", "w") as json_report:
        json_dic = {
            "treeData": [{
                "trees": [
                    to_write
                ]
            }],
        }

        json_report.write(json.dumps(json_dic, separators=(",", ":")))

    with open(".status", "w") as status_fh:
        status_fh.write("pass")

Response: This function is the main function of the process_newick template. It is the main function of the process_newick template. It is the main function of the process_newick template. It is the main function of the process_newick template. It is the main function of the process_newick template. It is the main function of the process_newick template.


Instruction:
How would you implement a function in Python 3 that
factorizes s. t. CUR

Response:
def factorize(self):
        """ Factorize s.t. CUR = data

            Updated Values
            --------------
            .C : updated values for C.
            .U : updated values for U.
            .R : updated values for R.
        """

        [prow, pcol] = self.sample_probability()

        self._rid = self.sample(self._rrank, prow)
        self._cid = self.sample(self._crank, pcol)

        self._cmdinit()

        self.computeUCR()


Instruction:
Can you generate the documentation for the following Python 3 function
def factorize(self):
        [prow, pcol] = self.sample_probability()
        self._rid = self.sample(self._rrank, prow)
        self._cid = self.sample(self._crank, pcol)

        self._rcnt = np.ones(len(self._rid))
        self._ccnt = np.ones(len(self._cid))

        self.computeUCR()

Response: Factorize s. t. CUR


Instruction:
Explain what the following Python 3 code does
def quickhull(sample):

    link = lambda a, b: np.concatenate((a, b[1:]))
    edge = lambda a, b: np.concatenate(([a], [b]))

    def dome(sample, base):
        h, t = base
        dists = np.dot(sample - h, np.dot(((0, -1), (1, 0)), (t - h)))
        outer = np.repeat(sample, dists > 0, axis=0)

        if len(outer):
            pivot = sample[np.argmax(dists)]
            return link(dome(outer, edge(h, pivot)),
                dome(outer, edge(pivot, t)))
        else:
            return base

    if len(sample) > 2:
        axis = sample[:, 0]
        base = np.take(sample, [np.argmin(axis), np.argmax(axis)], axis=0)
        return link(dome(sample, base),
            dome(sample, base[::-1]))
    else:
        return sample

Response: Find data points on the convex hull of a supplied data set.


Instruction:
Can you tell what is the following Python 3 function doing
def _map_w_to_data(self):

        # assign W to the next best data sample
        self._Wmapped_index = vq(self.data, self.W)
        self.Wmapped = np.zeros(self.W.shape)

        # do not directly assign, i.e. Wdist = self.data[:,sel]
        # as self might be unsorted (in non ascending order)
        # -> sorting sel would screw the matching to W if
        # self.data is stored as a hdf5 table (see h5py)
        for i, s in enumerate(self._Wmapped_index):
            self.Wmapped[:,i] = self.data[:,s]

Response: Map W to data points that are most similar to basis vectors W


Instruction:
Can you tell what is the following Python 3 function doing
def factorize(self, show_progress=False, compute_w=True, compute_h=True,
                  compute_err=True, niter=1):
        """ Factorize s.t. WH = data

            Parameters
            ----------
            show_progress : bool
                    print some extra information to stdout.
            compute_h : bool
                    iteratively update values for H.
            compute_w : bool
                    iteratively update values for W.
            compute_err : bool
                    compute Frobenius norm |data-WH| after each update and store
                    it to .ferr[k].

            Updated Values
            --------------
            .W : updated values for W.
            .H : updated values for H.
            .ferr : Frobenius norm |data-WH|.
        """

        AA.factorize(self, niter=1, show_progress=show_progress,
                  compute_w=compute_w, compute_h=compute_h,
                  compute_err=compute_err)

Response: Factorize s. t. WH = data


Instruction:
Create a Python 3 function for
updating the W attribute of the object with new values

Response:
def update_w(self):
        """ compute new W """

        def select_next(iterval):
            """ select the next best data sample using robust map
            or simply the max iterval ... """

            if self._robust_map:
                k = np.argsort(iterval)[::-1]
                d_sub = self.data[:,k[:self._robust_nselect]]
                self.sub.extend(k[:self._robust_nselect])

                # cluster d_sub
                kmeans_mdl = Kmeans(d_sub, num_bases=self._robust_cluster)
                kmeans_mdl.factorize(niter=10)

                # get largest cluster
                h = np.histogram(kmeans_mdl.assigned, range(self._robust_cluster+1))[0]
                largest_cluster = np.argmax(h)
                sel = pdist(kmeans_mdl.W[:, largest_cluster:largest_cluster+1], d_sub)
                sel = k[np.argmin(sel)]
            else:
                sel = np.argmax(iterval)

            return sel

        EPS = 10**-8

        if scipy.sparse.issparse(self.data):
            norm_data = np.sqrt(self.data.multiply(self.data).sum(axis=0))
            norm_data = np.array(norm_data).reshape((-1))
        else:
            norm_data = np.sqrt(np.sum(self.data**2, axis=0))

        self.select = []

        if self._method == 'pca' or self._method == 'aa':
            iterval = norm_data.copy()

        if self._method == 'nmf':
            iterval = np.sum(self.data, axis=0)/(np.sqrt(self.data.shape[0])*norm_data)
            iterval = 1.0 - iterval

        self.select.append(select_next(iterval))

        for l in range(1, self._num_bases):

            if scipy.sparse.issparse(self.data):
                c = self.data[:, self.select[-1]:self.select[-1]+1].T * self.data
                c = np.array(c.todense())
            else:
                c = np.dot(self.data[:,self.select[-1]], self.data)

            c = c/(norm_data * norm_data[self.select[-1]])

            if self._method == 'pca':
                c = 1.0 - np.abs(c)
                c = c * norm_data

            elif self._method == 'aa':
                c = (c*-1.0 + 1.0)/2.0
                c = c * norm_data

            elif self._method == 'nmf':
                c = 1.0 - np.abs(c)

            ### update the estimated volume
            iterval = c * iterval

            # detect the next best data point
            self.select.append(select_next(iterval))

            self._logger.info('cur_nodes: ' + str(self.select))

        # sort indices, otherwise h5py won't work
        self.W = self.data[:, np.sort(self.select)]

        # "unsort" it again to keep the correct order
        self.W = self.W[:, np.argsort(np.argsort(self.select))]


Instruction:
Implement a function in Python 3 to
factorize s. t. WH = data

Response:
def factorize(self, show_progress=False, compute_w=True, compute_h=True,
                  compute_err=True, robust_cluster=3, niter=1, robust_nselect=-1):
        """ Factorize s.t. WH = data

            Parameters
            ----------
            show_progress : bool
                    print some extra information to stdout.
                    False, default
            compute_h : bool
                    iteratively update values for H.
                    True, default
            compute_w : bool
                    iteratively update values for W.
                    default, True
            compute_err : bool
                    compute Frobenius norm |data-WH| after each update and store
                    it to .ferr[k].
            robust_cluster : int, optional
                    set the number of clusters for robust map selection.
                    3, default
            robust_nselect : int, optional
                    set the number of samples to consider for robust map
                    selection.
                    -1, default (automatically determine suitable number)

            Updated Values
            --------------
            .W : updated values for W.
            .H : updated values for H.
            .ferr : Frobenius norm |data-WH|.
        """
        self._robust_cluster = robust_cluster
        self._robust_nselect = robust_nselect

        if self._robust_nselect == -1:
            self._robust_nselect = np.round(np.log(self.data.shape[1])*2)

        AA.factorize(self, niter=1, show_progress=show_progress,
                  compute_w=compute_w, compute_h=compute_h,
                  compute_err=compute_err)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def processFlat(self):
        self.config["hier"] = False
        est_idxs, est_labels, F = self.process()
        assert est_idxs[0] == 0 and est_idxs[-1] == F.shape[1] - 1
        return self._postprocess(est_idxs, est_labels)

Response: Main process. for flat segmentation.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def processHierarchical(self):
        self.config["hier"] = True
        est_idxs, est_labels, F = self.process()
        for layer in range(len(est_idxs)):
            assert est_idxs[layer][0] == 0 and \
                est_idxs[layer][-1] == F.shape[1] - 1
            est_idxs[layer], est_labels[layer] = \
                self._postprocess(est_idxs[layer], est_labels[layer])
        return est_idxs, est_labels

Response: Main process. for hierarchial segmentation.


Instruction:
Can you generate the documentation for the following Python 3 function
def compute_gaussian_krnl(M):
    g = signal.gaussian(M, M // 3., sym=True)
    G = np.dot(g.reshape(-1, 1), g.reshape(1, -1))
    G[M // 2:, :M // 2] = -G[M // 2:, :M // 2]
    G[:M // 2, M // 2:] = -G[:M // 2, M // 2:]
    return G

Response: Creates a gaussian kernel following Foote s paper.


Instruction:
Given the following Python 3 function, write the documentation
def compute_ssm(X, metric="seuclidean"):
    D = distance.pdist(X, metric=metric)
    D = distance.squareform(D)
    D /= D.max()
    return 1 - D

Response: Computes the self - similarity matrix of X.


Instruction:
Given the following Python 3 function, write the documentation
def compute_nc(X, G):
    N = X.shape[0]
    M = G.shape[0]
    nc = np.zeros(N)

    for i in range(M // 2, N - M // 2 + 1):
        nc[i] = np.sum(X[i - M // 2:i + M // 2, i - M // 2:i + M // 2] * G)

    # Normalize
    nc += nc.min()
    nc /= nc.max()
    return nc

Response:
Computes the novelty curve from the self - similarity matrix X and
        the gaussian kernel G.


Instruction:
Write a Python 3 script to
obtain peaks from a novelty curve using an adaptive threshold.

Response:
def pick_peaks(nc, L=16):
    """Obtain peaks from a novelty curve using an adaptive threshold."""
    offset = nc.mean() / 20.

    nc = filters.gaussian_filter1d(nc, sigma=4)  # Smooth out nc

    th = filters.median_filter(nc, size=L) + offset
    #th = filters.gaussian_filter(nc, sigma=L/2., mode="nearest") + offset

    peaks = []
    for i in range(1, nc.shape[0] - 1):
        # is it a peak?
        if nc[i - 1] < nc[i] and nc[i] > nc[i + 1]:
            # is it above the threshold?
            if nc[i] > th[i]:
                peaks.append(i)
    #plt.plot(nc)
    #plt.plot(th)
    #for peak in peaks:
        #plt.axvline(peak)
    #plt.show()

    return peaks


Instruction:
Here you have a function in Python 3, explain what it does
def processFlat(self):
        # Preprocess to obtain features
        F = self._preprocess()

        # Normalize
        F = msaf.utils.normalize(F, norm_type=self.config["bound_norm_feats"])

        # Make sure that the M_gaussian is even
        if self.config["M_gaussian"] % 2 == 1:
            self.config["M_gaussian"] += 1

        # Median filter
        F = median_filter(F, M=self.config["m_median"])
        #plt.imshow(F.T, interpolation="nearest", aspect="auto"); plt.show()

        # Self similarity matrix
        S = compute_ssm(F)

        # Compute gaussian kernel
        G = compute_gaussian_krnl(self.config["M_gaussian"])
        #plt.imshow(S, interpolation="nearest", aspect="auto"); plt.show()

        # Compute the novelty curve
        nc = compute_nc(S, G)

        # Find peaks in the novelty curve
        est_idxs = pick_peaks(nc, L=self.config["L_peaks"])

        # Add first and last frames
        est_idxs = np.concatenate(([0], est_idxs, [F.shape[0] - 1]))

        # Empty labels
        est_labels = np.ones(len(est_idxs) - 1) * -1

        # Post process estimations
        est_idxs, est_labels = self._postprocess(est_idxs, est_labels)

        return est_idxs, est_labels

Response: Main function for processing the flat data.


Instruction:
Create a Python 3 function for
factorizing s. t. WH = data - WH

Response:
def factorize(self, show_progress=False, compute_w=True, compute_h=True,
                  compute_err=True, niter=1):
        """ Factorize s.t. WH = data

            Parameters
            ----------
            show_progress : bool
                    print some extra information to stdout.
            niter : int
                    number of iterations.
            compute_h : bool
                    iteratively update values for H.
            compute_w : bool
                    iteratively update values for W.
            compute_err : bool
                    compute Frobenius norm |data-WH| after each update and store
                    it to .ferr[k].

            Updated Values
            --------------
            .W : updated values for W.
            .H : updated values for H.
            .ferr : Frobenius norm |data-WH|.
        """
        if show_progress:
            self._logger.setLevel(logging.INFO)
        else:
            self._logger.setLevel(logging.ERROR)

        # create W and H if they don't already exist
        # -> any custom initialization to W,H should be done before
        if not hasattr(self,'W'):
               self.init_w()

        if not hasattr(self,'H'):
                self.init_h()

        if compute_err:
            self.ferr = np.zeros(niter)

        for i in range(niter):
            if compute_w:
                self.update_w()

            if compute_h:
                self.update_h()

            if compute_err:
                self.ferr[i] = self.frobenius_norm()
                self._logger.info('Iteration ' + str(i+1) + '/' + str(niter) +
                    ' FN:' + str(self.ferr[i]))
            else:
                self._logger.info('Iteration ' + str(i+1) + '/' + str(niter))


Instruction:
Write a Python 3 function that can
compute the novelty curve from the structural features X.

Response:
def compute_nc(X):
    """Computes the novelty curve from the structural features."""
    N = X.shape[0]
    # nc = np.sum(np.diff(X, axis=0), axis=1) # Difference between SF's

    nc = np.zeros(N)
    for i in range(N - 1):
        nc[i] = distance.euclidean(X[i, :], X[i + 1, :])

    # Normalize
    nc += np.abs(nc.min())
    nc /= float(nc.max())
    return nc


Instruction:
Given the following Python 3 function, write the documentation
def pick_peaks(nc, L=16, offset_denom=0.1):
    offset = nc.mean() * float(offset_denom)
    th = filters.median_filter(nc, size=L) + offset
    #th = filters.gaussian_filter(nc, sigma=L/2., mode="nearest") + offset
    #import pylab as plt
    #plt.plot(nc)
    #plt.plot(th)
    #plt.show()
    # th = np.ones(nc.shape[0]) * nc.mean() - 0.08
    peaks = []
    for i in range(1, nc.shape[0] - 1):
        # is it a peak?
        if nc[i - 1] < nc[i] and nc[i] > nc[i + 1]:
            # is it above the threshold?
            if nc[i] > th[i]:
                peaks.append(i)
    return peaks

Response: Obtain peaks from a novelty curve using an adaptive threshold.


Instruction:
Can you implement a function in Python 3 that
shifts circularly the X squre matrix in order to get a time - lag matrix.

Response:
def circular_shift(X):
    """Shifts circularly the X squre matrix in order to get a
        time-lag matrix."""
    N = X.shape[0]
    L = np.zeros(X.shape)
    for i in range(N):
        L[i, :] = np.asarray([X[(i + j) % N, j] for j in range(N)])
    return L


Instruction:
Create a Python 3 function to
time - delay embedding with m dimensions and tau delays.

Response:
def embedded_space(X, m, tau=1):
    """Time-delay embedding with m dimensions and tau delays."""
    N = X.shape[0] - int(np.ceil(m))
    Y = np.zeros((N, int(np.ceil(X.shape[1] * m))))
    for i in range(N):
        # print X[i:i+m,:].flatten().shape, w, X.shape
        # print Y[i,:].shape
        rem = int((m % 1) * X.shape[1])  # Reminder for float m
        Y[i, :] = np.concatenate((X[i:i + int(m), :].flatten(),
                                 X[i + int(m), :rem]))
    return Y


Instruction:
Can you create a Python 3 function that
processes the flat data for the segment indeces and return a dictionary of the number of times and labels.

Response:
def processFlat(self):
        """Main process.
        Returns
        -------
        est_idxs : np.array(N)
            Estimated times for the segment boundaries in frame indeces.
        est_labels : np.array(N-1)
            Estimated labels for the segments.
        """
        # Structural Features params
        Mp = self.config["Mp_adaptive"]   # Size of the adaptive threshold for
                                          # peak picking
        od = self.config["offset_thres"]  # Offset coefficient for adaptive
                                          # thresholding

        M = self.config["M_gaussian"]     # Size of gaussian kernel in beats
        m = self.config["m_embedded"]     # Number of embedded dimensions
        k = self.config["k_nearest"]      # k*N-nearest neighbors for the
                                          # recurrence plot

        # Preprocess to obtain features, times, and input boundary indeces
        F = self._preprocess()

        # Normalize
        F = U.normalize(F, norm_type=self.config["bound_norm_feats"])

        # Check size in case the track is too short
        if F.shape[0] > 20:

            if self.framesync:
                red = 0.1
                F_copy = np.copy(F)
                F = librosa.util.utils.sync(
                    F.T, np.linspace(0, F.shape[0], num=F.shape[0] * red),
                    pad=False).T

            # Emedding the feature space (i.e. shingle)
            E = embedded_space(F, m)
            # plt.imshow(E.T, interpolation="nearest", aspect="auto"); plt.show()

            # Recurrence matrix
            R = librosa.segment.recurrence_matrix(
                E.T,
                k=k * int(F.shape[0]),
                width=1,  # zeros from the diagonal
                metric="euclidean",
                sym=True).astype(np.float32)

            # Circular shift
            L = circular_shift(R)
            #plt.imshow(L, interpolation="nearest", cmap=plt.get_cmap("binary"))
            #plt.show()

            # Obtain structural features by filtering the lag matrix
            SF = gaussian_filter(L.T, M=M, axis=1)
            SF = gaussian_filter(L.T, M=1, axis=0)
            # plt.imshow(SF.T, interpolation="nearest", aspect="auto")
            #plt.show()

            # Compute the novelty curve
            nc = compute_nc(SF)

            # Find peaks in the novelty curve
            est_bounds = pick_peaks(nc, L=Mp, offset_denom=od)

            # Re-align embedded space
            est_bounds = np.asarray(est_bounds) + int(np.ceil(m / 2.))

            if self.framesync:
                est_bounds /= red
                F = F_copy
        else:
            est_bounds = []

        # Add first and last frames
        est_idxs = np.concatenate(([0], est_bounds, [F.shape[0] - 1]))
        est_idxs = np.unique(est_idxs)

        assert est_idxs[0] == 0 and est_idxs[-1] == F.shape[0] - 1

        # Empty labels
        est_labels = np.ones(len(est_idxs) - 1) * - 1

        # Post process estimations
        est_idxs, est_labels = self._postprocess(est_idxs, est_labels)

        # plt.figure(1)
        # plt.plot(nc);
        # [plt.axvline(p, color="m", ymin=.6) for p in est_bounds]
        # [plt.axvline(b, color="b", ymax=.6, ymin=.3) for b in brian_bounds]
        # [plt.axvline(b, color="g", ymax=.3) for b in ann_bounds]
        # plt.show()

        return est_idxs, est_labels


Instruction:
Can you tell what is the following Python 3 function doing
def _plot_formatting(title, est_file, algo_ids, last_bound, N, output_file):
    import matplotlib.pyplot as plt
    if title is None:
        title = os.path.basename(est_file).split(".")[0]
    plt.title(title)
    plt.yticks(np.arange(0, 1, 1 / float(N)) + 1 / (float(N) * 2))
    plt.gcf().subplots_adjust(bottom=0.22)
    plt.gca().set_yticklabels(algo_ids)
    plt.xlabel("Time (seconds)")
    plt.xlim((0, last_bound))
    plt.tight_layout()
    if output_file is not None:
        plt.savefig(output_file)
    plt.show()

Response:
Formats the plot with the correct axis labels title ticks and
    so on.


Instruction:
Explain what the following Python 3 code does
def plot_boundaries(all_boundaries, est_file, algo_ids=None, title=None,
                    output_file=None):
    """Plots all the boundaries.

    Parameters
    ----------
    all_boundaries: list
        A list of np.arrays containing the times of the boundaries, one array
        for each algorithm.
    est_file: str
        Path to the estimated file (JSON file)
    algo_ids : list
        List of algorithm ids to to read boundaries from.
        If None, all algorithm ids are read.
    title : str
        Title of the plot. If None, the name of the file is printed instead.
    """
    import matplotlib.pyplot as plt
    N = len(all_boundaries)  # Number of lists of boundaries
    if algo_ids is None:
        algo_ids = io.get_algo_ids(est_file)

    # Translate ids
    for i, algo_id in enumerate(algo_ids):
        algo_ids[i] = translate_ids[algo_id]
    algo_ids = ["GT"] + algo_ids

    figsize = (6, 4)
    plt.figure(1, figsize=figsize, dpi=120, facecolor='w', edgecolor='k')
    for i, boundaries in enumerate(all_boundaries):
        color = "b"
        if i == 0:
            color = "g"
        for b in boundaries:
            plt.axvline(b, i / float(N), (i + 1) / float(N), color=color)
        plt.axhline(i / float(N), color="k", linewidth=1)

    # Format plot
    _plot_formatting(title, est_file, algo_ids, all_boundaries[0][-1], N,
                     output_file)

Response: Plots all the boundaries.


Instruction:
Create a Python 3 function to
plot all the labels of the boundaries.

Response:
def plot_labels(all_labels, gt_times, est_file, algo_ids=None, title=None,
                output_file=None):
    """Plots all the labels.

    Parameters
    ----------
    all_labels: list
        A list of np.arrays containing the labels of the boundaries, one array
        for each algorithm.
    gt_times: np.array
        Array with the ground truth boundaries.
    est_file: str
        Path to the estimated file (JSON file)
    algo_ids : list
        List of algorithm ids to to read boundaries from.
        If None, all algorithm ids are read.
    title : str
        Title of the plot. If None, the name of the file is printed instead.
    """
    import matplotlib.pyplot as plt
    N = len(all_labels)  # Number of lists of labels
    if algo_ids is None:
        algo_ids = io.get_algo_ids(est_file)

    # Translate ids
    for i, algo_id in enumerate(algo_ids):
        algo_ids[i] = translate_ids[algo_id]
    algo_ids = ["GT"] + algo_ids

    # Index the labels to normalize them
    for i, labels in enumerate(all_labels):
        all_labels[i] = mir_eval.util.index_labels(labels)[0]

    # Get color map
    cm = plt.get_cmap('gist_rainbow')
    max_label = max(max(labels) for labels in all_labels)

    # To intervals
    gt_inters = utils.times_to_intervals(gt_times)

    # Plot labels
    figsize = (6, 4)
    plt.figure(1, figsize=figsize, dpi=120, facecolor='w', edgecolor='k')
    for i, labels in enumerate(all_labels):
        for label, inter in zip(labels, gt_inters):
            plt.axvspan(inter[0], inter[1], ymin=i / float(N),
                        ymax=(i + 1) / float(N), alpha=0.6,
                        color=cm(label / float(max_label)))
        plt.axhline(i / float(N), color="k", linewidth=1)

    # Draw the boundary lines
    for bound in gt_times:
        plt.axvline(bound, color="g")

    # Format plot
    _plot_formatting(title, est_file, algo_ids, gt_times[-1], N,
                     output_file)


Instruction:
Can you write a function in Python 3 where it
plots the results of one track with ground truth if it exists.

Response:
def plot_one_track(file_struct, est_times, est_labels, boundaries_id, labels_id,
                   title=None):
    """Plots the results of one track, with ground truth if it exists."""
    import matplotlib.pyplot as plt
    # Set up the boundaries id
    bid_lid = boundaries_id
    if labels_id is not None:
        bid_lid += " + " + labels_id
    try:
        # Read file
        jam = jams.load(file_struct.ref_file)
        ann = jam.search(namespace='segment_.*')[0]
        ref_inters, ref_labels = ann.to_interval_values()

        # To times
        ref_times = utils.intervals_to_times(ref_inters)
        all_boundaries = [ref_times, est_times]
        all_labels = [ref_labels, est_labels]
        algo_ids = ["GT", bid_lid]
    except:
        logging.warning("No references found in %s. Not plotting groundtruth"
                        % file_struct.ref_file)
        all_boundaries = [est_times]
        all_labels = [est_labels]
        algo_ids = [bid_lid]

    N = len(all_boundaries)

    # Index the labels to normalize them
    for i, labels in enumerate(all_labels):
        all_labels[i] = mir_eval.util.index_labels(labels)[0]

    # Get color map
    cm = plt.get_cmap('gist_rainbow')
    max_label = max(max(labels) for labels in all_labels)

    figsize = (8, 4)
    plt.figure(1, figsize=figsize, dpi=120, facecolor='w', edgecolor='k')
    for i, boundaries in enumerate(all_boundaries):
        color = "b"
        if i == 0:
            color = "g"
        for b in boundaries:
            plt.axvline(b, i / float(N), (i + 1) / float(N), color=color)
        if labels_id is not None:
            labels = all_labels[i]
            inters = utils.times_to_intervals(boundaries)
            for label, inter in zip(labels, inters):
                plt.axvspan(inter[0], inter[1], ymin=i / float(N),
                            ymax=(i + 1) / float(N), alpha=0.6,
                            color=cm(label / float(max_label)))
        plt.axhline(i / float(N), color="k", linewidth=1)

    # Format plot
    _plot_formatting(title, os.path.basename(file_struct.audio_file), algo_ids,
                     all_boundaries[0][-1], N, None)


Instruction:
Write a Python 3 script for
plotting a given tree containing hierarchical segmentation.

Response:
def plot_tree(T, res=None, title=None, cmap_id="Pastel2"):
    """Plots a given tree, containing hierarchical segmentation.

    Parameters
    ----------
    T: mir_eval.segment.tree
        A tree object containing the hierarchical segmentation.
    res: float
        Frame-rate resolution of the tree (None to use seconds).
    title: str
        Title for the plot. `None` for no title.
    cmap_id: str
        Color Map ID
    """
    import matplotlib.pyplot as plt
    def round_time(t, res=0.1):
        v = int(t / float(res)) * res
        return v

    # Get color map
    cmap = plt.get_cmap(cmap_id)

    # Get segments by level
    level_bounds = []
    for level in T.levels:
        if level == "root":
            continue
        segments = T.get_segments_in_level(level)
        level_bounds.append(segments)

    # Plot axvspans for each segment
    B = float(len(level_bounds))
    #plt.figure(figsize=figsize)
    for i, segments in enumerate(level_bounds):
        labels = utils.segment_labels_to_floats(segments)
        for segment, label in zip(segments, labels):
            #print i, label, cmap(label)
            if res is None:
                start = segment.start
                end = segment.end
                xlabel = "Time (seconds)"
            else:
                start = int(round_time(segment.start, res=res) / res)
                end = int(round_time(segment.end, res=res) / res)
                xlabel = "Time (frames)"
            plt.axvspan(start, end,
                        ymax=(len(level_bounds) - i) / B,
                        ymin=(len(level_bounds) - i - 1) / B,
                        facecolor=cmap(label))

    # Plot labels
    L = float(len(T.levels) - 1)
    plt.yticks(np.linspace(0, (L - 1) / L, num=L) + 1 / L / 2.,
               T.levels[1:][::-1])
    plt.xlabel(xlabel)
    if title is not None:
        plt.title(title)
    plt.gca().set_xlim([0, end])


Instruction:
Can you tell what is the following Python 3 function doing
def get_feat_segments(F, bound_idxs):
    # Make sure bound_idxs are not empty
    assert len(bound_idxs) > 0, "Boundaries can't be empty"

    # Make sure that boundaries are sorted
    bound_idxs = np.sort(bound_idxs)

    # Make sure we're not out of bounds
    assert bound_idxs[0] >= 0 and bound_idxs[-1] < F.shape[0], \
        "Boundaries are not correct for the given feature dimensions."

    # Obtain the segments
    feat_segments = []
    for i in range(len(bound_idxs) - 1):
        feat_segments.append(F[bound_idxs[i]:bound_idxs[i + 1], :])
    return feat_segments

Response: Returns a set of segments defined by the bound_idxs.


Instruction:
Write a Python 3 script for
giving a list of feature segments return a list of 2D - Fourier MagnitudeCoefs with the maximum size as main size and zero pad the rest.

Response:
def feat_segments_to_2dfmc_max(feat_segments, offset=4):
    """From a list of feature segments, return a list of 2D-Fourier Magnitude
    Coefs using the maximum segment size as main size and zero pad the rest.

    Parameters
    ----------
    feat_segments: list
        List of segments, one for each boundary interval.
    offset: int >= 0
        Number of frames to ignore from beginning and end of each segment.

    Returns
    -------
    fmcs: np.ndarray
        Tensor containing the 2D-FMC matrices, one matrix per segment.
    """
    if len(feat_segments) == 0:
        return []

    # Get maximum segment size
    max_len = max([feat_segment.shape[0] for feat_segment in feat_segments])

    fmcs = []
    for feat_segment in feat_segments:
        # Zero pad if needed
        X = np.zeros((max_len, feat_segment.shape[1]))

        # Remove a set of frames in the beginning an end of the segment
        if feat_segment.shape[0] <= offset or offset == 0:
            X[:feat_segment.shape[0], :] = feat_segment
        else:
            X[:feat_segment.shape[0] - offset, :] = \
                feat_segment[offset // 2:-offset // 2, :]

        # Compute the 2D-FMC
        try:
            fmcs.append(utils2d.compute_ffmc2d(X))
        except:
            logging.warning("Couldn't compute the 2D Fourier Transform")
            fmcs.append(np.zeros((X.shape[0] * X.shape[1]) // 2 + 1))

        # Normalize
        # fmcs[-1] = fmcs[-1] / float(fmcs[-1].max())

    return np.asarray(fmcs)


Instruction:
Can you tell what is the following Python 3 function doing
def compute_similarity(F, bound_idxs, dirichlet=False, xmeans=False, k=5,
                       offset=4):
    """Main function to compute the segment similarity of file file_struct.

    Parameters
    ----------
    F: np.ndarray
        Matrix containing one feature vector per row.
    bound_idxs: np.ndarray
        Array with the indeces of the segment boundaries.
    dirichlet: boolean
        Whether to use the dirichlet estimator of the number of unique labels.
    xmeans: boolean
        Whether to use the xmeans estimator of the number of unique labels.
    k: int > 0
        If the other two predictors are `False`, use fixed number of labels.
    offset: int >= 0
        Number of frames to ignore from beginning and end of each segment.

    Returns
    -------
    labels_est: np.ndarray
        Estimated labels, containing integer identifiers.
    """
    # Get the feature segments
    feat_segments = get_feat_segments(F, bound_idxs)

    # Get the 2D-FMCs segments
    fmcs = feat_segments_to_2dfmc_max(feat_segments, offset)
    if len(fmcs) == 0:
        return np.arange(len(bound_idxs) - 1)

    # Compute the labels using kmeans
    if dirichlet:
        k_init = np.min([fmcs.shape[0], k])
        # Only compute the dirichlet method if the fmc shape is small enough
        if fmcs.shape[1] > 500:
            labels_est = compute_labels_kmeans(fmcs, k=k)
        else:
            dpgmm = mixture.DPGMM(n_components=k_init, covariance_type='full')
            # dpgmm = mixture.VBGMM(n_components=k_init, covariance_type='full')
            dpgmm.fit(fmcs)
            k = len(dpgmm.means_)
            labels_est = dpgmm.predict(fmcs)
            # print("Estimated with Dirichlet Process:", k)
    if xmeans:
        xm = XMeans(fmcs, plot=False)
        k = xm.estimate_K_knee(th=0.01, maxK=8)
        labels_est = compute_labels_kmeans(fmcs, k=k)
        # print("Estimated with Xmeans:", k)
    else:
        labels_est = compute_labels_kmeans(fmcs, k=k)

    return labels_est

Response: Compute the similarity of a file with a set of features.


Instruction:
How would you explain what the following Python 3 function does
def processFlat(self):
        # Preprocess to obtain features, times, and input boundary indeces
        F = self._preprocess()

        # Normalize
        F = U.normalize(F, norm_type=self.config["label_norm_feats"],
                        floor=self.config["label_norm_floor"],
                        min_db=self.config["label_norm_min_db"])

        # Find the labels using 2D-FMCs
        est_labels = compute_similarity(F, self.in_bound_idxs,
                                        dirichlet=self.config["dirichlet"],
                                        xmeans=self.config["xmeans"],
                                        k=self.config["k"],
                                        offset=self.config["2dfmc_offset"])

        # Post process estimations
        self.in_bound_idxs, est_labels = self._postprocess(self.in_bound_idxs,
                                                           est_labels)

        return self.in_bound_idxs, est_labels

Response: Main function for processing the flat indeces and labels.


Instruction:
Can you create a Python 3 function that
fits the OLDA model to the given data and labels.

Response:
def fit(self, X, Y):
        '''Fit the OLDA model

        Parameters
        ----------
        X : array-like, shape [n_samples]
            Training data: each example is an n_features-by-* data array

        Y : array-like, shape [n_samples]
            Training labels: each label is an array of change-points
                             (eg, a list of segment boundaries)

        Returns
        -------
        self : object
        '''

        # Re-initialize the scatter matrices
        self.scatter_ordinal_ = None
        self.scatter_within_  = None

        # Reduce to partial-fit
        self.partial_fit(X, Y)

        return self


Instruction:
Can you tell what is the following Python 3 function doing
def partial_fit(self, X, Y):
        '''Partial-fit the OLDA model

        Parameters
        ----------
        X : array-like, shape [n_samples]
            Training data: each example is an n_features-by-* data array

        Y : array-like, shape [n_samples]
            Training labels: each label is an array of change-points
                             (eg, a list of segment boundaries)

        Returns
        -------
        self : object
        '''

        for (xi, yi) in itertools.izip(X, Y):

            prev_mean       = None
            prev_length     = None

            if self.scatter_within_ is None:
                # First round: initialize
                d, n = xi.shape

                if yi[0] > 0:
                    yi = np.concatenate([np.array([0]), yi])
                if yi[-1] < n:
                    yi = np.concatenate([yi, np.array([n])])

                self.scatter_within_  = self.sigma * np.eye(d)
                self.scatter_ordinal_ = np.zeros(d)

            # iterate over segments
            for (seg_start, seg_end) in zip(yi[:-1], yi[1:]):

                seg_length = seg_end - seg_start

                if seg_length < 2:
                    continue

                seg_mean = np.mean(xi[:, seg_start:seg_end], axis=1, keepdims=True)
                seg_cov  = np.cov(xi[:, seg_start:seg_end])
                self.scatter_within_ = self.scatter_within_ + seg_length * seg_cov

                if prev_mean is not None:
                    diff_ord = seg_mean - (prev_length * prev_mean + seg_length * seg_mean) / (prev_length + seg_length)
                    self.scatter_ordinal_ = self.scatter_ordinal_ + seg_length * np.dot(diff_ord, diff_ord.T)

                    diff_ord = prev_mean - (prev_length * prev_mean + seg_length * seg_mean) / (prev_length + seg_length)
                    self.scatter_ordinal_ = self.scatter_ordinal_ + prev_length * np.dot(diff_ord, diff_ord.T)

                prev_mean = seg_mean
                prev_length = seg_length

        e_vals, e_vecs = scipy.linalg.eig(self.scatter_ordinal_, self.scatter_within_)
        self.e_vals_ = e_vals
        self.e_vecs_ = e_vecs
        self.components_ = e_vecs.T
        return self

Response: Partial fit the OLDA model.


Instruction:
Given the following Python 3 function, write the documentation
def compute_features(self):
        linear_cqt = np.abs(librosa.cqt(
            self._audio, sr=self.sr, hop_length=self.hop_length,
            n_bins=self.n_bins, norm=self.norm, filter_scale=self.filter_scale)
                            ) ** 2
        cqt = librosa.amplitude_to_db(linear_cqt, ref=self.ref_power).T
        return cqt

Response:
Compute the features for a give
            time frame and beat.


Instruction:
Create a Python 3 function for
computing the features for a give time frame and beat.

Response:
def compute_features(self):
        """Actual implementation of the features.

        Returns
        -------
        mfcc: np.array(N, F)
            The features, each row representing a feature vector for a give
            time frame/beat.
        """
        S = librosa.feature.melspectrogram(self._audio,
                                           sr=self.sr,
                                           n_fft=self.n_fft,
                                           hop_length=self.hop_length,
                                           n_mels=self.n_mels)
        log_S = librosa.amplitude_to_db(S, ref=self.ref_power)
        mfcc = librosa.feature.mfcc(S=log_S, n_mfcc=self.n_mfcc).T
        return mfcc


Instruction:
How would you code a function in Python 3 to
compute the features for a give time frame and beat.

Response:
def compute_features(self):
        """Actual implementation of the features.

        Returns
        -------
        pcp: np.array(N, F)
            The features, each row representing a feature vector for a give
            time frame/beat.
        """
        audio_harmonic, _ = self.compute_HPSS()
        pcp_cqt = np.abs(librosa.hybrid_cqt(audio_harmonic,
                                            sr=self.sr,
                                            hop_length=self.hop_length,
                                            n_bins=self.n_bins,
                                            norm=self.norm,
                                            fmin=self.f_min)) ** 2
        pcp = librosa.feature.chroma_cqt(C=pcp_cqt,
                                         sr=self.sr,
                                         hop_length=self.hop_length,
                                         n_octaves=self.n_octaves,
                                         fmin=self.f_min).T
        return pcp


Instruction:
How would you code a function in Python 3 to
compute the features for a give time frame and beat.

Response:
def compute_features(self):
        """Actual implementation of the features.

        Returns
        -------
        tonnetz: np.array(N, F)
            The features, each row representing a feature vector for a give
            time frame/beat.
        """
        pcp = PCP(self.file_struct, self.feat_type, self.sr, self.hop_length,
                  self.n_bins, self.norm, self.f_min, self.n_octaves).features
        tonnetz = librosa.feature.tonnetz(chroma=pcp.T).T
        return tonnetz


Instruction:
Write a Python 3 function that can
compute the features for a give time frame.

Response:
def compute_features(self):
        """Actual implementation of the features.

        Returns
        -------
        tempogram: np.array(N, F)
            The features, each row representing a feature vector for a give
            time frame/beat.
        """
        return librosa.feature.tempogram(self._audio, sr=self.sr,
                                         hop_length=self.hop_length,
                                         win_length=self.win_length).T


Instruction:
Make a summary of the following Python 3 code
def read_estimations(est_file, boundaries_id, labels_id=None, **params):
    # Open file and read jams
    jam = jams.load(est_file)

    # Find correct estimation
    est = find_estimation(jam, boundaries_id, labels_id, params)
    if est is None:
        raise NoEstimationsError("No estimations for file: %s" % est_file)

    # Get data values
    all_boundaries, all_labels = est.to_interval_values()

    if params["hier"]:
        hier_bounds = defaultdict(list)
        hier_labels = defaultdict(list)
        for bounds, labels in zip(all_boundaries, all_labels):
            level = labels["level"]
            hier_bounds[level].append(bounds)
            hier_labels[level].append(labels["label"])
        # Order
        all_boundaries = []
        all_labels = []
        for key in sorted(list(hier_bounds.keys())):
            all_boundaries.append(np.asarray(hier_bounds[key]))
            all_labels.append(np.asarray(hier_labels[key]))

    return all_boundaries, all_labels

Response: Reads the estimations of an algorithm from a JAMS file and returns the boundaries and labels.


Instruction:
Implement a Python 3 function for
reading the boundary times and labels of a single audio file.

Response:
def read_references(audio_path, annotator_id=0):
    """Reads the boundary times and the labels.

    Parameters
    ----------
    audio_path : str
        Path to the audio file

    Returns
    -------
    ref_times : list
        List of boundary times
    ref_labels : list
        List of labels

    Raises
    ------
    IOError: if `audio_path` doesn't exist.
    """
    # Dataset path
    ds_path = os.path.dirname(os.path.dirname(audio_path))

    # Read references
    jam_path = os.path.join(ds_path, ds_config.references_dir,
                            os.path.basename(audio_path)[:-4] +
                            ds_config.references_ext)

    jam = jams.load(jam_path, validate=False)
    ann = jam.search(namespace='segment_.*')[annotator_id]
    ref_inters, ref_labels = ann.to_interval_values()

    # Intervals to times
    ref_times = utils.intervals_to_times(ref_inters)

    return ref_times, ref_labels


Instruction:
Make a summary of the following Python 3 code
def align_times(times, frames):
    dist = np.minimum.outer(times, frames)
    bound_frames = np.argmax(np.maximum(0, dist), axis=1)
    aligned_times = np.unique(bound_frames)
    return aligned_times

Response: Aligns the times to the closest frame times.


Instruction:
Write a Python 3 script for
finding the correct estimation from all the estimations contained in a JAMS file given the specified parameters.

Response:
def find_estimation(jam, boundaries_id, labels_id, params):
    """Finds the correct estimation from all the estimations contained in a
    JAMS file given the specified arguments.

    Parameters
    ----------
    jam : jams.JAMS
        JAMS object.
    boundaries_id : str
        Identifier of the algorithm used to compute the boundaries.
    labels_id : str
        Identifier of the algorithm used to compute the labels.
    params : dict
        Additional search parameters. E.g. {"feature" : "pcp"}.

    Returns
    -------
    ann : jams.Annotation
        Found estimation.
        `None` if it couldn't be found.
    """
    # Use handy JAMS search interface
    namespace = "multi_segment" if params["hier"] else "segment_open"
    # TODO: This is a workaround to issue in JAMS. Should be
    # resolved in JAMS 0.2.3, but for now, this works too.
    ann = jam.search(namespace=namespace).\
        search(**{"Sandbox.boundaries_id": boundaries_id}).\
        search(**{"Sandbox.labels_id": lambda x:
                  (isinstance(x, six.string_types) and
                   re.match(labels_id, x) is not None) or x is None})
    for key, val in zip(params.keys(), params.values()):
        if isinstance(val, six.string_types):
            ann = ann.search(**{"Sandbox.%s" % key: val})
        else:
            ann = ann.search(**{"Sandbox.%s" % key: lambda x: x == val})

    # Check estimations found
    if len(ann) > 1:
        logging.warning("More than one estimation with same parameters.")

    if len(ann) > 0:
        ann = ann[0]

    # If we couldn't find anything, let's return None
    if not ann:
        ann = None

    return ann


Instruction:
How would you code a function in Python 3 to
save the segment estimations in a JAMS file.

Response:
def save_estimations(file_struct, times, labels, boundaries_id, labels_id,
                     **params):
    """Saves the segment estimations in a JAMS file.

    Parameters
    ----------
    file_struct : FileStruct
        Object with the different file paths of the current file.
    times : np.array or list
        Estimated boundary times.
        If `list`, estimated hierarchical boundaries.
    labels : np.array(N, 2)
        Estimated labels (None in case we are only storing boundary
        evaluations).
    boundaries_id : str
        Boundary algorithm identifier.
    labels_id : str
        Labels algorithm identifier.
    params : dict
        Dictionary with additional parameters for both algorithms.
    """
    # Remove features if they exist
    params.pop("features", None)

    # Get duration
    dur = get_duration(file_struct.features_file)

    # Convert to intervals and sanity check
    if 'numpy' in str(type(times)):
        # Flat check
        inters = utils.times_to_intervals(times)
        assert len(inters) == len(labels), "Number of boundary intervals " \
            "(%d) and labels (%d) do not match" % (len(inters), len(labels))
        # Put into lists to simplify the writing process later
        inters = [inters]
        labels = [labels]
    else:
        # Hierarchical check
        inters = []
        for level in range(len(times)):
            est_inters = utils.times_to_intervals(times[level])
            inters.append(est_inters)
            assert len(inters[level]) == len(labels[level]), \
                "Number of boundary intervals (%d) and labels (%d) do not " \
                "match in level %d" % (len(inters[level]), len(labels[level]),
                                       level)

    # Create new estimation
    namespace = "multi_segment" if params["hier"] else "segment_open"
    ann = jams.Annotation(namespace=namespace)

    # Find estimation in file
    if os.path.isfile(file_struct.est_file):
        jam = jams.load(file_struct.est_file, validate=False)
        curr_ann = find_estimation(jam, boundaries_id, labels_id, params)
        if curr_ann is not None:
            curr_ann.data = ann.data  # cleanup all data
            ann = curr_ann  # This will overwrite the existing estimation
        else:
            jam.annotations.append(ann)
    else:
        # Create new JAMS if it doesn't exist
        jam = jams.JAMS()
        jam.file_metadata.duration = dur
        jam.annotations.append(ann)

    # Save metadata and parameters
    ann.annotation_metadata.version = msaf.__version__
    ann.annotation_metadata.data_source = "MSAF"
    sandbox = {}
    sandbox["boundaries_id"] = boundaries_id
    sandbox["labels_id"] = labels_id
    sandbox["timestamp"] = \
        datetime.datetime.today().strftime("%Y/%m/%d %H:%M:%S")
    for key in params:
        sandbox[key] = params[key]
    ann.sandbox = sandbox

    # Save actual data
    for i, (level_inters, level_labels) in enumerate(zip(inters, labels)):
        for bound_inter, label in zip(level_inters, level_labels):
            dur = float(bound_inter[1]) - float(bound_inter[0])
            label = chr(int(label) + 65)
            if params["hier"]:
                value = {"label": label, "level": i}
            else:
                value = label
            ann.append(time=bound_inter[0], duration=dur,
                       value=value)

    # Write results
    jam.save(file_struct.est_file)


Instruction:
Can you generate the documentation for the following Python 3 function
def get_all_boundary_algorithms():
    algo_ids = []
    for name in msaf.algorithms.__all__:
        module = eval(msaf.algorithms.__name__ + "." + name)
        if module.is_boundary_type:
            algo_ids.append(module.algo_id)
    return algo_ids

Response: Gets all the possible boundary algorithms in MSAF.


Instruction:
Can you tell what is the following Python 3 function doing
def get_all_label_algorithms():
    algo_ids = []
    for name in msaf.algorithms.__all__:
        module = eval(msaf.algorithms.__name__ + "." + name)
        if module.is_label_type:
            algo_ids.append(module.algo_id)
    return algo_ids

Response: Gets all the possible label algorithms in MSAF.


Instruction:
Here you have a function in Python 3, explain what it does
def get_configuration(feature, annot_beats, framesync, boundaries_id,
                      labels_id):
    """Gets the configuration dictionary from the current parameters of the
    algorithms to be evaluated."""
    config = {}
    config["annot_beats"] = annot_beats
    config["feature"] = feature
    config["framesync"] = framesync
    bound_config = {}
    if boundaries_id != "gt":
        bound_config = \
            eval(msaf.algorithms.__name__ + "." + boundaries_id).config
        config.update(bound_config)
    if labels_id is not None:
        label_config = \
            eval(msaf.algorithms.__name__ + "." + labels_id).config

        # Make sure we don't have parameter name duplicates
        if labels_id != boundaries_id:
            overlap = set(bound_config.keys()). \
                intersection(set(label_config.keys()))
            assert len(overlap) == 0, \
                "Parameter %s must not exist both in %s and %s algorithms" % \
                (overlap, boundaries_id, labels_id)
        config.update(label_config)
    return config

Response:
Gets the configuration dictionary from the current parameters of the the
    algorithms to be evaluated.


Instruction:
Write a Python 3 script to
get the files of the given dataset.

Response:
def get_dataset_files(in_path):
    """Gets the files of the given dataset."""
    # Get audio files
    audio_files = []
    for ext in ds_config.audio_exts:
        audio_files += glob.glob(
            os.path.join(in_path, ds_config.audio_dir, "*" + ext))

    # Make sure directories exist
    utils.ensure_dir(os.path.join(in_path, ds_config.features_dir))
    utils.ensure_dir(os.path.join(in_path, ds_config.estimations_dir))
    utils.ensure_dir(os.path.join(in_path, ds_config.references_dir))

    # Get the file structs
    file_structs = []
    for audio_file in audio_files:
        file_structs.append(FileStruct(audio_file))

    # Sort by audio file name
    file_structs = sorted(file_structs,
                          key=lambda file_struct: file_struct.audio_file)

    return file_structs


Instruction:
Write a Python 3 function for
reading hierarchical references from a jams file.

Response:
def read_hier_references(jams_file, annotation_id=0, exclude_levels=[]):
    """Reads hierarchical references from a jams file.

    Parameters
    ----------
    jams_file : str
        Path to the jams file.
    annotation_id : int > 0
        Identifier of the annotator to read from.
    exclude_levels: list
        List of levels to exclude. Empty list to include all levels.

    Returns
    -------
    hier_bounds : list
        List of the segment boundary times in seconds for each level.
    hier_labels : list
        List of the segment labels for each level.
    hier_levels : list
        List of strings for the level identifiers.
    """
    hier_bounds = []
    hier_labels = []
    hier_levels = []
    jam = jams.load(jams_file)
    namespaces = ["segment_salami_upper", "segment_salami_function",
                  "segment_open", "segment_tut", "segment_salami_lower"]

    # Remove levels if needed
    for exclude in exclude_levels:
        if exclude in namespaces:
            namespaces.remove(exclude)

    # Build hierarchy references
    for ns in namespaces:
        ann = jam.search(namespace=ns)
        if not ann:
            continue
        ref_inters, ref_labels = ann[annotation_id].to_interval_values()
        hier_bounds.append(utils.intervals_to_times(ref_inters))
        hier_labels.append(ref_labels)
        hier_levels.append(ns)

    return hier_bounds, hier_labels, hier_levels


Instruction:
Explain what the following Python 3 code does
def get_duration(features_file):
    with open(features_file) as f:
        feats = json.load(f)
    return float(feats["globals"]["dur"])

Response: Reads the duration of a given features file.


Instruction:
Explain what the following Python 3 code does
def write_mirex(times, labels, out_file):
    inters = msaf.utils.times_to_intervals(times)
    assert len(inters) == len(labels)
    out_str = ""
    for inter, label in zip(inters, labels):
        out_str += "%.3f\t%.3f\t%s\n" % (inter[0], inter[1], label)
    with open(out_file, "w") as f:
        f.write(out_str[:-1])

Response: Writes the results to a MIREX file.


Instruction:
Explain what the following Python 3 code does
def _get_dataset_file(self, dir, ext):
        audio_file_ext = "." + self.audio_file.split(".")[-1]
        base_file = os.path.basename(self.audio_file).replace(
            audio_file_ext, ext)
        return os.path.join(self.ds_path, dir, base_file)

Response: Gets the desired dataset file.


Instruction:
Explain what the following Python 3 code does
def processFlat(self):
        # Preprocess to obtain features (array(n_frames, n_features))
        F = self._preprocess()

        # Do something with the default parameters
        # (these are defined in the in the config.py file).
        assert self.config["my_param1"] == 1.0

        # Identify boundaries in frame indeces with the new algorithm
        my_bounds = np.array([0, F.shape[0] - 1])

        # Label the segments (use -1 to have empty segments)
        my_labels = np.ones(len(my_bounds) - 1) * -1

        # Post process estimations
        est_idxs, est_labels = self._postprocess(my_bounds, my_labels)

        # We're done!
        return est_idxs, est_labels

Response: Main function for processing the flat data.


Instruction:
Write a Python 3 script to
load a ground - truth segmentation and align times to the nearest detected beats.

Response:
def align_segmentation(beat_times, song):
    '''Load a ground-truth segmentation, and align times to the nearest
    detected beats.

    Arguments:
        beat_times -- array
        song -- path to the audio file

    Returns:
        segment_beats -- array
            beat-aligned segment boundaries

        segment_times -- array
            true segment times

        segment_labels -- array
            list of segment labels
    '''
    try:
        segment_times, segment_labels = msaf.io.read_references(song)
    except:
        return None, None, None
    segment_times = np.asarray(segment_times)

    # Map to intervals
    segment_intervals = msaf.utils.times_to_intervals(segment_times)

    # Map beats to intervals
    beat_intervals = np.asarray(zip(beat_times[:-1], beat_times[1:]))

    # Map beats to segments
    beat_segment_ids = librosa.util.match_intervals(beat_intervals,
                                                    segment_intervals)

    segment_beats = []
    segment_times_out = []
    segment_labels_out = []

    # print segment_times, beat_segment_ids, len(beat_times),
    # len(beat_segment_ids)
    for i in range(segment_times.shape[0]):
        hits = np.argwhere(beat_segment_ids == i)
        if len(hits) > 0 and i < len(segment_intervals) and \
                i < len(segment_labels):
            segment_beats.extend(hits[0])
            segment_times_out.append(segment_intervals[i, :])
            segment_labels_out.append(segment_labels[i])

    # Pull out the segment start times
    segment_beats = list(segment_beats)
    # segment_times_out = np.asarray(
    #  segment_times_out)[:, 0].squeeze().reshape((-1, 1))

    # if segment_times_out.ndim == 0:
    #    segment_times_out = segment_times_out[np.newaxis]
    segment_times_out = segment_times

    return segment_beats, segment_times_out, segment_labels_out


Instruction:
Implement a function in Python 3 to
estimate the beats using librosa.

Response:
def estimate_beats(self):
        """Estimates the beats using librosa.

        Returns
        -------
        times: np.array
            Times of estimated beats in seconds.
        frames: np.array
            Frame indeces of estimated beats.
        """
        # Compute harmonic-percussive source separation if needed
        if self._audio_percussive is None:
            self._audio_harmonic, self._audio_percussive = self.compute_HPSS()

        # Compute beats
        tempo, frames = librosa.beat.beat_track(
            y=self._audio_percussive, sr=self.sr,
            hop_length=self.hop_length)

        # To times
        times = librosa.frames_to_time(frames, sr=self.sr,
                                       hop_length=self.hop_length)

        # TODO: Is this really necessary?
        if len(times) > 0 and times[0] == 0:
            times = times[1:]
            frames = frames[1:]

        return times, frames


Instruction:
Given the following Python 3 function, write the documentation
def read_ann_beats(self):
        times, frames = (None, None)

        # Read annotations if they exist in correct folder
        if os.path.isfile(self.file_struct.ref_file):
            try:
                jam = jams.load(self.file_struct.ref_file)
            except TypeError:
                logging.warning(
                    "Can't read JAMS file %s. Maybe it's not "
                    "compatible with current JAMS version?" %
                    self.file_struct.ref_file)
                return times, frames
            beat_annot = jam.search(namespace="beat.*")

            # If beat annotations exist, get times and frames
            if len(beat_annot) > 0:
                beats_inters, _ = beat_annot[0].to_interval_values()
                times = beats_inters[:, 0]
                frames = librosa.time_to_frames(times, sr=self.sr,
                                                hop_length=self.hop_length)
        return times, frames

Response: Reads the annotated beats if available.


Instruction:
Given the following Python 3 function, write the documentation
def compute_beat_sync_features(self, beat_frames, beat_times, pad):
        if beat_frames is None:
            return None, None

        # Make beat synchronous
        beatsync_feats = librosa.util.utils.sync(self._framesync_features.T,
                                                 beat_frames, pad=pad).T

        # Assign times (and add last time if padded)
        beatsync_times = np.copy(beat_times)
        if beatsync_times.shape[0] != beatsync_feats.shape[0]:
            beatsync_times = np.concatenate((beatsync_times,
                                             [self._framesync_times[-1]]))
        return beatsync_feats, beatsync_times

Response: Compute the features for the beat - synchronous.


Instruction:
Make a summary of the following Python 3 code
def read_features(self, tol=1e-3):
        try:
            # Read JSON file
            with open(self.file_struct.features_file) as f:
                feats = json.load(f)

            # Store duration
            if self.dur is None:
                self.dur = float(feats["globals"]["dur"])

            # Check that we have the correct global parameters
            assert(np.isclose(
                self.dur, float(feats["globals"]["dur"]), rtol=tol))
            assert(self.sr == int(feats["globals"]["sample_rate"]))
            assert(self.hop_length == int(feats["globals"]["hop_length"]))
            assert(os.path.basename(self.file_struct.audio_file) ==
                   os.path.basename(feats["globals"]["audio_file"]))

            # Check for specific features params
            feat_params_err = FeatureParamsError(
                "Couldn't find features for %s id in file %s" %
                (self.get_id(), self.file_struct.features_file))
            if self.get_id() not in feats.keys():
                raise feat_params_err
            for param_name in self.get_param_names():
                value = getattr(self, param_name)
                if hasattr(value, '__call__'):
                    # Special case of functions
                    if value.__name__ != \
                            feats[self.get_id()]["params"][param_name]:
                        raise feat_params_err
                else:
                    if str(value) != \
                            feats[self.get_id()]["params"][param_name]:
                        raise feat_params_err

            # Store actual features
            self._est_beats_times = np.array(feats["est_beats"])
            self._est_beatsync_times = np.array(feats["est_beatsync_times"])
            self._est_beats_frames = librosa.core.time_to_frames(
                self._est_beats_times, sr=self.sr, hop_length=self.hop_length)
            self._framesync_features = \
                np.array(feats[self.get_id()]["framesync"])
            self._est_beatsync_features = \
                np.array(feats[self.get_id()]["est_beatsync"])

            # Read annotated beats if available
            if "ann_beats" in feats.keys():
                self._ann_beats_times = np.array(feats["ann_beats"])
                self._ann_beatsync_times = np.array(feats["ann_beatsync_times"])
                self._ann_beats_frames = librosa.core.time_to_frames(
                    self._ann_beats_times, sr=self.sr,
                    hop_length=self.hop_length)
                self._ann_beatsync_features = \
                    np.array(feats[self.get_id()]["ann_beatsync"])
        except KeyError:
            raise WrongFeaturesFormatError(
                "The features file %s is not correctly formatted" %
                self.file_struct.features_file)
        except AssertionError:
            raise FeaturesNotFound(
                "The features for the given parameters were not found in "
                "features file %s" % self.file_struct.features_file)
        except IOError:
            raise NoFeaturesFileError("Could not find features file %s",
                                      self.file_struct.features_file)

Response:
Reads the features from a file and stores them in the current
            object.


Instruction:
Explain what the following Python 3 code does
def write_features(self):
        out_json = collections.OrderedDict()
        try:
            # Only save the necessary information
            self.read_features()
        except (WrongFeaturesFormatError, FeaturesNotFound,
                NoFeaturesFileError):
            # We need to create the file or overwite it
            # Metadata
            out_json = collections.OrderedDict({"metadata": {
                "versions": {"librosa": librosa.__version__,
                             "msaf": msaf.__version__,
                             "numpy": np.__version__},
                "timestamp": datetime.datetime.today().strftime(
                    "%Y/%m/%d %H:%M:%S")}})

            # Global parameters
            out_json["globals"] = {
                "dur": self.dur,
                "sample_rate": self.sr,
                "hop_length": self.hop_length,
                "audio_file": self.file_struct.audio_file
            }

            # Beats
            out_json["est_beats"] = self._est_beats_times.tolist()
            out_json["est_beatsync_times"] = self._est_beatsync_times.tolist()
            if self._ann_beats_times is not None:
                out_json["ann_beats"] = self._ann_beats_times.tolist()
                out_json["ann_beatsync_times"] = self._ann_beatsync_times.tolist()
        except FeatureParamsError:
            # We have other features in the file, simply add these ones
            with open(self.file_struct.features_file) as f:
                out_json = json.load(f)
        finally:
            # Specific parameters of the current features
            out_json[self.get_id()] = {}
            out_json[self.get_id()]["params"] = {}
            for param_name in self.get_param_names():
                value = getattr(self, param_name)
                # Check for special case of functions
                if hasattr(value, '__call__'):
                    value = value.__name__
                else:
                    value = str(value)
                out_json[self.get_id()]["params"][param_name] = value

            # Actual features
            out_json[self.get_id()]["framesync"] = \
                self._framesync_features.tolist()
            out_json[self.get_id()]["est_beatsync"] = \
                self._est_beatsync_features.tolist()
            if self._ann_beatsync_features is not None:
                out_json[self.get_id()]["ann_beatsync"] = \
                    self._ann_beatsync_features.tolist()

            # Save it
            with open(self.file_struct.features_file, "w") as f:
                json.dump(out_json, f, indent=2)

Response: Saves the current features to a JSON file.


Instruction:
How would you implement a function in Python 3 that
returns the parameter names for these features avoiding the global parameters.

Response:
def get_param_names(self):
        """Returns the parameter names for these features, avoiding
        the global parameters."""
        return [name for name in vars(self) if not name.startswith('_') and
                name not in self._global_param_names]


Instruction:
Given the following Python 3 function, write the documentation
def _compute_framesync_times(self):
        self._framesync_times = librosa.core.frames_to_time(
            np.arange(self._framesync_features.shape[0]), self.sr,
            self.hop_length)

Response: Computes the framesync times based on the framesync features.


Instruction:
Can you implement a function in Python 3 that
computes all the features for all the beatsync and framesync files from the audio file.

Response:
def _compute_all_features(self):
        """Computes all the features (beatsync, framesync) from the audio."""
        # Read actual audio waveform
        self._audio, _ = librosa.load(self.file_struct.audio_file,
                                      sr=self.sr)

        # Get duration of audio file
        self.dur = len(self._audio) / float(self.sr)

        # Compute actual features
        self._framesync_features = self.compute_features()

        # Compute framesync times
        self._compute_framesync_times()

        # Compute/Read beats
        self._est_beats_times, self._est_beats_frames = self.estimate_beats()
        self._ann_beats_times, self._ann_beats_frames = self.read_ann_beats()

        # Beat-Synchronize
        pad = True  # Always append to the end of the features
        self._est_beatsync_features, self._est_beatsync_times = \
            self.compute_beat_sync_features(self._est_beats_frames,
                                            self._est_beats_times, pad)
        self._ann_beatsync_features, self._ann_beatsync_times = \
            self.compute_beat_sync_features(self._ann_beats_frames,
                                            self._ann_beats_times, pad)