Instruction:
How would you explain what the following Python 3 function does
def find_contamination(pair, output_folder, databases_folder, forward_id='_R1', threads=1, keep_files=False,
                       quality_cutoff=20, base_cutoff=2, base_fraction_cutoff=0.05, cgmlst_db=None, Xmx=None, tmpdir=None,
                       data_type='Illumina', use_rmlst=False):
    """
    This needs some documentation fairly badly, so here we go.
    :param pair: This has become a misnomer. If the input reads are actually paired, needs to be a list
    with the full filepath to forward reads at index 0 and full path to reverse reads at index 1.
    If reads are unpaired, should be a list of length 1 with the only entry being the full filepath to read set.
    :param output_folder: Folder where outputs (confindr log and report, and other stuff) will be stored.
    This will be created if it does not exist. (I think - should write a test that double checks this).
    :param databases_folder: Full path to folder where ConFindr's databases live. These files can be
    downloaded from figshare in .tar.gz format (https://ndownloader.figshare.com/files/11864267), and
    will be automatically downloaded if the script is run from the command line.
    :param forward_id: Identifier that marks reads as being in the forward direction for paired reads.
    Defaults to _R1
    :param threads: Number of threads to run analyses with. All parts of this pipeline scale pretty well,
    so more is better.
    :param keep_files: Boolean that says whether or not to keep temporary files.
    :param quality_cutoff: Integer of the phred score required to have a base count towards a multiallelic site.
    :param base_cutoff: Integer of number of bases needed to have a base be part of a multiallelic site.
    :param base_fraction_cutoff: Float of fraction of bases needed to have a base be part of a multiallelic site.
    If specified will be used in parallel with base_cutoff
    :param cgmlst_db: if None, we're using rMLST, if True, using some sort of custom cgMLST database. This requires some
    custom parameters.
    :param Xmx: if None, BBTools will use auto memory detection. If string, BBTools will use what's specified as their
    memory request.
    :param tmpdir: if None, any genus-specifc databases that need to be created will be written to ConFindr DB location.
    Otherwise, genus-specific databases will be written here.
    """
    if os.path.isfile(os.path.join(databases_folder, 'download_date.txt')):
        with open(os.path.join(databases_folder, 'download_date.txt')) as f:
            database_download_date = f.readline().rstrip()
    else:
        database_download_date = 'NA'
    log = os.path.join(output_folder, 'confindr_log.txt')
    if len(pair) == 2:
        sample_name = os.path.split(pair[0])[-1].split(forward_id)[0]
        paired = True
        logging.debug('Sample is paired. Sample name is {}'.format(sample_name))
    else:
        sample_name = os.path.split(pair[0])[-1].split('.')[0]
        paired = False
        logging.debug('Sample is unpaired. Sample name is {}'.format(sample_name))
    sample_tmp_dir = os.path.join(output_folder, sample_name)
    if not os.path.isdir(sample_tmp_dir):
        os.makedirs(sample_tmp_dir)

    logging.info('Checking for cross-species contamination...')
    if paired:
        genus = find_cross_contamination(databases_folder, pair, tmpdir=sample_tmp_dir, log=log, threads=threads)
    else:
        genus = find_cross_contamination_unpaired(databases_folder, reads=pair[0], tmpdir=sample_tmp_dir, log=log, threads=threads)
    if len(genus.split(':')) > 1:
        write_output(output_report=os.path.join(output_folder, 'confindr_report.csv'),
                     sample_name=sample_name,
                     multi_positions=0,
                     genus=genus,
                     percent_contam='NA',
                     contam_stddev='NA',
                     total_gene_length=0,
                     database_download_date=database_download_date)
        logging.info('Found cross-contamination! Skipping rest of analysis...\n')
        if keep_files is False:
            shutil.rmtree(sample_tmp_dir)
        return
    # Setup genus-specific databases, if necessary.
    if cgmlst_db is not None:
        # Sanity check that the DB specified is actually a file, otherwise, quit with appropriate error message.
        if not os.path.isfile(cgmlst_db):
            logging.error('ERROR: Specified cgMLST file ({}) does not exist. Please check the path and try again.'.format(cgmlst_db))
            quit(code=1)
        sample_database = cgmlst_db
    else:
        db_folder = databases_folder if tmpdir is None else tmpdir
        if not os.path.isdir(db_folder):
            os.makedirs(db_folder)
        if genus != 'NA':
            # Logic here is as follows: users can either have both rMLST databases, which cover all of bacteria,
            # cgmlst-derived databases, which cover only Escherichia, Salmonella, and Listeria (may add more at some
            # point), or they can have both. They can also set priority to either always use rMLST, or to use my
            # core-genome derived stuff and fall back on rMLST if they're trying to look at a genus I haven't created
            # a scheme for.

            # In the event rmlst databases have priority, always use them.
            if use_rmlst is True:
                sample_database = os.path.join(db_folder, '{}_db.fasta'.format(genus))
                if not os.path.isfile(sample_database):
                    if os.path.isfile(os.path.join(db_folder, 'gene_allele.txt')) and os.path.isfile(os.path.join(db_folder, 'rMLST_combined.fasta')):
                        logging.info('Setting up genus-specific database for genus {}...'.format(genus))
                        allele_list = find_genusspecific_allele_list(os.path.join(db_folder, 'gene_allele.txt'), genus)
                        setup_allelespecific_database(fasta_file=sample_database,
                                                      database_folder=db_folder,
                                                      allele_list=allele_list)
            else:
                # Check if a cgderived database is available. If not, try to use rMLST database.
                sample_database = os.path.join(db_folder, '{}_db_cgderived.fasta'.format(genus))
                if not os.path.isfile(sample_database):
                    sample_database = os.path.join(db_folder, '{}_db.fasta'.format(genus))
                    # Create genus specific database if it doesn't already exist and we have the necessary rMLST files.
                    if os.path.isfile(os.path.join(db_folder, 'rMLST_combined.fasta')) and os.path.isfile(os.path.join(db_folder, 'gene_allele.txt')) and not os.path.isfile(sample_database):
                        logging.info('Setting up genus-specific database for genus {}...'.format(genus))
                        allele_list = find_genusspecific_allele_list(os.path.join(db_folder, 'gene_allele.txt'), genus)
                        setup_allelespecific_database(fasta_file=sample_database,
                                                      database_folder=db_folder,
                                                      allele_list=allele_list)

        else:
            sample_database = os.path.join(db_folder, 'rMLST_combined.fasta')

    # If a user has gotten to this point and they don't have any database available to do analysis because
    # they don't have rMLST downloaded and we don't have a cg-derived database available, boot them with a helpful
    # message.
    if not os.path.isfile(sample_database):
        write_output(output_report=os.path.join(output_folder, 'confindr_report.csv'),
                     sample_name=sample_name,
                     multi_positions=0,
                     genus=genus,
                     percent_contam='NA',
                     contam_stddev='NA',
                     total_gene_length=0,
                     database_download_date=database_download_date)
        logging.info('Did not find databases for genus {genus}. You can download the rMLST database to get access to all '
                     'genera (see https://olc-bioinformatics.github.io/ConFindr/install/). Alternatively, if you have a '
                     'high-quality core-genome derived database for your genome of interest, we would be happy to '
                     'add it - open an issue at https://github.com/OLC-Bioinformatics/ConFindr/issues with the '
                     'title "Add genus-specific database: {genus}"\n'.format(genus=genus))
        if keep_files is False:
            shutil.rmtree(sample_tmp_dir)
        return

    # Extract rMLST reads and quality trim.
    logging.info('Extracting conserved core genes...')
    if paired:
        if Xmx is None:
            out, err, cmd = bbtools.bbduk_bait(reference=sample_database,
                                               forward_in=pair[0],
                                               reverse_in=pair[1],
                                               forward_out=os.path.join(sample_tmp_dir, 'rmlst_R1.fastq.gz'),
                                               reverse_out=os.path.join(sample_tmp_dir, 'rmlst_R2.fastq.gz'),
                                               threads=threads,
                                               returncmd=True)
        else:
            out, err, cmd = bbtools.bbduk_bait(reference=sample_database,
                                               forward_in=pair[0],
                                               reverse_in=pair[1],
                                               forward_out=os.path.join(sample_tmp_dir, 'rmlst_R1.fastq.gz'),
                                               reverse_out=os.path.join(sample_tmp_dir, 'rmlst_R2.fastq.gz'),
                                               threads=threads,
                                               Xmx=Xmx,
                                               returncmd=True)
    else:
        if data_type == 'Nanopore':
            forward_out = os.path.join(sample_tmp_dir, 'trimmed.fastq.gz')
        else:
            forward_out = os.path.join(sample_tmp_dir, 'rmlst.fastq.gz')
        if Xmx is None:
            out, err, cmd = bbtools.bbduk_bait(reference=sample_database, forward_in=pair[0],
                                               forward_out=forward_out,
                                               returncmd=True, threads=threads)
        else:
            out, err, cmd = bbtools.bbduk_bait(reference=sample_database, forward_in=pair[0],
                                               forward_out=forward_out, Xmx=Xmx,
                                               returncmd=True, threads=threads)

    write_to_logfile(log, out, err, cmd)
    logging.info('Quality trimming...')
    if data_type == 'Illumina':
        if paired:
            if Xmx is None:
                out, err, cmd = bbtools.bbduk_trim(forward_in=os.path.join(sample_tmp_dir, 'rmlst_R1.fastq.gz'),
                                                   reverse_in=os.path.join(sample_tmp_dir, 'rmlst_R2.fastq.gz'),
                                                   forward_out=os.path.join(sample_tmp_dir, 'trimmed_R1.fastq.gz'),
                                                   reverse_out=os.path.join(sample_tmp_dir, 'trimmed_R2.fastq.gz'),
                                                   threads=str(threads), returncmd=True)
            else:
                out, err, cmd = bbtools.bbduk_trim(forward_in=os.path.join(sample_tmp_dir, 'rmlst_R1.fastq.gz'),
                                                   reverse_in=os.path.join(sample_tmp_dir, 'rmlst_R2.fastq.gz'),
                                                   forward_out=os.path.join(sample_tmp_dir, 'trimmed_R1.fastq.gz'),
                                                   reverse_out=os.path.join(sample_tmp_dir, 'trimmed_R2.fastq.gz'), Xmx=Xmx,
                                                   threads=str(threads), returncmd=True)

        else:
            if Xmx is None:
                out, err, cmd = bbtools.bbduk_trim(forward_in=os.path.join(sample_tmp_dir, 'rmlst.fastq.gz'),
                                                   forward_out=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                                   returncmd=True, threads=threads)
            else:
                out, err, cmd = bbtools.bbduk_trim(forward_in=os.path.join(sample_tmp_dir, 'rmlst.fastq.gz'),
                                                   forward_out=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                                   returncmd=True, threads=threads, Xmx=Xmx)
        write_to_logfile(log, out, err, cmd)

    logging.info('Detecting contamination...')
    # Now do mapping in two steps - first, map reads back to database with ambiguous reads matching all - this
    # will be used to get a count of number of reads aligned to each gene/allele so we can create a custom rmlst file
    # with only the most likely allele for each gene.
    if not os.path.isfile(sample_database + '.fai'):  # Don't bother re-indexing, this only needs to happen once.
        pysam.faidx(sample_database)
    kma_database = sample_database.replace('.fasta', '') + '_kma'
    kma_report = os.path.join(sample_tmp_dir, 'kma_rmlst')
    if not os.path.isfile(kma_database + '.name'):  # The .name is one of the files KMA creates when making a database.
        cmd = 'kma index -i {} -o {}'.format(sample_database, kma_database)  # NOTE: Need KMA >=1.2.0 for this to work.
        out, err = run_cmd(cmd)
        write_to_logfile(log, out, err, cmd)

    # Run KMA.
    if paired:
        cmd = 'kma -ipe {forward_in} {reverse_in} -t_db {kma_database} -o {kma_report} ' \
              '-t {threads}'.format(forward_in=os.path.join(sample_tmp_dir, 'trimmed_R1.fastq.gz'),
                                    reverse_in=os.path.join(sample_tmp_dir, 'trimmed_R2.fastq.gz'),
                                    kma_database=kma_database,
                                    kma_report=kma_report,
                                    threads=threads)
        out, err = run_cmd(cmd)
        write_to_logfile(log, out, err, cmd)
    else:
        if data_type == 'Illumina':
            cmd = 'kma -i {input_reads} -t_db {kma_database} -o {kma_report} ' \
                  '-t {threads}'.format(input_reads=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                        kma_database=kma_database,
                                        kma_report=kma_report,
                                        threads=threads)
        else:
            # Recommended Nanopore settings from KMA repo: https://bitbucket.org/genomicepidemiology/kma
            cmd = 'kma -i {input_reads} -t_db {kma_database} -o {kma_report} -mem_mode -mp 20 -mrs 0.0 -bcNano ' \
                  '-t {threads}'.format(input_reads=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                        kma_database=kma_database,
                                        kma_report=kma_report,
                                        threads=threads)
        out, err = run_cmd(cmd)
        write_to_logfile(log, out, err, cmd)

    rmlst_report = os.path.join(output_folder, sample_name + '_rmlst.csv')
    gene_alleles = find_rmlst_type(kma_report=kma_report + '.res',
                                   rmlst_report=rmlst_report)

    with open(os.path.join(sample_tmp_dir, 'rmlst.fasta'), 'w') as f:
        for contig in SeqIO.parse(sample_database, 'fasta'):
            if contig.id in gene_alleles:
                f.write('>{}\n'.format(contig.id))
                f.write(str(contig.seq) + '\n')

    rmlst_gene_length = find_total_sequence_length(os.path.join(sample_tmp_dir, 'rmlst.fasta'))
    logging.debug('Total gene length is {}'.format(rmlst_gene_length))
    # Second step of mapping - Do a mapping of our baited reads against a fasta file that has only one allele per
    # rMLST gene.
    pysam.faidx(os.path.join(sample_tmp_dir, 'rmlst.fasta'))
    if paired:
        cmd = 'bbmap.sh ref={ref} in={forward_in} in2={reverse_in} out={outbam} threads={threads} mdtag ' \
              'nodisk'.format(ref=os.path.join(sample_tmp_dir, 'rmlst.fasta'),
                              forward_in=os.path.join(sample_tmp_dir, 'trimmed_R1.fastq.gz'),
                              reverse_in=os.path.join(sample_tmp_dir, 'trimmed_R2.fastq.gz'),
                              outbam=os.path.join(sample_tmp_dir, 'out_2.bam'),
                              threads=threads)
        if cgmlst_db is not None:
            # Lots of core genes seem to have relatives within a genome that are at ~70 percent identity - this means
            # that reads that shouldn't really map do, and cause false positives. Adding in this subfilter means that
            # reads can only have one mismatch, so they actually have to be from the right gene for this to work.
            cmd += ' subfilter=1'
        if Xmx:
            cmd += ' -Xmx{}'.format(Xmx)
        out, err = run_cmd(cmd)
        write_to_logfile(log, out, err, cmd)
    else:
        if data_type == 'Illumina':
            cmd = 'bbmap.sh ref={ref} in={forward_in} out={outbam} threads={threads} mdtag ' \
                  'nodisk'.format(ref=os.path.join(sample_tmp_dir, 'rmlst.fasta'),
                                  forward_in=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                  outbam=os.path.join(sample_tmp_dir, 'out_2.bam'),
                                  threads=threads)
            if cgmlst_db is not None:
                # Lots of core genes seem to have relatives within a genome that are at ~70 percent identity - this means
                # that reads that shouldn't really map do, and cause false positives. Adding in this subfilter means that
                # reads can only have one mismatch, so they actually have to be from the right gene for this to work.
                cmd += ' subfilter=1'
            if Xmx:
                cmd += ' -Xmx{}'.format(Xmx)
            out, err = run_cmd(cmd)
            write_to_logfile(log, out, err, cmd)
        else:
            cmd = 'minimap2 --MD -t {threads} -ax map-ont {ref} {reads} ' \
                  '> {outsam}'.format(ref=os.path.join(sample_tmp_dir, 'rmlst.fasta'),
                                      reads=os.path.join(sample_tmp_dir, 'trimmed.fastq.gz'),
                                      outsam=os.path.join(sample_tmp_dir, 'out_2.sam'),
                                      threads=threads)
            out, err = run_cmd(cmd)
            write_to_logfile(log, out, err, cmd)
            outbam = os.path.join(sample_tmp_dir, 'out_2.bam')
            # Apparently have to perform equivalent of a touch on this file for this to work.
            fh = open(outbam, 'w')
            fh.close()
            pysam.view('-b', '-o', outbam, os.path.join(sample_tmp_dir, 'out_2.sam'), save_stdout=outbam)
    pysam.sort('-o', os.path.join(sample_tmp_dir, 'contamination.bam'), os.path.join(sample_tmp_dir, 'out_2.bam'))
    pysam.index(os.path.join(sample_tmp_dir, 'contamination.bam'))
    # Now find number of multi-positions for each rMLST gene/allele combination
    multi_positions = 0

    # Run the BAM parsing in parallel! Some refactoring of the code would likely be a good idea so this
    # isn't quite so ugly, but it works.
    p = multiprocessing.Pool(processes=threads)
    bamfile_list = [os.path.join(sample_tmp_dir, 'contamination.bam')] * len(gene_alleles)
    # bamfile_list = [os.path.join(sample_tmp_dir, 'rmlst.bam')] * len(gene_alleles)
    reference_fasta_list = [os.path.join(sample_tmp_dir, 'rmlst.fasta')] * len(gene_alleles)
    quality_cutoff_list = [quality_cutoff] * len(gene_alleles)
    base_cutoff_list = [base_cutoff] * len(gene_alleles)
    base_fraction_list = [base_fraction_cutoff] * len(gene_alleles)
    multibase_dict_list = list()
    report_write_list = list()
    for multibase_dict, report_write in p.starmap(read_contig, zip(gene_alleles, bamfile_list, reference_fasta_list, quality_cutoff_list, base_cutoff_list, base_fraction_list), chunksize=1):
        multibase_dict_list.append(multibase_dict)
        report_write_list.append(report_write)
    p.close()
    p.join()

    # Write out report info.
    report_file = os.path.join(output_folder, sample_name + '_contamination.csv')
    with open(report_file, 'w') as r:
        r.write('{reference},{position},{bases},{coverage}\n'.format(reference='Gene',
                                                                     position='Position',
                                                                     bases='Bases',
                                                                     coverage='Coverage'))
        for item in report_write_list:
            for contamination_info in item:
                r.write(contamination_info)

    # Total up the number of multibase positions.
    for multibase_position_dict in multibase_dict_list:
        multi_positions += sum([len(snp_positions) for snp_positions in multibase_position_dict.values()])
    if cgmlst_db is None:
        snp_cutoff = int(rmlst_gene_length/10000) + 1
    else:
        snp_cutoff = 10
    if multi_positions >= snp_cutoff:
        percent_contam, contam_stddev = estimate_percent_contamination(contamination_report_file=report_file)
    else:
        percent_contam = 0
        contam_stddev = 0
    logging.info('Done! Number of contaminating SNVs found: {}\n'.format(multi_positions))
    write_output(output_report=os.path.join(output_folder, 'confindr_report.csv'),
                 sample_name=sample_name,
                 multi_positions=multi_positions,
                 genus=genus,
                 percent_contam=percent_contam,
                 contam_stddev=contam_stddev,
                 total_gene_length=rmlst_gene_length,
                 snp_cutoff=snp_cutoff,
                 cgmlst=cgmlst_db,
                 database_download_date=database_download_date)
    if keep_files is False:
        shutil.rmtree(sample_tmp_dir)

Response: This function finds the contamination of a pair of reads.


Instruction:
Create a Python 3 function for
functioning that writes the output of ConFindr to a CSV file.

Response:
def write_output(output_report, sample_name, multi_positions, genus, percent_contam, contam_stddev, total_gene_length,
                 database_download_date, snp_cutoff=3, cgmlst=None):
    """
    Function that writes the output generated by ConFindr to a report file. Appends to a file that already exists,
    or creates the file if it doesn't already exist.
    :param output_report: Path to CSV output report file. Should have headers SampleName,Genus,NumContamSNVs,
    ContamStatus,PercentContam, and PercentContamStandardDeviation, in that order.
    :param sample_name: string - name of sample
    :param multi_positions: integer - number of positions that were found to have more than one base present.
    :param genus: string - The genus of your sample
    :param percent_contam: float - Estimated percentage contamination
    :param contam_stddev: float - Standard deviation of percentage contamination
    :param total_gene_length: integer - number of bases examined to make a contamination call.
    :param cgmlst: If None, means that rMLST database was used, so use rMLST snp cutoff. Otherwise, some sort of cgMLST
    database was used, so use a different cutoff.
    """
    # If the report file hasn't been created, make it, with appropriate header.
    if not os.path.isfile(output_report):
        with open(os.path.join(output_report), 'w') as f:
            f.write('Sample,Genus,NumContamSNVs,ContamStatus,PercentContam,PercentContamStandardDeviation,BasesExamined,DatabaseDownloadDate\n')

    if multi_positions >= snp_cutoff or len(genus.split(':')) > 1:
        contaminated = True
    else:
        contaminated = False
    with open(output_report, 'a+') as f:
        f.write('{samplename},{genus},{numcontamsnvs},'
                '{contamstatus},{percent_contam},{contam_stddev},'
                '{gene_length},{database_download_date}\n'.format(samplename=sample_name,
                                                                  genus=genus,
                                                                  numcontamsnvs=multi_positions,
                                                                  contamstatus=contaminated,
                                                                  percent_contam=percent_contam,
                                                                  contam_stddev=contam_stddev,
                                                                  gene_length=total_gene_length,
                                                                  database_download_date=database_download_date))


Instruction:
Can you tell what is the following Python 3 function doing
def check_acceptable_xmx(xmx_string):
    acceptable_xmx = True
    acceptable_suffixes = ['K', 'M', 'G']
    if xmx_string[-1].upper() not in acceptable_suffixes:
        acceptable_xmx = False
        logging.error('ERROR: Memory must be specified as K (kilobytes), M (megabytes), or G (gigabytes). Your specified '
                      'suffix was {}.'.format(xmx_string[-1]))
    if '.' in xmx_string:
        acceptable_xmx = False
        logging.error('ERROR: Xmx strings must be integers, floating point numbers are not accepted.')
    if not str.isdigit(xmx_string[:-1]):
        acceptable_xmx = False
        logging.error('ERROR: The amount of memory requested was not an integer.')
    return acceptable_xmx

Response: Checks that the XMX string is acceptable by BBTools.


Instruction:
How would you explain what the following Python 3 function does
def char_code(columns, name=None):
    if name is None:
        name = 'Char Code Field (' + str(columns) + ' columns)'

    if columns <= 0:
        raise BaseException()

    char_sets = None
    for char_set in _tables.get_data('character_set'):
        regex = '[ ]{' + str(15 - len(char_set)) + '}' + char_set
        if char_sets is None:
            char_sets = regex
        else:
            char_sets += '|' + regex

    # Accepted sets
    _character_sets = pp.Regex(char_sets)
    _unicode_1_16b = pp.Regex('U\+0[0-8,A-F]{3}[ ]{' + str(columns - 6) + '}')
    _unicode_2_21b = pp.Regex('U\+0[0-8,A-F]{4}[ ]{' + str(columns - 7) + '}')

    # Basic field
    char_code_field = (_character_sets | _unicode_1_16b | _unicode_2_21b)

    # Parse action
    char_code_field = char_code_field.setParseAction(lambda s: s[0].strip())

    # Name
    char_code_field.setName(name)

    return char_code_field

Response: Returns an instance of the Character set code field.


Instruction:
Can you tell what is the following Python 3 function doing
def make_choice_validator(
        choices, default_key=None, normalizer=None):
    """
    Returns a callable that accepts the choices provided.

    Choices should be provided as a list of 2-tuples, where the first
    element is a string that should match user input (the key); the
    second being the value associated with the key.

    The callable by default will match, upon complete match the first
    value associated with the result will be returned.  Partial matches
    are supported.

    If a default is provided, that value will be returned if the user
    provided input is empty, i.e. the value that is mapped to the empty
    string.

    Finally, a normalizer function can be passed.  This normalizes all
    keys and validation value.
    """

    def normalize_all(_choices):
        # normalize all the keys for easier comparison
        if normalizer:
            _choices = [(normalizer(key), value) for key, value in choices]
        return _choices

    choices = normalize_all(choices)

    def choice_validator(value):
        if normalizer:
            value = normalizer(value)
        if not value and default_key:
            value = choices[default_key][0]
        results = []
        for choice, mapped in choices:
            if value == choice:
                return mapped
            if choice.startswith(value):
                results.append((choice, mapped))
        if len(results) == 1:
            return results[0][1]
        elif not results:
            raise ValueError('Invalid choice.')
        else:
            raise ValueError(
                'Choice ambiguous between (%s)' % ', '.join(
                    k for k, v in normalize_all(results))
            )

    return choice_validator

Response: Creates a callable that accepts the user input.


Instruction:
Write a Python 3 function that can
prompt user for question maybe choices and get answer.

Response:
def prompt(question, validator=None,
           choices=None, default_key=NotImplemented,
           normalizer=str.lower,
           _stdin=None, _stdout=None):
    """
    Prompt user for question, maybe choices, and get answer.

    Arguments:

    question
        The question to prompt.  It will only be prompted once.
    validator
        Defaults to None.  Must be a callable that takes in a value.
        The callable should raise ValueError when the value leads to an
        error, otherwise return a converted value.
    choices
        If choices are provided instead, a validator will be constructed
        using make_choice_validator along with the next default_value
        argument.  Please refer to documentation for that function.
    default_value
        See above.
    normalizer
        Defaults to str.lower.  See above.
    """

    def write_choices(choice_keys, default_key):
        _stdout.write('(')
        _stdout.write('/'.join(choice_keys))
        _stdout.write(') ')
        if default_key is not NotImplemented:
            _stdout.write('[')
            _stdout.write(choice_keys[default_key])
            _stdout.write('] ')

    if _stdin is None:
        _stdin = sys.stdin

    if _stdout is None:
        _stdout = sys.stdout

    _stdout.write(question)
    _stdout.write(' ')

    if not check_interactive():
        if choices and default_key is not NotImplemented:
            choice_keys = [choice for choice, mapped in choices]
            write_choices(choice_keys, default_key)
            display, answer = choices[default_key]
            _stdout.write(display)
            _stdout.write('\n')

            logger.warning(
                'non-interactive mode; auto-selected default option [%s]',
                display)
            return answer
        logger.warning(
            'interactive code triggered within non-interactive session')
        _stdout.write('Aborted.\n')
        return None

    choice_keys = []

    if validator is None:
        if choices:
            validator = make_choice_validator(
                choices, default_key, normalizer)
            choice_keys = [choice for choice, mapped in choices]
        else:
            validator = null_validator

    answer = NotImplemented
    while answer is NotImplemented:
        if choice_keys:
            write_choices(choice_keys, default_key)
        _stdout.flush()
        try:
            answer = validator(
                _stdin.readline().strip().encode(locale).decode(locale))
        except ValueError as e:
            _stdout.write('%s\n' % e)
            _stdout.write(question.splitlines()[-1])
            _stdout.write(' ')
        except KeyboardInterrupt:
            _stdout.write('Aborted.\n')
            answer = None

    return answer


Instruction:
Write a Python 3 script for
prompting end user with a diff of original and new json that may overwrite the file at the target_path.

Response:
def prompt_overwrite_json(original, new, target_path, dumps=json_dumps):
    """
    Prompt end user with a diff of original and new json that may
    overwrite the file at the target_path.  This function only displays
    a confirmation prompt and it is up to the caller to implement the
    actual functionality.  Optionally, a custom json.dumps method can
    also be passed in for output generation.
    """

    # generate compacted ndiff output.
    diff = '\n'.join(l for l in (
        line.rstrip() for line in difflib.ndiff(
            json_dumps(original).splitlines(),
            json_dumps(new).splitlines(),
        ))
        if l[:1] in '?+-' or l[-1:] in '{}' or l[-2:] == '},')
    basename_target = basename(target_path)
    return prompt(
        "Generated '%(basename_target)s' differs with '%(target_path)s'.\n\n"
        "The following is a compacted list of changes required:\n"
        "%(diff)s\n\n"
        "Overwrite '%(target_path)s'?" % locals(),
        choices=(
            ('Yes', True),
            ('No', False),
        ),
        default_key=1,
    )


Instruction:
Can you create a Python 3 function that
locates a single npm package to return its browser or main entry.

Response:
def locate_package_entry_file(working_dir, package_name):
    """
    Locate a single npm package to return its browser or main entry.
    """

    basedir = join(working_dir, 'node_modules', package_name)
    package_json = join(basedir, 'package.json')
    if not exists(package_json):
        logger.debug(
            "could not locate package.json for the npm package '%s' in the "
            "current working directory '%s'; the package may have been "
            "not installed, the build process may fail",
            package_name, working_dir,
        )
        return

    with open(package_json) as fd:
        package_info = json.load(fd)

    if ('browser' in package_info or 'main' in package_info):
        # assume the target file exists because configuration files
        # never lie /s
        return join(
            basedir,
            *(package_info.get('browser') or package_info['main']).split('/')
        )

    index_js = join(basedir, 'index.js')
    if exists(index_js):
        return index_js

    logger.debug(
        "package.json for the npm package '%s' does not contain a main "
        "entry point", package_name,
    )


Instruction:
Can you tell what is the following Python 3 function doing
def render_pictures(context, selection='recent', amount=3):
    pictures = Image.objects.filter(
        folder__id__in=Gallery.objects.filter(is_published=True).values_list(
            'folder__pk', flat=True))
    if selection == 'recent':
        context.update({
            'pictures': pictures.order_by('-uploaded_at')[:amount]
        })
    elif selection == 'random':
        context.update({
            'pictures': pictures.order_by('?')[:amount]
        })
    else:
        return None
    return context

Response: Template tag to render a list of pictures.


Instruction:
How would you explain what the following Python 3 function does
def add_header(self, entry):
        info = entry.split('\t')
        self.n_individuals = len(info)-9
        for i,v in enumerate(info[9:]):
            self.individuals[v] = i
        return self.n_individuals > 0

Response: Parses the VCF Header field and returns the number of samples in the VCF file


Instruction:
Implement a Python 3 function for
parsing an individual VCF entry and return a VCFEntry which contains information about the call.

Response:
def parse_entry(self, row):
        """Parse an individual VCF entry and return a VCFEntry which contains information about
        the call (such as alternative allele, zygosity, etc.)

        """
        var_call = VCFEntry(self.individuals)
        var_call.parse_entry(row)
        return var_call


Instruction:
Here you have a function in Python 3, explain what it does
def get_header(self, individual=-1):
        type_map = dict([(val,key) for key,val in self.meta.type_map.iteritems()])
        extra = '\n'.join(['##{0}'.format(i) for i in self.meta.extra])
        info = '\n'.join(['##INFO=<ID={0},Number={1},Type={2},Description={3}>'.format(key, val.get('num_entries','.'), type_map.get(val.get('type', '')), val.get('description')) for key,val in self.meta.info.iteritems()])
        filter = '\n'.join(['##FILTER=<ID={0},Description={1}>'.format(key, val.get('description','.')) for key,val in self.meta.filter.iteritems()])
        format = '\n'.join(['##FORMAT=<ID={0},Number={1},Type={2},Description={3}>'.format(key, val.get('num_entries','.'), type_map.get(val.get('type', '')), val.get('description')) for key,val in self.meta.format.iteritems()])
        alt = '\n'.join(['##ALT=<ID={0},Description={1}>'.format(key, val.get('description','.')) for key,val in self.meta.alt.iteritems()])
        header = '\t'.join(['#CHROM','POS','ID','REF','ALT','QUAL','FILTER','INFO','FORMAT'])
        if individual is not None:
            if individual == -1:
                individual = '\t'.join(self.individuals.keys())
            else:
                if isinstance(individual, int):
                    for i, v in self.individuals.iteritems():
                        if v == individual:
                            individual = i
                            break
            header += '\t'+individual
        return '\n'.join([extra, info, filter, format, alt, header])

Response: Returns the vcf header


Instruction:
Given the following Python 3 function, write the documentation
def add_info(self, entry):
        entry = entry[8:-1]
        info = entry.split(',')
        if len(info) < 4:
            return False
        for v in info:
            key, value = v.split('=', 1)
            if key == 'ID':
                self.info[value] = {}
                id_ = value
            elif key == 'Number':
                if value == 'A' or value == 'G':
                    value = -1
                self.info[id_]['num_entries'] = value
            elif key == 'Type':
                self.info[id_]['type'] = self.type_map[value]
            elif key == 'Description':
                self.info[id_]['description'] = value
                if len(info) > 4:
                    self.info[id_]['description'] += '; '.join(info[4:])
                break
        return True

Response: Parse and store the info field


Instruction:
Can you generate a brief explanation for the following Python 3 code
def add_filter(self, entry):
        entry = entry[10:-1]
        info = entry.split(',')
        if len(info) < 2:
            return False
        for v in info:
            key, value = v.split('=', 1)
            if key == 'ID':
                self.filter[value] = {}
                id_ = value
            elif key == 'Description':
                self.filter[id_]['description'] = value
                if len(info) > 2:
                    self.info[id_]['description'] += '; '.join(info[2:])
        return True

Response: Parse and store the filter field


Instruction:
Can you write a function in Python 3 where it
parses and store the alternative allele field

Response:
def add_alt(self, entry):
        """Parse and store the alternative allele field"""
        entry = entry[7:-1]
        info = entry.split(',')
        if len(info) < 2:
            return False
        for v in info:
            key, value = v.split('=', 1)
            if key == 'ID':
                self.alt[value] = {}
                id_ = value
            elif key == 'Description':
                self.alt[id_]['description'] = value
                if len(info) > 4:
                    self.alt[id_]['description'] += '; '.join(info[4:])
                break
        return True


Instruction:
Here you have a function in Python 3, explain what it does
def sample_string(self, individual=-1):
        base = str(self)
        extra = self.get_sample_info(individual=individual)
        extra = [':'.join([str(j) for j in i]) for i in zip(*extra.values())]
        return '\t'.join([base, '\t'.join(extra)])

Response: Returns the VCF entry as it appears in the vcf file


Instruction:
Write a Python 3 function that can
return the sample info of a given sample or all by default

Response:
def get_sample_info(self, individual=-1):
        """Returns the sample info of a given sample or all by default

        """
        if isinstance(individual, str):
            individual = self.individuals[individual]
        extra = OrderedDict()
        for format_ in self.format:
            index = getattr(self, format_)
            if index != -1:
                if format_ == 'GT':
                    d = self.genotype
                elif format_ == 'GQ':
                    d = self.genome_quality
                elif format_ == 'DP':
                    d = self.depth
                if individual == -1:
                    if len(d) != len(self.samples):
                        [self.parse_sample(i) for i in six.moves.range(len(self.samples))]
                    extra[format_] = [d[i] for i in six.moves.range(len(d))]
                else:
                    if individual not in d:
                        self.parse_sample(individual)
                    extra[format_] = [d[individual]]
        return extra


Instruction:
Write a Python 3 script to
return the alternative alleles of the individual as a list

Response:
def get_alt(self, individual=0, nucleotides_only=True):
        """Returns the alternative alleles of the individual as a list"""
        #not i.startswith(',') is put in to handle cases like <DEL:ME:ALU> where we have no alternate allele
        #but some reference
        if isinstance(individual, str):
            individual = self.individuals[individual]
        if nucleotides_only:
            return [self.alt[i-1].replace('.', '') for i in self.genotype[individual] if i > 0 and not self.alt[i-1].startswith('<')]
        else:
            return [self.alt[i-1].replace('.', '') for i in self.genotype[individual] if i > 0]


Instruction:
Can you tell what is the following Python 3 function doing
def get_alt_length(self, individual=0):
        if isinstance(individual, str):
            individual = self.individuals[individual]
        return [len(self.alt[i-1].replace('.','')) for i in self.genotype[individual] if i > 0 and not self.alt[i-1].startswith('<')]

Response: Returns the number of basepairs of each alternative allele


Instruction:
Explain what the following Python 3 code does
def get_alt_lengths(self):
        #this is a hack to store the # of individuals without having to actually store it
        out = []
        for i in six.moves.range(len(self.genotype)):
            valid_alt = self.get_alt_length(individual=i)
            if not valid_alt:
                out.append(None)
            else:
                out.append(max(valid_alt)-len(self.ref))
        return out

Response: Returns the longest length of the variant. For deletions returns is negative and insertions are +.


Instruction:
Here you have a function in Python 3, explain what it does
def has_snp(self, individual=0):
        if isinstance(individual, str):
            individual = self.individuals[individual]
        alts = self.get_alt(individual=individual)
        if alts:
            return [i != self.ref and len(i) == len(self.ref) for i in alts]
        return [False]

Response: Returns a boolean list of SNP status ordered by samples


Instruction:
How would you explain what the following Python 3 function does
def parse_entry(self, entry):
        entry = entry.split('\t')
        self.chrom, self.pos, self.id, self.ref, alt_, self.qual, filter_, info, self.format = entry[:9]
        self.samples = entry[9:]
        self.alt = alt_.split(',')
        if filter_ == 'PASS' or filter_ == '.':
            self.passed = True
        else:
            self.passed = filter_.split(';')
        self.info = info
        # currently unused
        #if info != '.':
            #info_l = info.split(';')
            #self.info = [v.split('=') if '=' in v else (v,1) for v in info_l]
        self.format = self.format.split(':')
        if 'GT' in self.format:
            self.GT = self.format.index('GT')
        if 'GQ' in self.format:
            self.GQ = self.format.index('GQ')
        if 'DP' in self.format:
            self.DP = self.format.index('DP')
        if 'FT' in self.format:
            self.FT = self.format.index('FT')

Response: This parses a VCF row and stores the relevant information


Instruction:
Explain what the following Python 3 code does
def add_child(self, child):
        child_id = getattr(child, 'id', None)
        if child_id:
            if not hasattr(self, 'children'):
                self.children = {}
            if child_id not in self.children:
                self.children[child_id] = child

Response: Add a child to the entry.


Instruction:
Can you write a function in Python 3 where it
handles the ISO formatting of an object.

Response:
def _iso_handler(obj):
    """
    Transforms an object into it's ISO format, if possible.

    If the object can't be transformed, then an error is raised for the JSON
    parser.

    This is meant to be used on datetime instances, but will work with any
    object having a method called isoformat.

    :param obj: object to transform into it's ISO format
    :return: the ISO format of the object
    """
    if hasattr(obj, 'isoformat'):
        result = obj.isoformat()
    else:
        raise TypeError("Unserializable object {} of type {}".format(obj,
                                                                     type(obj)))

    return result


Instruction:
Create a Python 3 function for
generating string of cwr format for all possible combinations of fields and entity.

Response:
def encode(self, entity):
        """
        Generate string of cwr format for all possible combinations of fields,
        accumulate and then elect the best. The best string it is who used most of all fields
        :param entity:
        :return:
        """
        possible_results = []
        entity_dict = self.get_entity_dict(entity)
        record_field_encoders = self.get_record_fields_encoders()
        for field_encoders in record_field_encoders:
            result = self.try_encode(field_encoders, entity_dict)
            if result:
                possible_results.append({'result': result, 'len': len(field_encoders)})
        cwr = self.head(entity) + self._get_best_result(possible_results) + "\r\n"
        return cwr


Instruction:
How would you code a function in Python 3 to
return the argparser for this instance.

Response:
def argparser(self):
        """
        For setting up the argparser for this instance.
        """

        if self.__argparser is None:
            self.__argparser = self.argparser_factory()
            self.init_argparser(self.__argparser)
        return self.__argparser


Instruction:
How would you implement a function in Python 3 that
produces an argparser for this type of Runtime.

Response:
def argparser_factory(self):
        """
        Produces argparser for this type of Runtime.
        """

        return ArgumentParser(
            prog=self.prog, description=self.__doc__, add_help=False,
        )


Instruction:
Here you have a function in Python 3, explain what it does
def init_argparser(self, argparser):

        def prepare_argparser():
            if argparser in self.argparser_details:
                return False
            result = self.argparser_details[argparser] = ArgumentParserDetails(
                {}, {}, {})
            return result

        def to_module_attr(ep):
            return '%s:%s' % (ep.module_name, '.'.join(ep.attrs))

        def register(name, runtime, entry_point):
            subparser = commands.add_parser(
                name, help=inst.description,
            )
            # Have to specify this separately because otherwise the
            # subparser will not have a proper description when it is
            # invoked as the root.
            subparser.description = inst.description

            # Assign values for version reporting system
            setattr(subparser, ATTR_ROOT_PKG, getattr(
                argparser, ATTR_ROOT_PKG, self.package_name))
            subp_info = []
            subp_info.extend(getattr(argparser, ATTR_INFO, []))
            subp_info.append((subparser.prog, entry_point.dist))
            setattr(subparser, ATTR_INFO, subp_info)

            try:
                try:
                    runtime.init_argparser(subparser)
                except RuntimeError as e:
                    # first attempt to filter out recursion errors; also if
                    # the stack frame isn't available the complaint about
                    # bad validation doesn't apply anyway.
                    frame = currentframe()
                    if (not frame or 'maximum recursion depth' not in str(
                            e.args)):
                        raise

                    if (not isinstance(runtime, Runtime) or (type(
                            runtime).entry_point_load_validated.__code__ is
                            Runtime.entry_point_load_validated.__code__)):
                        # welp, guess some other thing blew up then, or
                        # that the problem is definitely not caused by
                        # this runtime implementation.
                        # TODO figure out how to log this nicer via the
                        # self.log_debug_error without exploding the
                        # console like Megumin would have done.
                        raise

                    # assume the overridden method didn't do everything
                    # correctly then; would be great if there is a way
                    # to ensure that our thing would have been called.
                    cls = type(runtime)
                    logger.critical(
                        "Runtime subclass at entry_point '%s' has override "
                        "'entry_point_load_validated' without filtering out "
                        "its parent classes; this can be addressed by calling "
                        "super(%s.%s, self).entry_point_load_validated("
                        "entry_point) in its implementation, or simply don't "
                        "override that method to avoid infinite recursion.",
                        entry_point, cls.__module__, cls.__name__,
                    )
                    exc = RuntimeError(
                        "%r has an invalid 'entry_point_load_validated' "
                        "implementation: insufficient protection against "
                        "infinite recursion into self not provided" % runtime
                    )
                    # for Python 3 to not blow it up.
                    exc.__suppress_context__ = True
                    raise exc
            except Exception as e:
                self.log_debug_error(
                    "cannot register entry_point '%s' from '%s' as a "
                    "subcommand to '%s': %s: %s",
                    entry_point, entry_point.dist, argparser.prog,
                    e.__class__.__name__, e
                )
                # this is where naughty things happen: will be poking at
                # the parser internals to undo the damage that was done
                # first, pop the choices_actions as a help was provided
                commands._choices_actions.pop()
                # then pop the name that was mapped.
                commands._name_parser_map.pop(name)
            else:
                # finally record the completely initialized subparser
                # into the structure here if successful.
                subparsers[name] = subparser
                runtimes[name] = runtime
                entry_points[name] = entry_point

        details = prepare_argparser()
        if not details:
            logger.debug(
                'argparser %r has already been initialized against runner %r',
                argparser, self,
            )
            return
        subparsers, runtimes, entry_points = details

        super(Runtime, self).init_argparser(argparser)

        commands = argparser.add_subparsers(
            dest=self.action_key, metavar='<command>')
        # Python 3.7 has required set to True, which is correct in most
        # cases but this disables the manual handling for cases where a
        # command was not provided; also this generates a useless error
        # message that simply states "<command> is required" and forces
        # the program to exit.  As the goal of this suite of classes is
        # to act as a helpful CLI front end, force required to be False
        # to keep our manual handling and management of subcommands.
        # Setting this as a property for compatibility with Python<3.7,
        # as only in Python>=3.7 the add_subparsers can accept required
        # as an argument.
        commands.required = False

        for entry_point in self.iter_entry_points():
            inst = self.entry_point_load_validated(entry_point)
            if not inst:
                continue

            if entry_point.name in runtimes:
                reg_ep = entry_points[entry_point.name]
                reg_rt = runtimes[entry_point.name]

                if reg_rt is inst:
                    # this is fine, multiple packages declared the same
                    # thing with the same name.
                    logger.debug(
                        "duplicated registration of command '%s' via entry "
                        "point '%s' ignored; registered '%s', confict '%s'",
                        entry_point.name, entry_point, reg_ep.dist,
                        entry_point.dist,
                    )
                    continue

                logger.error(
                    "a calmjs runtime command named '%s' already registered.",
                    entry_point.name
                )
                logger.info("conflicting entry points are:")
                logger.info(
                    "'%s' from '%s' (registered)", reg_ep, reg_ep.dist)
                logger.info(
                    "'%s' from '%s' (conflict)", entry_point, entry_point.dist)
                # Fall back name should work if the class/instances are
                # stable.
                name = to_module_attr(entry_point)

                if name in runtimes:
                    # Maybe this is the third time this module is
                    # registered.  Test for its identity.
                    if runtimes[name] is not inst:
                        # Okay someone is having a fun time here mucking
                        # with data structures internal to here, likely
                        # (read hopefully) due to testing or random
                        # monkey patching (or module level reload).
                        logger.critical(
                            "'%s' is already registered but points to a "
                            "completely different instance; please try again "
                            "with verbose logging and note which packages are "
                            "reported as conflicted; alternatively this is a "
                            "forced situation where this Runtime instance has "
                            "been used or initialized improperly.",
                            name
                        )
                    else:
                        logger.debug(
                            "fallback command '%s' is already registered.",
                            name
                        )
                    continue

                logger.error(
                    "falling back to using full instance path '%s' as command "
                    "name, also registering alias for registered command", name
                )
                register(to_module_attr(reg_ep), reg_rt, reg_ep)
            else:
                name = entry_point.name

            register(name, inst, entry_point)

Response: Initialize the argparser for the internal version of the system.


Instruction:
How would you explain what the following Python 3 function does
def unrecognized_arguments_error(self, args, parsed, extras):

        # loop variants
        kwargs = vars(parsed)
        failed = list(extras)
        # initial values
        runtime, subparser, idx = (self, self.argparser, 0)
        # recursion not actually needed when it can be flattened.
        while isinstance(runtime, Runtime):
            cmd = kwargs.pop(runtime.action_key)
            # can happen if it wasn't set, or is set but from a default
            # value (thus not provided by args)
            action_idx = None if cmd not in args else args.index(cmd)
            if cmd not in args and cmd is not None:
                # this normally shouldn't happen, and the test case
                # showed that the parsing will not flip down to the
                # forced default subparser - this can remain a debug
                # message until otherwise.
                logger.debug(
                    "command for prog=%r is set to %r without being specified "
                    "as part of the input arguments - the following error "
                    "message may contain misleading references",
                    subparser.prog, cmd
                )
            subargs = args[idx:action_idx]
            subparsed, subextras = subparser.parse_known_args(subargs)
            if subextras:
                subparser.unrecognized_arguments_error(subextras)
                # since the failed arguments are in order
                failed = failed[len(subextras):]
                if not failed:
                    # have taken everything, quit now.
                    # also note that if cmd was really None it would
                    # cause KeyError below, but fortunately it also
                    # forced action_idx to be None which took all
                    # remaining tokens from failed, so definitely get
                    # out of here.
                    break

            # advance the values
            # note that any internal consistency will almost certainly
            # result in KeyError being raised.
            details = runtime.get_argparser_details(subparser)
            runtime = details.runtimes[cmd]
            subparser = details.subparsers[cmd]
            idx = action_idx + 1

        if failed:
            subparser.unrecognized_arguments_error(failed)
        sys.exit(2)

Response: This function is called by the argparse module to check if unrecognized arguments are present in the arguments list.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def error(self, argparser, target, message):

        warnings.warn(
            'Runtime.error is deprecated and will be removed by calmjs-4.0.0',
            DeprecationWarning)
        details = self.get_argparser_details(argparser)
        argparser = details.subparsers[target] if details else self.argparser
        argparser.error(message)

Response: This method is used to log an error message for the specified target parser.


Instruction:
Can you tell what is the following Python 3 function doing
def init_argparser_export_target(
            self, argparser,
            default=None,
            help='the export target',
            ):
        """
        Subclass could override this by providing alternative keyword
        arguments and call this as its super.  It should not reimplement
        this completely.  Example:

        def init_argparser_export_target(self, argparser):
            super(MyToolchainRuntime, self).init_argparser_export_target(
                argparser, default='my_default.js',
                help="the export target, default is 'my_default.js'",
            )

        Note that the above example will prevent its subclasses from
        directly using the definition of that class, but they _can_
        simply call the exact same super, or invoke ToolchainRuntime's
        init_argparser_* method directly.

        Arguments

        default
            The default export target.
        help
            The help text.
        """

        argparser.add_argument(
            '-w', '--overwrite', dest=EXPORT_TARGET_OVERWRITE,
            action='store_true',
            help='overwrite the export target without any confirmation',
        )

        argparser.add_argument(
            '--export-target', dest=EXPORT_TARGET,
            metavar=metavar(EXPORT_TARGET),
            default=default,
            help=help,
        )

Response: This method is used to initialize the argument parser for the specific locale.


Instruction:
Can you write a function in Python 3 where it
initializes the argument parser for the working directory.

Response:
def init_argparser_working_dir(
            self, argparser,
            explanation='',
            help_template=(
                'the working directory; %(explanation)s'
                'default is current working directory (%(cwd)s)'),
            ):
        """
        Subclass could an extra expanation on how this is used.

        Arguments

        explanation
            Explanation text for the default help template
        help_template
            A standard help message for this option.
        """

        cwd = self.toolchain.join_cwd()
        argparser.add_argument(
            '--working-dir', dest=WORKING_DIR,
            metavar=metavar(WORKING_DIR),
            default=cwd,
            help=help_template % {'explanation': explanation, 'cwd': cwd},
        )


Instruction:
How would you implement a function in Python 3 that
initializes the argument parser for the build directory.

Response:
def init_argparser_build_dir(
            self, argparser, help=(
                'the build directory, where all sources will be copied to '
                'as part of the build process; if left unspecified, the '
                'default behavior is to create a new temporary directory '
                'that will be removed upon conclusion of the build; if '
                'specified, it must be an existing directory and all files '
                'for the build will be copied there instead, overwriting any '
                'existing file, with no cleanup done after.'
            )):
        """
        For setting up build directory
        """

        argparser.add_argument(
            '--build-dir', default=None, dest=BUILD_DIR,
            metavar=metavar(BUILD_DIR), help=help,
        )


Instruction:
Here you have a function in Python 3, explain what it does
def init_argparser_optional_advice(
            self, argparser, default=[], help=(
                'a comma separated list of packages to retrieve optional '
                'advice from; the provided packages should have registered '
                'the appropriate entry points for setting up the advices for '
                'the toolchain; refer to documentation for the specified '
                'packages for details'
            )):
        """
        For setting up optional advice.
        """

        argparser.add_argument(
            '--optional-advice', default=default, required=False,
            dest=ADVICE_PACKAGES, action=StoreRequirementList,
            metavar='<advice>[,<advice>[...]]',
            help=help
        )

Response: Initializes the argument parser for optional advice.


Instruction:
Create a Python 3 function to
initialize the argument parser for the current version of the toolchain.

Response:
def init_argparser(self, argparser):
        """
        Other runtimes (or users of ArgumentParser) can pass their
        subparser into here to collect the arguments here for a
        subcommand.
        """

        super(ToolchainRuntime, self).init_argparser(argparser)

        # it is possible for subclasses to fully override this, but if
        # they are using this as the runtime to drive the toolchain they
        # should be prepared to follow the layout, but if they omit them
        # it should only result in the spec omitting these arguments.
        self.init_argparser_export_target(argparser)
        self.init_argparser_working_dir(argparser)
        self.init_argparser_build_dir(argparser)
        self.init_argparser_optional_advice(argparser)


Instruction:
Write a Python 3 function for
preparing a spec for usage with the generic ToolchainRuntime.

Response:
def prepare_spec(self, spec, **kwargs):
        """
        Prepare a spec for usage with the generic ToolchainRuntime.

        Subclasses should avoid overriding this; override create_spec
        instead.
        """

        self.prepare_spec_debug_flag(spec, **kwargs)
        self.prepare_spec_export_target_checks(spec, **kwargs)
        # defer the setup till the actual toolchain invocation
        spec.advise(SETUP, self.prepare_spec_advice_packages, spec, **kwargs)


Instruction:
Can you implement a function in Python 3 that
turns the provided kwargs into a spec ready for toolchain.

Response:
def kwargs_to_spec(self, **kwargs):
        """
        Turn the provided kwargs into arguments ready for toolchain.
        """

        spec = self.create_spec(**kwargs)
        self.prepare_spec(spec, **kwargs)
        return spec


Instruction:
Can you generate the documentation for the following Python 3 function
def init_argparser_package_names(self, argparser, help=(
                'names of the python package to generate artifacts for; '
                'note that the metadata directory for the specified '
                'packages must be writable')):
        """
        Default helper for setting up the package_names option.

        This is separate so that subclasses are not assumed for the
        purposes of artifact creation; they should consider modifying
        the default help message to reflect the fact.
        """

        argparser.add_argument(
            'package_names', metavar=metavar('package'), nargs='+', help=help)

Response: Initialize the argument parser for the package_names option.


Instruction:
Create a Python 3 function to
initialize the argument parser for the source registry flag.

Response:
def init_argparser_source_registry(
            self, argparser, default=None, help=(
                'comma separated list of registries to use for gathering '
                'JavaScript sources from the given Python packages'
            )):
        """
        For setting up the source registry flag.
        """

        argparser.add_argument(
            '--source-registry', default=default,
            dest=CALMJS_MODULE_REGISTRY_NAMES, action=StoreDelimitedList,
            metavar='<registry>[,<registry>[...]]',
            help=help,
        )

        argparser.add_argument(
            '--source-registries', default=default,
            dest=CALMJS_MODULE_REGISTRY_NAMES, action=StoreDelimitedList,
            help=SUPPRESS,
        )


Instruction:
Given the following Python 3 function, write the documentation
def init_argparser_loaderplugin_registry(
            self, argparser, default=None, help=(
                'the name of the registry to use for the handling of loader '
                'plugins that may be loaded from the given Python packages'
            )):
        """
        Default helper for setting up the loaderplugin registries flags.

        Note that this is NOT part of the init_argparser due to
        implementation specific requirements.  Subclasses should
        consider modifying the default value help message to cater to the
        toolchain it encapsulates.
        """

        argparser.add_argument(
            '--loaderplugin-registry', default=default,
            dest=CALMJS_LOADERPLUGIN_REGISTRY_NAME, action='store',
            metavar=metavar('registry'),
            help=help,
        )

Response: Initialize the given argument parser for loaderplugin registries.


Instruction:
Write a Python 3 script for
initializing the argument parser.

Response:
def init_argparser(self, argparser):
        """
        Other runtimes (or users of ArgumentParser) can pass their
        subparser into here to collect the arguments here for a
        subcommand.
        """

        super(SourcePackageToolchainRuntime, self).init_argparser(argparser)

        self.init_argparser_source_registry(argparser)
        self.init_argparser_package_names(argparser)


Instruction:
Can you write a function in Python 3 where it
returns a grammar for an Alphanumeric field.

Response:
def alphanum(columns, name=None, extended=False, isLast=False):
    """
    Creates the grammar for an Alphanumeric (A) field, accepting only the
    specified number of characters.

    By default Alphanumeric fields accept only ASCII characters, excluding
    lowercases. If the extended flag is set to True, then non-ASCII characters
    are allowed, but the no ASCII lowercase constraint is kept.

    This can be a compulsory field, in which case the empty string is
    disallowed.

    The text will be stripped of heading and trailing whitespaces.

    :param columns: number of columns for this field
    :param name: name for the field
    :param extended: indicates if this is the exceptional case where non-ASCII
    are allowed
    :return: grammar for this Alphanumeric field
    """

    if name is None:
        name = 'Alphanumeric Field'

    if columns < 0:
        # Can't be empty or have negative size
        raise BaseException()

    if isLast:
        columns = str('1,' + str(columns))

    # Checks if non-ASCII characters are allowed
    if not extended:
        # The regular expression just forbids lowercase characters
        field = pp.Regex('([\x00-\x60]|[\x7B-\x7F]){' + str(columns) + '}')
    else:
        # The regular expression forbids lowercase characters but allows
        # non-ASCII characters
        field = pp.Regex('([\x00-\x09]|[\x0E-\x60]|[\x7B-\x7F]|[^\x00-\x7F]){' +
                         str(columns) + '}')

    # Parse action
    field.setParseAction(lambda s: s[0].strip())

    # Compulsory field validation action
    if columns:
        field.addParseAction(lambda s: _check_not_empty(s[0]))

    # White spaces are not removed
    field.leaveWhitespace()

    # Name
    field.setName(name)

    return field


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _check_not_empty(string):
    string = string.strip()

    if len(string) == 0:
        message = 'The string should not be empty'
        raise pp.ParseException(message)

Response: Checks that the string is not empty.


Instruction:
Create a Python 3 function for
returning a grammar for a numeric field.

Response:
def numeric(columns, name=None):
    """
    Creates the grammar for a Numeric (N) field, accepting only the specified
    number of characters.

    This version only allows integers.

    :param columns: number of columns for this field
    :param name: name for the field
    :return: grammar for the integer numeric field
    """

    if name is None:
        name = 'Numeric Field'

    if columns <= 0:
        # Can't be empty or have negative size
        raise BaseException()

    # Only numbers are accepted
    field = pp.Regex('[0-9]{' + str(columns) + '}')

    # Parse action
    field.setParseAction(_to_int)
    field.leaveWhitespace()

    # Name
    field.setName(name)

    return field


Instruction:
Here you have a function in Python 3, explain what it does
def numeric_float(columns, nums_int, name=None):

    if name is None:
        name = 'Numeric Field'

    if columns <= 0:
        # Can't be empty or have negative size
        raise BaseException('Number of columns should be positive')

    if nums_int < 0:
        # Integer columns can't have negative size
        raise BaseException('Number of integer values should be positive or '
                            'zero')

    if columns < nums_int:
        # There are more integer numbers than columns
        message = 'The number of columns is %s and should be higher or ' \
                  'equal than the integers: %s' % (
                      columns, nums_int)
        raise BaseException(message)

    # Basic field
    field = pp.Word(pp.nums, exact=columns)

    # Parse action
    field.setParseAction(lambda n: _to_numeric_float(n[0], nums_int))

    # Compulsory field validation action
    field.addParseAction(lambda s: _check_above_value_float(s[0], 0))

    # Name
    field.setName(name)

    return field

Response: Returns a grammar for a numeric field with the specified number of characters.


Instruction:
Make a summary of the following Python 3 code
def _to_numeric_float(number, nums_int):
    index_end = len(number) - nums_int
    return float(number[:nums_int] + '.' + number[-index_end:])

Response: Transforms a string into a float.


Instruction:
How would you implement a function in Python 3 that
checks that the value parsed from the string is above a minimum value.

Response:
def _check_above_value_float(string, minimum):
    """
    Checks that the number parsed from the string is above a minimum.

    This is used on compulsory numeric fields.

    If the value is not above the minimum an exception is thrown.

    :param string: the field value
    :param minimum: minimum value
    """
    value = float(string)

    if value < minimum:
        message = 'The Numeric Field value should be above %s' % minimum
        raise pp.ParseException(message)


Instruction:
Implement a Python 3 function for
returning the grammar for a Boolean field

Response:
def boolean(name=None):
    """
    Creates the grammar for a Boolean (B) field, accepting only 'Y' or 'N'

    :param name: name for the field
    :return: grammar for the flag field
    """

    if name is None:
        name = 'Boolean Field'

    # Basic field
    field = pp.Regex('[YN]')

    # Parse action
    field.setParseAction(lambda b: _to_boolean(b[0]))

    # Name
    field.setName(name)

    return field


Instruction:
Here you have a function in Python 3, explain what it does
def _to_boolean(string):

    if string == 'Y':
        result = True
    elif string == 'N':
        result = False
    else:
        raise pp.ParseException(string, msg='Is not a valid boolean value')

    return result

Response: Transforms a string into a boolean value.


Instruction:
Given the following Python 3 function, write the documentation
def flag(name=None):

    if name is None:
        name = 'Flag Field'

    # Basic field
    field = pp.Regex('[YNU]')

    # Name
    field.setName(name)

    field.leaveWhitespace()

    return field

Response: Returns the grammar for a Flag field.


Instruction:
Can you tell what is the following Python 3 function doing
def date(name=None):

    if name is None:
        name = 'Date Field'

    # Basic field
    # This regex allows values from 00000101 to 99991231
    field = pp.Regex('[0-9][0-9][0-9][0-9](0[1-9]|1[0-2])'
                     '(0[1-9]|[1-2][0-9]|3[0-1])')

    # Parse action
    field.setParseAction(lambda d: datetime.datetime.strptime(d[0], '%Y%m%d')
                         .date())

    # Name
    field.setName(name)

    # White spaces are not removed
    field.leaveWhitespace()

    return field

Response: Returns a grammar for a Date field.


Instruction:
Can you write a function in Python 3 where it
returns a grammar for a Time or Duration field.

Response:
def time(name=None):
    """
    Creates the grammar for a Time or Duration (T) field, accepting only
    numbers in a certain pattern.

    :param name: name for the field
    :return: grammar for the date field
    """

    if name is None:
        name = 'Time Field'

    # Basic field
    # This regex allows values from 000000 to 235959
    field = pp.Regex('(0[0-9]|1[0-9]|2[0-3])[0-5][0-9][0-5][0-9]')

    # Parse action
    field.setParseAction(lambda t: datetime.datetime.strptime(t[0], '%H%M%S')
                         .time())

    # White spaces are not removed
    field.leaveWhitespace()

    # Name
    field.setName(name)

    return field


Instruction:
How would you implement a function in Python 3 that
creates a grammar for a Lookup field.

Response:
def lookup(values, name=None):
    """
    Creates the grammar for a Lookup (L) field, accepting only values from a
    list.

    Like in the Alphanumeric field, the result will be stripped of all heading
    and trailing whitespaces.

    :param values: values allowed
    :param name: name for the field
    :return: grammar for the lookup field
    """
    if name is None:
        name = 'Lookup Field'

    if values is None:
        raise ValueError('The values can no be None')

    # TODO: This should not be needed, it is just a patch. Fix this.
    try:
        v = values.asList()
        values = v
    except AttributeError:
        values = values

    # Only the specified values are allowed
    lookup_field = pp.oneOf(values)

    lookup_field.setName(name)

    lookup_field.setParseAction(lambda s: s[0].strip())

    lookup_field.leaveWhitespace()

    return lookup_field


Instruction:
Implement a function in Python 3 to
return a grammar for a blank field.

Response:
def blank(columns=1, name=None):
    """
    Creates the grammar for a blank field.

    These are for constant empty strings which should be ignored, as they are
    used just as fillers.

    :param columns: number of columns, which is the required number of
    whitespaces
    :param name: name for the field
    :return: grammar for the blank field
    """
    if name is None:
        name = 'Blank Field'

    field = pp.Regex('[ ]{' + str(columns) + '}')
    field.leaveWhitespace()
    field.suppress()

    field.setName(name)

    return field


Instruction:
Here you have a function in Python 3, explain what it does
def get_attribute(self, attribute, value=None, features=False):
        if attribute in self.filters:
            valid_gff_objects = self.fast_attributes[attribute] if not value else\
                [i for i in self.fast_attributes[attribute] if i.attributes.get(attribute, False) == value]
            if features:
                valid_ids = [gff_object.attributes.get(self.id_tag, None) for gff_object in valid_gff_objects]
                return [self.feature_map[gff_id] for gff_id in valid_ids if gff_id]
            else:
                return valid_gff_objects
        else:
            valid_gff_objects = [gff_object for gff_feature in self.feature_map.values()
                              for gff_object in gff_feature.features
                              if gff_object.attributes.get(attribute, False)]
            valid_gff_objects = valid_gff_objects if not value else [gff_object for gff_object in valid_gff_objects
                                                                     if gff_object.attributes[attribute] == value]
            if features:
                valid_ids = [gff_object.attributes.get(self.id_tag, None) for gff_object in valid_gff_objects]
                return [self.feature_map[gff_id] for gff_id in valid_ids if gff_id]
            else:
                return valid_gff_objects

Response: This method returns a list of GFF objects with the given attribute and if supplied those objects with the specified value. If not supplied those objects with the given value are returned.


Instruction:
Can you tell what is the following Python 3 function doing
def contains(self, chrom, start, end, overlap=True):
        d = self.positions.get(chrom,[])
        if overlap:
            return [vcf_entry for vcf_start, vcf_end in d
                    for vcf_entry in d[(vcf_start, vcf_end)]
                    if not (end < vcf_start or start > vcf_end)]
        else:
            return [vcf_entry for vcf_start, vcf_end in d
                    for vcf_entry in d[(vcf_start, vcf_end)]
                    if (vcf_start <= start and vcf_end >= end)]

Response: This method returns a list of VCFEntry objects which cover a specified location.


Instruction:
Given the following Python 3 function, write the documentation
def remove_variants(self, variants):
        chroms = set([i.chrom for i in variants])
        for chrom in chroms:
            if self.append_chromosome:
                chrom = 'chr%s' % chrom
            to_delete = [pos for pos in self.positions[chrom] if pos in variants]
            for pos in to_delete:
                del self.positions[chrom][pos]

Response: Remove a list of variants from the positions we are scanning


Instruction:
Can you generate the documentation for the following Python 3 function
def generate_handler_sourcepath(
            self, toolchain, spec, loaderplugin_sourcepath):
        """
        Attempt to locate the plugin source; returns a mapping of
        modnames to the absolute path of the located sources.
        """

        # TODO calmjs-4.0.0 consider formalizing to the method instead
        npm_pkg_name = (
            self.node_module_pkg_name
            if self.node_module_pkg_name else
            self.find_node_module_pkg_name(toolchain, spec)
        )

        if not npm_pkg_name:
            cls = type(self)
            registry_name = getattr(
                self.registry, 'registry_name', '<invalid_registry/handler>')
            if cls is NPMLoaderPluginHandler:
                logger.error(
                    "no npm package name specified or could be resolved for "
                    "loaderplugin '%s' of registry '%s'; please subclass "
                    "%s:%s such that the npm package name become specified",
                    self.name, registry_name, cls.__module__, cls.__name__,
                )
            else:
                logger.error(
                    "no npm package name specified or could be resolved for "
                    "loaderplugin '%s' of registry '%s'; implementation of "
                    "%s:%s may be at fault",
                    self.name, registry_name, cls.__module__, cls.__name__,
                )
            return {}

        working_dir = spec.get(WORKING_DIR, None)
        if working_dir is None:
            logger.info(
                "attempting to derive working directory using %s, as the "
                "provided spec is missing working_dir", toolchain
            )
            working_dir = toolchain.join_cwd()

        logger.debug("deriving npm loader plugin from '%s'", working_dir)

        target = locate_package_entry_file(working_dir, npm_pkg_name)
        if target:
            logger.debug('picked %r for loader plugin %r', target, self.name)
            # use the parent recursive lookup.
            result = super(
                NPMLoaderPluginHandler, self).generate_handler_sourcepath(
                    toolchain, spec, loaderplugin_sourcepath)
            result.update({self.name: target})
            return result

        # the expected package file is not found, use the logger to show
        # why.
        # Also note that any inner/chained loaders will be dropped.
        if exists(join(
                working_dir, 'node_modules', npm_pkg_name,
                'package.json')):
            logger.warning(
                "'package.json' for the npm package '%s' does not contain a "
                "valid entry point: sources required for loader plugin '%s' "
                "cannot be included automatically; the build process may fail",
                npm_pkg_name, self.name,
            )
        else:
            logger.warning(
                "could not locate 'package.json' for the npm package '%s' "
                "which was specified to contain the loader plugin '%s' in the "
                "current working directory '%s'; the missing package may "
                "be installed by running 'npm install %s' for the mean time "
                "as a workaround, though the package that owns that source "
                "file that has this requirement should declare an explicit "
                "dependency; the build process may fail",
                npm_pkg_name, self.name, working_dir,
                npm_pkg_name,
            )

        return {}

Response: Generate the path to the source file for the given loader plugin.


Instruction:
Here you have a function in Python 3, explain what it does
def store_records_for_package(self, entry_point, records):

        pkg_records_entry = self._dist_to_package_module_map(entry_point)
        pkg_records_entry.extend(
            rec for rec in records if rec not in pkg_records_entry)
        # TODO figure out a more efficient way to do this with a bit
        # more reuse.
        if entry_point.dist is not None:
            if entry_point.dist.project_name not in self.package_loader_map:
                self.package_loader_map[entry_point.dist.project_name] = []
            self.package_loader_map[entry_point.dist.project_name].append(
                entry_point.name)

Response: Stores the records for the given entry_point in the package_loader_map.


Instruction:
How would you explain what the following Python 3 function does
def default_file_encoder():
    config = CWRConfiguration()
    field_configs = config.load_field_config('table')
    field_configs.update(config.load_field_config('common'))

    field_values = CWRTables()

    for entry in field_configs.values():
        if 'source' in entry:
            values_id = entry['source']
            entry['values'] = field_values.get_data(values_id)

    record_configs = config.load_record_config('common')
    return CwrFileEncoder(record_configs, field_configs)

Response: Get default encoder cwr file


Instruction:
Can you create a Python 3 function that
encodes a FileTag object into a CWR file name.

Response:
def encode(self, tag):
        """
        Parses a CWR file name from a FileTag object.

        The result will be a string following the format CWyynnnnsss_rrr.Vxx,
        where the numeric sequence will have the length set on the encoder's
        constructor.

        :param tag: FileTag to parse
        :return: a string file name parsed from the FileTag
        """
        # Acquires sequence number
        sequence = str(tag.sequence_n)

        # If the sequence is bigger the max, it is cut
        if len(sequence) > self._sequence_l:
            sequence = sequence[:self._sequence_l]

        # If the sequence is smaller the max, it is padded with zeroes
        while len(sequence) < self._sequence_l:
            sequence = '0' + sequence

        # Acquires version
        version = str(tag.version)

        # If the version is too long only the first and last number are taken,
        # to remove decimal separator
        if len(version) > 2:
            version = version[:1] + version[-1:]

        # If the version is too short, it is padded with zeroes
        while len(version) < 2:
            version = '0' + version

        # Acquires year
        # Only the two last digits of the year are used
        year = str(tag.year)[-2:]

        # Acquires sender and receiver
        sender = tag.sender[:3]
        receiver = tag.receiver[:3]

        rule = self._header + year + sequence + sender
        rule = rule + self._ip_delimiter + receiver + ".V" + version

        return rule


Instruction:
Can you generate the documentation for the following Python 3 function
def encode(self, transmission):
        data = ''
        data += self._record_encode(transmission.header)
        for group in transmission.groups:
            data += self._record_encode(group.group_header)
            for transaction in group.transactions:
                for record in transaction:
                    data += self._record_encode(record)
            data += self._record_encode(group.group_trailer)
        data += self._record_encode(transmission.trailer)
        return data

Response: Encodes the data for a single object in the domain model.


Instruction:
Can you write a function in Python 3 where it
gets a random scan for all the tables

Response:
def getScans(self, modifications=True, fdr=True):
        """
        get a random scan
        """
        if not self.scans:
            for i in self:
                yield i
        else:
            for i in self.scans.values():
                yield i
        yield None


Instruction:
Write a Python 3 script to
return a scan object for a given title

Response:
def getScan(self, title, peptide=None):
        """
        allows random lookup
        """
        if self.ra.has_key(title):
            self.filename.seek(self.ra[title][0],0)
            toRead = self.ra[title][1]-self.ra[title][0]
            info = self.filename.read(toRead)
            scan = self.parseScan(info)
        else:
            return None
        return scan


Instruction:
Can you generate a brief explanation for the following Python 3 code
def parseScan(self, scan):
        setupScan = True
        foundCharge = False
        foundMass = False
        foundTitle = False
        scanObj = ScanObject()
        scanObj.ms_level = 2
        for row in scan.split('\n'):
            if not row:
                continue
            entry = row.strip().split('=')
            if len(entry) >= 2:
                if entry[0] == 'PEPMASS':
                    scanObj.mass = float(entry[1])
                    foundMass = True
                elif entry[0] == 'CHARGE':
                    scanObj.charge = entry[1]
                    foundCharge = True
                elif entry[0] == 'TITLE':
#                    if self.titleMap:
#                        pos = entry[1].find(',')
#                        title = self.titleMap[int(entry[1][:entry[1].find(',')])]
#                    else:
                    title = '='.join(entry[1:])
                    foundTitle = True
                    scanObj.title = title
                    scanObj.id = title
                elif entry[0] == 'RTINSECONDS':
                    scanObj.rt = float(entry[1])
            else:
                mz,intensity = self.scanSplit.split(row.strip())
                scanObj.scans.append((float(mz),float(intensity)))
        if foundCharge and foundMass and foundTitle:
            return scanObj
        return None

Response: Parses a scan string and returns a ScanObject object.


Instruction:
Can you create a Python 3 function that
gets a random scan from the database

Response:
def getScan(self, specId, peptide=None):
        """
        get a random scan
        """
        sql = self.base_sql + " where sh.SpectrumID = %d and p.Sequence = '%s'"%(int(specId),peptide)
        self.cur.execute(sql)
        i = self.cur.fetchone()
        if not i:
            return None
        scan = self.parseFullScan(i)
        scan.spectrumId = specId
        return scan


Instruction:
Can you generate the documentation for the following Python 3 function
def getScans(self, modifications=False, fdr=True):
        if fdr:
            sql = self.base_sql+"WHERE p.ConfidenceLevel >= {} and p.SearchEngineRank <= {} {}".format(self.clvl, self.srank, self.extra)
            try:
                self.cur.execute(sql)
            except sqlite3.OperationalError:
                sql = self.base_sql+"WHERE p.ConfidenceLevel >= {} {}".format(self.clvl, self.extra)
                self.cur.execute(sql)
        else:
            sql = self.base_sql
            self.cur.execute(sql)
        while True:
            # results = self.cur.fetchmany(1000)
            # if not results:
            #     break
            try:
                tup = self.cur.fetchone()
            except:
                sys.stderr.write('Error fetching scan:\n{}\n'.format(traceback.format_exc()))
            else:
                while tup is not None:
                    if tup is None:
                        break
                    if tup[1] is not None:
                        scan = self.parseFullScan(tup, modifications=modifications)
                        scan.spectrumId = tup[3]
                        yield scan
                    try:
                        tup = self.cur.fetchone()
                    except:
                        sys.stderr.write('Error fetching scan:\n{}\n'.format(traceback.format_exc()))
                if tup is None:
                    break
        yield None

Response: get a random set of scans


Instruction:
Here you have a function in Python 3, explain what it does
def parseFullScan(self, i, modifications=False):
        scanObj = PeptideObject()
        peptide = str(i[1])
        pid=i[2]
        scanObj.acc = self.protein_map.get(i[4], i[4])
        if pid is None:
            return None
        if modifications:
            sql = 'select aam.ModificationName,pam.Position,aam.DeltaMass from peptidesaminoacidmodifications pam left join aminoacidmodifications aam on (aam.AminoAcidModificationID=pam.AminoAcidModificationID) where pam.PeptideID=%s'%pid
            for row in self.conn.execute(sql):
                scanObj.addModification(peptide[row[1]], str(row[1]), str(row[2]), row[0])
        else:
            mods = self.mods.get(int(pid))
            if mods is not None:
                for modId, modPosition in zip(mods[0].split(','),mods[1].split(',')):
                    modEntry = self.modTable[str(modId)]
                    scanObj.addModification(peptide[int(modPosition)], modPosition, modEntry[1], modEntry[0])
            tmods = self.tmods.get(int(pid))
            if tmods is not None:
                for modIds in tmods:
                    for modId in modIds.split(','):
                        modEntry = self.modTable[str(modId)]
                        scanObj.addModification('[', 0, modEntry[1], modEntry[0])
        scanObj.peptide = peptide
        if self.decompressScanInfo(scanObj, i[0]):
            return scanObj
        return None

Response: Parses a full scan info for giving a Spectrum Obj for plotting. takes significantly longer since it has to unzip/parse xml


Instruction:
Can you tell what is the following Python 3 function doing
def getScans(self, modifications=True, fdr=True):
        if fdr:
            sql = "select sp.Spectrum, p.Sequence, p.PeptideID, p.SpectrumID from spectrumheaders sh left join spectra sp on (sp.UniqueSpectrumID=sh.UniqueSpectrumID) left join peptides p on (sh.SpectrumID=p.SpectrumID) WHERE p.ConfidenceLevel >= %d and p.SearchEngineRank <= %d" % (self.clvl, self.srank)
            try:
                self.cur.execute(sql)
            except sqlite3.OperationalError:
                sql = "select sp.Spectrum, p.Sequence, p.PeptideID, p.SpectrumID from spectrumheaders sh left join spectra sp on (sp.UniqueSpectrumID=sh.UniqueSpectrumID) left join peptides p on (sh.SpectrumID=p.SpectrumID) WHERE p.ConfidenceLevel >= %d" % self.clvl
                self.cur.execute(sql)
        else:
            sql = "select sp.Spectrum, p.Sequence, p.PeptideID, p.SpectrumID from spectrumheaders sh left join spectra sp on (sp.UniqueSpectrumID=sh.UniqueSpectrumID) left join peptides p on (sh.SpectrumID=p.SpectrumID)"
            self.cur.execute(sql)
        while True:
            results = self.cur.fetchmany(1000)
            if not results:
                break
            for tup in results:
                scan = self.parseFullScan(tup, modifications=modifications)
                scan.spectrumId = tup[3]
                yield scan
        yield None

Response: get a random scan from the database


Instruction:
Create a Python 3 function to
parse a full scan info for giving a Spectrum Obj for plotting. takes significantly longer since it has to unzip and parse xml

Response:
def parseFullScan(self, i, modifications=True):
        """
        parses scan info for giving a Spectrum Obj for plotting. takes significantly longer since it has to unzip/parse xml
        """
        scanObj = PeptideObject()
        peptide = str(i[1])
        pid=i[2]
        if modifications:
            sql = 'select aam.ModificationName,pam.Position,aam.DeltaMass from peptidesaminoacidmodifications pam left join aminoacidmodifications aam on (aam.AminoAcidModificationID=pam.AminoAcidModificationID) where pam.PeptideID=%s'%pid
            for row in self.conn.execute(sql):
                scanObj.addModification(peptide[row[1]], str(row[1]), str(row[2]), row[0])
        scanObj.peptide = peptide
        if self.decompressScanInfo(scanObj, i[0]):
            return scanObj
        return None


Instruction:
Create a Python 3 function for
attempting to resolve the distribution from the provided class in the class most naive way - this assumes that the module path to the class contains the name of the package that provided the class and the class itself.

Response:
def _cls_lookup_dist(cls):
    """
    Attempt to resolve the distribution from the provided class in the
    most naive way - this assumes the Python module path to the class
    contains the name of the package that provided the module and class.
    """

    frags = cls.__module__.split('.')
    for name in ('.'.join(frags[:x]) for x in range(len(frags), 0, -1)):
        dist = find_pkg_dist(name)
        if dist:
            return dist


Instruction:
Can you generate the documentation for the following Python 3 function
def verify_builder(builder):

    try:
        d = getcallargs(builder, package_names=[], export_target='some_path')
    except TypeError:
        return False
    return d == {'package_names': [], 'export_target': 'some_path'}

Response:
Verify that the provided builder has a signature that is at least
    compatible.


Instruction:
Can you tell what is the following Python 3 function doing
def extract_builder_result(builder_result, toolchain_cls=Toolchain):

    try:
        toolchain, spec = builder_result
    except Exception:
        return None, None
    if not isinstance(toolchain, toolchain_cls) or not isinstance(spec, Spec):
        return None, None
    return toolchain, spec

Response: Extract the builder result to produce a Toolchain and Spec instance.


Instruction:
Create a Python 3 function to
trace the versions of the involved packages for the provided toolchain instance.

Response:
def trace_toolchain(toolchain):
    """
    Trace the versions of the involved packages for the provided
    toolchain instance.
    """

    pkgs = []
    for cls in getmro(type(toolchain)):
        if not issubclass(cls, Toolchain):
            continue
        dist = _cls_lookup_dist(cls)
        value = {
            'project_name': dist.project_name,
            'version': dist.version,
        } if dist else {}
        key = '%s:%s' % (cls.__module__, cls.__name__)
        pkgs.append({key: value})
    return pkgs


Instruction:
Can you generate the documentation for the following Python 3 function
def get_artifact_filename(self, package_name, artifact_name):

        project_name = self.packages.normalize(package_name)
        return self.records.get((project_name, artifact_name))

Response: Returns the path to the artifact file that should be used for the given artifact name.