Instruction:
Write a Python 3 script for
returning the raw mempool of the endpoint

Response:
def get_raw_mempool(self, id=None, endpoint=None):
        """
        Returns the tx that are in the memorypool of the endpoint
        Args:
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            json object of the result or the error encountered in the RPC call
        """
        return self._call_endpoint(GET_RAW_MEMPOOL, id=id, endpoint=endpoint)


Instruction:
Write a Python 3 function for
getting a single transaction by hash.

Response:
def get_transaction(self, tx_hash, id=None, endpoint=None):
        """
        Look up a transaction by hash.
        Args:
            tx_hash: (str) hash in the form '58c634f81fbd4ae2733d7e3930a9849021840fc19dc6af064d6f2812a333f91d'
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use

        Returns:
            json: the transaction as a json object
        """
        return self._call_endpoint(GET_RAW_TRANSACTION, params=[tx_hash, 1], id=id, endpoint=endpoint)


Instruction:
Create a Python 3 function for
returning a storage item of a specified contract and storage key.

Response:
def get_storage(self, contract_hash, storage_key, id=None, endpoint=None):
        """
        Returns a storage item of a specified contract
        Args:
            contract_hash: (str) hash of the contract to lookup, for example 'd7678dd97c000be3f33e9362e673101bac4ca654'
            storage_key: (str) storage key to lookup, for example 'totalSupply'
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            bytearray: bytearray value of the storage item
        """
        result = self._call_endpoint(GET_STORAGE, params=[contract_hash, binascii.hexlify(storage_key.encode('utf-8')).decode('utf-8')], id=id, endpoint=endpoint)
        try:

            return bytearray(binascii.unhexlify(result.encode('utf-8')))
        except Exception as e:
            raise NEORPCException("could not decode result %s " % e)


Instruction:
How would you code a function in Python 3 to
get a transaction output by specified transaction hash and vout id

Response:
def get_tx_out(self, tx_hash, vout_id, id=None, endpoint=None):
        """
        Gets a transaction output by specified transaction hash and output index
        Args:
            tx_hash: (str) hash in the form '58c634f81fbd4ae2733d7e3930a9849021840fc19dc6af064d6f2812a333f91d'
            vout_id: (int) index of the transaction output in the transaction
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            json object of the result or the error encountered in the RPC call
        """
        return self._call_endpoint(GET_TX_OUT, params=[tx_hash, vout_id], id=id, endpoint=endpoint)


Instruction:
Make a summary of the following Python 3 code
def invoke_contract(self, contract_hash, params, id=None, endpoint=None):
        return self._call_endpoint(INVOKE, params=[contract_hash, params], id=id, endpoint=endpoint)

Response: Invoke a contract on the base class


Instruction:
Can you generate the documentation for the following Python 3 function
def invoke_contract_fn(self, contract_hash, operation, params=None, id=None, endpoint=None):
        return self._call_endpoint(INVOKE_FUNCTION, params=[contract_hash, operation, params if params else []], id=id, endpoint=endpoint)

Response: Invoke a contract function on the base class


Instruction:
Can you tell what is the following Python 3 function doing
def invoke_script(self, script, id=None, endpoint=None):
        return self._call_endpoint(INVOKE_SCRIPT, params=[script], id=id, endpoint=endpoint)

Response: Invoke a script that has been assembled by the caller.


Instruction:
Can you create a Python 3 function that
sends a serialized tx to the network endpoint

Response:
def send_raw_tx(self, serialized_tx, id=None, endpoint=None):
        """
        Submits a serialized tx to the network
        Args:
            serialized_tx: (str) a hexlified string of a transaction
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            bool: whether the tx was accepted or not
        """
        return self._call_endpoint(SEND_TX, params=[serialized_tx], id=id, endpoint=endpoint)


Instruction:
Create a Python 3 function to
validate an address string and return whether or not it is valid.

Response:
def validate_addr(self, address, id=None, endpoint=None):
        """
        returns whether or not addr string is valid

        Args:
            address: (str) address to lookup ( in format 'AXjaFSP23Jkbe6Pk9pPGT6NBDs1HVdqaXK')
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use

        Returns:
            json object of the result or the error encountered in the RPC call

        """
        return self._call_endpoint(VALIDATE_ADDR, params=[address], id=id, endpoint=endpoint)


Instruction:
Explain what the following Python 3 code does
def get_peers(self, id=None, endpoint=None):
        return self._call_endpoint(GET_PEERS, id=id, endpoint=endpoint)

Response: Get the current peers of a remote node.


Instruction:
How would you explain what the following Python 3 function does
def get_validators(self, id=None, endpoint=None):
        return self._call_endpoint(GET_VALIDATORS, id=id, endpoint=endpoint)

Response: Returns the current NEO consensus nodes information and voting status.


Instruction:
Explain what the following Python 3 code does
def get_version(self, id=None, endpoint=None):
        return self._call_endpoint(GET_VERSION, id=id, endpoint=endpoint)

Response: Get the current version of the current resource in the specified endpoint.


Instruction:
Given the following Python 3 function, write the documentation
def get_new_address(self, id=None, endpoint=None):
        return self._call_endpoint(GET_NEW_ADDRESS, id=id, endpoint=endpoint)

Response: Create new address in the specified region


Instruction:
Implement a function in Python 3 to
get the current wallet index height.

Response:
def get_wallet_height(self, id=None, endpoint=None):
        """
        Get the current wallet index height.
        Args:
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            json object of the result or the error encountered in the RPC call
        """
        return self._call_endpoint(GET_WALLET_HEIGHT, id=id, endpoint=endpoint)


Instruction:
Can you implement a function in Python 3 that
lists all the addresses in the current wallet.

Response:
def list_address(self, id=None, endpoint=None):
        """
        Lists all the addresses in the current wallet.
        Args:
            id: (int, optional) id to use for response tracking
            endpoint: (RPCEndpoint, optional) endpoint to specify to use
        Returns:
            json object of the result or the error encountered in the RPC call
        """
        return self._call_endpoint(LIST_ADDRESS, id=id, endpoint=endpoint)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def send_from(self, asset_id, addr_from, to_addr, value, fee=None, change_addr=None, id=None, endpoint=None):
        params = [asset_id, addr_from, to_addr, value]
        if fee:
            params.append(fee)
        if fee and change_addr:
            params.append(change_addr)
        elif not fee and change_addr:
            params.append(0)
            params.append(change_addr)
        return self._call_endpoint(SEND_FROM, params=params, id=id, endpoint=endpoint)

Response: Send a value to the specified address.


Instruction:
Given the following Python 3 function, write the documentation
def send_to_address(self, asset_id, to_addr, value, fee=None, change_addr=None, id=None, endpoint=None):
        params = [asset_id, to_addr, value]
        if fee:
            params.append(fee)
        if fee and change_addr:
            params.append(change_addr)
        elif not fee and change_addr:
            params.append(0)
            params.append(change_addr)
        return self._call_endpoint(SEND_TO_ADDRESS, params=params, id=id, endpoint=endpoint)

Response: Send a transfer to a specific address.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def send_many(self, outputs_array, fee=None, change_addr=None, id=None, endpoint=None):
        params = [outputs_array]
        if fee:
            params.append(fee)
        if fee and change_addr:
            params.append(change_addr)
        elif not fee and change_addr:
            params.append(0)
            params.append(change_addr)
        return self._call_endpoint(SEND_MANY, params=params, id=id, endpoint=endpoint)

Response: Send a list of outputs to the NEO network.


Instruction:
Can you tell what is the following Python 3 function doing
def choose_pool(mpi=False, processes=1, **kwargs):

    if mpi:
        if not MPIPool.enabled():
            raise SystemError("Tried to run with MPI but MPIPool not enabled.")

        pool = MPIPool(**kwargs)
        if not pool.is_master():
            pool.wait()
            sys.exit(0)

        log.info("Running with MPI on {0} cores".format(pool.size))
        return pool

    elif processes != 1 and MultiPool.enabled():
        log.info("Running with MultiPool on {0} cores".format(processes))
        return MultiPool(processes=processes, **kwargs)

    else:
        log.info("Running with SerialPool")
        return SerialPool(**kwargs)

Response: Choose between the different pools given options from e. g. argparse.


Instruction:
Implement a Python 3 function for
waiting for a master process and send the result to the callback.

Response:
def wait(self, callback=None):
        """Tell the workers to wait and listen for the master process. This is
        called automatically when using :meth:`MPIPool.map` and doesn't need to
        be called by the user.
        """
        if self.is_master():
            return

        worker = self.comm.rank
        status = MPI.Status()
        while True:
            log.log(_VERBOSE, "Worker {0} waiting for task".format(worker))

            task = self.comm.recv(source=self.master, tag=MPI.ANY_TAG,
                                  status=status)

            if task is None:
                log.log(_VERBOSE, "Worker {0} told to quit work".format(worker))
                break

            func, arg = task
            log.log(_VERBOSE, "Worker {0} got task {1} with tag {2}"
                    .format(worker, arg, status.tag))

            result = func(arg)

            log.log(_VERBOSE, "Worker {0} sending answer {1} with tag {2}"
                    .format(worker, result, status.tag))

            self.comm.ssend(result, self.master, status.tag)

        if callback is not None:
            callback()


Instruction:
Implement a function in Python 3 to
evaluate a function or callable on each task in parallel using MPI.

Response:
def map(self, worker, tasks, callback=None):
        """Evaluate a function or callable on each task in parallel using MPI.

        The callable, ``worker``, is called on each element of the ``tasks``
        iterable. The results are returned in the expected order (symmetric with
        ``tasks``).

        Parameters
        ----------
        worker : callable
            A function or callable object that is executed on each element of
            the specified ``tasks`` iterable. This object must be picklable
            (i.e. it can't be a function scoped within a function or a
            ``lambda`` function). This should accept a single positional
            argument and return a single object.
        tasks : iterable
            A list or iterable of tasks. Each task can be itself an iterable
            (e.g., tuple) of values or data to pass in to the worker function.
        callback : callable, optional
            An optional callback function (or callable) that is called with the
            result from each worker run and is executed on the master process.
            This is useful for, e.g., saving results to a file, since the
            callback is only called on the master thread.

        Returns
        -------
        results : list
            A list of results from the output of each ``worker()`` call.
        """

        # If not the master just wait for instructions.
        if not self.is_master():
            self.wait()
            return

        if callback is None:
            callback = _dummy_callback

        workerset = self.workers.copy()
        tasklist = [(tid, (worker, arg)) for tid, arg in enumerate(tasks)]
        resultlist = [None] * len(tasklist)
        pending = len(tasklist)

        while pending:
            if workerset and tasklist:
                worker = workerset.pop()
                taskid, task = tasklist.pop()
                log.log(_VERBOSE, "Sent task %s to worker %s with tag %s",
                        task[1], worker, taskid)
                self.comm.send(task, dest=worker, tag=taskid)

            if tasklist:
                flag = self.comm.Iprobe(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG)
                if not flag:
                    continue
            else:
                self.comm.Probe(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG)

            status = MPI.Status()
            result = self.comm.recv(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG,
                                    status=status)
            worker = status.source
            taskid = status.tag
            log.log(_VERBOSE, "Master received from worker %s with tag %s",
                    worker, taskid)

            callback(result)

            workerset.add(worker)
            resultlist[taskid] = result
            pending -= 1

        return resultlist


Instruction:
Write a Python 3 function that can
tell all the workers to quit.

Response:
def close(self):
        """ Tell all the workers to quit."""
        if self.is_worker():
            return

        for worker in self.workers:
            self.comm.send(None, worker, 0)


Instruction:
Can you create a Python 3 function that
updates the git revision string if the path is being imported directly from a git working copy.

Response:
def update_git_devstr(version, path=None):
    """
    Updates the git revision string if and only if the path is being imported
    directly from a git working copy.  This ensures that the revision number in
    the version string is accurate.
    """

    try:
        # Quick way to determine if we're in git or not - returns '' if not
        devstr = get_git_devstr(sha=True, show_warning=False, path=path)
    except OSError:
        return version

    if not devstr:
        # Probably not in git so just pass silently
        return version

    if 'dev' in version:  # update to the current git revision
        version_base = version.split('.dev', 1)[0]
        devstr = get_git_devstr(sha=False, show_warning=False, path=path)

        return version_base + '.dev' + devstr
    else:
        # otherwise it's already the true/release version
        return version


Instruction:
How would you explain what the following Python 3 function does
def get_git_devstr(sha=False, show_warning=True, path=None):

    if path is None:
        path = os.getcwd()
        if not _get_repo_path(path, levels=0):
            return ''

    if not os.path.isdir(path):
        path = os.path.abspath(os.path.dirname(path))

    if sha:
        # Faster for getting just the hash of HEAD
        cmd = ['rev-parse', 'HEAD']
    else:
        cmd = ['rev-list', '--count', 'HEAD']

    def run_git(cmd):
        try:
            p = subprocess.Popen(['git'] + cmd, cwd=path,
                                 stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE,
                                 stdin=subprocess.PIPE)
            stdout, stderr = p.communicate()
        except OSError as e:
            if show_warning:
                warnings.warn('Error running git: ' + str(e))
            return (None, b'', b'')

        if p.returncode == 128:
            if show_warning:
                warnings.warn('No git repository present at {0!r}! Using '
                              'default dev version.'.format(path))
            return (p.returncode, b'', b'')
        if p.returncode == 129:
            if show_warning:
                warnings.warn('Your git looks old (does it support {0}?); '
                              'consider upgrading to v1.7.2 or '
                              'later.'.format(cmd[0]))
            return (p.returncode, stdout, stderr)
        elif p.returncode != 0:
            if show_warning:
                warnings.warn('Git failed while determining revision '
                              'count: {0}'.format(_decode_stdio(stderr)))
            return (p.returncode, stdout, stderr)

        return p.returncode, stdout, stderr

    returncode, stdout, stderr = run_git(cmd)

    if not sha and returncode == 129:
        # git returns 129 if a command option failed to parse; in
        # particular this could happen in git versions older than 1.7.2
        # where the --count option is not supported
        # Also use --abbrev-commit and --abbrev=0 to display the minimum
        # number of characters needed per-commit (rather than the full hash)
        cmd = ['rev-list', '--abbrev-commit', '--abbrev=0', 'HEAD']
        returncode, stdout, stderr = run_git(cmd)
        # Fall back on the old method of getting all revisions and counting
        # the lines
        if returncode == 0:
            return str(stdout.count(b'\n'))
        else:
            return ''
    elif sha:
        return _decode_stdio(stdout)[:40]
    else:
        return _decode_stdio(stdout).strip()

Response: Returns a string containing the git version info of the current revision.


Instruction:
Write a Python 3 function for
giving a file or directory name determine the root of the git repository that corresponds to that file.

Response:
def _get_repo_path(pathname, levels=None):
    """
    Given a file or directory name, determine the root of the git repository
    this path is under.  If given, this won't look any higher than ``levels``
    (that is, if ``levels=0`` then the given path must be the root of the git
    repository and is returned if so.

    Returns `None` if the given path could not be determined to belong to a git
    repo.
    """

    if os.path.isfile(pathname):
        current_dir = os.path.abspath(os.path.dirname(pathname))
    elif os.path.isdir(pathname):
        current_dir = os.path.abspath(pathname)
    else:
        return None

    current_level = 0

    while levels is None or current_level <= levels:
        if os.path.exists(os.path.join(current_dir, '.git')):
            return current_dir

        current_level += 1
        if current_dir == os.path.dirname(current_dir):
            break

        current_dir = os.path.dirname(current_dir)

    return None


Instruction:
Make a summary of the following Python 3 code
def map(self, func, iterable, callback=None):
        return self._call_callback(callback, map(func, iterable))

Response: A wrapper around the built - in map function that provides consistent interface with the other Pool classes.


Instruction:
Implement a Python 3 function for
translating a single codon into a amino acid or stop M

Response:
def translate_codon(codon, aa_pos):
    """Translate a single codon into a single amino acid or stop '*'

    Parameters
    ----------
    codon : str
        Expected to be of length 3
    aa_pos : int
        Codon/amino acid offset into the protein (starting from 0)
    """
    # not handling rare Leucine or Valine starts!
    if aa_pos == 0 and codon in START_CODONS:
        return "M"
    elif codon in STOP_CODONS:
        return "*"
    else:
        return DNA_CODON_TABLE[codon]


Instruction:
Can you implement a function in Python 3 that
translates a nucleotide sequence into amino acid protein sequence.

Response:
def translate(
        nucleotide_sequence,
        first_codon_is_start=True,
        to_stop=True,
        truncate=False):
    """Translates cDNA coding sequence into amino acid protein sequence.

    Should typically start with a start codon but allowing non-methionine
    first residues since the CDS we're translating might have been affected
    by a start loss mutation.

    The sequence may include the 3' UTR but will stop translation at the first
    encountered stop codon.

    Parameters
    ----------
    nucleotide_sequence : BioPython Seq
        cDNA sequence

    first_codon_is_start : bool
        Treat the beginning of nucleotide_sequence (translates methionin)

    truncate : bool
        Truncate sequence if it's not a multiple of 3 (default = False)
    Returns BioPython Seq of amino acids
    """
    if not isinstance(nucleotide_sequence, Seq):
        nucleotide_sequence = Seq(nucleotide_sequence)

    if truncate:
        # if sequence isn't a multiple of 3, truncate it so BioPython
        # doesn't complain
        n_nucleotides = int(len(nucleotide_sequence) / 3) * 3
        nucleotide_sequence = nucleotide_sequence[:n_nucleotides]
    else:
        n_nucleotides = len(nucleotide_sequence)

    assert n_nucleotides % 3 == 0, \
        ("Expected nucleotide sequence to be multiple of 3"
         " but got %s of length %d") % (
            nucleotide_sequence,
            n_nucleotides)

    # passing cds=False to translate since we may want to deal with premature
    # stop codons
    protein_sequence = nucleotide_sequence.translate(to_stop=to_stop, cds=False)

    if first_codon_is_start and (
            len(protein_sequence) == 0 or protein_sequence[0] != "M"):
        if nucleotide_sequence[:3] in START_CODONS:
            # TODO: figure out when these should be made into methionines
            # and when left as whatever amino acid they normally code for
            # e.g. Leucine start codons
            # See: DOI: 10.1371/journal.pbio.0020397
            return "M" + protein_sequence[1:]
        else:
            raise ValueError(
                ("Expected first codon of %s to be start codon"
                 " (one of %s) but got %s") % (
                    protein_sequence[:10],
                    START_CODONS,
                    nucleotide_sequence))

    return protein_sequence


Instruction:
How would you explain what the following Python 3 function does
def find_first_stop_codon(nucleotide_sequence):
    n_mutant_codons = len(nucleotide_sequence) // 3
    for i in range(n_mutant_codons):
        codon = nucleotide_sequence[3 * i:3 * i + 3]
        if codon in STOP_CODONS:
            return i
    return -1

Response: Given a nucleotide sequence return the index of first stop codon.


Instruction:
How would you explain what the following Python 3 function does
def translate_in_frame_mutation(
        transcript,
        ref_codon_start_offset,
        ref_codon_end_offset,
        mutant_codons):
    """
    Returns:
        - mutant amino acid sequence
        - offset of first stop codon in the mutant sequence (or -1 if there was none)
        - boolean flag indicating whether any codons from the 3' UTR were used

    Parameters
    ----------
    transcript : pyensembl.Transcript
        Reference transcript to which a cDNA mutation should be applied.

    ref_codon_start_offset : int
        Starting (base 0) integer offset into codons (character triplets) of the
        transcript's reference coding sequence.

    ref_codon_end_offset : int
        Final (base 0) integer offset into codons of the transcript's
        reference coding sequence.

    mutant_codons : str
        Nucleotide sequence to replace the reference codons with
        (expected to have length that is a multiple of three)
    """
    mutant_stop_codon_index = find_first_stop_codon(mutant_codons)

    using_three_prime_utr = False

    if mutant_stop_codon_index != -1:
        mutant_codons = mutant_codons[:3 * mutant_stop_codon_index]
    elif ref_codon_end_offset > len(transcript.protein_sequence):
        # if the mutant codons didn't contain a stop but did mutate the
        # true reference stop codon then the translated sequence might involve
        # the 3' UTR
        three_prime_utr = transcript.three_prime_utr_sequence
        n_utr_codons = len(three_prime_utr) // 3
        # trim the 3' UTR sequence to have a length that is a multiple of 3
        truncated_utr_sequence = three_prime_utr[:n_utr_codons * 3]

        # note the offset of the first stop codon in the combined
        # nucleotide sequence of both the end of the CDS and the 3' UTR
        first_utr_stop_codon_index = find_first_stop_codon(truncated_utr_sequence)

        if first_utr_stop_codon_index > 0:
            # if there is a stop codon in the 3' UTR sequence and it's not the
            # very first codon
            using_three_prime_utr = True
            n_mutant_codons_before_utr = len(mutant_codons) // 3
            mutant_stop_codon_index = n_mutant_codons_before_utr + first_utr_stop_codon_index
            # combine the in-frame mutant codons with the truncated sequence of
            # the 3' UTR
            mutant_codons += truncated_utr_sequence[:first_utr_stop_codon_index * 3]
        elif first_utr_stop_codon_index == -1:
            # if there is no stop codon in the 3' UTR sequence
            using_three_prime_utr = True
            mutant_codons += truncated_utr_sequence

    amino_acids = translate(
        mutant_codons,
        first_codon_is_start=(ref_codon_start_offset == 0))

    return amino_acids, mutant_stop_codon_index, using_three_prime_utr

Response: Translate the mutant amino acid sequence into a single cDNA mutation sequence.


Instruction:
Write a Python 3 script for
scripting which loads variants and annotates them with overlapping genes. Example usage: varcode-genes --vcf mutect.vcf \ --vcf strelka.vcf \ --maf tcga_brca.maf \ --variant chr1 498584 C G \ --json-variants more_variants.json

Response:
def main(args_list=None):
    """
    Script which loads variants and annotates them with overlapping genes.

    Example usage:
        varcode-genes
            --vcf mutect.vcf \
            --vcf strelka.vcf \
            --maf tcga_brca.maf \
            --variant chr1 498584 C G \
            --json-variants more_variants.json
    """
    print_version_info()
    if args_list is None:
        args_list = sys.argv[1:]
    args = arg_parser.parse_args(args_list)
    variants = variant_collection_from_args(args)
    variants_dataframe = variants.to_dataframe()
    logger.info('\n%s', variants_dataframe)
    if args.output_csv:
        variants_dataframe.to_csv(args.output_csv, index=False)


Instruction:
Explain what the following Python 3 code does
def load_maf_dataframe(path, nrows=None, raise_on_error=True, encoding=None):
    require_string(path, "Path to MAF")

    n_basic_columns = len(MAF_COLUMN_NAMES)

    # pylint: disable=no-member
    # pylint gets confused by read_csv
    df = pandas.read_csv(
        path,
        comment="#",
        sep="\t",
        low_memory=False,
        skip_blank_lines=True,
        header=0,
        encoding=encoding)

    if len(df.columns) < n_basic_columns:
        error_message = (
            "Too few columns in MAF file %s, expected %d but got  %d : %s" % (
                path, n_basic_columns, len(df.columns), df.columns))
        if raise_on_error:
            raise ValueError(error_message)
        else:
            logging.warn(error_message)

    # check each pair of expected/actual column names to make sure they match
    for expected, actual in zip(MAF_COLUMN_NAMES, df.columns):
        if expected != actual:
            # MAFs in the wild have capitalization differences in their
            # column names, normalize them to always use the names above
            if expected.lower() == actual.lower():
                # using DataFrame.rename in Python 2.7.x doesn't seem to
                # work for some files, possibly because Pandas treats
                # unicode vs. str columns as different?
                df[expected] = df[actual]
                del df[actual]
            else:
                error_message = (
                    "Expected column %s but got %s" % (expected, actual))
                if raise_on_error:
                    raise ValueError(error_message)
                else:
                    logging.warn(error_message)

    return df

Response: Load the guaranteed columns of a TCGA MAF file into a DataFrame.


Instruction:
Can you tell what is the following Python 3 function doing
def load_maf(
        path,
        optional_cols=[],
        sort_key=variant_ascending_position_sort_key,
        distinct=True,
        raise_on_error=True,
        encoding=None):
    """
    Load reference name and Variant objects from MAF filename.

    Parameters
    ----------

    path : str
        Path to MAF (*.maf).

    optional_cols : list, optional
        A list of MAF columns to include as metadata if they are present in the MAF.
        Does not result in an error if those columns are not present.

    sort_key : fn
        Function which maps each element to a sorting criterion.
        Set to None to not to sort the variants.

    distinct : bool
        Don't keep repeated variants

    raise_on_error : bool
        Raise an exception upon encountering an error or just log a warning.

    encoding : str, optional
        Encoding to use for UTF when reading MAF file.
    """
    # pylint: disable=no-member
    # pylint gets confused by read_csv inside load_maf_dataframe
    maf_df = load_maf_dataframe(path, raise_on_error=raise_on_error, encoding=encoding)

    if len(maf_df) == 0 and raise_on_error:
        raise ValueError("Empty MAF file %s" % path)

    ensembl_objects = {}
    variants = []
    metadata = {}
    for _, x in maf_df.iterrows():
        contig = x.Chromosome
        if isnull(contig):
            error_message = "Invalid contig name: %s" % (contig,)
            if raise_on_error:
                raise ValueError(error_message)
            else:
                logging.warn(error_message)
                continue

        start_pos = x.Start_Position
        ref = x.Reference_Allele

        # it's possible in a MAF file to have multiple Ensembl releases
        # mixed in a single MAF file (the genome assembly is
        # specified by the NCBI_Build column)
        ncbi_build = x.NCBI_Build
        if ncbi_build in ensembl_objects:
            ensembl = ensembl_objects[ncbi_build]
        else:
            if isinstance(ncbi_build, int):
                reference_name = "B%d" % ncbi_build
            else:
                reference_name = str(ncbi_build)
            ensembl = infer_genome(reference_name)
            ensembl_objects[ncbi_build] = ensembl

        # have to try both Tumor_Seq_Allele1 and Tumor_Seq_Allele2
        # to figure out which is different from the reference allele
        if x.Tumor_Seq_Allele1 != ref:
            alt = x.Tumor_Seq_Allele1
        else:
            if x.Tumor_Seq_Allele2 == ref:
                error_message = (
                    "Both tumor alleles agree with reference %s: %s" % (
                        ref, x,))
                if raise_on_error:
                    raise ValueError(error_message)
                else:
                    logging.warn(error_message)
                    continue
            alt = x.Tumor_Seq_Allele2

        variant = Variant(
            contig,
            start_pos,
            str(ref),
            str(alt),
            ensembl=ensembl)

        # keep metadata about the variant and its TCGA annotation
        metadata[variant] = {
            'Hugo_Symbol': x.Hugo_Symbol,
            'Center': x.Center,
            'Strand': x.Strand,
            'Variant_Classification': x.Variant_Classification,
            'Variant_Type': x.Variant_Type,
            'dbSNP_RS': x.dbSNP_RS,
            'dbSNP_Val_Status': x.dbSNP_Val_Status,
            'Tumor_Sample_Barcode': x.Tumor_Sample_Barcode,
            'Matched_Norm_Sample_Barcode': x.Matched_Norm_Sample_Barcode,
        }
        for optional_col in optional_cols:
            if optional_col in x:
                metadata[variant][optional_col] = x[optional_col]

        variants.append(variant)

    return VariantCollection(
        variants=variants,
        source_to_metadata_dict={path: metadata},
        sort_key=sort_key,
        distinct=distinct)

Response: Loads the ensembl objects from a MAF file.


Instruction:
Write a Python 3 function that can
apply function to specified field of effect if it is not None otherwise return default.

Response:
def apply_to_field_if_exists(effect, field_name, fn, default):
    """
    Apply function to specified field of effect if it is not None,
    otherwise return default.
    """
    value = getattr(effect, field_name, None)
    if value is None:
        return default
    else:
        return fn(value)


Instruction:
Can you create a Python 3 function that
applies function to transcript associated with effect if it exists otherwise return default.

Response:
def apply_to_transcript_if_exists(effect, fn, default):
    """
    Apply function to transcript associated with effect,
    if it exists, otherwise return default.
    """
    return apply_to_field_if_exists(
        effect=effect,
        field_name="transcript",
        fn=fn,
        default=default)


Instruction:
Explain what the following Python 3 code does
def number_exons_in_associated_transcript(effect):
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: len(t.exons),
        default=0)

Response: Returns the number of exons in an associated transcript.


Instruction:
Write a Python 3 function that can
return the length of the coding sequence of the associated transcript.

Response:
def cds_length_of_associated_transcript(effect):
    """
    Length of coding sequence of transcript associated with effect,
    if there is one (otherwise return 0).
    """
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: len(t.coding_sequence) if (t.complete and t.coding_sequence) else 0,
        default=0)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def length_of_associated_transcript(effect):
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: len(t.sequence),
        default=0)

Response: Returns the length of spliced mRNA sequence of transcript associated with effect.


Instruction:
Create a Python 3 function for
returning the name of the associated transcript.

Response:
def name_of_associated_transcript(effect):
    """
    Name of transcript associated with effect,
    if there is one (otherwise return "").
    """
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: t.name,
        default="")


Instruction:
Write a Python 3 script for
returning True if effect has complete transcript False otherwise

Response:
def effect_has_complete_transcript(effect):
    """
    Parameters
    ----------
    effect : subclass of MutationEffect

    Returns True if effect has transcript and that transcript has complete CDS
    """
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: t.complete,
        default=False)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def effect_associated_with_protein_coding_gene(effect):
    return apply_to_gene_if_exists(
        effect=effect,
        fn=lambda g: g.biotype == "protein_coding",
        default=False)

Response: Returns True if effect is associated with a gene and that gene has a protein_coding biotype.


Instruction:
Here you have a function in Python 3, explain what it does
def effect_associated_with_protein_coding_transcript(effect):
    return apply_to_transcript_if_exists(
        effect=effect,
        fn=lambda t: t.biotype == "protein_coding",
        default=False)

Response: Returns True if effect is associated with a protein coding transcript.


Instruction:
Explain what the following Python 3 code does
def parse_transcript_number(effect):
    name = name_of_associated_transcript(effect)
    if "-" not in name:
        return 0
    parts = name.split("-")
    last_part = parts[-1]
    if last_part.isdigit():
        return int(last_part)
    else:
        return 0

Response:
Try to parse the number at the end of a transcript name associated with
    an effect. Returns 0 if the number is not found.


Instruction:
How would you explain what the following Python 3 function does
def multi_gene_effect_sort_key(effect):

    return tuple([
        effect_priority(effect),
        effect_associated_with_protein_coding_gene(effect),
        effect_associated_with_protein_coding_transcript(effect),
        effect_has_complete_transcript(effect),
        cds_length_of_associated_transcript(effect),
        length_of_associated_transcript(effect),
        number_exons_in_associated_transcript(effect),
        transcript_name_ends_with_01(effect),
        -parse_transcript_number(effect)
    ])

Response: This function returns a sort key for choosing the highest priority of an effect across multiple genes.


Instruction:
Make a summary of the following Python 3 code
def select_between_exonic_splice_site_and_alternate_effect(effect):
    if effect.__class__ is not ExonicSpliceSite:
        return effect
    if effect.alternate_effect is None:
        return effect
    splice_priority = effect_priority(effect)
    alternate_priority = effect_priority(effect.alternate_effect)
    if splice_priority > alternate_priority:
        return effect
    else:
        return effect.alternate_effect

Response: Select the most appropriate effect between an ExonicSpliceSite and an alternate effect.


Instruction:
Here you have a function in Python 3, explain what it does
def keep_max_priority_effects(effects):
    priority_values = map(effect_priority, effects)
    max_priority = max(priority_values)
    return [e for (e, p) in zip(effects, priority_values) if p == max_priority]

Response:
Given a list of MutationEffect subclasses and only keep the ones with the maximum priority
    effect type.


Instruction:
Create a Python 3 function to
apply each filter to the effect list sequentially.

Response:
def filter_pipeline(effects, filters):
    """
    Apply each filter to the effect list sequentially. If any filter
    returns zero values then ignore it. As soon as only one effect is left,
    return it.

    Parameters
    ----------
    effects : list of MutationEffect subclass instances

    filters : list of functions
        Each function takes a list of effects and returns a list of effects

    Returns list of effects
    """
    for filter_fn in filters:
        filtered_effects = filter_fn(effects)
        if len(effects) == 1:
            return effects
        elif len(filtered_effects) > 1:
            effects = filtered_effects
    return effects


Instruction:
How would you implement a function in Python 3 that
returns the first non - empty mutation effect that is a canonical transcript with both the maximum length CDS and maximum length full transcript sequence.

Response:
def top_priority_effect_for_single_gene(effects):
    """
    For effects which are from the same gene, check to see if there
    is a canonical transcript with both the maximum length CDS
    and maximum length full transcript sequence.

    If not, then use number of exons and transcript name as tie-breaking
    features.

    Parameters
    ----------
    effects : list of MutationEffect subclass instances

    Returns single effect object
    """

    # first filter effects to keep those on
    # 1) maximum priority effects
    # 2) protein coding genes
    # 3) protein coding transcripts
    # 4) complete transcripts
    #
    # If any of these filters drop all the effects then we move on to the next
    # filtering step.
    effects = filter_pipeline(
        effects=effects,
        filters=[
            keep_max_priority_effects,
            keep_effects_on_protein_coding_genes,
            keep_effects_on_protein_coding_transcripts,
            keep_effects_on_complete_transcripts,
        ],
    )

    if len(effects) == 1:
        return effects[0]

    # compare CDS length and transcript lengths of remaining effects
    # if one effect has the maximum of both categories then return it
    cds_lengths = [cds_length_of_associated_transcript(e) for e in effects]
    max_cds_length = max(cds_lengths)

    # get set of indices of all effects with maximum CDS length
    max_cds_length_indices = {
        i
        for (i, l) in enumerate(cds_lengths)
        if l == max_cds_length
    }

    seq_lengths = [length_of_associated_transcript(e) for e in effects]
    max_seq_length = max(seq_lengths)

    # get set of indices for all effects whose associated transcript
    # has maximum sequence length
    max_seq_length_indices = {
        i
        for (i, l) in enumerate(seq_lengths)
        if l == max_seq_length
    }

    # which effects have transcripts with both the longest CDS and
    # longest full transcript sequence?
    intersection_of_indices = \
        max_cds_length_indices.intersection(max_seq_length_indices)

    n_candidates = len(intersection_of_indices)
    if n_candidates == 1:
        best_index = intersection_of_indices.pop()
        return effects[best_index]

    elif n_candidates == 0:
        # if set of max CDS effects and max sequence length effects is disjoint
        # then let's try to do the tie-breaking sort over their union
        union_of_indices = max_cds_length_indices.union(max_seq_length_indices)
        candidate_effects = [effects[i] for i in union_of_indices]

    else:
        # if multiple effects have transcripts with the max CDS length and
        # the max full sequence length then run the tie-breaking sort
        # over all these candidates
        candidate_effects = [effects[i] for i in intersection_of_indices]

    # break ties by number of exons, whether name of transcript ends if "01",
    # and all else being equal, prefer transcript names that end with lower
    # numbers
    return max(
        candidate_effects,
        key=tie_breaking_sort_key_for_single_gene_effects)


Instruction:
How would you implement a function in Python 3 that
gives a collection of variant transcript effects return the highest priority effect.

Response:
def top_priority_effect(effects):
    """
    Given a collection of variant transcript effects,
    return the top priority object. ExonicSpliceSite variants require special
    treatment since they actually represent two effects -- the splicing modification
    and whatever else would happen to the exonic sequence if nothing else gets
    changed. In cases where multiple transcripts give rise to multiple
    effects, use a variety of filtering and sorting heuristics to pick
    the canonical transcript.
    """
    if len(effects) == 0:
        raise ValueError("List of effects cannot be empty")

    effects = map(
        select_between_exonic_splice_site_and_alternate_effect,
        effects)

    effects_grouped_by_gene = apply_groupby(
        effects, fn=gene_id_of_associated_transcript, skip_none=False)

    if None in effects_grouped_by_gene:
        effects_without_genes = effects_grouped_by_gene.pop(None)
    else:
        effects_without_genes = []

    # if we had any effects associated with genes then choose one of those
    if len(effects_grouped_by_gene) > 0:
        effects_with_genes = [
            top_priority_effect_for_single_gene(gene_effects)
            for gene_effects in effects_grouped_by_gene.values()
        ]
        return max(effects_with_genes, key=multi_gene_effect_sort_key)
    else:
        # if all effects were without genes then choose the best among those
        assert len(effects_without_genes) > 0
        return max(effects_without_genes, key=multi_gene_effect_sort_key)


Instruction:
Given the following Python 3 function, write the documentation
def to_dict(self):
        return dict(
            variants=self.variants,
            distinct=self.distinct,
            sort_key=self.sort_key,
            sources=self.sources,
            source_to_metadata_dict=self.source_to_metadata_dict)

Response: Returns a dictionary with the state of the object.


Instruction:
Can you create a Python 3 function that
creates another VariantCollection of the same class and with new elements.

Response:
def clone_with_new_elements(self, new_elements):
        """
        Create another VariantCollection of the same class and with
        same state (including metadata) but possibly different entries.

        Warning: metadata is a dictionary keyed by variants. This method
        leaves that dictionary as-is, which may result in extraneous entries
        or missing entries.
        """
        kwargs = self.to_dict()
        kwargs["variants"] = new_elements
        return self.from_dict(kwargs)


Instruction:
Can you create a Python 3 function that
returns EffectCollection containing all effects of this Variant on a transcript.

Response:
def effects(self, raise_on_error=True):
        """
        Parameters
        ----------
        raise_on_error : bool, optional
            If exception is raised while determining effect of variant on a
            transcript, should it be raised? This default is True, meaning
            errors result in raised exceptions, otherwise they are only logged.

        """
        return EffectCollection([
            effect
            for variant in self
            for effect in variant.effects(raise_on_error=raise_on_error)
        ])


Instruction:
Can you generate a brief explanation for the following Python 3 code
def gene_counts(self):
        return {
            gene_name: len(group)
            for (gene_name, group)
            in self.groupby_gene_name().items()
        }

Response:
Returns the number of elements overlapping each gene name. Expects the sequence to have
        an implementation of groupby_gene_name.


Instruction:
Can you tell what is the following Python 3 function doing
def filter_by_transcript_expression(
            self,
            transcript_expression_dict,
            min_expression_value=0.0):
        """
        Filters variants down to those which have overlap a transcript whose
        expression value in the transcript_expression_dict argument is greater
        than min_expression_value.

        Parameters
        ----------
        transcript_expression_dict : dict
            Dictionary mapping Ensembl transcript IDs to expression estimates
            (either FPKM or TPM)

        min_expression_value : float
            Threshold above which we'll keep an effect in the result collection
        """
        return self.filter_any_above_threshold(
            multi_key_fn=lambda variant: variant.transcript_ids,
            value_dict=transcript_expression_dict,
            threshold=min_expression_value)

Response: Returns a new VariantCollection containing only those variants which have overlap a transcript whose transcript_expression_value in the transcript_expression_dict argument is greater than min_expression_value.


Instruction:
How would you implement a function in Python 3 that
returns a new Collection containing only variants which have overlap a gene whose expression value in the transcript_expression_dict argument is greater than min_expression_value.

Response:
def filter_by_gene_expression(
            self,
            gene_expression_dict,
            min_expression_value=0.0):
        """
        Filters variants down to those which have overlap a gene whose
        expression value in the transcript_expression_dict argument is greater
        than min_expression_value.

        Parameters
        ----------
        gene_expression_dict : dict
            Dictionary mapping Ensembl gene IDs to expression estimates
            (either FPKM or TPM)

        min_expression_value : float
            Threshold above which we'll keep an effect in the result collection
        """
        return self.filter_any_above_threshold(
            multi_key_fn=lambda effect: effect.gene_ids,
            value_dict=gene_expression_dict,
            threshold=min_expression_value)


Instruction:
How would you implement a function in Python 3 that
returns True if the variants in this collection are exactly equal.

Response:
def exactly_equal(self, other):
        '''
        Comparison between VariantCollection instances that takes into account
        the info field of Variant instances.

        Returns
        ----------
        True if the variants in this collection equal the variants in the other
        collection. The Variant.info fields are included in the comparison.
        '''
        return (
            self.__class__ == other.__class__ and
            len(self) == len(other) and
            all(x.exactly_equal(y) for (x, y) in zip(self, other)))


Instruction:
Create a Python 3 function for
creating a single VariantCollection from multiple VariantCollection objects.

Response:
def _combine_variant_collections(cls, combine_fn, variant_collections, kwargs):
        """
        Create a single VariantCollection from multiple different collections.

        Parameters
        ----------

        cls : class
            Should be VariantCollection

        combine_fn : function
            Function which takes any number of sets of variants and returns
            some combination of them (typically union or intersection).

        variant_collections : tuple of VariantCollection

        kwargs : dict
            Optional dictionary of keyword arguments to pass to the initializer
            for VariantCollection.
        """
        kwargs["variants"] = combine_fn(*[set(vc) for vc in variant_collections])
        kwargs["source_to_metadata_dict"] = cls._merge_metadata_dictionaries(
            [vc.source_to_metadata_dict for vc in variant_collections])
        kwargs["sources"] = set.union(*([vc.sources for vc in variant_collections]))
        for key, value in variant_collections[0].to_dict().items():
            # If some optional parameter isn't explicitly specified as an
            # argument to union() or intersection() then use the same value
            # as the first VariantCollection.
            #
            # I'm doing this so that the meaning of VariantCollection.union
            # and VariantCollection.intersection with a single argument is
            # the identity function (rather than setting optional parameters
            # to their default values.
            if key not in kwargs:
                kwargs[key] = value
        return cls(**kwargs)


Instruction:
How would you code a function in Python 3 to
return the union of two VariantCollection objects.

Response:
def union(self, *others, **kwargs):
        """
        Returns the union of variants in a several VariantCollection objects.
        """
        return self._combine_variant_collections(
            combine_fn=set.union,
            variant_collections=(self,) + others,
            kwargs=kwargs)


Instruction:
Can you write a function in Python 3 where it
returns the intersection of two VariantCollection objects.

Response:
def intersection(self, *others, **kwargs):
        """
        Returns the intersection of variants in several VariantCollection objects.
        """
        return self._combine_variant_collections(
            combine_fn=set.intersection,
            variant_collections=(self,) + others,
            kwargs=kwargs)


Instruction:
Explain what the following Python 3 code does
def to_dataframe(self):
        def row_from_variant(variant):
            return OrderedDict([
                ("chr", variant.contig),
                ("start", variant.original_start),
                ("ref", variant.original_ref),
                ("alt", variant.original_alt),
                ("gene_name", ";".join(variant.gene_names)),
                ("gene_id", ";".join(variant.gene_ids))
            ])
        rows = [row_from_variant(v) for v in self]
        if len(rows) == 0:
            # TODO: return a DataFrame with the appropriate columns
            return pd.DataFrame()
        return pd.DataFrame.from_records(rows, columns=rows[0].keys())

Response: Build a DataFrame from this variant collection


Instruction:
Write a Python 3 script for
reusing the `trim_shared_prefix` function above to implement similar functionality for string suffixes. Given ref='ABC' and alt='BC', we first revese both strings: reverse_ref = 'CBA' reverse_alt = 'CB' and then the result of calling trim_shared_prefix will be: ('A', '', 'CB') We then reverse all three of the result strings to get back the shared suffix and both prefixes leading up to it: ('A', '', 'BC')

Response:
def trim_shared_suffix(ref, alt):
    """
    Reuse the `trim_shared_prefix` function above to implement similar
    functionality for string suffixes.

    Given ref='ABC' and alt='BC', we first revese both strings:
        reverse_ref = 'CBA'
        reverse_alt = 'CB'
    and then the result of calling trim_shared_prefix will be:
        ('A', '', 'CB')
    We then reverse all three of the result strings to get back
    the shared suffix and both prefixes leading up to it:
        ('A', '', 'BC')
    """
    n_ref = len(ref)
    n_alt = len(alt)
    n_min = min(n_ref, n_alt)
    i = 0
    while i < n_min and ref[-i - 1] == alt[-i - 1]:
        i += 1

    # i is length of shared suffix.
    if i == 0:
        return (ref, alt, '')
    return (ref[:-i], alt[:-i], ref[-i:])


Instruction:
Write a Python 3 script for
giving two nucleotide or amino acid strings identify the common prefix and suffix of the two nucleotide strings and return the unique components along with the prefix and suffix.

Response:
def trim_shared_flanking_strings(ref, alt):
    """
    Given two nucleotide or amino acid strings, identify
    if they have a common prefix, a common suffix, and return
    their unique components along with the prefix and suffix.

    For example, if the input ref = "SYFFQGR" and alt = "SYMLLFIFQGR"
    then the result will be:
        ("F", "MLLFI", "SY", "FQGR")
    """
    ref, alt, prefix = trim_shared_prefix(ref, alt)
    ref, alt, suffix = trim_shared_suffix(ref, alt)
    return ref, alt, prefix, suffix


Instruction:
Write a Python 3 script to
script which loads variants and annotates them with overlapping genes and predicted coding effects. Example usage: varcode --vcf mutect.vcf \ --vcf strelka.vcf \ --maf tcga_brca.maf \ --variant chr1 498584 C G \ --json-variants more_variants.json

Response:
def main(args_list=None):
    """
    Script which loads variants and annotates them with overlapping genes
    and predicted coding effects.

    Example usage:
        varcode
            --vcf mutect.vcf \
            --vcf strelka.vcf \
            --maf tcga_brca.maf \
            --variant chr1 498584 C G \
            --json-variants more_variants.json
    """
    print_version_info()
    if args_list is None:
        args_list = sys.argv[1:]

    args = arg_parser.parse_args(args_list)
    variants = variant_collection_from_args(args)
    effects = variants.effects()
    if args.only_coding:
        effects = effects.drop_silent_and_noncoding()
    if args.one_per_variant:
        variant_to_effect_dict = effects.top_priority_effect_per_variant()
        effects = effects.clone_with_new_elements(list(variant_to_effect_dict.values()))

    effects_dataframe = effects.to_dataframe()
    logger.info('\n%s', effects)
    if args.output_csv:
        effects_dataframe.to_csv(args.output_csv, index=False)


Instruction:
Given the following Python 3 function, write the documentation
def get_codons(
        variant,
        trimmed_cdna_ref,
        trimmed_cdna_alt,
        sequence_from_start_codon,
        cds_offset):
    """
    Returns indices of first and last reference codons affected by the variant,
    as well as the actual sequence of the mutated codons which replace those
    reference codons.

    Parameters
    ----------
    variant : Variant

    trimmed_cdna_ref : str
        Trimmed reference cDNA nucleotides affected by the variant

    trimmed_cdna_alt : str
        Trimmed alternate cDNA nucleotides which replace the reference

    sequence_from_start_codon : str
        cDNA nucleotide coding sequence

    cds_offset : int
        Integer offset into the coding sequence where ref is replace with alt
    """
    # index (starting from 0) of first affected reference codon
    ref_codon_start_offset = cds_offset // 3
    # which nucleotide of the first codon got changed?
    nucleotide_offset_into_first_ref_codon = cds_offset % 3

    n_ref_nucleotides = len(trimmed_cdna_ref)
    if n_ref_nucleotides == 0:
        if nucleotide_offset_into_first_ref_codon == 2:
            # if we're inserting between codons
            ref_codon_end_offset = ref_codon_start_offset
        else:
            # inserting inside a reference codon
            ref_codon_end_offset = ref_codon_start_offset + 1
        ref_codons = sequence_from_start_codon[
            ref_codon_start_offset * 3:ref_codon_end_offset * 3]
        # split the reference codon into nucleotides before/after insertion
        prefix = ref_codons[:nucleotide_offset_into_first_ref_codon + 1]
        suffix = ref_codons[nucleotide_offset_into_first_ref_codon + 1:]
    else:
        ref_codon_end_offset = (cds_offset + n_ref_nucleotides - 1) // 3 + 1
        # codons in the reference sequence
        ref_codons = sequence_from_start_codon[
            ref_codon_start_offset * 3:ref_codon_end_offset * 3]

        # We construct the new codons by taking the unmodified prefix
        # of the first ref codon, the unmodified suffix of the last ref codon
        # and sticking the alt nucleotides in between.
        # Since this is supposed to be an in-frame mutation, the concatenated
        # nucleotide string is expected to have a length that is a multiple of
        # three.
        prefix = ref_codons[:nucleotide_offset_into_first_ref_codon]

        offset_in_last_ref_codon = (cds_offset + n_ref_nucleotides - 1) % 3

        if offset_in_last_ref_codon == 0:
            suffix = ref_codons[-2:]
        elif offset_in_last_ref_codon == 1:
            suffix = ref_codons[-1:]
        else:
            suffix = ""
    mutant_codons = prefix + trimmed_cdna_alt + suffix
    assert len(mutant_codons) % 3 == 0, \
        "Expected in-frame mutation but got %s (length = %d)" % (
            mutant_codons, len(mutant_codons))
    return ref_codon_start_offset, ref_codon_end_offset, mutant_codons

Response: Returns the indices of first and last reference codons affected by the variant and the actual sequence of the mutated codons.


Instruction:
Can you implement a function in Python 3 that
predicts the coding effect of an in - frame nucleotide change in - frame nucleotide change in - frame sequence.

Response:
def predict_in_frame_coding_effect(
        variant,
        transcript,
        trimmed_cdna_ref,
        trimmed_cdna_alt,
        sequence_from_start_codon,
        cds_offset):
    """Coding effect of an in-frame nucleotide change

    Parameters
    ----------
    variant : Variant

    transcript : Transcript

    trimmed_cdna_ref : str
        Reference nucleotides from the coding sequence of the transcript

    trimmed_cdna_alt : str
        Nucleotides to insert in place of the reference nucleotides

    sequence_from_start_codon : Bio.Seq or str
        Transcript sequence from the CDS start codon (including the 3' UTR).
        This sequence includes the 3' UTR since a mutation may delete the stop
        codon and we'll have to translate past the normal end of the CDS to
        determine the new protein sequence.

    cds_offset : int
        Index of first ref nucleotide, starting from 0 = beginning of coding
        sequence. If variant is a pure insertion (no ref nucleotides) then this
        argument indicates the offset *after* which to insert the alt
        nucleotides.
    """
    ref_codon_start_offset, ref_codon_end_offset, mutant_codons = get_codons(
        variant=variant,
        trimmed_cdna_ref=trimmed_cdna_ref,
        trimmed_cdna_alt=trimmed_cdna_alt,
        sequence_from_start_codon=sequence_from_start_codon,
        cds_offset=cds_offset)

    mutation_affects_start_codon = (ref_codon_start_offset == 0)

    if mutation_affects_start_codon and mutant_codons[:3] not in START_CODONS:
        # if we changed a start codon to something else then
        # we no longer know where the protein begins (or even in
        # what frame).
        # TODO: use the Kozak consensus sequence or a predictive model
        # to identify the most likely start site
        return StartLoss(
            variant=variant,
            transcript=transcript)

    # rely on Ensembl's annotation of the protein sequence since we can't
    # easily predict whether the starting nucleotide is a methionine
    # (most common) or leucine
    aa_ref = transcript.protein_sequence[ref_codon_start_offset:ref_codon_end_offset]

    reference_protein_length = len(transcript.protein_sequence)

    aa_alt, mutant_stop_codon_index, using_three_prime_utr = \
        translate_in_frame_mutation(
            transcript=transcript,
            ref_codon_start_offset=ref_codon_start_offset,
            ref_codon_end_offset=ref_codon_end_offset,
            mutant_codons=mutant_codons)

    mutant_codons_contain_stop = mutant_stop_codon_index != -1

    # trim shared subsequences at the start and end of reference
    # and mutated amino acid sequences
    aa_ref, aa_alt, shared_prefix, shared_suffix = \
        trim_shared_flanking_strings(
            aa_ref,
            aa_alt)

    n_aa_ref = len(aa_ref)
    n_aa_alt = len(aa_alt)
    n_aa_shared = len(shared_prefix)

    is_insertion = (ref_codon_start_offset == ref_codon_end_offset)

    # index of first amino acid which is different from the reference
    aa_mutation_start_offset = (
        ref_codon_start_offset + n_aa_shared + is_insertion)

    if mutant_codons_contain_stop:
        mutant_stop_codon_index += n_aa_shared

    if mutation_affects_start_codon and (aa_ref == aa_alt):
            # Substitution between start codons gets special treatment since,
            # though superficially synonymous, this could still potentially
            # cause a start loss / change in reading frame and might be worth
            # closer scrutiny
            return AlternateStartCodon(
                variant=variant,
                transcript=transcript,
                ref_codon=transcript.sequence[:3],
                alt_codon=mutant_codons[:3])

    n_ref_amino_acids_after_mutated_site = (
        reference_protein_length - aa_mutation_start_offset - 1)

    if mutant_codons_contain_stop and (
            n_aa_alt <= n_ref_amino_acids_after_mutated_site):
        # if the new coding sequence contains a stop codon, then this is a
        # PrematureStop mutation if it decreases the length of the protein
        return PrematureStop(
            variant=variant,
            transcript=transcript,
            aa_mutation_start_offset=aa_mutation_start_offset,
            aa_ref=aa_ref,
            aa_alt=aa_alt)

    if (aa_mutation_start_offset > reference_protein_length) or (
            n_aa_ref == n_aa_alt == 0):
        # if inserted nucleotides go after original stop codon or if nothing
        # is changed in the amino acid sequence then this is a Silent variant
        return Silent(
            variant=variant,
            transcript=transcript,
            aa_pos=aa_mutation_start_offset,
            aa_ref=shared_prefix + shared_suffix)
    elif using_three_prime_utr:
        # if non-silent mutation is at the end of the protein then
        # should be a stop-loss
        return StopLoss(
            variant,
            transcript,
            aa_ref=aa_ref,
            aa_alt=aa_alt)
    elif n_aa_alt == 0:
        return Deletion(
            variant,
            transcript,
            aa_mutation_start_offset=aa_mutation_start_offset,
            aa_ref=aa_ref)
    elif n_aa_ref == 0:
        return Insertion(
            variant,
            transcript,
            aa_mutation_start_offset=aa_mutation_start_offset,
            aa_alt=aa_alt)
    elif n_aa_ref == n_aa_alt == 1:
        # simple substitution e.g. p.V600E
        return Substitution(
            variant,
            transcript,
            aa_mutation_start_offset=aa_mutation_start_offset,
            aa_ref=aa_ref,
            aa_alt=aa_alt)
    else:
        # multiple amino acids were substituted e.g. p.VQQ39FF
        return ComplexSubstitution(
            variant,
            transcript,
            aa_mutation_start_offset=aa_mutation_start_offset,
            aa_ref=aa_ref,
            aa_alt=aa_alt)


Instruction:
Here you have a function in Python 3, explain what it does
def add_variant_args(arg_parser):
    variant_arg_group = arg_parser.add_argument_group(
        title="Variants",
        description="Genomic variant files")

    variant_arg_group.add_argument(
        "--vcf",
        default=[],
        action="append",
        help="Genomic variants in VCF format")

    variant_arg_group.add_argument(
        "--maf",
        default=[],
        action="append",
        help="Genomic variants in TCGA's MAF format",)

    variant_arg_group.add_argument(
        "--variant",
        default=[],
        action="append",
        nargs=4,
        metavar=("CHR", "POS", "REF", "ALT"),
        help=(
            "Individual variant as 4 arguments giving chromsome, position, ref,"
            " and alt. Example: chr1 3848 C G. Use '.' to indicate empty alleles"
            " for insertions or deletions."))

    variant_arg_group.add_argument(
        "--genome",
        type=str,
        help=(
            "What reference assembly your variant coordinates are using. "
            "Examples: 'hg19', 'GRCh38', or 'mm9'. "
            "This argument is ignored for MAF files, since each row includes "
            "the reference. "
            "For VCF files, this is used if specified, and otherwise is guessed from "
            "the header. For variants specfied on the commandline with --variant, "
            "this option is required."))

    variant_arg_group.add_argument(
        "--download-reference-genome-data",
        action="store_true",
        default=False,
        help=(
            ("Automatically download genome reference data required for "
             "annotation using PyEnsembl. Otherwise you must first run "
             "'pyensembl install' for the release/species corresponding "
             "to the genome used in your VCF.")))

    variant_arg_group.add_argument(
        "--json-variants",
        default=[],
        action="append",
        help="Path to Varcode.VariantCollection object serialized as a JSON file.")

    return variant_arg_group

Response: Adds the commandline arguments for the variant command line.


Instruction:
Given the following Python 3 function, write the documentation
def insert_before(sequence, offset, new_residues):
    assert 0 < offset <= len(sequence), \
        "Invalid position %d for sequence of length %d" % (
            offset, len(sequence))
    prefix = sequence[:offset]
    suffix = sequence[offset:]
    return prefix + new_residues + suffix

Response: Mutate the given sequence by inserting the string new_residues before offset.


Instruction:
Can you tell what is the following Python 3 function doing
def insert_after(sequence, offset, new_residues):
    assert 0 <= offset < len(sequence), \
        "Invalid position %d for sequence of length %d" % (
            offset, len(sequence))
    prefix = sequence[:offset + 1]
    suffix = sequence[offset + 1:]
    return prefix + new_residues + suffix

Response: Mutate the given sequence by inserting new_residues after offset.


Instruction:
Can you tell what is the following Python 3 function doing
def substitute(sequence, offset, ref, alt):
    n_ref = len(ref)
    sequence_ref = sequence[offset:offset + n_ref]
    assert str(sequence_ref) == str(ref), \
        "Reference %s at offset %d != expected reference %s" % \
        (sequence_ref, offset, ref)
    prefix = sequence[:offset]
    suffix = sequence[offset + n_ref:]
    return prefix + alt + suffix

Response: Mutate a sequence by substituting given alt at instead of ref at the given position.


Instruction:
Implement a Python 3 function for
giving a list of assemblies identify the most recent one

Response:
def _most_recent_assembly(assembly_names):
    """
    Given list of (in this case, matched) assemblies, identify the most recent
       ("recency" here is determined by sorting based on the numeric element of the assembly name)
    """
    match_recency = [
        int(re.search('\d+', assembly_name).group())
        for assembly_name in assembly_names
    ]
    most_recent = [
        x for (y, x) in sorted(zip(match_recency, assembly_names), reverse=True)][0]
    return most_recent


Instruction:
Write a Python 3 script to
infer a canonical name from a string containing a reference name or path.

Response:
def infer_reference_name(reference_name_or_path):
    """
    Given a string containing a reference name (such as a path to
    that reference's FASTA file), return its canonical name
    as used by Ensembl.
    """
    # identify all cases where reference name or path matches candidate aliases
    reference_file_name = os.path.basename(reference_name_or_path)
    matches = {'file_name': list(), 'full_path': list()}
    for assembly_name in reference_alias_dict.keys():
        candidate_list = [assembly_name] + reference_alias_dict[assembly_name]
        for candidate in candidate_list:
            if candidate.lower() in reference_file_name.lower():
                matches['file_name'].append(assembly_name)
            elif candidate.lower() in reference_name_or_path.lower():
                matches['full_path'].append(assembly_name)
    # remove duplicate matches (happens due to overlapping aliases)
    matches['file_name'] = list(set(matches['file_name']))
    matches['full_path'] = list(set(matches['full_path']))
    # given set of existing matches, choose one to return
    # (first select based on file_name, then full path. If multiples, use most recent)
    if len(matches['file_name']) == 1:
        match = matches['file_name'][0]
    elif len(matches['file_name']) > 1:
        # separate logic for >1 vs 1 to give informative warning
        match = _most_recent_assembly(matches['file_name'])
        warn(
            ('More than one reference ({}) matches path in header ({}); '
             'the most recent one ({}) was used.').format(
                ','.join(matches['file_name']), reference_file_name, match))
    elif len(matches['full_path']) >= 1:
        # combine full-path logic since warning is the same
        match = _most_recent_assembly(matches['full_path'])
        warn((
            'Reference could not be matched against filename ({}); '
            'using best match against full path ({}).').format(
                reference_name_or_path, match))
    else:
        raise ValueError(
            "Failed to infer genome assembly name for %s" % reference_name_or_path)
    return match


Instruction:
Create a Python 3 function to
infer the Ensembl version of a given genome.

Response:
def infer_genome(genome_object_string_or_int):
    """
    If given an integer, return associated human EnsemblRelease for that
    Ensembl version.

    If given a string, return latest EnsemblRelease which has a reference
    of the same name.

    If given a PyEnsembl Genome, simply return it.
    """
    if isinstance(genome_object_string_or_int, Genome):
        return genome_object_string_or_int
    if is_integer(genome_object_string_or_int):
        return cached_release(genome_object_string_or_int)
    elif is_string(genome_object_string_or_int):
        # first infer the canonical reference name, e.g. mapping hg19 -> GRCh37
        # and then get the associated PyEnsembl Genome object
        reference_name = infer_reference_name(genome_object_string_or_int)
        return genome_for_reference_name(reference_name)
    else:
        raise TypeError(
            ("Expected genome to be an int, string, or pyensembl.Genome "
                "instance, got %s : %s") % (
                str(genome_object_string_or_int),
                type(genome_object_string_or_int)))


Instruction:
Write a Python 3 script to
return a dictionary representation of the object.

Response:
def to_dict(self):
        """
        We want the original values (un-normalized) field values while
        serializing since normalization will happen in __init__.
        """
        return dict(
            contig=self.original_contig,
            start=self.original_start,
            ref=self.original_ref,
            alt=self.original_alt,
            ensembl=self.ensembl,
            allow_extended_nucleotides=self.allow_extended_nucleotides,
            normalize_contig_name=self.normalize_contig_name)


Instruction:
Write a Python 3 script for
returning a short description of the genomic variant.

Response:
def short_description(self):
        """
        HGVS nomenclature for genomic variants
        More info: http://www.hgvs.org/mutnomen/
        """
        if self.is_insertion:
            return "chr%s g.%d_%dins%s" % (
                self.contig,
                self.start,
                self.start + 1,
                self.alt)
        elif self.is_deletion:
            return "chr%s g.%d_%ddel%s" % (
                self.contig,
                self.start,
                self.end,
                self.ref)
        elif self.ref == self.alt:
            return "chr%s g.%d%s" % (self.contig, self.start, self.ref)
        else:
            # substitution
            return "chr%s g.%d%s>%s" % (
                self.contig,
                self.start,
                self.ref,
                self.alt)


Instruction:
Can you generate the documentation for the following Python 3 function
def genes(self):
        if self._genes is None:
            self._genes = self.ensembl.genes_at_locus(
                self.contig, self.start, self.end)
        return self._genes

Response: Return a Gene object for all genes which overlap this variant.


Instruction:
Here you have a function in Python 3, explain what it does
def gene_ids(self):
        return self.ensembl.gene_ids_at_locus(
            self.contig, self.start, self.end)

Response: Returns a list of gene IDs that overlap this variant.


Instruction:
Make a summary of the following Python 3 code
def gene_names(self):
        return self.ensembl.gene_names_at_locus(
            self.contig, self.start, self.end)

Response: Return a list of all gene names that overlap this variant.


Instruction:
Write a Python 3 script for
returning True if this variant represents an insertion of nucleotides into the reference genome.

Response:
def is_insertion(self):
        """
        Does this variant represent the insertion of nucleotides into the
        reference genome?
        """
        # An insertion would appear in a VCF like C>CT, so that the
        # alternate allele starts with the reference nucleotides.
        # Since the nucleotide strings may be normalized in the constructor,
        # it's worth noting that the normalized form of this variant would be
        # ''>'T', so that 'T'.startswith('') still holds.
        return (len(self.ref) < len(self.alt)) and self.alt.startswith(self.ref)


Instruction:
Write a Python 3 function for
being this variant represent the deletion of nucleotides from the reference genome?

Response:
def is_deletion(self):
        """
        Does this variant represent the deletion of nucleotides from the
        reference genome?
        """
        # A deletion would appear in a VCF like CT>C, so that the
        # reference allele starts with the alternate nucleotides.
        # This is true even in the normalized case, where the alternate
        # nucleotides are an empty string.
        return (len(self.ref) > len(self.alt)) and self.ref.startswith(self.alt)


Instruction:
Can you create a Python 3 function that
is the variant a single nucleotide variant

Response:
def is_snv(self):
        """Is the variant a single nucleotide variant"""
        return (len(self.ref) == len(self.alt) == 1) and (self.ref != self.alt)


Instruction:
Write a Python 3 script to
am this variant and pyrimidine to pyrimidine change?

Response:
def is_transition(self):
        """Is this variant and pyrimidine to pyrimidine change or purine to purine change"""
        return self.is_snv and is_purine(self.ref) == is_purine(self.alt)


Instruction:
Can you write a function in Python 3 where it
is this variant a pyrimidine to purine change vice versa?

Response:
def is_transversion(self):
        """Is this variant a pyrimidine to purine change or vice versa"""
        return self.is_snv and is_purine(self.ref) != is_purine(self.alt)


Instruction:
Can you generate the documentation for the following Python 3 function
def variant_overlaps_interval(
        variant_start,
        n_ref_bases,
        interval_start,
        interval_end):
    """
    Does a variant overlap a given interval on the same chromosome?

    Parameters
    ----------
    variant_start : int
        Inclusive base-1 position of variant's starting location
        (or location before an insertion)

    n_ref_bases : int
        Number of reference bases affect by variant (used to compute
        end coordinate or determine whether variant is an insertion)

    interval_start : int
        Interval's inclusive base-1 start position

    interval_end : int
        Interval's inclusive base-1 end position
    """

    if n_ref_bases == 0:
        # insertions only overlap intervals which start before and
        # end after the insertion point, they must be fully contained
        # by the other interval
        return interval_start <= variant_start and interval_end >= variant_start
    variant_end = variant_start + n_ref_bases
    """
    if self._changes_exonic_splice_site(
            strand_ref,
            strand_alt,)
    """
    # overlap means other interval starts before this variant ends
    # and the interval ends after this variant starts
    return interval_start <= variant_end and interval_end >= variant_start

Response: Returns True if the given interval of a given variant is contained within the given base - 1 interval.


Instruction:
Given the following Python 3 function, write the documentation
def changes_exonic_splice_site(
        transcript_offset,
        transcript,
        transcript_ref,
        transcript_alt,
        exon_start_offset,
        exon_end_offset,
        exon_number):
    """Does the given exonic mutation of a particular transcript change a
    splice site?

    Parameters
    ----------
    transcript_offset : int
        Offset from start of transcript of first reference nucleotide
        (or the last nucleotide before an insertion)

    transcript : pyensembl.Transcript

    transcript_ref : str
        Reference nucleotides

    transcript_alt : alt
        Alternate nucleotides

    exon_start_offset : int
        Start offset of exon relative to beginning of transcript

    exon_end_offset : int
        End offset of exon relative to beginning of transcript

    exon_number : int
        Which exon in the order they form the transcript
    """
    # first we're going to make sure the variant doesn't disrupt the
    # splicing sequences we got from Divina et. al's
    #   Ab initio prediction of mutation-induced cryptic
    #   splice-site activation and exon skipping
    # (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2947103/)
    #
    # 5' splice site: MAG|GURAGU consensus
    #   M is A or C; R is purine; | is the exon-intron boundary
    #
    # 3' splice site: YAG|R
    #
    if exon_number > 1 and transcript_offset == exon_start_offset:
        # if this is any exon past the first, check to see if it lost
        # the purine on its left side
        #
        # the 3' splice site sequence has just a single purine on
        # the exon side
        if len(transcript_ref) > 0 and transcript_ref[0] in PURINE_NUCLEOTIDES:
            if len(transcript_alt) > 0:
                if transcript_alt[0] not in PURINE_NUCLEOTIDES:
                    return True
            else:
                # if the mutation is a deletion, are there ref nucleotides
                # afterward?
                offset_after_deletion = transcript_offset + len(transcript_ref)
                if len(transcript.sequence) > offset_after_deletion:
                    next_base = transcript.sequence[offset_after_deletion]
                    if next_base not in PURINE_NUCLEOTIDES:
                        return True

    if exon_number < len(transcript.exons):
        # if the mutation affects an exon whose right end gets spliced
        # to a next exon, check if the variant alters the exon side of
        # 5' consensus splicing sequence
        #
        # splicing sequence:
        #   MAG|GURAGU
        # M is A or C; R is purine; | is the exon-intron boundary
        #
        # TODO: check for overlap of two intervals instead of just
        # seeing if the mutation starts inside the exonic splice site
        if variant_overlaps_interval(
                variant_start=transcript_offset,
                n_ref_bases=len(transcript_ref),
                interval_start=exon_end_offset - 2,
                interval_end=exon_end_offset):
            end_of_reference_exon = transcript.sequence[
                exon_end_offset - 2:exon_end_offset + 1]

            if matches_exon_end_pattern(end_of_reference_exon):
                # if the last three nucleotides conform to the consensus
                # sequence then treat any deviation as an ExonicSpliceSite
                # mutation
                end_of_variant_exon = end_of_reference_exon
                if matches_exon_end_pattern(end_of_variant_exon):
                    # end of exon matches splicing signal, check if it still
                    # does after the mutation
                    return True

Response: Does the given exonic mutation of a particular transcript change a specific splice site?


Instruction:
Can you implement a function in Python 3 that
is the nucleotide a purine

Response:
def is_purine(nucleotide, allow_extended_nucleotides=False):
    """Is the nucleotide a purine"""
    if not allow_extended_nucleotides and nucleotide not in STANDARD_NUCLEOTIDES:
        raise ValueError(
            "{} is a non-standard nucleotide, neither purine or pyrimidine".format(nucleotide))
    return nucleotide in PURINE_NUCLEOTIDES


Instruction:
Can you write a function in Python 3 where it
normalizes a nucleotide string by converting all letters upper case checking to make sure all letters in the string are actually nucleotides.

Response:
def normalize_nucleotide_string(
        nucleotides,
        allow_extended_nucleotides=False,
        empty_chars=".-",
        treat_nan_as_empty=True):
    """
    Normalizes a nucleotide string by converting various ways of encoding empty
    strings into "", making all letters upper case, and checking to make sure
    all letters in the string are actually nucleotides.

    Parameters
    ----------
    nucleotides : str
        Sequence of nucleotides, e.g. "ACCTG"

    extended_nucleotides : bool
        Allow non-canonical nucleotide characters like 'X' for unknown base

    empty_chars : str
        Characters which encode empty strings, such as "." used in VCF format
        or "-" used in MAF format

    treat_nan_as_empty : bool
        Some MAF files represent deletions/insertions with NaN ref/alt values
    """
    if nucleotides in empty_chars:
        return ""
    elif treat_nan_as_empty and isinstance(nucleotides, float) and np.isnan(nucleotides):
        return ""

    require_string(nucleotides, name="nucleotide string")

    nucleotides = nucleotides.upper()

    if allow_extended_nucleotides:
        valid_nucleotides = EXTENDED_NUCLEOTIDES
    else:
        valid_nucleotides = STANDARD_NUCLEOTIDES

    if not set(nucleotides) <= valid_nucleotides:
        raise ValueError(
            "Invalid character(s) in nucleotide string: %s" % (
                ",".join(set(nucleotides) - valid_nucleotides),))

    return nucleotides


Instruction:
Create a Python 3 function to
load a VCF file into memory.

Response:
def load_vcf(
        path,
        genome=None,
        reference_vcf_key="reference",
        only_passing=True,
        allow_extended_nucleotides=False,
        include_info=True,
        chunk_size=10 ** 5,
        max_variants=None,
        sort_key=variant_ascending_position_sort_key,
        distinct=True):
    """
    Load reference name and Variant objects from the given VCF filename.

    Currently only local files are supported by this function (no http). If you
    call this on an HTTP URL, it will fall back to `load_vcf`.

    Parameters
    ----------

    path : str
        Path to VCF (*.vcf) or compressed VCF (*.vcf.gz).

    genome : {pyensembl.Genome, reference name, Ensembl version int}, optional
        Optionally pass in a PyEnsembl Genome object, name of reference, or
        PyEnsembl release version to specify the reference associated with a
        VCF (otherwise infer reference from VCF using reference_vcf_key)

    reference_vcf_key : str, optional
        Name of metadata field which contains path to reference FASTA
        file (default = 'reference')

    only_passing : boolean, optional
        If true, any entries whose FILTER field is not one of "." or "PASS" is
        dropped.

    allow_extended_nucleotides : boolean, default False
        Allow characters other that A,C,T,G in the ref and alt strings.

    include_info : boolean, default True
        Whether to parse the INFO and per-sample columns. If you don't need
        these, set to False for faster parsing.

    chunk_size: int, optional
        Number of records to load in memory at once.

    max_variants : int, optional
        If specified, return only the first max_variants variants.

    sort_key : fn
        Function which maps each element to a sorting criterion.
        Set to None to not to sort the variants.

    distinct : boolean, default True
        Don't keep repeated variants
    """

    require_string(path, "Path or URL to VCF")
    parsed_path = parse_url_or_path(path)

    if parsed_path.scheme and parsed_path.scheme.lower() != "file":
        # pandas.read_table nominally supports HTTP, but it tends to crash on
        # large files and does not support gzip. Switching to the python-based
        # implementation of read_table (with engine="python") helps with some
        # issues but introduces a new set of problems (e.g. the dtype parameter
        # is not accepted). For these reasons, we're currently not attempting
        # to load VCFs over HTTP with pandas directly, and instead download it
        # to a temporary file and open that.

        (filename, headers) = urllib.request.urlretrieve(path)
        try:
            # The downloaded file has no file extension, which confuses pyvcf
            # for gziped files in Python 3. We rename it to have the correct
            # file extension.
            new_filename = "%s.%s" % (
                filename, parsed_path.path.split(".")[-1])
            os.rename(filename, new_filename)
            filename = new_filename
            return load_vcf(
                filename,
                genome=genome,
                reference_vcf_key=reference_vcf_key,
                only_passing=only_passing,
                allow_extended_nucleotides=allow_extended_nucleotides,
                include_info=include_info,
                chunk_size=chunk_size,
                max_variants=max_variants,
                sort_key=sort_key,
                distinct=distinct)
        finally:
            logger.info("Removing temporary file: %s", filename)
            os.unlink(filename)

    # Loading a local file.
    # The file will be opened twice: first to parse the header with pyvcf, then
    # by pandas to read the data.

    # PyVCF reads the metadata immediately and stops at the first line with
    # data. We can close the file after that.
    handle = PyVCFReaderFromPathOrURL(path)
    handle.close()
    genome = infer_genome_from_vcf(
        genome,
        handle.vcf_reader,
        reference_vcf_key)

    df_iterator = read_vcf_into_dataframe(
        path,
        include_info=include_info,
        sample_names=handle.vcf_reader.samples if include_info else None,
        chunk_size=chunk_size)

    if include_info:
        def sample_info_parser(unparsed_sample_info_strings, format_string):
            """
            Given a format string like "GT:AD:ADP:DP:FS"
            and a list of sample info strings where each entry is like
            "0/1:3,22:T=3,G=22:25:33", return a dict that maps:
            sample name -> field name -> value. Uses pyvcf to parse the fields.
            """
            return pyvcf_calls_to_sample_info_list(
                handle.vcf_reader._parse_samples(
                    unparsed_sample_info_strings, format_string, None))
    else:
        sample_info_parser = None

    return dataframes_to_variant_collection(
        df_iterator,
        source_path=path,
        info_parser=handle.vcf_reader._parse_info if include_info else None,
        only_passing=only_passing,
        max_variants=max_variants,
        sample_names=handle.vcf_reader.samples if include_info else None,
        sample_info_parser=sample_info_parser,
        variant_kwargs={
            'ensembl': genome,
            'allow_extended_nucleotides': allow_extended_nucleotides},
        variant_collection_kwargs={
            'sort_key': sort_key,
            'distinct': distinct})


Instruction:
Make a summary of the following Python 3 code
def pyvcf_calls_to_sample_info_list(calls):
    return OrderedDict(
        (call.sample, call.data._asdict()) for call in calls)

Response: Given a list of pyvcf. model. _Call instances return a dict mapping each sample name to its per - sample info.


Instruction:
Can you write a function in Python 3 where it
loads a VariantCollection from a list of pandas dataframes.

Response:
def dataframes_to_variant_collection(
        dataframes,
        source_path,
        info_parser=None,
        only_passing=True,
        max_variants=None,
        sample_names=None,
        sample_info_parser=None,
        variant_kwargs={},
        variant_collection_kwargs={}):
    """
    Load a VariantCollection from an iterable of pandas dataframes.

    This takes an iterable of dataframes instead of a single dataframe to avoid
    having to load huge dataframes at once into memory. If you have a single
    dataframe, just pass it in a single-element list.

    Parameters
    ----------
    dataframes
        Iterable of dataframes (e.g. a generator). Expected columns are:
            ["CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER"]
        and 'INFO' if `info_parser` is not Null. Columns must be in this
        order.

    source_path : str
        Path of VCF file from which DataFrame chunks were generated.

    info_parser : string -> object, optional
        Callable to parse INFO strings.

    only_passing : boolean, optional
        If true, any entries whose FILTER field is not one of "." or "PASS" is
        dropped.

    max_variants : int, optional
        If specified, return only the first max_variants variants.

    sample_names : list of strings, optional
        Sample names. The final columns of the dataframe should match these.
        If specified, the per-sample info columns will be parsed. You must
        also specify sample_info_parser.

    sample_info_parser : string list * string -> dict, optional
        Callable to parse per-sample info columns.

    variant_kwargs : dict, optional
        Additional keyword paramters to pass to Variant.__init__

    variant_collection_kwargs : dict, optional
        Additional keyword parameters to pass to VariantCollection.__init__.
    """

    expected_columns = (
        ["CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER"] +
        (["INFO"] if info_parser else []))

    if info_parser and sample_names:
        if sample_info_parser is None:
            raise TypeError(
                "Must specify sample_info_parser if specifying sample_names")
        expected_columns.append("FORMAT")
        expected_columns.extend(sample_names)

    variants = []
    metadata = {}
    try:
        for chunk in dataframes:
            assert chunk.columns.tolist() == expected_columns,\
                "dataframe columns (%s) do not match expected columns (%s)" % (
                    chunk.columns, expected_columns)

            for tpl in chunk.itertuples():
                (i, chrom, pos, id_, ref, alts, qual, flter) = tpl[:8]
                if flter == ".":
                    flter = None
                elif flter == "PASS":
                    flter = []
                elif only_passing:
                    continue
                else:
                    flter = flter.split(';')
                if id_ == ".":
                    id_ = None
                qual = float(qual) if qual != "." else None
                alt_num = 0
                info = sample_info = None
                for alt in alts.split(","):
                    if alt != ".":
                        if info_parser is not None and info is None:
                            info = info_parser(tpl[8])  # INFO column
                            if sample_names:
                                # Sample name -> field -> value dict.
                                sample_info = sample_info_parser(
                                    list(tpl[10:]),  # sample info columns
                                    tpl[9],    # FORMAT column
                                )

                        variant = Variant(
                            chrom,
                            int(pos),  # want a Python int not numpy.int64
                            ref,
                            alt,
                            **variant_kwargs)
                        variants.append(variant)
                        metadata[variant] = {
                            'id': id_,
                            'qual': qual,
                            'filter': flter,
                            'info': info,
                            'sample_info': sample_info,
                            'alt_allele_index': alt_num,
                        }
                        if max_variants and len(variants) > max_variants:
                            raise StopIteration
                    alt_num += 1
    except StopIteration:
        pass

    return VariantCollection(
        variants=variants,
        source_to_metadata_dict={source_path: metadata},
        **variant_collection_kwargs)


Instruction:
Implement a function in Python 3 to
read the contents of a VCF file into a pandas dataframe.

Response:
def read_vcf_into_dataframe(
        path,
        include_info=False,
        sample_names=None,
        chunk_size=None):
    """
    Load the data of a VCF into a pandas dataframe. All headers are ignored.

    Parameters
    ----------
    path : str
        Path to local file. HTTP and other protocols are not implemented.

    include_info : boolean, default False
        If true, the INFO field is not parsed, but is included as a string in
        the resulting data frame. If false, the INFO field is omitted.

    sample_names: string list, optional
        Sample names. The final columns of the dataframe should match these.
        If specified (and include_info is also specified), the FORMAT and
        per-sample info columns will be included in the result dataframe.

    chunk_size : int, optional
        If buffering is desired, the number of rows per chunk.

    Returns
    ---------
    If chunk_size is None (the default), a dataframe with the contents of the
    VCF file. Otherwise, an iterable of dataframes, each with chunk_size rows.

    """
    vcf_field_types = OrderedDict()
    vcf_field_types['CHROM'] = str
    vcf_field_types['POS'] = int
    vcf_field_types['ID'] = str
    vcf_field_types['REF'] = str
    vcf_field_types['ALT'] = str
    vcf_field_types['QUAL'] = str
    vcf_field_types['FILTER'] = str
    if include_info:
        vcf_field_types['INFO'] = str
        if sample_names:
            vcf_field_types['FORMAT'] = str
            for name in sample_names:
                vcf_field_types[name] = str

    parsed_path = parse_url_or_path(path)
    if not parsed_path.scheme or parsed_path.scheme.lower() == "file":
        path = parsed_path.path
    else:
        raise NotImplementedError("Only local files are supported.")

    compression = None
    if path.endswith(".gz"):
        compression = "gzip"
    elif path.endswith(".bz2"):
        compression = "bz2"

    reader = pandas.read_table(
        path,
        compression=compression,
        comment="#",
        chunksize=chunk_size,
        dtype=vcf_field_types,
        names=list(vcf_field_types),
        usecols=range(len(vcf_field_types)))
    return reader


Instruction:
Explain what the following Python 3 code does
def stream_gzip_decompress_lines(stream):
    dec = zlib.decompressobj(zlib.MAX_WBITS | 16)
    previous = ""
    for compressed_chunk in stream:
        chunk = dec.decompress(compressed_chunk).decode()
        if chunk:
            lines = (previous + chunk).split("\n")
            previous = lines.pop()
            for line in lines:
                yield line
    yield previous

Response: Decompress a gzip stream into lines of text.


Instruction:
Here you have a function in Python 3, explain what it does
def infer_genome_from_vcf(genome, vcf_reader, reference_vcf_key):
    if genome:
        return infer_genome(genome)
    elif reference_vcf_key not in vcf_reader.metadata:
        raise ValueError("Unable to infer reference genome for %s" % (
            vcf_reader.filename,))
    else:
        reference_path = vcf_reader.metadata[reference_vcf_key]
        return infer_genome(reference_path)

Response: Helper function to make a pyensembl. Genome instance from a vcf file.