Instruction:
Implement a Python 3 function for
scanning all PWM sequences with this motif.

Response:
def pwm_scan_all(self, fa, cutoff=0.9, nreport=50, scan_rc=True):
        """Scan sequences with this motif.

        Scan sequences from a FASTA object with this motif. Less efficient
        than using a Scanner object. By setting the cutoff to 0.0 and
        nreport to 1, the best match for every sequence will be returned.
        The score, position and strand for every match is returned.

        Parameters
        ----------
        fa : Fasta object
            Fasta object to scan.
        cutoff : float , optional
            Cutoff to use for motif scanning. This cutoff is not specifically
            optimized and the strictness will vary a lot with motif lengh.
        nreport : int , optional
            Maximum number of matches to report.
        scan_rc : bool , optional
            Scan the reverse complement. True by default.

        Returns
        -------
        matches : dict
            Dictionary with motif matches. The score, position and strand for
            every match is returned.
        """
        c = self.pwm_min_score() + (self.pwm_max_score() - self.pwm_min_score()) * cutoff
        pwm = self.pwm
        matches = {}
        for name, seq in fa.items():
            matches[name] = []
            result = pfmscan(seq.upper(), pwm, c, nreport, scan_rc)
            for score,pos,strand in result:
                matches[name].append((pos,score,strand))
        return matches


Instruction:
Can you write a function in Python 3 where it
scans a FASTA file with this motif and save it to a GFF file.

Response:
def pwm_scan_to_gff(self, fa, gfffile, cutoff=0.9, nreport=50, scan_rc=True, append=False):
        """Scan sequences with this motif and save to a GFF file.

        Scan sequences from a FASTA object with this motif. Less efficient
        than using a Scanner object. By setting the cutoff to 0.0 and
        nreport to 1, the best match for every sequence will be returned.
        The output is save to a file in GFF format.

        Parameters
        ----------
        fa : Fasta object
            Fasta object to scan.
        gfffile : str
            Filename of GFF output file.
        cutoff : float , optional
            Cutoff to use for motif scanning. This cutoff is not specifically
            optimized and the strictness will vary a lot with motif lengh.
        nreport : int , optional
            Maximum number of matches to report.
        scan_rc : bool , optional
            Scan the reverse complement. True by default.
        append : bool , optional
            Append to GFF file instead of overwriting it. False by default.
        """
        if append:
            out = open(gfffile, "a")
        else:
            out = open(gfffile, "w")

        c = self.pwm_min_score() + (self.pwm_max_score() - self.pwm_min_score()) * cutoff
        pwm = self.pwm

        strandmap = {-1:"-","-1":"-","-":"-","1":"+",1:"+","+":"+"}
        gff_line = ("{}\tpfmscan\tmisc_feature\t{}\t{}\t{:.3f}\t{}\t.\t"
                    "motif_name \"{}\" ; motif_instance \"{}\"\n")
        for name, seq in fa.items():
            result = pfmscan(seq.upper(), pwm, c, nreport, scan_rc)
            for score, pos, strand in result:
                out.write(gff_line.format(
                    name,
                    pos,
                    pos + len(pwm),
                    score,
                    strandmap[strand],
                    self.id,
                    seq[pos:pos + len(pwm)]
                    ))
        out.close()


Instruction:
Create a Python 3 function for
returning the average of two motifs.

Response:
def average_motifs(self, other, pos, orientation, include_bg=False):
        """Return the average of two motifs.

        Combine this motif with another motif and return the average as a new
        Motif object. The position and orientatien need to be supplied. The pos
        parameter is the position of the second motif relative to this motif.

        For example, take the following two motifs:
        Motif 1: CATGYT
        Motif 2: GGCTTGY

        With position -2, the motifs are averaged as follows:
        xxCATGYT
        GGCTTGYx

        Parameters
        ----------
        other : Motif object
            Other Motif object.
        pos : int
            Position of the second motif relative to this motif.
        orientation : int
            Orientation, should be 1 or -1. If the orientation is -1 then the
            reverse complement of the other motif is used for averaging.
        include_bg : bool , optional
            Extend both motifs with background frequencies (0.25) before
            averaging. False by default.

        Returns
        -------
        motif : motif object
            New Motif object containing average motif.
        """
        # xxCATGYT
        # GGCTTGYx
        # pos = -2
        pfm1 = self.pfm[:]
        pfm2 = other.pfm[:]

        if orientation < 0:
            pfm2 = [row[::-1] for row in pfm2[::-1]]

        pfm1_count = float(np.sum(pfm1[0]))
        pfm2_count = float(np.sum(pfm2[0]))

        if include_bg:
            if len(pfm1) > len(pfm2) + pos:
                pfm2 += [[pfm2_count / 4.0 for x in range(4)] for i in range(-(len(pfm1) - len(pfm2) - pos), 0)]
            elif len(pfm2) + pos > len(pfm1):
                pfm1 += [[pfm1_count / 4.0 for x in range(4)] for i in range(-(len(pfm2) - len(pfm1) + pos), 0)]

            if pos < 0:
                pfm1 = [[pfm1_count / 4.0 for x in range(4)] for i in range(-pos)] + pfm1
            elif pos > 0:
                pfm2 = [[pfm2_count / 4.0 for x in range(4)] for i in range(pos)] + pfm2

        else:
            if len(pfm1) > len(pfm2) + pos:
                pfm2 += [[pfm1[i][x] / pfm1_count * (pfm2_count) for x in range(4)] for i in range(-(len(pfm1) - len(pfm2) - pos), 0)]
            elif len(pfm2) + pos > len(pfm1):
                pfm1 += [[pfm2[i][x] / pfm2_count * (pfm1_count) for x in range(4)] for i in range(-(len(pfm2) - len(pfm1) + pos), 0)]

            if pos < 0:
                pfm1 = [[pfm2[i][x] / pfm2_count * (pfm1_count) for x in range(4)] for i in range(-pos)] + pfm1
            elif pos > 0:
                pfm2 = [[pfm1[i][x] / pfm1_count * (pfm2_count) for x in range(4)] for i in range(pos)] + pfm2

        pfm = [[a + b for a,b in zip(x,y)] for x,y in zip(pfm1, pfm2)]

        m = Motif(pfm)
        m.id = m.to_consensus()
        return m


Instruction:
Can you tell what is the following Python 3 function doing
def _pwm_to_str(self, precision=4):
        if not self.pwm:
            return ""

        fmt = "{{:.{:d}f}}".format(precision)
        return "\n".join(
                ["\t".join([fmt.format(p) for p in row])
                for row in self.pwm]
                )

Response: Return string representation of the pwm.


Instruction:
How would you explain what the following Python 3 function does
def to_pwm(self, precision=4, extra_str=""):
        motif_id = self.id

        if extra_str:
            motif_id += "_%s" % extra_str

        if not self.pwm:
            self.pwm = [self.iupac_pwm[char]for char in self.consensus.upper()]

        return ">%s\n%s" % (
                motif_id,
                self._pwm_to_str(precision)
            )

Response: Return a string representation of the Amino Acid motif.


Instruction:
Explain what the following Python 3 code does
def to_img(self, fname, fmt="PNG", add_left=0, seqlogo=None, height=6):
        if not seqlogo:
            seqlogo = self.seqlogo
        if not seqlogo:
            raise ValueError("seqlogo not specified or configured")

        #TODO: split to_align function

        VALID_FORMATS = ["EPS", "GIF", "PDF", "PNG"]
        N = 1000
        fmt = fmt.upper()
        if not fmt in VALID_FORMATS:
            sys.stderr.write("Invalid motif format\n")
            return

        if fname[-4:].upper() == (".%s" % fmt):
            fname = fname[:-4]
        seqs = []
        if add_left == 0:
            seqs = ["" for i in range(N)]
        else:
            for nuc in ["A", "C", "T", "G"]:
                seqs += [nuc * add_left for i in range(N // 4)]

        for pos in range(len(self.pwm)):
            vals = [self.pwm[pos][0] * N]
            for i in range(1,4):
                vals.append(vals[i-1] + self.pwm[pos][i] * N)
            if vals[3] - N != 0:
                #print "Motif weights don't add up to 1! Error of %s%%" % ((vals[3] - n)/ n * 100)
                vals[3] = N
            for i in range(N):
                if i <= vals[0]:
                    seqs[i] += "A"
                elif i <= vals[1]:
                    seqs[i] += "C"
                elif i <= vals[2]:
                    seqs[i] += "G"
                elif i <= vals[3]:
                    seqs[i] += "T"

        f = NamedTemporaryFile(mode="w", dir=mytmpdir())
        for seq in seqs:
            f.write("%s\n" % seq)
        f.flush()
        makelogo = "{0} -f {1} -F {2} -c -a -h {3} -w {4} -o {5} -b -n -Y"
        cmd = makelogo.format(
                              seqlogo,
                              f.name,
                              fmt,
                              height,
                              len(self) + add_left,
                              fname)
        sp.call(cmd, shell=True)

Response: Create a sequence logo and save it to a file.


Instruction:
Can you create a Python 3 function that
creates a new Motif instance with shuffled positions.

Response:
def randomize(self):
        """Create a new motif with shuffled positions.

        Shuffle the positions of this motif and return a new Motif instance.

        Returns
        -------
        m : Motif instance
            Motif instance with shuffled positions.
        """
        random_pfm = [[c for c in row] for row in self.pfm]
        random.shuffle(random_pfm)
        m = Motif(pfm=random_pfm)
        m.id = "random"
        return m


Instruction:
Make a summary of the following Python 3 code
def maelstrom(args):
    infile = args.inputfile
    genome = args.genome
    outdir = args.outdir
    pwmfile = args.pwmfile
    methods = args.methods
    ncpus = args.ncpus

    if not os.path.exists(infile):
        raise ValueError("file {} does not exist".format(infile))

    if methods:
        methods = [x.strip() for x in methods.split(",")]

    run_maelstrom(infile, genome, outdir, pwmfile, methods=methods, ncpus=ncpus)

Response: Run the maelstrom method.


Instruction:
Explain what the following Python 3 code does
def zmq_send_data(socket, data, flags=0, copy=True, track=False):

    header, payload = [], []

    for key in sorted(data.keys()):
        arr = data[key]

        if not isinstance(arr, np.ndarray):
            raise DataError('Only ndarray types can be serialized')

        header.append(dict(dtype=str(arr.dtype),
                           shape=arr.shape,
                           key=key,
                           aligned=arr.flags['ALIGNED']))
        # Force contiguity
        payload.append(arr)

    # Send the header
    msg = [json.dumps(header).encode('ascii')]
    msg.extend(payload)

    return socket.send_multipart(msg, flags, copy=copy, track=track)

Response: Send data to a socket


Instruction:
Make a summary of the following Python 3 code
def zmq_recv_data(socket, flags=0, copy=True, track=False):

    data = dict()

    msg = socket.recv_multipart(flags=flags, copy=copy, track=track)

    headers = json.loads(msg[0].decode('ascii'))

    if len(headers) == 0:
        raise StopIteration

    for header, payload in zip(headers, msg[1:]):
        data[header['key']] = np.frombuffer(buffer(payload),
                                            dtype=header['dtype'])
        data[header['key']].shape = header['shape']
        if six.PY2:
            # Legacy python won't let us preserve alignment, skip this step
            continue
        data[header['key']].flags['ALIGNED'] = header['aligned']

    return data

Response: Receive data over a socket.


Instruction:
Write a Python 3 script for
yielding the data from the ZMQStreamer.

Response:
def iterate(self, max_iter=None):
        """
        Note: A ZMQStreamer does not activate its stream,
        but allows the zmq_worker to do that.

        Yields
        ------
        data : dict
            Data drawn from `streamer(max_iter)`.
        """
        context = zmq.Context()

        if six.PY2:
            warnings.warn('zmq_stream cannot preserve numpy array alignment '
                          'in Python 2', RuntimeWarning)

        try:
            socket = context.socket(zmq.PAIR)

            port = socket.bind_to_random_port('tcp://*',
                                              min_port=self.min_port,
                                              max_port=self.max_port,
                                              max_tries=self.max_tries)
            terminate = mp.Event()

            worker = mp.Process(target=SafeFunction(zmq_worker),
                                args=[port, self.streamer, terminate],
                                kwargs=dict(copy=self.copy,
                                            max_iter=max_iter))

            worker.daemon = True
            worker.start()

            # Yield from the queue as long as it's open
            while True:
                yield zmq_recv_data(socket)

        except StopIteration:
            pass

        except:
            # pylint: disable-msg=W0702
            six.reraise(*sys.exc_info())

        finally:
            terminate.set()
            worker.join(self.timeout)
            if worker.is_alive():
                worker.terminate()
            context.destroy()


Instruction:
Can you implement a function in Python 3 that
masks all lowercase nucleotides with N s

Response:
def hardmask(self):
        """ Mask all lowercase nucleotides with N's """
        p = re.compile("a|c|g|t|n")
        for seq_id in self.fasta_dict.keys():
            self.fasta_dict[seq_id] = p.sub("N", self.fasta_dict[seq_id])
        return self


Instruction:
Write a Python 3 function for
returning n random sequences from this Fasta object

Response:
def get_random(self, n, l=None):
        """ Return n random sequences from this Fasta object """
        random_f = Fasta()
        if l:
            ids = self.ids[:]
            random.shuffle(ids)
            i = 0
            while (i < n) and (len(ids) > 0):
                seq_id = ids.pop()
                if (len(self[seq_id]) >= l):
                    start = random.randint(0, len(self[seq_id]) - l)
                    random_f["random%s" % (i + 1)] = self[seq_id][start:start+l]
                    i += 1
            if len(random_f) != n:
                sys.stderr.write("Not enough sequences of required length")
                return
            else:
                return random_f

        else:
            choice = random.sample(self.ids, n)
            for i in range(n):
                random_f[choice[i]] = self[choice[i]]
        return random_f


Instruction:
Given the following Python 3 function, write the documentation
def writefasta(self, fname):
        f = open(fname, "w")
        fa_str = "\n".join([">%s\n%s" % (id, self._format_seq(seq)) for id, seq in self.items()])
        f.write(fa_str)
        f.close()

Response: Write the sequence list to a FASTA formatted file


Instruction:
Implement a function in Python 3 to
cluster motifs into a set of sequences.

Response:
def cluster_motifs(motifs, match="total", metric="wic", combine="mean", pval=True, threshold=0.95, trim_edges=False, edge_ic_cutoff=0.2, include_bg=True, progress=True, ncpus=None):
    """
    Clusters a set of sequence motifs. Required arg 'motifs' is a file containing
    positional frequency matrices or an array with motifs.

    Optional args:

    'match', 'metric' and 'combine' specify the method used to compare and score
    the motifs. By default the WIC score is used (metric='wic'), using the the
    score over the whole alignment (match='total'), with the total motif score
    calculated as the mean score of all positions (combine='mean').
    'match' can be either 'total' for the total alignment or 'subtotal' for the
    maximum scoring subsequence of the alignment.
    'metric' can be any metric defined in MotifComparer, currently: 'pcc', 'ed',
    'distance', 'wic' or 'chisq'
    'combine' determines how the total score is calculated from the score of
    individual positions and can be either 'sum' or 'mean'

    'pval' can be True or False and determines if the score should be converted to
    an empirical p-value

    'threshold' determines the score (or p-value) cutoff

    If 'trim_edges' is set to True, all motif edges with an IC below
    'edge_ic_cutoff' will be removed before clustering

    When computing the average of two motifs 'include_bg' determines if, at a
    position only present in one motif, the information in that motif should
    be kept, or if it should be averaged with background frequencies. Should
    probably be left set to True.

    """

    # First read pfm or pfm formatted motiffile
    if type([]) != type(motifs):
        motifs = read_motifs(motifs, fmt="pwm")

    mc = MotifComparer()

    # Trim edges with low information content
    if trim_edges:
        for motif in motifs:
            motif.trim(edge_ic_cutoff)

    # Make a MotifTree node for every motif
    nodes = [MotifTree(m) for m in motifs]

    # Determine all pairwise scores and maxscore per motif
    scores = {}
    motif_nodes = dict([(n.motif.id,n) for n in nodes])
    motifs = [n.motif for n in nodes]

    if progress:
        sys.stderr.write("Calculating initial scores\n")
    result = mc.get_all_scores(motifs, motifs, match, metric, combine, pval, parallel=True, ncpus=ncpus)

    for m1, other_motifs in result.items():
        for m2, score in other_motifs.items():
            if m1 == m2:
                if pval:
                    motif_nodes[m1].maxscore = 1 - score[0]
                else:
                    motif_nodes[m1].maxscore = score[0]
            else:
                if pval:
                    score = [1 - score[0]] + score[1:]
                scores[(motif_nodes[m1],motif_nodes[m2])] = score

    cluster_nodes = [node for node in nodes]
    ave_count = 1

    total = len(cluster_nodes)

    while len(cluster_nodes) > 1:
        l = sorted(scores.keys(), key=lambda x: scores[x][0])
        i = -1
        (n1, n2) = l[i]
        while n1 not in cluster_nodes or n2 not in cluster_nodes:
            i -= 1
            (n1,n2) = l[i]

        if len(n1.motif) > 0 and len(n2.motif) > 0:
            (score, pos, orientation) = scores[(n1,n2)]
            ave_motif = n1.motif.average_motifs(n2.motif, pos, orientation, include_bg=include_bg)

            ave_motif.trim(edge_ic_cutoff)

            # Check if the motif is not empty
            if len(ave_motif) == 0:
                ave_motif = Motif([[0.25,0.25,0.25,0.25]])

            ave_motif.id = "Average_%s" % ave_count
            ave_count += 1

            new_node = MotifTree(ave_motif)
            if pval:
                new_node.maxscore = 1 - mc.compare_motifs(new_node.motif, new_node.motif, match, metric, combine, pval)[0]
            else:
                new_node.maxscore = mc.compare_motifs(new_node.motif, new_node.motif, match, metric, combine, pval)[0]

            new_node.mergescore = score
            #print "%s + %s = %s with score %s" % (n1.motif.id, n2.motif.id, ave_motif.id, score)
            n1.parent = new_node
            n2.parent = new_node
            new_node.left = n1
            new_node.right = n2

            cmp_nodes = dict([(node.motif, node) for node in nodes if not node.parent])

            if progress:
                progress = (1 - len(cmp_nodes) / float(total)) * 100
                sys.stderr.write('\rClustering [{0}{1}] {2}%'.format(
                    '#' * (int(progress) // 10),
                    " " * (10 - int(progress) // 10),
                    int(progress)))

            result = mc.get_all_scores(
                    [new_node.motif],
                    list(cmp_nodes.keys()),
                    match,
                    metric,
                    combine,
                    pval,
                    parallel=True)

            for motif, n in cmp_nodes.items():
                x = result[new_node.motif.id][motif.id]
                if pval:
                    x = [1 - x[0]] + x[1:]
                scores[(new_node, n)] = x

            nodes.append(new_node)

        cluster_nodes = [node for node in nodes if not node.parent]

    if progress:
        sys.stderr.write("\n")
    root = nodes[-1]
    for node in [node for node in nodes if not node.left]:
        node.parent.checkMerge(root, threshold)

    return root


Instruction:
Make a summary of the following Python 3 code
def batch_length(batch):
    '''Determine the number of samples in a batch.

    Parameters
    ----------
    batch : dict
        A batch dictionary.  Each value must implement `len`.
        All values must have the same `len`.

    Returns
    -------
    n : int >= 0 or None
        The number of samples in this batch.
        If the batch has no fields, n is None.

    Raises
    ------
    PescadorError
        If some two values have unequal length
    '''
    n = None

    for value in six.itervalues(batch):
        if n is None:
            n = len(value)

        elif len(value) != n:
            raise PescadorError('Unequal field lengths')

    return n

Response: Determine the number of samples in a batch.


Instruction:
Write a Python 3 script to
activate a number of streams

Response:
def _activate(self):
        """Activates a number of streams"""
        self.distribution_ = 1. / self.n_streams * np.ones(self.n_streams)
        self.valid_streams_ = np.ones(self.n_streams, dtype=bool)

        self.streams_ = [None] * self.k

        self.stream_weights_ = np.zeros(self.k)
        self.stream_counts_ = np.zeros(self.k, dtype=int)
        # Array of pointers into `self.streamers`
        self.stream_idxs_ = np.zeros(self.k, dtype=int)

        for idx in range(self.k):

            if not (self.distribution_ > 0).any():
                break

            self.stream_idxs_[idx] = self.rng.choice(
                self.n_streams, p=self.distribution_)
            self.streams_[idx], self.stream_weights_[idx] = (
                self._new_stream(self.stream_idxs_[idx]))

        self.weight_norm_ = np.sum(self.stream_weights_)


Instruction:
Can you tell what is the following Python 3 function doing
def _new_stream(self, idx):
        '''Randomly select and create a stream.

        Parameters
        ----------
        idx : int, [0:n_streams - 1]
            The stream index to replace
        '''
        # instantiate
        if self.rate is not None:
            n_stream = 1 + self.rng.poisson(lam=self.rate)
        else:
            n_stream = None

        # If we're sampling without replacement, zero this one out
        # This effectively disables this stream as soon as it is chosen,
        # preventing it from being chosen again (unless it is revived)
        if not self.with_replacement:
            self.distribution_[idx] = 0.0

            # Correct the distribution
            if (self.distribution_ > 0).any():
                self.distribution_[:] /= np.sum(self.distribution_)

        return (self.streamers[idx].iterate(max_iter=n_stream),
                self.weights[idx])

Response: Randomly select and create a new stream.


Instruction:
Implement a function in Python 3 to
return the next available sample stream.

Response:
def _next_sample_index(self):
        """StochasticMux chooses its next sample stream randomly"""
        return self.rng.choice(self.n_active,
                               p=(self.stream_weights_ /
                                  self.weight_norm_))


Instruction:
Given the following Python 3 function, write the documentation
def _activate_stream(self, idx):
        '''Randomly select and create a stream.

        StochasticMux adds mode handling to _activate_stream, making it so that
        if we're not sampling "with_replacement", the distribution for this
        chosen streamer is set to 0, causing the streamer not to be available
        until it is exhausted.

        Parameters
        ----------
        idx : int, [0:n_streams - 1]
            The stream index to replace
        '''
        # Get the number of samples for this streamer.
        n_samples_to_stream = None
        if self.rate is not None:
            n_samples_to_stream = 1 + self.rng.poisson(lam=self.rate)

        # instantiate a new streamer
        streamer = self.streamers[idx].iterate(max_iter=n_samples_to_stream)
        weight = self.weights[idx]

        # If we're sampling without replacement, zero this one out
        # This effectively disables this stream as soon as it is chosen,
        # preventing it from being chosen again (unless it is revived)
        # if not self.with_replacement:
        if self.mode != "with_replacement":
            self.distribution_[idx] = 0.0

            # Correct the distribution
            if (self.distribution_ > 0).any():
                self.distribution_[:] /= np.sum(self.distribution_)

        return streamer, weight

Response: Randomly select and create a new streamer and weight.


Instruction:
Can you write a function in Python 3 where it
activates the current class.

Response:
def _activate(self):
        """ShuffledMux's activate is similar to StochasticMux,
        but there is no 'n_active', since all the streams are always available.
        """
        self.streams_ = [None] * self.n_streams

        # Weights of the active streams.
        # Once a stream is exhausted, it is set to 0.
        # Upon activation, this is just a copy of self.weights.
        self.stream_weights_ = np.array(self.weights, dtype=float)
        # How many samples have been drawn from each (active) stream.
        self.stream_counts_ = np.zeros(self.n_streams, dtype=int)

        # Initialize each active stream.
        for idx in range(self.n_streams):
            # Setup a new streamer at this index.
            self._new_stream(idx)

        self.weight_norm_ = np.sum(self.stream_weights_)


Instruction:
Can you generate the documentation for the following Python 3 function
def _next_sample_index(self):
        # Return the next streamer index where the streamer is not None,
        # wrapping around.
        idx = self.active_index_
        self.active_index_ += 1

        if self.active_index_ >= len(self.streams_):
            self.active_index_ = 0

        # Continue to increment if this streamer is exhausted (None)
        # This should never be infinite looping;
        # the `_streamers_available` check happens immediately
        # before this, so there should always be at least one not-None
        # streamer.
        while self.streams_[idx] is None:
            idx = self.active_index_
            self.active_index_ += 1

            if self.active_index_ >= len(self.streams_):
                self.active_index_ = 0

        return idx

Response: Rotates through each active sampler by incrementing the index and returning the next streamer index.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _new_stream(self, idx):
        # Get the stream index from the candidate pool
        stream_index = self.stream_idxs_[idx]

        # Activate the Streamer, and get the weights
        self.streams_[idx] = self.streamers[stream_index].iterate()

        # Reset the sample count to zero
        self.stream_counts_[idx] = 0

Response: Activates a new stream given the index into the stream pool.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _replace_stream(self, idx=None):
        self.streams_[idx] = None

        # Check if we've now exhausted all the streams.
        if not self._streamers_available():
            if self.mode == 'exhaustive':
                pass

            elif self.mode == "cycle":
                self._setup_streams(permute=False)

            elif self.mode == "permuted_cycle":
                self._setup_streams(permute=True)

Response: Called by BaseMux s iterate method when a stream is exhausted.


Instruction:
How would you explain what the following Python 3 function does
def _new_stream(self):
        '''Grab the next stream from the input streamers, and start it.

        Raises
        ------
        StopIteration
            When the input list or generator of streamers is complete,
            will raise a StopIteration. If `mode == cycle`, it
            will instead restart iterating from the beginning of the sequence.
        '''
        try:
            # Advance the stream_generator_ to get the next available stream.
            # If successful, this will make self.chain_streamer_.active True
            next_stream = six.advance_iterator(self.stream_generator_)

        except StopIteration:
            # If running with cycle, restart the chain_streamer_ after
            # exhaustion.
            if self.mode == "cycle":
                self.stream_generator_ = self.chain_streamer_.iterate()

                # Try again to get the next stream;
                # if it fails this time, just let it raise the StopIteration;
                # this means the streams are probably dead or empty.
                next_stream = six.advance_iterator(self.stream_generator_)

            # If running in exhaustive mode
            else:
                # self.chain_streamer_ should no longer be active, so
                # the outer loop should fall out without running.
                next_stream = None

        if next_stream is not None:
            # Start that stream, and return it.
            streamer = next_stream.iterate()

            # Activate the Streamer
            self.streams_[0] = streamer

            # Reset the sample count to zero
            self.stream_counts_[0] = 0

Response: Grab the next stream from the input streamers and start it.


Instruction:
Make a summary of the following Python 3 code
def split_and_save_datasets(X, Y, paths):
    shuffled_idxs = np.random.permutation(np.arange(len(X)))

    for i in range(len(paths)):
        # Take every len(paths) item, starting at i.
        # len(paths) is 3, so this would be [0::3], [1::3], [2::3]
        X_i = X[shuffled_idxs[i::len(paths)]]
        Y_i = Y[shuffled_idxs[i::len(paths)]]
        np.savez(paths[i], X=X_i, Y=Y_i)

Response: Shuffle X and Y into n datasets and save them to disk at the locations provided in paths.


Instruction:
Implement a Python 3 function for
generating data from an npz file.

Response:
def npz_generator(npz_path):
    """Generate data from an npz file."""
    npz_data = np.load(npz_path)
    X = npz_data['X']
    # Y is a binary maxtrix with shape=(n, k), each y will have shape=(k,)
    y = npz_data['Y']

    n = X.shape[0]

    while True:
        i = np.random.randint(0, n)
        yield {'X': X[i], 'Y': y[i]}


Instruction:
Can you implement a function in Python 3 that
computes the hypergeometric version of the formula.

Response:
def phyper(k, good, bad, N):
    """ Current hypergeometric implementation in scipy is broken, so here's the correct version """
    pvalues = [phyper_single(x, good, bad, N) for x in range(k + 1, N + 1)]
    return np.sum(pvalues)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def write_equalwidth_bedfile(bedfile, width, outfile):

    BUFSIZE = 10000
    f = open(bedfile)
    out = open(outfile, "w")
    lines = f.readlines(BUFSIZE)
    line_count = 0
    while lines:
        for line in lines:
            line_count += 1
            if not line.startswith("#") and not line.startswith("track") and not line.startswith("browser"):
                vals = line.strip().split("\t")
                try:
                    start, end = int(vals[1]), int(vals[2])
                except ValueError:
                    print("Error on line %s while reading %s. Is the file in BED or WIG format?" % (line_count, bedfile))
                    sys.exit(1)

                start = (start + end) // 2 - (width // 2)
                # This shifts the center, but ensures the width is identical... maybe not ideal
                if start < 0:
                    start = 0
                end = start + width
                # Keep all the other information in the bedfile if it's there
                if len(vals) > 3:
                    out.write("%s\t%s\t%s\t%s\n" % (vals[0], start, end, "\t".join(vals[3:])))
                else:
                    out.write("%s\t%s\t%s\n" % (vals[0], start, end))
        lines = f.readlines(BUFSIZE)

    out.close()
    f.close()

Response:
Read input from bedfile set the width of all entries to width and
    write the result to outfile.


Instruction:
Given the following Python 3 function, write the documentation
def calc_motif_enrichment(sample, background, mtc=None, len_sample=None, len_back=None):

    INF = "Inf"

    if mtc not in [None, "Bonferroni", "Benjamini-Hochberg", "None"]:
        raise RuntimeError("Unknown correction: %s" % mtc)

    sig = {}
    p_value  = {}
    n_sample = {}
    n_back = {}

    if not(len_sample):
        len_sample = sample.seqn()
    if not(len_back):
        len_back = background.seqn()

    for motif in sample.motifs.keys():
        p = "NA"
        s = "NA"
        q = len(sample.motifs[motif])
        m = 0
        if(background.motifs.get(motif)):
            m = len(background.motifs[motif])
            n = len_back - m
            k = len_sample
            p = phyper(q - 1, m, n, k)
            if p != 0:
                s = -(log(p)/log(10))
            else:
                s = INF
        else:
            s = INF
            p = 0.0

        sig[motif] = s
        p_value[motif] = p
        n_sample[motif] = q
        n_back[motif] = m

    if mtc == "Bonferroni":
        for motif in p_value.keys():
            if  p_value[motif] != "NA":
                p_value[motif] = p_value[motif] * len(p_value.keys())
                if p_value[motif] > 1:
                    p_value[motif] = 1
    elif mtc == "Benjamini-Hochberg":
        motifs = sorted(p_value.keys(), key=lambda x: -p_value[x])
        l = len(p_value)
        c = l
        for m in motifs:
            if  p_value[m] != "NA":
                p_value[m] = p_value[m] * l / c
            c -= 1

    return (sig, p_value, n_sample, n_back)

Response: Calculate enrichment based on hypergeometric distribution


Instruction:
Write a Python 3 function for
parsing a cutoff file and return a dictionary with motif id as key and cutoff as value

Response:
def parse_cutoff(motifs, cutoff, default=0.9):
    """ Provide either a file with one cutoff per motif or a single cutoff
        returns a hash with motif id as key and cutoff as value
    """

    cutoffs = {}
    if os.path.isfile(str(cutoff)):
        for i,line in enumerate(open(cutoff)):
            if line != "Motif\tScore\tCutoff\n":
                try:
                    motif,_,c = line.strip().split("\t")
                    c = float(c)
                    cutoffs[motif] = c
                except Exception as e:
                    sys.stderr.write("Error parsing cutoff file, line {0}: {1}\n".format(e, i + 1))
                    sys.exit(1)
    else:
        for motif in motifs:
            cutoffs[motif.id] = float(cutoff)

    for motif in motifs:
        if not motif.id in cutoffs:
            sys.stderr.write("No cutoff found for {0}, using default {1}\n".format(motif.id, default))
            cutoffs[motif.id] = default
    return cutoffs


Instruction:
Can you tell what is the following Python 3 function doing
def determine_file_type(fname):
    if not (isinstance(fname, str) or isinstance(fname, unicode)):
        raise ValueError("{} is not a file name!", fname)

    if not os.path.isfile(fname):
        raise ValueError("{} is not a file!", fname)

    ext = os.path.splitext(fname)[1].lower()
    if ext in ["bed"]:
        return "bed"
    elif ext in ["fa", "fasta"]:
        return "fasta"
    elif ext in ["narrowpeak"]:
        return "narrowpeak"

    try:
        Fasta(fname)
        return "fasta"
    except:
        pass
    # Read first line that is not a comment or an UCSC-specific line
    p = re.compile(r'^(#|track|browser)')
    with open(fname) as f:
        for line in f.readlines():
            line = line.strip()
            if not p.search(line):
                break
    region_p = re.compile(r'^(.+):(\d+)-(\d+)$')
    if region_p.search(line):
        return "region"
    else:
        vals = line.split("\t")
        if len(vals) >= 3:
            try:
                _, _ = int(vals[1]), int(vals[2])
            except ValueError:
                return "unknown"

            if len(vals) == 10:
                try:
                    _, _ = int(vals[4]), int(vals[9])
                    return "narrowpeak"
                except ValueError:
                    # As far as I know there is no 10-column BED format
                    return "unknown"
                    pass
            return "bed"

    # Catch-all
    return "unknown"

Response: Determine the file type of a UCSC file.


Instruction:
Implement a Python 3 function for
calculating md5 checksum of file.

Response:
def file_checksum(fname):
    """Return md5 checksum of file.

    Note: only works for files < 4GB.

    Parameters
    ----------
    filename : str
        File used to calculate checksum.

    Returns
    -------
        checkum : str
    """
    size = os.path.getsize(fname)
    with open(fname, "r+") as f:
        checksum = hashlib.md5(mmap.mmap(f.fileno(), size)).hexdigest()
    return checksum


Instruction:
Can you generate a brief explanation for the following Python 3 code
def download_annotation(genomebuild, gene_file):
    pred_bin = "genePredToBed"
    pred = find_executable(pred_bin)
    if not pred:
        sys.stderr.write("{} not found in path!\n".format(pred_bin))
        sys.exit(1)

    tmp = NamedTemporaryFile(delete=False, suffix=".gz")

    anno = []
    f = urlopen(UCSC_GENE_URL.format(genomebuild))
    p = re.compile(r'\w+.Gene.txt.gz')
    for line in f.readlines():
        m = p.search(line.decode())
        if m:
            anno.append(m.group(0))

    sys.stderr.write("Retrieving gene annotation for {}\n".format(genomebuild))
    url = ""
    for a in ANNOS:
        if a in anno:
            url = UCSC_GENE_URL.format(genomebuild) + a
            break
    if url:
        sys.stderr.write("Using {}\n".format(url))
        urlretrieve(
                url,
                tmp.name
                )

        with gzip.open(tmp.name) as f:
            cols = f.readline().decode(errors='ignore').split("\t")

        start_col = 1
        for i,col in enumerate(cols):
            if col == "+" or col == "-":
                start_col = i - 1
                break
        end_col = start_col + 10

        cmd = "zcat {} | cut -f{}-{} | {} /dev/stdin {}"
        print(cmd.format(tmp.name, start_col, end_col, pred, gene_file))
        sp.call(cmd.format(
            tmp.name, start_col, end_col, pred, gene_file),
            shell=True)

    else:
        sys.stderr.write("No annotation found!")

Response: Download gene annotation from UCSC based on genomebuild.


Instruction:
Here you have a function in Python 3, explain what it does
def _check_dir(self, dirname):
        if not os.path.exists(dirname):
            print("Directory %s does not exist!" % dirname)
            sys.exit(1)

Response: Check if directory exists and die if not


Instruction:
How would you implement a function in Python 3 that
indexes a single one - sequence fasta - file

Response:
def _make_index(self, fasta, index):
        """ Index a single, one-sequence fasta-file"""
        out = open(index, "wb")
        f = open(fasta)
        # Skip first line of fasta-file
        line = f.readline()
        offset = f.tell()
        line = f.readline()
        while line:
            out.write(pack(self.pack_char, offset))
            offset = f.tell()
            line = f.readline()
        f.close()
        out.close()


Instruction:
Given the following Python 3 function, write the documentation
def create_index(self,fasta_dir=None, index_dir=None):

        # Use default directories if they are not supplied
        if not fasta_dir:
            fasta_dir = self.fasta_dir

        if not index_dir:
            index_dir = self.index_dir

        # Can't continue if we still don't have an index_dir or fasta_dir
        if not fasta_dir:
            print("fasta_dir not defined!")
            sys.exit(1)

        if not index_dir:
            print("index_dir not defined!")
            sys.exit(1)

        index_dir = os.path.abspath(index_dir)
        fasta_dir = os.path.abspath(fasta_dir)

        self.index_dir = index_dir

        # Prepare index directory
        if not os.path.exists(index_dir):
            try:
                os.mkdir(index_dir)
            except OSError as e:
                if e.args[0] == 13:
                    sys.stderr.write("No permission to create index directory. Superuser access needed?\n")
                    sys.exit()
                else:
                    sys.stderr.write(e)

        # Directories need to exist
        self._check_dir(fasta_dir)
        self._check_dir(index_dir)

        # Get all fasta-files

        fastafiles = find_by_ext(fasta_dir, FASTA_EXT)
        if not(fastafiles):
            msg = "No fastafiles found in {} with extension in {}".format(
                                        fasta_dir, ",".join(FASTA_EXT))
            raise IOError(msg)

        # param_file will hold all the information about the location of the fasta-files, indeces and
        # length of the sequences
        param_file = os.path.join(index_dir, self.param_file)
        size_file = os.path.join(index_dir, self.size_file)

        try:
            out = open(param_file, "w")
        except IOError as e:
            if e.args[0] == 13:
                sys.stderr.write("No permission to create files in index directory. Superuser access needed?\n")
                sys.exit()
            else:
                sys.stderr.write(e)
        s_out = open(size_file, "w")

        for fasta_file in fastafiles:
            #sys.stderr.write("Indexing %s\n" % fasta_file)
            f = open(fasta_file)
            line = f.readline()
            if not line.startswith(">"):
                sys.stderr.write("%s is not a valid FASTA file, expected > at first line\n" % fasta_file)
                sys.exit()

            seqname = line.strip().replace(">", "")
            line = f.readline()
            line_size = len(line.strip())

            total_size = 0
            while line:
                line = line.strip()
                if line.startswith(">"):
                    sys.stderr.write("Sorry, can only index genomes with "
                    "one sequence per FASTA file\n%s contains multiple "
                    "sequences\n" % fasta_file)
                    sys.exit()

                total_size += len(line)
                line = f.readline()

            index_file = os.path.join(index_dir, "%s.index" % seqname)

            out.write("{}\t{}\t{}\t{}\t{}\n".format(
                seqname, fasta_file, index_file, line_size, total_size))
            s_out.write("{}\t{}\n".format(seqname, total_size))

            self._make_index(fasta_file, index_file)
            f.close()
        out.close()
        s_out.close()

        # Read the index we just made so we can immediately use it
        self._read_index_file()

Response: Create the index for the current object and store the results in index_dir


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _read_index_file(self):
        param_file = os.path.join(self.index_dir, self.param_file)
        with open(param_file) as f:
            for line in f.readlines():
                (name, fasta_file, index_file, line_size, total_size) = line.strip().split("\t")
                self.size[name] = int(total_size)
                self.fasta_file[name] = fasta_file
                self.index_file[name] = index_file
                self.line_size[name] = int(line_size)

Response: read the index file


Instruction:
Explain what the following Python 3 code does
def _read_seq_from_fasta(self, fasta, offset, nr_lines):
        fasta.seek(offset)
        lines = [fasta.readline().strip() for _ in range(nr_lines)]
        return "".join(lines)

Response: retrieve a number of lines from a fasta file - object starting at offset


Instruction:
Can you generate the documentation for the following Python 3 function
def get_sequences(self, chr, coords):
        # Check if we have an index_dir
        if not self.index_dir:
            print("Index dir is not defined!")
            sys.exit()

        # retrieve all information for this specific sequence
        fasta_file = self.fasta_file[chr]
        index_file = self.index_file[chr]
        line_size = self.line_size[chr]
        total_size = self.size[chr]
        index = open(index_file, "rb")
        fasta = open(fasta_file)

        seqs = []
        for coordset in coords:
            seq = ""
            for (start,end) in coordset:
                if start > total_size:
                    raise ValueError("%s: %s, invalid start, greater than sequence length!" % (chr,start))

                if start < 0:
                    raise ValueError("Invalid start, < 0!")

                if end > total_size:
                    raise ValueError("Invalid end, greater than sequence length!")

                seq += self._read(index, fasta, start, end, line_size)
            seqs.append(seq)
        index.close()
        fasta.close()

        return seqs

Response: Retrieve multiple sequences from same chr


Instruction:
Can you generate the documentation for the following Python 3 function
def get_sequence(self, chrom, start, end, strand=None):
        # Check if we have an index_dir
        if not self.index_dir:
            print("Index dir is not defined!")
            sys.exit()

        # retrieve all information for this specific sequence
        fasta_file = self.fasta_file[chrom]
        index_file = self.index_file[chrom]
        line_size = self.line_size[chrom]
        total_size = self.size[chrom]

        #print fasta_file, index_file, line_size, total_size
        if start > total_size:
            raise ValueError(
                    "Invalid start {0}, greater than sequence length {1} of {2}!".format(start, total_size, chrom))

        if start < 0:
            raise ValueError("Invalid start, < 0!")

        if end > total_size:
            raise ValueError(
                    "Invalid end {0}, greater than sequence length {1} of {2}!".format(end, total_size, chrom))

        index = open(index_file, "rb")
        fasta = open(fasta_file)
        seq = self._read(index, fasta, start, end, line_size)
        index.close()
        fasta.close()

        if strand and strand == "-":
            seq = rc(seq)
        return seq

Response: Retrieve a sequence from the index file


Instruction:
Write a Python 3 function that can
return the sizes of all sequences in the index or the size of chrom if chrom is not specified.

Response:
def get_size(self, chrom=None):
        """ Return the sizes of all sequences in the index, or the size of chrom if specified
        as an optional argument """
        if len(self.size) == 0:
            raise LookupError("no chromosomes in index, is the index correct?")

        if chrom:
            if chrom in self.size:
                return self.size[chrom]
            else:
                raise KeyError("chromosome {} not in index".format(chrom))
        total = 0
        for size in self.size.values():
            total += size

        return total


Instruction:
How would you implement a function in Python 3 that
returns an instance of a specific tool.

Response:
def get_tool(name):
    """
    Returns an instance of a specific tool.

    Parameters
    ----------
    name : str
        Name of the tool (case-insensitive).

    Returns
    -------
    tool : MotifProgram instance
    """
    tool = name.lower()
    if tool not in __tools__:
        raise ValueError("Tool {0} not found!\n".format(name))

    t = __tools__[tool]()

    if not t.is_installed():
        sys.stderr.write("Tool {0} not installed!\n".format(tool))

    if not t.is_configured():
        sys.stderr.write("Tool {0} not configured!\n".format(tool))

    return t


Instruction:
Implement a Python 3 function for
locating a tool in the current directory.

Response:
def locate_tool(name, verbose=True):
    """
    Returns the binary of a tool.

    Parameters
    ----------
    name : str
        Name of the tool (case-insensitive).

    Returns
    -------
    tool_bin : str
        Binary of tool.
    """
    m = get_tool(name)
    tool_bin = which(m.cmd)
    if tool_bin:
        if verbose:
            print("Found {} in {}".format(m.name, tool_bin))
        return tool_bin
    else:
        print("Couldn't find {}".format(m.name))


Instruction:
Write a Python 3 function that can
return the command used to run the tool.

Response:
def bin(self):
        """
        Get the command used to run the tool.

        Returns
        -------
        command : str
            The tool system command.
        """
        if self.local_bin:
            return self.local_bin
        else:
            return self.config.bin(self.name)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def is_installed(self):
        return self.is_configured() and os.access(self.bin(), os.X_OK)

Response: Check if the tool is installed.


Instruction:
Can you tell what is the following Python 3 function doing
def run(self, fastafile, params=None, tmp=None):
        if not self.is_configured():
            raise ValueError("%s is not configured" % self.name)

        if not self.is_installed():
            raise ValueError("%s is not installed or not correctly configured" % self.name)

        self.tmpdir = mkdtemp(prefix="{0}.".format(self.name), dir=tmp)
        fastafile = os.path.abspath(fastafile)

        try:
            return self._run_program(self.bin(), fastafile, params)
        except KeyboardInterrupt:
            return ([], "Killed", "Killed")

Response: Run the tool and predict motifs from a FASTA file.


Instruction:
Write a Python 3 function for
parsing parameters. Combine default and user-defined parameters.

Response:
def _parse_params(self, params=None):
        """
        Parse parameters.

        Combine default and user-defined parameters.
        """
        prm = self.default_params.copy()
        if params is not None:
            prm.update(params)

        if prm["background"]:
            # Absolute path, just to be sure
            prm["background"] =  os.path.abspath(prm["background"])
            prm["background"] = " --negSet {0} ".format(
                    prm["background"])

        prm["strand"] = ""
        if not prm["single"]:
            prm["strand"] = " --revcomp "

        return prm


Instruction:
Make a summary of the following Python 3 code
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)

        outfile = os.path.join(
                self.tmpdir,
                os.path.basename(fastafile.replace(".fa", ".pwm")))

        stdout = ""
        stderr = ""

        cmd = "%s %s %s --localization --batch %s %s" % (
            bin,
            self.tmpdir,
            fastafile,
            params["background"],
            params["strand"],
            )

        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
        out,err = p.communicate()
        stdout += out.decode()
        stderr += err.decode()

        motifs = []

        if os.path.exists(outfile):
            motifs = read_motifs(outfile, fmt="xxmotif")
            for m in motifs:
                m.id = "{0}_{1}".format(self.name, m.id)
        else:
            stdout += "\nMotif file {0} not found!\n".format(outfile)
            stderr += "\nMotif file {0} not found!\n".format(outfile)

        return motifs, stdout, stderr

Response: Run XXmotif and predict motifs from a FASTA file.


Instruction:
Write a Python 3 script for
parsing parameters. Combine default and user-defined parameters.

Response:
def _parse_params(self, params=None):
        """
        Parse parameters.

        Combine default and user-defined parameters.
        """
        prm = self.default_params.copy()
        if params is not None:
            prm.update(params)

        # Background file is essential!
        if not prm["background"]:
            print("Background file needed!")
            sys.exit()

        prm["background"] =  os.path.abspath(prm["background"])

        prm["strand"] = ""
        if prm["single"]:
            prm["strand"] = " -strand + "

        return prm


Instruction:
Make a summary of the following Python 3 code
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)

        outfile = NamedTemporaryFile(
                mode="w",
                dir=self.tmpdir,
                prefix= "homer_w{}.".format(params["width"])
                ).name

        cmd = "%s denovo -i %s -b %s -len %s -S %s %s -o %s -p 8" % (
            bin,
            fastafile,
            params["background"],
            params["width"],
            params["number"],
            params["strand"],
            outfile)

        stderr = ""
        stdout = "Running command:\n{}\n".format(cmd)

        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE, cwd=self.tmpdir)
        out,err = p.communicate()
        stdout += out.decode()
        stderr += err.decode()

        motifs = []

        if os.path.exists(outfile):
            motifs = read_motifs(outfile, fmt="pwm")
            for i, m in enumerate(motifs):
                m.id = "{}_{}_{}".format(self.name, params["width"], i + 1)

        return motifs, stdout, stderr

Response: Run Homer and predict motifs from a FASTA file.


Instruction:
Can you create a Python 3 function that
converts BioProspector output to motifs

Response:
def parse(self, fo):
        """
        Convert BioProspector output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing BioProspector output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []

        p = re.compile(r'^\d+\s+(\d+\.\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)')
        pwm = []
        motif_id = ""
        for line in fo.readlines():
            if line.startswith("Motif #"):
                if pwm:
                    m = Motif(pwm)
                    m.id = "BioProspector_w%s_%s" % (len(m), motif_id)
                    motifs.append(m)
                motif_id =  line.split("#")[1].split(":")[0]
                pwm = []
            else:
                m = p.search(line)
                if m:
                    pwm.append([float(m.group(x))/100.0 for x in range(1,5)])

        if pwm:
            m = Motif(pwm)
            m.id = "BioProspector_w%s_%s" % (len(m), motif_id)
            motifs.append(m)
        return motifs


Instruction:
Given the following Python 3 function, write the documentation
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)

        default_params = {"width":10}
        if params is not None:
            default_params.update(params)

        fgfile, summitfile, outfile = self._prepare_files(fastafile)

        current_path = os.getcwd()
        os.chdir(self.tmpdir)

        cmd = "{} -i {} -w {} -dna 4 -iteration 50 -chain 20 -seqprop -0.1 -strand 2 -peaklocation {} -t_dof 3 -dep 2".format(
                bin,
                fgfile,
                params['width'],
                summitfile)

        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
        stdout,stderr = p.communicate()

        os.chdir(current_path)

        motifs = []
        if os.path.exists(outfile):
            with open(outfile) as f:
                motifs = self.parse(f)
                for i,m in enumerate(motifs):
                    m.id = "HMS_w{}_{}".format(params['width'], i + 1)

        return motifs, stdout, stderr

Response: Run HMS and predict motifs from a FASTA file.


Instruction:
Write a Python 3 script to
convert HMS output to motifs

Response:
def parse(self, fo):
        """
        Convert HMS output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing HMS output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []
        m = [[float(x) for x in fo.readline().strip().split(" ")] for i in range(4)]
        matrix = [[m[0][i], m[1][i],m[2][i],m[3][i]] for i in range(len(m[0]))]
        motifs = [Motif(matrix)]
        motifs[-1].id = self.name

        return motifs


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)

        fgfile = os.path.join(self.tmpdir, "AMD.in.fa")
        outfile = fgfile + ".Matrix"
        shutil.copy(fastafile, fgfile)

        current_path = os.getcwd()
        os.chdir(self.tmpdir)

        stdout = ""
        stderr = ""

        cmd = "%s -F %s -B %s" % (
                bin,
                fgfile,
                params["background"],
                )
        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
        out,err = p.communicate()
        stdout += out.decode()
        stderr += err.decode()

        os.chdir(current_path)
        motifs = []
        if os.path.exists(outfile):
            f = open(outfile)
            motifs = self.parse(f)
            f.close()

        return motifs, stdout, stderr

Response: Run AMD and predict motifs from a FASTA file.


Instruction:
Write a Python 3 function for
converting AMD output to motifs AttributeNames

Response:
def parse(self, fo):
        """
        Convert AMD output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing AMD output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []

        #160:  112  CACGTGC      7.25   chr14:32308489-32308689
        p = re.compile(r'\d+\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)\s+([\d.]+)')
        wm = []
        name = ""
        for line in fo.readlines():
            if line.startswith("Motif") and line.strip().endswith(":"):
                if name:
                    motifs.append(Motif(wm))
                    motifs[-1].id = name
                    name = ""
                    wm = []
                name = "%s_%s" % (self.name, line.split(":")[0])
            else:
                m = p.search(line)
                if m:
                    wm.append([float(m.group(x)) for x in range(1,5)])
        motifs.append(Motif(wm))
        motifs[-1].id = name

        return motifs


Instruction:
Write a Python 3 script for
converting Improbizer output to motifs

Response:
def parse(self, fo):
        """
        Convert Improbizer output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing Improbizer output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []
        p = re.compile(r'\d+\s+@\s+\d+\.\d+\s+sd\s+\d+\.\d+\s+(\w+)$')

        line = fo.readline()
        while line and line.find("Color") == -1:
            m = p.search(line)
            if m:
                pwm_data = {}
                for i in range(4):
                    vals = [x.strip() for x in fo.readline().strip().split(" ") if x]
                    pwm_data[vals[0].upper()] = vals[1:]
                pwm = []
                for i in range(len(pwm_data["A"])):
                    pwm.append([float(pwm_data[x][i]) for x in ["A","C","G","T"]])
                motifs.append(Motif(pwm))
                motifs[-1].id = "%s_%s" % (self.name, m.group(1))
            line = fo.readline()

        return motifs


Instruction:
Can you generate the documentation for the following Python 3 function
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)

        tmp = NamedTemporaryFile(mode="w", dir=self.tmpdir, delete=False)
        shutil.copy(fastafile, tmp.name)
        fastafile = tmp.name

        current_path = os.getcwd()
        os.chdir(self.dir())

        motifs = []
        stdout = ""
        stderr = ""
        for wildcard in [0,1,2]:
            cmd = "%s -sample %s -background %s -directory %s -strand %s -wildcard %s" % (
                    bin,
                    fastafile,
                    params["background"],
                    self.tmpdir,
                    params["strand"],
                    wildcard,
                    )

            p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
            out,err = p.communicate()
            stdout += out.decode()
            stderr += err.decode()

            os.chdir(current_path)
            pwmfiles = glob.glob("{}/tmp*/result/*pwm".format(self.tmpdir))
            if len(pwmfiles) > 0:
                out_file = pwmfiles[0]
                stdout += "\nOutfile: {}".format(out_file)

                my_motifs = []
                if os.path.exists(out_file):
                    my_motifs = read_motifs(out_file, fmt="pwm")
                    for m in motifs:
                        m.id = "{}_{}".format(self.name, m.id)
                    stdout += "\nTrawler: {} motifs".format(len(motifs))

                # remove temporary files
                if os.path.exists(tmp.name):
                    os.unlink(tmp.name)

                for motif in my_motifs:
                    motif.id = "{}_{}_{}".format(self.name, wildcard, motif.id)

                motifs += my_motifs
            else:
                stderr += "\nNo outfile found"

        return motifs, stdout, stderr

Response: Run the tool and predict motifs from a FASTA file.


Instruction:
How would you implement a function in Python 3 that
runs the weeder and predict motifs from a FASTA file.

Response:
def _run_program(self, bin,fastafile, params=None):
        """
        Run Weeder and predict motifs from a FASTA file.

        Parameters
        ----------
        bin : str
            Command used to run the tool.

        fastafile : str
            Name of the FASTA input file.

        params : dict, optional
            Optional parameters. For some of the tools required parameters
            are passed using this dictionary.

        Returns
        -------
        motifs : list of Motif instances
            The predicted motifs.

        stdout : str
            Standard out of the tool.

        stderr : str
            Standard error of the tool.
        """
        params = self._parse_params(params)

        organism = params["organism"]
        weeder_organisms = {
            "hg18":"HS",
            "hg19":"HS",
            "hg38":"HS",
            "mm9":"MM",
            "mm10":"MM",
            "dm3":"DM",
            "dm5":"DM",
            "dm6":"DM",
            "yeast":"SC",
            "sacCer2":"SC",
            "sacCer3":"SC",
            "TAIR10":"AT",
            "TAIR11":"AT",
            }
        weeder_organism = weeder_organisms.get(organism, "HS")

        tmp = NamedTemporaryFile(dir=self.tmpdir)
        name = tmp.name
        tmp.close()
        shutil.copy(fastafile, name)
        fastafile = name

        cmd = "{} -f {} -O".format(
                self.cmd,
                fastafile,
                weeder_organism,
                )

        if params["single"]:
            cmd += " -ss"

        #print cmd
        stdout, stderr = "", ""
        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE, cwd=self.tmpdir)
        out,err = p.communicate()
        stdout += out.decode()
        stderr += err.decode()

        motifs = []
        if os.path.exists(fastafile + ".matrix.w2"):
            f = open(fastafile + ".matrix.w2")
            motifs = self.parse(f)
            f.close()

        for m in motifs:
            m.id = "{}_{}".format(self.name, m.id.split("\t")[0])

        for ext in [".w2", ".matrix.w2" ]:
            if os.path.exists(fastafile + ext):
                os.unlink(fastafile + ext)

        return motifs, stdout, stderr


Instruction:
Create a Python 3 function to
parse parameters. Combine default and user-defined parameters.

Response:
def _parse_params(self, params=None):
        """
        Parse parameters.

        Combine default and user-defined parameters.
        """
        prm = self.default_params.copy()
        if params is not None:
            prm.update(params)

        if prm["background_model"]:
            # Absolute path, just to be sure
            prm["background_model"] = os.path.abspath(prm["background_model"])
        else:
            if prm.get("organism", None):
                prm["background_model"] = os.path.join(
                        self.config.get_bg_dir(),
                        "{}.{}.bg".format(
                            prm["organism"],
                            "MotifSampler"))
            else:
                raise Exception("No background specified for {}".format(self.name))

        prm["strand"] = 1
        if prm["single"]:
            prm["strand"] = 0

        tmp = NamedTemporaryFile(dir=self.tmpdir)
        prm["pwmfile"] = tmp.name

        tmp2  = NamedTemporaryFile(dir=self.tmpdir)
        prm["outfile"] = tmp2.name

        return prm


Instruction:
Create a Python 3 function to
run MotifSampler and predict motifs from a FASTA file.

Response:
def _run_program(self, bin, fastafile, params=None):
        """
        Run MotifSampler and predict motifs from a FASTA file.

        Parameters
        ----------
        bin : str
            Command used to run the tool.

        fastafile : str
            Name of the FASTA input file.

        params : dict, optional
            Optional parameters. For some of the tools required parameters
            are passed using this dictionary.

        Returns
        -------
        motifs : list of Motif instances
            The predicted motifs.

        stdout : str
            Standard out of the tool.

        stderr : str
            Standard error of the tool.
        """
        params = self._parse_params(params)
        # TODO: test organism
        #cmd = "%s -f %s -b %s -m %s -w %s -n %s -o %s -s %s > /dev/null 2>&1" % (
        cmd = "%s -f %s -b %s -m %s -w %s -n %s -o %s -s %s" % (
                bin,
                fastafile,
                params["background_model"],
                params["pwmfile"],
                params["width"],
                params["number"],
                params["outfile"],
                params["strand"],
                )
        #print cmd
        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
        stdout, stderr = p.communicate()

        #stdout,stderr = "",""
        #p = Popen(cmd, shell=True)
        #p.wait()

        motifs = []
        if os.path.exists(params["outfile"]):
            with open(params["outfile"]) as f:
                motifs = self.parse_out(f)

        for motif in motifs:
            motif.id = "%s_%s" % (self.name, motif.id)

        return motifs, stdout, stderr


Instruction:
Write a Python 3 script to
convert MotifSampler output to motifs Arc Arc.

Response:
def parse(self, fo):
        """
        Convert MotifSampler output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing MotifSampler output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []

        pwm = []
        info = {}
        for line in fo.readlines():
            if line.startswith("#"):
                vals =  line.strip()[1:].split(" = ")
                if len(vals) > 1:
                    info[vals[0]] = vals[1]
            elif len(line) > 1:
                pwm.append([float(x) for x in line.strip().split("\t")])
            else:
                motifs.append(Motif())
                motifs[-1].consensus = info["Consensus"]
                motifs[-1].width = info["W"]
                motifs[-1].id = info["ID"]
                motifs[-1].pwm = pwm[:]
                pwm = []

        return motifs


Instruction:
Can you generate a brief explanation for the following Python 3 code
def parse_out(self, fo):
        motifs = []
        nucs = {"A":0,"C":1,"G":2,"T":3}
        pseudo = 0.0 # Should be 1/sqrt(# of seqs)
        aligns = {}
        for line in fo.readlines():
            if line.startswith("#"):
                pass
            elif len(line) > 1:
                vals = line.strip().split("\t")
                m_id, site = [x.strip().split(" ")[1].replace('"',"") for x in vals[8].split(";") if x]
                #if vals[6] == "+":
                if site.upper().find("N") == -1:
                    aligns.setdefault(m_id, []).append(site)
                #else:
                #    print site, rc(site)
                #    aligns.setdefault(id, []).append(rc(site))

        for m_id, align in aligns.items():
            #print id, len(align)

            width = len(align[0])
            pfm =  [[0 for x in range(4)] for x in range(width)]
            for row in align:
                for i in range(len(row)):
                    pfm[i][nucs[row[i]]] += 1
            total = float(len(align))
            pwm = [[(x + pseudo/4)/total+(pseudo) for x in row] for row in pfm]
            m = Motif()
            m.align = align[:]
            m.pwm = pwm[:]
            m.pfm = pfm[:]
            m.id = m_id
            motifs.append(m)
        return motifs

Response:
Convert MotifSampler output to motifs
         Arc Arc.


Instruction:
Can you tell what is the following Python 3 function doing
def _run_program(self, bin, fastafile, params=None):
        default_params = {"width":10, "number":10}
        if params is not None:
            default_params.update(params)

        new_file = os.path.join(self.tmpdir, "mdmodule_in.fa")
        shutil.copy(fastafile, new_file)

        fastafile = new_file
        pwmfile = fastafile + ".out"

        width = default_params['width']
        number = default_params['number']

        current_path = os.getcwd()
        os.chdir(self.tmpdir)
        cmd = "%s -i %s -a 1 -o %s -w %s -t 100 -r %s" % (bin, fastafile, pwmfile, width, number)
        p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
        stdout,stderr = p.communicate()

        stdout = "cmd: {}\n".format(cmd) + stdout.decode()

        motifs = []
        if os.path.exists(pwmfile):
            with open(pwmfile) as f:
                motifs = self.parse(f)

        os.chdir(current_path)

        for motif in motifs:
            motif.id = "%s_%s" % (self.name, motif.id)

        return motifs, stdout, stderr

Response: Run MDmodule and predict motifs from a FASTA file.


Instruction:
Write a Python 3 script for
parsing MDmodule output to motifs osity.

Response:
def parse(self, fo):
        """
        Convert MDmodule output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing MDmodule output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []
        nucs = {"A":0,"C":1,"G":2,"T":3}
        p = re.compile(r'(\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)\s+(\d+\.\d+)')
        pf = re.compile(r'>.+\s+[bf]\d+\s+(\w+)')

        pwm = []
        pfm = []
        align = []
        m_id = ""
        for line in fo.readlines():
            if line.startswith("Motif"):
                if m_id:
                    motifs.append(Motif())
                    motifs[-1].id = m_id
                    motifs[-1].pwm = pwm
                    motifs[-1].pfm = pfm
                    motifs[-1].align = align
                    pwm = []
                    pfm = []
                    align = []
                m_id = line.split("\t")[0]
            else:
                m = p.search(line)
                if m:
                    pwm.append([float(m.group(x))/100 for x in [2,3,4,5]])
                m = pf.search(line)
                if m:
                    if not pfm:
                        pfm = [[0 for x in range(4)] for x in range(len(m.group(1)))]
                    for i in range(len(m.group(1))):
                        pfm[i][nucs[m.group(1)[i]]] += 1

                    align.append(m.group(1))

        if pwm:
            motifs.append(Motif())
            motifs[-1].id = m_id
            motifs[-1].pwm = pwm
            motifs[-1].pfm = pfm
            motifs[-1].align = align

        return motifs


Instruction:
Implement a Python 3 function for
parsing parameters. Combine default and user-defined parameters.

Response:
def _parse_params(self, params=None):
        """
        Parse parameters.

        Combine default and user-defined parameters.
        """
        prm = self.default_params.copy()
        if params is not None:
            prm.update(params)

        return prm


Instruction:
Make a summary of the following Python 3 code
def _run_program(self, bin, fastafile, params=None):
        params = self._parse_params(params)
        basename = "munk_in.fa"

        new_file = os.path.join(self.tmpdir, basename)
        out = open(new_file, "w")
        f = Fasta(fastafile)
        for seq in f.seqs:
            header = len(seq) // 2
            out.write(">%s\n" % header)
            out.write("%s\n" % seq)
        out.close()

        fastafile = new_file
        outfile = fastafile + ".out"

        current_path = os.getcwd()
        os.chdir(self.dir())

        motifs = []
        # Max recommended by ChIPMunk userguide
        ncpus = 4
        stdout = ""
        stderr = ""
        for zoops_factor in ["oops", 0.0, 0.5, 1.0]:
            cmd = "{} {} {} y {} m:{} 100 10 1 {} 1>{}".format(
                bin,
                params.get("width", 8),
                params.get("width", 20),
                zoops_factor,
                fastafile,
                ncpus,
                outfile
                )
            #print("command: ", cmd)
            p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
            std = p.communicate()
            stdout = stdout + std[0].decode()
            stderr = stderr + std[1].decode()

            if "RuntimeException" in stderr:
                return [], stdout, stderr

            if os.path.exists(outfile):
                with open(outfile) as f:
                    motifs += self.parse(f)

        os.chdir(current_path)

        return motifs, stdout, stderr

Response: Run ChIPMunk and predict motifs from a FASTA file.


Instruction:
Can you generate the documentation for the following Python 3 function
def parse(self, fo):
        #KDIC|6.124756232026243
        #A|517.9999999999999 42.99999999999999 345.99999999999994 25.999999999999996 602.9999999999999 155.99999999999997 2.9999999999999996 91.99999999999999
        #C|5.999999999999999 4.999999999999999 2.9999999999999996 956.9999999999999 91.99999999999999 17.999999999999996 22.999999999999996 275.99999999999994
        #G|340.99999999999994 943.9999999999999 630.9999999999999 6.999999999999999 16.999999999999996 48.99999999999999 960.9999999999999 14.999999999999998
        #T|134.99999999999997 7.999999999999999 19.999999999999996 9.999999999999998 287.99999999999994 776.9999999999999 12.999999999999998 616.9999999999999
        #N|999.9999999999998
        line = fo.readline()
        if not line:
            return []

        while not line.startswith("A|"):
            line = fo.readline()
        matrix = []
        for _ in range(4):
            matrix.append([float(x) for x in line.strip().split("|")[1].split(" ")])
            line = fo.readline()
        #print matrix
        matrix = [[matrix[x][y] for x in range(4)] for y in range(len(matrix[0]))]
        #print matrix
        m = Motif(matrix)
        m.id = "ChIPMunk_w%s" % len(m)
        return [m]

Response: Parses ChIPMunk output to motifs


Instruction:
Here you have a function in Python 3, explain what it does
def _run_program(self, bin, fastafile, params=None):
        default_params = {}
        if params is not None:
            default_params.update(params)

        width = params.get("width", 8)
        basename = "posmo_in.fa"

        new_file = os.path.join(self.tmpdir, basename)
        shutil.copy(fastafile, new_file)

        fastafile = new_file
        #pwmfile = fastafile + ".pwm"

        motifs = []
        current_path = os.getcwd()
        os.chdir(self.tmpdir)
        for n_ones in range(4, min(width, 11), 2):
            x = "1" * n_ones
            outfile = "%s.%s.out" % (fastafile, x)
            cmd = "%s 5000 %s %s 1.6 2.5 %s 200" % (bin, x, fastafile, width)
            p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
            stdout, stderr = p.communicate()
            stdout = stdout.decode()
            stderr = stderr.decode()

            context_file = fastafile.replace(basename, "context.%s.%s.txt" % (basename, x))
            cmd = "%s %s %s simi.txt 0.88 10 2 10" % (bin.replace("posmo","clusterwd"), context_file, outfile)
            p = Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
            out, err = p.communicate()
            stdout += out.decode()
            stderr += err.decode()

            if os.path.exists(outfile):
                with open(outfile) as f:
                    motifs += self.parse(f, width, n_ones)

        os.chdir(current_path)

        return motifs, stdout, stderr

Response: Run Posmo and predict motifs from a FASTA file.


Instruction:
Write a Python 3 script to
convert Posmo output to motifs AttributeNames

Response:
def parse(self, fo, width, seed=None):
        """
        Convert Posmo output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing Posmo output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []

        lines = [fo.readline() for x in range(6)]
        while lines[0]:
            matrix = [[float(x) for x in line.strip().split("\t")] for line in lines[2:]]
            matrix = [[matrix[x][y] for x in range(4)] for y in range(len(matrix[0]))]
            m = Motif(matrix)
            m.trim(0.1)
            m.id = lines[0].strip().split(" ")[-1]
            motifs.append(m)
            lines = [fo.readline() for x in range(6)]

        for i,motif in enumerate(motifs):
            if seed:
                motif.id = "%s_w%s.%s_%s" % (self.name, width, seed, i + 1)
            else:
                motif.id = "%s_w%s_%s" % (self.name, width, i + 1)
            motif.trim(0.25)

        return motifs


Instruction:
Write a Python 3 function that can
convert GADEM output to motifs Arcs.

Response:
def parse(self, fo):
        """
        Convert GADEM output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing GADEM output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []
        nucs = {"A":0,"C":1,"G":2,"T":3}

        lines = fo.readlines()
        for i in range(0, len(lines), 5):
            align = []
            pwm = []
            pfm = []
            m_id = ""
            line = lines[i].strip()
            m_id = line[1:]
            number = m_id.split("_")[0][1:]
            if os.path.exists("%s.seq" % number):
                with open("%s.seq" % number) as f:
                    for l in f:
                        if "x" not in l and "n" not in l:
                            l = l.strip().upper()
                            align.append(l)
                            if not pfm:
                                pfm = [[0 for x in range(4)] for x in range(len(l))]
                            for p in range(len(l)):
                                pfm[p][nucs[l[p]]] += 1

            m = [l.strip().split(" ")[1].split("\t") for l in lines[i + 1: i + 5]]

            pwm = [[float(m[x][y]) for x in range(4)] for y in range(len(m[0]))]

            motifs.append(Motif(pwm))
            motifs[-1].id = "{}_{}".format(self.name, m_id)
            #motifs[-1].pwm = pwm
            if align:
                motifs[-1].pfm = pfm
                motifs[-1].align = align

        return motifs


Instruction:
Here you have a function in Python 3, explain what it does
def _run_program(self, bin, fastafile, params=None):
        fname = os.path.join(self.config.get_motif_dir(), "JASPAR2010_vertebrate.pwm")
        motifs = read_motifs(fname, fmt="pwm")

        for motif in motifs:
            motif.id = "JASPAR_%s" % motif.id
        return motifs, "", ""

Response: This function runs the JASPAR2010_vertebrate tool and returns a list of Motif instances and a string containing the output of the tool.


Instruction:
How would you code a function in Python 3 to
run MEME and predict motifs from a FASTA file.

Response:
def _run_program(self, bin, fastafile, params=None):
        """
        Run MEME and predict motifs from a FASTA file.

        Parameters
        ----------
        bin : str
            Command used to run the tool.

        fastafile : str
            Name of the FASTA input file.

        params : dict, optional
            Optional parameters. For some of the tools required parameters
            are passed using this dictionary.

        Returns
        -------
        motifs : list of Motif instances
            The predicted motifs.

        stdout : str
            Standard out of the tool.

        stderr : str
            Standard error of the tool.
        """
        default_params = {"width":10, "single":False, "number":10}
        if params is not None:
            default_params.update(params)

        tmp = NamedTemporaryFile(dir=self.tmpdir)
        tmpname = tmp.name

        strand = "-revcomp"
        width = default_params["width"]
        number = default_params["number"]

        cmd = [bin, fastafile, "-text","-dna","-nostatus","-mod", "zoops","-nmotifs", "%s" % number, "-w","%s" % width, "-maxsize", "10000000"]
        if not default_params["single"]:
            cmd.append(strand)

        #sys.stderr.write(" ".join(cmd) + "\n")
        p = Popen(cmd, bufsize=1, stderr=PIPE, stdout=PIPE)
        stdout,stderr = p.communicate()

        motifs = []
        motifs = self.parse(io.StringIO(stdout.decode()))

        # Delete temporary files
        tmp.close()

        return motifs, stdout, stderr


Instruction:
Write a Python 3 script to
convert MEME output to motifs Arcs.

Response:
def parse(self, fo):
        """
        Convert MEME output to motifs

        Parameters
        ----------
        fo : file-like
            File object containing MEME output.

        Returns
        -------
        motifs : list
            List of Motif instances.
        """
        motifs = []
        nucs = {"A":0,"C":1,"G":2,"T":3}

        p = re.compile('MOTIF.+MEME-(\d+)\s*width\s*=\s*(\d+)\s+sites\s*=\s*(\d+)')
        pa = re.compile('\)\s+([A-Z]+)')
        line = fo.readline()
        while line:
            m = p.search(line)
            align = []
            pfm = None
            if m:
                #print(m.group(0))
                id = "%s_%s_w%s" % (self.name, m.group(1), m.group(2))
                while not line.startswith("//"):
                    ma = pa.search(line)
                    if ma:
                        #print(ma.group(0))
                        l = ma.group(1)
                        align.append(l)
                        if not pfm:
                            pfm = [[0 for x in range(4)] for x in range(len(l))]
                        for pos in range(len(l)):
                            if l[pos] in nucs:
                                pfm[pos][nucs[l[pos]]] += 1
                            else:
                                for i in range(4):
                                    pfm[pos][i] += 0.25

                    line = fo.readline()

                motifs.append(Motif(pfm[:]))
                motifs[-1].id = id
                motifs[-1].align = align[:]
            line = fo.readline()

        return motifs


Instruction:
Create a Python 3 function to
scan regions in input table with motifs.

Response:
def scan_to_table(input_table, genome, scoring, pwmfile=None, ncpus=None):
    """Scan regions in input table with motifs.

    Parameters
    ----------
    input_table : str
        Filename of input table. Can be either a text-separated tab file or a
        feather file.

    genome : str
        Genome name. Can be either the name of a FASTA-formatted file or a
        genomepy genome name.

    scoring : str
        "count" or "score"

    pwmfile : str, optional
        Specify a PFM file for scanning.

    ncpus : int, optional
        If defined this specifies the number of cores to use.

    Returns
    -------
    table : pandas.DataFrame
        DataFrame with motif ids as column names and regions as index. Values
        are either counts or scores depending on the 'scoring' parameter.s
    """
    config = MotifConfig()

    if pwmfile is None:
        pwmfile = config.get_default_params().get("motif_db", None)
        if pwmfile is not None:
            pwmfile = os.path.join(config.get_motif_dir(), pwmfile)

    if pwmfile is None:
        raise ValueError("no pwmfile given and no default database specified")

    logger.info("reading table")
    if input_table.endswith("feather"):
        df = pd.read_feather(input_table)
        idx = df.iloc[:,0].values
    else:
        df = pd.read_table(input_table, index_col=0, comment="#")
        idx = df.index

    regions = list(idx)
    s = Scanner(ncpus=ncpus)
    s.set_motifs(pwmfile)
    s.set_genome(genome)
    s.set_background(genome=genome)

    nregions = len(regions)

    scores = []
    if scoring == "count":
        logger.info("setting threshold")
        s.set_threshold(fpr=FPR)
        logger.info("creating count table")
        for row in s.count(regions):
            scores.append(row)
        logger.info("done")
    else:
        s.set_threshold(threshold=0.0)
        logger.info("creating score table")
        for row in s.best_score(regions, normalize=True):
            scores.append(row)
        logger.info("done")

    motif_names = [m.id for m in read_motifs(pwmfile)]
    logger.info("creating dataframe")
    return pd.DataFrame(scores, index=idx, columns=motif_names)


Instruction:
Implement a function in Python 3 to
run maelstrom on a single input table.

Response:
def run_maelstrom(infile, genome, outdir, pwmfile=None, plot=True, cluster=False,
        score_table=None, count_table=None, methods=None, ncpus=None):
    """Run maelstrom on an input table.

    Parameters
    ----------
    infile : str
        Filename of input table. Can be either a text-separated tab file or a
        feather file.

    genome : str
        Genome name. Can be either the name of a FASTA-formatted file or a
        genomepy genome name.

    outdir : str
        Output directory for all results.

    pwmfile : str, optional
        Specify a PFM file for scanning.

    plot : bool, optional
        Create heatmaps.

    cluster : bool, optional
        If True and if the input table has more than one column, the data is
        clustered and the cluster activity methods are also run. Not
        well-tested.

    score_table : str, optional
        Filename of pre-calculated table with motif scores.

    count_table : str, optional
        Filename of pre-calculated table with motif counts.

    methods : list, optional
        Activity methods to use. By default are all used.

    ncpus : int, optional
        If defined this specifies the number of cores to use.
    """
    logger.info("Starting maelstrom")
    if infile.endswith("feather"):
        df = pd.read_feather(infile)
        df = df.set_index(df.columns[0])
    else:
        df = pd.read_table(infile, index_col=0, comment="#")

    # Check for duplicates
    if df.index.duplicated(keep=False).any():
        raise ValueError("Input file contains duplicate regions! "
                         "Please remove them.")

    if not os.path.exists(outdir):
        os.mkdir(outdir)

    if methods is None:
        methods = Moap.list_predictors()
    methods = [m.lower() for m in methods]

    shutil.copyfile(infile, os.path.join(outdir, "input.table.txt"))

    # Copy the motif informatuon
    pwmfile = pwmfile_location(pwmfile)
    if pwmfile:
        shutil.copy2(pwmfile, outdir)
        mapfile = re.sub(".p[fw]m$", ".motif2factors.txt", pwmfile)
        if os.path.exists(mapfile):
            shutil.copy2(mapfile, outdir)

    # Create a file with the number of motif matches
    if not count_table:
        count_table = os.path.join(outdir, "motif.count.txt.gz")
        if not os.path.exists(count_table):
            logger.info("Motif scanning (counts)")
            counts = scan_to_table(infile, genome, "count",
                pwmfile=pwmfile, ncpus=ncpus)
            counts.to_csv(count_table, sep="\t", compression="gzip")
        else:
            logger.info("Counts, using: %s", count_table)

    # Create a file with the score of the best motif match
    if not score_table:
        score_table = os.path.join(outdir, "motif.score.txt.gz")
        if not os.path.exists(score_table):
            logger.info("Motif scanning (scores)")
            scores = scan_to_table(infile, genome, "score",
                pwmfile=pwmfile, ncpus=ncpus)
            scores.to_csv(score_table, sep="\t", float_format="%.3f",
                compression="gzip")
        else:
            logger.info("Scores, using: %s", score_table)

    if cluster:
        cluster = False
        for method in methods:
            m = Moap.create(method, ncpus=ncpus)
            if m.ptype == "classification":
                cluster = True
                break
        if not cluster:
            logger.info("Skipping clustering, no classification methods")

    exps = []
    clusterfile = infile
    if df.shape[1] != 1:
        # More than one column
        for method in Moap.list_regression_predictors():
            if method in methods:
                m = Moap.create(method, ncpus=ncpus)
                exps.append([method, m.pref_table, infile])
                logger.debug("Adding %s", method)

        if cluster:
            clusterfile = os.path.join(outdir,
                    os.path.basename(infile) + ".cluster.txt")

            df[:] = scale(df, axis=0)
            names = df.columns
            df_changed = pd.DataFrame(index=df.index)
            df_changed["cluster"] = np.nan
            for name in names:
                df_changed.loc[(df[name] - df.loc[:,df.columns != name].max(1)) > 0.5, "cluster"] = name
            df_changed.dropna().to_csv(clusterfile, sep="\t")
    if df.shape[1] == 1 or cluster:
        for method in Moap.list_classification_predictors():
            if method in methods:
                m = Moap.create(method, ncpus=ncpus)
                exps.append([method, m.pref_table, clusterfile])

    if len(exps) == 0:
        logger.error("No method to run.")
        sys.exit(1)

    for method, scoring, fname in exps:
        try:
            if scoring == "count" and count_table:
                moap_with_table(fname, count_table, outdir, method, scoring, ncpus=ncpus)
            elif scoring == "score" and score_table:
                moap_with_table(fname, score_table, outdir, method, scoring, ncpus=ncpus)
            else:
                moap_with_bg(fname, genome, outdir, method, scoring, pwmfile=pwmfile, ncpus=ncpus)

        except Exception as e:
            logger.warn("Method %s with scoring %s failed", method, scoring)
            logger.warn(e)
            logger.warn("Skipping")
            raise
    dfs = {}
    for method, scoring,fname  in exps:
        t = "{}.{}".format(method,scoring)
        fname = os.path.join(outdir, "activity.{}.{}.out.txt".format(
                           method, scoring))
        try:
            dfs[t] = pd.read_table(fname, index_col=0, comment="#")
        except:
            logging.warn("Activity file for {} not found!\n".format(t))

    if len(methods) > 1:
        logger.info("Rank aggregation")
        df_p = df_rank_aggregation(df, dfs, exps)
        df_p.to_csv(os.path.join(outdir, "final.out.csv"), sep="\t")
    #df_p = df_p.join(m2f)

    # Write motif frequency table

    if df.shape[1] == 1:
        mcount = df.join(pd.read_table(count_table, index_col=0, comment="#"))
        m_group = mcount.groupby(df.columns[0])
        freq = m_group.sum() / m_group.count()
        freq.to_csv(os.path.join(outdir, "motif.freq.txt"), sep="\t")

    if plot and len(methods) > 1:
        logger.info("html report")
        maelstrom_html_report(
                outdir,
                os.path.join(outdir, "final.out.csv"),
                pwmfile
                )
        logger.info(os.path.join(outdir, "gimme.maelstrom.report.html"))


Instruction:
Given the following Python 3 function, write the documentation
def plot_heatmap(self, kind="final", min_freq=0.01, threshold=2, name=True, max_len=50, aspect=1, **kwargs):

        filt = np.any(np.abs(self.result) >= threshold, 1) & np.any(np.abs(self.freq.T) >= min_freq, 1)

        idx = self.result[filt].index

        cmap = "RdBu_r"
        if kind == "final":
            data = self.result
        elif kind == "freq":
            data = self.freq.T
            cmap = "Reds"
        elif kind in self.activity:
            data = self.activity[dtype]
            if kind in ["hypergeom.count", "mwu.score"]:
                cmap = "Reds"
        else:
            raise ValueError("Unknown dtype")

        #print(data.head())
        #plt.figure(
        m = data.loc[idx]
        if name:
            m["factors"] = [join_max(self.motifs[n].factors, max_len, ",", suffix=",(...)") for n in m.index]
            m = m.set_index("factors")
        h,w = m.shape
        cg = sns.clustermap(m, cmap=cmap, col_cluster=False,
                            figsize=(2 + w * 0.5 * aspect, 0.5 * h), linewidths=1,
                           **kwargs)
        cg.ax_col_dendrogram.set_visible(False)
        plt.setp(cg.ax_heatmap.yaxis.get_majorticklabels(), rotation=0);
        return cg

Response: Plot the clustered heatmap of predicted motif activity.


Instruction:
Implement a Python 3 function for
creating motif scores boxplot of different clusters.

Response:
def plot_scores(self, motifs, name=True, max_len=50):
        """Create motif scores boxplot of different clusters.
        Motifs can be specified as either motif or factor names.
        The motif scores will be scaled and plotted as z-scores.

        Parameters
        ----------
        motifs : iterable or str
            List of motif or factor names.

        name : bool, optional
            Use factor names instead of motif names for plotting.

        max_len : int, optional
            Truncate the list of factors to this maximum length.

        Returns
        -------

        g : FacetGrid
            Returns the seaborn FacetGrid object with the plot.
        """
        if self.input.shape[1] != 1:
            raise ValueError("Can't make a categorical plot with real-valued data")

        if type("") == type(motifs):
            motifs = [motifs]

        plot_motifs = []
        for motif in motifs:
            if motif in self.motifs:
                plot_motifs.append(motif)
            else:
                for m in self.motifs.values():
                    if motif in m.factors:
                        plot_motifs.append(m.id)

        data = self.scores[plot_motifs]
        data[:] = data.scale(data, axix=0)
        if name:
            data = data.T
            data["factors"] = [join_max(self.motifs[n].factors, max_len, ",", suffix=",(...)") for n in plot_motifs]
            data = data.set_index("factors").T

        data = pd.melt(self.input.join(data), id_vars=["cluster"])
        data.columns = ["cluster", "motif", "z-score"]
        g = sns.factorplot(data=data, y="motif", x="z-score", hue="cluster", kind="box", aspect=2)
        return g


Instruction:
Implement a function in Python 3 to
return version of package on pypi. python. org using json. Adapted from https://stackoverflow. com/a/34366589

Response:
def get_version(package, url_pattern=URL_PATTERN):
    """Return version of package on pypi.python.org using json. Adapted from https://stackoverflow.com/a/34366589"""
    req = requests.get(url_pattern.format(package=package))
    version = parse('0')
    if req.status_code == requests.codes.ok:
        # j = json.loads(req.text.encode(req.encoding))
        j = req.json()
        releases = j.get('releases', [])
        for release in releases:
            ver = parse(release)
            if not ver.is_prerelease:
                version = max(version, ver)
    return version


Instruction:
Implement a function in Python 3 to
convert arguments extracted from a parser to a dict and will dismiss arguments which default to NotSet.

Response:
def get_args(parser):
    """
    Converts arguments extracted from a parser to a dict,
    and will dismiss arguments which default to NOT_SET.

    :param parser: an ``argparse.ArgumentParser`` instance.
    :type parser: argparse.ArgumentParser
    :return: Dictionary with the configs found in the parsed CLI arguments.
    :rtype: dict
    """
    args = vars(parser.parse_args()).items()
    return {key: val for key, val in args if not isinstance(val, NotSet)}


Instruction:
Here you have a function in Python 3, explain what it does
def parse_value(val, parsebool=False):
    try:
        return int(val)
    except ValueError:
        pass
    try:
        return float(val)
    except:
        pass
    if parsebool:
        if re.match('yes|on', str(val), re.IGNORECASE):
            return True
        elif re.match('no|off', str(val), re.IGNORECASE):
            return False
    return val

Response: Parse input string and return int float or str depending on format.


Instruction:
Write a Python 3 function for
buffering read from socket. Reads all data available from buffer.

Response:
def socket_read(fp):
    """Buffered read from socket. Reads all data available from socket.

    @fp:     File pointer for socket.
    @return: String of characters read from buffer.

    """
    response = ''
    oldlen = 0
    newlen = 0
    while True:
        response += fp.read(buffSize)
        newlen = len(response)
        if newlen - oldlen == 0:
            break
        else:
            oldlen = newlen
    return response