Instruction:
Explain what the following Python 3 code does
def sam_conversions(self, sam_file, depth=True):
        cmd = self.tools.samtools + " view -bS " + sam_file + " > " + sam_file.replace(".sam", ".bam") + "\n"
        cmd += self.tools.samtools + " sort " + sam_file.replace(".sam", ".bam") + " -o " + sam_file.replace(".sam", "_sorted.bam") + "\n"
        cmd += self.tools.samtools + " index " + sam_file.replace(".sam", "_sorted.bam") + "\n"
        if depth:
            cmd += self.tools.samtools + " depth " + sam_file.replace(".sam", "_sorted.bam") + " > " + sam_file.replace(".sam", "_sorted.depth") + "\n"
        return cmd

Response: Convert sam files to bam files then sort and index them for later use.


Instruction:
Create a Python 3 function to
sort and index bam files for later use.

Response:
def bam_conversions(self, bam_file, depth=True):
        """
        Sort and index bam files for later use.

        :param bool depth: also calculate coverage over each position
        """
        cmd = self.tools.samtools + " view -h " + bam_file + " > " + bam_file.replace(".bam", ".sam") + "\n"
        cmd += self.tools.samtools + " sort " + bam_file + " -o " + bam_file.replace(".bam", "_sorted.bam") + "\n"
        cmd += self.tools.samtools + " index " + bam_file.replace(".bam", "_sorted.bam") + "\n"
        if depth:
            cmd += self.tools.samtools + " depth " + bam_file.replace(".bam", "_sorted.bam") + " > " + bam_file.replace(".bam", "_sorted.depth") + "\n"
        return cmd


Instruction:
Can you tell what is the following Python 3 function doing
def fastqc(self, file, output_dir):
        # You can find the fastqc help with fastqc --help
        try:
            pm = self.pm
        except AttributeError:
            # Do nothing, this is just for path construction.
            pass
        else:
            if not os.path.isabs(output_dir) and pm is not None:
                output_dir = os.path.join(pm.outfolder, output_dir)
        self.make_sure_path_exists(output_dir)
        return "{} --noextract --outdir {} {}".\
                format(self.tools.fastqc, output_dir, file)

Response: Create command to run fastqc on a FASTQ file with sequencing reads.


Instruction:
Create a Python 3 function for
creating pair of commands to run fastqc and organize files. The first command returned is the one that actually runs fastqc when it's executed; the second moves the output files to the output folder for the sample indicated. :param str input_bam: Path to file for which to run fastqc. :param str output_dir: Path to folder in which fastqc output will be written, and within which the sample's output folder lives. :param str sample_name: Sample name, which determines subfolder within output_dir for the fastqc files. :return list[str]: Pair of commands, to run fastqc and then move the files to their intended destination based on sample name.

Response:
def fastqc_rename(self, input_bam, output_dir, sample_name):
        """
        Create pair of commands to run fastqc and organize files.

        The first command returned is the one that actually runs fastqc when
        it's executed; the second moves the output files to the output
        folder for the sample indicated.

        :param str input_bam: Path to file for which to run fastqc.
        :param str output_dir: Path to folder in which fastqc output will be
            written, and within which the sample's output folder lives.
        :param str sample_name: Sample name, which determines subfolder within
            output_dir for the fastqc files.
        :return list[str]: Pair of commands, to run fastqc and then move the files to
            their intended destination based on sample name.
        """
        cmds = list()
        initial = os.path.splitext(os.path.basename(input_bam))[0]
        cmd1 = self.fastqc(input_bam, output_dir)
        cmds.append(cmd1)
        cmd2 = "if [[ ! -s {1}_fastqc.html ]]; then mv {0}_fastqc.html {1}_fastqc.html; mv {0}_fastqc.zip {1}_fastqc.zip; fi".format(
            os.path.join(output_dir, initial), os.path.join(output_dir, sample_name))
        cmds.append(cmd2)
        return cmds


Instruction:
Can you generate a brief explanation for the following Python 3 code
def samtools_index(self, bam_file):
        cmd = self.tools.samtools + " index {0}".format(bam_file)
        return cmd

Response: Index a bam file.


Instruction:
How would you explain what the following Python 3 function does
def skewer(
            self, input_fastq1, output_prefix, output_fastq1,
            log, cpus, adapters, input_fastq2=None, output_fastq2=None):
        """
        Create commands with which to run skewer.

        :param str input_fastq1: Path to input (read 1) FASTQ file
        :param str output_prefix: Prefix for output FASTQ file names
        :param str output_fastq1: Path to (read 1) output FASTQ file
        :param str log: Path to file to which to write logging information
        :param int | str cpus: Number of processing cores to allow
        :param str adapters: Path to file with sequencing adapters
        :param str input_fastq2: Path to read 2 input FASTQ file
        :param str output_fastq2: Path to read 2 output FASTQ file
        :return list[str]: Sequence of commands to run to trim reads with
            skewer and rename files as desired.
        """

        pe = input_fastq2 is not None
        mode = "pe" if pe else "any"
        cmds = list()
        cmd1 = self.tools.skewer + " --quiet"
        cmd1 += " -f sanger"
        cmd1 += " -t {0}".format(cpus)
        cmd1 += " -m {0}".format(mode)
        cmd1 += " -x {0}".format(adapters)
        cmd1 += " -o {0}".format(output_prefix)
        cmd1 += " {0}".format(input_fastq1)
        if input_fastq2 is None:
            cmds.append(cmd1)
        else:
            cmd1 += " {0}".format(input_fastq2)
            cmds.append(cmd1)
        if input_fastq2 is None:
            cmd2 = "mv {0} {1}".format(output_prefix + "-trimmed.fastq", output_fastq1)
            cmds.append(cmd2)
        else:
            cmd2 = "mv {0} {1}".format(output_prefix + "-trimmed-pair1.fastq", output_fastq1)
            cmds.append(cmd2)
            cmd3 = "mv {0} {1}".format(output_prefix + "-trimmed-pair2.fastq", output_fastq2)
            cmds.append(cmd3)
        cmd4 = "mv {0} {1}".format(output_prefix + "-trimmed.log", log)
        cmds.append(cmd4)
        return cmds

Response: This function will run the skewer command that will trim reads with the same name and write the new file.


Instruction:
Given the following Python 3 function, write the documentation
def filter_reads(self, input_bam, output_bam, metrics_file, paired=False, cpus=16, Q=30):
        nodups = re.sub("\.bam$", "", output_bam) + ".nodups.nofilter.bam"
        cmd1 = self.tools.sambamba + " markdup -t {0} -r --compression-level=0 {1} {2} 2> {3}".format(cpus, input_bam, nodups, metrics_file)
        cmd2 = self.tools.sambamba + ' view -t {0} -f bam --valid'.format(cpus)
        if paired:
            cmd2 += ' -F "not (unmapped or mate_is_unmapped) and proper_pair'
        else:
            cmd2 += ' -F "not unmapped'
        cmd2 += ' and not (secondary_alignment or supplementary) and mapping_quality >= {0}"'.format(Q)
        cmd2 += ' {0} |'.format(nodups)
        cmd2 += self.tools.sambamba + " sort -t {0} /dev/stdin -o {1}".format(cpus, output_bam)
        cmd3 = "if [[ -s {0} ]]; then rm {0}; fi".format(nodups)
        cmd4 = "if [[ -s {0} ]]; then rm {0}; fi".format(nodups + ".bai")
        return [cmd1, cmd2, cmd3, cmd4]

Response: Filter the input bam to output bam.


Instruction:
Here you have a function in Python 3, explain what it does
def run_spp(self, input_bam, output, plot, cpus):
        base = "{} {} -rf -savp".format(self.tools.Rscript, self.tools.spp)
        cmd = base + " -savp={} -s=0:5:500 -c={} -out={} -p={}".format(
            plot, input_bam, output, cpus)
        return cmd

Response: Run the SPP read peak analysis tool.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def plot_atacseq_insert_sizes(self, bam, plot, output_csv, max_insert=1500, smallest_insert=30):
        try:
            import pysam
            import numpy as np
            import matplotlib.mlab as mlab
            from scipy.optimize import curve_fit
            from scipy.integrate import simps
            import matplotlib
            matplotlib.use('Agg')
            import matplotlib.pyplot as plt
        except:
            print("Necessary Python modules couldn't be loaded.")
            return

        try:
            import seaborn as sns
            sns.set_style("whitegrid")
        except:
            pass

        def get_fragment_sizes(bam, max_insert=1500):
            frag_sizes = list()

            bam = pysam.Samfile(bam, 'rb')

            for i, read in enumerate(bam):
                if read.tlen < max_insert:
                    frag_sizes.append(read.tlen)
            bam.close()

            return np.array(frag_sizes)

        def mixture_function(x, *p):
            """
            Mixture function to model four gaussian (nucleosomal)
            and one exponential (nucleosome-free) distributions.
            """
            m1, s1, w1, m2, s2, w2, m3, s3, w3, m4, s4, w4, q, r = p
            nfr = expo(x, 2.9e-02, 2.8e-02)
            nfr[:smallest_insert] = 0

            return (mlab.normpdf(x, m1, s1) * w1 +
                    mlab.normpdf(x, m2, s2) * w2 +
                    mlab.normpdf(x, m3, s3) * w3 +
                    mlab.normpdf(x, m4, s4) * w4 +
                    nfr)

        def expo(x, q, r):
            """
            Exponential function.
            """
            return q * np.exp(-r * x)

        # get fragment sizes
        frag_sizes = get_fragment_sizes(bam)

        # bin
        numBins = np.linspace(0, max_insert, max_insert + 1)
        y, scatter_x = np.histogram(frag_sizes, numBins, density=1)
        # get the mid-point of each bin
        x = (scatter_x[:-1] + scatter_x[1:]) / 2

        # Parameters are empirical, need to check
        paramGuess = [
            200, 50, 0.7,  # gaussians
            400, 50, 0.15,
            600, 50, 0.1,
            800, 55, 0.045,
            2.9e-02, 2.8e-02  # exponential
        ]

        try:
            popt3, pcov3 = curve_fit(
                mixture_function, x[smallest_insert:], y[smallest_insert:],
                p0=paramGuess, maxfev=100000)
        except:
            print("Nucleosomal fit could not be found.")
            return

        m1, s1, w1, m2, s2, w2, m3, s3, w3, m4, s4, w4, q, r = popt3

        # Plot
        plt.figure(figsize=(12, 12))

        # Plot distribution
        plt.hist(frag_sizes, numBins, histtype="step", ec="k", normed=1, alpha=0.5)

        # Plot nucleosomal fits
        plt.plot(x, mlab.normpdf(x, m1, s1) * w1, 'r-', lw=1.5, label="1st nucleosome")
        plt.plot(x, mlab.normpdf(x, m2, s2) * w2, 'g-', lw=1.5, label="2nd nucleosome")
        plt.plot(x, mlab.normpdf(x, m3, s3) * w3, 'b-', lw=1.5, label="3rd nucleosome")
        plt.plot(x, mlab.normpdf(x, m4, s4) * w4, 'c-', lw=1.5, label="4th nucleosome")

        # Plot nucleosome-free fit
        nfr = expo(x, 2.9e-02, 2.8e-02)
        nfr[:smallest_insert] = 0
        plt.plot(x, nfr, 'k-', lw=1.5, label="nucleosome-free")

        # Plot sum of fits
        ys = mixture_function(x, *popt3)
        plt.plot(x, ys, 'k--', lw=3.5, label="fit sum")

        plt.legend()
        plt.xlabel("Fragment size (bp)")
        plt.ylabel("Density")
        plt.savefig(plot, bbox_inches="tight")

        # Integrate curves and get areas under curve
        areas = [
            ["fraction", "area under curve", "max density"],
            ["Nucleosome-free fragments", simps(nfr), max(nfr)],
            ["1st nucleosome", simps(mlab.normpdf(x, m1, s1) * w1), max(mlab.normpdf(x, m1, s1) * w1)],
            ["2nd nucleosome", simps(mlab.normpdf(x, m2, s2) * w1), max(mlab.normpdf(x, m2, s2) * w2)],
            ["3rd nucleosome", simps(mlab.normpdf(x, m3, s3) * w1), max(mlab.normpdf(x, m3, s3) * w3)],
            ["4th nucleosome", simps(mlab.normpdf(x, m4, s4) * w1), max(mlab.normpdf(x, m4, s4) * w4)]
        ]

        try:
            import csv

            with open(output_csv, "w") as f:
                writer = csv.writer(f)
                writer.writerows(areas)
        except:
            pass

Response: Plot the length of the fragmented sequence in a file.


Instruction:
Can you tell what is the following Python 3 function doing
def bam_to_bigwig(
            self, input_bam, output_bigwig, genome_sizes, genome,
            tagmented=False, normalize=False, norm_factor=1000):
        """
        Convert a BAM file to a bigWig file.

        :param str input_bam: path to BAM file to convert
        :param str output_bigwig: path to which to write file in bigwig format
        :param str genome_sizes: path to file with chromosome size information
        :param str genome: name of genomic assembly
        :param bool tagmented: flag related to read-generating protocol
        :param bool normalize: whether to normalize coverage
        :param int norm_factor: number of bases to use for normalization
        :return list[str]: sequence of commands to execute
        """
        # TODO:
        # addjust fragment length dependent on read size and real fragment size
        # (right now it asssumes 50bp reads with 180bp fragments)
        cmds = list()
        transient_file = os.path.abspath(re.sub("\.bigWig", "", output_bigwig))
        cmd1 = self.tools.bedtools + " bamtobed -i {0} |".format(input_bam)
        if not tagmented:
            cmd1 += " " + self.tools.bedtools + " slop -i stdin -g {0} -s -l 0 -r 130 |".format(genome_sizes)
            cmd1 += " fix_bedfile_genome_boundaries.py {0} |".format(genome)
        cmd1 += " " + self.tools.genomeCoverageBed + " {0}-bg -g {1} -i stdin > {2}.cov".format(
            "-5 " if tagmented else "",
            genome_sizes,
            transient_file
        )
        cmds.append(cmd1)
        if normalize:
            cmds.append("""awk 'NR==FNR{{sum+= $4; next}}{{ $4 = ($4 / sum) * {1}; print}}' {0}.cov {0}.cov | sort -k1,1 -k2,2n > {0}.normalized.cov""".format(transient_file, norm_factor))
        cmds.append(self.tools.bedGraphToBigWig + " {0}{1}.cov {2} {3}".format(transient_file, ".normalized" if normalize else "", genome_sizes, output_bigwig))
        # remove tmp files
        cmds.append("if [[ -s {0}.cov ]]; then rm {0}.cov; fi".format(transient_file))
        if normalize:
            cmds.append("if [[ -s {0}.normalized.cov ]]; then rm {0}.normalized.cov; fi".format(transient_file))
        cmds.append("chmod 755 {0}".format(output_bigwig))
        return cmds

Response: Convert a BAM file to a bigWig file.


Instruction:
Can you write a function in Python 3 where it
calculates the fraction of reads in peaks defined in input_bam and input_bed.

Response:
def calc_frip(self, input_bam, input_bed, threads=4):
        """
        Calculate fraction of reads in peaks.

        A file of with a pool of sequencing reads and a file with peak call
        regions define the operation that will be performed. Thread count
        for samtools can be specified as well.

        :param str input_bam: sequencing reads file
        :param str input_bed: file with called peak regions
        :param int threads: number of threads samtools may use
        :return float: fraction of reads in peaks defined in given peaks file
        """
        cmd = self.simple_frip(input_bam, input_bed, threads)
        return subprocess.check_output(cmd.split(" "), shell=True)


Instruction:
How would you implement a function in Python 3 that
uses MACS2 to call peaks in the specified sample.

Response:
def macs2_call_peaks(
                self, treatment_bams, output_dir, sample_name, genome,
                control_bams=None, broad=False, paired=False,
                pvalue=None, qvalue=None, include_significance=None):
        """
        Use MACS2 to call peaks.

        :param str | Iterable[str] treatment_bams: Paths to files with data to
            regard as treatment.
        :param str output_dir: Path to output folder.
        :param str sample_name: Name for the sample involved.
        :param str genome: Name of the genome assembly to use.
        :param str | Iterable[str] control_bams: Paths to files with data to
            regard as control
        :param bool broad: Whether to do broad peak calling.
        :param bool paired: Whether reads are paired-end
        :param float | NoneType pvalue: Statistical significance measure to
            pass as --pvalue to peak calling with MACS
        :param float | NoneType qvalue: Statistical significance measure to
            pass as --qvalue to peak calling with MACS
        :param bool | NoneType include_significance: Whether to pass a
            statistical significance argument to peak calling with MACS; if
            omitted, this will be True if the peak calling is broad or if
            either p-value or q-value is specified; default significance
            specification is a p-value of 0.001 if a significance is to be
            specified but no value is provided for p-value or q-value.
        :return str: Command to run.
        """
        sizes = {"hg38": 2.7e9, "hg19": 2.7e9, "mm10": 1.87e9, "dr7": 1.412e9, "mm9": 1.87e9}

        # Whether to specify to MACS2 a value for statistical significance
        # can be either directly indicated, but if not, it's determined by
        # whether the mark is associated with broad peaks. By default, we
        # specify a significance value to MACS2 for a mark associated with a
        # broad peak.
        if include_significance is None:
            include_significance = broad

        cmd = self.tools.macs2 + " callpeak -t {0}".format(treatment_bams if type(treatment_bams) is str else " ".join(treatment_bams))

        if control_bams is not None:
            cmd += " -c {0}".format(control_bams if type(control_bams) is str else " ".join(control_bams))

        if paired:
            cmd += " -f BAMPE "

        # Additional settings based on whether the marks is associated with
        # broad peaks
        if broad:
            cmd += " --broad --nomodel --extsize 73"
        else:
            cmd += " --fix-bimodal --extsize 180 --bw 200"

        if include_significance:
            # Allow significance specification via either p- or q-value,
            # giving preference to q-value if both are provided but falling
            # back on a default p-value if neither is provided but inclusion
            # of statistical significance measure is desired.
            if qvalue is not None:
                cmd += " --qvalue {}".format(qvalue)
            else:
                cmd += " --pvalue {}".format(pvalue or 0.00001)
        cmd += " -g {0} -n {1} --outdir {2}".format(sizes[genome], sample_name, output_dir)

        return cmd


Instruction:
Explain what the following Python 3 code does
def spp_call_peaks(
                self, treatment_bam, control_bam, treatment_name, control_name,
                output_dir, broad, cpus, qvalue=None):
        """
        Build command for R script to call peaks with SPP.

        :param str treatment_bam: Path to file with data for treatment sample.
        :param str control_bam: Path to file with data for control sample.
        :param str treatment_name: Name for the treatment sample.
        :param str control_name: Name for the control sample.
        :param str output_dir: Path to folder for output.
        :param str | bool broad: Whether to specify broad peak calling mode.
        :param int cpus: Number of cores the script may use.
        :param float qvalue: FDR, as decimal value
        :return str: Command to run.
        """
        broad = "TRUE" if broad else "FALSE"
        cmd = self.tools.Rscript + " `which spp_peak_calling.R` {0} {1} {2} {3} {4} {5} {6}".format(
            treatment_bam, control_bam, treatment_name, control_name, broad, cpus, output_dir
        )
        if qvalue is not None:
            cmd += " {}".format(qvalue)
        return cmd

Response: Build command to call peaks with SPP.


Instruction:
How would you explain what the following Python 3 function does
def get_read_type(self, bam_file, n=10):
        from collections import Counter
        try:
            p = subprocess.Popen([self.tools.samtools, 'view', bam_file],
                                 stdout=subprocess.PIPE)
            # Count paired alignments
            paired = 0
            read_length = Counter()
            while n > 0:
                line = p.stdout.next().split("\t")
                flag = int(line[1])
                read_length[len(line[9])] += 1
                if 1 & flag:  # check decimal flag contains 1 (paired)
                    paired += 1
                n -= 1
            p.kill()
        except IOError("Cannot read provided bam file.") as e:
            raise e
        # Get most abundant read read_length
        read_length = sorted(read_length)[-1]
        # If at least half is paired, return True
        if paired > (n / 2.):
            return "PE", read_length
        else:
            return "SE", read_length

Response: Get the read type and length of a specific read from a BAM file.


Instruction:
Explain what the following Python 3 code does
def parse_bowtie_stats(self, stats_file):
        import pandas as pd
        stats = pd.Series(index=["readCount", "unpaired", "unaligned", "unique", "multiple", "alignmentRate"])
        try:
            with open(stats_file) as handle:
                content = handle.readlines()  # list of strings per line
        except:
            return stats
        # total reads
        try:
            line = [i for i in range(len(content)) if " reads; of these:" in content[i]][0]
            stats["readCount"] = re.sub("\D.*", "", content[line])
            if 7 > len(content) > 2:
                line = [i for i in range(len(content)) if "were unpaired; of these:" in content[i]][0]
                stats["unpaired"] = re.sub("\D", "", re.sub("\(.*", "", content[line]))
            else:
                line = [i for i in range(len(content)) if "were paired; of these:" in content[i]][0]
                stats["unpaired"] = stats["readCount"] - int(re.sub("\D", "", re.sub("\(.*", "", content[line])))
            line = [i for i in range(len(content)) if "aligned 0 times" in content[i]][0]
            stats["unaligned"] = re.sub("\D", "", re.sub("\(.*", "", content[line]))
            line = [i for i in range(len(content)) if "aligned exactly 1 time" in content[i]][0]
            stats["unique"] = re.sub("\D", "", re.sub("\(.*", "", content[line]))
            line = [i for i in range(len(content)) if "aligned >1 times" in content[i]][0]
            stats["multiple"] = re.sub("\D", "", re.sub("\(.*", "", content[line]))
            line = [i for i in range(len(content)) if "overall alignment rate" in content[i]][0]
            stats["alignmentRate"] = re.sub("\%.*", "", content[line]).strip()
        except IndexError:
            pass
        return stats

Response: Parses Bowtie2 stats file and returns pandas Series with values.


Instruction:
Can you generate the documentation for the following Python 3 function
def parse_duplicate_stats(self, stats_file):
        import pandas as pd
        series = pd.Series()
        try:
            with open(stats_file) as handle:
                content = handle.readlines()  # list of strings per line
        except:
            return series
        try:
            line = [i for i in range(len(content)) if "single ends (among them " in content[i]][0]
            series["single-ends"] = re.sub("\D", "", re.sub("\(.*", "", content[line]))
            line = [i for i in range(len(content)) if " end pairs...   done in " in content[i]][0]
            series["paired-ends"] = re.sub("\D", "", re.sub("\.\.\..*", "", content[line]))
            line = [i for i in range(len(content)) if " duplicates, sorting the list...   done in " in content[i]][0]
            series["duplicates"] = re.sub("\D", "", re.sub("\.\.\..*", "", content[line]))
        except IndexError:
            pass
        return series

Response: Parses sambamba markdup output returns series with values.


Instruction:
Make a summary of the following Python 3 code
def parse_qc(self, qc_file):
        import pandas as pd
        series = pd.Series()
        try:
            with open(qc_file) as handle:
                line = handle.readlines()[0].strip().split("\t")  # list of strings per line
            series["NSC"] = line[-3]
            series["RSC"] = line[-2]
            series["qualityTag"] = line[-1]
        except:
            pass
        return series

Response: Parse the phantompeakqualtools QC table and return a pandas Series.


Instruction:
Create a Python 3 function to
count the number of peaks from a sample s peak file.

Response:
def get_peak_number(self, sample):
        """
        Counts number of peaks from a sample's peak file.

        :param pipelines.Sample sample: Sample object with "peaks" attribute.
        """
        proc = subprocess.Popen(["wc", "-l", sample.peaks], stdout=subprocess.PIPE)
        out, err = proc.communicate()
        sample["peakNumber"] = re.sub("\D.*", "", out)
        return sample


Instruction:
Make a summary of the following Python 3 code
def get_frip(self, sample):
        import pandas as pd
        with open(sample.frip, "r") as handle:
            content = handle.readlines()
        reads_in_peaks = int(re.sub("\D", "", content[0]))
        mapped_reads = sample["readCount"] - sample["unaligned"]
        return pd.Series(reads_in_peaks / mapped_reads, index="FRiP")

Response: Calculates the fraction of reads in peaks for a given sample.


Instruction:
Can you write a function in Python 3 where it
ignores interrupt and termination signals.

Response:
def _ignore_interrupts(self):
        """
        Ignore interrupt and termination signals. Used as a pre-execution
        function (preexec_fn) for subprocess.Popen calls that pypiper will
        control over (i.e., manually clean up).
        """
        signal.signal(signal.SIGINT, signal.SIG_IGN)
        signal.signal(signal.SIGTERM, signal.SIG_IGN)


Instruction:
Write a Python 3 script to
start a new pipeline with the given arguments.

Response:
def start_pipeline(self, args=None, multi=False):
        """
        Initialize setup. Do some setup, like tee output, print some diagnostics, create temp files.
        You provide only the output directory (used for pipeline stats, log, and status flag files).
        """
        # Perhaps this could all just be put into __init__, but I just kind of like the idea of a start function
        self.make_sure_path_exists(self.outfolder)

        # By default, Pypiper will mirror every operation so it is displayed both
        # on sys.stdout **and** to a log file. Unfortunately, interactive python sessions
        # ruin this by interfering with stdout. So, for interactive mode, we do not enable
        # the tee subprocess, sending all output to screen only.
        # Starting multiple PipelineManagers in the same script has the same problem, and
        # must therefore be run in interactive_mode.

        interactive_mode = multi or not hasattr(__main__, "__file__")
        if interactive_mode:
            print("Warning: You're running an interactive python session. "
                  "This works, but pypiper cannot tee the output, so results "
                  "are only logged to screen.")
        else:
            sys.stdout = Unbuffered(sys.stdout)
            # sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)  # Unbuffer output

            # The tee subprocess must be instructed to ignore TERM and INT signals;
            # Instead, I will clean up this process in the signal handler functions.
            # This is required because otherwise, if pypiper receives a TERM or INT,
            # the tee will be automatically terminated by python before I have a chance to
            # print some final output (for example, about when the process stopped),
            # and so those things don't end up in the log files because the tee
            # subprocess is dead. Instead, I will handle the killing of the tee process
            # manually (in the exit handler).

            # a for append to file

            tee = subprocess.Popen(
                ["tee", "-a", self.pipeline_log_file], stdin=subprocess.PIPE,
                preexec_fn=self._ignore_interrupts)

            # If the pipeline is terminated with SIGTERM/SIGINT,
            # make sure we kill this spawned tee subprocess as well.
            # atexit.register(self._kill_child_process, tee.pid, proc_name="tee")
            os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
            os.dup2(tee.stdin.fileno(), sys.stderr.fileno())

            self.tee = tee

        # For some reason, this exit handler function MUST be registered after
        # the one that kills the tee process.
        atexit.register(self._exit_handler)

        # A future possibility to avoid this tee, is to use a Tee class; this works for anything printed here
        # by pypiper, but can't tee the subprocess output. For this, it would require using threading to
        # simultaneously capture and display subprocess output. I shelve this for now and stick with the tee option.
        # sys.stdout = Tee(self.pipeline_log_file)

        # Record the git version of the pipeline and pypiper used. This gets (if it is in a git repo):
        # dir: the directory where the code is stored
        # hash: the commit id of the last commit in this repo
        # date: the date of the last commit in this repo
        # diff: a summary of any differences in the current (run) version vs. the committed version

        # Wrapped in try blocks so that the code will not fail if the pipeline or pypiper are not git repositories
        gitvars = {}
        try:
            # pypiper dir
            ppd = os.path.dirname(os.path.realpath(__file__))
            gitvars['pypiper_dir'] = ppd
            gitvars['pypiper_hash'] = subprocess.check_output("cd " + ppd + "; git rev-parse --verify HEAD 2>/dev/null", shell=True)
            gitvars['pypiper_date'] = subprocess.check_output("cd " + ppd + "; git show -s --format=%ai HEAD 2>/dev/null", shell=True)
            gitvars['pypiper_diff'] = subprocess.check_output("cd " + ppd + "; git diff --shortstat HEAD 2>/dev/null", shell=True)
            gitvars['pypiper_branch'] = subprocess.check_output("cd " + ppd + "; git branch | grep '*' 2>/dev/null", shell=True)
        except Exception:
            pass
        try:
            # pipeline dir
            pld = os.path.dirname(os.path.realpath(sys.argv[0]))
            gitvars['pipe_dir'] = pld
            gitvars['pipe_hash'] = subprocess.check_output("cd " + pld + "; git rev-parse --verify HEAD 2>/dev/null", shell=True)
            gitvars['pipe_date'] = subprocess.check_output("cd " + pld + "; git show -s --format=%ai HEAD 2>/dev/null", shell=True)
            gitvars['pipe_diff'] = subprocess.check_output("cd " + pld + "; git diff --shortstat HEAD 2>/dev/null", shell=True)
            gitvars['pipe_branch'] = subprocess.check_output("cd " + pld + "; git branch | grep '*' 2>/dev/null", shell=True)
        except Exception:
            pass

        # Print out a header section in the pipeline log:
        # Wrap things in backticks to prevent markdown from interpreting underscores as emphasis.
        # print("----------------------------------------")
        print("### [Pipeline run code and environment:]\n")
        print("* " + "Command".rjust(20) + ":  " + "`" + str(" ".join(sys.argv)) + "`")
        print("* " + "Compute host".rjust(20) + ":  " + platform.node())
        print("* " + "Working dir".rjust(20) + ":  " + os.getcwd())
        print("* " + "Outfolder".rjust(20) + ":  " + self.outfolder)

        self.timestamp("* " + "Pipeline started at".rjust(20) + ":  ")

        print("\n### [Version log:]\n")
        print("* " + "Python version".rjust(20) + ":  " + platform.python_version())
        try:
            print("* " + "Pypiper dir".rjust(20) + ":  " + "`" + gitvars['pypiper_dir'].strip() + "`")
            print("* " + "Pypiper version".rjust(20) + ":  " + __version__)
            print("* " + "Pypiper hash".rjust(20) + ":  " + str(gitvars['pypiper_hash']).strip())
            print("* " + "Pypiper branch".rjust(20) + ":  " + str(gitvars['pypiper_branch']).strip())
            print("* " + "Pypiper date".rjust(20) + ":  " + str(gitvars['pypiper_date']).strip())
            if "" != str(gitvars['pypiper_diff']):
                print("* " + "Pypiper diff".rjust(20) + ":  " + str(gitvars['pypiper_diff']).strip())
        except KeyError:
            # It is ok if keys aren't set, it means pypiper isn't in a  git repo.
            pass

        try:
            print("* " + "Pipeline dir".rjust(20) + ":  " + "`" + gitvars['pipe_dir'].strip() + "`")
            print("* " + "Pipeline version".rjust(20) + ":  " + str(self.pl_version))
            print("* " + "Pipeline hash".rjust(20) + ":  " + str(gitvars['pipe_hash']).strip())
            print("* " + "Pipeline branch".rjust(20) + ":  " + str(gitvars['pipe_branch']).strip())
            print("* " + "Pipeline date".rjust(20) + ":  " + str(gitvars['pipe_date']).strip())
            if (gitvars['pipe_diff'] != ""):
                print("* " + "Pipeline diff".rjust(20) + ":  " + str(gitvars['pipe_diff']).strip())
        except KeyError:
            # It is ok if keys aren't set, it means the pipeline isn't a git repo.
            pass

        # Print all arguments (if any)
        print("\n### [Arguments passed to pipeline:]\n")
        for arg, val in (vars(args) if args else dict()).items():
            argtext = "`{}`".format(arg)
            valtext = "`{}`".format(val)
            print("* {}:  {}".format(argtext.rjust(20), valtext))
        print("\n----------------------------------------\n")
        self._set_status_flag(RUN_FLAG)

        # Record the start in PIPE_profile and PIPE_commands output files so we
        # can trace which run they belong to

        with open(self.pipeline_commands_file, "a") as myfile:
            myfile.write("# Pipeline started at " + time.strftime("%m-%d %H:%M:%S", time.localtime(self.starttime)) + "\n\n")

        with open(self.pipeline_profile_file, "a") as myfile:
            myfile.write("# Pipeline started at " + time.strftime("%m-%d %H:%M:%S", time.localtime(self.starttime)) + "\n\n")


Instruction:
Create a Python 3 function to
configure state and files on disk to match current processing status.

Response:
def _set_status_flag(self, status):
        """
        Configure state and files on disk to match current processing status.

        :param str status: Name of new status designation for pipeline.
        """

        # Remove previous status flag file.
        flag_file_path = self._flag_file_path()
        try:
            os.remove(flag_file_path)
        except:
            # Print message only if the failure to remove the status flag
            # is unexpected; there's no flag for initialization, so we
            # can't remove the file.
            if self.status != "initializing":
                print("Could not remove flag file: '{}'".format(flag_file_path))
            pass

        # Set new status.
        prev_status = self.status
        self.status = status
        self._create_file(self._flag_file_path())
        print("\nChanged status from {} to {}.".format(
                prev_status, self.status))


Instruction:
Here you have a function in Python 3, explain what it does
def _flag_file_path(self, status=None):
        flag_file_name = "{}_{}".format(
                self.name, flag_name(status or self.status))
        return pipeline_filepath(self, filename=flag_file_name)

Response: Create path to flag file based on indicated or current status.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def run(self, cmd, target=None, lock_name=None, shell=None, nofail=False, clean=False, follow=None, container=None):

        # If the pipeline's not been started, skip ahead.
        if not self._active:
            cmds = [cmd] if isinstance(cmd, str) else cmd
            cmds_text = [c if isinstance(c, str) else " ".join(c) for c in cmds]
            print("Pipeline is inactive; skipping {} command(s):\n{}".
                  format(len(cmds), "\n".join(cmds_text)))
            return 0

        # Short-circuit if the checkpoint file exists and the manager's not
        # been configured to overwrite such files.
        if self.curr_checkpoint is not None:
            check_fpath = checkpoint_filepath(self.curr_checkpoint, self)
            if os.path.isfile(check_fpath) and not self.overwrite_checkpoints:
                print("Checkpoint file exists for '{}' ('{}'), and the {} has "
                      "been configured to not overwrite checkpoints; "
                      "skipping command '{}'".format(
                    self.curr_checkpoint, check_fpath,
                    self.__class__.__name__, cmd))
                return 0

        # TODO: consider making the logic such that locking isn't implied, or
        # TODO (cont.): that we can make it otherwise such that it's not
        # TODO (cont.): strictly necessary to provide target or lock_name.
        # The default lock name is based on the target name.
        # Therefore, a targetless command that you want
        # to lock must specify a lock_name manually.
        if target is None and lock_name is None:
            self.fail_pipeline(Exception(
                "You must provide either a target or a lock_name."))

        # Downstream code requires target to be a list, so convert if only
        # a single item was given
        if not is_multi_target(target) and target is not None:
            target = [target]

        # Downstream code requires a list of locks; convert
        if isinstance(lock_name, str):
            lock_name = [lock_name]

        # Default lock_name (if not provided) is based on the target file name,
        # but placed in the parent pipeline outfolder
        lock_name = lock_name or make_lock_name(target, self.outfolder)
        lock_files = [self._make_lock_path(ln) for ln in lock_name]

        process_return_code = 0
        local_maxmem = 0

        # Decide how to do follow-up.
        if not follow:
            call_follow = lambda: None
        elif not hasattr(follow, "__call__"):
            # Warn about non-callable argument to follow-up function.
            print("Follow-up function is not callable and won't be used: {}".
                  format(type(follow)))
            call_follow = lambda: None
        else:
            # Wrap the follow-up function so that the log shows what's going on.
            def call_follow():
                print("Follow:")
                follow()

        # The while=True loop here is unlikely to be triggered, and is just a
        # wrapper to prevent race conditions; the lock_file must be created by
        # the current loop. If not, we loop again and then re-do the tests.
        # The recover and newstart options inform the pipeline to run a command
        # in a scenario where it normally would not. We use these "local" flags
        # to allow us to report on the state of the pipeline in the first round
        # as normal, but then proceed on the next iteration through the outer
        # loop. The proceed_through_locks is a flag that is set if any lockfile
        # is found that needs to be recovered or overwritten. It instructs us to
        # ignore lock files on the next iteration.
        local_recover = False
        local_newstart = False
        proceed_through_locks = False

        while True:
            ##### Tests block
            # Base case: All targets exists and not set to overwrite targets break loop, don't run process.
            # os.path.exists returns True for either a file or directory; .isfile is file-only
            if target is not None and all([os.path.exists(t) for t in target]) \
                    and not any([os.path.isfile(l) for l in lock_files]) \
                    and not local_newstart:
                for tgt in target:
                    if os.path.exists(tgt): print("Target exists: `" + tgt + "`")
                if self.new_start:
                    print("New start mode; run anyway.")
                    # Set the local_newstart flag so the command will run anyway.
                    # Doing this in here instead of outside the loop allows us
                    # to still report the target existence.
                    local_newstart = True
                    continue
                # Normally we don't run the follow, but if you want to force. . .
                if self.force_follow:
                    call_follow()
                break  # Do not run command

            # Scenario 1: Lock file exists, but we're supposed to overwrite target; Run process.
            if not proceed_through_locks:
                for lock_file in lock_files:
                    recover_file = self._recoverfile_from_lockfile(lock_file)
                    if os.path.isfile(lock_file):
                        print("Found lock file: {}".format(lock_file))
                        if self.overwrite_locks:
                            print("Overwriting target. . .")
                            proceed_through_locks = True
                        elif os.path.isfile(recover_file):
                            print("Found dynamic recovery file ({}); "
                                  "overwriting target. . .".format(recover_file))
                            # remove the lock file which will then be promptly re-created for the current run.
                            local_recover = True
                            proceed_through_locks = True
                            # the recovery flag is now spent; remove so we don't accidentally re-recover a failed job
                            os.remove(recover_file)
                        else:  # don't overwrite locks
                            self._wait_for_lock(lock_file)
                            # when it's done loop through again to try one more
                            # time (to see if the target exists now)
                            continue

            # If you get to this point, the target doesn't exist, and the lock_file doesn't exist
            # (or we should overwrite). create the lock (if you can)
            # Initialize lock in master lock list
            for lock_file in lock_files:
                self.locks.append(lock_file)
                if self.overwrite_locks or local_recover:
                    self._create_file(lock_file)
                else:
                    try:
                        self._create_file_racefree(lock_file)  # Create lock
                    except OSError as e:
                        if e.errno == errno.EEXIST:  # File already exists
                            print ("Lock file created after test! Looping again: {}".format(
                                lock_file))

                            # Since a lock file was created by a different source,
                            # we need to reset this flag to re-check the locks.
                            proceed_through_locks = False
                            continue  # Go back to start

            ##### End tests block
            # If you make it past these tests, we should proceed to run the process.

            if target is not None:
                print("Target to produce: {}\n".format(",".join(['`'+x+'`' for x in target])))
            else:
                print("Targetless command, running...\n")

            if isinstance(cmd, list):  # Handle command lists
                for cmd_i in cmd:
                    list_ret, maxmem = \
                        self.callprint(cmd_i, shell, lock_file, nofail, container)
                    maxmem = max(maxmem) if isinstance(maxmem, Iterable) else maxmem
                    local_maxmem = max(local_maxmem,  maxmem)
                    list_ret = max(list_ret) if isinstance(list_ret, Iterable) else list_ret
                    process_return_code = max(process_return_code, list_ret)

            else:  # Single command (most common)
                process_return_code, local_maxmem = \
                    self.callprint(cmd, shell, lock_file, nofail, container)  # Run command
                if isinstance(process_return_code, list):
                    process_return_code = max(process_return_code)

            # For temporary files, you can specify a clean option to automatically
            # add them to the clean list, saving you a manual call to clean_add
            if target is not None and clean:
                for tgt in target:
                    self.clean_add(tgt)

            call_follow()
            for lock_file in lock_files:
                os.remove(lock_file)  # Remove lock file
                self.locks.remove(lock_file)

            # If you make it to the end of the while loop, you're done
            break

        return process_return_code

Response: This is the primary workhorse function of PipelineManager. It runs a command in a Docker container.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def checkprint(self, cmd, shell=None, nofail=False):

        self._report_command(cmd)

        likely_shell = check_shell(cmd, shell)

        if shell is None:
            shell = likely_shell

        if not shell:
            if likely_shell:
                print("Should this command run in a shell instead of directly in a subprocess?")
            cmd = shlex.split(cmd)

        try:
            return subprocess.check_output(cmd, shell=shell)
        except Exception as e:
            self._triage_error(e, nofail)

Response: This function is used to run a command in a specific shell and return the output of the command.


Instruction:
Can you implement a function in Python 3 that
prints the given command and then executes it.

Response:
def callprint(self, cmd, shell=None, lock_file=None, nofail=False, container=None):
        """
        Prints the command, and then executes it, then prints the memory use and
        return code of the command.

        Uses python's subprocess.Popen() to execute the given command. The shell argument is simply
        passed along to Popen(). You should use shell=False (default) where possible, because this enables memory
        profiling. You should use shell=True if you require shell functions like redirects (>) or stars (*), but this
        will prevent the script from monitoring memory use. The pipes (|) will be used to split the command into
        subprocesses run within python, so the memory profiling is possible.

        cmd can also be a series (a dict object) of multiple commands, which will be run in succession.

        :param str | Iterable[str] cmd: Bash command(s) to be run.
        :param str lock_file: a lock file name
        :param bool nofail: Should the pipeline bail on a nonzero return from a process? Default: False
            Nofail can be used to implement non-essential parts of the pipeline; if these processes fail,
            they will not cause the pipeline to bail out.
        :param bool shell: if the command should be run it its own shell, default: None (will try
            to determine based on the command)
        :param container: Named Docker container in which to execute.
        :param container: str
        """
        # The Popen shell argument works like this:
        # if shell=False, then we format the command (with split()) to be a list of command and its arguments.
        # Split the command to use shell=False;
        # leave it together to use shell=True;

        def get_mem_child_sum(proc):
            try:
                # get children processes
                children = proc.children(recursive=True)
                # get RSS memory of each child proc and sum all
                mem_sum = sum([x.memory_info().rss for x in children])
                # return in gigs
                return mem_sum/1e9
            except (psutil.NoSuchProcess, psutil.ZombieProcess) as e:
                print(e)
                print("Warning: couldn't add memory use for process: {}".format(proc.pid))
                return 0

        def display_memory(memval):
            return None if memval < 0 else "{}GB".format(round(memval, 3))

        def make_dict(command):
            a, s = (command, True) if check_shell(command, shell) else (shlex.split(command), False)
            return dict(args=a, stdout=subprocess.PIPE, shell=s)

        if container:
            cmd = "docker exec " + container + " " + cmd

        param_list = [make_dict(c) for c in split_by_pipes(cmd)] \
            if check_shell_pipes(cmd) else [dict(args=cmd, stdout=None, shell=True)]

        proc_name = get_proc_name(cmd)

        # stop_times = []
        processes = []
        running_processes = []
        start_time = time.time()
        for i in range(len(param_list)):
            running_processes.append(i)
            if i == 0:
                processes.append(psutil.Popen(preexec_fn=os.setpgrp, **param_list[i]))
            else:
                param_list[i]["stdin"] = processes[i - 1].stdout
                processes.append(psutil.Popen(preexec_fn=os.setpgrp, **param_list[i]))

            self.procs[processes[-1].pid] = {
                "proc_name": proc_name,
                "subproc_name" : get_proc_name(param_list[i]["args"]),
                "start_time": start_time,
                "container": container,
                "p": processes[-1]
            }

        self._report_command(cmd, [x.pid for x in processes])
            # Capture the subprocess output in
 tags to make it format nicely
            # if the markdown log file is displayed as HTML.
        print("
")

        local_maxmems = [-1] * len(running_processes)
        returncodes = [None] * len(running_processes)

        if not self.wait:
            print("
")
            ids = [x.pid for x in processes]
            print ("Not waiting for subprocess(es): " + str(ids))
            return [0, -1]

        def proc_wrapup(i):
            """
            :param i: internal ID number of the subprocess
            """
            returncode = processes[i].returncode
            current_pid = processes[i].pid

            info = "Process {pid} returned {ret}; memory: {mem}. ".format(
                pid=current_pid,
                ret=processes[i].returncode,
                mem=display_memory(local_maxmems[i]))

            # report process profile
            self._report_profile(self.procs[current_pid]["proc_name"], lock_file, time.time() - self.procs[current_pid]["start_time"], local_maxmems[i])

            # Remove this as a running subprocess
            del self.procs[current_pid]
            running_processes.remove(i)

            returncodes[i] = returncode
            return info

        sleeptime = .0001
        info = ""
        while running_processes:
            for i in running_processes:
                local_maxmems[i] = max(local_maxmems[i], (get_mem_child_sum(processes[i])))
                self.peak_memory = max(self.peak_memory, local_maxmems[i])
                if not self._attend_process(processes[i], sleeptime):
                    info += proc_wrapup(i)

            # the sleeptime is extremely short at the beginning and gets longer exponentially
            # (+ constant to prevent copious checks at the very beginning)
            # = more precise mem tracing for short processes
            sleeptime = min((sleeptime + 0.25) * 3 , 60/len(processes))

        # All jobs are done, print a final closing and job info
        stop_time = time.time()
        info += " Elapsed: " + str(datetime.timedelta(seconds=self.time_elapsed(start_time))) + "."
        info += " Peak memory: {pipe}.".format(pipe=display_memory(self.peak_memory))

        print("
")
        print(info)

        for rc in returncodes:
            if rc != 0:
                msg = "Subprocess returned nonzero result. Check above output for details"
                self._triage_error(SubprocessError(msg), nofail)

        return [returncodes, local_maxmems]


Instruction:
Write a Python 3 script to
debug function used in unit tests. :param p: A subprocess.Popen process. :param bool shell: If command requires should be run in its own shell. Optional. Default: False.

Response:
def _wait_for_process(self, p, shell=False):
        """
        Debug function used in unit tests.

        :param p: A subprocess.Popen process.
        :param bool shell: If command requires should be run in its own shell. Optional. Default: False.
        """
        local_maxmem = -1
        sleeptime = .5
        while p.poll() is None:
            if not shell:
                local_maxmem = max(local_maxmem, self._memory_usage(p.pid) / 1e6)
                # print("int.maxmem (pid:" + str(p.pid) + ") " + str(local_maxmem))
            time.sleep(sleeptime)
            sleeptime = min(sleeptime + 5, 60)

        self.peak_memory = max(self.peak_memory, local_maxmem)

        del self.procs[p.pid]

        info = "Process " + str(p.pid) + " returned: (" + str(p.returncode) + ")."
        if not shell:
            info += " Peak memory: (Process: " + str(round(local_maxmem, 3)) + "GB;"
            info += " Pipeline: " + str(round(self.peak_memory, 3)) + "GB)\n"

        print(info + "\n")
        if p.returncode != 0:
            raise Exception("Process returned nonzero result.")
        return [p.returncode, local_maxmem]


Instruction:
Write a Python 3 function for
waiting until the lock file exists and a dynamic recovery flag is spotted.

Response:
def _wait_for_lock(self, lock_file):
        """
        Just sleep until the lock_file does not exist or a lock_file-related dynamic recovery flag is spotted

        :param str lock_file: Lock file to wait upon.
        """
        sleeptime = .5
        first_message_flag = False
        dot_count = 0
        recover_file = self._recoverfile_from_lockfile(lock_file)
        while os.path.isfile(lock_file):
            if first_message_flag is False:
                self.timestamp("Waiting for file lock: " + lock_file)
                self._set_status_flag(WAIT_FLAG)
                first_message_flag = True
            else:
                sys.stdout.write(".")
                dot_count = dot_count + 1
                if dot_count % 60 == 0:
                    print("")  # linefeed
            # prevents the issue of pypier waiting for the lock file to be gone infinitely
            # in case the recovery flag is sticked by other pipeline when it's interrupted
            if os.path.isfile(recover_file):
                sys.stdout.write(" Dynamic recovery flag found")
                break
            time.sleep(sleeptime)
            sleeptime = min(sleeptime + 2.5, 60)

        if first_message_flag:
            self.timestamp("File unlocked.")
            self._set_status_flag(RUN_FLAG)


Instruction:
Write a Python 3 script to
print a message time and time elapsed for the current checkpoint.

Response:
def timestamp(self, message="", checkpoint=None,
                  finished=False, raise_error=True):
        """
        Print message, time, and time elapsed, perhaps creating checkpoint.

        This prints your given message, along with the current time, and time
        elapsed since the previous timestamp() call.  If you specify a
        HEADING by beginning the message with "###", it surrounds the message
        with newlines for easier readability in the log file. If a checkpoint
        is designated, an empty file is created corresponding to the name
        given. Depending on how this manager's been configured, the value of
        the checkpoint, and whether this timestamp indicates initiation or
        completion of a group of pipeline steps, this call may stop the
        pipeline's execution.

        :param str message: Message to timestamp.
        :param str checkpoint: Name of checkpoint; this tends to be something
            that reflects the processing logic about to be or having just been
            completed. Provision of an argument to this parameter means that
            a checkpoint file will be created, facilitating arbitrary starting
            and stopping point for the pipeline as desired.
        :param bool finished: Whether this call represents the completion of a
            conceptual unit of a pipeline's processing
        :param raise_error: Whether to raise exception if
            checkpoint or current state indicates that a halt should occur.
        """

        # Halt if the manager's state has been set such that this call
        # should halt the pipeline.
        if self.halt_on_next:
            self.halt(checkpoint, finished, raise_error=raise_error)

        # Determine action to take with respect to halting if needed.
        if checkpoint:
            if finished:
                # Write the file.
                self._checkpoint(checkpoint)
                self.prev_checkpoint = checkpoint
                self.curr_checkpoint = None
            else:
                self.prev_checkpoint = self.curr_checkpoint
                self.curr_checkpoint = checkpoint
                self._checkpoint(self.prev_checkpoint)
            # Handle the two halting conditions.
            if (finished and checkpoint == self.stop_after) or (not finished and checkpoint == self.stop_before):
                self.halt(checkpoint, finished, raise_error=raise_error)
            # Determine if we've started executing.
            elif checkpoint == self.start_point:
                self._active = True
            # If this is a prospective checkpoint, set the current checkpoint
            # accordingly and whether we should halt the pipeline on the
            # next timestamp call.
            if not finished and checkpoint == self.stop_after:
                self.halt_on_next = True

        elapsed = self.time_elapsed(self.last_timestamp)
        t = time.strftime("%m-%d %H:%M:%S")
        if checkpoint is None:
            msg = "{m} ({t}) elapsed: {delta_t} _TIME_".\
                    format(m=message, t=t, delta_t=elapsed)
        else:
            msg = "{m} ({t}) ({status} {stage}) elapsed: {delta_t} _TIME_".\
                format(m=message, t=t,
                       status="finished" if finished else "starting",
                       stage=checkpoint, delta_t=elapsed)
        if re.match("^###", message):
            msg = "\n{}\n".format(msg)
        print(msg)
        self.last_timestamp = time.time()


Instruction:
Can you implement a function in Python 3 that
writes a string to self. pipeline_profile_file.

Response:
def _report_profile(self, command, lock_name, elapsed_time, memory):
        """
        Writes a string to self.pipeline_profile_file.
        """
        message_raw = str(command) + "\t " + \
            str(lock_name) + "\t" + \
            str(datetime.timedelta(seconds = round(elapsed_time, 2))) + "\t " + \
            str(memory)

        with open(self.pipeline_profile_file, "a") as myfile:
            myfile.write(message_raw + "\n")


Instruction:
Given the following Python 3 function, write the documentation
def report_result(self, key, value, annotation=None):
        # Default annotation is current pipeline name.
        annotation = str(annotation or self.name)

        # In case the value is passed with trailing whitespace.
        value = str(value).strip()

        # keep the value in memory:
        self.stats_dict[key] = value
        message_raw = "{key}\t{value}\t{annotation}".format(
            key=key, value=value, annotation=annotation)

        message_markdown = "\n> `{key}`\t{value}\t{annotation}\t_RES_".format(
            key=key, value=value, annotation=annotation)

        print(message_markdown)

        # Just to be extra careful, let's lock the file while we we write
        # in case multiple pipelines write to the same file.
        self._safe_write_to_file(self.pipeline_stats_file, message_raw)

Response: Writes a string to self. pipeline_stats_file. A message is printed to self. stats_dict.


Instruction:
Can you write a function in Python 3 where it
writes a string to self. pipeline_objects_file. Used to report figures and others.

Response:
def report_object(self, key, filename, anchor_text=None, anchor_image=None,
       annotation=None):
        """
        Writes a string to self.pipeline_objects_file. Used to report figures and others.

        :param str key: name (key) of the object
        :param str filename: relative path to the file (relative to parent output dir)
        :param str anchor_text: text used as the link anchor test or caption to
            refer to the object. If not provided, defaults to the key.
        :param str anchor_image: a path to an HTML-displayable image thumbnail (so,
            .png or .jpg, for example). If a path, the path should be relative
            to the parent output dir.
        :param str annotation: By default, the figures will be annotated with the
            pipeline name, so you can tell which pipeline records which figures.
            If you want, you can change this.
        """

        # Default annotation is current pipeline name.
        annotation = str(annotation or self.name)

        # In case the value is passed with trailing whitespace.
        filename = str(filename).strip()
        if anchor_text:
            anchor_text = str(anchor_text).strip()
        else:
            anchor_text = str(key).strip()

        # better to use a relative path in this file
        # convert any absolute paths into relative paths
        relative_filename = os.path.relpath(filename, self.outfolder) \
                if os.path.isabs(filename) else filename

        if anchor_image:
            relative_anchor_image = os.path.relpath(anchor_image, self.outfolder) \
                if os.path.isabs(anchor_image) else anchor_image
        else:
            relative_anchor_image = "None"

        message_raw = "{key}\t{filename}\t{anchor_text}\t{anchor_image}\t{annotation}".format(
            key=key, filename=relative_filename, anchor_text=anchor_text,
            anchor_image=relative_anchor_image, annotation=annotation)

        message_markdown = "> `{key}`\t{filename}\t{anchor_text}\t{anchor_image}\t{annotation}\t_OBJ_".format(
            key=key, filename=relative_filename, anchor_text=anchor_text,
            anchor_image=relative_anchor_image,annotation=annotation)

        print(message_markdown)

        self._safe_write_to_file(self.pipeline_objects_file, message_raw)


Instruction:
Here you have a function in Python 3, explain what it does
def _safe_write_to_file(self, file, message):
        target = file
        lock_name = make_lock_name(target, self.outfolder)
        lock_file = self._make_lock_path(lock_name)

        while True:
            if os.path.isfile(lock_file):
                self._wait_for_lock(lock_file)
            else:
                try:
                    self.locks.append(lock_file)
                    self._create_file_racefree(lock_file)
                except OSError as e:
                    if e.errno == errno.EEXIST:
                        print ("Lock file created after test! Looping again.")
                        continue  # Go back to start

                # Proceed with file writing
                with open(file, "a") as myfile:
                    myfile.write(message + "\n")

                os.remove(lock_file)
                self.locks.remove(lock_file)

                # If you make it to the end of the while loop, you're done
                break

Response: Writes a string to a file safely.


Instruction:
Implement a function in Python 3 to
write a command to both stdout and to the commands log file

Response:
def _report_command(self, cmd, procs=None):
        """
        Writes a command to both stdout and to the commands log file
        (self.pipeline_commands_file).

        :param str cmd: command to report
        :param str | list[str] procs: process numbers for processes in the command
        """
        if isinstance(procs, list):
            procs = ",".join(map(str,procs))
        if procs:
            line = "\n> `{cmd}` ({procs})\n".format(cmd=str(cmd), procs=procs)
        else:
            line = "\n> `{cmd}`\n".format(cmd=str(cmd))
        print(line)

        with open(self.pipeline_commands_file, "a") as myfile:
            myfile.write(line + "\n\n")


Instruction:
Given the following Python 3 function, write the documentation
def _create_file_racefree(self, file):
        write_lock_flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
        os.open(file, write_lock_flags)

Response: Creates a file and throws an OSError if it already exists.


Instruction:
Can you write a function in Python 3 where it
creates path to the lock file with given name as base.

Response:
def _make_lock_path(self, lock_name_base):
        """
        Create path to lock file with given name as base.

        :param str lock_name_base: Lock file name, designed to not be prefixed
            with the lock file designation, but that's permitted.
        :return str: Path to the lock file.
        """

        # For lock prefix validation, separate file name from other path
        # components, as we care about the name prefix not path prefix.
        base, name = os.path.split(lock_name_base)

        lock_name = self._ensure_lock_prefix(name)
        if base:
            lock_name = os.path.join(base, lock_name)
        return pipeline_filepath(self, filename=lock_name)


Instruction:
Given the following Python 3 function, write the documentation
def _recoverfile_from_lockfile(self, lockfile):
        # Require that the lock file path be absolute, or at least relative
        # and starting with the pipeline output folder.
        if not (os.path.isabs(lockfile) or lockfile.startswith(self.outfolder)):
            lockfile = self._make_lock_path(lockfile)
        return lockfile.replace(LOCK_PREFIX, "recover." + LOCK_PREFIX)

Response: Create path to recovery file with given name as base.


Instruction:
Explain what the following Python 3 code does
def make_sure_path_exists(self, path):
        try:
            os.makedirs(path)
        except OSError as exception:
            if exception.errno != errno.EEXIST:
                raise

Response: Creates all directories in a path if it does not exist.


Instruction:
Here you have a function in Python 3, explain what it does
def _refresh_stats(self):

        # regex identifies all possible stats files.
        #regex = self.outfolder +  "*_stats.tsv"
        #stats_files = glob.glob(regex)
        #stats_files.insert(self.pipeline_stats_file) # last one is the current pipeline
        #for stats_file in stats_files:

        stats_file = self.pipeline_stats_file
        if os.path.isfile(self.pipeline_stats_file):
            with open(stats_file, 'r') as stat_file:
                for line in stat_file:
                    try:
                        # Someone may have put something that's not 3 columns in the stats file
                        # if so, shame on him, but we can just ignore it.
                        key, value, annotation  = line.split('\t')
                    except ValueError:
                        print("WARNING: Each row in a stats file is expected to have 3 columns")

                    if annotation.rstrip() == self.name or annotation.rstrip() == "shared":
                        self.stats_dict[key] = value.strip()

Response: Load up the stats sheet created for this pipeline and reads those stats into memory


Instruction:
Here you have a function in Python 3, explain what it does
def get_stat(self, key):

        try:
            return self.stats_dict[key]
        except KeyError:
            self._refresh_stats()
            try:
                return self.stats_dict[key]
            except KeyError:
                print("Missing stat '{}'".format(key))
                return None

Response: Retrieves a stat that was previously reported by the current pipeline run.


Instruction:
Given the following Python 3 function, write the documentation
def _checkpoint(self, stage):

        # For null stage, short-circuit and indicate no file write.
        # This handles case in which we're timestamping prospectively and
        # previously weren't in a stage.
        if stage is None:
            return False

        try:
            is_checkpoint = stage.checkpoint
        except AttributeError:
            # Maybe we have a raw function, not a stage.
            if hasattr(stage, "__call__"):
                stage = stage.__name__
            else:
                # Maybe we have a stage name not a Stage.
                # In that case, we can proceed as-is, with downstream
                # processing handling Stage vs. stage name disambiguation.
                # Here, though, warn about inputs that appear filename/path-like.
                # We can't rely on raw text being a filepath or filename,
                # because that would ruin the ability to pass stage name rather
                # than actual stage. We can issue a warning message based on the
                # improbability of a stage name containing the '.' that would
                # be expected to characterize the extension of a file name/path.
                base, ext = os.path.splitext(stage)
                if ext and "." not in base:
                    print("WARNING: '{}' looks like it may be the name or path of "
                          "a file; for such a checkpoint, use touch_checkpoint.".
                          format(stage))
        else:
            if not is_checkpoint:
                print("Not a checkpoint: {}".format(stage))
                return False
            stage = stage.name

        print("Checkpointing: '{}'".format(stage))
        if os.path.isabs(stage):
            check_fpath = stage
        else:
            check_fpath = checkpoint_filepath(stage, pm=self)
        return self._touch_checkpoint(check_fpath)

Response: This is the hook that is used to stop processing of a logical processing phase. This is the hook that is used to stop processing of a logical processing phase.


Instruction:
Make a summary of the following Python 3 code
def fail_pipeline(self, e, dynamic_recover=False):
        # Take care of any active running subprocess
        sys.stdout.flush()
        self._terminate_running_subprocesses()

        if dynamic_recover:
            # job was terminated, not failed due to a bad process.
            # flag this run as recoverable.
            if len(self.locks) < 1:
                # If there is no process locked, then recovery will be automatic.
                print("No locked process. Dynamic recovery will be automatic.")
            # make a copy of self.locks to iterate over since we'll be clearing them as we go
            # set a recovery flag for each lock.
            for lock_file in self.locks[:]:
                recover_file = self._recoverfile_from_lockfile(lock_file)
                print("Setting dynamic recover file: {}".format(recover_file))
                self._create_file(recover_file)
                self.locks.remove(lock_file)

        # Produce cleanup script
        self._cleanup(dry_run=True)

        # Finally, set the status to failed and close out with a timestamp
        if not self._failed:  # and not self._completed:
            self.timestamp("### Pipeline failed at: ")
            total_time = datetime.timedelta(seconds=self.time_elapsed(self.starttime))
            print("Total time: " + str(total_time))
            self._set_status_flag(FAIL_FLAG)

        raise e

Response: This function will stop the pipeline and set the status flag to failed and skips normal success completion procedure.


Instruction:
Make a summary of the following Python 3 code
def halt(self, checkpoint=None, finished=False, raise_error=True):
        self.stop_pipeline(PAUSE_FLAG)
        self._active = False
        if raise_error:
            raise PipelineHalt(checkpoint, finished)

Response:
Stop the pipeline before completion point.

        :param str checkpoint: Name of stage just reached or just completed.
        :param bool finished: Whether the indicated stage was just finished
            (True), or just reached (False)
        :param bool raise_error: Whether to raise an exception to truly
            halt execution.


Instruction:
Here you have a function in Python 3, explain what it does
def stop_pipeline(self, status=COMPLETE_FLAG):
        self._set_status_flag(status)
        self._cleanup()
        self.report_result("Time", str(datetime.timedelta(seconds=self.time_elapsed(self.starttime))))
        self.report_result("Success", time.strftime("%m-%d-%H:%M:%S"))
        print("\n##### [Epilogue:]")
        print("* " + "Total elapsed time".rjust(20) + ":  " + str(datetime.timedelta(seconds=self.time_elapsed(self.starttime))))
        # print("Peak memory used: " + str(memory_usage()["peak"]) + "kb")
        print("* " + "Peak memory used".rjust(20) + ":  " + str(round(self.peak_memory, 2)) + " GB")
        if self.halted:
            return
        self.timestamp("* Pipeline completed at: ".rjust(20))

Response: Terminate the healthy pipeline and report the status of the pipeline.


Instruction:
How would you implement a function in Python 3 that
functions for handling both SIGTERM and SIGINT

Response:
def _generic_signal_handler(self, signal_type):
        """
        Function for handling both SIGTERM and SIGINT
        """
        print("</pre>")
        message = "Got " + signal_type + ". Failing gracefully..."
        self.timestamp(message)
        self.fail_pipeline(KeyboardInterrupt(signal_type), dynamic_recover=True)
        sys.exit(1)


Instruction:
Here you have a function in Python 3, explain what it does
def _exit_handler(self):

        # TODO: consider handling sys.stderr/sys.stdout exceptions related to
        # TODO (cont.): order of interpreter vs. subprocess shutdown signal receipt.
        # TODO (cont.): see https://bugs.python.org/issue11380

        # Make the cleanup file executable if it exists
        if os.path.isfile(self.cleanup_file):
            # Make the cleanup file self destruct.
            with open(self.cleanup_file, "a") as myfile:
                myfile.write("rm " + self.cleanup_file + "\n")
            os.chmod(self.cleanup_file, 0o755)

        # If the pipeline hasn't completed successfully, or already been marked
        # as failed, then mark it as failed now.

        if not self._has_exit_status:
            print("Pipeline status: {}".format(self.status))
            self.fail_pipeline(Exception("Pipeline failure. See details above."))

        if self.tee:
            self.tee.kill()

Response: This function is called when the script is completing.


Instruction:
Explain what the following Python 3 code does
def _kill_child_process(self, child_pid, proc_name=None):

        # When we kill process, it turns into a zombie, and we have to reap it.
        # So we can't just kill it and then let it go; we call wait

        def pskill(proc_pid, sig=signal.SIGINT):
            parent_process = psutil.Process(proc_pid)
            for child_proc in parent_process.children(recursive=True):
                child_proc.send_signal(sig)
            parent_process.send_signal(sig)

        if child_pid is None:
            return

        if proc_name:
            proc_string = " ({proc_name})".format(proc_name=proc_name)

        # First a gentle kill
        sys.stdout.flush()
        still_running = self._attend_process(psutil.Process(child_pid), 0)
        sleeptime = .25
        time_waiting = 0

        while still_running and time_waiting < 3:
            try:
                if time_waiting > 2:
                    pskill(child_pid, signal.SIGKILL)
                    # print("pskill("+str(child_pid)+", signal.SIGKILL)")
                elif time_waiting > 1:
                    pskill(child_pid, signal.SIGTERM)
                    # print("pskill("+str(child_pid)+", signal.SIGTERM)")
                else:
                    pskill(child_pid, signal.SIGINT)
                    # print("pskill("+str(child_pid)+", signal.SIGINT)")

            except OSError:
                # This would happen if the child process ended between the check
                # and the next kill step
                still_running = False
                time_waiting = time_waiting + sleeptime

            # Now see if it's still running
            time_waiting = time_waiting + sleeptime
            if not self._attend_process(psutil.Process(child_pid), sleeptime):
                still_running = False

        if still_running:
            # still running!?
            print("Child process {child_pid}{proc_string} never responded"
                "I just can't take it anymore. I don't know what to do...".format(child_pid=child_pid,
                    proc_string=proc_string))
        else:
            if time_waiting > 0:
                note = "terminated after {time} sec".format(time=int(time_waiting))
            else:
                note = "was already terminated"

            msg = "Child process {child_pid}{proc_string} {note}.".format(
                child_pid=child_pid, proc_string=proc_string, note=note)
            print(msg)

Response: Kills a child process.


Instruction:
Create a Python 3 function for
adding files (or regexs) to a cleanup list, to delete when this pipeline completes successfully. When making a call with run that produces intermediate files that should be deleted after the pipeline completes, you flag these files for deletion with this command. Files added with clean_add will only be deleted upon success of the pipeline. :param str regex: A unix-style regular expression that matches files to delete (can also be a file name). :param bool conditional: True means the files will only be deleted if no other pipelines are currently running; otherwise they are added to a manual cleanup script called {pipeline_name}_cleanup.sh :param bool manual: True means the files will just be added to a manual cleanup script.

Response:
def clean_add(self, regex, conditional=False, manual=False):
        """
        Add files (or regexs) to a cleanup list, to delete when this pipeline completes successfully.
        When making a call with run that produces intermediate files that should be
        deleted after the pipeline completes, you flag these files for deletion with this command.
        Files added with clean_add will only be deleted upon success of the pipeline.

        :param str regex:  A unix-style regular expression that matches files to delete
            (can also be a file name).
        :param bool conditional: True means the files will only be deleted if no other
            pipelines are currently running; otherwise they are added to a manual cleanup script
            called {pipeline_name}_cleanup.sh
        :param bool manual: True means the files will just be added to a manual cleanup script.
        """
        if self.dirty:
            # Override the user-provided option and force manual cleanup.
            manual = True

        if not self.clean_initialized:
            # Make cleanup files relative to the cleanup script in case the result folder moves.
            with open(self.cleanup_file, "a") as myfile:
                clean_init = 'DIR="$(cd -P -- "$(dirname -- "$0")" && pwd -P)"'
                myfile.write(clean_init + "\n")
                myfile.write("cd ${DIR}\n")
                self.clean_initialized = True

        if manual:
            try:
                filenames = glob.glob(regex)
                for filename in filenames:
                    with open(self.cleanup_file, "a") as myfile:
                        relative_filename = os.path.relpath(filename, self.outfolder) \
                            if os.path.isabs(filename) else filename
                        if os.path.isfile(relative_filename):
                            myfile.write("rm " + relative_filename + "\n")
                        elif os.path.isdir(relative_filename):
                            # first, add all filenames in the directory
                            myfile.write("rm " + relative_filename + "/*\n")
                            # and the directory itself
                            myfile.write("rmdir " + relative_filename + "\n")
            except:
                pass
        elif conditional:
            self.cleanup_list_conditional.append(regex)
        else:
            self.cleanup_list.append(regex)
            # TODO: what's the "absolute" list?
            # Remove it from the conditional list if added to the absolute list
            while regex in self.cleanup_list_conditional:
                self.cleanup_list_conditional.remove(regex)


Instruction:
Given the following Python 3 function, write the documentation
def _cleanup(self, dry_run=False):

        if dry_run:
            # Move all unconditional cleans into the conditional list
            if len(self.cleanup_list) > 0:
                combined_list = self.cleanup_list_conditional + self.cleanup_list
                self.cleanup_list_conditional = combined_list
                self.cleanup_list = []

        if len(self.cleanup_list) > 0:
            print("\nCleaning up flagged intermediate files. . .")
            for expr in self.cleanup_list:
                print("\nRemoving glob: " + expr)
                try:
                    # Expand regular expression
                    files = glob.glob(expr)
                    # Remove entry from cleanup list
                    while files in self.cleanup_list:
                        self.cleanup_list.remove(files)
                    # and delete the files
                    for file in files:
                        if os.path.isfile(file):
                            print("`rm " + file + "`")
                            os.remove(os.path.join(file))
                        elif os.path.isdir(file):
                            print("`rmdir " + file + "`")
                            os.rmdir(os.path.join(file))
                except:
                    pass

        if len(self.cleanup_list_conditional) > 0:
            run_flag = flag_name(RUN_FLAG)
            flag_files = [fn for fn in glob.glob(self.outfolder + flag_name("*"))
                          if COMPLETE_FLAG not in os.path.basename(fn)
                          and not "{}_{}".format(self.name, run_flag) == os.path.basename(fn)]
            if len(flag_files) == 0 and not dry_run:
                print("\nCleaning up conditional list. . .")
                for expr in self.cleanup_list_conditional:
                    print("\nRemoving glob: " + expr)
                    try:
                        files = glob.glob(expr)
                        while files in self.cleanup_list_conditional:
                            self.cleanup_list_conditional.remove(files)
                        for file in files:
                            if os.path.isfile(file):
                                print("`rm " + file + "`")
                                os.remove(os.path.join(file))
                            elif os.path.isdir(file):
                                print("`rmdir " + file + "`")
                                os.rmdir(os.path.join(file))
                    except:
                        pass
            else:
                print("\nConditional flag found: " + str([os.path.basename(i) for i in flag_files]))
                print("\nThese conditional files were left in place:\n\n- " + "\n- ".join(self.cleanup_list_conditional))
                # Produce a cleanup script.
                no_cleanup_script = []
                for cleandir in self.cleanup_list_conditional:
                    try:
                        items_to_clean = glob.glob(cleandir)
                        for clean_item in items_to_clean:
                            with open(self.cleanup_file, "a") as clean_script:
                                if os.path.isfile(clean_item):
                                    clean_script.write("rm " + clean_item + "\n")
                                elif os.path.isdir(clean_item):
                                    clean_script.write("rmdir " + clean_item + "\n")
                    except Exception as e:
                        no_cleanup_script.append(cleandir)
                if no_cleanup_script:
                    print('\n\nCould not produce cleanup script for item(s):\n\n- ' + '\n- '.join(no_cleanup_script))

Response: Cleans up the intermediate files and deletes the files that are not in the output folder.


Instruction:
Can you generate the documentation for the following Python 3 function
def _memory_usage(self, pid='self', category="hwm", container=None):
        if container:
            # TODO: Put some debug output here with switch to Logger
            # since this is relatively untested.
            cmd = "docker stats " + container + " --format '{{.MemUsage}}' --no-stream"
            mem_use_str = subprocess.check_output(cmd, shell=True)
            mem_use = mem_use_str.split("/")[0].split()

            mem_num = re.findall('[\d\.]+', mem_use_str.split("/")[0])[0]
            mem_scale = re.findall('[A-Za-z]+', mem_use_str.split("/")[0])[0]

            #print(mem_use_str, mem_num, mem_scale)
            mem_num = float(mem_num)
            if mem_scale == "GiB":
                return mem_num * 1e6
            elif mem_scale == "MiB":
                return mem_num * 1e3
            elif mem_scale == "KiB":
                return mem_num
            else:
                # What type is this?
                return 0

        # Thanks Martin Geisler:
        status = None
        result = {'peak': 0, 'rss': 0, 'hwm': 0}

        try:
            # This will only work on systems with a /proc file system
            # (like Linux).
            # status = open('/proc/self/status')
            proc_spot = '/proc/%s/status' % pid
            status = open(proc_spot)
            for line in status:
                parts = line.split()
                key = parts[0][2:-1].lower()
                if key in result:
                    result[key] = int(parts[1])
        except:
            return 0

        finally:
            if status is not None:
                status.close()
        # print(result[category])
        return result[category]

Response: Return the memory usage of the process in kilobytes.


Instruction:
Can you generate the documentation for the following Python 3 function
def _triage_error(self, e, nofail):
        if not nofail:
            self.fail_pipeline(e)
        elif self._failed:
            print("This is a nofail process, but the pipeline was terminated for other reasons, so we fail.")
            raise e
        else:
            print(e)
            print("ERROR: Subprocess returned nonzero result, but pipeline is continuing because nofail=True")

Response: Print a message and decide what to do with the error.


Instruction:
Write a Python 3 function for
reading the requirements file for given requirements group.

Response:
def read_reqs_file(reqs_name):
    """ Read requirements file for given requirements group. """
    path_reqs_file = os.path.join(
            "requirements", "reqs-{}.txt".format(reqs_name))
    with open(path_reqs_file, 'r') as reqs_file:
        return [pkg.rstrip() for pkg in reqs_file.readlines()
                if not pkg.startswith("#")]


Instruction:
Explain what the following Python 3 code does
def _is_unordered(collection):
    if not isinstance(collection, Iterable):
        raise TypeError("Non-iterable alleged collection: {}".
                        format(type(collection)))
    return isinstance(collection, set) or \
           (isinstance(collection, dict) and
            not isinstance(collection, OrderedDict))

Response: Checks whether the given object is an unordered sequence of objects.


Instruction:
Can you generate the documentation for the following Python 3 function
def _parse_stage_spec(stage_spec):

    # The logic used here, a message to a user about how to specify Stage.
    req_msg = "Stage specification must be either a {0} itself, a " \
              "(<name>, {0}) pair, or a callable with a __name__ attribute " \
              "(e.g., a non-anonymous function)".format(Stage.__name__)

    # Simplest case is stage itself.
    if isinstance(stage_spec, Stage):
        return stage_spec.name, stage_spec

    # Handle alternate forms of specification.
    try:
        # Unpack pair of name and stage, requiring name first.
        name, stage = stage_spec
    except (TypeError, ValueError):
        # Normally, this sort of unpacking issue create a ValueError. Here,
        # though, we also need to catch TypeError since that's what arises
        # if an attempt is made to unpack a single function.
        # Attempt to parse stage_spec as a single named callable.
        try:
            name = stage_spec.__name__
        except AttributeError:
            raise TypeError(req_msg)
        else:
            # Control flow here indicates an anonymous function that was not
            # paired with a name. Prohibit that.
            if name == (lambda: None).__name__:
                raise TypeError(req_msg)
        stage = stage_spec

    # Ensure that the stage is callable.
    if not hasattr(stage, "__call__"):
        raise TypeError(req_msg)

    return name, Stage(stage, name=name)

Response: Parses a stage specification and returns a name and Stage instance.


Instruction:
Write a Python 3 function for
touching checkpoint file for given stage and provide timestamp message.

Response:
def checkpoint(self, stage, msg=""):
        """
        Touch checkpoint file for given stage and provide timestamp message.

        :param pypiper.Stage stage: Stage for which to mark checkpoint
        :param str msg: Message to embed in timestamp.
        :return bool: Whether a checkpoint file was written.
        """
        # Canonical usage model for Pipeline checkpointing through
        # implementations of this class is by automatically creating a
        # checkpoint when a conceptual unit or group of operations of a
        # pipeline completes, so fix the 'finished' parameter to the manager's
        # timestamp method to be True.
        return self.manager.timestamp(
                message=msg, checkpoint=stage.checkpoint_name, finished=True)


Instruction:
How would you explain what the following Python 3 function does
def completed_stage(self, stage):
        check_path = checkpoint_filepath(stage, self.manager)
        return os.path.exists(check_path)

Response: Determines whether this pipeline has completed the given stage.


Instruction:
Can you generate the documentation for the following Python 3 function
def list_flags(self, only_name=False):
        paths = glob.glob(os.path.join(self.outfolder, flag_name("*")))
        if only_name:
            return [os.path.split(p)[1] for p in paths]
        else:
            return paths

Response: List the flag files associated with this pipeline.


Instruction:
Make a summary of the following Python 3 code
def run(self, start_point=None, stop_before=None, stop_after=None):

        # Start the run with a clean slate of Stage status/label tracking.
        self._reset()

        # TODO: validate starting point against checkpoint flags for
        # TODO (cont.): earlier stages if the pipeline defines its stages as a
        # TODO (cont.): sequence (i.e., probably prohibit start point with
        # TODO (cont): nonexistent earlier checkpoint flag(s).)

        if stop_before and stop_after:
            raise IllegalPipelineExecutionError(
                    "Cannot specify both inclusive and exclusive stops.")

        if stop_before:
            stop = stop_before
            inclusive_stop = False
        elif stop_after:
            stop = stop_after
            inclusive_stop = True
        else:
            stop = None
            inclusive_stop = None

        # Ensure that a stage name--if specified--is supported.
        for s in [start_point, stop]:
            if s is None:
                continue
            name = parse_stage_name(s)
            if name not in self.stage_names:
                raise UnknownPipelineStageError(name, self)

        # Permit order-agnostic pipelines, but warn.
        if self._unordered and (start_point or stop_before or stop_after):
            print("WARNING: Starting and stopping points are nonsense for "
                  "pipeline with unordered stages.")

        # TODO: consider context manager based on start/stop points.

        # Determine where to start (but perhaps skip further based on
        # checkpoint completions.)
        start_index = self._start_index(start_point)
        stop_index = self._stop_index(stop, inclusive=inclusive_stop)
        assert stop_index <= len(self._stages)
        if start_index >= stop_index:
            raise IllegalPipelineExecutionError(
                    "Cannot start pipeline at or after stopping point")

        # TODO: consider storing just stage name rather than entire stage.
        # TODO (cont.): the bad case for whole-Stage is if associated data
        # TODO (cont.): (i.e., one or more args) are large.
        self.skipped.extend(self._stages[:start_index])

        # TODO: support both courses of action for non-continuous checkpoints.
        # TODO (cont.): That is, what if there's a stage with a checkpoint
        # TODO (cont.): file downstream of one without it? Naively, we'll
        # TODO (cont.): skip it, but we may want to re-run.
        skip_mode = True

        for stage in self._stages[start_index:stop_index]:

            # TODO: Note that there's no way to tell whether a non-checkpointed
            # TODO (cont.) Stage has been completed, and thus this seek
            # TODO (cont.) operation will find the first Stage, starting
            # TODO (cont.) the specified start point, either uncheckpointed or
            # TODO (cont.) for which the checkpoint file does not exist.
            # Look for checkpoint file.
            if skip_mode and self.completed_stage(stage):
                print("Skipping completed checkpoint stage: {}".format(stage))
                self.skipped.append(stage)
                continue

            # Once we've found where to being execution, ignore checkpoint
            # flags downstream if they exist since there may be dependence
            # between results from different stages.
            skip_mode = False

            print("Running stage: {}".format(stage))

            stage.run()
            self.executed.append(stage)
            self.checkpoint(stage)

        # Add any unused stages to the collection of skips.
        self.skipped.extend(self._stages[stop_index:])

        # Where we stopped determines the shutdown mode.
        if stop_index == len(self._stages):
            self.wrapup()
        else:
            self.halt(raise_error=False)

Response:
Run the pipeline, optionally specifying start and/or stop points.

        :param str start_point: Name of stage at which to begin execution.
        :param str stop_before: Name of stage at which to cease execution;
            exclusive, i.e. this stage is not run
        :param str stop_after: Name of stage at which to cease execution;
            inclusive, i.e. this stage is the last one run
        :raise IllegalPipelineExecutionError: If both inclusive (stop_after)
            and exclusive (stop_before) halting points are provided, or if that
            start stage is the same as or after the stop stage, raise an
            IllegalPipelineExecutionError.


Instruction:
Explain what the following Python 3 code does
def _start_index(self, start=None):
        if start is None:
            return 0
        start_stage = translate_stage_name(start)
        internal_names = [translate_stage_name(s.name) for s in self._stages]
        try:
            return internal_names.index(start_stage)
        except ValueError:
            raise UnknownPipelineStageError(start, self)

Response: Return the index of the first stage to run.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _stop_index(self, stop_point, inclusive):
        if not stop_point:
            # Null case, no stopping point
            return len(self._stages)
        stop_name = parse_stage_name(stop_point)
        try:
            stop_index = self.stage_names.index(stop_name)
        except ValueError:
            raise UnknownPipelineStageError(stop_name, self)
        return stop_index + 1 if inclusive else stop_index

Response: Determines the index of the next stop point in the sequence of Pipeline s stages that indicates that the stop point is the one that is not in the final stage.