Instruction:
Write a Python 3 function for
applying a n - dimensional clustering filter to the data.

Response:
def filter_clustering(self, analytes, filt=False, normalise=True,
                          method='kmeans', include_time=False, samples=None,
                          sort=True, subset=None, level='sample', min_data=10, **kwargs):
        """
        Applies an n - dimensional clustering filter to the data.

        Parameters
        ----------
        analytes : str
            The analyte(s) that the filter applies to.
        filt : bool
            Whether or not to apply existing filters to the data before
            calculating this filter.
        normalise : bool
            Whether or not to normalise the data to zero mean and unit
            variance. Reccomended if clustering based on more than 1 analyte.
            Uses `sklearn.preprocessing.scale`.
        method : str
            Which clustering algorithm to use:

            * 'meanshift': The `sklearn.cluster.MeanShift` algorithm.
              Automatically determines number of clusters
              in data based on the `bandwidth` of expected
              variation.
            * 'kmeans': The `sklearn.cluster.KMeans` algorithm. Determines
              the characteristics of a known number of clusters
              within the data. Must provide `n_clusters` to specify
              the expected number of clusters.
        level : str
            Whether to conduct the clustering analysis at the 'sample' or
            'population' level.
        include_time : bool
            Whether or not to include the Time variable in the clustering
            analysis. Useful if you're looking for spatially continuous
            clusters in your data, i.e. this will identify each spot in your
            analysis as an individual cluster.
        samples : optional, array_like or None
            Which samples to apply this filter to. If None, applies to all
            samples.
        sort : bool
            Whether or not you want the cluster labels to
            be sorted by the mean magnitude of the signals
            they are based on (0 = lowest)
        min_data : int
            The minimum number of data points that should be considered by
            the filter. Default = 10.
        **kwargs
            Parameters passed to the clustering algorithm specified by
            `method`.
        Meanshift Parameters
            bandwidth : str or float
                The bandwith (float) or bandwidth method ('scott' or 'silverman')
                used to estimate the data bandwidth.
            bin_seeding : bool
                Modifies the behaviour of the meanshift algorithm. Refer to
                sklearn.cluster.meanshift documentation.
        K-Means Parameters
            n_clusters : int
                The number of clusters expected in the data.

        Returns
        -------
        None
        """
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        if isinstance(analytes, str):
            analytes = [analytes]

        self.minimal_analytes.update(analytes)

        if level == 'sample':
            with self.pbar.set(total=len(samples), desc='Clustering Filter') as prog:
                for s in samples:
                    self.data[s].filter_clustering(analytes=analytes, filt=filt,
                                                normalise=normalise,
                                                method=method,
                                                include_time=include_time,
                                                min_data=min_data,
                                                sort=sort,
                                                **kwargs)
                    prog.update()

        if level == 'population':
            if isinstance(sort, bool):
                sort_by = 0
            else:
                sort_by = sort

            name = '_'.join(analytes) + '_{}'.format(method)

            self.fit_classifier(name=name, analytes=analytes, method=method,
                                subset=subset, filt=filt, sort_by=sort_by, **kwargs)

            self.apply_classifier(name=name, subset=subset)


Instruction:
Can you write a function in Python 3 where it
fits a clustering classifier on the data.

Response:
def fit_classifier(self, name, analytes, method, samples=None,
                       subset=None, filt=True, sort_by=0, **kwargs):
        """
        Create a clustering classifier based on all samples, or a subset.

        Parameters
        ----------
        name : str
            The name of the classifier.
        analytes : str or iterable
            Which analytes the clustering algorithm should consider.
        method : str
            Which clustering algorithm to use. Can be:

            'meanshift'
                The `sklearn.cluster.MeanShift` algorithm.
                Automatically determines number of clusters
                in data based on the `bandwidth` of expected
                variation.
            'kmeans'
                The `sklearn.cluster.KMeans` algorithm. Determines
                the characteristics of a known number of clusters
                within the data. Must provide `n_clusters` to specify
                the expected number of clusters.
        samples : iterable
            list of samples to consider. Overrides 'subset'.
        subset : str
            The subset of samples used to fit the classifier. Ignored if
            'samples' is specified.
        sort_by : int
            Which analyte the resulting clusters should be sorted
            by - defaults to 0, which is the first analyte.
        **kwargs :
            method-specific keyword parameters - see below.
        Meanshift Parameters
            bandwidth : str or float
                The bandwith (float) or bandwidth method ('scott' or 'silverman')
                used to estimate the data bandwidth.
            bin_seeding : bool
                Modifies the behaviour of the meanshift algorithm. Refer to
                sklearn.cluster.meanshift documentation.
        K - Means Parameters
            n_clusters : int
                The number of clusters expected in the data.

        Returns
        -------
        name : str
        """
        # isolate data
        if samples is not None:
            subset = self.make_subset(samples)

        self.get_focus(subset=subset, filt=filt)

        # create classifer
        c = classifier(analytes,
                       sort_by)
        # fit classifier
        c.fit(data=self.focus,
              method=method,
              **kwargs)

        self.classifiers[name] = c

        return name


Instruction:
Implement a function in Python 3 to
apply a clustering classifier based on all samples or a subset of samples.

Response:
def apply_classifier(self, name, samples=None, subset=None):
        """
        Apply a clustering classifier based on all samples, or a subset.

        Parameters
        ----------
        name : str
            The name of the classifier to apply.
        subset : str
            The subset of samples to apply the classifier to.
        Returns
        -------
        name : str
        """
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        c = self.classifiers[name]
        labs = c.classifier.ulabels_

        with self.pbar.set(total=len(samples), desc='Applying ' + name + ' classifier') as prog:
            for s in samples:
                d = self.data[s]
                try:
                    f = c.predict(d.focus)
                except ValueError:
                    # in case there's no data
                    f = np.array([-2] * len(d.Time))
                for l in labs:
                    ind = f == l
                    d.filt.add(name=name + '_{:.0f}'.format(l),
                            filt=ind,
                            info=name + ' ' + c.method + ' classifier',
                            params=(c.analytes, c.method))
                prog.update()

        return name


Instruction:
Can you create a Python 3 function that
applies a correlation filter to the data.

Response:
def filter_correlation(self, x_analyte, y_analyte, window=None,
                           r_threshold=0.9, p_threshold=0.05, filt=True,
                           samples=None, subset=None):
        """
        Applies a correlation filter to the data.

        Calculates a rolling correlation between every `window` points of
        two analytes, and excludes data where their Pearson's R value is
        above `r_threshold` and statistically significant.

        Data will be excluded where their absolute R value is greater than
        `r_threshold` AND the p - value associated with the correlation is
        less than `p_threshold`. i.e. only correlations that are statistically
        significant are considered.

        Parameters
        ----------
        x_analyte, y_analyte : str
            The names of the x and y analytes to correlate.
        window : int, None
            The rolling window used when calculating the correlation.
        r_threshold : float
            The correlation index above which to exclude data.
            Note: the absolute pearson R value is considered, so
            negative correlations below -`r_threshold` will also
            be excluded.
        p_threshold : float
            The significant level below which data are excluded.
        filt : bool
            Whether or not to apply existing filters to the data before
            calculating this filter.

        Returns
        -------
        None
        """
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        self.minimal_analytes.update([x_analyte, y_analyte])

        with self.pbar.set(total=len(samples), desc='Correlation Filter') as prog:
            for s in samples:
                self.data[s].filter_correlation(x_analyte, y_analyte,
                                                window=window,
                                                r_threshold=r_threshold,
                                                p_threshold=p_threshold,
                                                filt=filt)
                prog.update()


Instruction:
Here you have a function in Python 3, explain what it does
def correlation_plots(self, x_analyte, y_analyte, window=15, filt=True, recalc=False, samples=None, subset=None, outdir=None):
        if outdir is None:
            outdir = self.report_dir + '/correlations/'
        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        if subset is not None:
            samples = self._get_samples(subset)
        elif samples is None:
            samples = self.subsets['All_Analyses']
        elif isinstance(samples, str):
            samples = [samples]

        with self.pbar.set(total=len(samples), desc='Drawing Plots') as prog:
            for s in samples:
                f, a = self.data[s].correlation_plot(x_analyte=x_analyte, y_analyte=y_analyte,
                                                     window=window, filt=filt, recalc=recalc)
                f.savefig('{}/{}_{}-{}.pdf'.format(outdir, s, x_analyte, y_analyte))
                plt.close(f)
                prog.update()
        return

Response: Plots the local correlation between two analytes.


Instruction:
Implement a Python 3 function for
turning data filters on for particular analytes and samples.

Response:
def filter_on(self, filt=None, analyte=None, samples=None, subset=None, show_status=False):
        """
        Turns data filters on for particular analytes and samples.

        Parameters
        ----------
        filt : optional, str or array_like
            Name, partial name or list of names of filters. Supports
            partial matching. i.e. if 'cluster' is specified, all
            filters with 'cluster' in the name are activated.
            Defaults to all filters.
        analyte : optional, str or array_like
            Name or list of names of analytes. Defaults to all analytes.
        samples : optional, array_like or None
            Which samples to apply this filter to. If None, applies to all
            samples.

        Returns
        -------
        None
        """
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            try:
                self.data[s].filt.on(analyte, filt)
            except:
                warnings.warn("filt.on failure in sample " + s)

        if show_status:
            self.filter_status(subset=subset)
        return


Instruction:
Create a Python 3 function to
turn data filters off for particular analytes and samples.

Response:
def filter_off(self, filt=None, analyte=None, samples=None, subset=None, show_status=False):
        """
        Turns data filters off for particular analytes and samples.

        Parameters
        ----------
        filt : optional, str or array_like
            Name, partial name or list of names of filters. Supports
            partial matching. i.e. if 'cluster' is specified, all
            filters with 'cluster' in the name are activated.
            Defaults to all filters.
        analyte : optional, str or array_like
            Name or list of names of analytes. Defaults to all analytes.
        samples : optional, array_like or None
            Which samples to apply this filter to. If None, applies to all
            samples.

        Returns
        -------
        None
        """
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            try:
                self.data[s].filt.off(analyte, filt)
            except:
                warnings.warn("filt.off failure in sample " + s)

        if show_status:
            self.filter_status(subset=subset)
        return


Instruction:
Write a Python 3 function for
printing the current status of filters for the specified samples and optionally a subset.

Response:
def filter_status(self, sample=None, subset=None, stds=False):
        """
        Prints the current status of filters for specified samples.

        Parameters
        ----------
        sample : str
            Which sample to print.
        subset : str
            Specify a subset
        stds : bool
            Whether or not to include standards.
        """
        s = ''
        if sample is None and subset is None:
            if not self._has_subsets:
                s += 'Subset: All Samples\n\n'
                s += self.data[self.subsets['All_Samples'][0]].filt.__repr__()
            else:
                for n in sorted(str(sn) for sn in self._subset_names):
                    if n in self.subsets:
                        pass
                    elif int(n) in self.subsets:
                        n = int(n)
                        pass
                    s += 'Subset: ' + str(n) + '\n'
                    s += 'Samples: ' + ', '.join(self.subsets[n]) + '\n\n'
                    s += self.data[self.subsets[n][0]].filt.__repr__()
                if len(self.subsets['not_in_set']) > 0:
                    s += '\nNot in Subset:\n'
                    s += 'Samples: ' + ', '.join(self.subsets['not_in_set']) + '\n\n'
                    s += self.data[self.subsets['not_in_set'][0]].filt.__repr__()
            print(s)
            return

        elif sample is not None:
            s += 'Sample: ' + sample + '\n'
            s += self.data[sample].filt.__repr__()
            print(s)
            return

        elif subset is not None:
            if isinstance(subset, (str, int, float)):
                subset = [subset]
            for n in subset:
                s += 'Subset: ' + str(n) + '\n'
                s += 'Samples: ' + ', '.join(self.subsets[n]) + '\n\n'
                s += self.data[self.subsets[n][0]].filt.__repr__()
            print(s)
            return


Instruction:
Here you have a function in Python 3, explain what it does
def filter_clear(self, samples=None, subset=None):
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            self.data[s].filt.clear()

Response: Clears all data filters.


Instruction:
Explain what the following Python 3 code does
def filter_defragment(self, threshold, mode='include', filt=True, samples=None, subset=None):
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            f = self.data[s].filt.grab_filt(filt)
            self.data[s].filt.add(name='defrag_{:s}_{:.0f}'.format(mode, threshold),
                                  filt=filters.defrag(f, threshold, mode),
                                  info='Defrag {:s} filter with threshold {:.0f}'.format(mode, threshold),
                                  params=(threshold, mode, filt, samples, subset))

Response: Filter the data for which a specific number of points is less than threshold.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def filter_exclude_downhole(self, threshold, filt=True, samples=None, subset=None):
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            self.data[s].filter_exclude_downhole(threshold, filt)

Response: Exclude all points down -hole after the first excluded data point.


Instruction:
How would you explain what the following Python 3 function does
def filter_trim(self, start=1, end=1, filt=True, samples=None, subset=None):
        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in samples:
            self.data[s].filter_trim(start, end, filt)

Response: Removes points from the start and end of the filter regions.


Instruction:
Make a summary of the following Python 3 code
def filter_nremoved(self, filt=True, quiet=False):
        rminfo = {}
        for n in self.subsets['All_Samples']:
            s = self.data[n]
            rminfo[n] = s.filt_nremoved(filt)
        if not quiet:
            maxL = max([len(s) for s in rminfo.keys()])
            print('{string:{number}s}'.format(string='Sample ', number=maxL + 3) +
                  '{total:4s}'.format(total='tot') +
                  '{removed:4s}'.format(removed='flt') +
                  '{percent:4s}'.format(percent='%rm'))
            for k, (ntot, nfilt, pcrm) in rminfo.items():
                print('{string:{number}s}'.format(string=k, number=maxL + 3) +
                      '{total:4.0f}'.format(total=ntot) +
                      '{removed:4.0f}'.format(removed=nfilt) +
                      '{percent:4.0f}'.format(percent=pcrm))

        return rminfo

Response: Report how many data are removed by the active filters.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def optimise_signal(self, analytes, min_points=5,
                        threshold_mode='kde_first_max',
                        threshold_mult=1., x_bias=0, filt=True,
                        weights=None, mode='minimise',
                        samples=None, subset=None):
        """
        Optimise data selection based on specified analytes.

        Identifies the longest possible contiguous data region in
        the signal where the relative standard deviation (std) and
        concentration of all analytes is minimised.

        Optimisation is performed via a grid search of all possible
        contiguous data regions. For each region, the mean std and
        mean scaled analyte concentration ('amplitude') are calculated.

        The size and position of the optimal data region are identified
        using threshold std and amplitude values. Thresholds are derived
        from all calculated stds and amplitudes using the method specified
        by `threshold_mode`. For example, using the 'kde_max' method, a
        probability density function (PDF) is calculated for std and
        amplitude values, and the threshold is set as the maximum of the
        PDF. These thresholds are then used to identify the size and position
        of the longest contiguous region where the std is below the threshold,
        and the amplitude is either below the threshold.

        All possible regions of the data that have at least
        `min_points` are considered.

        For a graphical demonstration of the action of signal_optimiser,
        use `optimisation_plot`.

        Parameters
        ----------
        d : latools.D object
            An latools data object.
        analytes : str or array-like
            Which analytes to consider.
        min_points : int
            The minimum number of contiguous points to
            consider.
        threshold_mode : str
            The method used to calculate the optimisation
            thresholds. Can be 'mean', 'median', 'kde_max'
            or 'bayes_mvs', or a custom function. If a
            function, must take a 1D array, and return a
            single, real number.
        weights : array-like of length len(analytes)
            An array of numbers specifying the importance of
            each analyte considered. Larger number makes the
            analyte have a greater effect on the optimisation.
            Default is None.
        """
        if samples is not None:
            subset = self.make_subset(samples)
        samples = self._get_samples(subset)

        if isinstance(analytes, str):
            analytes = [analytes]

        self.minimal_analytes.update(analytes)

        errs = []

        with self.pbar.set(total=len(samples), desc='Optimising Data selection') as prog:
            for s in samples:
                e = self.data[s].signal_optimiser(analytes=analytes, min_points=min_points,
                                                  threshold_mode=threshold_mode, threshold_mult=threshold_mult,
                                                  x_bias=x_bias, weights=weights, filt=filt, mode=mode)
                if e != '':
                    errs.append(e)
                prog.update()

        if len(errs) > 0:
            print('\nA Few Problems:\n' + '\n'.join(errs) + '\n\n  *** Check Optimisation Plots ***')

Response: This method optimises the signal based on the specified analytes.


Instruction:
How would you code a function in Python 3 to
plot the result of signal_optimise.

Response:
def optimisation_plots(self, overlay_alpha=0.5, samples=None, subset=None, **kwargs):
        """
        Plot the result of signal_optimise.

        `signal_optimiser` must be run first, and the output
        stored in the `opt` attribute of the latools.D object.

        Parameters
        ----------
        d : latools.D object
            A latools data object.
        overlay_alpha : float
            The opacity of the threshold overlays. Between 0 and 1.
        **kwargs
            Passed to `tplot`
        """
        if samples is not None:
            subset = self.make_subset(samples)
        samples = self._get_samples(subset)

        outdir=self.report_dir + '/optimisation_plots/'
        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        with self.pbar.set(total=len(samples), desc='Drawing Plots') as prog:
            for s in samples:
                figs = self.data[s].optimisation_plot(overlay_alpha, **kwargs)

                n = 1
                for f, _ in figs:
                    if f is not None:
                        f.savefig(outdir + '/' + s + '_optim_{:.0f}.pdf'.format(n))
                        plt.close(f)
                    n += 1
                prog.update()
        return


Instruction:
Explain what the following Python 3 code does
def set_focus(self, focus_stage=None, samples=None, subset=None):
        if samples is not None:
            subset = self.make_subset(samples)

        if subset is None:
            subset = 'All_Analyses'

        samples = self._get_samples(subset)

        if focus_stage is None:
            focus_stage = self.focus_stage
        else:
            self.focus_stage = focus_stage

        for s in samples:
            self.data[s].setfocus(focus_stage)

Response: Set the focus attribute of the data file.


Instruction:
Create a Python 3 function to
collect all data from all analytes into a single array.

Response:
def get_focus(self, filt=False, samples=None, subset=None, nominal=False):
        """
        Collect all data from all samples into a single array.
        Data from standards is not collected.

        Parameters
        ----------
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        samples : str or list
            which samples to get
        subset : str or int
            which subset to get

        Returns
        -------
        None
        """

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        # t = 0
        focus = {'uTime': []}
        focus.update({a: [] for a in self.analytes})

        for sa in samples:
            s = self.data[sa]
            focus['uTime'].append(s.uTime)
            ind = s.filt.grab_filt(filt)
            for a in self.analytes:
                tmp = s.focus[a].copy()
                tmp[~ind] = np.nan
                focus[a].append(tmp)

        if nominal:
            self.focus.update({k: nominal_values(np.concatenate(v)) for k, v, in focus.items()})
        else:
            self.focus.update({k: np.concatenate(v) for k, v, in focus.items()})

        return


Instruction:
Can you generate the documentation for the following Python 3 function
def get_gradients(self, analytes=None, win=15, filt=False, samples=None, subset=None, recalc=True):
        if analytes is None:
            analytes = self.analytes

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        # check if gradients already calculated
        if all([self.data[s].grads_calced for s in samples]) and hasattr(self, 'gradients'):
            if not recalc:
                print("Using existing gradients. Set recalc=True to re-calculate.")
                return

        if not hasattr(self, 'gradients'):
            self.gradients = Bunch()

        # t = 0
        focus = {'uTime': []}
        focus.update({a: [] for a in analytes})

        with self.pbar.set(total=len(samples), desc='Calculating Gradients') as prog:
            for sa in samples:
                s = self.data[sa]
                focus['uTime'].append(s.uTime)
                ind = s.filt.grab_filt(filt)
                grads = calc_grads(s.uTime, s.focus, keys=analytes, win=win)
                for a in analytes:
                    tmp = grads[a]
                    tmp[~ind] = np.nan
                    focus[a].append(tmp)
                    s.grads = tmp
                s.grads_calced = True
                prog.update()

        self.gradients.update({k: np.concatenate(v) for k, v, in focus.items()})

        return

Response: Calculates all gradients for all analytes in the specified set of samples and returns a list of arrays.


Instruction:
How would you code a function in Python 3 to
plot a histogram of the gradients in all samples.

Response:
def gradient_histogram(self, analytes=None, win=15, filt=False, bins=None, samples=None, subset=None, recalc=True, ncol=4):
        """
        Plot a histogram of the gradients in all samples.

        Parameters
        ----------
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        bins : None or array-like
            The bins to use in the histogram
        samples : str or list
            which samples to get
        subset : str or int
            which subset to get
        recalc : bool
            Whether to re-calculate the gradients, or use existing gradients.

        Returns
        -------
        fig, ax
        """
        if analytes is None:
            analytes = [a for a in self.analytes if self.internal_standard not in a]
        if not hasattr(self, 'gradients'):
            self.gradients = Bunch()

        ncol = int(ncol)
        n = len(analytes)
        nrow = plot.calc_nrow(n, ncol)

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        self.get_gradients(analytes=analytes, win=win, filt=filt, subset=subset, recalc=recalc)

        fig, axs = plt.subplots(nrow, ncol, figsize=[3. * ncol, 2.5 * nrow])

        if not isinstance(axs, np.ndarray):
            axs = [axs]

        i = 0
        for a, ax in zip(analytes, axs.flatten()):
            d = nominal_values(self.gradients[a])
            d = d[~np.isnan(d)]

            m, u = unitpicker(d, focus_stage=self.focus_stage, denominator=self.internal_standard)

            if bins is None:
                ibins = np.linspace(*np.percentile(d * m, [1, 99]), 50)
            else:
                ibins = bins

            ax.hist(d * m, bins=ibins, color=self.cmaps[a])
            ax.axvline(0, ls='dashed', lw=1, c=(0,0,0,0.7))

            ax.set_title(a, loc='left')
            if ax.is_first_col():
                ax.set_ylabel('N')
            ax.set_xlabel(u + '/s')

            i += 1

        if i < ncol * nrow:
            for ax in axs.flatten()[i:]:
                ax.set_visible(False)

        fig.tight_layout()

        return fig, axs


Instruction:
Can you create a Python 3 function that
plots the two analytes against each other.

Response:
def crossplot(self, analytes=None, lognorm=True,
                  bins=25, filt=False, samples=None,
                  subset=None, figsize=(12, 12), save=False,
                  colourful=True, mode='hist2d', **kwargs):
        """
        Plot analytes against each other.

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to plot. Defaults to all analytes.
        lognorm : bool
            Whether or not to log normalise the colour scale
            of the 2D histogram.
        bins : int
            The number of bins in the 2D histogram.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        figsize : tuple
            Figure size (width, height) in inches.
        save : bool or str
            If True, plot is saves as 'crossplot.png', if str plot is
            saves as str.
        colourful : bool
            Whether or not the plot should be colourful :).
        mode : str
            'hist2d' (default) or 'scatter'

        Returns
        -------
        (fig, axes)
        """
        if analytes is None:
            analytes = self.analytes
        if self.focus_stage in ['ratio', 'calibrated']:
            analytes = [a for a in analytes if self.internal_standard not in a]

        # sort analytes
        try:
            analytes = sorted(analytes, key=lambda x: float(re.findall('[0-9.-]+', x)[0]))
        except IndexError:
            analytes = sorted(analytes)

        self.get_focus(filt=filt, samples=samples, subset=subset)

        fig, axes = plot.crossplot(dat=self.focus, keys=analytes, lognorm=lognorm,
                                   bins=bins, figsize=figsize, colourful=colourful,
                                   focus_stage=self.focus_stage, cmap=self.cmaps,
                                   denominator=self.internal_standard, mode=mode)

        if save or isinstance(save, str):
            if isinstance(save, str):
                fig.savefig(self.report_dir + '/' + save, dpi=200)
            else:
                fig.savefig(self.report_dir + '/crossplot.png', dpi=200)

        return fig, axes


Instruction:
Write a Python 3 function that can
plot the gradients of the analytes against each other.

Response:
def gradient_crossplot(self, analytes=None, win=15, lognorm=True,
                           bins=25, filt=False, samples=None,
                           subset=None, figsize=(12, 12), save=False,
                           colourful=True, mode='hist2d', recalc=True, **kwargs):
        """
        Plot analyte gradients against each other.

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to plot. Defaults to all analytes.
        lognorm : bool
            Whether or not to log normalise the colour scale
            of the 2D histogram.
        bins : int
            The number of bins in the 2D histogram.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        figsize : tuple
            Figure size (width, height) in inches.
        save : bool or str
            If True, plot is saves as 'crossplot.png', if str plot is
            saves as str.
        colourful : bool
            Whether or not the plot should be colourful :).
        mode : str
            'hist2d' (default) or 'scatter'
        recalc : bool
            Whether to re-calculate the gradients, or use existing gradients.

        Returns
        -------
        (fig, axes)
        """

        if analytes is None:
            analytes = self.analytes
        if self.focus_stage in ['ratio', 'calibrated']:
            analytes = [a for a in analytes if self.internal_standard not in a]

        # sort analytes
        try:
            analytes = sorted(analytes, key=lambda x: float(re.findall('[0-9.-]+', x)[0]))
        except IndexError:
            analytes = sorted(analytes)

        samples = self._get_samples(subset)

        # calculate gradients
        self.get_gradients(analytes=analytes, win=win, filt=filt, subset=subset, recalc=recalc)

        # self.get_focus(filt=filt, samples=samples, subset=subset)
        # grads = calc_grads(self.focus.uTime, self.focus, analytes, win)

        fig, axes = plot.crossplot(dat=self.gradients, keys=analytes, lognorm=lognorm,
                                   bins=bins, figsize=figsize, colourful=colourful,
                                   focus_stage=self.focus_stage, cmap=self.cmaps,
                                   denominator=self.internal_standard, mode=mode)

        if save:
            fig.savefig(self.report_dir + '/g_crossplot.png', dpi=200)

        return fig, axes


Instruction:
How would you explain what the following Python 3 function does
def histograms(self, analytes=None, bins=25, logy=False,
                   filt=False, colourful=True):
        """
        Plot histograms of analytes.

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to plot. Defaults to all analytes.
        bins : int
            The number of bins in each histogram (default = 25)
        logy : bool
            If true, y axis is a log scale.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        colourful : bool
            If True, histograms are colourful :)

        Returns
        -------
        (fig, axes)
        """
        if analytes is None:
            analytes = self.analytes
        if self.focus_stage in ['ratio', 'calibrated']:
            analytes = [a for a in analytes if self.internal_standard not in a]
        if colourful:
            cmap = self.cmaps
        else:
            cmap = None

        self.get_focus(filt=filt)
        fig, axes = plot.histograms(self.focus, keys=analytes,
                                    bins=bins, logy=logy, cmap=cmap)

        return fig, axes

Response: Plot histograms of analytes.


Instruction:
Can you tell what is the following Python 3 function doing
def filter_effect(self, analytes=None, stats=['mean', 'std'], filt=True):
        if analytes is None:
            analytes = self.analytes
        if isinstance(analytes, str):
            analytes = [analytes]

        # calculate filtered and unfiltered stats
        self.sample_stats(['La139', 'Ti49'], stats=stats, filt=False)
        suf = self.stats.copy()
        self.sample_stats(['La139', 'Ti49'], stats=stats, filt=filt)
        sf = self.stats.copy()

        # create dataframe for results
        cols = []
        for s in self.stats_calced:
            cols += ['unfiltered_{:}'.format(s), 'filtered_{:}'.format(s)]

        comp = pd.DataFrame(index=self.samples,
                            columns=pd.MultiIndex.from_arrays([cols, [None] * len(cols)]))

        # collate stats
        for k, v in suf.items():
            vf = sf[k]
            for i, a in enumerate(v['analytes']):
                for s in self.stats_calced:
                    comp.loc[k, ('unfiltered_{:}'.format(s), a)] = v[s][i,0]
                    comp.loc[k, ('filtered_{:}'.format(s), a)] = vf[s][i,0]
        comp.dropna(0, 'all', inplace=True)
        comp.dropna(1, 'all', inplace=True)
        comp.sort_index(1, inplace=True)

        # calculate filtered/unfiltered ratios
        rats = []
        for s in self.stats_calced:
            rat = comp.loc[:, 'filtered_{:}'.format(s)] / comp.loc[:, 'unfiltered_{:}'.format(s)]
            rat.columns = pd.MultiIndex.from_product([['{:}_ratio'.format(s)], rat.columns])
            rats.append(rat)

        # join it all up
        comp = comp.join(pd.concat(rats, 1))
        comp.sort_index(1, inplace=True)

        return comp.loc[:, (pd.IndexSlice[:], pd.IndexSlice[analytes])]

Response: Filter the effect of the active filter.


Instruction:
Can you write a function in Python 3 where it
plots analytes and samples as function of time.

Response:
def trace_plots(self, analytes=None, samples=None, ranges=False,
                    focus=None, outdir=None, filt=None, scale='log',
                    figsize=[10, 4], stats=False, stat='nanmean',
                    err='nanstd', subset='All_Analyses'):
        """
        Plot analytes as a function of time.

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to plot. Defaults to all analytes.
        samples: optional, array_like or str
            The sample(s) to plot. Defaults to all samples.
        ranges : bool
            Whether or not to show the signal/backgroudn regions
            identified by 'autorange'.
        focus : str
            The focus 'stage' of the analysis to plot. Can be
            'rawdata', 'despiked':, 'signal', 'background',
            'bkgsub', 'ratios' or 'calibrated'.
        outdir : str
            Path to a directory where you'd like the plots to be
            saved. Defaults to 'reports/[focus]' in your data directory.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        scale : str
            If 'log', plots the data on a log scale.
        figsize : array_like
            Array of length 2 specifying figure [width, height] in
            inches.
        stats : bool
            Whether or not to overlay the mean and standard deviations
            for each trace.
        stat, err: str
            The names of the statistic and error components to plot.
            Deafaults to 'nanmean' and 'nanstd'.

        Returns
        -------
        None
        """
        if focus is None:
            focus = self.focus_stage
        if outdir is None:
            outdir = self.report_dir + '/' + focus
        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        # if samples is not None:
        #     subset = self.make_subset(samples)

        if subset is not None:
            samples = self._get_samples(subset)
        elif samples is None:
            samples = self.subsets['All_Analyses']
        elif isinstance(samples, str):
            samples = [samples]

        with self.pbar.set(total=len(samples), desc='Drawing Plots') as prog:
            for s in samples:
                f, a = self.data[s].tplot(analytes=analytes, figsize=figsize,
                                        scale=scale, filt=filt,
                                        ranges=ranges, stats=stats,
                                        stat=stat, err=err, focus_stage=focus)
                # ax = fig.axes[0]
                # for l, u in s.sigrng:
                #     ax.axvspan(l, u, color='r', alpha=0.1)
                # for l, u in s.bkgrng:
                #     ax.axvspan(l, u, color='k', alpha=0.1)
                f.savefig(outdir + '/' + s + '_traces.pdf')
                # TODO: on older(?) computers raises
                # 'OSError: [Errno 24] Too many open files'
                plt.close(f)
                prog.update()
        return


Instruction:
Implement a Python 3 function for
plotting the gradients of the analytes in the specified directory.

Response:
def gradient_plots(self, analytes=None, win=15, samples=None, ranges=False,
                       focus=None, outdir=None,
                       figsize=[10, 4], subset='All_Analyses'):
        """
        Plot analyte gradients as a function of time.

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to plot. Defaults to all analytes.
        samples: optional, array_like or str
            The sample(s) to plot. Defaults to all samples.
        ranges : bool
            Whether or not to show the signal/backgroudn regions
            identified by 'autorange'.
        focus : str
            The focus 'stage' of the analysis to plot. Can be
            'rawdata', 'despiked':, 'signal', 'background',
            'bkgsub', 'ratios' or 'calibrated'.
        outdir : str
            Path to a directory where you'd like the plots to be
            saved. Defaults to 'reports/[focus]' in your data directory.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        scale : str
            If 'log', plots the data on a log scale.
        figsize : array_like
            Array of length 2 specifying figure [width, height] in
            inches.
        stats : bool
            Whether or not to overlay the mean and standard deviations
            for each trace.
        stat, err: str
            The names of the statistic and error components to plot.
            Deafaults to 'nanmean' and 'nanstd'.

        Returns
        -------
        None
        """
        if focus is None:
            focus = self.focus_stage
        if outdir is None:
            outdir = self.report_dir + '/' + focus + '_gradient'
        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        # if samples is not None:
        #     subset = self.make_subset(samples)

        if subset is not None:
            samples = self._get_samples(subset)
        elif samples is None:
            samples = self.subsets['All_Analyses']
        elif isinstance(samples, str):
            samples = [samples]

        with self.pbar.set(total=len(samples), desc='Drawing Plots') as prog:
            for s in samples:
                f, a = self.data[s].gplot(analytes=analytes, win=win, figsize=figsize,
                                        ranges=ranges, focus_stage=focus)
                # ax = fig.axes[0]
                # for l, u in s.sigrng:
                #     ax.axvspan(l, u, color='r', alpha=0.1)
                # for l, u in s.bkgrng:
                #     ax.axvspan(l, u, color='k', alpha=0.1)
                f.savefig(outdir + '/' + s + '_gradients.pdf')
                # TODO: on older(?) computers raises
                # 'OSError: [Errno 24] Too many open files'
                plt.close(f)
                prog.update()
        return


Instruction:
Write a Python 3 script to
plot filter reports for all analytes in the name.

Response:
def filter_reports(self, analytes, filt_str='all', nbin=5, samples=None,
                       outdir=None, subset='All_Samples'):
        """
        Plot filter reports for all filters that contain ``filt_str``
        in the name.
        """
        if outdir is None:
            outdir = self.report_dir + '/filters/' + filt_str
            if not os.path.isdir(self.report_dir + '/filters'):
                os.mkdir(self.report_dir + '/filters')
        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        with self.pbar.set(total=len(samples), desc='Drawing Plots') as prog:
            for s in samples:
                _ = self.data[s].filter_report(filt=filt_str,
                                            analytes=analytes,
                                            savedir=outdir,
                                            nbin=nbin)
                prog.update()
            # plt.close(fig)
        return


Instruction:
Implement a Python 3 function for
calculating sample statistics for the specified analytes.

Response:
def sample_stats(self, analytes=None, filt=True,
                     stats=['mean', 'std'],
                     eachtrace=True, csf_dict={}):
        """
        Calculate sample statistics.

        Returns samples, analytes, and arrays of statistics
        of shape (samples, analytes). Statistics are calculated
        from the 'focus' data variable, so output depends on how
        the data have been processed.

        Included stat functions:

        * :func:`~latools.stat_fns.mean`: arithmetic mean
        * :func:`~latools.stat_fns.std`: arithmetic standard deviation
        * :func:`~latools.stat_fns.se`: arithmetic standard error
        * :func:`~latools.stat_fns.H15_mean`: Huber mean (outlier removal)
        * :func:`~latools.stat_fns.H15_std`: Huber standard deviation (outlier removal)
        * :func:`~latools.stat_fns.H15_se`: Huber standard error (outlier removal)

        Parameters
        ----------
        analytes : optional, array_like or str
            The analyte(s) to calculate statistics for. Defaults to
            all analytes.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        stats : array_like
            take a single array_like input, and return a single statistic.
            list of functions or names (see above) or functions that
            Function should be able to cope with NaN values.
        eachtrace : bool
            Whether to calculate the statistics for each analysis
            spot individually, or to produce per - sample means.
            Default is True.

        Returns
        -------
        None
            Adds dict to analyse object containing samples, analytes and
            functions and data.
        """
        if analytes is None:
            analytes = self.analytes
        elif isinstance(analytes, str):
            analytes = [analytes]

        self.stats = Bunch()

        self.stats_calced = []
        stat_fns = Bunch()

        stat_dict = {'mean': np.nanmean,
                     'std': np.nanstd,
                     'nanmean': np.nanmean,
                     'nanstd': np.nanstd,
                     'se': stderr,
                     'H15_mean': H15_mean,
                     'H15_std': H15_std,
                     'H15_se': H15_se}

        for s in stats:
            if isinstance(s, str):
                if s in stat_dict.keys():
                    self.stats_calced.append(s)
                    stat_fns[s] = stat_dict[s]
                if s in csf_dict.keys():
                    self.stats_calced.append(s)
                    exec(csf_dict[s])
                    stat_fns[s] = eval(s)
            elif callable(s):
                self.stats_calced.append(s.__name__)
                stat_fns[s.__name__] = s
                if not hasattr(self, 'custom_stat_functions'):
                    self.custom_stat_functions = ''
                self.custom_stat_functions += inspect.getsource(s) + '\n\n\n\n'

        # calculate stats for each sample
        with self.pbar.set(total=len(self.samples), desc='Calculating Stats') as prog:
            for s in self.samples:
                if self.srm_identifier not in s:
                    self.data[s].sample_stats(analytes, filt=filt,
                                            stat_fns=stat_fns,
                                            eachtrace=eachtrace)

                    self.stats[s] = self.data[s].stats
                prog.update()


Instruction:
Can you generate the documentation for the following Python 3 function
def statplot(self, analytes=None, samples=None, figsize=None,
                 stat='mean', err='std', subset=None):
        """
        Function for visualising per-ablation and per-sample means.

        Parameters
        ----------
        analytes : str or iterable
            Which analyte(s) to plot
        samples : str or iterable
            Which sample(s) to plot
        figsize : tuple
            Figure (width, height) in inches
        stat : str
            Which statistic to plot. Must match
            the name of the functions used in
            'sample_stats'.
        err : str
            Which uncertainty to plot.
        subset : str
            Which subset of samples to plot.
        """
        if not hasattr(self, 'stats'):
            self.sample_stats()

        if analytes is None:
                analytes = self.analytes
        elif isinstance(analytes, str):
            analytes = [analytes]

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        analytes = [a for a in analytes if a !=
                    self.internal_standard]

        if figsize is None:
            figsize = (1.5 * len(self.stats), 3 * len(analytes))

        fig, axs = plt.subplots(len(analytes), 1, figsize=figsize)

        for ax, an in zip(axs, analytes):
            i = 0
            stab = self.getstats()
            m, u = unitpicker(np.percentile(stab.loc[:, an].dropna(), 25), 0.1,
                              focus_stage='calibrated',
                              denominator=self.internal_standard)
            for s in samples:
                if self.srm_identifier not in s:
                    d = self.stats[s]
                    if d[stat].ndim == 2:
                        n = d[stat].shape[-1]
                        x = np.linspace(i - .1 * n / 2, i + .1 * n / 2, n)
                    else:
                        x = [i]
                    a_ind = d['analytes'] == an

                    # plot individual ablations with error bars
                    ax.errorbar(x, d[stat][a_ind][0] * m,
                                yerr=d[err][a_ind][0] * m,
                                marker='o', color=self.cmaps[an],
                                lw=0, elinewidth=1)

                    ax.set_ylabel('%s / %s (%s )' % (pretty_element(an),
                                                     pretty_element(self.internal_standard),
                                                     u))

                    # plot whole - sample mean
                    if len(x) > 1:
                        # mean calculation with error propagation?
                        # umean = un.uarray(d[stat][a_ind][0] * m, d[err][a_ind][0] * m).mean()
                        # std = un.std_devs(umean)
                        # mean = un.nominal_values(umean)
                        mean = np.nanmean(d[stat][a_ind][0] * m)
                        std = np.nanstd(d[stat][a_ind][0] * m)
                        ax.plot(x, [mean] * len(x), c=self.cmaps[an], lw=2)
                        ax.fill_between(x, [mean + std] * len(x),
                                        [mean - std] * len(x),
                                        lw=0, alpha=0.2, color=self.cmaps[an])

                    # highlight each sample
                    if i % 2 == 1:
                        ax.axvspan(i - .5, i + .5, color=(0, 0, 0, 0.05), lw=0)

                    i += 1

            ax.set_xticks(np.arange(0, len(self.stats)))
            ax.set_xlim(-0.5, len(self.stats) - .5)

            ax.set_xticklabels(samples)

        return fig, ax

Response: Function for visualising per - abation and per - sample means.


Instruction:
Can you tell what is the following Python 3 function doing
def getstats(self, save=True, filename=None, samples=None, subset=None, ablation_time=False):
        slst = []

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        for s in self.stats_calced:
            for nm in [n for n in samples if self.srm_identifier
                       not in n]:
                if self.stats[nm][s].ndim == 2:
                    # make multi - index
                    reps = np.arange(self.stats[nm][s].shape[-1])
                    ss = np.array([s] * reps.size)
                    nms = np.array([nm] * reps.size)
                    # make sub - dataframe
                    stdf = pd.DataFrame(self.stats[nm][s].T,
                                        columns=self.stats[nm]['analytes'],
                                        index=[ss, nms, reps])
                    stdf.index.set_names(['statistic', 'sample', 'rep'],
                                         inplace=True)
                else:
                    stdf = pd.DataFrame(self.stats[nm][s],
                                        index=self.stats[nm]['analytes'],
                                        columns=[[s], [nm]]).T

                    stdf.index.set_names(['statistic', 'sample'],
                                         inplace=True)
                slst.append(stdf)
        out = pd.concat(slst)

        if ablation_time:
            ats = self.ablation_times(samples=samples, subset=subset)
            ats['statistic'] = 'nanmean'
            ats.set_index('statistic', append=True, inplace=True)
            ats = ats.reorder_levels(['statistic', 'sample', 'rep'])

            out = out.join(ats)

        out.drop(self.internal_standard, 1, inplace=True)

        if save:
            if filename is None:
                filename = 'stat_export.csv'
            out.to_csv(self.export_dir + '/' + filename)

        self.stats_df = out

        return out

Response: Return pandas dataframe of all sample statistics.


Instruction:
Create a Python 3 function to
export traces from all analytes to a single file.

Response:
def _minimal_export_traces(self, outdir=None, analytes=None,
                               samples=None, subset='All_Analyses'):
        """
        Used for exporting minimal dataset. DON'T USE.
        """
        if analytes is None:
            analytes = self.analytes
        elif isinstance(analytes, str):
            analytes = [analytes]

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        focus_stage = 'rawdata'
        # ud = 'counts'

        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        for s in samples:
            d = self.data[s].data[focus_stage]
            out = Bunch()

            for a in analytes:
                out[a] = d[a]

            out = pd.DataFrame(out, index=self.data[s].Time)
            out.index.name = 'Time'

            d = dateutil.parser.parse(self.data[s].meta['date'])
            header = ['# Minimal Reproduction Dataset Exported from LATOOLS on %s' %
                      (time.strftime('%Y:%m:%d %H:%M:%S')),
                      "# Analysis described in '../analysis.lalog'",
                      '# Run latools.reproduce to import analysis.',
                      '#',
                      '# Sample: %s' % (s),
                      '# Analysis Time: ' + d.strftime('%Y-%m-%d %H:%M:%S')]

            header = '\n'.join(header) + '\n'

            csv = out.to_csv()

            with open('%s/%s.csv' % (outdir, s), 'w') as f:
                f.write(header)
                f.write(csv)
        return


Instruction:
Write a Python 3 function for
functioning to export raw data from the main - dir - name_export directory.

Response:
def export_traces(self, outdir=None, focus_stage=None, analytes=None,
                      samples=None, subset='All_Analyses', filt=False, zip_archive=False):
        """
        Function to export raw data.

        Parameters
        ----------
        outdir : str
            directory to save toe traces. Defaults to 'main-dir-name_export'.
        focus_stage : str
            The name of the analysis stage to export.

            * 'rawdata': raw data, loaded from csv file.
            * 'despiked': despiked data.
            * 'signal'/'background': isolated signal and background data.
              Created by self.separate, after signal and background
              regions have been identified by self.autorange.
            * 'bkgsub': background subtracted data, created by
              self.bkg_correct
            * 'ratios': element ratio data, created by self.ratio.
            * 'calibrated': ratio data calibrated to standards, created by self.calibrate.

            Defaults to the most recent stage of analysis.
        analytes : str or array - like
            Either a single analyte, or list of analytes to export.
            Defaults to all analytes.
        samples : str or array - like
            Either a single sample name, or list of samples to export.
            Defaults to all samples.
        filt : str, dict or bool
            Either logical filter expression contained in a str,
            a dict of expressions specifying the filter string to
            use for each analyte or a boolean. Passed to `grab_filt`.
        """
        if analytes is None:
            analytes = self.analytes
        elif isinstance(analytes, str):
            analytes = [analytes]

        if samples is not None:
            subset = self.make_subset(samples)

        samples = self._get_samples(subset)

        if focus_stage is None:
            focus_stage = self.focus_stage

        if focus_stage in ['ratios', 'calibrated']:
            analytes = [a for a in analytes if a != self.internal_standard]

        if outdir is None:
            outdir = os.path.join(self.export_dir, 'trace_export')

        ud = {'rawdata': 'counts',
              'despiked': 'counts',
              'bkgsub': 'background corrected counts',
              'ratios': 'counts/count {:s}',
              'calibrated': 'mol/mol {:s}'}
        if focus_stage in ['ratios', 'calibrated']:
            ud[focus_stage] = ud[focus_stage].format(self.internal_standard)

        if not os.path.isdir(outdir):
            os.mkdir(outdir)

        for s in samples:
            d = self.data[s].data[focus_stage]
            ind = self.data[s].filt.grab_filt(filt)
            out = Bunch()

            for a in analytes:
                out[a] = nominal_values(d[a][ind])
                if focus_stage not in ['rawdata', 'despiked']:
                    out[a + '_std'] = std_devs(d[a][ind])
                    out[a + '_std'][out[a + '_std'] == 0] = np.nan

            out = pd.DataFrame(out, index=self.data[s].Time[ind])
            out.index.name = 'Time'

            header = ['# Sample: %s' % (s),
                      '# Data Exported from LATOOLS on %s' %
                      (time.strftime('%Y:%m:%d %H:%M:%S')),
                      '# Processed using %s configuration' % (self.config['config']),
                      '# Analysis Stage: %s' % (focus_stage),
                      '# Unit: %s' % ud[focus_stage]]

            header = '\n'.join(header) + '\n'

            csv = out.to_csv()

            with open('%s/%s_%s.csv' % (outdir, s, focus_stage), 'w') as f:
                f.write(header)
                f.write(csv)

        if zip_archive:
            utils.zipdir(outdir, delete=True)

        return


Instruction:
Make a summary of the following Python 3 code
def save_log(self, directory=None, logname=None, header=None):
        if directory is None:
            directory = self.export_dir
        if not os.path.isdir(directory):
            directory = os.path.dirname(directory)

        if logname is None:
            logname = 'analysis.lalog'

        if header is None:
            header = self._log_header()

        loc = logging.write_logfile(self.log, header,
                                    os.path.join(directory, logname))

        return loc

Response: Save the log file in specified location.


Instruction:
Create a Python 3 function for
exporting a minimal reproduction dataset and a minimal reproduction dataset.

Response:
def minimal_export(self, target_analytes=None, path=None):
        """
        Exports a analysis parameters, standard info and a minimal dataset,
        which can be imported by another user.

        Parameters
        ----------
        target_analytes : str or iterable
            Which analytes to include in the export. If specified, the export
            will contain these analytes, and all other analytes used during
            data processing (e.g. during filtering). If not specified,
            all analytes are exported.
        path : str
            Where to save the minimal export.
            If it ends with .zip, a zip file is created.
            If it's a folder, all data are exported to a folder.
        """
        if target_analytes is None:
            target_analytes = self.analytes
        if isinstance(target_analytes, str):
            target_analytes = [target_analytes]

        self.minimal_analytes.update(target_analytes)
        zip_archive = False

        # set up data path
        if path is None:
            path = self.export_dir + '/minimal_export.zip'
        if path.endswith('.zip'):
            path = path.replace('.zip', '')
            zip_archive = True
        if not os.path.isdir(path):
            os.mkdir(path)

        # export data
        self._minimal_export_traces(path + '/data', analytes=self.minimal_analytes)

         # define analysis_log header
        log_header = ['# Minimal Reproduction Dataset Exported from LATOOLS on %s' %
                      (time.strftime('%Y:%m:%d %H:%M:%S')),
                      'data_folder :: ./data/']

        if hasattr(self, 'srmdat'):
            log_header.append('srm_table :: ./srm.table')

            # export srm table
            els = np.unique([re.sub('[0-9]', '', a) for a in self.minimal_analytes])
            srmdat = []
            for e in els:
                srmdat.append(self.srmdat.loc[self.srmdat.element == e, :])
            srmdat = pd.concat(srmdat)

            with open(path + '/srm.table', 'w') as f:
                f.write(srmdat.to_csv())

        # save custom functions (of defined)
        if hasattr(self, 'custom_stat_functions'):
            with open(path + '/custom_stat_fns.py', 'w') as f:
                f.write(self.custom_stat_functions)
            log_header.append('custom_stat_functions :: ./custom_stat_fns.py')

        log_header.append('# Analysis Log Start: \n')

        # format sample_stats correctly
        lss = [(i, l) for i, l in enumerate(self.log) if 'sample_stats' in l]
        rep = re.compile("(.*'stats': )(\[.*?\])(.*)")
        for i, l in lss:
            self.log[i] = rep.sub(r'\1' + str(self.stats_calced) + r'\3', l)

        # save log
        self.save_log(path, 'analysis.lalog', header=log_header)

        if zip_archive:
            utils.zipdir(directory=path, delete=True)

        return


Instruction:
Here you have a function in Python 3, explain what it does
def by_regex(file, outdir=None, split_pattern=None, global_header_rows=0, fname_pattern=None, trim_tail_lines=0, trim_head_lines=0):
    # create output sirectory
    if outdir is None:
        outdir = os.path.join(os.path.dirname(file), 'split')
    if not os.path.exists(outdir):
        os.mkdir(outdir)

    # read input file
    with open(file, 'r') as f:
        lines = f.readlines()

    # get file extension
    extension = os.path.splitext(file)[-1]

    # grab global header rows
    global_header = lines[:global_header_rows]

    # find indices of lines containing split_pattern
    starts = []
    for i, line in enumerate(lines):
        if re.search(split_pattern, line):
            starts.append(i)
    starts.append(len(lines))  # get length of lines

    # split lines into segments based on positions of regex
    splits = {}
    for i in range(len(starts) - 1):
        m = re.search(fname_pattern, lines[starts[i]])
        if m:
            fname = m.groups()[0].strip()
        else:
            fname = 'no_name_{:}'.format(i)

        splits[fname] = global_header + lines[starts[i]:starts[i+1]][trim_head_lines:trim_tail_lines]

    # write files
    print('Writing files to: {:}'.format(outdir))
    for k, v in splits.items():
        fname = (k + extension).replace(' ', '_')
        with open(os.path.join(outdir, fname), 'w') as f:
            f.writelines(v)
        print('  {:}'.format(fname))

    print('Done.')

    return outdir

Response: Splits one long analysis file into multiple smaller ones.


Instruction:
Explain what the following Python 3 code does
def long_file(data_file, dataformat, sample_list, savedir=None, srm_id=None, **autorange_args):
    if isinstance(sample_list, str):
        if os.path.exists(sample_list):
            sample_list = np.genfromtxt(sample_list, dtype=str)
        else:
            raise ValueError('File {} not found.')
    elif not isinstance(sample_list, (list, np.ndarray)):
        raise ValueError('sample_list should be an array_like or a file.')

    if srm_id is not None:
        srm_replace = []
        for s in sample_list:
            if srm_id in s:
                s = srm_id
            srm_replace.append(s)
        sample_list = srm_replace

    _, _, dat, meta = read_data(data_file, dataformat=dataformat, name_mode='file')

    if 'date' in meta:
        d = dateutil.parser.parse(meta['date'])
    else:
        d = datetime.datetime.now()
    # autorange
    bkg, sig, trn, _ = autorange(dat['Time'], dat['total_counts'], **autorange_args)

    ns = np.zeros(sig.size)
    ns[sig] = np.cumsum((sig ^ np.roll(sig, 1)) & sig)[sig]

    n = int(max(ns))

    if len(sample_list) != n:
        warn('Length of sample list does not match number of ablations in file.\n' +
             'We will continue, but please make sure the assignments are correct.')

    # calculate split boundaries
    bounds = []
    lower = 0
    sn = 0
    next_sample = ''
    for ni in range(n-1):
        sample = sample_list[sn]
        next_sample = sample_list[sn + 1]

        if sample != next_sample:
            current_end = np.argwhere(dat['Time'] == dat['Time'][ns == ni + 1].max())[0]
            next_start = np.argwhere(dat['Time'] == dat['Time'][ns == ni + 2].min())[0]
            upper = (current_end + next_start) // 2

            bounds.append((sample, (int(lower), int(upper))))

            lower = upper + 1

        sn += 1

    bounds.append((sample_list[-1], (int(upper) + 1, len(ns))))

    # split up data
    sections = {}
    seen = {}
    for s, (lo, hi) in bounds:
        if s not in seen:
            seen[s] = 0
        else:
            seen[s] += 1
            s += '_{}'.format(seen[s])
        sections[s] = {'oTime': dat['Time'][lo:hi]}
        sections[s]['Time'] = sections[s]['oTime'] - np.nanmin(sections[s]['oTime'])
        sections[s]['rawdata'] = {}
        for k, v in dat['rawdata'].items():
            sections[s]['rawdata'][k] = v[lo:hi]
        sections[s]['starttime'] = d + datetime.timedelta(seconds=np.nanmin(sections[s]['oTime']))

    # save output
    if savedir is None:
        savedir = os.path.join(os.path.dirname(os.path.abspath(data_file)), os.path.splitext(os.path.basename(data_file))[0] + '_split')
    if not os.path.isdir(savedir):
        os.makedirs(savedir)

    header = ['# Long data file split by latools on {}'.format(datetime.datetime.now().strftime('%Y:%m:%d %H:%M:%S'))]
    if 'date' not in meta:
        header.append('# Warning: No date specified in file - Analysis Times are date file was split. ')
    else:
        header.append('# ')
        header.append('# ')
        header.append('# ')

    flist = [savedir]
    for s, dat in sections.items():
        iheader = header.copy()
        iheader.append('# Sample: {}'.format(s))
        iheader.append('# Analysis Time: {}'.format(dat['starttime'].strftime('%Y-%m-%d %H:%M:%S')))

        iheader = '\n'.join(iheader) + '\n'

        out = pd.DataFrame({analyte_2_namemass(k): v for k, v in dat['rawdata'].items()}, index=dat['Time'])
        out.index.name = 'Time'
        csv = out.to_csv()

        with open('{}/{}.csv'.format(savedir, s), 'w') as f:
            f.write(iheader)
            f.write(csv)
        flist.append('   {}.csv'.format(s))

    print("File split into {} sections.\n Saved to: {}\n\n Import using the 'REPRODUCE' configuration.".format(n, '\n'.join(flist)))
    return None

Response: Read a file and return a list of all possible long ablations.


Instruction:
Can you tell what is the following Python 3 function doing
def fold_map(self, fa: F[A], z: B, f: Callable[[A], B], g: Callable[[Z, B], Z]=operator.add) -> Z:
        ''' map `f` over the traversable, then fold over the result
        using the supplied initial element `z` and operation `g`,
        defaulting to addition for the latter.
        '''
        mapped = Functor.fatal(type(fa)).map(fa, f)
        return self.fold_left(mapped)(z)(g)

Response:
fold over the traversable then fold over the result
        using the supplied initial element z and operation g.


Instruction:
How would you explain what the following Python 3 function does
def pca_calc(nc, d):

    # check for and remove nans
    ind = ~np.apply_along_axis(any, 1, np.isnan(d))

    if any(~ind):
        pcs = np.full((d.shape[0], nc), np.nan)
        d = d[ind, :]

    pca = PCA(nc).fit(d)

    if any(~ind):
        pcs[ind, :] = pca.transform(d)
    else:
        pcs = pca.transform(d)

    return pca, pcs

Response: Calculates the pca of d.


Instruction:
Can you generate the documentation for the following Python 3 function
def pca_plot(pca, dt, xlabs=None, mode='scatter', lognorm=True):

    nc = pca.n_components
    f = np.arange(pca.n_features_)
    cs = list(itertools.combinations(range(nc), 2))

    ind = ~np.apply_along_axis(any, 1, np.isnan(dt))

    cylim = (pca.components_.min(), pca.components_.max())
    yd = cylim[1] - cylim[0]

    # Make figure
    fig, axs = plt.subplots(nc, nc, figsize=[3 * nc, nc * 3], tight_layout=True)

    for x, y in zip(*np.triu_indices(nc)):
        if x == y:
            tax = axs[x, y]
            tax.bar(f, pca.components_[x], 0.8)
            tax.set_xticks([])
            tax.axhline(0, zorder=-1, c=(0,0,0,0.6))

            # labels
            tax.set_ylim(cylim[0] - 0.2 * yd,
                         cylim[1] + 0.2 * yd)

            for xi, yi, lab in zip(f, pca.components_[x], xlabs):
                if yi > 0:
                    yo = yd * 0.03
                    va = 'bottom'
                else:
                    yo = yd * -0.02
                    va = 'top'

                tax.text(xi, yi + yo, lab, ha='center', va=va, rotation=90, fontsize=8)

        else:
            xv = dt[ind, x]
            yv = dt[ind, y]

            if mode == 'scatter':
                axs[x, y].scatter(xv, yv, alpha=0.2)
                axs[y, x].scatter(yv, xv, alpha=0.2)
            if mode == 'hist2d':
                if lognorm:
                    norm = mpl.colors.LogNorm()
                else:
                    norm = None
                axs[x, y].hist2d(xv, yv, 50, cmap=plt.cm.Blues, norm=norm)
                axs[y, x].hist2d(yv, xv, 50, cmap=plt.cm.Blues, norm=norm)

        if x == 0:
            axs[y, x].set_ylabel('PC{:.0f}'.format(y + 1))
        if y == nc - 1:
            axs[y, x].set_xlabel('PC{:.0f}'.format(x + 1))

    return fig, axs, xv, yv

Response: Plots a fitted PCA and all components.


Instruction:
Make a summary of the following Python 3 code
def calc_windows(fn, s, min_points):
    max_points = np.sum(~np.isnan(s))
    n_points = max_points - min_points

    out = np.full((n_points, s.size), np.nan)

    # skip nans, for speed
    ind = ~np.isnan(s)
    s = s[ind]

    for i, w in enumerate(range(min_points, s.size)):
        r = rolling_window(s, w, pad=np.nan)
        out[i, ind] = np.apply_along_axis(fn, 1, r)

    return out

Response: Calculate the number of windows in a set of contiguous regions.


Instruction:
Create a Python 3 function for
calculating the mean and std of a window.

Response:
def calc_window_mean_std(s, min_points, ind=None):
    """
    Apply fn to all contiguous regions in s that have at least min_points.
    """
    max_points = np.sum(~np.isnan(s))
    n_points = max_points - min_points

    mean = np.full((n_points, s.size), np.nan)
    std = np.full((n_points, s.size), np.nan)

    # skip nans, for speed
    if ind is None:
        ind = ~np.isnan(s)
    else:
        ind = ind & ~np.isnan(s)
    s = s[ind]

    for i, w in enumerate(range(min_points, s.size)):
        r = rolling_window(s, w, pad=np.nan)
        mean[i, ind] = r.sum(1) / w
        std[i, ind] = (((r - mean[i, ind][:, np.newaxis])**2).sum(1) / (w - 1))**0.5
        # mean[i, ind] = np.apply_along_axis(np.nanmean, 1, r)
        # std[i, ind] = np.apply_along_axis(np.nanstd, 1, r)

    return mean, std


Instruction:
How would you explain what the following Python 3 function does
def bayes_scale(s):
    if sum(~np.isnan(s)) > 1:
        bm, bv, bs = bayes_mvs(s[~np.isnan(s)])
        return (s - bm.statistic) / bs.statistic
    else:
        return np.full(s.shape, np.nan)

Response: Scale a sequence of data points using bayes_kvm statistics.


Instruction:
Write a Python 3 script for
computing the median scaling of a sequence of log entries.

Response:
def median_scaler(s):
    """
    Remove median, divide by IQR.
    """
    if sum(~np.isnan(s)) > 2:
        ss = s[~np.isnan(s)]
        median = np.median(ss)
        IQR = np.diff(np.percentile(ss, [25, 75]))
        return (s - median) / IQR
    else:
        return np.full(s.shape, np.nan)


Instruction:
Can you tell what is the following Python 3 function doing
def signal_optimiser(d, analytes, min_points=5,
                     threshold_mode='kde_first_max',
                     threshold_mult=1., x_bias=0,
                     weights=None, ind=None, mode='minimise'):
    """
    Optimise data selection based on specified analytes.

    Identifies the longest possible contiguous data region in
    the signal where the relative standard deviation (std) and
    concentration of all analytes is minimised.

    Optimisation is performed via a grid search of all possible
    contiguous data regions. For each region, the mean std and
    mean scaled analyte concentration ('amplitude') are calculated.

    The size and position of the optimal data region are identified
    using threshold std and amplitude values. Thresholds are derived
    from all calculated stds and amplitudes using the method specified
    by `threshold_mode`. For example, using the 'kde_max' method, a
    probability density function (PDF) is calculated for std and
    amplitude values, and the threshold is set as the maximum of the
    PDF. These thresholds are then used to identify the size and position
    of the longest contiguous region where the std is below the threshold,
    and the amplitude is either below the threshold.

    All possible regions of the data that have at least
    `min_points` are considered.

    For a graphical demonstration of the action of signal_optimiser,
    use `optimisation_plot`.

    Parameters
    ----------
    d : latools.D object
        An latools data object.
    analytes : str or array-like
        Which analytes to consider.
    min_points : int
        The minimum number of contiguous points to
        consider.
    threshold_mode : str
        The method used to calculate the optimisation
        thresholds. Can be 'mean', 'median', 'kde_max'
        or 'bayes_mvs', or a custom function. If a
        function, must take a 1D array, and return a
        single, real number.
    threshood_mult : float or tuple
        A multiplier applied to the calculated threshold
        before use. If a tuple, the first value is applied
        to the mean threshold, and the second is applied to
        the standard deviation threshold. Reduce this to make
        data selection more stringent.
    x_bias : float
        If non-zero, a bias is applied to the calculated statistics
        to prefer the beginning (if > 0) or end (if < 0) of the
        signal. Should be between zero and 1.
    weights : array-like of length len(analytes)
        An array of numbers specifying the importance of
        each analyte considered. Larger number makes the
        analyte have a greater effect on the optimisation.
        Default is None.
    ind : boolean array
        A boolean array the same length as the data. Where
        false, data will not be included.
    mode : str
        Whether to 'minimise' or 'maximise' the concentration
        of the elements.

    Returns
    -------
    dict, str : optimisation result, error message
    """
    errmsg = ''

    if isinstance(analytes, str):
        analytes = [analytes]

    if ind is None:
        ind = np.full(len(d.Time), True)

    # initial catch
    if not any(ind) or (np.diff(bool_2_indices(ind)).max() < min_points):
        errmsg = 'Optmisation failed. No contiguous data regions longer than {:.0f} points.'.format(min_points)
        return Bunch({'means': np.nan,
                      'stds': np.nan,
                      'mean_threshold': np.nan,
                      'std_threshold': np.nan,
                      'lims': np.nan,
                      'filt': ind,
                      'threshold_mode': threshold_mode,
                      'min_points': min_points,
                      'analytes': analytes,
                      'opt_centre': np.nan,
                      'opt_n_points': np.nan,
                      'weights': weights,
                      'optimisation_success': False,
                      'errmsg': errmsg}), errmsg

    msmeans, msstds = calculate_optimisation_stats(d, analytes, min_points, weights, ind, x_bias)

    # second catch
    if all(np.isnan(msmeans).flat) or all(np.isnan(msmeans).flat):
        errmsg = 'Optmisation failed. No contiguous data regions longer than {:.0f} points.'.format(min_points)
        return Bunch({'means': np.nan,
                      'stds': np.nan,
                      'mean_threshold': np.nan,
                      'std_threshold': np.nan,
                      'lims': np.nan,
                      'filt': ind,
                      'threshold_mode': threshold_mode,
                      'min_points': min_points,
                      'analytes': analytes,
                      'opt_centre': np.nan,
                      'opt_n_points': np.nan,
                      'weights': weights,
                      'optimisation_success': False,
                      'errmsg': errmsg}), errmsg

    # define thresholds
    valid = ['kde_first_max', 'kde_max', 'median', 'bayes_mvs', 'mean']
    n_under = 0
    i = np.argwhere(np.array(valid) == threshold_mode)[0, 0]
    o_threshold_mode = threshold_mode
    while (n_under <= 0) & (i < len(valid)):
        if threshold_mode == 'median':
            # median - OK, but best?
            std_threshold = np.nanmedian(msstds)
            mean_threshold = np.nanmedian(msmeans)
        elif threshold_mode == 'mean':
            # mean
            std_threshold = np.nanmean(msstds)
            mean_threshold = np.nanmean(msmeans)
        elif threshold_mode == 'kde_max':
            # maximum of gaussian kernel density estimator
            mkd = gaussian_kde(msmeans[~np.isnan(msmeans)].flat)
            xm = np.linspace(*np.percentile(msmeans.flatten()[~np.isnan(msmeans.flatten())], (1, 99)), 100)
            mdf = mkd.pdf(xm)
            mean_threshold = xm[np.argmax(mdf)]

            rkd = gaussian_kde(msstds[~np.isnan(msstds)])
            xr = np.linspace(*np.percentile(msstds.flatten()[~np.isnan(msstds.flatten())], (1, 99)), 100)
            rdf = rkd.pdf(xr)
            std_threshold = xr[np.argmax(rdf)]
        elif threshold_mode == 'kde_first_max':
            # first local maximum of gaussian kernel density estimator
            mkd = gaussian_kde(msmeans[~np.isnan(msmeans)].flat)
            xm = np.linspace(*np.percentile(msmeans.flatten()[~np.isnan(msmeans.flatten())], (1, 99)), 100)
            mdf = mkd.pdf(xm)
            inds = np.argwhere(np.r_[False, mdf[1:] > mdf[:-1]] &
                            np.r_[mdf[:-1] > mdf[1:], False] &
                            (mdf > 0.25 * mdf.max()))
            mean_threshold = xm[np.min(inds)]

            rkd = gaussian_kde(msstds[~np.isnan(msstds)])
            xr = np.linspace(*np.percentile(msstds.flatten()[~np.isnan(msstds.flatten())], (1, 99)), 100)
            rdf = rkd.pdf(xr)
            inds = np.argwhere(np.r_[False, rdf[1:] > rdf[:-1]] &
                            np.r_[rdf[:-1] > rdf[1:], False] &
                            (rdf > 0.25 * rdf.max()))
            std_threshold = xr[np.min(inds)]
        elif threshold_mode == 'bayes_mvs':
            # bayesian mvs.
            bm, _, bs = bayes_mvs(msstds[~np.isnan(msstds)])
            std_threshold = bm.statistic

            bm, _, bs = bayes_mvs(msmeans[~np.isnan(msmeans)])
            mean_threshold = bm.statistic
        elif callable(threshold_mode):
            std_threshold = threshold_mode(msstds[~np.isnan(msstds)].flatten())
            mean_threshold = threshold_mode(msmeans[~np.isnan(msmeans)].flatten())
        else:
            try:
                mean_threshold, std_threshold = threshold_mode
            except:
                raise ValueError('\nthreshold_mode must be one of:\n   ' + ', '.join(valid) + ',\na custom function, or a \n(mean_threshold, std_threshold) tuple.')

        # apply threshold_mult
        if isinstance(threshold_mult, (int, float)):
            std_threshold *= threshold_mult
            mean_threshold *= threshold_mult
        elif len(threshold_mult) == 2:
            mean_threshold *= threshold_mult[0]
            std_threshold *= threshold_mult[1]
        else:
            raise ValueError('\nthreshold_mult must be a float, int or tuple of length 2.')

        rind = (msstds < std_threshold)

        if mode == 'minimise':
            mind = (msmeans < mean_threshold)
        else:
            mind = (msmeans > mean_threshold)

        ind = rind & mind

        n_under = ind.sum()
        if n_under == 0:
            i += 1
            if i <= len(valid) - 1:
                threshold_mode = valid[i]
            else:
                errmsg = 'Optimisation failed. No of the threshold_mode would work. Try reducting min_points.'
                return Bunch({'means': np.nan,
                              'stds': np.nan,
                              'mean_threshold': np.nan,
                              'std_threshold': np.nan,
                              'lims': np.nan,
                              'filt': ind,
                              'threshold_mode': threshold_mode,
                              'min_points': min_points,
                              'analytes': analytes,
                              'opt_centre': np.nan,
                              'opt_n_points': np.nan,
                              'weights': weights,
                              'optimisation_success': False,
                              'errmsg': errmsg}), errmsg

    if i > 0:
        errmsg = "optimisation failed using threshold_mode='{:}', falling back to '{:}'".format(o_threshold_mode, threshold_mode)

    # identify max number of points within thresholds
    passing = np.argwhere(ind)
    opt_n_points = passing[:, 0].max()
    opt_centre = passing[passing[:, 0] == opt_n_points, 1].min()

    opt_n_points += min_points

    # centres, npoints = np.meshgrid(np.arange(msmeans.shape[1]),
    #                                np.arange(min_points, min_points + msmeans.shape[0]))
    # opt_n_points = npoints[ind].max()
    # plus/minus one point to allow some freedom to shift selection window.
    # cind = ind & (npoints == opt_n_points)
    # opt_centre = centres[cind].min()

    if opt_n_points % 2 == 0:
        lims = (opt_centre - opt_n_points // 2,
                opt_centre + opt_n_points // 2)
    else:
        lims = (opt_centre - opt_n_points // 2,
                opt_centre + opt_n_points // 2 + 1)

    filt = np.zeros(d.Time.shape, dtype=bool)
    filt[lims[0]:lims[1]] = True

    return Bunch({'means': msmeans,
                  'stds': msstds,
                  'mean_threshold': mean_threshold,
                  'std_threshold': std_threshold,
                  'lims': lims,
                  'filt': filt,
                  'threshold_mode': threshold_mode,
                  'min_points': min_points,
                  'analytes': analytes,
                  'opt_centre': opt_centre,
                  'opt_n_points': opt_n_points,
                  'weights': weights,
                  'optimisation_success': True,
                  'errmsg': errmsg}), errmsg

Response: This function is used to optimize the signal of the specified analytes.


Instruction:
Can you write a function in Python 3 where it
plots the result of signal_optimise.

Response:
def optimisation_plot(d, overlay_alpha=0.5, **kwargs):
    """
    Plot the result of signal_optimise.

    `signal_optimiser` must be run first, and the output
    stored in the `opt` attribute of the latools.D object.

    Parameters
    ----------
    d : latools.D object
        A latools data object.
    overlay_alpha : float
        The opacity of the threshold overlays. Between 0 and 1.
    **kwargs
        Passed to `tplot`
    """
    if not hasattr(d, 'opt'):
        raise ValueError('Please run `signal_optimiser` before trying to plot its results.')

    out = []
    for n, opt in d.opt.items():
        if not opt['optimisation_success']:
            out.append((None, None))

        else:
            # unpack variables
            means = opt['means']
            stds = opt['stds']
            min_points = opt['min_points']
            mean_threshold = opt['mean_threshold']
            std_threshold = opt['std_threshold']
            opt_centre = opt['opt_centre']
            opt_n_points = opt['opt_n_points']

            centres, npoints = np.meshgrid(np.arange(means.shape[1]), np.arange(min_points, min_points + means.shape[0]))
            rind = (stds < std_threshold)
            mind = (means < mean_threshold)

            # color scale and histogram limits
            mlim = np.percentile(means.flatten()[~np.isnan(means.flatten())], (0, 99))
            rlim = np.percentile(stds.flatten()[~np.isnan(stds.flatten())], (0, 99))

            cmr = plt.cm.Blues
            cmr.set_bad((0,0,0,0.3))

            cmm = plt.cm.Reds
            cmm.set_bad((0,0,0,0.3))

            # create figure
            fig = plt.figure(figsize=[7,7])

            ma = fig.add_subplot(3, 2, 1)
            ra = fig.add_subplot(3, 2, 2)

            # work out image limits
            nonan = np.argwhere(~np.isnan(means))
            xdif = np.ptp(nonan[:, 1])
            ydif = np.ptp(nonan[:, 0])
            extent = (nonan[:, 1].min() - np.ceil(0.1 * xdif),  # x min
                    nonan[:, 1].max() + np.ceil(0.1 * xdif),  # x max
                    nonan[:, 0].min() + min_points,  # y min
                    nonan[:, 0].max() + np.ceil(0.1 * ydif) + min_points)  # y max

            mm = ma.imshow(means, origin='bottomleft', cmap=cmm, vmin=mlim[0], vmax=mlim[1],
                        extent=(centres.min(), centres.max(), npoints.min(), npoints.max()))

            ma.set_ylabel('N points')
            ma.set_xlabel('Center')
            fig.colorbar(mm, ax=ma, label='Amplitude')

            mr = ra.imshow(stds, origin='bottomleft', cmap=cmr, vmin=rlim[0], vmax=rlim[1],
                        extent=(centres.min(), centres.max(), npoints.min(), npoints.max()))

            ra.set_xlabel('Center')
            fig.colorbar(mr, ax=ra, label='std')

            # view limits
            ra.imshow(~rind, origin='bottomleft', cmap=plt.cm.Greys, alpha=overlay_alpha,
                    extent=(centres.min(), centres.max(), npoints.min(), npoints.max()))
            ma.imshow(~mind, origin='bottomleft', cmap=plt.cm.Greys, alpha=overlay_alpha,
                    extent=(centres.min(), centres.max(), npoints.min(), npoints.max()))

            for ax in [ma, ra]:
                ax.scatter(opt_centre, opt_n_points, c=(1,1,1,0.7), edgecolor='k',marker='o')
                ax.set_xlim(extent[:2])
                ax.set_ylim(extent[-2:])

            # draw histograms
            mah = fig.add_subplot(3, 2, 3)
            rah = fig.add_subplot(3, 2, 4)

            mah.set_xlim(mlim)
            mbin = np.linspace(*mah.get_xlim(), 50)
            mah.hist(means.flatten()[~np.isnan(means.flatten())], mbin)
            mah.axvspan(mean_threshold, mah.get_xlim()[1], color=(0,0,0,overlay_alpha))

            mah.axvline(mean_threshold, c='r')
            mah.set_xlabel('Scaled Mean Analyte Conc')
            mah.set_ylabel('N')

            rah.set_xlim(rlim)
            rbin = np.linspace(*rah.get_xlim(), 50)
            rah.hist(stds.flatten()[~np.isnan(stds.flatten())], rbin)
            rah.axvspan(std_threshold, rah.get_xlim()[1], color=(0,0,0,0.4))
            rah.axvline(std_threshold, c='r')
            rah.set_xlabel('std')

            tax = fig.add_subplot(3,1,3)
            tplot(d, opt.analytes, ax=tax, **kwargs)
            tax.axvspan(*d.Time[[opt.lims[0], opt.lims[1]]], alpha=0.2)

            tax.set_xlim(d.Time[d.ns == n].min() - 3, d.Time[d.ns == n].max() + 3)

            fig.tight_layout()

            out.append((fig, (ma, ra, mah, rah, tax)))
    return out


Instruction:
Make a summary of the following Python 3 code
def noise_despike(sig, win=3, nlim=24., maxiter=4):
    if win % 2 != 1:
        win += 1  # win must be odd

    kernel = np.ones(win) / win  # make convolution kernel
    over = np.ones(len(sig), dtype=bool)  # initialize bool array
    # pad edges to avoid edge-effects
    npad = int((win - 1) / 2)
    over[:npad] = False
    over[-npad:] = False
    # set up monitoring
    nloops = 0
    # do the despiking
    while any(over) and (nloops < maxiter):
        rmean = np.convolve(sig, kernel, 'valid')  # mean by convolution
        rstd = rmean**0.5  # std = sqrt(signal), because count statistics
        # identify where signal > mean + std * nlim (OR signa < mean - std *
        # nlim)
        # | (sig[npad:-npad] < rmean - nlim * rstd)
        over[npad:-npad] = (sig[npad:-npad] > rmean + nlim * rstd)
        # if any are over, replace them with mean of neighbours
        if any(over):
            # replace with values either side
            # sig[over] = sig[np.roll(over, -1) | np.roll(over, 1)].reshape((sum(over), 2)).mean(1)
            # replace with mean
            sig[npad:-npad][over[npad:-npad]] = rmean[over[npad:-npad]]
            nloops += 1
        # repeat until no more removed.
    return sig

Response: This function is used to remove anomalous values from a signal.


Instruction:
Write a Python 3 function for
applying exponential decay filter to remove physically impossible data based on instrumental washout. The filter is re-applied until no more points are removed, or maxiter is reached. Parameters ---------- exponent : float Exponent used in filter tstep : float The time increment between data points. maxiter : int The maximum number of times the filter should be applied. Returns ------- None

Response:
def expdecay_despike(sig, expdecay_coef, tstep, maxiter=3):
    """
    Apply exponential decay filter to remove physically impossible data based on instrumental washout.

    The filter is re-applied until no more points are removed, or maxiter is reached.

    Parameters
    ----------
    exponent : float
        Exponent used in filter
    tstep : float
        The time increment between data points.
    maxiter : int
        The maximum number of times the filter should be applied.

    Returns
    -------
    None
    """
    # determine rms noise of data
    noise = np.std(sig[:5])  # initially, calculated based on first 5 points
    # expand the selection up to 50 points, unless it dramatically increases
    # the std (i.e. catches the 'laser on' region)
    for i in [10, 20, 30, 50]:
        inoise = np.std(sig[:i])
        if inoise < 1.5 * noise:
            noise = inoise
    rms_noise3 = 3 * noise

    i = 0
    f = True
    while (i < maxiter) and f:
        # calculate low and high possibles values based on exponential decay
        siglo = np.roll(sig * np.exp(tstep * expdecay_coef), 1)
        sighi = np.roll(sig * np.exp(-tstep * expdecay_coef), -1)

        # identify points that are outside these limits, beyond what might be explained
        # by noise in the data
        loind = (sig < siglo - rms_noise3) & (sig < np.roll(sig, -1) - rms_noise3)
        hiind = (sig > sighi + rms_noise3) & (sig > np.roll(sig, 1) + rms_noise3)

        # replace all such values with their preceding
        sig[loind] = sig[np.roll(loind, -1)]
        sig[hiind] = sig[np.roll(hiind, -1)]

        f = any(np.concatenate([loind, hiind]))
        i += 1

    return sig


Instruction:
Given the following Python 3 function, write the documentation
def _flat_map(self, f: Callable):
        ''' **f** must return the same stack type as **self.value** has.
        Iterates over the effects, sequences the inner instance
        successively to the top and joins with the outer instance.
        Example:
        List(Right(Just(1))) => List(Right(Just(List(Right(Just(5))))))
        => List(List(Right(Just(Right(Just(5))))))
        => List(Right(Just(Right(Just(5)))))
        => List(Right(Right(Just(Just(5)))))
        => List(Right(Just(Just(5))))
        => List(Right(Just(5)))
        Note: IO works only as outermost effect, as it cannot sequence
        '''
        index = List.range(self.depth + 1)
        g = index.fold_left(f)(lambda z, i: lambda a: a.map(z))
        nested = g(self.value)
        def sequence_level(z, depth, tpe):
            nesting = lambda z, i: lambda a: a.map(z).sequence(tpe)
            lifter = List.range(depth).fold_left(I)(nesting)
            return z // lifter
        def sequence_type(z, data):
            return lambda a: sequence_level(a, *data).map(z)
        h = self.all_effects.reversed.with_index.fold_left(I)(sequence_type)
        return h(nested)

Response: Map a function over the effects and return the same stack type as self. value.


Instruction:
Can you tell what is the following Python 3 function doing
def add(self, name, filt, info='', params=(), setn=None):

        iname = '{:.0f}_'.format(self.n) + name
        self.index[self.n] = iname

        if setn is None:
            setn = self.maxset + 1
        self.maxset = setn

        if setn not in self.sets.keys():
            self.sets[setn] = [iname]
        else:
            self.sets[setn].append(iname)

        # self.keys is not added to?
        self.components[iname] = filt
        self.info[iname] = info
        self.params[iname] = params
        for a in self.analytes:
            self.switches[a][iname] = False
        self.n += 1
        return

Response: Add a new entry to the set.


Instruction:
Can you create a Python 3 function that
removes the filter from the set that name belongs to the current set.

Response:
def remove(self, name=None, setn=None):
        """
        Remove filter.

        Parameters
        ----------
        name : str
            name of the filter to remove
        setn : int or True
            int: number of set to remove
            True: remove all filters in set that 'name' belongs to

        Returns
        -------
        None
        """
        if isinstance(name, int):
            name = self.index[name]

        if setn is not None:
            name = self.sets[setn]
            del self.sets[setn]
        elif isinstance(name, (int, str)):
            name = [name]

        if setn is True:
            for n in name:
                for k, v in self.sets.items():
                    if n in v:
                        name.append([m for m in v if m != n])

        for n in name:
            for k, v in self.sets.items():
                if n in v:
                    self.sets[k] = [m for m in v if n != m]
            del self.components[n]
            del self.info[n]
            del self.params[n]
            del self.keys[n]
            for a in self.analytes:
                del self.switches[a][n]
            return


Instruction:
Can you tell what is the following Python 3 function doing
def clear(self):
        self.components = {}
        self.info = {}
        self.params = {}
        self.switches = {}
        self.keys = {}
        self.index = {}
        self.sets = {}
        self.maxset = -1
        self.n = 0
        for a in self.analytes:
            self.switches[a] = {}
        return

Response: Clear all the related data.