Instruction:
Can you tell what is the following Python 3 function doing
def update_checkplot_objectinfo(cpf,
                                fast_mode=False,
                                findercmap='gray_r',
                                finderconvolve=None,
                                deredden_object=True,
                                custom_bandpasses=None,
                                gaia_submit_timeout=10.0,
                                gaia_submit_tries=3,
                                gaia_max_timeout=180.0,
                                gaia_mirror=None,
                                complete_query_later=True,
                                lclistpkl=None,
                                nbrradiusarcsec=60.0,
                                maxnumneighbors=5,
                                plotdpi=100,
                                findercachedir='~/.astrobase/stamp-cache',
                                verbose=True):
    '''This updates a checkplot objectinfo dict.

    Useful in cases where a previous round of GAIA/finderchart/external catalog
    acquisition failed. This will preserve the following keys in the checkplot
    if they exist::

        comments
        varinfo
        objectinfo.objecttags

    Parameters
    ----------

    cpf : str
        The path to the checkplot pickle to update.

    fast_mode : bool or float
        This runs the external catalog operations in a "fast" mode, with short
        timeouts and not trying to hit external catalogs that take a long time
        to respond. See the docstring for
        :py:func:`astrobase.checkplot.pkl_utils._pkl_finder_objectinfo` for
        details on how this works. If this is True, will run in "fast" mode with
        default timeouts (5 seconds in most cases). If this is a float, will run
        in "fast" mode with the provided timeout value in seconds.

    findercmap : str or matplotlib.cm.ColorMap object
        The Colormap object to use for the finder chart image.

    finderconvolve : astropy.convolution.Kernel object or None
        If not None, the Kernel object to use for convolving the finder image.

    deredden_objects : bool
        If this is True, will use the 2MASS DUST service to get extinction
        coefficients in various bands, and then try to deredden the magnitudes
        and colors of the object already present in the checkplot's objectinfo
        dict.

    custom_bandpasses : dict
        This is a dict used to provide custom bandpass definitions for any
        magnitude measurements in the objectinfo dict that are not automatically
        recognized by the `varclass.starfeatures.color_features` function. See
        its docstring for details on the required format.

    gaia_submit_timeout : float
        Sets the timeout in seconds to use when submitting a request to look up
        the object's information to the GAIA service. Note that if `fast_mode`
        is set, this is ignored.

    gaia_submit_tries : int
        Sets the maximum number of times the GAIA services will be contacted to
        obtain this object's information. If `fast_mode` is set, this is
        ignored, and the services will be contacted only once (meaning that a
        failure to respond will be silently ignored and no GAIA data will be
        added to the checkplot's objectinfo dict).

    gaia_max_timeout : float
        Sets the timeout in seconds to use when waiting for the GAIA service to
        respond to our request for the object's information. Note that if
        `fast_mode` is set, this is ignored.

    gaia_mirror : str
        This sets the GAIA mirror to use. This is a key in the
        :py:data:`astrobase.services.gaia.GAIA_URLS` dict which defines the URLs
        to hit for each mirror.

    complete_query_later : bool
        If this is True, saves the state of GAIA queries that are not yet
        complete when `gaia_max_timeout` is reached while waiting for the GAIA
        service to respond to our request. A later call for GAIA info on the
        same object will attempt to pick up the results from the existing query
        if it's completed. If `fast_mode` is True, this is ignored.

    lclistpkl : dict or str
        If this is provided, must be a dict resulting from reading a catalog
        produced by the `lcproc.catalogs.make_lclist` function or a str path
        pointing to the pickle file produced by that function. This catalog is
        used to find neighbors of the current object in the current light curve
        collection. Looking at neighbors of the object within the radius
        specified by `nbrradiusarcsec` is useful for light curves produced by
        instruments that have a large pixel scale, so are susceptible to
        blending of variability and potential confusion of neighbor variability
        with that of the actual object being looked at. If this is None, no
        neighbor lookups will be performed.

    nbrradiusarcsec : float
        The radius in arcseconds to use for a search conducted around the
        coordinates of this object to look for any potential confusion and
        blending of variability amplitude caused by their proximity.

    maxnumneighbors : int
        The maximum number of neighbors that will have their light curves and
        magnitudes noted in this checkplot as potential blends with the target
        object.

    plotdpi : int
        The resolution in DPI of the plots to generate in this function
        (e.g. the finder chart, etc.)

    findercachedir : str
        The path to the astrobase cache directory for finder chart downloads
        from the NASA SkyView service.

    verbose : bool
        If True, will indicate progress and warn about potential problems.

    Returns
    -------

    str
        Path to the updated checkplot pickle file.

    '''

    cpd = _read_checkplot_picklefile(cpf)

    if cpd['objectinfo']['objecttags'] is not None:
        objecttags = cpd['objectinfo']['objecttags'][::]
    else:
        objecttags = None

    varinfo = deepcopy(cpd['varinfo'])

    if 'comments' in cpd and cpd['comments'] is not None:
        comments = cpd['comments'][::]
    else:
        comments = None

    newcpd = _pkl_finder_objectinfo(cpd['objectinfo'],
                                    varinfo,
                                    findercmap,
                                    finderconvolve,
                                    cpd['sigclip'],
                                    cpd['normto'],
                                    cpd['normmingap'],
                                    fast_mode=fast_mode,
                                    deredden_object=deredden_object,
                                    custom_bandpasses=custom_bandpasses,
                                    gaia_submit_timeout=gaia_submit_timeout,
                                    gaia_submit_tries=gaia_submit_tries,
                                    gaia_max_timeout=gaia_max_timeout,
                                    gaia_mirror=gaia_mirror,
                                    complete_query_later=complete_query_later,
                                    lclistpkl=lclistpkl,
                                    nbrradiusarcsec=nbrradiusarcsec,
                                    maxnumneighbors=maxnumneighbors,
                                    plotdpi=plotdpi,
                                    findercachedir=findercachedir,
                                    verbose=verbose)

    #
    # don't update neighbors or finder chart if the new one is bad
    #
    if (newcpd['finderchart'] is None and
        cpd['finderchart'] is not None):
        newcpd['finderchart'] = deepcopy(
            cpd['finderchart']
        )

    if (newcpd['neighbors'] is None and
        cpd['neighbors'] is not None):
        newcpd['neighbors'] = deepcopy(
            cpd['neighbors']
        )

    #
    # if there's existing GAIA info, don't overwrite if the new objectinfo dict
    # doesn't have any
    #
    if (('failed' in newcpd['objectinfo']['gaia_status'] or
         ('gaiaid' in newcpd['objectinfo'] and
          newcpd['objectinfo']['gaiaid'] is None)) and
        'ok' in cpd['objectinfo']['gaia_status']):

        newcpd['objectinfo']['gaia_status'] = deepcopy(
            cpd['objectinfo']['gaia_status']
        )
        if 'gaiaid' in cpd['objectinfo']:
            newcpd['objectinfo']['gaiaid'] = deepcopy(
                cpd['objectinfo']['gaiaid']
            )
        newcpd['objectinfo']['gaiamag'] = deepcopy(
            cpd['objectinfo']['gaiamag']
        )
        newcpd['objectinfo']['gaia_absmag'] = deepcopy(
            cpd['objectinfo']['gaia_absmag']
        )
        newcpd['objectinfo']['gaia_parallax'] = deepcopy(
            cpd['objectinfo']['gaia_parallax']
        )
        newcpd['objectinfo']['gaia_parallax_err'] = deepcopy(
            cpd['objectinfo']['gaia_parallax_err']
        )
        newcpd['objectinfo']['gaia_pmra'] = deepcopy(
            cpd['objectinfo']['gaia_pmra']
        )
        newcpd['objectinfo']['gaia_pmra_err'] = deepcopy(
            cpd['objectinfo']['gaia_pmra_err']
        )
        newcpd['objectinfo']['gaia_pmdecl'] = deepcopy(
            cpd['objectinfo']['gaia_pmdecl']
        )
        newcpd['objectinfo']['gaia_pmdecl_err'] = deepcopy(
            cpd['objectinfo']['gaia_pmdecl_err']
        )

    if (not np.isfinite(newcpd['objectinfo']['gaia_neighbors']) and
        np.isfinite(cpd['objectinfo']['gaia_neighbors'])):
        newcpd['objectinfo']['gaia_neighbors'] = deepcopy(
            cpd['objectinfo']['gaia_neighbors']
        )
    if (not np.isfinite(newcpd['objectinfo']['gaia_closest_distarcsec']) and
        np.isfinite(cpd['objectinfo']['gaia_closest_distarcsec'])):
        newcpd['objectinfo']['gaia_closest_distarcsec'] = deepcopy(
            cpd['objectinfo']['gaia_closest_gmagdiff']
        )
    if (not np.isfinite(newcpd['objectinfo']['gaia_closest_gmagdiff']) and
        np.isfinite(cpd['objectinfo']['gaia_closest_gmagdiff'])):
        newcpd['objectinfo']['gaia_closest_gmagdiff'] = deepcopy(
            cpd['objectinfo']['gaia_closest_gmagdiff']
        )

    if (newcpd['objectinfo']['gaia_ids'] is None and
        cpd['objectinfo']['gaia_ids'] is not None):
        newcpd['objectinfo']['gaia_ids'] = deepcopy(
            cpd['objectinfo']['gaia_ids']
        )
    if (newcpd['objectinfo']['gaia_xypos'] is None and
        cpd['objectinfo']['gaia_xypos'] is not None):
        newcpd['objectinfo']['gaia_xypos'] = deepcopy(
            cpd['objectinfo']['gaia_xypos']
        )
    if (newcpd['objectinfo']['gaia_mags'] is None and
        cpd['objectinfo']['gaia_mags'] is not None):
        newcpd['objectinfo']['gaia_mags'] = deepcopy(
            cpd['objectinfo']['gaia_mags']
        )
    if (newcpd['objectinfo']['gaia_parallaxes'] is None and
        cpd['objectinfo']['gaia_parallaxes'] is not None):
        newcpd['objectinfo']['gaia_parallaxes'] = deepcopy(
            cpd['objectinfo']['gaia_parallaxes']
        )
    if (newcpd['objectinfo']['gaia_parallax_errs'] is None and
        cpd['objectinfo']['gaia_parallax_errs'] is not None):
        newcpd['objectinfo']['gaia_parallax_errs'] = deepcopy(
            cpd['objectinfo']['gaia_parallax_errs']
        )
    if (newcpd['objectinfo']['gaia_pmras'] is None and
        cpd['objectinfo']['gaia_pmras'] is not None):
        newcpd['objectinfo']['gaia_pmras'] = deepcopy(
            cpd['objectinfo']['gaia_pmras']
        )
    if (newcpd['objectinfo']['gaia_pmra_errs'] is None and
        cpd['objectinfo']['gaia_pmra_errs'] is not None):
        newcpd['objectinfo']['gaia_pmra_errs'] = deepcopy(
            cpd['objectinfo']['gaia_pmra_errs']
        )
    if (newcpd['objectinfo']['gaia_pmdecls'] is None and
        cpd['objectinfo']['gaia_pmdecls'] is not None):
        newcpd['objectinfo']['gaia_pmdecls'] = deepcopy(
            cpd['objectinfo']['gaia_pmdecls']
        )
    if (newcpd['objectinfo']['gaia_pmdecl_errs'] is None and
        cpd['objectinfo']['gaia_pmdecl_errs'] is not None):
        newcpd['objectinfo']['gaia_pmdecl_errs'] = deepcopy(
            cpd['objectinfo']['gaia_pmdecl_errs']
        )
    if (newcpd['objectinfo']['gaia_absolute_mags'] is None and
        cpd['objectinfo']['gaia_absolute_mags'] is not None):
        newcpd['objectinfo']['gaia_absolute_mags'] = deepcopy(
            cpd['objectinfo']['gaia_absolute_mags']
        )
    if (newcpd['objectinfo']['gaiak_colors'] is None and
        cpd['objectinfo']['gaiak_colors'] is not None):
        newcpd['objectinfo']['gaiak_colors'] = deepcopy(
            cpd['objectinfo']['gaiak_colors']
        )
    if (newcpd['objectinfo']['gaia_dists'] is None and
        cpd['objectinfo']['gaia_dists'] is not None):
        newcpd['objectinfo']['gaia_dists'] = deepcopy(
            cpd['objectinfo']['gaia_dists']
        )

    #
    # don't overwrite good SIMBAD info with bad
    #
    if ('failed' in newcpd['objectinfo']['simbad_status'] and
        'ok' in cpd['objectinfo']['simbad_status']):
        newcpd['objectinfo']['simbad_status'] = deepcopy(
            cpd['objectinfo']['simbad_status']
        )

    if (newcpd['objectinfo']['simbad_nmatches'] is None and
        cpd['objectinfo']['simbad_nmatches'] is not None):
        newcpd['objectinfo']['simbad_nmatches'] = deepcopy(
            cpd['objectinfo']['simbad_nmatches']
        )
    if (newcpd['objectinfo']['simbad_mainid'] is None and
        cpd['objectinfo']['simbad_mainid'] is not None):
        newcpd['objectinfo']['simbad_mainid'] = deepcopy(
            cpd['objectinfo']['simbad_mainid']
        )
    if (newcpd['objectinfo']['simbad_objtype'] is None and
        cpd['objectinfo']['simbad_objtype'] is not None):
        newcpd['objectinfo']['simbad_objtype'] = deepcopy(
            cpd['objectinfo']['simbad_objtype']
        )
    if (newcpd['objectinfo']['simbad_allids'] is None and
        cpd['objectinfo']['simbad_allids'] is not None):
        newcpd['objectinfo']['simbad_allids'] = deepcopy(
            cpd['objectinfo']['simbad_allids']
        )
    if (newcpd['objectinfo']['simbad_distarcsec'] is None and
        cpd['objectinfo']['simbad_distarcsec'] is not None):
        newcpd['objectinfo']['simbad_distarcsec'] = deepcopy(
            cpd['objectinfo']['simbad_distarcsec']
        )
    if (newcpd['objectinfo']['simbad_best_mainid'] is None and
        cpd['objectinfo']['simbad_best_mainid'] is not None):
        newcpd['objectinfo']['simbad_best_mainid'] = deepcopy(
            cpd['objectinfo']['simbad_best_mainid']
        )
    if (newcpd['objectinfo']['simbad_best_objtype'] is None and
        cpd['objectinfo']['simbad_best_objtype'] is not None):
        newcpd['objectinfo']['simbad_best_objtype'] = deepcopy(
            cpd['objectinfo']['simbad_best_objtype']
        )
    if (newcpd['objectinfo']['simbad_best_allids'] is None and
        cpd['objectinfo']['simbad_best_allids'] is not None):
        newcpd['objectinfo']['simbad_best_allids'] = deepcopy(
            cpd['objectinfo']['simbad_best_allids']
        )
    if (newcpd['objectinfo']['simbad_best_distarcsec'] is None and
        cpd['objectinfo']['simbad_best_distarcsec'] is not None):
        newcpd['objectinfo']['simbad_best_distarcsec'] = deepcopy(
            cpd['objectinfo']['simbad_best_distarcsec']
        )

    #
    # update the objectinfo dict
    #
    cpd.update(newcpd)
    cpd['objectinfo']['objecttags'] = objecttags
    cpd['comments'] = comments

    newcpf = _write_checkplot_picklefile(cpd, outfile=cpf)

    return newcpf

Response: This function updates the checkplot objectinfo dict with the information from the external catalog.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _read_pklc(lcfile):
    '''
    This just reads a light curve pickle file.

    Parameters
    ----------

    lcfile : str
        The file name of the pickle to open.

    Returns
    -------

    dict
        This returns an lcdict.

    '''

    if lcfile.endswith('.gz'):

        try:
            with gzip.open(lcfile,'rb') as infd:
                lcdict = pickle.load(infd)
        except UnicodeDecodeError:
            with gzip.open(lcfile,'rb') as infd:
                lcdict = pickle.load(infd, encoding='latin1')

    else:

        try:
            with open(lcfile,'rb') as infd:
                lcdict = pickle.load(infd)
        except UnicodeDecodeError:
            with open(lcfile,'rb') as infd:
                lcdict = pickle.load(infd, encoding='latin1')

    return lcdict

Response: This just reads a light curve pickle file and returns a dict.


Instruction:
Explain what the following Python 3 code does
def get_lcformat(formatkey, use_lcformat_dir=None):
    '''This loads an LC format description from a previously-saved JSON file.

    Parameters
    ----------

    formatkey : str
        The key used to refer to the LC format. This is part of the JSON file's
        name, e.g. the format key 'hat-csv' maps to the format JSON file:
        '<astrobase install path>/data/lcformats/hat-csv.json'.

    use_lcformat_dir : str or None
        If provided, must be the path to a directory that contains the
        corresponding lcformat JSON file for `formatkey`. If this is None, this
        function will look for lcformat JSON files corresponding to the given
        `formatkey`:

        - first, in the directory specified in this kwarg,
        - if not found there, in the home directory: ~/.astrobase/lcformat-jsons
        - if not found there, in: <astrobase install path>/data/lcformats

    Returns
    -------

    tuple
        A tuple of the following form is returned::

            (fileglob       : the file glob of the associated LC files,
             readerfunc_in  : the imported Python function for reading LCs,
             timecols       : list of time col keys to get from the lcdict,
             magcols        : list of mag col keys to get from the lcdict ,
             errcols        : list of err col keys to get from the lcdict,
             magsarefluxes  : True if the measurements are fluxes not mags,
             normfunc_in    : the imported Python function for normalizing LCs)

        All `astrobase.lcproc` functions can then use this tuple to dynamically
        import your LC reader and normalization functions to work with your LC
        format transparently.

    '''

    if isinstance(use_lcformat_dir, str):

        # look for the lcformat JSON
        lcformat_jsonpath = os.path.join(
            use_lcformat_dir,
            '%s.json' % formatkey
        )

        if not os.path.exists(lcformat_jsonpath):

            lcformat_jsonpath = os.path.join(
                os.path.expanduser('~/.astrobase/lcformat-jsons'),
                '%s.json' % formatkey
            )

            if not os.path.exists(lcformat_jsonpath):

                install_path = os.path.dirname(__file__)
                install_path = os.path.abspath(
                    os.path.join(install_path, '..', 'data','lcformats')
                )

                lcformat_jsonpath = os.path.join(
                    install_path,
                    '%s.json' % formatkey
                )

                if not os.path.exists(lcformat_jsonpath):
                    LOGERROR('could not find an lcformat JSON '
                             'for formatkey: %s in any of: '
                             'use_lcformat_dir, home directory, '
                             'astrobase installed data directory'
                             % formatkey)
                    return None

    else:

        lcformat_jsonpath = os.path.join(
            os.path.expanduser('~/.astrobase/lcformat-jsons'),
            '%s.json' % formatkey
        )

        if not os.path.exists(lcformat_jsonpath):

            install_path = os.path.dirname(__file__)
            install_path = os.path.abspath(
                os.path.join(install_path, '..', 'data','lcformats')
            )

            lcformat_jsonpath = os.path.join(
                install_path,
                '%s.json' % formatkey
            )

            if not os.path.exists(lcformat_jsonpath):
                LOGERROR('could not find an lcformat JSON '
                         'for formatkey: %s in any of: '
                         'use_lcformat_dir, home directory, '
                         'astrobase installed data directory'
                         % formatkey)
                return None

    # load the found lcformat JSON
    with open(lcformat_jsonpath) as infd:
        lcformatdict = json.load(infd)

    readerfunc_module = lcformatdict['lcreader_module']
    readerfunc = lcformatdict['lcreader_func']
    readerfunc_kwargs = lcformatdict['lcreader_kwargs']
    normfunc_module = lcformatdict['lcnorm_module']
    normfunc = lcformatdict['lcnorm_func']
    normfunc_kwargs = lcformatdict['lcnorm_kwargs']

    fileglob = lcformatdict['fileglob']
    timecols = lcformatdict['timecols']
    magcols = lcformatdict['magcols']
    errcols = lcformatdict['errcols']
    magsarefluxes = lcformatdict['magsarefluxes']

    # import all the required bits
    # see if we can import the reader module
    readermodule = _check_extmodule(readerfunc_module, formatkey)

    if not readermodule:
        LOGERROR("could not import the required "
                 "module: %s to read %s light curves" %
                 (readerfunc_module, formatkey))
        return None

    # then, get the function we need to read the light curve
    try:
        readerfunc_in = getattr(readermodule, readerfunc)
    except AttributeError:
        LOGEXCEPTION('Could not get the specified reader '
                     'function: %s for lcformat: %s '
                     'from module: %s'
                     % (formatkey, readerfunc_module, readerfunc))
        raise

    # see if we can import the normalization module
    if normfunc_module:
        normmodule = _check_extmodule(normfunc_module, formatkey)
        if not normmodule:
            LOGERROR("could not import the required "
                     "module: %s to normalize %s light curves" %
                     (normfunc_module, formatkey))
            return None

    else:
        normmodule = None

    # finally, get the function we need to normalize the light curve
    if normfunc_module and normfunc:
        try:
            normfunc_in = getattr(normmodule, normfunc)
        except AttributeError:
            LOGEXCEPTION('Could not get the specified norm '
                         'function: %s for lcformat: %s '
                         'from module: %s'
                         % (formatkey, normfunc_module, normfunc))
            raise

    else:
        normfunc_in = None

    # add in any optional kwargs that need to be there for readerfunc
    if isinstance(readerfunc_kwargs, dict):
        readerfunc_in = partial(readerfunc_in, **readerfunc_kwargs)

    # add in any optional kwargs that need to be there for normfunc
    if normfunc_in is not None:
        if isinstance(normfunc_kwargs, dict):
            normfunc_in = partial(normfunc_in, **normfunc_kwargs)

    # assemble the return tuple
    # this can be used directly by other lcproc functions
    returntuple = (
        fileglob,
        readerfunc_in,
        timecols,
        magcols,
        errcols,
        magsarefluxes,
        normfunc_in,
    )

    return returntuple

Response: This function loads an LC format description from a previously - saved JSON file.


Instruction:
Can you write a function in Python 3 where it
uploads a file to an Amazon S3 bucket.

Response:
def s3_put_file(local_file, bucket, client=None, raiseonfail=False):
    """This uploads a file to S3.

    Parameters
    ----------

    local_file : str
        Path to the file to upload to S3.

    bucket : str
        The AWS S3 bucket to upload the file to.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    raiseonfail : bool
        If True, will re-raise whatever Exception caused the operation to fail
        and break out immediately.

    Returns
    -------

    str or None
        If the file upload is successful, returns the s3:// URL of the uploaded
        file. If it failed, will return None.

    """

    if not client:
        client = boto3.client('s3')

    try:
        client.upload_file(local_file, bucket, os.path.basename(local_file))
        return 's3://%s/%s' % (bucket, os.path.basename(local_file))
    except Exception as e:
        LOGEXCEPTION('could not upload %s to bucket: %s' % (local_file,
                                                            bucket))

        if raiseonfail:
            raise

        return None


Instruction:
Explain what the following Python 3 code does
def s3_delete_file(bucket, filename, client=None, raiseonfail=False):

    if not client:
        client = boto3.client('s3')

    try:
        resp = client.delete_object(Bucket=bucket, Key=filename)
        if not resp:
            LOGERROR('could not delete file %s from bucket %s' % (filename,
                                                                  bucket))
        else:
            return resp['DeleteMarker']
    except Exception as e:
        LOGEXCEPTION('could not delete file %s from bucket %s' % (filename,
                                                                  bucket))
        if raiseonfail:
            raise

        return None

Response: This function deletes a file from an S3 bucket.


Instruction:
Given the following Python 3 function, write the documentation
def sqs_create_queue(queue_name, options=None, client=None):

    if not client:
        client = boto3.client('sqs')

    try:

        if isinstance(options, dict):
            resp = client.create_queue(QueueName=queue_name, Attributes=options)
        else:
            resp = client.create_queue(QueueName=queue_name)

        if resp is not None:
            return {'url':resp['QueueUrl'],
                    'name':queue_name}
        else:
            LOGERROR('could not create the specified queue: %s with options: %s'
                     % (queue_name, options))
            return None

    except Exception as e:
        LOGEXCEPTION('could not create the specified queue: %s with options: %s'
                     % (queue_name, options))
        return None

Response: This function creates an SQS queue.


Instruction:
Make a summary of the following Python 3 code
def sqs_delete_item(queue_url,
                    receipt_handle,
                    client=None,
                    raiseonfail=False):
    """This deletes a message from the queue, effectively acknowledging its
    receipt.

    Call this only when all messages retrieved from the queue have been
    processed, since this will prevent redelivery of these messages to other
    queue workers pulling fromn the same queue channel.

    Parameters
    ----------

    queue_url : str
        The SQS URL of the queue where we got the messages from. This should be
        the same queue used to retrieve the messages in `sqs_get_item`.

    receipt_handle : str
        The receipt handle of the queue message that we're responding to, and
        will acknowledge receipt of. This will be present in each message
        retrieved using `sqs_get_item`.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    raiseonfail : bool
        If True, will re-raise whatever Exception caused the operation to fail
        and break out immediately.

    Returns
    -------

    Nothing.

    """

    if not client:
        client = boto3.client('sqs')

    try:

        client.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=receipt_handle
        )

    except Exception as e:

        LOGEXCEPTION(
            'could not delete message with receipt handle: '
            '%s from queue: %s' % (receipt_handle, queue_url)
        )

        if raiseonfail:
            raise

Response: This function deletes a message from the SQS queue.


Instruction:
Make a summary of the following Python 3 code
def make_ec2_nodes(
        security_groupid,
        subnet_id,
        keypair_name,
        iam_instance_profile_arn,
        launch_instances=1,
        ami='ami-04681a1dbd79675a5',
        instance='t3.micro',
        ebs_optimized=True,
        user_data=None,
        wait_until_up=True,
        client=None,
        raiseonfail=False,
):
    """This makes new EC2 worker nodes.

    This requires a security group ID attached to a VPC config and subnet, a
    keypair generated beforehand, and an IAM role ARN for the instance. See:

    https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html

    Use `user_data` to launch tasks on instance launch.

    Parameters
    ----------

    security_groupid : str
        The security group ID of the AWS VPC where the instances will be
        launched.

    subnet_id : str
        The subnet ID of the AWS VPC where the instances will be
        launched.

    keypair_name : str
        The name of the keypair to be used to allow SSH access to all instances
        launched here. This corresponds to an already downloaded AWS keypair PEM
        file.

    iam_instance_profile_arn : str
        The ARN string corresponding to the AWS instance profile that describes
        the permissions the launched instances have to access other AWS
        resources. Set this up in AWS IAM.

    launch_instances : int
        The number of instances to launch in this request.

    ami : str
        The Amazon Machine Image ID that describes the OS the instances will use
        after launch. The default ID is Amazon Linux 2 in the US East region.

    instance : str
        The instance type to launch. See the following URL for a list of IDs:
        https://aws.amazon.com/ec2/pricing/on-demand/

    ebs_optimized : bool
        If True, will enable EBS optimization to speed up IO. This is usually
        True for all instances made available in the last couple of years.

    user_data : str or None
        This is either the path to a file on disk that contains a shell-script
        or a string containing a shell-script that will be executed by root
        right after the instance is launched. Use to automatically set up
        workers and queues. If None, will not execute anything at instance
        start up.

    wait_until_up : bool
        If True, will not return from this function until all launched instances
        are verified as running by AWS.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    raiseonfail : bool
        If True, will re-raise whatever Exception caused the operation to fail
        and break out immediately.

    Returns
    -------

    dict
        Returns launched instance info as a dict, keyed by instance ID.

    """

    if not client:
        client = boto3.client('ec2')

    # get the user data from a string or a file
    # note: boto3 will base64 encode this itself
    if isinstance(user_data, str) and os.path.exists(user_data):
        with open(user_data,'r') as infd:
            udata = infd.read()

    elif isinstance(user_data, str):
        udata = user_data

    else:
        udata = (
            '#!/bin/bash\necho "No user data provided. '
            'Launched instance at: %s UTC"' % datetime.utcnow().isoformat()
        )

    # fire the request
    try:
        resp = client.run_instances(
            ImageId=ami,
            InstanceType=instance,
            SecurityGroupIds=[
                security_groupid,
            ],
            SubnetId=subnet_id,
            UserData=udata,
            IamInstanceProfile={'Arn':iam_instance_profile_arn},
            InstanceInitiatedShutdownBehavior='terminate',
            KeyName=keypair_name,
            MaxCount=launch_instances,
            MinCount=launch_instances,
            EbsOptimized=ebs_optimized,
        )

        if not resp:
            LOGERROR('could not launch requested instance')
            return None

        else:

            instance_dict = {}

            instance_list = resp.get('Instances',[])

            if len(instance_list) > 0:

                for instance in instance_list:

                    LOGINFO('launched instance ID: %s of type: %s at: %s. '
                            'current state: %s'
                            % (instance['InstanceId'],
                               instance['InstanceType'],
                               instance['LaunchTime'].isoformat(),
                               instance['State']['Name']))

                    instance_dict[instance['InstanceId']] = {
                        'type':instance['InstanceType'],
                        'launched':instance['LaunchTime'],
                        'state':instance['State']['Name'],
                        'info':instance
                    }

            # if we're waiting until we're up, then do so
            if wait_until_up:

                ready_instances = []

                LOGINFO('waiting until launched instances are up...')

                ntries = 5
                curr_try = 0

                while ( (curr_try < ntries) or
                        ( len(ready_instances) <
                          len(list(instance_dict.keys()))) ):

                    resp = client.describe_instances(
                        InstanceIds=list(instance_dict.keys()),
                    )

                    if len(resp['Reservations']) > 0:
                        for resv in resp['Reservations']:
                            if len(resv['Instances']) > 0:
                                for instance in resv['Instances']:
                                    if instance['State']['Name'] == 'running':

                                        ready_instances.append(
                                            instance['InstanceId']
                                        )

                                        instance_dict[
                                            instance['InstanceId']
                                        ]['state'] = 'running'

                                        instance_dict[
                                            instance['InstanceId']
                                        ]['ip'] = instance['PublicIpAddress']

                                        instance_dict[
                                            instance['InstanceId']
                                        ]['info'] = instance

                    # sleep for a bit so we don't hit the API too often
                    curr_try = curr_try + 1
                    time.sleep(5.0)

                if len(ready_instances) == len(list(instance_dict.keys())):
                    LOGINFO('all instances now up.')
                else:
                    LOGWARNING(
                        'reached maximum number of tries for instance status, '
                        'not all instances may be up.'
                    )

            return instance_dict

    except ClientError as e:

        LOGEXCEPTION('could not launch requested instance')
        if raiseonfail:
            raise

        return None

    except Exception as e:

        LOGEXCEPTION('could not launch requested instance')
        if raiseonfail:
            raise

        return None

Response: This function creates EC2 worker nodes for the given instance.


Instruction:
Here you have a function in Python 3, explain what it does
def delete_ec2_nodes(
        instance_id_list,
        client=None
):
    """This deletes EC2 nodes and terminates the instances.

    Parameters
    ----------

    instance_id_list : list of str
        A list of EC2 instance IDs to terminate.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    Returns
    -------

    Nothing.

    """

    if not client:
        client = boto3.client('ec2')

    resp = client.terminate_instances(
        InstanceIds=instance_id_list
    )

    return resp

Response: This function deletes EC2 nodes and terminates the instances.


Instruction:
Can you generate the documentation for the following Python 3 function
def make_spot_fleet_cluster(
        security_groupid,
        subnet_id,
        keypair_name,
        iam_instance_profile_arn,
        spot_fleet_iam_role,
        target_capacity=20,
        spot_price=0.4,
        expires_days=7,
        allocation_strategy='lowestPrice',
        instance_types=SPOT_INSTANCE_TYPES,
        instance_weights=None,
        instance_ami='ami-04681a1dbd79675a5',
        instance_user_data=None,
        instance_ebs_optimized=True,
        wait_until_up=True,
        client=None,
        raiseonfail=False
):
    """This makes an EC2 spot-fleet cluster.

    This requires a security group ID attached to a VPC config and subnet, a
    keypair generated beforehand, and an IAM role ARN for the instance. See:

    https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html

    Use `user_data` to launch tasks on instance launch.

    Parameters
    ----------

    security_groupid : str
        The security group ID of the AWS VPC where the instances will be
        launched.

    subnet_id : str
        The subnet ID of the AWS VPC where the instances will be
        launched.

    keypair_name : str
        The name of the keypair to be used to allow SSH access to all instances
        launched here. This corresponds to an already downloaded AWS keypair PEM
        file.

    iam_instance_profile_arn : str
        The ARN string corresponding to the AWS instance profile that describes
        the permissions the launched instances have to access other AWS
        resources. Set this up in AWS IAM.

    spot_fleet_iam_role : str
        This is the name of AWS IAM role that allows the Spot Fleet Manager to
        scale up and down instances based on demand and instances failing,
        etc. Set this up in IAM.

    target_capacity : int
        The number of instances to target in the fleet request. The fleet
        manager service will attempt to maintain this number over the lifetime
        of the Spot Fleet Request.

    spot_price : float
        The bid price in USD for the instances. This is per hour. Keep this at
        about half the hourly on-demand price of the desired instances to make
        sure your instances aren't taken away by AWS when it needs capacity.

    expires_days : int
        The number of days this request is active for. All instances launched by
        this request will live at least this long and will be terminated
        automatically after.

    allocation_strategy : {'lowestPrice', 'diversified'}
        The allocation strategy used by the fleet manager.

    instance_types : list of str
        List of the instance type to launch. See the following URL for a list of
        IDs: https://aws.amazon.com/ec2/pricing/on-demand/

    instance_weights : list of float or None
        If `instance_types` is a list of different instance types, this is the
        relative weight applied towards launching each instance type. This can
        be used to launch a mix of instances in a defined ratio among their
        types. Doing this can make the spot fleet more resilient to AWS taking
        back the instances if it runs out of capacity.

    instance_ami : str
        The Amazon Machine Image ID that describes the OS the instances will use
        after launch. The default ID is Amazon Linux 2 in the US East region.

    instance_user_data : str or None
        This is either the path to a file on disk that contains a shell-script
        or a string containing a shell-script that will be executed by root
        right after the instance is launched. Use to automatically set up
        workers and queues. If None, will not execute anything at instance
        start up.

    instance_ebs_optimized : bool
        If True, will enable EBS optimization to speed up IO. This is usually
        True for all instances made available in the last couple of years.

    wait_until_up : bool
        If True, will not return from this function until the spot fleet request
        is acknowledged by AWS.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    raiseonfail : bool
        If True, will re-raise whatever Exception caused the operation to fail
        and break out immediately.

    Returns
    -------

    str or None
        This is the spot fleet request ID if successful. Otherwise, returns
        None.

    """

    fleetconfig = copy.deepcopy(SPOT_FLEET_CONFIG)
    fleetconfig['IamFleetRole'] = spot_fleet_iam_role
    fleetconfig['AllocationStrategy'] = allocation_strategy
    fleetconfig['TargetCapacity'] = target_capacity
    fleetconfig['SpotPrice'] = str(spot_price)
    fleetconfig['ValidUntil'] = (
        datetime.utcnow() + timedelta(days=expires_days)
    ).strftime(
        '%Y-%m-%dT%H:%M:%SZ'
    )

    # get the user data from a string or a file
    # we need to base64 encode it here
    if (isinstance(instance_user_data, str) and
        os.path.exists(instance_user_data)):
        with open(instance_user_data,'rb') as infd:
            udata = base64.b64encode(infd.read()).decode()

    elif isinstance(instance_user_data, str):
        udata = base64.b64encode(instance_user_data.encode()).decode()

    else:
        udata = (
            '#!/bin/bash\necho "No user data provided. '
            'Launched instance at: %s UTC"' % datetime.utcnow().isoformat()
        )
        udata = base64.b64encode(udata.encode()).decode()

    for ind, itype in enumerate(instance_types):

        thisinstance = SPOT_PERINSTANCE_CONFIG.copy()
        thisinstance['InstanceType'] = itype
        thisinstance['ImageId'] = instance_ami
        thisinstance['SubnetId'] = subnet_id
        thisinstance['KeyName'] = keypair_name
        thisinstance['IamInstanceProfile']['Arn'] = iam_instance_profile_arn
        thisinstance['SecurityGroups'][0] = {'GroupId':security_groupid}
        thisinstance['UserData'] = udata
        thisinstance['EbsOptimized'] = instance_ebs_optimized

        # get the instance weights
        if isinstance(instance_weights, list):
            thisinstance['WeightedCapacity'] = instance_weights[ind]

        fleetconfig['LaunchSpecifications'].append(thisinstance)

    #
    # launch the fleet
    #

    if not client:
        client = boto3.client('ec2')

    try:

        resp = client.request_spot_fleet(
            SpotFleetRequestConfig=fleetconfig,
        )

        if not resp:

            LOGERROR('spot fleet request failed.')
            return None

        else:

            spot_fleet_reqid = resp['SpotFleetRequestId']
            LOGINFO('spot fleet requested successfully. request ID: %s' %
                    spot_fleet_reqid)

            if not wait_until_up:
                return spot_fleet_reqid

            else:

                ntries = 10
                curr_try = 0

                while curr_try < ntries:

                    resp = client.describe_spot_fleet_requests(
                        SpotFleetRequestIds=[
                            spot_fleet_reqid
                        ]
                    )

                    curr_state = resp.get('SpotFleetRequestConfigs',[])

                    if len(curr_state) > 0:

                        curr_state = curr_state[0]['SpotFleetRequestState']

                        if curr_state == 'active':
                            LOGINFO('spot fleet with reqid: %s is now active' %
                                    spot_fleet_reqid)
                            break

                    LOGINFO(
                        'spot fleet not yet active, waiting 15 seconds. '
                        'try %s/%s' % (curr_try, ntries)
                    )
                    curr_try = curr_try + 1
                    time.sleep(15.0)

                return spot_fleet_reqid

    except ClientError as e:

        LOGEXCEPTION('could not launch spot fleet')
        if raiseonfail:
            raise

        return None

    except Exception as e:

        LOGEXCEPTION('could not launch spot fleet')
        if raiseonfail:
            raise

        return None

Response: This function creates an EC2 spot - fleet cluster.


Instruction:
Given the following Python 3 function, write the documentation
def delete_spot_fleet_cluster(
        spot_fleet_reqid,
        client=None,
):
    """
    This deletes a spot-fleet cluster.

    Parameters
    ----------

    spot_fleet_reqid : str
        The fleet request ID returned by `make_spot_fleet_cluster`.

    client : boto3.Client or None
        If None, this function will instantiate a new `boto3.Client` object to
        use in its operations. Alternatively, pass in an existing `boto3.Client`
        instance to re-use it here.

    Returns
    -------

    Nothing.

    """

    if not client:
        client = boto3.client('ec2')

    resp = client.cancel_spot_fleet_requests(
        SpotFleetRequestIds=[spot_fleet_reqid],
        TerminateInstances=True
    )

    return resp

Response: This function deletes a spot - fleet cluster.


Instruction:
Here you have a function in Python 3, explain what it does
def gcs_get_file(bucketname,
                 filename,
                 local_file,
                 altexts=None,
                 client=None,
                 service_account_json=None,
                 raiseonfail=False):
    """This gets a single file from a Google Cloud Storage bucket.

    Parameters
    ----------

    bucketname : str
        The name of the GCS bucket to download the file from.

    filename : str
        The full name of the file to download, including all prefixes.

    local_file : str
        Path to where the downloaded file will be stored.

    altexts : None or list of str
        If not None, this is a list of alternate extensions to try for the file
        other than the one provided in `filename`. For example, to get anything
        that's an .sqlite where .sqlite.gz is expected, use altexts=[''] to
        strip the .gz.

    client : google.cloud.storage.Client instance
        The instance of the Client to use to perform the download operation. If
        this is None, a new Client will be used. If this is None and
        `service_account_json` points to a downloaded JSON file with GCS
        credentials, a new Client with the provided credentials will be used. If
        this is not None, the existing Client instance will be used.

    service_account_json : str
        Path to a downloaded GCS credentials JSON file.

    raiseonfail : bool
        If True, will re-raise whatever Exception caused the operation to fail
        and break out immediately.

    Returns
    -------

    str
        Path to the downloaded filename or None if the download was
        unsuccessful.

    """

    if not client:

        if (service_account_json is not None and
            os.path.exists(service_account_json)):
            client = storage.Client.from_service_account_json(
                service_account_json
            )
        else:
            client = storage.Client()

    try:

        bucket = client.get_bucket(bucketname)
        blob = bucket.get_blob(filename)
        blob.download_to_filename(local_file)
        return local_file

    except Exception as e:

        for alt_extension in altexts:

            split_ext = os.path.splitext(filename)
            check_file = split_ext[0] + alt_extension
            try:
                bucket = client.get_bucket(bucket)
                blob = bucket.get_blob(check_file)
                blob.download_to_filename(
                    local_file.replace(split_ext[-1],
                                       alt_extension)
                )
                return local_file.replace(split_ext[-1],
                                          alt_extension)
            except Exception as e:
                pass

    else:

        LOGEXCEPTION('could not download gs://%s/%s' % (bucket, filename))

        if raiseonfail:
            raise

        return None

Response: This function downloads a single file from a Google Cloud Storage bucket.


Instruction:
Given the following Python 3 function, write the documentation
def read_hatlc(hatlc):
    '''
    This reads a consolidated HAT LC written by the functions above.

    Returns a dict.

    '''

    lcfname = os.path.basename(hatlc)

    # unzip the files first
    if '.gz' in lcfname:
        lcf = gzip.open(hatlc,'rb')
    elif '.bz2' in lcfname:
        lcf = bz2.BZ2File(hatlc, 'rb')
    else:
        lcf = open(hatlc,'rb')

    if '.fits' in lcfname and HAVEPYFITS:

        hdulist = pyfits.open(lcf)
        objectinfo = hdulist[0].header
        objectlc = hdulist[1].data
        lccols = objectlc.columns.names
        hdulist.close()
        lcf.close()

        lcdict = {}

        for col in lccols:
            lcdict[col] = np.array(objectlc[col])

        lcdict['hatid'] = objectinfo['hatid']
        lcdict['twomassid'] = objectinfo['2massid']
        lcdict['ra'] = objectinfo['ra']
        lcdict['dec'] = objectinfo['dec']
        lcdict['mags'] = [objectinfo[x] for x in ('vmag','rmag','imag',
                                                  'jmag','hmag','kmag')]
        lcdict['ndet'] = objectinfo['ndet']
        lcdict['hatstations'] = objectinfo['hats']
        lcdict['filters'] = objectinfo['filters']
        lcdict['columns'] = lccols

        return lcdict

    elif '.fits' in lcfname and not HAVEPYFITS:

        print("can't read %s since we don't have the pyfits module" % lcfname)
        return

    elif '.csv' in lcfname or '.hatlc' in lcfname:

        lcflines = lcf.read().decode().split('\n')
        lcf.close()

        # now process the read-in LC
        objectdata = [x for x in lcflines if x.startswith('#')]
        objectlc = [x for x in lcflines if not x.startswith('#')]
        objectlc = [x for x in objectlc if len(x) > 1]

        if '.csv' in lcfname:
            objectlc = [x.split(',') for x in objectlc]
        else:
            objectlc = [x.split() for x in objectlc]

        # transpose split rows to get columns
        objectlc = list(zip(*objectlc))

        # read the header to figure out the object's info and column names
        objectdata = [x.strip('#') for x in objectdata]
        objectdata = [x.strip() for x in objectdata]
        objectdata = [x for x in objectdata if len(x) > 0]

        hatid, twomassid = objectdata[0].split(' - ')
        ra, dec = objectdata[1].split(', ')
        ra = float(ra.split(' = ')[-1].strip(' deg'))
        dec = float(dec.split(' = ')[-1].strip(' deg'))

        vmag, rmag, imag, jmag, hmag, kmag = objectdata[2].split(', ')
        vmag = float(vmag.split(' = ')[-1])
        rmag = float(rmag.split(' = ')[-1])
        imag = float(imag.split(' = ')[-1])
        jmag = float(jmag.split(' = ')[-1])
        hmag = float(hmag.split(' = ')[-1])
        kmag = float(kmag.split(' = ')[-1])

        ndet = int(objectdata[3].split(': ')[-1])
        hatstations = objectdata[4].split(': ')[-1]

        filterhead_ind = objectdata.index('Filters used:')
        columnhead_ind = objectdata.index('Columns:')

        filters = objectdata[filterhead_ind:columnhead_ind]

        columndefs = objectdata[columnhead_ind+1:]

        columns = []
        for line in columndefs:

            colnum, colname, coldesc = line.split(' - ')
            columns.append(colname)

        lcdict = {}

        # now write all the columns to the output dictionary
        for ind, col in enumerate(columns):

            # this formats everything nicely using our existing column
            # definitions
            lcdict[col] = np.array([TEXTLC_OUTPUT_COLUMNS[col][3](x)
                                    for x in objectlc[ind]])

        # write the object metadata to the output dictionary
        lcdict['hatid'] = hatid
        lcdict['twomassid'] = twomassid.replace('2MASS J','')
        lcdict['ra'] = ra
        lcdict['dec'] = dec
        lcdict['mags'] = [vmag, rmag, imag, jmag, hmag, kmag]
        lcdict['ndet'] = ndet
        lcdict['hatstations'] = hatstations.split(', ')
        lcdict['filters'] = filters[1:]
        lcdict['cols'] = columns

        return lcdict

Response: Reads a consolidated HAT LC file and returns a dict.


Instruction:
Create a Python 3 function to
read a pickled LC file and returns a dictionary of the base class s attributes.

Response:
def read_fakelc(fakelcfile):
    '''
    This just reads a pickled fake LC.

    Parameters
    ----------

    fakelcfile : str
        The fake LC file to read.

    Returns
    -------

    dict
        This returns an lcdict.

    '''

    try:
        with open(fakelcfile,'rb') as infd:
            lcdict = pickle.load(infd)
    except UnicodeDecodeError:
        with open(fakelcfile,'rb') as infd:
            lcdict = pickle.load(infd, encoding='latin1')

    return lcdict


Instruction:
Implement a Python 3 function for
calculating the precision of the base log file.

Response:
def precision(ntp, nfp):
    '''
    This calculates precision.

    https://en.wikipedia.org/wiki/Precision_and_recall

    Parameters
    ----------

    ntp : int
        The number of true positives.

    nfp : int
        The number of false positives.

    Returns
    -------

    float
        The precision calculated using `ntp/(ntp + nfp)`.

    '''

    if (ntp+nfp) > 0:
        return ntp/(ntp+nfp)
    else:
        return np.nan


Instruction:
How would you implement a function in Python 3 that
calculates the recall of a single node in a sequence.

Response:
def recall(ntp, nfn):
    '''
    This calculates recall.

    https://en.wikipedia.org/wiki/Precision_and_recall

    Parameters
    ----------

    ntp : int
        The number of true positives.

    nfn : int
        The number of false negatives.

    Returns
    -------

    float
        The precision calculated using `ntp/(ntp + nfn)`.

    '''

    if (ntp+nfn) > 0:
        return ntp/(ntp+nfn)
    else:
        return np.nan


Instruction:
Can you implement a function in Python 3 that
calculates the Matthews correlation coefficient of two sets of true positives and false negatives.

Response:
def matthews_correl_coeff(ntp, ntn, nfp, nfn):
    '''
    This calculates the Matthews correlation coefficent.

    https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

    Parameters
    ----------

    ntp : int
        The number of true positives.

    ntn : int
        The number of true negatives

    nfp : int
        The number of false positives.

    nfn : int
        The number of false negatives.

    Returns
    -------

    float
        The Matthews correlation coefficient.

    '''

    mcc_top = (ntp*ntn - nfp*nfn)
    mcc_bot = msqrt((ntp + nfp)*(ntp + nfn)*(ntn + nfp)*(ntn + nfn))

    if mcc_bot > 0:
        return mcc_top/mcc_bot
    else:
        return np.nan


Instruction:
Can you generate the documentation for the following Python 3 function
def get_recovered_variables_for_magbin(simbasedir,
                                       magbinmedian,
                                       stetson_stdev_min=2.0,
                                       inveta_stdev_min=2.0,
                                       iqr_stdev_min=2.0,
                                       statsonly=True):
    '''This runs variability selection for the given magbinmedian.

    To generate a full recovery matrix over all magnitude bins, run this
    function for each magbin over the specified stetson_stdev_min and
    inveta_stdev_min grid.

    Parameters
    ----------

    simbasedir : str
        The input directory of fake LCs.

    magbinmedian : float
        The magbin to run the variable recovery for. This is an item from the
        dict from `simbasedir/fakelcs-info.pkl: `fakelcinfo['magrms'][magcol]`
        list for each magcol and designates which magbin to get the recovery
        stats for.

    stetson_stdev_min : float
        The minimum sigma above the trend in the Stetson J variability index
        distribution for this magbin to use to consider objects as variable.

    inveta_stdev_min : float
        The minimum sigma above the trend in the 1/eta variability index
        distribution for this magbin to use to consider objects as variable.

    iqr_stdev_min : float
        The minimum sigma above the trend in the IQR variability index
        distribution for this magbin to use to consider objects as variable.

    statsonly : bool
        If this is True, only the final stats will be returned. If False, the
        full arrays used to generate the stats will also be returned.

    Returns
    -------

    dict
        The returned dict contains statistics for this magbin and if requested,
        the full arrays used to calculate the statistics.

    '''

    # get the info from the simbasedir
    with open(os.path.join(simbasedir, 'fakelcs-info.pkl'),'rb') as infd:
        siminfo = pickle.load(infd)

    objectids = siminfo['objectid']
    varflags = siminfo['isvariable']
    sdssr = siminfo['sdssr']

    # get the column defs for the fakelcs
    timecols = siminfo['timecols']
    magcols = siminfo['magcols']
    errcols = siminfo['errcols']

    # register the fakelc pklc as a custom lcproc format
    # now we should be able to use all lcproc functions correctly
    fakelc_formatkey = 'fake-%s' % siminfo['lcformat']
    lcproc.register_lcformat(
        fakelc_formatkey,
        '*-fakelc.pkl',
        timecols,
        magcols,
        errcols,
        'astrobase.lcproc',
        '_read_pklc',
        magsarefluxes=siminfo['magsarefluxes']
    )

    # make the output directory if it doesn't exit
    outdir = os.path.join(simbasedir, 'recvar-threshold-pkls')
    if not os.path.exists(outdir):
        os.mkdir(outdir)

    # run the variability search
    varfeaturedir = os.path.join(simbasedir, 'varfeatures')
    varthreshinfof = os.path.join(
        outdir,
        'varthresh-magbinmed%.2f-stet%.2f-inveta%.2f.pkl' % (magbinmedian,
                                                             stetson_stdev_min,
                                                             inveta_stdev_min)
    )
    varthresh = varthreshold.variability_threshold(
        varfeaturedir,
        varthreshinfof,
        lcformat=fakelc_formatkey,
        min_stetj_stdev=stetson_stdev_min,
        min_inveta_stdev=inveta_stdev_min,
        min_iqr_stdev=iqr_stdev_min,
        verbose=False
    )

    # get the magbins from the varthresh info
    magbins = varthresh['magbins']

    # get the magbininds
    magbininds = np.digitize(sdssr, magbins)

    # bin the objects according to these magbins
    binned_objectids = []
    binned_actualvars = []
    binned_actualnotvars = []

    # go through all the mag bins and bin up the objectids, actual variables,
    # and actual not-variables
    for mbinind, _magi in zip(np.unique(magbininds),
                              range(len(magbins)-1)):

        thisbinind = np.where(magbininds == mbinind)

        thisbin_objectids = objectids[thisbinind]
        thisbin_varflags = varflags[thisbinind]

        thisbin_actualvars = thisbin_objectids[thisbin_varflags]
        thisbin_actualnotvars = thisbin_objectids[~thisbin_varflags]

        binned_objectids.append(thisbin_objectids)
        binned_actualvars.append(thisbin_actualvars)
        binned_actualnotvars.append(thisbin_actualnotvars)

    # this is the output dict
    recdict = {
        'simbasedir':simbasedir,
        'timecols':timecols,
        'magcols':magcols,
        'errcols':errcols,
        'magsarefluxes':siminfo['magsarefluxes'],
        'stetj_min_stdev':stetson_stdev_min,
        'inveta_min_stdev':inveta_stdev_min,
        'iqr_min_stdev':iqr_stdev_min,
        'magbinmedian':magbinmedian,
    }

    # now, for each magcol, find the magbin corresponding to magbinmedian, and
    # get its stats
    for magcol in magcols:

        # this is the index of the matching magnitude bin for the magbinmedian
        # provided
        magbinind = np.where(
            np.array(varthresh[magcol]['binned_sdssr_median']) == magbinmedian
        )

        magbinind = np.asscalar(magbinind[0])

        # get the objectids, actual vars and actual notvars in this magbin
        thisbin_objectids = binned_objectids[magbinind]
        thisbin_actualvars = binned_actualvars[magbinind]
        thisbin_actualnotvars = binned_actualnotvars[magbinind]

        # stetson recovered variables in this magbin
        stet_recoveredvars = varthresh[magcol][
            'binned_objectids_thresh_stetsonj'
        ][magbinind]

        # calculate TP, FP, TN, FN
        stet_recoverednotvars = np.setdiff1d(thisbin_objectids,
                                             stet_recoveredvars)

        stet_truepositives = np.intersect1d(stet_recoveredvars,
                                            thisbin_actualvars)
        stet_falsepositives = np.intersect1d(stet_recoveredvars,
                                             thisbin_actualnotvars)
        stet_truenegatives = np.intersect1d(stet_recoverednotvars,
                                            thisbin_actualnotvars)
        stet_falsenegatives = np.intersect1d(stet_recoverednotvars,
                                             thisbin_actualvars)

        # calculate stetson recall, precision, Matthews correl coeff
        stet_recall = recall(stet_truepositives.size,
                             stet_falsenegatives.size)

        stet_precision = precision(stet_truepositives.size,
                                   stet_falsepositives.size)

        stet_mcc = matthews_correl_coeff(stet_truepositives.size,
                                         stet_truenegatives.size,
                                         stet_falsepositives.size,
                                         stet_falsenegatives.size)

        # inveta recovered variables in this magbin
        inveta_recoveredvars = varthresh[magcol][
            'binned_objectids_thresh_inveta'
        ][magbinind]
        inveta_recoverednotvars = np.setdiff1d(thisbin_objectids,
                                               inveta_recoveredvars)

        inveta_truepositives = np.intersect1d(inveta_recoveredvars,
                                              thisbin_actualvars)
        inveta_falsepositives = np.intersect1d(inveta_recoveredvars,
                                               thisbin_actualnotvars)
        inveta_truenegatives = np.intersect1d(inveta_recoverednotvars,
                                              thisbin_actualnotvars)
        inveta_falsenegatives = np.intersect1d(inveta_recoverednotvars,
                                               thisbin_actualvars)

        # calculate inveta recall, precision, Matthews correl coeff
        inveta_recall = recall(inveta_truepositives.size,
                               inveta_falsenegatives.size)

        inveta_precision = precision(inveta_truepositives.size,
                                     inveta_falsepositives.size)

        inveta_mcc = matthews_correl_coeff(inveta_truepositives.size,
                                           inveta_truenegatives.size,
                                           inveta_falsepositives.size,
                                           inveta_falsenegatives.size)

        # iqr recovered variables in this magbin
        iqr_recoveredvars = varthresh[magcol][
            'binned_objectids_thresh_iqr'
        ][magbinind]
        iqr_recoverednotvars = np.setdiff1d(thisbin_objectids,
                                            iqr_recoveredvars)

        iqr_truepositives = np.intersect1d(iqr_recoveredvars,
                                           thisbin_actualvars)
        iqr_falsepositives = np.intersect1d(iqr_recoveredvars,
                                            thisbin_actualnotvars)
        iqr_truenegatives = np.intersect1d(iqr_recoverednotvars,
                                           thisbin_actualnotvars)
        iqr_falsenegatives = np.intersect1d(iqr_recoverednotvars,
                                            thisbin_actualvars)

        # calculate iqr recall, precision, Matthews correl coeff
        iqr_recall = recall(iqr_truepositives.size,
                            iqr_falsenegatives.size)

        iqr_precision = precision(iqr_truepositives.size,
                                  iqr_falsepositives.size)

        iqr_mcc = matthews_correl_coeff(iqr_truepositives.size,
                                        iqr_truenegatives.size,
                                        iqr_falsepositives.size,
                                        iqr_falsenegatives.size)

        # calculate the items missed by one method but found by the other
        # methods
        stet_missed_inveta_found = np.setdiff1d(inveta_truepositives,
                                                stet_truepositives)
        stet_missed_iqr_found = np.setdiff1d(iqr_truepositives,
                                             stet_truepositives)

        inveta_missed_stet_found = np.setdiff1d(stet_truepositives,
                                                inveta_truepositives)
        inveta_missed_iqr_found = np.setdiff1d(iqr_truepositives,
                                               inveta_truepositives)

        iqr_missed_stet_found = np.setdiff1d(stet_truepositives,
                                             iqr_truepositives)
        iqr_missed_inveta_found = np.setdiff1d(inveta_truepositives,
                                               iqr_truepositives)

        if not statsonly:

            recdict[magcol] = {
                # stetson J alone
                'stet_recoveredvars':stet_recoveredvars,
                'stet_truepositives':stet_truepositives,
                'stet_falsepositives':stet_falsepositives,
                'stet_truenegatives':stet_truenegatives,
                'stet_falsenegatives':stet_falsenegatives,
                'stet_precision':stet_precision,
                'stet_recall':stet_recall,
                'stet_mcc':stet_mcc,
                # inveta alone
                'inveta_recoveredvars':inveta_recoveredvars,
                'inveta_truepositives':inveta_truepositives,
                'inveta_falsepositives':inveta_falsepositives,
                'inveta_truenegatives':inveta_truenegatives,
                'inveta_falsenegatives':inveta_falsenegatives,
                'inveta_precision':inveta_precision,
                'inveta_recall':inveta_recall,
                'inveta_mcc':inveta_mcc,
                # iqr alone
                'iqr_recoveredvars':iqr_recoveredvars,
                'iqr_truepositives':iqr_truepositives,
                'iqr_falsepositives':iqr_falsepositives,
                'iqr_truenegatives':iqr_truenegatives,
                'iqr_falsenegatives':iqr_falsenegatives,
                'iqr_precision':iqr_precision,
                'iqr_recall':iqr_recall,
                'iqr_mcc':iqr_mcc,
                # true positive variables missed by one method but picked up by
                # the others
                'stet_missed_inveta_found':stet_missed_inveta_found,
                'stet_missed_iqr_found':stet_missed_iqr_found,
                'inveta_missed_stet_found':inveta_missed_stet_found,
                'inveta_missed_iqr_found':inveta_missed_iqr_found,
                'iqr_missed_stet_found':iqr_missed_stet_found,
                'iqr_missed_inveta_found':iqr_missed_inveta_found,
                # bin info
                'actual_variables':thisbin_actualvars,
                'actual_nonvariables':thisbin_actualnotvars,
                'all_objectids':thisbin_objectids,
                'magbinind':magbinind,

            }

        # if statsonly is set, then we only return the numbers but not the
        # arrays themselves
        else:

            recdict[magcol] = {
                # stetson J alone
                'stet_recoveredvars':stet_recoveredvars.size,
                'stet_truepositives':stet_truepositives.size,
                'stet_falsepositives':stet_falsepositives.size,
                'stet_truenegatives':stet_truenegatives.size,
                'stet_falsenegatives':stet_falsenegatives.size,
                'stet_precision':stet_precision,
                'stet_recall':stet_recall,
                'stet_mcc':stet_mcc,
                # inveta alone
                'inveta_recoveredvars':inveta_recoveredvars.size,
                'inveta_truepositives':inveta_truepositives.size,
                'inveta_falsepositives':inveta_falsepositives.size,
                'inveta_truenegatives':inveta_truenegatives.size,
                'inveta_falsenegatives':inveta_falsenegatives.size,
                'inveta_precision':inveta_precision,
                'inveta_recall':inveta_recall,
                'inveta_mcc':inveta_mcc,
                # iqr alone
                'iqr_recoveredvars':iqr_recoveredvars.size,
                'iqr_truepositives':iqr_truepositives.size,
                'iqr_falsepositives':iqr_falsepositives.size,
                'iqr_truenegatives':iqr_truenegatives.size,
                'iqr_falsenegatives':iqr_falsenegatives.size,
                'iqr_precision':iqr_precision,
                'iqr_recall':iqr_recall,
                'iqr_mcc':iqr_mcc,
                # true positive variables missed by one method but picked up by
                # the others
                'stet_missed_inveta_found':stet_missed_inveta_found.size,
                'stet_missed_iqr_found':stet_missed_iqr_found.size,
                'inveta_missed_stet_found':inveta_missed_stet_found.size,
                'inveta_missed_iqr_found':inveta_missed_iqr_found.size,
                'iqr_missed_stet_found':iqr_missed_stet_found.size,
                'iqr_missed_inveta_found':iqr_missed_inveta_found.size,
                # bin info
                'actual_variables':thisbin_actualvars.size,
                'actual_nonvariables':thisbin_actualnotvars.size,
                'all_objectids':thisbin_objectids.size,
                'magbinind':magbinind,
            }

    #
    # done with per magcol
    #

    return recdict

Response: This function runs variability selection for the given magbin and returns a dict containing the statistics for that variable.


Instruction:
Given the following Python 3 function, write the documentation
def plot_varind_gridsearch_magbin_results(gridsearch_results):
    '''This plots the gridsearch results from `variable_index_gridsearch_magbin`.

    Parameters
    ----------

    gridsearch_results : dict
        This is the dict produced by `variable_index_gridsearch_magbin` above.

    Returns
    -------

    dict
        The returned dict contains filenames of the recovery rate plots made for
        each variability index. These include plots of the precision, recall,
        and Matthews Correlation Coefficient over each magbin and a heatmap of
        these values over the grid points of the variability index stdev values
        arrays used.

    '''

    # get the result pickle/dict
    if (isinstance(gridsearch_results, str) and
        os.path.exists(gridsearch_results)):

        with open(gridsearch_results,'rb') as infd:
            gridresults = pickle.load(infd)

    elif isinstance(gridsearch_results, dict):

        gridresults = gridsearch_results

    else:
        LOGERROR('could not understand the input '
                 'variable index grid-search result dict/pickle')
        return None

    plotres = {'simbasedir':gridresults['simbasedir']}

    recgrid = gridresults['recovery']
    simbasedir = gridresults['simbasedir']

    for magcol in gridresults['magcols']:

        plotres[magcol] = {'best_stetsonj':[],
                           'best_inveta':[],
                           'best_iqr':[],
                           'magbinmedians':gridresults['magbinmedians']}

        # go through all the magbins
        for magbinind, magbinmedian in enumerate(gridresults['magbinmedians']):

            LOGINFO('plotting results for %s: magbin: %.3f' %
                    (magcol, magbinmedian))

            stet_mcc = np.array(
                [x[magcol]['stet_mcc']
                 for x in recgrid[magbinind]]
            )[::(gridresults['inveta_grid'].size *
                 gridresults['stetson_grid'].size)]
            stet_precision = np.array(
                [x[magcol]['stet_precision']
                 for x in recgrid[magbinind]]
            )[::(gridresults['inveta_grid'].size *
                 gridresults['stetson_grid'].size)]
            stet_recall = np.array(
                [x[magcol]['stet_recall']
                 for x in recgrid[magbinind]]
            )[::(gridresults['inveta_grid'].size *
                 gridresults['stetson_grid'].size)]
            stet_missed_inveta_found = np.array(
                [x[magcol]['stet_missed_inveta_found']
                 for x in recgrid[magbinind]]
            )[::(gridresults['inveta_grid'].size *
                 gridresults['stetson_grid'].size)]
            stet_missed_iqr_found = np.array(
                [x[magcol]['stet_missed_iqr_found']
                 for x in recgrid[magbinind]]
            )[::(gridresults['inveta_grid'].size *
                 gridresults['stetson_grid'].size)]

            inveta_mcc = np.array(
                [x[magcol]['inveta_mcc']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    ::gridresults['inveta_grid'].size
                ]
            inveta_precision = np.array(
                [x[magcol]['inveta_precision']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    ::gridresults['inveta_grid'].size
                ]
            inveta_recall = np.array(
                [x[magcol]['inveta_recall']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    ::gridresults['inveta_grid'].size
                ]
            inveta_missed_stet_found = np.array(
                [x[magcol]['inveta_missed_stet_found']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    ::gridresults['inveta_grid'].size
                ]
            inveta_missed_iqr_found = np.array(
                [x[magcol]['inveta_missed_iqr_found']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    ::gridresults['inveta_grid'].size
                ]

            iqr_mcc = np.array(
                [x[magcol]['iqr_mcc']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    :gridresults['inveta_grid'].size
                ]
            iqr_precision = np.array(
                [x[magcol]['iqr_precision']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    :gridresults['inveta_grid'].size
                ]
            iqr_recall = np.array(
                [x[magcol]['iqr_recall']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    :gridresults['inveta_grid'].size
                ]
            iqr_missed_stet_found = np.array(
                [x[magcol]['iqr_missed_stet_found']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    :gridresults['inveta_grid'].size
                ]
            iqr_missed_inveta_found = np.array(
                [x[magcol]['iqr_missed_inveta_found']
                 for x in recgrid[magbinind]]
            )[:(gridresults['iqr_grid'].size *
                gridresults['stetson_grid'].size)][
                    :gridresults['inveta_grid'].size
                ]

            fig = plt.figure(figsize=(6.4*5, 4.8*3))

            # FIRST ROW: stetson J plot

            plt.subplot(3,5,1)
            if np.any(np.isfinite(stet_mcc)):
                plt.plot(gridresults['stetson_grid'],
                         stet_mcc)
                plt.xlabel('stetson J stdev multiplier threshold')
                plt.ylabel('MCC')
                plt.title('MCC for stetson J')
            else:
                plt.text(0.5,0.5,
                         'stet MCC values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,2)
            if np.any(np.isfinite(stet_precision)):
                plt.plot(gridresults['stetson_grid'],
                         stet_precision)
                plt.xlabel('stetson J stdev multiplier threshold')
                plt.ylabel('precision')
                plt.title('precision for stetson J')
            else:
                plt.text(0.5,0.5,
                         'stet precision values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,3)
            if np.any(np.isfinite(stet_recall)):
                plt.plot(gridresults['stetson_grid'],
                         stet_recall)
                plt.xlabel('stetson J stdev multiplier threshold')
                plt.ylabel('recall')
                plt.title('recall for stetson J')
            else:
                plt.text(0.5,0.5,
                         'stet recall values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,4)
            if np.any(np.isfinite(stet_missed_inveta_found)):
                plt.plot(gridresults['stetson_grid'],
                         stet_missed_inveta_found)
                plt.xlabel('stetson J stdev multiplier threshold')
                plt.ylabel('# objects stetson missed but inveta found')
                plt.title('stetson J missed, inveta found')
            else:
                plt.text(0.5,0.5,
                         'stet-missed/inveta-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,5)
            if np.any(np.isfinite(stet_missed_iqr_found)):
                plt.plot(gridresults['stetson_grid'],
                         stet_missed_iqr_found)
                plt.xlabel('stetson J stdev multiplier threshold')
                plt.ylabel('# objects stetson missed but IQR found')
                plt.title('stetson J missed, IQR found')
            else:
                plt.text(0.5,0.5,
                         'stet-missed/IQR-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            # SECOND ROW: inveta plots

            plt.subplot(3,5,6)
            if np.any(np.isfinite(inveta_mcc)):
                plt.plot(gridresults['inveta_grid'],
                         inveta_mcc)
                plt.xlabel('inveta stdev multiplier threshold')
                plt.ylabel('MCC')
                plt.title('MCC for inveta')
            else:
                plt.text(0.5,0.5,
                         'inveta MCC values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,7)
            if np.any(np.isfinite(inveta_precision)):
                plt.plot(gridresults['inveta_grid'],
                         inveta_precision)
                plt.xlabel('inveta stdev multiplier threshold')
                plt.ylabel('precision')
                plt.title('precision for inveta')
            else:
                plt.text(0.5,0.5,
                         'inveta precision values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,8)
            if np.any(np.isfinite(inveta_recall)):
                plt.plot(gridresults['inveta_grid'],
                         inveta_recall)
                plt.xlabel('inveta stdev multiplier threshold')
                plt.ylabel('recall')
                plt.title('recall for inveta')
            else:
                plt.text(0.5,0.5,
                         'inveta recall values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,9)
            if np.any(np.isfinite(inveta_missed_stet_found)):
                plt.plot(gridresults['inveta_grid'],
                         inveta_missed_stet_found)
                plt.xlabel('inveta stdev multiplier threshold')
                plt.ylabel('# objects inveta missed but stetson found')
                plt.title('inveta missed, stetson J found')
            else:
                plt.text(0.5,0.5,
                         'inveta-missed-stet-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,10)
            if np.any(np.isfinite(inveta_missed_iqr_found)):
                plt.plot(gridresults['inveta_grid'],
                         inveta_missed_iqr_found)
                plt.xlabel('inveta stdev multiplier threshold')
                plt.ylabel('# objects inveta missed but IQR found')
                plt.title('inveta missed, IQR found')
            else:
                plt.text(0.5,0.5,
                         'inveta-missed-iqr-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            # THIRD ROW: inveta plots

            plt.subplot(3,5,11)
            if np.any(np.isfinite(iqr_mcc)):
                plt.plot(gridresults['iqr_grid'],
                         iqr_mcc)
                plt.xlabel('IQR stdev multiplier threshold')
                plt.ylabel('MCC')
                plt.title('MCC for IQR')
            else:
                plt.text(0.5,0.5,
                         'IQR MCC values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,12)
            if np.any(np.isfinite(iqr_precision)):
                plt.plot(gridresults['iqr_grid'],
                         iqr_precision)
                plt.xlabel('IQR stdev multiplier threshold')
                plt.ylabel('precision')
                plt.title('precision for IQR')
            else:
                plt.text(0.5,0.5,
                         'IQR precision values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,13)
            if np.any(np.isfinite(iqr_recall)):
                plt.plot(gridresults['iqr_grid'],
                         iqr_recall)
                plt.xlabel('IQR stdev multiplier threshold')
                plt.ylabel('recall')
                plt.title('recall for IQR')
            else:
                plt.text(0.5,0.5,
                         'IQR recall values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,14)
            if np.any(np.isfinite(iqr_missed_stet_found)):
                plt.plot(gridresults['iqr_grid'],
                         iqr_missed_stet_found)
                plt.xlabel('IQR stdev multiplier threshold')
                plt.ylabel('# objects IQR missed but stetson found')
                plt.title('IQR missed, stetson J found')
            else:
                plt.text(0.5,0.5,
                         'iqr-missed-stet-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplot(3,5,15)
            if np.any(np.isfinite(iqr_missed_inveta_found)):
                plt.plot(gridresults['iqr_grid'],
                         iqr_missed_inveta_found)
                plt.xlabel('IQR stdev multiplier threshold')
                plt.ylabel('# objects IQR missed but inveta found')
                plt.title('IQR missed, inveta found')
            else:
                plt.text(0.5,0.5,
                         'iqr-missed-inveta-found values are all nan '
                         'for this magbin',
                         transform=plt.gca().transAxes,
                         horizontalalignment='center',
                         verticalalignment='center')
                plt.xticks([])
                plt.yticks([])

            plt.subplots_adjust(hspace=0.25,wspace=0.25)

            plt.suptitle('magcol: %s, magbin: %.3f' % (magcol, magbinmedian))

            plotdir = os.path.join(gridresults['simbasedir'],
                                   'varindex-gridsearch-plots')
            if not os.path.exists(plotdir):
                os.mkdir(plotdir)

            gridplotf = os.path.join(
                plotdir,
                '%s-magbin-%.3f-var-recoverygrid-permagbin.png' %
                (magcol, magbinmedian)
            )

            plt.savefig(gridplotf,dpi=100,bbox_inches='tight')
            plt.close('all')

            # get the best values of MCC, recall, precision and their associated
            # stet, inveta
            stet_mcc_maxind = np.where(stet_mcc == np.max(stet_mcc))
            stet_precision_maxind = np.where(
                stet_precision == np.max(stet_precision)
            )
            stet_recall_maxind = np.where(stet_recall == np.max(stet_recall))

            best_stet_mcc = stet_mcc[stet_mcc_maxind]
            best_stet_precision = stet_mcc[stet_precision_maxind]
            best_stet_recall = stet_mcc[stet_recall_maxind]

            stet_with_best_mcc = gridresults['stetson_grid'][stet_mcc_maxind]
            stet_with_best_precision = gridresults['stetson_grid'][
                stet_precision_maxind
            ]
            stet_with_best_recall = (
                gridresults['stetson_grid'][stet_recall_maxind]
            )

            inveta_mcc_maxind = np.where(inveta_mcc == np.max(inveta_mcc))
            inveta_precision_maxind = np.where(
                inveta_precision == np.max(inveta_precision)
            )
            inveta_recall_maxind = (
                np.where(inveta_recall == np.max(inveta_recall))
            )

            best_inveta_mcc = inveta_mcc[inveta_mcc_maxind]
            best_inveta_precision = inveta_mcc[inveta_precision_maxind]
            best_inveta_recall = inveta_mcc[inveta_recall_maxind]

            inveta_with_best_mcc = gridresults['inveta_grid'][inveta_mcc_maxind]
            inveta_with_best_precision = gridresults['inveta_grid'][
                inveta_precision_maxind
            ]
            inveta_with_best_recall = gridresults['inveta_grid'][
                inveta_recall_maxind
            ]

            iqr_mcc_maxind = np.where(iqr_mcc == np.max(iqr_mcc))
            iqr_precision_maxind = np.where(
                iqr_precision == np.max(iqr_precision)
            )
            iqr_recall_maxind = (
                np.where(iqr_recall == np.max(iqr_recall))
            )

            best_iqr_mcc = iqr_mcc[iqr_mcc_maxind]
            best_iqr_precision = iqr_mcc[iqr_precision_maxind]
            best_iqr_recall = iqr_mcc[iqr_recall_maxind]

            iqr_with_best_mcc = gridresults['iqr_grid'][iqr_mcc_maxind]
            iqr_with_best_precision = gridresults['iqr_grid'][
                iqr_precision_maxind
            ]
            iqr_with_best_recall = gridresults['iqr_grid'][
                iqr_recall_maxind
            ]

            plotres[magcol][magbinmedian] = {
                # stetson
                'stet_grid':gridresults['stetson_grid'],
                'stet_mcc':stet_mcc,
                'stet_precision':stet_precision,
                'stet_recall':stet_recall,
                'stet_missed_inveta_found':stet_missed_inveta_found,
                'best_stet_mcc':best_stet_mcc,
                'stet_with_best_mcc':stet_with_best_mcc,
                'best_stet_precision':best_stet_precision,
                'stet_with_best_precision':stet_with_best_precision,
                'best_stet_recall':best_stet_recall,
                'stet_with_best_recall':stet_with_best_recall,
                # inveta
                'inveta_grid':gridresults['inveta_grid'],
                'inveta_mcc':inveta_mcc,
                'inveta_precision':inveta_precision,
                'inveta_recall':inveta_recall,
                'inveta_missed_stet_found':inveta_missed_stet_found,
                'best_inveta_mcc':best_inveta_mcc,
                'inveta_with_best_mcc':inveta_with_best_mcc,
                'best_inveta_precision':best_inveta_precision,
                'inveta_with_best_precision':inveta_with_best_precision,
                'best_inveta_recall':best_inveta_recall,
                'inveta_with_best_recall':inveta_with_best_recall,
                # iqr
                'iqr_grid':gridresults['iqr_grid'],
                'iqr_mcc':iqr_mcc,
                'iqr_precision':iqr_precision,
                'iqr_recall':iqr_recall,
                'iqr_missed_stet_found':iqr_missed_stet_found,
                'best_iqr_mcc':best_iqr_mcc,
                'iqr_with_best_mcc':iqr_with_best_mcc,
                'best_iqr_precision':best_iqr_precision,
                'iqr_with_best_precision':iqr_with_best_precision,
                'best_iqr_recall':best_iqr_recall,
                'iqr_with_best_recall':iqr_with_best_recall,
                # plot info
                'recoveryplot':gridplotf
            }

            # recommend inveta, stetson index, and iqr for this magbin

            # if there are multiple stets, choose the smallest one
            if stet_with_best_mcc.size > 1:
                plotres[magcol]['best_stetsonj'].append(stet_with_best_mcc[0])
            elif stet_with_best_mcc.size > 0:
                plotres[magcol]['best_stetsonj'].append(stet_with_best_mcc[0])
            else:
                plotres[magcol]['best_stetsonj'].append(np.nan)

            # if there are multiple best invetas, choose the smallest one
            if inveta_with_best_mcc.size > 1:
                plotres[magcol]['best_inveta'].append(inveta_with_best_mcc[0])
            elif inveta_with_best_mcc.size > 0:
                plotres[magcol]['best_inveta'].append(inveta_with_best_mcc[0])
            else:
                plotres[magcol]['best_inveta'].append(np.nan)

            # if there are multiple best iqrs, choose the smallest one
            if iqr_with_best_mcc.size > 1:
                plotres[magcol]['best_iqr'].append(iqr_with_best_mcc[0])
            elif iqr_with_best_mcc.size > 0:
                plotres[magcol]['best_iqr'].append(iqr_with_best_mcc[0])
            else:
                plotres[magcol]['best_iqr'].append(np.nan)

    # write the plotresults to a pickle
    plotrespicklef = os.path.join(simbasedir,
                                  'varindex-gridsearch-magbin-results.pkl')
    with open(plotrespicklef, 'wb') as outfd:
        pickle.dump(plotres, outfd, pickle.HIGHEST_PROTOCOL)

    # recommend the values of stetson J and inveta to use
    for magcol in gridresults['magcols']:

        LOGINFO('best stdev multipliers for each %s magbin:' % magcol)
        LOGINFO('magbin    inveta    stetson J    IQR')

        for magbin, inveta, stet, iqr in zip(
                plotres[magcol]['magbinmedians'],
                plotres[magcol]['best_inveta'],
                plotres[magcol]['best_stetsonj'],
                plotres[magcol]['best_iqr']):
            LOGINFO('%.3f    %.3f    %.3f    %.3f' % (magbin,
                                                      inveta,
                                                      stet,
                                                      iqr))

    return plotres

Response: This function plots the gridsearch results from variable index gridsearch_magbin.