Instruction:
Can you generate a brief explanation for the following Python 3 code
def list_endpoints(self, query=None):
    '''list all endpoints, providing a list of endpoints to the user to
       better filter the search. This function takes no arguments,
       as the user has not provided an endpoint id or query.
    '''
    bot.info('Please select an endpoint id to query from')

    endpoints = self._get_endpoints(query)

    # Iterate through endpoints to provide user a list

    bot.custom(prefix="Globus", message="Endpoints", color="CYAN")
    rows = []
    for kind,eps in endpoints.items():
        for epid,epmeta in eps.items():
            rows.append([epid, '[%s]' %kind, epmeta['name']])

    bot.table(rows)
    return rows

Response:
list all endpoints, providing a list of endpoints to the user to
       better filter the search. This function takes no arguments,
       as the user has not provided an endpoint id or query.


Instruction:
Write a Python 3 function for
sharing will use the client to get a shareable link for an image of choice.

Response:
def share(self, query, share_to=None):
    '''share will use the client to get a shareable link for an image of choice.
       the functions returns a url of choice to send to a recipient.
    '''

    names = parse_image_name(remove_uri(query))

    # Dropbox path is the path in storage with a slash
    dropbox_path = '/%s' % names['storage']

    # First ensure that exists
    if self.exists(dropbox_path) is True:

        # Create new shared link
        try:
            share = self.dbx.sharing_create_shared_link_with_settings(dropbox_path)

        # Already exists!
        except ApiError as err:
            share = self.dbx.sharing_create_shared_link(dropbox_path)

        bot.info(share.url)
    return share.url


Instruction:
Make a summary of the following Python 3 code
def _set_base(self):
        '''set the API base or default to use Docker Hub. The user is able
           to set the base, api version, and protocol via a settings file
           of environment variables:

           SREGISTRY_NVIDIA_BASE: defaults to nvcr.io
           SREGISTRY_NVIDIA_TOKEN: defaults to $oauthtoken
           SREGISTRY_NVIDIA_VERSION: defaults to v2
           SREGISTRY_NVIDIA_NO_HTTPS: defaults to not set (so https)

        '''
        base = self._get_setting('SREGISTRY_NVIDIA_BASE')
        version = self._get_setting('SREGISTRY_NVIDIA_VERSION')

        if base is None:
            base = "nvcr.io"

        if version is None:
            version = "v2"

        nohttps = self._get_setting('SREGISTRY_NVIDIA_NOHTTPS')
        if nohttps is None:
            nohttps = "https://"
        else:
            nohttps = "http://"

        # <protocol>://<base>/<version>
        self.base = "%s%s/%s" %(nohttps, base.strip('/'), version)

Response: set the base or default to use Docker Hub.


Instruction:
Write a Python 3 script to
update secrets will take a secrets credential file located at. sregistry or the environment variable SREGISTRY_CLIENT_SECRETS and update the current client secrets as well as the API base base.

Response:
def _update_secrets(self):
        '''update secrets will take a secrets credential file
           either located at .sregistry or the environment variable
           SREGISTRY_CLIENT_SECRETS and update the current client
           secrets as well as the associated API base. For the case of
           using Docker Hub, if we find a .docker secrets file, we update
           from there.
        '''
        # If the user has defined secrets, use them
        token = self._required_get_and_update('SREGISTRY_NVIDIA_TOKEN')
        username = self._get_and_update_setting('SREGISTRY_NVIDIA_USERNAME')

        if username is None:
            username = "$oauthtoken"

        # Option 1: the user exports username and password
        if token is not None:
            auth = basic_auth_header(username, token)
            self.headers.update(auth)


Instruction:
Write a Python 3 function that can
pull an image from a Globus endpoint

Response:
def pull(self, images, file_name=None, save=True, **kwargs):
    '''pull an image from a Globus endpoint. The user must have the default
       local endpoint set up. For example:

       6881ae2e-db26-11e5-9772-22000b9da45e:.singularity/shub/sherlock_vep.simg

    Parameters
    ==========
    images: refers to the globus endpoint id and image path.
    file_name: the user's requested name for the file. It can
               optionally be None if the user wants a default.
    save: if True, you should save the container to the database
          using self.add(). For globus this is the only option, and
          we don't have control over when this happens.

    Returns
    =======
    finished: a single container path, or list of paths
    '''

    # Ensure we have a transfer client
    if not hasattr(self, 'transfer_client'):
        self._init_transfer_client()

    if not isinstance(images,list):
        images = [images]

    bot.debug('Execution of PULL for %s images' %len(images))

    finished = []
    for image in images:

        # Split the name into endpoint and rest

        endpoint, remote = self._parse_endpoint_name(image)
        source = self.transfer_client.get_endpoint(endpoint)

        name = os.path.basename(remote)
        q = parse_image_name(name, default_collection=source['name'])

        # The user must have a personal endpoint

        endpoints = self._get_endpoints()

        if len(endpoints['my-endpoints']) == 0:
            bot.error('You must have a personal endpoint to transfer the container')
            sys.exit(1)

        # Take the first endpoint that is active

        dest = None
        for eid,contender in endpoints['my-endpoints'].items():
           if contender['gcp_connected'] is True:
               dest = contender
               break

        # Exit if none are active, required!

        if dest is None:
            bot.error('No activated local endpoints online! Start to transfer')
            sys.exit(1)

        # We need to know the full path of the endpoint

        base = self._get_endpoint_path(dest['id'])
        storage_folder = '%s/%s' %(base, q['collection'])
        self._create_endpoint_folder(dest['id'], storage_folder)

        label = "Singularity Registry Pull"
        tdata = globus_sdk.TransferData(self.transfer_client,
                                        source['id'],
                                        dest['id'],
                                        label=label,
                                        sync_level="checksum")

        image = os.path.join(base, q['storage'])
        tdata.add_item(remote, image)
        bot.info('Requesting transfer to %s' %q['storage'])
        transfer_result = self.transfer_client.submit_transfer(tdata)
        bot.info(transfer_result['message'])
        finished.append(transfer_result)

    if len(finished) == 1:
        finished = finished[0]
    return finished


Instruction:
Can you write a function in Python 3 where it
gets the path to the credential cache for the given client level.

Response:
def get_credential_cache():
    '''if the user has specified settings to provide a cache for credentials
       files, initialize it. The root for the folder is created if it doesn't
       exist. The path for the specific client is returned, and it's
       not assumed to be either a folder or a file (this is up to the
       developer of the client).
    '''
    from sregistry.defaults import ( CREDENTIAL_CACHE, SREGISTRY_CLIENT )

    client_credential_cache = None

    # Check 1: user can disable a credential cache on the client level
    if CREDENTIAL_CACHE is not None:
        env = 'SREGISTRY_DISABLE_CREDENTIAL_%s' %SREGISTRY_CLIENT.upper()
        if os.environ.get(env) is not None:
            bot.debug('[%s] cache disabled' %SREGISTRY_CLIENT)
            CREDENTIAL_CACHE = None

    # Check 2: user can disable a credential cache on the client level
    if CREDENTIAL_CACHE is not None:
        if not os.path.exists(CREDENTIAL_CACHE):
            mkdir_p(CREDENTIAL_CACHE)
        client_credential_cache = '%s/%s' %(CREDENTIAL_CACHE, SREGISTRY_CLIENT)
    if client_credential_cache is not None:
        bot.debug('credentials cache')
    return client_credential_cache


Instruction:
Can you generate a brief explanation for the following Python 3 code
def update_client_secrets(backend, updates, secrets=None, save=True):
    '''update client secrets will update the data structure for a particular
       authentication. This should only be used for a (quasi permanent) token
       or similar. The secrets file, if found, is updated and saved by default.
    '''
    if secrets is None:
        secrets = read_client_secrets()
    if backend not in secrets:
        secrets[backend] = {}
    secrets[backend].update(updates)

    # The update typically includes a save
    if save is True:
        secrets_file = get_secrets_file()
        if secrets_file is not None:
            write_json(secrets,secrets_file)

    return secrets

Response: update client secrets for a particular backend


Instruction:
Can you tell what is the following Python 3 function doing
def read_client_secrets():
    '''for private or protected registries, a client secrets file is required
       to be located at .sregistry. If no secrets are found, we use default
       of Singularity Hub, and return a dummy secrets.
    '''
    client_secrets = _default_client_secrets()

    # If token file not provided, check environment
    secrets = get_secrets_file()

    # If exists, load
    if secrets is not None:
        client_secrets = read_json(secrets)

    # Otherwise, initialize
    else:
        from sregistry.defaults import SREGISTRY_CLIENT_SECRETS
        write_json(client_secrets, SREGISTRY_CLIENT_SECRETS)

    return client_secrets

Response: Read client secrets from the secrets file.


Instruction:
Here you have a function in Python 3, explain what it does
def _init_client(self):
        '''init client will check if the user has defined a bucket that
           differs from the default, use the application credentials to
           get the bucket, and then instantiate the client.
        '''

        # Get storage and compute services
        self._get_services()

        env = 'SREGISTRY_GOOGLE_STORAGE_BUCKET'
        self._bucket_name = self._get_and_update_setting(env)

        # If the user didn't set in environment, use default
        if self._bucket_name is None:
            self._bucket_name = 'sregistry-%s' %os.environ['USER']

        self._get_bucket()

Response: init client will check if the user has defined a bucket that has changed from the default


Instruction:
How would you explain what the following Python 3 function does
def _get_services(self, version='v1'):
        '''get version 1 of the google compute and storage service

        Parameters
        ==========
        version: version to use (default is v1)
        '''
        self._bucket_service = storage.Client()
        creds = GoogleCredentials.get_application_default()
        self._storage_service = discovery_build('storage', version, credentials=creds)
        self._compute_service = discovery_build('compute', version, credentials=creds)

Response: get version 1 of the google compute and storage service


Instruction:
Write a Python 3 script to
get a bucket based on a bucket name. create it if it doesn t exist.

Response:
def _get_bucket(self):
        '''get a bucket based on a bucket name. If it doesn't exist, create it.
        '''

        # Case 1: The bucket already exists
        try:
            self._bucket = self._bucket_service.get_bucket(self._bucket_name)

        # Case 2: The bucket needs to be created
        except google.cloud.exceptions.NotFound:
            self._bucket = self._bucket_service.create_bucket(self._bucket_name)

        # Case 3: The bucket name is already taken
        except:
            bot.error('Cannot get or create %s' %self._bucket_name)
            sys.exit(1)

        return self._bucket


Instruction:
How would you explain what the following Python 3 function does
def delete_object(service, bucket_name, object_name):
    '''delete object will delete a file from a bucket

       Parameters
       ==========
       storage_service: the service obtained with get_storage_service
       bucket_name: the name of the bucket
       object_name: the "name" parameter of the object.

    '''
    try:
        operation = service.objects().delete(bucket=bucket_name,
                                             object=object_name).execute()
    except HttpError as e:
        pass
        operation = e
    return operation

Response: delete object from a bucket


Instruction:
Create a Python 3 function to
delete an image from Google Storage.

Response:
def delete(self, name):
    '''delete an image from Google Storage.

       Parameters
       ==========
       name: the name of the file (or image) to delete

    '''

    bot.debug("DELETE %s" % name)

    for file_object in files:
        if isinstance(file_object, dict):
            if "kind" in file_object:
                if file_object['kind'] == "storage#object":
                    object_name = "/".join(file_object['id'].split('/')[:-1])
                    object_name = re.sub('%s/' %self._bucket['name'],'', object_name,1)

                    delete_object(service=self._bucket_service,
                                  bucket_name=bucket['name'],
                                  object_name=object_name)


Instruction:
Can you generate the documentation for the following Python 3 function
def destroy(self, name):
    '''destroy an instance, meaning take down the instance and stop the build.

       Parameters
       ==========
       name: the name of the instance to stop building.
    '''

    instances = self._get_instances()
    project = self._get_project()
    zone = self._get_zone()
    found = False

    if 'items' in instances:
        for instance in instances['items']:
            if instance['name'] == name:
                found = True
                break

    if found:
        bot.info('Killing instance %s' %name)
        return self._compute_service.instances().delete(project=project,
                                                        zone=zone,
                                                        instance=name).execute()

Response:
destroy an instance, meaning take down the instance and stop the build.

       Parameters
       ==========
       name: the name of the instance to stop building.


Instruction:
Explain what the following Python 3 code does
def get_subparsers(parser):
    '''get_subparser will get a dictionary of subparsers, to help with printing help
    '''

    actions = [action for action in parser._actions
               if isinstance(action, argparse._SubParsersAction)]

    subparsers = dict()
    for action in actions:
        # get all subparsers and print help
        for choice, subparser in action.choices.items():
            subparsers[choice] = subparser

    return subparsers

Response: get_subparser will get a dictionary of subparsers to help with printing help


Instruction:
Can you generate the documentation for the following Python 3 function
def generate(self, delim='-', length=4, chars='0123456789'):
        '''
        Generate a robot name. Inspiration from Haikunator, but much more
                 poorly implemented ;)

        Parameters
        ==========
        delim: Delimiter
        length: TokenLength
        chars: TokenChars
        '''

        descriptor = self._select(self._descriptors)
        noun = self._select(self._nouns)
        numbers = ''.join((self._select(chars) for _ in range(length)))
        return delim.join([descriptor, noun, numbers])

Response:
Generate a robot name. Inspiration from Haikunator but much more
                 poorly implemented ; )


Instruction:
Make a summary of the following Python 3 code
def mkdir_p(path):
    '''mkdir_p attempts to get the same functionality as mkdir -p
    :param path: the path to create.
    '''
    try:
        os.makedirs(path)
    except OSError as e:
        if e.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else:
            bot.error("Error creating path %s, exiting." % path)
            sys.exit(1)

Response: mkdir - p attempts to get the same functionality as mkdir - p


Instruction:
Can you generate the documentation for the following Python 3 function
def get_tmpfile(requested_tmpdir=None, prefix=""):
    '''get a temporary file with an optional prefix. By default will be
       created in /tmp unless SREGISTRY_TMPDIR is set. By default, the file
       is closed (and just a name returned).

       Parameters
       ==========
       requested_tmpdir: an optional requested temporary directory, first
       priority as is coming from calling function.
       prefix: Given a need for a sandbox (or similar), prefix the file
       with this string.
    '''

    # First priority for the base goes to the user requested.
    tmpdir = get_tmpdir(requested_tmpdir)

    # If tmpdir is set, add to prefix
    if tmpdir is not None:
        prefix = os.path.join(tmpdir, os.path.basename(prefix))

    fd, tmp_file = tempfile.mkstemp(prefix=prefix)
    os.close(fd)

    return tmp_file

Response: get a temporary file with an optional prefix. By default the file is created in the default directory. By default the file is returned. By default the file is returned. By default the file is returned. By default the file is returned. By default the file is returned. By default the file is returned.


Instruction:
Write a Python 3 script to
get a temporary directory for an operation.

Response:
def get_tmpdir(requested_tmpdir=None, prefix="", create=True):
    '''get a temporary directory for an operation. If SREGISTRY_TMPDIR
       is set, return that. Otherwise, return the output of tempfile.mkdtemp

       Parameters
       ==========
       requested_tmpdir: an optional requested temporary directory, first
       priority as is coming from calling function.
       prefix: Given a need for a sandbox (or similar), we will need to
       create a subfolder *within* the SREGISTRY_TMPDIR.
       create: boolean to determine if we should create folder (True)
    '''
    from sregistry.defaults import SREGISTRY_TMPDIR

    # First priority for the base goes to the user requested.
    tmpdir = requested_tmpdir or SREGISTRY_TMPDIR

    prefix = prefix or "sregistry-tmp"
    prefix = "%s.%s" %(prefix, next(tempfile._get_candidate_names()))
    tmpdir = os.path.join(tmpdir, prefix)

    if not os.path.exists(tmpdir) and create is True:
        os.mkdir(tmpdir)

    return tmpdir


Instruction:
How would you code a function in Python 3 to
extract a tar archive to a specified output folder

Response:
def extract_tar(archive, output_folder, handle_whiteout=False):
    '''extract a tar archive to a specified output folder

        Parameters
        ==========
        archive: the archive file to extract
        output_folder: the output folder to extract to
        handle_whiteout: use docker2oci variation to handle whiteout files

    '''
    from .terminal import run_command

    # Do we want to remove whiteout files?
    if handle_whiteout is True:
        return _extract_tar(archive, output_folder)

    # If extension is .tar.gz, use -xzf
    args = '-xf'
    if archive.endswith(".tar.gz"):
        args = '-xzf'

    # Just use command line, more succinct.
    command = ["tar", args, archive, "-C", output_folder, "--exclude=dev/*"]
    if not bot.is_quiet():
        print("Extracting %s" % archive)

    return run_command(command)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _extract_tar(archive, output_folder):
    '''use blob2oci to handle whiteout files for extraction. Credit for this
       script goes to docker2oci by Olivier Freyermouth, and see script
       folder for license.

       Parameters
       ==========
       archive: the archive to extract
       output_folder the output folder (sandbox) to extract to

    '''
    from .terminal import ( run_command, which )

    result = which('blob2oci')
    if result['return_code'] != 0:
        bot.error('Cannot find blob2oci script on path, exiting.')
        sys.exit(1)

    script = result['message']
    command = ['exec' ,script, '--layer', archive, '--extract', output_folder]

    if not bot.is_quiet():
        print("Extracting %s" % archive)

    return run_command(command)

Response: extract a tar archive


Instruction:
Make a summary of the following Python 3 code
def get_file_hash(filename):
    '''find the SHA256 hash string of a file
    '''
    hasher = hashlib.sha256()
    with open(filename, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hasher.update(chunk)
    return hasher.hexdigest()

Response: find the SHA256 hash string of a file


Instruction:
Can you tell what is the following Python 3 function doing
def write_json(json_obj, filename, mode="w", print_pretty=True):
    '''write_json will (optionally,pretty print) a json object to file

       Parameters
       ==========
       json_obj: the dict to print to json
       filename: the output file to write to
       pretty_print: if True, will use nicer formatting
    '''
    with open(filename, mode) as filey:
        if print_pretty:
            filey.writelines(print_json(json_obj))
        else:
            filey.writelines(json.dumps(json_obj))
    return filename

Response: write_json will print a json object to file


Instruction:
How would you code a function in Python 3 to
read in a json file and returns the data structure as dict.

Response:
def read_json(filename, mode='r'):
    '''read_json reads in a json file and returns
       the data structure as dict.
    '''
    with open(filename, mode) as filey:
        data = json.load(filey)
    return data


Instruction:
Implement a function in Python 3 to
clean up will delete a list of files only if they exist

Response:
def clean_up(files):
    '''clean up will delete a list of files, only if they exist
    '''
    if not isinstance(files, list):
        files = [files]

    for f in files:
        if os.path.exists(f):
            bot.verbose3("Cleaning up %s" % f)
            os.remove(f)


Instruction:
Can you create a Python 3 function that
pulls an image from a docker hub

Response:
def pull(self, images, file_name=None, save=True, force=False, **kwargs):
    '''pull an image from a docker hub. This is a (less than ideal) workaround
       that actually does the following:

       - creates a sandbox folder
       - adds docker layers, metadata folder, and custom metadata to it
       - converts to a squashfs image with build

    the docker manifests are stored with registry metadata.

    Parameters
    ==========
    images: refers to the uri given by the user to pull in the format
    <collection>/<namespace>. You should have an API that is able to
    retrieve a container based on parsing this uri.
    file_name: the user's requested name for the file. It can
               optionally be None if the user wants a default.
    save: if True, you should save the container to the database
          using self.add()

    Returns
    =======
    finished: a single container path, or list of paths
    '''

    if not isinstance(images,list):
        images = [images]

    bot.debug('Execution of PULL for %s images' %len(images))

    # If used internally we want to return a list to the user.

    finished = []
    for image in images:

        q = parse_image_name( remove_uri(image),
                              default_collection='aws' )

        image_file = self._pull(file_name=file_name,
                                save=save,
                                force=force,
                                names=q,
                                kwargs=kwargs)

        finished.append(image_file)

    if len(finished) == 1:
        finished = finished[0]
    return finished


Instruction:
How would you implement a function in Python 3 that
pulls an image from a singularity registry

Response:
def pull(self, images, file_name=None, save=True, **kwargs):
    '''pull an image from a singularity registry

    Parameters
    ==========
    images: refers to the uri given by the user to pull in the format
    <collection>/<namespace>. You should have an API that is able to
    retrieve a container based on parsing this uri.
    file_name: the user's requested name for the file. It can
               optionally be None if the user wants a default.
    save: if True, you should save the container to the database
          using self.add()

    Returns
    =======
    finished: a single container path, or list of paths
    '''

    # Here we take an entire list or a single image by ensuring we have a list
    # This makes the client flexible to command line or internal function use,
    # for one or more images.
    if not isinstance(images,list):
        images = [images]

    bot.debug('Execution of PULL for %s images' %len(images))

    # If used internally we want to return a list to the user.
    finished = []
    for image in images:

        q = parse_image_name(remove_uri(image))

        # Verify image existence, and obtain id
        url = "..." # write your custom endpoint URL here
        bot.debug('Retrieving manifest at %s' %url)

        # You can use the client get function to retrieve a url manifest
        manifest = self._get(url)

        # it's good practice to add the url as a `selfLink`
        manifest['selfLink'] = url

        # Make sure to parse the response (manifest) in case it's not what
        # you expect!

        # If the user didn't provide a file, make one based on the names
        if file_name is None:
            file_name = q['storage'].replace('/','-')

        # You can then use the client download function to get the url
        # for some image in your manifest. In this example, it's in the `image`
        # field and we want to show the progress bar.
        image_file = self.download(url=manifest['image'],
                                   file_name=file_name,
                                   show_progress=True)

        # If the user is saving to local storage, you need to assumble the uri
        # here in the expected format <collection>/<namespace>:<tag>@<version>
        if save is True:
            image_uri = "%s/%s:%s" %(manifest['collection'],
                                     manifest['name'],
                                     manifest['tag'])

            # Importantly, the client add function will take the image file, the
            # uri, the download link, and any relevant metadata (dictionary)
            # for the database
            container = self.add(image_path = image_file, # the file path
                                 image_uri = image_uri,   # the full uri
                                 image_name = file_name, # a custom name?
                                 metadata = manifest,
                                 url = manifest['image'])

            # When the container is created, this is the path to the image
            image_file = container.image

        if os.path.exists(image_file):
            bot.debug('Retrieved image file %s' %image_file)
            bot.custom(prefix="Success!", message=image_file)
            finished.append(image_file)

    if len(finished) == 1:
        finished = finished[0]
    return finished


Instruction:
Write a Python 3 script for
pushing an image to an S3 endpoint

Response:
def push(self, path, name, tag=None):
    '''push an image to an S3 endpoint'''

    path = os.path.abspath(path)
    image = os.path.basename(path)
    bot.debug("PUSH %s" % path)

    if not os.path.exists(path):
        bot.error('%s does not exist.' %path)
        sys.exit(1)

    # Extract the metadata
    names = parse_image_name(remove_uri(name), tag=tag)
    image_size = os.path.getsize(path) >> 20

    # Create extra metadata, this is how we identify the image later
    # *important* bug in boto3 will return these capitalized
    # see https://github.com/boto/boto3/issues/1709
    metadata = {'sizemb': "%s" % image_size,
                'client': 'sregistry' }

    self.bucket.upload_file(path, names['storage_uri'], {"Metadata": metadata })


Instruction:
Can you write a function in Python 3 where it
gets a collection if it exists create it first.

Response:
def get_or_create_collection(self, name):
    '''get a collection if it exists. If it doesn't exist, create it first.

    Parameters
    ==========
    name: the collection name, usually parsed from get_image_names()['name']

    '''
    from sregistry.database.models import Collection
    collection = self.get_collection(name)

    # If it doesn't exist, create it
    if collection is None:
        collection = Collection(name=name)
        self.session.add(collection)
        self.session.commit()

    return collection


Instruction:
Implement a Python 3 function for
getting a collection if it exists otherwise return None

Response:
def get_collection(self, name):
    '''get a collection, if it exists, otherwise return None.
    '''
    from sregistry.database.models import Collection
    return Collection.query.filter(Collection.name == name).first()


Instruction:
Given the following Python 3 function, write the documentation
def get_container(self, name, collection_id, tag="latest", version=None):
    '''get a container, otherwise return None.
    '''
    from sregistry.database.models import Container
    if version is None:
        container = Container.query.filter_by(collection_id = collection_id,
                                              name = name,
                                              tag = tag).first()
    else:
        container = Container.query.filter_by(collection_id = collection_id,
                                              name = name,
                                              tag = tag,
                                              version = version).first()
    return container

Response: get a container if it exists otherwise return None.


Instruction:
Can you generate the documentation for the following Python 3 function
def get(self, name, quiet=False):
    '''Do a get for a container, and then a collection, and then return None
       if no result is found.
    Parameters
    ==========
    name: should coincide with either the collection name, or the container
          name with the collection. A query is done first for the collection,
          and then the container, and the path to the image file returned.
    '''
    from sregistry.database.models import Collection, Container
    names = parse_image_name( remove_uri (name) )

    # First look for a collection (required)
    collection = self.get_collection(name=names['collection'])
    container = None

    if collection is not None:
        container = self.get_container(collection_id=collection.id,
                                       name=names['image'],
                                       tag=names['tag'],
                                       version=names['version'])

        if container is not None and quiet is False:

            # The container image file exists [local]
            if container.image is not None:
                print(container.image)

            # The container has a url (but not local file)
            elif container.url is not None:
                print(container.url)
            else:
                bot.info('No storage file found for %s' %name)

    return container

Response: Do a get for a container and then a collection and then a file.


Instruction:
Here you have a function in Python 3, explain what it does
def images(self, query=None):
    '''List local images in the database, optionally with a query.

       Paramters
       =========
       query: a string to search for in the container or collection name|tag|uri

    '''
    from sregistry.database.models import Collection, Container

    rows = []
    if query is not None:
        like = "%" + query + "%"
        containers = Container.query.filter(or_(Container.name == query,
                                                Container.tag.like(like),
                                                Container.uri.like(like),
                                                Container.name.like(like))).all()
    else:
        containers = Container.query.all()

    if len(containers) > 0:
        message = "  [date]   [client]\t[uri]"
        bot.custom(prefix='Containers:', message=message, color="RED")
        for c in containers:
            uri = c.get_uri()
            created_at = c.created_at.strftime('%B %d, %Y')
            rows.append([created_at, "   [%s]" %c.client, uri])
        bot.table(rows)
    return containers

Response: List local images in the database optionally with a query.


Instruction:
Can you write a function in Python 3 where it
inspects a local image in the database which typically includes the basic fields in the model.

Response:
def inspect(self, name):
    '''Inspect a local image in the database, which typically includes the
       basic fields in the model.

    '''
    print(name)
    container = self.get(name)
    if container is not None:
        collection = container.collection.name
        fields = container.__dict__.copy()
        fields['collection'] = collection
        fields['metrics'] = json.loads(fields['metrics'])
        del fields['_sa_instance_state']
        fields['created_at'] = str(fields['created_at'])
        print(json.dumps(fields, indent=4, sort_keys=True))
        return fields


Instruction:
How would you implement a function in Python 3 that
renames performs a move and returns the container object

Response:
def rename(self, image_name, path):
    '''rename performs a move, but ensures the path is maintained in storage

       Parameters
       ==========
       image_name: the image name (uri) to rename to.
       path: the name to rename (basename is taken)

    '''
    container = self.get(image_name, quiet=True)

    if container is not None:
        if container.image is not None:

            # The original directory for the container stays the same
            dirname = os.path.dirname(container.image)

            # But we derive a new filename and uri

            names = parse_image_name( remove_uri (path) )
            storage = os.path.join( self.storage,
                                    os.path.dirname(names['storage']) )

            # This is the collection folder

            if not os.path.exists(storage):
                os.mkdir(storage)

            # Here we get the new full path, rename the container file

            fullpath = os.path.abspath(os.path.join(dirname, names['storage']))
            container = self.cp(move_to=fullpath,
                                container=container,
                                command="rename")

            # On successful rename of file, update the uri

            if container is not None:
                container.uri = names['uri']
                self.session.commit()
                return container

    bot.warning('%s not found' %(image_name))


Instruction:
Make a summary of the following Python 3 code
def mv(self, image_name, path):
    '''Move an image from it's current location to a new path.
       Removing the image from organized storage is not the recommended approach
       however is still a function wanted by some.

       Parameters
       ==========
       image_name: the parsed image name.
       path: the location to move the image to

    '''

    container = self.get(image_name, quiet=True)

    if container is not None:

        name = container.uri or container.get_uri()
        image = container.image or ''

        # Only continue if image file exists
        if os.path.exists(image):

            # Default assume directory, use image name and path fully
            filename = os.path.basename(image)
            filedir = os.path.abspath(path)

            # If it's a file, use filename provided
            if not os.path.isdir(path):
                filename = os.path.basename(path)
                filedir = os.path.dirname(path)

            # If directory is empty, assume $PWD
            if filedir == '':
                filedir = os.getcwd()

            # Copy to the fullpath from the storage
            fullpath = os.path.abspath(os.path.join(filedir,filename))
            return self.cp(move_to=fullpath,
                           container=container,
                           command="move")

    bot.warning('%s not found' %(image_name))

Response: Move an image from it s current location to a new location.


Instruction:
Can you tell what is the following Python 3 function doing
def rmi(self, image_name):
    '''Remove an image from the database and filesystem.
    '''
    container = self.rm(image_name, delete=True)
    if container is not None:
        bot.info("[rmi] %s" % container)

Response: Remove an image from the database and filesystem.


Instruction:
Make a summary of the following Python 3 code
def rm(self, image_name, delete=False):
    '''Remove an image from the database, akin to untagging the image. This
    does not delete the file from the cache, unless delete is set to True
    (as called by rmi).
    '''
    container = self.get(image_name)
    if container is not None:
        name = container.uri or container.get_uri()
        image = container.image
        self.session.delete(container)
        self.session.commit()
        if image is not None:
            if os.path.exists(image) and delete is True:
                os.remove(container.image)
            return image
        bot.info("[rm] %s" % name)

Response: Remove an image from the database.


Instruction:
How would you implement a function in Python 3 that
gets or create a container including the collection to add it to the database

Response:
def add(self, image_path=None,
              image_uri=None,
              image_name=None,
              url=None,
              metadata=None,
              save=True,
              copy=False):

    '''get or create a container, including the collection to add it to.
    This function can be used from a file on the local system, or via a URL
    that has been downloaded. Either way, if one of url, version, or image_file
    is not provided, the model is created without it. If a version is not
    provided but a file path is, then the file hash is used.

    Parameters
    ==========
    image_path: full path to image file
    image_name: if defined, the user wants a custom name (and not based on uri)
    metadata: any extra metadata to keep for the image (dict)
    save: if True, move the image to the cache if it's not there
    copy: If True, copy the image instead of moving it.

    image_name: a uri that gets parsed into a names object that looks like:

    {'collection': 'vsoch',
     'image': 'hello-world',
     'storage': 'vsoch/hello-world-latest.img',
     'tag': 'latest',
     'version': '12345'
     'uri': 'vsoch/hello-world:latest@12345'}

    After running add, the user will take some image in a working
    directory, add it to the database, and have it available for search
    and use under SREGISTRY_STORAGE/<collection>/<container>

    If the container was retrieved from a webby place, it should have version
    If no version is found, the file hash is used.
    '''

    from sregistry.database.models import (
        Container,
        Collection
    )

    # We can only save if the image is provided
    if image_path is not None:
        if not os.path.exists(image_path) and save is True:
            bot.error('Cannot find %s' %image_path)
            sys.exit(1)

    # An image uri is required for version, tag, etc.
    if image_uri is None:
        bot.error('You must provide an image uri <collection>/<namespace>')
        sys.exit(1)

    names = parse_image_name( remove_uri(image_uri) )
    bot.debug('Adding %s to registry' % names['uri'])

    # If Singularity is installed, inspect image for metadata
    metadata = self.get_metadata(image_path, names=names)
    collection = self.get_or_create_collection(names['collection'])

    # Get a hash of the file for the version, or use provided
    version = names.get('version')
    if version == None:
        if image_path != None:
            version = get_image_hash(image_path)
        else:
            version = ''  # we can't determine a version, not in API/no file
        names = parse_image_name( remove_uri(image_uri), version=version )

    # If save, move to registry storage first
    if save is True and image_path is not None:

        # If the user hasn't defined a custom name
        if image_name is None:
            image_name = self._get_storage_name(names)

        if copy is True:
            copyfile(image_path, image_name)
        else:
            shutil.move(image_path, image_name)

        image_path = image_name

    # Just in case the client didn't provide it, see if we have in metadata
    if url is None and "url" in metadata:
        url = metadata['url']

    # First check that we don't have one already!
    container = self.get_container(name=names['image'],
                                   collection_id=collection.id,
                                   tag=names['tag'],
                                   version=version)

    # The container did not exist, create it
    if container is None:
        action = "new"
        container = Container(metrics=json.dumps(metadata),
                              name=names['image'],
                              image=image_path,
                              client=self.client_name,
                              tag=names['tag'],
                              version=version,
                              url=url,
                              uri=names['uri'],
                              collection_id=collection.id)

        self.session.add(container)
        collection.containers.append(container)

    # The container existed, update it.
    else:
        action="update"
        metrics=json.loads(container.metrics)
        metrics.update(metadata)
        container.url= url
        container.client=self.client_name
        if image_path is not None:
            container.image=image_path
        container.metrics=json.dumps(metrics)

    self.session.commit()
    bot.info("[container][%s] %s" % (action,names['uri']))
    return container


Instruction:
Can you write a function in Python 3 where it
pushes an image to a Singularity registry

Response:
def push(self, path, name, tag=None):
    '''push an image to Singularity Registry'''

    path = os.path.abspath(path)
    image = os.path.basename(path)
    bot.debug("PUSH %s" % path)

    if not os.path.exists(path):
        bot.error('%s does not exist.' %path)
        sys.exit(1)

    # Interaction with a registry requires secrets
    self.require_secrets()

    # Extract the metadata
    names = parse_image_name(remove_uri(name), tag=tag)
    image_size = os.path.getsize(path) >> 20

# COLLECTION ###################################################################

    # If the registry is provided in the uri, use it
    if names['registry'] == None:
        names['registry'] = self.base

    # If the base doesn't start with http or https, add it
    names = self._add_https(names)

    # Prepare push request, this will return a collection ID if permission
    url = '%s/push/' % names['registry']
    auth_url = '%s/upload/chunked_upload' % names['registry']
    SREGISTRY_EVENT = self.authorize(request_type="push",
                                     names=names)

    # Data fields for collection
    fields = { 'collection': names['collection'],
               'name':names['image'],
               'tag': names['tag']}

    headers = { 'Authorization': SREGISTRY_EVENT }

    r = requests.post(auth_url, json=fields, headers=headers)

    # Always tell the user what's going on!
    message = self._read_response(r)
    print('\n[1. Collection return status {0} {1}]'.format(r.status_code, message))

    # Get the collection id, if created, and continue with upload
    if r.status_code != 200:
        sys.exit(1)

# UPLOAD #######################################################################

    url = '%s/upload' % names['registry'].replace('/api','')
    bot.debug('Seting upload URL to {0}'.format(url))

    cid = r.json()['cid']
    upload_to = os.path.basename(names['storage'])

    SREGISTRY_EVENT = self.authorize(request_type="upload",
                                     names=names)

    encoder = MultipartEncoder(fields={'SREGISTRY_EVENT': SREGISTRY_EVENT,
                                       'name': names['image'],
                                       'collection': str(cid),
                                       'tag': names['tag'],
                                       'file1': (upload_to, open(path, 'rb'), 'text/plain')})

    progress_callback = create_callback(encoder, self.quiet)
    monitor = MultipartEncoderMonitor(encoder, progress_callback)
    headers = {'Content-Type': monitor.content_type,
               'Authorization': SREGISTRY_EVENT }

    try:
        r = requests.post(url, data=monitor, headers=headers)
        r.raise_for_status()
        message = r.json()['message']
        print('\n[Return status {0} {1}]'.format(r.status_code, message))
    except requests.HTTPError as e:
        print('\nUpload failed: {0}.'.format(e))
    except KeyboardInterrupt:
        print('\nUpload cancelled.')
    except Exception as e:
        print(e)


Instruction:
Can you tell what is the following Python 3 function doing
def parse_header(recipe, header="from", remove_header=True):
    '''take a recipe, and return the complete header, line. If
       remove_header is True, only return the value.

       Parameters
       ==========
       recipe: the recipe file
       headers: the header key to find and parse
       remove_header: if true, remove the key

    '''
    parsed_header = None
    fromline = [x for x in recipe.split('\n') if "%s:" %header in x.lower()]

    # Case 1: We did not find the fromline
    if len(fromline) == 0:
        return ""

    # Case 2: We found it!
    if len(fromline) > 0:
        fromline = fromline[0]
        parsed_header = fromline.strip()

    # Does the user want to clean it up?
    if remove_header is True:
        parsed_header = fromline.split(':', 1)[-1].strip()
    return parsed_header

Response: take a recipe and return the complete header line.


Instruction:
How would you explain what the following Python 3 function does
def find_recipes(folders, pattern=None, base=None):
    '''find recipes will use a list of base folders, files,
       or patterns over a subset of content to find recipe files
       (indicated by Starting with Singularity

       Parameters
       ==========
        base: if defined, consider folders recursively below this level.

    '''
    # If the user doesn't provide a list of folders, use $PWD
    if folders is None:
        folders = os.getcwd()

    if not isinstance(folders,list):
        folders = [folders]

    manifest = dict()
    for base_folder in folders:

        # If we find a file, return the one file
        custom_pattern = None
        if os.path.isfile(base_folder):  # updates manifest
            manifest = find_single_recipe(filename=base_folder,
                                          pattern=pattern,
                                          manifest=manifest)
            continue

        # The user likely provided a custom pattern
        elif not os.path.isdir(base_folder):
            custom_pattern = base_folder.split('/')[-1:][0]
            base_folder = "/".join(base_folder.split('/')[0:-1])

        # If we don't trigger loop, we have directory
        manifest = find_folder_recipes(base_folder=base_folder,
                                       pattern=custom_pattern or pattern,
                                       manifest=manifest,
                                       base=base)

    return manifest

Response: find recipes will use a list of base folders files or patterns over a subset of content to find recipes


Instruction:
Given the following Python 3 function, write the documentation
def find_folder_recipes(base_folder,
                        pattern="Singularity",
                        manifest=None,
                        base=None):

    '''find folder recipes will find recipes based on a particular pattern.

       Parameters
       ==========
       base_folder: the base folder to recursively walk
       pattern: a default pattern to search for
       manifest: an already started manifest
       base: if defined, consider folders under this level recursively.

    '''

    # The user is not appending to an existing manifest
    if manifest is None:
        manifest = dict()

    for root, dirnames, filenames in os.walk(base_folder):

        for filename in fnmatch.filter(filenames, pattern):

            container_path = os.path.join(root, filename)
            if base is not None:
                container_base = container_path.replace(base,'').strip('/')
                collection = container_base.split('/')[0]
                recipe = os.path.basename(container_base)
                container_uri = "%s/%s" %(collection,recipe)
            else:
                container_uri = '/'.join(container_path.strip('/').split('/')[-2:])

            add_container = True

            # Add the most recently updated container
            if container_uri in manifest:
                if manifest[container_uri]['modified'] > os.path.getmtime(container_path):
                    add_container = False

            if add_container:
                manifest[container_uri] = {'path': os.path.abspath(container_path),
                                           'modified':os.path.getmtime(container_path)}

    return manifest

Response: find folder recipes based on a particular pattern


Instruction:
Given the following Python 3 function, write the documentation
def build(self, name,
                recipe="Singularity",
                context=None,
                preview=False):

    '''trigger a build on Google Cloud (builder then storage) given a name
       recipe, and Github URI where the recipe can be found.

       Parameters
       ==========
       recipe: the local recipe to build.
       name: should be the complete uri that the user has requested to push.
       context: the dependency files needed for the build. If not defined, only
                the recipe is uploaded.
       preview: if True, preview but don't run the build

       Environment
       ===========
       SREGISTRY_GOOGLE_BUILD_SINGULARITY_VERSION: the version of Singularity
           to use, defaults to 3.0.2-slim
       SREGISTRY_GOOGLE_BUILD_CLEANUP: after build, delete intermediate
           dependencies in cloudbuild bucket.

    '''
    bot.debug("BUILD %s" % recipe)

    # This returns a data structure with collection, container, based on uri
    names = parse_image_name(remove_uri(name))

    # Load the build configuration
    config = self._load_build_config(name=names['uri'], recipe=recipe)

    build_package = [recipe]
    if context not in [None, '', []]:

        # If the user gives a ., include recursive $PWD
        if '.' in context:
            context = glob(os.getcwd() + '/**/*', recursive=True)
        build_package = build_package + context

    package = create_build_package(build_package)

    # Does the package already exist? If the user cached, it might
    destination='source/%s' % os.path.basename(package)
    blob = self._build_bucket.blob(destination)

    # if it doesn't exist, upload it
    if not blob.exists() and preview is False:
        bot.log('Uploading build package!')
        manifest = self._upload(source=package,
                                bucket=self._build_bucket,
                                destination=destination)
    else:
        bot.log('Build package found in %s.' % self._build_bucket.name)

    # The source should point to the bucket with the .tar.gz, latest generation
    config["source"]["storageSource"]['bucket'] = self._build_bucket.name
    config["source"]["storageSource"]['object'] = destination

    # If not a preview, run the build and return the response
    if preview is False:
        config = self._run_build(config, self._bucket, names)

    # If the user wants to cache cloudbuild files, this will be set
    if not self._get_and_update_setting('SREGISTRY_GOOGLE_BUILD_CACHE'):
        blob.delete()

    # Clean up either way, return config or response
    shutil.rmtree(os.path.dirname(package))
    return config

Response: trigger a build on Google Cloud


Instruction:
Can you create a Python 3 function that
gives a list of files copy them to a temporary folder compress into a. tar. gz and rename based on the file hash.

Response:
def create_build_package(package_files):
    '''given a list of files, copy them to a temporary folder,
       compress into a .tar.gz, and rename based on the file hash.
       Return the full path to the .tar.gz in the temporary folder.

       Parameters
       ==========
       package_files: a list of files to include in the tar.gz

    '''
    # Ensure package files all exist
    for package_file in package_files:
        if not os.path.exists(package_file):
            bot.exit('Cannot find %s.' % package_file)

    bot.log('Generating build package for %s files...' % len(package_files))
    build_dir = get_tmpdir(prefix="sregistry-build")
    build_tar = '%s/build.tar.gz' % build_dir
    tar = tarfile.open(build_tar, "w:gz")

    # Create the tar.gz
    for package_file in package_files:
        tar.add(package_file)
    tar.close()

    # Get hash (sha256), and rename file
    sha256 = get_file_hash(build_tar)
    hash_tar =  "%s/%s.tar.gz" %(build_dir, sha256)
    shutil.move(build_tar, hash_tar)
    return hash_tar


Instruction:
Can you tell what is the following Python 3 function doing
def load_build_config(self, name, recipe):
    '''load a google compute config, meaning that we start with a template,
       and mimic the following example cloudbuild.yaml:

        steps:
        - name: "singularityware/singularity:${_SINGULARITY_VERSION}"
          args: ['build', 'julia-centos-another.sif', 'julia.def']
        artifacts:
          objects:
            location: 'gs://sregistry-gcloud-build-vanessa'
            paths: ['julia-centos-another.sif']

        Parameters
        ==========
        recipe: the local recipe file for the builder.
        name: the name of the container, based on the uri

    '''
    version_envar = 'SREGISTRY_GOOGLE_BUILD_SINGULARITY_VERSION'
    version = self._get_and_update_setting(version_envar, '3.0.2-slim')
    config = get_build_template()

    # Name is in format 'dinosaur/container-latest'

    # The command to give the builder, with image name
    container_name = '%s.sif' % name.replace('/','-', 1)
    config['steps'][0]['name'] = 'singularityware/singularity:%s' % version
    config['steps'][0]['args'] = ['build', container_name, recipe]

    config["artifacts"]["objects"]["location"] = "gs://%s" % self._bucket_name
    config["artifacts"]["objects"]["paths"] = [container_name]

    return config

Response: load a google compute config


Instruction:
How would you explain what the following Python 3 function does
def run_build(self, config, bucket, names):
    '''run a build, meaning creating a build. Retry if there is failure
    '''

    project = self._get_project()

    #          prefix,    message, color
    bot.custom('PROJECT', project, "CYAN")
    bot.custom('BUILD  ', config['steps'][0]['name'], "CYAN")

    response = self._build_service.projects().builds().create(body=config,
                                              projectId=project).execute()

    build_id = response['metadata']['build']['id']
    status = response['metadata']['build']['status']
    bot.log("build %s: %s" % (build_id, status))

    start = time.time()
    while status not in ['COMPLETE', 'FAILURE', 'SUCCESS']:
        time.sleep(15)
        response = self._build_service.projects().builds().get(id=build_id,
                                                  projectId=project).execute()

        build_id = response['id']
        status = response['status']
        bot.log("build %s: %s" % (build_id, status))

    end = time.time()
    bot.log('Total build time: %s seconds' % (round(end - start, 2)))

    # If successful, update blob metadata and visibility
    if status == 'SUCCESS':

        # Does the user want to keep the container private?
        env = 'SREGISTRY_GOOGLE_STORAGE_PRIVATE'
        blob = bucket.blob(response['artifacts']['objects']['paths'][0])

        # Make Public, if desired
        if self._get_and_update_setting(env) == None:
            blob.make_public()
            response['public_url'] = blob.public_url

        # Add the metadata directly to the object
        update_blob_metadata(blob, response, config, bucket, names)
        response['media_link'] = blob.media_link
        response['size'] = blob.size
        response['file_hash'] = blob.md5_hash

    return response

Response: run a build, meaning creating a build. Retry if there is failure


Instruction:
Make a summary of the following Python 3 code
def update_blob_metadata(blob, response, config, bucket, names):
    '''a specific function to take a blob, along with a SUCCESS response
       from Google build, the original config, and update the blob
       metadata with the artifact file name, dependencies, and image hash.
    '''

    manifest = os.path.basename(response['results']['artifactManifest'])
    manifest = json.loads(bucket.blob(manifest).download_as_string())

    metadata = {'file_hash': manifest['file_hash'][0]['file_hash'][0]['value'],
                'artifactManifest': response['results']['artifactManifest'],
                'location': manifest['location'],
                'storageSourceBucket': config['source']['storageSource']['bucket'],
                'storageSourceObject': config['source']['storageSource']['object'],
                'buildCommand': ' '.join(config['steps'][0]['args']),
                'builder': config['steps'][0]['name'],
                'media_link': blob.media_link,
                'self_link': blob.self_link,
                'size': blob.size,
                'name': names['tag_uri'],
                'type': "container"} # identifier that the blob is a container

    blob.metadata = metadata
    blob._properties['metadata'] = metadata
    blob.patch()

Response: a specific function to take a blob along with a SUCCESS response and update the blob s metadata with the artifact file name dependencies and image hash.


Instruction:
Create a Python 3 function to
query a Singularity registry for a list of images.

Response:
def search(self, query=None, args=None):
    '''query a Singularity registry for a list of images.
     If query is None, collections are listed.

    EXAMPLE QUERIES:
    '''

    # You can optionally better parse the image uri (query), but not
    # necessary
    # names = parse_image_name(remove_uri(query))

    if query is not None:

        # Here you might do a function that is a general list
        # Note that this means adding the function Client in __init__
        return self._container_query(query)

    # or default to listing (searching) all things.
    return self._search_all()


Instruction:
How would you explain what the following Python 3 function does
def parse_image_name(image_name,
                     tag=None,
                     version=None,
                     defaults=True,
                     ext="sif",
                     default_collection="library",
                     default_tag="latest",
                     base=None,
                     lowercase=True):

    '''return a collection and repo name and tag
    for an image file.

    Parameters
    =========
    image_name: a user provided string indicating a collection,
                image, and optionally a tag.
    tag: optionally specify tag as its own argument
         over-rides parsed image tag
    defaults: use defaults "latest" for tag and "library"
              for collection.
    base: if defined, remove from image_name, appropriate if the
          user gave a registry url base that isn't part of namespace.
    lowercase: turn entire URI to lowercase (default is True)
    '''

    # Save the original string
    original = image_name

    if base is not None:
        image_name = image_name.replace(base,'').strip('/')

    # If a file is provided, remove extension
    image_name = re.sub('[.](img|simg|sif)','', image_name)

    # Parse the provided name
    uri_regexes = [ _reduced_uri,
                    _default_uri,
                    _docker_uri ]

    for r in uri_regexes:
        match = r.match(image_name)
        if match:
            break

    if not match:
        bot.exit('Could not parse image "%s"! Exiting.' % image)

    # Get matches
    registry = match.group('registry')
    collection = match.group('collection')
    repo_name = match.group('repo')
    repo_tag = match.group('tag')
    version = match.group('version')

    # A repo_name is required
    assert(repo_name)

    # If a collection isn't provided
    collection = set_default(collection, default_collection, defaults)
    repo_tag = set_default(repo_tag, default_tag, defaults)

    # The collection, name must be all lowercase
    if lowercase:
        collection = collection.lower().rstrip('/')
        repo_name = repo_name.lower()
        repo_tag = repo_tag.lower()
    else:
        collection = collection.rstrip('/')

    if version != None:
        version = version.lower()

    # Piece together the uri base
    if registry == None:
        uri = "%s/%s" % (collection, repo_name)
    else:
        uri = "%s/%s/%s" % (registry, collection, repo_name)

    url = uri

    # Tag is defined
    if repo_tag != None:
        uri = "%s-%s" % (uri, repo_tag)
        tag_uri = "%s:%s" % (url, repo_tag)

    # Version is defined
    storage_version = None
    if version is not None:
        uri = "%s@%s" % (uri, version)
        tag_uri = "%s@%s" % (tag_uri, version)
        storage_version = "%s@%s.%s" % (tag_uri, version, ext)

    # A second storage URI honors the tag (:) separator

    storage = "%s.%s" %(uri, ext)
    storage_uri = "%s.%s" %(tag_uri, ext)
    result = {"collection": collection,
              "original": original,
              "registry": registry,
              "image": repo_name,
              "url": url,
              "tag": repo_tag,
              "version": version,
              "storage": storage,
              "storage_uri": storage_uri,
              "storage_version": storage_version or storage_uri,
              "tag_uri": tag_uri,
              "uri": uri}

    return result

Response: parse an image name and return a collection repo name and tag for an image file.


Instruction:
Make a summary of the following Python 3 code
def format_container_name(name, special_characters=None):
    '''format_container_name will take a name supplied by the user,
    remove all special characters (except for those defined by "special-characters"
    and return the new image name.
    '''
    if special_characters is None:
        special_characters = []
    return ''.join(e.lower()
                   for e in name if e.isalnum() or e in special_characters)

Response: Format a container name.


Instruction:
Explain what the following Python 3 code does
def get_uri(image):
    '''get the uri for an image, if within acceptable

       Parameters
       ==========
       image: the image uri, in the format <uri>://<registry>/<namespace>:<tag>

    '''
    # Ensure we have a string
    image = image or ''

    # Find uri prefix, including ://
    regexp = re.compile('^.+://')
    uri = regexp.match(image)

    if uri is not None:
        uri = (uri.group().lower()
                          .replace('_','-')
                          .replace('://',''))

        accepted_uris = ['aws',
                         'docker',
                         'http', 'https', # Must be allowed for pull
                         'dropbox',
                         'gitlab',
                         'globus',
                         'google-build',
                         'google-storage',
                         'google-drive',
                         'hub',
                         'nvidia',
                         'registry',
                         's3',
                         'swift']

        # Allow for Singularity compatability
        if "shub" in uri: uri = "hub"

        if uri not in accepted_uris:
            bot.warning('%s is not a recognized uri.' % uri)
            uri = None

    return uri

Response:
get the uri for an image if within acceptable
ATTRIBS


Instruction:
How would you explain what the following Python 3 function does
def pull(self, images, file_name=None, save=True, **kwargs):
    '''pull an image from storage using Swift. The image is found based on the
       storage uri

       Parameters
       ==========
       images: refers to the uri given by the user to pull in the format
       <collection>/<namespace>. You should have an API that is able to
       retrieve a container based on parsing this uri.
       file_name: the user's requested name for the file. It can
                  optionally be None if the user wants a default.
       save: if True, you should save the container to the database
             using self.add()

       Returns
       =======
       finished: a single container path, or list of paths

    '''
    force = False
    if "force" in kwargs:
        force = kwargs['force']

    if not isinstance(images, list):
        images = [images]

    bot.debug('Execution of PULL for %s images' % len(images))

    # If used internally we want to return a list to the user.
    finished = []
    for image in images:

        names = parse_image_name(remove_uri(image))

        # First try to get the collection
        collection = self._get_collection(names['collection'])
        if collection is None:
            bot.error('Collection %s does not exist.' % names['collection'])

            # Show the user collections he/she does have access to
            collections = self.get_collections()
            if collections:
                bot.info('Collections available to you: \n%s' %'\n'.join(collections))
            sys.exit(1)

        # Determine if the container exists in storage
        image_name = os.path.basename(names['storage'])

        try:
            obj_tuple = self.conn.get_object(names['collection'], image_name)
        except ClientException:
            bot.exit('%s does not exist.' % names['storage'])

        # Give etag as version if version not defined
        if names['version'] == None:
            names['version'] = obj_tuple[0]['etag']

        # If the user didn't provide a file, make one based on the names
        if file_name is None:
            file_name = self._get_storage_name(names)

        # If the file already exists and force is False
        if os.path.exists(file_name) and force is False:
            bot.error('Image exists! Remove first, or use --force to overwrite')
            sys.exit(1)

        # Write to file
        with open(file_name, 'wb') as filey:
            filey.write(obj_tuple[1])

        # If we save to storage, the uri is the dropbox_path
        if save is True:

            names.update(obj_tuple[0])
            container = self.add(image_path = file_name,
                                 image_uri = names['uri'],
                                 metadata = names)

            # When the container is created, this is the path to the image
            image_file = container.image

            if os.path.exists(image_file):
                bot.debug('Retrieved image file %s' %image_file)
                bot.custom(prefix="Success!", message=image_file)
                finished.append(image_file)

        else:
            bot.error('%s does not exist. Try sregistry search to see images.' % path)

    if len(finished) == 1:
        finished = finished[0]
    return finished

Response: pull an image from storage using Swift


Instruction:
Implement a Python 3 function for
pushing an image to Google Cloud Drive

Response:
def push(self, path, name, tag=None):
    '''push an image to Google Cloud Drive, meaning uploading it

    path: should correspond to an absolte image path (or derive it)
    name: should be the complete uri that the user has requested to push.
    tag: should correspond with an image tag. This is provided to mirror Docker
    '''
    # The root of the drive for containers (the parent folder)
    parent = self._get_or_create_folder(self._base)

    image = None
    path = os.path.abspath(path)
    bot.debug("PUSH %s" % path)

    if not os.path.exists(path):
        bot.error('%s does not exist.' %path)
        sys.exit(1)

    names = parse_image_name(remove_uri(name),tag=tag)
    if names['version'] is None:
        version = get_image_hash(path)
        names = parse_image_name(remove_uri(name), tag=tag, version=version)

    # Update metadata with names, flatten to only include labels
    metadata = self.get_metadata(path, names=names)
    metadata = metadata['data']
    metadata.update(names)
    metadata.update(metadata['attributes']['labels'])
    del metadata['attributes']

    file_metadata = {
        'name': names['storage'],
        'mimeType' : 'application/octet-stream',
        'parents': [parent['id']],
        'properties': metadata
    }

    media = MediaFileUpload(path,resumable=True)
    try:
        bot.spinner.start()
        image = self._service.files().create(body=file_metadata,
                                             media_body=media,
                                             fields='id').execute()

        # Add a thumbnail!
        thumbnail = get_thumbnail()

        with open(thumbnail, "rb") as f:
            body = { "contentHints": {
                     "thumbnail": { "image": base64.urlsafe_b64encode(f.read()).decode('utf8'),
                                     "mimeType": "image/png" }
                                  }}
            image = self._service.files().update(fileId=image['id'],
                                                 body = body).execute()

        bot.spinner.stop()
        print(image['name'])

    except HttpError:
        bot.error('Error uploading %s' %path)
        pass

    return image


Instruction:
Can you tell what is the following Python 3 function doing
def _set_base(self, zone=None):
        '''set the API base or default to use Docker Hub. The user is able
           to set the base, api version, and protocol via a settings file
           of environment variables:
        '''
        if hasattr(self.aws._client_config, 'region_name'):
            zone = self.aws._client_config.region_name

        aws_id = self._required_get_and_update('SREGISTRY_AWS_ID')
        aws_zone = self._required_get_and_update('SREGISTRY_AWS_ZONE', zone)
        version = self._get_setting('SREGISTRY_AWS_VERSION', 'v2')
        base = self._get_setting('SREGISTRY_AWS_BASE')

        if base is None:
            base = "%s.dkr.ecr.%s.amazonaws.com" % (aws_id, aws_zone)

        nohttps = self._get_setting('SREGISTRY_AWS_NOHTTPS')
        if nohttps is None:
            nohttps = "https://"
        else:
            nohttps = "http://"

        # <protocol>://<base>/<version>
        self.base = "%s%s/%s" %(nohttps, base.strip('/'), version)

Response: set the base or default to use Docker Hub.


Instruction:
Here you have a function in Python 3, explain what it does
def _update_secrets(self):
        '''update secrets will take a secrets credential file
           either located at .sregistry or the environment variable
           SREGISTRY_CLIENT_SECRETS and update the current client
           secrets as well as the associated API base. For the case of
           using Docker Hub, if we find a .docker secrets file, we update
           from there.
        '''
        bot.debug('Creating aws client...')
        try:
            from awscli.clidriver import create_clidriver
        except:
            bot.exit('Please install pip install sregistry[aws]')

        driver = create_clidriver()
        self.aws = driver.session.create_client('ecr')

Response:
update secrets will take a secrets credential file located at. sregistry or the environment variable SREGISTRY_CLIENT_SECRETS SREGISTRY_CLIENT_SECRETSS and update the current client
 secrets with the associated API base.


Instruction:
How would you explain what the following Python 3 function does
def get_logging_level():
    '''get_logging_level will configure a logging to standard out based on the user's
    selected level, which should be in an environment variable called
    MESSAGELEVEL. if MESSAGELEVEL is not set, the maximum level
    (5) is assumed (all messages).
    '''
    level = os.environ.get("MESSAGELEVEL", INFO)

    # User knows logging levels and set one
    if isinstance(level, int):
        return level

    # Otherwise it's a string
    if level == "CRITICAL":
        return CRITICAL
    elif level == "ABORT":
        return ABORT
    elif level == "ERROR":
        return ERROR
    elif level == "WARNING":
        return WARNING
    elif level == "LOG":
        return LOG
    elif level == "INFO":
        return INFO
    elif level == "QUIET":
        return QUIET
    elif level.startswith("VERBOSE"):
        return VERBOSE3
    elif level == "LOG":
        return LOG
    elif level == "DEBUG":
        return DEBUG

    return level

Response: This function returns the logging level for the current user.


Instruction:
Create a Python 3 function for
determining if a level should print to stderr includes all levels but INFO and QUIET

Response:
def emitError(self, level):
        '''determine if a level should print to
        stderr, includes all levels but INFO and QUIET'''
        if level in [ABORT,
                     ERROR,
                     WARNING,
                     VERBOSE,
                     VERBOSE1,
                     VERBOSE2,
                     VERBOSE3,
                     DEBUG]:
            return True
        return False


Instruction:
Write a Python 3 script to
emit is the main function to print the message optionally with a prefix :param level: the level of the message :param message: the message to print :param prefix: a prefix for the message

Response:
def emit(self, level, message, prefix=None, color=None):
        '''emit is the main function to print the message
        optionally with a prefix
        :param level: the level of the message
        :param message: the message to print
        :param prefix: a prefix for the message
        '''
        if color is None:
            color = level

        if prefix is not None:
            prefix = self.addColor(color, "%s " % (prefix))
        else:
            prefix = ""
            message = self.addColor(color, message)

        # Add the prefix
        message = "%s%s" % (prefix, message)

        if not message.endswith('\n'):
            message = "%s\n" % message

        # If the level is quiet, only print to error
        if self.level == QUIET:
            pass

        # Otherwise if in range print to stdout and stderr
        elif self.isEnabledFor(level):
            if self.emitError(level):
                self.write(self.errorStream, message)
            else:
                self.write(self.outputStream, message)

        # Add all log messages to history
        self.history.append(message)


Instruction:
Write a Python 3 function for
writing will write a message to a stream

Response:
def write(self, stream, message):
        '''write will write a message to a stream,
        first checking the encoding
        '''
        if isinstance(message, bytes):
            message = message.decode('utf-8')
        stream.write(message)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_logs(self, join_newline=True):
        ''''get_logs will return the complete history, joined by newline
        (default) or as is.
        '''
        if join_newline:
            return '\n'.join(self.history)
        return self.history

Response: get_logs will return the complete history of the assessment


Instruction:
Write a Python 3 script for
printing a table of entries.

Response:
def table(self, rows, col_width=2):
        '''table will print a table of entries. If the rows is
        a dictionary, the keys are interpreted as column names. if
        not, a numbered list is used.
        '''

        labels = [str(x) for x in range(1,len(rows)+1)]
        if isinstance(rows, dict):
            labels = list(rows.keys())
            rows = list(rows.values())

        for row in rows:
            label = labels.pop(0)
            label = label.ljust(col_width)
            message = "\t".join(row)
            self.custom(prefix=label,
                        message=message)


Instruction:
How would you explain what the following Python 3 function does
def push(self, path, name, tag=None):
    '''push an image to Singularity Registry

    path: should correspond to an absolte image path (or derive it)
    name: should be the complete uri that the user has requested to push.
    tag: should correspond with an image tag. This is provided to mirror Docker
    '''
    path = os.path.abspath(path)
    bot.debug("PUSH %s" % path)

    if not os.path.exists(path):
        bot.error('%s does not exist.' %path)
        sys.exit(1)

    # This returns a data structure with collection, container, based on uri
    names = parse_image_name(remove_uri(name),tag=tag)

    # use Singularity client, if exists, to inspect to extract metadata
    metadata = self.get_metadata(path, names=names)

    # If you want a spinner
    bot.spinner.start()
    # do your push request here. Generally you want to except a KeyboardInterrupt
    # and give the user a status from the response
    bot.spinner.stop()

Response: push an image to Singularity registry


Instruction:
Implement a Python 3 function for
pushing an image to a Globus endpoint

Response:
def push(self, path, name, tag=None):
    '''push an image to Globus endpoint. In this case, the name is the
       globus endpoint id and path.

       --name <endpointid>:/path/for/image

    '''

    # Split the name into endpoint and rest

    endpoint, remote = self._parse_endpoint_name(name)

    path = os.path.abspath(path)
    image = os.path.basename(path)
    bot.debug("PUSH %s" % path)

    # Flatten image uri into image name

    q = parse_image_name(image)

    if not os.path.exists(path):
        bot.error('%s does not exist.' %path)
        sys.exit(1)

    # Ensure we have a transfer client
    if not hasattr(self, 'transfer_client'):
        self._init_transfer_client()

    # The user must have a personal endpoint

    endpoints = self._get_endpoints()

    if len(endpoints['my-endpoints']) == 0:
        bot.error('You must have a personal endpoint to transfer the container')
        sys.exit(1)

    # Take the first endpoint that is active

    source_endpoint = None
    for eid,contender in endpoints['my-endpoints'].items():
       if contender['gcp_connected'] is True:
           source_endpoint = contender
           break

    # Exit if none are active, required!

    if source_endpoint is None:
        bot.error('No activated local endpoints online! Go online to transfer')
        sys.exit(1)

    # The destination endpoint should have an .singularity/shub folder set
    self._create_endpoint_cache(endpoint)

    # SREGISTRY_STORAGE must be an endpoint
    # if the image isn't already there, add it first

    added = self.add(image_path=path,
                     image_uri=q['uri'],
                     copy=True)

    label = "Singularity Registry Transfer for %s" %added.name
    tdata = globus_sdk.TransferData(self.transfer_client,
                                    source_endpoint['id'],
                                    endpoint,
                                    label=label,
                                    sync_level="checksum")
    image = ".singularity/shub/%s" %image
    tdata.add_item(added.image, image)
    bot.info('Requesting transfer from local %s to %s:%s' %(SREGISTRY_STORAGE,
                                                            endpoint, image))
    transfer_result = self.transfer_client.submit_transfer(tdata)
    bot.info(transfer_result['message'])
    return transfer_result


Instruction:
Can you create a Python 3 function that
returns a default template for some function in sregistry

Response:
def get_template(name):
    '''return a default template for some function in sregistry
       If there is no template, None is returned.

       Parameters
       ==========
       name: the name of the template to retrieve

    '''
    name = name.lower()
    templates = dict()

    templates['tarinfo'] = {"gid": 0,
                            "uid": 0,
                            "uname": "root",
                            "gname": "root",
                            "mode": 493}

    if name in templates:
        bot.debug("Found template for %s" % (name))
        return templates[name]
    else:
        bot.warning("Cannot find template %s" % (name))


Instruction:
Make a summary of the following Python 3 code
def download_layers(self, repo_name, digest=None, destination=None):
    ''' download layers is a wrapper to do the following for a client loaded
        with a manifest for an image:

        1. use the manifests to retrieve list of digests (get_digests)
        2. atomically download the list to destination (get_layers)

        This function uses the MultiProcess client to download layers
        at the same time.
    '''
    from sregistry.main.workers import Workers
    from sregistry.main.workers.aws import download_task

    # Obtain list of digets, and destination for download
    self._get_manifest(repo_name, digest)
    digests = self._get_digests(repo_name, digest)
    destination = self._get_download_cache(destination)

    # Create multiprocess download client
    workers = Workers()

    # Download each layer atomically
    tasks = []
    layers = []

    # Start with a fresh token
    self._update_token()

    for digest in digests:

        targz = "%s/%s.tar.gz" % (destination, digest['digest'])
        url = '%s/%s/blobs/%s' % (self.base, repo_name, digest['digest'])

        # Only download if not in cache already
        if not os.path.exists(targz):
            tasks.append((url, self.headers, targz))
        layers.append(targz)

    # Download layers with multiprocess workers
    if len(tasks) > 0:

        download_layers = workers.run(func=download_task,
                                      tasks=tasks)

    return layers, url

Response: Download layers from the cache


Instruction:
Write a Python 3 script for
getting the manifest via the aws client

Response:
def get_manifest(self, repo_name, tag):
    '''return the image manifest via the aws client, saved in self.manifest
    '''

    image = None
    repo = self.aws.describe_images(repositoryName=repo_name)
    if 'imageDetails' in repo:
        for contender in repo.get('imageDetails'):
            if tag in contender['imageTags']:
                image = contender
                break

    # if the image isn't found, we need to exit
    if image is None:
        bot.exit('Cannot find %s:%s, is the uri correct?' %(repo_name, digest))

    digest = image['imageDigest']
    digests = self.aws.batch_get_image(repositoryName=repo_name,
                                       imageIds=[{"imageDigest": digest,
                                                  "imageTag": tag}])

    self.manifest = json.loads(digests['images'][0]['imageManifest'])
    return self.manifest


Instruction:
Can you tell what is the following Python 3 function doing
def get_digests(self, repo_name, tag):
    '''
       return a list of layers from a manifest.
       The function is intended to work with both version
       1 and 2 of the schema. All layers (including redundant)
       are returned. By default, we try version 2 first,
       then fall back to version 1.

       For version 1 manifests: extraction is reversed

       Parameters
       ==========
       manifest: the manifest to read_layers from

    '''
    if not hasattr(self, 'manifest'):
        bot.error('Please retrieve manifest for the image first.')
        sys.exit(1)

    # version 2 manifest here!
    return self.manifest['layers']

Response: get_digests is a function that returns a list of layers from a manifest.


Instruction:
How would you explain what the following Python 3 function does
def prepare_metadata(metadata):
    '''prepare a key/value list of metadata for the request. The metadata
       object that comes in is only parsed one level.
    '''
    pairs = {
        'metadata': {
            'items': [{
                'key': 'client',
                'value': 'sregistry'
            }
           ]
        }
    }
    for key,val in metadata.items():
        if not isinstance(val,dict) and not isinstance(val,list):
            pairs['metadata']['items'].append({'key':key,'value':val})
        elif isinstance(val,dict):
            for k,v in val.items():
                if not isinstance(v,dict) and not isinstance(v,list):
                    pairs['metadata']['items'].append({'key':k,'value':v})

    return pairs

Response: prepare a key - value list of metadata for the request.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_build_template(name=None, manager='apt'):
    '''get a particular build template, by default we return templates
       that are based on package managers.

       Parameters
       ==========
       name: the full path of the template file to use.
       manager: the package manager to use in the template (yum or apt)

    '''
    base = get_installdir()
    if name is None:
        name = "%s/main/templates/build/singularity-builder-%s.sh" %(base,
                                                                     manager)

    if os.path.exists(name):
        bot.debug("Found template %s" %name)
        return ''.join(read_file(name))

    bot.warning("Template %s not found." %name)

Response:
get a particular build template by default we return templates
       that are based on package managers.


Instruction:
Write a Python 3 script to
extract metadata using Singularity inspect if the executable is found and return the names of the image

Response:
def get_metadata(self, image_file, names={}):
    '''extract metadata using Singularity inspect, if the executable is found.
       If not, return a reasonable default (the parsed image name)

       Parameters
       ==========
       image_file: the full path to a Singularity image
       names: optional, an extracted or otherwise created dictionary of
              variables for the image, likely from utils.parse_image_name

    '''
    metadata = dict()

    # We can't return anything without image_file or names
    if image_file is not None:
        if not os.path.exists(image_file):
            bot.error('Cannot find %s.' %image_file)
            return names or metadata

    # The user provided a file, but no names
    if not names:
        names = parse_image_name(remove_uri(image_file))

    # Look for the Singularity Executable
    singularity = which('singularity')['message']

    # Inspect the image, or return names only
    if os.path.exists(singularity) and image_file is not None:
        from spython.main import Client as Singularity

        # We try and inspect, but not required (wont work within Docker)
        try:
            Singularity.quiet = True
            updates = Singularity.inspect(image=image_file)
        except:
            bot.warning('Inspect command not supported, metadata not included.')
            updates = None

        # Try loading the metadata
        if updates is not None:
            try:
                updates = json.loads(updates)
                metadata.update(updates)
            except:
                pass

    metadata.update(names)
    return metadata


Instruction:
Implement a Python 3 function for
pulling an image from a docker hub

Response:
def _pull(self,
          file_name,
          names,
          save=True,
          force=False,
          uri="docker://",
          **kwargs):

    '''pull an image from a docker hub. This is a (less than ideal) workaround
       that actually does the following:

       - creates a sandbox folder
       - adds docker layers, metadata folder, and custom metadata to it
       - converts to a squashfs image with build

    the docker manifests are stored with registry metadata.

    Parameters
    ==========
    images: refers to the uri given by the user to pull in the format
    <collection>/<namespace>. You should have an API that is able to
    retrieve a container based on parsing this uri.
    file_name: the user's requested name for the file. It can
               optionally be None if the user wants a default.
    save: if True, you should save the container to the database
          using self.add()

    Returns
    =======
    finished: a single container path, or list of paths
    '''

    # Use Singularity to build the image, based on user preference
    if file_name is None:
        file_name = self._get_storage_name(names)

    # Determine if the user already has the image
    if os.path.exists(file_name) and force is False:
        bot.error('Image exists! Remove first, or use --force to overwrite')
        sys.exit(1)

    digest = names['version'] or names['tag']

    # Build from sandbox, prefix with sandbox
    sandbox = get_tmpdir(prefix="sregistry-sandbox")

    # First effort, get image via Sregistry
    layers = self._download_layers(names['url'], digest)

    # This is the url where the manifests were obtained
    url = self._get_manifest_selfLink(names['url'], digest)

    # Add environment to the layers
    envtar = self._get_environment_tar()
    layers = [envtar] + layers

    # Create singularity image from an empty folder
    for layer in layers:
        bot.info('Exploding %s' %layer)
        result = extract_tar(layer, sandbox, handle_whiteout=True)
        if result['return_code'] != 0:
            bot.error(result['message'])
            sys.exit(1)

    sudo = kwargs.get('sudo', False)

    # Build from a sandbox (recipe) into the image_file (squashfs)
    image_file = Singularity.build(image=file_name,
                                   recipe=sandbox,
                                   sudo=sudo)

    # Fall back to using Singularity
    if image_file is None:
        bot.info('Downloading with native Singularity, please wait...')
        image = image.replace('docker://', uri)
        image_file = Singularity.pull(image, pull_folder=sandbox)

    # Save to local storage
    if save is True:

        # Did we get the manifests?
        manifests = {}
        if hasattr(self, 'manifests'):
            manifests = self.manifests

        container = self.add(image_path = image_file,
                             image_uri = names['uri'],
                             metadata = manifests,
                             url = url)

        # When the container is created, this is the path to the image
        image_file = container.image

    if os.path.exists(image_file):
        bot.debug('Retrieved image file %s' %image_file)
        bot.custom(prefix="Success!", message=image_file)

    # Clean up sandbox
    shutil.rmtree(sandbox)

    return image_file


Instruction:
Create a Python 3 function for
updating secrets will take a secrets credential file located at. sregistry or the environment variable SREGISTRY_CLIENT_SECRETS and update the current client secrets as well as the associated API base.

Response:
def _update_secrets(self):
        '''update secrets will take a secrets credential file
           either located at .sregistry or the environment variable
           SREGISTRY_CLIENT_SECRETS and update the current client
           secrets as well as the associated API base. This is where you
           should do any customization of the secrets flie, or using
           it to update your client, if needed.
        '''
        # Get a setting for client myclient and some variable name VAR.
        # returns None if not set
        setting = self._get_setting('SREGISTRY_MYCLIENT_VAR')

        # Get (and if found in environment (1) settings (2) update the variable
        # It will still return None if not set
        setting = self._get_and_update_setting('SREGISTRY_MYCLIENT_VAR')

        # If you have a setting that is required and not found, you should exit.

        # Here is how to read all client secrets
        self.secrets = read_client_secrets()

        # If you don't want to use the shared settings file, you have your own.
        # Here is how to get if the user has a cache for you enabled, this
        # returns a path (enabled) or None (disabled) that you should honor
        # You can use this as a file path or folder and for both cases, you
        # need to create the file or folder
        if self._credential_cache is not None:
            bot.info("credential cache set to %s" %self._credential_cache)


Instruction:
Can you implement a function in Python 3 that
generates a repr string for the given class.

Response:
def _make_repr(class_name, *args, **kwargs):
    """
    Generate a repr string.

    Positional arguments should be the positional arguments used to
    construct the class. Keyword arguments should consist of tuples of
    the attribute value and default. If the value is the default, then
    it won't be rendered in the output.

    Here's an example::

        def __repr__(self):
            return make_repr('MyClass', 'foo', name=(self.name, None))

    The output of this would be something line ``MyClass('foo',
    name='Will')``.

    """
    arguments = [repr(arg) for arg in args]
    arguments.extend(
        "{}={!r}".format(name, value)
        for name, (value, default) in sorted(kwargs.items())
        if value != default
    )
    return "{}({})".format(class_name, ", ".join(arguments))


Instruction:
Write a Python 3 function that can
translate S3 errors to FSErrors.

Response:
def s3errors(path):
    """Translate S3 errors to FSErrors."""
    try:
        yield
    except ClientError as error:
        _error = error.response.get("Error", {})
        error_code = _error.get("Code", None)
        response_meta = error.response.get("ResponseMetadata", {})
        http_status = response_meta.get("HTTPStatusCode", 200)
        error_msg = _error.get("Message", None)
        if error_code == "NoSuchBucket":
            raise errors.ResourceError(path, exc=error, msg=error_msg)
        if http_status == 404:
            raise errors.ResourceNotFound(path)
        elif http_status == 403:
            raise errors.PermissionDenied(path=path, msg=error_msg)
        else:
            raise errors.OperationFailed(path=path, exc=error)
    except SSLError as error:
        raise errors.OperationFailed(path, exc=error)
    except EndpointConnectionError as error:
        raise errors.RemoteConnectionError(path, exc=error, msg="{}".format(error))


Instruction:
Implement a function in Python 3 to
create a S3File backed with a temporary file.

Response:
def factory(cls, filename, mode, on_close):
        """Create a S3File backed with a temporary file."""
        _temp_file = tempfile.TemporaryFile()
        proxy = cls(_temp_file, filename, mode, on_close=on_close)
        return proxy


Instruction:
Can you tell what is the following Python 3 function doing
def gravatar_url(user_or_email, size=GRAVATAR_DEFAULT_SIZE):
    if hasattr(user_or_email, 'email'):
        email = user_or_email.email
    else:
        email = user_or_email

    try:
        return escape(get_gravatar_url(email=email, size=size))
    except:
        return ''

Response: Builds a gravatar url from a user or email.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def gravatar(user_or_email, size=GRAVATAR_DEFAULT_SIZE, alt_text='', css_class='gravatar'):
    if hasattr(user_or_email, 'email'):
        email = user_or_email.email
    else:
        email = user_or_email

    try:
        url = escape(get_gravatar_url(email=email, size=size))
    except:
        return ''

    return mark_safe(
        '<img class="{css_class}" src="{src}" width="{width}"'
        ' height="{height}" alt="{alt}" />'.format(
            css_class=css_class, src=url, width=size, height=size, alt=alt_text
        )
    )

Response: Builds an image tag from an email address.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_gravatar_url(email, size=GRAVATAR_DEFAULT_SIZE, default=GRAVATAR_DEFAULT_IMAGE,
        rating=GRAVATAR_DEFAULT_RATING, secure=GRAVATAR_DEFAULT_SECURE):
    """
    Builds a url to a gravatar from an email address.

    :param email: The email to fetch the gravatar for
    :param size: The size (in pixels) of the gravatar to fetch
    :param default: What type of default image to use if the gravatar does not exist
    :param rating: Used to filter the allowed gravatar ratings
    :param secure: If True use https, otherwise plain http
    """
    if secure:
        url_base = GRAVATAR_SECURE_URL
    else:
        url_base = GRAVATAR_URL

    # Calculate the email hash
    email_hash = calculate_gravatar_hash(email)

    # Build querystring
    query_string = urlencode({
        's': str(size),
        'd': default,
        'r': rating,
    })

    # Build url
    url = '{base}avatar/{hash}.jpg?{qs}'.format(base=url_base,
            hash=email_hash, qs=query_string)

    return url

Response: Builds a url to a gravatar from an email address.


Instruction:
Write a Python 3 script to
return True if the user has a gravatar False otherwise.

Response:
def has_gravatar(email):
    """
    Returns True if the user has a gravatar, False if otherwise
    """
    # Request a 404 response if the gravatar does not exist
    url = get_gravatar_url(email, default=GRAVATAR_DEFAULT_IMAGE_404)

    # Verify an OK response was received
    try:
        request = Request(url)
        request.get_method = lambda: 'HEAD'
        return 200 == urlopen(request).code
    except (HTTPError, URLError):
        return False


Instruction:
Can you tell what is the following Python 3 function doing
def get_gravatar_profile_url(email, secure=GRAVATAR_DEFAULT_SECURE):
    if secure:
        url_base = GRAVATAR_SECURE_URL
    else:
        url_base = GRAVATAR_URL

    # Calculate the email hash
    email_hash = calculate_gravatar_hash(email)

    # Build url
    url = '{base}{hash}'.format(base=url_base, hash=email_hash)

    return url

Response: Builds a url to a gravatar profile from an email address.