Instruction:
Can you generate the documentation for the following Python 3 function
def write_object_to_file(self,
                             query_results,
                             filename,
                             fmt="csv",
                             coerce_to_timestamp=False,
                             record_time_added=False):
        """
        Write query results to file.

        Acceptable formats are:
            - csv:
                comma-separated-values file. This is the default format.
            - json:
                JSON array. Each element in the array is a different row.
            - ndjson:
                JSON array but each element is new-line delimited instead of comma delimited like in `json`

        This requires a significant amount of cleanup.
        Pandas doesn't handle output to CSV and json in a uniform way.
        This is especially painful for datetime types.
        Pandas wants to write them as strings in CSV, but as millisecond Unix timestamps.

        By default, this function will try and leave all values as they are represented in Salesforce.
        You use the `coerce_to_timestamp` flag to force all datetimes to become Unix timestamps (UTC).
        This is can be greatly beneficial as it will make all of your datetime fields look the same,
        and makes it easier to work with in other database environments

        :param query_results: the results from a SQL query
        :type query_results: list of dict
        :param filename: the name of the file where the data should be dumped to
        :type filename: str
        :param fmt: the format you want the output in. Default:  'csv'
        :type fmt: str
        :param coerce_to_timestamp: True if you want all datetime fields to be converted into Unix timestamps.
            False if you want them to be left in the same format as they were in Salesforce.
            Leaving the value as False will result in datetimes being strings. Default: False
        :type coerce_to_timestamp: bool
        :param record_time_added: True if you want to add a Unix timestamp field
            to the resulting data that marks when the data was fetched from Salesforce. Default: False
        :type record_time_added: bool
        :return: the dataframe that gets written to the file.
        :rtype: pd.Dataframe
        """
        fmt = fmt.lower()
        if fmt not in ['csv', 'json', 'ndjson']:
            raise ValueError("Format value is not recognized: {}".format(fmt))

        # this line right here will convert all integers to floats
        # if there are any None/np.nan values in the column
        # that's because None/np.nan cannot exist in an integer column
        # we should write all of our timestamps as FLOATS in our final schema
        df = pd.DataFrame.from_records(query_results, exclude=["attributes"])

        df.columns = [column.lower() for column in df.columns]

        # convert columns with datetime strings to datetimes
        # not all strings will be datetimes, so we ignore any errors that occur
        # we get the object's definition at this point and only consider
        # features that are DATE or DATETIME
        if coerce_to_timestamp and df.shape[0] > 0:
            # get the object name out of the query results
            # it's stored in the "attributes" dictionary
            # for each returned record
            object_name = query_results[0]['attributes']['type']

            self.log.info("Coercing timestamps for: %s", object_name)

            schema = self.describe_object(object_name)

            # possible columns that can be converted to timestamps
            # are the ones that are either date or datetime types
            # strings are too general and we risk unintentional conversion
            possible_timestamp_cols = [
                field['name'].lower()
                for field in schema['fields']
                if field['type'] in ["date", "datetime"] and field['name'].lower() in df.columns
            ]
            df[possible_timestamp_cols] = df[possible_timestamp_cols].apply(self._to_timestamp)

        if record_time_added:
            fetched_time = time.time()
            df["time_fetched_from_salesforce"] = fetched_time

        # write the CSV or JSON file depending on the option
        # NOTE:
        #   datetimes here are an issue.
        #   There is no good way to manage the difference
        #   for to_json, the options are an epoch or a ISO string
        #   but for to_csv, it will be a string output by datetime
        #   For JSON we decided to output the epoch timestamp in seconds
        #   (as is fairly standard for JavaScript)
        #   And for csv, we do a string
        if fmt == "csv":
            # there are also a ton of newline objects that mess up our ability to write to csv
            # we remove these newlines so that the output is a valid CSV format
            self.log.info("Cleaning data and writing to CSV")
            possible_strings = df.columns[df.dtypes == "object"]
            df[possible_strings] = df[possible_strings].apply(
                lambda x: x.str.replace("\r\n", "").str.replace("\n", "")
            )
            # write the dataframe
            df.to_csv(filename, index=False)
        elif fmt == "json":
            df.to_json(filename, "records", date_unit="s")
        elif fmt == "ndjson":
            df.to_json(filename, "records", lines=True, date_unit="s")

        return df

Response: Writes the results of a SQL query to a file.


Instruction:
Explain what the following Python 3 code does
def _read(self, ti, try_number, metadata=None):
        # Explicitly getting log relative path is necessary as the given
        # task instance might be different than task instance passed in
        # in set_context method.
        log_relative_path = self._render_filename(ti, try_number)
        remote_loc = os.path.join(self.remote_base, log_relative_path)

        try:
            remote_log = self.gcs_read(remote_loc)
            log = '*** Reading remote log from {}.\n{}\n'.format(
                remote_loc, remote_log)
            return log, {'end_of_log': True}
        except Exception as e:
            log = '*** Unable to read remote log from {}\n*** {}\n\n'.format(
                remote_loc, str(e))
            self.log.error(log)
            local_log, metadata = super()._read(ti, try_number)
            log += local_log
            return log, metadata

Response: Read the log from GCS and return it.


Instruction:
Can you write a function in Python 3 where it
reads the log at the remote_log_location and returns the contents of the file.

Response:
def gcs_read(self, remote_log_location):
        """
        Returns the log found at the remote_log_location.
        :param remote_log_location: the log's location in remote storage
        :type remote_log_location: str (path)
        """
        bkt, blob = self.parse_gcs_url(remote_log_location)
        return self.hook.download(bkt, blob).decode('utf-8')


Instruction:
Here you have a function in Python 3, explain what it does
def gcs_write(self, log, remote_log_location, append=True):
        if append:
            try:
                old_log = self.gcs_read(remote_log_location)
                log = '\n'.join([old_log, log]) if old_log else log
            except Exception as e:
                if not hasattr(e, 'resp') or e.resp.get('status') != '404':
                    log = '*** Previous log discarded: {}\n\n'.format(str(e)) + log

        try:
            bkt, blob = self.parse_gcs_url(remote_log_location)
            from tempfile import NamedTemporaryFile
            with NamedTemporaryFile(mode='w+') as tmpfile:
                tmpfile.write(log)
                # Force the file to be flushed, since we're doing the
                # upload from within the file context (it hasn't been
                # closed).
                tmpfile.flush()
                self.hook.upload(bkt, blob, tmpfile.name)
        except Exception as e:
            self.log.error('Could not write logs to %s: %s', remote_log_location, e)

Response: Writes the log to the remote storage.


Instruction:
How would you code a function in Python 3 to
give a Google Cloud Storage URL returns a tuple containing the corresponding bucket and blob name.

Response:
def parse_gcs_url(gsurl):
        """
        Given a Google Cloud Storage URL (gs://<bucket>/<blob>), returns a
        tuple containing the corresponding bucket and blob.
        """
        parsed_url = urlparse(gsurl)
        if not parsed_url.netloc:
            raise AirflowException('Please provide a bucket name')
        else:
            bucket = parsed_url.netloc
            blob = parsed_url.path.strip('/')
            return bucket, blob


Instruction:
Here you have a function in Python 3, explain what it does
def get_collection(self, mongo_collection, mongo_db=None):
        mongo_db = mongo_db if mongo_db is not None else self.connection.schema
        mongo_conn = self.get_conn()

        return mongo_conn.get_database(mongo_db).get_collection(mongo_collection)

Response: Fetches a mongo collection object for querying.


Instruction:
Make a summary of the following Python 3 code
def aggregate(self, mongo_collection, aggregate_query, mongo_db=None, **kwargs):
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.aggregate(aggregate_query, **kwargs)

Response: Runs an aggregation pipeline and returns the results


Instruction:
Write a Python 3 function that can
run a mongo find query and returns the results

Response:
def find(self, mongo_collection, query, find_one=False, mongo_db=None, **kwargs):
        """
        Runs a mongo find query and returns the results
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find
        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        if find_one:
            return collection.find_one(query, **kwargs)
        else:
            return collection.find(query, **kwargs)


Instruction:
Create a Python 3 function to
insert a single document into a mongo collection

Response:
def insert_one(self, mongo_collection, doc, mongo_db=None, **kwargs):
        """
        Inserts a single document into a mongo collection
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_one
        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.insert_one(doc, **kwargs)


Instruction:
Given the following Python 3 function, write the documentation
def insert_many(self, mongo_collection, docs, mongo_db=None, **kwargs):
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.insert_many(docs, **kwargs)

Response: Insert many docs into a mongo collection.


Instruction:
Explain what the following Python 3 code does
def update_one(self, mongo_collection, filter_doc, update_doc,
                   mongo_db=None, **kwargs):
        """
        Updates a single document in a mongo collection.
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.update_one

        :param mongo_collection: The name of the collection to update.
        :type mongo_collection: str
        :param filter_doc: A query that matches the documents to update.
        :type filter_doc: dict
        :param update_doc: The modifications to apply.
        :type update_doc: dict
        :param mongo_db: The name of the database to use.
            Can be omitted; then the database from the connection string is used.
        :type mongo_db: str

        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.update_one(filter_doc, update_doc, **kwargs)

Response: Updates a single document in a mongo collection.


Instruction:
Write a Python 3 function that can
replace a single document in a mongo collection.

Response:
def replace_one(self, mongo_collection, doc, filter_doc=None,
                    mongo_db=None, **kwargs):
        """
        Replaces a single document in a mongo collection.
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.replace_one

        .. note::
            If no ``filter_doc`` is given, it is assumed that the replacement
            document contain the ``_id`` field which is then used as filters.

        :param mongo_collection: The name of the collection to update.
        :type mongo_collection: str
        :param doc: The new document.
        :type doc: dict
        :param filter_doc: A query that matches the documents to replace.
            Can be omitted; then the _id field from doc will be used.
        :type filter_doc: dict
        :param mongo_db: The name of the database to use.
            Can be omitted; then the database from the connection string is used.
        :type mongo_db: str
        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        if not filter_doc:
            filter_doc = {'_id': doc['_id']}

        return collection.replace_one(filter_doc, doc, **kwargs)


Instruction:
Create a Python 3 function for
replacing many documents in a collection.

Response:
def replace_many(self, mongo_collection, docs,
                     filter_docs=None, mongo_db=None, upsert=False, collation=None,
                     **kwargs):
        """
        Replaces many documents in a mongo collection.

        Uses bulk_write with multiple ReplaceOne operations
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.bulk_write

        .. note::
            If no ``filter_docs``are given, it is assumed that all
            replacement documents contain the ``_id`` field which are then
            used as filters.

        :param mongo_collection: The name of the collection to update.
        :type mongo_collection: str
        :param docs: The new documents.
        :type docs: list[dict]
        :param filter_docs: A list of queries that match the documents to replace.
            Can be omitted; then the _id fields from docs will be used.
        :type filter_docs: list[dict]
        :param mongo_db: The name of the database to use.
            Can be omitted; then the database from the connection string is used.
        :type mongo_db: str
        :param upsert: If ``True``, perform an insert if no documents
            match the filters for the replace operation.
        :type upsert: bool
        :param collation: An instance of
            :class:`~pymongo.collation.Collation`. This option is only
            supported on MongoDB 3.4 and above.
        :type collation: pymongo.collation.Collation

        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        if not filter_docs:
            filter_docs = [{'_id': doc['_id']} for doc in docs]

        requests = [
            ReplaceOne(
                filter_docs[i],
                docs[i],
                upsert=upsert,
                collation=collation)
            for i in range(len(docs))
        ]

        return collection.bulk_write(requests, **kwargs)


Instruction:
Create a Python 3 function to
delete a single document in a mongo collection.

Response:
def delete_one(self, mongo_collection, filter_doc, mongo_db=None, **kwargs):
        """
        Deletes a single document in a mongo collection.
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.delete_one

        :param mongo_collection: The name of the collection to delete from.
        :type mongo_collection: str
        :param filter_doc: A query that matches the document to delete.
        :type filter_doc: dict
        :param mongo_db: The name of the database to use.
            Can be omitted; then the database from the connection string is used.
        :type mongo_db: str

        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.delete_one(filter_doc, **kwargs)


Instruction:
How would you code a function in Python 3 to
delete one or more documents in a mongo collection.

Response:
def delete_many(self, mongo_collection, filter_doc, mongo_db=None, **kwargs):
        """
        Deletes one or more documents in a mongo collection.
        https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.delete_many

        :param mongo_collection: The name of the collection to delete from.
        :type mongo_collection: str
        :param filter_doc: A query that matches the documents to delete.
        :type filter_doc: dict
        :param mongo_db: The name of the database to use.
            Can be omitted; then the database from the connection string is used.
        :type mongo_db: str

        """
        collection = self.get_collection(mongo_collection, mongo_db=mongo_db)

        return collection.delete_many(filter_doc, **kwargs)


Instruction:
Can you implement a function in Python 3 that
checks the mail folder for attachments with the given name.

Response:
def has_mail_attachment(self, name, mail_folder='INBOX', check_regex=False):
        """
        Checks the mail folder for mails containing attachments with the given name.

        :param name: The name of the attachment that will be searched for.
        :type name: str
        :param mail_folder: The mail folder where to look at.
        :type mail_folder: str
        :param check_regex: Checks the name for a regular expression.
        :type check_regex: bool
        :returns: True if there is an attachment with the given name and False if not.
        :rtype: bool
        """
        mail_attachments = self._retrieve_mails_attachments_by_name(name,
                                                                    mail_folder,
                                                                    check_regex,
                                                                    latest_only=True)
        return len(mail_attachments) > 0


Instruction:
Can you write a function in Python 3 where it
retrieves the mail s attachments in the mail folder by its name.

Response:
def retrieve_mail_attachments(self,
                                  name,
                                  mail_folder='INBOX',
                                  check_regex=False,
                                  latest_only=False,
                                  not_found_mode='raise'):
        """
        Retrieves mail's attachments in the mail folder by its name.

        :param name: The name of the attachment that will be downloaded.
        :type name: str
        :param mail_folder: The mail folder where to look at.
        :type mail_folder: str
        :param check_regex: Checks the name for a regular expression.
        :type check_regex: bool
        :param latest_only: If set to True it will only retrieve
                            the first matched attachment.
        :type latest_only: bool
        :param not_found_mode: Specify what should happen if no attachment has been found.
                               Supported values are 'raise', 'warn' and 'ignore'.
                               If it is set to 'raise' it will raise an exception,
                               if set to 'warn' it will only print a warning and
                               if set to 'ignore' it won't notify you at all.
        :type not_found_mode: str
        :returns: a list of tuple each containing the attachment filename and its payload.
        :rtype: a list of tuple
        """
        mail_attachments = self._retrieve_mails_attachments_by_name(name,
                                                                    mail_folder,
                                                                    check_regex,
                                                                    latest_only)
        if not mail_attachments:
            self._handle_not_found_mode(not_found_mode)

        return mail_attachments


Instruction:
Create a Python 3 function to
download the mail s attachments in the mail folder by its name.

Response:
def download_mail_attachments(self,
                                  name,
                                  local_output_directory,
                                  mail_folder='INBOX',
                                  check_regex=False,
                                  latest_only=False,
                                  not_found_mode='raise'):
        """
        Downloads mail's attachments in the mail folder by its name to the local directory.

        :param name: The name of the attachment that will be downloaded.
        :type name: str
        :param local_output_directory: The output directory on the local machine
                                       where the files will be downloaded to.
        :type local_output_directory: str
        :param mail_folder: The mail folder where to look at.
        :type mail_folder: str
        :param check_regex: Checks the name for a regular expression.
        :type check_regex: bool
        :param latest_only: If set to True it will only download
                            the first matched attachment.
        :type latest_only: bool
        :param not_found_mode: Specify what should happen if no attachment has been found.
                               Supported values are 'raise', 'warn' and 'ignore'.
                               If it is set to 'raise' it will raise an exception,
                               if set to 'warn' it will only print a warning and
                               if set to 'ignore' it won't notify you at all.
        :type not_found_mode: str
        """
        mail_attachments = self._retrieve_mails_attachments_by_name(name,
                                                                    mail_folder,
                                                                    check_regex,
                                                                    latest_only)

        if not mail_attachments:
            self._handle_not_found_mode(not_found_mode)

        self._create_files(mail_attachments, local_output_directory)


Instruction:
Create a Python 3 function to
get all attachments by name.

Response:
def get_attachments_by_name(self, name, check_regex, find_first=False):
        """
        Gets all attachments by name for the mail.

        :param name: The name of the attachment to look for.
        :type name: str
        :param check_regex: Checks the name for a regular expression.
        :type check_regex: bool
        :param find_first: If set to True it will only find the first match and then quit.
        :type find_first: bool
        :returns: a list of tuples each containing name and payload
                  where the attachments name matches the given name.
        :rtype: list of tuple
        """
        attachments = []

        for part in self.mail.walk():
            mail_part = MailPart(part)
            if mail_part.is_attachment():
                found_attachment = mail_part.has_matching_name(name) if check_regex \
                    else mail_part.has_equal_name(name)
                if found_attachment:
                    file_name, file_payload = mail_part.get_file()
                    self.log.info('Found attachment: {}'.format(file_name))
                    attachments.append((file_name, file_payload))
                    if find_first:
                        break

        return attachments


Instruction:
Given the following Python 3 function, write the documentation
def get_file(self):
        return self.part.get_filename(), self.part.get_payload(decode=True)

Response: Gets the file including name and payload.


Instruction:
Write a Python 3 function that can
write batch records to Kinesis Firehose

Response:
def put_records(self, records):
        """
        Write batch records to Kinesis Firehose
        """

        firehose_conn = self.get_conn()

        response = firehose_conn.put_record_batch(
            DeliveryStreamName=self.delivery_stream,
            Records=records
        )

        return response


Instruction:
Can you implement a function in Python 3 that
returns a generator that yields the statuses of the dependencies that are in the reschedule period.

Response:
def _get_dep_statuses(self, ti, session, dep_context):
        """
        Determines whether a task is ready to be rescheduled. Only tasks in
        NONE state with at least one row in task_reschedule table are
        handled by this dependency class, otherwise this dependency is
        considered as passed. This dependency fails if the latest reschedule
        request's reschedule date is still in future.
        """
        if dep_context.ignore_in_reschedule_period:
            yield self._passing_status(
                reason="The context specified that being in a reschedule period was "
                       "permitted.")
            return

        if ti.state not in self.RESCHEDULEABLE_STATES:
            yield self._passing_status(
                reason="The task instance is not in State_UP_FOR_RESCHEDULE or NONE state.")
            return

        task_reschedules = TaskReschedule.find_for_task_instance(task_instance=ti)
        if not task_reschedules:
            yield self._passing_status(
                reason="There is no reschedule request for this task instance.")
            return

        now = timezone.utcnow()
        next_reschedule_date = task_reschedules[-1].reschedule_date
        if now >= next_reschedule_date:
            yield self._passing_status(
                reason="Task instance id ready for reschedule.")
            return

        yield self._failing_status(
            reason="Task is not ready for reschedule yet but will be rescheduled "
                   "automatically. Current date is {0} and task will be rescheduled "
                   "at {1}.".format(now.isoformat(), next_reschedule_date.isoformat()))


Instruction:
Can you implement a function in Python 3 that
sends email using the default backend.

Response:
def send_email(to, subject, html_content,
               files=None, dryrun=False, cc=None, bcc=None,
               mime_subtype='mixed', mime_charset='utf-8', **kwargs):
    """
    Send email using backend specified in EMAIL_BACKEND.
    """
    path, attr = configuration.conf.get('email', 'EMAIL_BACKEND').rsplit('.', 1)
    module = importlib.import_module(path)
    backend = getattr(module, attr)
    to = get_email_address_list(to)
    to = ", ".join(to)

    return backend(to, subject, html_content, files=files,
                   dryrun=dryrun, cc=cc, bcc=bcc,
                   mime_subtype=mime_subtype, mime_charset=mime_charset, **kwargs)


Instruction:
Here you have a function in Python 3, explain what it does
def send_email_smtp(to, subject, html_content, files=None,
                    dryrun=False, cc=None, bcc=None,
                    mime_subtype='mixed', mime_charset='utf-8',
                    **kwargs):
    """
    Send an email with html content

    >>> send_email('test@example.com', 'foo', '<b>Foo</b> bar', ['/dev/null'], dryrun=True)
    """
    smtp_mail_from = configuration.conf.get('smtp', 'SMTP_MAIL_FROM')

    to = get_email_address_list(to)

    msg = MIMEMultipart(mime_subtype)
    msg['Subject'] = subject
    msg['From'] = smtp_mail_from
    msg['To'] = ", ".join(to)
    recipients = to
    if cc:
        cc = get_email_address_list(cc)
        msg['CC'] = ", ".join(cc)
        recipients = recipients + cc

    if bcc:
        # don't add bcc in header
        bcc = get_email_address_list(bcc)
        recipients = recipients + bcc

    msg['Date'] = formatdate(localtime=True)
    mime_text = MIMEText(html_content, 'html', mime_charset)
    msg.attach(mime_text)

    for fname in files or []:
        basename = os.path.basename(fname)
        with open(fname, "rb") as f:
            part = MIMEApplication(
                f.read(),
                Name=basename
            )
            part['Content-Disposition'] = 'attachment; filename="%s"' % basename
            part['Content-ID'] = '<%s>' % basename
            msg.attach(part)

    send_MIME_email(smtp_mail_from, recipients, msg, dryrun)

Response: Send an email with html content


Instruction:
Create a Python 3 function for
processing the result value of a get_record_or_record_or_record_or_record call.

Response:
def process_result_value(self, value, dialect):
        """
        Processes DateTimes from the DB making sure it is always
        returning UTC. Not using timezone.convert_to_utc as that
        converts to configured TIMEZONE while the DB might be
        running with some other setting. We assume UTC datetimes
        in the database.
        """
        if value is not None:
            if value.tzinfo is None:
                value = value.replace(tzinfo=utc)
            else:
                value = value.astimezone(utc)

        return value


Instruction:
How would you explain what the following Python 3 function does
def check_for_blob(self, container_name, blob_name, **kwargs):
        return self.connection.exists(container_name, blob_name, **kwargs)

Response: Check if a blob exists on Azure Blob Storage.


Instruction:
Here you have a function in Python 3, explain what it does
def check_for_prefix(self, container_name, prefix, **kwargs):
        matches = self.connection.list_blobs(container_name, prefix,
                                             num_results=1, **kwargs)
        return len(list(matches)) > 0

Response: Check if a prefix exists on Azure Blob storage.


Instruction:
Write a Python 3 script for
uploading a file to Azure Blob Storage.

Response:
def load_file(self, file_path, container_name, blob_name, **kwargs):
        """
        Upload a file to Azure Blob Storage.

        :param file_path: Path to the file to load.
        :type file_path: str
        :param container_name: Name of the container.
        :type container_name: str
        :param blob_name: Name of the blob.
        :type blob_name: str
        :param kwargs: Optional keyword arguments that
            `BlockBlobService.create_blob_from_path()` takes.
        :type kwargs: object
        """
        # Reorder the argument order from airflow.hooks.S3_hook.load_file.
        self.connection.create_blob_from_path(container_name, blob_name,
                                              file_path, **kwargs)


Instruction:
How would you code a function in Python 3 to
upload a string to Azure Blob Storage.

Response:
def load_string(self, string_data, container_name, blob_name, **kwargs):
        """
        Upload a string to Azure Blob Storage.

        :param string_data: String to load.
        :type string_data: str
        :param container_name: Name of the container.
        :type container_name: str
        :param blob_name: Name of the blob.
        :type blob_name: str
        :param kwargs: Optional keyword arguments that
            `BlockBlobService.create_blob_from_text()` takes.
        :type kwargs: object
        """
        # Reorder the argument order from airflow.hooks.S3_hook.load_string.
        self.connection.create_blob_from_text(container_name, blob_name,
                                              string_data, **kwargs)


Instruction:
Here you have a function in Python 3, explain what it does
def get_file(self, file_path, container_name, blob_name, **kwargs):
        return self.connection.get_blob_to_path(container_name, blob_name,
                                                file_path, **kwargs)

Response: Download a file from Azure Blob Storage.


Instruction:
How would you explain what the following Python 3 function does
def read_file(self, container_name, blob_name, **kwargs):
        return self.connection.get_blob_to_text(container_name,
                                                blob_name,
                                                **kwargs).content

Response: Reads a file from Azure Blob Storage and returns as a string.


Instruction:
Explain what the following Python 3 code does
def delete_file(self, container_name, blob_name, is_prefix=False,
                    ignore_if_missing=False, **kwargs):
        """
        Delete a file from Azure Blob Storage.

        :param container_name: Name of the container.
        :type container_name: str
        :param blob_name: Name of the blob.
        :type blob_name: str
        :param is_prefix: If blob_name is a prefix, delete all matching files
        :type is_prefix: bool
        :param ignore_if_missing: if True, then return success even if the
            blob does not exist.
        :type ignore_if_missing: bool
        :param kwargs: Optional keyword arguments that
            `BlockBlobService.create_blob_from_path()` takes.
        :type kwargs: object
        """

        if is_prefix:
            blobs_to_delete = [
                blob.name for blob in self.connection.list_blobs(
                    container_name, prefix=blob_name, **kwargs
                )
            ]
        elif self.check_for_blob(container_name, blob_name):
            blobs_to_delete = [blob_name]
        else:
            blobs_to_delete = []

        if not ignore_if_missing and len(blobs_to_delete) == 0:
            raise AirflowException('Blob(s) not found: {}'.format(blob_name))

        for blob_uri in blobs_to_delete:
            self.log.info("Deleting blob: " + blob_uri)
            self.connection.delete_blob(container_name,
                                        blob_uri,
                                        delete_snapshots='include',
                                        **kwargs)

Response: Delete a file from Azure Blob Storage.


Instruction:
Make a summary of the following Python 3 code
def get_conn(self):
        if self.conn is None:
            params = self.get_connection(self.ftp_conn_id)
            pasv = params.extra_dejson.get("passive", True)
            self.conn = ftplib.FTP(params.host, params.login, params.password)
            self.conn.set_pasv(pasv)

        return self.conn

Response: Returns a FTP connection object.


Instruction:
How would you implement a function in Python 3 that
returns a dictionary of all files in the remote directory

Response:
def describe_directory(self, path):
        """
        Returns a dictionary of {filename: {attributes}} for all files
        on the remote system (where the MLSD command is supported).

        :param path: full path to the remote directory
        :type path: str
        """
        conn = self.get_conn()
        conn.cwd(path)
        try:
            # only works in Python 3
            files = dict(conn.mlsd())
        except AttributeError:
            files = dict(mlsd(conn))
        return files


Instruction:
Can you generate a brief explanation for the following Python 3 code
def list_directory(self, path, nlst=False):
        conn = self.get_conn()
        conn.cwd(path)

        files = conn.nlst()
        return files

Response: Returns a list of files on the remote system.


Instruction:
Here you have a function in Python 3, explain what it does
def retrieve_file(
            self,
            remote_full_path,
            local_full_path_or_buffer,
            callback=None):
        """
        Transfers the remote file to a local location.

        If local_full_path_or_buffer is a string path, the file will be put
        at that location; if it is a file-like buffer, the file will
        be written to the buffer but not closed.

        :param remote_full_path: full path to the remote file
        :type remote_full_path: str
        :param local_full_path_or_buffer: full path to the local file or a
            file-like buffer
        :type local_full_path_or_buffer: str or file-like buffer
        :param callback: callback which is called each time a block of data
            is read. if you do not use a callback, these blocks will be written
            to the file or buffer passed in. if you do pass in a callback, note
            that writing to a file or buffer will need to be handled inside the
            callback.
            [default: output_handle.write()]
        :type callback: callable

        :Example::

            hook = FTPHook(ftp_conn_id='my_conn')

            remote_path = '/path/to/remote/file'
            local_path = '/path/to/local/file'

            # with a custom callback (in this case displaying progress on each read)
            def print_progress(percent_progress):
                self.log.info('Percent Downloaded: %s%%' % percent_progress)

            total_downloaded = 0
            total_file_size = hook.get_size(remote_path)
            output_handle = open(local_path, 'wb')
            def write_to_file_with_progress(data):
                total_downloaded += len(data)
                output_handle.write(data)
                percent_progress = (total_downloaded / total_file_size) * 100
                print_progress(percent_progress)
            hook.retrieve_file(remote_path, None, callback=write_to_file_with_progress)

            # without a custom callback data is written to the local_path
            hook.retrieve_file(remote_path, local_path)
        """
        conn = self.get_conn()

        is_path = isinstance(local_full_path_or_buffer, basestring)

        # without a callback, default to writing to a user-provided file or
        # file-like buffer
        if not callback:
            if is_path:
                output_handle = open(local_full_path_or_buffer, 'wb')
            else:
                output_handle = local_full_path_or_buffer
            callback = output_handle.write
        else:
            output_handle = None

        remote_path, remote_file_name = os.path.split(remote_full_path)
        conn.cwd(remote_path)
        self.log.info('Retrieving file from FTP: %s', remote_full_path)
        conn.retrbinary('RETR %s' % remote_file_name, callback)
        self.log.info('Finished retrieving file from FTP: %s', remote_full_path)

        if is_path and output_handle:
            output_handle.close()

Response: This method will retrieve the remote file to a local location.


Instruction:
Can you write a function in Python 3 where it
stores a local file to the remote location.

Response:
def store_file(self, remote_full_path, local_full_path_or_buffer):
        """
        Transfers a local file to the remote location.

        If local_full_path_or_buffer is a string path, the file will be read
        from that location; if it is a file-like buffer, the file will
        be read from the buffer but not closed.

        :param remote_full_path: full path to the remote file
        :type remote_full_path: str
        :param local_full_path_or_buffer: full path to the local file or a
            file-like buffer
        :type local_full_path_or_buffer: str or file-like buffer
        """
        conn = self.get_conn()

        is_path = isinstance(local_full_path_or_buffer, basestring)

        if is_path:
            input_handle = open(local_full_path_or_buffer, 'rb')
        else:
            input_handle = local_full_path_or_buffer
        remote_path, remote_file_name = os.path.split(remote_full_path)
        conn.cwd(remote_path)
        conn.storbinary('STOR %s' % remote_file_name, input_handle)

        if is_path:
            input_handle.close()


Instruction:
Can you create a Python 3 function that
renames a file from name to name

Response:
def rename(self, from_name, to_name):
        """
        Rename a file.

        :param from_name: rename file from name
        :param to_name: rename file to name
        """
        conn = self.get_conn()
        return conn.rename(from_name, to_name)


Instruction:
Write a Python 3 function for
returning a datetime object representing the last time the file was modified.

Response:
def get_mod_time(self, path):
        """
        Returns a datetime object representing the last time the file was modified

        :param path: remote file path
        :type path: string
        """
        conn = self.get_conn()
        ftp_mdtm = conn.sendcmd('MDTM ' + path)
        time_val = ftp_mdtm[4:]
        # time_val optionally has microseconds
        try:
            return datetime.datetime.strptime(time_val, "%Y%m%d%H%M%S.%f")
        except ValueError:
            return datetime.datetime.strptime(time_val, '%Y%m%d%H%M%S')


Instruction:
Make a summary of the following Python 3 code
def _create_dagruns(dag, execution_dates, state, run_id_template):
    # find out if we need to create any dag runs
    drs = DagRun.find(dag_id=dag.dag_id, execution_date=execution_dates)
    dates_to_create = list(set(execution_dates) - set([dr.execution_date for dr in drs]))

    for date in dates_to_create:
        dr = dag.create_dagrun(
            run_id=run_id_template.format(date.isoformat()),
            execution_date=date,
            start_date=timezone.utcnow(),
            external_trigger=False,
            state=state,
        )
        drs.append(dr)

    return drs

Response: Create all the dag runs for the given execution dates.


Instruction:
Create a Python 3 function to
set the state of a task instance and if needed its relatives.

Response:
def set_state(task, execution_date, upstream=False, downstream=False,
              future=False, past=False, state=State.SUCCESS, commit=False, session=None):
    """
    Set the state of a task instance and if needed its relatives. Can set state
    for future tasks (calculated from execution_date) and retroactively
    for past tasks. Will verify integrity of past dag runs in order to create
    tasks that did not exist. It will not create dag runs that are missing
    on the schedule (but it will as for subdag dag runs if needed).
    :param task: the task from which to work. task.task.dag needs to be set
    :param execution_date: the execution date from which to start looking
    :param upstream: Mark all parents (upstream tasks)
    :param downstream: Mark all siblings (downstream tasks) of task_id, including SubDags
    :param future: Mark all future tasks on the interval of the dag up until
        last execution date.
    :param past: Retroactively mark all tasks starting from start_date of the DAG
    :param state: State to which the tasks need to be set
    :param commit: Commit tasks to be altered to the database
    :param session: database session
    :return: list of tasks that have been created and updated
    """
    assert timezone.is_localized(execution_date)

    assert task.dag is not None
    dag = task.dag

    latest_execution_date = dag.latest_execution_date
    assert latest_execution_date is not None

    # determine date range of dag runs and tasks to consider
    end_date = latest_execution_date if future else execution_date

    if 'start_date' in dag.default_args:
        start_date = dag.default_args['start_date']
    elif dag.start_date:
        start_date = dag.start_date
    else:
        start_date = execution_date

    start_date = execution_date if not past else start_date

    if dag.schedule_interval == '@once':
        dates = [start_date]
    else:
        dates = dag.date_range(start_date=start_date, end_date=end_date)

    # find relatives (siblings = downstream, parents = upstream) if needed
    task_ids = [task.task_id]
    if downstream:
        relatives = task.get_flat_relatives(upstream=False)
        task_ids += [t.task_id for t in relatives]
    if upstream:
        relatives = task.get_flat_relatives(upstream=True)
        task_ids += [t.task_id for t in relatives]

    # verify the integrity of the dag runs in case a task was added or removed
    # set the confirmed execution dates as they might be different
    # from what was provided
    confirmed_dates = []
    drs = DagRun.find(dag_id=dag.dag_id, execution_date=dates)
    for dr in drs:
        dr.dag = dag
        dr.verify_integrity()
        confirmed_dates.append(dr.execution_date)

    # go through subdagoperators and create dag runs. We will only work
    # within the scope of the subdag. We wont propagate to the parent dag,
    # but we will propagate from parent to subdag.
    dags = [dag]
    sub_dag_ids = []
    while len(dags) > 0:
        current_dag = dags.pop()
        for task_id in task_ids:
            if not current_dag.has_task(task_id):
                continue

            current_task = current_dag.get_task(task_id)
            if isinstance(current_task, SubDagOperator):
                # this works as a kind of integrity check
                # it creates missing dag runs for subdagoperators,
                # maybe this should be moved to dagrun.verify_integrity
                drs = _create_dagruns(current_task.subdag,
                                      execution_dates=confirmed_dates,
                                      state=State.RUNNING,
                                      run_id_template=BackfillJob.ID_FORMAT_PREFIX)

                for dr in drs:
                    dr.dag = current_task.subdag
                    dr.verify_integrity()
                    if commit:
                        dr.state = state
                        session.merge(dr)

                dags.append(current_task.subdag)
                sub_dag_ids.append(current_task.subdag.dag_id)

    # now look for the task instances that are affected
    TI = TaskInstance

    # get all tasks of the main dag that will be affected by a state change
    qry_dag = session.query(TI).filter(
        TI.dag_id == dag.dag_id,
        TI.execution_date.in_(confirmed_dates),
        TI.task_id.in_(task_ids)).filter(
        or_(TI.state.is_(None),
            TI.state != state)
    )

    # get *all* tasks of the sub dags
    if len(sub_dag_ids) > 0:
        qry_sub_dag = session.query(TI).filter(
            TI.dag_id.in_(sub_dag_ids),
            TI.execution_date.in_(confirmed_dates)).filter(
            or_(TI.state.is_(None),
                TI.state != state)
        )

    if commit:
        tis_altered = qry_dag.with_for_update().all()
        if len(sub_dag_ids) > 0:
            tis_altered += qry_sub_dag.with_for_update().all()
        for ti in tis_altered:
            ti.state = state
    else:
        tis_altered = qry_dag.all()
        if len(sub_dag_ids) > 0:
            tis_altered += qry_sub_dag.all()

    return tis_altered


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _set_dag_run_state(dag_id, execution_date, state, session=None):
    DR = DagRun
    dr = session.query(DR).filter(
        DR.dag_id == dag_id,
        DR.execution_date == execution_date
    ).one()
    dr.state = state
    if state == State.RUNNING:
        dr.start_date = timezone.utcnow()
        dr.end_date = None
    else:
        dr.end_date = timezone.utcnow()
    session.merge(dr)

Response: Helper method that set the dag run state in the DB.


Instruction:
Here you have a function in Python 3, explain what it does
def set_dag_run_state_to_success(dag, execution_date, commit=False, session=None):
    res = []

    if not dag or not execution_date:
        return res

    # Mark the dag run to success.
    if commit:
        _set_dag_run_state(dag.dag_id, execution_date, State.SUCCESS, session)

    # Mark all task instances of the dag run to success.
    for task in dag.tasks:
        task.dag = dag
        new_state = set_state(task=task, execution_date=execution_date,
                              state=State.SUCCESS, commit=commit)
        res.extend(new_state)

    return res

Response: Set the dag run to success.


Instruction:
Make a summary of the following Python 3 code
def set_dag_run_state_to_failed(dag, execution_date, commit=False, session=None):
    res = []

    if not dag or not execution_date:
        return res

    # Mark the dag run to failed.
    if commit:
        _set_dag_run_state(dag.dag_id, execution_date, State.FAILED, session)

    # Mark only RUNNING task instances.
    TI = TaskInstance
    task_ids = [task.task_id for task in dag.tasks]
    tis = session.query(TI).filter(
        TI.dag_id == dag.dag_id,
        TI.execution_date == execution_date,
        TI.task_id.in_(task_ids)).filter(TI.state == State.RUNNING)
    task_ids_of_running_tis = [ti.task_id for ti in tis]
    for task in dag.tasks:
        if task.task_id not in task_ids_of_running_tis:
            continue
        task.dag = dag
        new_state = set_state(task=task, execution_date=execution_date,
                              state=State.FAILED, commit=commit)
        res.extend(new_state)

    return res

Response: Sets the dag run to failed.


Instruction:
Write a Python 3 script to
set the dag run to running.

Response:
def set_dag_run_state_to_running(dag, execution_date, commit=False, session=None):
    """
    Set the dag run for a specific execution date to running.
    :param dag: the DAG of which to alter state
    :param execution_date: the execution date from which to start looking
    :param commit: commit DAG and tasks to be altered to the database
    :param session: database session
    :return: If commit is true, list of tasks that have been updated,
             otherwise list of tasks that will be updated
    """
    res = []
    if not dag or not execution_date:
        return res

    # Mark the dag run to running.
    if commit:
        _set_dag_run_state(dag.dag_id, execution_date, State.RUNNING, session)

    # To keep the return type consistent with the other similar functions.
    return res


Instruction:
Can you generate the documentation for the following Python 3 function
def git_version(version):
    repo = None
    try:
        import git
        repo = git.Repo('.git')
    except ImportError:
        logger.warning('gitpython not found: Cannot compute the git version.')
        return ''
    except Exception as e:
        logger.warning('Cannot compute the git version. {}'.format(e))
        return ''
    if repo:
        sha = repo.head.commit.hexsha
        if repo.is_dirty():
            return '.dev0+{sha}.dirty'.format(sha=sha)
        # commit is clean
        return '.release:{version}+{sha}'.format(version=version, sha=sha)
    else:
        return 'no_git_version'

Response: Returns a version to identify the state of the git repo.


Instruction:
Explain what the following Python 3 code does
def execute(self, context):
        self.hook = DiscordWebhookHook(
            self.http_conn_id,
            self.webhook_endpoint,
            self.message,
            self.username,
            self.avatar_url,
            self.tts,
            self.proxy
        )
        self.hook.execute()

Response: Call the DiscordWebhookHook to post message


Instruction:
Can you implement a function in Python 3 that
validates the field is OK.

Response:
def _validate_field(self, validation_spec, dictionary_to_validate, parent=None,
                        force_optional=False):
        """
        Validates if field is OK.

        :param validation_spec: specification of the field
        :type validation_spec: dict
        :param dictionary_to_validate: dictionary where the field should be present
        :type dictionary_to_validate: dict
        :param parent: full path of parent field
        :type parent: str
        :param force_optional: forces the field to be optional
            (all union fields have force_optional set to True)
        :type force_optional: bool
        :return: True if the field is present
        """
        field_name = validation_spec['name']
        field_type = validation_spec.get('type')
        optional = validation_spec.get('optional')
        regexp = validation_spec.get('regexp')
        allow_empty = validation_spec.get('allow_empty')
        children_validation_specs = validation_spec.get('fields')
        required_api_version = validation_spec.get('api_version')
        custom_validation = validation_spec.get('custom_validation')

        full_field_path = self._get_field_name_with_parent(field_name=field_name,
                                                           parent=parent)
        if required_api_version and required_api_version != self._api_version:
            self.log.debug(
                "Skipping validation of the field '%s' for API version '%s' "
                "as it is only valid for API version '%s'",
                field_name, self._api_version, required_api_version)
            return False
        value = dictionary_to_validate.get(field_name)

        if (optional or force_optional) and value is None:
            self.log.debug("The optional field '%s' is missing. That's perfectly OK.", full_field_path)
            return False

        # Certainly down from here the field is present (value is not None)
        # so we should only return True from now on

        self._sanity_checks(children_validation_specs=children_validation_specs,
                            field_type=field_type,
                            full_field_path=full_field_path,
                            regexp=regexp,
                            allow_empty=allow_empty,
                            custom_validation=custom_validation,
                            value=value)

        if allow_empty is False:
            self._validate_is_empty(full_field_path, value)
        if regexp:
            self._validate_regexp(full_field_path, regexp, value)
        elif field_type == 'dict':
            if not isinstance(value, dict):
                raise GcpFieldValidationException(
                    "The field '{}' should be of dictionary type according to the "
                    "specification '{}' but it is '{}'".
                    format(full_field_path, validation_spec, value))
            if children_validation_specs is None:
                self.log.debug(
                    "The dict field '%s' has no nested fields defined in the "
                    "specification '%s'. That's perfectly ok - it's content will "
                    "not be validated.", full_field_path, validation_spec)
            else:
                self._validate_dict(children_validation_specs, full_field_path, value)
        elif field_type == 'union':
            if not children_validation_specs:
                raise GcpValidationSpecificationException(
                    "The union field '%s' has no nested fields "
                    "defined in specification '%s'. Unions should have at least one "
                    "nested field defined.", full_field_path, validation_spec)
            self._validate_union(children_validation_specs, full_field_path,
                                 dictionary_to_validate)
        elif field_type == 'list':
            if not isinstance(value, list):
                raise GcpFieldValidationException(
                    "The field '{}' should be of list type according to the "
                    "specification '{}' but it is '{}'".
                    format(full_field_path, validation_spec, value))
        elif custom_validation:
            try:
                custom_validation(value)
            except Exception as e:
                raise GcpFieldValidationException(
                    "Error while validating custom field '{}' specified by '{}': '{}'".
                    format(full_field_path, validation_spec, e))
        elif field_type is None:
            self.log.debug("The type of field '%s' is not specified in '%s'. "
                           "Not validating its content.", full_field_path, validation_spec)
        else:
            raise GcpValidationSpecificationException(
                "The field '{}' is of type '{}' in specification '{}'."
                "This type is unknown to validation!".format(
                    full_field_path, field_type, validation_spec))
        return True


Instruction:
Make a summary of the following Python 3 code
def validate(self, body_to_validate):
        try:
            for validation_spec in self._validation_specs:
                self._validate_field(validation_spec=validation_spec,
                                     dictionary_to_validate=body_to_validate)
        except GcpFieldValidationException as e:
            raise GcpFieldValidationException(
                "There was an error when validating: body '{}': '{}'".
                format(body_to_validate, e))
        all_field_names = [spec['name'] for spec in self._validation_specs
                           if spec.get('type') != 'union' and
                           spec.get('api_version') != self._api_version]
        all_union_fields = [spec for spec in self._validation_specs
                            if spec.get('type') == 'union']
        for union_field in all_union_fields:
            all_field_names.extend(
                [nested_union_spec['name'] for nested_union_spec in union_field['fields']
                 if nested_union_spec.get('type') != 'union' and
                 nested_union_spec.get('api_version') != self._api_version])
        for field_name in body_to_validate.keys():
            if field_name not in all_field_names:
                self.log.warning(
                    "The field '%s' is in the body, but is not specified in the "
                    "validation specification '%s'. "
                    "This might be because you are using newer API version and "
                    "new field names defined for that version. Then the warning "
                    "can be safely ignored, or you might want to upgrade the operator"
                    "to the version that supports the new API version.",
                    field_name, self._validation_specs)

Response: Validates that the body of the object is well - formed.


Instruction:
Can you create a Python 3 function that
returns the FileService object.

Response:
def get_conn(self):
        """Return the FileService object."""
        conn = self.get_connection(self.conn_id)
        service_options = conn.extra_dejson
        return FileService(account_name=conn.login,
                           account_key=conn.password, **service_options)


Instruction:
Write a Python 3 function for
checking if a directory exists on Azure File Share.

Response:
def check_for_directory(self, share_name, directory_name, **kwargs):
        """
        Check if a directory exists on Azure File Share.

        :param share_name: Name of the share.
        :type share_name: str
        :param directory_name: Name of the directory.
        :type directory_name: str
        :param kwargs: Optional keyword arguments that
            `FileService.exists()` takes.
        :type kwargs: object
        :return: True if the file exists, False otherwise.
        :rtype: bool
        """
        return self.connection.exists(share_name, directory_name,
                                      **kwargs)


Instruction:
Implement a Python 3 function for
checking if a file exists on Azure File Share.

Response:
def check_for_file(self, share_name, directory_name, file_name, **kwargs):
        """
        Check if a file exists on Azure File Share.

        :param share_name: Name of the share.
        :type share_name: str
        :param directory_name: Name of the directory.
        :type directory_name: str
        :param file_name: Name of the file.
        :type file_name: str
        :param kwargs: Optional keyword arguments that
            `FileService.exists()` takes.
        :type kwargs: object
        :return: True if the file exists, False otherwise.
        :rtype: bool
        """
        return self.connection.exists(share_name, directory_name,
                                      file_name, **kwargs)


Instruction:
Can you write a function in Python 3 where it
returns the list of directories and files stored on a Azure File Share.

Response:
def list_directories_and_files(self, share_name, directory_name=None, **kwargs):
        """
        Return the list of directories and files stored on a Azure File Share.

        :param share_name: Name of the share.
        :type share_name: str
        :param directory_name: Name of the directory.
        :type directory_name: str
        :param kwargs: Optional keyword arguments that
            `FileService.list_directories_and_files()` takes.
        :type kwargs: object
        :return: A list of files and directories
        :rtype: list
        """
        return self.connection.list_directories_and_files(share_name,
                                                          directory_name,
                                                          **kwargs)


Instruction:
Can you tell what is the following Python 3 function doing
def create_directory(self, share_name, directory_name, **kwargs):
        return self.connection.create_directory(share_name, directory_name, **kwargs)

Response: Creates a new directory on a Azure File Share.


Instruction:
Write a Python 3 script to
download a file from Azure File Share.

Response:
def get_file(self, file_path, share_name, directory_name, file_name, **kwargs):
        """
        Download a file from Azure File Share.

        :param file_path: Where to store the file.
        :type file_path: str
        :param share_name: Name of the share.
        :type share_name: str
        :param directory_name: Name of the directory.
        :type directory_name: str
        :param file_name: Name of the file.
        :type file_name: str
        :param kwargs: Optional keyword arguments that
            `FileService.get_file_to_path()` takes.
        :type kwargs: object
        """
        self.connection.get_file_to_path(share_name, directory_name,
                                         file_name, file_path, **kwargs)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_file_to_stream(self, stream, share_name, directory_name, file_name, **kwargs):
        self.connection.get_file_to_stream(share_name, directory_name,
                                           file_name, stream, **kwargs)

Response: Download a file from the given stream


Instruction:
Given the following Python 3 function, write the documentation
def load_file(self, file_path, share_name, directory_name, file_name, **kwargs):
        self.connection.create_file_from_path(share_name, directory_name,
                                              file_name, file_path, **kwargs)

Response: Upload a file to an Azure File Share.


Instruction:
How would you explain what the following Python 3 function does
def load_string(self, string_data, share_name, directory_name, file_name, **kwargs):
        self.connection.create_file_from_text(share_name, directory_name,
                                              file_name, string_data, **kwargs)

Response: This method uploads a string to the Azure File Share.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def load_stream(self, stream, share_name, directory_name, file_name, count, **kwargs):
        self.connection.create_file_from_stream(share_name, directory_name,
                                                file_name, stream, count, **kwargs)

Response: Uploads a stream to the Azure File Share.


Instruction:
Create a Python 3 function for
setting the file name and the logging handler to be used for airflow tasks.

Response:
def set_context(self, filename):
        """
        Provide filename context to airflow task handler.
        :param filename: filename in which the dag is located
        """
        local_loc = self._init_file(filename)
        self.handler = logging.FileHandler(local_loc)
        self.handler.setFormatter(self.formatter)
        self.handler.setLevel(self.level)

        if self._cur_date < datetime.today():
            self._symlink_latest_log_directory()
            self._cur_date = datetime.today()


Instruction:
Write a Python 3 script to
create log file and directory if required.

Response:
def _init_file(self, filename):
        """
        Create log file and directory if required.
        :param filename: task instance object
        :return: relative log path of the given task instance
        """
        relative_path = self._render_filename(filename)
        full_path = os.path.join(self._get_log_directory(), relative_path)
        directory = os.path.dirname(full_path)

        if not os.path.exists(directory):
            try:
                os.makedirs(directory)
            except OSError:
                if not os.path.isdir(directory):
                    raise

        if not os.path.exists(full_path):
            open(full_path, "a").close()

        return full_path


Instruction:
Make a summary of the following Python 3 code
def _parse_gcs_url(gsurl):

    parsed_url = urlparse(gsurl)
    if not parsed_url.netloc:
        raise AirflowException('Please provide a bucket name')
    else:
        bucket = parsed_url.netloc
        # Remove leading '/' but NOT trailing one
        blob = parsed_url.path.lstrip('/')
        return bucket, blob

Response: Given a Google Cloud Storage URL returns a tuple containing the corresponding bucket and blob name.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_conn(self):
        if not self._conn:
            self._conn = storage.Client(credentials=self._get_credentials())

        return self._conn

Response: Returns a Google Cloud Storage service object.


Instruction:
Write a Python 3 script for
copying an object from one bucket to another.

Response:
def copy(self, source_bucket, source_object, destination_bucket=None,
             destination_object=None):
        """
        Copies an object from a bucket to another, with renaming if requested.

        destination_bucket or destination_object can be omitted, in which case
        source bucket/object is used, but not both.

        :param source_bucket: The bucket of the object to copy from.
        :type source_bucket: str
        :param source_object: The object to copy.
        :type source_object: str
        :param destination_bucket: The destination of the object to copied to.
            Can be omitted; then the same bucket is used.
        :type destination_bucket: str
        :param destination_object: The (renamed) path of the object if given.
            Can be omitted; then the same name is used.
        :type destination_object: str
        """
        destination_bucket = destination_bucket or source_bucket
        destination_object = destination_object or source_object
        if source_bucket == destination_bucket and \
                source_object == destination_object:

            raise ValueError(
                'Either source/destination bucket or source/destination object '
                'must be different, not both the same: bucket=%s, object=%s' %
                (source_bucket, source_object))
        if not source_bucket or not source_object:
            raise ValueError('source_bucket and source_object cannot be empty.')

        client = self.get_conn()
        source_bucket = client.get_bucket(source_bucket)
        source_object = source_bucket.blob(source_object)
        destination_bucket = client.get_bucket(destination_bucket)
        destination_object = source_bucket.copy_blob(
            blob=source_object,
            destination_bucket=destination_bucket,
            new_name=destination_object)

        self.log.info('Object %s in bucket %s copied to object %s in bucket %s',
                      source_object.name, source_bucket.name,
                      destination_object.name, destination_bucket.name)


Instruction:
Create a Python 3 function to
have the same functionality as copy, except that will work on files over 5 TB, as well as when copying between locations and/or storage classes. destination_object can be omitted, in which case source_object is used. :param source_bucket: The bucket of the object to copy from. :type source_bucket: str :param source_object: The object to copy. :type source_object: str :param destination_bucket: The destination of the object to copied to. :type destination_bucket: str :param destination_object: The (renamed) path of the object if given. Can be omitted; then the same name is used. :type destination_object: str

Response:
def rewrite(self, source_bucket, source_object, destination_bucket,
                destination_object=None):
        """
        Has the same functionality as copy, except that will work on files
        over 5 TB, as well as when copying between locations and/or storage
        classes.

        destination_object can be omitted, in which case source_object is used.

        :param source_bucket: The bucket of the object to copy from.
        :type source_bucket: str
        :param source_object: The object to copy.
        :type source_object: str
        :param destination_bucket: The destination of the object to copied to.
        :type destination_bucket: str
        :param destination_object: The (renamed) path of the object if given.
            Can be omitted; then the same name is used.
        :type destination_object: str
        """
        destination_object = destination_object or source_object
        if (source_bucket == destination_bucket and
                source_object == destination_object):
            raise ValueError(
                'Either source/destination bucket or source/destination object '
                'must be different, not both the same: bucket=%s, object=%s' %
                (source_bucket, source_object))
        if not source_bucket or not source_object:
            raise ValueError('source_bucket and source_object cannot be empty.')

        client = self.get_conn()
        source_bucket = client.get_bucket(bucket_name=source_bucket)
        source_object = source_bucket.blob(blob_name=source_object)
        destination_bucket = client.get_bucket(bucket_name=destination_bucket)

        token, bytes_rewritten, total_bytes = destination_bucket.blob(
            blob_name=destination_object).rewrite(
            source=source_object
        )

        self.log.info('Total Bytes: %s | Bytes Written: %s',
                      total_bytes, bytes_rewritten)

        while token is not None:
            token, bytes_rewritten, total_bytes = destination_bucket.blob(
                blob_name=destination_object).rewrite(
                source=source_object, token=token
            )

            self.log.info('Total Bytes: %s | Bytes Written: %s',
                          total_bytes, bytes_rewritten)
        self.log.info('Object %s in bucket %s copied to object %s in bucket %s',
                      source_object.name, source_bucket.name,
                      destination_object, destination_bucket.name)


Instruction:
Explain what the following Python 3 code does
def download(self, bucket_name, object_name, filename=None):
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name)
        blob = bucket.blob(blob_name=object_name)

        if filename:
            blob.download_to_filename(filename)
            self.log.info('File downloaded to %s', filename)

        return blob.download_as_string()

Response: Download a file from Google Cloud Storage.


Instruction:
Here you have a function in Python 3, explain what it does
def upload(self, bucket_name, object_name, filename,
               mime_type='application/octet-stream', gzip=False):
        """
        Uploads a local file to Google Cloud Storage.

        :param bucket_name: The bucket to upload to.
        :type bucket_name: str
        :param object_name: The object name to set when uploading the local file.
        :type object_name: str
        :param filename: The local file path to the file to be uploaded.
        :type filename: str
        :param mime_type: The MIME type to set when uploading the file.
        :type mime_type: str
        :param gzip: Option to compress file for upload
        :type gzip: bool
        """

        if gzip:
            filename_gz = filename + '.gz'

            with open(filename, 'rb') as f_in:
                with gz.open(filename_gz, 'wb') as f_out:
                    shutil.copyfileobj(f_in, f_out)
                    filename = filename_gz

        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.blob(blob_name=object_name)
        blob.upload_from_filename(filename=filename,
                                  content_type=mime_type)

        if gzip:
            os.remove(filename)
        self.log.info('File %s uploaded to %s in %s bucket', filename, object_name, bucket_name)

Response: Uploads a local file to Google Cloud Storage.


Instruction:
Given the following Python 3 function, write the documentation
def exists(self, bucket_name, object_name):
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.blob(blob_name=object_name)
        return blob.exists()

Response: Checks for the existence of a file in Google Cloud Storage.


Instruction:
Create a Python 3 function for
checking if an object is updated after a given timestamp.

Response:
def is_updated_after(self, bucket_name, object_name, ts):
        """
        Checks if an blob_name is updated in Google Cloud Storage.

        :param bucket_name: The Google cloud storage bucket where the object is.
        :type bucket_name: str
        :param object_name: The name of the object to check in the Google cloud
            storage bucket.
        :type object_name: str
        :param ts: The timestamp to check against.
        :type ts: datetime.datetime
        """
        client = self.get_conn()
        bucket = storage.Bucket(client=client, name=bucket_name)
        blob = bucket.get_blob(blob_name=object_name)
        blob.reload()

        blob_update_time = blob.updated

        if blob_update_time is not None:
            import dateutil.tz

            if not ts.tzinfo:
                ts = ts.replace(tzinfo=dateutil.tz.tzutc())

            self.log.info("Verify object date: %s > %s", blob_update_time, ts)

            if blob_update_time > ts:
                return True

        return False


Instruction:
Write a Python 3 function for
deleting an object from the bucket.

Response:
def delete(self, bucket_name, object_name):
        """
        Deletes an object from the bucket.

        :param bucket_name: name of the bucket, where the object resides
        :type bucket_name: str
        :param object_name: name of the object to delete
        :type object_name: str
        """
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.blob(blob_name=object_name)
        blob.delete()

        self.log.info('Blob %s deleted.', object_name)


Instruction:
Implement a Python 3 function for
listing all objects in a given bucket with the given string prefix in name

Response:
def list(self, bucket_name, versions=None, max_results=None, prefix=None, delimiter=None):
        """
        List all objects from the bucket with the give string prefix in name

        :param bucket_name: bucket name
        :type bucket_name: str
        :param versions: if true, list all versions of the objects
        :type versions: bool
        :param max_results: max count of items to return in a single page of responses
        :type max_results: int
        :param prefix: prefix string which filters objects whose name begin with
            this prefix
        :type prefix: str
        :param delimiter: filters objects based on the delimiter (for e.g '.csv')
        :type delimiter: str
        :return: a stream of object names matching the filtering criteria
        """
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)

        ids = []
        pageToken = None
        while True:
            blobs = bucket.list_blobs(
                max_results=max_results,
                page_token=pageToken,
                prefix=prefix,
                delimiter=delimiter,
                versions=versions
            )

            blob_names = []
            for blob in blobs:
                blob_names.append(blob.name)

            prefixes = blobs.prefixes
            if prefixes:
                ids += list(prefixes)
            else:
                ids += blob_names

            pageToken = blobs.next_page_token
            if pageToken is None:
                # empty next page token
                break
        return ids


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_size(self, bucket_name, object_name):
        self.log.info('Checking the file size of object: %s in bucket_name: %s',
                      object_name,
                      bucket_name)
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.get_blob(blob_name=object_name)
        blob.reload()
        blob_size = blob.size
        self.log.info('The file size of %s is %s bytes.', object_name, blob_size)
        return blob_size

Response: Gets the size of a file in Google Cloud Storage.


Instruction:
Create a Python 3 function to
get the CRC32c checksum of an object in Google Cloud Storage bucket.

Response:
def get_crc32c(self, bucket_name, object_name):
        """
        Gets the CRC32c checksum of an object in Google Cloud Storage.

        :param bucket_name: The Google cloud storage bucket where the blob_name is.
        :type bucket_name: str
        :param object_name: The name of the object to check in the Google cloud
            storage bucket_name.
        :type object_name: str
        """
        self.log.info('Retrieving the crc32c checksum of '
                      'object_name: %s in bucket_name: %s', object_name, bucket_name)
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.get_blob(blob_name=object_name)
        blob.reload()
        blob_crc32c = blob.crc32c
        self.log.info('The crc32c checksum of %s is %s', object_name, blob_crc32c)
        return blob_crc32c


Instruction:
Write a Python 3 function that can
get the MD5 hash of an object in Google Cloud Storage bucket.

Response:
def get_md5hash(self, bucket_name, object_name):
        """
        Gets the MD5 hash of an object in Google Cloud Storage.

        :param bucket_name: The Google cloud storage bucket where the blob_name is.
        :type bucket_name: str
        :param object_name: The name of the object to check in the Google cloud
            storage bucket_name.
        :type object_name: str
        """
        self.log.info('Retrieving the MD5 hash of '
                      'object: %s in bucket: %s', object_name, bucket_name)
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name=bucket_name)
        blob = bucket.get_blob(blob_name=object_name)
        blob.reload()
        blob_md5hash = blob.md5_hash
        self.log.info('The md5Hash of %s is %s', object_name, blob_md5hash)
        return blob_md5hash


Instruction:
Here you have a function in Python 3, explain what it does
def create_bucket(self,
                      bucket_name,
                      resource=None,
                      storage_class='MULTI_REGIONAL',
                      location='US',
                      project_id=None,
                      labels=None
                      ):
        """
        Creates a new bucket. Google Cloud Storage uses a flat namespace, so
        you can't create a bucket with a name that is already in use.

        .. seealso::
            For more information, see Bucket Naming Guidelines:
            https://cloud.google.com/storage/docs/bucketnaming.html#requirements

        :param bucket_name: The name of the bucket.
        :type bucket_name: str
        :param resource: An optional dict with parameters for creating the bucket.
            For information on available parameters, see Cloud Storage API doc:
            https://cloud.google.com/storage/docs/json_api/v1/buckets/insert
        :type resource: dict
        :param storage_class: This defines how objects in the bucket are stored
            and determines the SLA and the cost of storage. Values include

            - ``MULTI_REGIONAL``
            - ``REGIONAL``
            - ``STANDARD``
            - ``NEARLINE``
            - ``COLDLINE``.

            If this value is not specified when the bucket is
            created, it will default to STANDARD.
        :type storage_class: str
        :param location: The location of the bucket.
            Object data for objects in the bucket resides in physical storage
            within this region. Defaults to US.

            .. seealso::
                https://developers.google.com/storage/docs/bucket-locations

        :type location: str
        :param project_id: The ID of the GCP Project.
        :type project_id: str
        :param labels: User-provided labels, in key/value pairs.
        :type labels: dict
        :return: If successful, it returns the ``id`` of the bucket.
        """

        self.log.info('Creating Bucket: %s; Location: %s; Storage Class: %s',
                      bucket_name, location, storage_class)

        client = self.get_conn()
        bucket = client.bucket(bucket_name=bucket_name)
        bucket_resource = resource or {}

        for item in bucket_resource:
            if item != "name":
                bucket._patch_property(name=item, value=resource[item])

        bucket.storage_class = storage_class
        bucket.labels = labels or {}
        bucket.create(project=project_id, location=location)
        return bucket.id

Response: Creates a new bucket in Google Cloud Storage.


Instruction:
Can you tell what is the following Python 3 function doing
def insert_bucket_acl(self, bucket_name, entity, role, user_project=None):
        self.log.info('Creating a new ACL entry in bucket: %s', bucket_name)
        client = self.get_conn()
        bucket = client.bucket(bucket_name=bucket_name)
        bucket.acl.reload()
        bucket.acl.entity_from_dict(entity_dict={"entity": entity, "role": role})
        if user_project:
            bucket.acl.user_project = user_project
        bucket.acl.save()

        self.log.info('A new ACL entry created in bucket: %s', bucket_name)

Response: Inserts a new ACL entry on the specified bucket.


Instruction:
Write a Python 3 function that can
insert an object acl entry on the specified object in the specified bucket.

Response:
def insert_object_acl(self, bucket_name, object_name, entity, role, user_project=None):
        """
        Creates a new ACL entry on the specified object.
        See: https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert

        :param bucket_name: Name of a bucket_name.
        :type bucket_name: str
        :param object_name: Name of the object. For information about how to URL encode
            object names to be path safe, see:
            https://cloud.google.com/storage/docs/json_api/#encoding
        :type object_name: str
        :param entity: The entity holding the permission, in one of the following forms:
            user-userId, user-email, group-groupId, group-email, domain-domain,
            project-team-projectId, allUsers, allAuthenticatedUsers
            See: https://cloud.google.com/storage/docs/access-control/lists#scopes
        :type entity: str
        :param role: The access permission for the entity.
            Acceptable values are: "OWNER", "READER".
        :type role: str
        :param user_project: (Optional) The project to be billed for this request.
            Required for Requester Pays buckets.
        :type user_project: str
        """
        self.log.info('Creating a new ACL entry for object: %s in bucket: %s',
                      object_name, bucket_name)
        client = self.get_conn()
        bucket = client.bucket(bucket_name=bucket_name)
        blob = bucket.blob(object_name)
        # Reload fetches the current ACL from Cloud Storage.
        blob.acl.reload()
        blob.acl.entity_from_dict(entity_dict={"entity": entity, "role": role})
        if user_project:
            blob.acl.user_project = user_project
        blob.acl.save()

        self.log.info('A new ACL entry created for object: %s in bucket: %s',
                      object_name, bucket_name)


Instruction:
Make a summary of the following Python 3 code
def compose(self, bucket_name, source_objects, destination_object):

        if not source_objects or not len(source_objects):
            raise ValueError('source_objects cannot be empty.')

        if not bucket_name or not destination_object:
            raise ValueError('bucket_name and destination_object cannot be empty.')

        self.log.info("Composing %s to %s in the bucket %s",
                      source_objects, destination_object, bucket_name)
        client = self.get_conn()
        bucket = client.get_bucket(bucket_name)
        destination_blob = bucket.blob(destination_object)
        destination_blob.compose(
            sources=[
                bucket.blob(blob_name=source_object) for source_object in source_objects
            ])

        self.log.info("Completed successfully.")

Response: This method is used to compose a list of existing object into a new object in the same storage bucket.


Instruction:
How would you explain what the following Python 3 function does
def argmin(arr, f):
    m = None
    i = None
    for idx, item in enumerate(arr):
        if item is not None:
            if m is None or f(item) < m:
                m = f(item)
                i = idx
    return i

Response: Return the index i in arr that minimizes f


Instruction:
Write a Python 3 function for
returning True if the secondary status message of a training job has changed.

Response:
def secondary_training_status_changed(current_job_description, prev_job_description):
    """
    Returns true if training job's secondary status message has changed.

    :param current_job_description: Current job description, returned from DescribeTrainingJob call.
    :type current_job_description: dict
    :param prev_job_description: Previous job description, returned from DescribeTrainingJob call.
    :type prev_job_description: dict

    :return: Whether the secondary status message of a training job changed or not.
    """
    current_secondary_status_transitions = current_job_description.get('SecondaryStatusTransitions')
    if current_secondary_status_transitions is None or len(current_secondary_status_transitions) == 0:
        return False

    prev_job_secondary_status_transitions = prev_job_description.get('SecondaryStatusTransitions') \
        if prev_job_description is not None else None

    last_message = prev_job_secondary_status_transitions[-1]['StatusMessage'] \
        if prev_job_secondary_status_transitions is not None \
        and len(prev_job_secondary_status_transitions) > 0 else ''

    message = current_job_description['SecondaryStatusTransitions'][-1]['StatusMessage']

    return message != last_message


Instruction:
Write a Python 3 function that can
return a string containing the start time and the secondary training job status message.

Response:
def secondary_training_status_message(job_description, prev_description):
    """
    Returns a string contains start time and the secondary training job status message.

    :param job_description: Returned response from DescribeTrainingJob call
    :type job_description: dict
    :param prev_description: Previous job description from DescribeTrainingJob call
    :type prev_description: dict

    :return: Job status string to be printed.
    """

    if job_description is None or job_description.get('SecondaryStatusTransitions') is None\
            or len(job_description.get('SecondaryStatusTransitions')) == 0:
        return ''

    prev_description_secondary_transitions = prev_description.get('SecondaryStatusTransitions')\
        if prev_description is not None else None
    prev_transitions_num = len(prev_description['SecondaryStatusTransitions'])\
        if prev_description_secondary_transitions is not None else 0
    current_transitions = job_description['SecondaryStatusTransitions']

    transitions_to_print = current_transitions[-1:] if len(current_transitions) == prev_transitions_num else \
        current_transitions[prev_transitions_num - len(current_transitions):]

    status_strs = []
    for transition in transitions_to_print:
        message = transition['StatusMessage']
        time_str = timezone.convert_to_utc(job_description['LastModifiedTime']).strftime('%Y-%m-%d %H:%M:%S')
        status_strs.append('{} {} - {}'.format(time_str, transition['Status'], message))

    return '\n'.join(status_strs)


Instruction:
Can you generate the documentation for the following Python 3 function
def tar_and_s3_upload(self, path, key, bucket):
        with tempfile.TemporaryFile() as temp_file:
            if os.path.isdir(path):
                files = [os.path.join(path, name) for name in os.listdir(path)]
            else:
                files = [path]
            with tarfile.open(mode='w:gz', fileobj=temp_file) as tar_file:
                for f in files:
                    tar_file.add(f, arcname=os.path.basename(f))
            temp_file.seek(0)
            self.s3_hook.load_file_obj(temp_file, key, bucket, replace=True)

Response: Tar the local file or directory and upload to s3


Instruction:
Create a Python 3 function to
configure the SageMaker operation s S3 resources.

Response:
def configure_s3_resources(self, config):
        """
        Extract the S3 operations from the configuration and execute them.

        :param config: config of SageMaker operation
        :type config: dict
        :rtype: dict
        """
        s3_operations = config.pop('S3Operations', None)

        if s3_operations is not None:
            create_bucket_ops = s3_operations.get('S3CreateBucket', [])
            upload_ops = s3_operations.get('S3Upload', [])
            for op in create_bucket_ops:
                self.s3_hook.create_bucket(bucket_name=op['Bucket'])
            for op in upload_ops:
                if op['Tar']:
                    self.tar_and_s3_upload(op['Path'], op['Key'],
                                           op['Bucket'])
                else:
                    self.s3_hook.load_file(op['Path'], op['Key'],
                                           op['Bucket'])


Instruction:
Write a Python 3 function for
checking if an S3 URL exists in the Airflow database.

Response:
def check_s3_url(self, s3url):
        """
        Check if an S3 URL exists

        :param s3url: S3 url
        :type s3url: str
        :rtype: bool
        """
        bucket, key = S3Hook.parse_s3_url(s3url)
        if not self.s3_hook.check_for_bucket(bucket_name=bucket):
            raise AirflowException(
                "The input S3 Bucket {} does not exist ".format(bucket))
        if key and not self.s3_hook.check_for_key(key=key, bucket_name=bucket)\
           and not self.s3_hook.check_for_prefix(
                prefix=key, bucket_name=bucket, delimiter='/'):
            # check if s3 key exists in the case user provides a single file
            # or if s3 prefix exists in the case user provides multiple files in
            # a prefix
            raise AirflowException("The input S3 Key "
                                   "or Prefix {} does not exist in the Bucket {}"
                                   .format(s3url, bucket))
        return True


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_log_conn(self):
        config = botocore.config.Config(retries={'max_attempts': 15})
        return self.get_client_type('logs', config=config)

Response: Establish an AWS connection to retrieve logs during training


Instruction:
Given the following Python 3 function, write the documentation
def create_training_job(self, config, wait_for_completion=True, print_log=True,
                            check_interval=30, max_ingestion_time=None):
        """
        Create a training job

        :param config: the config for training
        :type config: dict
        :param wait_for_completion: if the program should keep running until job finishes
        :type wait_for_completion: bool
        :param check_interval: the time interval in seconds which the operator
            will check the status of any SageMaker job
        :type check_interval: int
        :param max_ingestion_time: the maximum ingestion time in seconds. Any
            SageMaker jobs that run longer than this will fail. Setting this to
            None implies no timeout for any SageMaker job.
        :type max_ingestion_time: int
        :return: A response to training job creation
        """

        self.check_training_config(config)

        response = self.get_conn().create_training_job(**config)
        if print_log:
            self.check_training_status_with_log(config['TrainingJobName'],
                                                self.non_terminal_states,
                                                self.failed_states,
                                                wait_for_completion,
                                                check_interval, max_ingestion_time
                                                )
        elif wait_for_completion:
            describe_response = self.check_status(config['TrainingJobName'],
                                                  'TrainingJobStatus',
                                                  self.describe_training_job,
                                                  check_interval, max_ingestion_time
                                                  )

            billable_time = \
                (describe_response['TrainingEndTime'] - describe_response['TrainingStartTime']) * \
                describe_response['ResourceConfig']['InstanceCount']
            self.log.info('Billable seconds:{}'.format(int(billable_time.total_seconds()) + 1))

        return response

Response: Create a training job and return the response


Instruction:
Can you generate the documentation for the following Python 3 function
def create_tuning_job(self, config, wait_for_completion=True,
                          check_interval=30, max_ingestion_time=None):
        """
        Create a tuning job

        :param config: the config for tuning
        :type config: dict
        :param wait_for_completion: if the program should keep running until job finishes
        :type wait_for_completion: bool
        :param check_interval: the time interval in seconds which the operator
            will check the status of any SageMaker job
        :type check_interval: int
        :param max_ingestion_time: the maximum ingestion time in seconds. Any
            SageMaker jobs that run longer than this will fail. Setting this to
            None implies no timeout for any SageMaker job.
        :type max_ingestion_time: int
        :return: A response to tuning job creation
        """

        self.check_tuning_config(config)

        response = self.get_conn().create_hyper_parameter_tuning_job(**config)
        if wait_for_completion:
            self.check_status(config['HyperParameterTuningJobName'],
                              'HyperParameterTuningJobStatus',
                              self.describe_tuning_job,
                              check_interval, max_ingestion_time
                              )
        return response

Response: Create a tuning job in SageMaker


Instruction:
Can you generate a brief explanation for the following Python 3 code
def create_transform_job(self, config, wait_for_completion=True,
                             check_interval=30, max_ingestion_time=None):
        """
        Create a transform job

        :param config: the config for transform job
        :type config: dict
        :param wait_for_completion: if the program should keep running until job finishes
        :type wait_for_completion: bool
        :param check_interval: the time interval in seconds which the operator
            will check the status of any SageMaker job
        :type check_interval: int
        :param max_ingestion_time: the maximum ingestion time in seconds. Any
            SageMaker jobs that run longer than this will fail. Setting this to
            None implies no timeout for any SageMaker job.
        :type max_ingestion_time: int
        :return: A response to transform job creation
        """

        self.check_s3_url(config['TransformInput']['DataSource']['S3DataSource']['S3Uri'])

        response = self.get_conn().create_transform_job(**config)
        if wait_for_completion:
            self.check_status(config['TransformJobName'],
                              'TransformJobStatus',
                              self.describe_transform_job,
                              check_interval, max_ingestion_time
                              )
        return response

Response: Create a transform job


Instruction:
Write a Python 3 script to
create an endpoint in SageMaker

Response:
def create_endpoint(self, config, wait_for_completion=True,
                        check_interval=30, max_ingestion_time=None):
        """
        Create an endpoint

        :param config: the config for endpoint
        :type config: dict
        :param wait_for_completion: if the program should keep running until job finishes
        :type wait_for_completion: bool
        :param check_interval: the time interval in seconds which the operator
            will check the status of any SageMaker job
        :type check_interval: int
        :param max_ingestion_time: the maximum ingestion time in seconds. Any
            SageMaker jobs that run longer than this will fail. Setting this to
            None implies no timeout for any SageMaker job.
        :type max_ingestion_time: int
        :return: A response to endpoint creation
        """

        response = self.get_conn().create_endpoint(**config)
        if wait_for_completion:
            self.check_status(config['EndpointName'],
                              'EndpointStatus',
                              self.describe_endpoint,
                              check_interval, max_ingestion_time,
                              non_terminal_states=self.endpoint_non_terminal_states
                              )
        return response