Instruction:
Make a summary of the following Python 3 code
def run_load(self,
                 destination_project_dataset_table,
                 source_uris,
                 schema_fields=None,
                 source_format='CSV',
                 create_disposition='CREATE_IF_NEEDED',
                 skip_leading_rows=0,
                 write_disposition='WRITE_EMPTY',
                 field_delimiter=',',
                 max_bad_records=0,
                 quote_character=None,
                 ignore_unknown_values=False,
                 allow_quoted_newlines=False,
                 allow_jagged_rows=False,
                 schema_update_options=(),
                 src_fmt_configs=None,
                 time_partitioning=None,
                 cluster_fields=None,
                 autodetect=False):
        """
        Executes a BigQuery load command to load data from Google Cloud Storage
        to BigQuery. See here:

        https://cloud.google.com/bigquery/docs/reference/v2/jobs

        For more details about these parameters.

        :param destination_project_dataset_table:
            The dotted ``(<project>.|<project>:)<dataset>.<table>($<partition>)`` BigQuery
            table to load data into. If ``<project>`` is not included, project will be the
            project defined in the connection json. If a partition is specified the
            operator will automatically append the data, create a new partition or create
            a new DAY partitioned table.
        :type destination_project_dataset_table: str
        :param schema_fields: The schema field list as defined here:
            https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load
            Required if autodetect=False; optional if autodetect=True.
        :type schema_fields: list
        :param autodetect: Attempt to autodetect the schema for CSV and JSON
            source files.
        :type autodetect: bool
        :param source_uris: The source Google Cloud
            Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild
            per-object name can be used.
        :type source_uris: list
        :param source_format: File format to export.
        :type source_format: str
        :param create_disposition: The create disposition if the table doesn't exist.
        :type create_disposition: str
        :param skip_leading_rows: Number of rows to skip when loading from a CSV.
        :type skip_leading_rows: int
        :param write_disposition: The write disposition if the table already exists.
        :type write_disposition: str
        :param field_delimiter: The delimiter to use when loading from a CSV.
        :type field_delimiter: str
        :param max_bad_records: The maximum number of bad records that BigQuery can
            ignore when running the job.
        :type max_bad_records: int
        :param quote_character: The value that is used to quote data sections in a CSV
            file.
        :type quote_character: str
        :param ignore_unknown_values: [Optional] Indicates if BigQuery should allow
            extra values that are not represented in the table schema.
            If true, the extra values are ignored. If false, records with extra columns
            are treated as bad records, and if there are too many bad records, an
            invalid error is returned in the job result.
        :type ignore_unknown_values: bool
        :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not
            (false).
        :type allow_quoted_newlines: bool
        :param allow_jagged_rows: Accept rows that are missing trailing optional columns.
            The missing values are treated as nulls. If false, records with missing
            trailing columns are treated as bad records, and if there are too many bad
            records, an invalid error is returned in the job result. Only applicable when
            soure_format is CSV.
        :type allow_jagged_rows: bool
        :param schema_update_options: Allows the schema of the destination
            table to be updated as a side effect of the load job.
        :type schema_update_options: tuple
        :param src_fmt_configs: configure optional fields specific to the source format
        :type src_fmt_configs: dict
        :param time_partitioning: configure optional time partitioning fields i.e.
            partition by field, type and  expiration as per API specifications.
        :type time_partitioning: dict
        :param cluster_fields: Request that the result of this load be stored sorted
            by one or more columns. This is only available in combination with
            time_partitioning. The order of columns given determines the sort order.
        :type cluster_fields: list[str]
        """

        # bigquery only allows certain source formats
        # we check to make sure the passed source format is valid
        # if it's not, we raise a ValueError
        # Refer to this link for more details:
        #   https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat

        if schema_fields is None and not autodetect:
            raise ValueError(
                'You must either pass a schema or autodetect=True.')

        if src_fmt_configs is None:
            src_fmt_configs = {}

        source_format = source_format.upper()
        allowed_formats = [
            "CSV", "NEWLINE_DELIMITED_JSON", "AVRO", "GOOGLE_SHEETS",
            "DATASTORE_BACKUP", "PARQUET"
        ]
        if source_format not in allowed_formats:
            raise ValueError("{0} is not a valid source format. "
                             "Please use one of the following types: {1}"
                             .format(source_format, allowed_formats))

        # bigquery also allows you to define how you want a table's schema to change
        # as a side effect of a load
        # for more details:
        # https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schemaUpdateOptions
        allowed_schema_update_options = [
            'ALLOW_FIELD_ADDITION', "ALLOW_FIELD_RELAXATION"
        ]
        if not set(allowed_schema_update_options).issuperset(
                set(schema_update_options)):
            raise ValueError(
                "{0} contains invalid schema update options."
                "Please only use one or more of the following options: {1}"
                .format(schema_update_options, allowed_schema_update_options))

        destination_project, destination_dataset, destination_table = \
            _split_tablename(table_input=destination_project_dataset_table,
                             default_project_id=self.project_id,
                             var_name='destination_project_dataset_table')

        configuration = {
            'load': {
                'autodetect': autodetect,
                'createDisposition': create_disposition,
                'destinationTable': {
                    'projectId': destination_project,
                    'datasetId': destination_dataset,
                    'tableId': destination_table,
                },
                'sourceFormat': source_format,
                'sourceUris': source_uris,
                'writeDisposition': write_disposition,
                'ignoreUnknownValues': ignore_unknown_values
            }
        }

        time_partitioning = _cleanse_time_partitioning(
            destination_project_dataset_table,
            time_partitioning
        )
        if time_partitioning:
            configuration['load'].update({
                'timePartitioning': time_partitioning
            })

        if cluster_fields:
            configuration['load'].update({'clustering': {'fields': cluster_fields}})

        if schema_fields:
            configuration['load']['schema'] = {'fields': schema_fields}

        if schema_update_options:
            if write_disposition not in ["WRITE_APPEND", "WRITE_TRUNCATE"]:
                raise ValueError("schema_update_options is only "
                                 "allowed if write_disposition is "
                                 "'WRITE_APPEND' or 'WRITE_TRUNCATE'.")
            else:
                self.log.info(
                    "Adding experimental 'schemaUpdateOptions': %s",
                    schema_update_options
                )
                configuration['load'][
                    'schemaUpdateOptions'] = schema_update_options

        if max_bad_records:
            configuration['load']['maxBadRecords'] = max_bad_records

        # if following fields are not specified in src_fmt_configs,
        # honor the top-level params for backward-compatibility
        if 'skipLeadingRows' not in src_fmt_configs:
            src_fmt_configs['skipLeadingRows'] = skip_leading_rows
        if 'fieldDelimiter' not in src_fmt_configs:
            src_fmt_configs['fieldDelimiter'] = field_delimiter
        if 'ignoreUnknownValues' not in src_fmt_configs:
            src_fmt_configs['ignoreUnknownValues'] = ignore_unknown_values
        if quote_character is not None:
            src_fmt_configs['quote'] = quote_character
        if allow_quoted_newlines:
            src_fmt_configs['allowQuotedNewlines'] = allow_quoted_newlines

        src_fmt_to_configs_mapping = {
            'CSV': [
                'allowJaggedRows', 'allowQuotedNewlines', 'autodetect',
                'fieldDelimiter', 'skipLeadingRows', 'ignoreUnknownValues',
                'nullMarker', 'quote'
            ],
            'DATASTORE_BACKUP': ['projectionFields'],
            'NEWLINE_DELIMITED_JSON': ['autodetect', 'ignoreUnknownValues'],
            'PARQUET': ['autodetect', 'ignoreUnknownValues'],
            'AVRO': ['useAvroLogicalTypes'],
        }
        valid_configs = src_fmt_to_configs_mapping[source_format]
        src_fmt_configs = {
            k: v
            for k, v in src_fmt_configs.items() if k in valid_configs
        }
        configuration['load'].update(src_fmt_configs)

        if allow_jagged_rows:
            configuration['load']['allowJaggedRows'] = allow_jagged_rows

        return self.run_with_configuration(configuration)

Response: Executes a BigQuery load command to load data from a Google Cloud Storage object.


Instruction:
Can you tell what is the following Python 3 function doing
def run_with_configuration(self, configuration):
        jobs = self.service.jobs()
        job_data = {'configuration': configuration}

        # Send query and wait for reply.
        query_reply = jobs \
            .insert(projectId=self.project_id, body=job_data) \
            .execute(num_retries=self.num_retries)
        self.running_job_id = query_reply['jobReference']['jobId']
        if 'location' in query_reply['jobReference']:
            location = query_reply['jobReference']['location']
        else:
            location = self.location

        # Wait for query to finish.
        keep_polling_job = True
        while keep_polling_job:
            try:
                if location:
                    job = jobs.get(
                        projectId=self.project_id,
                        jobId=self.running_job_id,
                        location=location).execute(num_retries=self.num_retries)
                else:
                    job = jobs.get(
                        projectId=self.project_id,
                        jobId=self.running_job_id).execute(num_retries=self.num_retries)
                if job['status']['state'] == 'DONE':
                    keep_polling_job = False
                    # Check if job had errors.
                    if 'errorResult' in job['status']:
                        raise Exception(
                            'BigQuery job failed. Final error was: {}. The job was: {}'.
                            format(job['status']['errorResult'], job))
                else:
                    self.log.info('Waiting for job to complete : %s, %s',
                                  self.project_id, self.running_job_id)
                    time.sleep(5)

            except HttpError as err:
                if err.resp.status in [500, 503]:
                    self.log.info(
                        '%s: Retryable error, waiting for job to complete: %s',
                        err.resp.status, self.running_job_id)
                    time.sleep(5)
                else:
                    raise Exception(
                        'BigQuery job status check failed. Final error was: {}'.
                        format(err.resp.status))

        return self.running_job_id

Response: Executes a BigQuery SQL query and waits for the job to complete.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def cancel_query(self):
        jobs = self.service.jobs()
        if (self.running_job_id and
                not self.poll_job_complete(self.running_job_id)):
            self.log.info('Attempting to cancel job : %s, %s', self.project_id,
                          self.running_job_id)
            if self.location:
                jobs.cancel(
                    projectId=self.project_id,
                    jobId=self.running_job_id,
                    location=self.location).execute(num_retries=self.num_retries)
            else:
                jobs.cancel(
                    projectId=self.project_id,
                    jobId=self.running_job_id).execute(num_retries=self.num_retries)
        else:
            self.log.info('No running BigQuery jobs to cancel.')
            return

        # Wait for all the calls to cancel to finish
        max_polling_attempts = 12
        polling_attempts = 0

        job_complete = False
        while polling_attempts < max_polling_attempts and not job_complete:
            polling_attempts = polling_attempts + 1
            job_complete = self.poll_job_complete(self.running_job_id)
            if job_complete:
                self.log.info('Job successfully canceled: %s, %s',
                              self.project_id, self.running_job_id)
            elif polling_attempts == max_polling_attempts:
                self.log.info(
                    "Stopping polling due to timeout. Job with id %s "
                    "has not completed cancel and may or may not finish.",
                    self.running_job_id)
            else:
                self.log.info('Waiting for canceled job with id %s to finish.',
                              self.running_job_id)
                time.sleep(5)

Response: Cancel all started queries that have not yet completed.


Instruction:
How would you code a function in Python 3 to
get the schema for a given dataset. table.

Response:
def get_schema(self, dataset_id, table_id):
        """
        Get the schema for a given datset.table.
        see https://cloud.google.com/bigquery/docs/reference/v2/tables#resource

        :param dataset_id: the dataset ID of the requested table
        :param table_id: the table ID of the requested table
        :return: a table schema
        """
        tables_resource = self.service.tables() \
            .get(projectId=self.project_id, datasetId=dataset_id, tableId=table_id) \
            .execute(num_retries=self.num_retries)
        return tables_resource['schema']


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_tabledata(self, dataset_id, table_id,
                      max_results=None, selected_fields=None, page_token=None,
                      start_index=None):
        """
        Get the data of a given dataset.table and optionally with selected columns.
        see https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list

        :param dataset_id: the dataset ID of the requested table.
        :param table_id: the table ID of the requested table.
        :param max_results: the maximum results to return.
        :param selected_fields: List of fields to return (comma-separated). If
            unspecified, all fields are returned.
        :param page_token: page token, returned from a previous call,
            identifying the result set.
        :param start_index: zero based index of the starting row to read.
        :return: map containing the requested rows.
        """
        optional_params = {}
        if max_results:
            optional_params['maxResults'] = max_results
        if selected_fields:
            optional_params['selectedFields'] = selected_fields
        if page_token:
            optional_params['pageToken'] = page_token
        if start_index:
            optional_params['startIndex'] = start_index
        return (self.service.tabledata().list(
            projectId=self.project_id,
            datasetId=dataset_id,
            tableId=table_id,
            **optional_params).execute(num_retries=self.num_retries))

Response: Get the data of a given dataset. table and optionally with selected columns.


Instruction:
Explain what the following Python 3 code does
def run_table_delete(self, deletion_dataset_table,
                         ignore_if_missing=False):
        """
        Delete an existing table from the dataset;
        If the table does not exist, return an error unless ignore_if_missing
        is set to True.

        :param deletion_dataset_table: A dotted
            ``(<project>.|<project>:)<dataset>.<table>`` that indicates which table
            will be deleted.
        :type deletion_dataset_table: str
        :param ignore_if_missing: if True, then return success even if the
            requested table does not exist.
        :type ignore_if_missing: bool
        :return:
        """
        deletion_project, deletion_dataset, deletion_table = \
            _split_tablename(table_input=deletion_dataset_table,
                             default_project_id=self.project_id)

        try:
            self.service.tables() \
                .delete(projectId=deletion_project,
                        datasetId=deletion_dataset,
                        tableId=deletion_table) \
                .execute(num_retries=self.num_retries)
            self.log.info('Deleted table %s:%s.%s.', deletion_project,
                          deletion_dataset, deletion_table)
        except HttpError:
            if not ignore_if_missing:
                raise Exception('Table deletion failed. Table does not exist.')
            else:
                self.log.info('Table does not exist. Skipping.')

Response: Delete an existing table from the dataset.


Instruction:
Create a Python 3 function to
run an upsert operation on the table resource.

Response:
def run_table_upsert(self, dataset_id, table_resource, project_id=None):
        """
        creates a new, empty table in the dataset;
        If the table already exists, update the existing table.
        Since BigQuery does not natively allow table upserts, this is not an
        atomic operation.

        :param dataset_id: the dataset to upsert the table into.
        :type dataset_id: str
        :param table_resource: a table resource. see
            https://cloud.google.com/bigquery/docs/reference/v2/tables#resource
        :type table_resource: dict
        :param project_id: the project to upsert the table into.  If None,
            project will be self.project_id.
        :return:
        """
        # check to see if the table exists
        table_id = table_resource['tableReference']['tableId']
        project_id = project_id if project_id is not None else self.project_id
        tables_list_resp = self.service.tables().list(
            projectId=project_id, datasetId=dataset_id).execute(num_retries=self.num_retries)
        while True:
            for table in tables_list_resp.get('tables', []):
                if table['tableReference']['tableId'] == table_id:
                    # found the table, do update
                    self.log.info('Table %s:%s.%s exists, updating.',
                                  project_id, dataset_id, table_id)
                    return self.service.tables().update(
                        projectId=project_id,
                        datasetId=dataset_id,
                        tableId=table_id,
                        body=table_resource).execute(num_retries=self.num_retries)
            # If there is a next page, we need to check the next page.
            if 'nextPageToken' in tables_list_resp:
                tables_list_resp = self.service.tables()\
                    .list(projectId=project_id,
                          datasetId=dataset_id,
                          pageToken=tables_list_resp['nextPageToken'])\
                    .execute(num_retries=self.num_retries)
            # If there is no next page, then the table doesn't exist.
            else:
                # do insert
                self.log.info('Table %s:%s.%s does not exist. creating.',
                              project_id, dataset_id, table_id)
                return self.service.tables().insert(
                    projectId=project_id,
                    datasetId=dataset_id,
                    body=table_resource).execute(num_retries=self.num_retries)


Instruction:
Can you generate the documentation for the following Python 3 function
def run_grant_dataset_view_access(self,
                                      source_dataset,
                                      view_dataset,
                                      view_table,
                                      source_project=None,
                                      view_project=None):
        """
        Grant authorized view access of a dataset to a view table.
        If this view has already been granted access to the dataset, do nothing.
        This method is not atomic.  Running it may clobber a simultaneous update.

        :param source_dataset: the source dataset
        :type source_dataset: str
        :param view_dataset: the dataset that the view is in
        :type view_dataset: str
        :param view_table: the table of the view
        :type view_table: str
        :param source_project: the project of the source dataset. If None,
            self.project_id will be used.
        :type source_project: str
        :param view_project: the project that the view is in. If None,
            self.project_id will be used.
        :type view_project: str
        :return: the datasets resource of the source dataset.
        """

        # Apply default values to projects
        source_project = source_project if source_project else self.project_id
        view_project = view_project if view_project else self.project_id

        # we don't want to clobber any existing accesses, so we have to get
        # info on the dataset before we can add view access
        source_dataset_resource = self.service.datasets().get(
            projectId=source_project, datasetId=source_dataset).execute(num_retries=self.num_retries)
        access = source_dataset_resource[
            'access'] if 'access' in source_dataset_resource else []
        view_access = {
            'view': {
                'projectId': view_project,
                'datasetId': view_dataset,
                'tableId': view_table
            }
        }
        # check to see if the view we want to add already exists.
        if view_access not in access:
            self.log.info(
                'Granting table %s:%s.%s authorized view access to %s:%s dataset.',
                view_project, view_dataset, view_table, source_project,
                source_dataset)
            access.append(view_access)
            return self.service.datasets().patch(
                projectId=source_project,
                datasetId=source_dataset,
                body={
                    'access': access
                }).execute(num_retries=self.num_retries)
        else:
            # if view is already in access, do nothing.
            self.log.info(
                'Table %s:%s.%s already has authorized view access to %s:%s dataset.',
                view_project, view_dataset, view_table, source_project, source_dataset)
            return source_dataset_resource

Response: This method grants authorized view access of a dataset to a view table.


Instruction:
How would you code a function in Python 3 to
create an empty dataset.

Response:
def create_empty_dataset(self, dataset_id="", project_id="",
                             dataset_reference=None):
        """
        Create a new empty dataset:
        https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert

        :param project_id: The name of the project where we want to create
            an empty a dataset. Don't need to provide, if projectId in dataset_reference.
        :type project_id: str
        :param dataset_id: The id of dataset. Don't need to provide,
            if datasetId in dataset_reference.
        :type dataset_id: str
        :param dataset_reference: Dataset reference that could be provided
            with request body. More info:
            https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
        :type dataset_reference: dict
        """

        if dataset_reference:
            _validate_value('dataset_reference', dataset_reference, dict)
        else:
            dataset_reference = {}

        if "datasetReference" not in dataset_reference:
            dataset_reference["datasetReference"] = {}

        if not dataset_reference["datasetReference"].get("datasetId") and not dataset_id:
            raise ValueError(
                "{} not provided datasetId. Impossible to create dataset")

        dataset_required_params = [(dataset_id, "datasetId", ""),
                                   (project_id, "projectId", self.project_id)]
        for param_tuple in dataset_required_params:
            param, param_name, param_default = param_tuple
            if param_name not in dataset_reference['datasetReference']:
                if param_default and not param:
                    self.log.info(
                        "%s was not specified. Will be used default value %s.",
                        param_name, param_default
                    )
                    param = param_default
                dataset_reference['datasetReference'].update(
                    {param_name: param})
            elif param:
                _api_resource_configs_duplication_check(
                    param_name, param,
                    dataset_reference['datasetReference'], 'dataset_reference')

        dataset_id = dataset_reference.get("datasetReference").get("datasetId")
        dataset_project_id = dataset_reference.get("datasetReference").get(
            "projectId")

        self.log.info('Creating Dataset: %s in project: %s ', dataset_id,
                      dataset_project_id)

        try:
            self.service.datasets().insert(
                projectId=dataset_project_id,
                body=dataset_reference).execute(num_retries=self.num_retries)
            self.log.info('Dataset created successfully: In project %s '
                          'Dataset %s', dataset_project_id, dataset_id)

        except HttpError as err:
            raise AirflowException(
                'BigQuery job failed. Error was: {}'.format(err.content)
            )


Instruction:
Write a Python 3 function for
deleting a dataset of Big query in your project.

Response:
def delete_dataset(self, project_id, dataset_id):
        """
        Delete a dataset of Big query in your project.
        :param project_id: The name of the project where we have the dataset .
        :type project_id: str
        :param dataset_id: The dataset to be delete.
        :type dataset_id: str
        :return:
        """
        project_id = project_id if project_id is not None else self.project_id
        self.log.info('Deleting from project: %s  Dataset:%s',
                      project_id, dataset_id)

        try:
            self.service.datasets().delete(
                projectId=project_id,
                datasetId=dataset_id).execute(num_retries=self.num_retries)
            self.log.info('Dataset deleted successfully: In project %s '
                          'Dataset %s', project_id, dataset_id)

        except HttpError as err:
            raise AirflowException(
                'BigQuery job failed. Error was: {}'.format(err.content)
            )


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_dataset(self, dataset_id, project_id=None):

        if not dataset_id or not isinstance(dataset_id, str):
            raise ValueError("dataset_id argument must be provided and has "
                             "a type 'str'. You provided: {}".format(dataset_id))

        dataset_project_id = project_id if project_id else self.project_id

        try:
            dataset_resource = self.service.datasets().get(
                datasetId=dataset_id, projectId=dataset_project_id).execute(num_retries=self.num_retries)
            self.log.info("Dataset Resource: %s", dataset_resource)
        except HttpError as err:
            raise AirflowException(
                'BigQuery job failed. Error was: {}'.format(err.content))

        return dataset_resource

Response: Method returns dataset_resource if dataset exists and raised 404 error if dataset does not exist


Instruction:
Write a Python 3 script for
executing a BigQuery query and returns the job ID.

Response:
def execute(self, operation, parameters=None):
        """
        Executes a BigQuery query, and returns the job ID.

        :param operation: The query to execute.
        :type operation: str
        :param parameters: Parameters to substitute into the query.
        :type parameters: dict
        """
        sql = _bind_parameters(operation,
                               parameters) if parameters else operation
        self.job_id = self.run_query(sql)


Instruction:
Can you generate the documentation for the following Python 3 function
def executemany(self, operation, seq_of_parameters):
        for parameters in seq_of_parameters:
            self.execute(operation, parameters)

Response: Execute a BigQuery query multiple times with different parameters.


Instruction:
How would you explain what the following Python 3 function does
def next(self):
        if not self.job_id:
            return None

        if len(self.buffer) == 0:
            if self.all_pages_loaded:
                return None

            query_results = (self.service.jobs().getQueryResults(
                projectId=self.project_id,
                jobId=self.job_id,
                pageToken=self.page_token).execute(num_retries=self.num_retries))

            if 'rows' in query_results and query_results['rows']:
                self.page_token = query_results.get('pageToken')
                fields = query_results['schema']['fields']
                col_types = [field['type'] for field in fields]
                rows = query_results['rows']

                for dict_row in rows:
                    typed_row = ([
                        _bq_cast(vs['v'], col_types[idx])
                        for idx, vs in enumerate(dict_row['f'])
                    ])
                    self.buffer.append(typed_row)

                if not self.page_token:
                    self.all_pages_loaded = True

            else:
                # Reset all state since we've exhausted the results.
                self.page_token = None
                self.job_id = None
                self.page_token = None
                return None

        return self.buffer.pop(0)

Response: Returns the next entry in the buffer.


Instruction:
Can you write a function in Python 3 where it
fetches the next set of rows of a query result returning a list of tuples.

Response:
def fetchmany(self, size=None):
        """
        Fetch the next set of rows of a query result, returning a sequence of sequences
        (e.g. a list of tuples). An empty sequence is returned when no more rows are
        available. The number of rows to fetch per call is specified by the parameter.
        If it is not given, the cursor's arraysize determines the number of rows to be
        fetched. The method should try to fetch as many rows as indicated by the size
        parameter. If this is not possible due to the specified number of rows not being
        available, fewer rows may be returned. An :py:class:`~pyhive.exc.Error`
        (or subclass) exception is raised if the previous call to
        :py:meth:`execute` did not produce any result set or no call was issued yet.
        """
        if size is None:
            size = self.arraysize
        result = []
        for _ in range(size):
            one = self.fetchone()
            if one is None:
                break
            else:
                result.append(one)
        return result


Instruction:
How would you explain what the following Python 3 function does
def fetchall(self):
        result = []
        while True:
            one = self.fetchone()
            if one is None:
                break
            else:
                result.append(one)
        return result

Response: Fetch all rows of a query result returning them as a list of tuples.


Instruction:
Implement a function in Python 3 to
load the manifest file and registers the url_for_asset_ template tag.

Response:
def configure_manifest_files(app):
    """
    Loads the manifest file and register the `url_for_asset_` template tag.
    :param app:
    :return:
    """

    def parse_manifest_json():
        # noinspection PyBroadException
        try:
            global manifest
            manifest_file = os.path.join(os.path.dirname(__file__),
                                         'static/dist/manifest.json')
            with open(manifest_file, 'r') as f:
                manifest.update(json.load(f))

                for k in manifest.keys():
                    manifest[k] = os.path.join("dist", manifest[k])
        except Exception:
            print("Please make sure to build the frontend in "
                  "static/ directory and restart the server")
            pass

    def get_asset_url(filename):
        if app.debug:
            parse_manifest_json()
        return url_for('static', filename=manifest.get(filename, ''))

    parse_manifest_json()

    @app.context_processor
    def get_url_for_asset():
        """
        Template tag to return the asset URL.
        WebPack renders the assets after minification and modification
        under the static/dist folder.
        This template tag reads the asset name in manifest.json and returns
        the appropriate file.
        """
        return dict(url_for_asset=get_asset_url)


Instruction:
Implement a function in Python 3 to
query Postgres and returns a cursor to the results.

Response:
def _query_postgres(self):
        """
        Queries Postgres and returns a cursor to the results.
        """
        postgres = PostgresHook(postgres_conn_id=self.postgres_conn_id)
        conn = postgres.get_conn()
        cursor = conn.cursor()
        cursor.execute(self.sql, self.parameters)
        return cursor


Instruction:
How would you code a function in Python 3 to
take a cursor and writes results to a local file.

Response:
def _write_local_data_files(self, cursor):
        """
        Takes a cursor, and writes results to a local file.

        :return: A dictionary where keys are filenames to be used as object
            names in GCS, and values are file handles to local files that
            contain the data for the GCS objects.
        """
        schema = list(map(lambda schema_tuple: schema_tuple[0], cursor.description))
        tmp_file_handles = {}
        row_no = 0

        def _create_new_file():
            handle = NamedTemporaryFile(delete=True)
            filename = self.filename.format(len(tmp_file_handles))
            tmp_file_handles[filename] = handle
            return handle

        # Don't create a file if there is nothing to write
        if cursor.rowcount > 0:
            tmp_file_handle = _create_new_file()

            for row in cursor:
                # Convert datetime objects to utc seconds, and decimals to floats
                row = map(self.convert_types, row)
                row_dict = dict(zip(schema, row))

                s = json.dumps(row_dict, sort_keys=True).encode('utf-8')
                tmp_file_handle.write(s)

                # Append newline to make dumps BigQuery compatible.
                tmp_file_handle.write(b'\n')

                # Stop if the file exceeds the file size limit.
                if tmp_file_handle.tell() >= self.approx_max_file_size_bytes:
                    tmp_file_handle = _create_new_file()
                row_no += 1

        self.log.info('Received %s rows over %s files', row_no, len(tmp_file_handles))

        return tmp_file_handles


Instruction:
How would you code a function in Python 3 to
write the BigQuery schema for the results to the local file system.

Response:
def _write_local_schema_file(self, cursor):
        """
        Takes a cursor, and writes the BigQuery schema for the results to a
        local file system.

        :return: A dictionary where key is a filename to be used as an object
            name in GCS, and values are file handles to local files that
            contains the BigQuery schema fields in .json format.
        """
        schema = []
        for field in cursor.description:
            # See PEP 249 for details about the description tuple.
            field_name = field[0]
            field_type = self.type_map(field[1])
            field_mode = 'REPEATED' if field[1] in (1009, 1005, 1007,
                                                    1016) else 'NULLABLE'
            schema.append({
                'name': field_name,
                'type': field_type,
                'mode': field_mode,
            })

        self.log.info('Using schema for %s: %s', self.schema_filename, schema)
        tmp_schema_file_handle = NamedTemporaryFile(delete=True)
        s = json.dumps(schema, sort_keys=True).encode('utf-8')
        tmp_schema_file_handle.write(s)
        return {self.schema_filename: tmp_schema_file_handle}


Instruction:
Can you generate a brief explanation for the following Python 3 code
def convert_types(cls, value):
        if type(value) in (datetime.datetime, datetime.date):
            return time.mktime(value.timetuple())
        elif type(value) == datetime.time:
            formated_time = time.strptime(str(value), "%H:%M:%S")
            return datetime.timedelta(
                hours=formated_time.tm_hour,
                minutes=formated_time.tm_min,
                seconds=formated_time.tm_sec).seconds
        elif isinstance(value, Decimal):
            return float(value)
        else:
            return value

Response:
Takes a value from Postgres and converts it to a value that s safe for
        JSON or Google Cloud Storage / BigQuery.


Instruction:
Can you create a Python 3 function that
creates all the intermediate directories in a remote host.

Response:
def _make_intermediate_dirs(sftp_client, remote_directory):
    """
    Create all the intermediate directories in a remote host

    :param sftp_client: A Paramiko SFTP client.
    :param remote_directory: Absolute Path of the directory containing the file
    :return:
    """
    if remote_directory == '/':
        sftp_client.chdir('/')
        return
    if remote_directory == '':
        return
    try:
        sftp_client.chdir(remote_directory)
    except IOError:
        dirname, basename = os.path.split(remote_directory.rstrip('/'))
        _make_intermediate_dirs(sftp_client, dirname)
        sftp_client.mkdir(basename)
        sftp_client.chdir(basename)
        return


Instruction:
Given the following Python 3 function, write the documentation
def create_queue(self, queue_name, attributes=None):
        return self.get_conn().create_queue(QueueName=queue_name, Attributes=attributes or {})

Response: Create a queue using the connection object


Instruction:
Write a Python 3 function for
sending a message to the queue with optional delay

Response:
def send_message(self, queue_url, message_body, delay_seconds=0, message_attributes=None):
        """
        Send message to the queue

        :param queue_url: queue url
        :type queue_url: str
        :param message_body: the contents of the message
        :type message_body: str
        :param delay_seconds: seconds to delay the message
        :type delay_seconds: int
        :param message_attributes: additional attributes for the message (default: None)
            For details of the attributes parameter see :py:meth:`botocore.client.SQS.send_message`
        :type message_attributes: dict

        :return: dict with the information about the message sent
            For details of the returned value see :py:meth:`botocore.client.SQS.send_message`
        :rtype: dict
        """
        return self.get_conn().send_message(QueueUrl=queue_url,
                                            MessageBody=message_body,
                                            DelaySeconds=delay_seconds,
                                            MessageAttributes=message_attributes or {})


Instruction:
Here you have a function in Python 3, explain what it does
def _integrate_plugins():
    from airflow.plugins_manager import hooks_modules
    for hooks_module in hooks_modules:
        sys.modules[hooks_module.__name__] = hooks_module
        globals()[hooks_module._name] = hooks_module

Response: Integrate plugins to the context


Instruction:
Can you generate the documentation for the following Python 3 function
def run_command(self, run_with=None, join_args=False):
        run_with = run_with or []
        cmd = [" ".join(self._command)] if join_args else self._command
        full_cmd = run_with + cmd

        self.log.info('Running: %s', full_cmd)
        proc = subprocess.Popen(
            full_cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            universal_newlines=True,
            close_fds=True,
            env=os.environ.copy(),
            preexec_fn=os.setsid
        )

        # Start daemon thread to read subprocess logging output
        log_reader = threading.Thread(
            target=self._read_task_logs,
            args=(proc.stdout,),
        )
        log_reader.daemon = True
        log_reader.start()
        return proc

Response: Runs the task command.


Instruction:
Make a summary of the following Python 3 code
def on_finish(self):
        if self._cfg_path and os.path.isfile(self._cfg_path):
            if self.run_as_user:
                subprocess.call(['sudo', 'rm', self._cfg_path], close_fds=True)
            else:
                os.remove(self._cfg_path)

Response: A callback that should be called when this is done running.


Instruction:
How would you explain what the following Python 3 function does
def _main():
    # Parse arguments
    usage = "usage: nvd3.py [options]"
    parser = OptionParser(usage=usage,
                          version=("python-nvd3 - Charts generator with "
                                   "nvd3.js and d3.js"))
    parser.add_option("-q", "--quiet",
                      action="store_false", dest="verbose", default=True,
                      help="don't print messages to stdout")

    (options, args) = parser.parse_args()

Response: Main function for the nvd3 - charts generator.


Instruction:
Can you tell what is the following Python 3 function doing
def add_serie(self, y, x, name=None, extra=None, **kwargs):
        if not name:
            name = "Serie %d" % (self.serie_no)

        # For scatterChart shape & size fields are added in serie
        if 'shape' in kwargs or 'size' in kwargs:
            csize = kwargs.get('size', 1)
            cshape = kwargs.get('shape', 'circle')

            serie = [{
                'x': x[i],
                'y': j,
                'shape': cshape,
                'size': csize[i] if isinstance(csize, list) else csize
            } for i, j in enumerate(y)]
        else:
            if self.model == 'pieChart':
                serie = [{'label': x[i], 'value': y} for i, y in enumerate(y)]
            else:
                serie = [{'x': x[i], 'y': y} for i, y in enumerate(y)]

        data_keyvalue = {'values': serie, 'key': name}

        # multiChart
        # Histogram type='bar' for the series
        if 'type' in kwargs and kwargs['type']:
            data_keyvalue['type'] = kwargs['type']

        # Define on which Y axis the serie is related
        # a chart can have 2 Y axis, left and right, by default only one Y Axis is used
        if 'yaxis' in kwargs and kwargs['yaxis']:
            data_keyvalue['yAxis'] = kwargs['yaxis']
        else:
            if self.model != 'pieChart':
                data_keyvalue['yAxis'] = '1'

        if 'bar' in kwargs and kwargs['bar']:
            data_keyvalue['bar'] = 'true'

        if 'disabled' in kwargs and kwargs['disabled']:
            data_keyvalue['disabled'] = 'true'

        if 'color' in kwargs and kwargs['color']:
            data_keyvalue['color'] = kwargs['color']

        if extra:
            if self.model == 'pieChart':
                if 'color_list' in extra and extra['color_list']:
                    self.color_list = extra['color_list']

            if extra.get('date_format'):
                self.charttooltip_dateformat = extra['date_format']

            if extra.get('tooltip'):
                self.custom_tooltip_flag = True

                if self.model != 'pieChart':
                    _start = extra['tooltip']['y_start']
                    _end = extra['tooltip']['y_end']
                    _start = ("'" + str(_start) + "' + ") if _start else ''
                    _end = (" + '" + str(_end) + "'") if _end else ''

                    if self.model == 'linePlusBarChart':
                        if self.tooltip_condition_string:
                            self.tooltip_condition_string += stab(5)
                        self.tooltip_condition_string += stab(0) + "if(key.indexOf('" + name + "') > -1 ){\n" +\
                            stab(6) + "var y = " + _start + " String(graph.point.y) " + _end + ";\n" +\
                            stab(5) + "}\n"
                    elif self.model == 'cumulativeLineChart':
                        self.tooltip_condition_string += stab(0) + "if(key == '" + name + "'){\n" +\
                            stab(6) + "var y = " + _start + " String(e) " + _end + ";\n" +\
                            stab(5) + "}\n"
                    else:
                        self.tooltip_condition_string += stab(5) + "if(key == '" + name + "'){\n" +\
                            stab(6) + "var y = " + _start + " String(graph.point.y) " + _end + ";\n" +\
                            stab(5) + "}\n"

                if self.model == 'pieChart':
                    _start = extra['tooltip']['y_start']
                    _end = extra['tooltip']['y_end']
                    _start = ("'" + str(_start) + "' + ") if _start else ''
                    _end = (" + '" + str(_end) + "'") if _end else ''
                    self.tooltip_condition_string += "var y = " + _start + " String(y) " + _end + ";\n"

        # Increment series counter & append
        self.serie_no += 1
        self.series.append(data_keyvalue)

Response: add serie to the log file


Instruction:
Implement a function in Python 3 to
build HTML content only with no header or body tags.

Response:
def buildcontent(self):
        """Build HTML content only, no header or body tags. To be useful this
        will usually require the attribute `juqery_on_ready` to be set which
        will wrap the js in $(function(){<regular_js>};)
        """
        self.buildcontainer()
        # if the subclass has a method buildjs this method will be
        # called instead of the method defined here
        # when this subclass method is entered it does call
        # the method buildjschart defined here
        self.buildjschart()
        self.htmlcontent = self.template_content_nvd3.render(chart=self)


Instruction:
Given the following Python 3 function, write the documentation
def buildhtml(self):
        self.buildcontent()
        self.content = self.htmlcontent
        self.htmlcontent = self.template_page_nvd3.render(chart=self)

Response: Build the HTML page containing the HTML content


Instruction:
Can you write a function in Python 3 where it
generates HTML header content

Response:
def buildhtmlheader(self):
        """generate HTML header content"""
        self.htmlheader = ''
        # If the JavaScript assets have already been injected, don't bother re-sourcing them.
        global _js_initialized
        if '_js_initialized' not in globals() or not _js_initialized:
            for css in self.header_css:
                self.htmlheader += css
            for js in self.header_js:
                self.htmlheader += js


Instruction:
Can you create a Python 3 function that
generates HTML div with style

Response:
def buildcontainer(self):
        """generate HTML div"""
        if self.container:
            return

        # Create SVG div with style
        if self.width:
            if self.width[-1] != '%':
                self.style += 'width:%spx;' % self.width
            else:
                self.style += 'width:%s;' % self.width
        if self.height:
            if self.height[-1] != '%':
                self.style += 'height:%spx;' % self.height
            else:
                self.style += 'height:%s;' % self.height
        if self.style:
            self.style = 'style="%s"' % self.style

        self.container = self.containerheader + \
            '<div id="%s"><svg %s></svg></div>\n' % (self.name, self.style)


Instruction:
Given the following Python 3 function, write the documentation
def buildjschart(self):
        self.jschart = ''

        # add custom tooltip string in jschart
        # default condition (if build_custom_tooltip is not called explicitly with date_flag=True)
        if self.tooltip_condition_string == '':
            self.tooltip_condition_string = 'var y = String(graph.point.y);\n'

        # Include data
        self.series_js = json.dumps(self.series)

Response: generate javascript code for the chart


Instruction:
Can you tell what is the following Python 3 function doing
def create_x_axis(self, name, label=None, format=None, date=False, custom_format=False):
        axis = {}
        if custom_format and format:
            axis['tickFormat'] = format
        elif format:
            if format == 'AM_PM':
                axis['tickFormat'] = "function(d) { return get_am_pm(parseInt(d)); }"
            else:
                axis['tickFormat'] = "d3.format(',%s')" % format

        if label:
            axis['axisLabel'] = "'" + label + "'"

        # date format : see https://github.com/mbostock/d3/wiki/Time-Formatting
        if date:
            self.dateformat = format
            axis['tickFormat'] = ("function(d) { return d3.time.format('%s')"
                                  "(new Date(parseInt(d))) }\n"
                                  "" % self.dateformat)
            # flag is the x Axis is a date
            if name[0] == 'x':
                self.x_axis_date = True

        # Add new axis to list of axis
        self.axislist[name] = axis

        # Create x2Axis if focus_enable
        if name == "xAxis" and self.focus_enable:
            self.axislist['x2Axis'] = axis

Response: Create X - axis.


Instruction:
Can you generate the documentation for the following Python 3 function
def create_y_axis(self, name, label=None, format=None, custom_format=False):
        axis = {}

        if custom_format and format:
            axis['tickFormat'] = format
        elif format:
            axis['tickFormat'] = "d3.format(',%s')" % format

        if label:
            axis['axisLabel'] = "'" + label + "'"

        # Add new axis to list of axis
        self.axislist[name] = axis

Response: Create Y - axis for the current key.


Instruction:
Write a Python 3 script for
building HTML content only with no header or body tags.

Response:
def buildcontent(self):
        """Build HTML content only, no header or body tags. To be useful this
        will usually require the attribute `juqery_on_ready` to be set which
        will wrap the js in $(function(){<regular_js>};)
        """
        self.buildcontainer()
        # if the subclass has a method buildjs this method will be
        # called instead of the method defined here
        # when this subclass method is entered it does call
        # the method buildjschart defined here
        self.buildjschart()
        self.htmlcontent = self.template_chart_nvd3.render(chart=self)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_conn(self):
        conn = self.get_connection(self.sqlite_conn_id)
        conn = sqlite3.connect(conn.host)
        return conn

Response: Returns a sqlite connection object


Instruction:
How would you explain what the following Python 3 function does
def action_logging(f):
    @functools.wraps(f)
    def wrapper(*args, **kwargs):

        with create_session() as session:
            if g.user.is_anonymous:
                user = 'anonymous'
            else:
                user = g.user.username

            log = Log(
                event=f.__name__,
                task_instance=None,
                owner=user,
                extra=str(list(request.args.items())),
                task_id=request.args.get('task_id'),
                dag_id=request.args.get('dag_id'))

            if 'execution_date' in request.args:
                log.execution_date = pendulum.parse(
                    request.args.get('execution_date'))

            session.add(log)

        return f(*args, **kwargs)

    return wrapper

Response: Decorator to log user actions


Instruction:
Can you generate the documentation for the following Python 3 function
def has_dag_access(**dag_kwargs):
    def decorator(f):
        @functools.wraps(f)
        def wrapper(self, *args, **kwargs):
            has_access = self.appbuilder.sm.has_access
            dag_id = request.args.get('dag_id')
            # if it is false, we need to check whether user has write access on the dag
            can_dag_edit = dag_kwargs.get('can_dag_edit', False)

            # 1. check whether the user has can_dag_edit permissions on all_dags
            # 2. if 1 false, check whether the user
            #    has can_dag_edit permissions on the dag
            # 3. if 2 false, check whether it is can_dag_read view,
            #    and whether user has the permissions
            if (
                has_access('can_dag_edit', 'all_dags') or
                has_access('can_dag_edit', dag_id) or (not can_dag_edit and
                                                       (has_access('can_dag_read',
                                                                   'all_dags') or
                                                        has_access('can_dag_read',
                                                                   dag_id)))):
                return f(self, *args, **kwargs)
            else:
                flash("Access is Denied", "danger")
                return redirect(url_for(self.appbuilder.sm.auth_view.
                                        __class__.__name__ + ".login"))
        return wrapper
    return decorator

Response: Decorator to check whether the user has read or write permission on the dag.


Instruction:
How would you implement a function in Python 3 that
returns the last dag run for a dag.

Response:
def get_last_dagrun(dag_id, session, include_externally_triggered=False):
    """
    Returns the last dag run for a dag, None if there was none.
    Last dag run can be any type of run eg. scheduled or backfilled.
    Overridden DagRuns are ignored.
    """
    DR = DagRun
    query = session.query(DR).filter(DR.dag_id == dag_id)
    if not include_externally_triggered:
        query = query.filter(DR.external_trigger == False)  # noqa
    query = query.order_by(DR.execution_date.desc())
    return query.first()


Instruction:
How would you explain what the following Python 3 function does
def create_dagrun(self,
                      run_id,
                      state,
                      execution_date,
                      start_date=None,
                      external_trigger=False,
                      conf=None,
                      session=None):
        """
        Creates a dag run from this dag including the tasks associated with this dag.
        Returns the dag run.

        :param run_id: defines the the run id for this dag run
        :type run_id: str
        :param execution_date: the execution date of this dag run
        :type execution_date: datetime.datetime
        :param state: the state of the dag run
        :type state: airflow.utils.state.State
        :param start_date: the date this dag run should be evaluated
        :type start_date: datetime.datetime
        :param external_trigger: whether this dag run is externally triggered
        :type external_trigger: bool
        :param session: database session
        :type session: sqlalchemy.orm.session.Session
        """

        return self.get_dag().create_dagrun(run_id=run_id,
                                            state=state,
                                            execution_date=execution_date,
                                            start_date=start_date,
                                            external_trigger=external_trigger,
                                            conf=conf,
                                            session=session)

Response: Creates a dag run from this dag including the tasks associated with this dag.


Instruction:
Can you create a Python 3 function that
sends the message to SQS queue otos

Response:
def execute(self, context):
        """
        Publish the message to SQS queue

        :param context: the context object
        :type context: dict
        :return: dict with information about the message sent
            For details of the returned dict see :py:meth:`botocore.client.SQS.send_message`
        :rtype: dict
        """

        hook = SQSHook(aws_conn_id=self.aws_conn_id)

        result = hook.send_message(queue_url=self.sqs_queue,
                                   message_body=self.message_content,
                                   delay_seconds=self.delay_seconds,
                                   message_attributes=self.message_attributes)

        self.log.info('result is send_message is %s', result)

        return result


Instruction:
Write a Python 3 function that can
generate the HTML for a paging component.

Response:
def generate_pages(current_page, num_of_pages,
                   search=None, showPaused=None, window=7):
    """
    Generates the HTML for a paging component using a similar logic to the paging
    auto-generated by Flask managed views. The paging component defines a number of
    pages visible in the pager (window) and once the user goes to a page beyond the
    largest visible, it would scroll to the right the page numbers and keeps the
    current one in the middle of the pager component. When in the last pages,
    the pages won't scroll and just keep moving until the last page. Pager also contains
    <first, previous, ..., next, last> pages.
    This component takes into account custom parameters such as search and showPaused,
    which could be added to the pages link in order to maintain the state between
    client and server. It also allows to make a bookmark on a specific paging state.
    :param current_page:
        the current page number, 0-indexed
    :param num_of_pages:
        the total number of pages
    :param search:
        the search query string, if any
    :param showPaused:
        false if paused dags will be hidden, otherwise true to show them
    :param window:
        the number of pages to be shown in the paging component (7 default)
    :return:
        the HTML string of the paging component
    """

    void_link = 'javascript:void(0)'
    first_node = Markup("""<li class="paginate_button {disabled}" id="dags_first">
    <a href="{href_link}" aria-controls="dags" data-dt-idx="0" tabindex="0">&laquo;</a>
</li>""")

    previous_node = Markup("""<li class="paginate_button previous {disabled}" id="dags_previous">
    <a href="{href_link}" aria-controls="dags" data-dt-idx="0" tabindex="0">&lt;</a>
</li>""")

    next_node = Markup("""<li class="paginate_button next {disabled}" id="dags_next">
    <a href="{href_link}" aria-controls="dags" data-dt-idx="3" tabindex="0">&gt;</a>
</li>""")

    last_node = Markup("""<li class="paginate_button {disabled}" id="dags_last">
    <a href="{href_link}" aria-controls="dags" data-dt-idx="3" tabindex="0">&raquo;</a>
</li>""")

    page_node = Markup("""<li class="paginate_button {is_active}">
    <a href="{href_link}" aria-controls="dags" data-dt-idx="2" tabindex="0">{page_num}</a>
</li>""")

    output = [Markup('<ul class="pagination" style="margin-top:0px;">')]

    is_disabled = 'disabled' if current_page <= 0 else ''
    output.append(first_node.format(href_link="?{}"
                                    .format(get_params(page=0,
                                                       search=search,
                                                       showPaused=showPaused)),
                                    disabled=is_disabled))

    page_link = void_link
    if current_page > 0:
        page_link = '?{}'.format(get_params(page=(current_page - 1),
                                            search=search,
                                            showPaused=showPaused))

    output.append(previous_node.format(href_link=page_link,
                                       disabled=is_disabled))

    mid = int(window / 2)
    last_page = num_of_pages - 1

    if current_page <= mid or num_of_pages < window:
        pages = [i for i in range(0, min(num_of_pages, window))]
    elif mid < current_page < last_page - mid:
        pages = [i for i in range(current_page - mid, current_page + mid + 1)]
    else:
        pages = [i for i in range(num_of_pages - window, last_page + 1)]

    def is_current(current, page):
        return page == current

    for page in pages:
        vals = {
            'is_active': 'active' if is_current(current_page, page) else '',
            'href_link': void_link if is_current(current_page, page)
                         else '?{}'.format(get_params(page=page,
                                                      search=search,
                                                      showPaused=showPaused)),
            'page_num': page + 1
        }
        output.append(page_node.format(**vals))

    is_disabled = 'disabled' if current_page >= num_of_pages - 1 else ''

    page_link = (void_link if current_page >= num_of_pages - 1
                 else '?{}'.format(get_params(page=current_page + 1,
                                              search=search,
                                              showPaused=showPaused)))

    output.append(next_node.format(href_link=page_link, disabled=is_disabled))
    output.append(last_node.format(href_link="?{}"
                                   .format(get_params(page=last_page,
                                                      search=search,
                                                      showPaused=showPaused)),
                                   disabled=is_disabled))

    output.append(Markup('</ul>'))

    return Markup('\n'.join(output))


Instruction:
Write a Python 3 function for
returning a json response from a json serializable python object

Response:
def json_response(obj):
    """
    returns a json response from a json serializable python object
    """
    return Response(
        response=json.dumps(
            obj, indent=4, cls=AirflowJsonEncoder),
        status=200,
        mimetype="application/json")


Instruction:
Can you tell what is the following Python 3 function doing
def open_maybe_zipped(f, mode='r'):

    _, archive, filename = ZIP_REGEX.search(f).groups()
    if archive and zipfile.is_zipfile(archive):
        return zipfile.ZipFile(archive, mode=mode).open(filename)
    else:
        return io.open(f, mode=mode)

Response: Opens the given file if it is a zip file.


Instruction:
Can you write a function in Python 3 where it
uses by cache to get a unique key per URL

Response:
def make_cache_key(*args, **kwargs):
    """
    Used by cache to get a unique key per URL
    """
    path = request.path
    args = str(hash(frozenset(request.args.items())))
    return (path + args).encode('ascii', 'ignore')


Instruction:
Given the following Python 3 function, write the documentation
def get_conn(self):
        if not self._conn:
            self._conn = VideoIntelligenceServiceClient(credentials=self._get_credentials())
        return self._conn

Response: Returns Gcp Video Intelligence Service client


Instruction:
Implement a function in Python 3 to
perform video annotation. :param input_uri: Input video location. Currently, only Google Cloud Storage URIs are supported, which must be specified in the following format: ``gs://bucket-id/object-id``. :type input_uri: str :param input_content: The video data bytes. If unset, the input video(s) should be specified via ``input_uri``. If set, ``input_uri`` should be unset. :type input_content: bytes :param features: Requested video annotation features. :type features: list[google.cloud.videointelligence_v1.VideoIntelligenceServiceClient.enums.Feature] :param output_uri: Optional, location where the output (in JSON format) should be stored. Currently, only Google Cloud Storage URIs are supported, which must be specified in the following format: ``gs://bucket-id/object-id``. :type output_uri: str :param video_context: Optional, Additional video context and/or feature-specific parameters. :type video_context: dict or google.cloud.videointelligence_v1.types.VideoContext :param location: Optional, cloud region where annotation should take place. Supported cloud regions: us-east1, us-west1, europe-west1, asia-east1. If no region is specified, a region will be determined based on video file location. :type location: str :param retry: Retry object used to determine when/if to retry requests. If None is specified, requests will not be retried. :type retry: google.api_core.retry.Retry :param timeout: Optional, The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt. :type timeout: float :param metadata: Optional, Additional metadata that is provided to the method. :type metadata: seq[tuple[str, str]]

Response:
def annotate_video(
        self,
        input_uri=None,
        input_content=None,
        features=None,
        video_context=None,
        output_uri=None,
        location=None,
        retry=None,
        timeout=None,
        metadata=None,
    ):
        """
        Performs video annotation.

        :param input_uri: Input video location. Currently, only Google Cloud Storage URIs are supported,
            which must be specified in the following format: ``gs://bucket-id/object-id``.
        :type input_uri: str
        :param input_content: The video data bytes.
            If unset, the input video(s) should be specified via ``input_uri``.
            If set, ``input_uri`` should be unset.
        :type input_content: bytes
        :param features: Requested video annotation features.
        :type features: list[google.cloud.videointelligence_v1.VideoIntelligenceServiceClient.enums.Feature]
        :param output_uri: Optional, location where the output (in JSON format) should be stored. Currently,
            only Google Cloud Storage URIs are supported, which must be specified in the following format:
            ``gs://bucket-id/object-id``.
        :type output_uri: str
        :param video_context: Optional, Additional video context and/or feature-specific parameters.
        :type video_context: dict or google.cloud.videointelligence_v1.types.VideoContext
        :param location: Optional, cloud region where annotation should take place. Supported cloud regions:
            us-east1, us-west1, europe-west1, asia-east1.
            If no region is specified, a region will be determined based on video file location.
        :type location: str
        :param retry: Retry object used to determine when/if to retry requests.
            If None is specified, requests will not be retried.
        :type retry: google.api_core.retry.Retry
        :param timeout: Optional, The amount of time, in seconds, to wait for the request to complete.
            Note that if retry is specified, the timeout applies to each individual attempt.
        :type timeout: float
        :param metadata: Optional, Additional metadata that is provided to the method.
        :type metadata: seq[tuple[str, str]]
        """
        client = self.get_conn()
        return client.annotate_video(
            input_uri=input_uri,
            input_content=input_content,
            features=features,
            video_context=video_context,
            output_uri=output_uri,
            location_id=location,
            retry=retry,
            timeout=timeout,
            metadata=metadata,
        )


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _get_api_key(self):
        conn = self.get_connection(self.http_conn_id)
        api_key = conn.password
        if not api_key:
            raise AirflowException('Opsgenie API Key is required for this hook, '
                                   'please check your conn_id configuration.')
        return api_key

Response: Get Opsgenie API Key for creating alert.


Instruction:
Can you generate the documentation for the following Python 3 function
def get_conn(self, headers=None):
        conn = self.get_connection(self.http_conn_id)
        self.base_url = conn.host if conn.host else 'https://api.opsgenie.com'
        session = requests.Session()
        if headers:
            session.headers.update(headers)
        return session

Response: Get a connection to the OPS NIE API.


Instruction:
Write a Python 3 script to
execute the Opsgenie Alert call TTS

Response:
def execute(self, payload={}):
        """
        Execute the Opsgenie Alert call

        :param payload: Opsgenie API Create Alert payload values
            See https://docs.opsgenie.com/docs/alert-api#section-create-alert
        :type payload: dict
        """
        api_key = self._get_api_key()
        return self.run(endpoint='v2/alerts',
                        data=json.dumps(payload),
                        headers={'Content-Type': 'application/json',
                                 'Authorization': 'GenieKey %s' % api_key})


Instruction:
Can you write a function in Python 3 where it
executes the bash command in a temporary directory and return True if the command was successful False otherwise

Response:
def poke(self, context):
        """
        Execute the bash command in a temporary directory
        which will be cleaned afterwards
        """
        bash_command = self.bash_command
        self.log.info("Tmp dir root location: \n %s", gettempdir())
        with TemporaryDirectory(prefix='airflowtmp') as tmp_dir:
            with NamedTemporaryFile(dir=tmp_dir, prefix=self.task_id) as f:
                f.write(bytes(bash_command, 'utf_8'))
                f.flush()
                fname = f.name
                script_location = tmp_dir + "/" + fname
                self.log.info("Temporary script location: %s", script_location)
                self.log.info("Running command: %s", bash_command)
                sp = Popen(
                    ['bash', fname],
                    stdout=PIPE, stderr=STDOUT,
                    close_fds=True, cwd=tmp_dir,
                    env=self.env, preexec_fn=os.setsid)

                self.sp = sp

                self.log.info("Output:")
                line = ''
                for line in iter(sp.stdout.readline, b''):
                    line = line.decode(self.output_encoding).strip()
                    self.log.info(line)
                sp.wait()
                self.log.info("Command exited with return code %s", sp.returncode)

                return not sp.returncode


Instruction:
Can you create a Python 3 function that
builds the Opsgenie JSON payload.

Response:
def _build_opsgenie_payload(self):
        """
        Construct the Opsgenie JSON payload. All relevant parameters are combined here
        to a valid Opsgenie JSON payload.

        :return: Opsgenie payload (dict) to send
        """
        payload = {}

        for key in [
            "message", "alias", "description", "responders",
            "visibleTo", "actions", "tags", "details", "entity",
            "source", "priority", "user", "note"
        ]:
            val = getattr(self, key)
            if val:
                payload[key] = val
        return payload


Instruction:
Can you generate a brief explanation for the following Python 3 code
def execute(self, context):
        self.hook = OpsgenieAlertHook(self.opsgenie_conn_id)
        self.hook.execute(self._build_opsgenie_payload())

Response: Call the OpsgenieAlertHook to post message


Instruction:
Make a summary of the following Python 3 code
def get_conn(self):
        if not self.conn:
            self.conn = self.get_client_type('athena')
        return self.conn

Response:
check if aws conn exists already or create one and return it

        :return: boto3 session


Instruction:
Can you generate the documentation for the following Python 3 function
def run_query(self, query, query_context, result_configuration, client_request_token=None):
        response = self.conn.start_query_execution(QueryString=query,
                                                   ClientRequestToken=client_request_token,
                                                   QueryExecutionContext=query_context,
                                                   ResultConfiguration=result_configuration)
        query_execution_id = response['QueryExecutionId']
        return query_execution_id

Response: Run a presto query on athena with provided config and return query_execution_id


Instruction:
Can you tell what is the following Python 3 function doing
def check_query_status(self, query_execution_id):
        response = self.conn.get_query_execution(QueryExecutionId=query_execution_id)
        state = None
        try:
            state = response['QueryExecution']['Status']['State']
        except Exception as ex:
            self.log.error('Exception while getting query state', ex)
        finally:
            return state

Response: Check the status of a query execution. Returns None or one of valid query states.


Instruction:
Make a summary of the following Python 3 code
def get_query_results(self, query_execution_id):
        query_state = self.check_query_status(query_execution_id)
        if query_state is None:
            self.log.error('Invalid Query state')
            return None
        elif query_state in self.INTERMEDIATE_STATES or query_state in self.FAILURE_STATES:
            self.log.error('Query is in {state} state. Cannot fetch results'.format(state=query_state))
            return None
        return self.conn.get_query_results(QueryExecutionId=query_execution_id)

Response: Fetch submitted athena query results. Returns None if query is in intermediate state or if query is in failure state or if query is in cancelled state


Instruction:
Make a summary of the following Python 3 code
def poll_query_status(self, query_execution_id, max_tries=None):
        try_number = 1
        final_query_state = None  # Query state when query reaches final state or max_tries reached
        while True:
            query_state = self.check_query_status(query_execution_id)
            if query_state is None:
                self.log.info('Trial {try_number}: Invalid query state. Retrying again'.format(
                    try_number=try_number))
            elif query_state in self.INTERMEDIATE_STATES:
                self.log.info('Trial {try_number}: Query is still in an intermediate state - {state}'
                              .format(try_number=try_number, state=query_state))
            else:
                self.log.info('Trial {try_number}: Query execution completed. Final state is {state}'
                              .format(try_number=try_number, state=query_state))
                final_query_state = query_state
                break
            if max_tries and try_number >= max_tries:  # Break loop if max_tries reached
                final_query_state = query_state
                break
            try_number += 1
            sleep(self.sleep_time)
        return final_query_state

Response: Poll the status of the submitted athena query until the query reaches final state.


Instruction:
Make a summary of the following Python 3 code
def get_conn(self):
        if self.conn is None:
            cnopts = pysftp.CnOpts()
            if self.no_host_key_check:
                cnopts.hostkeys = None
            cnopts.compression = self.compress
            conn_params = {
                'host': self.remote_host,
                'port': self.port,
                'username': self.username,
                'cnopts': cnopts
            }
            if self.password and self.password.strip():
                conn_params['password'] = self.password
            if self.key_file:
                conn_params['private_key'] = self.key_file
            if self.private_key_pass:
                conn_params['private_key_pass'] = self.private_key_pass

            self.conn = pysftp.Connection(**conn_params)
        return self.conn

Response: Returns an SFTP connection object for the current object.


Instruction:
Given the following Python 3 function, write the documentation
def describe_directory(self, path):
        conn = self.get_conn()
        flist = conn.listdir_attr(path)
        files = {}
        for f in flist:
            modify = datetime.datetime.fromtimestamp(
                f.st_mtime).strftime('%Y%m%d%H%M%S')
            files[f.filename] = {
                'size': f.st_size,
                'type': 'dir' if stat.S_ISDIR(f.st_mode) else 'file',
                'modify': modify}
        return files

Response:
Returns a dictionary of all files
            for all files in the remote directory.


Instruction:
How would you implement a function in Python 3 that
returns a list of files on the remote system.

Response:
def list_directory(self, path):
        """
        Returns a list of files on the remote system.
        :param path: full path to the remote directory to list
        :type path: str
        """
        conn = self.get_conn()
        files = conn.listdir(path)
        return files


Instruction:
Make a summary of the following Python 3 code
def create_directory(self, path, mode=777):
        conn = self.get_conn()
        conn.mkdir(path, mode)

Response: Creates a directory on the remote system.


Instruction:
Write a Python 3 script for
transferring the remote file to a local location. If local_full_path is a string path, the file will be put at that location :param remote_full_path: full path to the remote file :type remote_full_path: str :param local_full_path: full path to the local file :type local_full_path: str

Response:
def retrieve_file(self, remote_full_path, local_full_path):
        """
        Transfers the remote file to a local location.
        If local_full_path is a string path, the file will be put
        at that location
        :param remote_full_path: full path to the remote file
        :type remote_full_path: str
        :param local_full_path: full path to the local file
        :type local_full_path: str
        """
        conn = self.get_conn()
        self.log.info('Retrieving file from FTP: %s', remote_full_path)
        conn.get(remote_full_path, local_full_path)
        self.log.info('Finished retrieving file from FTP: %s', remote_full_path)


Instruction:
Given the following Python 3 function, write the documentation
def store_file(self, remote_full_path, local_full_path):
        conn = self.get_conn()
        conn.put(local_full_path, remote_full_path)

Response: Stores a local file to the remote location.


Instruction:
Can you generate the documentation for the following Python 3 function
def __handle_rate_limit_exception(self, rate_limit_exception):
        retry_after = int(
            rate_limit_exception.response.headers.get('Retry-After', 60))
        self.log.info(
            "Hit Zendesk API rate limit. Pausing for %s seconds",
            retry_after
        )
        time.sleep(retry_after)

Response: Handle a Zendesk API rate limit exception.


Instruction:
Here you have a function in Python 3, explain what it does
def call(self, path, query=None, get_all_pages=True, side_loading=False):
        zendesk = self.get_conn()
        first_request_successful = False

        while not first_request_successful:
            try:
                results = zendesk.call(path, query)
                first_request_successful = True
            except RateLimitError as rle:
                self.__handle_rate_limit_exception(rle)

        # Find the key with the results
        keys = [path.split("/")[-1].split(".json")[0]]
        next_page = results['next_page']
        if side_loading:
            keys += query['include'].split(',')
        results = {key: results[key] for key in keys}

        if get_all_pages:
            while next_page is not None:
                try:
                    # Need to split because the next page URL has
                    # `github.zendesk...`
                    # in it, but the call function needs it removed.
                    next_url = next_page.split(self.__url)[1]
                    self.log.info("Calling %s", next_url)
                    more_res = zendesk.call(next_url)
                    for key in results:
                        results[key].extend(more_res[key])
                    if next_page == more_res['next_page']:
                        # Unfortunately zdesk doesn't always throw ZendeskError
                        # when we are done getting all the data. Sometimes the
                        # next just refers to the current set of results.
                        # Hence, need to deal with this special case
                        break
                    else:
                        next_page = more_res['next_page']
                except RateLimitError as rle:
                    self.__handle_rate_limit_exception(rle)
                except ZendeskError as ze:
                    if b"Use a start_time older than 5 minutes" in ze.msg:
                        # We have pretty up to date data
                        break
                    else:
                        raise ze

        return results

Response: Call Zendesk API and return the results.


Instruction:
Implement a function in Python 3 to
retrieve the partition values for a table.

Response:
def get_partitions(self,
                       database_name,
                       table_name,
                       expression='',
                       page_size=None,
                       max_items=None):
        """
        Retrieves the partition values for a table.

        :param database_name: The name of the catalog database where the partitions reside.
        :type database_name: str
        :param table_name: The name of the partitions' table.
        :type table_name: str
        :param expression: An expression filtering the partitions to be returned.
            Please see official AWS documentation for further information.
            https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-GetPartitions
        :type expression: str
        :param page_size: pagination size
        :type page_size: int
        :param max_items: maximum items to return
        :type max_items: int
        :return: set of partition values where each value is a tuple since
            a partition may be composed of multiple columns. For example:
            ``{('2018-01-01','1'), ('2018-01-01','2')}``
        """
        config = {
            'PageSize': page_size,
            'MaxItems': max_items,
        }

        paginator = self.get_conn().get_paginator('get_partitions')
        response = paginator.paginate(
            DatabaseName=database_name,
            TableName=table_name,
            Expression=expression,
            PaginationConfig=config
        )

        partitions = set()
        for page in response:
            for p in page['Partitions']:
                partitions.add(tuple(p['Values']))

        return partitions


Instruction:
How would you explain what the following Python 3 function does
def check_for_partition(self, database_name, table_name, expression):
        partitions = self.get_partitions(database_name, table_name, expression, max_items=1)

        if partitions:
            return True
        else:
            return False

Response: Checks whether a partition exists with the given expression.


Instruction:
Create a Python 3 function for
getting the information of the table in a hive database

Response:
def get_table(self, database_name, table_name):
        """
        Get the information of the table

        :param database_name: Name of hive database (schema) @table belongs to
        :type database_name: str
        :param table_name: Name of hive table
        :type table_name: str
        :rtype: dict

        >>> hook = AwsGlueCatalogHook()
        >>> r = hook.get_table('db', 'table_foo')
        >>> r['Name'] = 'table_foo'
        """

        result = self.get_conn().get_table(DatabaseName=database_name, Name=table_name)

        return result['Table']


Instruction:
How would you code a function in Python 3 to
get the physical location of the table in the specified database.

Response:
def get_table_location(self, database_name, table_name):
        """
        Get the physical location of the table

        :param database_name: Name of hive database (schema) @table belongs to
        :type database_name: str
        :param table_name: Name of hive table
        :type table_name: str
        :return: str
        """

        table = self.get_table(database_name, table_name)

        return table['StorageDescriptor']['Location']


Instruction:
Can you generate the documentation for the following Python 3 function
def cluster_status(self, cluster_identifier):
        conn = self.get_conn()
        try:
            response = conn.describe_clusters(
                ClusterIdentifier=cluster_identifier)['Clusters']
            return response[0]['ClusterStatus'] if response else None
        except conn.exceptions.ClusterNotFoundFault:
            return 'cluster_not_found'

Response: Return status of a cluster in the specified language


Instruction:
Implement a Python 3 function for
deleting a cluster and optionally create a snapshot

Response:
def delete_cluster(
            self,
            cluster_identifier,
            skip_final_cluster_snapshot=True,
            final_cluster_snapshot_identifier=''):
        """
        Delete a cluster and optionally create a snapshot

        :param cluster_identifier: unique identifier of a cluster
        :type cluster_identifier: str
        :param skip_final_cluster_snapshot: determines cluster snapshot creation
        :type skip_final_cluster_snapshot: bool
        :param final_cluster_snapshot_identifier: name of final cluster snapshot
        :type final_cluster_snapshot_identifier: str
        """
        response = self.get_conn().delete_cluster(
            ClusterIdentifier=cluster_identifier,
            SkipFinalClusterSnapshot=skip_final_cluster_snapshot,
            FinalClusterSnapshotIdentifier=final_cluster_snapshot_identifier
        )
        return response['Cluster'] if response['Cluster'] else None


Instruction:
Can you write a function in Python 3 where it
gets a list of snapshots for a cluster

Response:
def describe_cluster_snapshots(self, cluster_identifier):
        """
        Gets a list of snapshots for a cluster

        :param cluster_identifier: unique identifier of a cluster
        :type cluster_identifier: str
        """
        response = self.get_conn().describe_cluster_snapshots(
            ClusterIdentifier=cluster_identifier
        )
        if 'Snapshots' not in response:
            return None
        snapshots = response['Snapshots']
        snapshots = filter(lambda x: x['Status'], snapshots)
        snapshots.sort(key=lambda x: x['SnapshotCreateTime'], reverse=True)
        return snapshots


Instruction:
Can you generate a brief explanation for the following Python 3 code
def restore_from_cluster_snapshot(self, cluster_identifier, snapshot_identifier):
        response = self.get_conn().restore_from_cluster_snapshot(
            ClusterIdentifier=cluster_identifier,
            SnapshotIdentifier=snapshot_identifier
        )
        return response['Cluster'] if response['Cluster'] else None

Response: Restores a cluster from its snapshot


Instruction:
How would you explain what the following Python 3 function does
def create_cluster_snapshot(self, snapshot_identifier, cluster_identifier):
        response = self.get_conn().create_cluster_snapshot(
            SnapshotIdentifier=snapshot_identifier,
            ClusterIdentifier=cluster_identifier,
        )
        return response['Snapshot'] if response['Snapshot'] else None

Response: Creates a snapshot of a specific resource


Instruction:
Explain what the following Python 3 code does
def add_volume(self, volume):

        self._add_volume(name=volume.name, configs=volume.configs)

Response: Adds a new volume to the cache.


Instruction:
Can you write a function in Python 3 where it
adds a mount to the cache entry.

Response:
def add_mount(self,
                  volume_mount):
        """
        Args:
            volume_mount (VolumeMount):
        """
        self._add_mount(
            name=volume_mount.name,
            mount_path=volume_mount.mount_path,
            sub_path=volume_mount.sub_path,
            read_only=volume_mount.read_only
        )


Instruction:
Can you implement a function in Python 3 that
creates a job flow using the config from the EMR connection.

Response:
def create_job_flow(self, job_flow_overrides):
        """
        Creates a job flow using the config from the EMR connection.
        Keys of the json extra hash may have the arguments of the boto3
        run_job_flow method.
        Overrides for this config may be passed as the job_flow_overrides.
        """

        if not self.emr_conn_id:
            raise AirflowException('emr_conn_id must be present to use create_job_flow')

        emr_conn = self.get_connection(self.emr_conn_id)

        config = emr_conn.extra_dejson.copy()
        config.update(job_flow_overrides)

        response = self.get_conn().run_job_flow(**config)

        return response


Instruction:
How would you explain what the following Python 3 function does
def filter_for_filesize(result, size=None):
        if size:
            log = LoggingMixin().log
            log.debug(
                'Filtering for file size >= %s in files: %s',
                size, map(lambda x: x['path'], result)
            )
            size *= settings.MEGABYTE
            result = [x for x in result if x['length'] >= size]
            log.debug('HdfsSensor.poke: after size filter result is %s', result)
        return result

Response: Filter the result list to only return the files with size > = size.


Instruction:
Create a Python 3 function to
filter out the result of Snakebite ls for ignored extensions.

Response:
def filter_for_ignored_ext(result, ignored_ext, ignore_copying):
        """
        Will filter if instructed to do so the result to remove matching criteria

        :param result: list of dicts returned by Snakebite ls
        :type result: list[dict]
        :param ignored_ext: list of ignored extensions
        :type ignored_ext: list
        :param ignore_copying: shall we ignore ?
        :type ignore_copying: bool
        :return: list of dicts which were not removed
        :rtype: list[dict]
        """
        if ignore_copying:
            log = LoggingMixin().log
            regex_builder = r"^.*\.(%s$)$" % '$|'.join(ignored_ext)
            ignored_extensions_regex = re.compile(regex_builder)
            log.debug(
                'Filtering result for ignored extensions: %s in files %s',
                ignored_extensions_regex.pattern, map(lambda x: x['path'], result)
            )
            result = [x for x in result if not ignored_extensions_regex.match(x['path'])]
            log.debug('HdfsSensor.poke: after ext filter result is %s', result)
        return result


Instruction:
How would you explain what the following Python 3 function does
def execute(self, context):
        s3_conn = S3Hook(self.s3_conn_id)

        # Grab collection and execute query according to whether or not it is a pipeline
        if self.is_pipeline:
            results = MongoHook(self.mongo_conn_id).aggregate(
                mongo_collection=self.mongo_collection,
                aggregate_query=self.mongo_query,
                mongo_db=self.mongo_db
            )

        else:
            results = MongoHook(self.mongo_conn_id).find(
                mongo_collection=self.mongo_collection,
                query=self.mongo_query,
                mongo_db=self.mongo_db
            )

        # Performs transform then stringifies the docs results into json format
        docs_str = self._stringify(self.transform(results))

        # Load Into S3
        s3_conn.load_string(
            string_data=docs_str,
            key=self.s3_key,
            bucket_name=self.s3_bucket,
            replace=self.replace
        )

        return True

Response: Executes the task instance.


Instruction:
How would you explain what the following Python 3 function does
def _stringify(iterable, joinable='\n'):
        return joinable.join(
            [json.dumps(doc, default=json_util.default) for doc in iterable]
        )

Response:
Takes an iterable containing dictionaries and
            returns a stringified version using python join


Instruction:
Explain what the following Python 3 code does
def get_pool(name, session=None):
    if not (name and name.strip()):
        raise AirflowBadRequest("Pool name shouldn't be empty")

    pool = session.query(Pool).filter_by(pool=name).first()
    if pool is None:
        raise PoolNotFound("Pool '%s' doesn't exist" % name)

    return pool

Response: Get pool by a given name.


Instruction:
Write a Python 3 script for
creating a pool with a given parameters.

Response:
def create_pool(name, slots, description, session=None):
    """Create a pool with a given parameters."""
    if not (name and name.strip()):
        raise AirflowBadRequest("Pool name shouldn't be empty")

    try:
        slots = int(slots)
    except ValueError:
        raise AirflowBadRequest("Bad value for `slots`: %s" % slots)

    session.expire_on_commit = False
    pool = session.query(Pool).filter_by(pool=name).first()
    if pool is None:
        pool = Pool(pool=name, slots=slots, description=description)
        session.add(pool)
    else:
        pool.slots = slots
        pool.description = description

    session.commit()

    return pool


Instruction:
Can you write a function in Python 3 where it
deletes pool by a given name.

Response:
def delete_pool(name, session=None):
    """Delete pool by a given name."""
    if not (name and name.strip()):
        raise AirflowBadRequest("Pool name shouldn't be empty")

    pool = session.query(Pool).filter_by(pool=name).first()
    if pool is None:
        raise PoolNotFound("Pool '%s' doesn't exist" % name)

    session.delete(pool)
    session.commit()

    return pool


Instruction:
Create a Python 3 function for
converting a python dictionary to the proto object

Response:
def _dict_to_proto(py_dict, proto):
        """
        Converts a python dictionary to the proto supplied

        :param py_dict: The dictionary to convert
        :type py_dict: dict
        :param proto: The proto object to merge with dictionary
        :type proto: protobuf
        :return: A parsed python dictionary in provided proto format
        :raises:
            ParseError: On JSON parsing problems.
        """
        dict_json_str = json.dumps(py_dict)
        return json_format.Parse(dict_json_str, proto)


Instruction:
Can you generate the documentation for the following Python 3 function
def wait_for_operation(self, operation, project_id=None):
        self.log.info("Waiting for OPERATION_NAME %s", operation.name)
        time.sleep(OPERATIONAL_POLL_INTERVAL)
        while operation.status != Operation.Status.DONE:
            if operation.status == Operation.Status.RUNNING or operation.status == \
                    Operation.Status.PENDING:
                time.sleep(OPERATIONAL_POLL_INTERVAL)
            else:
                raise exceptions.GoogleCloudError(
                    "Operation has failed with status: %s" % operation.status)
            # To update status of operation
            operation = self.get_operation(operation.name, project_id=project_id or self.project_id)
        return operation

Response: Waits for an operation to complete and returns it.


Instruction:
Create a Python 3 function for
fetching the operation from Google Cloud

Response:
def get_operation(self, operation_name, project_id=None):
        """
        Fetches the operation from Google Cloud

        :param operation_name: Name of operation to fetch
        :type operation_name: str
        :param project_id: Google Cloud Platform project ID
        :type project_id: str
        :return: The new, updated operation from Google Cloud
        """
        return self.get_client().get_operation(project_id=project_id or self.project_id,
                                               zone=self.location,
                                               operation_id=operation_name)


Instruction:
Implement a function in Python 3 to
append labels to the provided Cluster Protobuf

Response:
def _append_label(cluster_proto, key, val):
        """
        Append labels to provided Cluster Protobuf

        Labels must fit the regex ``[a-z]([-a-z0-9]*[a-z0-9])?`` (current
         airflow version string follows semantic versioning spec: x.y.z).

        :param cluster_proto: The proto to append resource_label airflow
            version to
        :type cluster_proto: google.cloud.container_v1.types.Cluster
        :param key: The key label
        :type key: str
        :param val:
        :type val: str
        :return: The cluster proto updated with new label
        """
        val = val.replace('.', '-').replace('+', '-')
        cluster_proto.resource_labels.update({key: val})
        return cluster_proto