Instruction:
Create a Python 3 function to
decode a message into a Message object.

Response:
def decode(message):
        '''
        The decoder understands the comma-seperated format produced by the encoder and
        allocates the two values to the correct keys:
            data['hello'] = 'world'
            data['x'] = 10

        '''
        (hello, x) = message.payload.split(",")

        data = {}
        data['hello'] = hello
        data['x'] = x

        timestamp = datetime.now(pytz.timezone('UTC'))

        return Message(data, timestamp)


Instruction:
Write a Python 3 script to
retrieve the organization - specific status of each of the services offered by the IBM Watson IoT Platform.

Response:
def dataTransfer(self, start, end, detail=False):
        """
        Retrieve the organization-specific status of each of the services offered by the IBM Watson IoT Platform.
        In case of failure it throws APIException
        """

        r = self._apiClient.get(
            "api/v0002/usage/data-traffic?start=%s&end=%s&detail=%s"
            % (start.strftime("%Y-%m-%d"), end.strftime("%Y-%m-%d"), detail)
        )

        if r.status_code == 200:
            return DataTransferSummary(**r.json())
        else:
            raise ApiException(r)


Instruction:
Can you generate the documentation for the following Python 3 function
def initiate(self, request):
        url = MgmtRequests.mgmtRequests
        r = self._apiClient.post(url, request)

        if r.status_code == 202:
            return r.json()
        else:
            raise ApiException(r)

Response: Initiate a device management request.


Instruction:
Make a summary of the following Python 3 code
def delete(self, requestId):
        url = MgmtRequests.mgmtSingleRequest % (requestId)
        r = self._apiClient.delete(url)

        if r.status_code == 204:
            return True
        else:
            raise ApiException(r)

Response: Delete a single request from the device management service.


Instruction:
Explain what the following Python 3 code does
def get(self, requestId):
        url = MgmtRequests.mgmtSingleRequest % (requestId)
        r = self._apiClient.get(url)

        if r.status_code == 200:
            return r.json()
        else:
            raise ApiException(r)

Response: This method returns details of a device management request. It accepts requestId as parameter


Instruction:
How would you implement a function in Python 3 that
gets a list of device mangaement request device statuses.

Response:
def getStatus(self, requestId, typeId=None, deviceId=None):
        """
        Get a list of device management request device statuses.
        Get an individual device mangaement request device status.
        """
        if typeId is None or deviceId is None:
            url = MgmtRequests.mgmtRequestStatus % (requestId)
            r = self._apiClient.get(url)

            if r.status_code == 200:
                return r.json()
            else:
                raise ApiException(r)
        else:
            url = MgmtRequests.mgmtRequestSingleDeviceStatus % (requestId, typeId, deviceId)
            r = self._apiClient.get(url)

            if r.status_code == 200:
                return r.json()
            else:
                raise ApiException(r)


Instruction:
Create a Python 3 function for
forcing a flush of the index to storage. Renders index inaccessible.

Response:
def close(self):
        """Force a flush of the index to storage. Renders index
        inaccessible."""
        if self.handle:
            self.handle.destroy()
            self.handle = None
        else:
            raise IOError("Unclosable index")


Instruction:
Make a summary of the following Python 3 code
def insert(self, id, coordinates, obj=None):
        p_mins, p_maxs = self.get_coordinate_pointers(coordinates)
        data = ctypes.c_ubyte(0)
        size = 0
        pyserialized = None
        if obj is not None:
            size, data, pyserialized = self._serialize(obj)
        core.rt.Index_InsertData(self.handle, id, p_mins, p_maxs,
                                 self.properties.dimension, data, size)

Response: Inserts an entry into the index with the given coordinates.


Instruction:
Write a Python 3 function for
returning the number of objects that intersect the given coordinates.

Response:
def count(self, coordinates):
        """Return number of objects that intersect the given coordinates.

        :param coordinates: sequence or array
            This may be an object that satisfies the numpy array
            protocol, providing the index's dimension * 2 coordinate
            pairs representing the `mink` and `maxk` coordinates in
            each dimension defining the bounds of the query window.

        The following example queries the index for any objects any objects
        that were stored in the index intersect the bounds given in the
        coordinates::

            >>> from rtree import index
            >>> idx = index.Index()
            >>> idx.insert(4321,
            ...            (34.3776829412, 26.7375853734, 49.3776829412,
            ...             41.7375853734),
            ...            obj=42)

            >>> print(idx.count((0, 0, 60, 60)))
            1

        """
        p_mins, p_maxs = self.get_coordinate_pointers(coordinates)

        p_num_results = ctypes.c_uint64(0)

        core.rt.Index_Intersects_count(self.handle,
                                       p_mins,
                                       p_maxs,
                                       self.properties.dimension,
                                       ctypes.byref(p_num_results))

        return p_num_results.value


Instruction:
Explain what the following Python 3 code does
def intersection(self, coordinates, objects=False):

        if objects:
            return self._intersection_obj(coordinates, objects)

        p_mins, p_maxs = self.get_coordinate_pointers(coordinates)

        p_num_results = ctypes.c_uint64(0)

        it = ctypes.pointer(ctypes.c_int64())

        core.rt.Index_Intersects_id(self.handle,
                                    p_mins,
                                    p_maxs,
                                    self.properties.dimension,
                                    ctypes.byref(it),
                                    ctypes.byref(p_num_results))
        return self._get_ids(it, p_num_results.value)

Response: Return ids or objects in the index that intersect the given coordinates.


Instruction:
Can you write a function in Python 3 where it
returns the k nearest objects to the given coordinates.

Response:
def nearest(self, coordinates, num_results=1, objects=False):
        """Returns the ``k``-nearest objects to the given coordinates.

        :param coordinates: sequence or array
            This may be an object that satisfies the numpy array
            protocol, providing the index's dimension * 2 coordinate
            pairs representing the `mink` and `maxk` coordinates in
            each dimension defining the bounds of the query window.

        :param num_results: integer
            The number of results to return nearest to the given coordinates.
            If two index entries are equidistant, *both* are returned.
            This property means that :attr:`num_results` may return more
            items than specified

        :param objects: True / False / 'raw'
            If True, the nearest method will return index objects that
            were pickled when they were stored with each index entry, as
            well as the id and bounds of the index entries.
            If 'raw', it will return the object as entered into the database
            without the :class:`rtree.index.Item` wrapper.

        Example of finding the three items nearest to this one::

            >>> from rtree import index
            >>> idx = index.Index()
            >>> idx.insert(4321, (34.37, 26.73, 49.37, 41.73), obj=42)
            >>> hits = idx.nearest((0, 0, 10, 10), 3, objects=True)
        """
        if objects:
            return self._nearest_obj(coordinates, num_results, objects)
        p_mins, p_maxs = self.get_coordinate_pointers(coordinates)

        p_num_results = ctypes.pointer(ctypes.c_uint64(num_results))

        it = ctypes.pointer(ctypes.c_int64())

        core.rt.Index_NearestNeighbors_id(self.handle,
                                          p_mins,
                                          p_maxs,
                                          self.properties.dimension,
                                          ctypes.byref(it),
                                          p_num_results)

        return self._get_ids(it, p_num_results.contents.value)


Instruction:
How would you code a function in Python 3 to
return the bounds of the index .

Response:
def get_bounds(self, coordinate_interleaved=None):
        """Returns the bounds of the index

        :param coordinate_interleaved: If True, the coordinates are turned
            in the form [xmin, ymin, ..., kmin, xmax, ymax, ..., kmax],
            otherwise they are returned as
            [xmin, xmax, ymin, ymax, ..., ..., kmin, kmax].  If not specified,
            the :attr:`interleaved` member of the index is used, which
            defaults to True.
        """
        if coordinate_interleaved is None:
            coordinate_interleaved = self.interleaved
        return _get_bounds(
            self.handle, core.rt.Index_GetBounds, coordinate_interleaved)


Instruction:
Given the following Python 3 function, write the documentation
def delete(self, id, coordinates):
        p_mins, p_maxs = self.get_coordinate_pointers(coordinates)
        core.rt.Index_DeleteData(
            self.handle, id, p_mins, p_maxs, self.properties.dimension)

Response: Deletes items from the index with the given id within the given coordinates.


Instruction:
Given the following Python 3 function, write the documentation
def deinterleave(self, interleaved):
        assert len(interleaved) % 2 == 0, ("must be a pairwise list")
        dimension = len(interleaved) // 2
        di = []
        for i in range(dimension):
            di.extend([interleaved[i], interleaved[i + dimension]])
        return di

Response: Return a new list of the elements in the given interleaved list.


Instruction:
How would you explain what the following Python 3 function does
def interleave(self, deinterleaved):
        assert len(deinterleaved) % 2 == 0, ("must be a pairwise list")
        #  dimension = len(deinterleaved) / 2
        interleaved = []
        for i in range(2):
            interleaved.extend([deinterleaved[i + j]
                                for j in range(0, len(deinterleaved), 2)])
        return interleaved

Response: Interleave a set of two - element lists.


Instruction:
How would you code a function in Python 3 to
please override this method.

Response:
def destroy(self, context, returnError):
        """please override"""
        returnError.contents.value = self.IllegalStateError
        raise NotImplementedError("You must override this method.")


Instruction:
Can you generate the documentation for the following Python 3 function
def loadByteArray(self, context, page, resultLen, resultData, returnError):
        returnError.contents.value = self.IllegalStateError
        raise NotImplementedError("You must override this method.")

Response: loadByteArray is a wrapper for loadByteArray


Instruction:
Can you generate the documentation for the following Python 3 function
def storeByteArray(self, context, page, len, data, returnError):
        returnError.contents.value = self.IllegalStateError
        raise NotImplementedError("You must override this method.")

Response: Store a byte array.


Instruction:
Explain what the following Python 3 code does
def deleteByteArray(self, context, page, returnError):
        returnError.contents.value = self.IllegalStateError
        raise NotImplementedError("You must override this method.")

Response: please override this method.


Instruction:
Write a Python 3 script to
please override this method.

Response:
def flush(self, context, returnError):
        """please override"""
        returnError.contents.value = self.IllegalStateError
        raise NotImplementedError("You must override this method.")


Instruction:
Make a summary of the following Python 3 code
def insert(self, obj, coordinates):
        try:
            count = self._objects[id(obj)] + 1
        except KeyError:
            count = 1
        self._objects[id(obj)] = (count, obj)
        return super(RtreeContainer, self).insert(id(obj), coordinates, None)

Response: Inserts an item into the index with the given coordinates.


Instruction:
Make a summary of the following Python 3 code
def intersection(self, coordinates, bbox=False):
        if bbox == False:
            for id in super(RtreeContainer,
                            self).intersection(coordinates, bbox):
                yield self._objects[id][1]
        elif bbox == True:
            for value in super(RtreeContainer,
                               self).intersection(coordinates, bbox):
                value.object = self._objects[value.id][1]
                value.id = None
                yield value
        else:
            raise ValueError(
                "valid values for the bbox argument are True and False")

Response: Return the ids or objects in the index that intersect the given coordinates.


Instruction:
Implement a function in Python 3 to
delete the item from the container within the specified coordinates.

Response:
def delete(self, obj, coordinates):
        """Deletes the item from the container within the specified
        coordinates.

        :param obj: object
            Any object.

        :param coordinates: sequence or array
            Dimension * 2 coordinate pairs, representing the min
            and max coordinates in each dimension of the item to be
            deleted from the index. Their ordering will depend on the
            index's :attr:`interleaved` data member.
            These are not the coordinates of a space containing the
            item, but those of the item itself. Together with the
            id parameter, they determine which item will be deleted.
            This may be an object that satisfies the numpy array protocol.

        Example::

            >>> from rtree import index
            >>> idx = index.RtreeContainer()
            >>> idx.delete(object(),
            ...            (34.3776829412, 26.7375853734, 49.3776829412,
            ...             41.7375853734))
            Traceback (most recent call last):
             ...
            IndexError: object is not in the index

        """
        try:
            count = self._objects[id(obj)] - 1
        except KeyError:
            raise IndexError('object is not in the index')
        if count == 0:
            del self._objects[obj]
        else:
            self._objects[id(obj)] = (count, obj)
        return super(RtreeContainer, self).delete(id, coordinates)


Instruction:
Can you generate the documentation for the following Python 3 function
def check_return(result, func, cargs):
    "Error checking for Error calls"
    if result != 0:
        s = rt.Error_GetLastErrorMsg().decode()
        msg = 'LASError in "%s": %s' % \
            (func.__name__, s)
        rt.Error_Reset()
        raise RTreeError(msg)
    return True

Response: Error checking for Error calls


Instruction:
Here you have a function in Python 3, explain what it does
def check_void_done(result, func, cargs):
    "Error checking for void* returns that might be empty with no error"
    if rt.Error_GetErrorCount():
        s = rt.Error_GetLastErrorMsg().decode()
        msg = 'Error in "%s": %s' % (func.__name__, s)
        rt.Error_Reset()
        raise RTreeError(msg)
    return result

Response: Error checking for void * returns that might be empty with no error


Instruction:
Make a summary of the following Python 3 code
def load(self):

        if isinstance(self.application, str):
            return util.import_app(self.application)
        else:
            return self.application

Response: Attempt an import of the specified application


Instruction:
Given the following Python 3 function, write the documentation
def init_app(self, app):
        if not hasattr(app, 'extensions'):
            app.extensions = {}

        if 'common' in app.extensions:
            raise RuntimeError("Flask-Common extension already initialized")

        app.extensions['common'] = self
        self.app = app

        if 'COMMON_FILESERVER_DISABLED' not in app.config:
            with app.test_request_context():

                # Configure WhiteNoise.
                app.wsgi_app = WhiteNoise(app.wsgi_app, root=url_for('static', filename='')[1:])

        self.cache = Cache(app, config={'CACHE_TYPE': app.config.get("COMMON_CACHE_TYPE", 'simple')})

        @app.before_request
        def before_request_callback():
            request.start_time = maya.now()

        @app.after_request
        def after_request_callback(response):
            if 'COMMON_POWERED_BY_DISABLED' not in current_app.config:
                response.headers['X-Powered-By'] = 'Flask'
            if 'COMMON_PROCESSED_TIME_DISABLED' not in current_app.config:
                response.headers['X-Processed-Time'] = maya.now().epoch - request.start_time.epoch
            return response

        @app.route('/favicon.ico')
        def favicon():
            return redirect(url_for('static', filename='favicon.ico'), code=301)

Response: Initializes the Flask application with Common.


Instruction:
Write a Python 3 function that can
serve the Flask application.

Response:
def serve(self, workers=None, **kwargs):
        """Serves the Flask application."""
        if self.app.debug:
            print(crayons.yellow('Booting Flask development server...'))
            self.app.run()

        else:
            print(crayons.yellow('Booting Gunicorn...'))

            # Start the web server.
            server = GunicornServer(
                self.app, workers=workers or number_of_gunicorn_workers(),
                worker_class='egg:meinheld#gunicorn_worker', **kwargs)
            server.run()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def to_native(self, value):
        context_request = None
        if self.context:
            context_request = self.context.get('request', None)
        return build_versatileimagefield_url_set(
            value,
            self.sizes,
            request=context_request
        )

Response: Convert the value to a native version of the url field.


Instruction:
Explain what the following Python 3 code does
def crop_on_centerpoint(self, image, width, height, ppoi=(0.5, 0.5)):
        ppoi_x_axis = int(image.size[0] * ppoi[0])
        ppoi_y_axis = int(image.size[1] * ppoi[1])
        center_pixel_coord = (ppoi_x_axis, ppoi_y_axis)
        # Calculate the aspect ratio of `image`
        orig_aspect_ratio = float(
            image.size[0]
        ) / float(
            image.size[1]
        )
        crop_aspect_ratio = float(width) / float(height)

        # Figure out if we're trimming from the left/right or top/bottom
        if orig_aspect_ratio >= crop_aspect_ratio:
            # `image` is wider than what's needed,
            # crop from left/right sides
            orig_crop_width = int(
                (crop_aspect_ratio * float(image.size[1])) + 0.5
            )
            orig_crop_height = image.size[1]
            crop_boundary_top = 0
            crop_boundary_bottom = orig_crop_height
            crop_boundary_left = center_pixel_coord[0] - (orig_crop_width // 2)
            crop_boundary_right = crop_boundary_left + orig_crop_width
            if crop_boundary_left < 0:
                crop_boundary_left = 0
                crop_boundary_right = crop_boundary_left + orig_crop_width
            elif crop_boundary_right > image.size[0]:
                crop_boundary_right = image.size[0]
                crop_boundary_left = image.size[0] - orig_crop_width

        else:
            # `image` is taller than what's needed,
            # crop from top/bottom sides
            orig_crop_width = image.size[0]
            orig_crop_height = int(
                (float(image.size[0]) / crop_aspect_ratio) + 0.5
            )
            crop_boundary_left = 0
            crop_boundary_right = orig_crop_width
            crop_boundary_top = center_pixel_coord[1] - (orig_crop_height // 2)
            crop_boundary_bottom = crop_boundary_top + orig_crop_height
            if crop_boundary_top < 0:
                crop_boundary_top = 0
                crop_boundary_bottom = crop_boundary_top + orig_crop_height
            elif crop_boundary_bottom > image.size[1]:
                crop_boundary_bottom = image.size[1]
                crop_boundary_top = image.size[1] - orig_crop_height
        # Cropping the image from the original image
        cropped_image = image.crop(
            (
                crop_boundary_left,
                crop_boundary_top,
                crop_boundary_right,
                crop_boundary_bottom
            )
        )
        # Resizing the newly cropped image to the size specified
        # (as determined by `width`x`height`)
        return cropped_image.resize(
            (width, height),
            Image.ANTIALIAS
        )

Response: Crop an image on the centerpoint.


Instruction:
Can you create a Python 3 function that
processes an image and return a BytesIO instance of the image cropped to width and height.

Response:
def process_image(self, image, image_format, save_kwargs,
                      width, height):
        """
        Return a BytesIO instance of `image` cropped to `width` and `height`.

        Cropping will first reduce an image down to its longest side
        and then crop inwards centered on the Primary Point of Interest
        (as specified by `self.ppoi`)
        """
        imagefile = BytesIO()
        palette = image.getpalette()
        cropped_image = self.crop_on_centerpoint(
            image,
            width,
            height,
            self.ppoi
        )

        # Using ImageOps.fit on GIFs can introduce issues with their palette
        # Solution derived from: http://stackoverflow.com/a/4905209/1149774
        if image_format == 'GIF':
            cropped_image.putpalette(palette)

        cropped_image.save(
            imagefile,
            **save_kwargs
        )

        return imagefile


Instruction:
Can you tell what is the following Python 3 function doing
def process_image(self, image, image_format, save_kwargs,
                      width, height):
        """
        Return a BytesIO instance of `image` that fits in a bounding box.

        Bounding box dimensions are `width`x`height`.
        """
        imagefile = BytesIO()
        image.thumbnail(
            (width, height),
            Image.ANTIALIAS
        )
        image.save(
            imagefile,
            **save_kwargs
        )
        return imagefile

Response: Process an image and return a BytesIO instance of the image that fits in a bounding box.


Instruction:
How would you explain what the following Python 3 function does
def process_image(self, image, image_format, save_kwargs={}):
        imagefile = BytesIO()
        inv_image = ImageOps.invert(image)
        inv_image.save(
            imagefile,
            **save_kwargs
        )
        return imagefile

Response: Return a BytesIO instance of image with inverted colors.


Instruction:
How would you code a function in Python 3 to
ensure data is prepped properly before handing off to ImageField.

Response:
def to_python(self, data):
        """Ensure data is prepped properly before handing off to ImageField."""
        if data is not None:
            if hasattr(data, 'open'):
                data.open()
            return super(VersatileImageFormField, self).to_python(data)


Instruction:
Write a Python 3 script to
process the field s placeholder image.

Response:
def process_placeholder_image(self):
        """
        Process the field's placeholder image.

        Ensures the placeholder image has been saved to the same storage class
        as the field in a top level folder with a name specified by
        settings.VERSATILEIMAGEFIELD_SETTINGS['placeholder_directory_name']

        This should be called by the VersatileImageFileDescriptor __get__.
        If self.placeholder_image_name is already set it just returns right away.
        """
        if self.placeholder_image_name:
            return

        placeholder_image_name = None
        placeholder_image = self.placeholder_image
        if placeholder_image:
            if isinstance(placeholder_image, OnStoragePlaceholderImage):
                name = placeholder_image.path
            else:
                name = placeholder_image.image_data.name
            placeholder_image_name = os.path.join(
                VERSATILEIMAGEFIELD_PLACEHOLDER_DIRNAME, name
            )
            if not self.storage.exists(placeholder_image_name):
                self.storage.save(
                    placeholder_image_name,
                    placeholder_image.image_data
                )
        self.placeholder_image_name = placeholder_image_name


Instruction:
Here you have a function in Python 3, explain what it does
def pre_save(self, model_instance, add):
        file = super(VersatileImageField, self).pre_save(model_instance, add)
        self.update_ppoi_field(model_instance)
        return file

Response: Return field s value just before saving.


Instruction:
Can you generate the documentation for the following Python 3 function
def update_ppoi_field(self, instance, *args, **kwargs):
        # Nothing to update if the field doesn't have have a ppoi
        # dimension field.
        if not self.ppoi_field:
            return

        # getattr will call the VersatileImageFileDescriptor's __get__ method,
        # which coerces the assigned value into an instance of
        # self.attr_class(VersatileImageFieldFile in this case).
        file = getattr(instance, self.attname)

        # file should be an instance of VersatileImageFieldFile or should be
        # None.
        ppoi = None
        if file and not isinstance(file, tuple):
            if hasattr(file, 'ppoi'):
                ppoi = file.ppoi

        # Update the ppoi field.
        if self.ppoi_field:
            setattr(instance, self.ppoi_field, ppoi)

Response: Update the field s ppoi field.


Instruction:
Can you implement a function in Python 3 that
saves the data sent from the MultiValueField forms that set ppoi values.

Response:
def save_form_data(self, instance, data):
        """
        Handle data sent from MultiValueField forms that set ppoi values.

        `instance`: The model instance that is being altered via a form
        `data`: The data sent from the form to this field which can be either:
        * `None`: This is unset data from an optional field
        * A two-position tuple: (image_form_data, ppoi_data)
            * `image_form-data` options:
                * `None` the file for this field is unchanged
                * `False` unassign the file form the field
            * `ppoi_data` data structure:
                * `%(x_coordinate)sx%(y_coordinate)s': The ppoi data to
                  assign to the unchanged file

        """
        to_assign = data
        if data and isinstance(data, tuple):
            # This value is coming from a MultiValueField
            if data[0] is None:
                # This means the file hasn't changed but we need to
                # update the ppoi
                current_field = getattr(instance, self.name)
                if data[1]:
                    current_field.ppoi = data[1]
                to_assign = current_field
            elif data[0] is False:
                # This means the 'Clear' checkbox was checked so we
                # need to empty the field
                to_assign = ''
            else:
                # This means there is a new upload so we need to unpack
                # the tuple and assign the first position to the field
                # attribute
                to_assign = data[0]
        super(VersatileImageField, self).save_form_data(instance, to_assign)


Instruction:
Create a Python 3 function for
returning a formfield with the default values set for the VersatileImageField.

Response:
def formfield(self, **kwargs):
        """Return a formfield."""
        # This is a fairly standard way to set up some defaults
        # while letting the caller override them.
        defaults = {}
        if self.ppoi_field:
            defaults['form_class'] = SizedImageCenterpointClickDjangoAdminField
        if kwargs.get('widget') is AdminFileWidget:
            # Ensuring default admin widget is skipped (in favor of using
            # SizedImageCenterpointClickDjangoAdminField's default widget as
            # the default widget choice for use in the admin).
            # This is for two reasons:
            # 1. To prevent 'typical' admin users (those who want to use
            #    the PPOI 'click' widget by default) from having to
            #    specify a formfield_overrides for each ModelAdmin class
            #    used by each model that has a VersatileImageField.
            # 2. If a VersatileImageField does not have a ppoi_field specified
            #    it will 'fall back' to a ClearableFileInput anyways.
            # If admin users do, in fact, want to force use of the
            # AdminFileWidget they can simply subclass AdminFileWidget and
            # specify it in their ModelAdmin.formfield_overrides (though,
            # if that's the case, why are they using VersatileImageField in
            # the first place?)
            del kwargs['widget']
        defaults.update(kwargs)
        return super(VersatileImageField, self).formfield(**defaults)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def value_to_string(self, obj):
        if DJANGO_VERSION > (1, 9):
            value = self.value_from_object(obj)
        else:
            value = self._get_val_from_obj(obj)
        return self.get_prep_value(value)

Response: Prepare field for serialization.


Instruction:
Write a Python 3 function that can
print out a Yum - style progress bar.

Response:
def cli_progress_bar(start, end, bar_length=50):
    """
    Prints out a Yum-style progress bar (via sys.stdout.write).
    `start`: The 'current' value of the progress bar.
    `end`: The '100%' value of the progress bar.
    `bar_length`: The size of the overall progress bar.

    Example output with start=20, end=100, bar_length=50:
    [###########----------------------------------------] 20/100 (100%)

    Intended to be used in a loop. Example:
    end = 100
    for i in range(end):
        cli_progress_bar(i, end)

    Based on an implementation found here:
        http://stackoverflow.com/a/13685020/1149774
    """
    percent = float(start) / end
    hashes = '#' * int(round(percent * bar_length))
    spaces = '-' * (bar_length - len(hashes))
    stdout.write(
        "\r[{0}] {1}/{2} ({3}%)".format(
            hashes + spaces,
            start,
            end,
            int(round(percent * 100))
        )
    )
    stdout.flush()


Instruction:
Can you generate a brief explanation for the following Python 3 code
def _prewarm_versatileimagefield(size_key, versatileimagefieldfile):
        versatileimagefieldfile.create_on_demand = True
        try:
            url = get_url_from_image_key(versatileimagefieldfile, size_key)
        except Exception:
            success = False
            url_or_filepath = versatileimagefieldfile.name
            logger.exception('Thumbnail generation failed',
                             extra={'path': url_or_filepath})
        else:
            success = True
            url_or_filepath = url
        return (success, url_or_filepath)

Response: Pre - warm the image if it was successfully created.


Instruction:
Explain what the following Python 3 code does
def warm(self):
        num_images_pre_warmed = 0
        failed_to_create_image_path_list = []
        total = self.queryset.count() * len(self.size_key_list)
        for a, instance in enumerate(self.queryset, start=1):
            for b, size_key in enumerate(self.size_key_list, start=1):
                success, url_or_filepath = self._prewarm_versatileimagefield(
                    size_key,
                    reduce(getattr, self.image_attr.split("."), instance)
                )
                if success is True:
                    num_images_pre_warmed += 1
                    if self.verbose:
                        cli_progress_bar(num_images_pre_warmed, total)
                else:
                    failed_to_create_image_path_list.append(url_or_filepath)

                if a * b == total:
                    stdout.write('\n')

        stdout.flush()
        return (num_images_pre_warmed, failed_to_create_image_path_list)

Response: Warm the storage class with the contents of the VersatileImageField field.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def autodiscover():
    from importlib import import_module
    from django.apps import apps
    from django.utils.module_loading import module_has_submodule

    for app_config in apps.get_app_configs():
        # Attempt to import the app's module.

        try:
            before_import_sizedimage_registry = copy.copy(
                versatileimagefield_registry._sizedimage_registry
            )
            before_import_filter_registry = copy.copy(
                versatileimagefield_registry._filter_registry
            )
            import_module('%s.versatileimagefield' % app_config.name)
        except Exception:
            # Reset the versatileimagefield_registry to the state before the
            # last import as this import will have to reoccur on the next
            # request and this could raise NotRegistered and AlreadyRegistered
            # exceptions (see django ticket #8245).
            versatileimagefield_registry._sizedimage_registry = \
                before_import_sizedimage_registry
            versatileimagefield_registry._filter_registry = \
                before_import_filter_registry

            # Decide whether to bubble up this error. If the app just
            # doesn't have the module in question, we can ignore the error
            # attempting to import it, otherwise we want it to bubble up.
            if module_has_submodule(app_config.module, 'versatileimagefield'):
                raise

Response: Autodiscover the versatile image field modules.


Instruction:
Make a summary of the following Python 3 code
def register_sizer(self, attr_name, sizedimage_cls):
        if attr_name.startswith(
            '_'
        ) or attr_name in self.unallowed_sizer_names:
            raise UnallowedSizerName(
                "`%s` is an unallowed Sizer name. Sizer names cannot begin "
                "with an underscore or be named any of the "
                "following: %s." % (
                    attr_name,
                    ', '.join([
                        name
                        for name in self.unallowed_sizer_names
                    ])
                )
            )
        if not issubclass(sizedimage_cls, SizedImage):
            raise InvalidSizedImageSubclass(
                'Only subclasses of versatileimagefield.datastructures.'
                'SizedImage may be registered with register_sizer'
            )

        if attr_name in self._sizedimage_registry:
            raise AlreadyRegistered(
                'A SizedImage class is already registered to the `%s` '
                'attribute. If you would like to override this attribute, '
                'use the unregister method' % attr_name
            )
        else:
            self._sizedimage_registry[attr_name] = sizedimage_cls

Response: Registers a new SizedImage subclass with the given attribute name.


Instruction:
Can you generate the documentation for the following Python 3 function
def register_filter(self, attr_name, filterimage_cls):
        if attr_name.startswith('_'):
            raise UnallowedFilterName(
                '`%s` is an unallowed Filter name. Filter names cannot begin '
                'with an underscore.' % attr_name
            )
        if not issubclass(filterimage_cls, FilteredImage):
            raise InvalidFilteredImageSubclass(
                'Only subclasses of FilteredImage may be registered as '
                'filters with VersatileImageFieldRegistry'
            )

        if attr_name in self._filter_registry:
            raise AlreadyRegistered(
                'A ProcessedImageMixIn class is already registered to the `%s`'
                ' attribute. If you would like to override this attribute, '
                'use the unregister method' % attr_name
            )
        else:
            self._filter_registry[attr_name] = filterimage_cls

Response: Register a new FilteredImage subclass with the given attribute name.


Instruction:
Make a summary of the following Python 3 code
def unregister_filter(self, attr_name):
        if attr_name not in self._filter_registry:
            raise NotRegistered(
                'No FilteredImage subclass is registered to %s' % attr_name
            )
        else:
            del self._filter_registry[attr_name]

Response: Unregister the FilteredImage subclass currently assigned to attr_name.


Instruction:
How would you explain what the following Python 3 function does
def url(self):
        if not self.name and self.field.placeholder_image_name:
            return self.storage.url(self.field.placeholder_image_name)

        return super(VersatileImageMixIn, self).url

Response: Returns the appropriate URL for this VersatileImageFieldFile.


Instruction:
Can you tell what is the following Python 3 function doing
def ppoi(self, value):
        ppoi = validate_ppoi(
            value,
            return_converted_tuple=True
        )
        if ppoi is not False:
            self._ppoi_value = ppoi
            self.build_filters_and_sizers(ppoi, self.create_on_demand)

Response: Primary Point of Interest ( ppoi setter.


Instruction:
Explain what the following Python 3 code does
def build_filters_and_sizers(self, ppoi_value, create_on_demand):
        name = self.name
        if not name and self.field.placeholder_image_name:
            name = self.field.placeholder_image_name
        self.filters = FilterLibrary(
            name,
            self.storage,
            versatileimagefield_registry,
            ppoi_value,
            create_on_demand
        )
        for (
            attr_name,
            sizedimage_cls
        ) in iteritems(versatileimagefield_registry._sizedimage_registry):
            setattr(
                self,
                attr_name,
                sizedimage_cls(
                    path_to_image=name,
                    storage=self.storage,
                    create_on_demand=create_on_demand,
                    ppoi=ppoi_value
                )
            )

Response: Build the filters and sizers for a field.


Instruction:
Implement a function in Python 3 to
return the location where filtered images are stored.

Response:
def get_filtered_root_folder(self):
        """Return the location where filtered images are stored."""
        folder, filename = os.path.split(self.name)
        return os.path.join(folder, VERSATILEIMAGEFIELD_FILTERED_DIRNAME, '')


Instruction:
Can you tell what is the following Python 3 function doing
def get_sized_root_folder(self):
        folder, filename = os.path.split(self.name)
        return os.path.join(VERSATILEIMAGEFIELD_SIZED_DIRNAME, folder, '')

Response: Return the location where sized images are stored.


Instruction:
Write a Python 3 script to
return the location where filtered + sized images are stored.

Response:
def get_filtered_sized_root_folder(self):
        """Return the location where filtered + sized images are stored."""
        sized_root_folder = self.get_sized_root_folder()
        return os.path.join(
            sized_root_folder,
            VERSATILEIMAGEFIELD_FILTERED_DIRNAME
        )


Instruction:
Make a summary of the following Python 3 code
def delete_matching_files_from_storage(self, root_folder, regex):
        if not self.name:   # pragma: no cover
            return
        try:
            directory_list, file_list = self.storage.listdir(root_folder)
        except OSError:   # pragma: no cover
            pass
        else:
            folder, filename = os.path.split(self.name)
            basename, ext = os.path.splitext(filename)
            for f in file_list:
                if not f.startswith(basename) or not f.endswith(ext):   # pragma: no cover
                    continue
                tag = f[len(basename):-len(ext)]
                assert f == basename + tag + ext
                if regex.match(tag) is not None:
                    file_location = os.path.join(root_folder, f)
                    self.storage.delete(file_location)
                    cache.delete(
                        self.storage.url(file_location)
                    )
                    print(
                        "Deleted {file} (created from: {original})".format(
                            file=os.path.join(root_folder, f),
                            original=self.name
                        )
                    )

Response: Delete files in root_folder which match regex before file ext.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def validate_ppoi_tuple(value):
    valid = True
    while valid is True:
        if len(value) == 2 and isinstance(value, tuple):
            for x in value:
                if x >= 0 and x <= 1:
                    pass
                else:
                    valid = False
            break
        else:
            valid = False
    return valid

Response: Validates that a tuple is a valid PPOI.


Instruction:
Can you implement a function in Python 3 that
converts a string with format x y x y to a two position tuple.

Response:
def validate_ppoi(value, return_converted_tuple=False):
    """
    Converts, validates and optionally returns a string with formatting:
    '%(x_axis)dx%(y_axis)d' into a two position tuple.

    If a tuple is passed to `value` it is also validated.

    Both x_axis and y_axis must be floats or ints greater
    than 0 and less than 1.
    """

    valid_ppoi = True
    to_return = None
    if isinstance(value, tuple):
        valid_ppoi = validate_ppoi_tuple(value)
        if valid_ppoi:
            to_return = value
    else:
        tup = tuple()
        try:
            string_split = [
                float(segment.strip())
                for segment in value.split('x')
                if float(segment.strip()) >= 0 and float(segment.strip()) <= 1
            ]
        except Exception:
            valid_ppoi = False
        else:
            tup = tuple(string_split)

        valid_ppoi = validate_ppoi_tuple(tup)

        if valid_ppoi:
            to_return = tup
    if not valid_ppoi:
        raise ValidationError(
            message=INVALID_CENTERPOINT_ERROR_MESSAGE % str(value),
            code='invalid_ppoi'
        )
    else:
        if to_return and return_converted_tuple is True:
            return to_return


Instruction:
Given the following Python 3 function, write the documentation
def preprocess(self, image, image_format):
        save_kwargs = {'format': image_format}

        # Ensuring image is properly rotated
        if hasattr(image, '_getexif'):
            exif_datadict = image._getexif()  # returns None if no EXIF data
            if exif_datadict is not None:
                exif = dict(exif_datadict.items())
                orientation = exif.get(EXIF_ORIENTATION_KEY, None)
                if orientation == 3:
                    image = image.transpose(Image.ROTATE_180)
                elif orientation == 6:
                    image = image.transpose(Image.ROTATE_270)
                elif orientation == 8:
                    image = image.transpose(Image.ROTATE_90)

        # Ensure any embedded ICC profile is preserved
        save_kwargs['icc_profile'] = image.info.get('icc_profile')

        if hasattr(self, 'preprocess_%s' % image_format):
            image, addl_save_kwargs = getattr(
                self,
                'preprocess_%s' % image_format
            )(image=image)
            save_kwargs.update(addl_save_kwargs)

        return image, save_kwargs

Response: Preprocesses an image and returns a PIL Image instance and a dictionary of keyword arguments to be used when saving the image.


Instruction:
Can you write a function in Python 3 where it
receives a PIL Image instance of a GIF and return a 2 - tuple with the image and kwargs.

Response:
def preprocess_GIF(self, image, **kwargs):
        """
        Receive a PIL Image instance of a GIF and return 2-tuple.

        Args:
            * [0]: Original Image instance (passed to `image`)
            * [1]: Dict with a transparency key (to GIF transparency layer)
        """
        if 'transparency' in image.info:
            save_kwargs = {'transparency': image.info['transparency']}
        else:
            save_kwargs = {}
        return (image, save_kwargs)


Instruction:
Create a Python 3 function for
receiving a PIL Image instance of a JPEG and returns a 2 - tuple with the image and kwargs.

Response:
def preprocess_JPEG(self, image, **kwargs):
        """
        Receive a PIL Image instance of a JPEG and returns 2-tuple.

        Args:
            * [0]: Image instance, converted to RGB
            * [1]: Dict with a quality key (mapped to the value of `QUAL` as
                   defined by the `VERSATILEIMAGEFIELD_JPEG_RESIZE_QUALITY`
                   setting)
        """
        save_kwargs = {
            'progressive': VERSATILEIMAGEFIELD_PROGRESSIVE_JPEG,
            'quality': QUAL
        }
        if image.mode != 'RGB':
            image = image.convert('RGB')
        return (image, save_kwargs)


Instruction:
Can you generate a brief explanation for the following Python 3 code
def retrieve_image(self, path_to_image):
        image = self.storage.open(path_to_image, 'rb')
        file_ext = path_to_image.rsplit('.')[-1]
        image_format, mime_type = get_image_metadata_from_file_ext(file_ext)

        return (
            Image.open(image),
            file_ext,
            image_format,
            mime_type
        )

Response: Return a PIL Image instance stored at path_to_image.


Instruction:
Make a summary of the following Python 3 code
def save_image(self, imagefile, save_path, file_ext, mime_type):
        file_to_save = InMemoryUploadedFile(
            imagefile,
            None,
            'foo.%s' % file_ext,
            mime_type,
            imagefile.tell(),
            None
        )
        file_to_save.seek(0)
        self.storage.save(save_path, file_to_save)

Response: Save an image to the local storage at save_path.


Instruction:
Make a summary of the following Python 3 code
def ppoi_as_str(self):
        return "%s__%s" % (
            str(self.ppoi[0]).replace('.', '-'),
            str(self.ppoi[1]).replace('.', '-')
        )

Response: Return PPOI value as a string.


Instruction:
Can you create a Python 3 function that
creates a resized image.

Response:
def create_resized_image(self, path_to_image, save_path_on_storage,
                             width, height):
        """
        Create a resized image.

        `path_to_image`: The path to the image with the media directory to
                         resize. If `None`, the
                         VERSATILEIMAGEFIELD_PLACEHOLDER_IMAGE will be used.
        `save_path_on_storage`: Where on self.storage to save the resized image
        `width`: Width of resized image (int)
        `height`: Desired height of resized image (int)
        `filename_key`: A string that will be used in the sized image filename
                        to signify what operation was done to it.
                        Examples: 'crop' or 'scale'
        """
        image, file_ext, image_format, mime_type = self.retrieve_image(
            path_to_image
        )

        image, save_kwargs = self.preprocess(image, image_format)

        imagefile = self.process_image(
            image=image,
            image_format=image_format,
            save_kwargs=save_kwargs,
            width=width,
            height=height
        )
        self.save_image(imagefile, save_path_on_storage, file_ext, mime_type)


Instruction:
Implement a function in Python 3 to
render the widget as an HTML string.

Response:
def render(self, name, value, attrs=None, renderer=None):
        """
        Render the widget as an HTML string.

        Overridden here to support Django < 1.11.
        """
        if self.has_template_widget_rendering:
            return super(ClearableFileInputWithImagePreview, self).render(
                name, value, attrs=attrs, renderer=renderer
            )
        else:
            context = self.get_context(name, value, attrs)
            return render_to_string(self.template_name, context)


Instruction:
Implement a function in Python 3 to
get the context to render this widget with.

Response:
def get_context(self, name, value, attrs):
        """Get the context to render this widget with."""
        if self.has_template_widget_rendering:
            context = super(ClearableFileInputWithImagePreview, self).get_context(name, value, attrs)
        else:
            # Build the context manually.
            context = {}
            context['widget'] = {
                'name': name,
                'is_hidden': self.is_hidden,
                'required': self.is_required,
                'value': self._format_value(value),
                'attrs': self.build_attrs(self.attrs, attrs),
                'template_name': self.template_name,
                'type': self.input_type,
            }

        # It seems Django 1.11's ClearableFileInput doesn't add everything to the 'widget' key, so we can't use it
        # in MultiWidget. Add it manually here.
        checkbox_name = self.clear_checkbox_name(name)
        checkbox_id = self.clear_checkbox_id(checkbox_name)

        context['widget'].update({
            'checkbox_name': checkbox_name,
            'checkbox_id': checkbox_id,
            'is_initial': self.is_initial(value),
            'input_text': self.input_text,
            'initial_text': self.initial_text,
            'clear_checkbox_label': self.clear_checkbox_label,
        })

        if value and hasattr(value, "url"):
            context['widget'].update({
                'hidden_field_id': self.get_hidden_field_id(name),
                'point_stage_id': self.get_point_stage_id(name),
                'ppoi_id': self.get_ppoi_id(name),
                'sized_url': self.get_sized_url(value),
                'image_preview_id': self.image_preview_id(name),
            })

        return context


Instruction:
Can you implement a function in Python 3 that
builds an attribute dictionary.

Response:
def build_attrs(self, base_attrs, extra_attrs=None):
        """Build an attribute dictionary."""
        attrs = base_attrs.copy()
        if extra_attrs is not None:
            attrs.update(extra_attrs)
        return attrs


Instruction:
Write a Python 3 script for
returning the filename that is resized according to width height and filename_key.

Response:
def get_resized_filename(filename, width, height, filename_key):
    """
    Return the 'resized filename' (according to `width`, `height` and
    `filename_key`) in the following format:
    `filename`-`filename_key`-`width`x`height`.ext
    """
    try:
        image_name, ext = filename.rsplit('.', 1)
    except ValueError:
        image_name = filename
        ext = 'jpg'

    resized_template = "%(filename_key)s-%(width)dx%(height)d"
    if ext.lower() in ['jpg', 'jpeg']:
        resized_template = resized_template + "-%(quality)d"

    resized_key = resized_template % ({
        'filename_key': filename_key,
        'width': width,
        'height': height,
        'quality': QUAL
    })

    return "%(image_name)s-%(image_key)s.%(ext)s" % ({
        'image_name': image_name,
        'image_key': post_process_image_key(resized_key),
        'ext': ext
    })


Instruction:
Can you create a Python 3 function that
returns a path_to_image location on storage as dictated by width height and filename_key

Response:
def get_resized_path(path_to_image, width, height,
                     filename_key, storage):
    """
    Return a `path_to_image` location on `storage` as dictated by `width`, `height`
    and `filename_key`
    """
    containing_folder, filename = os.path.split(path_to_image)

    resized_filename = get_resized_filename(
        filename,
        width,
        height,
        filename_key
    )

    joined_path = os.path.join(*[
        VERSATILEIMAGEFIELD_SIZED_DIRNAME,
        containing_folder,
        resized_filename
    ]).replace(' ', '')  # Removing spaces so this path is memcached friendly

    return joined_path


Instruction:
Write a Python 3 function that can
return the filtered filename according to filename_key

Response:
def get_filtered_filename(filename, filename_key):
    """
    Return the 'filtered filename' (according to `filename_key`)
    in the following format:
    `filename`__`filename_key`__.ext
    """
    try:
        image_name, ext = filename.rsplit('.', 1)
    except ValueError:
        image_name = filename
        ext = 'jpg'
    return "%(image_name)s__%(filename_key)s__.%(ext)s" % ({
        'image_name': image_name,
        'filename_key': filename_key,
        'ext': ext
    })


Instruction:
Can you generate a brief explanation for the following Python 3 code
def get_filtered_path(path_to_image, filename_key, storage):
    containing_folder, filename = os.path.split(path_to_image)

    filtered_filename = get_filtered_filename(filename, filename_key)
    path_to_return = os.path.join(*[
        containing_folder,
        VERSATILEIMAGEFIELD_FILTERED_DIRNAME,
        filtered_filename
    ])
    # Removing spaces so this path is memcached key friendly
    path_to_return = path_to_return.replace(' ', '')
    return path_to_return

Response: Return the path to the filtered file


Instruction:
Can you write a function in Python 3 where it
validates a list of size keys.

Response:
def validate_versatileimagefield_sizekey_list(sizes):
    """
    Validate a list of size keys.

    `sizes`: An iterable of 2-tuples, both strings. Example:
    [
        ('large', 'url'),
        ('medium', 'crop__400x400'),
        ('small', 'thumbnail__100x100')
    ]
    """
    try:
        for key, size_key in sizes:
            size_key_split = size_key.split('__')
            if size_key_split[-1] != 'url' and (
                'x' not in size_key_split[-1]
            ):
                raise InvalidSizeKey(
                    "{0} is an invalid size. All sizes must be either "
                    "'url' or made up of at least two segments separated "
                    "by double underscores. Examples: 'crop__400x400', "
                    "filters__invert__url".format(size_key)
                )
    except ValueError:
        raise InvalidSizeKeySet(
            '{} is an invalid size key set. Size key sets must be an '
            'iterable of 2-tuples'.format(str(sizes))
        )
    return list(set(sizes))


Instruction:
Write a Python 3 script to
build a URL from image_key.

Response:
def get_url_from_image_key(image_instance, image_key):
    """Build a URL from `image_key`."""
    img_key_split = image_key.split('__')
    if 'x' in img_key_split[-1]:
        size_key = img_key_split.pop(-1)
    else:
        size_key = None
    img_url = reduce(getattr, img_key_split, image_instance)
    if size_key:
        img_url = img_url[size_key].url
    return img_url


Instruction:
Can you tell what is the following Python 3 function doing
def build_versatileimagefield_url_set(image_instance, size_set, request=None):
    size_set = validate_versatileimagefield_sizekey_list(size_set)
    to_return = {}
    if image_instance or image_instance.field.placeholder_image:
        for key, image_key in size_set:
            img_url = get_url_from_image_key(image_instance, image_key)
            if request is not None:
                img_url = request.build_absolute_uri(img_url)
            to_return[key] = img_url
    return to_return

Response: Builds a dictionary of urls corresponding to size_set.


Instruction:
Given the following Python 3 function, write the documentation
def get_rendition_key_set(key):
    try:
        rendition_key_set = IMAGE_SETS[key]
    except KeyError:
        raise ImproperlyConfigured(
            "No Rendition Key Set exists at "
            "settings.VERSATILEIMAGEFIELD_RENDITION_KEY_SETS['{}']".format(key)
        )
    else:
        return validate_versatileimagefield_sizekey_list(rendition_key_set)

Response: Returns a validated and prepped Rendition Key Set for the given key.


Instruction:
Can you write a function in Python 3 where it
takes a raw Instruction and translates it into a human readable text representation.

Response:
def format_instruction(insn):
    """
    Takes a raw `Instruction` and translates it into a human readable text
    representation. As of writing, the text representation for WASM is not yet
    standardized, so we just emit some generic format.
    """
    text = insn.op.mnemonic

    if not insn.imm:
        return text

    return text + ' ' + ', '.join([
        getattr(insn.op.imm_struct, x.name).to_string(
            getattr(insn.imm, x.name)
        )
        for x in insn.op.imm_struct._meta.fields
    ])


Instruction:
Explain what the following Python 3 code does
def format_function(
    func_body,
    func_type=None,
    indent=2,
    format_locals=True,
):
    """
    Takes a `FunctionBody` and optionally a `FunctionType`, yielding the string
    representation of the function line by line. The function type is required
    for formatting function parameter and return value information.
    """
    if func_type is None:
        yield 'func'
    else:
        param_section = ' (param {})'.format(' '.join(
            map(format_lang_type, func_type.param_types)
        )) if func_type.param_types else ''
        result_section = ' (result {})'.format(
            format_lang_type(func_type.return_type)
        ) if func_type.return_type else ''
        yield 'func' + param_section + result_section

    if format_locals and func_body.locals:
        yield '(locals {})'.format(' '.join(itertools.chain.from_iterable(
            itertools.repeat(format_lang_type(x.type), x.count)
            for x in func_body.locals
        )))

    level = 1
    for cur_insn in decode_bytecode(func_body.code):
        if cur_insn.op.flags & INSN_LEAVE_BLOCK:
            level -= 1
        yield ' ' * (level * indent) + format_instruction(cur_insn)
        if cur_insn.op.flags & INSN_ENTER_BLOCK:
            level += 1

Response:
Yields a string
    representation of a function.


Instruction:
Write a Python 3 script to
decode raw bytecode yielding Instructions.

Response:
def decode_bytecode(bytecode):
    """Decodes raw bytecode, yielding `Instruction`s."""
    bytecode_wnd = memoryview(bytecode)
    while bytecode_wnd:
        opcode_id = byte2int(bytecode_wnd[0])
        opcode = OPCODE_MAP[opcode_id]

        if opcode.imm_struct is not None:
            offs, imm, _ = opcode.imm_struct.from_raw(None, bytecode_wnd[1:])
        else:
            imm = None
            offs = 0

        insn_len = 1 + offs
        yield Instruction(opcode, imm, insn_len)
        bytecode_wnd = bytecode_wnd[insn_len:]


Instruction:
Can you generate the documentation for the following Python 3 function
def decode_module(module, decode_name_subsections=False):
    module_wnd = memoryview(module)

    # Read & yield module header.
    hdr = ModuleHeader()
    hdr_len, hdr_data, _ = hdr.from_raw(None, module_wnd)
    yield ModuleFragment(hdr, hdr_data)
    module_wnd = module_wnd[hdr_len:]

    # Read & yield sections.
    while module_wnd:
        sec = Section()
        sec_len, sec_data, _ = sec.from_raw(None, module_wnd)

        # If requested, decode name subsections when encountered.
        if (
            decode_name_subsections and
            sec_data.id == SEC_UNK and
            sec_data.name == SEC_NAME
        ):
            sec_wnd = sec_data.payload
            while sec_wnd:
                subsec = NameSubSection()
                subsec_len, subsec_data, _ = subsec.from_raw(None, sec_wnd)
                yield ModuleFragment(subsec, subsec_data)
                sec_wnd = sec_wnd[subsec_len:]
        else:
            yield ModuleFragment(sec, sec_data)

        module_wnd = module_wnd[sec_len:]

Response: Decodes raw WASM modules yielding ModuleFragments.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def deprecated_func(func):

    # We use a mutable container here to work around Py2's lack of
    # the `nonlocal` keyword.
    first_usage = [True]

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        if first_usage[0]:
            warnings.warn(
                "Call to deprecated function {}.".format(func.__name__),
                DeprecationWarning,
            )
            first_usage[0] = False
        return func(*args, **kwargs)

    return wrapper

Response: Deprecates a function printing a warning on the first usage.


Instruction:
How would you code a function in Python 3 to
send an action to the server.

Response:
def send_action(self, action, as_list=None, **kwargs):
        """Send an :class:`~panoramisk.actions.Action` to the server:

        :param action: an Action or dict with action name and parameters to
                       send
        :type action: Action or dict or Command
        :param as_list: If True, the action Future will retrieve all responses
        :type as_list: boolean
        :return: a Future that will receive the response
        :rtype: asyncio.Future

        :Example:

        To retrieve answer in a coroutine::

            manager = Manager()
            resp = yield from manager.send_action({'Action': 'Status'})

        With a callback::

            manager = Manager()
            future = manager.send_action({'Action': 'Status'})
            future.add_done_callback(handle_status_response)

        See https://wiki.asterisk.org/wiki/display/AST/AMI+Actions for
        more information on actions
        """
        action.update(kwargs)
        return self.protocol.send(action, as_list=as_list)


Instruction:
Implement a Python 3 function for
sending a command to the server.

Response:
def send_command(self, command, as_list=False):
        """Send a :class:`~panoramisk.actions.Command` to the server::

            manager = Manager()
            resp = manager.send_command('http show status')

        Return a response :class:`~panoramisk.message.Message`.
        See https://wiki.asterisk.org/wiki/display/AST/ManagerAction_Command
        """
        action = actions.Action({'Command': command, 'Action': 'Command'},
                                as_list=as_list)
        return self.send_action(action)


Instruction:
Can you write a function in Python 3 where it
sends a command to the AGI server.

Response:
def send_agi_command(self, channel, command, as_list=False):
        """Send a :class:`~panoramisk.actions.Command` to the server:

        :param channel: Channel name where to launch command.
               Ex: 'SIP/000000-00000a53'
        :type channel: String
        :param command: command to launch. Ex: 'GET VARIABLE async_agi_server'
        :type command: String
        :param as_list: If True, the action Future will retrieve all responses
        :type as_list: boolean
        :return: a Future that will receive the response
        :rtype: asyncio.Future

        :Example:

        ::

            manager = Manager()
            resp = manager.send_agi_command('SIP/000000-00000a53',
                                            'GET VARIABLE async_agi_server')

        Return a response :class:`~panoramisk.message.Message`.
        See https://wiki.asterisk.org/wiki/display/AST/Asterisk+11+ManagerAction_AGI
        """
        action = actions.Command({'Action': 'AGI',
                                  'Channel': channel,
                                  'Command': command},
                                 as_list=as_list)
        return self.send_action(action)


Instruction:
Here you have a function in Python 3, explain what it does
def connect(self):
        if self.loop is None:  # pragma: no cover
            self.loop = asyncio.get_event_loop()
        t = asyncio.Task(
            self.loop.create_connection(
                self.config['protocol_factory'],
                self.config['host'], self.config['port'],
                ssl=self.config['ssl']),
            loop=self.loop)
        t.add_done_callback(self.connection_made)
        return t

Response: connect to the server


Instruction:
Can you tell what is the following Python 3 function doing
def register_event(self, pattern, callback=None):
        def _register_event(callback):
            if not self.callbacks[pattern]:
                self.patterns.append((pattern,
                                     re.compile(fnmatch.translate(pattern))))
            self.callbacks[pattern].append(callback)
            return callback
        if callback is not None:
            return _register_event(callback)
        else:
            return _register_event

Response: register an event pattern with a callback


Instruction:
Can you implement a function in Python 3 that
closes the connection to the database.

Response:
def close(self):
        """Close the connection"""
        if self.pinger:
            self.pinger.cancel()
            self.pinger = None
        if getattr(self, 'protocol', None):
            self.protocol.close()


Instruction:
Write a Python 3 function for
sending a command to FastAGI.

Response:
def send_command(self, command):
        """Send a command for FastAGI request:

        :param command: Command to launch on FastAGI request. Ex: 'EXEC StartMusicOnHolds'
        :type command: String

        :Example:

        ::

            @asyncio.coroutine
            def call_waiting(request):
                print(['AGI variables:', request.headers])
                yield from request.send_command('ANSWER')
                yield from request.send_command('EXEC StartMusicOnHold')
                yield from request.send_command('EXEC Wait 10')

        """
        command += '\n'
        self.writer.write(command.encode(self.encoding))
        yield from self.writer.drain()

        agi_result = yield from self._read_result()
        # If Asterisk returns `100 Trying...`, wait for next the response.
        while agi_result.get('status_code') == 100:
            agi_result = yield from self._read_result()

        # when we got AGIUsageError the following line contains some indication
        if 'error' in agi_result and agi_result['error'] == 'AGIUsageError':
            buff_usage_error = yield from self.reader.readline()
            agi_result['msg'] += buff_usage_error.decode(self.encoding)

        return agi_result


Instruction:
Here you have a function in Python 3, explain what it does
def _read_result(self):
        response = yield from self.reader.readline()
        return parse_agi_result(response.decode(self.encoding)[:-1])

Response: Parse the response from the AGI and parse it.


Instruction:
Make a summary of the following Python 3 code
def add_route(self, path, endpoint):
        assert callable(endpoint), endpoint
        if path in self._route:
            raise ValueError('A route already exists.')
        if not asyncio.iscoroutinefunction(endpoint):
            endpoint = asyncio.coroutine(endpoint)
        self._route[path] = endpoint

Response: Add a route for FastAGI requests.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def del_route(self, path):
        if path not in self._route:
            raise ValueError('This route doesn\'t exist.')
        del(self._route[path])

Response: Delete a route for FastAGI requests.


Instruction:
Make a summary of the following Python 3 code
def handler(self, reader, writer):
        buffer = b''
        while b'\n\n' not in buffer:
            buffer += yield from reader.read(self.buf_size)
        lines = buffer[:-2].decode(self.default_encoding).split('\n')
        headers = OrderedDict([
            line.split(': ', 1) for line in lines if ': ' in line
        ])

        agi_network_script = headers.get('agi_network_script')
        log.info('Received FastAGI request from %r for "%s" route',
                 writer.get_extra_info('peername'), agi_network_script)
        log.debug("Asterisk Headers: %r", headers)

        if agi_network_script is not None:
            route = self._route.get(agi_network_script)
            if route is not None:
                request = Request(app=self,
                                  headers=headers,
                                  reader=reader, writer=writer,
                                  encoding=self.default_encoding)
                try:
                    yield from route(request)
                except BaseException:
                    log.exception(
                        'An exception has been raised for the request "%s"',
                        agi_network_script
                    )
            else:
                log.error('No route for the request "%s"', agi_network_script)
        else:
            log.error('No agi_network_script header for the request')
        log.debug("Closing client socket")
        writer.close()

Response: A coroutine handler to handle a FastAGI request.


Instruction:
Write a Python 3 function that can
parse AGI results using Regular expression.

Response:
def parse_agi_result(line):
    """Parse AGI results using Regular expression.

    AGI Result examples::

        100 result=0 Trying...

        200 result=0

        200 result=-1

        200 result=132456

        200 result= (timeout)

        510 Invalid or unknown command

        520-Invalid command syntax. Proper usage follows:
        int() argument must be a string, a bytes-like object or a number, not
        'NoneType'

        HANGUP

    """
    # print("--------------\n", line)
    if line == 'HANGUP':
        return {'error': 'AGIResultHangup',
                'msg': 'User hungup during execution'}

    kwargs = dict(code=0, response="", line=line)
    m = re_code.search(line)
    try:
        kwargs.update(m.groupdict())
    except AttributeError:
        # None has no attribute groupdict
        pass
    return agi_code_check(**kwargs)


Instruction:
How would you implement a function in Python 3 that
checks the AGI code and return a dict to help on error handling.

Response:
def agi_code_check(code=None, response=None, line=None):
    """
    Check the AGI code and return a dict to help on error handling.
    """
    code = int(code)
    response = response or ""
    result = {'status_code': code, 'result': ('', ''), 'msg': ''}
    if code == 100:
        result['msg'] = line
    elif code == 200:
        for key, value, data in re_kv.findall(response):
            result[key] = (value, data)
            # If user hangs up... we get 'hangup' in the data
            if data == 'hangup':
                return {
                    'error': 'AGIResultHangup',
                    'msg': 'User hungup during execution'}
            elif key == 'result' and value == '-1':
                return {
                    'error': 'AGIAppError',
                    'msg': 'Error executing application, or hangup'}
    elif code == 510:
        result['error'] = 'AGIInvalidCommand'
    elif code == 520:
        # AGI Usage error
        result['error'] = 'AGIUsageError'
        result['msg'] = line
    else:
        # Unhandled code or undefined response
        result['error'] = 'AGIUnknownError'
        result['msg'] = line
    return result


Instruction:
How would you code a function in Python 3 to
reset all the related objects to the given uid.

Response:
def reset(cls, uid=None):
        """Mostly used for unit testing. Allow to use a static uuid and reset
        all counter"""
        for instance in cls.instances:
            if uid:
                instance.uid = uid
            instance.generator = instance.get_generator()


Instruction:
Given the following Python 3 function, write the documentation
def get_instances(self):
        return ["<%s prefix:%s (uid:%s)>" % (self.__class__.__name__,
                                             i.prefix, self.uid)
                for i in self.instances]

Response: Return a list of all the instances in this database.


Instruction:
Write a Python 3 function that can
return True if a response is Success or Follows

Response:
def success(self):
        """return True if a response status is Success or Follows:

        .. code-block:: python

            >>> resp = Message({'Response': 'Success'})
            >>> print(resp.success)
            True
            >>> resp['Response'] = 'Failed'
            >>> resp.success
            False
        """
        if 'event' in self:
            return True
        if self.response in self.success_responses:
            return True
        return False


Instruction:
Can you generate the documentation for the following Python 3 function
def getdict(self, key):
        values = self.get(key, None)
        if not isinstance(values, list):
            raise TypeError("{0} must be a list. got {1}".format(key, values))
        result = utils.CaseInsensitiveDict()
        for item in values:
            k, v = item.split('=', 1)
            result[k] = v
        return result

Response: Convert a multi values header to a case - insensitive dict


Instruction:
Here you have a function in Python 3, explain what it does
def run_setup(script_name, script_args=None, stop_after="run"):
    if stop_after not in ('init', 'config', 'commandline', 'run'):
        raise ValueError("invalid value for 'stop_after': %r" % stop_after)

    core._setup_stop_after = stop_after

    save_argv = sys.argv
    glocals = copy(globals())
    glocals['__file__'] = script_name
    glocals['__name__'] = "__main__"
    try:
        try:
            sys.argv[0] = script_name
            if script_args is not None:
                sys.argv[1:] = script_args
            f = open(script_name)
            try:
                exec(f.read(), glocals, glocals)
            finally:
                f.close()
        finally:
            sys.argv = save_argv
            core._setup_stop_after = None
    except Exception:
        logging.warn("Exception when running setup.", exc_info=True)

    if core._setup_distribution is None:
        raise RuntimeError(
            "'distutils.core.setup()' was never called -- "
            "perhaps '%s' is not a Distutils setup script?" %
            script_name)

    # I wonder if the setup script's namespace -- g and l -- would be of
    # any interest to callers?
    return core._setup_distribution

Response: Run a setup script in a somewhat controlled environment and return the Distribution instance that drives things.


Instruction:
Create a Python 3 function for
returning data from a package directory.

Response:
def get_data(path):
    """
    Returns data from a package directory.
    'path' should be an absolute path.
    """
    # Run the imported setup to get the metadata.
    with FakeContext(path):
        with SetupMonkey() as sm:
            try:
                distro = run_setup('setup.py', stop_after='config')

                metadata = {'_setuptools': sm.used_setuptools}

                for k, v in distro.metadata.__dict__.items():
                    if k[0] == '_' or not v:
                        continue
                    if all(not x for x in v):
                        continue
                    metadata[k] = v

                if sm.used_setuptools:
                    for extras in ['cmdclass', 'zip_safe', 'test_suite']:
                        v = getattr(distro, extras, None)
                        if v is not None and v not in ([], {}):
                            metadata[extras] = v

            except ImportError as e:
                # Either there is no setup py, or it's broken.
                logging.exception(e)
                metadata = {}

    return metadata


Instruction:
Here you have a function in Python 3, explain what it does
def get_primary_keys(model):
    mapper = model.__mapper__
    return [mapper.get_property_by_column(column) for column in mapper.primary_key]

Response: Get primary key properties for a SQLAlchemy model.


Instruction:
Here you have a function in Python 3, explain what it does
def _deserialize(self, value, *args, **kwargs):
        if not isinstance(value, dict):
            if len(self.related_keys) != 1:
                self.fail(
                    "invalid",
                    value=value,
                    keys=[prop.key for prop in self.related_keys],
                )
            value = {self.related_keys[0].key: value}
        if self.transient:
            return self.related_model(**value)
        try:
            result = self._get_existing_instance(
                self.session.query(self.related_model), value
            )
        except NoResultFound:
            # The related-object DNE in the DB, but we still want to deserialize it
            # ...perhaps we want to add it to the DB later
            return self.related_model(**value)
        return result

Response: Deserialize a serialized value to a model instance.


Instruction:
Can you generate the documentation for the following Python 3 function
def _get_existing_instance(self, query, value):
        if self.columns:
            result = query.filter_by(
                **{prop.key: value.get(prop.key) for prop in self.related_keys}
            ).one()
        else:
            # Use a faster path if the related key is the primary key.
            result = query.get([value.get(prop.key) for prop in self.related_keys])
            if result is None:
                raise NoResultFound
        return result

Response: Retrieve the related object from an existing instance in the DB.


Instruction:
Write a Python 3 function for
adding keyword arguments to kwargs based on the passed in SQLAlchemy column.

Response:
def _add_column_kwargs(self, kwargs, column):
        """Add keyword arguments to kwargs (in-place) based on the passed in
        `Column <sqlalchemy.schema.Column>`.
        """
        if column.nullable:
            kwargs["allow_none"] = True
        kwargs["required"] = not column.nullable and not _has_default(column)

        if hasattr(column.type, "enums"):
            kwargs["validate"].append(validate.OneOf(choices=column.type.enums))

        # Add a length validator if a max length is set on the column
        # Skip UUID columns
        # (see https://github.com/marshmallow-code/marshmallow-sqlalchemy/issues/54)
        if hasattr(column.type, "length"):
            try:
                python_type = column.type.python_type
            except (AttributeError, NotImplementedError):
                python_type = None
            if not python_type or not issubclass(python_type, uuid.UUID):
                kwargs["validate"].append(validate.Length(max=column.type.length))

        if hasattr(column.type, "scale"):
            kwargs["places"] = getattr(column.type, "scale", None)


Instruction:
Write a Python 3 script for
adding keyword arguments to kwargs based on the passed in relationship Property.

Response:
def _add_relationship_kwargs(self, kwargs, prop):
        """Add keyword arguments to kwargs (in-place) based on the passed in
        relationship `Property`.
        """
        nullable = True
        for pair in prop.local_remote_pairs:
            if not pair[0].nullable:
                if prop.uselist is True:
                    nullable = False
                break
        kwargs.update({"allow_none": nullable, "required": not nullable})


Instruction:
Can you generate the documentation for the following Python 3 function
def get_declared_fields(mcs, klass, cls_fields, inherited_fields, dict_cls):
        opts = klass.opts
        Converter = opts.model_converter
        converter = Converter(schema_cls=klass)
        declared_fields = super(SchemaMeta, mcs).get_declared_fields(
            klass, cls_fields, inherited_fields, dict_cls
        )
        fields = mcs.get_fields(converter, opts, declared_fields, dict_cls)
        fields.update(declared_fields)
        return fields

Response:
Updates declared fields with fields converted from the SQLAlchemy model
            passed as the model class Meta option.


Instruction:
Can you write a function in Python 3 where it
retrieves an existing record by primary key.

Response:
def get_instance(self, data):
        """Retrieve an existing record by primary key(s). If the schema instance
        is transient, return None.

        :param data: Serialized data to inform lookup.
        """
        if self.transient:
            return None
        props = get_primary_keys(self.opts.model)
        filters = {prop.key: data.get(prop.key) for prop in props}
        if None not in filters.values():
            return self.session.query(self.opts.model).filter_by(**filters).first()
        return None


Instruction:
Implement a function in Python 3 to
split serialized attrs to ensure association proxies are passed separately.

Response:
def _split_model_kwargs_association(self, data):
        """Split serialized attrs to ensure association proxies are passed separately.

        This is necessary for Python < 3.6.0, as the order in which kwargs are passed
        is non-deterministic, and associations must be parsed by sqlalchemy after their
        intermediate relationship, unless their `creator` has been set.

        Ignore invalid keys at this point - behaviour for unknowns should be
        handled elsewhere.

        :param data: serialized dictionary of attrs to split on association_proxy.
        """
        association_attrs = {
            key: value
            for key, value in iteritems(data)
            # association proxy
            if hasattr(getattr(self.opts.model, key, None), "remote_attr")
        }
        kwargs = {
            key: value
            for key, value in iteritems(data)
            if (hasattr(self.opts.model, key) and key not in association_attrs)
        }
        return kwargs, association_attrs


Instruction:
Create a Python 3 function for
deleting old stellar tables that are not used anymore

Response:
def gc():
    """Deletes old stellar tables that are not used anymore"""
    def after_delete(database):
        click.echo("Deleted table %s" % database)

    app = get_app()
    upgrade_from_old_version(app)
    app.delete_orphan_snapshots(after_delete)


Instruction:
Implement a Python 3 function for
taking a snapshot of the database

Response:
def snapshot(name):
    """Takes a snapshot of the database"""
    app = get_app()
    upgrade_from_old_version(app)
    name = name or app.default_snapshot_name

    if app.get_snapshot(name):
        click.echo("Snapshot with name %s already exists" % name)
        sys.exit(1)
    else:
        def before_copy(table_name):
            click.echo("Snapshotting database %s" % table_name)
        app.create_snapshot(name, before_copy=before_copy)


Instruction:
Create a Python 3 function for
returning a list of snapshots

Response:
def list():
    """Returns a list of snapshots"""
    snapshots = get_app().get_snapshots()

    click.echo('\n'.join(
        '%s: %s' % (
            s.snapshot_name,
            humanize.naturaltime(datetime.utcnow() - s.created_at)
        )
        for s in snapshots
    ))


Instruction:
Can you tell what is the following Python 3 function doing
def restore(name):
    app = get_app()

    if not name:
        snapshot = app.get_latest_snapshot()
        if not snapshot:
            click.echo(
                "Couldn't find any snapshots for project %s" %
                load_config()['project_name']
            )
            sys.exit(1)
    else:
        snapshot = app.get_snapshot(name)
        if not snapshot:
            click.echo(
                "Couldn't find snapshot with name %s.\n"
                "You can list snapshots with 'stellar list'" % name
            )
            sys.exit(1)

    # Check if slaves are ready
    if not snapshot.slaves_ready:
        if app.is_copy_process_running(snapshot):
            sys.stdout.write(
                'Waiting for background process(%s) to finish' %
                snapshot.worker_pid
            )
            sys.stdout.flush()
            while not snapshot.slaves_ready:
                sys.stdout.write('.')
                sys.stdout.flush()
                sleep(1)
                app.db.session.refresh(snapshot)
            click.echo('')
        else:
            click.echo('Background process missing, doing slow restore.')
            app.inline_slave_copy(snapshot)

    app.restore(snapshot)
    click.echo('Restore complete.')

Response: Restores the database from a snapshot


Instruction:
Can you implement a function in Python 3 that
initializes the current language.

Response:
def init():
    """Initializes Stellar configuration."""
    while True:
        url = click.prompt(
            "Please enter the url for your database.\n\n"
            "For example:\n"
            "PostgreSQL: postgresql://localhost:5432/\n"
            "MySQL: mysql+pymysql://root@localhost/"
        )
        if url.count('/') == 2 and not url.endswith('/'):
            url = url + '/'

        if (
            url.count('/') == 3 and
            url.endswith('/') and
            url.startswith('postgresql://')
        ):
            connection_url = url + 'template1'
        else:
            connection_url = url

        engine = create_engine(connection_url, echo=False)
        try:
            conn = engine.connect()
        except OperationalError as err:
            click.echo("Could not connect to database: %s" % url)
            click.echo("Error message: %s" % err.message)
            click.echo('')
        else:
            break

    if engine.dialect.name not in SUPPORTED_DIALECTS:
        click.echo("Your engine dialect %s is not supported." % (
            engine.dialect.name
        ))
        click.echo("Supported dialects: %s" % (
            ', '.join(SUPPORTED_DIALECTS)
        ))

    if url.count('/') == 3 and url.endswith('/'):
        while True:
            click.echo("You have the following databases: %s" % ', '.join([
                db for db in list_of_databases(conn)
                if not db.startswith('stellar_')
            ]))

            db_name = click.prompt(
                "Please enter the name of the database (eg. projectdb)"
            )
            if database_exists(conn, db_name):
                break
            else:
                click.echo("Could not find database %s" % db_name)
                click.echo('')
    else:
        db_name = url.rsplit('/', 1)[-1]
        url = url.rsplit('/', 1)[0] + '/'

    name = click.prompt(
        'Please enter your project name (used internally, eg. %s)' % db_name,
        default=db_name
    )

    raw_url = url

    if engine.dialect.name == 'postgresql':
        raw_url = raw_url + 'template1'

    with open('stellar.yaml', 'w') as project_file:
        project_file.write(
            """
project_name: '%(name)s'
tracked_databases: ['%(db_name)s']
url: '%(raw_url)s'
stellar_url: '%(url)sstellar_data'
            """.strip() %
            {
                'name': name,
                'raw_url': raw_url,
                'url': url,
                'db_name': db_name
            }
        )

    click.echo("Wrote stellar.yaml")
    click.echo('')
    if engine.dialect.name == 'mysql':
        click.echo("Warning: MySQL support is still in beta.")
    click.echo("Tip: You probably want to take a snapshot: stellar snapshot")


Instruction:
Write a Python 3 script for
updating indexes after each epoch for shuffling

Response:
def on_epoch_end(self) -> None:
        'Updates indexes after each epoch for shuffling'
        self.indexes = np.arange(self.nrows)
        if self.shuffle:
            np.random.shuffle(self.indexes)


Instruction:
Can you create a Python 3 function that
defines the default function for cleaning text. This function operates over a list.

Response:
def textacy_cleaner(text: str) -> str:
    """
    Defines the default function for cleaning text.

    This function operates over a list.
    """
    return preprocess_text(text,
                           fix_unicode=True,
                           lowercase=True,
                           transliterate=True,
                           no_urls=True,
                           no_emails=True,
                           no_phone_numbers=True,
                           no_numbers=True,
                           no_currency_symbols=True,
                           no_punct=True,
                           no_contractions=False,
                           no_accents=True)


Instruction:
Can you generate the documentation for the following Python 3 function
def apply_parallel(func: Callable,
                   data: List[Any],
                   cpu_cores: int = None) -> List[Any]:
    """
    Apply function to list of elements.

    Automatically determines the chunk size.
    """
    if not cpu_cores:
        cpu_cores = cpu_count()

    try:
        chunk_size = ceil(len(data) / cpu_cores)
        pool = Pool(cpu_cores)
        transformed_data = pool.map(func, chunked(data, chunk_size), chunksize=1)
    finally:
        pool.close()
        pool.join()
        return transformed_data

Response: Apply function to list of elements.


Instruction:
Write a Python 3 function that can
generate a function that will clean and tokenize text.

Response:
def process_text_constructor(cleaner: Callable,
                             tokenizer: Callable,
                             append_indicators: bool,
                             start_tok: str,
                             end_tok: str):
    """Generate a function that will clean and tokenize text."""
    def process_text(text):
        if append_indicators:
            return [[start_tok] + tokenizer(cleaner(doc)) + [end_tok] for doc in text]
        return [tokenizer(cleaner(doc)) for doc in text]

    return process_text


Instruction:
How would you code a function in Python 3 to
combine the cleaner and tokenizer.

Response:
def process_text(self, text: List[str]) -> List[List[str]]:
        """Combine the cleaner and tokenizer."""
        process_text = process_text_constructor(cleaner=self.cleaner,
                                                tokenizer=self.tokenizer,
                                                append_indicators=self.append_indicators,
                                                start_tok=self.start_tok,
                                                end_tok=self.end_tok)
        return process_text(text)


Instruction:
Can you generate the documentation for the following Python 3 function
def parallel_process_text(self, data: List[str]) -> List[List[str]]:
        process_text = process_text_constructor(cleaner=self.cleaner,
                                                tokenizer=self.tokenizer,
                                                append_indicators=self.append_indicators,
                                                start_tok=self.start_tok,
                                                end_tok=self.end_tok)
        n_cores = self.num_cores
        return flattenlist(apply_parallel(process_text, data, n_cores))

Response: Apply cleaner - > tokenizer.


Instruction:
Given the following Python 3 function, write the documentation
def generate_doc_length_stats(self):
        heuristic = self.heuristic_pct
        histdf = (pd.DataFrame([(a, b) for a, b in self.document_length_histogram.items()],
                               columns=['bin', 'doc_count'])
                  .sort_values(by='bin'))
        histdf['cumsum_pct'] = histdf.doc_count.cumsum() / histdf.doc_count.sum()

        self.document_length_stats = histdf
        self.doc_length_huerestic = histdf.query(f'cumsum_pct >= {heuristic}').bin.head(1).values[0]
        logging.warning(' '.join(["Setting maximum document length to",
                                  f'{self.doc_length_huerestic} based upon',
                                  f'heuristic of {heuristic} percentile.\n',
                                  'See full histogram by insepecting the',
                                  "`document_length_stats` attribute."]))
        self.padding_maxlen = self.doc_length_huerestic

Response: Analyze the document length statistics for padding strategy


Instruction:
Make a summary of the following Python 3 code
def fit(self,
            data: List[str],
            return_tokenized_data: bool = False) -> Union[None, List[List[str]]]:
        """
        TODO: update docs

        Apply cleaner and tokenzier to raw data and build vocabulary.

        Parameters
        ----------
        data : List[str]
            These are raw documents, which are a list of strings. ex:
            [["The quick brown fox"], ["jumps over the lazy dog"]]
        return_tokenized_data : bool
            Return the tokenized strings.  This is primarly used for debugging
            purposes.

        Returns
        -------
        None or List[List[str]]
            if return_tokenized_data=True then will return tokenized documents,
            otherwise will not return anything.
        """
        self.__clear_data()
        now = get_time()
        logging.warning(f'....tokenizing data')
        tokenized_data = self.parallel_process_text(data)

        if not self.padding_maxlen:
            # its not worth the overhead to parallelize document length counts
            length_counts = map(count_len, tokenized_data)
            self.document_length_histogram = Counter(length_counts)
            self.generate_doc_length_stats()

        # Learn corpus on single thread
        logging.warning(f'(1/2) done. {time_diff(now)} sec')
        logging.warning(f'....building corpus')
        now = get_time()
        self.indexer = custom_Indexer(num_words=self.keep_n)
        self.indexer.fit_on_tokenized_texts(tokenized_data)

        # Build Dictionary accounting For 0 padding, and reserve 1 for unknown and rare Words
        self.token2id = self.indexer.word_index
        self.id2token = {v: k for k, v in self.token2id.items()}
        self.n_tokens = max(self.indexer.word_index.values())

        # logging
        logging.warning(f'(2/2) done. {time_diff(now)} sec')
        logging.warning(f'Finished parsing {self.indexer.document_count:,} documents.')

        if return_tokenized_data:
            return tokenized_data

Response: Fit the corpus on a list of strings.


Instruction:
Can you generate a brief explanation for the following Python 3 code
def token_count_pandas(self):
        freq_df = pd.DataFrame.from_dict(self.indexer.word_counts, orient='index')
        freq_df.columns = ['count']
        return freq_df.sort_values('count', ascending=False)

Response: See token counts as pandas dataframe


Instruction:
Can you tell what is the following Python 3 function doing
def fit_transform(self,
                      data: List[str]) -> List[List[int]]:
        """
        Apply cleaner and tokenzier to raw data, build vocabulary and return
        transfomred dataset that is a List[List[int]].  This will use
        process-based-threading on all available cores.

        ex:
        >>> data = [["The quick brown fox"], ["jumps over the lazy dog"]]
        >>> pp = preprocess(maxlen=5, no_below=0)
        >>> pp.fit_transform(data)
        # 0 padding is applied
        [[0, 2, 3, 4, 5], [6, 7, 2, 8, 9]]

        Parameters
        ----------
        data : List[str]
            These are raw documents, which are a list of strings. ex:
            [["The quick brown fox"], ["jumps over the lazy dog"]]

        Returns
        -------
        numpy.array with shape (number of documents, max_len)

        """
        tokenized_data = self.fit(data, return_tokenized_data=True)

        logging.warning(f'...fit is finished, beginning transform')
        now = get_time()
        indexed_data = self.indexer.tokenized_texts_to_sequences(tokenized_data)
        logging.warning(f'...padding data')
        final_data = self.pad(indexed_data)
        logging.warning(f'done. {time_diff(now)} sec')
        return final_data

Response: Fit the tokenizer and tokenzier to raw data and build vocabulary and return the transfomred dataset that is a List [ int ].


Instruction:
Here you have a function in Python 3, explain what it does
def transform(self, data: List[str]) -> List[List[int]]:
        tokenized_data = self.process_text(data)
        indexed_data = self.indexer.tokenized_texts_to_sequences(tokenized_data)
        return self.pad(indexed_data)

Response: Transform a list of documents into a list of lists of lists of lists of integers.


Instruction:
Explain what the following Python 3 code does
def transform_parallel(self, data: List[str]) -> List[List[int]]:
        logging.warning(f'...tokenizing data')
        tokenized_data = self.parallel_process_text(data)
        logging.warning(f'...indexing data')
        indexed_data = self.indexer.tokenized_texts_to_sequences(tokenized_data)
        logging.warning(f'...padding data')
        return self.pad(indexed_data)

Response: Transform list of documents into list of sequence ids.


Instruction:
Explain what the following Python 3 code does
def pad(self, docs: List[List[int]]) -> List[List[int]]:
        # First apply indexing on all the rows then pad_sequnces (i found this
        # faster than trying to do these steps on each row
        return pad_sequences(docs,
                             maxlen=self.padding_maxlen,
                             dtype=self.padding_dtype,
                             padding=self.padding,
                             truncating=self.truncating,
                             value=self.padding_value)

Response: Vectorize and apply padding on a set of tokenized doucments that are not already in the cache.


Instruction:
Given the following Python 3 function, write the documentation
def tokenized_texts_to_sequences(self, tok_texts):
        res = []
        for vect in self.tokenized_texts_to_sequences_generator(tok_texts):
            res.append(vect)
        return res

Response: Transforms tokenized text to a sequence of integers.